ARCHERY as a community-defined RTEs registry

New in version 6.5.

Community-defined RTEs is a new concept that aimed to automate software environment provisioning for distributed computing infrastructures.

Community-defined RTEs are prepared and digitally singed by community. All RTEs are than indexed in the registry to be discoverable. The ARC CE admin can deploy such RTEs with a signle ARC Control Tool command after establishing a trust chain with community.

This document describes how to use ARCHERY as a Community-defined software environment registry.

1. Signing keys

The trust-chain between community and site-admin is based on the digital signatures. All Community-defined RTEs are supposed to be signed using OpenPGP standard for signatures. Technical implementation relies on the GNU Privacy Guard (GPG) software.

If you already have GPG keypair it can be used to sign community RTEs.

If not, generate a new keypair with:

[user ~]$ gpg --gen-key

2. Preparing RTEs

Community-defined RTE script is any script that follows RunTime Environments in ARC6 design for both naming scheme and the content.

In many production cases community RTEs require additional files that need to be deployed on ARC CE.

Such files can be listed within RTE script metadata (at the top of file), using the download keyword, for example:

# download: url:https://example.org/mySW.sif checksum:sha256:1c0176d901301be17f86df3330f121d0ae603652eced38e8dbae413f75fad670

This will instruct the arcctl to download the mySW.sif singularity image from https://example.org/, verify the file checksum and place it into the software deployment directory for this community.

During the job execution, these downloaded files are available to payload.

Path to the directory containing these files stored in the RUNTIME_JOB_SWDIR variable in the job context.

Note

If RTE requires complete software package bundle it is advised to have Singularity container or at least a single tarball to be deployed on the ARC CE. Than write a simple commands for RTE stage 1 to extract the files or define paths to the container image.

3. Signing and publishing RTEs

The ARCHERY Manage Tool completely automates the RTEs signing and publishing process.

Just do it

In case your RTEs are organized in the typical directory-based structure, all you need to do is to add following configuration into the JSON topology configuration file for ARCHERY:

{
  "software": {
     "rtes_dir": "/home/community/rtesroot"
  }
}

This configuration will instruct archery-manage to:

  • export public key from GPG database (default location) to be published
  • index all RTEs available in the specified directory
  • extract descriptions from RTE scripts if available (to be published)
  • sign all RTE scripts with GPG
  • embedds public key, RTEs index and signed RTEs content into the ARCHERY zone

Customize the process

GPG location and key

In case your GPG database is not in the standard location and/or you have several keypairs in the database use the following options to point archery-manage to the right place:

{
  "software": {
     "gpg_home": "/srv/cummunity-gpg",
     "gpg_keyid": "community@example.org",
     "rtes_dir": "/home/community/rtesroot"
  }
}

Manually define public key

In case you are not going to sign RTEs on the same machine you probably have to define the community public key manually.

One way to do it - provide URL that points to the key location:

{
  "software": {
     "pubkey_url": "https://example.org/gpg.public.key",
     "rtes_dir": "/home/community/rtesroot"
  }
}

Or you can put base64-encoded key value [1] directly into the configuration:

[1]Use gpg --export | base64 -w0 to get value
{
  "software": {
     "pubkey": "mQENBF4y4ssBCADN9317J1KiEZSkDX2T00sCjtRLcL46XcN2llxHPFJm43ziG2GVqFHdoLQEhoN4ozvfNKxndTFkNTh6aP2CdG9ml0mzsEuHc1SYqrB+uSSOrVPz3LF95tjVJgJTFJQmY/hLD5/M12Iq4ayYdBBwLJFdZDg/PSclX3kRFS4CDDvVtlXfQ1Jc/XLaVOk7nFDdbTbFEvfenMgcen/cxmlj1GI7IplNoiFoBjLP72NwbPW/WdxGkPzHO+9ypW1WSYHNTL9442zXypcOYyfRhxqae5D+pSIEmptu7279Hr9u7OlhYBRQW4uGI7CTiuqWxBYUKVK1s0UyjB79EJRtzN5c+fgZABEBAAG0L01vbER5bkdyaWQgVmlydHVhbCBMYWIgPHN1cHBvcnRAbW9sZHluZ3JpZC5vcmc+iQFUBBMBCAA+FiEEOkfw1OQG2FTtqq21j9bdV6pWp3UFAl4y4ssCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQj9bdV6pWp3Ve9AgAwnAiecFLWznKg8n9/VBE5jYENAA7uRe3hxeBqvlm0wmohBQ2QTDSPgwuZqp7M5kiK/FPf8q0QtyZGoYlCGUQc0dsBXyRjnHOsUiTKl0M9/PDc+u+Y2teR2XibdQ8ja8CedU09NT5Lbhx+MSv2azLSxLjB7zAVEzPCfxgcXX3G5zfNFsO4+5FoR1GsoGirEN9dkZAxGa71hMh6bEkqAqJrK+wpDPqGlB9mmwi2MsDnfd5E3N2rl5Z8rDDoXL5tdlk0kKN9iaycXqCEZU7gs1v+fSe8N/RBDV85aiE4RLeCZ+LQL5IND6ftezQuw3GfRzy8w5EycGCpfWoAr+D5WBhibkBDQReMuLLAQgAr9onAVIzORKjbCapUGVCiTsNpTNzKsevMiwxqVanyr3iZLpEgGXrP7yLjxvKY4ow2nZMr9AdqyPAA==",
     "rtes_dir": "/home/community/rtesroot"
  }
}

Host signed RTEs instead of embedding into DNS

The RTE scripts are typically small enoygh to be emedded directly into the ARCHERY DNS zone. But if you want to publish it on the web instead, just use signed_rtes_url option.

{
  "software": {
     "signed_rtes_url": "https://example.org/rtes/",
     "rtes_dir": "/home/community/rtesroot"
  }
}

In this case ARCHERY will store only the references to signed RTE scripts in https://example.org/rtes/ location.

Signed RTEs srcipts will be saved locally in the signed directory [2] and NEEDS TO BE manually uploaded to the web-hosting to be actually accessible in that location.

[2]can be also redefined with additional signed_rtes_dir option

Manually define RTEs

Instead of organizing RTEs in the directory-like structure, they can be explicitely described in the configuration one-by-one. This allows more flexibility, as you can point to already existing and signed RTEs published anywhere.

To manully describe community RTEs define rtes array in the configuration:

{
  "software": {
    "rtes": [
      {
        "name": "APPS/COMMUNITY/SW1-1.0.0",
        "description": "Community SW1",
        "url": "https://example.org/rtes/SW1-1.0.0.signed",
      },
      {
        "name": "APPS/COMMUNITY/SW2-1.7.0",
        "description": "Community SW2",
        "data": "owGbwMvMwMHYf+1u+Kqw5aWMaxmPJ3EEG+ka6xnoGcRZrbinrJCSWpxclFlQkpmfZ6Xgm5/jUpnnXpSZohBspBCcX15SnliUqqCRklmkyQVUm1+el5OfmKJQWpRjlVFSUlBspa+fm5+TUpmXDtSjl1+Url9UklqsX2ykV5yZppCckZqcXVyaa1WckWhkamZlmGxgaG6WYmlgaGxgmJRqaJ5mYZaSZmxsbJBmaGSYYpCYamZgbGZqlJqcmmJskWxOubdHvkVqmRzX9vn/FwA=",
      },
      {
        "name": "APPS/COMMUNITY/SW3-0.2.1",
        "path": "/home/community/rtes/APPS/COMMUNITY/SW3-0.2.1"
      }
    ]
  }
}

Array contains objects describing RTEs. Each object defines the name, optionally description and points to the content of community-defined RTE script, using one of the following options:

  • url - provide URL to signed RTE script
  • data - provide base64-encoded signed RTE script content
  • path - local path to plain (unsigned) RTE script location

To sign RTE script without archery-manage, you can use following command:

[console ~]# gpg --output rtescript.signed --sign rtescript.sh

It is possible to use both rtes array and rtes_dir simultaneously.