ARCHERY as a community-defined RTEs registry
New in version 6.5.
Community-defined RTEs is a new concept that aimed to automate software environment provisioning for distributed computing infrastructures.
Community-defined RTEs are prepared and digitally singed by community. All RTEs are than indexed in the registry to be discoverable. The ARC CE admin can deploy such RTEs with a signle ARC Control Tool command after establishing a trust chain with community.
This document describes how to use ARCHERY as a Community-defined software environment registry.
1. Signing keys
The trust-chain between community and site-admin is based on the digital signatures. All Community-defined RTEs are supposed to be signed using OpenPGP standard for signatures. Technical implementation relies on the GNU Privacy Guard (GPG) software.
If you already have GPG keypair it can be used to sign community RTEs.
If not, generate a new keypair with:
[user ~]$ gpg --gen-key
2. Preparing RTEs
Community-defined RTE script is any script that follows RunTime Environments in ARC6 design for both naming scheme and the content.
In many production cases community RTEs require additional files that need to be deployed on ARC CE.
Such files can be listed within RTE script metadata (at the top of file), using the download
keyword, for example:
# download: url:https://example.org/mySW.sif checksum:sha256:1c0176d901301be17f86df3330f121d0ae603652eced38e8dbae413f75fad670
This will instruct the arcctl
to download the mySW.sif
singularity image from https://example.org/
, verify the file checksum and place it into the software deployment directory for this community.
During the job execution, these downloaded files are available to payload.
Path to the directory containing these files stored in the RUNTIME_JOB_SWDIR
variable in the job context.
Note
If RTE requires complete software package bundle it is advised to have Singularity container or at least a single tarball to be deployed on the ARC CE. Than write a simple commands for RTE stage 1 to extract the files or define paths to the container image.
3. Signing and publishing RTEs
The ARCHERY Manage Tool completely automates the RTEs signing and publishing process.
Just do it
In case your RTEs are organized in the typical directory-based structure, all you need to do is to add following configuration into the JSON topology configuration file for ARCHERY:
{
"software": {
"rtes_dir": "/home/community/rtesroot"
}
}
This configuration will instruct archery-manage
to:
export public key from GPG database (default location) to be published
index all RTEs available in the specified directory
extract descriptions from RTE scripts if available (to be published)
sign all RTE scripts with GPG
embedds public key, RTEs index and signed RTEs content into the ARCHERY zone
Customize the process
GPG location and key
In case your GPG database is not in the standard location and/or you have several keypairs in the database use the following options to point archery-manage
to the right place:
{
"software": {
"gpg_home": "/srv/cummunity-gpg",
"gpg_keyid": "community@example.org",
"rtes_dir": "/home/community/rtesroot"
}
}
Manually define public key
In case you are not going to sign RTEs on the same machine you probably have to define the community public key manually.
One way to do it - provide URL that points to the key location:
{
"software": {
"pubkey_url": "https://example.org/gpg.public.key",
"rtes_dir": "/home/community/rtesroot"
}
}
Or you can put base64-encoded key value [1] directly into the configuration:
{
"software": {
"pubkey": "mQENBF4y4ssBCADN9317J1KiEZSkDX2T00sCjtRLcL46XcN2llxHPFJm43ziG2GVqFHdoLQEhoN4ozvfNKxndTFkNTh6aP2CdG9ml0mzsEuHc1SYqrB+uSSOrVPz3LF95tjVJgJTFJQmY/hLD5/M12Iq4ayYdBBwLJFdZDg/PSclX3kRFS4CDDvVtlXfQ1Jc/XLaVOk7nFDdbTbFEvfenMgcen/cxmlj1GI7IplNoiFoBjLP72NwbPW/WdxGkPzHO+9ypW1WSYHNTL9442zXypcOYyfRhxqae5D+pSIEmptu7279Hr9u7OlhYBRQW4uGI7CTiuqWxBYUKVK1s0UyjB79EJRtzN5c+fgZABEBAAG0L01vbER5bkdyaWQgVmlydHVhbCBMYWIgPHN1cHBvcnRAbW9sZHluZ3JpZC5vcmc+iQFUBBMBCAA+FiEEOkfw1OQG2FTtqq21j9bdV6pWp3UFAl4y4ssCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQj9bdV6pWp3Ve9AgAwnAiecFLWznKg8n9/VBE5jYENAA7uRe3hxeBqvlm0wmohBQ2QTDSPgwuZqp7M5kiK/FPf8q0QtyZGoYlCGUQc0dsBXyRjnHOsUiTKl0M9/PDc+u+Y2teR2XibdQ8ja8CedU09NT5Lbhx+MSv2azLSxLjB7zAVEzPCfxgcXX3G5zfNFsO4+5FoR1GsoGirEN9dkZAxGa71hMh6bEkqAqJrK+wpDPqGlB9mmwi2MsDnfd5E3N2rl5Z8rDDoXL5tdlk0kKN9iaycXqCEZU7gs1v+fSe8N/RBDV85aiE4RLeCZ+LQL5IND6ftezQuw3GfRzy8w5EycGCpfWoAr+D5WBhibkBDQReMuLLAQgAr9onAVIzORKjbCapUGVCiTsNpTNzKsevMiwxqVanyr3iZLpEgGXrP7yLjxvKY4ow2nZMr9AdqyPAA==",
"rtes_dir": "/home/community/rtesroot"
}
}
Host signed RTEs instead of embedding into DNS
The RTE scripts are typically small enoygh to be emedded directly into the ARCHERY DNS zone. But if you want to publish it on the web instead, just use signed_rtes_url
option.
{
"software": {
"signed_rtes_url": "https://example.org/rtes/",
"rtes_dir": "/home/community/rtesroot"
}
}
In this case ARCHERY will store only the references to signed RTE scripts in https://example.org/rtes/
location.
Signed RTEs srcipts will be saved locally in the signed
directory [2] and NEEDS TO BE manually uploaded to the web-hosting to be actually accessible in that location.
can be also redefined with additional signed_rtes_dir
option
Manually define RTEs
Instead of organizing RTEs in the directory-like structure, they can be explicitely described in the configuration one-by-one. This allows more flexibility, as you can point to already existing and signed RTEs published anywhere.
To manully describe community RTEs define rtes
array in the configuration:
{
"software": {
"rtes": [
{
"name": "APPS/COMMUNITY/SW1-1.0.0",
"description": "Community SW1",
"url": "https://example.org/rtes/SW1-1.0.0.signed",
},
{
"name": "APPS/COMMUNITY/SW2-1.7.0",
"description": "Community SW2",
"data": "owGbwMvMwMHYf+1u+Kqw5aWMaxmPJ3EEG+ka6xnoGcRZrbinrJCSWpxclFlQkpmfZ6Xgm5/jUpnnXpSZohBspBCcX15SnliUqqCRklmkyQVUm1+el5OfmKJQWpRjlVFSUlBspa+fm5+TUpmXDtSjl1+Url9UklqsX2ykV5yZppCckZqcXVyaa1WckWhkamZlmGxgaG6WYmlgaGxgmJRqaJ5mYZaSZmxsbJBmaGSYYpCYamZgbGZqlJqcmmJskWxOubdHvkVqmRzX9vn/FwA=",
},
{
"name": "APPS/COMMUNITY/SW3-0.2.1",
"path": "/home/community/rtes/APPS/COMMUNITY/SW3-0.2.1"
}
]
}
}
Array contains objects describing RTEs. Each object defines the name
, optionally description
and points to the content of community-defined RTE script, using one of the following options:
url
- provide URL to signed RTE script
data
- provide base64-encoded signed RTE script content
path
- local path to plain (unsigned) RTE script location
To sign RTE script without archery-manage
, you can use following command:
[console ~]# gpg --output rtescript.signed --sign rtescript.sh
It is possible to use both rtes
array and rtes_dir
simultaneously.