Quickstart ARC: towards distributed computing in a few minutes - token edition
Scared of distributed computing complexities?
With ARC7 you can setup a Computing Element and try common distributed computing workflows in just a few minutes!
ARC7 comes with so-called zero configuration included and works out of the box without any configuration at all.
You can try ARC by using the legacy x509 user certificate, or with the newer Jason Web Token capability. The procedure below splits into x509 versus token at Step 4. The two require slightly different configuration options on the ARC server, and different procedures to aquire the authentication document (certificate or token).
The ARC server can be set up to accept both user x509 certificates and user tokens in paralell, or just one of the two. This is up to you.
Note
The zero configured A-REX comes with the REST interface enabled. It runs on port 443, so make sure it is not firewalled if you want to submit jobs from a remote client host.
Step 1. Enable NorduGrid ARC7 repos
Prepare your system to install via the NorduGrid Repositories.
Note
Alpha and release-candidate packages are in testing repository, so please make sure it is enabled, e.g. on RHEL-based systems you can use dnf --enablerepo=nordugrid-testing to enable it for one transaction or dnf config-manager --enable nordugrid-testing to enable permanently.
If you want to test ARC7 including all latest developments, set up your repository to include the nightly builds following Using ARC packages from nightly builds instructions.
Step 2. Install A-REX
ARC Resource-coupled EXecution service (A-REX) is a core component that manages authentication, authorization and job life cycle. It is enough to have A-REX installed to have a minimal computing element:
[root ~]# dnf -y install nordugrid-arc-arex
Step 3. Run A-REX
To start ARC services just run:
[root ~]# arcctl service start --as-configured
You can check if A-REX is running with:
[root ~]# arcctl service list
arc-arex (Installed, Disabled, Running)
arc-arex-ws (Installed, Disabled, Running)
arc-datadelivery-service (Not installed, Disabled, Stopped)
arc-infosys-ldap (Not installed, Disabled, Stopped)
Note
arcctltool automates many ARC CE operations and is designed with bash-completion in mind. If you would like to use ARC in production it is advised to have completion enabled:
[root ~]# dnf install -y bash-completion python-argcomplete
[root ~]# activate-global-python-argcomplete
Step 4. Set up a test jwt token issuer
For the zero-conf setup we will use ARC’s inbuilt test-token issuer to submit a token.
arcctl test-jwt init
Take note of the “Issuer URL” printed out when you domain
arcctl test-jwt info
Example of output where issuer is https://arc.example.org/arc/testjwt/6897a567:
Issuer URL: https://arc.example.org/arc/testjwt/6897a567
JWKS:
{
"keys": [
{
"e": "AQAB",
"kid": "testjwt",
"kty": "RSA",
"n": "hufyTzvUJI0mb2wwIkfJJ3Uha3q3rv2x-KpJIza_SVsFkzFAP7NT_r-YbNMhiNVbUTMe-AKH0JezQLd6bd4ITRu5JVNioj6KFMHCCwwlWAuFWnk9IdjP9COZht2KDxZzE4IQc7WrvP-tC_VJSnWztqXjmrxaR-8N51Kg4t7AqDSn_ntZPru3O6vg4lK7-6VrQcSpIeRwxduNm9czIfyToIalrzy186x3Ucuf2hktLKssHyEhbW2MzikqOMzTpPR5RWNccU4zWN6DyTFWCdDZTQSF30kuX0vmtW7ded7uaRn3cyO0zR59Zye1LR7r4hMUomLLZHykZPmwdxAB5lyq2w",
"use": "sig"
}
]
}
Step 5. Configure A-REX to handle token authentication
We will here use as an example the jwt issuer from Step 4, namely https://arc.example.org/arc/testjwt/6897a567
[authgroup:zero]
authtokens = * hhttps://arc.example.org/arc/testjwt/6897a567 * * *
When you set up your production ready service later on, you can be more specific with the audience, scope and group parameters, again see details in the [authgroup] authokens section.
Step 6. Enable publicinfo
Enable access to public information for non-authorized users through the REST interface by adding the [arex/ws/publicinfo] block. Place it just below the [arex/ws/jobs] block.
[arex/ws/publicinfo]
Step 7. Restart A-REX
Restart A-REX services to activate the configuration changes
[root ~]# arcctl service restart -a
Step 8. Install nordugrid-arc-client
You can test submission from the host running A-REX or from any other host in the network. In any case you must install the ARC client.
Install ARC client tools on the client host:
[root ~]# dnf -y install nordugrid-arc-client
It is technically possible to submit jobs from the root account, however it is advised to use a dedicated regular user. For the test installation we use the default special user nobody, but for production you will use dedicated regular users.
Warning
The zero configured A-REX comes with the REST interface enabled. It runs on port 443, so make sure it is not firewalled if you want to submit jobs from a remote client host.
You can start with the information query about your newly installed ARC computing element [1]:
[user ~]$ arcinfo -c https://arc.example.org/arex
Computing service:
Information endpoint: https://arc.example.org:443/arex
Submission endpoint: https://arc.example.org:443/arex (status: ok, interface: org.nordugrid.arcrest)
Step 9. Get a submission token
As the user to submit the job, get a submission token issued by the test-issuer. We assume here that you are running the client on the same machine as the ARC-CE server. Otherwise you must export the test-jwt issuer CA with ` arcctl test-jwt export` and follow instructions.
export BEARER_TOKEN=$(sudo arcctl test-jwt token)
Step 10. Submit a job and check that it is running
A simple job can be submitted with the arctest tool:
[user ~]$ arctest -J 2 -C https://arc.example.org/arex
Job submitted with jobid: https://arc.example.org:443/arex/rest/1.0/jobs/f77b3d1b1efb
The job status can be checked with the arcstat tool:
[user ~]$ arcstat https://arc.example.org:443/arex/rest/1.0/jobs/f77b3d1b1efb
Job: https://arc.example.org:443/arex/rest/1.0/jobs/f77b3d1b1efb
Name: arctest2
State: Running
Status of 1 jobs was queried, 1 jobs returned information
To fetch the job’s stdout run arccat tool:
[user ~]$ arccat https://arc.example.org:443/arex/rest/1.0/jobs/f77b3d1b1efb
HOSTNAME=arc.example.org
GRID_GLOBAL_JOBURL=https://arc.example.org:443/arex/f77b3d1b1efb
MALLOC_ARENA_MAX=2
PWD=/var/spool/arc/sessiondir/f77b3d1b1efb
SYSTEMD_EXEC_PID=374194
<output_omitted>
Step 11. Play more with the ARC Computing Element
As an admin you might frequently need to extract information from the logs and directories that ARC computing element uses. The brief list of the relevant paths can be obtained from:
[root ~]# arcctl config brief
ARC Storage Areas:
Control directory:
/var/spool/arc/jobstatus
Session directories:
/var/spool/arc/sessiondir
Scratch directory on Worker Node:
Not configured
Additional user-defined RTE directories:
Not configured
ARC Log Files:
A-REX Service log:
/var/log/arc/arex.log
A-REX Jobs log:
/var/log/arc/arex-jobs.log
A-REX Helpers log:
/var/log/arc/job.helper.errors
A-REX WS Interface log:
/var/log/arc/ws-interface.log
Infosys Infoproviders log:
/var/log/arc/infoprovider.log
To get information and manage jobs on A-REX server, the arcctl job is useful.
Operations include but is not limited to:
Listing jobs:
[root ~]# arcctl job list
f5ab040cdc51
f617259d58ec
<output omitted>
[root ~]# arcctl job list --long
f5ab040cdc51 FINISHED arctest2 https://wlcg.cloud.cnaf.infn.it//b9f1e5e2-a8f0-4332-bd9d-58bd63898cc6
f617259d58ec FINISHED arctest2 https://wlcg.cloud.cnaf.infn.it//b9f1e5e2-a8f0-4332-bd9d-58bd63898cc6
<output omitted>
Job general information:
[root ~]# arcctl job info f77b3d1b1efb
Name : arctest2
Owner : https://wlcg.cloud.cnaf.infn.it//b9f1e5e2-a8f0-4332-bd9d-58bd63898cc6
State : FINISHED
LRMS ID : 376176
Modified : 2023-06-02 16:07:05
Job log:
[root ~]# arcctl job log f77b3d1b1efb
2023-06-02T14:06:51Z Job state change UNDEFINED -> ACCEPTED Reason: (Re)Accepting new job
2023-06-02T14:06:51Z Job state change ACCEPTED -> PREPARING Reason: Starting job processing
2023-06-02T14:06:51Z Job state change PREPARING -> SUBMIT Reason: Pre-staging finished, passing job to LRMS
----- exiting submit_fork_job -----
2023-06-02T14:06:53Z Job state change SUBMIT -> INLRMS Reason: Job is passed to LRMS
---------- Output of the job wrapper script -----------
Detecting resource accounting method available for the job.
Looking for /usr/bin/time tool for accounting measurements
GNU time found and will be used for job accounting.
------------------------- End of output -------------------------
2023-06-02T14:07:05Z Job state change INLRMS -> FINISHING Reason: Job finished executing in LRMS
2023-06-02T14:07:05Z Job state change FINISHING -> FINISHED Reason: Stage-out finished.
A-REX logs that mentions the job:
[root ~]# arcctl job log f77b3d1b1efb --service
### /var/log/arc/arex.log:
[2023-06-02 16:06:51] [Arc] [INFO] [374270/3] f77b3d1b1efb: State: ACCEPTED: parsing job description
[2023-06-02 16:06:51] [Arc] [INFO] [374270/3] f77b3d1b1efb: State: ACCEPTED: moving to PREPARING
[2023-06-02 16:06:51] [Arc] [INFO] [374270/3] f77b3d1b1efb: State: PREPARING from ACCEPTED
[2023-06-02 16:06:51] [Arc] [INFO] [374270/3] f77b3d1b1efb: State: SUBMIT from PREPARING
[2023-06-02 16:06:51] [Arc] [INFO] [374270/3] f77b3d1b1efb: state SUBMIT: starting child: /usr/share/arc/submit-fork-job
[2023-06-02 16:06:53] [Arc] [INFO] [374270/3] f77b3d1b1efb: state SUBMIT: child exited with code 0
[2023-06-02 16:06:53] [Arc] [INFO] [374270/3] f77b3d1b1efb: State: INLRMS from SUBMIT
[2023-06-02 16:07:05] [Arc] [INFO] [374270/3] f77b3d1b1efb: Job finished
[2023-06-02 16:07:05] [Arc] [INFO] [374270/3] f77b3d1b1efb: State: FINISHING from INLRMS
[2023-06-02 16:07:05] [Arc] [INFO] [374270/3] f77b3d1b1efb: State: FINISHED from FINISHING
### /var/log/arc/ws-interface.log:
Getting job attributes:
[root ~]# arcctl job attr f77b3d1b1efb jobname
arctest2
Get production ready
Now you are ready to Install production ARC7 Computing Element!