Quickstart ARC: towards distributed computing in a few minutes - x509 edition

Scared of distributed computing complexities?

With ARC7 you can setup a Computing Element and try common distributed computing workflows in just a few minutes!

ARC7 comes with so-called zero configuration included and works out of the box without any configuration at all.

You can try ARC by using the legacy x509 user certificate, or with the newer Jason Web Token capability. The procedure below splits into x509 versus token at Step 4. The two require slightly different configuration options on the ARC server, and different procedures to aquire the authentication document (certificate or token).

The ARC server can be set up to accept both user x509 certificates and user tokens in paralell, or just one of the two. This is up to you.

Note

The zero configured A-REX comes with the REST interface enabled. It runs on port 443, so make sure it is not firewalled if you want to submit jobs from a remote client host.

Step 1. Enable NorduGrid ARC7 repos

Prepare your system to install via the NorduGrid Repositories.

Note

Alpha and release-candidate packages are in testing repository, so please make sure it is enabled, e.g. on RHEL-based systems you can use dnf --enablerepo=nordugrid-testing to enable it for one transaction or dnf config-manager --enable nordugrid-testing to enable permanently.

If you want to test ARC7 including all latest developments, set up your repository to include the nightly builds following Using ARC packages from nightly builds instructions.

Step 2. Install A-REX

ARC Resource-coupled EXecution service (A-REX) is a core component that manages authentication, authorization and job life cycle. It is enough to have A-REX installed to have a minimal computing element:

[root ~]# dnf -y install nordugrid-arc-arex

Step 3. Run A-REX

To start ARC services just run:

[root ~]# arcctl service start --as-configured

You can check if A-REX is running with:

[root ~]# arcctl service list
arc-arex                         (Installed, Disabled, Running)
arc-arex-ws                      (Installed, Disabled, Running)
arc-datadelivery-service         (Not installed, Disabled, Stopped)
arc-infosys-ldap                 (Not installed, Disabled, Stopped)

Note

arcctl tool automates many ARC CE operations and is designed with bash-completion in mind. If you would like to use ARC in production it is advised to have completion enabled:
[root ~]# dnf install -y bash-completion python-argcomplete
[root ~]# activate-global-python-argcomplete

Step 4. Generate user x509 certificate and key for testing

Grid services and users authentication heavily relies on cryptography and uses certificates/keys for each entity. ARC7 comes with Test Certificate Authority on board that can issue the test user certificates easily.

The ARC7 zero configuration implements a default closed approach defining the special authorization object called authgroup.

During the test-user certificate generation, arcctl test-ca will automatically add the issued certificate subject to the testCA.allowed-subjects file, opening the job submission possiblity to the test-user transparently. the testCA.allowed-subjects can be found in your /etc/grid-security folder.

No other subject will be able to submit to your system before you change the authgroup settings in arc.conf.

You can test submission from the host running A-REX or from any other host in the network.

Testing from the host running A-REX

It is technically possible to submit jobs from the root account, however it is advised to use a dedicated regular user. Here we assume that you use a dedicated regular user. In the example below we use our regular user user01. You should replace this username with the username of your own regular user.

To generate test certificate/key and install it to standard location inside local user’s home directory run:

[root ~]# arcctl test-ca usercert --install-user user01
User certificate and key are installed to default /home/user01/.globus location for user user01.

Testing from any other host

In order to submit jobs from any other host (not the one running A-REX) you need to transfer the (test) user certificate and the CA-files to this other host.

On the A-REX host generate a user certificate/key:

[root ~]# arcctl test-ca usercert --export-tar
User certificate and key are exported to testcert-09160712.tar.gz.
To use it with arc* tools on the other machine, copy the tarball and run the following commands:
  tar xzf testcert-09160712.tar.gz
  source arc-test-certs/setenv.sh

Transfer the tarball to the client host and on the client host execute the commands suggested in the arcctl output:

[user ~]$ tar xzf /tmp/testcert-09160712.tar.gz
[user ~]$ source arc-test-certs/setenv.sh

Note

The zero configured A-REX comes with the REST interface enabled. It runs on port 443, so make sure it is not firewalled to be able to be used from another client host.

Step 5. Install the nordugrid-arc-client

Install ARC client tools on the client host:

[root ~]# dnf -y install nordugrid-arc-client

It is technically possible to submit jobs from the root account, however it is advised to use a dedicated regular user. Here we assume that you use a dedicated regular user.

Note

The zero configured A-REX comes with the REST interface enabled. It runs on port 443, so make sure it is not firewalled if you want to submit jobs from a remote client host.

You can start with the information query about your newly installed ARC computing element [1]:

[user ~]$ arcinfo -c https://arc.example.org/arex
Computing service:
  Information endpoint: https://arc.example.org:443/arex
  Submission endpoint: https://arc.example.org:443/arex (status: ok, interface: org.nordugrid.arcrest)
[1]Examples uses arc.example.org as a domain name for A-REX host

Step 6. Submit a job and check it is running

To submit a job, or perform any other action towards the ARC server you need a so-called proxy-certificate which is a Single Sign-On token for distributed grid-infrastructure. It is generated in the following way:

[user ~]$ arcproxy
Your identity: /DC=org/DC=nordugrid/DC=ARC/O=TestCA/CN=Test User 50350053
Proxy generation succeeded
Your proxy is valid until: 2023-06-03 01:10:38

A simple job can be submitted with the arctest tool:

[user ~]$ arctest -J 2 -C https://arc.example.org/arex
Submitting test-job 2:
Job submitted with jobid: https://arc.example.org:443/arex/rest/1.0/jobs/d16c6c2858ec

The job status can be checked with the arcstat tool:

[user ~]$ arcstat https://arc.example.org:443/arex/rest/1.0/jobs/d16c6c2858ec
Job: https://arc.example.org:443/arex/d16c6c2858ec
 Name: arctest2
 State: Running

Status of 1 jobs was queried, 1 jobs returned information

To fetch the job’s stdout run arccat tool:

[user ~]$ arccat https://arc.example.org:443/arex//arex/rest/1.0/jobs/d16c6c2858ec
HOSTNAME=arc.example.org
GRID_GLOBAL_JOBURL=https://arc.example.org:443/arex/d16c6c2858ec
MALLOC_ARENA_MAX=2
PWD=/var/spool/arc/sessiondir/d16c6c2858ec
SYSTEMD_EXEC_PID=374003
<output_omitted>

Step 7. Play more with the ARC Computing Element

As an admin you might frequently need to extract information from the logs and directories that ARC computing element uses. The brief list of the relevant paths can be obtained from:

[root ~]# arcctl config brief
ARC Storage Areas:
    Control directory:
        /var/spool/arc/jobstatus
    Session directories:
        /var/spool/arc/sessiondir
    Scratch directory on Worker Node:
        Not configured
    Additional user-defined RTE directories:
        Not configured
ARC Log Files:
    A-REX Service log:
        /var/log/arc/arex.log
    A-REX Jobs log:
        /var/log/arc/arex-jobs.log
    A-REX Helpers log:
        /var/log/arc/job.helper.errors
    A-REX WS Interface log:
        /var/log/arc/ws-interface.log
    Infosys Infoproviders log:
        /var/log/arc/infoprovider.log

To get information and manage jobs on A-REX server, the arcctl job is useful. Operations include but is not limited to:

  • Listing jobs:
[root ~]# arcctl job list
d1475fb1dc51
d16c6c2858ec
<output omitted>

[root ~]# arcctl job list --long
d1475fb1dc51      FINISHED   arctest2                                /DC=org/DC=nordugrid/DC=ARC/O=TestCA/CN=Test User 50350053
d16c6c2858ec      FINISHED   arctest2                                /DC=org/DC=nordugrid/DC=ARC/O=TestCA/CN=Test User 50350053
<output omitted>
  • Job general information:
[root ~]# arcctl job info d16c6c2858ec
Name         : arctest2
Owner        : /DC=org/DC=nordugrid/DC=ARC/O=TestCA/CN=Test User 50350053
State        : FINISHED
LRMS ID      : 9
Modified     : 2023-06-02 13:24:45
  • Job log:
[root ~]# arcctl job log d16c6c2858ec
2023-06-02T11:24:28Z Job state change UNDEFINED -> ACCEPTED   Reason: (Re)Accepting new job
2023-06-02T11:24:28Z Job state change ACCEPTED -> PREPARING   Reason: Starting job processing
2023-06-02T11:24:28Z Job state change PREPARING -> SUBMIT   Reason: Pre-staging finished, passing job to LRMS
----- exiting submit_fork job -----

2023-06-02T11:24:28Z Job state change SUBMIT -> INLRMS   Reason: Job is passed to LRMS
---------- Output of the job wrapper script -----------
Detecting resource accounting method available for the job.
Looking for /usr/bin/time tool for accounting measurements
GNU time found and will be used for job accounting.
------------------------- End of output -------------------------
2023-06-02T11:24:45Z Job state change INLRMS -> FINISHING   Reason: Job finished executing in LRMS
2023-06-02T11:24:45Z Job state change FINISHING -> FINISHED   Reason: Stage-out finished.
  • A-REX logs that mentions the job:
 [root ~]# arcctl job log d16c6c2858ec --service
 ### /var/log/arc/arex.log:
 [2023-06-02 13:24:28] [Arc] [INFO] [357383/3] d16c6c2858ec: State: ACCEPTED: parsing job description
 [2023-06-02 13:24:28] [Arc] [INFO] [357383/3] d16c6c2858ec: State: ACCEPTED: moving to PREPARING
 [2023-06-02 13:24:28] [Arc] [INFO] [357383/3] d16c6c2858ec: State: PREPARING from ACCEPTED
 [2023-06-02 13:24:28] [Arc] [INFO] [357383/3] d16c6c2858ec: State: SUBMIT from PREPARING
 [2023-06-02 13:24:28] [Arc] [INFO] [357383/3] d16c6c2858ec: state SUBMIT: starting child: /usr/share/arc/submit-SLURM-job
 [2023-06-02 13:24:28] [Arc] [INFO] [357383/3] d16c6c2858ec: state SUBMIT: child exited with code 0
 [2023-06-02 13:24:28] [Arc] [INFO] [357383/3] d16c6c2858ec: State: INLRMS from SUBMIT
 [2023-06-02 13:24:45] [Arc] [INFO] [357383/3] d16c6c2858ec: Job finished
 [2023-06-02 13:24:45] [Arc] [INFO] [357383/3] d16c6c2858ec: State: FINISHING from INLRMS
 [2023-06-02 13:24:45] [Arc] [INFO] [357383/3] d16c6c2858ec: State: FINISHED from FINISHING
### /var/log/arc/ws-interface.log:
  • Getting job attributes:
[root ~]# arcctl job attr d16c6c2858ec jobname
arctest2


Get production ready

Now you are ready to Install production ARC7 Computing Element!