In this tutorial we will take an existing HPC cluster, install and configure ARC to become a production grid site, and test that it is working by submitting some test-jobs from a remote ARC client server.
This is the first tutorial with the not yet released ARC 7 and with using tokens. ARC still works as before with x509 user certificates, the differences will be pointed out.
There are many ways to set up ARC for production depending on the infrastructure you are installing ARC in front of. Sites will need different configuration options as a result. For a detailed overview of getting a production ready site the ARC installation and configuration guide is a good source of reference, in addition to the ARC configuration reference document.
If you are following this tutorial as part of the EGI 2023 conference, then a set of test-clusters have been prepared for you to use. They are already set up with the necessary prerequisites. You find the list of endpoints and can claim one, through this google spreadsheet.
In this tutorial we will set up a site with a set of recommended and commonly used configuration options.
To follow this tutorial you should have a functional cluster with the following required components:
For tutorial part 1 and 2¶
- A way to obtain a token
- Already installed and working batch system (in this tutorial we will focus on SLURM or HTCondor, but other supported batch systems are also ok)
- One server for ARC installation - this server could also host the batch system (be the SLURM master or HTCondor CM)
- The ARC server must be able to submit jobs to the batch system
- The ARC server must have an x509 host certificate
- Optionally a separate server for the SLURM master/HTCondor CM
- Minimally 1 compute node
- Shared filesystem between the ARC server and the compute nodes
For tutorial part 3¶
- Requirements from part 1 and 2
- Minimally 1 server for remote datadelivery service
- Part 0: Install ARC nightly build repo
- Part 1: Zero-conf test-setup using tokens (from Step 2)
- Part 2: Production ready setup with LOCAL ARC datadelivery
- Part 3: Production ready setup with REMOTE ARC datadelivery service
- Part 4: Brief mention of the powerful statecallout and RunTimeEvironment functionality that comes with ARC.
No tokens? Using x509 instead?¶
- Prequisite: An x509 user certificate or use the zero-conf test user certificate
- Part 1: Zero-conf test-setup using x509 (from Step 2)
All commands in an bash file¶
For the EGI test-sites set up for this tutorial, all commands that we have gone through in this tutorial are aggregated in the two scripts belows. You can download these on the ARC-CE and execute them as root.
Warning: this will only work for the prepared EGI test-servers, as configuration values for other sites will be different.