Tutorial Part 3 - Production ready setup with remote ARC datadelivery service

For this part of the tutorial you need at least one remove datadelivery server set up. It makes sense that the session directory and the cache directory is hosted on this server and shared to the ARC-CE and the compute nodes.

Install ARC Datadelivery service

Step 1. Setup nightly repo

On the remote data delivery server, install the nordugrid nightly repo following the same procedure as for the ARC-CE, namely this one.

Step 2. Install ARC packages and CA’s

[root ~]# dnf install -y nordugrid-arc-datadelivery-service

Create the log-directory

mkdir /var/log/arc

Step 3. Configure ARC Datadelivery service

On the remote data delivery server you must create the following minimal arc.conf:

[common]
hostname = THIS_IP

[datadelivery-service]
allowed_ip = IP_OF_ARC_CE
transfer_dir = /grid/cache1
transfer_dir = /grid/session1

secure=no

In our case we have made it simple, we do not require a host certificate (secure=no) and we have not set any FQDN either for this remote datadelivery server, so we use the ip-adress as hostname. The allowed_ip is the ip-adress of the ARC-CE(s) that we will allow requests from. Make sure to add the actual IP’s for your servers.

Step 4. Install CA’s

To trust the host certificate of the ARC-CE install CA’s.

Step 4a. Install igtf bundle

[root ~]# arcctl deploy igtf-ca classic --installrepo igtf

Step 4b. Install test-CA certificates from ARC-CE

Note

This step is only relevant for the tutorial if you are using the test-host certificates on the ARC-CE. Otherwise the igtf bundle should be enough.

Since we are still running with the test host certificate issued by the test-CA on the ARC-CE we must also transfer these from the ARC-CE to the datadelivery server.

On the ARC-CE

On the A-REX host print out the Test-CA certificate:

[user@arc-ce]  arcctl test-ca info -o ca-cert
-----BEGIN CERTIFICATE-----
MIIFyTCCA7GgAwIBAgIUeLkSbksS9r3raPvkT2rR0ep06X8wDQYJKoZIhvcNAQEM
BQAwdDETMBEGCgmSJomT8ixkARkWA29yZzEZMBcGCgmSJomT8ixkARkWCW5vcmR1
<output omitted>
TJ9f0I8ktHACLvLvJE9SIDWs2zPo8o4cmvLBAtxe+jaijn22THtpLLUSXt1ozexS
ZHGFtsUBuIoNzXoRXxJwkGBA1ZpLBbOpjyp6PzNcTPYFG51+EHTUMPkbfyQ5
-----END CERTIFICATE-----

Copy this output to your clipboard

On the ARC datadelivery server

On the remote ARC datadelivery server paste in the output from the step below as shown in the example here:

[root@data-delivery ~]# arcctl deploy ca-cert
[2024-10-25 21:40:50,328] [ARCCTL.ThirdParty.Deploy] [INFO] [726706] [Reading CA Certificate PEM data from stdin]
-----BEGIN CERTIFICATE-----
MIIFyTCCA7GgAwIBAgIUeLkSbksS9r3raPvkT2rR0ep06X8wDQYJKoZIhvcNAQEM
BQAwdDETMBEGCgmSJomT8ixkARkWA29yZzEZMBcGCgmSJomT8ixkARkWCW5vcmR1
<output omitted>
TJ9f0I8ktHACLvLvJE9SIDWs2zPo8o4cmvLBAtxe+jaijn22THtpLLUSXt1ozexS
ZHGFtsUBuIoNzXoRXxJwkGBA1ZpLBbOpjyp6PzNcTPYFG51+EHTUMPkbfyQ5
-----END CERTIFICATE-----
[2024-10-25 21:40:54,173] [ARCCTL.ThirdParty.Deploy] [INFO] [726706] [CA Certificate for /DC=org/DC=nordugrid/DC=ARC/O=TestCA/CN=ARC TestCA fdb0a5e3 is deployed successfully to /etc/grid-security/certificates/ARCTestCAfdb0a5e3.pem]

This will create all necessary files in your x509_cert_dir and allow your remote client to trust the ARC-CE.

Step 5. Start the remote Datadelivery service

[root ~]# arcctl service start -a

Step 6. Configure remote datadelivery service 0n the ARC-CE

For the ARC-CE we must add the following option to the [arex/data-staging] block - using the FQDN is you have that, or the ip-adress if not. Since we have set secure=no we use http otherwise https is used.

deliveryservice=http://<ip-remote-datadelivery-server>:<configured-port>/datadeliveryservice

For instance:

deliveryservice=http://158.37.63.219:443/datadeliveryservice

Step 7. Restart the remote ARC-CE

[root ~]# arcctl restart service -a

That is it. Now you have a remote data delivery server set up, and the ARC-CE only takes care of the high level logic of the transfers, and not the actual transfers themselves.

For more details on the A-REX data transfer framework, see here, and for the remote data deliver service, see here.