JWT-only Infrastructure for small community based on ARC

This document describes the example setup of the infrastructure for a small research community based on ARC support for OAuth2 JWT tokens.

Intro: JWT tokens, IdPs and ARC Test-JWT

JWT Tokens are signed by the identity provider (IdP). Production infrastructure usually relies on the dedicated IdP hosted and managed by community. One of the common options is the Keycloak.

In this example, we will use the test-jwt subsystem of the ARC Control Tool instead of the centrally managed IdP.

The Test JWT can be seen as a private IdP on the local client computer. Each client in the small community has its own private IdP.

In the x509 world it can be compared to the own CA for every client, trusted by servers. Every generated JWT token can be compared to a short-lived client certificate signed with such dedicated CA.

Prelude: Infrastructure actors

digraph { fontcolor = royalblue; fontsize=12; node [shape=Rectangle, fontcolor=black]; subgraph clusterClient { rankdir=TB; label="Client"; fontsize=12; arcsub [label="arcsub"]; testjwt [label="arcctl test-jwt"]; curl [label="curl"]; arcsub -> testjwt [color="blue", style="dashed", dir=back]; testjwt -> curl [color="blue", style="dashed"]; } subgraph clusterServer { rankdir=TB; label="Server"; fontsize=12; arex [label="ARC CE", shape=Rectangle]; webdav [label="Storage (Apache WebDAV)", shape=Rectangle, rank=3]; arex -> webdav [color="red"]; webdav -> arex [color="red"]; } arcsub -> arex [constraint=false, color="darkgreen"]; curl -> webdav [constraint=false, color="red"]; }

Client: user on the machine (laptop, VM, etc) with the ARC Client and ARC Control Tool installed. Each client has its own Test-JWT Issuer trusted by both the ARC CE and Storage.

ARC CE: tokens-only REST-only ARC 7 installation. Trusts the client(s) Test-JWTs.

Storage: the storage element providing WebDAV access to files with OAuth2 authentication. Trusts the client(s) Test-JWTs. This tutorial provides the example configuration for Apache.

For sake of simpliciy we will install Apache and ARC CE on the same host.

Both client and server host needs to enable the NorduGrid Repositories before moving forward.

Verse: Test-JWT IdP

Setting up the private IdP(s) with ARC Test-JWT is very straight-forward.

It is private to the user, so it should be run as a normal user account by design. All Test-JWT files (keys) resides in the user $HOME directory.

Install the client packages:

[root@token-clinet ~]# dnf install nordugrid-arc-client nordugrid-arc-arcctl

Init the Test-JWT Issuer [1]:

[andrii@token-client ~]$ arcctl test-jwt init
[2024-10-23 09:19:59,164] [ARCCTL.JWTIssuer] [INFO] [502] [Showing the JWKS for JWT Issuer https://token-client2/arc/testjwt/169d01c/1001]
{
  "keys": [
    {
      "e": "AQAB",
      "kid": "testjwt",
      "kty": "RSA",
      "n": "8cPfW5Mla0vPWT7m6Mbhx54lNid65BFYS02uwuXF....",
      "use": "sig"
    }
  ]
}
[2024-10-23 09:19:59,164] [ARCCTL.TestJWT] [INFO] [502] [Generating deployment command to be executed on ARC CE to trust the Test JWT issuer https://token-client2/arc/testjwt/169d01c/1001]
arcctl deploy jwt-issuer --deploy-conf test-jwt://H4sIAJ+jGGcC/6XSXY+iMBQG4P/C9RKgguLc8SGojGYQUIfNhiAUrEBBKKAY//vUdSd7PfGq503OedKc9sYUkIRxSELm7cagpmlhzbwxR0Kq5o3jSJlBzEY5gpgALqwjj...

Note

Notice that test-jwt init prints the command to run on ARC CE. You can output it again with arcctl test-jwt export.

For the Apache configuration the JWK key will be needed as an escaped string. You can automate typing backslashes with jq:

[andrii@token-client ~]$ arcctl test-jwt info | jq '.keys[] | tostring'
[2024-10-23 09:23:36,291] [ARCCTL.JWTIssuer] [INFO] [507] [Showing the JWKS for JWT Issuer https://token-client2/arc/testjwt/169d01c/1001]
"{\"e\":\"AQAB\",\"kid\":\"testjwt\",\"kty\":\"RSA\",\"n\":\"8cPfW5Mla0vPWT7m6Mbhx54lNid65BFYS02uwuXFazHCDNLe1f1N...\",\"use\":\"sig\"}"

Before using the client, the new short-lived (12 houts by defaults) JWT access token needs to be generated. The ARC client expect it in the BEARER_TOKEN variable, so the shortcut is:

[andrii@token-client ~]$ export BEARER_TOKEN=$(arcctl test-jwt token)

Note

HINT: The Claims in the JWT token are fully customizable. For example, to include the meaningfull username you can run arcctl test-jwt token -n "UiO aCT". Find more info in Test JWT Howto using ARC Control Tool.

Bridge: ARC CE

The ARC CE installation is as easy as:

[root@arc-jwt ~]# dnf -y install nordugrid-arc-arex

During the package installation the zero-conf (for testing only) is installed and the Test-CA hostkeys are generated and available at /etc/grid-security/testCA-hostkey.pem and /etc/grid-security/testCA-hostcert.pem.

To make the ARC CE trust the client’s private IdP, copy the arcctl deploy command printed out by Test-JWT on client host, run it on the ARC CE and then restart the services:

[root@arc-jwt ~]# arcctl deploy jwt-issuer --deploy-conf test-jwt://H4sIAJ+jGGcC/6XSXY+iMBQG4P/C9RKgguLc8SGojGYQUIfNhiAUrEBBKKAY....
[2024-10-23 09:20:09,364] [ARCCTL.ThirdParty.Deploy] [INFO] [41083] [ARC CE now trust JWT signatures of https://token-client2/arc/testjwt/169d01c/1001 issuer.]
[2024-10-23 09:20:09,364] [ARCCTL.JWTIssuer] [INFO] [41083] [Auth configuration for JWT issuer https://token-client2/arc/testjwt/169d01c/1001 has been written to /etc/arc.conf.d/10-testjwt-62641c8b.conf]
[2024-10-23 09:20:09,364] [ARCCTL.JWTIssuer] [WARNING] [41083] [ARC services restart is needed to apply configuration changes.]

[root@arc-jwt ~]# arcctl service restart -a

If there are multiple clients - run the deploy command for every client used in the infrastructure.

All users are mapped to nobody in zero-conf. For a production ARC CE this requires adjustments via editing the config (/etc/arc.conf.d/10-testjwt-62641c8b.conf in this example). See Configure authorization and mapping rules for examples of production authgroups and mapping.

Note

As we have Test-CA hostkeys, the client need to trust the ARC CE Test CA. To do it:

  1. Run arcctl test-ca info -o ca-cert on the ARC CE and copy the CA certificate

  2. Run arcctl deploy ca-cert on the client and paste the certificate

Note

The JWT-only case needs certificates only for hosts, like any regurlar web-service. As users are excluded, a small community can easily rely on e.g. LetsEncrypt to manage certificates in production. Use system CA bundle in this case.

Chorus: Apache + WebDAV + mod_oauth2 = Storage Element

For an Apache based WebDAV storage elemnt with OAuth2, first we need to install the necessary packages.

Unline the OIDC module, mod_oauth2 is not included in EPEL. Package installation is done from GitHub releases.

[root@arc-jwt ~]# dnf -y install httpd mod_ssl
[root@arc-jwt ~]# dnf -y install https://github.com/OpenIDC/mod_oauth2/releases/download/v4.0.0/mod_oauth2-4.0.0-1.el9.x86_64.rpm \
                                 https://github.com/OpenIDC/liboauth2/releases/download/v2.0.0/liboauth2-2.0.0-1.el9.x86_64.rpm \
                                 https://github.com/OpenIDC/liboauth2/releases/download/v2.0.0/liboauth2-apache-2.0.0-1.el9.x86_64.rpm

Start the Apache configuration with setting up TLS. For this example setup, we just link the ARC CE Test-CA generated hostcert/key to the location defined in default /etc/httpd/conf.d/ssl.conf:

[root@arc-jwt ~]# ln -s /etc/grid-security/testCA-hostkey.pem /etc/pki/tls/private/localhost.key
[root@arc-jwt ~]# ln -s /etc/grid-security/testCA-hostcert.pem /etc/pki/tls/certs/localhost.crt

As Apache is running on the same host as the ARC CE that also listens on 443, change the port to 8443 in /etc/httpd/conf.d/ssl.conf. It is advised to disable port 80 as well as a welcome page.

Once TLS is sorted out, WebDAV and OAuth2 configuration can be added like this:

[root@arc-jwt ~]# mkdir /srv/webdav
[root@arc-jwt ~]# echo "test" > /srv/webdav/test
[root@arc-jwt ~]# chown -R apache:apache /srv/webdav/

[root@arc-jwt ~]# cat <<EOF > /etc/httpd/conf.d/webdav.conf
Alias /webdav /srv/webdav

DavLockDB "/var/lib/httpd/DavLockDB"

<Directory /srv/webdav>
  DAV On
  AuthType oauth2
  OAuth2TokenVerify jwk "{\"e\":\"AQAB\",\"kid\":\"testjwt\",\"kty\":\"RSA\",\"n\":\"oy24GkRH3vs7DjGcp55wis2k-pET4...\",\"use\":\"sig\"}"
  OAuth2TokenVerify jwk "{\"e\":\"AQAB\",\"kid\":\"testjwt\",\"kty\":\"RSA\",\"n\":\"8cPfW5Mla0vPWT7m6Mbhx54lNid65...\",\"use\":\"sig\"}"
  <RequireAll>
    Require valid-user
  </RequireAll>
</Directory>
EOF

[root@arc-jwt ~]# systemctl restart httpd

The OAuth2TokenVerify contains the JWK key of the trusted Test-JWT Issuer. It can be specified multiple times, if there are multiple clients (multiple issuers).

You can verify that Apache WebDAV is working via curl:

[andrii@token-client ~]$ export BEARER_TOKEN=$(arcctl test-jwt token)
[andrii@token-client ~]$ curl -k -H "Authorization: bearer ${BEARER_TOKEN}" https://arc-jwt.local:8443/webdav/test
test

Coda: Submitting jobs with data staging

The example xRSL is inspired by the arctest job, but adding stage-in and stage-out to our WebDAV storage:

[andrii@token-client ~]$ cat << EOF > davtest.xrsl
&( executable = "/usr/bin/env" )
( jobname = "davtest" )
( stdout = "stdout" )
( join = "yes" )
( inputFiles = ("test" "https://arc-jwt.local:8443/webdav/test"))
( outputFiles = ("stdout" "https://arc-jwt.local:8443/webdav/test-output" "overwrite=yes"))
EOF

The ARC 7 client is delegating the JWT credentials for data-staging automatically, so the submission itself is just a generic arcsub call, nothing specific is needed.

[andrii@token-client ~]$ export BEARER_TOKEN=$(arcctl test-jwt token)

[andrii@token-client ~]$ arcsub -C https://arc-jwt.local/arex davtest.xrsl
Job submitted with jobid: https://arc-jwt.local:443/arex/rest/1.0/jobs/6ab04a5f9869

[andrii@token-client ~]$ arcstat https://arc-jwt.local:443/arex/rest/1.0/jobs/6ab04a5f9869
Job: https://arc-jwt.local:443/arex/rest/1.0/jobs/6ab04a5f9869
Name: davtest
State: Finished
Exit Code: 0

Once the job is finished, the ARC CE uploaded the output of env to WebDAV. We can get the file using curl:

[andrii@token-client ~]$ curl -k -H "Authorization: bearer ${BEARER_TOKEN}" https://arc-jwt.local:8443/webdav/test-output
HOSTNAME=arc-jwt.local
GRID_GLOBAL_JOBURL=https://arc-jwt.local:443/arex/6ab04a5f9869
...

Additionally, let’s do some server-side checks related to job data transfers.

Starting with simple storage location check:

[root@arc-jwt ~]# ls -l /srv/webdav/
total 12
-rw-r--r-- 1 apache apache 944 Oct 23 23:29 test-output
-rw-r--r-- 1 apache apache   5 Oct 23 23:23 test

Apache logs, showing the WebDav requests (note username from JWT token in the logs):

[root@arc-jwt ~]# cat /var/log/httpd/ssl_access_log
fe80::d937:cdd1:ce2d:806e - Test User 81681464 [23/Oct/2024:23:29:20 +0200] "PROPFIND https://arc-jwt.local:8443/webdav/test HTTP/1.1" 207 816
fe80::d937:cdd1:ce2d:806e - Test User 81681464 [23/Oct/2024:23:29:21 +0200] "GET https://arc-jwt.local:8443/webdav/test HTTP/1.1" 200 5
fe80::d937:cdd1:ce2d:806e - Test User 81681464 [23/Oct/2024:23:29:34 +0200] "DELETE https://arc-jwt.local:8443/webdav/test-output HTTP/1.1" 404 196
fe80::d937:cdd1:ce2d:806e - Test User 81681464 [23/Oct/2024:23:29:34 +0200] "PUT https://arc-jwt.local:8443/webdav/test-output HTTP/1.1" 201 232
fe80::d937:cdd1:ce2d:806e - Test User 81681464 [23/Oct/2024:23:29:34 +0200] "PROPFIND https://arc-jwt.local:8443/webdav/test-output HTTP/1.1" 207 825

ARC CE job logs, showing data transfers (redacted to show only relevant information):

[root@arc-jwt ~]# arcctl job log https://arc-jwt.local:443/arex/rest/1.0/jobs/6ab04a5f9869
2024-10-23T21:29:20Z Job state change UNDEFINED -> ACCEPTED   Reason: (Re)Accepting new job
2024-10-23T21:29:20Z Job state change ACCEPTED -> PREPARING   Reason: Starting job processing
[2024-10-23 23:29:20] [INFO] [41237/5] DTR 7c07...ce0d: Scheduler received new DTR 7c076250-e5d6-4d04-8d45-9fe866dcce0d with source: https://arc-jwt.local:8443/webdav/test, destination: file:/var/spool/arc/sessiondir/6ab04a5f9869/test, assigned to transfer share _default-download with priority 25
[2024-10-23 23:29:21] [INFO] [41237/4] DTR 7c07...ce0d: Transfer finished: 5 bytes transferred
[2024-10-23 23:29:21] [INFO] [41237/6] DTR 7c07...ce0d: 6ab04a5f9869: All downloads finished successfully
2024-10-23T21:29:21Z Job state change PREPARING -> SUBMIT   Reason: Pre-staging finished, passing job to LRMS
2024-10-23T21:29:22Z Job state change SUBMIT -> INLRMS   Reason: Job is passed to LRMS
2024-10-23T21:29:33Z Job state change INLRMS -> FINISHING   Reason: Job finished executing in LRMS
[2024-10-23 23:29:34] [INFO] [41237/5] DTR 93d8...b3bb: Scheduler received new DTR 93d85ed3-e121-4a97-9dd6-7a4f3d60b3bb with source: file:/var/spool/arc/sessiondir/6ab04a5f9869/stdout, destination: https://arc-jwt.local:8443/webdav/test-output, assigned to transfer share _default-upload with priority 25
[2024-10-23 23:29:34] [INFO] [41237/4] DTR 93d8...b3bb: Transfer finished: 946 bytes transferred : checksum adler32:49ad28a1
[2024-10-23 23:29:34] [INFO] [41237/6] DTR 93d8...b3bb: 6ab04a5f9869: All uploads finished successfully
2024-10-23T21:29:34Z Job state change FINISHING -> FINISHED   Reason: Stage-out finished.

Accounting data related to data transfers:

[root@arc-jwt ~]# arcctl accounting job transfers https://arc-jwt.local:443/arex/6ab04a5f9869
Data transfers (downloads) performed during A-REX stage-in:
  https://arc-jwt.local:8443/webdav/test:
    Size: 5
    Download timeframe: 2024-10-23 21:29:20 - 2024-10-23 21:29:21
Data transfers (uploads) performed during A-REX stage-out:
  https://arc-jwt.local:8443/webdav/test-output:
    Size: 946
    Upload timeframe: 2024-10-23 21:29:33 - 2024-10-23 21:29:34