ARC CE REST interface specification

Note

The current interface version is 1.0

Warning

This specification corresponds to production version under development! The technical preview version currently released does not comply the content of this document.

The REST API endpoint

The various functionalities of the service are accessible through HTTP(S) URL built upon following pattern:

<service endpoint URL>/rest/<version>/<functionality>

  • <service endpoint URL> represents mounting point of the service and may look like https://arc.example.org:443/arex.
  • <version> is two parts number separated by dot. Current version is 1.0.
  • <functionality> is one of keywords defined below.

Further the part <service endpoint URL>/rest/<version> is referred as <base>.

All parts of URL to the right of hostname are case-sensitive.

Depending on request Accept HTTP header, information in the response rendered in the different ways (see details for each functionality below):

  • GET and POST requests with Accept: application/json and Accept: application/xml return JSON and XML response rendering respectively
  • GET request with Accept: text/html header returns HTML with some useful information for web browser compatibility [1]
[1]TODO: clarify is it still the usefull case if we drop per-job URLs and only support POST. This functionality can be implemented with HTML that submit POST form but as it is API first of all, consider is it needed at all?

Description of functionalities and operations

Warning

TODO: Aleksandr - Add errors everywhere!

TODO: Aleksandr - Add real request/response examples everywhere after actual implementation!.

Supported versions

GET <service endpoint URL>/rest

Operations:

  • GET - returns list of supported REST API versions
  • POST, PUT, DELETE - not supported

Example response:

The XML response is like:

<versions>
  <version>1.0</version>
  <version>1.1</version>
  <version>1.2</version>
</versions>

Note

TODO: Aleksandr defines consistent JSON rendering based on actual implementation

The JSON is:

[ "1.0", "1.1", "1.2" ]

Obtaining CE resource information

GET <base URL>/info/[?schema=glue2]

Operations:

  • GET - retrieve generic information about cluster properties. It accepts the optional schema parameter with the following values: glue2, crr. Currently by the default the information is served as a GLUE2 document.

    XML or JSON returned according to request [2] headers.

  • HEAD - supported

  • PUT, POST, DELETE - not supported.

[2]For JSON response it is automatically converted into JSON with top node removed and attributes ignored.

Example QUERY:

GET https://host.domain.org:443/arex/rest/1.0/info/?schema=glue2 HTTP/1.1
Accept: application/xml

The XML response is:

<InfoRoot>
  <Domains xmlns="http://schemas.ogf.org/glue/2009/03/spec_2.0_r1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  xsi:schemaLocation="https://raw.github.com/OGF-GLUE/XSD/master/schema/GLUE2.xsd">
    <AdminDomain BaseType="Domain" CreationTime="2018-11-06T20:26:46Z" Validity="10800">
      <ID>urn:ad:UNDEFINEDVALUE</ID>
      <Name>UNDEFINEDVALUE</Name>
      <Distributed>false</Distributed>
      <Services>
        <ComputingService BaseType="Service" CreationTime="2018-11-06T20:26:46Z" Validity="10800">
          <ID>urn:ogf:ComputingService:arc.zero:arex</ID>
          <Capability>data.transfer.cepush.srm</Capability>
          <Capability>executionmanagement.jobmanager</Capability>
 ... output omitted ...

Operating jobs

GET <base URL>/jobs/[?state=<state1>[&state=<state2>[…]]]

POST <base URL>/jobs/?action=new

POST <base URL>/jobs/?action={info|status|kill|clean|restart}

Operations:

  • GET - get list of jobs
  • HEAD - supported
  • POST - job submission and management
  • PUT, DELETE - not supported

Get list of jobs

GET <base URL>/jobs/ retrieves list of jobs belonging to authenticated user as text/plain, application/xml or application/json. Returned document contains list of job IDs.

With text/html rendered HTML page is returned.

It accepts the optional state parameters. When defined the returned document contains only jobs in the requested state(s).

Note

TODO: Define states specification based on used and implemened ARC states. Simplisity is a design goal.

Example QUERY:

GET https://host.domain.org:443/arex/rest/1.0/jobs HTTP/1.1
Accept: application/xml

The XML response is:

<jobs>
  <job>
    <id>1234567890abcdef</id>
  </job>
  <job>
    <id>fedcba0987654321</id>
  </job>
</jobs>

Note

TODO: Aleksandr defines consistent JSON rendering based on actual implementation

The JSON is:

[
  "1234567890abcdef",
  "fedcba0987654321"
]

Job submission (create a new job)

POST <base URL>/jobs/?action=new initiates creation of a new job instance or multiple jobs.

Request body contains job description(s), in one of the supported formats: ADL as Content-type: application/xml or XRSL as Content-type: applicaton/rsl.

On successful submission response is 200 and response body contains array job ID and status elements according to requested format (XML/JSON).

To pass multiple job descriptions in document body of the same type:

[3]TODO: check whether | or + should be used to merge multiple xrsl. If | is correct, xRSL documentation should be fixed accordingly to implementation.

Note

TODO: request/response examples

Jobs management

POST <base URL>/jobs/?action={info|status|kill|clean|restart} - job management operations supporting arrays of jobs.

Request body contains list of jobids as text/plain or JSON/XML (e.g. output of GET <base URL>/jobs/ can be reused).

Response depends on the requested action:

Job info

POST <base URL>/jobs/?action=info retrieves full information about job(s) according to the GLUE2 activity information XML document, or in JSON format.

Response contains an array of job info documents.

Job status

POST <base URL>/jobs/?action=status retrieves information about job(s) current state.

Response contain and array of job ID and status elements according to requested format (XML/JSON).

Killing jobs

POST <base URL>/jobs/?action=kill send a request to kill job(s).

The response code is 202 to indicate request is queued for later execution. Response body is empty(?).

Clean job files

POST <base URL>/jobs/?action=clean send a request to clean job(s) files.

The response code is 202 to indicate request is queued for later execution. Response body is empty(?).

Restart job

POST <base URL>/jobs/?action=restart send a request to restart job(s).

The response code is 202 to indicate request is queued for later execution. Response body is empty(?).

Job delegations

POST <base URL>/jobs/?action=delegations - retrieves list of delegations associated with the job.

Response contains and array of job IDs with associated delegation IDs:

Note

TODO: Aleksandr defines consistent JSON rendering based on actual implementation

{
  "1234567890abcdef": [
      "1234567", "4567890"
  ],
  "fedcba0987654321": [
      "0987654"
  ]
}

File operations

Files belonging to specific job are operated using <base URL>/jobs/<job id> URL.

Working with session directory

GET <base URL>/jobs/<job id>/session/<path>

DELETE <base URL>/jobs/<job id>/session/<path>

PUT <base URL>/jobs/<job id>/session/<path>

Operations:

  • GET, HEAD, PUT, DELETE - supported for files stored in job’s session directory and perform usual actions.
  • GET, HEAD - for directories retrieves list of stored files (consider WebDAV for format)
  • DELETE - for directories removes whole directory
  • PUT - for directory not supported.
  • POST - not supported.

A-REX control directory files access for debugging

GET <base URL>/jobs/<job id>/debug/<file type>

Operations:

  • GET - return the content of file in A-REX control directory for requested jobID
  • HEAD - supported
  • POST, PUT, DELETE - not supported

The <file type> matches the controldir file suffix and can be one of the following:

  • failed
  • local
  • errors
  • description
  • diag
  • comment
  • status
  • acl
  • xml
  • input
  • output
  • input_status
  • output_status
  • statistics

Delegation functionality

GET <base URL>/delegations

POST <base URL>/delegations/?action=new

POST <base URL>/delegations/?action=get,renew,delete

Operations:

  • GET - retrieves list of delegations belonging to authenticated user
  • HEAD - supported
  • POST - manage delegations
  • PUT, DELETE - not supported

PUT <base URL>/delegations/<delegation id>

Operations:

  • GET, HEAD, POST - not supported
  • PUT - store delegation public part for particular delegation ID

Get list of delegations

GET <base URL>/delegations - retrieves list of delegations belonging to authenticated user

QUERY:

GET https://host.domain.org:443/arex/rest/1.0/delegations HTTP/1.1
Accept: application/xml

The XML response is:

<delegations>
  <delegation>
    <id>1234567890abcdef</id>
  </delegation>
  <delegation>
    <id>fedcba0987654321</id>
  </delegation>
</delegations>

The JSON formatted response (make consistent across specification):

[
  "1234567890abcdef",
  "fedcba0987654321"
]

New delegation

Delegation is a 2-step process. Step 1 generates pair of private/public keys on server side and communicates X.509 certificate request to the client. Client sings public key and stores delegated certificate to finish delegation procedure.

1 step
POST <base URL>/delegations/?action=new starts a new delegation process (1st step). Response is 201 and contains certificate request of application/x-pem-file type and URL of delegation in Location HTTP header with assigned delegation id.
2 step
PUT <base URL>/delegations/<delegation id> stores public part (2nd step). Request body contains signed certificate (Content-type: application/x-pem-file). Response is 200 on success.

Delegations management

Warning

TODO: Aleksandr - clarification is needed for the entire workflow. How job is associated with delegation ID?

Do we need bulk operations for delegation? For bulk jobs submission it seams the better approach is to share the same delegation.

In this sense following operations probably need to be replaced with per-ID URLs.

POST <base URL>/delegations/?action=get,renew,delete used to manage ARC CE delegations.

Request body contains list of delegation ids as text/plain or JSON/XML (e.g. output of GET <base URL>/delegations can be reused).

Response is structured depending on the action:

Get delegation(s)
POST <base URL>/delegations/?action=get returns public part of the stored delegations as application/x-pem-file. The Multipart content type can be used to return meny delegations.
Renew delegation(s)
POST <base URL>/delegations/?action=renew initiates renewal of delegation(s). Response is 200 with certificate request of application/x-pem-file type. The Multipart content type can be used to return meny certificate requests.
Delete delegation(s)
POST <base URL>/delegations/?action=delete removes delegation(s). Response is 200 with no body expected (or body represent a state of each delegation removal?).

Requirements followed during interface design

  1. Support for versioning: via URL paths like https://arc.zero:443/arex/rest/1.0/jobs
  2. Usable with simple tools (wget, curl)
  3. Accessible through Web Browser. Only basic functionality. Returns simple HTML page.
  4. Friendly to common HTTP REST frameworks
  5. Interactive access to session directory (with proper protection and WebDAV basic support). Should also work with arcls, arccp.
  6. Machine readable error/result codes/messages - so far HTTP codes seems to be enough
  7. Re-use of existing software modules (no drastic changes to information representation and jobs handling)
  8. Support for different response formats: html, xml, json - automatic conversion from internal XML is enough for first version

Plans for functionality extension post version 1.0

  1. More effective bulk operations: with HTTP v2, will require HTTP v2 development for HED, this feature is postponed till next versions
  2. Resource information functionality: consider filtering through URL options, consider supporting references (relative URLs) to underlying resources.
  3. Scalability for many jobs and delegations: consider paging and filtering through URL options
  4. Jobs: consider a way to provide list of all jobs per site or per VO to special monitoring agents
  5. Add hold action for jobs management once it will be implemented
  6. For sessiondir access add PATCH for files to modifies part of files. Body format need to be defined, all files treated as binary, currently support only non-standard PUT with ranges.