How to submit the job?

The heterogeneous nature of the distributed computing requires the formal job description in order to match resources and execution environment of the particular CE to job needs.

The client tools than, based on the job description, select submission endpoint and pass job to chosen CE for processing.

Submission sequence

Make sure you have client tools installed and credentials configured first.

Than follow the sequence below.

Generate proxy-certificate

To submit a job, or perform any other action towards the ARC server you need a so-called proxy-certificate which is a Single Sign-On token for distributed grid-infrastructure. It is generated using arcproxy tool.

Note

In most production cases you need to add VO affiliation information to proxy certificate with -S <VO> option.

[user ~]$ arcproxy -S myVO
Your identity: /DC=org/DC=nordugrid/DC=ARC/O=TestCA/CN=My Cert
Contacting VOMS server (named myVO): voms.myvo.example.org on port: 15001
Proxy generation succeeded
Your proxy is valid until: 2020-02-29 06:31:54

Create job description

ARC supports the Extended Resource Specification Language (xRSL) and EMI Activity Description Language (ADL) [1] for specifying job descriptions.

[1]

See section 9.3 of EMI-ES document for complete ADL specification

According to the syntax you should write xRSL or ADL file, describing your job files and resource requirements.

For example, the xRSL job description for arctest -J 2 job is:

&( executable = "/usr/bin/env" )( jobname = "arctest2" )( stdout = "stdout" )( join = "yes" )( gmlog = "gmlog" )

The same description in ADL:

<?xml version="1.0"?>
<ActivityDescription xmlns="http://www.eu-emi.eu/es/2010/12/adl" xmlns:emiestypes="http://www.eu-emi.eu/es/2010/12/types" xmlns:nordugrid-adl="http://www.nordugrid.org/es/2011/12/nordugrid-adl">
  <ActivityIdentification>
    <Name>arctest2</Name>
  </ActivityIdentification>
  <Application>
    <Executable>
      <Path>/usr/bin/env</Path>
    </Executable>
    <Output>stdout</Output>
    <Error>stdout</Error>
    <LoggingDirectory>gmlog</LoggingDirectory>
  </Application>
</ActivityDescription>

Run arcsub

The arcsub tool is responsible for job submission. This includes selecting (matchmaking) resources based on the job description and sending the job description and all job files to selected submission endpoint.

In the simplest case you can use a top-level Nordugrid ARCHERY registry to find CE in entire e-Inftrastrucutre that fits your job and allows you to use it resources:

[user ~]$ arcsub --registry nordugrid.org myjobdescription.xrsl
Job submitted with jobid: https://arc.example.org:443/arex/e932ad551df1

To specify the CEs you are aiming to submit jobs to more precisely, please read the next section for details.

Specify CEs for job submission

The arcsub accepts a set of options that allow to select target CE for the job submission.

There CEs can be passed manually (using -C, --computing-element=ce option) or fetched from the registries (-Y, --registry=registry).

The submission endpoint type (actual job submission interface) and information endpoint type (interface to query the information for matchmaking and brokering) can be defined as well, if you want to use some interface explicitely.

Following is a detailed description of how options works together to select submission endpoint, including the examples of usage.

Computing Element

The Computing Element (-C, --computing-element=CE) used to specify computing element name or a a complete endpoint URL directly.

Supported options:

• CE name (given as FQDN of the cluster frontend)

• URL of info or jobsubmission endpoint on the CE

The URL is one of the valid info or jobsubmission URLs of the computing element. The usage of URL is needed when the services are ruining on non-default ports or path.

Except the case when info-endpoint-type is NONE, the URL is interpreted as info-endpoint URL. In case of --info-endpoint-type=NONE, the URL represents the jobsubmission endpoint URL.

If only the --computing-element is specified without any preferred info-endpoint-type or submission-endpoint-type then arcsub will fetch resource info from all the available info endpoints parallel AND select from the discovered and supported (via available plugin) jobsubmission interfaces.

Examples:

[user ~]$ arcsub --computing-element my.cluster.org

will trigger a submission to a randomly selected jobsubmission-endpoint-type advertised via local infosys and supported by installed submission plugins.

[user ~]$ arcsub --computing-element ldap://my.cluster.org:389/o=glue

will trigger a submission to a available jobsubmission endpoint discovered from the info interface running on the LDAP port 389.

[user ~]$ arcsub --submission-endpoint-type arcrest --computing-element https://my.cluster.org/arex

will triger a submission to the ARC REST jobsubmission endpoint after the infosys discovery completed against the specified ARC REST GLUE2 info endpoint URL. The URL above is interpreted as info URL.

[user ~]$ arcsub --submission-endpoint-type arcrest --info-endpoint-type NONE --computing-element https://my.cluster.org/arex

will triger a submission to the ARC REST jobsubmission endpoint specified via the given URL. Info discovery is turned off. The URL is interpreted as jobsubmission endpoint URL.

Registry

The -Y, --registry=registry option used to specify service endpoint URL registries.

It takes FQDN’s of ARCHERY servers with optional specification of protocol.

Supported options:

• ARCHERY: arcsub --registry nordugrid.org or arcsub --registry dns://nordugrid.org

Note

Support for EGIIS, EMIR and BDII service catalogues are REMOVED and the corresponding plugins are not part of ARC7.

In case only a short hostname given with --registry without specifying the full protocol URL (ldap or dns), then arcsub will attempt to contact both ldap://hostname and dns://hostname.

Info endpoint type

Use the -Q, --info-endpoint-type=type for choosing the cluster/resource information retrieval interface type.

Supported options:

org.nordugrid.arcrest
org.nordugrid.ldap.nordugrid
org.nordugrid.ldap.glue2
org.nordugrid.internal
NONE

The special value NONE will disable the local infosys queries during job submission.

Note

Support for emies and gridftp information retrieval is REMOVED and the corresponding plugins deleted as of ARC7.

Note

The org.nordugrid prefix can be omitted, e.g. both org.nordugrid.arcrest and arcrest should be accepted.

Submission endpoint type

In ARC 7, only the arcrest submission endpoint type is available. If you are using an ARC 7 client to submit to an ARC 7 server it is therefore not necessary to specify submission endpoint. If you are submitting the job with an ARC 6 client to an ARC 7 server, you might want to specify the submission endpoint type:

Use -T, --submission-endpoint-type=type for choosing the jobsubmission endpoint type.

Supported options:

org.nordugrid.arcrest
org.nordugrid.internal

Note

org.nordugrid prefix can be omitted, e.g. both org.nordugrid.arcrest and arcrest should be accepted.

Note

support for EMI-ES and GridFTP submission interface types are REMOVED and the corresponding plugins deleted as of ARC7.