Job Description Language (xRSL)

In the distributed computing job are submitted to a large number of very different computing resources, which are often widely distributed geographically.

In order to abstract from the heterogeneous nature of these resources, a high-level job description language is necessary. This document introduces the xRSL job description language.

Note

The EMI Activity Description Language (ADL) is also supported by ARC (refer to section 9.3 of EMI-ES document for complete ADL specification)

The diversity of resources implies special requirements for a proper description of a job, introducing many new options as opposed to a conventional high-performance computing center use case.

The NorduGrid project introduced the extended Resource Specification Language (xRSL). xRSL adopts the general syntax of RSL language developed by Globus Alliance but extend and redesign some attributes and introduce the two levels of job option specifications:

User-side xRSL, i.e., the set of attributes specified by a user in a job-specific file. This file is interpreted by a Client and after the necessary modifications is passed to the ARC computing service – A-REX

Server-side xRSL, i.e., the set of attributes pre-processed by a client, and ready to be interpreted by the A-REX. Effectively, this is an internal job representation of ARC.

A user only has to know the user-side part, and utilize it to describe the grid jobs. A-REX, however, uses slightly different notations, supplied by the client tools; therefore developers of such tools must take care of converting user-submitted xRSL to the internal job description (Server-side xRSL).

In what follows, description of the xRSL is given, using the following notations:

`<xxxx>`	parameter to be substituted with a corresponding string or a number
`[xxxx]`	optional parameter
`xxx\|yyy\|zzz`	list of possible values of a parameter
`-"-`	same as above

xRSL syntax and rules

A job is described by means of xRSL attributes, which can be either passed via a command-line, or, more conveniently, be collected in a so-called xRSL-file (*.xrsl).

Such a file contains a plain list of attribute-value pairs and boolean operators & (for AND) and | (for OR). Attribute names are case-insensitive.

If the attributes are specified from the command line, the entire description must be enclosed either in single or in double quotes. Single quotes enclosure is completely analogous to xRSL-file usage, while double quotes imply standard shell expansion of the enclosed string. This has implications when strings contain local shell variables: they will not be expanded unless the task description is entered from the command line and is enclosed in double quotes.

Attribute value

An attribute-value pair is a key element of specification. It consists of an expression that assigns one or more values to an attribute, and is enclosed in round brackets:

(attribute="value")

For multi-valued attributes:

(attribute="value1" "value2")

List of values

Certain attributes do not have assigned value; instead, they contain a list of values that should be put in proper correspondence with each other:

(attribute=("value1" "value2")("value3" "value4"))

In the example above, value1 and value3 are put in correspondence to value2 and value4 respectively, according to the context of the attribute.

Quoting

Values should be enclosed in quotes if they contain blank spaces or special characters.

The special characters are:

+   &   |   (   )   =   <   >   !   "   '   ^   #   $

To quote a string containing special characters, you can use either single or double quotes. If your string, however, contains both such quotes, you can define any character as an own delimiter, by preceding it with the “carat” (^) character: attribute=^*My "good" value^* makes use of a carat-escaped asterisk as a delimiter.

Implicit conjunction

An xRSL job description starts with an ampersand (&) , to indicate implicit conjunction of all the attributes:

&(attribute1=value1)(attribute2="value 2")...

Whenever a disjunct-request of two or more attributes is needed, the following construction can be used:

(|(attribute="value1")(attribute="value2")...)

Note

Only few selected attributes (indicated further in the document) can be requested by the user multiple times, like in the disjunct request example above. Most attributes must be unique, i.e., appear only once in the job description document.

Operators

In expressions, the following relational operators are allowed, in general:

=   !=   >   <   >=   <=

However, most attributes can only be used with equality operator =. For few attributes (as indicated in the document), some other operators can be used in client-side job description as well.

Blank spaces

The xRSL attributes can be written in a single string, or split in lines arbitrary; blank spaces between and inside (attribute="value") relations are ignored.

Comments

Commented lines should start with (* and be closed with *):

(*attribute="value1"*)

Comments can not be nested.

Multible jobs in one description

Multiple job description in one file can be specified via multi-request operator +, which should precede multiple job descriptions:

+(&(...))(&(...))(&(...))

User-side xRSL attributes

The following attributes can be specified in a user’s xRSL script. Some have to be modified by the client tool before being passed to the A-REX.

Attribute names are case-insensitive, although assigned values may well be case-sensitive, if they represent file names, environment variables etc.

Note

It is possible to use unsupported attributes in job description. Standard ARC client submission commands (arcsub and arcresub) must be used with a command line option -U in order to accept unknown attributes. Without this command line option, the client tool will concider job description invalid if it contains unsupported attributes.

executable

Unique:	yes
Operators:	`=`
User xRSL:	`(executable=<string>)`
A-REX xRSL:	`-"-`
Example:	`(executable="local_to_job.exe")`

The executable to be submitted as a main task to a Local Resource Management System (LRMS).

string - file name (including path), local to the computing element (CE)

Executable is a file that has to be executed as the main process of the task. It could be either a pre-compiled binary, or a script. Users may transfer their own executables, or use the ones known to be already installed on the remote system (CE).

If an executable has to be transferred to the destination site (CE) from some source, it has to be specified in the inputFiles list. If it is not specified in inputFiles, the source is expected to be local to the user (client) and will be added as such to the inputFiles list by the ARC Client.

If the file name starts with a leading slash (/), it is considered to be the full path to the executable at the destination site (CE); otherwise the location of the file is relative to the session directory (where job input and files are stored).

If the xRSL string is entered from the command line and is enclosed in
double quotes, standard shell expansion of variables takes place. That
is, if the file name contains an environment variable ($...),
the value of this variable is resolved locally, but if the name itself
is also enclosed in double quotes, it will be resolved at the remote
computing element:
(executable=$ROOT_DIR/myprog.exe) – $ROOT_DIR is resolved
locally (will cause errors if the path does not exist at the
execution machine)
(executable="$ROOT_DIR/myprog.exe") – $ROOT_DIR will be
resolved remotely

To use executable at the destination site without specifying absolute path a trivial wrapper script can be used as the executable input file:

#!/bin/sh

# load local environment
module load openmpi/latest

# pass-through all the arguments
mpirun "$@"

Other option is to use dummy environment variable: (executable="${PROG_PATH}${PROG_PATH:+/}myprog.exe") – $PROG_PATH will be resolved remotely and possibly provided by relevant RTE or be an empty string when left undefined. In the latter case myprog.exe will be searched in the $PATH list on the destination site (CE).

arguments

Unique:	yes
Operators:	`=`
User xRSL:	`(arguments=<string> [string] ... )`
A-REX xRSL:	`(arguments=<executable> <string> [string] ... )`
Example:	`(arguments="10000" $(ATLAS)/input.dat)`

List of the arguments for the executable.

string - an argument

executable - the executable to be run by LRMS, taken by the ARC Client from the user-specified executable attribute

inputFiles

Unique:	yes
Operators:	`=`
User xRSL:	`(inputFiles=(<filename> <source> [option] ... ) ... )`
A-REX xRSL:	`(inputFiles=(<filename> <URL> [option] ... )` `(<filename> [size][.checksum]) ... )`
Example:	`(inputFiles=("local_to_job" "gsiftp://se1.lu.se/p1/remote.1" "threads=5")` `("local_to_job.dat" "/scratch/local_to_me.dat")` `("same_name_as_in_my_current_dir" ""))`

List of files to be copied to the computing element before job execution.

filename - destination file name, local to the computing element and always relative to the session directory

source - source of the file: (remote URLs, or a path, local to the submission node). If void ("", use the quotes!), the input file is taken from the submission directory.

option - URL options for source. The ARC Client converts source and any options given here to a URL with the syntax described here. Additionally delegationid=<id> can be specified here to assign X.509 delegation identifier to be used for corresponding URL.

URL - URL of the file

size - file size in bytes

checksum - file checksum (as returned by cksum)

If the inputFiles list does not contain the standard input file (as specified by stdin) and/or the executable file (as specified by executable), an ARC client must append these files to the list. If the <source> is a URL, any options given by option are added to it, then it is passed by the ARC Client to the A-REX as shown in the example above. A-REX recognizes all URLs except file:///.

Internally, the client must forward the (<filename> <source> [option] ...) request to the execution service without changes, unless <source> is a local path, void ("") or file:///. In case <source> is a local path, void ("") or file:///, the client must extract file size and checksum, and substitute the <source> string with [size][.checksum]. In the unlikely case when it is impossible to extract file size, the <source> string must be substituted by a void one ("").

Note

Please note that the inputFiles attribute is not meant to operate with directories, for reasons of access control and checksum verifications. You must specify a pair ("<local_to_job>" "<source>") for each file.

executables

Unique:	yes
Operators:	`=`
User xRSL:	`(executables=<string> [string] ...)`
A-REX xRSL:	`-"-`
Example:	`(executables="myscript.sh" "myjob.exe")`

List of files from the inputFiles set, which will be given executable permissions.

string - file name, local to the computing element and relative to the session directory

If the executable file (as specified in executable and if relative to the session directory) is not in the executables list, it will be added to the list by the ARC Client.

cache

Unique:	yes
Operators:	`=`
User xRSL:	`(cache="yes"\|"no")`
A-REX xRSL:	`-"-`
Example:	`(cache="yes")`

Specifies whether input files specified in the inputFiles should be placed by default in the cache or not. This affects all input files, even those described by executables.

If not specified, default value is yes.

Note

Cached files can not be modified by jobs by default. If your job has to modify input files, please use the (readonly=no) URL option for those files. This option does not affect whether or not the file is cached.

outputFiles

Unique:	yes
Operators:	`=`
User xRSL:	`(outputFiles=(<string> <URL> [option] ... ) ... )`
A-REX xRSL:	`(outputFiles=(<string> <URL>) ... )`
Example:	`(outputFiles=("local_to_job.dat" "gsiftp://se1.uio.no/stored.dat")` `("local_to_job_dir/" ""))`

List of files to be retrieved by the user or uploaded by the A-REX and optionally indexed (registered) in a data indexing service.

string - file name, local to the Computing Element (CE). If this string ends with a backslash / and <URL> is empty, the entire directory will be kept at the execution site. If however this string ends with a backslash / but the <URL> is a remote location, the contents of the directory are transferred to the destination.

URL - destination URL of the remote file; if void ("", use the quotes!), the file is kept for manual retrieval. Note that this can not be a local file:// URL.

option - URL options for destination URL. See URL Options for possible values. When the destination is an indexing service, a physical file location may be specified by the additional option location. This option can be given multiple times. The CE will attempt to upload the file to the specified locations in the order they are given until one succeeds. Options specified after a location option only affect that location. Another additional delegationid=<id> option can be specified here to assign X.509 delegation identifier to be used for corresponding URL.Before passing to the A-REX, the ARC Client adds to URL any options and locations given here, using the syntax described in URL Options.

If the list does not contain standard output, standard error file names and A-REX log-files directory name (as specified by stdout, stderr and gmlog), the ARC Client appends these items to the outputFiles list. If the <URL> is not specified (void, "", use the quotes!), files will be kept on the CE and should be downloaded by the user via the ARC Client. If specified name of file ends with /, the entire directory is kept.

A convenient way to keep the entire job directory at the remote site for a manual retrieval is to specify (outputfiles=(/ )).

In some cases, the list of output files may only be known after the job has completed. ARC allows a user to specify a list of output files dynamically in a file or files in the session directory as part of their job. The file(s) containing the output file information can be defined in the xRSL script as the path to the file relative to the session directory preceeded by ’@’. The format of these files is lines of 2 values separated by a space. The first value contains name of the output file relative to the session directory and the second value is a URL to which the file will be uploaded.

Example:

(outputFiles=("@output.files" "")

output.files is generated by the user and contains

file1 gsiftp://grid.domain.org/file1

file2 gsiftp://grid.domain.org/file2

After the job completes, the file output.files in the session directory will be read and any files described within will be uploaded to the given URLs.

cpuTime

Unique:	yes
Operators:	`=`
User xRSL:	`(cpuTime=<time>)`
A-REX xRSL:	`(cpuTime=<tttt>)`
Example:	`(cpuTime="240")`

Maximal CPU time request for the job. For a multi-processor job, this is a sum over all requested processors.

time - time (in minutes if no unit is specified)

tttt - time converted by the ARC Client from time to seconds.

The client converts time specified in the user-side XRSL file to seconds. If no time unit is specified, the client assumes the time given in minutes. Otherwise, a text format is accepted, i.e., any of the following will be interpreted properly (make sure to enclose such strings in quotes!):

week
days
days, 12 hours
hour, 30 minutes
hours
days
minutes

If both cpuTime and wallTime are specified, the ARC Client converts them both. cpuTime can not be specified together with gridTime or benchmarks.

Note

This attribute should be used to direct the job to a system with sufficient CPU resources, typically, a batch queue with the sufficient upper time limit. Jobs exceeding this maximum most likely will be terminated by remote systems! If time limits are not specified, the limit is not set and jobs can run as long as the system settings allow (note that in this case you can not avoid queues with too short time limits).

wallTime

Unique:	yes
Operators:	`=`
User xRSL:	`(wallTime=<time>)`
A-REX xRSL:	`(wallTime=<tttt>)`
Example:	`(wallTime="240")`

Maximal wall clock time request for the job.

time - time (in minutes if no unit is specified)

tttt - time converted by the ARC Client to seconds

The client converts time specified in the user-side XRSL file seconds. If no time unit is specified, the client assumes the time given in minutes. Otherwise, a text format is accepted, i.e., any of the following will be interpreted properly (make sure to enclose such strings in quotes!):

week
days
days, 12 hours
hour, 30 minutes
hours
days
minutes

If both cpuTime and wallTime are specified, the ARC Client converts them both. wallTime can not be specified together with gridTime or benchmarks. If only wallTime is specified, but not cpuTime, the corresponding cpuTime value is evaluated by the ARC Client and added to the job description.

Note

This attribute should be used to direct the job to a system with sufficient CPU resources, typically, a batch queue with the sufficient upper time limit. Jobs exceeding this maximum most likely will be terminated by remote systems! If time limits are not specified, the limit is not set and jobs can run as long as the system settings allow (note that in this case you can not avoid queues with too short time limits).

gridTime

Unique:	yes
Operators:	`=`
User xRSL:	`(gridTime=<time>)`
A-REX xRSL:	none
Example:	`(gridTime="2 h")`

Maximal CPU time request for the job scaled to the 2.8 GHz Intel Pentium 4 processor.

time - time (in minutes if no unit is specified)

The attribute is completely analogous to cpuTime, except that it will be recalculated to the actual CPU time request for each queue, depending on the published processor clock speed.

gridTime can not be specified together with cpuTime or wallTime. If only gridTime is specified, but not cpuTime, the corresponding cpuTime value is evaluated by the ARC Client and added to the job description.

benchmarks

Unique:	yes
Operators:	`=`
User xRSL:	`(benchmarks=(<string> <value> <time>) ... )`
A-REX xRSL:
Example:	`(benchmarks=("mybenchmark" "10" "1 hour, 30 minutes"))`

Evaluate a job’s cpuTime based on benchmark values.

string - benchmark name

value - benchmark value of reference machine

time - the cpuTime the job requires on the reference machine

benchmarks can not be specified together with cpuTime or wallTime. If only benchmarks is specified, but not cpuTime, the corresponding cpuTime value is evaluated by the ARC Client and added to the job description.

memory

Unique:	yes
Operators:	`=`
User xRSL:	`(memory=<integer>)`
A-REX xRSL:	`-"-`
Example:	`(memory>="500")`

Memory required for the job, per count for parallel jobs.

integer - size (Mbytes)

Note

Similarly to cpuTime, this attribute should be used to direct a job to a resource with a sufficient capacity. Jobs exceeding this memory limit will most likely be terminated by the remote system.

disk

Unique:	no
Operators:	`= != > < >= <=`
User xRSL:	`(disk=<integer>)`
A-REX xRSL:	none
Example:	`(disk="500")`

Disk space required for the job.

integer - disk space, Mbytes

Note

This attribute is used at the job submission time to find a system with sufficient disk space. However, it does not guarantee that this space will be available at the end of the job, as most known systems do not allow for disk space allocation. Eventually, a remote system can terminate a job that exceeds the requested disk space.

runTimeEnvironment

Required runtime environment.

string - environment name

The site to submit the job to will be chosen by the ARC Client among those advertising specified runtime environments. Before starting the job, the A-REX will set up environment variables and paths according to those requested. Runtime environment names are defined by Virtual Organizations, and tend to be organized in name spaces.

To request several environments, repeat the attribute string:

(runTimeEnvironment="ENV1")(runTimeEnvironment="ENV2") etc.

To make a disjunct-request, use a boolean expression:

(|(runTimeEnvironment="env1")(runTimeEnvironment="env2")).

You can use >= or <= operators: job will be submitted to any suitable site that satisfies such requirements, and among the available at the sites runtime environments, the highest version satisfying a requirement will be requested in the pre-processed xRSL script.

Runtime environment string interpretation is case-insensitive. If a runtime environment string consists of a name and a version number, a partial specification is possible: it is sufficient to request only the name and use > or >= operators to select the highest version.

The optional arguments can be supplied as additional strings after runtime environment string. On the server side they are passed to software responsible for preparing corresponding infrastructure.

middleware

Unique:	no
Operators:	`= != > < >= <=`
User xRSL:	`(middleware=<string>)`
A-REX xRSL:	only `=` is allowed
Example:	`(middleware="nordugrid-arc-0.5.99")`

Required middleware version. Make sure to specify full name and version number.

string

Grid middleware name.

The site to submit the job to will be chosen by the ARC Client among those advertising specified middleware. Usage is identical to that of the runTimeEnvironment. Use the >= operator to request a version “equal or higher”.

opsys

Unique:	no
Operators:	`= != > < >= <=`
User xRSL:	`(opsys=<string>)`
A-REX xRSL:	only `=` is allowed
Example:	`(opsys="FC3")`

Required operating system.

string

Operating system name and version.

The site to submit the job to will be chosen by the ARC Client among those advertising specified operating system. Usage is identical to that of runTimeEnvironment and middleware. Use the >= operator to request a version “equal or higher”.

stdin

Unique:	yes
Operators:	`=`
User xRSL:	`(stdin=<string>)`
A-REX xRSL:	`-"-`
Example:	`(stdin="myinput.dat")`

The standard input file.

string

file name, local to the computing element.

The standard input file should be listed in the inputFiles attribute; otherwise it will be forced to that list by the ARC Client.

stdout

Unique:	yes
Operators:	`=`
User xRSL:	`(stdout=<string>)`
A-REX xRSL:	`-"-`
Example:	`(stdout="myoutput.txt")`

The standard output file.

string

file name, local to the computing element and relative to the session directory.

The standard output file should be listed in the outputFiles attribute; otherwise it will be forced to that list by the ARC Client. If the standard output is not defined, ARC Client assigns a name.

stderr

Unique:	yes
Operators:	`=`
User xRSL:	`(stderr=<string>)`
A-REX xRSL:	`-"-`
Example:	`(stderr="myjob.err")`

The standard error file.

string

file name, local to the computing element and relative to the session directory.

The standard error file should be listed as an outputFiles attribute; otherwise it will be forced to that list by the ARC Client. If the standard error is not defined, ARC Client assigns a name. If join is specified with value yes, ARC Client adds stderr to the pre-processed xRSL script with the same value as stdout.

join

Unique:	yes
Operators:	`=`
User xRSL:	`(join="yes"\|"no")`
A-REX xRSL:	none
Example:	`(join="yes")`

If yes, joins stderr and stdout files into the stdout one. Default is no.

gmlog

Unique:	yes
Operators:	`=`
User xRSL:	`(gmlog=<string>)`
A-REX xRSL:	`-"-`
Example:	`(gmlog="myjob.log")`

A name of the directory containing grid-specific diagnostics per job.

string

a directory, local to the computing element and relative to the session directory

This directory is kept in the session directory to be available for retrieval (ARC Client forces it to the list if outputFiles)

jobName

Unique:	yes
Operators:	`=`
User xRSL:	`(jobName=<string>)`
A-REX xRSL:	`-"-`
Example:	`(jobName="My Job nr. 1")`

User-specified job name.

string

job name

This name is meant for convenience of the user. It can be used to select the job while using the ARC Client. It is also available through the Information System.

ftpThreads

Unique:	yes
Operators:	`=`
User xRSL:	`(ftpThreads=<integer>)`
A-REX xRSL:	`-"-`
Example:	`(ftpThreads="4")`

Defines how many parallel streams will be used by the A-REX during gsiftp and http(s|g)transfers of files.

integer

a number from 1 to 10

If not specified, parallelism is not used.

acl

Unique:	no
Operators:	`=`
User xRSL:	`(acl=<xml>)`
A-REX xRSL:	`-"-`
Example:	`(acl="<?xml version=""1.0""?>` `<gacl version=""0.0.1""><entry><any-user></any-user>` `<allow><write/><read/><list/><admin/></allow></entry></gacl>")`

Makes use of GACL rules to list users who are allowed to access and control job in addition to job’s owner. Access and control levels are specified per user. any-user tag refers to any user authorized at the execution cluster. To get more information about GACL please refer to http://www.gridsite.org.

xml

a GACL-compliant XML string defining access control list

Following job control levels can be specified via acl:

`write`	allows to modify contents of job data (job directory) and control job flow (cancel, clean, etc.)
`read`	allows to read content of job data (contents of job directory)
`list`	allows to list files available for the job (contents of job directory)
`admin`	allows to do everything – full equivalence to job ownership

queue

Unique:	yes
Operators:	`= !=`
User xRSL:	`(queue=<string>)`
A-REX xRSL:	only `=` is allowed
Example:	`(queue="pclong")`

The name of the remote batch queue.

Use only when you are sure that the queue by this name does exist.

string

known queue name

While users are not expected to specify queue in job descriptions, this attribute must be present in the Server-side xRSL. In fact, this is primarily an internal attribute, added to the job description by client tools after resource discovery and matchmaking. Still, users can specify this attribute to explicitly force job submission to a queue: when specified explicitly by the user, this value will not be overwritten by the ARC Client, and an attempt will be made to submit the job to the specified queue.

If for some reason (e.g. due to a client tool error) queue is absent from the Server-side xRSL, A-REX on the selected cluster will attempt to submit the job to the default queue if such is specified in the A-REX configuration.

startTime

Unique:	yes
Operators:	`=`
User xRSL:	`(startTime=<time>)`
A-REX xRSL:	`(startTime=<tttt>)`
Example:	`(startTime="2002-05-25 21:30:00")`

Time to start job processing by the Grid Manager, such as e.g. start downloading input files.

`time`	time string, YYYY-MM-DD hh:mm:ss
`tttt`	time string, YYYYMMDDhhmmss[Z] (converted by the ARC Client from `time`)

Actual job processing on a worker node starts depending on local scheduling mechanisms, but not sooner than startTime.

lifeTime

Unique:	yes
Operators:	`=`
User xRSL:	`(lifeTime=<time>)`
A-REX xRSL:	`(lifeTime=<tttt>)`
Example:	`(lifeTime="2 weeks")`

Maximal time to keep job files (the session directory) on the gatekeeper upon job completion.

`time`	time (in minutes if no unit is specified)
`tttt`	time (seconds, converted by the ARC Client from `time`)

Typical life time is 1 day (24 hours). Specified life time can not exceed local settings.

notify

Unique:	yes
Operators:	`=`
User xRSL:	`(notify=<string> [string] ... )`
A-REX xRSL:	`-"-`
Example:	`(notify="be your.name@your.domain.com")`

Request e-mail notifications on job status change.

string

string of the format: [b][q][f][e][c][d] user1@domain1 [user2@domain2] ...

Here flags indicating the job status are:

`b`	begin (PREPARING)
`q`	queued (INLRMS)
`f`	finalizing (FINISHING)
`e`	end (FINISHED)
`c`	cancellation (CANCELLED)
`d`	deleted (DELETED)

When no notification flags are specified, default value of eb will be used, i.e., notifications will be sent at the job’s beginning and at its end.

No more than 3 e-mail addresses per status change accepted.

rerun

Unique:	yes
Operators:	`=`
User xRSL:	`(rerun=<integer>)`
A-REX xRSL:	`-"-`
Example:	`(rerun="2")`

Number of reruns (if a system failure occurs).

integer

an integer number

If not specified, the default is 0. Default maximal allowed value is 5. The job may be rerun after failure in any state for which reruning has sense. To initiate rerun user has to use the arcresume command.

architecture

Unique:	no
Operators:	`= !=`
User xRSL:	`(architecture=<string>)`
A-REX xRSL:	none
Example:	`(architecture="i686")`

Request a specific architecture.

string

architecture (e.g., as produced by uname -a)

nodeAccess

Unique:	yes
Operators:	`=`
User xRSL:	`(nodeAccess="inbound"\|"outbound")`
A-REX xRSL:	none
Example:	`(nodeAccess="inbound")`

Request cluster nodes with inbound or outbound IP connectivity. If both are needed, a conjunct request should be specified.

dryRun

Unique:	yes
Operators:	`=`
User xRSL:	`(dryRun="yes"\|"no")`
A-REX xRSL:	`-"-`
Example:	`(dryRun="yes")`

If yes, do dry-run: job description is sent to the optimal destination, input files are transferred, but no actual job submission to LRMS is made. Typically used for xRSL and communication validation.

rsl_substitution

Unique:	no
Operators:	`=`
User xRSL:	`(rsl_substitution=(<string1> <string2>))`
A-REX xRSL:	`-"-`
Example:	`(rsl_substitution=("ATLAS" "/opt/atlas"))`

Substitutes <string2> with <string1> for internal RSL use.

`string1`	new internal RSL variable
`string2`	any string, e.g., existing combination of variables or a path

Use this attribute to define variables that simplify xRSL editing, e.g. when same path is used in several values, typically in inputFiles. Only one pair per substitution is allowed. To request several substitution, concatenate such requests. Bear in mind that substitution must be defined prior to actual use of a new variable string1.

After the substitution is defined, it should be used in a way similar to shell variables in scripts: enclosed in round brackets, preceded with a dollar sign, without quotes:

(inputfiles=("myfile" $(ATLAS)/data/somefile))

Unlike the environment attribute, rsl_substitution definition is only used by the client and is valid inside xRSL script. It can not be used to define environment or shell variable at the execution site.

environment

Unique:	no
Operators:	`=`
User xRSL:	`(environment=(<VAR> <string>) [(<VAR> <string>)] ... )`
A-REX xRSL:	`-"-`
Example:	`(environment=("ATLSRC" "/opt/atlas/src")` `("ALISRC" "/opt/alice/src"))`

Defines execution shell environment variables.

`VAR`	new variable name
`string`	any string, e.g., existing combination of variables or a path

Use this to define variables at an execution site. Unlike the rsl_substitution attribute, it can not be used to define variables on the client side.

count

Unique:	yes
Operators:	`=`
User xRSL:	`(count=<integer>)`
A-REX xRSL:	`-"-`
Example:	`(count="4")`

Specifies amount of sub-jobs to be submitted for parallel tasks.

countpernode

Unique:	yes
Operators:	`=`
User xRSL:	`(countpernode=<integer>)`
A-REX xRSL:	`-"-`
Example:	`(countpernode="2")`

Specifies amount of sub-jobs per node to be submitted for parallel tasks. Note: The count attribute must be specified when this attribute is specified.

exclusiveexecution

Unique:	yes
Operators:	`=`
User xRSL:	`(exclusiveexecution="yes"\|"no")`
A-REX xRSL:	`-"-`
Example:	`(exclusiveexecution="yes")`

Specifies whether the node should be allocated for exclusive use by the job.

Note

for the setting to be valid and take effect one must in addition specify count and countpernode.

jobreport

Unique:	yes
Operators:	`=`
User xRSL:	`(jobreport=<URL>)`
A-REX xRSL:	`-"-`
Example:	`(jobreport="https://grid.uio.no:8001/logger")`

Specifies an URL for an accounting service to send reports about job to. The default is set up in the cluster configuration.

URL

URL

It is up to users to make sure the requested accounting service accepts reports from the set of clusters they intend to use.

credentialserver

Unique:	yes
Operators:	`=`
User xRSL:	`(credentialserver=<URL>)`
A-REX xRSL:	`-"-`
Example:	`(credentialserver="myproxy://myproxy.nordugrid.org;username=user")`

Specifies an URL which Grid Manager may contact to renew/extend delegated proxy of job. Only MyProxy servers are supported.

URL

URL of MyProxy server

It is up to a user to make sure the specified MyProxy server will accept requests from Grid Manager to renew expired credentials. URL may contain options username and credname to specify user name and credentials name which Grid Manager should pass to MyProxy server. If username is not specified DN of user credentials is used instead.

priority

Unique:	yes
Operators:	`=`
User xRSL:	`(priority=<integer>)`
A-REX xRSL:	`-"-`
Example:	`(priority="80")`

Specifies priority given to this job during staging of input and output files when the new data staging framework is used by A-REX. Values are limited to between 1 (lowest priority) and 100 (highest priority). Default if this attribute is not specified is 50.

Server-side attributes

Note

It is strongly advised to aviod specifying following attributes in the client xRSL.

The following attributes are a part of the internal ARC job representation, and must be provided by ARC client tools and passed to the A-REX. Developers of new ARC client tools and utilities must make sure these attributes are added to the user job description before it is submitted to a A-REX.

sstdin

A-REX xRSL:	`(sstdin=<filename>)`
Example:	`(sstdin="myinput.dat")`

Internal attribute for the standard input. Can also be spelled stdinput. Only needed for GRAM compatibility, not used by ARC as such.

filename - standard input file name

action

A-REX xRSL:	`(action="request"\|"cancel"\|"clean"\|"renew"\|"restart")`
Example:	`(action="request")`

Action to be taken by the gatekeeper: submit the job, cancel job execution, clear the results of the job (also cancels the job), renew the proxy of the job, or restart the job from a previous failed state.

savestate

A-REX xRSL:	`(savestate="yes"\|"no")`
Example:	`(savestate="yes")`

If yes, input RSL is stored in a temporary file at the gatekeeper. Must be always set as yes in the current implementation. Only needed for GRAM compatibility, not used by ARC as such.

lrmstype

A-REX xRSL:	`(lrmstype=<string>)`
Example:	`(lrmstype="pbs")`

LRMS type, indicating which submission script is to be invoked.

string - LRMS type

hostName

A-REX xRSL:	`(hostname=<string>)`
Example:	`(hostName="grid.quark.lu.se")`

Name (e.g. as returned by the Linux hostname command) of the client machine from which the submission was made.

string - client host name, as passed by the ARC client

jobid

A-REX xRSL:	`(jobid=<string>)`
Example:	`(jobid="grid.quark.lu.se:2119/jobmanager-ng/157111017133827")`

Unique job identification string, needed for cancellation and clean-up.

string - global job ID

It can also be provided during submission of the job and should be unique to a computing element (cluster).

clientxrsl

A-REX xRSL:	`(clientxrsl=<string>)`
Example:	`(clientxrsl="&(executable=/bin/echo)(arguments=boo)")`

Job description xRSL string as submitted by the user, before being pre-processed by the client.

string - original xRSL description submitted by the user

This attribute is added by the User Interface during pre-processing, and is used for job re-submission in order to repeat brokering and matchmaking.

clientsoftware

A-REX xRSL:	`(clientsoftware=<string>)`
Example:	`(clientsoftware="nordugrid-arc-0.5.39")`

Version of ARC client used to submit the job.

string - version string

This attribute is added by the User Interface during pre-processing.

delegationid

A-REX xRSL:	`(delegationid=<id>)`
Example:	`(delegationid="a6f3e8fc920de023fbb432ea520")`

X.509 delegation identifier for all data staging operations and job itself.

id - delegation identifier obtained during delegation procedure (for applicable job submission interfaces)

This attribute is added by the User Interface during pre-processing.

Examples

User-side xRSL script

&
(* test run: if "yes", only submits RSL without actual job start *)
     (dryRun="no")
(* some local variables defined for further convenience *)
     (rsl_substitution=("TOPDIR" "/home/johndoe"))
     (rsl_substitution=("NGTEST" $(TOPDIR)/ngtest))
     (rsl_substitution=("BIGFILE" "/scratch/johndoe/100mb.tmp"))
(* some environment variables, to be used by the job *)
     (environment=("ATLAS" "/opt/atlas") ("CERN" "/cern"))
(* the main executable file to be staged in and submitted to the PBS *)
     (executable="checkall.sh")
(* the arguments for the executable above *)
     (arguments="pal")
(* files to be staged in before the execution *)
     (inputFiles = ("be_kaons" "")
     ("file1" gsiftp://grid.uio.no$(TOPDIR)/remfile.txt)
      ("bigfile.dat" $(BIGFILE) )    )
(* files to be given executable permissions after staging in *)
     (executables="be_kaons")
(* files to be staged out after the execution *)
     (outputFiles=
        ("file1" "gsiftp://grid.tsl.uu.se/tmp/file1.tmp")
        ("100mb.tmp" "rls://rls.nordugrid.org:39281/test/bigfile")
        ("be_kaons.hbook"  gsiftp://ce1.grid.org$(NGTEST)/kaons.hbook)   )
(* user-specified job name *)
     (jobName="NGtest")
(* standard input file *)
     (stdin="myinput.dat")
(* standard output file *)
     (stdout="myoutput.dat")
(* standard error file *)
     (stderr="myerror.dat")
(* A-REX logs directory name *)
     (gmlog="gmlog")
(* flag whether to merge stdout and stderr *)
     (join="no")
(* request e-mail notification on status change *)
     (notify="bqfe john.doe@gmail.com jane.doe@mail.org")
(* maximal CPU time required for the job, minutes for PBS*)
     (CpuTime="60")
(* maximal time for the session directory to exist on the remote node, days *)
     (lifeTime="7")
 (* memory required for the job, per count, Mbytes *)
     (Memory="200")
(* wall time to start job processing *)
     (startTime="2002-04-28 17:15:00")
(* disk space required for the job, Mbytes *)
     (Disk="500")
(* required architecture of the execution node *)
     (architecture="i686")
(* required run-time environment *)
     (runTimeEnvironment="APPS/HEP/Atlas-1.1")
(* number of re-runs, in case of a system failure *)
     (rerun="2")

Server-side xRSL script

Note that a client tool must do matchmaking and modify correspondingly the job document before submitting it to the matching resource. Specifically, a client tool has to:

expand all the rsl_substitution values

add double quotes to all strings

insert queue attribute in case such is missing

make sure every logical or comparison operator is expanded and replaced with a deterministic = statement

streamline blank spaces

Note

Comment lines are removed from the Server-side xRSL! Below they are shown to explain details.

&
(* saves RSL in a temporary file if "yes" *)
     ("savestate" = "yes" )
(* job submission to be performed if action is "request" *)
     ("action" = "request" )
(* submission host name *)
     ("hostname" = "ce1.grid.org" )
(* client software version *)
     ("clientsoftware" = "nordugrid-arc-0.6.0.3" )
(* walltime value added by the client, in seconds *)
     ("walltime" = "3600" )
(* test run: if "yes", only submits RSL without actual job start *)
     ("dryRun" = "no" )
(* some local variables defined for further convenience *)
     ("rsl_substitution" = ("TOPDIR" "/home/johndoe" ) )
     ("rsl_substitution" = ("NGTEST" "/home/johndoe/ngtest" ) )
     ("rsl_substitution" = ("BIGFILE" "/scratch/johndoe/100mb.tmp" ) )
(* some environment variables, to be used by the job *)
     ("environment" = ("ATLAS" "/opt/atlas" ) ("CERN" "/cern" ) )
(* executable *)
     ("executable" = "checkall.sh" )
(* arguments *)
     ("arguments" = "pal" )
(* files to be staged in before the execution *)
     ("inputfiles" = ("checkall.sh" "279320" )
        ("myinput.dat" "39806" )
        ("be_kaons" "8807" )
        ("file1" "gsiftp://grid.uio.no/home/johndoe/remfile.txt" )
        ("bigfile.dat" "104857600" )
     )
(* files to be given executable permissions after staging in *)
     ("executables" = "checkall.sh" "be_kaons" )
(* files to be staged out after the execution *)
     ("outputfiles" = ("file1" "gsiftp://grid.tsl.uu.se/tmp/file1.tmp" )
        ("100mb.tmp" "rls://rls.nordugrid.org:39281/test/bigfile" )
        ("be_kaons.hbook" "gsiftp://ce1.grid.org/home/johndoe/ngtest/kaons.hbook" )
        ("myoutput.dat" "" )
        ("myerror.dat" "" )
    )
(* user-specified job name *)
     ("jobName" = "NGtest" )
(* standard input file *)
     ("stdin" = "myinput.dat" )
(* standard output file *)
     ("stdout"="myoutput.dat" )
(* standard error file *)
     ("stderr" = "myerror.dat" )
(* flag whether to merge stdout and stderr *)
     ("join" = "no" )
(* request e-mail notification on status change *)
     ("notify" = "bqfe john.doe@gmail.com jane.doe@mail.org" )
(* specific queue to submit the job *)
     ("queue" = "atlas" )
(* CPU time required for the job, converted into seconds *)
     ("cputime" = "3600" )
(* maximal time for the session directory to exist on the remote node, seconds *)
     ("lifetime" = "604800" )
(* memory required for the job, per count, Mbytes *)
     ("memory" = "200" )
(* wall time to start job processing *)
     ("startTime" = "20020128171500" )
(* disk space required for the job, Mbytes *)
     ("disk" = "500" )
(* required architecture of the execution node *)
     ("architecture" = "i686" )
(* required run-time environment *)
     ("runtimeenvironment" = "APPS/HEP/Atlas-1.1" )
(* number of re-runs, in case of a system failure *)
     ("rerun" = "2" )
(* original client xRSL with expanded string substitutions; shortened here *)
     ("clientxrsl" = "&(""dryrun"" = ""no"" )(... )(""rerun"" = ""2"" )" )