JURA Accounting Technical Details

General accounting configuration and operations flows are described in Accounting with JURA. This section contains more technical details about implementation of each component of accounting subsystem.

Records processing and publishing

JURA Processing and Publishing records

Fig. 20 ARC CE accounting: records creation, processing and publishing

AREX Accounting Records (AAR job log files)

The A-REX Accounting Records (AAR) are job log files generated by A-REX. AAR is the only source of accounting information for JURA. AARs are written by A-REX based on job data available, including .diag files that backend scripts creates based on batch system data and/or GNU time utility measurements. These job log files reside under the <control_dir>/logs directory. The name of the AAR job log files consist of the ID of the job and a random string to avoid collision of multiple job log files for the same job: <jobid>.<random>.

The AAR job log file consists of name=value lines, where value is either a job-related resource consumption data or a static info like name, ID or proxy certificate.

A-REX generates at least two job log files for each job: one at the time of job submission, another one after the job finishes, and possibly others at various job events. Please note JURA makes use only one of the AREX generated files belonging to the same job: the one that corresponds to the FINISHED job event (such state is indicated by the status={completed|failed|aborted}).

JURA initial AAR processing

A-REX periodically runs jura that loop over available A-REX job log records in the <control_dir>/logs. JURA opens all the files and processes only those that corresponds to a FINISHED job state.

JURA converts AARs to per-job per-destnation extended AARs that contains the target information from arc.conf as well. One extended AAR is generated per accounting target. The extended AAR job log files named <jobid>.<random>_<random2> where first <random> is taken from original AAR.

The original AAR <jobid>.<random> file is deleted once per-destination extended AAR logs are created by JURA.


JURA as part of the initial processing deletes all files corresponding to non-finished job states.

JURA Publihsing loop

JURA publishing subsystem loop over extended AAR logs in the A-REX <control_dir>/logs directory. JURA generates records in the Usage Record (UR) format proposed by the Open Grid Forum (OGF) for SGAS or Compute Accounting Record (CAR) XML for APEL.

The extended AAR job log file is deleted once record is successfully submitted, thus preventing multiple insertion of same usage records. If submission to destination fails, the extended AAR log files are kept, so another attempt is made upon a subsequent run of JURA. This mechanism will be repeated until the expiration time passes at which point the next execution of JURA removes the file without processing.

Please note that the JURA publishing loop is backward compatible with ARC 5 implementation.

Reporting to SGAS

SGAS has a simple custom web service interface loosely based on WS-ResourceProperties. JURA uses the insertion method of this interface to report URs directly using ARC HTTP client implementation. The corresponding processed extended AAR job log files are deleted after receiving a non-fault response from the service.

To increase communication efficiency JURA sends URs in batches. SGAS accepts a batch of URs in a single request. The batch is an XML element called UsageRecords, containing elements representing URs.

The process of handling batches is the following: JURA does not send all usage records immediately after generation, but instead collects them in a batch until reaching the maximal number of records or until running out of job log files. The maximal number of URs in a batch can be set as a urbatchsize configuration parameter of SGAS target.

Reporting to APEL

APEL uses the SSM framework for communication.

JURA send records to APEL by means of invoking helper ssmsend process that uses SSM python librarires developed by APEL.

ARC ships minimal set of SSM libraries along with A-REX binary packages to allow SSM usage. If SSM binary packages from APEL are availble for your OS (e.g. EL6), you can install this packages and they will be used instead of those shipped with ARC automatically.

JURA prepares the messages to be sent by ssmsend and puts them info SSM Outgoing directory located in the /var/spool/arc/ssm/<destination hostname>/outgoing/00000000/. Generated messages are XML based CAR records with file name format <YYYYMMDDhhmmss>.

Reporting to APEL also works with sending records in batches. The default urbatchsize value is set to 1000 according to APEL recommendatations.

Accounting archive

JURA Archiving records

Fig. 21 ARC CE accounting: records archiving

TODO: jura-archive-manager

Republishing process

JURA Republishing records

Fig. 22 ARC CE accounting: records republishing

TODO: Describe republishing workflow in bit more details


The JURA executable runs with the same user privileges as the A-REX. The owner of a job log file is the local user mapped for the submitter entity of the corresponding job. Since these files contain confidential data, A-REX restricts access to them allowing only read access for the job owner, thus when JURA is executed by A-REX it is allowed to read and delete job log files.

All usage records are submitted using the X.509 credentials specified by the value of x509_ set of confiurartion options of arc.conf. No proxies are used for communcation with accouting services.

The only access restriction made by a SGAS service is matching the Distinguished Name of the client (in this context JURA) with a set of trusted DNs. When access is granted, policies are then applied by SGAS, allowing either publishing and/or querying rights. Clients with publishing right can insert any UR, regardless of content. By default, querying right only allows retrieving URs pertaining to jobs submitted by the querying entity.

Implementation and API

JURA as part of the ARC software stack is written in C++, and utilizes the functionality provided by the ARC libraries, including secure HTTPS communication provided by the ARC plugable TLS and HTTP modules.

The modular design is also present in the usage reporting part of the JURA code, making it possible to extend JURAs support of accounting services. To create a JURA module one should simply write a C++ class which inherits from the abstract Arc::Destination class, and it must extend the two methods:

  • static Arc::Destination* Arc::Destination::createDestination(Arc::JobLogFile&)
  • void Arc::Destination::report(Arc::JobLogFile&)

The static createDestination method should initialize a object of the specialized class, using the configuration options specified in the passed Arc::LogFile object, and the memory allocated by the method should be freed by the caller. Then the report method should carry out the transfer of the UR, represented by the JobLogfile object, to the accounting service.

JURA archive manager is written in Python and share the classes for managing ARC components with ARC Control Tool.


In the following list some issues which limits the functionality of JURA is described:

  • The current implementation of JURA and A-REX supports only one expiration time for all the reporting destinations. Even though the configuration enables the specification of different expiration values per reporting destination, it is not taken into account by the system, the last value is used as the common expiration time value.
  • It is not possible to use different credentials per destinations.
  • If you are updating from ARC5 with an old jura accounting archive already containing records, the conversion process to index the archive structure will be initiated and will cause serious system load until finished. To avoid old archive conversion, you can move records before update.
  • Some optional UR properties are not supported.
  • Memory can be reported incorrectly with buggy GNU “time” results.

Definition of the A-REX Accounting Record including JURA attribute mappings to SGAS and APEL

Table 4 Attributes used in current implementation
A-REX Accounting Record (AAR) SGAS OGF-UR APEL CAR Content description
ngjobid RecordIdentity is composed of ngjobid and hostname taken from the headnode. RecordIdentity is composed of ngjobid and hostname taken from the headnode. The global unique jobid assigned by AREX.
globalid JobIdentity.GlobalJobId JobIdentity.GlobalJobId The global unique jobid assigned by AREX. For gridftp inteface it contains full URL.
localid JobIdentity.LocalJobId JobIdentity.LocalJobId LRMS job ID
jobname JobName JobName User specified job name
headnode MachineName MachineName, SubmitHost, Site The A-REX job submission endpoint URL used for this job
lrms not used Infrastructure (used as a part of it) The LRMS behind A-REX
queue Queue Queue The name of the LRMS queue of the job
nodename Host Host WN name(s) as given by LRMS separated by :
clienthost SubmitHost (port removed) not used Client connection socket from the client to A-REX
usersn UserIdentity.GobalUserName UserIdentity.GobalUserName The global user identity, at the moment it is the SN from the certificate
localuser UserIdentity.LocalUserId UserIdentity.LocalUserId The mapped local userid
usercert UserIdentity.VO and child structures UserIdentity.Group and UserIdentity.GroupAttribute contains the full proxy cert chain
projectname ProjectName UserIdentity.GroupAttribute User-defined name of the project the job belongs to
status Status Status The terminal state of an A-REX job: aborted, failed, completed
exitcode not used ExitStatus The exit code of the payload in the LRMS
submissiontime StartTime StartTime The timestamp of job acceptance at A-REX
endtime EndTime EndTime The timestamp when the job reached the terminal state in A-REX
nodecount NodeCount NodeCount Number of allocated worker nodes
inputfile FileTransfers not used Details of downloaded inputfile: url, size, transfer start, transfer end, downloaded from cache
outputfile FileTransfers not used Details of uploaded outputtfile: url, size, transfer start, transfer end
usedmemory Memory Memory Maximum virtual memory used by the job
usedmaxresident Memory Memory Maximum resident memory used by the job
usedaverageresident Memory Memory To be dropped from the AAR schema
usedwalltime WallDuration WallDuration The measured clocktime ellapsed during the execution of the job in the LRMS. No matter on how many cores, processors, nodes the user job ran on.
usedcputime CpuDuration CpuDuration (with type all) The total CPU time consumed by the job. If the job ran on many cores/processors/nodes, all separate consumptions shall be aggregated in this value.
usedusercputime CpuDuration (with type user but should not be there) CpuDuration (with type user) The user part of the usedcputime
usedkernelcputime CpuDuration (with type system but should not be there) CpuDuration (with type system) The kernel part of the usedcputime
Table 5 Proposed NEW attributes
A-REX Accounting Record (AAR) SGAS OGF-UR APEL CAR Content description
cores Processors Processors The number of cores allocated to the job
systemsoftware     The type and version of the system software (i.e. opsys, glibc, compiler, or the entire container wrapping the system software)
WNinstance ServiceLevel   Coarse-grain characterization tag for the WorkerNode, e.g. BigMemory or t2.micro (aka Amazon instance type)
RTEs     List of used RTEs, including default ones as well.
data-stagein-volume Network class has something similar   The total volume of downloaded job input data in GBs
data-stagein-time     The time spent by the DTR sysem to download input data for the job
data-stageout-volume Network class has something similar   The total volume of uploaded job output data in GBs
data-stageout-time     The time spent by the DTR sysem to upload output data of the job
lrms-submissiont-time     The timestamp when the job was handed over to the LRMS system
lrmstarttime     The timestamp when the payload starts in the LRMS
lrmsendtime     The timestamp when the payload completed in the LRMS
authtokenattributes UserIdentity.VO and child structures UserIdentity.Group and UserIdentity.GroupAttribute to be implemented to store e.g. VO atributes and simillar capabilities
submissioninterface     to be implemented, at the same time clean up globalid that depends on submissioninterface type
benchmark Benchmark ServiceLevel The type and the corresponding benchmark value of the assigned WN
usedscratchspace StorageUsageBlock   The used size of scratch dir at the end of the job termination in the LRMS.
Table 6 NOT USED SGAS or APEL attributes
Charge Charge
Swap Swap