Legacy JURA Accounting Technical Details

WARNING: This component was deprecated in ARC 6.4 and completely removed in ARC 6.8!

Warning

Information in this chapter is relevant only for 6.0-6.3 ARC releases.

Starting from ARC 6.4 release the next generation accounting subsystem with local accounting database will be introduced. Make sure you are reading the documentation that match your ARC CE release version.

General accounting configuration and operations flows are described in Accounting with legacy JURA. This section contains more technical details about implementation of each component of accounting subsystem.

Records processing and publishing

JURA Processing and Publishing records

Fig. 25 ARC CE accounting: records creation, processing and publishing

AREX Accounting Records (AAR job log files)

The A-REX Accounting Records (AAR) are job log files generated by A-REX. AAR is the only source of accounting information for JURA. AARs are written by A-REX based on job data available, including .diag files that backend scripts creates based on batch system data and/or GNU time utility measurements. These job log files reside under the <control_dir>/logs directory. The name of the AAR job log files consist of the ID of the job and a random string to avoid collision of multiple job log files for the same job: <jobid>.<random>.

The AAR job log file consists of name=value lines, where value is either a job-related resource consumption data or a static info like name, ID or proxy certificate.

A-REX generates at least two job log files for each job: one at the time of job submission, another one after the job finishes, and possibly others at various job events. Please note JURA makes use only one of the AREX generated files belonging to the same job: the one that corresponds to the FINISHED job event (such state is indicated by the status={completed|failed|aborted}).

JURA initial AAR processing

A-REX periodically runs jura that loop over available A-REX job log records in the <control_dir>/logs. JURA opens all the files and processes only those that corresponds to a FINISHED job state.

JURA converts AARs to per-job per-destnation extended AARs that contains the target information from arc.conf as well. One extended AAR is generated per accounting target. The extended AAR job log files named <jobid>.<random>_<random2> where first <random> is taken from original AAR.

The original AAR <jobid>.<random> file is deleted once per-destination extended AAR logs are created by JURA.

Note

JURA as part of the initial processing deletes all files corresponding to non-finished job states.

JURA Publihsing loop

JURA publishing subsystem loop over extended AAR logs in the A-REX <control_dir>/logs directory. JURA generates records in the Usage Record (UR) format proposed by the Open Grid Forum (OGF) for SGAS or Compute Accounting Record (CAR) XML for APEL.

The extended AAR job log file is deleted once record is successfully submitted, thus preventing multiple insertion of same usage records. If submission to destination fails, the extended AAR log files are kept, so another attempt is made upon a subsequent run of JURA. This mechanism will be repeated until the expiration time passes at which point the next execution of JURA removes the file without processing.

Please note that the JURA publishing loop is backward compatible with ARC 5 implementation.

Reporting to SGAS

SGAS has a simple custom web service interface loosely based on WS-ResourceProperties. JURA uses the insertion method of this interface to report URs directly using ARC HTTP client implementation. The corresponding processed extended AAR job log files are deleted after receiving a non-fault response from the service.

To increase communication efficiency JURA sends URs in batches. SGAS accepts a batch of URs in a single request. The batch is an XML element called UsageRecords, containing elements representing URs.

The process of handling batches is the following: JURA does not send all usage records immediately after generation, but instead collects them in a batch until reaching the maximal number of records or until running out of job log files. The maximal number of URs in a batch can be set as a urbatchsize configuration parameter of SGAS target.

Reporting to APEL

APEL uses the SSM framework for communication.

JURA send records to APEL by means of invoking helper ssmsend process that uses SSM python librarires developed by APEL.

ARC ships minimal set of SSM libraries along with A-REX binary packages to allow SSM usage. If SSM binary packages from APEL are availble for your OS (e.g. EL6), you can install this packages and they will be used instead of those shipped with ARC automatically.

JURA prepares the messages to be sent by ssmsend and puts them info SSM Outgoing directory located in the /var/spool/arc/ssm/<destination hostname>/outgoing/00000000/. Generated messages are XML based CAR records with file name format <YYYYMMDDhhmmss>.

Reporting to APEL also works with sending records in batches. The default urbatchsize value is set to 1000 according to APEL recommendatations.

Accounting archive

JURA Archiving records

Fig. 26 ARC CE accounting: records archiving

After records are archived by jura a dedicated jura-archive-manager process manages the archive layout and archive database.

Republishing process

JURA Republishing records

Fig. 27 ARC CE accounting: records republishing

Republihsing is a complex workflow that involves:

  • archive database to find out which subset of records should be republished

  • exporting necessary records from archive structure to temporary flat directory

  • trigering jura with corresponding configuration to grab records from temporary directory and send them to target

  • cleanup after republihsing

This chain is controlled from arcctl code.

Security

The JURA executable runs with the same user privileges as the A-REX. The owner of a job log file is the local user mapped for the submitter entity of the corresponding job. Since these files contain confidential data, A-REX restricts access to them allowing only read access for the job owner, thus when JURA is executed by A-REX it is allowed to read and delete job log files.

All usage records are submitted using the X.509 credentials specified by the value of x509_ set of confiurartion options of arc.conf. No proxies are used for communcation with accouting services.

The only access restriction made by a SGAS service is matching the Distinguished Name of the client (in this context JURA) with a set of trusted DNs. When access is granted, policies are then applied by SGAS, allowing either publishing and/or querying rights. Clients with publishing right can insert any UR, regardless of content. By default, querying right only allows retrieving URs pertaining to jobs submitted by the querying entity.

Implementation and API

JURA as part of the ARC software stack is written in C++, and utilizes the functionality provided by the ARC libraries, including secure HTTPS communication provided by the ARC plugable TLS and HTTP modules.

The modular design is also present in the usage reporting part of the JURA code, making it possible to extend JURAs support of accounting services. To create a JURA module one should simply write a C++ class which inherits from the abstract Arc::Destination class, and it must extend the two methods:

  • static Arc::Destination* Arc::Destination::createDestination(Arc::JobLogFile&)

  • void Arc::Destination::report(Arc::JobLogFile&)

The static createDestination method should initialize a object of the specialized class, using the configuration options specified in the passed Arc::LogFile object, and the memory allocated by the method should be freed by the caller. Then the report method should carry out the transfer of the UR, represented by the JobLogfile object, to the accounting service.

JURA archive manager is written in Python and share the classes for managing ARC components with ARC Control Tool.

Limitations

In the following list some issues which limits the functionality of JURA is described:

  • The current implementation of JURA and A-REX supports only one expiration time for all the reporting destinations. Even though the configuration enables the specification of different expiration values per reporting destination, it is not taken into account by the system, the last value is used as the common expiration time value.

  • It is not possible to use different credentials per destinations.

  • If you are updating from ARC5 with an old jura accounting archive already containing records, the conversion process to index the archive structure will be initiated and will cause serious system load until finished. To avoid old archive conversion, you can move records before update.

  • Some optional UR properties are not supported.

  • Memory can be reported incorrectly with buggy GNU “time” results.