Accounting with legacy JURA

WARNING: This component was deprecated in ARC 6.4 and completely removed in ARC 6.8!

Warning

Information in this chapter is relevant only for 6.0-6.3 ARC releases.

Starting from ARC 6.4 release the Accounting Subsystem with local accounting database will be introduced. Make sure you are reading the documentation that match your ARC CE release version.

The Job Usage Reporter of ARC (JURA) is a component which is capable to create standard-complient usage records from job usage information provided by the A-REX (Job Log files) and send the records to remote accounting services.

JURA is capable of creating two types of usage records from the job log files:

  • Usage Record 1.0 (UR) XML format to be sent to an SGAS LUTS (Logging and Usage Tracking Service)

  • Compute Accounting Record (CAR) XML format to be sent to APEL

Overview

Figure shows the overview of accouting workflow in ARC. More details can be found in Legacy JURA Accounting Technical Details.

Accounting records processing in ARC CE

Fig. 7 Accounting records processing in ARC CE: creating, processing, publishing and archiving

When accounting is enabled, the sequence of events is the following:

  1. A-REX writes the per-job accouting records to Job Logs directory (logs directory inside the controldir).

  2. A-REX periodically runs jura. Default and minimum period is hourly but can be increased with urdelivery_frequency option.

  3. Jura reads jobs accounting information from Job Logs and depending on the targets cofigured in arc.conf creates accounting records according to required format.

  4. Jura publishes accounting records to the configured accounting services.

  5. If records archiving is enabled, JURA also puts the records into Accounting Archive

  6. When archiving functionalily is used, A-REX also runs the jura-archive-manager process which manages the archive structure and indexes the records in the local database. The local accounting database can then be used to lookup statistics or to republish data.

Enabling accounting records reporing

JURA is a stand-alone binary application that is a part of the A-REX package and can be enabled with [arex/jura] block.

JURA has a dedicated log file as defined by the logfile option. Log rotation is set for the default /var/log/arc/jura.log location. Accounting logs can be also viewed with arcctl:

Changed in version 6.4: Extra legacy argument is required to access legacy accounting subsystem functionality. Use arcctl accounting legacy logs.

[root ~]# arcctl accounting logs

Accounting services for sending the records are configured with dedicated sub-blocks. You need to define a separate block with an unique targetname for every target server used.

The usage record of each job will be reported to all of the destinations, unless the vofiler option is configured for some of them to filter records by VO name.

Configuring reporting to SGAS

The SGAS sub-block enables and configures an SGAS accounting server as a target destination to which JURA will send properly formatted Usage Record 1.0 (UR) XML records.

The targeturl option is the only mandatory parameter to configure an SGAS target. In the specific setup cases you can also apply VO filtering and set prefix for local job IDs.

Example:

[arex/jura/sgas: NeIC]
targeturl = https://grid.uio.no:8001/logger
urbatchsize = 80

Configuring reporting to APEL

The APEL sub-block enables and configures an APEL accounting server as a target destination to which JURA will send properly formatted Compute Accounting Record (CAR) XML records.

The targeturl option defines the APEL broker URL to send records to. Unlike the native APEL client that relies on Top-BDII infrastructure in runtime to find the blockers, ARC uses a dedicated target URL in the configuration. However, it is possible to look up available targets with arcctl:

[root ~]# arcctl accounting apel-brokers
http://mq.cro-ngi.hr:6163/
http://broker-prod1.argo.grnet.gr:6163/
[root ~]# arcctl accounting apel-brokers --ssl
https://mq.cro-ngi.hr:6162/
https://broker-prod1.argo.grnet.gr:6162/

SSL or non-SSL connecton should be defined accordingly with the reference_arex_jura_apel_use_ssl option.

You also need the GOCDB name of the resource and the APEL topic to which JURA will publish accounting records. For a correct production accounting setup it is recommended to specify resource benchmarking results.

Example:

[arex/jura/apel: EGI]
targeturl = https://mq.cro-ngi.hr:6162
topic = /queue/global.accounting.cpu.central
gocdb_name = RAL-LCG2
benchmark_type = Si2k
benchmark_value = 2625.00
use_ssl = yes

Note

Pubishing to APEL relies on SSM framework that has own logfile viewable with arcctl accounting logs --ssm.

Records archiving

The archiving functionality allows to store generated usage records in a specified directory on the disk.

The reference_removed_arex_jura_archiving enables this functionality. Directory path for the archivedJ ura usage records is defined with the reference_arex_jura_archiving_archivedir option.

Once archive is enabled, Jura writes generated usage records to files named usagerecord[CAR].<jobid>.<random>. If a A-REX job log file is processed repeatedly – for example because of temporary connection failures to the remote accounting service – and a respective usage record archive file already exists, then the usage record is not generated again. Instead, the content of the archive file is used without change (the creation time stamp is also retained).

Note

Records archive can grow very rapidly on heavy loaded sites with huge ammount of jobs comming daily. To limit the archiving time, use the reference_arex_jura_archiving_archivettl option.

To better organize archived records and index the accounting data in the local database, the jura-archive-manager helper process is run periodically by A-REX when archive is enabled.

The archive manager:

  • reads the resource usage data and stores it to an indexed SQLite database file accounting.db.

  • moves files according to the following directory structure: YYYY-MM/DD/<jj>/<kkkkk>.<random>.{UR|CAR}, where jj is first 2 characters of <jobid> and kkkkk is the rest of <jobid>

Lookup local accounting data

Data in the local accounting database can be viewed with arcctl for each type of destination. Timeframe of interest and many other filters can be specified, e.g.:

Changed in version 6.3: Extra legacy argument is required to access legacy accounting subsystem functionlity. Use arcctl accounting legacy stats.

[root ~]# arcctl accounting stats -t apel -b 2018-11-01 -e 2018-11-30
Statistics for APEL jobs from 2018-11-01 00:00:02 till 2018-11-29 23:59:13:
  Number of jobs:           355168
  Total WallTime: 48146 days, 18:06:50
  Total CPUTime:  71927 days, 10:33:33
[root ~]# arcctl accounting stats -t apel -v
ops
belle
atlas
ops.ndgf.org
[root ~]# arcctl accounting stats -t apel --filter-vo belle
Statistics for APEL jobs from 2015-04-13 07:28:52 till 2019-04-02 00:25:57:
  Number of jobs:           869400
  Total WallTime: 216866 days, 3:33:33
  Total CPUTime:  193604 days, 19:40:58

Republishing records

When something goes wrong with accounting services, network etc, it may be needed to republish local records again.

Warning

Republishing is ONLY possible when Records archiving is enabled and records still exist for the required time period.

The ARC Control Tool should be used to republish data. It is tightly integrated with the accounting archive database: arcctl will look up needed records in the database for the defined republishing period, prepare them for republishing and than invoke jura with correct configuration settings.

Warning

It is strongly recommended to AVOID running republishing in ARC5-way, executing the jura command manually. Even if you manage to get it working, it can cause unpredictable results.

Define the timeframe of interest and run the command to rebuplish data:

Changed in version 6.4: Extra legacy argument is required to access legacy accounting subsystem functionality. Use arcctl accounting legacy republish.

[root ~]# arcctl accounting republish -b 2018-12-01 -e 2018-12-31 -s https://sgas.grid.org.ua:8443/sgas
[root ~]# arcctl -d DEBUG accounting republish -b 2019-04-01 -e 2019-04-10 -a https://mq.cro-ngi.hr:6162