Release Notes for NorduGrid ARC 6.0.0

May 29, 2019

We are very happy to announce the first release of the ARC 6 release series.

ARC 6 comes with major re-engineering in the code base leading to improvements in the deployment, configuration, maintenance and scalability of ARC CE. It introduces many new features while some legacy interfaces, components and features have been obsoleted and removed.

Despite all the new features and code changes the supported ARC 6 CE interfaces are unchanged - the latest ARC 5 clients are compatible with an ARC 6 CE and vice versa.

With ARC 6 we hope to have greatly enhanced the ease of setting up, configuring and managing an ARC production site, and to have improved the reliability and scalability of the ARC CE by the internal restructuring ARC has undergone.

Important Changes

• Complete overhaul of server-side ARC CE configuration:
  • Configuration has been completely reworked, cleaned up, regrouped and made more streamlined.
  • In particular, the way authorization and user mapping is handled has been greatly improved. Worth mentioning is the support of higher granularity vo-based authorization, and that mapping via gridmap files is no longer required nor recommended.
  • ARC now comes with a so-called zero configuration - a preconfigured minimal ARC setup automatically installed with ARC, including test-CA and test-host certificate for immediate test submission.
  • Default values throughout ARC have had a big tidy up and are now handled consistently through the ARC runtime configuration.
  • Configuration blocks are now used to enable (turn on) or disable a functionality, a service or an interface. For example, the NorduGrid schema is no longer published unless it is enabled via the corresponding configuration block.
  • Validation of the ARC configuration is enforced by A-REX startup scripts. If you have configuration errors then A-REX will not start, and you will be pointed to the error.
• Scalability and manageability improvements:
  • The internal job-loop in A-Rex has been re-engineered to be event-driven
  • xrootd plugin has been re-written to improve data transfer performance
  • Consistent handling of server-side logging and improved logfile structure
  • Reworked startup scripts
  • Streamlined package name scheme, where e.g. ldap services have been separated out
• A new framework for RunTimeEnvironments (RTE):
  • The RTE framework has gotten a redesigned architecture and largely extended functionality (default, installed, enabled RTEs, introduction of RTE parameters).
  • Runtime environments (RTEs) must be explicitly enabled through the new arcctl tool (see below)
  • There are several system RTEs installed together with ARC which you can enable on demand. These are:
    • ENV/CANDYPOND: makes ARC Candypond ("Cache and deliver your pilot on-demand data") client available on the Worker Nodes and ready to be used
    • ENV/CONDOR/DOCKER: enables submission to Docker universe in HTCondor backend
    • ENV/LRMS-SCRATCH: enables the usage of local WN scratch directory defined by LRMS
    • ENV/PROXY: copies proxy certificate to the job session directory
    • ENV/RTE: copies RunTimeEnvironment scripts to the job session directory
• ARCCTL, a new server-side management and control tool for sysadmins has been developed.
  • arcctl is meant to be the central one-stop-shop tool for administrating an ARC CE
  • With this tool you can handle RTEs, the jobs, accounting, ARC services and many other things related to an ARC CE.
  • arcctl also offers help with integrating 3rd party services and components with an ARC CE such as handling CAs, VOMS, etc..
• JURA Accounting subsystem improvements:
  • Accounting configuration, archive and operations has been restructured to improve typical operations.
  • Improved republishing via arcctl.
  • Better integration with APEL SSM.
  • Archive structure is relying on a database for republishing and gathering stats locally.
• ARCHERY, the new DNS-based service endpoint catalogue for ARC
  • archery-manage package to populate the ARCHERY registry with ARC CE endpoint info
  • Official top-level DNS registry under the nordugrid.org domain
  • ARCHERY-compatible clients to submit and monitor jobs (arc cli, ldap monitor)
  • ARCHERY replaces the OBSOLETED EGIIS service of the ARC 5 release series.
• TECHNOLOGY PREVIEW components:
  • The REST interface - enabled together with the A-Rex Web-Service.
  • The INTERNAL interface - a special interface aimed for restrictive HPC sites, to be used with a local installation of the ARC Control Tower.
  • Candypond - "Cache and deliver your pilot on-demand data" service to bring the power of ARC Cache to the pilot job universe.
  • Experimental Python LRMS for SLURM with SSH access support.
• The ARC client and SDK:
  • The client has undergone a major internal cleanup, large number of submission, target retrieval and job description plugins got removed as a result of OBSOLETING third-party interfaces and technologies in ARC.
  • New plugins for ARCHERY and the REST And INTERNAL interfaces were implemented.
  • The arcstat cli of the ARC6 client and the underlying SDK now handles walltime as (walltime)*(number of cores) for multicore jobs.

Removed and Obsoleted Components

• Large number of configuration options of arc.conf got DELETED during the server-side configuration rework. A detailed list of those options are available in the arc.conf.DELETED file.
• The separate gangliarc component is discontinued, instead we now offer ganglia integrated into AREX.
• NorduGrid no longer distributes a source code bundle including documentation, gangliarc and Nagios plugins.
• CREAM related plugins and server-side components.
• UNICORE related plugins and server-side components.
• All the Windows and Solaris related ARC code and build.
• JAVA bindings for ARC SDK.
• Support for BES, WSRF and other non-EMIES WS flavours as job management interfaces.
• Support for JDL, ARCJSDL, RSL job description dialects: ARC keeps only XRSL and ADL as supported job description languages.
• EMIR indexing service including ARC CE registration to EMIR and ARC SDK plugins.
• Server-side EGIIS, including ARC CE registration to EGIIS. BUT: keep EGIIS plugins in the client.
• GLUE1 support from the ARC client SDK, server-side support for GLUE1 temporarily kept but labelled as OBSOLETE.
• ARC SDK support to obtain information from top-BDII.
• Publishing nordugrid-authuser objects in the NorduGrid LDAP schema

Known Issues and Notes

• If you are updating from ARC5 with an old JURA accounting archive already containing records, the conversion process to index the archive structure will be initiated and will cause serious system load until finished. To avoid old archive conversion, you can move records before update.
• Due to granularity of information in GLUE2 and NorduGrid schema, the GLUE2 schema is more accurate and the returned values might differ by +/- 1 min between the two schemas
• Nagios probes have their own release cycle and are no longer included in the NorduGrid ARC release. The Nagios probes source-code together with the installation instructions can be found here.

Bugs Fixed Since ARC 5

A long (but not exhaustive) list of bugs fixed since ARC 5 can be found here and at the end of these release notes.

Documentation

The ARC 6 documentation has been rewritten in a modern format and is placed at a new location.

Please note that although the most important parts of the ARC 6 documentation are in place, it is still not fully complete. If you have questions, please contact us!

Installing ARC 6

We recommend to install ARC release using the Nordugrid repository.

• Start with the basics
• Get production ready

We provide binary builds for the following systems:

• Debian: 7.0 - 9.0 (i386 and amd64)
• Fedora: 12 - 30 (i386 and x86_64)
• CentOS: EL6 (i386 and x86_64) and EL7 (x86_64)
• Ubuntu: 14.04 - 18.10 (i386 and amd64, both LTS and interim releases)

Future Support of ARC 5-series

Now that ARC 6.0.0 is released, we will only provide security updates of ARC 5.

In particular:

1. No new feature development is planned or going on for ARC5 and no bug-fixing development will happen on ARC5 code base in the future except for security issues.
2. Security fixes for ARC5 will be provided till end of June 2020.
3. Production Sites already running ARC 5 will be able to get deployment and configuration troubleshooting help via GGUS till end June 2021. This we call "operational site support".
4. ARC5 is available in EPEL7 and will stay there. EPEL8 will only contain ARC 6.

Getting in Touch

If you need help with the configuration file or have any other questions related to ARC 6, please contact us via

• Email: (Subscribe)
• Skype channel dedicated to ARC 6 support

List of Bugs Fixed Since ARC 5

• 895 ARC Administrative tools
• 1421 No log of backend script problems
• 3034 arcstat -s is case-sensitive
• 3106 arcproxy could be more expressive when it finds a problem
• 3360 SGE and LL backends fail to report correct node OS/system information in GLUE2 infosystem
• 3384 Support for per-queue authorisation configuration and publishing
• 3451 Configure number of cpus manually
• 3476 Crashes in multiple ARC components
• 3545 Patch for Correct Cores Parsing
• 3557 undetected job submission in case of heavy filesystem load
• 3565 Allow setting default VO in ~/.arc/client.conf to be used in arcproxy
• 3566 Implement RTEs processing without shared directory
• 3569 Exporting CPU/Wall time limits to Glue2/BDII
• 3570 Arcsub used 100GB memory
• 3584 JURA: create one log file per job, not per submission
• 3624 Data delivery service can only listen to one network interface.
• 3626 Force GLUE2ComputingManagerTotalLogicalCPUs to be totalcpus when this value is defined in arc.conf
• 3632 arcproxy fails in Ubunt 16.04, 16.10 and recent Debian systems
• 3637 arcget with multiple jobs crashes
• 3643 The watchdog crashes
• 3662 arcsub crashes
• 3667 JSON output for arcstat
• 3674 settings in client.conf ignored
• 3675 Problems retrieving jobs with arcget 1
• 3676 Problems retrieving jobs with arcget 2 (first byte missing)
• 3677 arcproxy fails retrieving attributes from voms
• 3682 Better error message when DN not in gridmap file
• 3690 Warnings about missing information on deleted jobs
• 3695 Slowness with arccp and the xrootd protocol
• 3700 ARC1ClusterInfo.pm uses netstat
• 3702 Cannot use arc data commands without certificates
• 3707 Seg fault Triolith - related or not to the data-staging problems on Triolith
• 3713 Malformed jobs.dat entries
• 3722 Xenial repo for the (nordugrid) ARC source is not correct
• 3756 package update failed to restart A-REX
• 3772 Can't use different credentials within one submission process when jobs require user input files
• 3773 Enabling arex-ganglia breaks controldir access
• 3778 arcctl not working if arc code configured with --disable-swig flag and installed with make install
• 3788 Poor performance with arccp and HTTPS
• 3812 A-REX hangs inside XRootd after fork

Previous releases

Details of previous releases can be found at the ARC Releases page