ARC5 to ARC6 Migration Guide

ARC 6 comes with major re-engineering in the code base leading to improvements in the deployment, configuration, maintenance and scalability of ARC CE. It introduces new features while some legacy interfaces, components and features have been obsoleted and removed.

Despite all the code changes the supported ARC 6 CE interfaces are unchanged as well as the structure of the control directory, session directory, accounting logs, etc. This makes it possible to upgrade ARC5 to ARC6 without a complete re-installation “from scratch”.

Note

Despite the fact that we haven’t observed any issues during normal package upgrades it is recommended to perform a clean install “from scratch” wherever possible in order to eliminate hidden bugs and ARC5 leftovers.

Following sections provides you with the necessary info about the major steps of an ARC5 to ARC6 migration.

Quick reference

Here is a quick 6 step checklist for what to do:

  1. Rewrite your arc.conf: you can send your configuration to us and we will help to rework it for ARC6.
  2. Learn the ARC6 packages and services names.
  3. Get familiar with arcctl that will simplify your life as an ARC6 administrator.
  4. Don’t forget to enable RunTime Environments (RTE) with arcctl rte enable.
  5. Be aware of accounting subsystem changes.
  6. Say “Good Bye!” to EGIIS, the ARC community is migrating over to a DNS-based ARCHERY system.

The details:

Configuration file

Because of the complete overhaul of the server-side ARC CE configuration, an existing ARC5 configuration can not be used with an ARC6 installation, it will immediately fail during the configuration validation process of the ARC service startup. Therefore you must create a new ARC6 configuration from scratch.

Note

There is no automatic converter from an old ARC5 configuration file to the new ARC6 syntax. The main reason for this is that there are many conceptual changes rather than simple option and block renaming changes.

We offer support in configuration rework via nordugrid-discuss mailing list or ARC6 Support Skype Channel.

Just send your configuration to us!

When reworking the configuration take notice of these syntax changes:

  • no more quotes in option values
  • blocks are now not only grouping options, but have a enable/disable functionality
  • in block names keywords are separated by / to represent subblocks (if such exist), while block identifiers are separated by : from keywords

Note

It is advised to read the General configuration structure section of ARC configuration reference document before starting any rewrite.

There are also changes in the configuration approach in several areas. The major highlights are:

  • The way authorization and user mapping is handled has been greatly reworked and improved. In particular mapping via gridmap files is NO longer required nor recommended. If you still have gridmap files in your configuration consider whether you really want to keep this approach in ARC6.
  • Jura accounting configuration is now configured within dedicated blocks.
  • Infosys schemas are now also enabled/disabled with blocks. In particular the default ARC5 NorduGrid LDAP schema should be explicitly enabled with [infosys/nordugrid] block.

Warning

ARC6 configuration also comes with consistent defaults for everything. If a block is enabled all configuration options in this block that has no specific value in arc.conf get their default values.

Configuration with all defaults included can be printed with arcctl config dump.

Notice that any changes in configuration requires restart off all affected ARC services.

Note

Config validation is enforced during A-REX start or can be run manually with arcctl config validate. So don’t worry if you had misspelled something, the validator will catch it!

Operating ARC services

ARC6 also comes with the a rewised consistent packages and services naming scheme. We made the service and package names more clear and consistent with configuration block names in arc.conf.

Log file locations and internal format have been reviewed to provide consistency across the ARC subsystems. Log file locations can be printed with arcctl config brief.

ARC Control Tool

To simplify common ARC CE operations, ARC6 introduces arcctl - the single entry point to administrate and work with various ARC and third-party components. The arcctl tool helps ARC site admins by wrapping typical complex operations and unifying ARC CE administration under a single utility.

The arcctl tool was designed with BASH-completion in mind. It completes subsystem names, arguments, jobIDs, certificate DNs, RTE names, etc.

Note

To get familiar with some of the arcctl functionality it is advised to follow Try ARC6: towards distributed computing in a few minutes tutorial. Using the arcctl tool to administrate an ARC6 site is recommended even for the most experienced ARC5 admins.

RunTime Environments

The new framework for RunTime Environments (RTE) has gotten a redesigned architecture and a largely extended functionality (default, installed, enabled RTEs, introduction of RTE parameters).

The major conceptual and behaviour differences are:

  • in ARC6 RTE directories are local to the ARC CE head node and SHOULD NOT be shared to worker nodes
  • in ARC6 the RTE concept got extended by the introduction of RTE types such as default, installed, enabled, dummy, etc.. An ordinary ARC5 RTE would correspond to an installed user-defined ARC6 RTE. Please read the documentation.
  • All installed ARC5 RTEs (available in a directory) should be enabled explicitly (e.g. arcctl rte enable ENV/PROXY) to be used on an ARC6 CE.

Note

You can use wildcards to enable all RTEs you have in one shot: arcctl rte enable '*'

Accounting

In ARC 6.0 Jura accounting had redesigned archive structure that introduces an archived records database and comes with a redesigned record republishing to accounting services, in addition to the already mentioned configuration changes.

To make ARC 6.0-6.3 accounting compatible with ARC5 jura, an automatic archive conversion is done by an additional jura-archive-manager process run by A-REX.

Warning

The JURA archive conversion consumes CPU and I/O resources of your ARC CE. If you are upgrading and have a lot of records in archive already expect an additional load during conversion.

In ARC 6.4 the next generation accounting subsystem had been introduces. Instead of publihsed records archiving it introduced local SQLite database that stores A-REX accounting records for each job for both on-site CE auditing and publishing.

Warning

If you want to keep old archived usage records from ARC 5 release it is advised to upgrade from ARC 5 to ARC 6.3 first, wait for records conversion completion, than update to ARC 6.4 or higher versions.

Retiring the EGIIS

All configuration blocks and services in ARC5 that correspond to registration to EGIIS are now gone in ARC6.

ARCHERY, the new DNS-based service endpoint catalogue for ARC is replacing EGIIS and it works top-to-bottom without explicit registration, i.e. nothing registration related needs to be configured in the arc.conf of an ARC6 CE.

If your CE had been previously registered in the country-level NorduGrid EGIIS your ARC CE registration is already migrated to the nordugrid.org ARCHERY instance.

If you find your CE missing (e.g. in the nordugrid top-level monitor) please contact the country-level ARCHERY instance admins as described in this document.