Main changes in ARC 6 compared to ARC 5

There are many improvements and changes in the new ARC release, and with these we hope to have greatly enhanced the ease of setting up, configuring and managing an ARC production site, and to have improved the reliability and scalability of the ARC CE by the internal restructuring ARC has undergone.

Note

Despite all the new features and code changes, the supported ARC 6 CE interfaces are unchanged - the latest ARC 5 clients are compatible with an ARC 6 CE and vice versa.

  • Complete overhaul of server-side ARC CE configuration:
    • Configuration has been completely reworked, cleaned up, regrouped and made more streamlined.
    • In particular, the way authorization and user mapping is handled has been greatly improved. Worth mentioning is the support of higher granularity vo-based authorization, and that mapping via gridmap files is no longer required nor recommended.
    • ARC now comes with a so-called zero configuration - a preconfigured minimal ARC setup automatically installed with ARC, including test-CA and test-host certificate for immediate test submission.
    • Default values throughout ARC have had a big tidy up and are now handled consistently through the ARC runtime configuration.
    • Configuration blocks are now used to enable (turn on) or disable a functionality, a service or an interface. For example, the NorduGrid schema is no longer published unless it is enabled via the corresponding configuration block.
    • Validation of the ARC configuration is enforced by A-REX startup scripts. If you have configuration errors then A-REX will not start, and you will be pointed to the error.
  • Scalability and manageability improvements:
    • The internal job-loop in A-Rex has been re-engineered to be event-driven
    • xrootd plugin has been re-written to improve data transfer performance
    • Consistent handling of server-side logging and improved logfile structure
    • Reworked startup scripts
    • Streamlined package name scheme, where e.g. ldap services have been separated out
  • The RTE framework has got a redesigned architecture and a largely extended functionality (default, installed and enabled RTEs, introduction of RTE parameters)

  • A new framework for RunTimeEnvironments (RTE):
    • The RTE framework has gotten a redesigned architecture and largely extended functionality (default, installed, enabled RTEs, introduction of RTE parameters).

    Note

    RTEs must in ARC 6 be explicitly enabled through the new arcctl tool as a separate step after installation.

    • There are several system RTEs installed together with ARC which you can enable on demand. These are:
      • ENV/CANDYPOND: makes ARC Candypond (“Cache and deliver your pilot on-demand data”) client available on the Worker Nodes and ready to be used
      • ENV/CONDOR/DOCKER: enables submission to Docker universe in HTCondor backend
      • ENV/LRMS-SCRATCH: enables the usage of local WN scratch directory defined by LRMS
      • ENV/PROXY: copies proxy certificate to the job session directory
      • ENV/RTE: copies RunTimeEnvironment scripts to the job session directory
  • ARCCTL, a new server-side management and control tool for sysadmins has been developed.
    • arcctl is meant to be the central one-stop-shop tool for administrating an ARC CE
    • With this tool you can handle RTEs, the jobs, accounting, ARC services and many other things related to an ARC CE.
    • arcctl also offers help with integrating 3rd party services and components with an ARC CE such as handling CAs, VOMS, etc..
  • JURA Accounting subsystem improvements:
    • Accounting configuration, archive and operations has been restructured to improve typical operations.
    • Improved republishing via arcctl.
    • Better integration with APEL SSM.
    • Archive structure is relying on a database for republishing and gathering stats locally.
  • ARCHERY, the new DNS-based service endpoint catalogue for ARC
    • archery-manage package to populate the ARCHERY registry with ARC CE endpoint info
    • Official top-level DNS registry under the nordugrid.org domain
    • ARCHERY-compatible clients to submit and monitor jobs (arc cli, ldap monitor)
    • ARCHERY replaces the OBSOLETED EGIIS service of the ARC 5 release series.
  • TECHNOLOGY PREVIEW components:
    • The REST interface - enabled together with the A-Rex Web-Service.
    • The INTERNAL interface - a special interface aimed for restrictive HPC sites, to be used with a local installation of the ARC Control Tower.
    • Candypond - “Cache and deliver your pilot on-demand data” service to bring the power of ARC Cache to the pilot job universe.
    • Experimental Python LRMS for SLURM with SSH access support.
  • The ARC client and SDK:
    • The client has undergone a major internal cleanup, large number of submission, target retrieval and job description plugins got removed as a result of OBSOLETING third-party interfaces and technologies in ARC.
    • New plugins for ARCHERY and the REST And INTERNAL interfaces were implemented.
    • The arcstat cli of the ARC6 client and the underlying SDK now handles walltime as (walltime)*(number of cores) for multicore jobs.

REMOVED and obsoleted components

  • Large number of configuration options of arc.conf got DELETED during the server-side configuration rework. A detailed list of those options are available in the arc.conf.DELETED file.
  • The separate gangliarc component is discontinued, instead we now offer ganglia integrated into AREX.
  • Nordugrid no longer distributes a source code bundle including documentation, gangliarc and nagios.
  • CREAM related plugins and server-side components.
  • UNICORE related plugins and server-side components.
  • All the windows & solaris related ARC code and build.
  • JAVA bindings for ARC SDK.
  • Support for BES, WSRF and other non-EMIES WS flavours as job management interfaces.
  • Support for JDL, ARCJSDL, RSL job description dialects: ARC keeps only XRSL and ADL as supported job description languages.
  • EMIR indexing service including ARC CE registration to EMIR and ARC SDK plugins.
  • Server-side EGIIS, including ARC CE registration to EGIIS. BUT: keep EGIIS plugins in the client.
  • GLUE1 support from the ARC client SDK, server-side support for GLUE1 temporarily kept but labelled as OBSOLETE.
  • ARC SDK support to obtain information from top-BDII.
  • Publishing nordugrid-authuser objects in the NorduGrid LDAP schema

List of bugs fixed since ARC 5

A rather long (but not exhaustive) list of bugs fixed since ARC 5 can be found here: http://www.nordugrid.org/arc/arc6/common/changelog/bugs-6.0.0.html