ARC v11.05 Release Announcement

May 10, 2011


The Advanced Resource Connector (ARC) middleware, introduced by NorduGrid, is an open source software solution enabling production quality computational and data Grids since May 2002. The previous production ARC release, version 0.8.3.1 was released on December 21, 2010.

The 11.05 release of the ARC software represents a major change, for the first time in history of ARC introducing a very substantially re-written computing element (execution service), and for the second time - a re-written client component. Moreover, distribution and version conventions of ARC and its packages change in order to accommodate latest developments in European Grid middlewares. Code repository underwent major changes as well. For the reasons above, careful upgrade from older ARC releases is advised, following the suggested migration instructions. While in general the upgrade procedure is reasonably smooth, you may encounter difficulties in case your system was configured in a severely customized manner.

New version convention, packaging and distribution

In previous ARC releases, release number was identical to that of the individual package version. This caused certain limitations and confusion. Starting with 11.05, ARC releases follow the numbering scheme familiar for Ubuntu users: first two digits indicate release year (2011), next two digits separated by a period indicate release month (5 for May), and eventual next two digits after a period denote incremental minor release version (absent for the major release).

Packages constituting an ARC release changed their contents, some keeping the names (when the contents was completely analogous, like nordugrid-arc-client), and some changing the name. Packages containing preview "nox" components in ARC 0.8 series became regular nordugrid-arc packages. Details of new packaging convention can be found in NorduGrid Wiki.

Package version number follows the traditional approach: underscoring the major step forward, most ARC packages in release 11.05 acquire version number 1.0.0. For packages that preserve their names it makes an auto-upgrade technically possible. Package numbers are in general independent from each other, such that ARC releases in future will see more packages with differing versions. A change of a package version does not necessarily trigger a new ARC release. More details on version numbering and rules can be found in NorduGrid Wiki.

To prevent accidental auto-upgrade, NorduGrid switches to the new repository structure, introducing new repository for each major release. This also allows to maintain support for older releases, if necessary.

Several key ARC 11.05 packages built for Scientific Linux 5 x86_64 are also distributed with the EMI-1 release. Package naming, numbering and dependencies in EMI is identical to those in ARC releases.

Previously existing meta-packages are re-defined to include new components:

New features highlights

ARC 11.05 collects and integrates several innovative next generation services and client tools into a consistent software release. All these components were available for preview in ARC 0.8 releases ("nox" packages), and now are mature enough to become mainstream.

ARC 11.05 adds a new client that can be installed on both Microsoft Windows and Mac OS X. It can also submit jobs both through the old gridftp interface of ARC 0.8 as well as through the new, more standard-conforming, WS-based interface. Old client tools (as in ARC 0.8 series) are available still, as a compatibility package. Standalone client tar-ball includes both old and new clients.

The Grid Manager based computing element has been replaced by a re-factored and improved version called A-REX (ARC Resource-coupled EXecution service).

Information system now supports publishing information following the GLUE2 standard, and support for GLUE 1.2 is improved for compatibility with gLite within EMI.

As an experimental feature, new data staging framework is included in A-REX. It aims to address shortcomings presented by the older uploader/downloader methods, but so far it is turned off by default.

A number of other new features are introduced (see the following sections), prompting some changes in the configuration. More details can be found in migration guides.

IMPORTANT change: With this ARC release, the Grid Manager based compute element will reach its End of Life support. Please (re-)read the previously communicated change and implications.

ARC components: detailed new features and deployment notes

ARC components term is introduced to denote logical grouping of tools and services. The components are:

For installation purposes, they are grouped into meta-packages (gridftpd server is included into Compute Element meta-package, and Core is a common dependency).

Apart of these, the following services are available:

The following changes are common for several components:

ARC Core

New features:

Deployment notes:

Note that this installation removes grid-crl-update and replaces it with fetch-crl which renders the nordugrid-arc-ca-utils package empty. For backward compatibility, fetch-crl is enabled through cron.

ARC Compute Element

New features:

Deployment notes:

For sites deploying ARC 0.8.3 or ARC 0.8.3.1 releases, a smooth upgrade path via Yum is provided. Since this will imply switch from Grid Manager to A-REX, careful check is advised. Please refer to Grid Manager EOL notes for details.

To deploy a functional computing element with the current production interfaces one needs to install the new nordugrid-arc-arex package together with ARIS and gridftp server packages, or the nordugrid-arc-compute-element meta-package. ARIS and gridftp are necessary to provide backward interface compatibility. It is also recommended to deploy the gridmap-utils package (included in the meta-package).

To start the A-REX service use /etc/init.d/a-rex start (there is a compatibility grid-manager symlink provided, too).

By default, A-REX uses configuration file from the old Grid Manager. Configurations taken from previous ARC CE deployments should work out of the box, though there were numerous changes to the configuration parameters and options. None of the changes is backward incompatible. Details are available in migration notes (see Additional documents section).

None of the experimental features is turned on by default

ARC Information System

New features:

Deployment notes:

ARIS is deployed by installing the nordugrid-arc-aris package. If you are upgrading from an earlier release, the update should be automatic from any of the previous packages (nordugrid-arc-infosys-ldap, nordugrid-arc-nox-ldap-infosys).

EGIIS is deployed by installing the nordugrid-arc-egiis package. If you are updating from an earlier release, the update from nordugrid-arc-nox-giis should be automatic. A meta package nordugrid-arc-information-index is also provided for easy deployment.

In order to keep publishing the same information as in GRIS before, no configuration changes are needed.

New configuration options were introduced for enabling publishing new information in ARIS. In order to enable or disable different information schemas in the output, the following variables can be set in the [infosys] section of the arc.conf (defaults as indicated below): infosys_nordugrid="enable"
infosys_glue12="disable"
infosys_glue2_ldap="disable"
 In order to publish site BDII information a new [infosys/site/] section should be added to the arc.conf configuration file, containing a unique_id option identifying the site and a URL option giving the address to the resource BDII.

ARC Clients

New features:

Deployment notes:

The new clients (arc*) are available from the nordugrid-arc-client package.

The behaviour and interface of the new arc* clients slightly differ from the old ones, so for compatibility reasons the ng* still distributed in the nordugrid-arc-compat package

The libarcclient and libarcdata2 libraries are available from the nordugrid-arc package.

The DMCs and different plugins are available from nordugrid-arc-plugins-needed and nordugrid-arc-plugins-globus (the latter requires a number of Globus packages).

The old arclib is now replaced by the new libraries and tagged as obsolete (though still available).

ARC gridftp server

New features:

Fixed bugs

In addition, a few documentation-related issues are fixed.

Known issues

README file distributed with the main package contains some obsolete information.

GLUE2 XML document published by A-REX is incomplete due to the absence of official XML rendering.

The new arc*-commands are not 100% compatible with the old ng* commands. Consult NorduGrid Wiki for details.

Recursive copy using ngcp -r deletes existing files (see bug 2132). Use caution when using ngcp -r or use arccp -r instead.

Source code distributed by EMI can only be built using SWIG 1.3.29 (not above) If your system has a newer SWIG, please use source distributed by NorduGrid instead.

Client error messages are still suboptimal; please report inadequate messages to NorduGrid Bugzilla.

There are additional non-critical smaller issues registered in NorduGrid Bugzilla; before reporting a bug, always try to cross-check whether it was reported already.

Availability

Source

ARC release 11.05 consists of the following source packages:

Source code for main components is available from here: svn co http://svn.nordugrid.org/repos/nordugrid/arc1/tags/1.0.0 arc-1.0.0

Source for the compatibility package (old client) is available from: svn co \
http://svn.nordugrid.org/repos/nordugrid/arc0/tags/compat_1.0.0 arc_compat-1.0.0

Documentation source (mostly LaTeX) is available from: svn co http://svn.nordugrid.org/repos/nordugrid/doc/tags/1.0.1 arc_doc-1.0.1

GUI arcjobtool is available from: svn co \
http://laportal.svn.sourceforge.net/svnroot/laportal/tags/arcjobtool-0.3.0-arc/\
arcjobtool_0.3.0

Repositories

See detailed description at the NorduGrid site.

These repositories provide binary packages for:

Scientific Linux and CentOS are implicitly supported through corresponding RedHat repositories.

Microsoft Windows and Mac OS X versions are available from same repositories for clients and some services.

Standalone client tar-balls for Linux are available from respective repositories for each flavor.

Additional documents

NorduGrid homepage