ARC v0.6 Release Announcement

May 22, 2007

Advanced Resource Connector (ARC) version 0.6 is the second stable release of this grid middleware. The first stable release (v0.4) was announced on April 13 2004. After more than three years of development, NorduGrid Collaboration presents this new release that contains some new features and hundreds of bug fixes.

ARC release 0.6 is not substantially different from version 0.4, and middleware is based on the same technology and consists of the same key components. While underlying protocols are the same, there is only minimal backwards compatibility between client tools and services of the two stacks, hence an upgrade is strongly recommended.

ARC release 0.6 was available for preview and tests since December 15 2006; during this period it was shown to perform substantially better than previous versions, offering more functionality.

Description

The Advanced Resource Connector as of version 0.6 is an open source software solution that enables production quality computational grids of wide range in size and purpose. The middleware integrates computing resources (commodity computing clusters managed by a batch system or standalone workstations) and, to a lesser extent, storage services, making them available via a common secure grid layer. The middleware builds upon standard open source solutions like the OpenLDAP, OpenSSL, and Globus Toolkit 4 pre-WS libraries. It relies on well-tested pre-OGSA grid technologies in creating unique ARC-specific services and tools. ARC developers strive to achieve simplicity, non-invasiveness, high performance, stability and reliability. With release 0.6 come enhanced scalability and better conformance to community standards. ARC middleware is officialy supported on all major Linux flavors, and is known to operate smoothly on other Linux systems, with a variety of batch job management systems.

Features

ARC provides a reliable implementation of fundamental grid services, such as information services, resource discovery and monitoring, job submission and management, brokering, basic data and resource management. Most of these services are provided through the security layer of GSI.

ARC implements several services and tools essential for a production quality middleware:

• Computing Service, consisting of ARC-specific grid-manager and GridFTP server.
• Storage Service integrated with data indexing system (Smart Storage Element).
• Information system, consisting of NorduGrid information model and providers.
• Client tool implemented as a CLI with integrated resource discovery, matchmaking and brokering, making use of ARC xRSL language for job description; it also has basic data manipulation functionalities.
• Persistent computing resource usage logging system.
• Real-time monitoring system that relies entirely on the information system.

ARC main features are:

• Acknowledged simplicity in deployment and maintenance on a wide range of computing resources of various configurations.
• Highly sophisticated grid gatekeeper for computing resources (grid-manager), capable of most complex job and data management.
operations, with comprehensive security features and extensibility
• Absence of centralized workload management service.
• Reliable information system that accurately reflects up-to date system status, suitable for both resource discovery and monitoring.
• Easy to install, small and yet very powerful client tool making intelligent use of the distributed information.

Changes since ARC v0.4

New components:

• Client C++ library for unified and easy access to ARC-enabled resources for third party applications. Bindings for Python and Perl are provided as well. Basic Java library is also available.
• Optional: minimal HTTPs/g server, used by Web Services.
• Optional: Smart Storage Element with SRM-compliant interface. A Web Service. Uses GSI-secured HTTP protocol for data transer and can be integrated with data indexing systems, such as Globus RLS.
• Optional: Globus RLS is adopted as the default data indexing system.
• Optional: Logger service to store job information. A Web Service. Complete with client.

New functionalities and features:

• Support for more sources in authorization framework - VOMS, LCAS, LCMAPS, MyProxy. Extensible to any third-party authorization framework.
• Support for JSDL 1.0 with ARC-specific extensions.
• Major re-write of the client code base introduce modularity, multithreaded implementation and extensibility and localization.
• Unified configuration for ARC services and clients.
• Support for more service types in all data management tools - SRM v1.1 with both HTTPg implementations, LFC, Fireman.
• Support for more LRMS types - Condor, SGE, LSF, fork.
• Enhanced job processing algorithm - better handling of faults.
• Many new attributes and some schema changes in information system which enhance client functionality and LDAP compatibility.
• Major rewrite of Grid Monitor increases it's robustness and performance. Multiple localizations are added.
• New packaging and distribution: changed package names, complete re-arrangement of package contents, new Yum repositories, upgraded external dependencies.
• Possibility to run services as a non-privileged user.
• Overall improved usability: server logs, notifications, client command line options.
• Multiple documentation updates.

Fixed bugs:

• In total, 351 bugs were reported and fixed during development, of them 24 blocking, 14 critical, 25 major, 256 normal, 32 minor and trivial ones. While some were introduced during development, most existed in ARC 0.4. See NorduGrid bug tracking system for details.

System requirements

The middleware is tested on a variety of Linux systems. While it should work on other Unix-like systems, this release was not tested on non-Linux machines.

• For a computing service, you will need a Linux cluster running a Local Resource Management System or a standalone Linux box configured with "fork" job submission. It can be shared with other services.
• For a storage service, you will need a conventional disk array with Linux front-end, or simply a Linux box with some storage capacity. It can be shared with other services.
• For all the optional services, a shared Linux box is sufficient.
• For a client any recent Linux box will do.

Availability

The middleware is free to be deployed anywhere by anybody. Pre-built binary releases for a dozen of Linux platforms can be downloaded from http://download.nordugrid.org or via NorduGrid RPM repositories.

The software is released under GNU General Public License (GPL).

The NorduGrid repository hosts the source code, and provides all the essential external software which are not part of a standard Linux distribution.

Installation notes

NorduGrid ARC packages fall into two main kinds: the server and the client part. Server is typically installed on a computing resource by a system administrator, while the client can be set up anywhere and needs no system administrator privileges. Detailed installation instructions are distributed with the middleware documentation and are available at the Web site:

• Server: http://www.nordugrid.org/documents/ng-server-install.html
• Client: http://www.nordugrid.org/documents/ng-client-install.html

Known issues

• Data-proximity brokering disappeared.
• Non-interoperable with other known grid middlewares.
• Portability is limited by the Globus Toolkit 4 libraries which means the middleware is restricted to few Unixes and is tested only on Linux.
• Needs non-standard OpenLDAP and OpenSSL libraries (Globus legacy).
• Only partial support for clusters without a shared file system.
• No support for routing queues in PBS.
• Storage Element is limited to POSIX file systems.
• No cluster-level sandboxing is provided for grid jobs apart from the standard Unix access controls. No easily implementable fine grained authorization for computing resources.
• No access control for the information system.
• Information system represents only two services: cluster and storage.
• Inconsistent treatment for node-sharing queues.
• LDAP port and information schema hardwired in the client libary.
• No bulk data manipulation tools.
• Ad-hoc data management tools with insufficient resource description.
• Inconsistencies in service discovery and authorization.
• No official workload management system.
• No user management system.
• No GUI.
• For community authorization services, only VOMS is tested.
• For data management services, only RLS is tested.

Support

User support and site installation assistance is provided via the request tracking system available at . In addition, NorduGrid runs a couple of mailing lists, among which the nordugrid-discuss mailing list is a general forum for all kind of issues related to ARC middleware.

NorduGrid deploys the Bugzilla problem tracking system. Feature and enhancement requests, as well as problems, should be reported there.

Contact information is kept updated on the NorduGrid web site.