About Advanced Resource Connector

The Advanced Resource Connector (ARC) middleware integrates computing resources (usually, computing clusters managed by a batch system or standalone workstations) and storage facilities, making them available via a secure common Grid layer. Complete documentation, including component descriptions, installation details, usage instructions, articles and presentations, can be found at the documentation page.

Until recently, ARC relied on largely proprietary technologies predating Web Service ones. Today, European Middleware Initiative replaces all pre-WS ARC components with the new ones. For backwards compatibility with deployed services, both pre-WS and WS-technologies are available in EMI ARC releases.

Background

The ARC middleware by NorduGrid (in the past called NorduGrid middleware) is a software solution that uses Grid technologies to enable sharing and federation of computing and storage resources distributed across different administrative and application domains. ARC is used to create Grid infrastructures of various scope and complexity, from campus to national Grids. It is distributed under the Apache v2.0 license.

In a nutshell, ARC is:

• A general purpose, Open Source, lightweight, portable middleware solution
• A reliable production-quality and scalable implementation of fundamental Grid services
• Facilitator of cross-organisational distributed computing infrastructure solutions
• Strongly commited to open standards and interoperability
• In continuous use since 2002

Selected ARC components

• ARC Compute Element (ARC CE)
• HED
• Chelonia

ARC design principles

• User- and performance-driven development:
• start with something simple that works for users and add functionality gradually
• Simplicity is a key in planning ARC development
• No single point of failure or bottlenecks - stable by design
• Non-intrusive and easy to maintain on the server side
• Tools and utilities respect local policies, in particular, security policies
• Flexible, powerful and simple on the client side
• Provide a lightweight, stable, robust and fault-tolerant system

ARC development

• ARC development is coordinated by NorduGrid, with contributions coming from many projects, most notably: NDGF, KnowARC, EMI and others.
• Current goal is to gradually upgrade established and well performing and yet technologically inflexible components by improved and extended ones, based on Web-Service architecture
• The new ARC components features:
• Service Oriented Architecture
• Modular structure
• Self-sufficient core components
• Business friendly open source license (Apache 2.0)
• Portability – runs on all Linux variants, with most components also available for Solaris, Windows and Mac OS
• Integration into Fedora (EPEL), Debian and Ubuntu

 

Internal structure of ARC components

• All communication between client and server side is done with Web services
• User clients are developed using the client libraries
• ARC offers both a command line interface (CLI), as well as a GUI
• Python and Java bindings allows easy development of application-specific clients
• On the the server side the HED (Hosting Environment Daemon) handles the communication between services and the outside world
• HED dispatches messages to services which in turn communicate with backends and through them local batch systems
• Chelonia is the ARC distributed, lightweight, self-healing storage solution.

Pre-Web-service ARC

Pre-WS ARC provides a reliable implementation of the fundamental Grid services, such as information services, resource discovery and monitoring, job submission and management, brokering and data management and resource management, using customized and largely proprietary approaches. Most of these services are provided through the security layer of the GSI. The middleware builds upon standard Open Source solutions like the OpenLDAP, OpenSSL, SASL and Globus Toolkit^® (GT) libraries. Pre-WS ARC provides solutions essential for a production quality middleware: the Grid Manager, gridftpd (the ARC/NorduGrid GridFTP server), the information model and providers (NorduGrid schema), User Interface and broker (a "personal" broker integrated into the user interface), extended Resource Specification Language (xRSL), and the monitoring system.

Pre-WS ARC main components are:

1. Grid services running on the resources: the Grid Manager, gridftpd and the information services. Grid jobs are submitted to the cluster through gridftpd and a separate session directory is created for each job. The Grid session directories are made available through the gridftpd server during and after job execution. The Grid Manager is a service running on a computing resource taking care of jobs, session directories and the input data cache area. Information services are implemented as efficient scripts populating the NorduGrid information database stored in the Globus-specific OpenLDAP backends.
2. Indexing services for the resources and data: a special usage of an OpenLDAP database allows to build a hierarchical mesh of Grid-connected sites. For data indexing, ARC middleware relies on 3rd-party solutions, such as e.g. the LHC File Catalog (LFC). ARC client tools and the Grid Manager are capable of interacting with these services.
3. Clients making intelligent use of the distributed information and data available on the Grid. ARC comes with a light-weight command line client, the User Interface. The ARC User Interface is a set of tools to submit, monitor and manage jobs on the Grid, move data around and query resource information. The User Interface comes with a built-in broker, which is able to select the best matching resource for a job. Grid job requirements are expressed via xRSL. Another special client is the Grid Monitor, which uses any Web browser as an agent to periodically query the distributed information system and present the results as a set of inter-linked Web pages.

ARC is designed to be a scalable, non-intrusive and portable solution. The development is user- and application-driven, with the main requirements being those of performance, stability, useability and portability. As a result of this approach, the standalone client is available for a dozen of platforms and can be installed in a few minutes. The server installation does not require a full site reconfiguration. The middleware can be built on any platform where the external software packages (like GT libraries) are available. While being deployed on a large production Grid and being used by real users, the middleware is naturally undergoing continous real life tests.

Last, but not least, ARC is designed with international users in mind: the client tools are available in several languages. The Grid Monitor is translated to more than 10 languages, and the User Interface is available not just in English, but also in Russian and Swedish. Volunteers to provide more translations are always welcomed!

.

User support, feature/bug reports, documentation

User support and site installation assistance is provided via EGI support system: please report your issues using GGUS request tracker. In addition, NorduGrid runs several mailing lists among which nordugrid-discuss is a general forum for all kind of issues related to the ARC middleware.

NorduGrid deploys the Bugzilla problem tracking system. Feature and enhancement requests, as well as discovered problems, should be reported there.

Research papers, overview talks, reference manuals, user guides, installation instructions, conference presenations, FAQ and even tutorial materials can be fetched from the documents section.