ARC SDK
|
ARC data staging components form a complete data transfer management system. Whereas ARC data library (libarcdata) is a library for data access, enabling several types of operation on data files on the Grid using a variety of access protocols, ARC data staging (libarcdatastaging) is a framework for managed data transfer to and from the Grid. The data staging system is designed to run as a persistent process, to execute data transfers on demand. Data transfers are defined and fed into the system, and then notification is given when they complete. No knowledge is required of the internal workings of the Grid, a user only needs to specify URLs representing the source and destination of the transfer.
The system is highly configurable and features an intelligent priority, fair-share and error handling mechanism, as well as the ability to spread data transfer across multiple hosts using ARC's DataDelivery service. It is used by ARC's Computing Element (A-REX) for pre- and post- job data transfer of input and output files. Note that this system is primarily for data transfer to and from local files and that third-party transfer is not supported. It is designed for the case of pulling or pushing data between the Grid and a local file system, rather than a service for transfer between two Grid storage elements. It is possible to transfer data between two remote endpoints, but all data flows through the client.
Simple examples of how to use libarcdatastaging are shown for several languages in the DTR examples page. In all the examples a Generator class receives as input a source and destination, and creates a DTR which describes the data transfer. It is then passed to the Scheduler and the Generator defines a receiveDTR() method for the Scheduler to calls to notify that the transfer has finished. The examples all allow using the Generator as a basic copy tool from the command line to copy a single file.
For more information see http://wiki.nordugrid.org/index.php/Data_Staging
Data Structures | |
class | DataStaging::DataDelivery |
DataDelivery transfers data between specified physical locations. More... | |
class | DataStaging::DataDeliveryComm |
This class provides an abstract interface for the Delivery layer. More... | |
struct | DataStaging::DataDeliveryComm::Status |
Plain C struct to pass information from executing process back to main thread. More... | |
class | DataStaging::DataDeliveryCommHandler |
Singleton class handling all active DataDeliveryComm objects. More... | |
class | DataStaging::DataDeliveryLocalComm |
This class starts, monitors and controls a local Delivery process. More... | |
class | DataStaging::DataDeliveryRemoteComm |
This class contacts a remote service to make a Delivery request. More... | |
class | DataStaging::TransferParameters |
Represents limits and properties of a DTR transfer. These generally apply to all DTRs. More... | |
class | DataStaging::DTRCacheParameters |
The configured cache directories. More... | |
class | DataStaging::DTRCallback |
The base class from which all callback-enabled classes should be derived. More... | |
class | DataStaging::DTR |
Data Transfer Request. More... | |
class | DataStaging::DTRList |
Global list of all active DTRs in the system. More... | |
class | DataStaging::DTRStatus |
Class representing the status of a DTR. More... | |
class | DataStaging::DTRErrorStatus |
A class to represent error states reported by various components. More... | |
class | DataStaging::Processor |
The Processor performs pre- and post-transfer operations. More... | |
class | DataStaging::Scheduler |
The Scheduler is the control centre of the data staging framework. More... | |
class | DataStaging::TransferSharesConf |
TransferSharesConf describes the configuration of TransferShares. More... | |
class | DataStaging::TransferShares |
TransferShares is used to implement fair-sharing and priorities. More... | |
Typedefs | |
typedef Arc::ThreadedPointer< DTR > | DataStaging::DTR_ptr |
Provides automatic memory management of DTRs and thread-safe destruction. More... | |
typedef Arc::ThreadedPointer < Arc::Logger > | DataStaging::DTRLogger |
The DTR's Logger object can be used outside the DTR object with DTRLogger. More... | |
Enumerations | |
enum | DataStaging::StagingProcesses { DataStaging::GENERATOR, DataStaging::SCHEDULER, DataStaging::PRE_PROCESSOR, DataStaging::DELIVERY, DataStaging::POST_PROCESSOR } |
Components of the data staging framework. More... | |
enum | DataStaging::ProcessState { DataStaging::INITIATED, DataStaging::RUNNING, DataStaging::TO_STOP, DataStaging::STOPPED } |
Internal state of StagingProcesses. More... | |
enum | DataStaging::CacheState { DataStaging::CACHEABLE, DataStaging::NON_CACHEABLE, DataStaging::CACHE_ALREADY_PRESENT, DataStaging::CACHE_DOWNLOADED, DataStaging::CACHE_LOCKED, DataStaging::CACHE_SKIP, DataStaging::CACHE_NOT_USED } |
Represents possible cache states of this DTR. More... | |
typedef Arc::ThreadedPointer<DTR> DataStaging::DTR_ptr |
Provides automatic memory management of DTRs and thread-safe destruction.
Represents possible cache states of this DTR.
Components of the data staging framework.