ARC SDK
Data Structures | Functions
ARC data library (libarcdata)

Detailed Description

libarcdata is a library for access to data on the Grid. It provides a uniform interface to several types of Grid storage and catalogs using various protocols. The protocols usable on a given system depend on the packages installed. The interface can be used to read, write, list, transfer and delete data to and from storage systems and catalogs.

The library uses ARC's dynamic plugin mechanism to load plugins for specific protocols only when required at runtime. These plugins are called Data Manager Components (DMCs). The DataHandle class takes care of automatically loading the required DMC at runtime to create a DataPoint object representing a resource accessible through a given protocol. DataHandle should always be used instead of DataPoint directly.

To create a new DMC for a protocol which is not yet supported see the instruction and examples in the DataPoint class documentation. This documentation also gives a complete overview of the interface.

The following protocols are currently supported in standard distributions of ARC.

DataMover provides a simple high-level interface to copy files. Fine-grained control over data transfer is shown in the following example:

#include <iostream>
#include <arc/data/DataPoint.h>
#include <arc/data/DataHandle.h>
#include <arc/data/DataBuffer.h>
using namespace Arc;
int main(int argc, char** argv) {
#define DESIRED_SIZE 512
if (argc != 2) {
std::cerr<<"Usage: partial_copy filename"<<std::endl;
return -1;
}
Arc::UserConfig usercfg;
URL url(argv[1]);
DataHandle handle(url,usercfg);
if(!handle || !(*handle)) {
std::cerr<<"Unsupported URL protocol or malformed URL"<<std::endl;
return -1;
};
handle->SetSecure(false); // GridFTP servers generally do not have encrypted data channel
FileInfo info;
if(!handle->Stat(info)) {
std::cerr<<"Failed Stat"<<std::endl;
return -1;
};
unsigned long long int fsize = handle->GetSize();
if(fsize == (unsigned long long int)-1) {
std::cerr<<"file size is not available"<<std::endl;
return -1;
};
if(fsize == 0) {
std::cerr<<"file is empty"<<std::endl;
return -1;
};
if(fsize > DESIRED_SIZE) {
handle->Range(fsize-DESIRED_SIZE,fsize-1);
};
DataBuffer buffer;
if(!handle->StartReading(buffer)) {
std::cerr<<"Failed to start reading"<<std::endl;
return -1;
};
for(;;) {
int n;
unsigned int length;
unsigned long long int offset;
if(!buffer.for_write(n,length,offset,true)) {
break;
};
std::cout<<"BUFFER: "<<offset<<": "<<length<<" :"<<std::string((const char*)(buffer[n]),length)<<std::endl;
buffer.is_written(n);
};
if(buffer.error()) {
std::cerr<<"Transfer failed"<<std::endl;
};
handle->StopReading();
return 0;
}

And the same example in python

1 import arc
2 import sys
3 
4 if len(sys.argv) != 2:
5  print "Usage: python partial_copy.py filename"
6  sys.exit(1)
7 
8 desired_size = 512
9 usercfg = arc.UserConfig()
10 url = arc.URL(sys.argv[1])
11 handle = arc.DataHandle(url,usercfg)
12 point = handle.__ref__()
13 point.SetSecure(False) # GridFTP servers generally do not have encrypted data channel
14 info = arc.FileInfo("")
15 point.Stat(info)
16 print "Name: ", info.GetName()
17 fsize = info.GetSize()
18 if fsize > desired_size:
19  point.Range(fsize-desired_size,fsize-1)
20 buffer = arc.DataBuffer()
21 point.StartReading(buffer)
22 while True:
23  n = 0
24  length = 0
25  offset = 0
26  ( r, n, length, offset, buf) = buffer.for_write(True)
27  if not r: break
28  print "BUFFER: ", offset, ": ", length, " :", buf
29  buffer.is_written(n);
30 point.StopReading()

Data Structures

class  Arc::DataBuffer
 Represents set of buffers. More...
 
class  Arc::DataCallback
 Callbacks to be used when there is not enough space on the local filesystem. More...
 
class  Arc::DataHandle
 This class is a wrapper around the DataPoint class. More...
 
class  Arc::DataMover
 DataMover provides an interface to transfer data between two DataPoints. More...
 
class  Arc::DataPoint
 A DataPoint represents a data resource and is an abstraction of a URL. More...
 
class  Arc::DataPointDirect
 DataPointDirect represents "physical" data objects. More...
 
class  Arc::DataPointIndex
 DataPointIndex represents "index" data objects, e.g. catalogs. More...
 
class  Arc::DataSpeed
 Keeps track of average and instantaneous transfer speed. More...
 
class  Arc::DataStatus
 Status code returned by many DataPoint methods. More...
 
struct  Arc::CacheParameters
 Contains data on the parameters of a cache. More...
 
class  Arc::FileCache
 FileCache provides an interface to all cache operations. More...
 
class  Arc::FileCacheHash
 FileCacheHash provides methods to make hashes from strings. More...
 
class  Arc::FileInfo
 FileInfo stores information about files (metadata). More...
 
class  Arc::URLMap
 URLMap allows mapping certain patterns of URLs to other URLs. More...
 

Functions

std::ostream & Arc::operator<< (std::ostream &o, const DataStatus &d)
 Write a human-friendly readable string with all error information to o. More...
 

Function Documentation

std::ostream& Arc::operator<< ( std::ostream &  o,
const DataStatus &  d 
)
inline

Write a human-friendly readable string with all error information to o.