ARC SDK
|
A DataPoint represents a data resource and is an abstraction of a URL. More...
#include <arc/data/DataPoint.h>
Public Types | |
enum | DataPointAccessLatency { ACCESS_LATENCY_ZERO, ACCESS_LATENCY_SMALL, ACCESS_LATENCY_LARGE } |
Describes the latency to access this URL. More... | |
enum | DataPointInfoType { INFO_TYPE_MINIMAL = 0, INFO_TYPE_NAME = 1, INFO_TYPE_TYPE = 2, INFO_TYPE_TIMES = 4, INFO_TYPE_CONTENT = 8, INFO_TYPE_CKSUM = 128, INFO_TYPE_ACCESS = 16, INFO_TYPE_STRUCT = 32, INFO_TYPE_REST = 64, INFO_TYPE_ALL = 255 } |
Describes type of information about URL to request. More... | |
typedef void(* | TransferCallback) (unsigned long long int bytes_transferred) |
Callback for use in protocol-internal or 3rd party transfers. More... | |
Public Member Functions | |
virtual DataStatus | Transfer (const URL &otherendpoint, bool source, TransferCallback callback=NULL) |
Do a transfer in a single operation. More... | |
virtual | ~DataPoint () |
Destructor. More... | |
virtual const URL & | GetURL () const |
Returns the URL that was passed to the constructor. More... | |
virtual const UserConfig & | GetUserConfig () const |
Returns the UserConfig that was passed to the constructor. More... | |
virtual bool | SetURL (const URL &url) |
Assigns new URL. More... | |
virtual std::string | str () const |
Returns a string representation of the DataPoint. More... | |
virtual | operator bool () const |
Is DataPoint valid? More... | |
virtual bool | operator! () const |
Is DataPoint valid? More... | |
virtual DataStatus | PrepareReading (unsigned int timeout, unsigned int &wait_time) |
Prepare DataPoint for reading. More... | |
virtual DataStatus | PrepareWriting (unsigned int timeout, unsigned int &wait_time) |
Prepare DataPoint for writing. More... | |
virtual DataStatus | StartReading (DataBuffer &buffer)=0 |
Start reading data from URL. More... | |
virtual DataStatus | StartWriting (DataBuffer &buffer, DataCallback *space_cb=NULL)=0 |
Start writing data to URL. More... | |
virtual DataStatus | StopReading ()=0 |
Stop reading. More... | |
virtual DataStatus | StopWriting ()=0 |
Stop writing. More... | |
virtual DataStatus | FinishReading (bool error=false) |
Finish reading from the URL. More... | |
virtual DataStatus | FinishWriting (bool error=false) |
Finish writing to the URL. More... | |
virtual DataStatus | Check (bool check_meta)=0 |
Query the DataPoint to check if object is accessible. More... | |
virtual DataStatus | Remove ()=0 |
Remove/delete object at URL. More... | |
virtual DataStatus | Stat (FileInfo &file, DataPointInfoType verb=INFO_TYPE_ALL)=0 |
Retrieve information about this object. More... | |
virtual DataStatus | Stat (std::list< FileInfo > &files, const std::list< DataPoint *> &urls, DataPointInfoType verb=INFO_TYPE_ALL)=0 |
Retrieve information about several DataPoints. More... | |
virtual DataStatus | List (std::list< FileInfo > &files, DataPointInfoType verb=INFO_TYPE_ALL)=0 |
List hierarchical content of this object. More... | |
virtual DataStatus | CreateDirectory (bool with_parents=false)=0 |
Create a directory. More... | |
virtual DataStatus | Rename (const URL &newurl)=0 |
Rename a URL. More... | |
virtual void | ReadOutOfOrder (bool v)=0 |
Allow/disallow DataPoint to read data out of order. More... | |
virtual bool | WriteOutOfOrder () const =0 |
Returns true if DataPoint supports receiving data out of order during writing. More... | |
virtual void | SetAdditionalChecks (bool v)=0 |
Allow/disallow additional checks on a source DataPoint before transfer. More... | |
virtual bool | GetAdditionalChecks () const =0 |
Returns true unless SetAdditionalChecks() was set to false. More... | |
virtual void | SetSecure (bool v)=0 |
Allow/disallow heavy security (data encryption) during data transfer. More... | |
virtual bool | GetSecure () const =0 |
Returns true if heavy security during data transfer is allowed. More... | |
virtual void | Passive (bool v)=0 |
Set passive transfers for FTP-like protocols. More... | |
virtual DataStatus | GetFailureReason (void) const |
Returns reason of transfer failure, as reported by callbacks. More... | |
virtual void | Range (unsigned long long int start=0, unsigned long long int end=0)=0 |
Set range of bytes to retrieve. More... | |
virtual DataStatus | Resolve (bool source)=0 |
Resolves index service URL into list of ordinary URLs. More... | |
virtual DataStatus | Resolve (bool source, const std::list< DataPoint *> &urls)=0 |
Resolves several index service URLs. More... | |
virtual bool | Registered () const =0 |
Returns true if file is registered in indexing service (only known after Resolve()) More... | |
virtual DataStatus | PreRegister (bool replication, bool force=false)=0 |
Index service pre-registration. More... | |
virtual DataStatus | PostRegister (bool replication)=0 |
Index service post-registration. More... | |
virtual DataStatus | PreUnregister (bool replication)=0 |
Index service pre-unregistration. More... | |
virtual DataStatus | Unregister (bool all)=0 |
Index service unregistration. More... | |
virtual bool | CheckSize () const |
Check if meta-information 'size' is available. More... | |
virtual void | SetSize (const unsigned long long int val) |
Set value of meta-information 'size'. More... | |
virtual unsigned long long int | GetSize () const |
Get value of meta-information 'size'. More... | |
virtual bool | CheckCheckSum () const |
Check if meta-information 'checksum' is available. More... | |
virtual void | SetCheckSum (const std::string &val) |
Set value of meta-information 'checksum'. More... | |
virtual const std::string & | GetCheckSum () const |
Get value of meta-information 'checksum'. More... | |
virtual const std::string | DefaultCheckSum () const |
Default checksum type (varies by protocol) More... | |
virtual bool | CheckModified () const |
Check if meta-information 'modification time' is available. More... | |
virtual void | SetModified (const Time &val) |
Set value of meta-information 'modification time'. More... | |
virtual const Time & | GetModified () const |
Get value of meta-information 'modification time'. More... | |
virtual bool | CheckValid () const |
Check if meta-information 'validity time' is available. More... | |
virtual void | SetValid (const Time &val) |
Set value of meta-information 'validity time'. More... | |
virtual const Time & | GetValid () const |
Get value of meta-information 'validity time'. More... | |
virtual void | SetAccessLatency (const DataPointAccessLatency &latency) |
Set value of meta-information 'access latency'. More... | |
virtual DataPointAccessLatency | GetAccessLatency () const |
Get value of meta-information 'access latency'. More... | |
virtual long long int | BufSize () const =0 |
Get suggested buffer size for transfers. More... | |
virtual int | BufNum () const =0 |
Get suggested number of buffers for transfers. More... | |
virtual bool | Cache () const |
Returns true if file is cacheable. More... | |
virtual bool | Local () const =0 |
Returns true if file is local, e.g. file:// urls. More... | |
virtual bool | ReadOnly () const =0 |
Returns true if file is readonly. More... | |
virtual int | GetTries () const |
Returns number of retries left. More... | |
virtual void | SetTries (const int n) |
Set number of retries. More... | |
virtual void | NextTry () |
Decrease number of retries left. More... | |
virtual bool | RequiresCredentials () const |
Returns true if some kind of credentials are needed to use this DataPoint. More... | |
virtual bool | RequiresCredentialsInFile () const |
Returns true if credentials must be stored in files. More... | |
virtual bool | IsIndex () const =0 |
Check if URL is an Indexing Service. More... | |
virtual bool | IsStageable () const |
Check if URL should be staged or queried for Transport URL (TURL) More... | |
virtual bool | SupportsTransfer () const |
Returns true if DataPoint supports internal transfer. More... | |
virtual bool | AcceptsMeta () const =0 |
Check if endpoint can have any use from meta information. More... | |
virtual bool | ProvidesMeta () const =0 |
Check if endpoint can provide at least some meta information directly. More... | |
virtual void | SetMeta (const DataPoint &p) |
Copy meta information from another object. More... | |
virtual void | ResetMeta () |
Reset meta information to default (undefined) values. More... | |
virtual bool | CompareMeta (const DataPoint &p) const |
Compare meta information from another object. More... | |
virtual std::vector< URL > | TransferLocations () const |
Returns physical file(s) to read/write, if different from CurrentLocation() More... | |
virtual void | ClearTransferLocations () |
Clear list of physical file(s) to read/write. More... | |
virtual const URL & | CurrentLocation () const =0 |
Returns current (resolved) URL. More... | |
virtual const std::string & | CurrentLocationMetadata () const =0 |
Returns meta information used to create current URL. More... | |
virtual DataPoint * | CurrentLocationHandle () const =0 |
Returns a pointer to the DataPoint representing the current location. More... | |
virtual DataStatus | CompareLocationMetadata () const =0 |
Compare metadata of DataPoint and current location. More... | |
virtual bool | NextLocation ()=0 |
Switch to next location in list of URLs. More... | |
virtual bool | LocationValid () const =0 |
Returns false no more locations are left and out of retries. More... | |
virtual bool | LastLocation ()=0 |
Returns true if the current location is the last. More... | |
virtual bool | HaveLocations () const =0 |
Returns true if number of resolved URLs is not 0. More... | |
virtual DataStatus | AddLocation (const URL &url, const std::string &meta)=0 |
Add URL representing physical replica to list of locations. More... | |
virtual DataStatus | RemoveLocation ()=0 |
Remove current URL from list. More... | |
virtual DataStatus | RemoveLocations (const DataPoint &p)=0 |
Remove locations present in another DataPoint object. More... | |
virtual DataStatus | ClearLocations ()=0 |
Remove all locations. More... | |
virtual int | AddCheckSumObject (CheckSum *cksum)=0 |
Add a checksum object which will compute checksum during data transfer. More... | |
virtual const CheckSum * | GetCheckSumObject (int index) const =0 |
Get CheckSum object at given position in list. More... | |
virtual void | SortLocations (const std::string &pattern, const URLMap &url_map)=0 |
Sort locations according to the specified pattern and URLMap. More... | |
virtual void | AddURLOptions (const std::map< std::string, std::string > &options) |
Add URL options to this DataPoint's URL object. More... | |
Static Public Member Functions | |
static DataStatus | Transfer3rdParty (const URL &source, const URL &destination, const UserConfig &usercfg, TransferCallback callback=NULL) |
Perform third party transfer. More... | |
Protected Member Functions | |
DataPoint (const URL &url, const UserConfig &usercfg, PluginArgument *parg) | |
Constructor. More... | |
virtual DataStatus | Transfer3rdParty (const URL &source, const URL &destination, TransferCallback callback=NULL) |
Perform third party transfer. More... | |
Protected Attributes | |
URL | url |
URL supplied in constructor. More... | |
const UserConfig | usercfg |
UserConfig supplied in constructor. More... | |
unsigned long long int | size |
Size of object represented by DataPoint. More... | |
std::string | checksum |
Checksum of object represented by DataPoint. More... | |
Time | modified |
Modification time of object represented by DataPoint. More... | |
Time | valid |
Validity time of object represented by DataPoint. More... | |
DataPointAccessLatency | access_latency |
Access latency of object represented by DataPoint. More... | |
int | triesleft |
Retries left for data transfer. More... | |
DataStatus | failure_code |
Result of data read/write carried out in separate thread. More... | |
bool | cache |
Whether this DataPoint is cacheable. More... | |
bool | stageable |
Whether this DataPoint requires staging. More... | |
std::set< std::string > | valid_url_options |
Valid URL options. Subclasses should add their own specific options to this list. More... | |
Static Protected Attributes | |
static Logger | logger |
Logger object. More... | |
A DataPoint represents a data resource and is an abstraction of a URL.
DataPoint uses ARC's Plugin mechanism to dynamically load the required Data Manager Component (DMC) when necessary. A DMC typically defines a subclass of DataPoint (e.g. DataPointHTTP) and is responsible for a specific protocol (e.g. http). DataPoints should not be used directly, instead the DataHandle wrapper class should be used, which automatically loads the correct DMC. Examples of how to use DataPoint methods are shown in the DataHandle documentation.
DataPoint defines methods for access to the data resource. To transfer data between two DataPoints, DataMover::Transfer() can be used.
There are two subclasses of DataPoint, DataPointDirect and DataPointIndex. None of these three classes can be instantiated directly. DataPointDirect and its subclasses handle "physical" resources through protocols such as file, http and gsiftp. These classes implement methods such as StartReading() and StartWriting(). DataPointIndex and its subclasses handle resources such as indexes and catalogs and implement methods like Resolve() and PreRegister().
When creating a new DMC, a subclass of either DataPointDirect or DataPointIndex should be created, and the appropriate methods implemented. DataPoint itself has no direct external dependencies, but plugins may rely on third-party components. The new DMC must also add itself to the list of available plugins and provide an Instance() method which returns a new instance of itself, if the supplied arguments are valid for the protocol. Here is an example skeleton implementation of a new DMC for protocol MyProtocol which represents a physical resource accessible through protocol my://
typedef void(* Arc::DataPoint::TransferCallback) (unsigned long long int bytes_transferred) |
Callback for use in protocol-internal or 3rd party transfers.
Will be called periodically during the transfer with the number of bytes transferred so far.
bytes_transferred | the number of bytes transferred so far |
Describes the latency to access this URL.
For now this value is one of a small set specified by the enumeration. In the future with more sophisticated protocols or information it could be replaced by a more fine-grained list of possibilities such as an int value.
Enumerator | |
---|---|
ACCESS_LATENCY_ZERO | URL can be accessed instantly. |
ACCESS_LATENCY_SMALL | URL has low (but non-zero) access latency, for example staged from disk. |
ACCESS_LATENCY_LARGE | URL has a large access latency, for example staged from tape. |
Describes type of information about URL to request.
|
virtual |
Destructor.
|
protected |
|
pure virtual |
Check if endpoint can have any use from meta information.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
Add a checksum object which will compute checksum during data transfer.
cksum | object which will compute checksum. Should not be destroyed until DataPointer itself. |
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
pure virtual |
Add URL representing physical replica to list of locations.
url | Location URL to add. |
meta | Location meta information. |
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
virtual |
|
pure virtual |
Get suggested number of buffers for transfers.
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
pure virtual |
Get suggested buffer size for transfers.
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
virtual |
Returns true if file is cacheable.
|
pure virtual |
Query the DataPoint to check if object is accessible.
If check_meta is true this method will also try to provide meta information about the object. Note that for many protocols an access check also provides meta information and so check_meta may have no effect.
check_meta | If true then the method will try to retrieve meta data during the check. |
Implemented in Arc::DataPointDelegate, and Arc::DataPointIndex.
|
virtual |
Check if meta-information 'checksum' is available.
|
virtual |
Check if meta-information 'modification time' is available.
|
virtual |
Check if meta-information 'size' is available.
|
virtual |
Check if meta-information 'validity time' is available.
|
pure virtual |
Remove all locations.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
inlinevirtual |
Clear list of physical file(s) to read/write.
To be used with protocols which re-direct to different URLs such as Transport URLs (TURLs). Has no effect for other protocols.
Reimplemented in Arc::DataPointIndex.
|
pure virtual |
Compare metadata of DataPoint and current location.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
virtual |
Compare meta information from another object.
Undefined values are not used for comparison.
p | object to which to compare. |
|
pure virtual |
Create a directory.
If the protocol supports it, this method creates the last directory in the path to the URL. It assumes the last component of the path is a file-like object and not a directory itself, unless the path ends in a directory separator. If with_parents is true then all missing parent directories in the path will also be created. The access control on the new directories is protocol-specific and may vary depending on protocol.
with_parents | If true then all missing directories in the path are created |
Implemented in Arc::DataPointDelegate.
|
pure virtual |
Returns current (resolved) URL.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
Returns a pointer to the DataPoint representing the current location.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
Returns meta information used to create current URL.
Usage differs between different indexing services.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
virtual |
Default checksum type (varies by protocol)
Reimplemented in Arc::DataPointDelegate.
|
virtual |
Finish reading from the URL.
Must be called after transfer of physical file has completed if PrepareReading() was called, to free resources, release requests that were made during preparation etc.
error | If true then action is taken depending on the error. |
Reimplemented in Arc::DataPointIndex.
|
virtual |
Finish writing to the URL.
Must be called after transfer of physical file has completed if PrepareWriting() was called, to free resources, release requests that were made during preparation etc.
error | if true then action is taken depending on the error, for example cleaning the file from the storage |
Reimplemented in Arc::DataPointIndex.
|
virtual |
Get value of meta-information 'access latency'.
Reimplemented in Arc::DataPointIndex.
|
pure virtual |
Returns true unless SetAdditionalChecks() was set to false.
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
virtual |
Get value of meta-information 'checksum'.
|
pure virtual |
Get CheckSum object at given position in list.
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
virtual |
Returns reason of transfer failure, as reported by callbacks.
This could be different from the failure returned by the methods themselves.
|
virtual |
Get value of meta-information 'modification time'.
|
pure virtual |
Returns true if heavy security during data transfer is allowed.
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
virtual |
Get value of meta-information 'size'.
|
virtual |
Returns number of retries left.
|
virtual |
Returns the URL that was passed to the constructor.
|
virtual |
Returns the UserConfig that was passed to the constructor.
|
virtual |
Get value of meta-information 'validity time'.
|
pure virtual |
Returns true if number of resolved URLs is not 0.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
Check if URL is an Indexing Service.
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
virtual |
Check if URL should be staged or queried for Transport URL (TURL)
Reimplemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
pure virtual |
Returns true if the current location is the last.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
List hierarchical content of this object.
If the DataPoint represents a directory or something similar its contents will be listed and put into files. If the DataPoint is file- like an error will be returned.
files | will contain list of file names and requested attributes. There may be more attributes than requested. There may be less if object can't provide particular information. |
verb | defines attribute types which method must try to retrieve. It is not a failure if some attributes could not be retrieved due to limitation of protocol or access control. |
Implemented in Arc::DataPointDelegate.
|
pure virtual |
Returns true if file is local, e.g. file:// urls.
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
pure virtual |
Returns false no more locations are left and out of retries.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
Switch to next location in list of URLs.
At last location switch to first if number of allowed retries is not exceeded.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
virtual |
Decrease number of retries left.
|
virtual |
Is DataPoint valid?
|
virtual |
Is DataPoint valid?
|
pure virtual |
Set passive transfers for FTP-like protocols.
v | true if passive should be used. |
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
pure virtual |
Index service post-registration.
Used for same purpose as PreRegister. Should be called after actual transfer of file successfully finished to finalise registration in an index service.
replication | if true, the file is being replicated between two locations registered in Indexing Service under the same name. |
Implemented in Arc::DataPointDirect.
|
virtual |
Prepare DataPoint for reading.
This method should be implemented by protocols which require preparation or staging of physical files for reading. It can act synchronously or asynchronously (if protocol supports it). In the first case the method will block until the file is prepared or the specified timeout has passed. In the second case the method can return with a ReadPrepareWait status before the file is prepared. The caller should then wait some time (a hint from the remote service may be given in wait_time) and call PrepareReading() again to poll for the preparation status, until the file is prepared. In this case it is also up to the caller to decide when the request has taken too long and if so cancel it by calling FinishReading(). When file preparation has finished, the physical file(s) to read from can be found from TransferLocations().
timeout | If non-zero, this method will block until either the file has been prepared successfully or the timeout has passed. A zero value means that the caller would like to call and poll for status. |
wait_time | If timeout is zero (caller would like asynchronous operation) and ReadPrepareWait is returned, a hint for how long to wait before a subsequent call may be given in wait_time. |
Reimplemented in Arc::DataPointIndex.
|
virtual |
Prepare DataPoint for writing.
This method should be implemented by protocols which require preparation of physical files for writing. It can act synchronously or asynchronously (if protocol supports it). In the first case the method will block until the file is prepared or the specified timeout has passed. In the second case the method can return with a WritePrepareWait status before the file is prepared. The caller should then wait some time (a hint from the remote service may be given in wait_time) and call PrepareWriting() again to poll for the preparation status, until the file is prepared. In this case it is also up to the caller to decide when the request has taken too long and if so cancel or abort it by calling FinishWriting(true). When file preparation has finished, the physical file(s) to write to can be found from TransferLocations().
timeout | If non-zero, this method will block until either the file has been prepared successfully or the timeout has passed. A zero value means that the caller would like to call and poll for status. |
wait_time | If timeout is zero (caller would like asynchronous operation) and WritePrepareWait is returned, a hint for how long to wait before a subsequent call may be given in wait_time. |
Reimplemented in Arc::DataPointIndex.
|
pure virtual |
Index service pre-registration.
This function registers the physical location of a file into an indexing service. It should be called before the actual transfer to that location happens.
replication | if true, the file is being replicated between two locations registered in the indexing service under the same name. |
force | if true, perform registration of a new file even if it already exists. Should be used to fix failures in indexing service. |
Implemented in Arc::DataPointDirect.
|
pure virtual |
Index service pre-unregistration.
Should be called if file transfer failed. It removes changes made by PreRegister().
replication | if true, the file is being replicated between two locations registered in Indexing Service under the same name. |
Implemented in Arc::DataPointDirect.
|
pure virtual |
Check if endpoint can provide at least some meta information directly.
Implemented in Arc::DataPointDelegate, Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
Set range of bytes to retrieve.
Default values correspond to whole file. Both start and end bytes are included in the range, i.e. start - end + 1 bytes will be read.
start | byte to start from |
end | byte to end at |
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
pure virtual |
Returns true if file is readonly.
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
pure virtual |
Allow/disallow DataPoint to read data out of order.
If set to true then data may be read from source out of order or in parallel from multiple threads. For a transfer between two DataPoints this should only be set to true if WriteOutOfOrder() returns true for the destination. Only certain protocols support this option.
v | true if allowed (default is false). |
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
pure virtual |
Returns true if file is registered in indexing service (only known after Resolve())
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
Remove/delete object at URL.
Implemented in Arc::DataPointDelegate, and Arc::DataPointIndex.
|
pure virtual |
Remove current URL from list.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
Remove locations present in another DataPoint object.
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
Rename a URL.
This method renames the file or directory specified in the constructor to the new name specified in newurl. It only performs namespace operations using the paths of the two URLs and in general ignores any differences in protocol and host between them. It is assumed that checks that the URLs are consistent are done by the caller of this method. This method does not do any data transfer and is only implemented for protocols which support renaming as an atomic namespace operation.
newurl | The new name for the URL |
Implemented in Arc::DataPointDelegate.
|
virtual |
Returns true if some kind of credentials are needed to use this DataPoint.
Reimplemented in Arc::DataPointDelegate.
|
virtual |
Returns true if credentials must be stored in files.
Some protocols require that credentials exist in files (in which case this method returns true), whereas others can use credentials in memory (then this method returns false).
|
virtual |
Reset meta information to default (undefined) values.
p | object from which information is taken. |
|
pure virtual |
Resolves index service URL into list of ordinary URLs.
Also obtains meta information about the file if possible. Resolve should be called for both source and destination URLs before a transfer. If source is true an error is returned if the file does not exist.
source | true if DataPoint object represents source of information. |
Implemented in Arc::DataPointDirect.
|
pure virtual |
Resolves several index service URLs.
Can use bulk calls if protocol allows. The protocols and hosts of all the DataPoints in urls must be the same and the same as this DataPoint's protocol and host. This method can be called on any of the urls, for example urls.front()->Resolve(true, urls);
source | true if DataPoint objects represent source of information |
urls | List of DataPoints to resolve. Protocols and hosts must match and match this DataPoint's protocol and host. |
Implemented in Arc::DataPointDirect.
|
virtual |
Set value of meta-information 'access latency'.
|
pure virtual |
Allow/disallow additional checks on a source DataPoint before transfer.
If set to true, extra checks will be performed in DataMover::Transfer() before data transfer starts on for example existence of the source file (and probably other checks too).
v | true if allowed (default is true). |
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
virtual |
Set value of meta-information 'checksum'.
Reimplemented in Arc::DataPointIndex.
|
virtual |
Copy meta information from another object.
Already defined values are not overwritten.
p | object from which information is taken. |
Reimplemented in Arc::DataPointIndex.
|
virtual |
Set value of meta-information 'modification time'.
|
pure virtual |
Allow/disallow heavy security (data encryption) during data transfer.
v | true if allowed (default depends on protocol). |
Implemented in Arc::DataPointIndex, and Arc::DataPointDirect.
|
virtual |
Set value of meta-information 'size'.
Reimplemented in Arc::DataPointIndex.
|
virtual |
Set number of retries.
Reimplemented in Arc::DataPointIndex.
|
virtual |
Assigns new URL.
The main purpose of this method is to reuse an existing connection for accessing a different object on the same server. The DataPoint implementation does not have to implement this method. If the supplied URL is not suitable or method is not implemented false is returned.
url | New URL |
|
virtual |
Set value of meta-information 'validity time'.
|
pure virtual |
Sort locations according to the specified pattern and URLMap.
See DataMover::set_preferred_pattern for a more detailed explanation of pattern matching. Locations present in url_map are preferred over others.
pattern | a set of strings, separated by |, to match against. |
url_map | map of URLs to local URLs |
Implemented in Arc::DataPointDirect, and Arc::DataPointIndex.
|
pure virtual |
Start reading data from URL.
A separate thread to transfer data will be created. No other operation can be performed while reading is in progress. Progress of the transfer should be followed using the DataBuffer object.
buffer | operation will use this buffer to put information into. Should not be destroyed before StopReading() was called and returned. If StopReading() is not called explicitly to release buffer it will be released in destructor of DataPoint which also usually calls StopReading(). |
Implemented in Arc::DataPointDelegate, and Arc::DataPointIndex.
|
pure virtual |
Start writing data to URL.
A separate thread to transfer data will be created. No other operation can be performed while writing is in progress. Progress of the transfer should be followed using the DataBuffer object.
buffer | operation will use this buffer to get information from. Should not be destroyed before StopWriting() was called and returned. If StopWriting() is not called explicitly to release buffer it will be released in destructor of DataPoint which also usually calls StopWriting(). |
space_cb | callback which is called if there is not enough space to store data. May not implemented for all protocols. |
Implemented in Arc::DataPointDelegate, and Arc::DataPointIndex.
|
pure virtual |
Retrieve information about this object.
If the DataPoint represents a directory or something similar, information about the object itself and not its contents will be obtained.
file | will contain object name and requested attributes. There may be more attributes than requested. There may be less if object can't provide particular information. |
verb | defines attribute types which method must try to retrieve. It is not a failure if some attributes could not be retrieved due to limitation of protocol or access control. |
Implemented in Arc::DataPointDelegate.
|
pure virtual |
Retrieve information about several DataPoints.
If a DataPoint represents a directory or something similar, information about the object itself and not its contents will be obtained. This method can use bulk operations if the protocol supports it. The protocols and hosts of all the DataPoints in urls must be the same and the same as this DataPoint's protocol and host. This method can be called on any of the urls, for example urls.front()->Stat(files, urls); Calling this method with an empty list of urls returns success if the protocol supports bulk Stat, and an error if it does not and this can be used as a check for bulk support.
files | will contain objects' names and requested attributes. There may be more attributes than requested. There may be less if objects can't provide particular information. The order of this list matches the order of urls. If a stat of any url fails then the corresponding FileInfo in this list will evaluate to false. |
urls | list of DataPoints to stat. Protocols and hosts must match and match this DataPoint's protocol and host. |
verb | defines attribute types which method must try to retrieve. It is not a failure if some attributes could not be retrieved due to limitation of protocol or access control. |
Implemented in Arc::DataPointDirect.
|
pure virtual |
Stop reading.
Must be called after corresponding StartReading() method, either after all data is transferred or to cancel transfer. Use buffer object to find out when data is transferred.
Implemented in Arc::DataPointDelegate, and Arc::DataPointIndex.
|
pure virtual |
Stop writing.
Must be called after corresponding StartWriting() method, either after all data is transferred or to cancel transfer. Use buffer object to find out when data is transferred.
Implemented in Arc::DataPointDelegate, and Arc::DataPointIndex.
|
virtual |
Returns a string representation of the DataPoint.
|
virtual |
Returns true if DataPoint supports internal transfer.
Reimplemented in Arc::DataPointIndex.
|
virtual |
Do a transfer in a single operation.
This method is designed for plugins which support doing transfers between different endpoints in a single operation, rather than two separate DataPoints communicating with each other via a DataBuffer.
endpoint | Other endpoint to transfer to or from |
source | Whether this DataPoint is the source (true) or destination (false) |
callback | Optional monitoring callback |
Reimplemented in Arc::DataPointDelegate, and Arc::DataPointIndex.
|
static |
Perform third party transfer.
Credentials are delegated to the destination and it pulls data from the source, i.e. data flows directly between source and destination instead of through the client. A callback function can be supplied to monitor progress. This method blocks until the transfer is complete. It is static because third party transfer requires different DMC plugins than those loaded by DataHandle for the same protocol. The third party transfer plugins are loaded internally in this method.
source | Source URL to pull data from |
destination | Destination URL which pulls data to itself |
usercfg | Configuration information |
callback | Optional monitoring callback |
|
protectedvirtual |
Perform third party transfer.
This method is protected because the static version should be used instead to load the correct DMC plugin for third party transfer.
source | Source URL to pull data from |
destination | Destination URL which pulls data to itself |
callback | Optional monitoring callback |
Reimplemented in Arc::DataPointDelegate.
|
virtual |
Returns physical file(s) to read/write, if different from CurrentLocation()
To be used with protocols which re-direct to different URLs such as Transport URLs (TURLs). The list is initially filled by PrepareReading and PrepareWriting. If this list is non-empty then real transfer should use a URL from this list. It is up to the caller to choose the best URL and instantiate new DataPoint for handling it. For consistency protocols which do not require redirections return original URL. For protocols which need redirection calling StartReading and StartWriting will use first URL in the list.
Reimplemented in Arc::DataPointIndex.
|
pure virtual |
Index service unregistration.
Remove information about file registered in indexing service.
all | if true, information about file itself is (LFN) is removed. Otherwise only particular physical instance in CurrentLocation() is unregistered. |
Implemented in Arc::DataPointDirect.
|
pure virtual |
Returns true if DataPoint supports receiving data out of order during writing.
Implemented in Arc::DataPointDelegate, Arc::DataPointIndex, and Arc::DataPointDirect.
|
protected |
Access latency of object represented by DataPoint.
|
protected |
Whether this DataPoint is cacheable.
|
protected |
Checksum of object represented by DataPoint.
|
protected |
Result of data read/write carried out in separate thread.
|
protected |
Size of object represented by DataPoint.
|
protected |
Whether this DataPoint requires staging.
|
protected |
Retries left for data transfer.
|
protected |
UserConfig supplied in constructor.
|
protected |
Valid URL options. Subclasses should add their own specific options to this list.