DataPointDirect represents "physical" data objects. More...

#include <arc/data/DataPointDirect.h>

Inheritance diagram for Arc::DataPointDirect:

Public Member Functions
virtual bool	IsIndex () const
	Check if URL is an Indexing Service. More...

virtual bool	IsStageable () const
	Check if URL should be staged or queried for Transport URL (TURL) More...

virtual long long int	BufSize () const
	Get suggested buffer size for transfers. More...

virtual int	BufNum () const
	Get suggested number of buffers for transfers. More...

virtual bool	Local () const
	Returns true if file is local, e.g. file:// urls. More...

virtual bool	ReadOnly () const
	Returns true if file is readonly. More...

virtual void	ReadOutOfOrder (bool v)
	Allow/disallow DataPoint to read data out of order. More...

virtual bool	WriteOutOfOrder ()
	Returns true if DataPoint supports receiving data out of order during writing. More...

virtual void	SetAdditionalChecks (bool v)
	Allow/disallow additional checks on a source DataPoint before transfer. More...

virtual bool	GetAdditionalChecks () const
	Returns true unless SetAdditionalChecks() was set to false. More...

virtual void	SetSecure (bool v)
	Allow/disallow heavy security (data encryption) during data transfer. More...

virtual bool	GetSecure () const
	Returns true if heavy security during data transfer is allowed. More...

virtual void	Passive (bool v)
	Set passive transfers for FTP-like protocols. More...

virtual void	Range (unsigned long long int start=0, unsigned long long int end=0)
	Set range of bytes to retrieve. More...

virtual int	AddCheckSumObject (CheckSum *cksum)
	Add a checksum object which will compute checksum during data transfer. More...

virtual const CheckSum *	GetCheckSumObject (int index) const
	Get CheckSum object at given position in list. More...

virtual DataStatus	Stat (std::list< FileInfo > &files, const std::list< DataPoint * > &urls, DataPointInfoType verb=INFO_TYPE_ALL)
	Retrieve information about several DataPoints. More...

virtual DataStatus	Resolve (bool source)
	Resolves index service URL into list of ordinary URLs. More...

virtual DataStatus	Resolve (bool source, const std::list< DataPoint * > &urls)
	Resolves several index service URLs. More...

virtual bool	Registered () const
	Returns true if file is registered in indexing service (only known after Resolve()) More...

virtual DataStatus	PreRegister (bool replication, bool force=false)
	Index service pre-registration. More...

virtual DataStatus	PostRegister (bool replication)
	Index service post-registration. More...

virtual DataStatus	PreUnregister (bool replication)
	Index service pre-unregistration. More...

virtual DataStatus	Unregister (bool all)
	Index service unregistration. More...

virtual bool	AcceptsMeta () const
	Check if endpoint can have any use from meta information. More...

virtual bool	ProvidesMeta () const
	Check if endpoint can provide at least some meta information directly. More...

virtual const URL &	CurrentLocation () const
	Returns current (resolved) URL. More...

virtual DataPoint *	CurrentLocationHandle () const
	Returns a pointer to the DataPoint representing the current location. More...

virtual const std::string &	CurrentLocationMetadata () const
	Returns meta information used to create current URL. More...

virtual DataStatus	CompareLocationMetadata () const
	Compare metadata of DataPoint and current location. More...

virtual bool	NextLocation ()
	Switch to next location in list of URLs. More...

virtual bool	LocationValid () const
	Returns false no more locations are left and out of retries. More...

virtual bool	HaveLocations () const
	Returns true if number of resolved URLs is not 0. More...

virtual bool	LastLocation ()
	Returns true if the current location is the last. More...

virtual DataStatus	AddLocation (const URL &url, const std::string &meta)
	Add URL representing physical replica to list of locations. More...

virtual DataStatus	RemoveLocation ()
	Remove current URL from list. More...

virtual DataStatus	RemoveLocations (const DataPoint &p)
	Remove locations present in another DataPoint object. More...

virtual DataStatus	ClearLocations ()
	Remove all locations. More...

virtual void	SortLocations (const std::string &, const URLMap &)
	Sort locations according to the specified pattern and URLMap. More...

Public Member Functions inherited from Arc::DataPoint
virtual	~DataPoint ()
	Destructor. More...

virtual const URL &	GetURL () const
	Returns the URL that was passed to the constructor. More...

virtual const UserConfig &	GetUserConfig () const
	Returns the UserConfig that was passed to the constructor. More...

virtual bool	SetURL (const URL &url)
	Assigns new URL. More...

virtual std::string	str () const
	Returns a string representation of the DataPoint. More...

virtual	operator bool () const
	Is DataPoint valid? More...

virtual bool	operator! () const
	Is DataPoint valid? More...

virtual DataStatus	PrepareReading (unsigned int timeout, unsigned int &wait_time)
	Prepare DataPoint for reading. More...

virtual DataStatus	PrepareWriting (unsigned int timeout, unsigned int &wait_time)
	Prepare DataPoint for writing. More...

virtual DataStatus	StartReading (DataBuffer &buffer)=0
	Start reading data from URL. More...

virtual DataStatus	StartWriting (DataBuffer &buffer, DataCallback *space_cb=NULL)=0
	Start writing data to URL. More...

virtual DataStatus	StopReading ()=0
	Stop reading. More...

virtual DataStatus	StopWriting ()=0
	Stop writing. More...

virtual DataStatus	FinishReading (bool error=false)
	Finish reading from the URL. More...

virtual DataStatus	FinishWriting (bool error=false)
	Finish writing to the URL. More...

virtual DataStatus	Check (bool check_meta)=0
	Query the DataPoint to check if object is accessible. More...

virtual DataStatus	Remove ()=0
	Remove/delete object at URL. More...

virtual DataStatus	Stat (FileInfo &file, DataPointInfoType verb=INFO_TYPE_ALL)=0
	Retrieve information about this object. More...

virtual DataStatus	List (std::list< FileInfo > &files, DataPointInfoType verb=INFO_TYPE_ALL)=0
	List hierarchical content of this object. More...

virtual DataStatus	CreateDirectory (bool with_parents=false)=0
	Create a directory. More...

virtual DataStatus	Rename (const URL &newurl)=0
	Rename a URL. More...

virtual DataStatus	GetFailureReason (void) const
	Returns reason of transfer failure, as reported by callbacks. More...

virtual bool	CheckSize () const
	Check if meta-information 'size' is available. More...

virtual void	SetSize (const unsigned long long int val)
	Set value of meta-information 'size'. More...

virtual unsigned long long int	GetSize () const
	Get value of meta-information 'size'. More...

virtual bool	CheckCheckSum () const
	Check if meta-information 'checksum' is available. More...

virtual void	SetCheckSum (const std::string &val)
	Set value of meta-information 'checksum'. More...

virtual const std::string &	GetCheckSum () const
	Get value of meta-information 'checksum'. More...

virtual const std::string	DefaultCheckSum () const
	Default checksum type (varies by protocol) More...

virtual bool	CheckModified () const
	Check if meta-information 'modification time' is available. More...

virtual void	SetModified (const Time &val)
	Set value of meta-information 'modification time'. More...

virtual const Time &	GetModified () const
	Get value of meta-information 'modification time'. More...

virtual bool	CheckValid () const
	Check if meta-information 'validity time' is available. More...

virtual void	SetValid (const Time &val)
	Set value of meta-information 'validity time'. More...

virtual const Time &	GetValid () const
	Get value of meta-information 'validity time'. More...

virtual void	SetAccessLatency (const DataPointAccessLatency &latency)
	Set value of meta-information 'access latency'. More...

virtual DataPointAccessLatency	GetAccessLatency () const
	Get value of meta-information 'access latency'. More...

virtual bool	Cache () const
	Returns true if file is cacheable. More...

virtual int	GetTries () const
	Returns number of retries left. More...

virtual void	SetTries (const int n)
	Set number of retries. More...

virtual void	NextTry ()
	Decrease number of retries left. More...

virtual bool	RequiresCredentials () const
	Returns true if some kind of credentials are needed to use this DataPoint. More...

virtual bool	RequiresCredentialsInFile () const
	Returns true if credentials must be stored in files. More...

virtual void	SetMeta (const DataPoint &p)
	Copy meta information from another object. More...

virtual bool	CompareMeta (const DataPoint &p) const
	Compare meta information from another object. More...

virtual std::vector< URL >	TransferLocations () const
	Returns physical file(s) to read/write, if different from CurrentLocation() More...

virtual void	ClearTransferLocations ()
	Clear list of physical file(s) to read/write. More...

virtual void	AddURLOptions (const std::map< std::string, std::string > &options)
	Add URL options to this DataPoint's URL object. More...

Protected Member Functions
	DataPointDirect (const URL &url, const UserConfig &usercfg, PluginArgument *parg)

Protected Member Functions inherited from Arc::DataPoint
	DataPoint (const URL &url, const UserConfig &usercfg, PluginArgument *parg)
	Constructor. More...

virtual DataStatus	Transfer3rdParty (const URL &source, const URL &destination, Callback3rdParty callback=NULL)
	Perform third party transfer. More...

Protected Attributes
DataBuffer *	buffer

long long int	bufsize

int	bufnum

bool	local

bool	readonly

bool	linkable

bool	is_secure

bool	force_secure

bool	force_passive

bool	additional_checks

bool	allow_out_of_order

unsigned long long int	range_start

unsigned long long int	range_end

std::list< CheckSum * >	checksums

Protected Attributes inherited from Arc::DataPoint
URL	url
	URL supplied in constructor. More...

const UserConfig	usercfg
	UserConfig supplied in constructor. More...

unsigned long long int	size
	Size of object represented by DataPoint. More...

std::string	checksum
	Checksum of object represented by DataPoint. More...

Time	modified
	Modification time of object represented by DataPoint. More...

Time	valid
	Validity time of object represented by DataPoint. More...

DataPointAccessLatency	access_latency
	Access latency of object represented by DataPoint. More...

int	triesleft
	Retries left for data transfer. More...

DataStatus	failure_code
	Result of data read/write carried out in separate thread. More...

bool	cache
	Whether this DataPoint is cacheable. More...

bool	stageable
	Whether this DataPoint requires staging. More...

std::set< std::string >	valid_url_options
	Valid URL options. Subclasses should add their own specific options to this list. More...

Additional Inherited Members
Public Types inherited from Arc::DataPoint
enum	DataPointAccessLatency { ACCESS_LATENCY_ZERO, ACCESS_LATENCY_SMALL, ACCESS_LATENCY_LARGE }
	Describes the latency to access this URL. More...

enum	DataPointInfoType { INFO_TYPE_MINIMAL = 0, INFO_TYPE_NAME = 1, INFO_TYPE_TYPE = 2, INFO_TYPE_TIMES = 4, INFO_TYPE_CONTENT = 8, INFO_TYPE_ACCESS = 16, INFO_TYPE_STRUCT = 32, INFO_TYPE_REST = 64, INFO_TYPE_ALL = 127 }
	Describes type of information about URL to request. More...

typedef void(*	Callback3rdParty )(unsigned long long int bytes_transferred)
	Callback for use in 3rd party transfer. More...

Static Public Member Functions inherited from Arc::DataPoint
static DataStatus	Transfer3rdParty (const URL &source, const URL &destination, const UserConfig &usercfg, Callback3rdParty callback=NULL)
	Perform third party transfer. More...

Static Protected Attributes inherited from Arc::DataPoint
static Logger	logger
	Logger object. More...

Detailed Description

DataPointDirect represents "physical" data objects.

This class should never be used directly, instead inherit from it to provide a class for a specific access protocol.

Member Function Documentation

virtual bool Arc::DataPointDirect::AcceptsMeta ( ) const

virtual

Check if endpoint can have any use from meta information.

Implements Arc::DataPoint.

virtual int Arc::DataPointDirect::AddCheckSumObject ( CheckSum * cksum )

virtual

Add a checksum object which will compute checksum during data transfer.

Parameters

cksum object which will compute checksum. Should not be destroyed until DataPointer itself.

Returns: integer position in the list of checksum objects.

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::AddLocation	(	const URL &	url,
		const std::string &	meta
	)

virtual

Add URL representing physical replica to list of locations.

Parameters

url	Location URL to add.
meta	Location meta information.

Returns: LocationAlreadyExistsError if location already exists, otherwise success

Implements Arc::DataPoint.

virtual int Arc::DataPointDirect::BufNum ( ) const

virtual

Get suggested number of buffers for transfers.

Implements Arc::DataPoint.

virtual long long int Arc::DataPointDirect::BufSize ( ) const

virtual

Get suggested buffer size for transfers.

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::ClearLocations ( )

virtual

Remove all locations.

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::CompareLocationMetadata ( ) const

virtual

Compare metadata of DataPoint and current location.

Returns: inconsistency error or error encountered during operation, or success

Implements Arc::DataPoint.

virtual const URL& Arc::DataPointDirect::CurrentLocation ( ) const

virtual

Returns current (resolved) URL.

Implements Arc::DataPoint.

virtual DataPoint* Arc::DataPointDirect::CurrentLocationHandle ( ) const

virtual

Returns a pointer to the DataPoint representing the current location.

Implements Arc::DataPoint.

virtual const std::string& Arc::DataPointDirect::CurrentLocationMetadata ( ) const

virtual

Returns meta information used to create current URL.

Usage differs between different indexing services.

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::GetAdditionalChecks ( ) const

virtual

Returns true unless SetAdditionalChecks() was set to false.

Implements Arc::DataPoint.

virtual const CheckSum* Arc::DataPointDirect::GetCheckSumObject ( int index ) const

virtual

Get CheckSum object at given position in list.

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::GetSecure ( ) const

virtual

Returns true if heavy security during data transfer is allowed.

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::HaveLocations ( ) const

virtual

Returns true if number of resolved URLs is not 0.

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::IsIndex ( ) const

virtual

Check if URL is an Indexing Service.

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::IsStageable ( ) const

virtual

Check if URL should be staged or queried for Transport URL (TURL)

Reimplemented from Arc::DataPoint.

virtual bool Arc::DataPointDirect::LastLocation ( )

virtual

Returns true if the current location is the last.

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::Local ( ) const

virtual

Returns true if file is local, e.g. file:// urls.

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::LocationValid ( ) const

virtual

Returns false no more locations are left and out of retries.

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::NextLocation ( )

virtual

Switch to next location in list of URLs.

At last location switch to first if number of allowed retries is not exceeded.

Returns: false if no retries left.

Implements Arc::DataPoint.

virtual void Arc::DataPointDirect::Passive ( bool v )

virtual

Set passive transfers for FTP-like protocols.

Parameters

v	true if passive should be used.

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::PostRegister ( bool replication )

virtual

Index service post-registration.

Used for same purpose as PreRegister. Should be called after actual transfer of file successfully finished to finalise registration in an index service.

Parameters

replication if true, the file is being replicated between two locations registered in Indexing Service under the same name.

Returns: success if post-registration succeeded

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::PreRegister	(	bool	replication,
		bool	force = `false`
	)

virtual

Index service pre-registration.

This function registers the physical location of a file into an indexing service. It should be called before the actual transfer to that location happens.

Parameters

replication	if true, the file is being replicated between two locations registered in the indexing service under the same name.
force	if true, perform registration of a new file even if it already exists. Should be used to fix failures in indexing service.

Returns: success if pre-registration succeeded

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::PreUnregister ( bool replication )

virtual

Index service pre-unregistration.

Should be called if file transfer failed. It removes changes made by PreRegister().

Parameters

replication if true, the file is being replicated between two locations registered in Indexing Service under the same name.

Returns: success if pre-unregistration succeeded

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::ProvidesMeta ( ) const

virtual

Check if endpoint can provide at least some meta information directly.

Implements Arc::DataPoint.

virtual void Arc::DataPointDirect::Range	(	unsigned long long int	start = `0`,
		unsigned long long int	end = `0`
	)

virtual

Set range of bytes to retrieve.

Default values correspond to whole file. Both start and end bytes are included in the range, i.e. start - end + 1 bytes will be read.

Parameters

start	byte to start from
end	byte to end at

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::ReadOnly ( ) const

virtual

Returns true if file is readonly.

Implements Arc::DataPoint.

virtual void Arc::DataPointDirect::ReadOutOfOrder ( bool v )

virtual

Allow/disallow DataPoint to read data out of order.

If set to true then data may be read from source out of order or in parallel from multiple threads. For a transfer between two DataPoints this should only be set to true if WriteOutOfOrder() returns true for the destination. Only certain protocols support this option.

Parameters

v	true if allowed (default is false).

Implements Arc::DataPoint.

virtual bool Arc::DataPointDirect::Registered ( ) const

virtual

Returns true if file is registered in indexing service (only known after Resolve())

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::RemoveLocation ( )

virtual

Remove current URL from list.

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::RemoveLocations ( const DataPoint & p )

virtual

Remove locations present in another DataPoint object.

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::Resolve ( bool source )

virtual

Resolves index service URL into list of ordinary URLs.

Also obtains meta information about the file if possible. Resolve should be called for both source and destination URLs before a transfer. If source is true an error is returned if the file does not exist.

Parameters

source true if DataPoint object represents source of information.

Returns: success if DataPoint was successfully resolved

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::Resolve	(	bool	source,
		const std::list< DataPoint * > &	urls
	)

virtual

Resolves several index service URLs.

Can use bulk calls if protocol allows. The protocols and hosts of all the DataPoints in urls must be the same and the same as this DataPoint's protocol and host. This method can be called on any of the urls, for example urls.front()->Resolve(true, urls);

Parameters

source	true if DataPoint objects represent source of information
urls	List of DataPoints to resolve. Protocols and hosts must match and match this DataPoint's protocol and host.

Returns: success if any DataPoint was successfully resolved

Implements Arc::DataPoint.

virtual void Arc::DataPointDirect::SetAdditionalChecks ( bool v )

virtual

Allow/disallow additional checks on a source DataPoint before transfer.

If set to true, extra checks will be performed in DataMover::Transfer() before data transfer starts on for example existence of the source file (and probably other checks too).

Parameters

v	true if allowed (default is true).

Implements Arc::DataPoint.

virtual void Arc::DataPointDirect::SetSecure ( bool v )

virtual

Allow/disallow heavy security (data encryption) during data transfer.

Parameters

v	true if allowed (default depends on protocol).

Implements Arc::DataPoint.

virtual void Arc::DataPointDirect::SortLocations	(	const std::string &	pattern,
		const URLMap &	url_map
	)

inlinevirtual

Sort locations according to the specified pattern and URLMap.

See DataMover::set_preferred_pattern for a more detailed explanation of pattern matching. Locations present in url_map are preferred over others.

Parameters

pattern	a set of strings, separated by \|, to match against.
url_map	map of URLs to local URLs

Implements Arc::DataPoint.

virtual DataStatus Arc::DataPointDirect::Stat	(	std::list< FileInfo > &	files,
		const std::list< DataPoint * > &	urls,
		DataPointInfoType	verb = `INFO_TYPE_ALL`
	)

virtual

Retrieve information about several DataPoints.

If a DataPoint represents a directory or something similar, information about the object itself and not its contents will be obtained. This method can use bulk operations if the protocol supports it. The protocols and hosts of all the DataPoints in urls must be the same and the same as this DataPoint's protocol and host. This method can be called on any of the urls, for example urls.front()->Stat(files, urls); Calling this method with an empty list of urls returns success if the protocol supports bulk Stat, and an error if it does not and this can be used as a check for bulk support.

Parameters

files	will contain objects' names and requested attributes. There may be more attributes than requested. There may be less if objects can't provide particular information. The order of this list matches the order of urls. If a stat of any url fails then the corresponding FileInfo in this list will evaluate to false.
urls	list of DataPoints to stat. Protocols and hosts must match and match this DataPoint's protocol and host.
verb	defines attribute types which method must try to retrieve. It is not a failure if some attributes could not be retrieved due to limitation of protocol or access control.