|
|
NorduGrid Note |
|
proposal |
|||
|
|
||
Replication Tool |
||
Oxana Smirnova* |
||
|
||
|
A wish list for the real replication tool, no matter whether it uses RC, or RLS, or whatever else for replica management service. |
|
Replica |
A copy of an original data (usually, a file) |
Collection |
A set of data (files), sharing certain common features |
Location |
Physical address for a file or a set of files; a collection may span several locations |
Server |
In this context, a replica database server. Such a database is assumed to contain lists of collections, their contents, and physical locations (handles) for each replica |
<name> |
A human-readable string |
<server contact> |
A string identifying server location and other necessary attributes |
<metadata file> |
A (formatted) file containing metadata associated to a replica |
Let's call this tool "ngreplica". The purpose of it is to copy, rename, move and delete files located on the Grid. Operations should be performed not only on single files, but on groups of files as well, specified either by a wildcard expression, or as a logical collection (logical dataset).
Parallellism option should be there, it is just omitted it from the examples, but something like "-p <nr of threads>" should be one of the options.
Third-party transfers should be enabled, too.
Database server locations, ports, distinguished names or whatever should be read by ngreplica from, e.g., a configuration file (.ngreplicarc), and not entered from a command line (such an option should be kept for advanced users).
List collections on a server (default is the one in .ngreplicarc file, otherwise a contact string is specified, <server contact>)
ngreplica --list-collections <name with wildcards> [-s <server contact>]
Add a collection
ngreplica --add-collection <name> [-s <server contact>]
Rename a collection
ngreplica --rename-collection <old name>=<new name> [-s <server contact>]
Remove a collection (optionally erasing its contents physically)
ngreplica --delete-collection <name> [--erase-contents] [-s <server contact>]
List locations for a collection (or whatever is an analogy):
ngreplica --list-locations <name with wildcards> -c <collection name> [-s <server contact>]
Add a location
ngreplica --add-location <name>:<location> -c <collection name> [-s <server contact>]
Rename a location
ngreplica --rename-location <old name>=<new name> -c <collection name> [-s <server contact>]
Remove a location
ngreplica --delete-location <name> -c <collection name> [-s <server contact>]
List LFNs in a collection
ngreplica --list-lfn <name with wildcards> -c <collection name> [-s <server contact>]
Add a file replica to a collection:
File is referred to as URL; hence the last part is the actual file name and the default logical file name
Logical file name can be assigned manually
If such a LFN already exists, the physical location is registered as another replica
If LFN does not exist, it is created - either from the file name, or from whatever was requested
ngreplica --replicate <URL> -c <collection name> [-l <LFN>] [-s <server contact>]
Simple copy
ngreplica --copy-file <URL1> <URL2>
Get file; with wildcard names for files and collections
ngreplica --get-file -l <LFN> <local file name> [-c <collection name>] [-s <server contact>]
List file attributes, including metadata and locations, for an LFN
ngreplica --list-file -l <LFN> -c <collection name> [-s <server contact>]
Delete physical location(s) for an LFN (assume they are numbered or tagged somehow); if all are deleted, LFN is erased, too
ngreplica --delete-file -l <LFN> -n <number(s)> -c <collection name> [-s <server contact>]
Rename LFN
ngreplica --rename-lfn <old LFN>=<new LFN> -c <collection name> [-s <server contact>]
Delete LFN (optionally, with files)
ngreplica --delete-lfn <LFN> -c <collection name> [--erase-files] [-s <server contact>]
Move a file to another collection
ngreplica --move-collections <old collection>=<new collection> -l <LFN> [-s <server contact>]
Add metadata to an LFN (in case they are not there - should be able to start a metadata discovery "spider", or take the data from a file, <metadata file>)
ngreplica --add-metadata -l <LFN> -c <collection name> [-f <metadata file>] [-s <server contact>]