NorduGrid Note


proposal

2003-05-18




Replication Tool

Oxana Smirnova*



A wish list for the real replication tool, no matter whether it uses RC, or RLS, or whatever else for replica management service.






  1. Definitions

    Replica

    A copy of an original data (usually, a file)

    Collection

    A set of data (files), sharing certain common features

    Location

    Physical address for a file or a set of files; a collection may span several locations

    Server

    In this context, a replica database server. Such a database is assumed to contain lists of collections, their contents, and physical locations (handles) for each replica

    <name>

    A human-readable string

    <server contact>

    A string identifying server location and other necessary attributes

    <metadata file>

    A (formatted) file containing metadata associated to a replica

  2. Functionality

Let's call this tool "ngreplica". The purpose of it is to copy, rename, move and delete files located on the Grid. Operations should be performed not only on single files, but on groups of files as well, specified either by a wildcard expression, or as a logical collection (logical dataset).

Parallellism option should be there, it is just omitted it from the examples, but something like "-p <nr of threads>" should be one of the options.

Third-party transfers should be enabled, too.

Database server locations, ports, distinguished names or whatever should be read by ngreplica from, e.g., a configuration file (.ngreplicarc), and not entered from a command line (such an option should be kept for advanced users).



  1. List collections on a server (default is the one in .ngreplicarc file, otherwise a contact string is specified, <server contact>)

    ngreplica --list-collections <name with wildcards> [-s <server contact>]

  2. Add a collection

    ngreplica --add-collection <name> [-s <server contact>]

  3. Rename a collection

    ngreplica --rename-collection <old name>=<new name> [-s <server contact>]

  4. Remove a collection (optionally erasing its contents physically)

    ngreplica --delete-collection <name> [--erase-contents] [-s <server contact>]

  5. List locations for a collection (or whatever is an analogy):

    ngreplica --list-locations <name with wildcards> -c <collection name> [-s <server contact>]

  6. Add a location

    ngreplica --add-location <name>:<location> -c <collection name> [-s <server contact>]

  7. Rename a location

    ngreplica --rename-location <old name>=<new name> -c <collection name> [-s <server contact>]

  8. Remove a location

    ngreplica --delete-location <name> -c <collection name> [-s <server contact>]

  9. List LFNs in a collection

    ngreplica --list-lfn <name with wildcards> -c <collection name> [-s <server contact>]

  10. Add a file replica to a collection:

    ngreplica --replicate <URL> -c <collection name> [-l <LFN>] [-s <server contact>]

  1. Simple copy

    ngreplica --copy-file <URL1> <URL2>

  2. Get file; with wildcard names for files and collections

    ngreplica --get-file -l <LFN> <local file name> [-c <collection name>] [-s <server contact>]

  3. List file attributes, including metadata and locations, for an LFN

    ngreplica --list-file -l <LFN> -c <collection name> [-s <server contact>]

  4. Delete physical location(s) for an LFN (assume they are numbered or tagged somehow); if all are deleted, LFN is erased, too

    ngreplica --delete-file -l <LFN> -n <number(s)> -c <collection name> [-s <server contact>]

  5. Rename LFN

    ngreplica --rename-lfn <old LFN>=<new LFN> -c <collection name> [-s <server contact>]

  6. Delete LFN (optionally, with files)

    ngreplica --delete-lfn <LFN> -c <collection name> [--erase-files] [-s <server contact>]

  7. Move a file to another collection

    ngreplica --move-collections <old collection>=<new collection> -l <LFN> [-s <server contact>]

  8. Add metadata to an LFN (in case they are not there - should be able to start a metadata discovery "spider", or take the data from a file, <metadata file>)

ngreplica --add-metadata -l <LFN> -c <collection name> [-f <metadata file>] [-s <server contact>]