Changes between Version 26 and Version 27 of Tutorial/DataMgt


Ignore:
Timestamp:
Sep 26, 2010, 3:24:17 PM (14 years ago)
Author:
/C=FR/O=CNRS/OU=UMR8607/CN=Charles Loomis/emailAddress=loomis@…
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Tutorial/DataMgt

    v26 v27  
    66Grid services participating to data management are the ''File Catalog'' (LFC) and the ''Storage Element'' (SE).
    77 * File Catalog: this service manages file names used by the user and associates them with replicas. Every grid file with a catalog entry can have several replicas. Catalog file names are Unix-like.
    8  * Storage Element: this is the service responsible for actually managing the data. It handles access to one file replica. Several SE implementations are available, interfacing with different back-ends like disks, distributed file systems, MSS (tapes). A SE is access through the ''SRM'' interface that has several advanced features like space reservation, file pinning, staging from MSS...
     8 * Storage Element: this is the service responsible for actually managing the data. It handles access to one file replica. Several SE implementations are available, interfacing with different back-ends like disks, distributed file systems, MSS (tapes). A SE is accessed through the ''SRM'' interface that has several advanced features like space reservation, file pinning, and staging from MSS.
    99
    1010
    1111== Main Data Management Commands ==
    1212
    13 A set of commands, known as ''lcg_utils'', are the main user-oriented tools to handle file and data management. They are completly independent of the actual SE implementation used and they take care of consistency between file catalog and storage elements. For example, they create logical file names in catalog when creating a file, they update a catalog entry when adding a new replica, they remove an entry from the catalog if the interaction with the SE is not successful...
     13A set of commands, known as ''lcg_utils'', are the main user-oriented tools to handle file and data management. They are completly independent of the actual SE implementation used and they take care of consistency between file catalog and storage elements. For example, they create logical file names in catalog when creating a file, they update a catalog entry when adding a new replica, they remove an entry from the catalog if the interaction with the SE is not successful, etc.
    1414
    1515''Note: each SE implementation provides its private set of client commands. They are not described here, even though some of them are independent of the SE implementation and may work with others. They are not needed, except in very specific circumstances.''
    1616
    17 Most of the commands, when referring to grid files stored on a SE, can use one of the following identifier for the file:
     17Most of the commands, when referring to grid files stored on a SE, can use one of the following identifiers for the file:
    1818 * Logical File Name (LFN): entry name in the file catalog, with a Unix-like syntax and starting with `/grid/`''voname''. A LFN must be prefixed `lfn:`.
    1919 * GUID: immutable identifier associated with a catalog entry (doesn't change when the LFN is changed/renamed). A GUID must be prefixed `guid:`.
    … …  
    2323
    2424Main commands are:
    25  * `lcg-cr` (copy and register): this command copies a file available locally (it can also copy a file from another SE) and create a new file on a SE, registering it in a file catalog. Copy destination '''must be''' a storage element. Even though the source is a file on a SE, this creates a new file and not a new replica of the existing file. By default logical file name registered in the LFC is generated but it can be set explicitly using option `-l` (''directories'' in the LFN must be created with `lfc-mkdir`, as in Unix they must be created with `mkdir` before being used). The storage element to use for the destination can be specified with option `-d`. If ommitted, the default SE (as specified by environment variable `VO_VONAME_DEFAULT_SE) for the VO is used. If  `-d` is present but specifies only a host name, the file name on the SE is generated. It is recommended to use option `-v` (verbose) to get details about the copy operation and ease the troubleshooting in case of errors. 
     25 * `lcg-cr` (copy and register): this command copies a file available locally (it can also copy a file from another SE) and create a new file on a SE, registering it in a file catalog. Copy destination '''must be''' a storage element. Even if the source is a file on a SE, this creates a new file and not a new replica of the existing file. By default logical file name registered in the LFC is generated but it can be set explicitly using option `-l`. The storage element to use for the destination can be specified with option `-d`. If omitted, the default SE (as specified by environment variable `VO_VONAME_DEFAULT_SE`) for the VO is used. If  `-d` is present but specifies only a host name, the file name on the SE is generated. It is recommended to use option `-v` (verbose) to get details about the copy operation and ease the troubleshooting in case of errors. 
    2626 * `lcg-cp` (copy): feature and syntax very similar to Unix `cp` command. Source and destination can be either a local file or a file on a SE, with any combination. Conversely to `lcg-cr` destination is not registered into a catalog and cannot be a LFN (the input file may be identified with a LFN or GUID if it resides on a SE). This command is mainly used to get a local copy of a SE file on a UI or WN.
    27  * `lcg-rep`: this command allow to add a new replica to an existing file registered in a file catalog. Syntax is similar to `lcg-cr` but the source '''must be''' a LFN. But conversely to `lcg-cr`, this command updates an existing entry in the file catalog but doesn't create a new one.
     27 * `lcg-rep`: this command allows adding a new replica to an existing file registered in a file catalog. Syntax is similar to `lcg-cr` but the source '''must be''' a LFN. But conversely to `lcg-cr`, this command updates an existing entry in the file catalog but doesn't create a new one.
    2828 * `lcg-lr` (list replica): list all the replicas associated with a given LFN or GUID. 
    29  * `lcg-ls` : feature and syntax very similar to Unix `ls` command.
    30  * `lcg-del` : removes a file replica from a SE and optionally remove entry from LFC after removing all replicas.
     29 * `lcg-ls` : feature a syntax very similar to Unix `ls` command.
     30 * `lcg-del` : removes a file replica from a SE and optionally removes an entry from the LFC after removing all replicas.
    3131 * `lcg-lg` : returns the GUID associated with a LFN.
    3232
    33 All these commands rely on BDII to guess defaults according to the VO used and to find the appropriate parameters to talk with the selected SE. In case BDII service is not functioning properly this may cause the commands to fail. Option `--nobdii` to `lcg-xxx` commands may help but requires to add a lot of parameters like SE port to use...
     33All these commands rely on BDII to guess defaults according to the VO used and to find the appropriate parameters to talk with the selected SE. In case the BDII service is not functioning properly this may cause the commands to fail. Option `--nobdii` to `lcg-xxx` commands may help but requires adding a lot of parameters to the command like SE port to use, etc.
    3434
    3535All the commands have an online help available with option `--help` or through the `man` command.