Data Management

Table of Contents

1. Main Data Management Commands
2. Using File Catalog
   1. LFC client main commands
3. Access Data on the Grid from a Job
4. GFAL

Grid services participating to data management are the File Catalog (LFC) and the Storage Element (SE).

• File Catalog: this service manages file names used by the user and associates them with replicas. Every grid file with a catalog entry can have several replicas. Catalog file names are Unix-like.
• Storage Element: this is the service responsible for actually managing the data. It handles access to one file replica. Several SE implementations are available, interfacing with different back-ends like disks, distributed file systems, MSS (tapes). A SE is access through the SRM interface that has several advanced features like space reservation, file pinning, staging from MSS...

Main Data Management Commands

A set of commands, known as lcg_utils, are the main user-oriented tools to handle file and data management. They are completly independent of the actual SE implementation used and they take care of consistency between file catalog and storage elements. For example, they create logical file names in catalog when creating a file, they update a catalog entry when adding a new replica, they remove an entry from the catalog if the interaction with the SE is not successful...

Note: each SE implementation provides its private set of client commands. They are not described here, even though some of them are independent of the SE implementation and may work with others. They are not needed, except in very specific circumstances.

Most of the commands, when referring to grid files stored on a SE, can use one of the following identifier for the file:

• Logical File Name (LFN): entry name in the file catalog, with a Unix-like syntax and starting with /grid/voname. A LFN must be prefixed lfn:.
• GUID: immutable identifier associated with a catalog entry (doesn't change when the LFN is changed/renamed). A GUID must be prefixed guid:.
• SURL: name of a replica on a particular SE. It has a URL format like srm://se.host.name/path/to/file.

When referring to local file, the prefix file: is optional and relative file names are supported.

Main commands are:

• lcg-cr (copy and register): this command copies a file available locally (it can also copy a file from another SE) and create a new file on a SE, registering it in a file catalog. Copy destination must be a storage element. Even though the source is a file on a SE, this creates a new file and not a new replica of the existing file. By default logical file name registered in the LFC is generated but it can be set explicitly using option -l (directories in the LFN must be created with lfc-mkdir, as in Unix they must be created with mkdir before being used). The storage element to use for the destination can be specified with option -d. If ommitted, the default SE (as specified by environment variable VO_VONAME_DEFAULT_SE) for the VO is used. If -d is present but specifies only a host name, the file name on the SE is generated. It is recommended to use option -v` (verbose) to get details about the copy operation and ease the troubleshooting in case of errors.
• lcg-cp (copy): feature and syntax very similar to Unix cp command. Source and destination can be either a local file or a file on a SE, with any combination. Conversely to lcg-cr destination is not registered into a catalog and cannot be a LFN (the input file may be identified with a LFN or GUID if it resides on a SE). This command is mainly used to get a local copy of a SE file on a UI or WN.
• lcg-rep: this command allow to add a new replica to an existing file registered in a file catalog. Syntax is similar to lcg-cr but the source must be a LFN. But conversely to lcg-cr, this command updates an existing entry in the file catalog but doesn't create a new one.
• lcg-lr (list replica): list all the replicas associated with a given LFN or GUID.
• lcg-ls : feature and syntax very similar to Unix ls command.
• lcg-del : removes a file replica from a SE and optionally remove entry from LFC after removing all replicas.
• lcg-lg : returns the GUID associated with a LFN.

All these commands rely on BDII to guess defaults according to the VO used and to find the appropriate parameters to talk with the selected SE. In case BDII service is not functioning properly this may cause the commands to fail. Option --nobdii to lcg-xxx commands may help but requires to add a lot of parameters like SE port to use...

All the commands have an online help available with option --help or through the man command.

Exercises :

1. Display information about avaialable SE resources for VO vo.lal.in2p3.fr, using command lcg-infosites.
   • How many SEs are available for VO vo.lal.in2p3.fr ?
   • Find the same information for VO dteam. How many SEs are available for this VO ?
2. Create a text file and copy it on a SE using command lcg-cr, using option -l to define the logical name (see above for the logical name format). On which SE is the file located? Try to explicitly define the destination SE.
3. Check that the file(s) is present in the catalog and list its replicas, using command lcg-lr. This should return SURL for all the replicas (a SURL starts with prefix srm:).
4. Display detailed information about the file, using command lcg-ls.
5. Find the GUID associated with the previous files.
6. Replicate the file on another SE open to the VO (using the information got with lcg-infosites).
7. Check that the new replica has been added to the catalog entry for the file, using command lcg-lr as previously. Compare the information returned.
8. Copy locally the file from the SE, using command lcg-cp. Do it using the GUID, the LFN and a replica SURL.
9. Suppress one of the replica using command lcg-del and check the result with lcg-lr.
10. Suppress the other replica using command lcg-del and check the result with lcg-lr.
11. Recreate a file with 2 replicas and try to suppress all the replicas at once. Try to do a lcg-lr and a lcg-ls.

Using File Catalog

The File Catalog service (also known as replica catalog) in gLite is implemented using LFC. LFC service has a client implemented through commands whose names start with lfc-. These commands are installed on every UI and WN.

These LFC commands are seldom used: LFC interaction is normally done with commands handling the necessary interactions with both LFC and SE services to implement actual file management (mainly the so-called lcg_utils commands whose names start with lcg-.). There is one exception, lfc-mkdir, that must be used to create directories before creating LFN in them (lfc-mkdir supports a -p option to create parent directories, similar to the same option in Unix mkdir).

All of these commands require a valid VOMS proxy with a valid VOMS extension (use voms-proxy-init --voms), else you need to use option --vo with lcg-xxx commands. lcg-xxx commands locate the LFC server from the VO used. To use a LFC server other than the default (central) one or with lfc-xxx commands, define environment variable LFC_HOST to the host name of the LFC server to use.

Note: this is not recommended to define LFC_HOST when this is not necessary as your command may fail if the default server is moved. An alternative to define LFC_HOST for using lfc-xxx commands is to prefix the LFN with lfc.host.dom.ain:.

LFC client main commands

The main LFC commands that may be used under normal circumstances are:

• lfc-mkdir [-]: create a directory in LFC and optionally its parents.
• lfs-ls [-l]: similar to Unix ls command. Gives the content of a directory and optionally detailed information in a Unix like format. User and group are the DN of the owner and the FQAN of the primary VOMS group.
• lfc-getacl / lfc-setacl: LFC namespace supports Posix-like ACLs. See the man page for more information on the syntax, in particular for lfc-setacl. A file name has one ACL but a directory has 2 different ACLs: the ACL controlling access to the directory and the default ACL applied to new files (new sub-directories inherit the parent ACL).

Access Data on the Grid from a Job

In the job description, this is possible to describe the data available on the grid that the job need to access. Data files can be identified by a LFN, a GUID or a SURL. This description is taken into account by WMS to select the appropriate site: a CE will be selected only if one the file replica is on one of the close SE for the CE (this is a configuration information for each CE). Then the job can retrieve the list of files provided, the replicas used... with command glite-brokerinfo (available only on WNs).

To specify the grid files the job needs access to, it is necessary to add the 2 following lines to the JDL:

DataAccessProtocol ={ "gsiftp","rfio"};
InputData = {"file1", "file2", ...};

DataAccessProtocol is a list of transfer/access protocol that the job will use to access the data. The protocols in the example are the most common but other possibilities are xroot, dcap, https... There is no default and this clause is required if InputData is present.

File names specified in InputData can be LFN, GUID or SURL. The standard syntax is used with the corresponding prefix lfn:, guid: or srm:. See above.

Note: files specified in InputData are not copied to the WN, conversely to files in InputSandbox. This is the responsability of the job to do what is relevant and appropriate.

glite-brokerinfo command accept many options, the main ones being:

• getCE
• getDataAccessProtocol
• getInputData
• getSEs
• getCloseSEs
• `getSEFreeSpace <SE>
• getLFN2SFN <LFN>
• getSEProtocols <SE>

Exercises:

1. Modify one of your existing JDL file and add clauses described above with reference to a file you created with lcg-cr commands.
2. Use glite-wms-job-list-match command to display the list of sites which can execute the job. Compare with the list without the InputData clause.
3. Execute the job after adding glite-brokerinfo closeSEs and glite-brokerinfo getInputData commands to check the information passed to the job.

GFAL

GFAL (Grid File Access Library) is a set of low-level and high-level APIs providing:

• Posix-like funtions to access files: gfal_open(), gfal_read(), gfal_write(), gfal_close()... These functions use a SURL to identify the file.
• API to the file catalog to do LFN/GUID to SURL translation
• API to the advanced features of SRM

GFAL is available for C/C++, Python and Perl. There is no Java implementation. In fact, lcg-xxx commands are wrapper above GFAL.

Documentation with examples is available in man pages for the functions.