Pool Management And Pool Selection (2012)
In this section we will show some basic operations once the dCache instance is installed. With help of these simple tests you will get in touch with the elementary operations for dCache
On connecting to the dCache head node, you should see the following prompt:
...
dCache Admin (VII) (user=admin)
[[gks-xxx.scc.kit.edu]] (local) admin >
The last line is the prompt indicating that you are logged in as user admin and are not connected to any specific cell (hence "local"). To move to a particular well-known cell, use the cd command. For example, to switch to the PnfsManager cell, issue the command cd PnfsManager:
[gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager [gks-xxx.scc.kit.edu] (PnfsManager) admin >
All cells implement at least two commands: help and info. The help command displays a list of all commands that this cell understands along with what arguments are accepted. info displays some basic information about the cell. The format of this basic information depends on which cell you are connected to.
[gks-xxx.scc.kit.edu] (PnfsManager) admin > help say <arg-0> cacheinfoof <pnfsid>|<globalPath> flags remove <pnfsId> <key> [...] flags ls <pnfsId> flags set <pnfsId> <key=value> [...] rename # rename <old name> <new name> dumpthreadqueues [<threadId>] ....... [gks-xxx.scc.kit.edu] (PnfsManager) admin > info --- pnfs-manager (Request processor) --- $Revision: 15129 $ NameSpace Provider: diskCacheV111.namespace.provider.PermissionHandlerNameSpaceProvider@cd2192 CacheLocation Provider: ......... ......... Statistics: PnfsManagerV3 requests failed PnfsSetLengthMessage 0 0 PnfsAddCacheLocationMessage 0 0 PnfsCreateEntryMessage 0 0 PnfsGetParentMessage 0 0 PnfsFlagMessage 0 0 PnfsGetCacheLocationsMessage 0 0 PnfsSetStorageInfoMessage 0 0 PnfsMapPathMessage 0 0 PnfsGetChecksumMessage 0 0 PnfsGetCacheStatisticsMessage 0 0 PnfsGetChecksumAllMessage 0 0 PnfsSetChecksumMessage 0 0 PoolFileFlushedMessage 0 0 PnfsGetFileMetaDataMessage 0 0 PnfsDeleteEntryMessage 0 0 PnfsUpdateCacheStatisticsMessage 0 0 PnfsClearCacheLocationMessage 0 0 PnfsGetStorageInfoMessage 0 0 PnfsSetFileMetaDataMessage 0 0 PnfsCreateDirectoryMessage 0 0 PnfsRenameMessage 0 0 Total 0 0 PnfsManagerV3.Folded requests failed Total 0 0 [gks-xxx.scc.kit.edu] (PnfsManager) admin >
When navigating to another cell, one must first leave the current cell. This is achieved using the two dots (..) command. Please note that there is no cd preceeding the two dots.
[gks-xxx.scc.kit.edu] (PoolManagaer) admin > .. [gks-xxx.scc.kit.edu] (local) admin > cd PoolManager
Obtain a list of commands that begin with "pm"
[gks-xxx.scc.kit.edu] (PoolManager) admin > help pm pm ls [<section>] [-l] pm destroy <partitionName> # destroys parameter partition pm set [<partitionName>] OPTIONS # help pm set [dcachetogo.dcache.org] (PoolManager) admin > ..
Finding the location of a certain file
In this example, we will locate the file /pnfs/dcache.org/data/worldwritable/dccp.tst If you have deleted this file, please recreate it by copying /etc/group into dCache. First, we need to discover the file's pnfs-ID. This is a unique number, expressed in hexadecimal digits, that is used within dCache. To look up a file's ID, one must ask the PnfsManager. Navigate to this cell, if you are not already there. Within PnfsManager issue pnfsidof to determinate the file's pnfs-ID, then use the cacheinfoof command to determine on which pool/s the file is available. Please keep in mind that the pnfs-ID and pool location may be different to our example below.
[gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager [gks-xxx.scc.kit.edu] (PnfsManager) admin > pnfsidof /pnfs/dcache.org/data/worldwritable/dccp.tst 00003DD91CF131A742658424E65175CCD5B0 [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/worldwritable/dccp.tst pool1
To confirm this, navigate to the pool and use the rep ls and pf (short hand for pathfinder, which is available in the PnfsManager cell) commands to list all replicas and their metadata on that pool.
[gks-xxx.scc.kit.edu] (PnfsManager) admin > ..
[gks-xxx.scc.kit.edu] (local) admin > cd pool1
[gks-xxx.scc.kit.edu] (pool1) admin > rep ls 00003DD91CF131A742658424E65175CCD5B0
00003DD91CF131A742658424E65175CCD5B0 <-P---------L(0)[0]> 616248 si={<Unknown>:<Unknown>}
[gks-xxx.scc.kit.edu] (pool1) admin > pf 00003DD91CF131A742658424E65175CCD5B0
/pnfs/dcache.org/data/worldwritable/dccp.tst
Assigning pools to a directory
This section describes how to configure dCache such that the directory in which a file is created determines which pools are a valid choice for storing the data.
Introducing tags
The dCache file system supports some advanced features not present with many standard filesystems, namely directory tags. A directory tag, or tag for short, is a keyword:value pair assigned to a directory. Tags are inherited by sub-directories on creation and changing a tag of a directory will also change them in each sub-directory, unless the inheritance chain is broken by changing a tag of a directory directly. Directories in the namespace should never be moved, since this will mess up the inheritance structure and eventually break the whole system. You can list all the available tags by accessing the contents of a special file in each directory: ".(tags)()". On a freshly installed dCache instance — as we have it on our virtual machines — no tags defined.
Establishing a directory's storage class
Certain tags have a specific meaning. In particular, the two tags OSMTemplate and sGroup are used to determine the storage descriptor, from which the storage class is determined. The storage class is important as it can be used when identifying pools where data of a specific user is written to.
The OSMTemplate tag value will typically contain two words on a single line: the phrase "StoreName" followed by some value. The value of sGroup provides the storage group. Together they provide the storage descriptor as "store name:storage group". For disk-only systems, the HSM osm is assumed. This implies that the storage class is store “name:storage group@osm”. For example, given the following tag values
.(tag)(OSMTemplate):StoreName foo .(tag)(sGroup):BAR
the storage class is “foo:BAR@osm”.
Operation with tags
To assign a value to a keyword, simply send that value to the corresponding ".(tag)(tagname)" file. If previously the tag did not exist, it is created automatically:
root@gks-xxx.scc.kit.edu:~# cd /pnfs/dcache.org/data/worldwritable/ root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# echo "StoreName world" > '.(tag)(OSMTemplate)' root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# echo "ANY" > '.(tag)(sGroup)'
list the tags in the /pnfs/dcache.org/data/worldwritable/ directory.
root@gks-xxx.scc.kit.edu: ~# cat "/pnfs/dcache.org/data/worldwritable/.(tags)()" .(tag)(OSMTemplate) .(tag)(sGroup)
In the above example, we defined in the directory /pnfs/dcache.org/data/worldwritable/ two tags: OSMTemplate and sGroup. To obtain the value of these tags, just examine the contents of the file: ".(tag)(tagname)". For example, the contents of file ".(tag)(sGroup)" contains the value of the sGroup tag:
root@gks-xxx.scc.kit.edu:~# cd /pnfs/dcache.org/data/worldwritable/ root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# cat ".(tag)(sGroup)" ANY
The following handy trick lists all tags and their values in the current directory (please be aware of the differnt kind of quotation marks used here).
[root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# grep "" `cat '.(tags)()'` .(tag)(OSMTemplate):StoreName world .(tag)(sGroup):ANY
Selecting pools based on storage class
When faced with a data operation (such as writing data into dCache) the Pool Selection Unit (psu), part of the PoolManager, is used to determine which pools are suitable for that request. It is the Pool Selection Unit that must be configured so pools are selected based on the request's storage class. These are the steps to carry out, in order to assign one pool exclusively to each of two fictive experiments exp1 and exp2.
Create two different directories (if you plan to use dcap set permissions for unauthorized access.)
root@gks-xxx.scc.kit.edu:~# mkdir -m 777 /pnfs/dcache.org/data/exp{1,2}
Set different storage classes for both directories.
root@gks-xxx.scc.kit.edu:~# cd /pnfs/dcache.org/data/exp1 root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# echo "StoreName exp1" > ".(tag)(OSMTemplate)" root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# echo GENERATED > ".(tag)(sGroup)" root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# cd /pnfs/dcache.org/data/exp2 root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp2# echo "StoreName exp2" > ".(tag)(OSMTemplate)" root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# echo GENERATED > ".(tag)(sGroup)"
Next, the PoolManager must be configured in such a way that it will select pools based on the two storage classes: "exp1:GENERATED@osm" and "exp2:GENERATED@osm". To do this, log into the admin interface and navigate to the PoolManager.
root@dcachetogo:/pnfs/dcache.org/data/exp2# ssh -1 -c blowfish -p 22223 -1 admin@localhost
admin@localhost's password:
dCache Admin (VII) (user=admin)
[gks-xxx.scc.kit.edu] (local) admin > cd PoolManager
[gks-xxx.scc.kit.edu] (PooolManager) admin >
We have only default settings which allows us to write the data from anywhere to each available pool. Let us create the storage classes: exp1:GENERATED@osm and exp1:GENERATED@osm
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu create unit -store exp1:GENERATED@osm [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create unit -store exp2:GENERATED@osm
Units and unit groups
Unit groups are sets of predicates (logic conditions) that must be met before a link is acceptable for some data transfer. Unit groups contain one or more units, each of which are simple tests. At least one unit within a units group must be true for the unit group to be suitable; this is a boolean "or" operation. The command "psu ls -l ugroup" will list all currently configured unit groups:
[dcachetogo.dcache.org] (PoolManager) admin > psu ls -l ugroup any-store unitList : *@* (type=Store;canonical=*@*;uGroups=1) world-net unitList : 0.0.0.0/0.0.0.0 (type=Net;canonical=0.0.0.0/0.0.0.0;uGroups=1) any-protocol unitList : */* (type=Protocol;canonical=*/*;uGroups=1)
Define a new unit group.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu create ugroup exp1-groups [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto ugroup exp1-groups exp1:GENERATED@osm [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create ugroup exp2-groups [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto ugroup exp2-groups exp2:GENERATED@osm
Validate our changes
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu ls -l ugroup any-store unitList : *@* (type=Store;canonical=*@*;uGroups=1) world-net unitList : 0.0.0.0/0.0.0.0 (type=Net;canonical=0.0.0.0/0.0.0.0;uGroups=1) exp2-groups unitList : exp2:GENERATED@osm (type=Store;canonical=exp2:GENERATED@osm;uGroups=1) any-protocol unitList : */* (type=Protocol;canonical=*/*;uGroups=1) exp1-groups unitList : exp1:GENERATED@osm (type=Store;canonical=exp1:GENERATED@osm;uGroups=1)
The unit group “exp1-group” is satisfactory (i.e. is true) if the disk operation involves the storage class “exp1:GENERATED@osm”. The unit group “exp2-groups” has similar constraints.
Links
Links may contain one or more unit groups. When choosing a link for some data operation, all a link's unit groups must be satisfied for that link to be considered; this is a Boolean "and" operation. If a link contains unit group “exp1–groups” then that link will only be selected if the file operation involves a directory where the storage class exp1:GENERATED@osm is configured. Likewise, if a link contains unit group “exp2-groups” then a similar condition holds. These unit groups are sufficient for our needs. Lets create the links.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link exp1-link world-net any-protocol exp1-groups [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link exp1-link -readpref=20 -writepref=20 -cachepref=20 -p2ppref=-1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link exp2-link world-net exp2-groups any-protocol [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link exp2-link -readpref=10 -writepref=10 -cachepref=10 -p2ppref=-1
To be sure that everything was done properly.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu ls link -l exp1-link readPref : 20 cachePref : 20 writePref : 20 p2pPref : -1 section : None linkGroup : None UGroups : world-net (links=3;units=1) any-protocol (links=3;units=1) exp1-groups (links=1;units=1) default-link readPref : 10 cachePref : 10 writePref : 10 p2pPref : -1 section : None linkGroup : None UGroups : any-store (links=1;units=2) world-net (links=3;units=1) any-protocol (links=3;units=1) exp2-link readPref : 10 cachePref : 10 writePref : 10 p2pPref : -1 section : None linkGroup : None UGroups : world-net (links=3;units=1) any-protocol (links=3;units=1) exp2-groups (links=1;units=1)
These two links are ideal for setting up pools such that they are for exclusive use by their respective VOs. Note how the link “exp1-link” contains the unit group “exp1-groups”. Recall that this ensures that this link is only ever selected if the file operation involves the storage class “exp1:GENERATED@osm”. This means that, for “exp1-link” to be available, the file operation must be within the directory that has tags defining an acceptable storage class. For our configuration, storage class ”exp1:GENERATED@osm” is only available within the directory /pnfs/dcache.org/data/exp1.
Next, we see how the pools are assigned to different links by listing the pool groups.
Pool groups
The default configuration has all pools assigned to the pool group default. This pool group is assigned to the link “default–link”, so is available for all file operations.
[dcachetogo.dcache.org] (PoolManager) admin > psu ls -l pgroup
To reserve a pool for the exp1 VO it must be removed from the default pool group and assigned to the exp1 pool group. Doing this makes the pool available for the “exp1-link” link, which will be selected preferentially for operations involving the exp1 VO storage class exp1:GENERATED@osm. The following will assign pool pool1 exclusively for VO exp1 and pool2 for VO exp2. This configuration leaves no pools for general usage.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu create pgroup exp1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu removefrom pgroup default pool1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto pgroup exp1 pool1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link exp1-link exp1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create pgroup exp2 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu removefrom pgroup default pool2 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto pgroup exp2 pool2 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link exp2-link exp2 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu ls -l pgroup default linkList : default-link (pref=10/10/-1/10;;ugroups=3;pools=1) poolList : exp2 linkList : exp2-link (pref=10/10/-1/10;;ugroups=3;pools=1) poolList : pool2 (enabled=true;active=11;rdOnly=false;links=0;pgroups=1;hsm=[];mode=enabled) exp1 linkList : exp1-link (pref=20/20/-1/20;;ugroups=3;pools=1) poolList : pool1 (enabled=true;active=14;rdOnly=false;links=0;pgroups=1;hsm=[];mode=enabled) .....
Make sure to save all this settings in PoolManager otherwise all these configuration will be lost if PoolManager is restarted:
[gks-xxx.scc.kit.edu] (PoolManager) admin > save
Lets test that it is working, copy a file three times at exp1 and check via admin interface where it is stored.
ui_user@gks-xxx.scc.kit.edu:~$ dccp /bin/bash dcap://localhost:22125/pnfs/dcache.org/data/exp1/exp1_f1.tst 868404 bytes (848 kiB) in 0 seconds ui_user@gks-xxx.scc.kit.edu:~$ cp /bin/bash /pnfs/dcache.org/data/exp1/exp1_f2.tst ui_user@gks-xxx.scc.kit.edu:~$ cp /bin/bash /pnfs/dcache.org/data/exp1/exp1_f3.tst
[gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp1/exp1_f1.tst pool1 [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp1/exp1_f2.tst pool1 [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp1/exp1_f3.tst pool1 [gks-xxx.scc.kit.edu] (PnfsManager) admin >
and repeat this task for folder /pnfs/dcache.org/data/exp2.
With current configuration no pool is left for general use. See what happens if you try to write data on the /pnfs/dcache.org/data/worldwritable directory.
ui_user@dcachetogo:/root$ dccp /bin/bash dcap://localhost:22125/pnfs/dcache.org/data/worldwritable/testbin-1 Failed open file in the dCache. Can't open destination file : "No write pool available for <size=0;new=true;stored=false;sClass=<Unknown>:<Unknown>;cClass=-;hsm=osm;accessLatency=NEARLINE;retentionPolicy=CUSTODIAL;uid=500;onerror=default;truncate=;alloc-size=868404;gid=500;path=/pnfs/dcache.org/data/worldwritable/testbin-1;timeout=-1;mode=0755;store=<Unknown>;group=<Unknown>;bfid=<Unknown>;> in the linkGroup [none]" System error: Input/output error
Configure read/write pools
On this example we demonstrate how PoolManager handles transfer requests, how it separates read and write operations on the pool level. Create a new pool as was described on the chapter Installation with name pool3. The "read/write/cached" restrictions applies in the PoolManager on the "links" level. Thus we bind pool3 to a new link which allows only read and cached operations.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu removefrom pgroup default pool3 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create pgroup read-pools [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto pgroup read-pools pool3 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link read-link world-net exp2-groups any-protocol [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link read-link -readpref=10 -writepref=0 -cachepref=10 -p2ppref=-1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link read-link read-pools
pool2 will be reconfigured to permit write operation only.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link write-link world-net exp2-groups any-protocol [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link write-link -readpref=0 -writepref=10 -cachepref=0 -p2ppref=-1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link write-link exp2 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu remove link exp2-link
With the current configuration of the PoolManager all data for exp2 will be stored on the pool2, but read operations from exp2 directory first will create an additional (cached) copy of a file on pool3, that will serve the data transfer in the end.
Write a new file...
ui_user@gks-xxx.scc.kit.edu:~$ dccp /bin/bash dcap://localhost:22125/pnfs/dcache.org/data/exp2/exp2_f5.tst
... and check for replicas.
[gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp2/exp2_f5.tst pool2
There is a bug in the current version of dCache, which we have to workaround first. This workaround requires to save the configuration of each pool and restart the poolDomain.
[gks-xxx.scc.kit.edu] (PnfsManager) admin > .. [gks-xxx.scc.kit.edu] (local) admin > cd pool1 [gks-035.scc.kit.edu] (pool1) admin > save [gks-035.scc.kit.edu] (pool1) admin > .. [gks-xxx.scc.kit.edu] (local) admin > cd pool2 [gks-035.scc.kit.edu] (pool2) admin > save [gks-035.scc.kit.edu] (pool2) admin > .. [gks-xxx.scc.kit.edu] (local) admin > cd pool3 [gks-035.scc.kit.edu] (pool3) admin > save [gks-035.scc.kit.edu] (pool3) admin > .. [gks-xxx.scc.kit.edu] (local) admin > logoff [root@gks-xxx dcache]# dcache restart gks-xxxDomain
Read the file and observe dCache enforcing the configuration.
dccp dcap://localhost:22125/pnfs/dcache.org/data/exp2/exp2_f5.tst /dev/null [gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp2/exp2_f5.tst pool2 pool3
Persistent configuration
All changes done after the last we saved the PoolManager was done "on-the-run". If we would stop dCache now, all these settings would be lost and the pools will again belong to the default link and pool groups, using the permissive units/unit groups from the installation. In order to make dCache working with the same configuration, we need another flat textfile.
dCache will parse the file /var/lib/dcache/config/poolmanager.conf on startup, if it is existing, otherwise it will apply said default configuration. So you can create a poolmanager.conf beforehand, listing all the commands we have issued so far, or you can tell dCache to save the current configuration.
[gks-xxx.scc.kit.edu] (PoolManager) admin > save
The basic pool configuration is done.
