Difference between revisions of "Pool Management And Pool Selection (2012)"
From Gridkaschool
(Created page with "{| align="right" style="margin-left:20px; margin-bottom:5px;" | __TOC__ |} In this section we will show some basic operations once the dCache instance is installed. With help of…") |
(→Operation with tags) |
||
| (8 intermediate revisions by the same user not shown) | |||
| Line 4: | Line 4: | ||
In this section we will show some basic operations once the dCache instance is installed. With help of these simple tests you will get in touch with the elementary operations for dCache |
In this section we will show some basic operations once the dCache instance is installed. With help of these simple tests you will get in touch with the elementary operations for dCache |
||
| − | === Getting in touch with the namespace === |
||
| − | To store file system information dCache uses its own namespace service which is mounted on the local machine via NFS under the <tt>/pnfs</tt> mounting point. It can easily be checked with <tt>df</tt>. If that is not the case for you, please head over to the previous lecture [[Installing dCache (2012)]] and read the section [[Installing dCache (2012)#Export of namespace file system|Export of namespace file system]]. |
||
| − | <pre> |
||
| − | root@dcachetogo ~# df /pnfs/dcache.org/data/ |
||
| − | Filesystem 1K-blocks Used Available Use% Mounted on |
||
| − | dcachetogo.dcache.org:/pnfs |
||
| − | 1099511627776 576 1099511627200 1% /pnfs |
||
| − | </pre> |
||
| − | As was written on in previous session NFS4.1 allows to mount dCache as a regular NFS server providing standard POSIX file access. |
||
| − | Lets us check the permissions of directory: <tt>/pnfs/dcache.org/data/</tt> |
||
| − | <pre> |
||
| − | root@dcachetogo:/pnfs/dcache.org/data# ls -ls /pnfs/dcache.org/ |
||
| − | total 1 |
||
| − | 1 drwxr-xr-x 2 root root 512 Sep 4 22:35 data |
||
| − | </pre> |
||
| − | also read, write and delete any file: |
||
| − | |||
| − | <pre> |
||
| − | root@dcachetogo:~# cp /bin/sh /pnfs/dcache.org/data/data.tst |
||
| − | root@dcachetogo:~# diff /bin/sh /pnfs/dcache.org/data/data.tst |
||
| − | root@dcachetogo:~# rm /pnfs/dcache.org/data/data.tst |
||
| − | rm: remove regular file `/pnfs/dcache.org/data/data.tst'? y |
||
| − | root@dcachetogo:~# |
||
| − | </pre> |
||
| − | |||
| − | === Copying data using dCache dcap protocol === |
||
| − | dCache supports different protocols for data transfer. For demonstration we will use the ''dcap'' transfer protocol, the simple, yet unsecured, native protocol of dCache. |
||
| − | |||
| − | All accesses with this protocol is mapped to "nobody", so we have to create worldwritable directory. |
||
| − | <pre> |
||
| − | # mkdir -m 777 /pnfs/dcache.org/data/worldwritable |
||
| − | </pre> |
||
| − | |||
| − | Now we will attempt to copy a modest sized file (the shell executable <tt>/bin/bash</tt>) from the local machine into dCache. The data will be stored within one of the pools, a new entry in the namespace is created and the stored data is linked against this namespace entry. |
||
| − | <pre> |
||
| − | root@dcachetogo:~# dccp /bin/bash dcap://localhost:22125/pnfs/dcache.org/data/worldwritable/testbin-1 |
||
| − | 868404 bytes (848 kiB) in 1 seconds (848 kiB/s) |
||
| − | </pre> |
||
| − | |||
| − | After the transfer has successfully finished, the file's metadata is stored in the namespace. This should be visible on the dCache node underneath the <tt>/pnfs</tt> directory. After copying the file back, you can check that the file has the same checksum value as the source file. |
||
| − | <pre> |
||
| − | ui_user@dcachetogo:~$ dccp dcap://localhost:22125/pnfs/dcache.org/data/worldwritable/testbin-1 /tmp/deleteme |
||
| − | ui_user@dcachetogo:~$ md5sum /bin/bash /tmp/deleteme |
||
| − | 109425e8b673c46d9bef5d4d070cae22 /bin/bash |
||
| − | 109425e8b673c46d9bef5d4d070cae22 /tmp/deleteme |
||
| − | </pre> |
||
| − | |||
| − | The file may be deleted in a number of ways; the simplest is to remove the namespace entry: |
||
| − | <pre> |
||
| − | root@dcachetogo:~# dccp /bin/bash dcap://localhost:22125/pnfs/dcache.org/data/worldwritable/testbin-2 |
||
| − | 868404 bytes (848 kiB) in 1 seconds (848 kiB/s) |
||
| − | root@dcachetogo ~# rm /pnfs/dcache.org/data/worldwritable/testbin-2 |
||
| − | </pre> |
||
| − | A file being removed from the namespace will trigger the removal of the file from whichever pool it is stored on. This process is not instantaneous; under normal conditions, it may take up to a minute for the file on a pool to be removed. This delay is deliberate as it allows dCache to prioritise read operations over delete operations, so preventing file deletions from overly affecting read performance. |
||
| − | |||
| − | == Administrating dCache == |
||
| − | In this section, we introduce the ''administration interface'' of dCache. This provides a rich command-driven interface that allows you to control many features of dCache. After completing this section you will... |
||
| − | * understand more advanced concepts, such as cells, links and poolgroups, |
||
| − | * understand how to interact with dCache using the admin interface and |
||
| − | * know how to configure dCache for more advanced deployments. |
||
| − | |||
| − | |||
| − | === Basic orientation === |
||
| − | The core mechanism for administrating a dCache instance is the admin interface. This is a service you may connect to using an ssh client. Using the admin interface, you may communicate with the various components making up dCache, query their status and update their behaviour. Although dCache is a distributed system, you only ever connect to a single node; dCache will route your messages internally. |
||
| − | |||
| − | === Connecting to the admin interface === |
||
| − | The admin interface uses the non-default port 22223 and the non-default cipher blowfish. This interface includes a default account with username <tt>admin</tt> and password ''dickerelch''. A suitable command for connecting to your dCache instance is: |
||
| − | <pre> |
||
| − | root@dcachetogo:~# ssh -1 -c blowfish -p 22223 -1 admin@localhost |
||
| − | admin@localhost's password: |
||
| − | dCache Admin (VII) (user=admin) |
||
| − | |||
| − | [dcachetogo.dcache.org] (local) admin > logoff |
||
| − | Connection to localhost closed. |
||
| − | root@dcachetogo:~# |
||
| − | </pre> |
||
=== Navigating within the admin interface === |
=== Navigating within the admin interface === |
||
| Line 87: | Line 11: | ||
... |
... |
||
dCache Admin (VII) (user=admin) |
dCache Admin (VII) (user=admin) |
||
| − | [ |
+ | [[gks-xxx.scc.kit.edu]] (local) admin > |
</pre> |
</pre> |
||
| Line 93: | Line 17: | ||
To move to a particular well-known cell, use the <tt>cd</tt> command. For example, to switch to the <tt>PnfsManager</tt> cell, issue the command <tt>cd PnfsManager</tt>: |
To move to a particular well-known cell, use the <tt>cd</tt> command. For example, to switch to the <tt>PnfsManager</tt> cell, issue the command <tt>cd PnfsManager</tt>: |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > |
</pre> |
</pre> |
||
All cells implement at least two commands: <tt>help</tt> and <tt>info</tt>. The <tt>help</tt> command displays a list of all commands that this cell understands along with what arguments are accepted. <tt>info</tt> displays some basic information about the cell. The format of this basic information depends on which cell you are connected to. |
All cells implement at least two commands: <tt>help</tt> and <tt>info</tt>. The <tt>help</tt> command displays a list of all commands that this cell understands along with what arguments are accepted. <tt>info</tt> displays some basic information about the cell. The format of this basic information depends on which cell you are connected to. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > help |
say <arg-0> |
say <arg-0> |
||
cacheinfoof <pnfsid>|<globalPath> |
cacheinfoof <pnfsid>|<globalPath> |
||
| Line 108: | Line 32: | ||
dumpthreadqueues [<threadId>] |
dumpthreadqueues [<threadId>] |
||
....... |
....... |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > info |
--- pnfs-manager (Request processor) --- |
--- pnfs-manager (Request processor) --- |
||
$Revision: 15129 $ |
$Revision: 15129 $ |
||
| Line 142: | Line 66: | ||
PnfsManagerV3.Folded requests failed |
PnfsManagerV3.Folded requests failed |
||
Total 0 0 |
Total 0 0 |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > |
</pre> |
</pre> |
||
When navigating to another cell, one must first leave the current cell. This is achieved using the two dots (<tt>..</tt>) command. Please note that there is no <tt>cd</tt> preceeding the two dots. |
When navigating to another cell, one must first leave the current cell. This is achieved using the two dots (<tt>..</tt>) command. Please note that there is no <tt>cd</tt> preceeding the two dots. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManagaer) admin > .. |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (local) admin > cd PoolManager |
</pre> |
</pre> |
||
Obtain a list of commands that begin with "pm" |
Obtain a list of commands that begin with "pm" |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > help pm |
pm ls [<section>] [-l] |
pm ls [<section>] [-l] |
||
pm destroy <partitionName> # destroys parameter partition |
pm destroy <partitionName> # destroys parameter partition |
||
pm set [<partitionName>] OPTIONS # help pm set |
pm set [<partitionName>] OPTIONS # help pm set |
||
| − | [dcachetogo.dcache.org] (PoolManager) admin > |
+ | [dcachetogo.dcache.org] (PoolManager) admin > .. |
</pre> |
</pre> |
||
=== Finding the location of a certain file === |
=== Finding the location of a certain file === |
||
| − | In this example, we will locate the file <tt style="white-space:nowrap;">/pnfs/dcache.org/data/worldwritable/ |
+ | In this example, we will locate the file <tt style="white-space:nowrap;"> /pnfs/dcache.org/data/worldwritable/dccp.tst </tt> If you have deleted this file, please recreate it by [[Installing_dCache_(2012)#File_transfer_into_dCache|copying]] <tt>/etc/group</tt> into dCache. First, we need to discover the file's pnfs-ID. This is a unique number, expressed in hexadecimal digits, that is used within dCache. To look up a file's ID, one must ask the <tt>PnfsManager</tt>. Navigate to this cell, if you are not already there. Within <tt>PnfsManager</tt> issue <tt>pnfsidof</tt> to determinate the file's pnfs-ID, then use the <tt>cacheinfoof</tt> command to determine on which pool/s the file is available. Please keep in mind that the pnfs-ID and pool location may be different to our example below. |
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > pnfsidof /pnfs/dcache.org/data/worldwritable/dccp.tst |
00003DD91CF131A742658424E65175CCD5B0 |
00003DD91CF131A742658424E65175CCD5B0 |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/worldwritable/dccp.tst |
pool1 |
pool1 |
||
</pre> |
</pre> |
||
| Line 170: | Line 94: | ||
To confirm this, navigate to the pool and use the <tt>rep ls</tt> and <tt>pf</tt> (short hand for <tt>pathfinder</tt>, which is available in the <tt>PnfsManager</tt> cell) commands to list all replicas and their metadata on that pool. |
To confirm this, navigate to the pool and use the <tt>rep ls</tt> and <tt>pf</tt> (short hand for <tt>pathfinder</tt>, which is available in the <tt>PnfsManager</tt> cell) commands to list all replicas and their metadata on that pool. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > .. |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (local) admin > cd pool1 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (pool1) admin > rep ls 00003DD91CF131A742658424E65175CCD5B0 |
00003DD91CF131A742658424E65175CCD5B0 <-P---------L(0)[0]> 616248 si={<Unknown>:<Unknown>} |
00003DD91CF131A742658424E65175CCD5B0 <-P---------L(0)[0]> 616248 si={<Unknown>:<Unknown>} |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (pool1) admin > pf 00003DD91CF131A742658424E65175CCD5B0 |
| − | /pnfs/dcache.org/data/worldwritable/ |
+ | /pnfs/dcache.org/data/worldwritable/dccp.tst |
| + | |||
</pre> |
</pre> |
||
| Line 200: | Line 125: | ||
To assign a value to a keyword, simply send that value to the corresponding "<tt style="white-space:nowrap;">.(tag)(<span style="font-family:monospace; color:green; font-style:italic;">tagname</span>)</tt>" file. If previously the tag did not exist, it is created automatically: |
To assign a value to a keyword, simply send that value to the corresponding "<tt style="white-space:nowrap;">.(tag)(<span style="font-family:monospace; color:green; font-style:italic;">tagname</span>)</tt>" file. If previously the tag did not exist, it is created automatically: |
||
<pre> |
<pre> |
||
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:~# cd /pnfs/dcache.org/data/worldwritable/ |
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# echo "StoreName world" > '.(tag)(OSMTemplate)' |
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# echo "ANY" > '.(tag)(sGroup)' |
</pre> |
</pre> |
||
list the tags in the <tt>/pnfs/dcache.org/data/worldwritable/</tt> directory. |
list the tags in the <tt>/pnfs/dcache.org/data/worldwritable/</tt> directory. |
||
<pre> |
<pre> |
||
| − | root@ |
+ | root@gks-xxx.scc.kit.edu: ~# cat "/pnfs/dcache.org/data/worldwritable/.(tags)()" |
.(tag)(OSMTemplate) |
.(tag)(OSMTemplate) |
||
.(tag)(sGroup) |
.(tag)(sGroup) |
||
| Line 214: | Line 139: | ||
In the above example, we defined in the directory <tt>/pnfs/dcache.org/data/worldwritable/</tt> two tags: ''OSMTemplate'' and ''sGroup''. To obtain the value of these tags, just examine the contents of the file: "<tt style="white-space:nowrap;">.(tag)(<span style="font-family:monospace; color:green; font-style:italic;">tagname</span>)</tt>". For example, the contents of file "<tt style="white-space:nowrap;">.(tag)(sGroup)</tt>" contains the value of the ''sGroup'' tag: |
In the above example, we defined in the directory <tt>/pnfs/dcache.org/data/worldwritable/</tt> two tags: ''OSMTemplate'' and ''sGroup''. To obtain the value of these tags, just examine the contents of the file: "<tt style="white-space:nowrap;">.(tag)(<span style="font-family:monospace; color:green; font-style:italic;">tagname</span>)</tt>". For example, the contents of file "<tt style="white-space:nowrap;">.(tag)(sGroup)</tt>" contains the value of the ''sGroup'' tag: |
||
<pre> |
<pre> |
||
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:~# cd /pnfs/dcache.org/data/worldwritable/ |
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# cat ".(tag)(sGroup)" |
ANY |
ANY |
||
</pre> |
</pre> |
||
| Line 221: | Line 146: | ||
The following handy trick lists all tags and their values in the current directory (please be aware of the differnt kind of quotation marks used here). |
The following handy trick lists all tags and their values in the current directory (please be aware of the differnt kind of quotation marks used here). |
||
<pre> |
<pre> |
||
| − | [root@ |
+ | [root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# grep "" `cat '.(tags)()'` |
.(tag)(OSMTemplate):StoreName world |
.(tag)(OSMTemplate):StoreName world |
||
.(tag)(sGroup):ANY |
.(tag)(sGroup):ANY |
||
| Line 232: | Line 157: | ||
Create two different directories (if you plan to use dcap set permissions for unauthorized access.) |
Create two different directories (if you plan to use dcap set permissions for unauthorized access.) |
||
<pre> |
<pre> |
||
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:~# mkdir -m 777 /pnfs/dcache.org/data/exp{1,2} |
</pre> |
</pre> |
||
Set different storage classes for both directories. |
Set different storage classes for both directories. |
||
<pre> |
<pre> |
||
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:~# cd /pnfs/dcache.org/data/exp1 |
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# echo "StoreName exp1" > ".(tag)(OSMTemplate)" |
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# echo GENERATED > ".(tag)(sGroup)" |
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# cd /pnfs/dcache.org/data/exp2 |
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp2# echo "StoreName exp2" > ".(tag)(OSMTemplate)" |
| − | root@ |
+ | root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# echo GENERATED > ".(tag)(sGroup)" |
</pre> |
</pre> |
||
| Line 253: | Line 178: | ||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (local) admin > cd PoolManager |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PooolManager) admin > |
</pre> |
</pre> |
||
| Line 260: | Line 185: | ||
Let us create the storage classes: <tt>exp1:GENERATED@osm</tt> and <tt>exp1:GENERATED@osm</tt> |
Let us create the storage classes: <tt>exp1:GENERATED@osm</tt> and <tt>exp1:GENERATED@osm</tt> |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create unit -store exp1:GENERATED@osm |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create unit -store exp2:GENERATED@osm |
</pre> |
</pre> |
||
| Line 281: | Line 206: | ||
Define a new unit group. |
Define a new unit group. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create ugroup exp1-groups |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto ugroup exp1-groups exp1:GENERATED@osm |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create ugroup exp2-groups |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto ugroup exp2-groups exp2:GENERATED@osm |
</pre> |
</pre> |
||
Validate our changes |
Validate our changes |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu ls -l ugroup |
any-store |
any-store |
||
unitList : |
unitList : |
||
| Line 313: | Line 238: | ||
If a link contains unit group “<tt style="white-space:nowrap;">exp1–groups</tt>” then that link will only be selected if the file operation involves a directory where the storage class <tt style="white-space:nowrap;">exp1:GENERATED@osm</tt> is configured. Likewise, if a link contains unit group “<tt style="white-space:nowrap;">exp2-groups</tt>” then a similar condition holds. These unit groups are sufficient for our needs. Lets create the links. |
If a link contains unit group “<tt style="white-space:nowrap;">exp1–groups</tt>” then that link will only be selected if the file operation involves a directory where the storage class <tt style="white-space:nowrap;">exp1:GENERATED@osm</tt> is configured. Likewise, if a link contains unit group “<tt style="white-space:nowrap;">exp2-groups</tt>” then a similar condition holds. These unit groups are sufficient for our needs. Lets create the links. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link exp1-link world-net any-protocol exp1-groups |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link exp1-link -readpref=20 -writepref=20 -cachepref=20 -p2ppref=-1 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link exp2-link world-net exp2-groups any-protocol |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link exp2-link -readpref=10 -writepref=10 -cachepref=10 -p2ppref=-1 |
</pre> |
</pre> |
||
To be sure that everything was done properly. |
To be sure that everything was done properly. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu ls link -l |
exp1-link |
exp1-link |
||
readPref : 20 |
readPref : 20 |
||
| Line 369: | Line 294: | ||
The following will assign pool <tt style="white-space:nowrap;">pool1</tt> exclusively for VO <tt style="white-space:nowrap;">exp1</tt> and <tt style="white-space:nowrap;">pool2</tt> for VO <tt style="white-space:nowrap;">exp2</tt>. This configuration leaves no pools for general usage. |
The following will assign pool <tt style="white-space:nowrap;">pool1</tt> exclusively for VO <tt style="white-space:nowrap;">exp1</tt> and <tt style="white-space:nowrap;">pool2</tt> for VO <tt style="white-space:nowrap;">exp2</tt>. This configuration leaves no pools for general usage. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create pgroup exp1 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu removefrom pgroup default pool1 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto pgroup exp1 pool1 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link exp1-link exp1 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create pgroup exp2 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu removefrom pgroup default pool2 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto pgroup exp2 pool2 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link exp2-link exp2 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu ls -l pgroup |
default |
default |
||
linkList : |
linkList : |
||
| Line 394: | Line 319: | ||
pool1 (enabled=true;active=14;rdOnly=false;links=0;pgroups=1;hsm=[];mode=enabled) |
pool1 (enabled=true;active=14;rdOnly=false;links=0;pgroups=1;hsm=[];mode=enabled) |
||
..... |
..... |
||
| + | </pre> |
||
| − | |||
| + | Make sure to save all this settings in PoolManager otherwise all these configuration will be lost if PoolManager is restarted: |
||
| − | [dcachetogo.dcache.org] (PoolManager) admin > save |
||
| + | <pre> |
||
| + | [gks-xxx.scc.kit.edu] (PoolManager) admin > save |
||
</pre> |
</pre> |
||
Lets test that it is working, copy a file three times at <tt style="white-space:nowrap;">exp1</tt> and check via admin interface where it is stored. |
Lets test that it is working, copy a file three times at <tt style="white-space:nowrap;">exp1</tt> and check via admin interface where it is stored. |
||
<pre> |
<pre> |
||
| − | ui_user@ |
+ | ui_user@gks-xxx.scc.kit.edu:~$ dccp /bin/bash dcap://localhost:22125/pnfs/dcache.org/data/exp1/exp1_f1.tst |
868404 bytes (848 kiB) in 0 seconds |
868404 bytes (848 kiB) in 0 seconds |
||
| − | ui_user@ |
+ | ui_user@gks-xxx.scc.kit.edu:~$ cp /bin/bash /pnfs/dcache.org/data/exp1/exp1_f2.tst |
| − | ui_user@ |
+ | ui_user@gks-xxx.scc.kit.edu:~$ cp /bin/bash /pnfs/dcache.org/data/exp1/exp1_f3.tst |
</pre> |
</pre> |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp1/exp1_f1.tst |
pool1 |
pool1 |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp1/exp1_f2.tst |
pool1 |
pool1 |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp1/exp1_f3.tst |
pool1 |
pool1 |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > |
</pre> |
</pre> |
||
| Line 433: | Line 360: | ||
The "read/write/cached" restrictions applies in the <tt>PoolManager</tt> on the "links" level. Thus we bind ''pool3'' to a new link which allows only read and cached operations. |
The "read/write/cached" restrictions applies in the <tt>PoolManager</tt> on the "links" level. Thus we bind ''pool3'' to a new link which allows only read and cached operations. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu removefrom pgroup default pool3 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create pgroup read-pools |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto pgroup read-pools pool3 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link read-link world-net exp2-groups any-protocol |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link read-link -readpref=10 -writepref=0 -cachepref=10 -p2ppref=-1 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link read-link read-pools |
</pre> |
</pre> |
||
pool2 will be reconfigured to permit write operation only. |
pool2 will be reconfigured to permit write operation only. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link write-link world-net exp2-groups any-protocol |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link write-link -readpref=0 -writepref=10 -cachepref=0 -p2ppref=-1 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link write-link exp2 |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > psu remove link exp2-link |
</pre> |
</pre> |
||
With the current configuration of the <tt>PoolManager</tt> all data for exp2 will be stored on the pool2, but read operations from exp2 directory first will create an additional (cached) copy of a file on pool3, that will serve the data transfer in the end. |
With the current configuration of the <tt>PoolManager</tt> all data for exp2 will be stored on the pool2, but read operations from exp2 directory first will create an additional (cached) copy of a file on pool3, that will serve the data transfer in the end. |
||
| Line 451: | Line 378: | ||
Write a new file... |
Write a new file... |
||
<pre> |
<pre> |
||
| − | ui_user@ |
+ | ui_user@gks-xxx.scc.kit.edu:~$ dccp /bin/bash dcap://localhost:22125/pnfs/dcache.org/data/exp2/exp2_f5.tst |
</pre> |
</pre> |
||
... and check for replicas. |
... and check for replicas. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp2/exp2_f5.tst |
pool2 |
pool2 |
||
</pre> |
</pre> |
||
| + | There is a bug in the current version of dCache, which we have to workaround first. This workaround requires to save the configuration of each pool and restart the poolDomain. |
||
| + | <pre> |
||
| + | [gks-xxx.scc.kit.edu] (PnfsManager) admin > .. |
||
| + | [gks-xxx.scc.kit.edu] (local) admin > cd pool1 |
||
| + | [gks-035.scc.kit.edu] (pool1) admin > save |
||
| + | [gks-035.scc.kit.edu] (pool1) admin > .. |
||
| + | [gks-xxx.scc.kit.edu] (local) admin > cd pool2 |
||
| + | [gks-035.scc.kit.edu] (pool2) admin > save |
||
| + | [gks-035.scc.kit.edu] (pool2) admin > .. |
||
| + | [gks-xxx.scc.kit.edu] (local) admin > cd pool3 |
||
| + | [gks-035.scc.kit.edu] (pool3) admin > save |
||
| + | [gks-035.scc.kit.edu] (pool3) admin > .. |
||
| + | [gks-xxx.scc.kit.edu] (local) admin > logoff |
||
| + | [root@gks-xxx dcache]# dcache restart gks-xxxDomain |
||
| + | </pre> |
||
| + | |||
Read the file and observe dCache enforcing the configuration. |
Read the file and observe dCache enforcing the configuration. |
||
<pre> |
<pre> |
||
dccp dcap://localhost:22125/pnfs/dcache.org/data/exp2/exp2_f5.tst /dev/null |
dccp dcap://localhost:22125/pnfs/dcache.org/data/exp2/exp2_f5.tst /dev/null |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager |
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp2/exp2_f5.tst |
pool2 pool3 |
pool2 pool3 |
||
</pre> |
</pre> |
||
=== Persistent configuration === |
=== Persistent configuration === |
||
| − | All |
+ | All changes done after the last we saved the PoolManager was done "on-the-run". If we would stop dCache now, all these settings would be lost and the pools will again belong to the default link and pool groups, using the permissive units/unit groups from the installation. In order to make dCache working with the same configuration, we need another flat textfile. |
dCache will parse the file <tt>/var/lib/dcache/config/poolmanager.conf</tt> on startup, if it is existing, otherwise it will apply said default configuration. So you can create a <tt>poolmanager.conf</tt> beforehand, listing all the commands we have issued so far, or you can tell dCache to save the current configuration. |
dCache will parse the file <tt>/var/lib/dcache/config/poolmanager.conf</tt> on startup, if it is existing, otherwise it will apply said default configuration. So you can create a <tt>poolmanager.conf</tt> beforehand, listing all the commands we have issued so far, or you can tell dCache to save the current configuration. |
||
<pre> |
<pre> |
||
| − | [ |
+ | [gks-xxx.scc.kit.edu] (PoolManager) admin > save |
</pre> |
</pre> |
||
Latest revision as of 13:39, 24 August 2012
In this section we will show some basic operations once the dCache instance is installed. With help of these simple tests you will get in touch with the elementary operations for dCache
On connecting to the dCache head node, you should see the following prompt:
...
dCache Admin (VII) (user=admin)
[[gks-xxx.scc.kit.edu]] (local) admin >
The last line is the prompt indicating that you are logged in as user admin and are not connected to any specific cell (hence "local"). To move to a particular well-known cell, use the cd command. For example, to switch to the PnfsManager cell, issue the command cd PnfsManager:
[gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager [gks-xxx.scc.kit.edu] (PnfsManager) admin >
All cells implement at least two commands: help and info. The help command displays a list of all commands that this cell understands along with what arguments are accepted. info displays some basic information about the cell. The format of this basic information depends on which cell you are connected to.
[gks-xxx.scc.kit.edu] (PnfsManager) admin > help say <arg-0> cacheinfoof <pnfsid>|<globalPath> flags remove <pnfsId> <key> [...] flags ls <pnfsId> flags set <pnfsId> <key=value> [...] rename # rename <old name> <new name> dumpthreadqueues [<threadId>] ....... [gks-xxx.scc.kit.edu] (PnfsManager) admin > info --- pnfs-manager (Request processor) --- $Revision: 15129 $ NameSpace Provider: diskCacheV111.namespace.provider.PermissionHandlerNameSpaceProvider@cd2192 CacheLocation Provider: ......... ......... Statistics: PnfsManagerV3 requests failed PnfsSetLengthMessage 0 0 PnfsAddCacheLocationMessage 0 0 PnfsCreateEntryMessage 0 0 PnfsGetParentMessage 0 0 PnfsFlagMessage 0 0 PnfsGetCacheLocationsMessage 0 0 PnfsSetStorageInfoMessage 0 0 PnfsMapPathMessage 0 0 PnfsGetChecksumMessage 0 0 PnfsGetCacheStatisticsMessage 0 0 PnfsGetChecksumAllMessage 0 0 PnfsSetChecksumMessage 0 0 PoolFileFlushedMessage 0 0 PnfsGetFileMetaDataMessage 0 0 PnfsDeleteEntryMessage 0 0 PnfsUpdateCacheStatisticsMessage 0 0 PnfsClearCacheLocationMessage 0 0 PnfsGetStorageInfoMessage 0 0 PnfsSetFileMetaDataMessage 0 0 PnfsCreateDirectoryMessage 0 0 PnfsRenameMessage 0 0 Total 0 0 PnfsManagerV3.Folded requests failed Total 0 0 [gks-xxx.scc.kit.edu] (PnfsManager) admin >
When navigating to another cell, one must first leave the current cell. This is achieved using the two dots (..) command. Please note that there is no cd preceeding the two dots.
[gks-xxx.scc.kit.edu] (PoolManagaer) admin > .. [gks-xxx.scc.kit.edu] (local) admin > cd PoolManager
Obtain a list of commands that begin with "pm"
[gks-xxx.scc.kit.edu] (PoolManager) admin > help pm pm ls [<section>] [-l] pm destroy <partitionName> # destroys parameter partition pm set [<partitionName>] OPTIONS # help pm set [dcachetogo.dcache.org] (PoolManager) admin > ..
Finding the location of a certain file
In this example, we will locate the file /pnfs/dcache.org/data/worldwritable/dccp.tst If you have deleted this file, please recreate it by copying /etc/group into dCache. First, we need to discover the file's pnfs-ID. This is a unique number, expressed in hexadecimal digits, that is used within dCache. To look up a file's ID, one must ask the PnfsManager. Navigate to this cell, if you are not already there. Within PnfsManager issue pnfsidof to determinate the file's pnfs-ID, then use the cacheinfoof command to determine on which pool/s the file is available. Please keep in mind that the pnfs-ID and pool location may be different to our example below.
[gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager [gks-xxx.scc.kit.edu] (PnfsManager) admin > pnfsidof /pnfs/dcache.org/data/worldwritable/dccp.tst 00003DD91CF131A742658424E65175CCD5B0 [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/worldwritable/dccp.tst pool1
To confirm this, navigate to the pool and use the rep ls and pf (short hand for pathfinder, which is available in the PnfsManager cell) commands to list all replicas and their metadata on that pool.
[gks-xxx.scc.kit.edu] (PnfsManager) admin > ..
[gks-xxx.scc.kit.edu] (local) admin > cd pool1
[gks-xxx.scc.kit.edu] (pool1) admin > rep ls 00003DD91CF131A742658424E65175CCD5B0
00003DD91CF131A742658424E65175CCD5B0 <-P---------L(0)[0]> 616248 si={<Unknown>:<Unknown>}
[gks-xxx.scc.kit.edu] (pool1) admin > pf 00003DD91CF131A742658424E65175CCD5B0
/pnfs/dcache.org/data/worldwritable/dccp.tst
Assigning pools to a directory
This section describes how to configure dCache such that the directory in which a file is created determines which pools are a valid choice for storing the data.
Introducing tags
The dCache file system supports some advanced features not present with many standard filesystems, namely directory tags. A directory tag, or tag for short, is a keyword:value pair assigned to a directory. Tags are inherited by sub-directories on creation and changing a tag of a directory will also change them in each sub-directory, unless the inheritance chain is broken by changing a tag of a directory directly. Directories in the namespace should never be moved, since this will mess up the inheritance structure and eventually break the whole system. You can list all the available tags by accessing the contents of a special file in each directory: ".(tags)()". On a freshly installed dCache instance — as we have it on our virtual machines — no tags defined.
Establishing a directory's storage class
Certain tags have a specific meaning. In particular, the two tags OSMTemplate and sGroup are used to determine the storage descriptor, from which the storage class is determined. The storage class is important as it can be used when identifying pools where data of a specific user is written to.
The OSMTemplate tag value will typically contain two words on a single line: the phrase "StoreName" followed by some value. The value of sGroup provides the storage group. Together they provide the storage descriptor as "store name:storage group". For disk-only systems, the HSM osm is assumed. This implies that the storage class is store “name:storage group@osm”. For example, given the following tag values
.(tag)(OSMTemplate):StoreName foo .(tag)(sGroup):BAR
the storage class is “foo:BAR@osm”.
Operation with tags
To assign a value to a keyword, simply send that value to the corresponding ".(tag)(tagname)" file. If previously the tag did not exist, it is created automatically:
root@gks-xxx.scc.kit.edu:~# cd /pnfs/dcache.org/data/worldwritable/ root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# echo "StoreName world" > '.(tag)(OSMTemplate)' root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# echo "ANY" > '.(tag)(sGroup)'
list the tags in the /pnfs/dcache.org/data/worldwritable/ directory.
root@gks-xxx.scc.kit.edu: ~# cat "/pnfs/dcache.org/data/worldwritable/.(tags)()" .(tag)(OSMTemplate) .(tag)(sGroup)
In the above example, we defined in the directory /pnfs/dcache.org/data/worldwritable/ two tags: OSMTemplate and sGroup. To obtain the value of these tags, just examine the contents of the file: ".(tag)(tagname)". For example, the contents of file ".(tag)(sGroup)" contains the value of the sGroup tag:
root@gks-xxx.scc.kit.edu:~# cd /pnfs/dcache.org/data/worldwritable/ root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# cat ".(tag)(sGroup)" ANY
The following handy trick lists all tags and their values in the current directory (please be aware of the differnt kind of quotation marks used here).
[root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/worldwritable# grep "" `cat '.(tags)()'` .(tag)(OSMTemplate):StoreName world .(tag)(sGroup):ANY
Selecting pools based on storage class
When faced with a data operation (such as writing data into dCache) the Pool Selection Unit (psu), part of the PoolManager, is used to determine which pools are suitable for that request. It is the Pool Selection Unit that must be configured so pools are selected based on the request's storage class. These are the steps to carry out, in order to assign one pool exclusively to each of two fictive experiments exp1 and exp2.
Create two different directories (if you plan to use dcap set permissions for unauthorized access.)
root@gks-xxx.scc.kit.edu:~# mkdir -m 777 /pnfs/dcache.org/data/exp{1,2}
Set different storage classes for both directories.
root@gks-xxx.scc.kit.edu:~# cd /pnfs/dcache.org/data/exp1 root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# echo "StoreName exp1" > ".(tag)(OSMTemplate)" root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# echo GENERATED > ".(tag)(sGroup)" root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# cd /pnfs/dcache.org/data/exp2 root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp2# echo "StoreName exp2" > ".(tag)(OSMTemplate)" root@gks-xxx.scc.kit.edu:/pnfs/dcache.org/data/exp1# echo GENERATED > ".(tag)(sGroup)"
Next, the PoolManager must be configured in such a way that it will select pools based on the two storage classes: "exp1:GENERATED@osm" and "exp2:GENERATED@osm". To do this, log into the admin interface and navigate to the PoolManager.
root@dcachetogo:/pnfs/dcache.org/data/exp2# ssh -1 -c blowfish -p 22223 -1 admin@localhost
admin@localhost's password:
dCache Admin (VII) (user=admin)
[gks-xxx.scc.kit.edu] (local) admin > cd PoolManager
[gks-xxx.scc.kit.edu] (PooolManager) admin >
We have only default settings which allows us to write the data from anywhere to each available pool. Let us create the storage classes: exp1:GENERATED@osm and exp1:GENERATED@osm
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu create unit -store exp1:GENERATED@osm [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create unit -store exp2:GENERATED@osm
Units and unit groups
Unit groups are sets of predicates (logic conditions) that must be met before a link is acceptable for some data transfer. Unit groups contain one or more units, each of which are simple tests. At least one unit within a units group must be true for the unit group to be suitable; this is a boolean "or" operation. The command "psu ls -l ugroup" will list all currently configured unit groups:
[dcachetogo.dcache.org] (PoolManager) admin > psu ls -l ugroup any-store unitList : *@* (type=Store;canonical=*@*;uGroups=1) world-net unitList : 0.0.0.0/0.0.0.0 (type=Net;canonical=0.0.0.0/0.0.0.0;uGroups=1) any-protocol unitList : */* (type=Protocol;canonical=*/*;uGroups=1)
Define a new unit group.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu create ugroup exp1-groups [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto ugroup exp1-groups exp1:GENERATED@osm [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create ugroup exp2-groups [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto ugroup exp2-groups exp2:GENERATED@osm
Validate our changes
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu ls -l ugroup any-store unitList : *@* (type=Store;canonical=*@*;uGroups=1) world-net unitList : 0.0.0.0/0.0.0.0 (type=Net;canonical=0.0.0.0/0.0.0.0;uGroups=1) exp2-groups unitList : exp2:GENERATED@osm (type=Store;canonical=exp2:GENERATED@osm;uGroups=1) any-protocol unitList : */* (type=Protocol;canonical=*/*;uGroups=1) exp1-groups unitList : exp1:GENERATED@osm (type=Store;canonical=exp1:GENERATED@osm;uGroups=1)
The unit group “exp1-group” is satisfactory (i.e. is true) if the disk operation involves the storage class “exp1:GENERATED@osm”. The unit group “exp2-groups” has similar constraints.
Links
Links may contain one or more unit groups. When choosing a link for some data operation, all a link's unit groups must be satisfied for that link to be considered; this is a Boolean "and" operation. If a link contains unit group “exp1–groups” then that link will only be selected if the file operation involves a directory where the storage class exp1:GENERATED@osm is configured. Likewise, if a link contains unit group “exp2-groups” then a similar condition holds. These unit groups are sufficient for our needs. Lets create the links.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link exp1-link world-net any-protocol exp1-groups [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link exp1-link -readpref=20 -writepref=20 -cachepref=20 -p2ppref=-1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link exp2-link world-net exp2-groups any-protocol [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link exp2-link -readpref=10 -writepref=10 -cachepref=10 -p2ppref=-1
To be sure that everything was done properly.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu ls link -l exp1-link readPref : 20 cachePref : 20 writePref : 20 p2pPref : -1 section : None linkGroup : None UGroups : world-net (links=3;units=1) any-protocol (links=3;units=1) exp1-groups (links=1;units=1) default-link readPref : 10 cachePref : 10 writePref : 10 p2pPref : -1 section : None linkGroup : None UGroups : any-store (links=1;units=2) world-net (links=3;units=1) any-protocol (links=3;units=1) exp2-link readPref : 10 cachePref : 10 writePref : 10 p2pPref : -1 section : None linkGroup : None UGroups : world-net (links=3;units=1) any-protocol (links=3;units=1) exp2-groups (links=1;units=1)
These two links are ideal for setting up pools such that they are for exclusive use by their respective VOs. Note how the link “exp1-link” contains the unit group “exp1-groups”. Recall that this ensures that this link is only ever selected if the file operation involves the storage class “exp1:GENERATED@osm”. This means that, for “exp1-link” to be available, the file operation must be within the directory that has tags defining an acceptable storage class. For our configuration, storage class ”exp1:GENERATED@osm” is only available within the directory /pnfs/dcache.org/data/exp1.
Next, we see how the pools are assigned to different links by listing the pool groups.
Pool groups
The default configuration has all pools assigned to the pool group default. This pool group is assigned to the link “default–link”, so is available for all file operations.
[dcachetogo.dcache.org] (PoolManager) admin > psu ls -l pgroup
To reserve a pool for the exp1 VO it must be removed from the default pool group and assigned to the exp1 pool group. Doing this makes the pool available for the “exp1-link” link, which will be selected preferentially for operations involving the exp1 VO storage class exp1:GENERATED@osm. The following will assign pool pool1 exclusively for VO exp1 and pool2 for VO exp2. This configuration leaves no pools for general usage.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu create pgroup exp1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu removefrom pgroup default pool1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto pgroup exp1 pool1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link exp1-link exp1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create pgroup exp2 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu removefrom pgroup default pool2 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto pgroup exp2 pool2 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link exp2-link exp2 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu ls -l pgroup default linkList : default-link (pref=10/10/-1/10;;ugroups=3;pools=1) poolList : exp2 linkList : exp2-link (pref=10/10/-1/10;;ugroups=3;pools=1) poolList : pool2 (enabled=true;active=11;rdOnly=false;links=0;pgroups=1;hsm=[];mode=enabled) exp1 linkList : exp1-link (pref=20/20/-1/20;;ugroups=3;pools=1) poolList : pool1 (enabled=true;active=14;rdOnly=false;links=0;pgroups=1;hsm=[];mode=enabled) .....
Make sure to save all this settings in PoolManager otherwise all these configuration will be lost if PoolManager is restarted:
[gks-xxx.scc.kit.edu] (PoolManager) admin > save
Lets test that it is working, copy a file three times at exp1 and check via admin interface where it is stored.
ui_user@gks-xxx.scc.kit.edu:~$ dccp /bin/bash dcap://localhost:22125/pnfs/dcache.org/data/exp1/exp1_f1.tst 868404 bytes (848 kiB) in 0 seconds ui_user@gks-xxx.scc.kit.edu:~$ cp /bin/bash /pnfs/dcache.org/data/exp1/exp1_f2.tst ui_user@gks-xxx.scc.kit.edu:~$ cp /bin/bash /pnfs/dcache.org/data/exp1/exp1_f3.tst
[gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp1/exp1_f1.tst pool1 [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp1/exp1_f2.tst pool1 [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp1/exp1_f3.tst pool1 [gks-xxx.scc.kit.edu] (PnfsManager) admin >
and repeat this task for folder /pnfs/dcache.org/data/exp2.
With current configuration no pool is left for general use. See what happens if you try to write data on the /pnfs/dcache.org/data/worldwritable directory.
ui_user@dcachetogo:/root$ dccp /bin/bash dcap://localhost:22125/pnfs/dcache.org/data/worldwritable/testbin-1 Failed open file in the dCache. Can't open destination file : "No write pool available for <size=0;new=true;stored=false;sClass=<Unknown>:<Unknown>;cClass=-;hsm=osm;accessLatency=NEARLINE;retentionPolicy=CUSTODIAL;uid=500;onerror=default;truncate=;alloc-size=868404;gid=500;path=/pnfs/dcache.org/data/worldwritable/testbin-1;timeout=-1;mode=0755;store=<Unknown>;group=<Unknown>;bfid=<Unknown>;> in the linkGroup [none]" System error: Input/output error
Configure read/write pools
On this example we demonstrate how PoolManager handles transfer requests, how it separates read and write operations on the pool level. Create a new pool as was described on the chapter Installation with name pool3. The "read/write/cached" restrictions applies in the PoolManager on the "links" level. Thus we bind pool3 to a new link which allows only read and cached operations.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu removefrom pgroup default pool3 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create pgroup read-pools [gks-xxx.scc.kit.edu] (PoolManager) admin > psu addto pgroup read-pools pool3 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link read-link world-net exp2-groups any-protocol [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link read-link -readpref=10 -writepref=0 -cachepref=10 -p2ppref=-1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link read-link read-pools
pool2 will be reconfigured to permit write operation only.
[gks-xxx.scc.kit.edu] (PoolManager) admin > psu create link write-link world-net exp2-groups any-protocol [gks-xxx.scc.kit.edu] (PoolManager) admin > psu set link write-link -readpref=0 -writepref=10 -cachepref=0 -p2ppref=-1 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu add link write-link exp2 [gks-xxx.scc.kit.edu] (PoolManager) admin > psu remove link exp2-link
With the current configuration of the PoolManager all data for exp2 will be stored on the pool2, but read operations from exp2 directory first will create an additional (cached) copy of a file on pool3, that will serve the data transfer in the end.
Write a new file...
ui_user@gks-xxx.scc.kit.edu:~$ dccp /bin/bash dcap://localhost:22125/pnfs/dcache.org/data/exp2/exp2_f5.tst
... and check for replicas.
[gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp2/exp2_f5.tst pool2
There is a bug in the current version of dCache, which we have to workaround first. This workaround requires to save the configuration of each pool and restart the poolDomain.
[gks-xxx.scc.kit.edu] (PnfsManager) admin > .. [gks-xxx.scc.kit.edu] (local) admin > cd pool1 [gks-035.scc.kit.edu] (pool1) admin > save [gks-035.scc.kit.edu] (pool1) admin > .. [gks-xxx.scc.kit.edu] (local) admin > cd pool2 [gks-035.scc.kit.edu] (pool2) admin > save [gks-035.scc.kit.edu] (pool2) admin > .. [gks-xxx.scc.kit.edu] (local) admin > cd pool3 [gks-035.scc.kit.edu] (pool3) admin > save [gks-035.scc.kit.edu] (pool3) admin > .. [gks-xxx.scc.kit.edu] (local) admin > logoff [root@gks-xxx dcache]# dcache restart gks-xxxDomain
Read the file and observe dCache enforcing the configuration.
dccp dcap://localhost:22125/pnfs/dcache.org/data/exp2/exp2_f5.tst /dev/null [gks-xxx.scc.kit.edu] (local) admin > cd PnfsManager [gks-xxx.scc.kit.edu] (PnfsManager) admin > cacheinfoof /pnfs/dcache.org/data/exp2/exp2_f5.tst pool2 pool3
Persistent configuration
All changes done after the last we saved the PoolManager was done "on-the-run". If we would stop dCache now, all these settings would be lost and the pools will again belong to the default link and pool groups, using the permissive units/unit groups from the installation. In order to make dCache working with the same configuration, we need another flat textfile.
dCache will parse the file /var/lib/dcache/config/poolmanager.conf on startup, if it is existing, otherwise it will apply said default configuration. So you can create a poolmanager.conf beforehand, listing all the commands we have issued so far, or you can tell dCache to save the current configuration.
[gks-xxx.scc.kit.edu] (PoolManager) admin > save
The basic pool configuration is done.
