Preparing access to the BWDAHub: Difference between revisions
Felix.bach (talk | contribs) |
|||
| (8 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
[[BWDAHub|<small>BWDAHub</small>]] |
[[BWDAHub|<small>BWDAHub</small>]] |
||
<small>Preparing access to the BWDAHub</small |
<small>Preparing access to the BWDAHub</small> |
||
[[Quickstart guide for gtransfer|<small>Quickstart guide for gtransfer</small>]] |
[[Quickstart guide for gtransfer|<small>Quickstart guide for gtransfer</small>]] |
||
[[Quickstart guide for gsatellite|<small>Quickstart guide for gsatellite</small>]] |
[[Quickstart guide for gsatellite|<small>Quickstart guide for gsatellite</small>]] |
||
---- |
---- |
||
| Line 8: | Line 8: | ||
== Introduction == |
== Introduction == |
||
For using the BWDataArchiv (BWDA) service it is mandatory to use certificates. X.509 certificates allow the mapping of user credentials to real persons and are issued or signed by so-called registration authorities (RAs). Due to this mapping certificates allow for very trustworthy communication. Please see [https://info.pca.dfn.de/grid-ras.html 1] for a listing of Grid RAs available in Germany. |
For using gridFTP with the BWDataArchiv (BWDA) service it is mandatory to use certificates. X.509 certificates allow the mapping of user credentials to real persons and are issued or signed by so-called registration authorities (RAs). Due to this mapping certificates allow for very trustworthy communication. Please see [https://info.pca.dfn.de/grid-ras.html 1] for a listing of Grid RAs available in Germany. |
||
<div style="border:1px solid #808080; margin:5px 3px 0px 3px; padding:3px 5px 2px 5px; background:#CEEEEE">'''NOTE:''' Commands or options in brackets are optional and may depend on your currently used UID or your choice. The <span style="color:red">$</span> character marks the shell prompt for a non-root user in Linux. |
<div style="border:1px solid #808080; margin:5px 3px 0px 3px; padding:3px 5px 2px 5px; background:#CEEEEE">'''NOTE:''' Commands or options in brackets are optional and may depend on your currently used UID or your choice. The <span style="color:red">$</span> character marks the shell prompt for a non-root user in Linux. |
||
</div> |
</div> |
||
== GSI proxy credential == |
== GSI proxy credential == |
||
| Line 25: | Line 24: | ||
...or their affiliated RAs is required for data transfers. Other CAs can be accepted locally at the BWDataArchiv after manual configuration by BWDA personnel, but you also need to make sure that your derived GPC is also accepted by the source/destination GridFTP service (depending on the direction of the GridFTP data transfer). |
...or their affiliated RAs is required for data transfers. Other CAs can be accepted locally at the BWDataArchiv after manual configuration by BWDA personnel, but you also need to make sure that your derived GPC is also accepted by the source/destination GridFTP service (depending on the direction of the GridFTP data transfer). |
||
Please contact your IT department on how to acquire such a personal X.509 certificate. After receiving your personal X.509 certificate you need to forward the certificate's distinguished name (DN) to the BWDA personnel in order to activate access to the BWDAHub. To determine the DN you can use the following openssl command on your personal X.509 certificate: |
Please contact your IT department on how to acquire such a personal X.509 certificate. After receiving your personal X.509 certificate you need to forward the certificate's distinguished name (DN) to the BWDA personnel in order to activate access to the BWDAHub. You can do this either using the webinterface https://www.rda.kit.edu/bwDA or by sending an email to bwarchiv-support[at]lists.kit.edu. Note that it may take some days until your DN will be usable for data transfers. To determine the DN you can use the following openssl command on your personal X.509 certificate: |
||
<pre> |
<pre> |
||
| Line 59: | Line 58: | ||
* <span style="color:red"><code>$ grid-proxy-init [-valid <NUMBER_OF_HOURS:NUMBER_OF_MINUTES>]</code></span> |
* <span style="color:red"><code>$ grid-proxy-init [-valid <NUMBER_OF_HOURS:NUMBER_OF_MINUTES>]</code></span> |
||
* Your GSI proxy certficate will be valid for the given number of hours and minutes or 12 hours by default |
* Your GSI proxy certficate will be valid for the given number of hours and minutes or 12 hours by default |
||
== Install and configure gsissh == |
== Install and configure gsissh == |
||
| Line 101: | Line 99: | ||
If your download has a different hash value, don't use it and contact the [mailto:support-bwarchiv@lists.kit.edu BWDA personnel] for further assistance. |
If your download has a different hash value, don't use it and contact the [mailto:support-bwarchiv@lists.kit.edu BWDA personnel] for further assistance. |
||
<div style="border:1px solid #808080; margin:5px 3px 0px 3px; padding:3px 5px 2px 5px; background:#CEEEEE"> |
|||
'''NOTE:''' The original tarball was extended to also include the CA certificate of "GridKa-CA" under the old subject DN hash as name, which was and still is used by older OpenSSL versions (prior to v1.0.0). Just for reference, this '''older''' tarball's SHA256 hash value was: |
|||
</div> |
|||
<pre> |
|||
52136e8943f03b8accfc8573273786a84fe6ee50f4ad33a9a45e8d379d5199a8 |
|||
</pre> |
|||
Now untar it with <span style="color:red"><code>$ tar -xzf certificates.tar.gz</code></span> |
Now untar it with <span style="color:red"><code>$ tar -xzf certificates.tar.gz</code></span> |
||
== Login to the BWDAHub == |
== Login to the BWDAHub == |
||
| Line 119: | Line 108: | ||
<pre> |
<pre> |
||
$ gsissh bwdahub.lsdf.kit.edu |
$ gsissh myusername@bwdahub.lsdf.kit.edu |
||
Last login: Thu May 19 14:22:00 2016 from userhost.domain.tld |
Last login: Thu May 19 14:22:00 2016 from userhost.domain.tld |
||
+-[Welcome]-------------------------------------------------------------------+ |
+-[Welcome]-------------------------------------------------------------------+ |
||
| Line 169: | Line 158: | ||
=== Alternative GSI proxy credential creation === |
=== Alternative GSI proxy credential creation === |
||
Download the tool [https://www.rda.kit.edu/ |
Download the tool [https://www.rda.kit.edu/downloads/genproxy.bash genproxy] from our website (right klick on the link and use "Save Link as..." for downloading the file because otherwise it is displayed by the browser as it is a plain text bash script file). Before continuing, check that the computed SHA256 hash for the content of the tool matches the value given below (e.g. on Linux use <span style="color:red"><code>$ sha256sum genproxy</code></span>): |
||
<pre> |
<pre> |
||
| Line 177: | Line 166: | ||
If your download has a different hash value, don't use it and contact the [mailto:support-bwarchiv@lists.kit.edu BWDA personnel] for further assistance. |
If your download has a different hash value, don't use it and contact the [mailto:support-bwarchiv@lists.kit.edu BWDA personnel] for further assistance. |
||
Make the tool executable (<span style="color:red"><code>chmod +x genproxy</code></span>) and include it in your <span style="color:red"><code>PATH</code></span> environment variable so that you can use it from everywhere. Some Linux distributions also automatically include executables placed in <span style="color:red"><code>$HOME/bin</code></span> in your <span style="color:red"><code>PATH</code></span> environment variable. Alternatively copy <span style="color:red"><code>genproxy</code></span> to your <span style="color:red"><code>$HOME/.globus</code></span> directory and use it from this directory exclusively. Don't forget to prepend the path to the tool then or <span style="color:red"><code>./</code></span> when calling it from there directly. |
Make the tool executable (<span style="color:red"><code>$ chmod +x genproxy</code></span>) and include it in your <span style="color:red"><code>PATH</code></span> environment variable so that you can use it from everywhere. Some Linux distributions also automatically include executables placed in <span style="color:red"><code>$HOME/bin</code></span> in your <span style="color:red"><code>PATH</code></span> environment variable. Alternatively copy <span style="color:red"><code>genproxy</code></span> to your <span style="color:red"><code>$HOME/.globus</code></span> directory and use it from this directory exclusively. Don't forget to prepend the path to the tool then or <span style="color:red"><code>./</code></span> when calling it from there directly. |
||
When finished make sure your personal certificate (<span style="color:red"><code>usercert.pem</code></span>) and private key (<span style="color:red"><code>userkey.pem</code></span>) are available as PEM files in <span style="color:red"><code>$HOME/.globus</code></span>. If this is the case, you can now create a GSI proxy credential by just calling <span style="color:red"><code>genproxy</code></span> and entering the pass phrase to decrypt your private key: |
When finished make sure your personal certificate (<span style="color:red"><code>usercert.pem</code></span>) and private key (<span style="color:red"><code>userkey.pem</code></span>) are available as PEM files in <span style="color:red"><code>$HOME/.globus</code></span>. If this is the case, you can now create a GSI proxy credential by just calling <span style="color:red"><code>genproxy</code></span> and entering the pass phrase to decrypt your private key: |
||
| Line 189: | Line 178: | ||
</pre> |
</pre> |
||
<div style="border:1px solid #808080; margin:5px 3px 0px 3px; padding:3px 5px 2px 5px; background:#CEEEEE">'''NOTE:''' By default the <span style="color:red"><code>genproxy</code></span> tool uses a "non-guessable" name for the generated GPC. This is important on multi-user hosts because otherwise symlink attacks are possible in <span style="color:red"><code>/tmp</code></span> which can expose the GPC to other users. If you're running <span style="color:red"><code>genproxy</code></span> from a single user host, you can also predefine the path and name of the GPC in the environment variable <span style="color:red"><code>X509_USER_PROXY</code></span> (for example with <span style="color:red"><code>export X509_USER_PROXY="$HOME/.globus/mygpc"</code></span>).</div> |
<div style="border:1px solid #808080; margin:5px 3px 0px 3px; padding:3px 5px 2px 5px; background:#CEEEEE">'''NOTE:''' By default the <span style="color:red"><code>genproxy</code></span> tool uses a "non-guessable" name for the generated GPC. This is important on multi-user hosts because otherwise symlink attacks are possible in <span style="color:red"><code>/tmp</code></span> which can expose the GPC to other users. If you're running <span style="color:red"><code>genproxy</code></span> from a single user host, you can also predefine the path and name of the GPC in the environment variable <span style="color:red"><code>X509_USER_PROXY</code></span> (for example with <span style="color:red"><code>$ export X509_USER_PROXY="$HOME/.globus/mygpc"</code></span>).</div> |
||
=== Copy GSI proxy credential to the BWDAHub === |
=== Copy GSI proxy credential to the BWDAHub === |
||
Latest revision as of 15:56, 2 August 2019
BWDAHub Preparing access to the BWDAHub Quickstart guide for gtransfer Quickstart guide for gsatellite
Introduction
For using gridFTP with the BWDataArchiv (BWDA) service it is mandatory to use certificates. X.509 certificates allow the mapping of user credentials to real persons and are issued or signed by so-called registration authorities (RAs). Due to this mapping certificates allow for very trustworthy communication. Please see 1 for a listing of Grid RAs available in Germany.
GSI proxy credential
For accessing the BWDAHub via gsissh and performing your data transfers with GridFTP you need a GSI proxy credential (GPC) signed by your personal X.509 certificate. Please see 2 for more information about GSI proxy certificates.
I.e. this means that you first need a personal X.509 certificate signed by your organization or institute. In addition the source and destination GridFTP services must be able to verify your GPC to enable the data transfer. By default a GPC derived from a personal X.509 certificate issued by the two German grid certificate authorities:
- "DFN-Verein PCA Grid - G01"
- "GridKa-CA"
...or their affiliated RAs is required for data transfers. Other CAs can be accepted locally at the BWDataArchiv after manual configuration by BWDA personnel, but you also need to make sure that your derived GPC is also accepted by the source/destination GridFTP service (depending on the direction of the GridFTP data transfer).
Please contact your IT department on how to acquire such a personal X.509 certificate. After receiving your personal X.509 certificate you need to forward the certificate's distinguished name (DN) to the BWDA personnel in order to activate access to the BWDAHub. You can do this either using the webinterface https://www.rda.kit.edu/bwDA or by sending an email to bwarchiv-support[at]lists.kit.edu. Note that it may take some days until your DN will be usable for data transfers. To determine the DN you can use the following openssl command on your personal X.509 certificate:
$ openssl x509 -noout -subject -in <YOUR_PERSONAL_X509_CERTIFICATE_FILE>
Procedure (Linux)
Install the globus-proxy-utils package
RHEL and compatible:
- Activate EPEL repository (see 3 for details) or the Globus Alliance repository (see 4 for details)
- install package with
$ [sudo] yum install globus-proxy-utils
Debian and compatible:
- Activate the Globus Alliance repository (see 4 for details)
- install package with
$ [sudo] apt-get install globus-proxy-utils
Create a GSI proxy credential
Place your personal X.509 certficate and corresponding private key as PKCS#12 keystore in the directory $HOME/.globus (you usually can export such a keystore from your web browser!). File system access permissions are important, so please follow the next commands exactly:
$ mkdir $HOME/.globus; chmod 0700 $HOME/.globus$ umask 0177; touch $HOME/.globus/usercred.p12- Now export your keystore to the file
$HOME/.globus/usercred.p12
Actually create your GPC:
$ grid-proxy-init [-valid <NUMBER_OF_HOURS:NUMBER_OF_MINUTES>]- Your GSI proxy certficate will be valid for the given number of hours and minutes or 12 hours by default
Install and configure gsissh
Gsissh is a modified version of ssh which allows authentication with a GPC.
Procedure (Linux)
Install the gsi-openssh-clients package
RHEL and compatible:
- Activate EPEL repository (see 3 for details) or the Globus Alliance repository (see 4 for details)
- Install package with
$ [sudo] yum install gsi-openssh-clients
Debian and compatible:
- Activate the Globus Alliance repository (see 4 for details)
- Install package with
$ [sudo] apt-get install gsi-openssh-clients - On Debian additionally install the
libglobus-usage0package with$ [sudo] apt-get install libglobus-usage0
Configure the trusted CA certficates directory
When accessing a gsissh service on a remote site the gsissh client checks the authenticity of the host certificate offered before continuing with authentication of the user. To be able to verify the offered host certificate, the client needs to trust the certificate of the CA that signed the host certificate. The BWDAHub is hosted by KIT in Karlsruhe and hence its host certificate was signed by the "GridKa-CA". Hence your gsissh client does only need to trust the CA certificate of the "GridKa-CA" to successfully verify the host certificate of the BWDAHub.
First create the needed directory for the CA certificate:
[$ mkdir -p $HOME/.globus; chmod 0700 $HOME/.globus] $ mkdir $HOME/.globus/certificates
Then download the tarball containing the necessary certificate and support files via your web browser and place it in $HOME/.globus/certificates. The SHA256 hash of the tarball is:
154bf3698be4502dffa68b65e94d9e5e1c3be5b0e73e045294d24c523041b445
If your download has a different hash value, don't use it and contact the BWDA personnel for further assistance.
Now untar it with $ tar -xzf certificates.tar.gz
Login to the BWDAHub
After following the descriptions made above you will be able to login to the BWDAHub with the following command:
$ gsissh myusername@bwdahub.lsdf.kit.edu
Last login: Thu May 19 14:22:00 2016 from userhost.domain.tld
+-[Welcome]-------------------------------------------------------------------+
| |
| BWDAHub (bwdahub.lsdf.kit.edu) |
| |
+-[Contact]-------------------------------------------------------------------+
| |
| General support: |
| |
| * <support-bwarchiv@lists.kit.edu> |
| |
+-[Docs]----------------------------------------------------------------------+
| |
| Before you start, please have a look at the documentation available in: |
| |
| /usr/share/doc/bwdahub-0.5.0 |
| |
| * gtransfer-quickstart.md |
| * gsatellite-quickstart.md |
| |
+-[News]----------------------------------------------------------------------+
| |
| 2016-09-06: |
| New gtransfer version (v0.7.0) installed on the BWDAHub. With this version |
| it's now also possible to specify the user account on GridFTP servers when |
| using host aliases. I.e.: |
| |
| `$ gt -s user1@my-gridftp:/~/files/* -d user2@bwda:/~/files` |
| |
| ...will now also work. |
| |
| For more details about the new release visit: |
| |
| `https://github.com/fr4nk5ch31n3r/gtransfer/releases/tag/v0.7.0` |
| |
+-----------------------------------------------------------------------------+
INFO: Disk quotas for user user (uid 123):
Filesystem blocks quota limit grace files quota limit grace
/dev/sda4 123M 1024M 1536M 1627 0 0 OK
INFO: Creating limited delegated GSI proxy credential "/home/user/.gsatellite/tmp/defaultGsiProxyCredential" for gtransfer data transfer jobs... OK
[user@archive-gftp-fuse ~]$
I wasn't able to install the required Globus tools
If the Globus tools cannot be installed or if there are no binaries for your platform, please follow the next steps to get access to the BWDAHub and the tools installed there.
Alternative GSI proxy credential creation
Download the tool genproxy from our website (right klick on the link and use "Save Link as..." for downloading the file because otherwise it is displayed by the browser as it is a plain text bash script file). Before continuing, check that the computed SHA256 hash for the content of the tool matches the value given below (e.g. on Linux use $ sha256sum genproxy):
98b4232f709331527a5edcddc3d3561180e001bb634e775cab085d7a24d29344
If your download has a different hash value, don't use it and contact the BWDA personnel for further assistance.
Make the tool executable ($ chmod +x genproxy) and include it in your PATH environment variable so that you can use it from everywhere. Some Linux distributions also automatically include executables placed in $HOME/bin in your PATH environment variable. Alternatively copy genproxy to your $HOME/.globus directory and use it from this directory exclusively. Don't forget to prepend the path to the tool then or ./ when calling it from there directly.
When finished make sure your personal certificate (usercert.pem) and private key (userkey.pem) are available as PEM files in $HOME/.globus. If this is the case, you can now create a GSI proxy credential by just calling genproxy and entering the pass phrase to decrypt your private key:
Example:
$ genproxy Your identity: /C=DE/O=Grid/OU=University #1/CN=User Enter pass phrase for /home/user/.globus/userkey.pem: Your proxy `/tmp/x509up_p6729.file8x0ds0.1' is valid until: Wed Aug 17 12:31:23 CEST 2016
genproxy tool uses a "non-guessable" name for the generated GPC. This is important on multi-user hosts because otherwise symlink attacks are possible in /tmp which can expose the GPC to other users. If you're running genproxy from a single user host, you can also predefine the path and name of the GPC in the environment variable X509_USER_PROXY (for example with $ export X509_USER_PROXY="$HOME/.globus/mygpc").Copy GSI proxy credential to the BWDAHub
ssh.When finished with the GSI proxy credential creation, copy the created GSI proxy credential to the BWDAHub using scp and the exact name for the destination file (X509_USER_PROXY) as given in the following command:
$ scp /tmp/x509up_p6729.file8x0ds0.1 user@bwdahub.lsdf.kit.edu:.globus/X509_USER_PROXY
...or when you used the environment variable X509_USER_PROXY also with the following command:
$ scp "$X509_USER_PROXY" user@bwdahub.lsdf.kit.edu:.globus/X509_USER_PROXY
Logging in to the BWDAHub using ssh
Now you are ready to login to the BWDAHub and using the tools installed there with ssh alone.
ssh you need to use your username on the BWDAHub, as otherwise ssh tries to login with your local username.$ ssh user@bwdahub.lsdf.kit.edu
Last login: Thu May 19 14:22:00 2016 from userhost.domain.tld
+-[Welcome]-------------------------------------------------------------------+
| |
| BWDAHub (bwdahub.lsdf.kit.edu) |
| |
+-[Contact]-------------------------------------------------------------------+
| |
| General support: |
| |
| * <support-bwarchiv@lists.kit.edu> |
| |
+-[Docs]----------------------------------------------------------------------+
| |
| Before you start, please have a look at the documentation available in: |
| |
| /usr/share/doc/bwdahub-0.5.0 |
| |
| * gtransfer-quickstart.md |
| * gsatellite-quickstart.md |
| |
+-[News]----------------------------------------------------------------------+
| |
| 2016-09-06: |
| New gtransfer version (v0.7.0) installed on the BWDAHub. With this version |
| it's now also possible to specify the user account on GridFTP servers when |
| using host aliases. I.e.: |
| |
| `$ gt -s user1@my-gridftp:/~/files/* -d user2@bwda:/~/files` |
| |
| ...will now also work. |
| |
| For more details about the new release visit: |
| |
| `https://github.com/fr4nk5ch31n3r/gtransfer/releases/tag/v0.7.0` |
| |
+-----------------------------------------------------------------------------+
INFO: Disk quotas for user user (uid 123):
Filesystem blocks quota limit grace files quota limit grace
/dev/sda4 123M 1024M 1536M 1627 0 0 OK
INFO: Creating limited delegated GSI proxy credential "/home/user/.gsatellite/tmp/defaultGsiProxyCredential" for gtransfer data transfer jobs... OK
[user@archive-gftp-fuse ~]$
back to BWDAHub back to bwDataArchiv back to Using bwDataArchiv
