Quickstart guide for gtransfer: Difference between revisions

From Lsdf
Jump to navigationJump to search
(Link zu bwDAHub oben)
m (Gt on the BWDAHub is configured to retry failed transfers ten times.)
 
(6 intermediate revisions by the same user not shown)
Line 3: Line 3:
&nbsp;&nbsp;<small>Quickstart guide for gtransfer</small><br\>
&nbsp;&nbsp;<small>Quickstart guide for gtransfer</small><br\>
&nbsp;&nbsp;[[Quickstart guide for gsatellite|<small>Quickstart guide for gsatellite</small>]]
&nbsp;&nbsp;[[Quickstart guide for gsatellite|<small>Quickstart guide for gsatellite</small>]]
----


= Basics =
= Basics =
Line 13: Line 16:


3. To recursively copy all files and directories below a given directory, add the <span style="color:red"><code>-r</code></span> option to the gt command line.
3. To recursively copy all files and directories below a given directory, add the <span style="color:red"><code>-r</code></span> option to the gt command line.



<div style="border:1px solid #808080; margin:5px 3px 0px 3px; padding:3px 5px 2px 5px; background:#CEEEEE">'''NOTE:''' Please note that since 2016-04-27 we've changed the default behaviour of gtransfer to also encrypt the data channel by default. This change was done by creating shell aliases for the <span style="color:red"><code>gt</code></span> and <span style="color:red"><code>gtransfer</code></span> commands that include the <span style="color:red"><code>-e</code></span> option. You can list all active aliases with the <span style="color:red"><code>alias</code></span> command.
<div style="border:1px solid #808080; margin:5px 3px 0px 3px; padding:3px 5px 2px 5px; background:#CEEEEE">'''NOTE:''' Please note that since 2016-04-27 we've changed the default behaviour of gtransfer to also encrypt the data channel by default. This change was done by creating shell aliases for the <span style="color:red"><code>gt</code></span> and <span style="color:red"><code>gtransfer</code></span> commands that include the <span style="color:red"><code>-e</code></span> option. You can list all active aliases with the <span style="color:red"><code>alias</code></span> command.
Line 25: Line 27:


To revert back to the new behaviour, simply exit your current session and relogin, or - if you already included the <span style="color:red"><code>unalias</code></span> command in your shell configuration file - comment or remove the respective line in your shell configuration file, exit your current session and relogin.
To revert back to the new behaviour, simply exit your current session and relogin, or - if you already included the <span style="color:red"><code>unalias</code></span> command in your shell configuration file - comment or remove the respective line in your shell configuration file, exit your current session and relogin.

<div style="border:1px solid #808080; margin:5px 3px 0px 3px; padding:3px 5px 2px 5px; background:#CEEEEE">
'''NOTE:''' Shell aliases are not expanded inside of shell scripts. And gtransfer jobs ran by [[Quickstart guide for gsatellite|gsatellite]] are just shell scripts that call the <span style="color:red"><code>gt(ransfer)</code></span> command. This means that in order to use data channel encryption for gtransfer jobs, you have to add the option <span style="color:red"><code>-e</code></span> to your gt command line inside of the job script.
</div>



= Hints =
= Hints =


== I have multiple user accounts at a remote GridFTP server. How can I choose a specific account? ==

This can be done by inserting a <span style="color:red"><code><USER>@</code></span> portion into your GridFTP URLs or prefixing host aliases with <span style="color:red"><code><USER>@</code></span>. Replace <span style="color:red"><code><USER></code></span> with your desired username on the remote site.

Examples:

* GridFTP URL:
<span style="color:red"><code>gsiftp://gridftp.domain.tld:2811/~/files/</code></span> => <span style="color:red"><code>gsiftp://user1@gridftp.domain.tld:2811/~/files/*</code></span>
* Host alias:
<span style="color:red"><code>my-gridftp:/~/files/</code></span> => <span style="color:red"><code>user1@my-gridftp:/~/files/</code></span>



== Can gtransfer automatically create non-existing directories on the destination side? ==
== Can gtransfer automatically create non-existing directories on the destination side? ==


Yes, this is possible and activated by default. Just enter the desired name or path in your destination URL and gtransfer will automatically create non-existing directories on the destination side (with the help of [http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/user/#globus-url-copy globus-url-copy]).
Yes, this is possible and activated by default. Just enter the desired name or path in your destination URL and gtransfer will automatically create non-existing directories on the destination side (with the help of [http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/user/#globus-url-copy globus-url-copy]).



== Use host aliases for your GridFTP servers (including the BWDA) ==
== Use host aliases for your GridFTP servers (including the BWDA) ==
Line 40: Line 61:


You can use them instead of the longer host part of a GridFTP URL in the source and destination URLs, e.g. you can use <span style="color:red"><code>bwda:/~/</code></span> instead of <span style="color:red"><code>gsiftp://archive-sftp.lsdf.kit.edu:2811/~/</code></span> . To create your own host aliases, please refer to the host aliases documentation linked below.
You can use them instead of the longer host part of a GridFTP URL in the source and destination URLs, e.g. you can use <span style="color:red"><code>bwda:/~/</code></span> instead of <span style="color:red"><code>gsiftp://archive-sftp.lsdf.kit.edu:2811/~/</code></span> . To create your own host aliases, please refer to the host aliases documentation linked below.



== What if the gtransfer command fails during a data transfer? ==
== What if the gtransfer command fails during a data transfer? ==
[http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/user/#globus-url-copy Globus-url-copy] - the tool gtransfer actually uses through [https://github.com/fr4nk5ch31n3r/tgftp tgftp] to transfer data - is configured by gtransfer to retry the transfer of files that failed to transfer successfully to the destination GridFTP server. And if that fails, gtransfer will retry the whole process three times until giving up on the transfer. And even if that happens, you can later continue a failed or interrupted transfer by simply issuing the very same gtransfer command. Gtransfer stores state information about a transfer in your home directory below <span style="color:red"><code>.gtransfer</code></span> . So this mechanism will work on the same machine and with the same user account and as long as the state files are not touched in between.
[http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/user/#globus-url-copy Globus-url-copy] - the tool gtransfer actually uses through [https://github.com/fr4nk5ch31n3r/tgftp tgftp] to transfer data - is configured by gtransfer to retry the transfer of files that failed to transfer successfully to the destination GridFTP server. And if that fails, gtransfer will retry the whole process ten times until giving up on the transfer. And even if that happens, you can later continue a failed or interrupted transfer by simply issuing the very same gtransfer command. Gtransfer stores state information about a transfer in your home directory below <span style="color:red"><code>.gtransfer</code></span> . So this mechanism will work on the same machine and with the same user account and as long as the state files are not touched in between.



== What if I need to interrupt a data transfer? ==
== What if I need to interrupt a data transfer? ==
Line 52: Line 73:


= Documentation =
= Documentation =



== General ==
== General ==


* [https://github.com/fr4nk5ch31n3r/gtransfer/ gtransfer GitHub repository and README]
* [https://github.com/fr4nk5ch31n3r/gtransfer/ gtransfer GitHub repository and README]



== Man pages ==
== Man pages ==


Man(ual) pages are also available locally on the BWDAHub. Simply enter <span style="color:red"><code>man</code></span> and the name of the manpage (e.g. <span style="color:red"><code>gtransfer</code></span> or <span style="color:red"><code>dpath</code></span> ) to read a specific page. If man pages with the same name exist in different sections you also have to specify the section number after the <span style="color:red"><code>man</code></span> command but before the name of the man page to read a man page from a specific section. E.g. to read the <span style="color:red"><code>dparam(5)</code></span> man page - which contains the file format description for dparams - you would enter <span style="color:red"><code>man 5 dparam</code></span> .
Man(ual) pages are also available locally on the BWDAHub. Simply enter <span style="color:red"><code>man</code></span> and the name of the manpage (e.g. <span style="color:red"><code>gtransfer</code></span> or <span style="color:red"><code>dpath</code></span> ) to read a specific page. If man pages with the same name exist in different sections you also have to specify the section number after the <span style="color:red"><code>man</code></span> command but before the name of the man page to read a man page from a specific section. E.g. to read the <span style="color:red"><code>dparam(5)</code></span> man page - which contains the file format description for dparams - you would enter <span style="color:red"><code>man 5 dparam</code></span> .



=== Section 1 ===
=== Section 1 ===
Line 67: Line 91:
* [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/dpath.1.md dpath(1)]
* [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/dpath.1.md dpath(1)]
* [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/halias.1.md halias(1)]
* [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/halias.1.md halias(1)]



=== Section 5 ===
=== Section 5 ===
Line 72: Line 97:
* [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/dparam.5.md dparam(5)]
* [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/dparam.5.md dparam(5)]
* [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/dpath.5.md dpath(5)]
* [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/dpath.5.md dpath(5)]



== Special functionality ==
== Special functionality ==


* [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/host-aliases.md Host aliases]
* [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/host-aliases.md Host aliases]

----[[BWDAHub|<small>back to BWDAHub</small>]]&nbsp;&nbsp;&nbsp;[[BwDataArchiv|<small>back to bwDataArchiv</small>]]&nbsp;&nbsp;&nbsp;[[Using bwDataArchiv|<small>back to Using bwDataArchiv</small>]]

----
[[BWDAHub|<small>back to BWDAHub</small>]]&nbsp;&nbsp;&nbsp;[[BwDataArchiv|<small>back to bwDataArchiv</small>]]&nbsp;&nbsp;&nbsp;[[Using bwDataArchiv|<small>back to Using bwDataArchiv</small>]]

Latest revision as of 10:15, 6 December 2016

  BWDAHub<br\>   Preparing access to the BWDAHub<br\>   Quickstart guide for gtransfer<br\>   Quickstart guide for gsatellite



Basics

1. Type gt and hit the ENTER/RETURN key to get a brief usage message. Use gt --help and man gt to get a description of all options.

2. To start a transfer, enter gt hit the SPACE key and then hit the TAB key three times. You'll get a listing of all available options. Start with -s to enter the source address. The - character was already provided by the gt bash completion. After entering s hit the SPACE key and enter your source address, e.g. gsiftp://gridftp.domain.tld:2811 . You can also hit the TAB key two times to get the preconfigured GridFTP source server addresses or host aliases. Add /~/ and then hit the TAB key two times to get a listing of the files and directories in your home directory on the remote server. Depending on the latency, it can take a few seconds until you see results and this will only work if your GSI proxy certificate is considered valid by the remote GridFTP server and you are trying to list a directory where you have rx (read and execute) permissions. Type in the beginning of your desired file or directory and hit the TAB key to complete the name. If you want to copy all files in a directory, add /* or just / to the end of the path. Now continue with the destination address. Add -d to the command line, hit the SPACE key and continue with the destination address just like you entered the source address. Enter a / at the end of the destination path.

3. To recursively copy all files and directories below a given directory, add the -r option to the gt command line.

NOTE: Please note that since 2016-04-27 we've changed the default behaviour of gtransfer to also encrypt the data channel by default. This change was done by creating shell aliases for the gt and gtransfer commands that include the -e option. You can list all active aliases with the alias command.

Though encrypting the data channel increases security for delicate data during transfer time, it also decreases performance considerably. Therefore if you don't need encryption for your data during transfer time, you can deactivate this new behaviour by using the following command (enter it without the shell prompt!):

$ unalias gtransfer gt

This will reset to the original behaviour of gtransfer (no data channel encryption) for your current session. To make this change permanent, include the command after the shell prompt in your shell configuration file ($HOME/.bash_profile for the bash) at the end of the file.

To revert back to the new behaviour, simply exit your current session and relogin, or - if you already included the unalias command in your shell configuration file - comment or remove the respective line in your shell configuration file, exit your current session and relogin.

NOTE: Shell aliases are not expanded inside of shell scripts. And gtransfer jobs ran by gsatellite are just shell scripts that call the gt(ransfer) command. This means that in order to use data channel encryption for gtransfer jobs, you have to add the option -e to your gt command line inside of the job script.


Hints

I have multiple user accounts at a remote GridFTP server. How can I choose a specific account?

This can be done by inserting a <USER>@ portion into your GridFTP URLs or prefixing host aliases with <USER>@. Replace <USER> with your desired username on the remote site.

Examples:

  • GridFTP URL:

gsiftp://gridftp.domain.tld:2811/~/files/ => gsiftp://user1@gridftp.domain.tld:2811/~/files/*

  • Host alias:

my-gridftp:/~/files/ => user1@my-gridftp:/~/files/


Can gtransfer automatically create non-existing directories on the destination side?

Yes, this is possible and activated by default. Just enter the desired name or path in your destination URL and gtransfer will automatically create non-existing directories on the destination side (with the help of globus-url-copy).


Use host aliases for your GridFTP servers (including the BWDA)

There are already two host aliases defined which point to the archive provided by the BWDataArchiv project:

  • bwda:
  • bwa: (the old name!)

You can use them instead of the longer host part of a GridFTP URL in the source and destination URLs, e.g. you can use bwda:/~/ instead of gsiftp://archive-sftp.lsdf.kit.edu:2811/~/ . To create your own host aliases, please refer to the host aliases documentation linked below.


What if the gtransfer command fails during a data transfer?

Globus-url-copy - the tool gtransfer actually uses through tgftp to transfer data - is configured by gtransfer to retry the transfer of files that failed to transfer successfully to the destination GridFTP server. And if that fails, gtransfer will retry the whole process ten times until giving up on the transfer. And even if that happens, you can later continue a failed or interrupted transfer by simply issuing the very same gtransfer command. Gtransfer stores state information about a transfer in your home directory below .gtransfer . So this mechanism will work on the same machine and with the same user account and as long as the state files are not touched in between.

What if I need to interrupt a data transfer?

You can always interrupt a gtransfer data transfer by hitting CTRL+C during a data transfer, which effectively sends a SIGINT to the gtransfer process group and interrupts the data transfer. You can continue the transfer from where it was interrupted by issuing the very same gtransfer command - as with failed transfers described above. The same restrictions - same host, same user account, no fiddling with the state files in between - apply here.


Documentation

General


Man pages

Man(ual) pages are also available locally on the BWDAHub. Simply enter man and the name of the manpage (e.g. gtransfer or dpath ) to read a specific page. If man pages with the same name exist in different sections you also have to specify the section number after the man command but before the name of the man page to read a man page from a specific section. E.g. to read the dparam(5) man page - which contains the file format description for dparams - you would enter man 5 dparam .


Section 1


Section 5


Special functionality



back to BWDAHub   back to bwDataArchiv   back to Using bwDataArchiv