Quickstart guide for gtransfer

  BWDAHub<br\>   Preparing access to the BWDAHub<br\>   Quickstart guide for gtransfer<br\>   Quickstart guide for gsatellite

---

Basics

1. Type gt and hit the ENTER/RETURN key to get a brief usage message. Use gt --help and man gt to get a description of all options.

2. To start a transfer, enter gt hit the SPACE key and then hit the TAB key three times. You'll get a listing of all available options. Start with -s to enter the source address. The - character was already provided by the gt bash completion. After entering s hit the SPACE key and enter your source address, e.g. gsiftp://gridftp.domain.tld:2811 . You can also hit the TAB key two times to get the preconfigured GridFTP source server addresses or host aliases. Add /~/ and then hit the TAB key two times to get a listing of the files and directories in your home directory on the remote server. Depending on the latency, it can take a few seconds until you see results and this will only work if your GSI proxy certificate is considered valid by the remote GridFTP server and you are trying to list a directory where you have rx (read and execute) permissions. Type in the beginning of your desired file or directory and hit the TAB key to complete the name. If you want to copy all files in a directory, add /* or just / to the end of the path. Now continue with the destination address. Add -d to the command line, hit the SPACE key and continue with the destination address just like you entered the source address. Enter a / at the end of the destination path.

3. To recursively copy all files and directories below a given directory, add the -r option to the gt command line.

NOTE: Please note that since 2016-04-27 we've changed the default behaviour of gtransfer to also encrypt the data channel by default. This change was done by creating shell aliases for the gt and gtransfer commands that include the -e option. You can list all active aliases with the alias command.

Though encrypting the data channel increases security for delicate data during transfer time, it also decreases performance considerably. Therefore if you don't need encryption for your data during transfer time, you can deactivate this new behaviour by using the following command (enter it without the shell prompt!):

$ unalias gtransfer gt

This will reset to the original behaviour of gtransfer (no data channel encryption) for your current session. To make this change permanent, include the command after the shell prompt in your shell configuration file ($HOME/.bash_profile for the bash) at the end of the file.

To revert back to the new behaviour, simply exit your current session and relogin, or - if you already included the unalias command in your shell configuration file - comment or remove the respective line in your shell configuration file, exit your current session and relogin.

NOTE: Shell aliases are not expanded inside of shell scripts. And gtransfer jobs ran by gsatellite are just shell scripts that call the gt(ransfer) command. This means that in order to use data channel encryption for gtransfer jobs, you have to add the option -e to your gt command line inside of the job script.

Hints

I have multiple user accounts at a remote GridFTP server. How can I choose a specific account?

This can be done by inserting a <USER>@ portion into your GridFTP URLs or prefixing host aliases with <USER>@. Replace <USER> with your desired username on the remote site.

Examples:

• GridFTP URL:

gsiftp://gridftp.domain.tld:2811/~/files/ => gsiftp://user1@gridftp.domain.tld:2811/~/files/*

• Host alias:

my-gridftp:/~/files/ => user1@my-gridftp:/~/files/

Can gtransfer automatically create non-existing directories on the destination side?

Yes, this is possible and activated by default. Just enter the desired name or path in your destination URL and gtransfer will automatically create non-existing directories on the destination side (with the help of globus-url-copy).

Use host aliases for your GridFTP servers (including the BWDA)

There are already two host aliases defined which point to the archive provided by the BWDataArchiv project:

• bwda:
• bwa: (the old name!)

You can use them instead of the longer host part of a GridFTP URL in the source and destination URLs, e.g. you can use bwda:/~/ instead of gsiftp://archive-sftp.lsdf.kit.edu:2811/~/ . To create your own host aliases, please refer to the host aliases documentation linked below.

What if the gtransfer command fails during a data transfer?

Globus-url-copy - the tool gtransfer actually uses through tgftp to transfer data - is configured by gtransfer to retry the transfer of files that failed to transfer successfully to the destination GridFTP server. And if that fails, gtransfer will retry the whole process ten times until giving up on the transfer. And even if that happens, you can later continue a failed or interrupted transfer by simply issuing the very same gtransfer command. Gtransfer stores state information about a transfer in your home directory below .gtransfer . So this mechanism will work on the same machine and with the same user account and as long as the state files are not touched in between.

What if I need to interrupt a data transfer?

You can always interrupt a gtransfer data transfer by hitting CTRL+C during a data transfer, which effectively sends a SIGINT to the gtransfer process group and interrupts the data transfer. You can continue the transfer from where it was interrupted by issuing the very same gtransfer command - as with failed transfers described above. The same restrictions - same host, same user account, no fiddling with the state files in between - apply here.

Documentation

General

• gtransfer GitHub repository and README

Man pages

Man(ual) pages are also available locally on the BWDAHub. Simply enter man and the name of the manpage (e.g. gtransfer or dpath ) to read a specific page. If man pages with the same name exist in different sections you also have to specify the section number after the man command but before the name of the man page to read a man page from a specific section. E.g. to read the dparam(5) man page - which contains the file format description for dparams - you would enter man 5 dparam .

Section 1

• gtransfer(1)
• dparam(1)
• dpath(1)
• halias(1)

Section 5

• dparam(5)
• dpath(5)

Special functionality

• Host aliases

---

back to BWDAHub   back to bwDataArchiv   back to Using bwDataArchiv