Quickstart guide for gsatellite: Difference between revisions

From Lsdf
Jump to navigationJump to search
No edit summary
 
(13 intermediate revisions by 2 users not shown)
Line 1: Line 1:
&nbsp;&nbsp;[[BWDAHub|<small>BWDAHub</small>]]<br\>
&nbsp;&nbsp;[[Preparing access to the BWDAHub|<small>Preparing access to the BWDAHub</small>]]<br\>
&nbsp;&nbsp;[[Quickstart guide for gtransfer|<small>Quickstart guide for gtransfer</small>]]<br\>
&nbsp;&nbsp;<small>Quickstart guide for gsatellite</small>
----


= Job creation =
= Job creation =


For the gtransfer job creation first use gtransfer's bash completion to construct the transfer command as usual but don't actually start it. When finished with the source and destination URLs and your desired [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/gtransfer.1.md#options gtransfer options] copy the gtransfer command line to a text file and add the following header to it:
For the gtransfer job creation first use gtransfer's bash completion to construct the transfer command as usual but don't actually start it. When finished with the source and destination URLs and your desired [https://github.com/fr4nk5ch31n3r/gtransfer/blob/master/share/doc/gtransfer.1.md#options gtransfer options] (don't forget to add <span style="color:red"><code>-e</code></span> to the gt command line if an encrypted data transfer is desired) copy the gtransfer command line to a text file and add the following header to it:


<pre>
<pre style="color: grey">
#!/bin/bash
#!/bin/bash
#GSAT -T gtransfer
#GSAT -T gtransfer
Line 10: Line 17:
Example gtransfer job file contents:
Example gtransfer job file contents:


<pre>

#!/bin/bash
<pre style="color: grey">
#GSAT -T gtransfer
#!/bin/bash
gt -s my-source-host:/path/to/files/* \
#GSAT -T gtransfer</pre>
gt -s my-source-host:/path/to/files/* \
-d my-dest-archive:/~/public/ \
-e -v
-d my-dest-archive:/~/public/ \
</pre>
-v




= Start the job scheduler =
= Start the job scheduler =
Line 50: Line 55:
$ gqsub gsat-jobs/gt.job
$ gqsub gsat-jobs/gt.job
00002
00002



Upon successful submission <span style="color:red"><code>gqsub</code></span> will return the job id of the submitted job.
Upon successful submission <span style="color:red"><code>gqsub</code></span> will return the job id of the submitted job.

<div style="border:1px solid #808080; margin:5px 3px 0px 3px; padding:3px 5px 2px 5px; background:#CEEEEE">
'''NOTE:''' In order to perform a data transfer between GridFTP servers, a gtransfer job needs a GSI proxy credential (GPC) that is valid during the whole data transfer - i.e. a GPC similar to the one you usually use when logging in to the BWDAHub via GSISSH.

During login a limited delegated GPC is automatically created for this purpose from the one you use for login (see your login messages for details). "Limited" in this context means that it can only be used for GridFTP services by default and not for logins via GSISSH.

By default the limited delegated GPC will have a lifetime of 12 hours at maximum but this also depends on the lifetime of the GPC used for login. If it expires earlier, the limited delegated GPC will also expire earlier.

This also means that for a long running transfer (e.g. one that takes more than 12 hours to complete) you have to relogin to the BWDAHub within 12 hours after the limited delegated GPC was created in order to recreate a new limited delegated GPC from the GPC used for login. If your locally existing GPC would expire earlier, you should also recreate a new one locally before logging in to the BWDAHub to refreshen the limited delegated GPC there.

You can configure the lifetime of the limited delegated GPC by creating the file <span style="color:red"><code>dgpc-lifetime</code></span> in <span style="color:red"><code>$HOME/.globus</code></span>. The format of the file is a single line consisting of the desired lifetime in hours and minutes (the format is <span style="color:red"><code>HH:MM</code></span>, e.g.: enter <span style="color:red"><code>12:00</code></span> for 12 hours or <span style="color:red"><code>24:00</code></span> for 1 day). After creation and configuration of this file you need to relogin for activation of this change.
</div>




Line 66: Line 82:
running 00002 bwdahub.lsdf.kit.edu gt.job
running 00002 bwdahub.lsdf.kit.edu gt.job


With the opion <span style="color:red"><code>-f</code></span> you will get a *full* listing of all job attributes. You can limit the output to the job of interest by adding the job id:
With the opion <span style="color:red"><code>-f</code></span> you will get a ''full'' listing of all job attributes. You can limit the output to the job of interest by adding the job id:


$ gqstat -f 00002
$ gqstat -f 00002
Line 76: Line 92:
job.pid="7491"
job.pid="7491"
job.start="1435048776"
job.start="1435048776"



= Follow job execution =
= Follow job execution =
Line 108: Line 123:
....................................................................
....................................................................
............................................................
............................................................

You can exit by hitting CTRL+C.




----
You can exit by hitting Ctrl + C.
[[BWDAHub|<small>back to BWDAHub</small>]]&nbsp;&nbsp;&nbsp;[[BwDataArchiv|<small>back to bwDataArchiv</small>]]&nbsp;&nbsp;&nbsp;[[Using bwDataArchiv|<small>back to Using bwDataArchiv</small>]]

Latest revision as of 10:24, 6 December 2016

  BWDAHub<br\>   Preparing access to the BWDAHub<br\>   Quickstart guide for gtransfer<br\>   Quickstart guide for gsatellite



Job creation

For the gtransfer job creation first use gtransfer's bash completion to construct the transfer command as usual but don't actually start it. When finished with the source and destination URLs and your desired gtransfer options (don't forget to add -e to the gt command line if an encrypted data transfer is desired) copy the gtransfer command line to a text file and add the following header to it:

#!/bin/bash
#GSAT -T gtransfer

Example gtransfer job file contents:

#!/bin/bash
#GSAT -T gtransfer
gt -s my-source-host:/path/to/files/* \
   -d my-dest-archive:/~/public/ \
   -e -v

Start the job scheduler

To start the gsatellite scheduler - if it is not already running - issue the following command (the $ character just marks the shell prompt!):

$ gsatlcd --start

You can check that the job scheduler is running by using the --status option of gsatlcd. This should issue a message similar to the following, when it is running and nothing if it is not running.

$ gsatlcd --status
gsatlc hostname="bwdahub.lsdf.kit.edu"
gsatlc PID="7160"


Enabling email notifications

After starting the job scheduler but before submitting the gtransfer job, you can configure your email address to enable email notification for job start and job termination events. Create a file named myEmailAddress in the directory .gsatellite in your home directory and enter your email address.

Example contents:

john.doe@mail.domain.tld

After configuration you will receive an email after a job was started and after it terminated execution.


Job submission

For job submission you can use the gqsub command. Simply point gqsub to the gtransfer job created a few steps earlier and hit Enter.

$ gqsub gsat-jobs/gt.job
00002

Upon successful submission gqsub will return the job id of the submitted job.

NOTE: In order to perform a data transfer between GridFTP servers, a gtransfer job needs a GSI proxy credential (GPC) that is valid during the whole data transfer - i.e. a GPC similar to the one you usually use when logging in to the BWDAHub via GSISSH.

During login a limited delegated GPC is automatically created for this purpose from the one you use for login (see your login messages for details). "Limited" in this context means that it can only be used for GridFTP services by default and not for logins via GSISSH.

By default the limited delegated GPC will have a lifetime of 12 hours at maximum but this also depends on the lifetime of the GPC used for login. If it expires earlier, the limited delegated GPC will also expire earlier.

This also means that for a long running transfer (e.g. one that takes more than 12 hours to complete) you have to relogin to the BWDAHub within 12 hours after the limited delegated GPC was created in order to recreate a new limited delegated GPC from the GPC used for login. If your locally existing GPC would expire earlier, you should also recreate a new one locally before logging in to the BWDAHub to refreshen the limited delegated GPC there.

You can configure the lifetime of the limited delegated GPC by creating the file dgpc-lifetime in $HOME/.globus. The format of the file is a single line consisting of the desired lifetime in hours and minutes (the format is HH:MM, e.g.: enter 12:00 for 12 hours or 24:00 for 1 day). After creation and configuration of this file you need to relogin for activation of this change.


Check job status

You can check the status of the job with the gqstat command. Without any options this gives a summary table of all jobs that ran or are still running:

$ gqstat
   job.state job.id             job.execHost                 job.name
------------ ------ ------------------------ ------------------------
finished     00000  bwdahub.lsdf.kit.edu     succeed.job.example     
failed       00001  bwdahub.lsdf.kit.edu     fail.job.example        
running      00002  bwdahub.lsdf.kit.edu     gt.job                  

With the opion -f you will get a full listing of all job attributes. You can limit the output to the job of interest by adding the job id:

$ gqstat -f 00002
job.id="00002"
job.name="gt.job"
job.dir="/home/user/.gsatellite/gschedule/jobs/00002.d"
job.state="running"
job.execHost="bwdahub.lsdf.kit.edu"
job.pid="7491"
job.start="1435048776"

Follow job execution

To follow job execution in near real-time, you can use the tail tool and the job output files available in the job directory. E.g. to follow the standard output of the running gt job you can use the following command:

$ tail -f /home/user/.gsatellite/gschedule/jobs/00002.d/job.stdout

gt [.]: Transfer step: 0

gt [.]: tgftp transfer command:
tgftp --log-filename "tgftp_transfer_7499__step_0.log" --force-log-o
verwrite --post-command "if [[ ! -s /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile ]]; then rm /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile; fi" -- -dbg -cc 16 -tcp-bs 4M -p 16 -fast 
-restart -rst-retries 1 -dumpfile /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile -stall-timeout 30 -sync -sync-level 1 -f 
/home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.transferList
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
............................................................

You can exit by hitting CTRL+C.



back to BWDAHub   back to bwDataArchiv   back to Using bwDataArchiv