Quickstart guide for gsatellite: Difference between revisions

From Lsdf
Jump to navigationJump to search
(Formatierung beim zweiten Code-Block ist noch nicht o.k.)
No edit summary
Line 24: Line 24:
To start the gsatellite scheduler - if it is not already running - issue the following command (the <span style="color:red"><code>$</code></span> character just marks the shell prompt!):
To start the gsatellite scheduler - if it is not already running - issue the following command (the <span style="color:red"><code>$</code></span> character just marks the shell prompt!):


$ gsatlcd --start
<code>
$ gsatlcd --start
</code>


You can check that the job scheduler is running by using the <span style="color:red"><code>--status</code></span> option of gsatlcd. This should issue a message similar to the following, when it is running and nothing if it is not running.
You can check that the job scheduler is running by using the <span style="color:red"><code>--status</code></span> option of gsatlcd. This should issue a message similar to the following, when it is running and nothing if it is not running.


$ gsatlcd --status
<code>
gsatlc hostname="bwdahub.lsdf.kit.edu"
$ gsatlcd --status
gsatlc hostname="bwdahub.lsdf.kit.edu"
gsatlc PID="7160"
gsatlc PID="7160"
</code>




Line 43: Line 39:
Example contents:
Example contents:


john.doe@mail.domain.tld
<code>
john.doe@mail.domain.tld
</code>


After configuration you will receive an email after a job was started and after it terminated execution.
After configuration you will receive an email after a job was started and after it terminated execution.
Line 54: Line 48:
For job submission you can use the <span style="color:red"><code>gqsub</code></span> command. Simply point <span style="color:red"><code>gqsub</code></span> to the gtransfer job created a few steps earlier and hit Enter.
For job submission you can use the <span style="color:red"><code>gqsub</code></span> command. Simply point <span style="color:red"><code>gqsub</code></span> to the gtransfer job created a few steps earlier and hit Enter.


$ gqsub gsat-jobs/gt.job
<code>
00002
$ gqsub gsat-jobs/gt.job

00002
</code>


Upon successful submission <span style="color:red"><code>gqsub</code></span> will return the job id of the submitted job.
Upon successful submission <span style="color:red"><code>gqsub</code></span> will return the job id of the submitted job.
Line 66: Line 59:
You can check the status of the job with the <span style="color:red"><code>gqstat</code></span> command. Without any options this gives a summary table of all jobs that ran or are still running:
You can check the status of the job with the <span style="color:red"><code>gqstat</code></span> command. Without any options this gives a summary table of all jobs that ran or are still running:


$ gqstat
<code>
job.state job.id job.execHost job.name
$ gqstat
------------ ------ ------------------------ ------------------------
job.state job.id job.execHost job.name
finished 00000 bwdahub.lsdf.kit.edu succeed.job.example
------------ ------ ------------------------ ------------------------
finished 00000 bwdahub.lsdf.kit.edu succeed.job.example
failed 00001 bwdahub.lsdf.kit.edu fail.job.example
failed 00001 bwdahub.lsdf.kit.edu fail.job.example
running 00002 bwdahub.lsdf.kit.edu gt.job
running 00002 bwdahub.lsdf.kit.edu gt.job
</code>


With the opion <span style="color:red"><code>-f</code></span> you will get a *full* listing of all job attributes. You can limit the output to the job of interest by adding the job id:
With the opion <span style="color:red"><code>-f</code></span> you will get a *full* listing of all job attributes. You can limit the output to the job of interest by adding the job id:


$ gqstat -f 00002
<code>
$ gqstat -f 00002
job.id="00002"
job.id="00002"
job.name="gt.job"
job.dir="/home/user/.gsatellite/gschedule/jobs/00002.d"
job.name="gt.job"
job.state="running"
job.dir="/home/user/.gsatellite/gschedule/jobs/00002.d"
job.execHost="bwdahub.lsdf.kit.edu"
job.state="running"
job.pid="7491"
job.execHost="bwdahub.lsdf.kit.edu"
job.pid="7491"
job.start="1435048776"
job.start="1435048776"
</code>




Line 93: Line 82:
To follow job execution in near real-time, you can use the <span style="color:red"><code>tail</code></span> tool and the job output files available in the job directory. E.g. to follow the standard output of the running gt job you can use the following command:
To follow job execution in near real-time, you can use the <span style="color:red"><code>tail</code></span> tool and the job output files available in the job directory. E.g. to follow the standard output of the running gt job you can use the following command:


$ tail -f /home/user/.gsatellite/gschedule/jobs/00002.d/job.stdout
<code>
$ tail -f /home/user/.gsatellite/gschedule/jobs/00002.d/job.stdout
gt [.]: Transfer step: 0

gt [.]: Transfer step: 0
gt [.]: tgftp transfer command:
tgftp --log-filename "tgftp_transfer_7499__step_0.log" --force-log-o
verwrite --post-command "if [[ ! -s /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile ]]; then rm /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile; fi" -- -dbg -cc 16 -tcp-bs 4M -p 16 -fast
-restart -rst-retries 1 -dumpfile /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile -stall-timeout 30 -sync -sync-level 1 -f
/home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.transferList
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
............................................................


gt [.]: tgftp transfer command:
tgftp --log-filename "tgftp_transfer_7499__step_0.log" --force-log-o
verwrite --post-command "if [[ ! -s /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile ]]; then rm /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile; fi" -- -dbg -cc 16 -tcp-bs 4M -p 16 -fast
-restart -rst-retries 1 -dumpfile /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile -stall-timeout 30 -sync -sync-level 1 -f
/home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.transferList
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
............................................................
</code>


You can exit by hitting Ctrl + C.
You can exit by hitting Ctrl + C.

Revision as of 16:52, 6 July 2016

Job creation

For the gtransfer job creation first use gtransfer's bash completion to construct the transfer command as usual but don't actually start it. When finished with the source and destination URLs and your desired gtransfer options copy the gtransfer command line to a text file and add the following header to it:

#!/bin/bash
#GSAT -T gtransfer

Example gtransfer job file contents:


 #!/bin/bash
 #GSAT -T gtransfer
gt -s my-source-host:/path/to/files/* \
   -d my-dest-archive:/~/public/ \
   -v


Start the job scheduler

To start the gsatellite scheduler - if it is not already running - issue the following command (the $ character just marks the shell prompt!):

$ gsatlcd --start

You can check that the job scheduler is running by using the --status option of gsatlcd. This should issue a message similar to the following, when it is running and nothing if it is not running.

$ gsatlcd --status
gsatlc hostname="bwdahub.lsdf.kit.edu"
gsatlc PID="7160"


Enabling email notifications

After starting the job scheduler but before submitting the gtransfer job, you can configure your email address to enable email notification for job start and job termination events. Create a file named myEmailAddress in the directory .gsatellite in your home directory and enter your email address.

Example contents:

john.doe@mail.domain.tld

After configuration you will receive an email after a job was started and after it terminated execution.


Job submission

For job submission you can use the gqsub command. Simply point gqsub to the gtransfer job created a few steps earlier and hit Enter.

$ gqsub gsat-jobs/gt.job
00002


Upon successful submission gqsub will return the job id of the submitted job.


Check job status

You can check the status of the job with the gqstat command. Without any options this gives a summary table of all jobs that ran or are still running:

$ gqstat
   job.state job.id             job.execHost                 job.name
------------ ------ ------------------------ ------------------------
finished     00000  bwdahub.lsdf.kit.edu     succeed.job.example     
failed       00001  bwdahub.lsdf.kit.edu     fail.job.example        
running      00002  bwdahub.lsdf.kit.edu     gt.job                  

With the opion -f you will get a *full* listing of all job attributes. You can limit the output to the job of interest by adding the job id:

$ gqstat -f 00002
job.id="00002"
job.name="gt.job"
job.dir="/home/user/.gsatellite/gschedule/jobs/00002.d"
job.state="running"
job.execHost="bwdahub.lsdf.kit.edu"
job.pid="7491"
job.start="1435048776"


Follow job execution

To follow job execution in near real-time, you can use the tail tool and the job output files available in the job directory. E.g. to follow the standard output of the running gt job you can use the following command:

$ tail -f /home/user/.gsatellite/gschedule/jobs/00002.d/job.stdout

gt [.]: Transfer step: 0

gt [.]: tgftp transfer command:
tgftp --log-filename "tgftp_transfer_7499__step_0.log" --force-log-o
verwrite --post-command "if [[ ! -s /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile ]]; then rm /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile; fi" -- -dbg -cc 16 -tcp-bs 4M -p 16 -fast 
-restart -rst-retries 1 -dumpfile /home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.dumpfile -stall-timeout 30 -sync -sync-level 1 -f 
/home/user/.gtransfer/tmp/
b3229ce5e1c76098d133ad29b5390120bc8077f1/3e9b70048804cf87ab18be4e23a
aa7483b75c02f.transferList
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
............................................................


You can exit by hitting Ctrl + C.