LSDF Online Storage: HPC: Difference between revisions
From Lsdf
Jump to navigationJump to search
No edit summary |
(Add migration note) Tag: Replaced |
||
| Line 1: | Line 1: | ||
This page has been updated and migrated to the new docs site: |
|||
<H1>LSDF_Online_Storage for HPC Cluster</H1> |
|||
https://www.lsdf.kit.edu/docs/hpc/ |
|||
The LSDF storage is permanently mounted on the login nodes and the data movers. |
|||
Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set. |
|||
<!-- |
|||
<H2>rdata</H2> |
|||
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T |
|||
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load. |
|||
'''Examples:''' |
|||
<pre> |
|||
>rdata "ls $LSDFPROJECTS" |
|||
>rdata "ls $LSDFHOME/*.c" |
|||
>rdata "cp foo $LSDFPROJECTS/ProjectPath/" |
|||
or |
|||
>rdata |
|||
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“ |
|||
</pre> |
|||
The command |
|||
<pre> |
|||
>man rdata |
|||
</pre> |
|||
shows how to use the command rdata. |
|||
<pre> |
|||
rdata(1) Cluster Commands rdata(1) |
|||
NAME |
|||
rdata - File operations are executed on a data mover node of the cluster |
|||
SYNOPSIS |
|||
rdata [ COMMAND ] |
|||
DESCRIPTION |
|||
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected. |
|||
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended. |
|||
COMMANDS |
|||
The following commands are allowed in COMMAND of rdata: |
|||
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command. |
|||
help Output of a short help text. |
|||
cd Changing the directory. |
|||
quit exit Termination of rdata. |
|||
ENVIRONMENT |
|||
The following user specific environment variables are available with rdata: |
|||
$HOME home directory |
|||
$WORK work directory |
|||
$UC1WORK uc1 work directory |
|||
$PROJECT fh1 project directory |
|||
$BWFILESTORAGE $BWFS bwFileStorage directory |
|||
$LSDF LSDF for Data Intensive Science - LSDF Path |
|||
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory |
|||
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory |
|||
EXAMPLES |
|||
rdata |
|||
rdata 'ls $BWFILESTORAGE/*.c' |
|||
rdata 'cp $WORK/foo $BWFILESTORAGE' |
|||
RETURNCODE |
|||
rdata returns the exit status of the last command or 127 when the last command was not found. |
|||
remote-fs-utils April 2013 |
|||
</pre> |
|||
--> |
|||
<H2>Mount on the login nodes</H2> |
|||
'''Environment variables:'''<br> |
|||
LSDFPROJECTS=/lsdf/kit/inst/projects <br> |
|||
LSDFHOME=/lsdf/kit/inst/xy1234 <br> |
|||
LSDF=/lsdf <br> |
|||
'''Limits:'''<br> |
|||
Cpu time limit: 3600s <br> |
|||
Virtual memory limit: 8-12GB |
|||
<H2>Batch Jobs</H2> |
|||
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''". |
|||
* [https://wiki.bwhpc.de/e/BwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage bwUniCluster] |
|||
* [https://wiki.scc.kit.edu/hpc/index.php?title=ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage ForHLR] |
|||
<hr> |
|||
<br> |
|||
<br> |
|||
<br> |
|||
<br> |
|||
[[Category:LSDF_Online_Storage|HPC]] |
|||
Latest revision as of 18:51, 21 February 2023
This page has been updated and migrated to the new docs site:
