LSDF Online Storage: HPC: Difference between revisions
From Lsdf
Jump to navigationJump to search
No edit summary |
No edit summary |
||
| Line 82: | Line 82: | ||
</pre> |
</pre> |
||
<H2>Mount on the login nodes(only on[http://www.scc.kit.edu/dienste/forhlr.php ForHLR]) </H2> |
<H2>Mount on the login nodes(only on [http://www.scc.kit.edu/dienste/forhlr.php ForHLR]) </H2> |
||
'''Limits:'''<br> |
'''Limits:'''<br> |
||
| Line 88: | Line 88: | ||
Virtual memory limit: 8GB |
Virtual memory limit: 8GB |
||
<H2>Batch Jobs (only |
<H2>Batch Jobs (only on [http://www.scc.kit.edu/dienste/forhlr.php ForHLR])</H2> |
||
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''". |
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''". |
||
Revision as of 10:22, 11 June 2019
LSDF_Online_Storage for HPC Cluster
The LSDF storage is permanently mounted on the login nodes and the data movers.
Both for interactive and for batch use, the environment variables $LSDF, $LSDFHOME and $LSDFPROJECTS are set.
rdata
Users of the filesystem LSDF Online Storage can furthermore transfer data to bwUniCluster and ForHLR via the tool rdata. T
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load.
Examples:
>rdata "ls $LSDFPROJECTS" >rdata "ls $LSDFHOME/*.c" >rdata "cp foo $LSDFPROJECTS/ProjectPath/" or >rdata > cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“
The command
>man rdata
shows how to use the command rdata.
rdata(1) Cluster Commands rdata(1)
NAME
rdata - File operations are executed on a data mover node of the cluster
SYNOPSIS
rdata [ COMMAND ]
DESCRIPTION
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.
COMMANDS
The following commands are allowed in COMMAND of rdata:
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.
help Output of a short help text.
cd Changing the directory.
quit exit Termination of rdata.
ENVIRONMENT
The following user specific environment variables are available with rdata:
$HOME home directory
$WORK work directory
$UC1WORK uc1 work directory
$PROJECT fh1 project directory
$BWFILESTORAGE $BWFS bwFileStorage directory
$LSDF LSDF for Data Intensive Science - LSDF Path
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory
EXAMPLES
rdata
rdata 'ls $BWFILESTORAGE/*.c'
rdata 'cp $WORK/foo $BWFILESTORAGE'
RETURNCODE
rdata returns the exit status of the last command or 127 when the last command was not found.
remote-fs-utils April 2013
Mount on the login nodes(only on ForHLR)
Limits:
Cpu time limit: 3600s
Virtual memory limit: 8GB
Batch Jobs (only on ForHLR)
On the compute nodes, the LSDF usage is indicated to the batch system via "#SBATCH --constraint=LSDF".
