Difference between revisions of "LSDF Online Storage: HPC"
Revision as of 14:27, 9 August 2019
LSDF_Online_Storage for HPC Cluster
The LSDF storage is permanently mounted on the login nodes and the data movers.
Both for interactive and for batch use, the environment variables $LSDF, $LSDFHOME and $LSDFPROJECTS are set.
Users of the filesystem LSDF Online Storage can furthermore transfer data to bwUniCluster and ForHLR via the tool rdata. T
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load.
>rdata "ls $LSDFPROJECTS" >rdata "ls $LSDFHOME/*.c" >rdata "cp foo $LSDFPROJECTS/ProjectPath/" or >rdata > cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“
shows how to use the command rdata.
rdata(1) Cluster Commands rdata(1) NAME rdata - File operations are executed on a data mover node of the cluster SYNOPSIS rdata [ COMMAND ] DESCRIPTION The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected. Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended. COMMANDS The following commands are allowed in COMMAND of rdata: cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command. help Output of a short help text. cd Changing the directory. quit exit Termination of rdata. ENVIRONMENT The following user specific environment variables are available with rdata: $HOME home directory $WORK work directory $UC1WORK uc1 work directory $PROJECT fh1 project directory $BWFILESTORAGE $BWFS bwFileStorage directory $LSDF LSDF for Data Intensive Science - LSDF Path $LSDFPROJECTS LSDF for Data Intensive Science - Project Directory $LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory EXAMPLES rdata rdata 'ls $BWFILESTORAGE/*.c' rdata 'cp $WORK/foo $BWFILESTORAGE' RETURNCODE rdata returns the exit status of the last command or 127 when the last command was not found. remote-fs-utils April 2013
Mount on the login nodes(only on ForHLR)
Cpu time limit: 3600s
Virtual memory limit: 8GB
Batch Jobs (only on ForHLR)
On the compute nodes, the LSDF usage is indicated to the batch system via "#SBATCH --constraint=LSDF".