LSDF Online Storage: HPC: Difference between revisions
From Lsdf
Jump to navigationJump to search
(Created page with "<H1>LSDF_Online_Storage for HPC Cluster</H1> The LSDF storage is permanently mounted on the login nodes and the data movers. Both for interactive and for batch use, the envi...") |
No edit summary |
||
| Line 82: | Line 82: | ||
</pre> |
</pre> |
||
<H2> |
<H2>Mounton the login nodes</H2> |
||
Cpu time limit: 3600s |
|||
Virtual memory limit: 8GB |
|||
<H2>Batch Jobs</H2> |
<H2>Batch Jobs</H2> |
||
Revision as of 10:06, 11 June 2019
LSDF_Online_Storage for HPC Cluster
The LSDF storage is permanently mounted on the login nodes and the data movers.
Both for interactive and for batch use, the environment variables $LSDF, $LSDFHOME and $LSDFPROJECTS are set.
rdata
Users of the filesystem LSDF Online Storage can furthermore transfer data to bwUniCluster and ForHLR via the tool rdata. Therefore the environment variables $LSDF, $LSDFHOME and $LSDFPROJECTS are set.
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load.
Examples:
>rdata "ls $LSDFPROJECTS" >rdata "ls $LSDFHOME/*.c" >rdata "cp foo $LSDFPROJECTS/ProjectPath/" or >rdata > cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“
The command
>man rdata
shows how to use the command rdata.
rdata(1) Cluster Commands rdata(1)
NAME
rdata - File operations are executed on a data mover node of the cluster
SYNOPSIS
rdata [ COMMAND ]
DESCRIPTION
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.
COMMANDS
The following commands are allowed in COMMAND of rdata:
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.
help Output of a short help text.
cd Changing the directory.
quit exit Termination of rdata.
ENVIRONMENT
The following user specific environment variables are available with rdata:
$HOME home directory
$WORK work directory
$UC1WORK uc1 work directory
$PROJECT fh1 project directory
$BWFILESTORAGE $BWFS bwFileStorage directory
$LSDF LSDF for Data Intensive Science - LSDF Path
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory
EXAMPLES
rdata
rdata 'ls $BWFILESTORAGE/*.c'
rdata 'cp $WORK/foo $BWFILESTORAGE'
RETURNCODE
rdata returns the exit status of the last command or 127 when the last command was not found.
remote-fs-utils April 2013
Mounton the login nodes
Cpu time limit: 3600s Virtual memory limit: 8GB
Batch Jobs
