LSDF Online Storage: HPC

LSDF_Online_Storage for HPC Cluster

The LSDF storage is permanently mounted on the login nodes and the data movers.

Both for interactive and for batch use, the environment variables $LSDF, $LSDFHOME and $LSDFPROJECTS are set.

rdata

Users of the filesystem LSDF Online Storage can furthermore transfer data to bwUniCluster and ForHLR via the tool rdata. Therefore the environment variables $LSDF, $LSDFHOME and $LSDFPROJECTS are set.

The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load.

Examples:

>rdata "ls $LSDFPROJECTS"
>rdata "ls $LSDFHOME/*.c"
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"
or
>rdata 
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“
 

The command

>man rdata

shows how to use the command rdata.

rdata(1)                                                                   Cluster Commands                                                                  rdata(1)

NAME
       rdata - File operations are executed on a data mover node of the cluster

SYNOPSIS
       rdata [ COMMAND ]

DESCRIPTION
       The program executes the specified command (with arguments) on a data mover node.  As remote host, a host with the lowest load is selected.

       Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.

COMMANDS
       The following commands are allowed in COMMAND of rdata:

       cp rm mv ls mkdir rmdir pwd echo rsync     Execution of the corresponding system command.

       help                                       Output of a short help text.

       cd                                         Changing the directory.

       quit exit                                  Termination of rdata.

ENVIRONMENT
       The following user specific environment variables are available with rdata:

       $HOME                                 home directory

       $WORK                                 work directory

       $UC1WORK                              uc1 work directory

       $PROJECT                              fh1 project directory

       $BWFILESTORAGE $BWFS                  bwFileStorage directory

       $LSDF                                 LSDF for Data Intensive Science - LSDF Path

       $LSDFPROJECTS                         LSDF for Data Intensive Science - Project Directory

       $LSDFHOME                             LSDF for Data Intensive Science - LSDF Home Directory

EXAMPLES
       rdata

       rdata 'ls $BWFILESTORAGE/*.c'

       rdata 'cp $WORK/foo $BWFILESTORAGE'

RETURNCODE
       rdata returns the exit status of the last command or 127 when the last command was not found.

remote-fs-utils                                                               April 2013  

Mount on the login nodes(only ForHLR)

Limits: Cpu time limit: 3600s Virtual memory limit: 8GB

Batch Jobs (only ForHLR)

On the compute nodes, the LSDF usage is indicated to the batch system via "#SBATCH --constraint=LSDF".

• Slurm HPC Workload Manager

---