LSDF Online Storage: HPC: Difference between revisions

From Lsdf
Jump to navigationJump to search
No edit summary
No edit summary
Line 7: Line 7:
<H2>rdata</H2>
<H2>rdata</H2>


Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. Therefore the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load.
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load.
Line 84: Line 84:
<H2>Mount on the login nodes(only [http://www.scc.kit.edu/dienste/forhlr.php ForHLR]) </H2>
<H2>Mount on the login nodes(only [http://www.scc.kit.edu/dienste/forhlr.php ForHLR]) </H2>


Limits:
'''Limits:'''
Cpu time limit: 3600s
Cpu time limit: 3600s <br>
Virtual memory limit: 8GB
Virtual memory limit: 8GB



Revision as of 10:14, 11 June 2019

LSDF_Online_Storage for HPC Cluster

The LSDF storage is permanently mounted on the login nodes and the data movers.

Both for interactive and for batch use, the environment variables $LSDF, $LSDFHOME and $LSDFPROJECTS are set.

rdata

Users of the filesystem LSDF Online Storage can furthermore transfer data to bwUniCluster and ForHLR via the tool rdata. T

The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load.

Examples:

>rdata "ls $LSDFPROJECTS"
>rdata "ls $LSDFHOME/*.c"
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"
or
>rdata 
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“
 

The command

>man rdata

shows how to use the command rdata.

rdata(1)                                                                   Cluster Commands                                                                  rdata(1)

NAME
       rdata - File operations are executed on a data mover node of the cluster

SYNOPSIS
       rdata [ COMMAND ]

DESCRIPTION
       The program executes the specified command (with arguments) on a data mover node.  As remote host, a host with the lowest load is selected.

       Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.

COMMANDS
       The following commands are allowed in COMMAND of rdata:

       cp rm mv ls mkdir rmdir pwd echo rsync     Execution of the corresponding system command.

       help                                       Output of a short help text.

       cd                                         Changing the directory.

       quit exit                                  Termination of rdata.

ENVIRONMENT
       The following user specific environment variables are available with rdata:

       $HOME                                 home directory

       $WORK                                 work directory

       $UC1WORK                              uc1 work directory

       $PROJECT                              fh1 project directory

       $BWFILESTORAGE $BWFS                  bwFileStorage directory

       $LSDF                                 LSDF for Data Intensive Science - LSDF Path

       $LSDFPROJECTS                         LSDF for Data Intensive Science - Project Directory

       $LSDFHOME                             LSDF for Data Intensive Science - LSDF Home Directory

EXAMPLES
       rdata

       rdata 'ls $BWFILESTORAGE/*.c'

       rdata 'cp $WORK/foo $BWFILESTORAGE'

RETURNCODE
       rdata returns the exit status of the last command or 127 when the last command was not found.

remote-fs-utils                                                               April 2013  

Mount on the login nodes(only ForHLR)

Limits: Cpu time limit: 3600s
Virtual memory limit: 8GB

Batch Jobs (only ForHLR)

On the compute nodes, the LSDF usage is indicated to the batch system via "#SBATCH --constraint=LSDF".