LSDF Online Storage: HPC: Difference between revisions

From Lsdf
Jump to navigationJump to search
No edit summary
(Add migration note)
Tag: Replaced
 
(13 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This page has been updated and migrated to the new docs site:
<H1>LSDF_Online_Storage for HPC Cluster</H1>


https://www.lsdf.kit.edu/docs/hpc/
The LSDF storage is permanently mounted on the login nodes and the data movers.

Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.

<H2>rdata</H2>

Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. Therefore the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load.

'''Examples:'''
<pre>
>rdata "ls $LSDFPROJECTS"
>rdata "ls $LSDFHOME/*.c"
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"
or
>rdata
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“
</pre>
The command
<pre>
>man rdata
</pre>
shows how to use the command rdata.
<pre>
rdata(1) Cluster Commands rdata(1)

NAME
rdata - File operations are executed on a data mover node of the cluster

SYNOPSIS
rdata [ COMMAND ]

DESCRIPTION
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.

Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.

COMMANDS
The following commands are allowed in COMMAND of rdata:

cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.

help Output of a short help text.

cd Changing the directory.

quit exit Termination of rdata.

ENVIRONMENT
The following user specific environment variables are available with rdata:

$HOME home directory

$WORK work directory

$UC1WORK uc1 work directory

$PROJECT fh1 project directory

$BWFILESTORAGE $BWFS bwFileStorage directory

$LSDF LSDF for Data Intensive Science - LSDF Path

$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory

$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory

EXAMPLES
rdata

rdata 'ls $BWFILESTORAGE/*.c'

rdata 'cp $WORK/foo $BWFILESTORAGE'

RETURNCODE
rdata returns the exit status of the last command or 127 when the last command was not found.

remote-fs-utils April 2013
</pre>

<H2>Mount on the login nodes(only [http://www.scc.kit.edu/dienste/forhlr.php ForHLR]) </H2>

Limits:
Cpu time limit: 3600s
Virtual memory limit: 8GB

<H2>Batch Jobs (only [http://www.scc.kit.edu/dienste/forhlr.php ForHLR])</H2>

On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''".
* [https://wiki.scc.kit.edu/hpc/index.php/ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage Slurm HPC Workload Manager]

<hr>
<br>
<br>
<br>
<br>

[[Category:LSDF_Online_Storage|HPC]]

Latest revision as of 18:51, 21 February 2023

This page has been updated and migrated to the new docs site:

https://www.lsdf.kit.edu/docs/hpc/