Difference between revisions of "LSDF Online Storage: HPC"

From Lsdf
(Add migration note)
(Tag: Replaced)
 
(4 intermediate revisions by one other user not shown)
Line 1: Line 1:
  +
This page has been updated and migrated to the new docs site:
<H1>LSDF_Online_Storage for HPC Cluster</H1>
 
   
  +
https://www.lsdf.kit.edu/docs/hpc/
The LSDF storage is permanently mounted on the login nodes and the data movers.
 
 
Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.
 
 
<!--
 
<H2>rdata</H2>
 
 
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T
 
 
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load.
 
 
'''Examples:'''
 
<pre>
 
>rdata "ls $LSDFPROJECTS"
 
>rdata "ls $LSDFHOME/*.c"
 
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"
 
or
 
>rdata
 
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“
 
</pre>
 
The command
 
<pre>
 
>man rdata
 
</pre>
 
shows how to use the command rdata.
 
<pre>
 
rdata(1) Cluster Commands rdata(1)
 
 
NAME
 
rdata - File operations are executed on a data mover node of the cluster
 
 
SYNOPSIS
 
rdata [ COMMAND ]
 
 
DESCRIPTION
 
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.
 
 
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.
 
 
COMMANDS
 
The following commands are allowed in COMMAND of rdata:
 
 
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.
 
 
help Output of a short help text.
 
 
cd Changing the directory.
 
 
quit exit Termination of rdata.
 
 
ENVIRONMENT
 
The following user specific environment variables are available with rdata:
 
 
$HOME home directory
 
 
$WORK work directory
 
 
$UC1WORK uc1 work directory
 
 
$PROJECT fh1 project directory
 
 
$BWFILESTORAGE $BWFS bwFileStorage directory
 
 
$LSDF LSDF for Data Intensive Science - LSDF Path
 
 
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory
 
 
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory
 
 
EXAMPLES
 
rdata
 
 
rdata 'ls $BWFILESTORAGE/*.c'
 
 
rdata 'cp $WORK/foo $BWFILESTORAGE'
 
 
RETURNCODE
 
rdata returns the exit status of the last command or 127 when the last command was not found.
 
 
remote-fs-utils April 2013
 
</pre>
 
-->
 
 
<H2>Mount on the login nodes</H2>
 
 
'''Limits:'''<br>
 
Cpu time limit: 3600s <br>
 
Virtual memory limit: 8GB
 
 
<H2>Batch Jobs</H2>
 
 
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''".
 
* [https://wiki.scc.kit.edu/hpc/index.php/ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage Slurm HPC Workload Manager]
 
* [https://wiki.bwhpc.de/e/BwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage bwUniCluster]
 
* [https://wiki.scc.kit.edu/hpc/index.php?title=ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage ForHLR]
 
<hr>
 
<br>
 
<br>
 
<br>
 
<br>
 
 
[[Category:LSDF_Online_Storage|HPC]]
 

Latest revision as of 18:51, 21 February 2023

This page has been updated and migrated to the new docs site:

https://www.lsdf.kit.edu/docs/hpc/