Difference between revisions of "LSDF Online Storage: HPC"
From Lsdf
(Add migration note) (Tag: Replaced) |
|||
Line 1: | Line 1: | ||
+ | This page has been updated and migrated to the new docs site: |
||
− | <H1>LSDF_Online_Storage for HPC Cluster</H1> |
||
+ | https://www.lsdf.kit.edu/docs/hpc/ |
||
− | The LSDF storage is permanently mounted on the login nodes and the data movers. |
||
− | |||
− | Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set. |
||
− | |||
− | <!-- |
||
− | <H2>rdata</H2> |
||
− | |||
− | Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T |
||
− | |||
− | The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load. |
||
− | |||
− | '''Examples:''' |
||
− | <pre> |
||
− | >rdata "ls $LSDFPROJECTS" |
||
− | >rdata "ls $LSDFHOME/*.c" |
||
− | >rdata "cp foo $LSDFPROJECTS/ProjectPath/" |
||
− | or |
||
− | >rdata |
||
− | > cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“ |
||
− | </pre> |
||
− | The command |
||
− | <pre> |
||
− | >man rdata |
||
− | </pre> |
||
− | shows how to use the command rdata. |
||
− | <pre> |
||
− | rdata(1) Cluster Commands rdata(1) |
||
− | |||
− | NAME |
||
− | rdata - File operations are executed on a data mover node of the cluster |
||
− | |||
− | SYNOPSIS |
||
− | rdata [ COMMAND ] |
||
− | |||
− | DESCRIPTION |
||
− | The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected. |
||
− | |||
− | Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended. |
||
− | |||
− | COMMANDS |
||
− | The following commands are allowed in COMMAND of rdata: |
||
− | |||
− | cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command. |
||
− | |||
− | help Output of a short help text. |
||
− | |||
− | cd Changing the directory. |
||
− | |||
− | quit exit Termination of rdata. |
||
− | |||
− | ENVIRONMENT |
||
− | The following user specific environment variables are available with rdata: |
||
− | |||
− | $HOME home directory |
||
− | |||
− | $WORK work directory |
||
− | |||
− | $UC1WORK uc1 work directory |
||
− | |||
− | $PROJECT fh1 project directory |
||
− | |||
− | $BWFILESTORAGE $BWFS bwFileStorage directory |
||
− | |||
− | $LSDF LSDF for Data Intensive Science - LSDF Path |
||
− | |||
− | $LSDFPROJECTS LSDF for Data Intensive Science - Project Directory |
||
− | |||
− | $LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory |
||
− | |||
− | EXAMPLES |
||
− | rdata |
||
− | |||
− | rdata 'ls $BWFILESTORAGE/*.c' |
||
− | |||
− | rdata 'cp $WORK/foo $BWFILESTORAGE' |
||
− | |||
− | RETURNCODE |
||
− | rdata returns the exit status of the last command or 127 when the last command was not found. |
||
− | |||
− | remote-fs-utils April 2013 |
||
− | </pre> |
||
− | --> |
||
− | |||
− | <H2>Mount on the login nodes</H2> |
||
− | |||
− | '''Environment variables:'''<br> |
||
− | LSDFPROJECTS=/lsdf/kit/inst/projects <br> |
||
− | LSDFHOME=/lsdf/kit/inst/xy1234 <br> |
||
− | LSDF=/lsdf <br> |
||
− | |||
− | '''Limits:'''<br> |
||
− | Cpu time limit: 3600s <br> |
||
− | Virtual memory limit: 8-12GB |
||
− | |||
− | <H2>Batch Jobs</H2> |
||
− | |||
− | On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''". |
||
− | * [https://wiki.bwhpc.de/e/BwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage bwUniCluster] |
||
− | * [https://wiki.scc.kit.edu/hpc/index.php?title=ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage ForHLR] |
||
− | <hr> |
||
− | <br> |
||
− | <br> |
||
− | <br> |
||
− | <br> |
||
− | |||
− | [[Category:LSDF_Online_Storage|HPC]] |
Latest revision as of 18:51, 21 February 2023
This page has been updated and migrated to the new docs site: