https://wiki.scc.kit.edu/lsdf/api.php?action=feedcontributions&user=Obholz&feedformat=atomLsdf - User contributions [en]2024-03-28T19:42:07ZUser contributionsMediaWiki 1.31.16https://wiki.scc.kit.edu/lsdf/index.php?title=Category:LSDF_Online_Storage&diff=5816Category:LSDF Online Storage2024-02-12T10:42:28Z<p>Obholz: </p>
<hr />
<div><br />
[[File:lsdf_online_storage_services.png|right|top|frameless|250px]]<br />
<br />
[[File:Lsdf_online_storage_services_2.jpg|right|top|thumb|250px|LSDF Online Storage @ KIT (SCC)]]<br />
<br />
The LSDF is a central storage system for large research data from data-intensive scientific disciplines. It is available to KIT researchers, their partners in the federal state and the Helmholtz Association as a multi-petabyte storage system. The service LSDF Online Storage enables KIT users and their partners in the state of Baden-Württemberg to access the storage resources of the LSDF. The service is used by institutes with research funded by the Helmholtz Association as well as insitutes at the university level of the KIT from different scientific disciplines. The project is sponsored and funded by the Ministry of Science, Research and the Arts of Baden-Württemberg (MWK), the German Research Foundation (DFG) and the Helmholtz Association of German Research Centres. It is operated by the Scientific Computing Center (SCC) at KIT. <br />
<br />
The LSDF enables scientists to store and access their research data via different services. These services, tailored to the requirements of the researchers, are offered on top of the LSDF storage. Important features are in particular high storage capacity, fast access to and transfer of the data, easy data access and management capabilities, as well as the connection to modern processing infrastructures such as the high-performance computing clusters based at KIT like the bwUniCluster or the ForHLR II.<br />
<br />
Access to the data is possible via standard protocols. Permissions to access the data are controlled by the data owners using group memberships and access control lists (ACLs). The LSDF Online Storage currently offers 10PB of usable storage capacity with a combined maximum read/write speed of the storage system of 25GByte/s. Today it is used by about 400 scientists.<br />
<br />
In order to use the LSDF Online Storage Service, interested users must first register for the service. Registration takes place via [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf bwIDM web interface]. Registration is only possible if a special Entitlement has been assigned to the user previously. Users usually receive this Entitlement after consultation with their responsible ITB or group representative or, if no project has yet been set up, by a separate request to the service operator.<br />
<br />
If you have any questions regarding registration, requesting a new storage project or further questions, please contact the service provider via the [https://bw-support.scc.kit.edu/?mode=ticket_submit&respounit=LSDF%20Online%20Storage BW support portal].<br />
<br />
The setup and size of the available storage space is determined by agreements between the users and the service provider and technically realized by quotas. For new projects a Data Management Plan (DMP) is developed and stored in coordination with the Institute of the Users and the representatives of the service provider. The DMPs are used for reporting and applying for funding to operate, maintain and expand the LSDF. The storage space of the LSDF Online Storage is currently usually available free of charge.<br />
<br />
Further information on the range of services and the usage of the service can be found in this Wiki the [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf user manual] and the current [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf terms of use].<br />
<br />
<br />
{| style="width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;"<br />
<!-- ACCESS --><br />
| style="width:50%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;" |<br />
{| style="width:100%; vertical-align:top; background:#f5fffa;"<br />
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">LSDF Online Storage Projects</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf Signing up to exising storage projects] (for users)<br />
* [https://www.lsdf.kit.edu/os/storagerequest/new/ Requesting new storage projects] (for project admins)<br />
* [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf Terms of use] (German version)<br />
* [[LSDF_Online_Storage:_Acknowledgements|Acknowledgements]]<br />
|-<br />
<!-- Documentation --><br />
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Documentation</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf User manual] (German version)<br />
|-<br />
<!-- Support --><br />
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Getting Help</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [https://bw-support.scc.kit.edu/?mode=ticket_submit&respounit=LSDF%20Online%20Storage Baden Württemberg Support Portal]<br />
|-<br />
<!-- Web sites<br />
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Web sites</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [http://www.scc.kit.edu/forschung/11843.php Large Scale Data Facility (LSDF)] (German version)<br />
* [http://www.scc.kit.edu/en/research/11843.php Large Scale Data Facility (LSDF)] (English version)<br />
* [http://www.scc.kit.edu/dienste/11228.php LSDF Online Storage] (German version)<br />
* [http://www.scc.kit.edu/en/services/11228.php LSDF Online Storage] (English version)<br />
|-<br />
--><br />
<!-- Monitoring<br />
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Monitoring</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [https://icinga.scc.kit.edu/ Icinga]<br />
|---><br />
|}<br />
<br />
| style="border:1px solid transparent;" |<br />
| style="width:60%; border:1px solid #BBBBBB; background:#f5faff; vertical-align:top;"|<br />
{| style="width:100%; vertical-align:top; background:#f5faff;"<br />
<br />
<!-- PROTOCOLS --><br />
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Accessing Data</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [[LSDF_Online_Storage:_Secure_Shell|Secure Shell (ssh, scp, sftp, sshfs)]]<br />
* [[LSDF_Online_Storage:_WebDAV|WebDAV / HTTPS]]<br><br />
* [[LSDF_Online_Storage:_SMB_/_CIFS|SMB / CIFS]] (KIT intern only)<br />
* [[LSDF_Online_Storage_NFS|NFS]] (KIT intern only)<br />
* [[LSDF_Online_Storage:_HPC|on the KIT HPC Clusters]]<br />
* [[LSDF_Online_Storage:_HDF_AAI|using the AAI of the Helmholtz Data Fedaration (HDF)]]<br />
<br />
|-<br />
<!-- SNAPSHOTS AND BACKUP --><br />
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Snapshots/Backup</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [[LSDF_Online_Storage:_Snapshots|Snapshots]] (for data retrieving and restoring)<br />
* [[LSDF_Online_Storage_Backup|Backup]]<br />
|-<br />
<!-- Miscellaneous --><br />
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Miscellaneous</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [[LSDF_Online_Storage_Filesystem|Filesystem characteristics]]<br />
* [[LSDF_Online_Storage_Quota|Quota]]<br />
|}<br />
|}<br />
<br><br />
<br><br />
-----<br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=Category:LSDF_Online_Storage&diff=5737Category:LSDF Online Storage2021-02-17T13:22:49Z<p>Obholz: </p>
<hr />
<div><br />
[[File:lsdf_online_storage_services.png|right|top|frameless|250px]]<br />
<br />
[[File:Lsdf_online_storage_services_2.jpg|right|top|thumb|250px|LSDF Online Storage @ KIT (SCC)]]<br />
<br />
The LSDF is a central storage system for large research data from data-intensive scientific disciplines. It is available to KIT researchers, their partners in the federal state and the Helmholtz Association as a multi-petabyte storage system. The service LSDF Online Storage enables KIT users and their partners in the state of Baden-Württemberg to access the storage resources of the LSDF. The service is used by institutes with research funded by the Helmholtz Association as well as insitutes at the university level of the KIT from different scientific disciplines. The project is sponsored and funded by the Ministry of Science, Research and the Arts of Baden-Württemberg (MWK), the German Research Foundation (DFG) and the Helmholtz Association of German Research Centres. It is operated by the Steinbuch Centre for Computing (SCC) at KIT.<br />
<br />
The LSDF enables scientists to store and access their research data via different services. These services, tailored to the requirements of the researchers, are offered on top of the LSDF storage. Important features are in particular high storage capacity, fast access to and transfer of the data, easy data access and management capabilities, as well as the connection to modern processing infrastructures such as the high-performance computing clusters based at KIT like the bwUniCluster or the ForHLR II.<br />
<br />
Access to the data is possible via standard protocols. Permissions to access the data are controlled by the data owners using group memberships and access control lists (ACLs). The LSDF Online Storage currently offers 10PB of usable storage capacity with a combined maximum read/write speed of the storage system of 25GByte/s. Today it is used by about 400 scientists.<br />
<br />
In order to use the LSDF Online Storage Service, interested users must first register for the service. Registration takes place via [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf bwIDM web interface]. Registration is only possible if a special Entitlement has been assigned to the user previously. Users usually receive this Entitlement after consultation with their responsible ITB or group representative or, if no project has yet been set up, by a separate request to the service operator.<br />
<br />
If you have any questions regarding registration, requesting a new storage project or further questions, please contact the service provider via the [https://bw-support.scc.kit.edu/?mode=ticket_submit&respounit=LSDF%20Online%20Storage BW support portal].<br />
<br />
The setup and size of the available storage space is determined by agreements between the users and the service provider and technically realized by quotas. For new projects a Data Management Plan (DMP) is developed and stored in coordination with the Institute of the Users and the representatives of the service provider. The DMPs are used for reporting and applying for funding to operate, maintain and expand the LSDF. The storage space of the LSDF Online Storage is currently usually available free of charge.<br />
<br />
Further information on the range of services and the usage of the service can be found in this Wiki the [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf user manual] and the current [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf terms of use].<br />
<br />
<br />
{| style="width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;"<br />
<!-- ACCESS --><br />
| style="width:50%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;" |<br />
{| style="width:100%; vertical-align:top; background:#f5fffa;"<br />
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">LSDF Online Storage Projects</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf Signing up to exising storage projects] (for users)<br />
* [https://www.lsdf.kit.edu/os/storagerequest/new/ Requesting new storage projects] (for project admins)<br />
* [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf Terms of use] (German version)<br />
* [[LSDF_Online_Storage:_Acknowledgements|Acknowledgements]]<br />
|-<br />
<!-- Documentation --><br />
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Documentation</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf User manual] (German version)<br />
|-<br />
<!-- Support --><br />
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Getting Help</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [https://bw-support.scc.kit.edu/?mode=ticket_submit&respounit=LSDF%20Online%20Storage Baden Württemberg Support Portal]<br />
|-<br />
<!-- Web sites<br />
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Web sites</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [http://www.scc.kit.edu/forschung/11843.php Large Scale Data Facility (LSDF)] (German version)<br />
* [http://www.scc.kit.edu/en/research/11843.php Large Scale Data Facility (LSDF)] (English version)<br />
* [http://www.scc.kit.edu/dienste/11228.php LSDF Online Storage] (German version)<br />
* [http://www.scc.kit.edu/en/services/11228.php LSDF Online Storage] (English version)<br />
|-<br />
--><br />
<!-- Monitoring<br />
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Monitoring</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [https://icinga.scc.kit.edu/ Icinga]<br />
|---><br />
|}<br />
<br />
| style="border:1px solid transparent;" |<br />
| style="width:60%; border:1px solid #BBBBBB; background:#f5faff; vertical-align:top;"|<br />
{| style="width:100%; vertical-align:top; background:#f5faff;"<br />
<br />
<!-- PROTOCOLS --><br />
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Accessing Data</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [[LSDF_Online_Storage:_Secure_Shell|Secure Shell (ssh, scp, sftp, sshfs)]]<br />
* [[LSDF_Online_Storage:_WebDAV|WebDAV / HTTPS]]<br><br />
* [[LSDF_Online_Storage:_SMB_/_CIFS|SMB / CIFS]] (KIT intern only)<br />
* [[LSDF_Online_Storage_NFS|NFS]] (KIT intern only)<br />
* [[LSDF_Online_Storage:_HPC|on the KIT HPC Clusters]] ([http://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster], [http://www.scc.kit.edu/dienste/forhlr.php ForHLR])<br />
<br />
|-<br />
<!-- SNAPSHOTS AND BACKUP --><br />
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Snapshots/Backup</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [[LSDF_Online_Storage:_Snapshots|Snapshots]] (for data retrieving and restoring)<br />
* [[LSDF_Online_Storage_Backup|Backup]]<br />
|-<br />
<!-- Miscellaneous --><br />
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Miscellaneous</div><br />
|-<br />
| style="color:#000; padding:2px 5px;" | <br />
* [[LSDF_Online_Storage_Filesystem|Filesystem characteristics]]<br />
* [[LSDF_Online_Storage_Quota|Quota]]<br />
|}<br />
|}<br />
<br><br />
<br><br />
-----<br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_HPC&diff=5710LSDF Online Storage: HPC2020-12-15T10:54:39Z<p>Obholz: </p>
<hr />
<div><H1>LSDF_Online_Storage for HPC Cluster</H1><br />
<br />
The LSDF storage is permanently mounted on the login nodes and the data movers.<br />
<br />
Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.<br />
<br />
<!--<br />
<H2>rdata</H2><br />
<br />
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T<br />
<br />
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load. <br />
<br />
'''Examples:'''<br />
<pre><br />
>rdata "ls $LSDFPROJECTS"<br />
>rdata "ls $LSDFHOME/*.c"<br />
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"<br />
or<br />
>rdata <br />
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“<br />
</pre><br />
The command <br />
<pre><br />
>man rdata<br />
</pre> <br />
shows how to use the command rdata. <br />
<pre><br />
rdata(1) Cluster Commands rdata(1)<br />
<br />
NAME<br />
rdata - File operations are executed on a data mover node of the cluster<br />
<br />
SYNOPSIS<br />
rdata [ COMMAND ]<br />
<br />
DESCRIPTION<br />
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.<br />
<br />
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.<br />
<br />
COMMANDS<br />
The following commands are allowed in COMMAND of rdata:<br />
<br />
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.<br />
<br />
help Output of a short help text.<br />
<br />
cd Changing the directory.<br />
<br />
quit exit Termination of rdata.<br />
<br />
ENVIRONMENT<br />
The following user specific environment variables are available with rdata:<br />
<br />
$HOME home directory<br />
<br />
$WORK work directory<br />
<br />
$UC1WORK uc1 work directory<br />
<br />
$PROJECT fh1 project directory<br />
<br />
$BWFILESTORAGE $BWFS bwFileStorage directory<br />
<br />
$LSDF LSDF for Data Intensive Science - LSDF Path<br />
<br />
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory<br />
<br />
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory<br />
<br />
EXAMPLES<br />
rdata<br />
<br />
rdata 'ls $BWFILESTORAGE/*.c'<br />
<br />
rdata 'cp $WORK/foo $BWFILESTORAGE'<br />
<br />
RETURNCODE<br />
rdata returns the exit status of the last command or 127 when the last command was not found.<br />
<br />
remote-fs-utils April 2013 <br />
</pre><br />
--><br />
<br />
<H2>Mount on the login nodes</H2> <br />
<br />
'''Environment variables:'''<br><br />
LSDFPROJECTS=/lsdf/kit/inst/projects <br><br />
LSDFHOME=/lsdf/kit/inst/xy1234 <br><br />
LSDF=/lsdf <br><br />
<br />
'''Limits:'''<br><br />
Cpu time limit: 3600s <br><br />
Virtual memory limit: 8-12GB<br />
<br />
<H2>Batch Jobs</H2> <br />
<br />
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''".<br />
* [https://wiki.bwhpc.de/e/BwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage bwUniCluster]<br />
* [https://wiki.scc.kit.edu/hpc/index.php?title=ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage ForHLR]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|HPC]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_HPC&diff=5709LSDF Online Storage: HPC2020-12-15T10:54:17Z<p>Obholz: </p>
<hr />
<div><H1>LSDF_Online_Storage for HPC Cluster</H1><br />
<br />
The LSDF storage is permanently mounted on the login nodes and the data movers.<br />
<br />
Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.<br />
<br />
<!--<br />
<H2>rdata</H2><br />
<br />
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T<br />
<br />
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load. <br />
<br />
'''Examples:'''<br />
<pre><br />
>rdata "ls $LSDFPROJECTS"<br />
>rdata "ls $LSDFHOME/*.c"<br />
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"<br />
or<br />
>rdata <br />
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“<br />
</pre><br />
The command <br />
<pre><br />
>man rdata<br />
</pre> <br />
shows how to use the command rdata. <br />
<pre><br />
rdata(1) Cluster Commands rdata(1)<br />
<br />
NAME<br />
rdata - File operations are executed on a data mover node of the cluster<br />
<br />
SYNOPSIS<br />
rdata [ COMMAND ]<br />
<br />
DESCRIPTION<br />
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.<br />
<br />
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.<br />
<br />
COMMANDS<br />
The following commands are allowed in COMMAND of rdata:<br />
<br />
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.<br />
<br />
help Output of a short help text.<br />
<br />
cd Changing the directory.<br />
<br />
quit exit Termination of rdata.<br />
<br />
ENVIRONMENT<br />
The following user specific environment variables are available with rdata:<br />
<br />
$HOME home directory<br />
<br />
$WORK work directory<br />
<br />
$UC1WORK uc1 work directory<br />
<br />
$PROJECT fh1 project directory<br />
<br />
$BWFILESTORAGE $BWFS bwFileStorage directory<br />
<br />
$LSDF LSDF for Data Intensive Science - LSDF Path<br />
<br />
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory<br />
<br />
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory<br />
<br />
EXAMPLES<br />
rdata<br />
<br />
rdata 'ls $BWFILESTORAGE/*.c'<br />
<br />
rdata 'cp $WORK/foo $BWFILESTORAGE'<br />
<br />
RETURNCODE<br />
rdata returns the exit status of the last command or 127 when the last command was not found.<br />
<br />
remote-fs-utils April 2013 <br />
</pre><br />
--><br />
<br />
<H2>Mount on the login nodes</H2> <br />
<br />
'''Environment variables:'''<br><br />
LSDFPROJECTS=/lsdf/kit/inst/projects <br><br />
LSDFHOME=/lsdf/kit/inst/fg9068 <br><br />
LSDF=/lsdf <br><br />
<br />
'''Limits:'''<br><br />
Cpu time limit: 3600s <br><br />
Virtual memory limit: 8-12GB<br />
<br />
<H2>Batch Jobs</H2> <br />
<br />
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''".<br />
* [https://wiki.bwhpc.de/e/BwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage bwUniCluster]<br />
* [https://wiki.scc.kit.edu/hpc/index.php?title=ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage ForHLR]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|HPC]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_HPC&diff=5708LSDF Online Storage: HPC2020-12-15T10:53:44Z<p>Obholz: </p>
<hr />
<div><H1>LSDF_Online_Storage for HPC Cluster</H1><br />
<br />
The LSDF storage is permanently mounted on the login nodes and the data movers.<br />
<br />
Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.<br />
<br />
<!--<br />
<H2>rdata</H2><br />
<br />
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T<br />
<br />
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load. <br />
<br />
'''Examples:'''<br />
<pre><br />
>rdata "ls $LSDFPROJECTS"<br />
>rdata "ls $LSDFHOME/*.c"<br />
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"<br />
or<br />
>rdata <br />
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“<br />
</pre><br />
The command <br />
<pre><br />
>man rdata<br />
</pre> <br />
shows how to use the command rdata. <br />
<pre><br />
rdata(1) Cluster Commands rdata(1)<br />
<br />
NAME<br />
rdata - File operations are executed on a data mover node of the cluster<br />
<br />
SYNOPSIS<br />
rdata [ COMMAND ]<br />
<br />
DESCRIPTION<br />
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.<br />
<br />
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.<br />
<br />
COMMANDS<br />
The following commands are allowed in COMMAND of rdata:<br />
<br />
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.<br />
<br />
help Output of a short help text.<br />
<br />
cd Changing the directory.<br />
<br />
quit exit Termination of rdata.<br />
<br />
ENVIRONMENT<br />
The following user specific environment variables are available with rdata:<br />
<br />
$HOME home directory<br />
<br />
$WORK work directory<br />
<br />
$UC1WORK uc1 work directory<br />
<br />
$PROJECT fh1 project directory<br />
<br />
$BWFILESTORAGE $BWFS bwFileStorage directory<br />
<br />
$LSDF LSDF for Data Intensive Science - LSDF Path<br />
<br />
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory<br />
<br />
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory<br />
<br />
EXAMPLES<br />
rdata<br />
<br />
rdata 'ls $BWFILESTORAGE/*.c'<br />
<br />
rdata 'cp $WORK/foo $BWFILESTORAGE'<br />
<br />
RETURNCODE<br />
rdata returns the exit status of the last command or 127 when the last command was not found.<br />
<br />
remote-fs-utils April 2013 <br />
</pre><br />
--><br />
<br />
<H2>Mount on the login nodes</H2> <br />
<br />
'''Environment variables:'''<br><br />
LSDFPROJECTS=/lsdf/kit/inst/projects<br />
LSDFHOME=/lsdf/kit/inst/fg9068<br />
LSDF=/lsdf<br />
<br />
'''Limits:'''<br><br />
Cpu time limit: 3600s <br><br />
Virtual memory limit: 8-12GB<br />
<br />
<H2>Batch Jobs</H2> <br />
<br />
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''".<br />
* [https://wiki.bwhpc.de/e/BwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage bwUniCluster]<br />
* [https://wiki.scc.kit.edu/hpc/index.php?title=ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage ForHLR]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|HPC]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_HPC&diff=5707LSDF Online Storage: HPC2020-12-15T10:50:15Z<p>Obholz: </p>
<hr />
<div><H1>LSDF_Online_Storage for HPC Cluster</H1><br />
<br />
The LSDF storage is permanently mounted on the login nodes and the data movers.<br />
<br />
Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.<br />
<br />
<!--<br />
<H2>rdata</H2><br />
<br />
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T<br />
<br />
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load. <br />
<br />
'''Examples:'''<br />
<pre><br />
>rdata "ls $LSDFPROJECTS"<br />
>rdata "ls $LSDFHOME/*.c"<br />
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"<br />
or<br />
>rdata <br />
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“<br />
</pre><br />
The command <br />
<pre><br />
>man rdata<br />
</pre> <br />
shows how to use the command rdata. <br />
<pre><br />
rdata(1) Cluster Commands rdata(1)<br />
<br />
NAME<br />
rdata - File operations are executed on a data mover node of the cluster<br />
<br />
SYNOPSIS<br />
rdata [ COMMAND ]<br />
<br />
DESCRIPTION<br />
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.<br />
<br />
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.<br />
<br />
COMMANDS<br />
The following commands are allowed in COMMAND of rdata:<br />
<br />
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.<br />
<br />
help Output of a short help text.<br />
<br />
cd Changing the directory.<br />
<br />
quit exit Termination of rdata.<br />
<br />
ENVIRONMENT<br />
The following user specific environment variables are available with rdata:<br />
<br />
$HOME home directory<br />
<br />
$WORK work directory<br />
<br />
$UC1WORK uc1 work directory<br />
<br />
$PROJECT fh1 project directory<br />
<br />
$BWFILESTORAGE $BWFS bwFileStorage directory<br />
<br />
$LSDF LSDF for Data Intensive Science - LSDF Path<br />
<br />
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory<br />
<br />
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory<br />
<br />
EXAMPLES<br />
rdata<br />
<br />
rdata 'ls $BWFILESTORAGE/*.c'<br />
<br />
rdata 'cp $WORK/foo $BWFILESTORAGE'<br />
<br />
RETURNCODE<br />
rdata returns the exit status of the last command or 127 when the last command was not found.<br />
<br />
remote-fs-utils April 2013 <br />
</pre><br />
--><br />
<br />
<H2>Mount on the login nodes</H2> <br />
<br />
'''Limits:'''<br><br />
Cpu time limit: 3600s <br><br />
Virtual memory limit: 8GB<br />
<br />
<H2>Batch Jobs</H2> <br />
<br />
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''".<br />
* [https://wiki.bwhpc.de/e/BwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage bwUniCluster]<br />
* [https://wiki.scc.kit.edu/hpc/index.php?title=ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage ForHLR]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|HPC]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_HPC&diff=5706LSDF Online Storage: HPC2020-12-15T10:13:45Z<p>Obholz: </p>
<hr />
<div><H1>LSDF_Online_Storage for HPC Cluster</H1><br />
<br />
The LSDF storage is permanently mounted on the login nodes and the data movers.<br />
<br />
Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.<br />
<br />
<!--<br />
<H2>rdata</H2><br />
<br />
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T<br />
<br />
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load. <br />
<br />
'''Examples:'''<br />
<pre><br />
>rdata "ls $LSDFPROJECTS"<br />
>rdata "ls $LSDFHOME/*.c"<br />
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"<br />
or<br />
>rdata <br />
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“<br />
</pre><br />
The command <br />
<pre><br />
>man rdata<br />
</pre> <br />
shows how to use the command rdata. <br />
<pre><br />
rdata(1) Cluster Commands rdata(1)<br />
<br />
NAME<br />
rdata - File operations are executed on a data mover node of the cluster<br />
<br />
SYNOPSIS<br />
rdata [ COMMAND ]<br />
<br />
DESCRIPTION<br />
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.<br />
<br />
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.<br />
<br />
COMMANDS<br />
The following commands are allowed in COMMAND of rdata:<br />
<br />
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.<br />
<br />
help Output of a short help text.<br />
<br />
cd Changing the directory.<br />
<br />
quit exit Termination of rdata.<br />
<br />
ENVIRONMENT<br />
The following user specific environment variables are available with rdata:<br />
<br />
$HOME home directory<br />
<br />
$WORK work directory<br />
<br />
$UC1WORK uc1 work directory<br />
<br />
$PROJECT fh1 project directory<br />
<br />
$BWFILESTORAGE $BWFS bwFileStorage directory<br />
<br />
$LSDF LSDF for Data Intensive Science - LSDF Path<br />
<br />
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory<br />
<br />
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory<br />
<br />
EXAMPLES<br />
rdata<br />
<br />
rdata 'ls $BWFILESTORAGE/*.c'<br />
<br />
rdata 'cp $WORK/foo $BWFILESTORAGE'<br />
<br />
RETURNCODE<br />
rdata returns the exit status of the last command or 127 when the last command was not found.<br />
<br />
remote-fs-utils April 2013 <br />
</pre><br />
--><br />
<br />
<H2>Mount on the login nodes</H2> <br />
<br />
'''Limits:'''<br><br />
Cpu time limit: 3600s <br><br />
Virtual memory limit: 8GB<br />
<br />
<H2>Batch Jobs</H2> <br />
<br />
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''".<br />
* [https://wiki.scc.kit.edu/hpc/index.php/ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage Slurm HPC Workload Manager]<br />
* [https://wiki.bwhpc.de/e/BwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage bwUniCluster]<br />
* [https://wiki.scc.kit.edu/hpc/index.php?title=ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage ForHLR]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|HPC]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_HPC&diff=5705LSDF Online Storage: HPC2020-12-15T08:52:17Z<p>Obholz: </p>
<hr />
<div><H1>LSDF_Online_Storage for HPC Cluster</H1><br />
<br />
The LSDF storage is permanently mounted on the login nodes and the data movers.<br />
<br />
Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.<br />
<br />
<!--<br />
<H2>rdata</H2><br />
<br />
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T<br />
<br />
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load. <br />
<br />
'''Examples:'''<br />
<pre><br />
>rdata "ls $LSDFPROJECTS"<br />
>rdata "ls $LSDFHOME/*.c"<br />
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"<br />
or<br />
>rdata <br />
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“<br />
</pre><br />
The command <br />
<pre><br />
>man rdata<br />
</pre> <br />
shows how to use the command rdata. <br />
<pre><br />
rdata(1) Cluster Commands rdata(1)<br />
<br />
NAME<br />
rdata - File operations are executed on a data mover node of the cluster<br />
<br />
SYNOPSIS<br />
rdata [ COMMAND ]<br />
<br />
DESCRIPTION<br />
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.<br />
<br />
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.<br />
<br />
COMMANDS<br />
The following commands are allowed in COMMAND of rdata:<br />
<br />
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.<br />
<br />
help Output of a short help text.<br />
<br />
cd Changing the directory.<br />
<br />
quit exit Termination of rdata.<br />
<br />
ENVIRONMENT<br />
The following user specific environment variables are available with rdata:<br />
<br />
$HOME home directory<br />
<br />
$WORK work directory<br />
<br />
$UC1WORK uc1 work directory<br />
<br />
$PROJECT fh1 project directory<br />
<br />
$BWFILESTORAGE $BWFS bwFileStorage directory<br />
<br />
$LSDF LSDF for Data Intensive Science - LSDF Path<br />
<br />
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory<br />
<br />
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory<br />
<br />
EXAMPLES<br />
rdata<br />
<br />
rdata 'ls $BWFILESTORAGE/*.c'<br />
<br />
rdata 'cp $WORK/foo $BWFILESTORAGE'<br />
<br />
RETURNCODE<br />
rdata returns the exit status of the last command or 127 when the last command was not found.<br />
<br />
remote-fs-utils April 2013 <br />
</pre><br />
--><br />
<br />
<H2>Mount on the login nodes</H2> <br />
<br />
'''Limits:'''<br><br />
Cpu time limit: 3600s <br><br />
Virtual memory limit: 8GB<br />
<br />
<H2>Batch Jobs</H2> <br />
<br />
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''".<br />
* [https://wiki.scc.kit.edu/hpc/index.php/ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage Slurm HPC Workload Manager]<br />
<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|HPC]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SSHFS_mount&diff=5674LSDF Online Storage: SSHFS mount2020-07-03T07:20:53Z<p>Obholz: Redirected page to LSDF Online Storage: Secure Shell#Using SSHFS on Linux or Mac OS</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_Secure_Shell#Using_SSHFS_on_Linux_or_Mac_OS]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|SSHFS mount]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_WebDAV_mount&diff=5646LSDF Online Storage: WebDAV mount2020-03-17T15:40:27Z<p>Obholz: Redirected page to LSDF Online Storage: WebDAV#Mounting a WebDAV Share on Linux with davfs2</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_WebDAV#Mounting_a_WebDAV_Share_on_Linux_with_davfs2]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|WebDAV mount]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SCP&diff=5645LSDF Online Storage: SCP2020-03-17T15:37:15Z<p>Obholz: Redirected page to LSDF Online Storage: Secure Shell#Using SCP on Linux or Mac OS</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_Secure_Shell#Using_SCP_on_Linux_or_Mac_OS]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|SCP]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_DaviX&diff=5644LSDF Online Storage: DaviX2020-03-17T15:35:38Z<p>Obholz: Redirected page to LSDF Online Storage: WebDAV#Using the DaviX Command Line Client on Linux and Mac OS</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_WebDAV#Using_the_DaviX_Command_Line_Client_on_Linux_and_Mac_OS]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|DaviX]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SCP&diff=5643LSDF Online Storage: SCP2020-03-17T15:34:11Z<p>Obholz: Redirected page to LSDF Online Storage: WebDAV#Using the DaviX Command Line Client on Linux and Mac OS</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_WebDAV#Using_the_DaviX_Command_Line_Client_on_Linux_and_Mac_OS]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|DaviX]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SCP&diff=5642LSDF Online Storage: SCP2020-03-17T15:32:10Z<p>Obholz: Redirected page to LSDF Online Storage: Secure Shell#Using SCP on Linux or Mac OS</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_Secure_Shell#Using_SCP_on_Linux_or_Mac_OS]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|SCP]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SCP&diff=5641LSDF Online Storage: SCP2020-03-17T15:30:18Z<p>Obholz: </p>
<hr />
<div>#REDIRECT [http://wiki.scc.kit.edu/lsdf/index.php/LSDF_Online_Storage:_WebDAV#Using_the_DaviX_Command_Line_Client_on_Linux_and_Mac_OS]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|DaviX]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_CIFS_MOUNT&diff=5640LSDF Online Storage: CIFS MOUNT2020-03-17T15:03:11Z<p>Obholz: Blanked the page</p>
<hr />
<div></div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_cifs_mount&diff=5639LSDF Online Storage: cifs mount2020-03-17T15:02:38Z<p>Obholz: Redirected page to LSDF Online Storage: SMB / CIFS#Mounting a Remote File System on Linux .28as root.29</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_SMB_/_CIFS#Mounting_a_Remote_File_System_on_Linux_.28as_root.29]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|cifs mount]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_CIFS_MOUNT&diff=5638LSDF Online Storage: CIFS MOUNT2020-03-17T15:01:12Z<p>Obholz: Redirected page to LSDF Online Storage: SMB / CIFS#Mounting a Remote File System on Linux .28as root.29</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_SMB_/_CIFS#Mounting_a_Remote_File_System_on_Linux_.28as_root.29]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|cifs mount]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_sambaclient&diff=5637LSDF Online Storage: sambaclient2020-03-17T14:58:04Z<p>Obholz: Redirected page to LSDF Online Storage: SMB / CIFS#Using SMB.2FCIFS on Linux with the Samba Command Line Client</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_SMB_/_CIFS#Using_SMB.2FCIFS_on_Linux_with_the_Samba_Command_Line_Client]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|sambaclient]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_sambaclient&diff=5636LSDF Online Storage: sambaclient2020-03-17T14:57:25Z<p>Obholz: Created page with "http://wiki.scc.kit.edu/lsdf/index.php/ #REDIRECT LSDF_Online_Storage:_SMB_/_CIFS#Using_SMB.2FCIFS_on_Linux_with_the_Samba_Command_Line_Client <hr> <br> <br> <br> <br> C..."</p>
<hr />
<div>http://wiki.scc.kit.edu/lsdf/index.php/<br />
#REDIRECT [[LSDF_Online_Storage:_SMB_/_CIFS#Using_SMB.2FCIFS_on_Linux_with_the_Samba_Command_Line_Client]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|sambaclient]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SCP&diff=5635LSDF Online Storage: SCP2020-03-17T14:56:02Z<p>Obholz: Redirected page to LSDF Online Storage: Secure Shell#Using SCP on Linux or Mac OS</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_Secure_Shell#Using_SCP_on_Linux_or_Mac_OS]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|SCP]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SSH&diff=5634LSDF Online Storage: SSH2020-03-17T14:52:26Z<p>Obholz: Redirected page to LSDF Online Storage: Secure Shell</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_Secure_Shell]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|SSH]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SCP&diff=5633LSDF Online Storage: SCP2020-03-17T14:50:19Z<p>Obholz: Redirected page to LSDF Online Storage: Secure Shell#Using SSHFS on Linux or Mac OS</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_Secure_Shell#Using_SSHFS_on_Linux_or_Mac_OS]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|SSHFS]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SSH&diff=5632LSDF Online Storage: SSH2020-03-17T14:48:59Z<p>Obholz: Created page with "#REDIRECT [LSDF_Online_Storage:_Secure_Shell]] <hr> <br> <br> <br> <br> SSH"</p>
<hr />
<div>#REDIRECT [LSDF_Online_Storage:_Secure_Shell]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|SSH]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SFTP&diff=5631LSDF Online Storage: SFTP2020-03-17T14:25:28Z<p>Obholz: Redirected page to LSDF Online Storage: Secure Shell#Using SFTP on Linux or Mac OS</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_Secure_Shell#Using_SFTP_on_Linux_or_Mac_OS]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|SFTP]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_SCP&diff=5630LSDF Online Storage: SCP2020-03-17T14:22:29Z<p>Obholz: Redirected page to LSDF Online Storage: Secure Shell#Using SCP on Linux or Mac OS</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_Secure_Shell#Using_SCP_on_Linux_or_Mac_OS]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|SCP]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage_lftp&diff=5629LSDF Online Storage lftp2020-03-17T14:08:38Z<p>Obholz: Blanked the page</p>
<hr />
<div></div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_lftp&diff=5628LSDF Online Storage: lftp2020-03-17T14:07:13Z<p>Obholz: Created page with "<H1>Using lftp from UNIX client</H1> '''lftp''' is a command-line file transfer program (FTP client) for UNIX and Unix-like systems. Besides FTP, it also supports HTTP, HTT..."</p>
<hr />
<div><H1>Using lftp from UNIX client</H1><br />
<br />
'''lftp''' is a command-line file transfer program (FTP client) for UNIX and Unix-like systems. <br />
<br />
Besides FTP, it also supports HTTP, HTTPS and '''SFTP''' etc.<br />
<br />
In addition to features common in advanced FTP clients, such as recursively mirroring entire directory trees and resuming downloads, lftp also supports more advanced functionality. Transfers can be scheduled for execution at a later time, bandwidth can be throttled, transfer queues can be created, and Unix shell-like job control is supported. The client can be used interactively or automated with scripts.<br />
<br />
It also has an option called segmented file transfer that allows more than one connection for the same file, bypassing a maximum download speed per file when some servers establish a maximum speed per connection<br />
<br />
<h2>Installation</h2><br />
<br />
'''Debian:''' <br />
<pre><br />
apt-get install lftp<br />
</pre><br />
'''Suse-Linux:'''<br />
<pre><br />
zypper install lftp<br />
</pre><br />
'''RedHat:'''<br />
<pre><br />
yum install lftp<br />
</pre><br />
<br />
<h2>Documentation</h2><br />
<br />
*lftp home: https://lftp.yar.ru/<br />
*lftp man page: https://lftp.yar.ru/lftp-man.html <br />
*lftp documentation wiki: http://linux.overshoot.tv/wiki/networking/lftp<br />
<br />
<h2>lftp interactive shell</h2><br />
You can launch lftp by typing just lftp and then using an '''open''' command to take you to your target site or you can provide the target's name on the same line<br />
<pre><br />
lftp<br />
lftp :~> open -u xy1234 sftp://os-login.lsdf.kit.edu<br />
Password: <br />
lftp xy1234@os-login.lsdf.kit.edu:~> help<br />
</pre><br />
or<br />
<pre><br />
lftp -u xy1234 sftp://os-login.lsdf.kit.edu<br />
Password: <br />
lftp 1234@os-login.lsdf.kit.edu:~>help<br />
</pre><br />
<br />
<h2>Using lftp to mirror/transfer files</h2><br />
<br />
<pre><br />
#!/bin/bash<br />
# <<EOF below is the functional equivalent of hitting .Enter. on your keyboard.<br />
# It allows the rest of the commands to be executed once connected<br />
#<br />
# mirror [OPTS] [source [target]]<br />
# mirror OPTS:<br />
# -c, --continue continue a mirror job if possible<br />
# -e, --delete delete files not present at remote site<br />
# -R, --reverse reverse mirror (put files)<br />
# -P, --parallel[=N] download N files in parallel<br />
# -v, --verbose[=level] verbose operation<br />
# -I GP, --include-glob GP include matching files<br />
# -r, --no-recursion donât go to subdirectories<br />
# ....<br />
lftp -e 'mirror -R /home/localuser/LocalDirToMirror ~/TargetDir' -u YourUsername,YourPassword sftp://os-login.lsdf.kit.edu <<EOF<br />
quit 0<br />
EOF<br />
</pre><br />
<br />
'''Example 1:'''<br />
<pre><br />
#!/bin/bash<br />
lftp -e 'mirror -P 2 -Re ./testdir ~/backupdir' -u 'xy1234:password' sftp://os-login.lsdf.kit.edu <<EOF<br />
quit 0<br />
EOF<br />
</pre><br />
<br />
'''Example 2:'''<br />
<pre><br />
#!/bin/bash<br />
lftp -e "mirror -ceR -I '*.tar' ./testdir ~/backupdir;quit" -u 'xy1234:password' sftp://os-login.lsdf.kit.edu<br />
</pre><br />
<br />
<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|lftp]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_Secure_Shell&diff=5627LSDF Online Storage: Secure Shell2020-03-17T13:42:48Z<p>Obholz: </p>
<hr />
<div>Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server. <br />
<br />
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.<br />
<br />
The login cluster are directly accessible by end users. <br />
These nodes are used for interactive login, file management, data retrieving and restoring. <br><br />
Three nodes are dedicated to this service but they are all accessible via one host name and a DNS round-robin alias distributes the login sessions to the different login nodes.<br />
<br />
'''Environment variables:'''<br />
<pre><br />
$SHELL=/bin/bash<br />
$USER=xy1234<br />
$LSDF=/lsdf<br />
$HOME=/lsdf/kit/inst/xy1234<br />
$LSDFPROJECTS=/lsdf/kit/inst/projects<br />
$SNAPSHOTS=/lsdf/kit/snapshots<br />
$PROJECTSNAPSHOTS=/lsdf/kit/inst/projects/.snapshots<br />
</pre><br />
<br />
<hr><br />
__FORCETOC__<br />
<br />
== Using SSH on Linux or Mac OS ==<br />
<br />
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal<br />
<pre><br />
$ ssh <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
where ''<USERNAME>'' needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server. <br />
<br />
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:<br />
<pre><br />
$ ssh-keygen<br />
</pre><br />
The newly generated key should be password protected.<br />
<br />
To copy the public part of your key pair to the login server issue the following command:<br />
<pre><br />
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
Subsequent logins will use the key instead of a password to authenticate to the login server.<br />
<br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/ssh.1 man page].<br />
<br />
== Using SCP on Linux or Mac OS ==<br />
<br />
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF<br />
Online Storage, you can execute the following commands:<br />
<pre><br />
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/<br />
or<br />
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)><br />
</pre><br />
Please be aware that symbolic links will usually not be copied as links.<br />
<br />
'''Examples:''' <br><br />
<pre><br />
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/<br />
or<br />
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/<br />
</pre><br />
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.<br />
<br />
Further details on the usage of the SCP command line client can be found on the [https://man.openbsd.org/scp.1 man page].<br />
<br />
== Using SFTP on Linux or Mac OS ==<br />
<br />
The following example illustrates the usage of SFTP in a linux shell:<br />
<pre><br />
$ sftp xy1234@os-login.lsdf.kit.edu<br />
Connecting to os-login.lsdf.kit.edu<br><br />
xy1234@os-login.lsdf.kit.edu's password: <br />
<br />
sftp> ls<br />
snapshots<br />
temp test<br />
<br />
sftp> help<br />
...<br />
<br />
sftp> put myfile<br />
<br />
sftp> get myfile<br />
</pre><br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].<br />
<br />
<br />
== Using rsync on Linux or Mac OS ==<br />
<br />
'''rsync''' is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
'''Attention: rsync''' with '''-a''' (or '''--group''') option write the data with wrong GID (from source).<br />
<br />
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:<br />
<pre><br />
#In the LSDF: set sticky bit and group, if it is missing or wrong:<br />
$ chmod -g+s /lsdf_path/<br />
$ chgrp lsdf-project-group /lsdf_path/<br />
<br />
#On the client host: rsync via ssh (workaround with '-a --no-g --chmod=Dg+s')<br />
$ rsync -uva --no-g --chmod=Dg+s --rsh="ssh -c es128-gcm@openssh.com -p 22" /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<hr><br />
To '''reach maximum bandwidth''' we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission. <br><br><br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
$ echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work and delete data you can use the option '''--delete'''. The option '''--delete''' deletes files in the target directory if they were deleted in the source directory.<br />
<br />
== Using SSHFS on Linux or Mac OS ==<br />
<br />
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.<br />
<br />
=== Installation ===<br />
<br />
SSHFS is available on many linux distribution via the standard software distribution channels.<br />
<br />
On Debian/Ubuntu like systems:<br />
<pre><br />
$ apt-get install sshfs<br />
</pre><br />
<br />
On RedHat/CentOS like systems:<br />
<pre><br />
$ yum install fuse-sshfs<br />
</pre><br />
<br />
=== Mounting a Remote File System Interactively (as normal user) ===<br />
<br />
To mount a remote file system via command line:<br />
<pre><br />
$ mkdir mountpoint<br />
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ ./mointpoint<br />
</pre><br />
<br />
=== Mounting a Remote File System with ''/etc/fstab'' (as root) ===<br />
<br />
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:<br />
<pre><br />
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects/ <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the <br />
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory). <br />
<br />
'''For example:'''<br />
<pre><br />
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
<br />
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):<br />
<pre><br />
$ mount /mnt/mountpoint<br />
</pre><br />
<br />
== Using Secure Shell with Windows ==<br />
<br />
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):<br />
<br />
{| class="wikitable"<br />
| '''MobaXterm'''<br />
| https://mobaxterm.mobatek.net/<br />
|-<br />
| '''PuTTY'''<br />
| https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html<br />
|-<br />
| '''WinSCP'''<br />
| http://winscp.net/eng/download.php<br />
|-<br />
| '''FileZilla'''<br />
| http://sourceforge.net/projects/filezilla/<br />
|-<br />
| '''WebDrive'''<br />
| https://southrivertech.com/products/webdrive/download/<br />
|-<br />
| '''Cygwin'''<br />
| http://cygwin.com/install.html<br />
|-<br />
| '''Windows 10 subsystem for Linux'''<br />
| https://msdn.microsoft.com/de-de/commandline/wsl/install_guide<br />
|}<br />
<br />
The following software packages provide network drive functionality based on SFTP:<br />
<br />
{| class="wikitable"<br />
| '''WebDrive''' (for Windows, Mac, iOS, Android)<br />
| http://www.southrivertechnologies.com/download/downloadwd.html<br />
|-<br />
| '''SFTPNetDrive''' (for Windows)<br />
| http://www.sftpnetdrive.com/<br />
|-<br />
| '''NetDrive''' (for Windows)<br />
| http://www.netdrive.net/<br />
|-<br />
| '''ExpanDrive''' (for Windows and Mac)<br />
| http://www.expandrive.com/expandrive<br />
|-<br />
| '''MountainDuck''' (for Windows and Mac)<br />
| https://mountainduck.io/<br />
|}<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]<br />
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]<br />
<br />
[[Category:LSDF_Online_Storage|Secure Shell]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_Secure_Shell&diff=5626LSDF Online Storage: Secure Shell2020-03-17T13:36:39Z<p>Obholz: </p>
<hr />
<div>Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server. <br />
<br />
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.<br />
<br />
The login cluster are directly accessible by end users. <br />
These nodes are used for interactive login, file management, data retrieving and restoring. <br><br />
Three nodes are dedicated to this service but they are all accessible via one host name and a DNS round-robin alias distributes the login sessions to the different login nodes.<br />
<br />
'''Environment Variables:'''<br />
<pre><br />
$SHELL=/bin/bash<br />
$USER=xy1234<br />
$LSDF=/lsdf<br />
$HOME=/lsdf/kit/inst/xy1234<br />
$LSDFPROJECTS=/lsdf/kit/inst/projects<br />
$SNAPSHOTS=/lsdf/kit/snapshots<br />
$PROJECTSNAPSHOTS=/lsdf/kit/inst/projects/.snapshots<br />
</pre><br />
<br />
<hr><br />
__FORCETOC__<br />
<br />
== Using SSH on Linux or Mac OS ==<br />
<br />
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal<br />
<pre><br />
$ ssh <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
where ''<USERNAME>'' needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server. <br />
<br />
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:<br />
<pre><br />
$ ssh-keygen<br />
</pre><br />
The newly generated key should be password protected.<br />
<br />
To copy the public part of your key pair to the login server issue the following command:<br />
<pre><br />
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
Subsequent logins will use the key instead of a password to authenticate to the login server.<br />
<br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/ssh.1 man page].<br />
<br />
== Using SCP on Linux or Mac OS ==<br />
<br />
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF<br />
Online Storage, you can execute the following commands:<br />
<pre><br />
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/<br />
or<br />
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)><br />
</pre><br />
Please be aware that symbolic links will usually not be copied as links.<br />
<br />
'''Examples:''' <br><br />
<pre><br />
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/<br />
or<br />
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/<br />
</pre><br />
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.<br />
<br />
Further details on the usage of the SCP command line client can be found on the [https://man.openbsd.org/scp.1 man page].<br />
<br />
== Using SFTP on Linux or Mac OS ==<br />
<br />
The following example illustrates the usage of SFTP in a linux shell:<br />
<pre><br />
$ sftp xy1234@os-login.lsdf.kit.edu<br />
Connecting to os-login.lsdf.kit.edu<br><br />
xy1234@os-login.lsdf.kit.edu's password: <br />
<br />
sftp> ls<br />
snapshots<br />
temp test<br />
<br />
sftp> help<br />
...<br />
<br />
sftp> put myfile<br />
<br />
sftp> get myfile<br />
</pre><br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].<br />
<br />
<br />
== Using rsync on Linux or Mac OS ==<br />
<br />
'''rsync''' is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
'''Attention: rsync''' with '''-a''' (or '''--group''') option write the data with wrong GID (from source).<br />
<br />
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:<br />
<pre><br />
#In the LSDF: set sticky bit and group, if it is missing or wrong:<br />
$ chmod -g+s /lsdf_path/<br />
$ chgrp lsdf-project-group /lsdf_path/<br />
<br />
#On the client host: rsync via ssh (workaround with '-a --no-g --chmod=Dg+s')<br />
$ rsync -uva --no-g --chmod=Dg+s --rsh="ssh -c es128-gcm@openssh.com -p 22" /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<hr><br />
To '''reach maximum bandwidth''' we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission. <br><br><br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
$ echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work and delete data you can use the option '''--delete'''. The option '''--delete''' deletes files in the target directory if they were deleted in the source directory.<br />
<br />
== Using SSHFS on Linux or Mac OS ==<br />
<br />
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.<br />
<br />
=== Installation ===<br />
<br />
SSHFS is available on many linux distribution via the standard software distribution channels.<br />
<br />
On Debian/Ubuntu like systems:<br />
<pre><br />
$ apt-get install sshfs<br />
</pre><br />
<br />
On RedHat/CentOS like systems:<br />
<pre><br />
$ yum install fuse-sshfs<br />
</pre><br />
<br />
=== Mounting a Remote File System Interactively (as normal user) ===<br />
<br />
To mount a remote file system via command line:<br />
<pre><br />
$ mkdir mountpoint<br />
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ ./mointpoint<br />
</pre><br />
<br />
=== Mounting a Remote File System with ''/etc/fstab'' (as root) ===<br />
<br />
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:<br />
<pre><br />
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects/ <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the <br />
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory). <br />
<br />
'''For example:'''<br />
<pre><br />
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
<br />
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):<br />
<pre><br />
$ mount /mnt/mountpoint<br />
</pre><br />
<br />
== Using Secure Shell with Windows ==<br />
<br />
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):<br />
<br />
{| class="wikitable"<br />
| '''MobaXterm'''<br />
| https://mobaxterm.mobatek.net/<br />
|-<br />
| '''PuTTY'''<br />
| https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html<br />
|-<br />
| '''WinSCP'''<br />
| http://winscp.net/eng/download.php<br />
|-<br />
| '''FileZilla'''<br />
| http://sourceforge.net/projects/filezilla/<br />
|-<br />
| '''WebDrive'''<br />
| https://southrivertech.com/products/webdrive/download/<br />
|-<br />
| '''Cygwin'''<br />
| http://cygwin.com/install.html<br />
|-<br />
| '''Windows 10 subsystem for Linux'''<br />
| https://msdn.microsoft.com/de-de/commandline/wsl/install_guide<br />
|}<br />
<br />
The following software packages provide network drive functionality based on SFTP:<br />
<br />
{| class="wikitable"<br />
| '''WebDrive''' (for Windows, Mac, iOS, Android)<br />
| http://www.southrivertechnologies.com/download/downloadwd.html<br />
|-<br />
| '''SFTPNetDrive''' (for Windows)<br />
| http://www.sftpnetdrive.com/<br />
|-<br />
| '''NetDrive''' (for Windows)<br />
| http://www.netdrive.net/<br />
|-<br />
| '''ExpanDrive''' (for Windows and Mac)<br />
| http://www.expandrive.com/expandrive<br />
|-<br />
| '''MountainDuck''' (for Windows and Mac)<br />
| https://mountainduck.io/<br />
|}<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]<br />
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]<br />
<br />
[[Category:LSDF_Online_Storage|Secure Shell]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage_Quota&diff=5625LSDF Online Storage Quota2020-03-17T13:29:38Z<p>Obholz: </p>
<hr />
<div><H1> Quota settings</H1><br />
<br />
<H2> LSDF User Storage</H2><br />
<br />
Dafaul User qouta (per user) : soft limit = 100GiB, hard limit = 400GiB <br />
<br />
A user can query his user quota on the login cluster:<br />
<pre><br />
>ssh loginname@os-login.lsdf.kit.edu<br />
>userquotas<br />
</pre><br />
<hr><br />
<br />
<H2>LSDF Project Storage</H2><br />
<br />
'''Fileset quota''' (per KIT institute according to storage agreement)<br />
<br><br />
A user can query his fileset quotas on the login cluster:<br />
<pre><br />
>ssh loginname@os-login.lsdf.kit.edu<br />
>projectquotas<br />
</pre><br />
<br><br />
'''Group quotas''' (per project)<br />
<br><br />
A project admins can query his group quota on<br />
https://www.lsdf.kit.edu/os/storgaeprojects<br />
<br><br><br />
'''User quotas''' (if required) <br />
<H2>LSDF Storage Request</H2><br />
<br />
A higher quota for a institute or user can be requested via support page: <br><br />
'''https://bw-support.scc.kit.edu/''' <br><br />
or <br><br />
via storage request page: <br><br />
'''https://www.lsdf.kit.edu/os/storageprojects/'''<br />
<hr><br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|Quota]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage_Quota&diff=5624LSDF Online Storage Quota2020-03-17T13:28:36Z<p>Obholz: </p>
<hr />
<div><H1> Quota settings</H1><br />
<br />
<H2> LSDF User Storage</H2><br />
<br />
Dafaul User qouta (per user) : soft limit = 100GiB, hard limit = 400GiB <br />
<br />
A user can query his user quota on the login cluster:<br />
<pre><br />
>ssh loginname@os-login.lsdf.kit.edu<br />
>userquotas<br />
</pre><br />
<hr><br />
<br />
<H2>LSDF Project Storage</H2><br />
<br />
'''Fileset quota''' (per KIT institute according to storage agreement)<br />
<br><br />
A user can query his fileset quotas on the login cluster:<br />
<pre><br />
>ssh loginname@os-login.lsdf.kit.edu<br />
>projectquotas<br />
</pre><br />
<br><br />
'''Group quotas''' per project<br />
A project admins can query his group quota on<br />
https://www.lsdf.kit.edu/os/storgaeprojects<br />
<br><br />
'''user quotas''' (if required) <br />
<H2>LSDF Storage Request</H2><br />
<br />
A higher quota for a institute or user can be requested via support page: <br><br />
'''https://bw-support.scc.kit.edu/''' <br><br />
or <br><br />
via storage request page: <br><br />
'''https://www.lsdf.kit.edu/os/storageprojects/'''<br />
<hr><br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|Quota]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_HPC&diff=5623LSDF Online Storage: HPC2020-03-17T12:37:45Z<p>Obholz: </p>
<hr />
<div><H1>LSDF_Online_Storage for HPC Cluster</H1><br />
<br />
The LSDF storage is permanently mounted on the login nodes and the data movers.<br />
<br />
Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.<br />
<br />
<H2>rdata</H2><br />
<br />
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T<br />
<br />
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load. <br />
<br />
'''Examples:'''<br />
<pre><br />
>rdata "ls $LSDFPROJECTS"<br />
>rdata "ls $LSDFHOME/*.c"<br />
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"<br />
or<br />
>rdata <br />
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“<br />
</pre><br />
The command <br />
<pre><br />
>man rdata<br />
</pre> <br />
shows how to use the command rdata. <br />
<pre><br />
rdata(1) Cluster Commands rdata(1)<br />
<br />
NAME<br />
rdata - File operations are executed on a data mover node of the cluster<br />
<br />
SYNOPSIS<br />
rdata [ COMMAND ]<br />
<br />
DESCRIPTION<br />
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.<br />
<br />
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.<br />
<br />
COMMANDS<br />
The following commands are allowed in COMMAND of rdata:<br />
<br />
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.<br />
<br />
help Output of a short help text.<br />
<br />
cd Changing the directory.<br />
<br />
quit exit Termination of rdata.<br />
<br />
ENVIRONMENT<br />
The following user specific environment variables are available with rdata:<br />
<br />
$HOME home directory<br />
<br />
$WORK work directory<br />
<br />
$UC1WORK uc1 work directory<br />
<br />
$PROJECT fh1 project directory<br />
<br />
$BWFILESTORAGE $BWFS bwFileStorage directory<br />
<br />
$LSDF LSDF for Data Intensive Science - LSDF Path<br />
<br />
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory<br />
<br />
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory<br />
<br />
EXAMPLES<br />
rdata<br />
<br />
rdata 'ls $BWFILESTORAGE/*.c'<br />
<br />
rdata 'cp $WORK/foo $BWFILESTORAGE'<br />
<br />
RETURNCODE<br />
rdata returns the exit status of the last command or 127 when the last command was not found.<br />
<br />
remote-fs-utils April 2013 <br />
</pre><br />
<br />
<H2>Mount on the login nodes</H2> <br />
<br />
'''Limits:'''<br><br />
Cpu time limit: 3600s <br><br />
Virtual memory limit: 8GB<br />
<br />
<H2>Batch Jobs</H2> <br />
<br />
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''".<br />
* [https://wiki.scc.kit.edu/hpc/index.php/ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage Slurm HPC Workload Manager]<br />
<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|HPC]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_HPC&diff=5622LSDF Online Storage: HPC2020-03-17T12:36:55Z<p>Obholz: </p>
<hr />
<div><H1>LSDF_Online_Storage for HPC Cluster</H1><br />
<br />
The LSDF storage is permanently mounted on the login nodes and the data movers.<br />
<br />
Both for interactive and for batch use, the environment variables '''$LSDF''', '''$LSDFHOME''' and '''$LSDFPROJECTS''' are set.<br />
<br />
<H2>rdata</H2><br />
<br />
Users of the filesystem LSDF Online Storage can furthermore transfer data to '''bwUniCluster''' and '''ForHLR''' via the tool '''rdata'''. T<br />
<br />
The command rdata executes the filesystem operations on special "data mover" nodes and distributes the load. <br />
<br />
'''Examples:'''<br />
<pre><br />
>rdata "ls $LSDFPROJECTS"<br />
>rdata "ls $LSDFHOME/*.c"<br />
>rdata "cp foo $LSDFPROJECTS/ProjectPath/"<br />
or<br />
>rdata <br />
> cp -a $WORK/path $LSDFPROJECTS/ProjectPath/“<br />
</pre><br />
The command <br />
<pre><br />
>man rdata<br />
</pre> <br />
shows how to use the command rdata. <br />
<pre><br />
rdata(1) Cluster Commands rdata(1)<br />
<br />
NAME<br />
rdata - File operations are executed on a data mover node of the cluster<br />
<br />
SYNOPSIS<br />
rdata [ COMMAND ]<br />
<br />
DESCRIPTION<br />
The program executes the specified command (with arguments) on a data mover node. As remote host, a host with the lowest load is selected.<br />
<br />
Interactive mode: If rdata is invoked without arguments, rdata reads line by line from standard input (like a restricted shell); at EOF rdata will be ended.<br />
<br />
COMMANDS<br />
The following commands are allowed in COMMAND of rdata:<br />
<br />
cp rm mv ls mkdir rmdir pwd echo rsync Execution of the corresponding system command.<br />
<br />
help Output of a short help text.<br />
<br />
cd Changing the directory.<br />
<br />
quit exit Termination of rdata.<br />
<br />
ENVIRONMENT<br />
The following user specific environment variables are available with rdata:<br />
<br />
$HOME home directory<br />
<br />
$WORK work directory<br />
<br />
$UC1WORK uc1 work directory<br />
<br />
$PROJECT fh1 project directory<br />
<br />
$BWFILESTORAGE $BWFS bwFileStorage directory<br />
<br />
$LSDF LSDF for Data Intensive Science - LSDF Path<br />
<br />
$LSDFPROJECTS LSDF for Data Intensive Science - Project Directory<br />
<br />
$LSDFHOME LSDF for Data Intensive Science - LSDF Home Directory<br />
<br />
EXAMPLES<br />
rdata<br />
<br />
rdata 'ls $BWFILESTORAGE/*.c'<br />
<br />
rdata 'cp $WORK/foo $BWFILESTORAGE'<br />
<br />
RETURNCODE<br />
rdata returns the exit status of the last command or 127 when the last command was not found.<br />
<br />
remote-fs-utils April 2013 <br />
</pre><br />
<br />
<H2>Mount on the login nodes(only on [http://www.scc.kit.edu/dienste/forhlr.php ForHLR]) </H2> <br />
<br />
'''Limits:'''<br><br />
Cpu time limit: 3600s <br><br />
Virtual memory limit: 8GB<br />
<br />
<H2>Batch Jobs</H2> <br />
<br />
On the compute nodes, the LSDF usage is indicated to the batch system via "'''#SBATCH --constraint=LSDF'''".<br />
* [https://wiki.scc.kit.edu/hpc/index.php/ForHLR_-_SLURM_Batch_Jobs#LSDF_Online_Storage Slurm HPC Workload Manager]<br />
<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|HPC]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_Secure_Shell&diff=5621LSDF Online Storage: Secure Shell2020-03-17T12:11:48Z<p>Obholz: /* Using rsync on Linux or Mac OS */</p>
<hr />
<div>Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server. <br />
<br />
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.<br />
<br />
__FORCETOC__<br />
<br />
== Using SSH on Linux or Mac OS ==<br />
<br />
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal<br />
<pre><br />
$ ssh <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
where ''<USERNAME>'' needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server. <br />
<br />
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:<br />
<pre><br />
$ ssh-keygen<br />
</pre><br />
The newly generated key should be password protected.<br />
<br />
To copy the public part of your key pair to the login server issue the following command:<br />
<pre><br />
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
Subsequent logins will use the key instead of a password to authenticate to the login server.<br />
<br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/ssh.1 man page].<br />
<br />
== Using SCP on Linux or Mac OS ==<br />
<br />
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF<br />
Online Storage, you can execute the following commands:<br />
<pre><br />
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/<br />
or<br />
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)><br />
</pre><br />
Please be aware that symbolic links will usually not be copied as links.<br />
<br />
'''Examples:''' <br><br />
<pre><br />
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/<br />
or<br />
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/<br />
</pre><br />
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.<br />
<br />
Further details on the usage of the SCP command line client can be found on the [https://man.openbsd.org/scp.1 man page].<br />
<br />
== Using SFTP on Linux or Mac OS ==<br />
<br />
The following example illustrates the usage of SFTP in a linux shell:<br />
<pre><br />
$ sftp xy1234@os-login.lsdf.kit.edu<br />
Connecting to os-login.lsdf.kit.edu<br><br />
xy1234@os-login.lsdf.kit.edu's password: <br />
<br />
sftp> ls<br />
snapshots<br />
temp test<br />
<br />
sftp> help<br />
...<br />
<br />
sftp> put myfile<br />
<br />
sftp> get myfile<br />
</pre><br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].<br />
<br />
<br />
== Using rsync on Linux or Mac OS ==<br />
<br />
'''rsync''' is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
'''Attention: rsync''' with '''-a''' (or '''--group''') option write the data with wrong GID (from source).<br />
<br />
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:<br />
<pre><br />
#In the LSDF: set sticky bit and group, if it is missing or wrong:<br />
$ chmod -g+s /lsdf_path/<br />
$ chgrp lsdf-project-group /lsdf_path/<br />
<br />
#On the client host: rsync via ssh (workaround with '-a --no-g --chmod=Dg+s')<br />
$ rsync -uva --no-g --chmod=Dg+s --rsh="ssh -c es128-gcm@openssh.com -p 22" /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<hr><br />
To '''reach maximum bandwidth''' we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission. <br><br><br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
$ echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work and delete data you can use the option '''--delete'''. The option '''--delete''' deletes files in the target directory if they were deleted in the source directory.<br />
<br />
== Using SSHFS on Linux or Mac OS ==<br />
<br />
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.<br />
<br />
=== Installation ===<br />
<br />
SSHFS is available on many linux distribution via the standard software distribution channels.<br />
<br />
On Debian/Ubuntu like systems:<br />
<pre><br />
$ apt-get install sshfs<br />
</pre><br />
<br />
On RedHat/CentOS like systems:<br />
<pre><br />
$ yum install fuse-sshfs<br />
</pre><br />
<br />
=== Mounting a Remote File System Interactively (as normal user) ===<br />
<br />
To mount a remote file system via command line:<br />
<pre><br />
$ mkdir mountpoint<br />
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ ./mointpoint<br />
</pre><br />
<br />
=== Mounting a Remote File System with ''/etc/fstab'' (as root) ===<br />
<br />
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:<br />
<pre><br />
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects/ <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the <br />
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory). <br />
<br />
'''For example:'''<br />
<pre><br />
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
<br />
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):<br />
<pre><br />
$ mount /mnt/mountpoint<br />
</pre><br />
<br />
== Using Secure Shell with Windows ==<br />
<br />
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):<br />
<br />
{| class="wikitable"<br />
| '''MobaXterm'''<br />
| https://mobaxterm.mobatek.net/<br />
|-<br />
| '''PuTTY'''<br />
| https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html<br />
|-<br />
| '''WinSCP'''<br />
| http://winscp.net/eng/download.php<br />
|-<br />
| '''FileZilla'''<br />
| http://sourceforge.net/projects/filezilla/<br />
|-<br />
| '''WebDrive'''<br />
| https://southrivertech.com/products/webdrive/download/<br />
|-<br />
| '''Cygwin'''<br />
| http://cygwin.com/install.html<br />
|-<br />
| '''Windows 10 subsystem for Linux'''<br />
| https://msdn.microsoft.com/de-de/commandline/wsl/install_guide<br />
|}<br />
<br />
The following software packages provide network drive functionality based on SFTP:<br />
<br />
{| class="wikitable"<br />
| '''WebDrive''' (for Windows, Mac, iOS, Android)<br />
| http://www.southrivertechnologies.com/download/downloadwd.html<br />
|-<br />
| '''SFTPNetDrive''' (for Windows)<br />
| http://www.sftpnetdrive.com/<br />
|-<br />
| '''NetDrive''' (for Windows)<br />
| http://www.netdrive.net/<br />
|-<br />
| '''ExpanDrive''' (for Windows and Mac)<br />
| http://www.expandrive.com/expandrive<br />
|-<br />
| '''MountainDuck''' (for Windows and Mac)<br />
| https://mountainduck.io/<br />
|}<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]<br />
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]<br />
<br />
[[Category:LSDF_Online_Storage|Secure Shell]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_Secure_Shell&diff=5620LSDF Online Storage: Secure Shell2020-03-17T12:10:08Z<p>Obholz: /* Using rsync on Linux or Mac OS */</p>
<hr />
<div>Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server. <br />
<br />
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.<br />
<br />
__FORCETOC__<br />
<br />
== Using SSH on Linux or Mac OS ==<br />
<br />
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal<br />
<pre><br />
$ ssh <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
where ''<USERNAME>'' needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server. <br />
<br />
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:<br />
<pre><br />
$ ssh-keygen<br />
</pre><br />
The newly generated key should be password protected.<br />
<br />
To copy the public part of your key pair to the login server issue the following command:<br />
<pre><br />
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
Subsequent logins will use the key instead of a password to authenticate to the login server.<br />
<br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/ssh.1 man page].<br />
<br />
== Using SCP on Linux or Mac OS ==<br />
<br />
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF<br />
Online Storage, you can execute the following commands:<br />
<pre><br />
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/<br />
or<br />
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)><br />
</pre><br />
Please be aware that symbolic links will usually not be copied as links.<br />
<br />
'''Examples:''' <br><br />
<pre><br />
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/<br />
or<br />
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/<br />
</pre><br />
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.<br />
<br />
Further details on the usage of the SCP command line client can be found on the [https://man.openbsd.org/scp.1 man page].<br />
<br />
== Using SFTP on Linux or Mac OS ==<br />
<br />
The following example illustrates the usage of SFTP in a linux shell:<br />
<pre><br />
$ sftp xy1234@os-login.lsdf.kit.edu<br />
Connecting to os-login.lsdf.kit.edu<br><br />
xy1234@os-login.lsdf.kit.edu's password: <br />
<br />
sftp> ls<br />
snapshots<br />
temp test<br />
<br />
sftp> help<br />
...<br />
<br />
sftp> put myfile<br />
<br />
sftp> get myfile<br />
</pre><br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].<br />
<br />
<br />
== Using rsync on Linux or Mac OS ==<br />
<br />
'''rsync''' is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
'''Attention: rsync''' with '''-a''' (or '''--group''') option write the data with wrong GID (from source).<br />
<br />
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:<br />
<pre><br />
#In the LSDF: set sticky bit and group, if it is missing or wrong:<br />
$ chmod -g+s /lsdf_path/<br />
$ chgrp lsdf-project-group /lsdf_path/<br />
<br />
#On the client host: rsync via ssh (workaround with '-a --no-g --chmod=Dg+s')<br />
$ rsync -uva --no-g --chmod=Dg+s --rsh="ssh -c es128-gcm@openssh.com -p 22" /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<hr><br />
To '''reach maximum bandwidth''' we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission. <br><br><br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
$ echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work and delete data you can use the option ''--delete''. The option ''--delete'' deletes files in the target directory if they were deleted in the source directory.<br />
<br />
== Using SSHFS on Linux or Mac OS ==<br />
<br />
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.<br />
<br />
=== Installation ===<br />
<br />
SSHFS is available on many linux distribution via the standard software distribution channels.<br />
<br />
On Debian/Ubuntu like systems:<br />
<pre><br />
$ apt-get install sshfs<br />
</pre><br />
<br />
On RedHat/CentOS like systems:<br />
<pre><br />
$ yum install fuse-sshfs<br />
</pre><br />
<br />
=== Mounting a Remote File System Interactively (as normal user) ===<br />
<br />
To mount a remote file system via command line:<br />
<pre><br />
$ mkdir mountpoint<br />
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ ./mointpoint<br />
</pre><br />
<br />
=== Mounting a Remote File System with ''/etc/fstab'' (as root) ===<br />
<br />
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:<br />
<pre><br />
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects/ <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the <br />
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory). <br />
<br />
'''For example:'''<br />
<pre><br />
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
<br />
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):<br />
<pre><br />
$ mount /mnt/mountpoint<br />
</pre><br />
<br />
== Using Secure Shell with Windows ==<br />
<br />
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):<br />
<br />
{| class="wikitable"<br />
| '''MobaXterm'''<br />
| https://mobaxterm.mobatek.net/<br />
|-<br />
| '''PuTTY'''<br />
| https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html<br />
|-<br />
| '''WinSCP'''<br />
| http://winscp.net/eng/download.php<br />
|-<br />
| '''FileZilla'''<br />
| http://sourceforge.net/projects/filezilla/<br />
|-<br />
| '''WebDrive'''<br />
| https://southrivertech.com/products/webdrive/download/<br />
|-<br />
| '''Cygwin'''<br />
| http://cygwin.com/install.html<br />
|-<br />
| '''Windows 10 subsystem for Linux'''<br />
| https://msdn.microsoft.com/de-de/commandline/wsl/install_guide<br />
|}<br />
<br />
The following software packages provide network drive functionality based on SFTP:<br />
<br />
{| class="wikitable"<br />
| '''WebDrive''' (for Windows, Mac, iOS, Android)<br />
| http://www.southrivertechnologies.com/download/downloadwd.html<br />
|-<br />
| '''SFTPNetDrive''' (for Windows)<br />
| http://www.sftpnetdrive.com/<br />
|-<br />
| '''NetDrive''' (for Windows)<br />
| http://www.netdrive.net/<br />
|-<br />
| '''ExpanDrive''' (for Windows and Mac)<br />
| http://www.expandrive.com/expandrive<br />
|-<br />
| '''MountainDuck''' (for Windows and Mac)<br />
| https://mountainduck.io/<br />
|}<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]<br />
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]<br />
<br />
[[Category:LSDF_Online_Storage|Secure Shell]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_Secure_Shell&diff=5619LSDF Online Storage: Secure Shell2020-03-17T11:46:34Z<p>Obholz: /* Using SCP on Linux or Mac OS */</p>
<hr />
<div>Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server. <br />
<br />
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.<br />
<br />
__FORCETOC__<br />
<br />
== Using SSH on Linux or Mac OS ==<br />
<br />
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal<br />
<pre><br />
$ ssh <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
where ''<USERNAME>'' needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server. <br />
<br />
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:<br />
<pre><br />
$ ssh-keygen<br />
</pre><br />
The newly generated key should be password protected.<br />
<br />
To copy the public part of your key pair to the login server issue the following command:<br />
<pre><br />
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
Subsequent logins will use the key instead of a password to authenticate to the login server.<br />
<br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/ssh.1 man page].<br />
<br />
== Using SCP on Linux or Mac OS ==<br />
<br />
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF<br />
Online Storage, you can execute the following commands:<br />
<pre><br />
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/<br />
or<br />
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)><br />
</pre><br />
Please be aware that symbolic links will usually not be copied as links.<br />
<br />
'''Examples:''' <br><br />
<pre><br />
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/<br />
or<br />
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/<br />
</pre><br />
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.<br />
<br />
Further details on the usage of the SCP command line client can be found on the [https://man.openbsd.org/scp.1 man page].<br />
<br />
== Using SFTP on Linux or Mac OS ==<br />
<br />
The following example illustrates the usage of SFTP in a linux shell:<br />
<pre><br />
$ sftp xy1234@os-login.lsdf.kit.edu<br />
Connecting to os-login.lsdf.kit.edu<br><br />
xy1234@os-login.lsdf.kit.edu's password: <br />
<br />
sftp> ls<br />
snapshots<br />
temp test<br />
<br />
sftp> help<br />
...<br />
<br />
sftp> put myfile<br />
<br />
sftp> get myfile<br />
</pre><br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].<br />
<br />
<br />
== Using rsync on Linux or Mac OS ==<br />
<br />
rsync is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:<br />
<pre><br />
In the LSDF:<br />
$ chmod -g+s /lsdf_path/<br />
<br />
On the client host:<br />
$ rsync -uva --no-g --chmod=Dg+s /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
$ echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work and delete data you can use the option ''--delete''. The option ''--delete'' deletes files in the target directory if they were deleted in the source directory.<br />
<br />
<br />
<br />
== Using SSHFS on Linux or Mac OS ==<br />
<br />
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.<br />
<br />
=== Installation ===<br />
<br />
SSHFS is available on many linux distribution via the standard software distribution channels.<br />
<br />
On Debian/Ubuntu like systems:<br />
<pre><br />
$ apt-get install sshfs<br />
</pre><br />
<br />
On RedHat/CentOS like systems:<br />
<pre><br />
$ yum install fuse-sshfs<br />
</pre><br />
<br />
=== Mounting a Remote File System Interactively (as normal user) ===<br />
<br />
To mount a remote file system via command line:<br />
<pre><br />
$ mkdir mountpoint<br />
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ ./mointpoint<br />
</pre><br />
<br />
=== Mounting a Remote File System with ''/etc/fstab'' (as root) ===<br />
<br />
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:<br />
<pre><br />
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects/ <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the <br />
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory). <br />
<br />
'''For example:'''<br />
<pre><br />
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
<br />
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):<br />
<pre><br />
$ mount /mnt/mountpoint<br />
</pre><br />
<br />
== Using Secure Shell with Windows ==<br />
<br />
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):<br />
<br />
{| class="wikitable"<br />
| '''MobaXterm'''<br />
| https://mobaxterm.mobatek.net/<br />
|-<br />
| '''PuTTY'''<br />
| https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html<br />
|-<br />
| '''WinSCP'''<br />
| http://winscp.net/eng/download.php<br />
|-<br />
| '''FileZilla'''<br />
| http://sourceforge.net/projects/filezilla/<br />
|-<br />
| '''WebDrive'''<br />
| https://southrivertech.com/products/webdrive/download/<br />
|-<br />
| '''Cygwin'''<br />
| http://cygwin.com/install.html<br />
|-<br />
| '''Windows 10 subsystem for Linux'''<br />
| https://msdn.microsoft.com/de-de/commandline/wsl/install_guide<br />
|}<br />
<br />
The following software packages provide network drive functionality based on SFTP:<br />
<br />
{| class="wikitable"<br />
| '''WebDrive''' (for Windows, Mac, iOS, Android)<br />
| http://www.southrivertechnologies.com/download/downloadwd.html<br />
|-<br />
| '''SFTPNetDrive''' (for Windows)<br />
| http://www.sftpnetdrive.com/<br />
|-<br />
| '''NetDrive''' (for Windows)<br />
| http://www.netdrive.net/<br />
|-<br />
| '''ExpanDrive''' (for Windows and Mac)<br />
| http://www.expandrive.com/expandrive<br />
|-<br />
| '''MountainDuck''' (for Windows and Mac)<br />
| https://mountainduck.io/<br />
|}<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]<br />
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]<br />
<br />
[[Category:LSDF_Online_Storage|Secure Shell]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_Secure_Shell&diff=5618LSDF Online Storage: Secure Shell2020-03-17T11:42:46Z<p>Obholz: /* Using SSHFS on Linux or Mac OS */</p>
<hr />
<div>Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server. <br />
<br />
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.<br />
<br />
__FORCETOC__<br />
<br />
== Using SSH on Linux or Mac OS ==<br />
<br />
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal<br />
<pre><br />
$ ssh <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
where ''<USERNAME>'' needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server. <br />
<br />
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:<br />
<pre><br />
$ ssh-keygen<br />
</pre><br />
The newly generated key should be password protected.<br />
<br />
To copy the public part of your key pair to the login server issue the following command:<br />
<pre><br />
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
Subsequent logins will use the key instead of a password to authenticate to the login server.<br />
<br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/ssh.1 man page].<br />
<br />
== Using SCP on Linux or Mac OS ==<br />
<br />
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF<br />
Online Storage, you can execute the following commands:<br />
<pre><br />
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/<br />
or<br />
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)><br />
</pre><br />
Please be aware that symbolic links will usually not be copied as links.<br />
<br />
'''Examples:''' <br><br />
<pre><br />
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/<br />
or<br />
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/<br />
</pre><br />
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.<br />
<br />
Further details on the usage of the SCP command line client can be found on the [https://man.openbsd.org/scp man page].<br />
<br />
== Using SFTP on Linux or Mac OS ==<br />
<br />
The following example illustrates the usage of SFTP in a linux shell:<br />
<pre><br />
$ sftp xy1234@os-login.lsdf.kit.edu<br />
Connecting to os-login.lsdf.kit.edu<br><br />
xy1234@os-login.lsdf.kit.edu's password: <br />
<br />
sftp> ls<br />
snapshots<br />
temp test<br />
<br />
sftp> help<br />
...<br />
<br />
sftp> put myfile<br />
<br />
sftp> get myfile<br />
</pre><br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].<br />
<br />
<br />
== Using rsync on Linux or Mac OS ==<br />
<br />
rsync is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:<br />
<pre><br />
In the LSDF:<br />
$ chmod -g+s /lsdf_path/<br />
<br />
On the client host:<br />
$ rsync -uva --no-g --chmod=Dg+s /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
$ echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work and delete data you can use the option ''--delete''. The option ''--delete'' deletes files in the target directory if they were deleted in the source directory.<br />
<br />
<br />
<br />
== Using SSHFS on Linux or Mac OS ==<br />
<br />
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.<br />
<br />
=== Installation ===<br />
<br />
SSHFS is available on many linux distribution via the standard software distribution channels.<br />
<br />
On Debian/Ubuntu like systems:<br />
<pre><br />
$ apt-get install sshfs<br />
</pre><br />
<br />
On RedHat/CentOS like systems:<br />
<pre><br />
$ yum install fuse-sshfs<br />
</pre><br />
<br />
=== Mounting a Remote File System Interactively (as normal user) ===<br />
<br />
To mount a remote file system via command line:<br />
<pre><br />
$ mkdir mountpoint<br />
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ ./mointpoint<br />
</pre><br />
<br />
=== Mounting a Remote File System with ''/etc/fstab'' (as root) ===<br />
<br />
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:<br />
<pre><br />
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects/ <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the <br />
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory). <br />
<br />
'''For example:'''<br />
<pre><br />
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
<br />
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):<br />
<pre><br />
$ mount /mnt/mountpoint<br />
</pre><br />
<br />
== Using Secure Shell with Windows ==<br />
<br />
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):<br />
<br />
{| class="wikitable"<br />
| '''MobaXterm'''<br />
| https://mobaxterm.mobatek.net/<br />
|-<br />
| '''PuTTY'''<br />
| https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html<br />
|-<br />
| '''WinSCP'''<br />
| http://winscp.net/eng/download.php<br />
|-<br />
| '''FileZilla'''<br />
| http://sourceforge.net/projects/filezilla/<br />
|-<br />
| '''WebDrive'''<br />
| https://southrivertech.com/products/webdrive/download/<br />
|-<br />
| '''Cygwin'''<br />
| http://cygwin.com/install.html<br />
|-<br />
| '''Windows 10 subsystem for Linux'''<br />
| https://msdn.microsoft.com/de-de/commandline/wsl/install_guide<br />
|}<br />
<br />
The following software packages provide network drive functionality based on SFTP:<br />
<br />
{| class="wikitable"<br />
| '''WebDrive''' (for Windows, Mac, iOS, Android)<br />
| http://www.southrivertechnologies.com/download/downloadwd.html<br />
|-<br />
| '''SFTPNetDrive''' (for Windows)<br />
| http://www.sftpnetdrive.com/<br />
|-<br />
| '''NetDrive''' (for Windows)<br />
| http://www.netdrive.net/<br />
|-<br />
| '''ExpanDrive''' (for Windows and Mac)<br />
| http://www.expandrive.com/expandrive<br />
|-<br />
| '''MountainDuck''' (for Windows and Mac)<br />
| https://mountainduck.io/<br />
|}<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]<br />
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]<br />
<br />
[[Category:LSDF_Online_Storage|Secure Shell]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_Secure_Shell&diff=5617LSDF Online Storage: Secure Shell2020-03-17T11:39:14Z<p>Obholz: /* Using SSHFS on Linux or Mac OS */</p>
<hr />
<div>Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server. <br />
<br />
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.<br />
<br />
__FORCETOC__<br />
<br />
== Using SSH on Linux or Mac OS ==<br />
<br />
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal<br />
<pre><br />
$ ssh <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
where ''<USERNAME>'' needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server. <br />
<br />
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:<br />
<pre><br />
$ ssh-keygen<br />
</pre><br />
The newly generated key should be password protected.<br />
<br />
To copy the public part of your key pair to the login server issue the following command:<br />
<pre><br />
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
Subsequent logins will use the key instead of a password to authenticate to the login server.<br />
<br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/ssh.1 man page].<br />
<br />
== Using SCP on Linux or Mac OS ==<br />
<br />
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF<br />
Online Storage, you can execute the following commands:<br />
<pre><br />
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/<br />
or<br />
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)><br />
</pre><br />
Please be aware that symbolic links will usually not be copied as links.<br />
<br />
'''Examples:''' <br><br />
<pre><br />
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/<br />
or<br />
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/<br />
</pre><br />
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.<br />
<br />
Further details on the usage of the SCP command line client can be found on the [https://man.openbsd.org/scp man page].<br />
<br />
== Using SFTP on Linux or Mac OS ==<br />
<br />
The following example illustrates the usage of SFTP in a linux shell:<br />
<pre><br />
$ sftp xy1234@os-login.lsdf.kit.edu<br />
Connecting to os-login.lsdf.kit.edu<br><br />
xy1234@os-login.lsdf.kit.edu's password: <br />
<br />
sftp> ls<br />
snapshots<br />
temp test<br />
<br />
sftp> help<br />
...<br />
<br />
sftp> put myfile<br />
<br />
sftp> get myfile<br />
</pre><br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].<br />
<br />
<br />
== Using rsync on Linux or Mac OS ==<br />
<br />
rsync is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:<br />
<pre><br />
In the LSDF:<br />
$ chmod -g+s /lsdf_path/<br />
<br />
On the client host:<br />
$ rsync -uva --no-g --chmod=Dg+s /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
$ echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work and delete data you can use the option ''--delete''. The option ''--delete'' deletes files in the target directory if they were deleted in the source directory.<br />
<br />
<br />
<br />
== Using SSHFS on Linux or Mac OS ==<br />
<br />
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.<br />
<br />
=== Installation ===<br />
<br />
SSHFS is available on many linux distribution via the standard software distribution channels.<br />
<br />
On Debian/Ubuntu like systems:<br />
<pre><br />
$ apt-get install sshfs<br />
</pre><br />
<br />
On RedHat/CentOS like systems:<br />
<pre><br />
$ yum install fuse-sshfs<br />
</pre><br />
<br />
=== Mounting a Remote File System Interactively (as normal user) ===<br />
<br />
To mount a remote file system via command line:<br />
<pre><br />
$ mkdir mountpoint<br />
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ ./mointpoint<br />
</pre><br />
<br />
=== Mounting a Remote File System with ''/etc/fstab'' (as root) ===<br />
<br />
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:<br />
<pre><br />
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects/ <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the <br />
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory). <br />
<br />
For example:<br />
<pre><br />
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
<br />
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):<br />
<pre><br />
$ mount /mnt/mountpoint<br />
</pre><br />
<br />
== Using Secure Shell with Windows ==<br />
<br />
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):<br />
<br />
{| class="wikitable"<br />
| '''MobaXterm'''<br />
| https://mobaxterm.mobatek.net/<br />
|-<br />
| '''PuTTY'''<br />
| https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html<br />
|-<br />
| '''WinSCP'''<br />
| http://winscp.net/eng/download.php<br />
|-<br />
| '''FileZilla'''<br />
| http://sourceforge.net/projects/filezilla/<br />
|-<br />
| '''WebDrive'''<br />
| https://southrivertech.com/products/webdrive/download/<br />
|-<br />
| '''Cygwin'''<br />
| http://cygwin.com/install.html<br />
|-<br />
| '''Windows 10 subsystem for Linux'''<br />
| https://msdn.microsoft.com/de-de/commandline/wsl/install_guide<br />
|}<br />
<br />
The following software packages provide network drive functionality based on SFTP:<br />
<br />
{| class="wikitable"<br />
| '''WebDrive''' (for Windows, Mac, iOS, Android)<br />
| http://www.southrivertechnologies.com/download/downloadwd.html<br />
|-<br />
| '''SFTPNetDrive''' (for Windows)<br />
| http://www.sftpnetdrive.com/<br />
|-<br />
| '''NetDrive''' (for Windows)<br />
| http://www.netdrive.net/<br />
|-<br />
| '''ExpanDrive''' (for Windows and Mac)<br />
| http://www.expandrive.com/expandrive<br />
|-<br />
| '''MountainDuck''' (for Windows and Mac)<br />
| https://mountainduck.io/<br />
|}<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]<br />
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]<br />
<br />
[[Category:LSDF_Online_Storage|Secure Shell]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_Secure_Shell&diff=5616LSDF Online Storage: Secure Shell2020-03-17T11:19:50Z<p>Obholz: /* Using SSH on Linux or Mac OS */</p>
<hr />
<div>Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server. <br />
<br />
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.<br />
<br />
__FORCETOC__<br />
<br />
== Using SSH on Linux or Mac OS ==<br />
<br />
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal<br />
<pre><br />
$ ssh <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
where ''<USERNAME>'' needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server. <br />
<br />
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:<br />
<pre><br />
$ ssh-keygen<br />
</pre><br />
The newly generated key should be password protected.<br />
<br />
To copy the public part of your key pair to the login server issue the following command:<br />
<pre><br />
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
Subsequent logins will use the key instead of a password to authenticate to the login server.<br />
<br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/ssh.1 man page].<br />
<br />
== Using SCP on Linux or Mac OS ==<br />
<br />
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF<br />
Online Storage, you can execute the following commands:<br />
<pre><br />
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/<br />
or<br />
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)><br />
</pre><br />
Please be aware that symbolic links will usually not be copied as links.<br />
<br />
'''Examples:''' <br><br />
<pre><br />
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/<br />
or<br />
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/<br />
</pre><br />
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.<br />
<br />
Further details on the usage of the SCP command line client can be found on the [https://man.openbsd.org/scp man page].<br />
<br />
== Using SFTP on Linux or Mac OS ==<br />
<br />
The following example illustrates the usage of SFTP in a linux shell:<br />
<pre><br />
$ sftp xy1234@os-login.lsdf.kit.edu<br />
Connecting to os-login.lsdf.kit.edu<br><br />
xy1234@os-login.lsdf.kit.edu's password: <br />
<br />
sftp> ls<br />
snapshots<br />
temp test<br />
<br />
sftp> help<br />
...<br />
<br />
sftp> put myfile<br />
<br />
sftp> get myfile<br />
</pre><br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].<br />
<br />
<br />
== Using rsync on Linux or Mac OS ==<br />
<br />
rsync is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:<br />
<pre><br />
In the LSDF:<br />
$ chmod -g+s /lsdf_path/<br />
<br />
On the client host:<br />
$ rsync -uva --no-g --chmod=Dg+s /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
$ echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work and delete data you can use the option ''--delete''. The option ''--delete'' deletes files in the target directory if they were deleted in the source directory.<br />
<br />
<br />
<br />
== Using SSHFS on Linux or Mac OS ==<br />
<br />
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.<br />
<br />
=== Installation ===<br />
<br />
SSHFS is available on many linux distribution via the standard software distribution channels.<br />
<br />
On Debian/Ubuntu like systems:<br />
<pre><br />
$ apt-get install sshfs<br />
</pre><br />
<br />
On RedHat/CentOS like systems:<br />
<pre><br />
$ yum install fuse-sshfs<br />
</pre><br />
<br />
=== Mounting a Remote File System Interactively (as normal user) ===<br />
<br />
To mount a remote file system via command line:<br />
<pre><br />
$ mkdir mountpoint<br />
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects ./mointpoint<br />
</pre><br />
<br />
=== Mounting a Remote File System with ''/etc/fstab'' (as root) ===<br />
<br />
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:<br />
<pre><br />
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the <br />
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory). <br />
<br />
For example:<br />
<pre><br />
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/scc/projects /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
<br />
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):<br />
<pre><br />
$ mount /mnt/mountpoint<br />
</pre><br />
<br />
<br />
<br />
== Using Secure Shell with Windows ==<br />
<br />
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):<br />
<br />
{| class="wikitable"<br />
| '''MobaXterm'''<br />
| https://mobaxterm.mobatek.net/<br />
|-<br />
| '''PuTTY'''<br />
| https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html<br />
|-<br />
| '''WinSCP'''<br />
| http://winscp.net/eng/download.php<br />
|-<br />
| '''FileZilla'''<br />
| http://sourceforge.net/projects/filezilla/<br />
|-<br />
| '''WebDrive'''<br />
| https://southrivertech.com/products/webdrive/download/<br />
|-<br />
| '''Cygwin'''<br />
| http://cygwin.com/install.html<br />
|-<br />
| '''Windows 10 subsystem for Linux'''<br />
| https://msdn.microsoft.com/de-de/commandline/wsl/install_guide<br />
|}<br />
<br />
The following software packages provide network drive functionality based on SFTP:<br />
<br />
{| class="wikitable"<br />
| '''WebDrive''' (for Windows, Mac, iOS, Android)<br />
| http://www.southrivertechnologies.com/download/downloadwd.html<br />
|-<br />
| '''SFTPNetDrive''' (for Windows)<br />
| http://www.sftpnetdrive.com/<br />
|-<br />
| '''NetDrive''' (for Windows)<br />
| http://www.netdrive.net/<br />
|-<br />
| '''ExpanDrive''' (for Windows and Mac)<br />
| http://www.expandrive.com/expandrive<br />
|-<br />
| '''MountainDuck''' (for Windows and Mac)<br />
| https://mountainduck.io/<br />
|}<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]<br />
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]<br />
<br />
[[Category:LSDF_Online_Storage|Secure Shell]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_Secure_Shell&diff=5615LSDF Online Storage: Secure Shell2020-03-17T09:33:19Z<p>Obholz: /* Using SCP on Linux or Mac OS */</p>
<hr />
<div>Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server. <br />
<br />
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.<br />
<br />
__FORCETOC__<br />
<br />
== Using SSH on Linux or Mac OS ==<br />
<br />
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal<br />
<pre><br />
$ ssh <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
where ''<USERNAME>'' needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server. <br />
<br />
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:<br />
<pre><br />
$ ssh-keygen<br />
</pre><br />
The newly generated key should be password protected.<br />
<br />
To copy the public part of your key pair to the login server issue the following command:<br />
<pre><br />
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
Subsequent logins will use the key instead of a password to authenticate to the login server.<br />
<br />
<br />
'''Attention:''' Posix style file commands (for instance '''chmod''') must not be applied on files on a CIFS share. This may revoke sharing and Access Lists (ACLs), steering the access to the files.<br />
<br />
<br />
<br />
== Using SCP on Linux or Mac OS ==<br />
<br />
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF<br />
Online Storage, you can execute the following commands:<br />
<pre><br />
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/<br />
or<br />
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)><br />
</pre><br />
Please be aware that symbolic links will usually not be copied as links.<br />
<br />
'''Examples:''' <br><br />
<pre><br />
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/<br />
or<br />
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/<br />
</pre><br />
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.<br />
<br />
Further details on the usage of the SCP command line client can be found on the [https://man.openbsd.org/scp man page].<br />
<br />
== Using SFTP on Linux or Mac OS ==<br />
<br />
The following example illustrates the usage of SFTP in a linux shell:<br />
<pre><br />
$ sftp xy1234@os-login.lsdf.kit.edu<br />
Connecting to os-login.lsdf.kit.edu<br><br />
xy1234@os-login.lsdf.kit.edu's password: <br />
<br />
sftp> ls<br />
snapshots<br />
temp test<br />
<br />
sftp> help<br />
...<br />
<br />
sftp> put myfile<br />
<br />
sftp> get myfile<br />
</pre><br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].<br />
<br />
<br />
== Using rsync on Linux or Mac OS ==<br />
<br />
rsync is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:<br />
<pre><br />
In the LSDF:<br />
$ chmod -g+s /lsdf_path/<br />
<br />
On the client host:<br />
$ rsync -uva --no-g --chmod=Dg+s /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
$ echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work and delete data you can use the option ''--delete''. The option ''--delete'' deletes files in the target directory if they were deleted in the source directory.<br />
<br />
<br />
<br />
== Using SSHFS on Linux or Mac OS ==<br />
<br />
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.<br />
<br />
=== Installation ===<br />
<br />
SSHFS is available on many linux distribution via the standard software distribution channels.<br />
<br />
On Debian/Ubuntu like systems:<br />
<pre><br />
$ apt-get install sshfs<br />
</pre><br />
<br />
On RedHat/CentOS like systems:<br />
<pre><br />
$ yum install fuse-sshfs<br />
</pre><br />
<br />
=== Mounting a Remote File System Interactively (as normal user) ===<br />
<br />
To mount a remote file system via command line:<br />
<pre><br />
$ mkdir mountpoint<br />
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects ./mointpoint<br />
</pre><br />
<br />
=== Mounting a Remote File System with ''/etc/fstab'' (as root) ===<br />
<br />
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:<br />
<pre><br />
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the <br />
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory). <br />
<br />
For example:<br />
<pre><br />
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/scc/projects /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
<br />
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):<br />
<pre><br />
$ mount /mnt/mountpoint<br />
</pre><br />
<br />
<br />
<br />
== Using Secure Shell with Windows ==<br />
<br />
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):<br />
<br />
{| class="wikitable"<br />
| '''MobaXterm'''<br />
| https://mobaxterm.mobatek.net/<br />
|-<br />
| '''PuTTY'''<br />
| https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html<br />
|-<br />
| '''WinSCP'''<br />
| http://winscp.net/eng/download.php<br />
|-<br />
| '''FileZilla'''<br />
| http://sourceforge.net/projects/filezilla/<br />
|-<br />
| '''WebDrive'''<br />
| https://southrivertech.com/products/webdrive/download/<br />
|-<br />
| '''Cygwin'''<br />
| http://cygwin.com/install.html<br />
|-<br />
| '''Windows 10 subsystem for Linux'''<br />
| https://msdn.microsoft.com/de-de/commandline/wsl/install_guide<br />
|}<br />
<br />
The following software packages provide network drive functionality based on SFTP:<br />
<br />
{| class="wikitable"<br />
| '''WebDrive''' (for Windows, Mac, iOS, Android)<br />
| http://www.southrivertechnologies.com/download/downloadwd.html<br />
|-<br />
| '''SFTPNetDrive''' (for Windows)<br />
| http://www.sftpnetdrive.com/<br />
|-<br />
| '''NetDrive''' (for Windows)<br />
| http://www.netdrive.net/<br />
|-<br />
| '''ExpanDrive''' (for Windows and Mac)<br />
| http://www.expandrive.com/expandrive<br />
|-<br />
| '''MountainDuck''' (for Windows and Mac)<br />
| https://mountainduck.io/<br />
|}<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]<br />
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]<br />
<br />
[[Category:LSDF_Online_Storage|Secure Shell]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_Secure_Shell&diff=5614LSDF Online Storage: Secure Shell2020-03-17T09:18:37Z<p>Obholz: /* Using SCP on Linux or Mac OS */</p>
<hr />
<div>Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server. <br />
<br />
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.<br />
<br />
__FORCETOC__<br />
<br />
== Using SSH on Linux or Mac OS ==<br />
<br />
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal<br />
<pre><br />
$ ssh <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
where ''<USERNAME>'' needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server. <br />
<br />
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:<br />
<pre><br />
$ ssh-keygen<br />
</pre><br />
The newly generated key should be password protected.<br />
<br />
To copy the public part of your key pair to the login server issue the following command:<br />
<pre><br />
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu<br />
</pre><br />
Subsequent logins will use the key instead of a password to authenticate to the login server.<br />
<br />
<br />
'''Attention:''' Posix style file commands (for instance '''chmod''') must not be applied on files on a CIFS share. This may revoke sharing and Access Lists (ACLs), steering the access to the files.<br />
<br />
<br />
<br />
== Using SCP on Linux or Mac OS ==<br />
<br />
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF<br />
Online Storage, you can execute the following commands:<br />
<pre><br />
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/<br />
or<br />
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)><br />
</pre><br />
Please be aware that symbolic links will usually not be copied as links.<br />
<br />
'''Examples:''' <br><br />
<pre><br />
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/<br />
or<br />
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/<br />
</pre><br />
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.<br />
<br />
== Using SFTP on Linux or Mac OS ==<br />
<br />
The following example illustrates the usage of SFTP in a linux shell:<br />
<pre><br />
$ sftp xy1234@os-login.lsdf.kit.edu<br />
Connecting to os-login.lsdf.kit.edu<br><br />
xy1234@os-login.lsdf.kit.edu's password: <br />
<br />
sftp> ls<br />
snapshots<br />
temp test<br />
<br />
sftp> help<br />
...<br />
<br />
sftp> put myfile<br />
<br />
sftp> get myfile<br />
</pre><br />
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].<br />
<br />
<br />
== Using rsync on Linux or Mac OS ==<br />
<br />
rsync is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:<br />
<pre><br />
In the LSDF:<br />
$ chmod -g+s /lsdf_path/<br />
<br />
On the client host:<br />
$ rsync -uva --no-g --chmod=Dg+s /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
$ echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work and delete data you can use the option ''--delete''. The option ''--delete'' deletes files in the target directory if they were deleted in the source directory.<br />
<br />
<br />
<br />
== Using SSHFS on Linux or Mac OS ==<br />
<br />
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.<br />
<br />
=== Installation ===<br />
<br />
SSHFS is available on many linux distribution via the standard software distribution channels.<br />
<br />
On Debian/Ubuntu like systems:<br />
<pre><br />
$ apt-get install sshfs<br />
</pre><br />
<br />
On RedHat/CentOS like systems:<br />
<pre><br />
$ yum install fuse-sshfs<br />
</pre><br />
<br />
=== Mounting a Remote File System Interactively (as normal user) ===<br />
<br />
To mount a remote file system via command line:<br />
<pre><br />
$ mkdir mountpoint<br />
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects ./mointpoint<br />
</pre><br />
<br />
=== Mounting a Remote File System with ''/etc/fstab'' (as root) ===<br />
<br />
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:<br />
<pre><br />
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the <br />
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory). <br />
<br />
For example:<br />
<pre><br />
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/scc/projects /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0<br />
</pre><br />
<br />
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):<br />
<pre><br />
$ mount /mnt/mountpoint<br />
</pre><br />
<br />
<br />
<br />
== Using Secure Shell with Windows ==<br />
<br />
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):<br />
<br />
{| class="wikitable"<br />
| '''MobaXterm'''<br />
| https://mobaxterm.mobatek.net/<br />
|-<br />
| '''PuTTY'''<br />
| https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html<br />
|-<br />
| '''WinSCP'''<br />
| http://winscp.net/eng/download.php<br />
|-<br />
| '''FileZilla'''<br />
| http://sourceforge.net/projects/filezilla/<br />
|-<br />
| '''WebDrive'''<br />
| https://southrivertech.com/products/webdrive/download/<br />
|-<br />
| '''Cygwin'''<br />
| http://cygwin.com/install.html<br />
|-<br />
| '''Windows 10 subsystem for Linux'''<br />
| https://msdn.microsoft.com/de-de/commandline/wsl/install_guide<br />
|}<br />
<br />
The following software packages provide network drive functionality based on SFTP:<br />
<br />
{| class="wikitable"<br />
| '''WebDrive''' (for Windows, Mac, iOS, Android)<br />
| http://www.southrivertechnologies.com/download/downloadwd.html<br />
|-<br />
| '''SFTPNetDrive''' (for Windows)<br />
| http://www.sftpnetdrive.com/<br />
|-<br />
| '''NetDrive''' (for Windows)<br />
| http://www.netdrive.net/<br />
|-<br />
| '''ExpanDrive''' (for Windows and Mac)<br />
| http://www.expandrive.com/expandrive<br />
|-<br />
| '''MountainDuck''' (for Windows and Mac)<br />
| https://mountainduck.io/<br />
|}<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]<br />
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]<br />
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]<br />
<br />
[[Category:LSDF_Online_Storage|Secure Shell]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_rsync&diff=5613LSDF Online Storage: rsync2020-03-16T08:47:26Z<p>Obholz: </p>
<hr />
<div><H1>Using rsync from UNIX client via the login-cluster</H1><br />
<br />
'''rsync''' is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote '''rsync''' daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amoun of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the desti-nation file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
'''Attention:''' '''rsync''' with '''-a''' (or '''--group''') option write the data with wrong GID (from source).<br />
<br />
'''Example:'''<br />
<pre><br />
#set sticky bit and group, if it is missing or wrong:<br />
>ssh xy1234@os-login.lsdf.kit.edu<br />
>chmod -g+s /lsdf_path/<br />
>chgrp lsdf-project-group /lsdf_path/<br />
<br />
#rsync via ssh (workaround with '-a --no-g --chmod=Dg+s')<br />
>rsync -uva --no-g --chmod=Dg+s --rsh="ssh -c es128-gcm@openssh.com -p 22" /source_path/source_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<br />
<hr><br />
To '''reach maximum bandwidth''' we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission. <br><br><br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
> echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work in bwgrid and delete data there you can use the option ‚''--delete''‘. With this your copy in LSDF Online Storage will be kept up to date. The option ‚''--delete''‘ deletes files in the target directory if they were deleted in the source directory.<br />
<br />
<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|rsync]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_HTTPS&diff=5595LSDF Online Storage: HTTPS2019-12-19T12:50:40Z<p>Obholz: </p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_WebDAV#Using_WebDAV_from_a_Web_Browser]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|HTTPS]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_HTTPS&diff=5594LSDF Online Storage: HTTPS2019-12-19T12:47:25Z<p>Obholz: Redirected page to LSDF Online Storage: WebDAV#Using WebDAV from a Web Browser</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_WebDAV#Using_WebDAV_from_a_Web_Browser]]<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|LSDF Online Storage|HTTPS]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage_rdata&diff=5593LSDF Online Storage rdata2019-12-19T12:43:04Z<p>Obholz: Redirected page to LSDF Online Storage: HPC#rdata</p>
<hr />
<div>#REDIRECT [[LSDF_Online_Storage:_HPC#rdata]]<br />
<!--http://wiki.scc.kit.edu/lsdf/index.php/LSDF_Online_Storage_rdata--><br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|RDATA]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage_rdata&diff=5592LSDF Online Storage rdata2019-12-19T12:27:17Z<p>Obholz: Redirected page to Category:LSDF Online Storage: HPC</p>
<hr />
<div>#REDIRECT [[:Category:LSDF_Online_Storage:_HPC]]<br />
<!--http://wiki.scc.kit.edu/lsdf/index.php/LSDF_Online_Storage_rdata--><br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|RDATA]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage_rdata&diff=5591LSDF Online Storage rdata2019-12-19T12:26:42Z<p>Obholz: Redirected page to Category:LSDF Online Storage</p>
<hr />
<div>#REDIRECT [[:Category:LSDF_Online_Storage|HPC]]<br />
<!--http://wiki.scc.kit.edu/lsdf/index.php/LSDF_Online_Storage_rdata--><br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
<br />
[[Category:LSDF_Online_Storage|RDATA]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_WebDAV&diff=5590LSDF Online Storage: WebDAV2019-12-19T09:57:35Z<p>Obholz: </p>
<hr />
<div>WebDAV or Web-based Distributed Authoring and Versioning is an extension of the Hypertext Transfer Protocol (HTTP) allowing to access files over the internet. WebDAV access to the LSDF Online Storage is provided via a cluster of servers reachable at the URL '''https://os-webdav.lsdf.kit.edu''' or '''davs://os-webdav.lsdf.kit.edu''' depending on the client used.<br />
<br />
__FORCETOC__<br />
<br />
<br />
== Prerequisites and Restrictions == <br />
<br />
Please note the following requirements and restrictions when using the LSDF WebDAV service:<br />
<br />
The WebDAV protocol in general does suport the listing of special files like symbolic links, i.e. symbolic links present in the LSDF file space will not be listed with most of the tools described on this page. Nevertheless it is still possible to access the content behind symbolic links. Therefore, it is recommended to access project directories directly, e.g. when mounting them in the file system or mapping them as network drives.<br />
<br />
<br />
== Using WebDAV from a Web Browser ==<br />
<br />
To access the starting page enter the following URL in your web browser<br />
<pre><br />
https://os-webdav.lsdf.kit.edu/<br />
</pre><br />
Project directories are accessible at the usual location:<br />
<pre><br />
https://os-webdav.lsdf.kit.edu/<OE>/<inst>/projects/<PROJECTNAME>/<br />
</pre><br />
where <br />
''<OE>'' is the name of your organizational unit, e.g. kit, bs, fiz ...;<br />
"<inst>" is the name of your institution, e.g. scc, iai, ...;<br />
''<PROJECTNAME>'' ist the name of the project. <br />
Your home directory is accessible at the following URL:<br />
<pre><br />
https://os-webdav.lsdf.kit.edu/<OE>/<inst>/<USERNAME>/<br />
</pre><br />
where ''<USERNAME>'' is you username: xy1234, bs_abcd,....<br />
<br />
<br />
<br />
== Using the DaviX Command Line Client on Linux and Mac OS ==<br />
<br />
The DaviX project provides a library and a set of client tools for file management over <br />
HTTP-based protocols. The command line client tools can be either installed as a regular <br />
package in you linux distribution, e.g. in RedHat based distributions with<br />
<pre><br />
$ yum install -y epel-release<br />
$ yum install davix<br />
</pre><br />
or from the [https://dmc.web.cern.ch/projects/davix/home home page of the DaviX project].<br />
<br />
<br />
<br />
=== Listing a Directory ===<br />
<br />
The content of a directory or individual files can be listed with the following command:<br />
<pre><br />
$ davix-ls -l davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234<br />
Basic authentication - server is asking for username and password:<br />
Login: xy1234<br />
Password: <br />
-rwxrwxrwx 0 10485760 2019-05-20 09:29:18 a_small_file<br />
</pre><br />
<br />
<br />
<br />
=== Copying Files ===<br />
<br />
To copy a file (e.g. testfile) from the WebDAV share to you local directory:<br />
<pre><br />
$ davix-get davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/testfile testfile<br />
Basic authentication - server is asking for username and password:<br />
Login: xy1234<br />
Password: <br />
Performing Read operation on: davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/testfile<br />
[==================================] 100% 10MiB/10MiB 0B/s <br />
</pre><br />
<br />
<br />
<br />
=== Recursively Uploading Files ===<br />
<br />
The following example shows how to recursively upload files with 6 transfers in parallel:<br />
<pre><br />
$ davix-put -r6 a_directory davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/a_new_directory<br />
Basic authentication - server is asking for username and password:<br />
Login: xy1234<br />
Password: <br />
Populating task queue for a_directory/ Files processed: 0 <br />
</pre><br />
<br />
<br />
<br />
=== Recursively downloading files ===<br />
<br />
Files and directories can also be downloaded recursively. In the following example a directory tree will be downloaded recursively with 6 transfers in parallel:<br />
<pre><br />
$ davix-get -r6 davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/a_new_directory downloaded_dir<br />
Basic authentication - server is asking for username and password:<br />
Login: xy1234<br />
Password: <br />
Crawling davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/a_new_directory/ Files processed: 1 <br />
</pre><br />
<br />
<br />
<br />
== Connection with via File Manager on Linux == <br />
<br />
A number of different linux file managers support WebDAV shares natively. Among others these are<br />
<br />
* '''Nautilus''' (Gnome)<br />
* '''Dolphin/Konqueror''' (KDE / Kubuntu)<br />
* '''Thunar''' (Xfce / Xubuntu)<br />
<br />
To mount a LSDF WebDAV share (e.g. a project directory) with Nautilus follow the steps outlined below:<br />
<br />
# Open ''Nautilus''.<br />
# Click ''+ Other Location'' on the left hand pane.<br />
# In the text box ''Connect to Server'' enter the URL to your project folder <nowiki>davs://os-webdav.lsdf.kit.edu/<OE>/<inst>/projects/<PROJECT></nowiki>, where ''<OE>'' is you organizational unit and ''<PROJECT>'' is the name of your project directory, e.g. <nowiki>davs://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject</nowiki><br />
# Click ''Connect''<br />
<br />
The share will be mounted on the local system and accessible both in Nautilus and from the directory ''/run/user/<UID>/gvfs'', where ''<UID>'' is the id of your user account.<br />
<br />
<br />
<br />
== Mounting a WebDAV Share on Linux with davfs2 ==<br />
<br />
davfs2 is a file system driver that allows mounting a WebDAV server as a local file system, like a disk drive. This way, applications can access resources on a WebDAV server without knowing anything about HTTP or WebDAV. davfs2 does extensive caching to make the file system responsive, to avoid unnecessary network traffic and to prevent data loss, and to cope for slow or unreliable connections. <br />
<br />
davfs2 has a system wide configuration file ''/etc/davfs2/davfs2.conf'' and a user configuration file ''~/.davfs2/davfs2.conf''. The user configuration file is used in addition to the system configuration file when invoked by a normal user. Documentation on all options can be found in the [https://linux.die.net/man/5/davfs2.conf manual page].<br />
<br />
<br />
<br />
=== Installation ===<br />
<br />
On CentOS or RedHat like systems install the package davfs2 which is available in the epel repository:<br />
<pre><br />
$ yum install -y epel-release<br />
$ yum install davfs2<br />
</pre><br />
<br />
<br />
<br />
=== Mounting (as root) ===<br />
<br />
To mount an LSDF WebDAV project or user share issue the following command as root:<br />
<pre><br />
$ mount.davfs https://os-webdav.lsdf.kit.edu/<OE>/<inst>/projects/<PROJECTNAME> <MOUNTPOINT><br />
</pre><br />
where ''<OE>'' is the name of your organizational unit, e.g. kit, ''<PROJECTNAME>'' is the name of the project, and ''<MOUNTPOINT>'' is a directory. For example:<br />
<pre><br />
$ mkdir /mnt/my_lsdf_project<br />
$ mount.davfs https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project<br />
Please enter the username to authenticate with server<br />
https://os-webdav.lsdf.kit.edu/ or hit enter for none.<br />
Username: xy1234<br />
Please enter the password to authenticate user fg9068 with server<br />
https://os-webdav.lsdf.kit.edu/ or hit enter for none.<br />
Password: <br />
</pre><br />
<br />
As an alternative it is possible to preconfigure a WebDAV mount. For the project mount described in the example above you would have to add the following line to the file ''/etc/fstab'':<br />
<pre><br />
https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project davfs rw,noauto,user 0 0<br />
</pre><br />
User authentication information needs to be stored in the file ''/etc/davfs2/secrets'':<br />
<pre><br />
$ cat /etc/davfs2/secrets | grep os-webdav<br />
https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project xy1234 ***<br />
</pre><br />
Afterwards create the mount point directory and mount the share with the command<br />
$ mount /mnt/my_lsdf_project<br />
</pre><br />
<br />
<br />
<br />
=== Unmounting (as root) ===<br />
<br />
To unmount a mounted WebDAV share issue the following command:<br />
<pre><br />
$ umount.davfs /mnt/my_lsdf_project<br />
</pre><br />
<br />
<br />
<br />
=== Mouting (as normal user) ===<br />
<br />
To mount a share as an unprivileged user, the user account needs to be a member of the group ''davfs2'':<br />
<pre><br />
$ usermod -a -G davfs2 xy1234<br />
</pre><br />
The mount has to be defined in the file ''/etc/fstab''. To mount a project directory add for example the following line to the file:<br />
<pre><br />
https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project davfs rw,noauto,user 0 0<br />
</pre><br />
User specific authentication information can be stored file '~/'.davfs2/secrets'':<br />
<pre><br />
$ mkdir ~/.davfs2/secrets<br />
$ chmod 600 ~/.davfs2/secrets<br />
$ cat ~/.davfs2/secrets<br />
https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project xy1234 ***<br />
</pre><br />
To mount the share issue the command<br />
<pre><br />
$ mount /mnt/my_lsdf_project<br />
</pre><br />
<br />
== Using WebDAV on Windows ==<br />
<br />
To mount a LSDF WebDAV share (e.g. a project directory) and map it to a drive in windows follow the steps outlined below:<br />
<br />
# Open a ''File Explorer'' and select ''This PC'' on the left hand pane.<br />
# In the Menu select ''Computer => Map Network Drive''.<br />
# Click ''Connect to a Web site that you can use to store your documents and pictures.''<br />
# Click ''Next''<br />
# Click ''Choose a custom network location'' and afterwards click ''Next''.<br />
# Enter the URL to you project folder <nowiki>https://os-webdav.lsdf.kit.edu/kit/<OE>/projects/<PROJECT></nowiki>, where ''<OE>'' is you organizational unit and ''<PROJECT>'' is the name of your project directory, e.g. <nowiki>https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject</nowiki><br />
# Click Next<br />
# Enter a name for the network location and click ''Next''<br />
# Click ''Finish''<br />
<br />
<br />
<br />
== Public Shares ==<br />
<br />
Scientific data in LSDF storage projects can be shared publicly with HTTPS/WebDAV. Public projects can be reached under the URL https://os-webdav.lsdf.kit.edu/public. Please [https://bw-support.scc.kit.edu/?mode=ticket_submit&respounit=LSDF%20Online%20Storage contact the LSDF support] to publish an existing storage project.<br />
<br />
<br />
<br />
== External Links ==<br />
<br />
* [http://savannah.nongnu.org/projects/davfs2 davfs2 project home page]<br />
* [https://dmc.web.cern.ch/projects/davix/home DaviX project home page]<br />
<br />
<br />
[[Category:LSDF_Online_Storage|WebDAV]]</div>Obholzhttps://wiki.scc.kit.edu/lsdf/index.php?title=LSDF_Online_Storage:_rsync&diff=5589LSDF Online Storage: rsync2019-12-19T09:56:02Z<p>Obholz: </p>
<hr />
<div><H1>Using rsync from UNIX client via the login-cluster</H1><br />
<br />
'''rsync''' is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote '''rsync''' daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amoun of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.<br />
<br />
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the desti-nation file directly when the quick check indicates that the files data does not need to be updated.<br />
<br />
'''Attention:''' '''rsync''' with '''-a''' (or '''--group''') option write the data with wrong GID (from source).<br />
<br />
'''Example:'''<br />
<pre><br />
#set sticky bit and group, if it is missing or wrong:<br />
>ssh xy1234@os-login.lsdf.kit.edu<br />
>chmod -g+s /lsdf_path/<br />
>chgrp lsdf-project-group /lsdf_path/<br />
<br />
#rsync via ssh (workaround wich '-a --no-g --chmod=Dg+s')<br />
>rsync -uva --no-g --chmod=Dg+s --rsh="ssh -c es128-gcm@openssh.com -p 22" /source_path/source_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err<br />
<br />
xy1234@os-login.lsdf.kit.edu's password:<br />
</pre><br />
<br />
<hr><br />
To '''reach maximum bandwidth''' we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission. <br><br><br />
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.<br />
<pre><br />
> echo $?<br />
</pre><br />
<br />
<br><br />
If you continue to work in bwgrid and delete data there you can use the option ‚''--delete''‘. With this your copy in LSDF Online Storage will be kept up to date. The option ‚''--delete''‘ deletes files in the target directory if they were deleted in the source directory.<br />
<br />
<br />
<hr><br />
<br><br />
<br><br />
<br><br />
<br><br />
[[Category:LSDF_Online_Storage|rsync]]</div>Obholz