Difference between revisions of "LSDF Online Storage: Secure Shell"

From Lsdf
(16 intermediate revisions by 2 users not shown)
Line 3: Line 3:
 
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.
 
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address '''os-login.lsdf.kit.edu'''.
   
  +
The login cluster are directly accessible by end users.
__FORCETOC__
 
  +
These nodes are used for interactive login, file management, data retrieving and restoring. <br>
  +
Three nodes are dedicated to this service but they are all accessible via one host name and a DNS round-robin alias distributes the login sessions to the different login nodes.
   
  +
'''Environment variables:'''
  +
<pre>
  +
$SHELL=/bin/bash
  +
$USER=xy1234
  +
$LSDF=/lsdf
  +
$HOME=/lsdf/kit/inst/xy1234
  +
$LSDFPROJECTS=/lsdf/kit/inst/projects
  +
$SNAPSHOTS=/lsdf/kit/snapshots
  +
$PROJECTSNAPSHOTS=/lsdf/kit/inst/projects/.snapshots
  +
</pre>
   
  +
<hr>
  +
__FORCETOC__
   
 
== Using SSH on Linux or Mac OS ==
 
== Using SSH on Linux or Mac OS ==
Line 27: Line 41:
 
Subsequent logins will use the key instead of a password to authenticate to the login server.
 
Subsequent logins will use the key instead of a password to authenticate to the login server.
   
  +
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/ssh.1 man page].
   
  +
== Using SCP on Linux or Mac OS ==
'''Attention:''' Posix style file commands (for instance '''chmod''') must not be applied on files on a CIFS share. This may revoke sharing and Access Lists (ACLs), steering the access to the files.
 
   
  +
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF
  +
Online Storage, you can execute the following commands:
  +
<pre>
  +
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/
  +
or
  +
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)>
  +
</pre>
  +
Please be aware that symbolic links will usually not be copied as links.
  +
  +
'''Examples:''' <br>
  +
<pre>
  +
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/
  +
or
  +
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/
  +
</pre>
  +
To reach maximum bandwidth we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission.
  +
  +
Further details on the usage of the SCP command line client can be found on the [https://man.openbsd.org/scp.1 man page].
  +
  +
== Using SFTP on Linux or Mac OS ==
  +
  +
The following example illustrates the usage of SFTP in a linux shell:
  +
<pre>
  +
$ sftp xy1234@os-login.lsdf.kit.edu
  +
Connecting to os-login.lsdf.kit.edu<br>
  +
xy1234@os-login.lsdf.kit.edu's password:
  +
  +
sftp> ls
  +
snapshots
  +
temp test
  +
  +
sftp> help
  +
...
  +
  +
sftp> put myfile
  +
  +
sftp> get myfile
  +
</pre>
  +
Further details on the usage of the SFTP command line client can be found on the [https://man.openbsd.org/sftp.1 man page].
  +
  +
  +
== Using rsync on Linux or Mac OS ==
  +
  +
'''rsync''' is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. '''rsync''' is widely used for backups and mirroring and as an improved copy command for everyday use.
  +
  +
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.
  +
  +
'''Attention: rsync''' with '''-a''' (or '''--group''') option write the data with wrong GID (from source).
  +
  +
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:
  +
<pre>
  +
#In the LSDF: set sticky bit and group, if it is missing or wrong:
  +
$ chmod -g+s /lsdf_path/
  +
$ chgrp lsdf-project-group /lsdf_path/
  +
  +
#On the client host: rsync via ssh (workaround with '-a --no-g --chmod=Dg+s')
  +
$ rsync -uva --no-g --chmod=Dg+s --rsh="ssh -c es128-gcm@openssh.com -p 22" /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err
  +
  +
xy1234@os-login.lsdf.kit.edu's password:
  +
</pre>
  +
<hr>
  +
To '''reach maximum bandwidth''' we recommend ssh ciphers '''arcfour''', '''aes128-gcm@openssh.com''' or '''aes128-cbc''' algorithm for the encryption of the transmission. <br><br>
  +
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.
  +
<pre>
  +
$ echo $?
  +
</pre>
   
  +
<br>
  +
If you continue to work and delete data you can use the option '''--delete'''. The option '''--delete''' deletes files in the target directory if they were deleted in the source directory.
   
 
== Using SSHFS on Linux or Mac OS ==
 
== Using SSHFS on Linux or Mac OS ==
Line 55: Line 138:
 
<pre>
 
<pre>
 
$ mkdir mountpoint
 
$ mkdir mountpoint
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects ./mointpoint
+
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ ./mointpoint
 
</pre>
 
</pre>
   
Line 62: Line 145:
 
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:
 
Add (as root) a line to your ''/etc/fstab'' describing how to mount the remote folder:
 
<pre>
 
<pre>
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0
+
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects/ <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0
 
</pre>
 
</pre>
 
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the
 
where ''<UID'' is the numerical user id of the user the remote mount will be mapped to, ''<GID>'' is the numerical group id of the
 
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory).
 
user the remote mount will be mapped to, ''<USERNAME>'' is the user name user for the SSH login, ''<MOUNTPOINT>'' is the local mount point (directory).
   
For example:
+
'''For example:'''
 
<pre>
 
<pre>
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/scc/projects /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0
+
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0
 
</pre>
 
</pre>
   
Line 76: Line 159:
 
$ mount /mnt/mountpoint
 
$ mount /mnt/mountpoint
 
</pre>
 
</pre>
 
 
   
 
== Using Secure Shell with Windows ==
 
== Using Secure Shell with Windows ==
Line 125: Line 206:
 
|}
 
|}
   
== External Links ==
 
   
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell]
 
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem]
 
   
  +
== External Links ==
  +
  +
* [http://www.openssh.com/manual.html OpenSSH Manual Pages]
  +
* [https://en.wikipedia.org/wiki/Secure_Shell Secure Shell on Wikipedia]
  +
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SSH File Transfer Protocol on Wikipedia]
  +
* [https://en.wikipedia.org/wiki/SSHFS Secure Shell Filesystem on Wikipedia]
   
 
[[Category:LSDF_Online_Storage|Secure Shell]]
 
[[Category:LSDF_Online_Storage|Secure Shell]]

Revision as of 15:42, 17 March 2020

Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server.

SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address os-login.lsdf.kit.edu.

The login cluster are directly accessible by end users. These nodes are used for interactive login, file management, data retrieving and restoring.
Three nodes are dedicated to this service but they are all accessible via one host name and a DNS round-robin alias distributes the login sessions to the different login nodes.

Environment variables:

$SHELL=/bin/bash
$USER=xy1234
$LSDF=/lsdf
$HOME=/lsdf/kit/inst/xy1234
$LSDFPROJECTS=/lsdf/kit/inst/projects
$SNAPSHOTS=/lsdf/kit/snapshots
$PROJECTSNAPSHOTS=/lsdf/kit/inst/projects/.snapshots


Using SSH on Linux or Mac OS

To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal

$ ssh <USERNAME>@os-login.lsdf.kit.edu

where <USERNAME> needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server.

To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:

$ ssh-keygen

The newly generated key should be password protected.

To copy the public part of your key pair to the login server issue the following command:

$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu

Subsequent logins will use the key instead of a password to authenticate to the login server.

Further details on the usage of the SFTP command line client can be found on the man page.

Using SCP on Linux or Mac OS

SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF Online Storage, you can execute the following commands:

$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/
or
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)>

Please be aware that symbolic links will usually not be copied as links.

Examples:

>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/
or
>scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename  ~/

To reach maximum bandwidth we recommend ssh ciphers arcfour, aes128-gcm@openssh.com or aes128-cbc algorithm for the encryption of the transmission.

Further details on the usage of the SCP command line client can be found on the man page.

Using SFTP on Linux or Mac OS

The following example illustrates the usage of SFTP in a linux shell:

$ sftp  xy1234@os-login.lsdf.kit.edu
Connecting to os-login.lsdf.kit.edu<br>
xy1234@os-login.lsdf.kit.edu's password: 

sftp> ls
snapshots
temp test

sftp> help
...

sftp> put myfile

sftp> get myfile

Further details on the usage of the SFTP command line client can be found on the man page.


Using rsync on Linux or Mac OS

rsync is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.

Attention: rsync with -a (or --group) option write the data with wrong GID (from source).

The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:

#In the LSDF: set sticky bit and group, if it is missing or wrong:
$ chmod -g+s  /lsdf_path/
$ chgrp lsdf-project-group /lsdf_path/

#On the client host: rsync via ssh (workaround with '-a --no-g --chmod=Dg+s')
$ rsync -uva --no-g --chmod=Dg+s  --rsh="ssh -c es128-gcm@openssh.com -p 22"  /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err

xy1234@os-login.lsdf.kit.edu's password:

To reach maximum bandwidth we recommend ssh ciphers arcfour, aes128-gcm@openssh.com or aes128-cbc algorithm for the encryption of the transmission.

As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.

$ echo $?


If you continue to work and delete data you can use the option --delete. The option --delete deletes files in the target directory if they were deleted in the source directory.

Using SSHFS on Linux or Mac OS

SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.

Installation

SSHFS is available on many linux distribution via the standard software distribution channels.

On Debian/Ubuntu like systems:

$ apt-get install sshfs

On RedHat/CentOS like systems:

$ yum install fuse-sshfs

Mounting a Remote File System Interactively (as normal user)

To mount a remote file system via command line:

$ mkdir mountpoint
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ ./mointpoint

Mounting a Remote File System with /etc/fstab (as root)

Add (as root) a line to your /etc/fstab describing how to mount the remote folder:

sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects/     <MOUNTPOINT>    fuse    uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0

where <UID is the numerical user id of the user the remote mount will be mapped to, <GID> is the numerical group id of the user the remote mount will be mapped to, <USERNAME> is the user name user for the SSH login, <MOUNTPOINT> is the local mount point (directory).

For example:

sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/   /mnt/mountpoint    fuse    uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0

In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):

$ mount /mnt/mountpoint

Using Secure Shell with Windows

Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):

MobaXterm https://mobaxterm.mobatek.net/
PuTTY https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
WinSCP http://winscp.net/eng/download.php
FileZilla http://sourceforge.net/projects/filezilla/
WebDrive https://southrivertech.com/products/webdrive/download/
Cygwin http://cygwin.com/install.html
Windows 10 subsystem for Linux https://msdn.microsoft.com/de-de/commandline/wsl/install_guide

The following software packages provide network drive functionality based on SFTP:

WebDrive (for Windows, Mac, iOS, Android) http://www.southrivertechnologies.com/download/downloadwd.html
SFTPNetDrive (for Windows) http://www.sftpnetdrive.com/
NetDrive (for Windows) http://www.netdrive.net/
ExpanDrive (for Windows and Mac) http://www.expandrive.com/expandrive
MountainDuck (for Windows and Mac) https://mountainduck.io/


External Links