LSDF Online Storage: Secure Shell

Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server.

SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address os-login.lsdf.kit.edu.

Using SSH on Linux or Mac OS

To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal

$ ssh <USERNAME>@os-login.lsdf.kit.edu

where <USERNAME> needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server.

To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:

$ ssh-keygen

The newly generated key should be password protected.

To copy the public part of your key pair to the login server issue the following command:

$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu

Subsequent logins will use the key instead of a password to authenticate to the login server.

Attention: Posix style file commands (for instance chmod) must not be applied on files on a CIFS share. This may revoke sharing and Access Lists (ACLs), steering the access to the files.

Using SCP on Linux or Mac OS

SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF Online Storage, you can execute the following commands:

$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/
$ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)>

Please be aware that symbolic links will usually not be copied as links.

Using SFTP on Linux or Mac OS

The following example illustrates the usage of SFTP in a linux shell:

$ sftp  xy1234@os-login.lsdf.kit.edu
Connecting to os-login.lsdf.kit.edu<br>
xy1234@os-login.lsdf.kit.edu's password: 

sftp> ls
snapshots
temp test

sftp> help
...

sftp> put myfile

sftp> get myfile

Further details on the usage of the SFTP command line client can be found on the man page.

Using rsync on Linux or Mac OS

rsync is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.

The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:

In the LSDF:
$ chmod -g+s  /lsdf_path/

On the client host:
$ rsync -uva --no-g --chmod=Dg+s /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err

xy1234@os-login.lsdf.kit.edu's password:

As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.

$ echo $?

If you continue to work and delete data you can use the option --delete. The option --delete deletes files in the target directory if they were deleted in the source directory.

Using SSHFS on Linux or Mac OS

SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.

Installation

SSHFS is available on many linux distribution via the standard software distribution channels.

On Debian/Ubuntu like systems:

$ apt-get install sshfs

On RedHat/CentOS like systems:

$ yum install fuse-sshfs

Mounting a Remote File System Interactively (as normal user)

To mount a remote file system via command line:

$ mkdir mountpoint
$ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects ./mointpoint

Mounting a Remote File System with /etc/fstab (as root)

Add (as root) a line to your /etc/fstab describing how to mount the remote folder:

sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects    <MOUNTPOINT>    fuse    uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0

where <UID is the numerical user id of the user the remote mount will be mapped to, <GID> is the numerical group id of the user the remote mount will be mapped to, <USERNAME> is the user name user for the SSH login, <MOUNTPOINT> is the local mount point (directory).

For example:

sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/scc/projects   /mnt/mountpoint    fuse    uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0

In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):

$ mount /mnt/mountpoint

Using Secure Shell with Windows

Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):

MobaXterm	https://mobaxterm.mobatek.net/
PuTTY	https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
WinSCP	http://winscp.net/eng/download.php
FileZilla	http://sourceforge.net/projects/filezilla/
WebDrive	https://southrivertech.com/products/webdrive/download/
Cygwin	http://cygwin.com/install.html
Windows 10 subsystem for Linux	https://msdn.microsoft.com/de-de/commandline/wsl/install_guide

The following software packages provide network drive functionality based on SFTP:

WebDrive (for Windows, Mac, iOS, Android)	http://www.southrivertechnologies.com/download/downloadwd.html
SFTPNetDrive (for Windows)	http://www.sftpnetdrive.com/
NetDrive (for Windows)	http://www.netdrive.net/
ExpanDrive (for Windows and Mac)	http://www.expandrive.com/expandrive
MountainDuck (for Windows and Mac)	https://mountainduck.io/

External Links

• OpenSSH Manual Pages
• Secure Shell on Wikipedia
• SSH File Transfer Protocol on Wikipedia
• Secure Shell Filesystem on Wikipedia