LSDF Online Storage: Secure Shell
Secure Shell or SSH refers to both a cryptographic network protocol as well a number of applications that can be used to establish a secure connect with a service provider like the LSDF Online Storage. SSH is typically used to login interactively to a remote machine and execute commands, but it can also be used to transfer file using the protocols SSH File Transfer Protocol (SFTP) or Secure Copy (SCP). The SSH Filesystem (SSHFS) is a file system client based on the SSH protocol which allows to locally mount and interact with directories and files located on a remote server.
SSH access to the LSDF Online Storage is provided via a cluster of login servers reachable at the address os-login.lsdf.kit.edu.
The login cluster are directly accessible by end users.
These nodes are used for interactive login, file management, data retrieving and restoring.
Three nodes are dedicated to this service but they are all accessible via one host name and a DNS round-robin alias distributes the login sessions to the different login nodes.
Environment Variables:
$SHELL=/bin/bash $USER=xy1234 $LSDF=/lsdf $HOME=/lsdf/kit/inst/xy1234 $LSDFPROJECTS=/lsdf/kit/inst/projects $SNAPSHOTS=/lsdf/kit/snapshots $PROJECTSNAPSHOTS=/lsdf/kit/inst/projects/.snapshots
Using SSH on Linux or Mac OS
To interactively login to the LSDF Online Storage login cluster enter the following command in a linux shell or Mac OS terminal
$ ssh <USERNAME>@os-login.lsdf.kit.edu
where <USERNAME> needs to be replaced with your actual user name, e.g. xy1234. The command above will ask for a password to grant access to the login server.
To avoid being asked for a password on every login, password-less access can be configured by distributing a public key to the login server. To create a new set of keys issue the following command:
$ ssh-keygen
The newly generated key should be password protected.
To copy the public part of your key pair to the login server issue the following command:
$ ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@os-login.lsdf.kit.edu
Subsequent logins will use the key instead of a password to authenticate to the login server.
Further details on the usage of the SFTP command line client can be found on the man page.
Using SCP on Linux or Mac OS
SCP is a protocol that allows secure data transfer between a local and a remote computer or between two remote computers. To transfer the data to or from the LSDF Online Storage, you can execute the following commands:
$ scp <LOCAL PATH>/<FILES(S)> <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH>/ or $ scp <USERNAME>@os-login.lsdf.kit.edu:<REMOTE PATH> <LOCAL PATH>/<FILES(S)>
Please be aware that symbolic links will usually not be copied as links.
Examples:
>scp -c arcfour ~/filename xy1234@os-login.lsdf.kit.edu:~/ or >scp -c arcfour xy1234@os-login.lsdf.kit.edu:~/filename ~/
To reach maximum bandwidth we recommend ssh ciphers arcfour, aes128-gcm@openssh.com or aes128-cbc algorithm for the encryption of the transmission.
Further details on the usage of the SCP command line client can be found on the man page.
Using SFTP on Linux or Mac OS
The following example illustrates the usage of SFTP in a linux shell:
$ sftp xy1234@os-login.lsdf.kit.edu Connecting to os-login.lsdf.kit.edu<br> xy1234@os-login.lsdf.kit.edu's password: sftp> ls snapshots temp test sftp> help ... sftp> put myfile sftp> get myfile
Further details on the usage of the SFTP command line client can be found on the man page.
Using rsync on Linux or Mac OS
rsync is a fast and versatile file copying tool. It can copy locally, to or from another host over any remote shell, or to or from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does not need to be updated.
Attention: rsync with -a (or --group) option write the data with wrong GID (from source).
The following example synchronizes the local directory /user_path/user_dir/ with the LSDF directory /lsdf_path/target_dir/ and sets the correct owning group in the LSDF:
#In the LSDF: set sticky bit and group, if it is missing or wrong: $ chmod -g+s /lsdf_path/ $ chgrp lsdf-project-group /lsdf_path/ #On the client host: rsync via ssh (workaround with '-a --no-g --chmod=Dg+s') $ rsync -uva --no-g --chmod=Dg+s --rsh="ssh -c es128-gcm@openssh.com -p 22" /user_path/user_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err xy1234@os-login.lsdf.kit.edu's password:
To reach maximum bandwidth we recommend ssh ciphers arcfour, aes128-gcm@openssh.com or aes128-cbc algorithm for the encryption of the transmission.
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.
$ echo $?
If you continue to work and delete data you can use the option --delete. The option --delete deletes files in the target directory if they were deleted in the source directory.
Using SSHFS on Linux or Mac OS
SSHFS is a file system implementation allowing users to mount and interact with directories and files located on a remote server over a normal ssh connection. Using SSHFS a mounted remote file system behaves similar to other local volumes or files systems. In situations where higher performance and / or simultaneous access to the LSDF Online Storage by many users are required other protocols like NFS, CIFS or WebDAV might be better suited.
Installation
SSHFS is available on many linux distribution via the standard software distribution channels.
On Debian/Ubuntu like systems:
$ apt-get install sshfs
On RedHat/CentOS like systems:
$ yum install fuse-sshfs
Mounting a Remote File System Interactively (as normal user)
To mount a remote file system via command line:
$ mkdir mountpoint $ sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ ./mointpoint
Mounting a Remote File System with /etc/fstab (as root)
Add (as root) a line to your /etc/fstab describing how to mount the remote folder:
sshfs <USERNAME>@os-login.lsdf.kit.edu:/lsdf/kit/<INSTITUTE>/projects/ <MOUNTPOINT> fuse uid=<UID>,gid=<GID>,umask=0,allow_other,defaults,auto 0 0
where <UID is the numerical user id of the user the remote mount will be mapped to, <GID> is the numerical group id of the user the remote mount will be mapped to, <USERNAME> is the user name user for the SSH login, <MOUNTPOINT> is the local mount point (directory).
For example:
sshfs xy1234@os-login.lsdf.kit.edu:/lsdf/kit/inst/projects/ /mnt/mountpoint fuse uid=7777,gid=12345,umask=0,allow_other,defaults,auto 0 0
In order to make the above example work, password-less ssh login with keys needs to be properly configured. Mount the remote file system by issuing the following command (as root):
$ mount /mnt/mountpoint
Using Secure Shell with Windows
Please find below a list of Secure Shell clients for Windows and MacOS (without any claim to completeness):
| MobaXterm | https://mobaxterm.mobatek.net/ |
| PuTTY | https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html |
| WinSCP | http://winscp.net/eng/download.php |
| FileZilla | http://sourceforge.net/projects/filezilla/ |
| WebDrive | https://southrivertech.com/products/webdrive/download/ |
| Cygwin | http://cygwin.com/install.html |
| Windows 10 subsystem for Linux | https://msdn.microsoft.com/de-de/commandline/wsl/install_guide |
The following software packages provide network drive functionality based on SFTP:
| WebDrive (for Windows, Mac, iOS, Android) | http://www.southrivertechnologies.com/download/downloadwd.html |
| SFTPNetDrive (for Windows) | http://www.sftpnetdrive.com/ |
| NetDrive (for Windows) | http://www.netdrive.net/ |
| ExpanDrive (for Windows and Mac) | http://www.expandrive.com/expandrive |
| MountainDuck (for Windows and Mac) | https://mountainduck.io/ |
