LSDF Online Storage: rsync
Using rsync from UNIX client via the login-cluster
rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amoun of data sent over the network by sending only the differences between the source files and the existing files in the destination. rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm by (default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the desti-nation file directly when the quick check indicates that the files data does not need to be updated.
Attention: rsync with -a (or --group) option write the data with wrong GID (from source).
Example:
#set sticky bit and group, if it is missing or wrong: >ssh xy1234@os-login.lsdf.kit.edu >chmod -g+s /lsdf_path/ >chgrp lsdf-project-group /lsdf_path/ #rsync via ssh (workaround with '-a --no-g --chmod=Dg+s') >rsync -uva --no-g --chmod=Dg+s --rsh="ssh -c es128-gcm@openssh.com -p 22" /source_path/source_dir/ xy1234@os-login.lsdf.kit.edu:/lsdf_path/target_dir/ 1>/tmp/xy1234.msg 2>/tmp/xy1234.err xy1234@os-login.lsdf.kit.edu's password:
To reach maximum bandwidth we recommend ssh ciphers arcfour, aes128-gcm@openssh.com or aes128-cbc algorithm for the encryption of the transmission.
As during all transfers errors might occur, we recommend controlling the correct transfer by checking the return value and rerun the transfer if it is not zero.
> echo $?
If you continue to work in bwgrid and delete data there you can use the option ‚--delete‘. With this your copy in LSDF Online Storage will be kept up to date. The option ‚--delete‘ deletes files in the target directory if they were deleted in the source directory.
