Difference between revisions of "LSDF Online Storage: WebDAV"

From Lsdf
(27 intermediate revisions by 2 users not shown)
Line 4: Line 4:
   
   
== Prerequisites ==
+
== Prerequisites and Restrictions ==
   
 
Please note the following requirements and restrictions when using the LSDF WebDAV service:
 
Please note the following requirements and restrictions when using the LSDF WebDAV service:
   
* The WebDAV protocol in general does suport the listing of special files like symbolic links. I.e. symbolic links links will not be listed with most of the tools described on this page. Nevertheless it is possible to access the content behind symbolic links. Therefor, it is recommended to access project directories directly, e.g. when mounting them ni the file system.
+
The WebDAV protocol in general does suport the listing of special files like symbolic links, i.e. symbolic links present in the LSDF file space will not be listed with most of the tools described on this page. Nevertheless it is still possible to access the content behind symbolic links. Therefore, it is recommended to access project directories directly, e.g. when mounting them in the file system or mapping them as network drives.
   
   
Line 19: Line 19:
 
Project directories are accessible at the usual location:
 
Project directories are accessible at the usual location:
 
<pre>
 
<pre>
https://os-webdav.lsdf.kit.edu/kit/<OE>/projects/<PROJECTNAME>
+
https://os-webdav.lsdf.kit.edu/<OE>/<inst>/projects/<PROJECTNAME>/
 
</pre>
 
</pre>
  +
where
where ''<OE>'' is the name of your organizational unit, e.g. scc, and
 
  +
''<OE>'' is the name of your organizational unit, e.g. kit, bs, fiz ...;
  +
"<inst>" is the name of your institution, e.g. scc, iai, ...;
 
''<PROJECTNAME>'' ist the name of the project.
 
''<PROJECTNAME>'' ist the name of the project.
 
Your home directory is accessible at the following URL:
 
Your home directory is accessible at the following URL:
 
<pre>
 
<pre>
https://os-webdav.lsdf.kit.edu/kit/inst/<OE>/<USERNAME>/
+
https://os-webdav.lsdf.kit.edu/<OE>/<inst>/<USERNAME>/
 
</pre>
 
</pre>
where ''<USERNAME>'' is you KIT username.
+
where ''<USERNAME>'' is you username: xy1234, bs_abcd,....
   
   
   
== Using WebDAV on Linux ==
+
== Using the DaviX Command Line Client on Linux and Mac OS ==
 
 
=== Using DaviX in the Command Line ===
 
   
 
The DaviX project provides a library and a set of client tools for file management over
 
The DaviX project provides a library and a set of client tools for file management over
Line 47: Line 46:
   
   
==== Listing a Directory ====
+
=== Listing a Directory ===
   
 
The content of a directory or individual files can be listed with the following command:
 
The content of a directory or individual files can be listed with the following command:
Line 58: Line 57:
 
</pre>
 
</pre>
   
==== Copying a File ====
 
   
  +
To copy a file from the WebDAV share to you local directory:
 
  +
=== Copying Files ===
  +
  +
To copy a file (e.g. testfile) from the WebDAV share to you local directory:
 
<pre>
 
<pre>
 
$ davix-get davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/testfile testfile
 
$ davix-get davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/testfile testfile
Line 70: Line 71:
 
</pre>
 
</pre>
   
==== Recursively Uploading Files ====
 
   
  +
Uploading files recursively with 6 transfers in parallel:
 
  +
=== Recursively Uploading Files ===
  +
  +
The following example shows how to recursively upload files with 6 transfers in parallel:
 
<pre>
 
<pre>
 
$ davix-put -r6 a_directory davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/a_new_directory
 
$ davix-put -r6 a_directory davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/a_new_directory
Line 81: Line 84:
 
</pre>
 
</pre>
   
==== Recursively downloading files ====
 
   
  +
Files and directories can be downloaded recursively
 
  +
=== Recursively downloading files ===
  +
  +
Files and directories can also be downloaded recursively. In the following example a directory tree will be downloaded recursively with 6 transfers in parallel:
 
<pre>
 
<pre>
 
$ davix-get -r6 davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/a_new_directory downloaded_dir
 
$ davix-get -r6 davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/a_new_directory downloaded_dir
Line 94: Line 99:
   
   
  +
== Connection with via File Manager on Linux ==
=== Mounting a WebDAV Share ===
 
   
  +
A number of different linux file managers support WebDAV shares natively. Among others these are
davfs2 is a file system driver that allows mounting a WebDAV server as a local file system, like a disk drive. This way, applications can access resources on a WebDAV server without knowing anything about HTTP or WebDAV. davfs2 does extensive caching to make the file system responsive, to avoid unnecessary network traffic and to prevent data loss, and to cope for slow or unreliable connections.
 
   
  +
* '''Nautilus''' (Gnome)
  +
* '''Dolphin/Konqueror''' (KDE / Kubuntu)
  +
* '''Thunar''' (Xfce / Xubuntu)
   
  +
To mount a LSDF WebDAV share (e.g. a project directory) with Nautilus follow the steps outlined below:
==== Installation ====
 
   
  +
# Open ''Nautilus''.
On CentOS or RedHat like systems install the package davfs2 which is available in the epel repository:
 
  +
# Click ''+ Other Location'' on the left hand pane.
$ yum install -y epel-release
 
  +
# In the text box ''Connect to Server'' enter the URL to your project folder <nowiki>davs://os-webdav.lsdf.kit.edu/<OE>/<inst>/projects/<PROJECT></nowiki>, where ''<OE>'' is you organizational unit and ''<PROJECT>'' is the name of your project directory, e.g. <nowiki>davs://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject</nowiki>
$ yum install davfs2
 
  +
# Click ''Connect''
   
  +
The share will be mounted on the local system and accessible both in Nautilus and from the directory ''/run/user/<UID>/gvfs'', where ''<UID>'' is the id of your user account.
   
==== Mounting a Share (as root) ====
 
   
To mount an LSDF WebDAV share issue the following command as root:
 
 
$ mount.davfs https://os-webdav.lsdf.kit.edu/ /mnt/webdav >/dev/null 2>&1
 
Please enter the username to authenticate with server
 
https://os-webdav.lsdf.kit.edu/ or hit enter for none.
 
Username: xy1234
 
Please enter the password to authenticate user fg9068 with server
 
https://os-webdav.lsdf.kit.edu/ or hit enter for none.
 
Password:
 
   
  +
== Mounting a WebDAV Share on Linux with davfs2 ==
or
 
   
  +
davfs2 is a file system driver that allows mounting a WebDAV server as a local file system, like a disk drive. This way, applications can access resources on a WebDAV server without knowing anything about HTTP or WebDAV. davfs2 does extensive caching to make the file system responsive, to avoid unnecessary network traffic and to prevent data loss, and to cope for slow or unreliable connections.
$ mount.davfs -o username=xy1234,uid=xxx,gid=yyy,rw https://os-webdav.lsdf.kit.edu/inst/projects /mnt/webdav
 
   
  +
davfs2 has a system wide configuration file ''/etc/davfs2/davfs2.conf'' and a user configuration file ''~/.davfs2/davfs2.conf''. The user configuration file is used in addition to the system configuration file when invoked by a normal user. Documentation on all options can be found in the [https://linux.die.net/man/5/davfs2.conf manual page].
   
  +
'''Mount WebDAV share using fstab'''
 
  +
  +
=== Installation ===
  +
  +
On CentOS or RedHat like systems install the package davfs2 which is available in the epel repository:
 
<pre>
 
<pre>
  +
$ yum install -y epel-release
>cat /etc/fstab | grep webdav
 
  +
$ yum install davfs2
https://os-webdav.lsdf.kit.edu/kit/inst/projects /mnt/webdav davfs rw,noauto,user,uid=xy1234 0 0
 
  +
</pre>
   
>mkdir /mnt/webdav
 
   
>cat /etc/davfs2/secrets | grep os-webdav
 
https://os-webdav.lsdf.kit.edu/kit/inst/projects xy1234 ***
 
   
  +
=== Mounting (as root) ===
>mount /mnt/webdav >/dev/null 2>&1
 
  +
  +
To mount an LSDF WebDAV project or user share issue the following command as root:
  +
<pre>
  +
$ mount.davfs https://os-webdav.lsdf.kit.edu/<OE>/<inst>/projects/<PROJECTNAME> <MOUNTPOINT>
  +
</pre>
  +
where ''<OE>'' is the name of your organizational unit, e.g. kit, ''<PROJECTNAME>'' is the name of the project, and ''<MOUNTPOINT>'' is a directory. For example:
  +
<pre>
  +
$ mkdir /mnt/my_lsdf_project
  +
$ mount.davfs https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project
  +
Please enter the username to authenticate with server
  +
https://os-webdav.lsdf.kit.edu/ or hit enter for none.
  +
Username: xy1234
  +
Please enter the password to authenticate user fg9068 with server
  +
https://os-webdav.lsdf.kit.edu/ or hit enter for none.
  +
Password:
 
</pre>
 
</pre>
   
  +
As an alternative it is possible to preconfigure a WebDAV mount. For the project mount described in the example above you would have to add the following line to the file ''/etc/fstab'':
'''WebDAV umount:'''
 
 
<pre>
 
<pre>
  +
https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project davfs rw,noauto,user 0 0
>umount.davfs /mnt/webdav
 
 
</pre>
 
</pre>
  +
User authentication information needs to be stored in the file ''/etc/davfs2/secrets'':
 
<H3>Mount WebDAV share as user</H3>
 
 
'''davfs2''' makes mounting by unprivileged users as easy and secure as possible.
 
 
User must be a member of the davfs2 group:
 
 
<pre>
 
<pre>
  +
$ cat /etc/davfs2/secrets | grep os-webdav
>usermod -a -G davfs2 xy1234
 
  +
https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project xy1234 ***
>grep davfs2 /etc/group
 
  +
</pre>
  +
Afterwards create the mount point directory and mount the share with the command
  +
$ mount /mnt/my_lsdf_project
 
</pre>
 
</pre>
   
'''Example:'''
 
<pre>
 
>cat /etc/fstab | grep webdav
 
https://os-webdav.lsdf.kit.edu/kit/inst/projects /mnt/webdav davfs rw,noauto,user,uid=xy1234 0 0
 
   
>chmod 600 /lsdf/kit/inst/xy1234/.davfs2/secrets
 
   
  +
=== Unmounting (as root) ===
>cat /lsdf/kit/inst/xy1234/.davfs2/secrets | grep os-webdav
 
https://os-webdav.lsdf.kit.edu/kit/inst/projects xy1234 ***
 
   
  +
To unmount a mounted WebDAV share issue the following command:
>mount /mnt/webdav >/dev/null 2>&1
 
  +
<pre>
 
>umount /mnt/webdav
+
$ umount.davfs /mnt/my_lsdf_project
umount: /mnt/webdav: umount failed: Operation not permitted
 
 
</pre>
 
</pre>
   
You can mount the above-mentioned WebDAV share using '''sudo''' utility.
 
   
=== WebDAV client cadaver ===
 
   
  +
=== Mouting (as normal user) ===
'''cadaver''' is a command-line WebDAV client, with support for file upload, download, on-screen display, in-place editing, namespace operations (move/copy), collection creation and deletion, property manipulation, and resource locking.
 
   
  +
To mount a share as an unprivileged user, the user account needs to be a member of the group ''davfs2'':
'''Installation:'''
 
 
<pre>
 
<pre>
  +
$ usermod -a -G davfs2 xy1234
yum install cadaver
 
 
</pre>
 
</pre>
  +
The mount has to be defined in the file ''/etc/fstab''. To mount a project directory add for example the following line to the file:
 
'''Example:'''
 
 
<pre>
 
<pre>
cadaver https://os-webdav.lsdf.kit.edu/kit/inst/xy1234
+
https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project davfs rw,noauto,user 0 0
  +
</pre>
dav:/kit/inst/xy1234/> help
 
  +
User specific authentication information can be stored file '~/'.davfs2/secrets'':
Available commands:
 
  +
<pre>
Available commands:
 
  +
$ mkdir ~/.davfs2/secrets
ls cd pwd put get mget mput
 
  +
$ chmod 600 ~/.davfs2/secrets
edit less mkcol cat delete rmcol copy
 
  +
$ cat ~/.davfs2/secrets
move lock unlock discover steal showlocks version
 
  +
https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project xy1234 ***
checkin checkout uncheckout history label propnames chexec
 
  +
</pre>
propget propdel propset search set open close
 
  +
To mount the share issue the command
echo quit unset lcd lls lpwd logout
 
  +
<pre>
help describe about
 
  +
$ mount /mnt/my_lsdf_project
Aliases: rm=delete, mkdir=mkcol, mv=move, cp=copy, more=less, quit=exit=bye
 
 
</pre>
 
</pre>
   
=== Connection via File Manager ===
+
== Using WebDAV on Windows ==
   
  +
To mount a LSDF WebDAV share (e.g. a project directory) and map it to a drive in windows follow the steps outlined below:
* '''Nautilus''' (Gnome)
 
* '''Kubuntu (Dolphin/Konqueror)''' (KDE)
 
* '''Xubuntu (Thunar)''' (Xfce)
 
* '''Lubuntu (PCManFM)'''
 
*
 
   
  +
# Open a ''File Explorer'' and select ''This PC'' on the left hand pane.
  +
# In the Menu select ''Computer => Map Network Drive''.
  +
# Click ''Connect to a Web site that you can use to store your documents and pictures.''
  +
# Click ''Next''
  +
# Click ''Choose a custom network location'' and afterwards click ''Next''.
  +
# Enter the URL to you project folder <nowiki>https://os-webdav.lsdf.kit.edu/kit/<OE>/projects/<PROJECT></nowiki>, where ''<OE>'' is you organizational unit and ''<PROJECT>'' is the name of your project directory, e.g. <nowiki>https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject</nowiki>
  +
# Click Next
  +
# Enter a name for the network location and click ''Next''
  +
# Click ''Finish''
   
   
   
== Using WebDAV on Windows ==
+
== Public Shares ==
   
  +
Scientific data in LSDF storage projects can be shared publicly with HTTPS/WebDAV. Public projects can be reached under the URL https://os-webdav.lsdf.kit.edu/public. Please [https://bw-support.scc.kit.edu/?mode=ticket_submit&respounit=LSDF%20Online%20Storage contact the LSDF support] to publish an existing storage project.
=== Add network address ===
 
   
'''Example:'''
 
Computer => Add network address => Continue => https://os-webdab.lsdf.kit.edu/kit/inst/projects
 
   
=== Network drive over WebDAV ===
 
   
  +
== External Links ==
'''Examples:'''
 
<pre>
 
>net use * https://os-webdav.lsdf.kit.edu/kit/inst/projects /user:xy1234
 
Laufwerk W: ist jetzt mit https://os-webdav.lsdf.kit.edu/ verbunden.
 
Der Befehl wurde erfolgreich ausgeführt.
 
   
  +
* [http://savannah.nongnu.org/projects/davfs2 davfs2 project home page]
or
 
  +
* [https://dmc.web.cern.ch/projects/davix/home DaviX project home page]
 
>net use * \\os-webdav.lsdf.kit.edu@SSL\ /user:xy1234
 
...
 
</pre>
 
   
   
[[Category:LSDF_Online_Storage]]
+
[[Category:LSDF_Online_Storage|WebDAV]]

Revision as of 11:57, 19 December 2019

WebDAV or Web-based Distributed Authoring and Versioning is an extension of the Hypertext Transfer Protocol (HTTP) allowing to access files over the internet. WebDAV access to the LSDF Online Storage is provided via a cluster of servers reachable at the URL https://os-webdav.lsdf.kit.edu or davs://os-webdav.lsdf.kit.edu depending on the client used.



Prerequisites and Restrictions

Please note the following requirements and restrictions when using the LSDF WebDAV service:

The WebDAV protocol in general does suport the listing of special files like symbolic links, i.e. symbolic links present in the LSDF file space will not be listed with most of the tools described on this page. Nevertheless it is still possible to access the content behind symbolic links. Therefore, it is recommended to access project directories directly, e.g. when mounting them in the file system or mapping them as network drives.


Using WebDAV from a Web Browser

To access the starting page enter the following URL in your web browser

https://os-webdav.lsdf.kit.edu/

Project directories are accessible at the usual location:

https://os-webdav.lsdf.kit.edu/<OE>/<inst>/projects/<PROJECTNAME>/

where <OE> is the name of your organizational unit, e.g. kit, bs, fiz ...; "<inst>" is the name of your institution, e.g. scc, iai, ...; <PROJECTNAME> ist the name of the project. Your home directory is accessible at the following URL:

https://os-webdav.lsdf.kit.edu/<OE>/<inst>/<USERNAME>/

where <USERNAME> is you username: xy1234, bs_abcd,....


Using the DaviX Command Line Client on Linux and Mac OS

The DaviX project provides a library and a set of client tools for file management over HTTP-based protocols. The command line client tools can be either installed as a regular package in you linux distribution, e.g. in RedHat based distributions with

$ yum install -y epel-release
$ yum install davix

or from the home page of the DaviX project.


Listing a Directory

The content of a directory or individual files can be listed with the following command:

$ davix-ls -l davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234
Basic authentication - server is asking for username and password:
Login: xy1234
Password: 
-rwxrwxrwx 0     10485760   2019-05-20 09:29:18 a_small_file


Copying Files

To copy a file (e.g. testfile) from the WebDAV share to you local directory:

$ davix-get davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/testfile testfile
Basic authentication - server is asking for username and password:
Login: xy1234
Password: 
Performing Read operation on: davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/testfile
[==================================] 100%     10MiB/10MiB         0B/s                     


Recursively Uploading Files

The following example shows how to recursively upload files with 6 transfers in parallel:

$ davix-put -r6  a_directory davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/a_new_directory
Basic authentication - server is asking for username and password:
Login: xy1234
Password: 
Populating task queue for a_directory/      Files processed:          0                     


Recursively downloading files

Files and directories can also be downloaded recursively. In the following example a directory tree will be downloaded recursively with 6 transfers in parallel:

$ davix-get -r6 davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/a_new_directory downloaded_dir
Basic authentication - server is asking for username and password:
Login: xy1234
Password: 
Crawling davs://os-webdav.lsdf.kit.edu/kit/scc/xy1234/a_new_directory/      Files processed:          1    


Connection with via File Manager on Linux

A number of different linux file managers support WebDAV shares natively. Among others these are

  • Nautilus (Gnome)
  • Dolphin/Konqueror (KDE / Kubuntu)
  • Thunar (Xfce / Xubuntu)

To mount a LSDF WebDAV share (e.g. a project directory) with Nautilus follow the steps outlined below:

  1. Open Nautilus.
  2. Click + Other Location on the left hand pane.
  3. In the text box Connect to Server enter the URL to your project folder davs://os-webdav.lsdf.kit.edu/<OE>/<inst>/projects/<PROJECT>, where <OE> is you organizational unit and <PROJECT> is the name of your project directory, e.g. davs://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject
  4. Click Connect

The share will be mounted on the local system and accessible both in Nautilus and from the directory /run/user/<UID>/gvfs, where <UID> is the id of your user account.


Mounting a WebDAV Share on Linux with davfs2

davfs2 is a file system driver that allows mounting a WebDAV server as a local file system, like a disk drive. This way, applications can access resources on a WebDAV server without knowing anything about HTTP or WebDAV. davfs2 does extensive caching to make the file system responsive, to avoid unnecessary network traffic and to prevent data loss, and to cope for slow or unreliable connections.

davfs2 has a system wide configuration file /etc/davfs2/davfs2.conf and a user configuration file ~/.davfs2/davfs2.conf. The user configuration file is used in addition to the system configuration file when invoked by a normal user. Documentation on all options can be found in the manual page.


Installation

On CentOS or RedHat like systems install the package davfs2 which is available in the epel repository:

$ yum install -y epel-release
$ yum install davfs2


Mounting (as root)

To mount an LSDF WebDAV project or user share issue the following command as root:

$ mount.davfs https://os-webdav.lsdf.kit.edu/<OE>/<inst>/projects/<PROJECTNAME> <MOUNTPOINT>

where <OE> is the name of your organizational unit, e.g. kit, <PROJECTNAME> is the name of the project, and <MOUNTPOINT> is a directory. For example:

$ mkdir /mnt/my_lsdf_project
$ mount.davfs https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project
Please enter the username to authenticate with server
https://os-webdav.lsdf.kit.edu/ or hit enter for none.
 Username: xy1234
Please enter the password to authenticate user fg9068 with server
https://os-webdav.lsdf.kit.edu/ or hit enter for none.
 Password:  

As an alternative it is possible to preconfigure a WebDAV mount. For the project mount described in the example above you would have to add the following line to the file /etc/fstab:

https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project davfs rw,noauto,user 0 0

User authentication information needs to be stored in the file /etc/davfs2/secrets:

$ cat /etc/davfs2/secrets | grep os-webdav
https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project    xy1234     ***

Afterwards create the mount point directory and mount the share with the command $ mount /mnt/my_lsdf_project


Unmounting (as root)

To unmount a mounted WebDAV share issue the following command:

$ umount.davfs  /mnt/my_lsdf_project


Mouting (as normal user)

To mount a share as an unprivileged user, the user account needs to be a member of the group davfs2:

$ usermod -a -G davfs2 xy1234

The mount has to be defined in the file /etc/fstab. To mount a project directory add for example the following line to the file:

https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project davfs rw,noauto,user 0 0

User specific authentication information can be stored file '~/'.davfs2/secrets:

$ mkdir ~/.davfs2/secrets
$ chmod 600 ~/.davfs2/secrets
$ cat ~/.davfs2/secrets
https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject /mnt/my_lsdf_project    xy1234     ***

To mount the share issue the command

$ mount /mnt/my_lsdf_project

Using WebDAV on Windows

To mount a LSDF WebDAV share (e.g. a project directory) and map it to a drive in windows follow the steps outlined below:

  1. Open a File Explorer and select This PC on the left hand pane.
  2. In the Menu select Computer => Map Network Drive.
  3. Click Connect to a Web site that you can use to store your documents and pictures.
  4. Click Next
  5. Click Choose a custom network location and afterwards click Next.
  6. Enter the URL to you project folder https://os-webdav.lsdf.kit.edu/kit/<OE>/projects/<PROJECT>, where <OE> is you organizational unit and <PROJECT> is the name of your project directory, e.g. https://os-webdav.lsdf.kit.edu/kit/scc/projects/myproject
  7. Click Next
  8. Enter a name for the network location and click Next
  9. Click Finish


Public Shares

Scientific data in LSDF storage projects can be shared publicly with HTTPS/WebDAV. Public projects can be reached under the URL https://os-webdav.lsdf.kit.edu/public. Please contact the LSDF support to publish an existing storage project.


External Links