Difference between revisions of "Category:LSDF Online Storage"

From Lsdf
(Add link to HDF AAI)
 
(29 intermediate revisions by 2 users not shown)
Line 1: Line 1:
  +
  +
[[File:lsdf_online_storage_services.png|right|top|frameless|250px]]
  +
  +
[[File:Lsdf_online_storage_services_2.jpg|right|top|thumb|250px|LSDF Online Storage @ KIT (SCC)]]
   
 
The LSDF is a central storage system for large research data from data-intensive scientific disciplines. It is available to KIT researchers, their partners in the federal state and the Helmholtz Association as a multi-petabyte storage system. The service LSDF Online Storage enables KIT users and their partners in the state of Baden-Württemberg to access the storage resources of the LSDF. The service is used by institutes with research funded by the Helmholtz Association as well as insitutes at the university level of the KIT from different scientific disciplines. The project is sponsored and funded by the Ministry of Science, Research and the Arts of Baden-Württemberg (MWK), the German Research Foundation (DFG) and the Helmholtz Association of German Research Centres. It is operated by the Steinbuch Centre for Computing (SCC) at KIT.
 
The LSDF is a central storage system for large research data from data-intensive scientific disciplines. It is available to KIT researchers, their partners in the federal state and the Helmholtz Association as a multi-petabyte storage system. The service LSDF Online Storage enables KIT users and their partners in the state of Baden-Württemberg to access the storage resources of the LSDF. The service is used by institutes with research funded by the Helmholtz Association as well as insitutes at the university level of the KIT from different scientific disciplines. The project is sponsored and funded by the Ministry of Science, Research and the Arts of Baden-Württemberg (MWK), the German Research Foundation (DFG) and the Helmholtz Association of German Research Centres. It is operated by the Steinbuch Centre for Computing (SCC) at KIT.
  +
  +
The LSDF enables scientists to store and access their research data via different services. These services, tailored to the requirements of the researchers, are offered on top of the LSDF storage. Important features are in particular high storage capacity, fast access to and transfer of the data, easy data access and management capabilities, as well as the connection to modern processing infrastructures such as the high-performance computing clusters based at KIT like the bwUniCluster or the ForHLR II.
   
 
Access to the data is possible via standard protocols. Permissions to access the data are controlled by the data owners using group memberships and access control lists (ACLs). The LSDF Online Storage currently offers 10PB of usable storage capacity with a combined maximum read/write speed of the storage system of 25GByte/s. Today it is used by about 400 scientists.
 
Access to the data is possible via standard protocols. Permissions to access the data are controlled by the data owners using group memberships and access control lists (ACLs). The LSDF Online Storage currently offers 10PB of usable storage capacity with a combined maximum read/write speed of the storage system of 25GByte/s. Today it is used by about 400 scientists.
Line 6: Line 12:
 
In order to use the LSDF Online Storage Service, interested users must first register for the service. Registration takes place via [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf bwIDM web interface]. Registration is only possible if a special Entitlement has been assigned to the user previously. Users usually receive this Entitlement after consultation with their responsible ITB or group representative or, if no project has yet been set up, by a separate request to the service operator.
 
In order to use the LSDF Online Storage Service, interested users must first register for the service. Registration takes place via [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf bwIDM web interface]. Registration is only possible if a special Entitlement has been assigned to the user previously. Users usually receive this Entitlement after consultation with their responsible ITB or group representative or, if no project has yet been set up, by a separate request to the service operator.
   
If you have any questions regarding registration, requesting a new storage project or further questions, please contact the service provider via the [https://bw-support.scc.kit.edu/ BW support portal].
+
If you have any questions regarding registration, requesting a new storage project or further questions, please contact the service provider via the [https://bw-support.scc.kit.edu/?mode=ticket_submit&respounit=LSDF%20Online%20Storage BW support portal].
   
 
The setup and size of the available storage space is determined by agreements between the users and the service provider and technically realized by quotas. For new projects a Data Management Plan (DMP) is developed and stored in coordination with the Institute of the Users and the representatives of the service provider. The DMPs are used for reporting and applying for funding to operate, maintain and expand the LSDF. The storage space of the LSDF Online Storage is currently usually available free of charge.
 
The setup and size of the available storage space is determined by agreements between the users and the service provider and technically realized by quotas. For new projects a Data Management Plan (DMP) is developed and stored in coordination with the Institute of the Users and the representatives of the service provider. The DMPs are used for reporting and applying for funding to operate, maintain and expand the LSDF. The storage space of the LSDF Online Storage is currently usually available free of charge.
Line 12: Line 18:
 
Further information on the range of services and the usage of the service can be found in this Wiki the [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf user manual] and the current [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf terms of use].
 
Further information on the range of services and the usage of the service can be found in this Wiki the [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf user manual] and the current [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf terms of use].
   
  +
{| style="width: 100%; border-spacing: 5px;"
 
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |
 
[[File:LSDF_Online_Storage.jpg|center|border|700px|LSDF Online Storage KIT (SCC)]]
 
|-
 
| style="text-align:center; color:#000;vertical-align:middle;" |<span style="font-size:100%">'''LSDF Online Storage © KIT (SCC)''' </span>
 
|}
 
 
{| style="width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;"
 
{| style="width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;"
 
<!-- ACCESS -->
 
<!-- ACCESS -->
 
| style="width:50%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;" |
 
| style="width:50%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;" |
 
{| style="width:100%; vertical-align:top; background:#f5fffa;"
 
{| style="width:100%; vertical-align:top; background:#f5fffa;"
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Access</div>
+
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">LSDF Online Storage Projects</div>
 
|-
 
|-
 
| style="color:#000; padding:2px 5px;" |
 
| style="color:#000; padding:2px 5px;" |
* [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf LSDF Online Storage registration]
+
* [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf Signing up to exising storage projects] (for users)
  +
* [https://www.lsdf.kit.edu/os/storagerequest/new/ Requesting new storage projects] (for project admins)
|-
 
  +
* [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf Terms of use] (German version)
<!-- Storage Request -->
 
  +
* [[LSDF_Online_Storage:_Acknowledgements|Acknowledgements]]
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Storage Request</div>
 
|-
 
| style="color:#000; padding:2px 5px;" |
 
* [https://www.lsdf.kit.edu/os/storagerequest/ Storage Request] (for project admins)
 
 
|-
 
|-
 
<!-- Documentation -->
 
<!-- Documentation -->
Line 37: Line 35:
 
|-
 
|-
 
| style="color:#000; padding:2px 5px;" |
 
| style="color:#000; padding:2px 5px;" |
* [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf terms of use] (German version)
+
* [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf User manual] (German version)
* [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf user manual] (German version)
 
 
|-
 
|-
 
<!-- Support -->
 
<!-- Support -->
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Support/Ticket System</div>
+
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Getting Help</div>
 
|-
 
|-
 
| style="color:#000; padding:2px 5px;" |
 
| style="color:#000; padding:2px 5px;" |
* [https://bw-support.scc.kit.edu BW support portal]
+
* [https://bw-support.scc.kit.edu/?mode=ticket_submit&respounit=LSDF%20Online%20Storage Baden Württemberg Support Portal]
 
|-
 
|-
<!-- Web sites -->
+
<!-- Web sites
 
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Web sites</div>
 
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Web sites</div>
 
|-
 
|-
Line 55: Line 52:
 
* [http://www.scc.kit.edu/en/services/11228.php LSDF Online Storage] (English version)
 
* [http://www.scc.kit.edu/en/services/11228.php LSDF Online Storage] (English version)
 
|-
 
|-
  +
-->
 
<!-- Monitoring
 
<!-- Monitoring
 
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Monitoring</div>
 
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Monitoring</div>
Line 68: Line 66:
   
 
<!-- PROTOCOLS -->
 
<!-- PROTOCOLS -->
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Protocols</div>
+
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Accessing Data</div>
 
|-
 
|-
 
| style="color:#000; padding:2px 5px;" |
 
| style="color:#000; padding:2px 5px;" |
  +
* [[LSDF_Online_Storage:_Secure_Shell|Secure Shell (ssh, scp, sftp, sshfs)]]
'''for LSDF login cluster os-login.lsdf.kit.edu:'''
 
  +
* [[LSDF_Online_Storage:_WebDAV|WebDAV / HTTPS]]<br>
* [[LSDF_Online_Storage_SCP|SCP]], [[LSDF_Online_Storage_SFTP|SFTP]], [[LSDF_Online_Storage_SSH|SSH]], [[LSDF_Online_Storage_SSHFS|SSHFS]]
 
  +
* [[LSDF_Online_Storage:_SMB_/_CIFS|SMB / CIFS]] (KIT intern only)
<!--
 
  +
* [[LSDF_Online_Storage_NFS|NFS]] (KIT intern only)
* [[LSDF_Online_Storage_HTTPS|HTTPS]] (https://os.lsdf.kit.edu for downloads only) -->
 
  +
* [[LSDF_Online_Storage:_HPC|on the KIT HPC Clusters]] ([http://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster], [http://www.scc.kit.edu/dienste/forhlr.php ForHLR])
'''for LSDF cluster os.lsdf.kit.edu:'''
 
  +
* [[LSDF_Online_Storage:_HDF_AAI|using the AAI of the Helmholtz Data Fedaration (HDF)]]
* [[LSDF_Online_Storage_CIFS|SMB/CIFS]], [[LSDF_Online_Storage_NFS|NFS]] (KIT intern)
 
  +
|-
 
<!-- TOOLS -->
 
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Tools</div>
 
|-
 
| style="color:#000; padding:2px 5px;" |
 
'''for HPC Cluster:'''
 
<br>
 
* [[LSDF_Online_Storage_rdata|rdata]] (for [http://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster], [http://www.scc.kit.edu/dienste/forhlr.php ForHLR]) <!--[http://hpcmon.scc.kit.edu HPC-Auslastung] --> <br>
 
'''for SCP/SFTP/SSH protocol:'''
 
* [http://www.openssh.com/ OpenSSH]
 
*[http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty suite] (for Windows and Unix)
 
*[https://winscp.net/eng/download.php WinSCP] (for Windows)
 
*[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux)
 
*[http://cygwin.com/install.html Cygwin] (for Windows)
 
'''network drive over SFTP:'''
 
*[https://www.southrivertechnologies.com/download/downloadwd.html WebDrive] (for Windows, Mac, iOS, Android)
 
*[http://www.sftpnetdrive.com/ SFTPNetDrive] (for Windows)
 
*[http://www.netdrive.net/ NetDrive] (for Windows)
 
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac)
 
*[https://mountainduck.io MountainDuck] (for Windows and Mac)
 
'''network drive over SMB/CIFS:'''
 
*[https://www.bwhpc-c5.de/wiki/index.php/LSDF_Online_Storage_CIFS#Using_CIFS_for_WINDOWS_client__.28only_KIT_intern_or_KIT-VPN.29 Windows Explorer] (KIT intern)
 
'''for UNIX client:'''
 
* [[LSDF_Online_Storage_SCP|scp]], [[LSDF_Online_Storage_SFTP|sftp]], [[LSDF_Online_Storage_SSH|ssh]], [[LSDF_Online_Storage_SSHFS|sshfs mount]]
 
* [[LSDF_Online_Storage_CIFS#SMB_Client|smbclient]], [[LSDF_Online_Storage_CIFS#Mount_a_Share|cifs mount]] (KIT intern)
 
* [[LSDF_Online_Storage_rsync|rsync]], [[LSDF_Online_Storage_lftp|lftp]]
 
'''for WINDOWS client:'''
 
*[[LSDF_Online_Storage_RoboMirror|RoboMirror]] (for backup and restore)
 
 
|-
 
|-
 
<!-- SNAPSHOTS AND BACKUP -->
 
<!-- SNAPSHOTS AND BACKUP -->
Line 110: Line 81:
 
|-
 
|-
 
| style="color:#000; padding:2px 5px;" |
 
| style="color:#000; padding:2px 5px;" |
* [[LSDF_Online_Storage_Snapshots|Snapshots]] (for data retrieving and restoring)
+
* [[LSDF_Online_Storage:_Snapshots|Snapshots]] (for data retrieving and restoring)
 
* [[LSDF_Online_Storage_Backup|Backup]]
 
* [[LSDF_Online_Storage_Backup|Backup]]
 
|-
 
|-
Line 117: Line 88:
 
|-
 
|-
 
| style="color:#000; padding:2px 5px;" |
 
| style="color:#000; padding:2px 5px;" |
  +
* [[LSDF_Online_Storage_Filesystem|Filesystem characteristics]]
* [[LSDF_Online_Storage_Login_Cluster|Login Cluster]]
 
* [[LSDF_Online_Storage_Filesystem|Filesystem]]
 
 
* [[LSDF_Online_Storage_Quota|Quota]]
 
* [[LSDF_Online_Storage_Quota|Quota]]
<!-- * [[LSDF_Online_Storage_Price|Price]] -->
 
 
|}
 
|}
 
|}
 
|}
Line 128: Line 97:
 
<br>
 
<br>
 
<br>
 
<br>
  +
  +
[[Category:LSDF_Online_Storage]]

Latest revision as of 13:28, 2 March 2021

Lsdf online storage services.png
LSDF Online Storage @ KIT (SCC)

The LSDF is a central storage system for large research data from data-intensive scientific disciplines. It is available to KIT researchers, their partners in the federal state and the Helmholtz Association as a multi-petabyte storage system. The service LSDF Online Storage enables KIT users and their partners in the state of Baden-Württemberg to access the storage resources of the LSDF. The service is used by institutes with research funded by the Helmholtz Association as well as insitutes at the university level of the KIT from different scientific disciplines. The project is sponsored and funded by the Ministry of Science, Research and the Arts of Baden-Württemberg (MWK), the German Research Foundation (DFG) and the Helmholtz Association of German Research Centres. It is operated by the Steinbuch Centre for Computing (SCC) at KIT.

The LSDF enables scientists to store and access their research data via different services. These services, tailored to the requirements of the researchers, are offered on top of the LSDF storage. Important features are in particular high storage capacity, fast access to and transfer of the data, easy data access and management capabilities, as well as the connection to modern processing infrastructures such as the high-performance computing clusters based at KIT like the bwUniCluster or the ForHLR II.

Access to the data is possible via standard protocols. Permissions to access the data are controlled by the data owners using group memberships and access control lists (ACLs). The LSDF Online Storage currently offers 10PB of usable storage capacity with a combined maximum read/write speed of the storage system of 25GByte/s. Today it is used by about 400 scientists.

In order to use the LSDF Online Storage Service, interested users must first register for the service. Registration takes place via bwIDM web interface. Registration is only possible if a special Entitlement has been assigned to the user previously. Users usually receive this Entitlement after consultation with their responsible ITB or group representative or, if no project has yet been set up, by a separate request to the service operator.

If you have any questions regarding registration, requesting a new storage project or further questions, please contact the service provider via the BW support portal.

The setup and size of the available storage space is determined by agreements between the users and the service provider and technically realized by quotas. For new projects a Data Management Plan (DMP) is developed and stored in coordination with the Institute of the Users and the representatives of the service provider. The DMPs are used for reporting and applying for funding to operate, maintain and expand the LSDF. The storage space of the LSDF Online Storage is currently usually available free of charge.

Further information on the range of services and the usage of the service can be found in this Wiki the user manual and the current terms of use.


LSDF Online Storage Projects
Documentation
Getting Help
Accessing Data
Snapshots/Backup
Miscellaneous