Category:LSDF Online Storage: Difference between revisions

From Lsdf
Jump to navigationJump to search
No edit summary
No edit summary
 
(35 intermediate revisions by 4 users not shown)
Line 2: Line 2:
[[File:lsdf_online_storage_services.png|right|top|frameless|250px]]
[[File:lsdf_online_storage_services.png|right|top|frameless|250px]]


⚫
[[File:Lsdf_online_storage_services_2.jpg|right|top|thumb|250px|LSDF Online Storage @ KIT (SCC)]]
⚫
The LSDF is a central storage system for large research data from data-intensive scientific disciplines. It is available to KIT researchers, their partners in the federal state and the Helmholtz Association as a multi-petabyte storage system. The service LSDF Online Storage enables KIT users and their partners in the state of Baden-Württemberg to access the storage resources of the LSDF. The service is used by institutes with research funded by the Helmholtz Association as well as insitutes at the university level of the KIT from different scientific disciplines. The project is sponsored and funded by the Ministry of Science, Research and the Arts of Baden-Württemberg (MWK), the German Research Foundation (DFG) and the Helmholtz Association of German Research Centres. It is operated by the Steinbuch Centre for Computing (SCC) at KIT.

⚫
The LSDF is a central storage system for large research data from data-intensive scientific disciplines. It is available to KIT researchers, their partners in the federal state and the Helmholtz Association as a multi-petabyte storage system. The service LSDF Online Storage enables KIT users and their partners in the state of Baden-Württemberg to access the storage resources of the LSDF. The service is used by institutes with research funded by the Helmholtz Association as well as insitutes at the university level of the KIT from different scientific disciplines. The project is sponsored and funded by the Ministry of Science, Research and the Arts of Baden-Württemberg (MWK), the German Research Foundation (DFG) and the Helmholtz Association of German Research Centres. It is operated by the Scientific Computing Center (SCC) at KIT.

The LSDF enables scientists to store and access their research data via different services. These services, tailored to the requirements of the researchers, are offered on top of the LSDF storage. Important features are in particular high storage capacity, fast access to and transfer of the data, easy data access and management capabilities, as well as the connection to modern processing infrastructures such as the high-performance computing clusters based at KIT like the bwUniCluster or the ForHLR II.


Access to the data is possible via standard protocols. Permissions to access the data are controlled by the data owners using group memberships and access control lists (ACLs). The LSDF Online Storage currently offers 10PB of usable storage capacity with a combined maximum read/write speed of the storage system of 25GByte/s. Today it is used by about 400 scientists.
Access to the data is possible via standard protocols. Permissions to access the data are controlled by the data owners using group memberships and access control lists (ACLs). The LSDF Online Storage currently offers 10PB of usable storage capacity with a combined maximum read/write speed of the storage system of 25GByte/s. Today it is used by about 400 scientists.
Line 8: Line 12:
In order to use the LSDF Online Storage Service, interested users must first register for the service. Registration takes place via [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf bwIDM web interface]. Registration is only possible if a special Entitlement has been assigned to the user previously. Users usually receive this Entitlement after consultation with their responsible ITB or group representative or, if no project has yet been set up, by a separate request to the service operator.
In order to use the LSDF Online Storage Service, interested users must first register for the service. Registration takes place via [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf bwIDM web interface]. Registration is only possible if a special Entitlement has been assigned to the user previously. Users usually receive this Entitlement after consultation with their responsible ITB or group representative or, if no project has yet been set up, by a separate request to the service operator.


If you have any questions regarding registration, requesting a new storage project or further questions, please contact the service provider via the [https://bw-support.scc.kit.edu/ BW support portal].
If you have any questions regarding registration, requesting a new storage project or further questions, please contact the service provider via the [https://bw-support.scc.kit.edu/?mode=ticket_submit&respounit=LSDF%20Online%20Storage BW support portal].


The setup and size of the available storage space is determined by agreements between the users and the service provider and technically realized by quotas. For new projects a Data Management Plan (DMP) is developed and stored in coordination with the Institute of the Users and the representatives of the service provider. The DMPs are used for reporting and applying for funding to operate, maintain and expand the LSDF. The storage space of the LSDF Online Storage is currently usually available free of charge.
The setup and size of the available storage space is determined by agreements between the users and the service provider and technically realized by quotas. For new projects a Data Management Plan (DMP) is developed and stored in coordination with the Institute of the Users and the representatives of the service provider. The DMPs are used for reporting and applying for funding to operate, maintain and expand the LSDF. The storage space of the LSDF Online Storage is currently usually available free of charge.
Line 14: Line 18:
Further information on the range of services and the usage of the service can be found in this Wiki the [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf user manual] and the current [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf terms of use].
Further information on the range of services and the usage of the service can be found in this Wiki the [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf user manual] and the current [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf terms of use].



{| style="width: 100%; border-spacing: 5px;"
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |
⚫
[[File:LSDF_Online_Storage.jpg|center|border|700px|LSDF Online Storage KIT (SCC)]]
|-
| style="text-align:center; color:#000;vertical-align:middle;" |<span style="font-size:100%">'''LSDF Online Storage © KIT (SCC)''' </span>
|}
{| style="width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;"
{| style="width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;"
<!-- ACCESS -->
<!-- ACCESS -->
| style="width:50%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;" |
| style="width:50%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;" |
{| style="width:100%; vertical-align:top; background:#f5fffa;"
{| style="width:100%; vertical-align:top; background:#f5fffa;"
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Access</div>
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">LSDF Online Storage Projects</div>
|-
|-
| style="color:#000; padding:2px 5px;" |
| style="color:#000; padding:2px 5px;" |
* [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf LSDF Online Storage registration]
* [https://bwidm.scc.kit.edu/user/register-service.xhtml?ssn=lsdf Signing up to exising storage projects] (for users)
⚫
* [https://www.lsdf.kit.edu/os/storageprojects/new/ Requesting new storage projects] (for project admins)
|-
* [https://www.scc.kit.edu/downloads/SDM/Nutzungsbedingungen-LSDF-Online-Storage.pdf Terms of use] (German)
<!-- Storage Request -->
* [https://www.scc.kit.edu/downloads/SDM/Terms-of-Use-LSDF-Online-Storage.pdf Terms of use] (English)
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Storage Request</div>
* [[LSDF_Online_Storage:_Acknowledgements|Acknowledgements]]
|-
| style="color:#000; padding:2px 5px;" |
⚫
* [https://www.lsdf.kit.edu/os/storagerequest/ Storage Request] (for project admins)
|-
|-
<!-- Documentation -->
<!-- Documentation -->
Line 39: Line 36:
|-
|-
| style="color:#000; padding:2px 5px;" |
| style="color:#000; padding:2px 5px;" |
* [http://www.scc.kit.edu/downloads/sdm/Nutzungsbedingungen-LSDF-Online-Storage.pdf terms of use] (German version)
* [http://www.scc.kit.edu/downloads/SDM/Nutzerhandbuch-LSDF-Online-Storage.pdf User manual] (German)
* [http://www.scc.kit.edu/downloads/sdm/Nutzerhandbuch-LSDF-Online-Storage.pdf user manual] (German version)
* [https://www.scc.kit.edu/downloads/SDM/User-Manual-LSDF-Online-Storage.pdf User manual] (English)
|-
|-
<!-- Support -->
<!-- Support -->
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Support/Ticket System</div>
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Getting Help</div>
|-
|-
| style="color:#000; padding:2px 5px;" |
| style="color:#000; padding:2px 5px;" |
* [https://bw-support.scc.kit.edu BW support portal]
* [https://bw-support.scc.kit.edu/?mode=ticket_submit&respounit=LSDF%20Online%20Storage Baden Württemberg Support Portal]
|-
|-
<!-- Web sites -->
<!-- Web sites -->
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Web sites</div>
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Web sites</div>
|-
|-
| style="color:#000; padding:2px 5px;" |
| style="color:#000; padding:2px 5px;" |
* [http://www.scc.kit.edu/forschung/11843.php Large Scale Data Facility (LSDF)] (German version)
* [http://www.scc.kit.edu/dienste/11228.php LSDF Online Storage] (German)
* [http://www.scc.kit.edu/en/research/11843.php Large Scale Data Facility (LSDF)] (English version)
* [http://www.scc.kit.edu/en/services/11228.php LSDF Online Storage] (English)
* [http://www.scc.kit.edu/dienste/11228.php LSDF Online Storage] (German version)
* [https://www.scc.kit.edu/dienste/17035.php Data services - selection according to application scenarios] (German)
* [http://www.scc.kit.edu/en/services/11228.php LSDF Online Storage] (English version)
* [https://www.scc.kit.edu/en/services/17035.php Data services - selection according to application scenarios] (English)
* [https://www.scc.kit.edu/services/17037.php Data services - selection according to specifications] (German)
* [https://www.scc.kit.edu/en/services/17037.php Data services - selection according to specifications] (English)
|-
|-

<!-- Monitoring
<!-- Monitoring
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Monitoring</div>
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Monitoring</div>
Line 70: Line 70:


<!-- PROTOCOLS -->
<!-- PROTOCOLS -->
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Protocols</div>
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Accessing Data</div>
|-
|-
| style="color:#000; padding:2px 5px;" |
| style="color:#000; padding:2px 5px;" |
* [[LSDF_Online_Storage:_Secure_Shell|Secure Shell (ssh, scp, sftp, sshfs)]]
'''for LSDF login cluster os-login.lsdf.kit.edu:'''
* [[LSDF_Online_Storage:_WebDAV|WebDAV / HTTPS]]<br>
* [[LSDF_Online_Storage_SCP|SCP]], [[LSDF_Online_Storage_SFTP|SFTP]], [[LSDF_Online_Storage_SSH|SSH]], [[LSDF_Online_Storage_SSHFS|SSHFS]]
* [[LSDF_Online_Storage:_SMB_/_CIFS|SMB / CIFS]] (KIT intern only)
<!--
⚫
* [[LSDF_Online_Storage_NFS|NFS]] (KIT intern only)
* [[LSDF_Online_Storage_HTTPS|HTTPS]] (https://os.lsdf.kit.edu for downloads only) -->
* [[LSDF_Online_Storage:_HPC|on the KIT HPC Clusters]]
'''for LSDF cluster os.lsdf.kit.edu:'''
* [[LSDF_Online_Storage:_HDF_AAI|using the AAI of the Helmholtz Data Fedaration (HDF)]]
⚫
* [[LSDF_Online_Storage_CIFS|SMB/CIFS]], [[LSDF_Online_Storage_NFS|NFS]] (KIT intern)

|-
<!-- TOOLS -->
| style="padding:2px;" | <div style="margin:3px; background:#cedff2; font-size:120%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;">Tools</div>
|-
| style="color:#000; padding:2px 5px;" |
'''for HPC Cluster:'''
<br>
* [[LSDF_Online_Storage_rdata|rdata]] (for [http://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster], [http://www.scc.kit.edu/dienste/forhlr.php ForHLR]) <!--[http://hpcmon.scc.kit.edu HPC-Auslastung] --> <br>
'''for SCP/SFTP/SSH protocol:'''
* [http://www.openssh.com/ OpenSSH]
*[http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty suite] (for Windows and Unix)
*[https://winscp.net/eng/download.php WinSCP] (for Windows)
*[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux)
*[http://cygwin.com/install.html Cygwin] (for Windows)
'''network drive over SFTP:'''
*[https://www.southrivertechnologies.com/download/downloadwd.html WebDrive] (for Windows, Mac, iOS, Android)
*[http://www.sftpnetdrive.com/ SFTPNetDrive] (for Windows)
*[http://www.netdrive.net/ NetDrive] (for Windows)
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac)
*[https://mountainduck.io MountainDuck] (for Windows and Mac)
'''network drive over SMB/CIFS:'''
*[https://www.bwhpc-c5.de/wiki/index.php/LSDF_Online_Storage_CIFS#Using_CIFS_for_WINDOWS_client__.28only_KIT_intern_or_KIT-VPN.29 Windows Explorer] (KIT intern)
'''for UNIX client:'''
* [[LSDF_Online_Storage_SCP|scp]], [[LSDF_Online_Storage_SFTP|sftp]], [[LSDF_Online_Storage_SSH|ssh]], [[LSDF_Online_Storage_SSHFS|sshfs mount]]
* [[LSDF_Online_Storage_CIFS#SMB_Client|smbclient]], [[LSDF_Online_Storage_CIFS#Mount_a_Share|cifs mount]] (KIT intern)
* [[LSDF_Online_Storage_rsync|rsync]], [[LSDF_Online_Storage_lftp|lftp]]
'''for WINDOWS client:'''
*[[LSDF_Online_Storage_RoboMirror|RoboMirror]] (for backup and restore)
|-
|-
<!-- SNAPSHOTS AND BACKUP -->
<!-- SNAPSHOTS AND BACKUP -->
Line 112: Line 85:
|-
|-
| style="color:#000; padding:2px 5px;" |
| style="color:#000; padding:2px 5px;" |
* [[LSDF_Online_Storage_Snapshots|Snapshots]] (for data retrieving and restoring)
* [[LSDF_Online_Storage:_Snapshots|Snapshots]] (for data retrieving and restoring)
* [[LSDF_Online_Storage_Backup|Backup]]
* [[LSDF_Online_Storage_Backup|Backup]]
|-
|-
Line 119: Line 92:
|-
|-
| style="color:#000; padding:2px 5px;" |
| style="color:#000; padding:2px 5px;" |
⚫
* [[LSDF_Online_Storage_Filesystem|Filesystem characteristics]]
* [[LSDF_Online_Storage_Login_Cluster|Login Cluster]]
⚫
* [[LSDF_Online_Storage_Filesystem|Filesystem]]
* [[LSDF_Online_Storage_Quota|Quota]]
* [[LSDF_Online_Storage_Quota|Quota]]
<!-- * [[LSDF_Online_Storage_Price|Price]] -->
|}
|}
|}
|}

Latest revision as of 10:48, 8 October 2024

Lsdf online storage services.png
LSDF Online Storage @ KIT (SCC)

The LSDF is a central storage system for large research data from data-intensive scientific disciplines. It is available to KIT researchers, their partners in the federal state and the Helmholtz Association as a multi-petabyte storage system. The service LSDF Online Storage enables KIT users and their partners in the state of Baden-Württemberg to access the storage resources of the LSDF. The service is used by institutes with research funded by the Helmholtz Association as well as insitutes at the university level of the KIT from different scientific disciplines. The project is sponsored and funded by the Ministry of Science, Research and the Arts of Baden-Württemberg (MWK), the German Research Foundation (DFG) and the Helmholtz Association of German Research Centres. It is operated by the Scientific Computing Center (SCC) at KIT.

The LSDF enables scientists to store and access their research data via different services. These services, tailored to the requirements of the researchers, are offered on top of the LSDF storage. Important features are in particular high storage capacity, fast access to and transfer of the data, easy data access and management capabilities, as well as the connection to modern processing infrastructures such as the high-performance computing clusters based at KIT like the bwUniCluster or the ForHLR II.

Access to the data is possible via standard protocols. Permissions to access the data are controlled by the data owners using group memberships and access control lists (ACLs). The LSDF Online Storage currently offers 10PB of usable storage capacity with a combined maximum read/write speed of the storage system of 25GByte/s. Today it is used by about 400 scientists.

In order to use the LSDF Online Storage Service, interested users must first register for the service. Registration takes place via bwIDM web interface. Registration is only possible if a special Entitlement has been assigned to the user previously. Users usually receive this Entitlement after consultation with their responsible ITB or group representative or, if no project has yet been set up, by a separate request to the service operator.

If you have any questions regarding registration, requesting a new storage project or further questions, please contact the service provider via the BW support portal.

The setup and size of the available storage space is determined by agreements between the users and the service provider and technically realized by quotas. For new projects a Data Management Plan (DMP) is developed and stored in coordination with the Institute of the Users and the representatives of the service provider. The DMPs are used for reporting and applying for funding to operate, maintain and expand the LSDF. The storage space of the LSDF Online Storage is currently usually available free of charge.

Further information on the range of services and the usage of the service can be found in this Wiki the user manual and the current terms of use.


LSDF Online Storage Projects
Documentation
Getting Help
Web sites
Accessing Data
Snapshots/Backup
Miscellaneous





Subcategories

This category has only the following subcategory.

L

Retrieved from "https://wiki.scc.kit.edu/lsdf/index.php?title=Category:LSDF_Online_Storage&oldid=5823"