ForHLR - User Access

From KIT HPC Wiki
Jump to: navigation, search

ForHLR is Baden-Württemberg's general purpose tier 2 high performance computing (HPC) cluster that can only be accessed by completing a project proposal that is examined by a referee of a scientific steering committee.
To log on ForHLR (II) an user account is required. All coworkers of a project manager can apply for an account.



ForHLR Phase I wiring by Robert Barthel, copyright: KIT (SCC)
ForHLR I wiring © KIT (SCC)


1 How to get Access

(German description on website http://www.scc.kit.edu/dienste/proposals.php)

Employees of Universities and research institutes can get access to HPC-system ForHLR II via a proposal. A test project, that is granted without an evaluation of a steering committee, can also be applied with a fixed amount of CPU-hours.

If the project manager is an employee of a foreign research organization, a german project partner is necessary to provide a proposal.

IMPORTANT: At the end of the time frame a status report must be provided (10-15 pages)!


Access to ForHLR II system by Hartmut Häfner, copyright: KIT (SCC)


4 steps for access to ForHLR II:

  1. Completion of the „Online Proposal Form“ (press send-button to submit the form).
    Resources to be granted are CPU-time and permanent diskspace!
    1. T ALTERNATIVELY: Completion of the „Online Test Project Form“ (press send-button to submit the form).
      Resource to be granted is the permanent diskspace!
  2. Writing the extended project description (3-5 pages).
    PDF-file must be send to forhlr-projects∂lists.kit.edu!
  3. Fill out of ForHLR access form for each project collaborator.
    Form must be send per fax to SCC-ServiceDesk or scanned to haefner∂kit.edu!
  4. Personal registration for ForHLR II on website https://bwidm.scc.kit.edu


Brief Summary

Complete steps 1+3+4, to get preliminary accounts with 25000 CPU-hours for all project members for testing.
Fulfill steps 1+2+3+4, to get accounts with 100000 CPU-hours for all project members during the granting phase.
Fulfill steps 1.T+3+4, to get accounts with 500000 CPU-hours for all project members for one year.


1.1 Extended Description


After execution of steps 1+2 all project collaborators can get (unconfirmed) accounts on ForHLR II. The access form for ForHLR II located at http://www.scc.kit.edu/hotline/formulare.php must be filled, signed by the project manager and sent to SCC-ServiceDesk (steps 3(+4)). The project manager determines the project collaborators. Up to the decision of the steering committee MIN(100000, requested_CPU-hours*1/10) CPU-hours are available. In case of a positive decision of the steering committee the accounts will be confirmed and all project collaborators can access the system within the limits of the granted resources.

We allow preliminary accounts to planned projects with a maximum lifespan of 4 weeks to estimate the needed resources of a project. Upon receipt of a project proposal within a 4-week period preliminary accounts will be converted into (unconfirmed) accounts, otherwise the preliminary accounts will be closed. By implementing the steps 1+3(+4) preliminary accounts will be activated. Please note that only 25000 CPU-hours are available for planned projects with preliminary accounts.

We allow test projects for smaller projects without evaluation of a steering committee and with a maximum lifespan of 1 year. By implementing the steps 1.T+3(+4) project accounts will be activated. Please note that only 500000 CPU-hours are available for test projects (without possibility of a modification). Test projects only get once the above mentioned CPU-hours.

Granting access and issuing a user account for ForHLR II always requires the registration on the KIT service website https://bwidm.scc.kit.edu (step 4).

Please fill the online form on the website

   http://www.scc.kit.edu/forschung/4971.php

Further needed informations can be found on the online form.


Please fill the online test project form on the website

   http://www.scc.kit.edu/forschung/10950.php

Further needed informations can be found on the online form.

If you have already filed a proposal sucessfully and your granted CPU-hours or granted permanent diskspace is exhausted, you have to file a follow-on proposal. Only a "short" extended project description of about 2 pages is required; therefore this kind of proposal can only be used, if the field of research within the project does'nt change substantially.

Please fill the follow-on online form on the website

   http://www.scc.kit.edu/forschung/6160.php

The time frame of all projects always ends at 6/30. If a projext is granted in the first half year, it will end at 6/30 of the next year. If a project in contrast ends in the second half year, it will end at 6/30 of the next but one year. About end of the time frame (a notification per e-mail will be sent to the project manager) a status report must be provided! If CPU-time and permanent diskspace is not exhausted at the end of the time frame of a project, the time frame of a project will be automatically extended by one year after acceptance of the status report by the responsible referee. If the granted CPU-time is exhausted or the project requires substantially more permanent diskspace, a follow-on proposal must be submitted!

1.2 Step A: Filling the Access Form

Each collaborator of a project and the project manager, if he wants access to ForHLR II, have to fill the form:


1.3 Step B: Web Registration, service password and 2-factor authentication

After completing step A, i.e., after successfull issueing of the ForHLR entitlement, you have to register yourself for the service. To do so please visit https://bwidm.scc.kit.edu and complete the following steps.

1. Select your home organization from the list on the main page and click Proceed or Fortfahren. If you have got a GuP-account of KIT (GuP stands for Gäste-und-Partner), please select KIT. (If you are from a non-governmental organization or from an organization outside of Baden-Württemberg, you always will get a GuP-account.)

FH2 access login bwidm select institution.png


2. You will be directed to the Identity Provider of your home organisation (or KIT). Enter the user ID / username and password of your home organisation (or KIT) - this is usually the same password used for your e-mail account and other services - and click on Login, Einloggen or something similar.

3. You will be redirected back to the registration website https://bwidm.scc.kit.edu/. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution (or KIT) is providing to the central system. Please check that all data is valid and then click on Continue or Weiter.

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "FH2". Click on Register or Registrieren to start the registration process.

FH2 access login bwidm service list.png


5. Since August 13, 2020 a 2-factor authentication mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

FH2 access login bwidm registration requirements 2fa.png

Click on the Link or on the My Tokens link in the main menu. The instructions for registering a new 2FA token can be found on the following page: ForHLR - User Access/2FA Tokens. Please complete them before proceeding.

6. Make sure all requirements are met by checking the Requirements box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please contact the support channels.

FH2 access login bwidm registration requirements.png


7. Read the Terms of Use (Nutzungsbedingungen und -richtlinien), check the box besides I have read and accepted the terms of use and click on Register or Registrieren.

8. Set a service password for the ForHLR II and click on Save or Speichern. Logging in with the password of your home organisation is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

FH2 access login bwidm service password.png





1.4 Changing the Service Password

Your ForHLR II password is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new ForHLR II password via the registration website https://bwidm.scc.kit.edu by carrying out the following steps:

  1. Go to https://bwidm.scc.kit.edu and select your home organization
  2. Authenticate yourself via the user id / username and password provided by your home institution
  3. Find the entry FH2 and select Set Service Password
  4. Enter the new password, repeat it and click Save button.
  5. If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
  6. Proceed to log in using the new password


1.5 Contact

If you have questions or problems concerning registration, please contact your local hotline.


2 Establishing network access

Access to ForHLR II is limited to IP addresses from the campus networks of the home institutions of our users. If you are on your campus network (e.g. in your office or on the campus WiFi) you should be able to connect to ForHLR II without restrictions. If you are outside of one of those networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).

If your home institution is a non-governmental organization or an organization outside of Baden-Württemberg, you must set up a VPN-channel to KIT. Then you can establish a VPN connection to KIT and login to ForHLR2.
If your home institution is not on our allow list but should be, please contact the support channels.

3 Login

After finishing the web registration ForHLR II is ready for your SSH based login. ForHLR II has five dedicated login nodes. The selection of the login node is done automatically and is based on a round-robin scheduling. Logging in another time the session might run on another login node.

Only the secure shell SSH is allowed to login. Other protocols like telnet or rlogin are not allowed for security reasons.


A connection to ForHLR II can be established by the command:

$ ssh <UserID>@forhlr2.scc.kit.edu

If you are using OpenSSH (usually installed on Linux based systems) and you want to use a GUI-based application on ForHLR II can be established by the command:

$ ssh -X <UserID>@forhlr2.scc.kit.edu

If you are in more than one project group, your home directory (the directory you see after login) will be the first project name. With the command

$ id

you can see all project groups you are in. Changing the project group can be done by the command

$ newgrp <new_project_group>

Changing the project group does not mean to change the accounting behaviour of your batch jobs. If you want to debit a specific account and a specific project group respectively you have to use the sbatch option -A (--account=...) when starting batch jobs. See also https://wiki.scc.kit.edu/hpc/index.php/ForHLR_-_SLURM_Batch_Jobs#sbatch_Command_Parameters.


3.1 Usernames

Your username will be the same as the one provided by your home institution, but prefixed with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on ForHLR II is kn_ab1234.

The following list contains all prefixes currently in use:

Home organization <UserID>
Universität Freiburg fr_username
Universität Heidelberg hd_username
Universität Hohenheim ho_username
KIT username (without any prefix)
Universität Konstanz kn_username
Universität Mannheim ma_username
Universität Stuttgart st_username
Universität Tübingen tu_username
Universität Ulm ul_username
Hochschule Aalen aa_username
Hochschule Albstadt-Sigmaringen as_username
Hochschule Esslingen es_username
Hochschule Furtwangen hf_username
Hochschule Karlsruhe hk_username
HTWG Konstanz ht_username
Hochschule Reutlingen hr_username
Hochschule Rottenburg ro_username
Hochschule für Technik Stuttgart hs_username
Hochschule Ulm hu_username



3.2 Example login process

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a One-Time Password. Generate one using the Software or Hardware Token registered on the bwIDM system (see ForHLR - User Access/2FA Tokens) and enter it after the Your OTP: prompt.

2. The systems asks for your service password. Enter it after the Password: prompt.

3. You are greeted by the ForHLR II banner followed by a shell.

The result should look like this:

FH2 access login example.png


3.3 Troubleshooting

Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead

Likely cause: You are most likely not on a network from which access to the ForHLR II system is allowed. Please check if you might have to establish a VPN connection first.


Issue: The system asks for the One-Time Password multiple times

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.


Issue: The system asks for the service password multiple times

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution.


Issue: There is an error message by the pam_ses_open.sh skript

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

  • Log into bwIDM, look for the FH2 entry and click on Registry info. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.



3.4 Allowed activities on login nodes

The login nodes of ForHLR II are the access point to the compute system and to your ForHLR II $HOME directory. The login nodes are shared with all users of ForHLR II. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

  • short compilation of your program code and
  • short pre- and postprocessing of your batch jobs.


To guarantee usability for all users of ForHLR II you must not run your compute jobs on the login nodes. Compute jobs must be submitted to the queueing system. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the queueing system.

3.5 SSH Keys

In contrast to many other systems it is no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

ForHLR - User Access/SSH Keys

4 Important Project Figures

Your project has four important operating figures:

  1. the granted CPU-hours (exactly core-hours),
  2. the consumed CPU-hours (exactly core-hours),
  3. the granted permanent disk space (in your project-directory),
  4. the consumed permanent disk space (in your project-directory).

The first two numbers can be queried by two below briefly described Slurm-commands. The third and fourth number can be found in the file /project/fh2-project-<your_acronym>/diskusage.

The granted CPU-minutes(!) of all the projects you are involved in can be queried by the command

$ sacctmgr -p list assoc

The used CPU-hours ( or CPU-minutes) of all the projects you are involved in can be queried by the command

$ sreport [-p] --tres cpu cluster AccountUtilizationByUser Start=2019-02-18 [-t hours]

If the used CPU-minutes of your (chosen) project are higher than the granted CPU-minutes, batch jobs will stay PENDING in the queue with the error message "AssocGrpCPUMinutesLimit" till the granted CPU-minutes will be raised!

HENCE IMPORTANT FOR PROJECT MANAGERS: Please send at the latest a follow-on online proposal via the website
http://www.scc.kit.edu/dienste/proposals.php, if the error message "AssocGrpCPUMinutesLimit" for your submitted batchjobs occurs! Please remind that your account is only filled up with MIN(100000, 10% of required CPU-hours) CPU-hours during the review period!

In the file diskusage many numbers are specified; the first two numbers are important:

  1. granted_permanent_disk_space (for directory /project/fh{1|2}-project-<your_acronym>),
  2. consumed_permanent_disk_space (of directory /project/fh{1|2}-project-<your_acronym>).

If the second number is greater than the first number, you can't write data in an arbitrary subdirectory of directory
/project/fh2-project-<your_acronym> anymore. This also means that batch jobs will crash if they write data into this directory tree!

5 First steps on ForHLR II

For setting up your environment to use compilers and installed software please visit:


For guides on how to submit compute jobs to ForHLR II please visit:



Back to top