ForHLR - User Access

From KIT HPC Wiki
Jump to: navigation, search

ForHLR is Baden-Württemberg's general purpose tier 2 high performance computing (HPC) cluster that can only be accessed by completing a project proposal that is examined by a referee of a scientific steering committee.
To log on ForHLR (II) an user account is required. All coworkers of a project manager can apply for an account.



ForHLR Phase I wiring by Robert Barthel, copyright: KIT (SCC)
ForHLR I wiring © KIT (SCC)


1 How to get Access

(German description on website http://www.scc.kit.edu/dienste/proposals.php)

Employees of Universities and research institutes can get access to HPC-system ForHLR II via a proposal. A test project, that is granted without an evaluation of a steering committee, can also be applied with a fixed amount of CPU-hours.

If the project manager is an employee of a foreign research organization, a german project partner is necessary to provide a proposal.

IMPORTANT: At the end of the time frame a status report must be provided (10-15 pages)!


Access to ForHLR II system by Hartmut Häfner, copyright: KIT (SCC)


4 steps for access to ForHLR II:

  1. Completion of the „Online Proposal Form“ (press send-button to submit the form).
    Resources to be granted are CPU-time and permanent diskspace!
    1. T ALTERNATIVELY: Completion of the „Online Test Project Form“ (press send-button to submit the form).
      Resource to be granted is the permanent diskspace!
  2. Writing the extended project description (3-5 pages).
    PDF-file must be send to forhlr-projects∂lists.kit.edu!
  3. Fill out of ForHLR access form for each project collaborator.
    Form must be send per fax to SCC-ServiceDesk or scanned to haefner∂kit.edu!
  4. Personal registration for ForHLR II on website https://bwidm.scc.kit.edu


Brief Summary

Complete steps 1+3+4, to get preliminary accounts with 25000 CPU-hours for all project members for testing.
Fulfill steps 1+2+3+4, to get accounts with 100000 CPU-hours for all project members during the granting phase.
Fulfill steps 1.T+3+4, to get accounts with 500000 CPU-hours for all project members for one year.


Extended Description
After execution of steps 1+2 all project collabotators can get (unconfirmed) accounts on ForHLR II. Thereto the access form for ForHLR II, that can be reached via the website
http://www.scc.kit.edu/hotline/formulare.php, must be filled, signed by the project manager and send to SCC-ServiceDesk (steps 3(+4)). The project manager determines the project collaborators. Up to the decision of the steering committee MIN(100000, requested_CPU-hours*1/10) CPU-hours are available. In case of a positive decision of the steering committee the accounts will be confirmed and all project collaborators can access the system within the limits of the granted resources.

We allow preliminary accounts to planned projects with a maximum lifespan of 4 weeks to estimate the needed resources of a project. Upon receipt of a project proposal within a 4-week period preliminary accounts will be converted into (unconfirmed) accounts, otherwise the preliminary accounts will be closed. By implementing the steps 1+3(+4) preliminary accounts will be activated. Please note that only 25000 CPU-hours are available for planned projects with preliminary accounts.

We allow test projects for smaller projects without evaluation of a steering committee and with a maximum lifespan of 1 year. By implementing the steps 1.T+3(+4) project accounts will be activated. Please note that only 500000 CPU-hours are available for test projects (without possibility of a modification). Test projects only get once the above mentioned CPU-hours.

Granting access and issuing a user account for ForHLR II always requires the registration on the KIT service website https://bwidm.scc.kit.edu (step 4).


Please fill the online form on the website

   http://www.scc.kit.edu/forschung/4971.php

Further needed informations can be found on the online form.


Please fill the online test project form on the website

   http://www.scc.kit.edu/forschung/10950.php

Further needed informations can be found on the online form.


If you have already filed a proposal sucessfully and your granted CPU-hours or granted permanent diskspace is exhausted, you have to file a follow-on proposal. Only a "short" extended project description of about 2 pages is required; therefore this kind of proposal can only be used, if the field of research within the project does'nt change substantially.

Please fill the follow-on online form on the website

   http://www.scc.kit.edu/forschung/6160.php


The time frame of all projects always ends at 6/30. If a projext is granted in the first half year, it will end at 6/30 of the next year. If a project in contrast ends in the second half year, it will end at 6/30 of the next but one year. About end of the time frame (a notification per e-mail will be sent to the project manager) a status report must be provided! If CPU-time and permanent diskspace is not exhausted at the end of the time frame of a project, the time frame of a project will be automatically extended by one year after acceptance of the status report by the responsible referee. If the granted CPU-time is exhausted or the project requires substantially more permanent diskspace, a follow-on proposal must be submitted!

1.1 Step A: Filling the Access Form

Each collaborator of a project and the project manager, if he wants access to ForHLR II, have to fill the form:


1.2 Step B: Web Registration

After step A, i.e., after issueing the ForHLR II entitlement, please visit:

  • https://bwidm.scc.kit.edu
    1. Select your home organization from the list and click Proceed
    2. You will be directed to the Identity Provider of your home organisation
    3. Enter your home-organisational user ID / username and your home-organisational password and click Login button
    4. You will be redirected back to the registration website https://bwidm.scc.kit.edu/
    5. Select FH2 Service Description (on the left hand side)
    6. Click Register
    7. Finally
      • for all non-KIT members mandatorily set a service password for authentication on ForHLR II
      • for KIT members optionally set a service password for authentication on ForHLR II



1.3 Changing Password

By default, for KIT members your ForHLR II password to log on matches that one of your KIT account, while for all non-KIT members your ForHLR II password is that one you saved during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new ForHLR II password via the registration website https://bwidm.scc.kit.edu by carrying out the following steps:

  1. visit https://bwidm.scc.kit.edu and select your home organization
  2. authenticate yourself via your home-organizational user id / username and your home-organizational password
  3. find on the left side ForHLR II and select Set Service Password
  4. set new service, i.e. ForHLR II, password, repeat it and click Save button.
  5. the page answers e.g. "Das Passwort wurde bei dem Dienst geändert" ("password has been changed")
  6. proceed to log in using the new password in the next step


1.4 Contact

If you have questions or problems concerning registration, please contact your local hotline.

2 Login

After finishing the web registration ForHLR II is ready for your SSH based login. ForHLR II has five dedicated login nodes. The selection of the login node is done automatically and is based on a round-robin scheduling. Logging in another time the session might run on another login node.

Only the secure shell SSH is allowed to login. Other protocols like telnet or rlogin are not allowed for security reasons.


A connection to ForHLR II can be established by the command:

$ ssh <UserID>@forhlr2.scc.kit.edu

If you are using OpenSSH (usually installed on Linux based systems) and you want to use a GUI-based application on ForHLR II can be established by the command:

$ ssh -X <UserID>@forhlr2.scc.kit.edu

If you are in more than one project group, your home directory (the directory you see after login) will be the first project name. With the command

$ id

you can see all project groups you are in. Changing the project group can be done by the command

$ newgrp <new_project_group>

Changing the project group does not mean to change the accounting behaviour of your batch jobs. If you want to debit a specific account and a specific project group respectively you have to use the sbatch option -A (--account=...) when starting batch jobs. See also https://wiki.scc.kit.edu/hpc/index.php/ForHLR_-_SLURM_Batch_Jobs#sbatch_Command_Parameters.


2.1 About UserID / Username

<UserID> of the ssh command is a placeholder for your username at your home organization together with a prefix as followed:

  • Freiburg: <UserID> = fr_username
  • Heidelberg: <UserID> = hd_username
  • Hohenheim: <UserID> = ho_username
  • KIT: <UserID> = username (without any prefix)
  • Konstanz: <UserID> = kn_username
  • Mannheim: <UserID> = ma_username
  • Stuttgart: <UserID> = st_username
  • Tübingen: <UserID> = tu_username
  • Ulm: <UserID> = ul_username


2.2 Allowed activities on login nodes

The login nodes of ForHLR II are the access point to the compute system and to your ForHLR II $HOME directory. The login nodes are shared with all users of ForHLR II. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

  • short compilation of your program code and
  • short pre- and postprocessing of your batch jobs.


To guarantee usability for all users of ForHLR II you must not run your compute jobs on the login nodes. Compute jobs must be submitted to the queueing system. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the queueing system.

3 Important Project Figures

Your project has four important operating figures:

  1. the granted CPU-hours (exactly core-hours),
  2. the consumed CPU-hours (exactly core-hours),
  3. the granted permanent disk space (in your project-directory),
  4. the consumed permanent disk space (in your project-directory).

The first two numbers can be queried by two below briefly described Slurm-commands. The third and fourth number can be found in the file /project/fh2-project-<your_acronym>/diskusage.

The granted CPU-minutes(!) of all the projects you are involved in can be queried by the command

$ sacctmgr -p list assoc

The used CPU-hours ( or CPU-minutes) of all the projects you are involved in can be queried by the command

$ sreport [-p] --tres cpu cluster AccountUtilizationByUser Start=2019-02-18 [-t hours]

If the used CPU-minutes of your (chosen) project are higher than the granted CPU-minutes, batch jobs will stay PENDING in the queue with the error message "AssocGrpCPUMinutesLimit" till the granted CPU-minutes will be raised!

HENCE IMPORTANT FOR PROJECT MANAGERS: Please send at the latest a follow-on online proposal via the website
http://www.scc.kit.edu/dienste/proposals.php, if the error message "AssocGrpCPUMinutesLimit" for your submitted batchjobs occurs! Please remind that your account is only filled up with MIN(100000, 10% of required CPU-hours) CPU-hours during the review period!

In the file diskusage many numbers are specified; the first two numbers are important:

  1. granted_permanent_disk_space (for directory /project/fh{1|2}-project-<your_acronym>),
  2. consumed_permanent_disk_space (of directory /project/fh{1|2}-project-<your_acronym>).

If the second number is greater than the first number, you can't write data in an arbitrary subdirectory of directory
/project/fh2-project-<your_acronym> anymore. This also means that batch jobs will crash if they write data into this directory tree!

4 First steps on ForHLR II

For setting up your environment to use compilers and installed software please visit:


For guides on how to submit compute jobs to ForHLR II please visit:



Back to top