Migration to RHEL8 operating system (Tier-2)

During the first half of 2022, we will migrate the oldest HPC-UGent Tier-2 clusters that are using CentOS 7 as operating system to RHEL8 (Red Hat Enterprise Linux 8).

This includes clusters swalot, skitty, victini, kirlia, and joltik (see also the infrastructure overview), as well as the Tier-2 login nodes.

Motivation

Migrating to RHEL8 is done to bring all clusters in line with the most recent clusters that are already running RHEL8 (doduo, accelgor, slaking), as well as the Tier-1 cluster Hortense.

This makes the maintenance of the HPC-UGent Tier-2 infrastructure significantly easier, since we only need to take into account a single operating system version going forward.

Impact on login nodes

The default HPC-UGent Tier-2 login nodes (which you access with an SSH client via login.hpc.ugent.be)
will be switched to RHEL8 on Monday 25 April 2022 at 07:00 CEST.

Next to the switch to a more recent operating system (which should be transparent w.r.t. user experience),
this also involves some changes that will impact the login experience.

1. The host keys of the login nodes will change.
   
   As a result of this, your SSH client will produce a (scary looking) warning that the host key it has on record does not match the current host key for login.hpc.ugent.be .
   
   You should verify whether the new host key that is presented is correct, and if so accept the new host key.
   The fingerprint of the new host key for the RHEL8 login nodes should be one of the following:
   
   * ECDSA:
   e6:d2:9c:d8:e7:59:45:03:4a:1f:dc:96:62:29:9c:5f (MD5)
   

   C8TVx0w8UjGgCQfCmEUaOPxJGNMqv2PXLyBNODe5eOQ (SHA256)
   67:6c:af:23:cc:a1:72:09:f5:45:f1:60:08:e8:98:ca:31:87:58:6c (SHA1)

   * ED25519:
   19:28:76:94:52:9d:ff:7d:fb:8b:27:b6:d7:69:42:eb (MD5)
   8AJg3lPN27y6i+um7rFx3xoy42U8ZgqNe4LsEycHILA (SHA256)
   e3:cc:07:64:78:80:28:ec:b8:a8:8f:49:44:d1:1e:dc:cc:0b:c5:6b (SHA1)

   * RSA:
   10:2f:31:21:04:75:cb:ed:67:e0:d5:0c:a1:5a:f4:78 (MD5)
   W8Wz0/FkkCR2ulN7+w8tNI9M0viRgFr2YlHrhKD2Dd0 (SHA256)
   53:25:8c:1e:72:8b:ce:87:3e:54:12:44:a7:13:1a:89:e4:15:b6:8e (SHA1)
   
   

   See also section 10.6 in the HPC-UGent user manual .
   
   On Linux and macOS, you can use this command to remove the cached fingerprint(s) for login.hpc.ugent.be :
   

   ssh-keygen -R login.hpc.ugent.be
   

   When using PuTTy on Windows, you should see a clear warning popup window that mentions the fingerprint of the currently presented host key, with an easy way to accept the new host key after verifying it.
   
   
2. Stricter user limits will be enforced on the RHEL8 login nodes.
   
   To encourage only using the login nodes as an entry portal to the HPC-UGent infrastructure, stricter user limits will be enforced on the RHEL8 login nodes.
   This includes (per user):
   * max. of 24 hours of CPU time;
   * max. 8 GB of memory in use;
   * max. 80 processes;
   * max. 10 open sessions;
   If needed, these limits may be tweaked later.
   
   Note that the recently added interactive debug cluster "slaking" can be used as an alternative to the login nodes for resource-intensive tasks;
   for more information, see Chapter 22 in the HPC-UGent user manual .
   
   
3. Active sessions will be terminated when the switch is done.
   
   When login.hpc.ugent.be is switched to the RHEL8 login nodes, all active login sessions will be terminated.
   You will need to start a new login session to connect to the HPC-UGent infrastructure.
4. Warning + lag when using PuTTy
   
   Several people using PuTTy on Windows have reported seeing the following warning message after logging in:
   PuTTY X11 proxy: unable to connect to forwarded X server: Network error: Connection refused
   In addition, the responsiveness of the login session is very poor/laggy: you often need to type in a character multiple times before it appears in the session, which makes fluent typing impossible.
   
   The underlying cause seems to be logging in with X11 forwarding enabled in PuTTY, without having an X server (like Xming) running on your system.
   X11 forwarding is only enabled to run graphical user interfaces (GUIs) on the login nodes and forwarding the windows to your own system.
   
   To fix this problem, you should disable X11 forwarding in your PuTTy configuration (and save that change in your PuTTy session), as follows:
   i. In "Session", choose your saved session to connect to the HPC-UGent login nodes, and hit "Load";
   ii. Go to "Connection" -> "SSH" -> "X11" in your PuTTy session;
   iii. Untick the "Enable X11 forwarding" check box on top;
   iv. Go back to "Session", and hit "Save" to save the change;
   v. Try logging in again.
   You should no longer see the PuTTY X11 proxy warning message when logging in, and typing should be fluent now.
   
   For running GUI applications on the login nodes, we strongly recommend using the HPC-UGent web portal instead, see Chapter 8 in the HPC-UGent user manual .
   
   

Impact on central software stack

The migration to RHEL8 as operating system should not impact your workflow, everything will basically be working as it did before (incl. job submission, etc.).

However, there will be impact on the availability of software that is made available via modules.

Software that was installed with a compiler toolchain older than the ones we started using in the 2nd half of 2019 will no longer be available once the clusters have been updated to RHEL8.

This includes all software installations on top of a toolchain that is older than foss/2019b, gompi/2019b, intel/2019b, iimpi/2019b, GCC(core)/8.3.0 .

The "module" command will produce a clear warning when you are loading modules that are using a toolchain that will no longer be available after the cluster has been migrated to RHEL8.
For example:

foss/2019a:
       ___________________________________
      /  This module will soon no longer  \
      \  be available on this cluster!    /
       -----------------------------------
             \   ^__^
              \  (xx)\_______
                 (__)\       )\/\
                  U  ||----w |
                     ||     ||

If you require software that is currently only available with an older toolchain on the HPC-UGent Tier-2 clusters that are still running CentOS 7,
check via "module avail" if a more recent version is installed that you can switch to, or submit a software installation request so we can provide a more recent installation of that software which you can adopt.

Planning

We plan to migrate the HPC-UGent Tier-2 clusters that are still using CentOS 7 to RHEL 8 one by one, following the schedule outlined below.

Questions

If you have any questions regarding the migration of the oldest HPC-UGent Tier-2 clusters and Tier-2 login nodes to RHEL8,
don't hesitate to contact the HPC-UGent support team via hpc@ugent.be .