Introducing AlmaLinux ELevate

AlmaLinux today announced the introduction of their new feature ELevate which allows you to migrate between major versions of RHEL based distributions from 7.x to 8.x. This was previously not possible under CentOS and had came up during the CentOS 6.x end of life and in 2024 will come up again in the CentOS 7.x end of life. Until ELevate the only way to move to the latest version of RHEL based distributions was to deploy a new server with the desired version then configure and migrate everything to the new server.

How AlmaLinux ELevate accomplishes this feat is by utilizing Red Hat’s Leapp framework with a community created library and service for the migration set required for it. ELevate relies on the package evolution service (PES) which allows you to download, customize and submit new data sets for packages. This will allow maintainers and users alike make migrations smooth and easy. ELevate is currently under development and recommends you test your migration scenarios before attempting it. That being said let’s take a quick look at migrating a stock CentOS 7.x Hawk Host cloud compute to AlmaLinux 8.x

The first step in the upgrade process is to make sure we have a completely upgraded system which I currently have:

[root@el7conversion ~]# uname -rv
3.10.0-1160.45.1.el7.x86_64 #1 SMP Wed Oct 13 17:20:51 UTC 2021
[root@el7conversion ~]# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
[root@el7conversion ~]# 

It is now time to install the elevate-release packages with the project repository and GPG key:

yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el7.noarch.rpm

The next step is to install the required packages which in our case we wish to upgrade to AlmaLinux 8.x:

yum install -y leapp-upgrade leapp-data-almalinux

We then from here want to do a preupgrade check and see if we meet the minimum requirements to perform the upgrade:

leapp preupgrade

Upon this finishing I’m advised of the following:

============================================================
                     UPGRADE INHIBITED                      
============================================================

Upgrade has been inhibited due to the following problems:
    1. Inhibitor: Possible problems with remote login using root account
    2. Inhibitor: Detected loaded kernel drivers which have been removed in RHEL 8. Upgrade cannot proceed.
    3. Inhibitor: Missing required answers in the answer file
Consult the pre-upgrade report for details and possible remediation.

============================================================
                     UPGRADE INHIBITED                      
============================================================


Debug output written to /var/log/leapp/leapp-preupgrade.log

============================================================
                           REPORT                           
============================================================

A report has been generated at /var/log/leapp/leapp-report.json
A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                       END OF REPORT                        
============================================================

Answerfile has been generated at /var/log/leapp/answerfile

It looks like unfortunately the base system is not compatible and requires a few steps before we can proceed with the upgrade. This can be checked at /var/log/leapp/leapp-report.txt and in our case it contained the following items:

Risk Factor: high (inhibitor)
Title: Possible problems with remote login using root account
Summary: OpenSSH configuration file does not explicitly state the option PermitRootLogin in sshd_config file, which will default in RHEL8 to "prohibit-password".
Remediation: [hint] If you depend on remote root logins using passwords, consider setting up a different user for remote administration or adding "PermitRootLogin yes" to sshd_config.
Key: 3d21e8cc9e1c09dc60429de7716165787e99515f

We can fix this by running the following:

echo PermitRootLogin yes | sudo tee -a /etc/ssh/sshd_config

Risk Factor: high (inhibitor)
Title: Detected loaded kernel drivers which have been removed in RHEL 8. Upgrade cannot proceed.
Summary: Support for the following RHEL 7 device drivers has been removed in RHEL 8: 
     - floppy
     - pata_acpi
Please see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/considerations_in_adopting_rhel_8/index#removed-device-drivers_hardware-enablement for details.
Remediation: [hint] Please disable detected kernel drivers in order to proceed with the upgrade process using the rmmod or modprobe -r.
Key: b6fd580136aaf67fa42d68fb75b27f6e13f47c2d

We have two drivers which are not compatible which on our compute we don’t need so we can run the following:

rmmod floppy
rmmod pata_acpi

Risk Factor: high (inhibitor)
Title: Missing required answers in the answer file
Summary: One or more sections in answerfile are missing user choices: remove_pam_pkcs11_module_check.confirm
For more information consult https://leapp.readthedocs.io/en/latest/dialogs.html
Remediation: [hint] Please register user choices with leapp answer cli command or by manually editing the answerfile.
[command] leapp answer --section remove_pam_pkcs11_module_check.confirm=True
Key: d35f6c6b1b1fa6924ef442e3670d90fa92f0d54b

This requires a leap answer which we’re fine with so we can run the following:

leapp answer --section remove_pam_pkcs11_module_check.confirm=True

Risk Factor: high
Title: Packages from unknown repositories may not be installed
Summary: 1 packages may not be installed or upgraded due to repositories unknown to leapp:
- kernel-uek (repoid: ol8-uek)
Remediation: [hint] Please file a bug in http://bugzilla.redhat.com/ for leapp-repository component of the Red Hat Enterprise Linux product.
Key: 9a2b05abf8f45fd7915e52542887bb334bb218ea

In our case while rated as high it had no impact on our upgrade.

Risk Factor: high
Title: Difference in Python versions and support in RHEL 8
Summary: In RHEL 8, there is no 'python' command. Python 3 (backward incompatible) is the primary Python version and Python 2 is available with limited support and limited set of packages. Read more here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/#using-python3
Remediation: [hint] Please run "alternatives --set python /usr/bin/python3" after upgrade
Key: 0c98585b1d8d252eb540bf61560094f3495351f5

We can safely ignore this as it’s a base image and we’re not using Python 2 or Python 3 ourselves.

Risk Factor: high
Title: Leapp could not identify where GRUB core is located
Summary: We assume GRUB core is located on the same device as /boot. Leapp needs to update GRUB core as it is not done automatically on legacy (BIOS) systems. 
Remediation: [hint] Please run "grub2-install <GRUB_DEVICE> command manually after upgrade

After the upgrade is completely we’ll following the grub2-install instructions.

Title: Packages not signed by Red Hat found on the system
Summary: The following packages have not been signed by Red Hat and may be removed during the upgrade process in case Red Hat-signed packages to be removed during the upgrade depend on them:
- elevate-release
- gpg-pubkey
- leapp
- leapp-data-almalinux
- leapp-deps
- leapp-upgrade-el7toel8
- leapp-upgrade-el7toel8-deps
- python2-leapp
Key: 13f0791ae5f19f50e7d0d606fb6501f91b1efb2c

In our case this can be ignored as these are leap related packages.

Risk Factor: medium
Title: chrony using default configuration
Summary: default chrony configuration in RHEL8 uses leapsectz directive, which cannot be used with leap smearing NTP servers, and uses a single pool directive instead of four server directives
Key: c4222ebd18730a76f6bc7b3b66df898b106e6554

This is purely informative and in our case can be safely ignored.

Risk Factor: low
Title: Grep has incompatible changes in the next major version
Summary: If a file contains data improperly encoded for the current locale, and this is discovered before any of the file's contents are output, grep now treats the file as binary.
The 'grep -P' no longer reports an error and exits when given invalid UTF-8 data. Instead, it considers the data to be non-matching.
In locales with multibyte character encodings other than UTF-8, grep -P now reports an error and exits instead of misbehaving.
When searching binary data, grep now may treat non-text bytes as line terminators. This can boost performance significantly.
The 'grep -z' no longer automatically treats the byte '\200' as binary data.
Context no longer excludes selected lines omitted because of -m. For example, 'grep "^" -m1 -A1' now outputs the first two input lines, not just the first line.

Remediation: [hint] Please update your scripts to be compatible with the changes.
Key: 94665a499e2eeee35eca3e7093a7abe183384b16

Another case of good information to know but this would matter more if we had scripts on the server that were utilizing grep.

Risk Factor: low
Title: Postfix has incompatible changes in the next major version
Summary: Postfix 3.x has so called "compatibility safety net" that runs Postfix programs with backwards-compatible default settings. It will log a warning whenever backwards-compatible default setting may be required for continuity of service. Based on this logging the system administrator can decide if any backwards-compatible settings need to be made permanent in main.cf or master.cf, before turning off the backwards-compatibility safety net.
The backward compatibility safety net is by default turned off in Red Hat Enterprise Linux 8.
It can be turned on by running:  "postconf -e compatibility_level=0
It can be turned off by running: "postconf -e compatibility_level=2

In the Postfix MySQL database client, the default "option_group" value has changed to "client", i.e. it now reads options from the [client] group from the MySQL configuration file. To disable it, set "option_group" to the empty string.

The postqueue command no longer forces all message arrival times to be reported in UTC. To get the old behavior, set TZ=UTC in main.cf:import_environment.

Postfix 3.2 enables elliptic curve negotiation. This changes the default smtpd_tls_eecdh_grade setting to "auto", and introduces a new parameter "tls_eecdh_auto_curves" with the names of curves that may be negotiated.

The "master.cf" chroot default value has changed from "y" (yes) to "n" (no). This applies to master.cf services where chroot field is not explicitly specified.

The "append_dot_mydomain" default value has changed from "yes" to "no". You may need changing it to "yes" if senders cannot use complete domain names in e-mail addresses.

The "relay_domains" default value has changed from "$mydestination" to the empty value. This could result in unexpected "Relay access denied" errors or ETRN errors, because now will postfix by default relay only for the localhost.

The "mynetworks_style" default value has changed from "subnet" to "host". This parameter is used to implement the "permit_mynetworks" feature. The change could result in unexpected "access denied" errors, because postfix will now by default trust only the local machine, not the remote SMTP clients on the same IP subnetwork.

Postfix now supports dynamically loaded database plugins. Plugins are shipped in individual RPM sub-packages. Correct database plugins have to be installed, otherwise the specific database client will not work. For example for PostgreSQL map to work, the postfix-pgsql RPM package has to be installed.

Key: 5721e0a07a67d82cf7e5ea6f17662cd4f82e0a33

We can ignore this in our case as we have a stock CentOS 7.x server and won’t need to worry about this.

Risk Factor: info
Title: SElinux disabled
Summary: SElinux disabled, continuing...
Key: 4f25fea9b15b9d1d07d52cc1de02073f295dac3

Another purely informative message letting us know we already have SElinux disabled.

Risk Factor: info
Title: Current PAM and nsswitch.conf configuration will be kept.
Summary: There is a new tool called authselect in RHEL8 that replaced authconfig. The upgrade process was unable to find an authselect profile that would be equivalent to your current configuration. Therefore your configuration will be left intact.
Key: 40c4ab1da4a30dc1ca40e543f6385e1336d8810c

This is another case of useful information to know but we can ignore for our case. We’ve finally gone through everything at this point and it’s time to run the actual upgrade.

leapp upgrade

In our case as per the warning we need to run the following before our reboot:

grub2-mkconfig -o /boot/grub2/grub.cfg

We now can perform a reboot and using the console available in our client area we can watch the upgrade process. In my case we need to switch the kernel to the upgrade one that was created by Leap. Once it was booting the upgrade process started and once completed the server rebooted. Upon logging into it again we can confirm it is now using AlmaLinux 8:

[root@el7conversion ~]# uname -rv
4.18.0-305.19.1.el8_4.x86_64 #1 SMP Wed Sep 15 11:28:53 EDT 2021
[root@el7conversion ~]# cat /etc/redhat-release 
AlmaLinux release 8.4 (Electric Cheetah)

In our case it worked quite nicely but we were upgrading from a base CentOS 7.x to a base AlmaLinux 8.x installation. This being unnecessary to do in our case as we already offer an AlmaLinux 8 operating system image. If however you’re looking to move from CentOS 7.x to AlmaLinux 8 this may definitely be a great option for you. As per their recommendation though make sure to test before upgrading as well as making backups in the event the upgrade process does not work for you.