How to monitor servers with broken TLS in Checkmk 2.3

With Checkmk 2.3.0 we bumped the version of OpenSSL from 1.1 to 3.0. This means much stricter defaults regarding security relevant ciphers, handshake and renegotiation settings. You might experience failing active checks for all kinds of services, including but not limited to check_http, check_smtp and several others. Here is how to work around potential problems.

Since the humble beginnings of Checkmk, OMD has been a way of packaging all components of a monitoring system. This not only included the monitoring core, the web interface, and the database components. It also meant packaging libraries such as OpenSSL or the Python interpreter in specific versions that are consistent for all supported operating systems. This component consistency is required to guarantee consistent behavior on rather old systems like the still supported RHEL 8 while simultaneously supporting new systems like the soon-to-be released Ubuntu 24.04.

One major update that accompanied the 2.3.0 update of Checkmk is the change from OpenSSL 1.1 to 3.0. The OpenSSL developers worked hard on removing old ciphers, legacy protocols and extensions and switching to more modern (read: more secure) defaults. This inevitably breaks compatibility to older software or hardware, resulting in failed active checks in Checkmk.

In general these failing checks should be used as an indicator that the monitored host either has to be updated to more modern TLS libraries or at least TLS specific settings should be adjusted to more secure defaults. During internal tests with the Checkmk 2.3.0 beta, this has already worked in most cases.

However, changing TLS settings might not be possible in all cases. Reasons such as maintaining a broad compatibility with other older systems can be valid. For systems where neither updates nor configuration changes are possible, you might want to change the behavior of OpenSSL 3.x to be able to communicate with these systems. This is done by overwriting the OpenSSL default configuration with a more permissive custom configuration that will only be applied to affected hosts.

OMD[heute]:~$ cat << EOF > etc/unsafe_openssl.cnf 
# DISCLAIMER:
# This configuration is _very_ insecure.
# Please reconsider if it is possible to upgrade the monitored system instead.
# Proceed at your own risk.

# Tell openssl which section to check
openssl_conf = openssl_init

[openssl_init]
# Which section to use for tls/ssl related options
ssl_conf = ssl_configuration

[ssl_configuration]
# system_default will be applied during any creation of the SSL_CTX structure
system_default = policy

[policy]
# To read up on these options go to:
# https://www.openssl.org/docs/man1.1.1/man3/SSL_CONF_cmd.html

MinProtocol = TLSv1
Options = UnsafeLegacyRenegotiation

# The Ciphers and the seclevel are explained here:
# https://www.openssl.org/docs/man1.1.1/man1/ciphers.html
# https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_security_level.html
CipherString = DEFAULT:@SECLEVEL=0
EOF

The example provided should work in most cases. If it does not work, we recommend trying to figure out a solution that works with the standard checks that are using OpenSSL 3.0. This might involve further changes to the configuration file and thus lead to the aforementioned multiple configuration files for differing problems. We would appreciate it if you publish your solutions in the Checkmk Forum. Please do not forget to specify the software or device version your configuration targets.

As you have learned, the active check simply means running a program that adheres to the specifications of the Monitoring Plug-ins project. If you have no success with changing the OpenSSL 3 configuration for the service you want to check, you are free to try installing a monitoring plug-ins package from other sources other than Checkmk. These will not interfere with those supplied by Checkmk.

Before doing so, please consider:

1. You might hide security issues that urgently need to be addressed.

2. Taking a lazy approach might prevent us from creating a proper archive of OpenSSL 3 configuration files for widely used software and devices.

You have been warned but still want to continue? The effort needed can vary greatly: Installation might be as easy as using a distribution supplied package. For example, Ubuntu 20.04 provides monitoring plug-ins 2.2 as a binary package linked against OpenSSL 1.1. Testing this combination is virtually no effort, so it might be worth giving it a try. On newer distributions, it can be necessary to compile both OpenSSL 1.1 and the monitoring plug-ins, which might require lots of compiler tweaking, source patching and thus tedious trial and error.

While it might be necessary to break compatibility sometimes for a few affected systems to achieve better security for the majority of systems, the open approach of Checkmk with its adherence to commonly used standards allows the creation of easy and target-oriented workarounds.