A better check_http – the technical background

The check_http maintained at Monitoring-Plugins.org is one of the most important active checks for monitoring systems that have their roots in Nagios. In addition to Checkmk, these systems include Icinga, Naemon and Zabbix. In this first part you can read the technical reasoning for the decision to start a new implementation, and learn everything you need to know about installation on monitoring systems other than Checkmk and about using it on the command line.

This blog article is based on a 30-minute talk at the OSMC. A second part will soon cover the configuration via Checkmk's GUI and suggestions for the most efficient migration of existing rules for check_http.

The previous check_http is over 20 years old. Over time, various developers have added support for SSL, proxies and other features. One consequence is that the check does not evaluate some command line parameters for certain calls and gives precedence to specific ones (for example, checking both certificate lifetime and status code is not possible in a single call). In Checkmk we display such restrictions in the GUI, so our users know immediately that two rules – and thus two check calls – are necessary. Improving the login options and revising the behavior when using SSL would have required a lot of work. 

The last point is particularly important to many Checkmk users: they expect the check to behave like a browser. For example, it should switch to a CRIT state if an SSL-secured call fails in the browser. 

When deciding whether to work on the existing check or start a new implementation, we also had to consider compatibility. Since achieving bug compatibility would be challenging for a check in use for over 20 years, we chose a new implementation. Other factors included the heavy burden a large number of pull requests would place on maintainers and the fact that additional dependencies could impact the portability of the monitoring plug-ins. We therefore decided to carry out a new implementation in-house.

The next step was to choose the programming language for the new check. C/C++, Python and Rust are the languages we primarily use at Checkmk, with a current shift away from C/C++ and a focus on Rust for performance-critical components. C/C++ would have allowed to add the check – potentially as check_httpv2 – in the collection of monitoring plug-ins, but this method is more difficult to maintain than the other two options. 

Python has the largest programming community within Checkmk, but implementing it as a special agent would have broken compatibility with other monitoring systems. The decisive factor was that the current way of calling both active checks and special agents – namely via a created and terminated process per call – was not competitive enough with Python in terms of performance. 

 In the end, Rust won the race: it offers performance similar to C/C++, and has already proven its value in the company with the Agent Controller. This modern, compiled language had everything going for it.

If you are using a monitoring system that does not or does not yet include check_httpv2, you can simply retrofit it. Programs written in Rust can be compiled in just a few minutes thanks to the quickly installed tool chain. The source code available under GPLv2 is contained in the Checkmk Github repository, our example shows the installation under Debian and Ubuntu.

First, ensure the presence of some dependencies:

apt install git curl
apt install build-essential libssl-dev pkg-config

Next, install and make the Rust toolchain available in the current shell environment:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
. “$HOME/.cargo/env”

Access the source codes by cloning the complete Checkmk repository locally. This takes a few minutes –  enough to prepare a hot drink:

We recommend using the latest stable branch, at the time of writing this blog article this was 2.3.0:

git checkout 2.3.0

This is followed by the build and thus the consumption of the hot drink prepared above:

cd checkmk/packages/check-http/
cargo build --release

You can execute the freshly compiled binary directly from the build directory:

ls -lah target/release/
time ./target/release/check_http --url https://docs.checkmk.com/
echo $?

If the test succeeds, install the new check to a suitable directory, such as /usr/local/bin or the directory containing the monitoring plug-ins. We recommend naming it check_httpv2:

install -m 0755 ./target/release/check_http /usr/local/bin/check_httpv2

The biggest advantage over the familiar check_http is its ability to evaluate various server configuration, certificate status and load parameters in a wider range of scenarios. For example, in addition to a response time threshold of 0.2s (WARN) to 0.5s (CRIT), you can also verify the response code 200, TLS 1.3 availability and a certificate validity period of at least 7 or 15 days:

check_httpv2 \
    --timeout 2 \
    --response-time-levels 0.2,1.0 \
    --min-tls-version tls13 \
    --certificate-levels 15,7 \
    --status-code 200 \
    --url https://docs.checkmk.com/

For a list of all command line parameters, use the --help option. Alternatively, you can set up a test installation with Checkmk (this can be done in five minutes), configure check_httpv2 there and retrieve the generated command line from the service details:

1. Navigate to ‘Check HTTP web service’ in the setup search
2. Create a rule with the desired settings for an accessible endpoint
3. Find the command line parameters in the service details for the check

In addition to check_httpv2, we have also developed check_cert, another active check written in Rust. Use it not only to check certificates for remaining validity and other aspects such as key length or issuer, even if no HTTPS endpoint exposes the certificates. 

The source codes are located in checkmk/packages/check-cert. Compiling and determining the command line is similar to check_httpv2. Due to the focus on checking certificates, more detailed methods will be added here rather than in check_httpv2.