Let’s Encrypt certificate monitoring with Checkmk

Is a fully-fledged monitoring solution like Checkmk a bit of overkill just for monitoring Let’s Encrypt certificates? No way! Checkmk is easy to set up with its single package installation, has moderate hardware requirements and is easily extendable. Especially when you begin to notice that simple ‘uptime only’ dashboards no longer fulfill all of your requirements, you will be happy to add agent based monitoring to those hosts where it matters – and don’t have to switch monitoring solutions to do so.

For monitoring the health of the Checkmk server itself, we recommend at least installing and properly configuring a monitoring agent. Checkmk includes agent packages for most Linux distributions. These packages can be downloaded from within the Checkmk web interface. Instead of downloading and then again uploading, you can simply take the installation package from the file system:

find /opt/omd/ -name '*agent*.deb'

And install it (make sure you are the root user):

apt install /opt/omd/versions/2.3.0pXX.cre/share/check_mk/agents/check-mk-agent_2.3.0pXX-1_all.deb

Now head to the Checkmk GUI located at

http://hostname/sitename

Login as cmkadmin with the password you just defined. Go to Setup > Hosts and create a first folder to contain the Checkmk server itself. I used infrastructure as folder name. In that folder, create a host localhost that uses the defaults. Click Save & run service discovery.

Note: In Checkmk you might use Help > Show inline help to read detailed information on the available options for each setting.

In the service discovery, you will usually have to click Accept all twice to confirm the services found. Now click on the yellow octagon indicating that some changes have not been activated. In Checkmk changes are not automatically activated upon saving to enable admins to Activate changes that may depend on each other.

You might have noticed that the Check_MK Agent service (yes, this one uses the old spelling) has the WARN state. This is because the communication between the Checkmk agent and server is not encrypted at this point. Go to a root shell one final time to enable mTLS by registering the agent to the site:

cmk-agent-ctl register --hostname localhost --server localhost --site my1stcheckmk --user cmkadmin

Attempting to register at localhost, port 8000. Server certificate details:

PEM-encoded certificate:
-----BEGIN CERTIFICATE-----
MIIFSDCCAzCgAwIBAgIUHG9nA6WWO8J6TySCuhF4LBWTCQEwDQYJKoZIhvcNAQEN
q5oZixF2aR7DVvTKz6W8d72wlaOmNTzmlxmBBeWc9keMCCuUD9jNPBKQ8oJg0R2v
1b9kYDypqoukSYZV
-----END CERTIFICATE-----

Issued by:
	Site 'my1stcheckmk' local CA
Issued to:
	my1stcheckmk
Validity:
	From Fri, 14 Feb 2025 16:35:36 +0000
	To   Wed, 14 Feb 2035 16:35:36 +0000

Do you want to establish this connection? [Y/n]
> y

Checkmk uses a rules-based approach for configuring nearly everything. Criteria for rules can be folders, labels and how objects are named. For the certificate monitoring, it makes sense to create a folder letsencrypt, where all hosts that correspond to certificates will be placed. This time, take care with the details for the folder and set No API integrations, no Checkmk agent. This is the most direct way to define that all hosts in this folder are monitored only with active checks. Active checks are those checks that directly check an exposed service and not interpret agent output. So, no agent installation as done previously for localhost is needed for the hosts that will be placed into this folder.

As the next step, we recommend adding the hosts that should be checked: Navigate to Setup > Hosts, there open the new folder and click on the Hosts menu. Here you find an entry Import hosts via CSV file. With this you can either upload a CSV file or paste the contents of a CSV file into a text entry field. If you are only adding a dozen or so hosts, you might just type the hostnames, one per line into the field. Name the single column by typing ‘hostname’ in the first line:

hostname
docs.checkmk.com
checkmk.com
download.checkmk.com
play.checkmk.com
trust.checkmk.com

Now you can experience the full power of rules: Open the Setup menu and type ‘certificate' in the search field. Below HTTP, TCP, Email you will be presented with Check certificates. Click on this suggestion to edit/add rules.

You can now add a new rule for the certificate checks. Follow the suggestions from the following screenshot regarding prefix, name and endpoint. Here we have created a warning at 22 days of remaining certificate lifetime. If you attempt to renew daily, you might use 28 days here. Finally, of most importance is the condition: Apply the rule to the newly created folder without any further limitation. This associates the rule with the parameters specified to each and any host in this folder.

Of course, having to look at a dashboard to detect problems is not the smartest way for identifying issues. Proper notifications and alerting is really the way to go, and email still dominates notification methods. For this reason, it makes sense to begin with emails.

There is one significant difference between Checkmk Raw and Checkmk Cloud regarding email: Checkmk Raw relies entirely on the system MTA (‘mail transfer agent’), so as a minimum you will need to configure something like the relay-only Nullmailer. Alternatively, for HTML email Checkmk Cloud provides so-called ‘synchronous delivery’ which cuts out a local MTA and instead directly delivers to a smarthost. This is easier to configure and allows for better detection of failed email deliveries.

At this point we highly recommend creating a normal monitoring user that has an email address in order to separate administration tasks from monitoring tasks. Fill out at least a name, user name and email address, and then add this user to the Everything contact group. When your monitoring environment is getting larger, you may want to create dedicated contact groups for certain projects and remove the Everything contact group from all users.

For notifications, one default rule to notify all contacts of a host/service will already be enabled. Since you added the new user to the Everything group, they will be notified if a service changes to WARN or CRIT. Click on the pencil symbol to edit the default rule – you will almost certainly want to change the from field in the emails generated.

To check whether notifications actually arrive at their destination, go to Setup > Events > Notifications. There you will find a button Test notifications. Click it to select a host and a service, set the checkmark for HTML/ASCII and go for it. A few seconds later, you will be presented with a protocol showing who received the email and – of course – also find an email in your inbox.