Seven check plug-ins for Linux server monitoring

Linux is really popular on servers – the Open Source operating system has a reputation for being stable and reliable. Still, if you're a sysadmin and looking after one or more Linux servers, you probably want to monitor your machines' health and performance (and receive alerts when necessary).

Checkmk supports various Linux distributions. The server component is available as installation package (.deb and .rpm), as virtual appliance, and as a Docker container. The Linux version of the Checkmk agent is also easy to install: It's a simple shell script (/usr/bin/check_mk_agent) that invokes various other commands (like df, mount, ps, etc.) to collect data about the file system, the CPU, network devices, etc.

It is also possible to extend the agent's functionality with plug-ins. In this blog post, we are going to introduce seven Checkmk plug-ins that come in handy when monitoring Linux servers.

Plug-ins at your service

Checkmk plug-ins are small scripts or programs invoked by the agent to collect additional data about the monitored machine(s). There are two types of plug-ins:

• Check plug-ins
• Agent plug-ins

About 2,000 check plug-ins are shipped with Checkmk, i.e. they're on the server. In addition to the check plug-ins, approximately 100 agent plug-ins exist. They need to be installed on the monitored system. Most plug-ins mentioned in this blog post are agent plug-ins. Some of them require a configuration (​​​​​​in/etc/check_mk) to function correctly, others work out of the box. Please have a look at the documentation of the plug-ins or check the source code to find out more about their configuration options.

If you're using Checkmk Raw, you can install the plug-ins by copying them to the plug-ins directory of the Linux agent (/usr/lib/check_mk_agent/plugins); please remember to make the files executable (chmod +x). Since some of the plug-ins have a very long runtime and don't need to be refreshed every minute, you can store them in subdirectories named after a number of seconds to execute them asynchronously.

Customers of Checkmk Enterprise or Checkmk Cloud can use the agent bakery to install the plug-ins and generate the configuration file(s) automatically.

To find and download the agent plug-ins in Checkmk Raw and how to automatically roll out plug-ins using one of the commercial Checkmk editions via Agent Bakery, please refer to our manual.

And here are our top 7 plug-ins for monitoring Linux servers with Checkmk.

1. Package Management

All three plug-ins reach the warning state for regular package updates and critical for security updates.

Tip: It's a good idea to install the plug-ins with asynchronous execution; it's not really necessary to check for updates more than once every day.

2. Inventory

3. Number of logged-in users

If you'd like to monitor the number of users who are currently logged in, you can install the plug-in mk_logins. It combines the two commands "who" and "wc -l" to receive information about logged-in users. In Checkmk Enterprise or Checkmk Cloud, you can deploy the agent plug-in with the rule Number of logged-in users (Linux) under Setup.

The default configuration gives out a warning if the number exceeds 20, and it reaches the critical level if the number is 30 or more. You can adjust the rule and redefine numbers as you see fit. For example, set the threshold to 1 for servers where normally nobody logs in. Also, it's possible to configure time periods and specify different warn/crit levels.

Tip: On one of my servers, I've modified the plug-in so that it excludes my own IP address of the dial-up connection by adding a simple grep command:

#!/bin/bash
[...]
if type who >/dev/null; then
    echo "<<<logins>>>"
    who | grep -v 203.0.113.0 | wc -l
fi

4. SSH Daemon

If your SSH server doesn't use /etc/ssh/sshd_config but a different config file, simply modify the plug-in in your favourite text editor and change the SSHD_CONFIG variable.

Whatever your configuration is, do check if the firewall is open to allow connections to the SSH port (by default 22, but it is advisable to change it). Run on the host to be monitored:

iptables -nvL

and if you notice dropped packets in the INPUT chain on the SSH port, it needs to be opened with:

iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPT

With ufw, it is just a matter of running

ufw allow 22/tcp

Or, if you are using firewalld:

firewall-cmd --permanent --add-port=22/tcp

Checkmk can check for a misconfigured firewall, iptables or one of its modern frontends like ufw and firewalld, with the mk_iptables agent plug-in.

5. MySQL Databases

Do not forget the agent plug-in itself, that needs to be installed for the check plug-ins to work: mk_mysql. As usual, put the agent in /usr/lib/check_mk_agent/plugins, make sure it is executable, and you are ready to keep an eye on your MySQL databases. Or, with Checkmk Enterprise or Checkmk Cloud, bake the plug-in directly into the agent package for easy installation. To do so you need to search and configure the agent rule MySQL Databases under Setup.

6. LVM

7. SMART monitoring