Linux plug-ins for server, container, and file monitoring

Checkmk currently has over 2,000 plug-ins that allow you to collect relevant metrics from your deployed systems. Additional plug-ins can also be used to extend the functionality of the monitoring agent. To get you started with monitoring on Linux and to show you what’s possible with Checkmk, we wrote about Linux plug-ins for server monitoring before.

However, that article was just the tip of what’s possible to monitor under Linux. Plenty more plug-ins exist. In the previous article, we showed you how to use a series of plug-ins, including:

• How to check for updates in your Linux distribution.
• How to collect inventory information.
• How to check logged-in users.
• How to make sure SSH is properly configured and working.
• How to monitor MySQL databases, LVM volumes and
• how to check the health of your drives through S.M.A.R.T. monitoring.

In the second article of the series, we want to delve into file systems (network ones included), Docker, and web server monitoring. Unless otherwise specified, the focus will be on Linux monitoring. Let’s see how to monitor them with Checkmk!

File system operations

Monitoring a file or folder from time to time is important. Maybe you need to check if a cache file is not being created or to be notified if a file has not been modified within a certain amount of time. Linux has had a standard way to monitor changes in files and folders since 2005: inotify. The main agent plug-in shipped with Checkmk, mk_inotify, is built upon inotify and is a prerequisite for monitoring a series of file system operations.

For actual monitoring of certain files and folders, two other components are needed: the Python module pynotify and the plug-in inotify. Pynotify is available through pip as usual, or packaged by your distribution. To install it, simply use:

$ sudo pip install pyinotify

The inotify plug-in will use it to gather info about specific file operations like access, open, create, modify. It is possible to specify what files and folders to monitor and which operations you want to be informed about. Checkmk will create a service for each file and folder you want monitored. That is done through the agent rule Monitor file operations with Inotify (Linux), for the Enterprise Edition.

NGINX and Apache web servers

For internal or external access, a web server is usually installed on one or more servers in any infrastructure. Nowadays, the two most widespread ones are NGINX and Apache. Both are easy to monitor with Checkmk.

For NGINX, the agent plug-in is named nginx_status and, as easily guessed, monitors the global status of the NGINX process (not the servers) running on a host. It does so by connecting to NGINX and collecting info from the stub status module. First, this component needs to be enabled for Checkmk to monitor your web server.

To check if it is, use:

nginx -V 2>&1 | grep -o with-http_stub_status_modulee

If you see an output, it is. If not, it can be quickly enabled by editing the NGINX configuration file (on Debian and derivatives is at /etc/nginx/sites-enabled/default) under the server block with:

location /nginx_status { stub_status on; access_log off; allow 127.0.0.1; deny all; }

Change 127.0.0.1 with the address of your Checkmk main location. Reload NGINX and the path http://127.0.0.1/nginx_status is open to connections, which Checkmk will use to connect and collect metrics. To install the agent plug-in in the Enterprise Edition, configure the NGINX webservers (Linux) agent rule, and you are ready to monitor NGINX with Checkmk.

Monitoring Apache works similarly. The Apache status module should be enabled, which is usually the case under Debian/Ubuntu. The /server_status path is then open to monitoring via Checkmk. The required agent plug-in is apache_status. The agent rule is Apache webservers (Linux) for the CEE. Once everything is set up, you will see a series of useful metrics from Apache. Number of slots, workers, accesses, CPU load for each worker, number of connections, servers and more can be monitored with the apache_status plug-in.

Docker nodes and containers

It is rare to have an infrastructure these days that doesn’t involve containers and the respective orchestration. On locale or on remote, Docker containers are used for many tasks. There are plenty of Linux servers that exploit their power. By now, it may be superfluous to say that containers can be, and need to be, monitored with Checkmk.

The main agent plug-in to have is mk_docker.py. It is the base executable that allows the other plug-ins to gather information about your Docker nodes and containers. The plug-ins to monitor nodes are:

• docker_node_disk_usage monitors the number and size of total and active containers, images, local volumes, and build cache for a Docker node.
• docker_node_info_containers monitors the total number of containers on a node, including paused and stopped ones.
• docker_node_info is the basic plug-in to monitor whether the Docker application is running or not.

For containers, Checkmk offers:

• docker_container_cpu monitors CPU utilization of a Docker container. docker_container_cpu_cgroupv2 does the same, but for cgroup v2.
• docker_container_diskstat monitors the throughput of block devices of Docker containers. It is possible to monitor every single disk or a summary check of them all. docker_container_diskstat_cgroupv2 is the equivalent for cgroup v2.
• docker_container_status_health monitors the health status of containers. It makes use of Docker's HEALTHCHECK API.
• docker_container_mem makes sure, unlike what you might expect by the name, that the Docker container respects memory usage limits of the node or as configured for the container itself. It does not measure general memory usage. Here is also a version for cgroup v2 available, docker_container_mem_cgroupv2.
• docker_container_status checks if a container is running or not and is therefore the most basic check plug-in for Docker monitoring.
• docker_container_status_uptime monitors the uptime of Docker containers. It can be configured to alert when the uptime goes below or above certain levels.

The agent rule in the CEE is named Docker node and containers.

NFS exports (Linux and Solaris)

NFS is the simplest way to share folders and their relative files on Linux and related operating systems. It has been around for what seems like forever and has been the classic method to export a directory to multiple hosts for years. NFS is present on any Linux distribution, and monitoring it is supported by Checkmk.

nfsexports is the agent plug-in for keeping your NFS exports on track and, once installed, will inform you of the availability of each export as defined in /etc/exports or through exportfs. It will alert you in case of malfunctioning in the NFS server daemon or portmapper, and if any of the exports is not available. Configure it through the NFS4 exports (Linux, Solaris) agent rule in the CEE.

To independently monitor them, one Checkmk service can be created for each export. 

NFS IO stats