We use cookies to ensure that we give you the best experience on our website.  Visit our Privacy Policy to learn more. If you continue to use this site, we will assume that you are okay with it.

Your choices regarding cookies on this site.
Your preferences have been updated.
In order for the changes to take effect completely please clear your browser cookies and cache. Then reload the page.

Monitoring Kubernetes

Checkmk Manual
Last updated: August 29 2019

Search in the manual

1. Introduction

The great success of Docker has led to people using Docker on an ever-larger scale. In contrast to virtual machines such as VMWare, its very low overhead makes the container ‚cheap’ and thus almost a mass-product. It goes without saying that a good tool for orchestrating the containers is essential. For the majority, the open source tool Kubernetes will be the tool of choice.

Checkmk supports monitoring of Kubernetes from version 1.5.0p12. The focus is currently on states and metrics that are especially interesting for the administrator. The following Check plug-ins are available:

There are some new plugins since version 1.6.0:

2. Setting up the monitoring

2.1. Service-Account

To set up a Kubernetes cluster in Checkmk you first need to have a service account and a related Cluster Role in Kubernetes so that Checkmk can access the API. We have created the file check_mk_rbac.yaml for you as a ready template which you will find in the „Treasures“, in the directory share/doc/check_mk/treasures/kubernetes directory, or online here. Its first part looks something like this:

apiVersion: v1
kind: Namespace
  name: check-mk
kind: ServiceAccount
[...approx 80 further lines...]

We use check-mk here as Name and Namespace respectively.

Load this file onto your Kubernetes cluster with the kubectl command:

user@host:~$ kubectl apply -f check_mk_rbac.yaml
namespace/check-mk created
serviceaccount/check-mk created
clusterrole.rbac.authorization.k8s.io/check-mk created
clusterrolebinding.rbac.authorization.k8s.io/check-mk created

If you use the Google Kubernetes engine, it may be that you receive an "Error from server (Forbidden): error when creating get "check_mk_rbac.yaml": response. In this case you must first extend your user’s permissions. This is done with the following command (replacing MYNAME with your Google login name):

user@host:~$ kubectl create clusterrolebinding MYNAME-cluster-admin-binding --clusterrole=cluster-admin --user=MYNAME@example.org

If all has gone well, you can query the new service account with kubectl get serviceaccounts:

user@host:~$ kubectl get serviceaccounts check-mk -n check-mk -o yaml
apiVersion: v1
kind: ServiceAccount
    kubectl.kubernetes.io/last-applied-configuration: |
  creationTimestamp: "2019-01-23T08:16:05Z"
  name: check-mk
  namespace: check-mk
  resourceVersion: "4004661"
  selfLink: /api/v1/namespaces/check-mk/serviceaccounts/check-mk
  uid: 218179a3-1ee7-11e9-bf43-080027a5f141
- name: check-mk-token-z9hbp

There you will also find the name of the associated Secrets. This has the form ‚check-mk-token-ID‘ (here in the example check-mk-token-z9hbp). The ID for the Secret is generated automatically by Kubernetes. You can then use the contents of the Secrets with the get secrets query:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml
apiVersion: v1
  namespace: Y2hlY2stbWs=
kind: Secret
    kubernetes.io/service-account.name: check-mk
    kubernetes.io/service-account.uid: 218179a3-1ee7-11e9-bf43-080027a5f141
  creationTimestamp: "2019-01-23T08:16:06Z"
  name: check-mk-token-z9hbp
  namespace: check-mk
  resourceVersion: "4004660"
  selfLink: /api/v1/namespaces/check-mk/secrets/check-mk-token-z9hbp
  uid: 2183cee6-1ee7-11e9-bf43-080027a5f141
type: kubernetes.io/service-account-token

The output will include the base64 encoded CA certificate (ca.crt), and the base64 encoded tokens (token) for the account. You can choose the certificate from the output of get secret – e.g. with the following command cut it out, and immediately convert it to the form you need to import into Checkmk:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | grep "ca.crt" | cut -f4 -d' ' | base64 --decode

2.2. Importing a certificate into Checkmk

For Checkmk to accept the Kubernetes CA certificate, you must add it to WATO at Global Settings ➳ Site Management ➳ Trusted certificate authorities for SSL.

Without the correct import of the CA, the Checkmk service of the Kubernetes cluster will fail with and certificate verify failed:

2.3. Entering a password (Token) in Checkmk

The best way to save the service account token is to use WATO's password storage. This is the safest option, since the deposit and the use of the passwords is organizationally separate. Alternatively, enter the password directly in plain text when creating the rule (see below).

The following command line truncates the password directly from the output of get secrets:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | grep "token:" | cut -f4 -d' ' | base64 --decode

If you are working directly under Linux, you can also enter |xsel--clipboard. Then the password is not output, but copied directly to Clipboard (as if you had copied with the mouse):

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | grep "token:" | cut -f4 -d' ' | base64 --decode | xsel --clipboard

Tip: If you have the command line tool jq installed, the whole thing is a bit easier. jq is e.g. on Debian/Ubuntu in the package of the same name. It is a tool that can access JSON data in a structured way. This is the command line:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | jq -r .secrets[0].name

The ‚password‘ really is that long. Add it, for example, under the ID kubernetes in the password storage:

2.4. Adding a Kubernetes-Cluster to the Monitoring

The monitoring under Checkmk functions in two levels. The Kubernetes Cluster itself is monitored as a host. For the individual Kubernetes nodes we use the piggyback principle. That means each node is monitored as a separate host in Checkmk. The monitoring data from these hosts are not retrieved separately from Kubernetes, but instead derived from the data from the Kubernetes cluster.

Because Kubernetes cannot be queried over the normal Checkmk agent – you need the Kubernetes Special Agent – which is also known as the Datasource Program. In this case Checkmk does not contact the destination host as usual over TCP port 6556, instead it invokes a utility program that interfaces with the target system via a Kubernetes application-specific API.

The procedure is as follows:

  1. Create a host in Checkmk for the Kubernetes master (Kubernetes Control Plane).
  2. Create a rule that assigns the special agent for Kubernetes to this Kubernetes host.

This rule can be found in WATO at {{Host & Service Parameters|Datasource Programs|Kubernetes}}. In the properties of the rule you either enter the password in plain text, or select it via the password storage if you filed it there earlier.

You do not normally need any further information. The functions of the other options are best found in the Online Help ICON [icon_help.png].

If you now call the service configuration at the Kubernets-host (Discovery) in the WATO, you should already find some of the services:

2.5. New in Version 1.6.0

Version 1.6.0 also supports the monitoring of Pods, Services and Deployments in Checkmk. These are each represented as hosts. We recommend that you create this host automatically with the new dynamic configuration.

The configuration will now look like this:

The Custom URL prefix has, for example, the format https://mykuber01.comp.lan. If you do not specify this, Checkmk will use HTTPS as the protocol and use the IP address of the Kubernetes host instead of a host name in Checkmk. This new configuration alternatively allows HTTP (unsafe), and working with a name instead of an IP address.

2.6. Monitoring the nodes

So that the nodes are also monitored, you must also create them as hosts in WATO You can do this (from Checkmk version 1.6.0) with the new Dynamic Configuration Daemon (DCD). Or you simply create these as hosts by hand.

It is important that the hostnames in Checkmk exactly match the names of the Kubernetes nodes. You can easily get these names from the Kubernetes host’s Nodes service.

Unless you have a Checkmk agent installed on the nodes themselves (which would generally be rather unusual), you will need to set the Check_MK Agent to No agent.

3. Hardware/software inventory

The Kubernetes integration in Checkmk also supports the [Inventory|hardware/software inventory]. In version 1.5.0p12 this is limited to the Kubernetes roles. More plug-ins are planned.

4. Removing Checkmk

If you want to remove Checkmk’s service account and cluster role from Kubernetes, this can be performed with the following commands:

user@host:~$ kubectl -f check_mk_rbac.yaml
namespace "check-mk" deleted
serviceaccount "check-mk" deleted
clusterrole.rbac.authorization.k8s.io "check-mk" deleted
clusterrolebinding.rbac.authorization.k8s.io "check-mk" deleted