We use cookies to ensure that we give you the best experience on our website.  Visit our Privacy Policy to learn more. If you continue to use this site, we will assume that you are okay with it.

Your choices regarding cookies on this site.
Your preferences have been updated.
In order for the changes to take effect completely please clear your browser cookies and cache. Then reload the page.

Monitoring Kubernetes

1. Introduction

The great success of Docker has led to people using Docker on an ever-larger scale. In contrast to virtual machines such as VMWare, its very low overhead makes the container ‘cheap’, and thus almost a mass-product. It goes without saying that a good tool for orchestrating the containers is essential. For the majority, the open source tool Kubernetes will be the tool of choice.

Important: The latest Kubernetes versions 1.18 and 1.19 are currently not supported, due to the fact that the official Kubernetes Python Client is not yet compatible with Kubernetes' newest API.

Source: Kubernetes Python Client @ github

RFC: Release Kubernetes Python Clients in parallel for released Kubernetes version

Checkmk supports monitoring of Kubernetes from Version 1.5.0p12. The focus is currently on states and metrics that are especially interesting for the administrator. The following Check plug-ins are available:

Some new plug-ins have been created since Version 1.6.0:

2. Setting up the monitoring

2.1. Service-Account

To set up a Kubernetes cluster in Checkmk you first need to have a service account and a related Cluster Role in Kubernetes, so that Checkmk can access the API. We have created the file check_mk_rbac.yaml for you as a ready template which you will find in the ‘Treasures’, in the share/doc/check_mk/treasures/kubernetes directory, or online here. The first part of this file looks something like this:

share/doc/check_mk/treasures/kubernetes/check_mk_rbac.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: check-mk
---
kind: ServiceAccount
[...approx 80 further lines...]

We use check-mk here as the Name and Namespace respectively.

Load this file onto your Kubernetes cluster with the kubectl command:

user@host:~$ kubectl apply -f check_mk_rbac.yaml
namespace/check-mk created
serviceaccount/check-mk created
clusterrole.rbac.authorization.k8s.io/check-mk created
clusterrolebinding.rbac.authorization.k8s.io/check-mk created

If you use the Google Kubernetes engine, it may be that you receive an "Error from server (Forbidden): error when creating get check_mk_rbac.yaml": response. In this case you must first extend your user’s permissions. This is done with the following command (replacing MYNAME with your Google login name):

user@host:~$ kubectl create clusterrolebinding MYNAME-cluster-admin-binding --clusterrole=cluster-admin --user=MYNAME@example.org

If all has gone well, you can query the new service account with kubectl get serviceaccounts:

user@host:~$ kubectl get serviceaccounts check-mk -n check-mk -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},
"name":"check-mk","namespace":"check-mk"}}
  creationTimestamp: "2019-01-23T08:16:05Z"
  name: check-mk
  namespace: check-mk
  resourceVersion: "4004661"
  selfLink: /api/v1/namespaces/check-mk/serviceaccounts/check-mk
  uid: 218179a3-1ee7-11e9-bf43-080027a5f141
secrets:
- name: check-mk-token-z9hbp

There you will also find the name of the associated Secrets. This has the form ‘check-mk-token-ID ’ (here in the example check-mk-token-z9hbp). The ID for the Secret is generated automatically by Kubernetes. You can then use the contents of the Secrets with the get secrets query:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml
apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQVRBTkJna3Foa2lHO...
  namespace: Y2hlY2stbWs=
  token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklpSjkuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObG...
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: check-mk
    kubernetes.io/service-account.uid: 218179a3-1ee7-11e9-bf43-080027a5f141
  creationTimestamp: "2019-01-23T08:16:06Z"
  name: check-mk-token-z9hbp
  namespace: check-mk
  resourceVersion: "4004660"
  selfLink: /api/v1/namespaces/check-mk/secrets/check-mk-token-z9hbp
  uid: 2183cee6-1ee7-11e9-bf43-080027a5f141
type: kubernetes.io/service-account-token

The output will include the base64 encoded CA certificate (ca.crt), and the base64 encoded tokens (token) for the account. You can choose the certificate from the output of get secret – e.g. with the following command cut it out, and immediately convert it to the form you need to import into Checkmk:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | grep "ca.crt" | cut -f4 -d' ' | base64 --decode
-----BEGIN CERTIFICATE-----
MIIC5zCCAc+gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
a3ViZUNBMB4XDTE4MDkxMDE2MDAwMVoXDTI4MDkwODE2MDAwMVowFTETMBEGA1UE
AxMKbWluaWt1YmVDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK9Z
iG0gNZK5VU94a0E6OrUqxOQRdkv6S6vG3LnuozdgNfxsEetR9bMGu15DWaSa40JX
FbC5RxzNq/W9B2pPmkAlAguqHvayn7lNWjoF5P+31tucIxs3AOfBsLetyCJQduYD
jbe1v1/KCn/4YUzk99cW0ivPqnwVHBoMPUfVof8yA00RJugH6lMZL3kmOkD5AtRH
FTThW9riAlJATBofLfkgRnUEpfb3u1xF9vYEDwKkcV91ealZowJ/BciuxM2F8RIg
LdwF/vOh6a+4Cu8adTyQ8mAryfVPDhFBhbsg+BXRykhNzNDPruC+9wAG/50vg4kV
4wFpkPOkOCvB8ROYelkCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMB0GA1UdJQQW
MBQGCCsGAQUFBwMCBggrBgEFBQcDATAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
DQEBCwUAA4IBAQAeNwON8SACLl2SB8t8P4/heKdR3Hyg3hlAOSGjsyo396goAPS1
t6IeCzWZ5Z/LsF7o8y9g8A7blUvARLysmmWOre3X4wDuPvH7jrYt+PUjq+RNeeUX
5R1XAyFfuVcWstT5HpKXdh6U6HfzGpKS1JoFkySrYARhJ+MipJUKNrQLESNqdxBK
4gLCdFxutTTFYkKf6crfIkHoDfXfurMo+wyEYE4Yeh8KRSQWvaKTdab4UvMwlUbO
+8wFZRe08faBqyvavH31KfmkBLZbMMM5r4Jj0Z6a56qZDuiMzlkCl6rmKynQeFzD
KKvQHZazKf1NdcCqKOoU+eh6q6dI9uVFZybG
-----END CERTIFICATE-----

2.2. Importing a certificate into Checkmk

For Checkmk to accept the Kubernetes CA certificate, you must add it to WATO at Global Settings ➳ Site Management ➳ Trusted certificate authorities for SSL.

Without the correct import of the CA, the Checkmk service of the Kubernetes cluster will fail with and certificate verify failed:

2.3. Entering a password (Token) in Checkmk

The best way to save the service account token is to use WATO’s password storage. This is the safest option, since the deposit and the use of the passwords is organisationally separate. Alternatively, enter the password directly in plain text when creating the rule (see below).

The following command line truncates the password directly from the output of get secrets:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | grep "token:" | cut -f4 -d' ' | base64 --decode
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjaGVjay1tayIsI
mt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjaGVjay1tay10b2tlbi16OWhicCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5
hbWUiOiJjaGVjay1tayIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjIxODE3OWEzLTFlZTctMTFlOS1iZjQzLTA4MDAyN2E1ZjE0MSIsInN1YiI6I
nN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjaGVjay1tazpjaGVjay1tayJ9.gcLEH8jjUloTeaAj-U_kRAmRVIiETTk89ujViriGtllnv2iKF12p0L9ybT1fO-1Vx7XyU8jneQRO9lZw8JbhVmaPjrkEc8
kAcUdpGERUHmVFG-yj3KhOwMMUSyfg6wAeBLvj-y1-_pMJEVkVbylYCP6xoLh_rpf75JkAicZTDmhkBNOtSf9ZMjxEmL6kzNYvPwz76szLJUg_ZC636OA2Z47qREUtdNVLyutls7ZVLzuluS2rnfoP
JEVp_hN3PXTRei0F5rNeA01wmgWtDfo0xALZ-GfvEQ-O6GjNwHDlsqYmgtz5rC23cWLAf6MtETfyeEJjRqwituhqUJ9Jp7ZHgQ%

If you are working directly under Linux, you can also enter | xsel--clipboard. Then the password is not output, but copied directly to the clipboard (as if you had copied with the mouse):

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | grep "token:" | cut -f4 -d' ' | base64 --decode | xsel --clipboard

Tip: If you have the command line tool jq installed, the whole process is a bit easier. jq is e.g. on Debian/Ubuntu in the package of the same name. This is a tool that can access JSON data in a structured way.

This is the command line:

user@host:~$ kubectl get secrets check-mk-token-z9hbp -n check-mk -o yaml | jq -r .secrets[0].name

The ‘password’ really is that long. Add it, for example, under the ID kubernetes in the password storage:

2.4. Adding a Kubernetes-Cluster to the Monitoring

The monitoring under Checkmk functions in two levels. The Kubernetes Cluster itself is monitored as a host. For the individual Kubernetes nodes we use the piggyback principle. That means each node is monitored as a separate host in Checkmk. The monitoring data from these hosts are not retrieved separately from Kubernetes, but instead derived from the data from the Kubernetes cluster.

Because Kubernetes cannot be queried over the normal Checkmk agent – you need the Kubernetes Special Agent – which is also known as the Datasource Program. In this case Checkmk does not contact the destination host as usual over TCP port 6556, instead it invokes a utility program that interfaces with the target system via a Kubernetes application-specific API.

The procedure is as follows:

  1. Create a host in Checkmk for the Kubernetes master (Kubernetes Control Plane).
  2. Create a rule that assigns the special agent for Kubernetes to this Kubernetes host.

This rule can be found in WATO at Host & Service Parameters ➳ Datasource Programs ➳ Kubernetes. In the properties of the rule you either enter the password in plain text, or select it via the password storage if you filed it there earlier.

You do not normally need any further information. The functions of the other options are best found in the Online Help .

If you now call the service configuration at the Kubernets-host (Discovery) in the WATO, you should already find some of the services:

2.5. New in Version 1.6.0

Version 1.6.0 also supports the monitoring of Pods, Services and Deployments in Checkmk. These are each represented as hosts. We recommend that you create this host automatically with the new dynamic configuration.

The configuration will now look like this:

The Custom URL prefix has, for example, the format https://mykuber01.comp.lan. If you do not specify this, Checkmk will use HTTPS as the protocol and use the IP address of the Kubernetes host instead of a host name in Checkmk. As an alternative, this new configuration allows HTTP (unsafe), and working with a name instead of an IP address

The Custom path prefix is a path which is appended to the URL. A path prefix is important for Rancher, for example, because there are several Kubernetes clusters that can be included. An individual’s API clusters can then be reached – e.g., at /k8s/cluster/mycluster.

2.6. Monitoring the nodes

So that the nodes are also monitored, you must also create them as hosts in WATO You can do this (from Checkmk Version 1.6.0) with the new Dynamic Configuration Daemon (DCD). Or you simply create these as hosts by hand.

It is important that the hostnames in Checkmk exactly match the names of the Kubernetes nodes. You can easily get these names from the Kubernetes host’s Nodes service.

By the way – with the Access to agents ➳ General settings ➳ Hostname translation for piggybacked hosts ruleset you can define rules very flexibly, creating them based on the host names contained in piggy data. This means that you can use host names in Checkmk that do not match the names of the nodes.

Unless you have a Checkmk agent installed on the nodes themselves (which would generally be rather unusual), you will need to set the Check_MK Agent to No agent.

2.7. Labels in Kubernetes

In the future – from Version 1.6.0+ – Checkmk for Kubernetes will automatically discover labels for nodes, pods, services etc. The labels are defined in the same way as in Docker and have the form cmk/kubernetes_object:OBJECT.

In order to use the advantages of labels for Kubernetes monitoring in Version 1.6.0, you can use the Monitoring Configuration ➳ Host Checks ➳ Host labels rule set to change the behavior of Version 1.6.0+ manually. To do this you must create a new label in one rule for each OBJECT and assign it to the corresponding Kubernetes host. Overall you need the following labels:

  • cmk/kubernetes_object:node
  • cmk/kubernetes_object:service
  • cmk/kubernetes_object:deployment
  • cmk/kubernetes_object:pod
  • cmk/kubernetes_object:daemon_set
  • cmk/kubernetes_object:stateful_set

With the labels for nodes it is recommended to select the folder in which the Kubernetes nodes are located, or to specify all nodes directly in ‘Explicit hosts’. For the remaining objects you can simply use a regular expression for the prefix of the Piggyback Hosts in ‘Explicit hosts’ (e.g. ~pod_ for pods). After updating to Version 1.6.0+ you can remove the rules you have created.

A final note: the prefix cmk/ is usually the internal namespace of Checkmk, to which you should not add any labels. So that you can use the same rules before and after the update to Version 1.6.0+, it is advisable to make a small exception at this point.

3. Hardware/software inventory

The Kubernetes integration in Checkmk also supports the [Inventory|hardware/software inventory]. In Version 1.5.0p12 this is limited to the Kubernetes roles. More plug-ins are planned.

4. Removing Checkmk

If you want to remove Checkmk’s service account and cluster role from Kubernetes, this can be performed with the following commands:

user@host:~$ kubectl delete -f check_mk_rbac.yaml
namespace "check-mk" deleted
serviceaccount "check-mk" deleted
clusterrole.rbac.authorization.k8s.io "check-mk" deleted
clusterrolebinding.rbac.authorization.k8s.io "check-mk" deleted

5. Kubernetes in OpenShift installations

5.1. Creating a project

OpenShift is a product line of container application platforms for cloud computing developed by Red Hat, which is based, among other things, on Kubernetes.

From Version 1.5.0p13 Checkmk can also monitor an OpenShift-based Kubernetes. The procedure is very similar to that described above, but differs in some details when setting up the cluster for monitoring. You can create your own project for monitoring in OpenShift. This can be performed from the command line with:

root@linux# oc new-project check-mk
Now using project "check-mk" on server "https://192.168.42.62:8443".

You can add applications to this project with the 'new-app' command.
For example, try:

    oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git

to build a new example application in Ruby.

5.2. Creating a service account

The next step is to create a service account. This is done with:

root@linux# oc create serviceaccount check-mk
serviceaccount/check-mk created

5.3. Assigning cluster roles

Openshift already has the cluster-reader ClusterRole, with read rights which you can use for the service account – it is thus not necessary to set up a new ClusterRole. You can assign the cluster-reader ClusterRole to the service account with the following command:

oc adm policy add-cluster-role-to-user cluster-reader -z check-mk
cluster role ‘cluster-reader’ added: ‘check-mk’

5.4. Next steps

The remaining steps for the inclusion of the cluster in the monitoring are as described at the beginning of this article. However, you always use the Openshift tool as a command – oc – in the command line instead of the kubectl described in the article (e.g. when querying the service account and the token). You can output the IP address and the cluster’s port with oc status.

6. Kubernetes in Rancher installations

6.1. Create a Service-Account

With Rancher, setting up of the monitoring in Checkmk is basically identical to the setting up directly via Kubernetes variant described above. Here as well you need the service account so that Checkmk can access the cluster. You create this directly in the Rancher web interface, where subsequently you will also find its token and certificate. These you can then import as described into Checkmk.

In Rancher, first navigate to Global ➳ Security ➳ Roles ➳ Cluster to create a new role, checkmk.

For convenience, clone the Cluster Owner role.

Under Grant Resources revoke the Create, Delete, Patch and Update rights from the cloned role.

Now create a new checkmk Rancher user under Global ➳ Users ➳ Add User. In Global Permissions select the User-Base option to grant the user only the most necessary reading rights.

6.2. Assign cluster roles

Next, switch to your cluster and click on Edit in the cluster menu at the top right. Here you can use Add Member to add the newly-created user checkmk with the corresponding role checkmk to the cluster.

6.3. Next steps

Then log in to Rancher with the new user, go to the cluster and click on Kubeconfig File. Here are three details you need for monitoring in Checkmk:

  • clusters ➳ cluster ➳ server: URL/path information for the Checkmk rule.
  • clusters ➳ cluster ➳ certificate-authority-data: A Base64-encoded certificate.
  • users ➳ user ➳ token: The access password in the form of a bearer token.

You still have to decode the certificate – for example, on the command line with base64 --decode, or in one of the many online services. From here the setup in Checkmk corresponds to the procedure for pure Kubernetes use found in the chapter Importing a certificate into Checkmk.

7. Monitoring Kubernetes via the Event Console

7.1. Adding a Rancher Cluster

If you manage your Kubernetes clusters with Rancher, you can use the Event Console to monitor events in Rancher. In the Rancher interface you can easily activate the connection for an entire cluster or for individual projects.

Navigate either to your cluster or to a project under Project/Namespaces and there call Tools ➳ Logging. The configuration is identical in both cases, only the heading of the Cluster Logging or Project Logging page shows where you are. Select Syslog as the destination and first enter the Endpoint in the configuration mask – here your Checkmk server’s IP address including port 514 – for example 192.168.178.100:514 Leave the protocol as UDP. Under Program enter the desired name for the log as it should appear in the Event Console. Finally, you define the log level under Log Severity – for testing it is recommended to use Notice here in order to also get definite and immediate entries into the system.

A corresponding Event Console rule must run in Checkmk so that the data also arrives in the monitoring. For example, you can test the value Match syslog application (tag) in the area Matching Criteria to test the log names just assigned under Program.

In the Checkmk interface you can now see the events in your cluster or project in the Events view, which you can access via the Views widgets and Tactical Overview. The log name specified in the Rancher configuration under Program appears in the Application column.

7.2. Include other clusters

If the clusters were not set up with an administration like Rancher, you can have them report to the Event Console using Fluentd. Fluentd is an open source, universal logging solution that can collect data for Elasticsearch for example, but also for the syslog format. You can easily run Fluentd as a container using a Kubernetes daemon set.

First, clone the Fluentd repository:

user@host:~$ git clone https://github.com/fluent/fluentd-kubernetes-daemonset

This contains various configuration files in YAML format, and the associated Docker files. To connect to Checkmk, in the DaemonSet configuration fluentd-kubernetes-daemonset/fluentd-daemonset-syslog.yaml you only have to set the SYSLOG_HOST value in line 70. Enter the host name or IP address of the Syslog Endpoint/Checkmk server here – 192.168.178.101 for example. Leave the the port at 514, and the protocol at UDP.

fluentd-kubernetes-daemonset/fluentd-daemonset-syslog.yaml (gekürzt)
---
containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-syslog
        env:
          - name:  SYSLOG_HOST
            value: "192.168.178.101"
          - name:  SYSLOG_PORT
            value: "514"
          - name:  SYSLOG_PROTOCOL
            value: "udp"
---

Then use the DeamonSet with the kubectl tool:

user@host:~$ kubectl apply -f fluentd-kubernetes-daemonset/fluentd-daemonset-syslog.yaml

Depending on the cluster, it will take a little time until the Fluentd container is running on each node. You will then need another Event console rule which will bring the data into the monitoring. For testing purposes, the fluentd value offers itself as a filter for Match syslog application (tag) in the Matching Criteria area to get all of the events from the Fluentd instances. In this case set the application to fluentd instead of Rancher2. You will then find the result as described in above under Views ➳ Even Console ➳ Events or in the Tactical Overview – this time with the new application name: