Catch up on the latest product updates, best practices, and expert insights from the Checkmk Conference #12 – Watch the livestream recordings now
back to website

Werk #22907: New Kubernetes monitoring architecture introduced with Checkmk 3.0

Component Checks & agents
Title New Kubernetes monitoring architecture introduced with Checkmk 3.0
Date Oct 28, 2026
Level Prominent Change
Class New Feature
Compatibility Incompatible - Manual interaction might be required
Checkmk versions & editions
3.0.0b1
Not yet released 		Checkmk Community, Checkmk Pro, Checkmk Ultimate, Checkmk Cloud, Checkmk Ultimate MT

As announced in Werk #22179, the former agent_kube special agent is now being replaced. All of the work performed by agent_kube is now done by a new in-cluster agent called Checkmk Kubernetes Agent.

Main features of the new Kubernetes monitoring architecture

The new Kubernetes monitoring in Checkmk 3.0 provides new ways of connecting Checkmk to your cluster and will allow you to scale in ways that were not possible before. With the new Kubernetes Quick Setup, setting it up is easier and faster than ever before.

Quick Setup for easy deployment of the Checkmk Kubernetes Agent

Checkmk 3.0 also introduces a new Kubernetes Quick Setup which guides you through setting up Checkmk Kubernetes Agent in your cluster.

Go to Setup > Quick Setup > Kubernetes and follow the wizard filling in the necessary information, and Checkmk will generate a Helm values.yaml file for you along with the required authentication Secret manifest for your environment. Just download these files, run the commands that the Quick Setup provides, and start monitoring your cluster.

Pull mode

The former agent_kube special agent has been replaced with agent_kube_v2 which can query Checkmk Kubernetes Agent and pull monitoring data from it. Simply deploy the agent into your cluster, configure authentication, expose a Service pointing at the agent, and let Checkmk pull your monitoring data in from the agent.

Push mode

For Checkmk Ultimate and Checkmk Cloud customers, support for push mode is available, allowing you to push monitoring data from your cluster directly into Checkmk. You no longer need to expose anything in your cluster to Checkmk (over the public internet, in the case of Checkmk Cloud) in order to monitor it. The agent connects to Checkmk, not the other way around.

OpenTelemetry

For Checkmk Ultimate and Checkmk Cloud customers, support for monitoring your cluster's performance metrics via OpenTelemetry (OTel) is available, allowing you to monitor pods and their containers without creating Checkmk hosts. This reduces load on the monitoring core substantially. Enable Checkmk's built-in OpenTelemetry collector (see the user guide for its prerequisites) and configure Checkmk Kubernetes Agent via Helm to push data into the collector. You can view these metrics in custom graphs allowing you to quickly identify and troubleshoot problems that occur in your workloads. These metrics are pushed with the standard Kubernetes OpenTelemetry resource attributes, allowing you to apply filters on your graphs based on namespace, node, workload, among others.

Selective monitoring of Kubernetes objects

Ensure that your most important workloads are created as Checkmk hosts while avoiding the configuration churn caused by highly dynamic objects. Simply add a Kubernetes annotation to selectively include or exclude specific objects from host creation. This for example allows you to create hosts for selected pods while skipping the rest. The data for these skipped objects is still accounted for in the workload aggregation hosts such as node, deployment, daemonset, namespace, and cluster.

Performance and scalability

Checkmk Kubernetes Agent has been designed to work on large-scale clusters and to keep memory utilization as low as possible. It is comprised of 2 components, the node scraper and the cluster aggregator. A DaemonSet runs a very small scraper process on each node to collect and ship metrics about the node and its running workloads to the cluster aggregator. The aggregator caches the state of the cluster, so its memory usage is small by default and grows linearly with the size of your cluster.

The design of Checkmk Kubernetes Agent also places much less strain on your cluster's API server than agent_kube in the past. The agent makes use of Kubernetes watches which are a way to subscribe to updates in real-time without having to re-scan the entire cluster for the current state each Checkmk check cycle. When the agent's aggregator process begins, it will perform an initial one-time list on the cluster for all monitored kinds and then maintain that list via incremental, real-time updates for as long as the process runs. If a network interruption occurs, it will automatically negotiate with Kubernetes to synchronize internal state and "catch up" on missed events.

Migration

A manual migration is required from the old special agent to the new in-cluster agent

The new agent produces the same hosts and services as before, so you can switch without interrupting the monitoring of your Kubernetes objects.

  1. Note the cluster name used in your existing Kubernetes rule and the name of your cluster host. Use the same names below, otherwise your Kubernetes objects will be created as new hosts.

  2. In Setup, delete the existing cluster host and disable the old Kubernetes rule, but do not activate changes yet. Your current monitoring keeps running until you activate. Historic metrics are kept. Keep your existing dynamic host management connection, since it continues to manage the hosts it created.

  3. Run the Kubernetes Quick Setup with these names and deploy the new agent as instructed. If you deploy it into the same namespace as the old agent, choose a configuration name that differs from the name of your old Helm release, otherwise Helm replaces the old release instead of installing next to it. The Quick Setup uses the cluster name as the host name by default. If your old cluster host has a different name, select "Use an explicit host name" under "Checkmk host name" and enter the old name.

  4. Activate all pending changes together, then run a service discovery on the cluster host.

  5. Once the new monitoring works, uninstall the old Helm release.

If you prefer not to use the Quick Setup, deploy the new Helm chart yourself, reconfigure the existing cluster host for the new agent, and disable the old rule in the same activation.

If your previous dynamic host management connection was only used for Kubernetes, you can later delete it together with the hosts it created. The connection created by the Quick Setup then recreates these hosts. Deleting only the connection leaves its hosts unmanaged.

All configuration of monitoring data now happens in-cluster, via the Helm chart.

  • Your old (and now deprecated) Kubernetes special agent rule set is no longer used and there is no direct replacement for it in Checkmk itself.

  • Properties such as which Kubernetes kinds or namespaces are monitored are now configured via the Helm chart and require redeploying the Helm chart when modifying them.

  • An initial values.yaml can be generated via the new Kubernetes Quick Setup mentioned above. It is expected that most administrators will then maintain this file via GitOps tooling and modify it as their environment and monitoring needs change. You can see the full list of configurable options by running the command helm show values oci://ghcr.io/checkmk/charts/checkmk-agent --version '~3.0.0'.

Monitoring history and rules remain

  • Your history and rules will still apply once you set up the new agent, provided you use the same cluster name (used for generating piggyback host names in Checkmk) and cluster host name (the host that "owns" the cluster in Checkmk). Configure these during the Quick Setup wizard or using the clusterName and clusterHostName top-level fields in your values.yaml file when deploying the Helm chart.

Deprecation of agent_kube

  • As mentioned in Werk #22179, we have deprecated the old special agent (agent_kube) in Checkmk 3.0 with plans to remove it in Checkmk 3.1.

No cluster-specific specialization

  • Special support for OpenShift has been removed for the reason that it is simply no longer necessary: Checkmk Kubernetes Agent works out of the box in OpenShift without needing special security policies in the vast majority of cases. The agent is Kubernetes-flavor-agnostic and designed to work in every cluster.

Versioning

The major and minor version of the agent correspond to the Checkmk release it supports: Checkmk Kubernetes Agent 3.0.* is intended to work with all Checkmk 3.0 patch-levels; 3.1.* is intended to work with all Checkmk 3.1 patch-levels, and so on.

To the list of all Werks