Werk #19125: OTel: Add data point limit to handle high cardinality metrics
| Component | Checks & agents | ||||
| Title | OTel: Add data point limit to handle high cardinality metrics | ||||
| Date | Mar 17, 2026 | ||||
| Level | Trivial Change | ||||
| Class | Bug Fix | ||||
| Compatibility | Compatible - no manual interaction needed | ||||
| Checkmk versions & editions |
|
When Checkmk hosts created by a DCD connection monitor OpenTelemetry metrics,
a metric with many unique attribute combinations (data point attributes such as
user_id, request_id, endpoint, or pod_name) could produce tens of
thousands of data points. Previously, this caused service discovery to time
out with no clear indication of the root cause.
This affects you if your OTel instrumentation attaches high-cardinality attributes to metrics — for example:
- Per-user or per-session identifiers (
user_id,session_id,account_id) - Per-request identifiers (
request_id,trace_id) - URL paths containing IDs (
/api/users/1001,/api/users/1002, …) - Ephemeral infrastructure identifiers (
pod_name,container_id,ip_address)
A limit of 10,000 data points per metric is now enforced. If a metric exceeds this limit, service discovery will fail with an explicit error message instead of timing out:
More than 10000 data points found for metric '{name}'. Aborting to avoid overloading the system. To reduce the cardinality of this metric, either drop high-cardinality attributes at the OTel instrumentation or collector level, or narrow down the data points by configuring data point attribute filters in the DCD connection settings.
If you encounter this error, you have two options:
-
Drop high-cardinality attributes upstream — configure an attribute filter in your OTel collector pipeline to remove attributes such as
user_id,request_id, orpod_namebefore they reach Checkmk. -
Filter data points in the DCD connection — add data point attribute filters in the DCD connection settings (e.g.
method=GET) to restrict which data points are fetched per host.