Checkmk Conference #6 goes digital. Get your tickets here!
From version 1.6.0 Checkmk includes the new concept of Labels. A host can contain any number of Labels, and these are very similar to Tags:
- Labels are ‘attached’ to hosts in the same way as tags.
- Labels can be used in the same way as tags as conditions for rules.
- Labels are constructed similarly to tags on the key:value principle.
So why is there a new concept here? Well - the IT world is changing and becoming much more dynamic. Cloud and container systems like Azure, AWS and Kubernetes autonomously create and delete objects – which correspond to hosts in Checkmk. In these technologies tags/labels play a big role because they make the connections between the monitored objects and their functions. The host names, on the other hand, are increasing random and meaningless.
Checkmk can create such dynamic hosts automatically with the new Dynamic Configuration Daemon, and also receive information about any labels/tags that are already present. These labels/tags can then be used for rule conditions, searches, evaluations, and other tasks.
Of course the question arises – why we do not simply map such dynamic labels to the existing concept of host tags. In fact that is also a very obvious conclusion at first look. However host tags have a very important property which would make that very difficult and complicated: Checkmk rigidly defines which tag groups and tags are present. Everything is well-defined. Every host has exactly one tag from every group. Everyone can rely on it. Typing errors in the spelling of tags cannot occur – not even with hosts which do not stick to the scheme – because its compliance is strictly controlled by WATO. This is very important and useful in very heterogeneous environments with many thousands of manually-managed hosts.
By contrast, dynamic labels from Kubernetes and Co are effectively ‘freeform’, and even if they do follow a scheme, this is unknown to Checkmk. In addition you might be monitoring several different platforms, which in turn use labels in very different ways.
That is why a new Checkmk-Labels concept has been introduced which suits this growing dynamic. Of course you can also use the labels even if you don’t use connections to cloud environments.
Here are the features of Labels:
- Labels do not have to be predefined anywhere. There is no fixed scheme for labels. Everything is free form. Everything is allowed.
- Each host can have as many labels as you like. These can be maintained manually, defined via rules, or created automatically.
- Labels are structured according to the key:value principle. Each host may only have one value per key. So a host that has the label foo:bar cannot at the same time have foo:bar2.
- Unlike the host properties, both the key and the value - except for the colon (:) - may contain any characters.
- There is no distinction between ID and title (display name). The name/key of the label is at the same time its ID and display name.
Labels have the following tasks:
- They form a basis for conditions in configuration rules (for example, all hosts with the label os:windows should be monitored in the same way).
- It is very easy to store additional information or comments about a host (for example, Location:RZ 74/123/xyz) and to display this in views for example.
- Labels are passed to Grafana over the new connector and can be used there for filtering, aggregation, etc.
2. Creating Labels
2.1. Explicit Labels
A host can be assigned labels in three different ways. The first of these is simple: In the mask in which you create a host, you can arbitrarily define as many labels as desired:
To specify the labels, activate the checkbox attribute as usual, click on the Add some label text, and there define your labels. Enter the labels according to the Name:value scheme, and close each label by pressing enter.
You can edit an existing label by double-clicking it in its text, or delete it with the little cross.
Note: Both the name and the value of a label may include any character - except the colon (:)! However you should think carefully about the use of characters and upper/lower case, since if you later define conditions via labels, then the spelling of both the name and the value must be strictly observed.
Of course labels can also be inherited from a folder. Like other attributes, labels can be in subfolders or at the host, then as needed be overwritten again - individually in fact. If for example, the label location:munich is set in the folder, this will be inherited by all hosts in this folder which do not themselves already have the label location defined. Other labels a host may have will remain unaffected.
Explicitly-defined labels for hosts or folders appear violet in the list of hosts:
2.2. Creating labels via rules
As usual in Checkmk, attributes can also be assigned to hosts and services by rules. This will make you independent of the folder structure, and it also applies to the labels. There is a Host labels rule set for this function.
The following rule adds the hw:real label to all hosts in the Bayern folder which have the Hardware type is real hardware host tag:
You may have noticed that labels cannot be used in the conditions for this rule! This is not a mistake, but intentional, and it avoids recursive dependencies and other anomalies.
2.3. Labels in the discovery check
If you have enabled the discovery check – which is the default for new installations – it will warn you when new host labels are found that have not yet been added to the WATO properties of a host. This will look like this, for example:
You have two options for responding to this warning. The first is to add the new labels by calling the service configuration of the host in WATO and updating the configuration of the labels with the Update host labels button. The discovery check will then be OK again the next time it runs (up to a two-hour delay), even if you have not yet activated the changes.
If this affects many hosts at once, you will certainly not want to visit the service configuration for each one. The best way to do this is to run Bulk discovery and select the Add unmonitored services and new host labels mode.
The second way to get the discovery check green is to reconfigure it so that it no longer prompts for new labels. To do this, go to the Monitoring Configuration ➳ Inventory and Check_MK settings ➳ Periodic service discovery ruleset, and edit the existing rule – there you will find the Severity of new host labels option:
2.4. Sequence of label assignment
Theoretically, the same label may be defined with different values in multiple sources simultaneously. That's why there is the following order of priority:
- First of all, explicit labels, i.e., those that you define for the host or folder directly in WATO.
- In the second place are labels that are created by rules.
- In the last place are automatic labels.
These precedence rules give you the ultimate control over the labels.
3. Labels as conditions in rules
An important function of labels is the same as with tags, namely their ‘Use’ condition in rules. This is especially true for automatically-generated labels, because they perform their monitoring fully-automatically according to information from AWS, Azure, Kubernetes and co.
The following example shows a rule condition that only applies when the host has the label state:bavaria, but not the label environment:test:
You can use both labels and tags in a rule. These will be automatically AND-linked. The rule only applies if both conditions are met simultaneously.
Please note that the exact spelling of the labels is important. Since labels are freeform, and therefore WATO cannot know exactly which labels really exist, it cannot recognize typing errors. If that causes isolated problems it may be more effective if you work with tags, since these work with selection boxes instead of with text input.
4. Labels in Views
So far we have only talked about the configuration. The labels are also visible in the monitoring itself. This starts with the host details:
Since the labels are also clickable, they are not just for appearance: With a click you will be forwarded to a search for all hosts with this particular label. You can also do something similar in the Views’ search function – here there is a new search box that will enable you to for search for labels. The entry is made here using an interactive search for all existing labels:
5. Service Labels
Services can also have labels. These are similar to the host labels, however with a few small differences:
- You cannot define service labels explicitly. These can only be created by rules (Service labels), or automatically.
- You cannot currently formulate any conditions via service labels, however this will soon be possible.
6. Labels in Grafana
For Grafana Datasource is currently being developed with which you can access the historic metrics from Checkmk directly from Grafana. If you use these Grafana automatically receives the information about all host and service labels. This allows you to more easily group Checkmk metrics and work with templates in Grafana.