Checkmk Conference #6 goes digital. Get your tickets here!
The monitoring of Windows servers was one of the first important tasks for Checkmk. As for all other server operating systems Checkmk therefore also provides its own agent for monitoring Windows. Compared to monitoring with WMI or SNMP this has several benefits – it is:
- minimalist, since it uses minimal resources of RAM, CPU and network.
- secure, because it does not read data from the network. This prevents any external access to the system.
- comprehensive, because it has access to vital data that is not acessible via WMI or SNMP.
- easily extendable, because you can write extensions in any programming or scripting language.
- open, because even though the agent is delivered as an executable file, you always have access to the source code, and you can in principle also compile the agent yorself, and have access to the functionality at any time. No unknown code is used.
The installation and setup of the agent is very simple and has only a few steps, because to function it needs no additional libraries for example. It is also delivered with a basic configuration which is sufficient for most applications.
In operation the agent consists of a Windows service which starts the supplied program – the agent. When invoked the agent collects data from the local system and provides it to the monitoring through port 6556, in the same way as other agents.
New from 1.6.0 is that the agent has been completely redesigned. This Article therefore mainly refers to the current (old) agent. The new agent basically functions in the same way as the old one. The differences are described below in their own section. There you will also find information about a migration if you have the old agent installed on your systems.
For compatibility reasons, only the major versions of Windows are supported. The following table lists these explicitly:
|NT 5.1||Windows XP||Old Agent|
|NT 5.2||Windows XP x64 Edition||Old Agent|
|NT 5.2||Windows Server 2003||Old Agent|
|NT 6.0||Windows Vista||New Agent|
|NT 6.0||Windows Server 2008||New Agent|
|NT 6.1||Windows 7||New Agent|
|NT 6.1||Windows 2008 R2||New Agent|
|NT 6.2||Windows 8||New Agent|
|NT 6.2||Windows Server 2012||New Agent|
|NT 6.3||Windows 8.1||New Agent|
|NT 6.3||Windows 2012 R2||New Agent|
|NT 6.4 / NT 10.0||Windows 10||New Agent|
|NT 6.4 / NT 10.0||Windows Server 2016||New Agent|
|NT 6.4 / NT 10.0||Windows Server 2019||New Agent|
2. Installing the agent
2.1. Various installation options
Checkmk offers several ways to install the Windows Agent – from a manual installation of the individual parts,
up to a fully-automatic deployment. Some of these possibilities are only available in the
|Supplied MSI package||Easy installation of a standard agent with manual configuration via configuration files.||X||X|
|MSI package from the Agent Bakery||Configuration over the GUI – an individual configuration of each host is possible.||X|
|Automatic Updates||The package from the Agent Bakery is first installed by hand or by script, and automatically-updated from then on.||X|
2.2. Installation via MSI package
Regardless of whether you create the MSI packages over the Agent Bakery, or configure manually on each host – you first need the installation file. These can be found under Monitoring Agents in WATO. If you are using one of the Enterprise Editions and a version that is older than 1.4.0, take the detour via the Unpackaged agents or Agent files button:
In Packaged Agents you can find the required file check_mk_agent.msi. Download this file to the host and start the installation. In principle, you just have to follow the menu, and read and agree to the license terms of GNU GENERAL PUBLIC LICENSE. In the Destination Folder menu item you can specify an alternative file path over which the agent is to be installed. Otherwise it will be installed using the default path %ProgramFiles (x86)%\checkmk\. This path is used for compatibility reasons, and is independent of whether the agent is to be installed on a 32-bit or 64-bit operating system. The installation routine automatically selects the right agent.
After installation the agent will be started immediately as a Windows service, and be ready to monitor the system.
Windows provides the ability to automate installations of MSI packages via msiexec. An automated installation might look like this, for example:
C:\Users\hhirsch\Downloads\> msiexec /i check_mk_agent.msi /qn
In some cases Checkmk can not access a Windows host, even though the package was installed correctly and the service is actually running (see also in the section about the error diagnosis). In such cases the firewall could be the problem. Unfortunately the agent itself can not test if it is accessible from the outside. Therefore check, and if necessary, set a firewall rule for the agent in the Windows Firewall with Advanced Security (WF.msc). Alternatively, you can also perform this step and set the rule directly on the command line. If necessary customize the following command to your custom installation path:
C:\Windows\System32> netsh advfirewall firewall add rule name="Check_MK" ^ More? description="Monitoring" dir=in localport=6556 protocol=tcp action=allow ^ More? program="%ProgramFiles(x86)%\checkmk\service\check_mk_agent.exe" ^ More? profile=private,domain enable=yes OK.
2.3. Installation using the Agent Bakery
2.4. Automatic updates
If you use the Agent Bakery you can set up automatic updates for the agent. This is described in its own article.
3. Architecture of the agents
The agent’s directories
The agent is divided over two parts of the file system:
- C:\Program Files (x86)\checkmk\service\: Program-specific files are installed here. Customization is not necessary.
- C:\ProgramData\checkmk\agent\: Host-specific files are stored here. The behavior of the agent is configured here, and plug-ins, logs, etc., are also stored in this directory. Note: This directory is normally marked as invisible by the system.
The agent’s configuration files
For the agent configuration three files are read sequentially and hierarchically:
- C:\Program Files (x86)\checkmk\service\check_mk.yml: The default configuration is stored here. You must not change these.
- C:\ProgramData\checkmk\agent\bakery\check_mk.bakery.yml: This file is created by the agent bakery, and it may override a default value from the previous file.
- C:\ProgramData\checkmk\agent\check_mk.user.yml: In this file you can make manual customizations to test a setting or extension on a host. This file is read in from the Bakery after configuration, and overwrites it if necessary.
As you may already have recognised by the of the configuration files’ file extension, YAML is used for the configuration format. We decided, starting from version 1.6.0, that this format makes it easier to use structured data than with the classic INI format.
For the manual work with the agent, only the last configuration file (check_mk.user.yaml) is relevant, because it is the last to be read in and thus it has the last word. If the agent bakery is not used, this is in fact the only file in which customization with the configuration of the agent may be made.
4. Installation of the old agent
4.1. Why a second agent?
In earlier versions of Checkmk the agent had a different architecture. This has been working well for a long time, and from version 1.6.0 it has been replaced by a new one in order to tidy up loose ends, to simplify the configuration, and ultimately to have better tools on hand – for example for help in being better able to track down configuration errors.
For compatibility reasons the old agent is still included in Checkmk because only it can reliably monitor those old platforms like Windows XP and Windows 2003. These two systems will no longer by supported by the new agent. In addition, having an already-installed old agent should make a migration to the current one easier. This is still compatible with Checkmk, so that an update of your Checkmk server to version 1.6.0 does not automatically require an agent update.
4.2. Special features of the agent up to version 1.5.0
The old Windows agent differs from the new agent in the following points:
- Different usage of directories. In the old agent the installation directory and the configuration directory are the same. Only the C:\Program Files (x86)\check_mk\ directory will be used.
- As a result the available plug-ins are not automatically installed, but must be individually downloaded from the Checkmk server and correctly stored.
- In the old agent the configuration is recorded in an initialization file (check_mk.ini). The default configuration and the Agent Bakery use this identical file. Local customizations can be made using the check_mk.user.ini file, which must be in the same directory.
- The possibilities for going deeper into the agent are very limited.
4.3. Migration to the new standard agent
The migration from an already-installed agent to the new 1.6.0 agent is very simple. Simply call the installation package for the new agent (check_mk_agent.msi), and follow the instructions as usual. During the installation you will always be asked if existing installations of the old agent should also be directly uninstalled. This option is not enabled by default:
If to be on the safe side you still want to keep the old agent, do not use this option. The old agent will then only be stopped and be disabled. Regardless, the configuration of the old agent will, as a part of the installation process, be transfered to the new format and stored as a user configuration (check_mk.user.yml). This gives you the option of checking the conversion with the aid of the original file. Once you are sure the conversion was successful, you can uninstall the old agent.
You would probably only want to make such a manual check of individual hosts in a group. If you are sure that the conversion works correctly, you can accordingly uninstall the old agent automatically. You will then save yourself the manual uninstallation, and can simply replace the old agent with the new one.
5. Testing and fault diagnosis
5.1. Checking the configuration
To check if the configuration was read in as you expected, call the agent with the showconfig option. With this option you not only get an output of the configuration as it is currently being used by the agent – the environment variables in use, as well as the configuration files will always be shown.
If only a certain part of the configuration is of interest, restrict the output to a specific part. Here, for example, it is checked whether the options for the ‘ps’ section have been set correctly:
C:\Program Files x86\checkmk\service> .\check_mk_agent.exe showconfig ps # Environment Variables: # MK_LOCALDIR="C:\ProgramData\checkmk\agent\local" # MK_STATEDIR="C:\ProgramData\checkmk\agent\state" # MK_PLUGINSDIR="C:\ProgramData\checkmk\agent\plugins" # MK_TEMPDIR="C:\ProgramData\checkmk\agent\tmp" # MK_LOGDIR="C:\ProgramData\checkmk\agent\log" # MK_CONFDIR="C:\ProgramData\checkmk\agent\config" # MK_SPOOLDIR="C:\ProgramData\checkmk\agent\spool" # MK_INSTALLDIR="C:\ProgramData\checkmk\agent\install" # MK_MSI_PATH="C:\ProgramData\checkmk\agent\update" # Loaded Config Files: # system: 'C:\Program Files (x86)\checkmk\service\check_mk.yml' # bakery: 'C:\ProgramData\checkmk\agent\bakery' # user : 'C:\ProgramData\checkmk\agent\check_mk.user.yml' # ps enabled: yes use_wmi: yes full_path: no
5.2. Testing the agent
There are several ways under Windows to test the agent’s functionality. With the help option you will get a detailed overview of which diagnostic options the agent provides. The most important of these are presented here.
Use the test option to directly run the agent locally and see immediately if an output can be generated without errors. To save space only the first lines of output are listed in the example here:
C:\Program Files x86\checkmk\service> .\check_mk_agent.exe test <<<check_mk>>> Version: 1.6.0b8 BuildDate: Sep 4 2019 AgentOS: windows Hostname: MSEDGEWIN10 Architecture: 64bit WorkingDirectory: C:\Program Files (x86)\checkmk\service
Testing from the monitoring server
If a problem does not exist locally, with the -io option you have another way to check the agent from outside. This option starts the agent as a short-term service, and then logs every connection made to this service from outside. In this way you can verify if a request really has reached the host. Please note that for this test to work the agent’s Windows service must not be running – therefore stop the service before carrying out the test:
C:\Program Files x86\checkmk\service> .\check_mk_agent.exe check -io testing 10 seconds Starting IO ipv6:false, used port:6556 Connected from '192.168.42.1' ipv6 :false -> queue Put on queue, size is  Found connection on queue, in queue left Connected from '192.168.42.1' ipv6:false <- queue No data to send Shutting down IO... Stopping execution Exiting process queue cma::world::ExternalPort::ioThreadProc: terminated from outside IO ends...
5.3. Further debugging options
The agent offers more options for finding out many details about the concrete behavior of the agent than those already described. With the help option, among other things you get a detailed and complete list of the available options beyond those described here.
6. Incorporation of classic check plug-ins
6.1. The basic configration
Under Windows you can continue to use its Nagios-based plug-ins on a host if there is no counterpart in Checkmk. The mechanism for this is quite simple – you use the MRPE feature of Checkmk which behaves analogously to the NRPE of Nagios.
The inclusion of MRPE plug-ins is enabled by default. If you do not want to use this feature you can deactivate it in the configuration file by adding the following definition:
mrpe: enabled: no
Limiting the execution time
Sometimes the runtime of a script or Nagios plug-in is unpredictable, and in the worst case, a plug-in never ends. To maintain control here you can limit the maximum runtime of the MRPE plug-ins. The value shown here is also the default value in seconds. So adjust it only if you want to set a shorter or longer interval:
mrpe: # enabled: yes timeout: 60
6.2. Executing plug-ins via MRPE
To tell the agent where the file to be run can be found, and how to call it, add an entry in the MRPE configuration:
mrpe: config: - check = MyServiceName 'C:\ProgramData\CheckMK\Agent\mrpe\my_check_plugin.bat' -w 10 -c 20 MyParameter
It is not a requirement that the file be stored in the the agent’s directory, even if it makes sense to store everything in a common location. In this configuration example you can see the following elements for the relevant line:
|MyServiceName||The service name as it should be displayed in Checkmk|
|'C:\ProgramData\CheckMK\Agent\mrpe\my_check_plugin.bat'||The script or program to be called. Since file paths and filenames under Windows may contain spaces, the parentheses mark the identity of the expression.|
|-w 10 -c 20||Options have been passed to this script – in this case a threshold of 10 for WARN, and a threshold of 20 for CRIT.|
|MyParameter||To finalise, the script has been given a parameter that does not belong to a specific option.|
After you have set up the MRPE plug-in, it will be immediately active, without needing a reboot of the agent, and it will be added to the output. In the service discovery you will now automatically find your new service:
6.3. MRPE with the Agent Bakery
Alternatively to configuring directly on a host in the user-specific configuration file, you can also define your MRPE plug-ins directly in the web interface. Use the Monitoring Agents ➳ Generic Options ➳ Execute MRPE Checks ruleset for this. The necessary entry is then generated automatically in the Bakery configuration file.
7. Extending agent plug-ins
7.1. What are plug-ins?
The standard agent contains a whole series of sections which provide monitoring data for various check plug-ins, which are then found automatically by a Service Discovery and output as services. Above all, the important monitoring of the operating system belongs to these.
In addition there is the possibility of extending agents with agent plug-ins. These are small scripts or programs called by the agent to extend the agent with additional sections of monitoring data. The Checkmk project already provides quite a few such plug-ins which – if they are installed and configured correctly – with a service disovery also automatically create new services.
Why are these plug-ins not tightly-integrated with the standard agents? For each of the plug-ins one of the following reasons applies:
- The plug-in can only get its data via internal interfaces which the standard agent does not provide (Powershell for example).
- The plug-in in any case needs a configuration, without which it would not work (mk_oracle.ps1 for example).
- The plug-in is so specialised that most users do not need it (citrix_licenses.vbs for example).
7.2. Installing plug-ins manually
As already mentioned, Checkmk provides a whole series of plug-ins for Windows as standard. You can find these on the monitored host in the agent’s installation directory. There all available plug-ins are always stored directly with the agent so that they are also directly available for use: C:\Program Files (x86)\check_mk\service\plugins. Alternatively you can also find the plug-ins on the Checkmk server itself in local/share/check_mk/agents/windows/plugins. These are also available via the agents download page in WATO under the Windows Agent - Plugins box (as described at the beginning of this article):
For all of the standard agent plug-ins we provide there are also suitable matching check plug-ins which can evaluate the collected data and generate services – thus you do not have to install any extras on the Checkmk server.
Important: Have a look at an agent plug-in before you install it on a host – you will often find important information on the correct use of the plug-in.
The actual installation is then easy. Copy the desired plug-in either from the Checkmk server or from the installation directory to C:\ProgramData\CheckMK\Agent\plugins. If the plug-in is in this directory it will be called automatically by the agent, and a new section will be created in the agent’s output. This section usually has the same name as the plug-in. Complex plug-ins – for example, mk_oracle.ps1 – even create a whole series of new sections.
7.3. Configuring plug-ins
Some plug-ins require a configuration file in C:\ProgramData\CheckMK\Agent\config for them to work correctly. For others a configuration is optional – for example mssql.vbs) – this allows for special features or customizations. Other plug-ins function without further steps because they have different sources for their information:
- Documentation of the associated check plug-ins in the WATO module Check plugins
- Comments in the plug-in itself (often very helpful!)
- A matching article in this guide – for example, on monitoring Oracle
7.4. Customizing the execution of a special plug-in
Each plug-in can be executed in different modes. The following options are available. The setting printed in bold is the default value:
|pattern||'@user\*.ps1'||Sets the ranges of the following options. Here you can also work with wildcards. The options below will then apply to all plug-ins that match the expression. It is determined upfront whether the plug-in should be executed directly from the installation directory or from the data directory.|
|run||yes/no||Determines whether the execution of a plug-in should be suppressed.|
|async||yes/no||Executes a plug-in asynchronously and stores the data in a file. With synchronous execution, the output is passed directly to the agent.|
|timeout||60||Sets the maximum execution time – if this is exceeded the plug-in is terminated, even if no output has been produced. The default value orients itself to the agent’s default query interval.|
|cache_age||60||Specifies in seconds how long an output is valid. If async is activated, a cache of ??? seconds is created.|
|retry_count||1||The number of times a plug-in can fail before discarding an output from the cache.|
|description||‘Text’||Here you can enter a free text/comment to add to the logs.|
A configuration for the Veeam plug-in looks like this, for example. This extract is truncated here and contains only the part relevant for this example:
plugins: enabled: yes execution: - pattern: $CUSTOM_PLUGINS_PATH$\veeam_backup_status.ps1 async: yes timeout: 120 cache_age: 300 retry_count: 2
The plug-in, as defined above, will be executed asynchronously every five minutes (300 seconds), and may not exceed two minutes (120 seconds) running time. If the plug-in runs in this timeout, it will make a second attempt to get a result.
7.5. Installing plug-ins using the Bakery
The plug-ins supplied by Checkmk can be configured via the Agent Bakery. This ensures both the installation of the plug-in itself, as well as the correct creation of the configuration file if one should be necessary.
Each plug-in is configured via an agent rule. You can find the matching rule sets in Monitoring agentes ➳ Agent plugins:
7.6. Executing plug-ins manually
Since agent plug-ins are executable programs, you can also execute them manually for testing and diagnostic purposes. There are plug-ins however which need certain environment variables set by the agent, e.g. to find its configuration file. If necessary, set these variables by hand if they are needed in a script or program.
8.1. Preliminary considerations
As with the Linux agent, access to the agent for Windows also needs to be secure. After all, these are potentially sensitive servers that must be protected from external attacks. For this reason, the same basic principles as under Linux also apply here. Likewise with Windows, the agent does not read any data from the network, so an attacker can never infiltrate commands or scripts via the monitoring port 6556.
If the monitored system is queried over an insecure (internet) connection, additional measures will be necessary. The agent has an optional built-in encryption of the transmitted data to protect it from attacks affecting network traffic. On newer Windows versions the native SSH is additionally possible, so that an encryption over the entire connection duration can be guaranteed, as already familiar under Linux.
8.2. Restricting access via IP addresses
The restriction to certain IP addresses can certainly be configured via the Firewall, additionally however, the agent also has the ability to simply ignore requests from foreign IP addresses. Just add the following restriction to the global options in the configuration file. Note that other parameters can be set before or after this in the configuration file and only the relevant excerpt is shown here:
global: only_from: 127.0.0.1/32 192.168.42.73/32
As you can see in the example, in principle you can allow as many subnets as you like. For example, with a /32 enter a subnet of size 1, so that only this single address is allowed, while the use of 192.168.42.0/24 will allow all addresses between 192.168.42.0 and 192.168.42.255.
In the Agent Bakery you can configure the permitted IP addresses with the Monitoring agents ➳ Rules ➳ Generic options ➳ Restrict agent access via IP address rule set in WATO.
Of course an attacker can very easily fake his IP address and thus get a connection to the agent. However it is very unlikely that he will get an answer – because any answer will be routed to the real monitoring server. Or if the attacker actually gets an answer the CMK server receives no data and will very quickly report an error.
8.3. Calling via SSH
Newer versions of Windows have native support for SSH, but even with older versions, using Cygwin you can retrofit an SSH server and replicate an identical configuration to that possible under Linux. For the setup please see the latest Cygwin or Microsoft support. Once an SSH server has been started and is reachable, further setup is identical to that under Linux – you set up the authorized_keys on the monitored host, and restrict access to the execution of the agent.
8.4. Built-in encryption
From Checkmk version 1.4.0 the Windows Agent (and also its Linux counterpart) encrypts its data without any additional tools. This is not strictly a substitute for access control – but since an attacker cannot send commands and cannot do anything with encrypted output data anyway – it comes very close to one.
The effort in using the encryption, and the additional CPU load necessary, are both less than the method using SSH as described above – which we however still recommend using for transmission over the internet.
The encryption of course requires a suitable configuration both on the agent and on the server.
This can either be done by hand (
Setting up without the Bakery
Even without the Agent Bakery, the first step is performed in WATO. Create a rule in the Host & Service Parameters ➳ Access to agents ➳ Encryption rule set. The rule should apply to all hosts for which you would like to use encryption. SNMP hosts ignore this setting, so you do not have to explicitly exclude it.
The Encryption for agent setting is important. As long as you leave the rule on the default Disable setting, everything of course remains unaltered. You can choose between:
- Enable: Encryption is enabled, but data from agents without encryption will continue to be accepted.
- Enforce: Encryption is turned on, and only encrypted data is accepted.
It makes sense to start with Enable first. As soon as you believe all of the agents are using encryption, switch to Enforce to find any hosts which still send data in clear text.
The encryption works with a common password that you specify here, and which must be saved in plain text (‘shared secret’) on the Checkmk server as well as in the agent’s configuration. Choose a random password and keep it ready for the second step – configuring the agent.
On the Windows server add the password to the agent’s configuration. This also goes into the global options:
global: encrypted: yes passphrase: MyPassword
Now you can perform the following tests (see also the article on the command line in Checkmk:
- A call to check_mk_agent on the target system must produce a jumble of characters.
- A telnet myhost123 6556 from the Checkmk server must issue the same jumble of characters.
- A cmk -d myshost123 on the Checkmk server must display the clean plain text data.
Setting up with the Bakery
Setting up encryption with the agent bakery is very easy. With the creation of the rule as just described you are basically done. You only need to bake and distribute the new agents. The file /etc/check_mk/encryption.cfg will be automatically-created and built into the agent packages for you.
9. Monitoring Windows with SNMP
There are a few situations where monitoring via SNMP in addition to the normal agent may be useful. And that is in fact the case if either your own application software or a hardware monitoring tool from the server’s manufacturer can only provide monitoring data via SNMP and – either due to the Windows version in use, or because there are no commandlets for the application – a query through Powershell is not possible.
In such a case, add the setting SNMP for the appropriate connection type (snmpv2/3 or snmpv1) to the properties of the host in the DATA SOURCES box in WATO . In versions older than 1.5.0 the box is called Host tags and will be altered to Dual: Check_MK Agent + SNMP. Services which are available via both SNMP and Checkmk agent (e.g. CPU utilization, file systems, network adapters) will then be fetched automatically from the Checkmk agent and not via SNMP. This will automatically avoid a double transfer.
10. Files and directories
10.1. Paths on the monitored host
|C:\Program Files (x86)\checkmk\service\||Installation directory for the program-specific files. The actual agent check_mk_agent.exe is also stored here.|
|C:\ProgramData\checkmk\agent\||Installation directory for the host-specific files. Here are extensions, logs and configuration files specific to this host.|
|C:\ProgrammData\checkmk\agent\check_mk.user.yml||Configuration changes made by the user are stored here.|
|C:\ProgrammData\checkmk\agent\bakery\check_mk.bakery.yml||Configuration alterations from the bakery are stored here.|
|C:\ProgrammData\checkmk\agent\plugins||Here plug-ins which are to be executed automatically by the agent are stored.|
|C:\ProgrammData\checkmk\agent\local||The directory for your own local-scripts|
|C:\ProgrammData\checkmk\agent\mrpe||MRPE extensions can be saved here.|
|C:\ProgrammData\checkmk\agent\backup||The user configuration creates a backup here after each change to the Checkmk agent service.|
10.2. Paths on the Checkmk server
|local/share/check_mk/agents/custom/||Base directory for own files that should be delivered with a baked agent.|
|share/check_mk/agents/windows/||The agents and their MSI packages are stored here. In this directory you will also find configuration examples and all plug-ins for the agent.|