Monitoring VMware with Checkmk

Checkmk can create the monitoring hosts for your virtual machines (VMs) automatically. For the host names, it uses the display names from VMware vCenter. A consistent naming policy with DNS names is thus an important basis for this tutorial. You should follow this tutorial regarding naming concepts. It explains the criteria you should consider when naming your systems. However, do not change your systems' display names in VMware without first reading the linked article to the end.

The first step is to create a folder in Checkmk to manage your VMs in monitoring.

• Under Setup -> All hosts click Add folder and enter a name of your choice under Title. I named the folder ‘vm’.
• Check the box at Checkmk agent / API integrations and select the ‘Configured API integrations and Checkmk agent’ option.
• Click Save.

You have now created a folder in Checkmk and defined that Checkmk expects the data from APIs and the agent for all hosts in this folder. This will still allow you to later install the Checkmk agent on the VMs to retrieve additional monitoring data.

Now add vCenter to this folder as the first host.

• Open the folder you have just created by clicking on it.
• Click on Add host.
• Enter the display name of vCenter under Hostname. In my example, ‘muc-vm-vcenter-001.lan.domain.net’.
• Since the display names are DNS resolvable, you can leave all other settings as is and click 'Save & go to folder'.

If you have multiple vCenter servers in use, you can repeat this step, editing the host name for each vCenter server. However, depending on the distribution and structure of your IT environment, it may also make sense to set up the folder structure in Checkmk differently and, for example, first create a folder for each location. In that case, you should again use a separate folder with VMs for each location and not create all VMs from all sites in one folder.

If your display names in VMware are not resolvable by DNS, you would have to store vCenter's IP address under IPv4 address in Checkmk before proceeding. A non-stringent naming concept always leads to problems in the long run and makes it difficult to correctly assign monitoring data later, for example. I therefore recommend tailoring your naming concept according to the instructions linked at the beginning of this tutorial.

Now you need to grant Checkmk access to vCenter, and decide what data it should retrieve from there.

• Click again on Setup and search for ‘vmware’.
• Then click the ‘VMWare ESX via vSphere’ rule.
• Click on Add rule.
• Under VMWare ESX via vSphere, edit the items in order: Under vSphere User name, enter your prepared user account for vCenter. In my example this is ‘monitoring@vsphere.local’ followed by the password under vSphere secret.
• Since you are using VMware vCenter, you must set the Type of query selection to ‘Queried host is the vCenter’.
• You only need to check the box at TCP Port number if you have set the port for HTTPS to a different port, but this is not necessary in most environments.
• For SSL certificate checking you can stay with the default ‘Use hostname’ setting if you are using a certificate issued by a certificate authority for vCenter. If this is not the case, you would have to select the ‘Deactivated’ option under SSL certificate checking. Keep in mind that by default, VMware vCenter starts with HTTPS and a self-signed certificate.
• With the Connect Timeout option, you can tell Checkmk to abort in the event of excessive response times. You won't normally need this option, so don't check it.
• Under Retrieve information about, decide for which systems Checkmk should retrieve data from vCenter. Deselect the ‘Host Systems’ option so that the ESXi hosts do not end up in your VMs folder. You will create these hosts separately later. ‘License Usage’ allows you to retrieve the overall status of vCenter licenses in use. This is only required for very large datacenters with over provisioned licenses, so I do not use this option here.
• Leave the options for monitoring Placeholder VMs and Display ESX Host power state on disabled. You should, however, enable the VM snapshot summary option here, since forgotten snapshots can sometimes lead to dramatic problems.
• By default, Checkmk takes the display names configured in vCenter as the host names. Under Piggyback name of virtual machines, you can tell Checkmk to use the names as stored in the ESXi hosts. I don't need this option in this tutorial and don't recommend using it, as the VMware interface is prone to errors here in practice.
• Checkmk does not allow spaces in host names, but display names in VMware can theoretically contain blank characters (spaces). If you are using DNS names in VMware (according to my recommendation), you cannot include any spaces. If you have nevertheless used spaces in VMware, you can use Spaces in hostnames to decide how Checkmk should handle them.

• Under Conditions, select your vCenter server under ‘Explicit hosts’. If you have multiple vCenter servers in use and use the same credentials for all read-only accounts, you can query all vCenter servers together with a single rule. Simply add the appropriate vCenter hosts here. Alternatively, you will need to configure a new rule for each vCenter server and adjust the credentials and explicit host for each.
• Click Save to save the rule.

Checkmk can now retrieve the data from vCenter via the VMware API. For this, you should start the automatic service discovery in Checkmk for your vCenter host.

• Go to Setup -> Hosts and click on your folder. You should now see the vCenter host.
• Click on the yellow icon for your vCenter host right in the middle of the toolbar.

• Wait for Checkmk to complete the automatic service discovery and click Accept all.
• Important at this point: Do not activate the ‘pending changes’ yet, for now read this chapter to the end.

You have now included all VMs as monitoring services – and perhaps more than that. For example, the VMware API is not able to distinguish between templates and VMs. Therefore, if you have created templates in vCenter, in the monitoring Checkmk will identify your templates as powered off VMs in addition to the real VMs and would by default assume an error.

You could manually disable the individual services in the templates following the service discovery. However, if you use a consistent naming concept in VMware, you will reach your goal faster with the disable service rule in Checkmk. This will also exclude any templates that you may create in the future from the monitoring.

For my purposes, I have prefixed all template display names in VMware with ‘template’. This makes removing the template services from monitoring child's play:

• Click on Setup. Search for ‘disable’ and click on ‘Disabled services’.
• Click on Add rule.
• Under Conditions, select the Explicit hosts option and enter your vCenter host there. If you use multiple vCenter servers, you can enter multiple hosts here. In my case, I enter ‘muc-vm-vcenter-001.lan.domain.net’.
• Then activate the checkbox at Services. Based on the service names, you can now disable services. I use the regex ‘VM template’. This disables all services with the prefix ‘template’ for vCenter. Note that Checkmk adds the abbreviation ‘VM’ prefixed to the VM service name.
• Save the rule with Save.

• Now, start the service discovery for the vCenter host again as before under Setup -> Hosts. Click again on the yellow icon in the middle in front of your vCenter host.
• Compared to previously, your template services are now all disabled. Click Accept all.

Checkmk uses the piggyback mechanism, with which it transfers information based on the host names to the appropriate hosts in the monitoring. This can be, for example, the data on the individual VMs from the ESXi host or vCenter.

However, to do this, you must first create the VMs as hosts in Checkmk, since they are currently only visible as services within your vCenter server. The host names in Checkmk must correspond to the display names in VMware. Only then can Checkmk correctly assign the piggyback data.

You could create the hosts for the VMs manually, but it can also be done more easily via the Dynamic Configuration Daemon (DCD) mentioned at the beginning of this tutorial, with which Checkmk can create the hosts for your VMs automatically:

• Under Setup, search for ‘dynamic’ and click ‘Dynamic host management’. Click on Add connection.
• Under General Properties, choose any Unique ID and a Title. For my example, I chose ‘vmware-vcenter’ and ‘vm’.

• Under Connection Properties, you must click ‘Add new element’ once to allow Checkmk to automatically create hosts.
• In addition, you must adjust the advanced options at Connection Properties. To do this, click on ‘show more’.
• Under Piggyback creation options, select the destination folder for your VMs. In my case ‘vm’.
• At Hosts attributes to set, attributes for Monitoring agents: SNMP, Monitoring agents: Checkmk agent / API integrations, Monitoring agents: Piggyback and Network address: IP address family are predefined. Delete the four defaults, since you do not need these for the VMware monitoring. To do this, click once on each of the four X symbols for the individual attributes, so that they disappear.
• Enable the Only add matching hosts option and use regex to create hosts only for the VMs. If you followed the link to the recommendations for a naming concept, you can enter " .*-vm" here. Leave the other settings as they are.
• Click on Save.

Now retrieve the monitoring data from the ESXi hosts. To do this, first create a folder for the ESXi hosts.

• Under Setup -> All hosts you again create a folder with Add folder. As its Title I use ‘esxi’ here. You can use a name of your choice.
• Click on Save.

You cannot install Checkmk agents on ESXi hosts, so you can keep the default setting for this folder.

Create a rule to be able to retrieve the data via the VMware API:

• Click Setup and again search for ‘vmware’. Click on ‘VMWare ESX via vSphere’.
• Click on Add rule.
• As the vSphere User name, you use your read-only account. For my account, this is called ‘checkmk’ for all ESXi hosts. This is followed by the appropriate password under vSphere secret.
• For Type of query you retain the default settings ‘Queried host is a host system’.
• Connect Timeout can be left as is.
• Under Retrieve information about... there are important differences compared to the previous configuration for vCenter. You now leave ‘Host Systems’ and ‘Performance Counters’ enabled, but uncheck ‘Virtual Machines’ and ‘Datastores’. You already get this information from vCenter.
• Piggyback name of virtual machines and VM snapshot summary also remain disabled.

• This time select the folder you just created for the ESXI hosts under Conditions at Folder. In my example this is ‘esxi’. Since I use the same credentials for all ESXi hosts, I can apply the rule to the folder. If you use different credentials, you must define the rule with the respective credentials for each ESXi host.
• Click Save to store this rule.

You can now add your hypervisors as hosts in Checkmk. In this tutorial, I manually add them as hosts to the monitoring. If you have more than five or ten ESXi hosts, you can import them via a CSV file. Follow these instructions in the Checkmk manual.

• Go to Setup -> Hosts and select the folder to which you applied the rule for the ESXi hosts, in my case ‘esxi’.
• Click on Add host.
• As the Hostname enter the display name for the first ESXi host, for my example ‘muc-sr1-esxi-001.lan.domain.net’.
• Click on Save & go to service configuration.
• Wait until Checkmk has completed the automatic service discovery.
• Click Accept All, but do not close this view yet.

For ESXi hosts, you should disable the ‘Hardware Sensors’ monitoring services for all ESXi hosts at this point. Since you later monitor the server hardware via the server's management board, you do not need VMware's hardware monitoring. Moreover, this does not always work reliably and often generates false alarms.

• Remove the ‘Hardware Sensors’ monitoring service again. Simply click on the red X symbol in front of the ‘Hardware Sensors’ monitoring service. This will then be moved to the Disabled services area.
• There, click on the action menu, the third icon on the ‘Hardware Sensors’ service.

You will now see the disabled services for this host. These are controlled by the ‘Disabled services’ rule.

• Click on the first icon under Action for your monitoring rule for the hardware sensors. You can recognize the appropriate rule by the text under Conditions. In my case it says ‘Host name is muc-sr1-esxi-001.lan.domain.net Service name is Hardware Sensors’.

• Now customize the rule. Scroll down and under Conditions, specify that Checkmk applies this rule not only to this particular ESXi host, but to all hosts in your ESXi folder.
  • At Folder, select your folder for the ESXi hosts, in my case ‘esxi’.
  • Uncheck the box at Explicit hosts.
• Click on Save.

You have now set up a monitoring for your VMs and ESXi hosts. There are more rules in Checkmk, however, that make VMware monitoring easier for you on a day-to-day basis. I would like to show you how to use two of these rules now.

In the previously mentioned blog on naming concepts, I use an underscore as a prefix with display names in VMware for VMs that are no longer in use and which will soon be deleted. For example, if you are planning to delete the test system ‘muc-vm-test-01.lan.domain.net’ soon, add an underscore and its new display name will be ‘ _muc-vm-test-01.lan.domain.net’.

This is then also an indication that this VM should be disabled. By default, Checkmk assumes that a VM should be online. You can use the ‘ESX host and virtual machine states’ rule to customize the defaults for warnings and critical states for selected services. Here, I show you how to use the ‘ESX host and virtual machine states’ rule to customize the configuration for all systems with an underscore prefix in their names.

• Click Setup and search for ‘esx’. Click ‘ESX host and virtual machine states’.
• Then click Add rule.
• Under Target states, set the following items:
  • For Powered ON, select the ‘WARN’ option.
  • For Powered OFF, select the ‘OK’ option.

• Under Conditions, select the checkbox at Name of the VM/HostSystem.
• Enter ‘VM _’ as the condition in the input field.
• Click on Save.

Checkmk automatically adds the abbreviation ’VM’ from the service name to the display name of each VM. By using the regex ‘VM _’, Checkmk modifies the configuration of the Status for all VM services with the prefix ‘_’. These services should now be offline, and you will not get false alerts in Checkmk.

In the future, if you want to take a VM offline permanently, all you need to do in the VMware client is to add an underscore prefix to its display name and run a new Service Discovery. Checkmk will then recognize that this VM's monitoring service should be offline.

Of course, this is just an example of how to apply this rule. In the same way, you could also configure the status of replica servers from VMware. The Enable timespecific parameters option also gives you the ability to set specific times for your VMs/hosts in Checkmk, such as when they are scheduled to be DOWN during maintenance windows.

You have already used the Disable Service rule to exclude the Hardware Sensors service. I would also like to show you how to use a single rule to permanently remove unwanted VMs from monitoring. The key is again a consistent naming concept. In my example, this is all VMs with the prefix ‘#’.

• Click on Setup. Search for ‘Disable’ and there click on ‘Disabled services’.
• Click on Add rule.
• Under Conditions, activate the checkbox at Services. I use the regex "VM #".
• Store this rule with Save.

You have now created all VMs and ESXi hosts in two separate folders in your monitoring. A fully comprehensive VMware monitoring also includes the monitoring of the server hardware. I recommend querying the appropriate information via the server's management board and creating a single host for each hardware server. I query the data via SNMP. Checkmk also supports queries via IPMI.

• Again, create a new folder under Setup and name it as you wish. In my example, I use HPE servers, so I name the folder ‘ilo’. In Dell, for example, the management boards are called ‘iDRAC’.
  • Under Monitoring agents, enable Checkmk agent / API integrations and select ‘No API integrations, no Checkmk agent’.
  • In addition, activate the checkbox at SNMP and select your SNMP version. For my example, it is "SNMP v2 or v3".
  • Store the folder with Save.

I don't use SNMPv3, instead I use v2c and the default SNMP community ‘public’ for all of my management boards. This is the default configuration in Checkmk. Hence, I don't need to enter any more data when creating the hosts. If you have a different SNMP community and/or use SNMPv3, then check the box at SNMP credentials and add the appropriate information.

• Under Setup, click again on Hosts and click on the folder you just created.
• Now, click Add Host to create the host for your first management board.
• Under Hostname, add the name of your management board, in my example ‘muc-sr1-esxi-001-ilo.lan.domain.net’.
• Click Save & go to service configuration, and then wait for the service discovery.
• Click on Accept all.
• Repeat this procedure for each hardware server with a management board.

Since you have already stored the appropriate configuration in the folder, all of the hosts inherit the correct settings, which saves time. If you use a number of different SNMP communities for your management boards, you would always have to configure them individually when adding the individual hosts. A configuration via the folder, on the other hand, saves configuration effort.

You can see that some changes have accumulated in the monitoring. You should accept the changes in Checkmk:

• Again, go to the yellow icon in the top right corner, and again activate the ‘pending changes’ by clicking Activate on selected sites.

For example, the hardware server's host view looks like this in my monitoring:

At this point, I would also like to show you how to monitor all the backups for your VMs. I use Veeam as a backup tool and have set up a hardware backup server called ‘muc-sr2-veeam-001.lan.domain.net’. I use the Checkmk agent plug-in for Veeam:

• Add your backup server as a host to the monitoring. To do this, go to Setup and click Add host.
• Enter your backup server name as the Hostname. In my example ‘muc-sr2-veeam-001.lan.domain.net’.
• Click on Save & go to folder.
• Click on Setup again. Search for ‘veeam’. Click on ‘Veeam Backup Status (Windows)’.
• Under Conditions you should assign this rule to your backup server, so in this tutorial I choose ‘muc-sr2-veeam-001.lan.domain.net’ under Explicit hosts. Otherwise you can leave all other settings as they are.
• Store the rule with Save.

• Now go to Setup -> Agents -> Windows, Linux, Solaris, AIX.
• You can see from the orange button that Checkmk's Agent Bakery is ready to bake an agent for you.
• Click on ‘Bake agents’.

• Wait until the agent finishes baking and appears as an option for selection. Then download the Windows agent with the configured Veeam plug-in.

• Install the agent on your Veeam server.
• Once the agent has been installed, go to Setup -> Hosts and select your Veeam server's host. Start the automatic service discovery for your Veeam server.
• Click on Accept all.

The Veeam plug-in not only monitors the Veeam jobs on the Veeam server, but also transmits services as piggyback data for the VM hosts. In order for Checkmk to include the ‘Veeam Client’ service for each VM host in the monitoring, you must start a service discovery for all VM hosts. The easiest way to do this is with a ‘Bulk Discovery’ for your VM folder.

• Go to Setup -> Hosts and open your VM folder.
• Click Hosts in the top menu bar. Select ‘Discover services’.

• In the Bulk discovery window, you can accept all default options and click on ‘Start’.
• Following a service discovery, you will see a summary in the Background job details window. Checkmk should have detected some new services.
• Confirm the 'pending changes' again.
• Checkmk will now monitor the operation of Veeam, but will also capture the appropriate piggyback data from the Veeam host and distribute it to the VM hosts in Checkmk.

For example, the backup service now appears in my vCenter server's host view. Checkmk detects that Veeam has not made a backup for 40 hours. This is again where a consistent naming concept comes into play, because it allows Checkmk to distribute the data correctly without the need for further configuration.