Hospital network relies on open source monitoring in Docker containers
Who is Paracelsus?
“Medically competent, humanly committed.” This is the motto of the Paracelsus Clinics, and with this approach, the IT team is building up a monitoring system for their clinic network. The goal is centralized IT monitoring in order to provide the medical staff with an ideal basis for treating patients.
I am always surprised by how easily Checkmk can be used in day-to-day operations. Especially if you are familiar with other open-source tools, Checkmk is a boon. The administration of hosts is really simple.
The IT managers needed maximum visibility in complex IT environments, with numerous medical devices distributed across multiple locations. The IT team therefore decided to switch to Checkmk in 2015.
In 2020, the IT team will be monitoring assets in ten clinics, with 600 hosts and a total of 15,000 services. Paracelsus relies exclusively on the Checkmk Raw Edition, and operates all monitoring instances in Docker containers.
After their positive experiences at the Paracelsus Clinic Henstedt-Ulzburg, the IT team is working on switching the monitoring at other Paracelsus Clinic locations in Germany to Checkmk.
The highest standards in healthcare
The healthcare industry has always been a pioneer in digitization. For example, much of the communication between different healthcare devices already takes place via digital channels. IT departments must be able to ensure efficient processes, particularly in view of the ever-increasing volumes of data. If a device malfunctions, the success of patient treatment or even their well-being is at risk. IT managers must therefore be able to identify problems quickly and with absolute precision.
The IT team at the Paracelsus Clinic in Henstedt-Ulzburg solved exactly this challenge with Checkmk. In 2015 they switched from Nagios and Icinga to the Checkmk Raw Edition. The main reason for the change was the time-consuming configuration of monitoring with the old software, which could no longer be handled by the IT team.
Paracelsus has the goal of mapping the information stream, including the complete lifecycle of individual files, in its monitoring. To do so, one has to understand the special IT communication in healthcare: Usually, medical devices use control systems that are set up on virtualized middleware machines. For medical devices, these management systems are the only way to interact with the network. The control systems then distribute the input from the medical devices, such as X-rays, over the network, and then transmit the data to communication servers, for example via Java interfaces. These interfaces then pass the information on to subsystems such as hospital information systems (HIS). All of these diverse elements needed to be monitored.
The challenge
The IT team was looking for an open-source monitoring solution that would meet the strict data protection and IT security requirements in the healthcare sector. At the same time, the IT monitoring must be easy to use. The tools previously used were complex to configure. In addition, IT monitoring had to provide comprehensive visibility and be technologically mature enough to monitor even unusual assets such as medical devices.
Moving away from Nagios
What spoke for Checkmk was that it can integrate plug-ins from Nagios and Icinga without any problems, so that only minimal customizations were necessary. But the main reason was the easy configuration via Checkmk’s WATO (Web Administration Tool), as well as the large number of Checkmk plug-ins that are available. The Paracelsus IT team quickly noticed that the official Checkmk plug-ins often work more precisely and are a better choice compared to the legacy plug-ins from the old monitoring solutions.
Paracelsus uses Checkmk agents or integrations with manufacturer interfaces for its monitoring. For example, the IT team monitors appliances such as Cisco ASA via the integration of Checkmk. A great help for Paracelsus is the Checkmk plug-in for monitoring individual files (fileinfo check). This has greatly facilitated the practical implementation. In the network area, the IT team also monitors some devices like switches via SNMP. The Paracelsus team has an overview of all ports and can ensure that these are working properly.
The ability to retrieve and combine detailed information from multiple sources is used by Paracelsus to narrow down the sources of errors as precisely as possible. Paracelsus relies on predefined HL7 standards in all clinics. All exchanged data follow the same structure and are always processed by the end devices according to predefined patterns. The IT team monitors compliance with these processing patterns.
The solution
In 2015 Paracelsus decided to use the Checkmk Raw Edition at their clinic in Henstedt-Ulzburg. The IT department there replaced Nagios and Icinga with Checkmk, and has subsequently been continuously expanding its monitoring. Since 2018 Paracelsus has been working on the integration of additional locations. Currently, the IT team is already monitoring 600 hosts in 10 clinics throughout Germany with Checkmk. Paracelsus relies exclusively on the deployment of Checkmk in Docker containers.
Monitoring complex procedures
If a file is not accepted by the receiving device or server, or if it cannot be forwarded, it remains in local storage. The IT team therefore checks, for example, the time stamps on the files or the sizes of certain folders on the systems. Monitoring detects if data stays in one place for too long or if directories become too large. It also detects if a file’s destination is not accessible or if servers receive too many requests.
Through the comprehensive monitoring of individual documents and all networked systems, Paracelsus can precisely pinpoint errors at any time. The alarm thresholds depend on the respective system. Some medical devices delete files after successful processing, others have waiting lists (message queues). Accordingly, Paracelsus adapts the monitoring to the device type or its files. Thanks to the experience of the Paracelsus team and the easy configuration of Checkmk, even the setup of complex threshold values is easy to manage. Furthermore, conditions such as time periods can also be included.
The advantages
Paracelsus is impressed by the usability and the continuous visibility of all assets. Compared to previous solutions, the IT team saves a lot of time through the improved usability of Checkmk. In addition, by using the open source version of Checkmk, there are no costs for software licenses.
Plans for the future
Paracelsus made a conscious decision to implement the system via Docker, since all clinics of the group use container infrastructures in many other areas. For example, every clinic already has a Kubernetes cluster, which is also monitored with Checkmk. Paracelsus also uses Checkmk integrations with Grafana and Prometheus. This allows the creation of meaningful visualizations and an interlocking with the monitoring tools of the development teams.
Paracelsus wants to provide as many branches as possible with Checkmk instances. In some hospitals Checkmk replaces further legacy tools, in others, there has not been real monitoring up until now. In any case, the local IT managers welcome the use of Checkmk, as IT problems can be detected faster and even avoided with the monitoring. Checkmk is also able to identify unused resources and systems.
Paracelsus is focussing on the inclusion of further hospitals into its monitoring. It also plans to monitor cloud assets. Paracelsus already uses AWS and Azure Cloud and hosts Prometheus in the Azure Cloud. Therefore, in the future, the hospital group also intends to monitor its AWS and Azure Assets with Checkmk.
The standard set of over 1,900 official plug-ins alone is incredibly powerful and a great help to us.