Werk #19417: Piggyback Hub: Fix crash when many piggyback hosts are updated at once
| Component | Checks & agents | ||||||
| Title | Piggyback Hub: Fix crash when many piggyback hosts are updated at once | ||||||
| Date | Apr 7, 2026 | ||||||
| Level | Trivial Change | ||||||
| Class | Bug Fix | ||||||
| Compatibility | Compatible - no manual interaction needed | ||||||
| Checkmk versions & editions |
|
On sites with a large number of piggybacked hosts, the piggyback hub process could crash when many of them were updated at the same time. The hub now recovers gracefully: Any updates that were missed during the burst are detected and reprocessed automatically.
If you see the following message in the piggyback hub logs under
var/log/piggyback-hub.log, it means that the recovery path was triggered:
Too many messages for the piggyback-hub to progress at once, rescanning data. Consider raising /proc/sys/fs/inotify/max_queued_events.
In systems where a large number of piggybacked hosts are updated at the same
time, the fs.inotify.max_queued_events limit should be raised to reduce how
often the recovery path is triggered.
The value should be at least the number of piggyback hosts on the site, ideally
at least double that number to account for bursts.
To do so, run:
sysctl -w fs.inotify.max_queued_events=65536
Additionally, the piggyback hub registers an inotify watch for each piggybacked
host directory. If fs.inotify.max_user_watches is lower than the number of
piggybacked hosts, the hub may fail to monitor all of them. Check the current
value with:
sysctl fs.inotify.max_user_watches
and raise it if necessary:
sysctl -w fs.inotify.max_user_watches=65536
To make these changes persist across reboots, add the respective lines to
/etc/sysctl.conf (or a file in /etc/sysctl.d/):
fs.inotify.max_queued_events = 65536
fs.inotify.max_user_watches = 65536