Werk #15702: UNIX agents/mk_oracle: Validate termination of cached plugins

Component	Checks & agents
Title	UNIX agents/mk_oracle: Validate termination of cached plugins
Date	Jul 5, 2023
Level	Trivial Change
Class	Bug Fix
Compatibility	Compatible - no manual interaction needed
Checkmk versions & editions	2.3.0b1 Not yet released Checkmk Raw (CRE), Checkmk Enterprise (CEE), Checkmk Cloud (CCE), Checkmk MSP (CME) 2.3.0b1 Checkmk Raw (CRE), Checkmk Enterprise (CEE), Checkmk Cloud (CCE), Checkmk MSP (CME) 2.2.0p7 Checkmk Raw (CRE), Checkmk Enterprise (CEE), Checkmk Cloud (CCE), Checkmk MSP (CME) 2.1.0p31 Checkmk Raw (CRE), Checkmk Enterprise (CEE), Checkmk MSP (CME)

Many UNIX agents (linux, solaris, aix, freebsd, openwrt) as well as the mk_oracle agent plugin share the same functionality when running asynchronous/cached plugins:
After they got invoked as subprocesses, they will be watched by subsequent calls of the agent.
If they're still running after twice the cache time, they'll be terminated with kill -9.

Previously, the termination has not been verified.
As a result, in situations where a processes couldn't be terminated with kill -9, the agent created new processes unlimitedly.

This situation is now mitigated by checking for success after terminating a timed out process.
If the termination failed, no further process of the same plugin will be started.

Additionally, timed out processes, as well as processes that failed to be terminated, will now be reported to the monitoring server.
Affected plugins will be displayed at the "Check_MK Agent" service, alongside a WARN state.

To the list of all Werks