IT alert fatigue: What causes it and how to fix it at the source

Somewhere in your monitoring system, there is a notification everyone has learned to ignore. The nightly backup pushes a server past its CPU threshold at 2 a.m., the service goes WARN, and by the time anyone looks, it is OK again. Nobody wants to raise the threshold, because the backup really is that heavy and a higher limit could hide a real problem. It keeps notifying every night.

It is not the only one. Overnight, patterns like this can add up to a few dozen notifications, and you acknowledge each one in about four seconds without reading it, the way the queue has taught you to handle most of what arrives.

The notification that actually matters lands in the same queue, looks like the rest, and gets cleared the same way. That is IT alert fatigue, and the noise behind it has four causes you can fix.

IT alert fatigue comes from notification noise, and that noise has four root causes: static thresholds, missing dependency modeling, flapping, and alert storms. Let’s look at them in a bit more detail.

• Static thresholds: WARN and CRIT levels that were set once and never adjusted to the environment push services into WARN or CRIT on every brief CPU or disk spike that recovers on its own. The spikes are real, just not worth acting on, and you probably tolerate them because the alternative is raising the threshold and missing something real.
• Missing or incorrect dependency modeling: Your monitoring doesn’t know that 50 hosts depend on one switch, so when the switch fails, you get 51 notifications instead of one.
• Flapping: A service hovering on its threshold flips OK to CRIT to OK and notifies on every flip. One borderline disk can produce a dozen notifications overnight, all about the same problem.
• Alert storms: One incident produces dozens of separate notifications within minutes, with no parent-child relationship to collapse them. A storage array slows down and every database service on every host it serves goes CRIT at once, and a second tool monitoring the same hosts reports it all again. The flood buries the notification that matters.

Static thresholds, missing dependency modeling, flapping, and alert storms rarely fix themselves. They build up in monitoring that has grown over years, where thresholds were set once, dependencies were never modeled, new tools were added alongside old ones, and nobody has had time to clean up. That is why the noise is usually worst in the monitoring that has been running longest.

For more information on how problematic false positives can be, check out our piece explaining the nature of false positives and negatives (spoiler: neither is good).

The main cost of IT alert fatigue is a slow or missed response to a real incident, because the notifications before it weren’t worth opening.

The incident gets worse while it waits. As long as nobody opens the notification, nobody starts on the fix. The disk that filled overnight is a failed service by morning, and everything depending on it has been failing alongside it.

At the same time, your team slows down. Hours go into checking notifications that turn out to be nothing. Nobody fixes whatever generates them, so the volume holds steady. For example, a Checkmk customer’s IT operations team was previously getting 8,000 to 10,000 notifications a month on its legacy monitoring, most of them false alarms. That is the kind of load that turns on-call into something people leave jobs over. The hours spent sifting are only the visible cost. The hidden costs are everything the team can’t get to: root causes stay unfixed, and technical debt builds while people chase the noise.

You reduce IT alert fatigue by fixing the four root causes at the source, then routing whatever survives so every notification reaches someone who can act on it.

Tuning against a wall of red is guesswork. If the setup is new, or you have inherited one that was never tuned, start by bringing it to a stable state: fix the real problems the monitoring has uncovered, clear the false positives, and keep notifications to a small group until everything is OK, then open them up to the whole team. The Checkmk User Guide walks through that sequence in its notification basics.

Then, in this order:

1. Retune WARN and CRIT levels to the load the system actually runs at, and delay or average short spikes so a peak has to persist before anyone hears about it.
2. Check that flap detection is on, so a service crossing its threshold repeatedly stops notifying on every crossing. In Checkmk it is active by default; in other tools you may have to enable it.
3. Model parent-child dependencies, so a failed switch notifies once and marks everything behind it UNREACH.
4. Consolidate overlapping monitors, and let incident routing bundle related notifications into one.

The order is important. Consolidation and bundling on top of untuned monitoring give you tidier noise.

Notifications that survive all four fixes are usually worth someone’s attention, but not everyone’s, and not all at once. Route each one to the people responsible for the system it comes from, through contact groups rather than a shared inbox, so a database notification reaches the database team and nobody else. Then sort by urgency: anything that needs action now goes to whoever is on call, the non-urgent waits for office hours, and the rest goes to a ticket or a dashboard as a record.

The specific moves (how long to delay, what to average over, and how to wire parent and child hosts) are in Checkmk’s guide to fine-tuning notifications.

Checkmk reduces monitoring alert noise with six built-in mechanisms, so each of the four causes has an answer: default thresholds and predictive levels for static thresholds, parent-child host dependencies for missing dependency modeling, flap detection for flapping, and bulk and delayed notifications for alert storms and short spikes.

• Default thresholds that fit the type of service: Auto-discovery finds the services on each host and proposes each one with meaningful default thresholds already set, drawn from 2,000+ built-in integrations. They are a sensible start for that type of service, not a match for your specific load, so you still need to retune WARN and CRIT where your environment runs differently. Many check plug-ins can also average a metric over a period before the thresholds apply, so a brief peak has to persist to matter.
• Predictive levels (Checkmk Pro, Ultimate, and Cloud editions): Instead of one static WARN and CRIT, Checkmk derives the thresholds from the service’s own history, so the backup load at 2 a.m. is normal and the same load at 2 p.m. is not. However, they do need some monitoring history before they become useful.
• Parent-child host relationships: When all of a host’s parents go DOWN, the host is flagged UNREACH and its notifications are suppressed, so a failed switch notifies once instead of once per host behind it.
• Flap detection: A service changing state repeatedly is moved into a flapping state. You get one notification when it starts flapping and one when it stops, and nothing for the flips in between. It is on by default.
• Bulk notifications: Notifications that arrive within a short time are bundled into one. That catches the floods parent-child dependencies do not, where the affected hosts are not behind a single failed parent.
• Delayed notifications: A notification only goes out if the problem is still there after a set time, so a spike that recovers in two minutes never reaches anyone.

What survives goes out through one alerting engine to the tools you already use: Slack, Microsoft Teams, PagerDuty, ilert, and Splunk On-Call in every edition, ServiceNow and Jira in the Checkmk Pro, Ultimate, and Cloud editions.

That is what the mechanisms look like on paper. The IT operations team we mentioned earlier used to sift through 8,000 to 10,000 notifications a month; after moving to Checkmk and tuning its alerting with the tools Checkmk provides, it brought that down to about 2,000: 75 to 80% fewer notifications, with no loss in quality of service.