When we onboard a client, one of the first things we ask for is read access to wherever their alerts currently land. Not to judge — to count. The number is almost always somewhere between 150 and 500 a day, and the reaction is almost always the same: a slightly embarrassed "yes, we know, we've been meaning to clean that up".
The record so far is 380 a day, in one Slack channel, at a company of about seventy people. Nine months later that same client gets six. Their engineers did not get better at ignoring noise. We deleted the noise.
And here is the part that surprises people: over the same period their outage count went down, not up. Fewer alerts, fewer outages. That is not a coincidence and it is not a paradox.
Why volume makes things worse, not just annoying
An alert has exactly one job: to move a human being from not-looking to looking. Everything else about it — the formatting, the severity label, the integration it came from — is decoration on that single function.
Once a channel produces more alerts than a person can read, the channel stops performing that function entirely. Not partially. Entirely. The human adaptation is not "read faster", it is "stop reading", and it happens within about two weeks of the volume crossing the line.
A channel with 380 alerts a day and a channel with zero alerts a day have the same effect on human behaviour. The difference is that the second one is honest about it.
Worse, the noisy channel creates a false sense of coverage. Everybody knows alerts exist. Everybody assumes someone would notice. The organisation feels monitored while being, functionally, unmonitored — which is how you get a 58-minute outage discovered by a customer email.
Where 380 alerts a day actually comes from
Nobody sets out to build this. It accumulates, and always in the same four ways:
- Defaults nobody turned off. A tool is installed, its out-of-the-box rules fire on every host, and the rules were written by a vendor who has never seen your workload. CPU above 80% for five minutes is not an incident on a batch processing box; it is Tuesday.
- Alerts added after an incident, never removed. Something broke, somebody added a check, the check has fired 4,000 times since and been acted on twice. Nobody deletes it, because deleting an alert feels like a decision you could be blamed for.
- One failure fanning out into forty notifications. A database goes away and every service that talks to it alerts independently. Forty messages, one cause, and the actual signal is buried in the middle.
- Flapping. A check that fires and clears every few minutes across a network wobble. In one week it produces more messages than everything else combined.
None of these are stupid mistakes. Each one was locally reasonable at the moment it was made. That is exactly why the pile grows without anybody deciding to build it.
The four rules
We now apply these before any alert is allowed to reach a human, on every client, no exceptions granted on grounds of "but this one is important".
1. Only customer-visible symptoms may page
A page — a phone that rings at three in the morning — is reserved for things a customer is currently unable to do. Checkout failing. Login failing. The product not loading. That is the whole list.
Everything else is a ticket in a queue, read during working hours by someone with coffee. A disk at 85% is a ticket. A queue growing steadily is a ticket. Replication lag of four seconds is a ticket. None of them will be worse in six hours; all of them will be handled better by a rested person.
This one rule typically removes 80% of the volume, and it is the one clients resist hardest, because every alert has an owner who believes theirs is the exception.
2. Two consecutive failures, from two locations
A single failed check is not evidence. Networks wobble, DNS hiccups, a third party has a bad second. If the second run from a different city also fails, something is genuinely wrong.
The cost is real and worth naming honestly: with checks every three minutes, this rule adds up to three minutes to detection. We think three minutes is a fair price for an on-call engineer who still believes the phone when it rings.
3. Every page carries a runbook link, or it does not exist
If nobody can write down what a person should do when this fires, then nobody knows what to do when it fires, and waking them up achieves nothing but a tired engineer.
This rule is brutal in practice and it is our favourite, because it kills alerts through pure administrative friction. Roughly a third of existing alerts die at this step — not because anyone argued they were useless, but because nobody could be found who was willing to write the paragraph.
4. Review the actionability rate every month
For every alert rule, one number: of the times it fired, how often did a human do something as a result?
rule checkout-synthetic fired 2 acted 2 100% keep
rule cpu-high-web-tier fired 611 acted 3 0.5% delete
rule disk-80pct-all fired 288 acted 11 3.8% demote to ticket
Below 10%, the rule is deleted or demoted to a ticket. No discussion, no exceptions, and specifically no "but what if". The rule fired 611 times and produced three actions; it is not protecting anybody, it is training them to ignore the channel it lives in.
Why the outages went down
Because detection is not a technical property of your monitoring. It is a property of whether a human being looks when something arrives.
At 380 alerts a day, the median time between an alert firing and a human reading it was, at that client, about four hours — and for anything that arrived overnight, until the morning. At six a day, it is under two minutes, because six a day means every single one is real and everybody knows it.
They did not buy better monitoring. They deleted 374 alerts a day and got a functioning smoke detector for the first time.
Where to start on Monday
You do not need us to do any of this, and it takes about half a day:
- Count what arrived in the last seven days, grouped by rule. The distribution will be uglier than you expect — usually three rules account for most of it.
- For the top ten by volume, ask when someone last acted on one. If nobody remembers, that is your answer.
- Move everything that is not a customer-visible symptom out of the paging channel and into a queue. Do it in one change, not gradually, so the difference is obvious.
- For what survives, write the runbook paragraph. Whatever nobody will write for, delete.
The hard part is not technical. It is being willing to delete an alert and own that decision. Which is genuinely uncomfortable — right up until the first night the phone rings and somebody picks it up on the second ring, because it has been three weeks since the last one and it has never once been nothing.