A managed support desk lives or dies by its alerts. When you look after infrastructure for a dozen client companies at once, every server, database and payment gateway in that portfolio can send a message to your team, and on a bad week they all do. Over the years our engineers have learned, often the hard way, that the desks which respond best are rarely the ones with the most monitoring. They are the ones that receive fewer, better alerts. Here are eight lessons, each drawn from the kind of shift that support engineers tend to remember for a long time.
1. The alert you scrolled past is usually the one that mattered
Ravi had been on the evening shift for three hours, acknowledging warnings about a retail client’s cache nodes every few minutes. They always cleared on their own. At 9:15 a message arrived about the client’s order database replication falling behind. It sat between two cache warnings and looked just like them. He cleared it with the others. By the time the client rang, orders placed in the last hour were missing from their reports. The lesson our desk took was not “be more careful”. It was that a channel where the routine and the serious look identical guarantees this outcome sooner or later.
2. Count what fires before deciding what to fix
After that night, the team lead exported a month of alerts across all clients and sorted them by rule. A small group of rules accounted for most of the volume, and almost none of those had ever led to a ticket, a change or a phone call. Without that list, every conversation about noise had been a matter of opinion. With it, the next steps were obvious.
3. One fault deserves one message
When a client’s storage array degraded one afternoon, our desk received separate notifications from the virtual machines, the databases on them, the application health checks and the backup jobs. Four engineers started investigating four different symptoms of the same cause. Configuring the alerting to group notifications by client and affected system turned that cascade into a single incident with its symptoms listed underneath, and the next storage problem was handled by one person, calmly, with the full picture in front of them.
4. Retire inherited alarms without guilt
Managed services teams often inherit monitoring from whoever ran a client’s systems before. Anjali once found a rule warning about a disk on a server that had been switched off for most of a year; it fired whenever the check failed to reach it, which was always. Nobody had questioned it because nobody felt it was theirs to delete. Now every rule must have an owner, and rules that repeatedly fire without action are reviewed, documented and removed. Clients are told what was retired and why.
5. Write down what is worth waking someone for
Our night engineers used to make judgement calls on every page. That is exhausting, and under pressure the easiest judgement is to wait. Each client contract now includes a short statement of what justifies an immediate response, usually some variation of “customers cannot use the core service, or will not be able to soon”. Everything else becomes a ticket for the morning shift. Engineers say the biggest change is not the reduced volume but the confidence of knowing that a page at 3am is genuinely meant for them.
6. Planned work should never page anyone
One Sunday, a client’s own team ran a planned database upgrade without telling us. Our desk spent an hour chasing what looked like an outage. Since then, change windows are shared in advance and alerts for the affected systems are muted for that period, with the silence expiring automatically. Unplanned surprises still happen, but planned ones no longer masquerade as emergencies. Our article on building resilient operations through cloud and reliability practice covers how change coordination fits into a broader support model.
7. Give every surviving alert a next step
A message that says “high latency on api-03” forces the responder to start from scratch. A message that names the client, the service, the likely user impact and a link to a short runbook lets them begin work in seconds. We now refuse to put any new rule on the pager unless it comes with that context. The effect is subtle but real: engineers open alerts readily again because they have learned that each one will help them act.
8. Make the review a habit, not a project
The first clean-up felt like a big achievement, and within a few months the noise had started creeping back as new clients and new rules arrived. What finally kept it under control was a short fortnightly review: every page from the last two weeks, one question each time, “did this deserve to interrupt someone?” Bigger platform migrations, like the ones walked through in our piece on modernising enterprise cloud operations with Kubernetes and automation, now include an alert review as a standard step.
Where our engineers keep learning
Several people on the desk read TheAIOps.com, whose AIOps and IT operations writing includes articles on correlating and filtering alerts in large environments; the ideas there have shaped how we group notifications across clients.
The quieter desk
None of these lessons involved buying anything new. They came from listening to the engineers who carry the pager and taking their frustration seriously. Today the evening shift sees far fewer messages, and when one arrives, people look at it properly. For a support desk responsible for other companies’ livelihoods, that attentiveness is the whole job. Ravi still works nights, by the way. He says the shift feels completely different now, because the alerts finally mean what they say.
