This is an invented story, offered as an illustration and not as a record of any real client. A small support team at a managed-services firm takes over the day-to-day running of an online booking platform. On day one they receive a folder of dashboards, a pager rota and a list of passwords. What they do not receive is any statement of who looks after the monitoring system. Over the next ninety days they learn, step by step, why a discipline called ObservabilityOps exists.
Week one: the quiet failure
The team lead, a methodical engineer, starts with a simple question during the first handover call: “If the alerting system itself stopped working tonight, how would we find out?” The outgoing contractor pauses and admits that they would probably learn from a customer. No one had ever tested the path from a failing service to a human phone.
That admission shapes everything. The team reads around the subject and lands on the Ops Atlas description of ObservabilityOps, which frames the job as operating the observability platform itself. The Atlas notes that this is its own reading, not an official standard, and the team appreciates the candour: they are looking for a working vocabulary, not a badge.
Their first deliverable is humble. They add a rule that fires permanently and routes to a dedicated channel, the kind of heartbeat that the Atlas calls a Watchdog pattern. They also schedule one black-box test that sends a synthetic booking through the public site and expects a confirmation. If either goes quiet, they are paged. Total effort: half a day.
Week three: the inventory nobody wanted
Next the team writes down everything that emits signals. The result surprises the client’s product manager: eleven services, four log formats, two metrics back ends from different eras, and a dashboard folder with more than two hundred panels. Using a simple three-column sheet (source, destination, reader) they discover that three data streams have no reader at all.
The support lead applies a lesson familiar from service work in general: a list is also a conversation starter. In the review with the client they ask for a named person on the client side to co-own the platform, because the Atlas suggests that when only a couple of teams are involved, a named on-call owner is a fair choice if the stack is self-hosted or the client is regulated. Here the stack is self-hosted, so the answer is yes.
Role clarity of this kind echoes the themes in a guide that sorts out DevOps role titles and responsibilities. The team uses it to draft a one-paragraph description of the new duty, so the client can see it is a real job and not a hobby.
Week five: the label that ate the budget
A routine look at storage growth reveals a steady upward line. One metric tracking booking attempts has acquired a label holding the session identifier. Each session creates a new series, and the series count has ballooned. The team recalls the Atlas guidance on keeping label cardinality low, roughly ten or fewer distinct values per label as a norm, with a closer look warranted past a hundred.
They fix it in the least dramatic way possible. The identifier moves into traces and logs, where high-variety values belong, and a limit at the scrape level ensures that a future mistake does not take the platform down. They also tell the developers why, which matters more than the patch: a bounded label is a design choice, not an optimisation.
Week seven: the logs that would not line up
During a minor incident, engineers spend forty minutes matching log lines from the web tier and the payment service by timestamp. The postmortem notes that neither service writes the trace and span identifiers as separate fields. The Atlas’s logging advice is the remedy: use structured records with severity and trace identifiers as top-level fields, recombine multi-line stack traces, and handle severity carefully, since unparsed lines carry a zero value that a filter must treat explicitly.
The team publishes a short logging standard, two pages long, and offers a template that every new service can copy. Within a month, three of the eleven services have adopted it, and a single search now follows a booking from the browser to the bank callback.
Week nine: pruning the pager
By now the pager is the team’s loudest complaint. They go through the last quarter of notifications and count how many led to action. The tally comes to roughly one in three, much like the example in the Atlas where 15 of 42 pages were actionable. They set a rule inspired by the Google reliability guidance quoted there: signals nobody uses are candidates for removal. Over three meetings they delete forty-odd rules, demote another dozen to daily summaries, and convert two noisy component alerts into one symptom alert tied to the booking confirmation test.
Several weeks later, the pager volume has dropped by more than half and, more tellingly, the engineers start answering it faster.
Week eleven: money enters the room
The client’s finance contact asks why the monitoring bill rose. The team has no clean answer, so they build one. They calculate a unit cost: monthly monitoring spend divided by booking requests served. They show it next to coverage measures such as the share of services reporting standard fields. The Atlas describes the same approach, with an arithmetic example of a monthly bill of 18,000 serving 900 million requests, about 20 per million. The key is that unit cost is read together with coverage, never alone, so that a drop does not simply mean blind spots.
They also introduce a lightweight showback: a monthly one-page summary that tells each service team how many series and gigabytes it generated. No charges, just visibility. Behaviour shifts quickly when numbers carry a team’s name.
A pause for the awkward part
Not everything went smoothly in our imagined quarter. Two developers resisted the logging standard, arguing that it slowed feature work, and one dashboard owner refused to let a favourite panel be retired. The support lead handled both the same way: by asking what question the thing answered and who had last used it. When nobody could name a recent use, the panel went; when a developer showed that a field saved them time during a real incident, the standard gained a champion. Gentle evidence beat policy every time, and it kept the client’s own engineers on side for the work that followed.
Week thirteen: the handover that sticks
Once the quarter closes, the team walks the client through a brief written report. It lists the named owner, the inventory, the heartbeat and outside probe, the logging standard, the pruned alert set, and the unit-cost trend. A 30/60/90 outline in the Atlas turns out to resemble what they did, more by accident than design, though the team admits their order was driven by pain rather than a plan. In hindsight they would do it in the suggested order, with the cheap items first.
To keep improvement going, they move the collector and alert configuration into version control, add a review step, and schedule a quarterly pruning session. A reference like a summary of core principles for effective software delivery supports the argument they make to the client: small, reviewed changes beat heroic fixes.
What the story is meant to illustrate
- The most valuable early step was a test of the alert path, not a new tool.
- Ownership came before technology; a named person turned a pile of dashboards into a service.
- Most improvements were cheap: removing a label, adding fields, deleting rules.
- Cost conversations went better with a unit measure and a visible owner for each number.
No real team is quite this tidy, and the Atlas itself offers cases with very different shapes. Still, the arc rings true for anyone who has inherited a monitoring stack: the first honest question about who watches the watchers is the beginning of the work, and it rarely takes a big budget to answer it.