This is a story about a fictional support team, composite and invented, but built from patterns that managed-services engineers will recognise. The team looked after the web platform for a mid-sized regional retailer: a handful of servers, a database, a content system and a nightly set of batch jobs. For two years they ran on reaction. Something broke, they fixed it, and they moved on. Then they changed one thing: they put routine maintenance on a schedule. What follows is how that went, including the awkward parts, because practical lessons are rarely in the tidy version.
The year of firefighting
The first sign that the old approach was failing was a Monday morning in autumn. A certificate on the retailer’s checkout domain had expired over the weekend. Orders dropped to nearly nothing for four hours while the on-call engineer traced the cause, found the right account, generated a new certificate and installed it. The fix itself took twenty minutes. Finding out why it had happened took the rest of the morning, because nobody had recorded who owned the renewal.
The post-incident review listed a familiar set of contributing factors. The renewal reminder went to a mailbox belonging to an engineer who had left. The staging environment had never exercised the renewal path. And the team had no regular moment at which anyone looked at expiry dates. It was not a skills problem. Everyone on the team knew how certificates worked. It was a calendar problem.
Over the following months the pattern repeated in different costumes: a disk filled with old logs on a Friday evening; a library fell so far behind that a security fix needed a three-day upgrade; a database backup had been failing silently for nineteen days. Each incident had a different surface and the same root, which was that upkeep was optional and therefore skipped.
The decision: one standing slot
The team lead did something unglamorous. She added a ninety-minute recurring meeting called “Platform care” on the first Tuesday of every month, and she named a different engineer as owner each time, with the previous owner as deputy. The invitation contained a link to a checklist document and nothing else. No slides, no agenda debates.
The checklist began as a list of the previous year’s incidents turned into questions. Is anything expiring in the next sixty days? Are backups succeeding, and has one been restored recently? How full are the disks? Are any libraries more than two minor versions behind? Are there accounts or keys belonging to people who no longer work here? Seven questions, each answerable in minutes.
The first slot: messy, useful
The first session ran two and a half hours and felt chaotic. The inventory of certificates turned up a fourth domain nobody remembered, whose certificate expired in eleven days. The backup check found that restores had never been tested on the new database version. The access review discovered two service accounts with administrator rights and no known owner.
Several engineers grumbled that the time would have been better spent on feature tickets. The lead let them grumble and asked them to write down what the session found. That list, pinned in the team channel, ended the argument within a week: three of the findings would have caused outages within the quarter.
The routine settles in
By the third month the team had a rhythm, and it had shape. Weekly, whoever was on call spent fifteen minutes scanning failed jobs, disk growth and anything expiring within thirty days. Monthly, the “Platform care” slot handled updates in staging, then production; rotated secrets; trimmed logs; and rehearsed a restore of one table. Quarterly, they added a half-day for bigger upgrades: a database minor version, a base image refresh and a review of the monitoring thresholds that had grown noisy.
The sessions got shorter, from two and a half hours to ninety minutes, and then to an hour. Part of that was familiarity; part was automation of the dullest checks, such as a script that listed certificate end dates across every endpoint. The team kept the human review anyway, because a script that nobody reads is another thing that can silently fail.
What the numbers did, in plain terms
The team deliberately avoided building an elaborate dashboard. They kept a tally in the notes at the end of each session: unexpected expiries, disk emergencies, emergency patches, and total time spent. Within two quarters the emergency counts fell to zero for the categories the slot covered, and the monthly time stabilised near an hour. They did not publish any grand percentages, because the sample was one team and a handful of months, but the direction was unambiguous, and so was the change in the on-call experience. Nights were quieter.
Their approach echoed ideas found in reliability engineering writing. A piece making the case for operations staff to adopt code-defined infrastructure and reliability engineering covers the broader thinking, and the team’s habit of writing infrastructure changes down as code made each monthly update easier to review and reverse.
The conversation with the retailer
One more thing changed. The support team began sending the retailer a three-line note after each session: what was checked, what was changed, and what was coming up. The retailer’s operations manager had never received anything like it. Previously she heard from the team only when something went wrong, which coloured every contact with worry. The monthly note turned the relationship into something steadier, and it made the later conversation about renewing the support contract far easier, because the value was written down in dated lines rather than argued from memory.
The note also exposed decisions that belonged to the customer, such as whether to retire an old microsite that kept generating certificate renewals. That kind of decision is exactly what a regular slot surfaces and an emergency never does.
Where it nearly fell apart
In the sixth month a major promotion at the retailer collided with the first Tuesday. The product manager asked, reasonably, whether platform care could wait. The lead’s answer became team policy: a slot may move once, to a date announced before the original passes, and may shrink but not vanish. They ran a fifteen-minute version of the checklist, which covered expiry dates and backups only, and rebooked the full session for the following week. The routine survived because it was designed to bend.
There was also a quieter risk: the checklist grew. Every incident added a line, and after a few months it ran to thirty items. The lead pruned it ruthlessly, moving rare items to the quarterly list and deleting anything that had never found a problem. A checklist that cannot be finished in the slot is a checklist that will be abandoned.
What the story shows about roles and principles
Notice who did the work. It was not a special reliability hero; it rotated across ordinary support engineers, which spread knowledge and prevented the routine from depending on one person. For readers mapping that idea onto job titles, the explanation of how different DevOps job titles and career paths relate may help. And the underlying principles are not exotic. Small changes, frequent feedback and shared ownership all appear in the overview of core principles for delivering software effectively.
Lessons you can take without the fiction
- Convert incidents into questions. Last year’s outages are your first checklist.
- Name an owner and a deputy. Rotation spreads knowledge.
- Keep the slot short and fixed. Ninety minutes on a known day beats an ambitious open-ended afternoon.
- Let it shrink, not vanish. Busy weeks get the fifteen-minute version.
- Prune the list. If an item never finds anything, move or remove it.
- Write down what you find. The record is what silences the sceptics.
Support organisations also meet customers who want help with the digital side of running a business. Some firms describe that offer in broad terms; BHEEKHO, a business-style site, outlines web, cloud and AI-integration work in sections on a single page, and its own presentation of web and cloud solutions is an example of how such an outfit might introduce itself. Whoever delivers the service, the same discipline applies to the dates on the calendar.
If you run a support team, your equivalent of that first Tuesday is waiting. Make the invitation today, link a draft checklist built from your last three incidents, and give the first session two hours rather than one. It will be messy and it will be useful, and by the third month it will be the quietest meeting of your week.