What follows is an invented story, assembled for illustration, about a managed-support team and a client who runs a handful of machine-learning models. Every name, number and date in the narrative is made up. The practices it leans on are real and come from the Ops Atlas article on MLOps, and I cite the Atlas wherever a claim is about the field rather than about the story. The point is to show how a support contract changes shape when what you are supporting is a model rather than a server.
The setup
The client is a mid-sized online retailer. Its data team trains three models: a demand forecast, a returns-risk score and a search re-ranker. They had been run for a year by the people who built them, which meant that when those people went on leave, they were run by nobody. The retailer signs a managed-support agreement with a team I will call the support crew. The crew’s brief, in the contract’s words, is “keep the models healthy”. Its first job is to turn that sentence into something measurable.
Week one: the inventory
The crew starts with a table. For each model it records the owner, the data source, the place the output goes and the people who read it. By day three the table reveals the first problem. The returns-risk score is read by the warehouse planning tool, by a marketing export and by a spreadsheet maintained by someone in finance. Only one of those three consumers had ever been mentioned to the data team.
The Atlas has a name for this: undeclared consumers, listed among the anti-patterns because a model that unknown systems depend on can never be retired safely. The crew’s response is modest and effective. It keeps the consumer list as a living document, restricts who can read the output, and announces changes ahead of time. That one table is worth the first week’s fee.
Weeks two and three: what can be rebuilt?
Next the crew asks each model’s author to rebuild the live version from scratch while it watches. The demand forecast rebuilds cleanly, landing within the agreed tolerance on its quality metrics. The returns-risk model does not: its training notebook reads a file from a shared drive that has since been overwritten. The crew finds an older copy in a backup and writes down its fingerprint, then sets up a small record beside each model with the code version, the data fingerprint, parameters and environment.
The crew’s lead is careful about wording in the report. “Reproducible” is defined as quality metrics within a stated margin rather than identical output, because, as the Atlas explains, a training run may involve randomness and even a major deep-learning library disclaims repeatability across releases. Clients appreciate a definition they can check.
Week four: an expensive mistake nearly happens
The retailer’s data scientist proposes a nightly job that retrains the search re-ranker and promotes the result automatically. The crew pushes back with evidence. The Atlas warns that an automatic retrain-and-promote loop can cause outages through feedback loops, because whatever the model decides ends up as tomorrow’s training material. A re-ranker is a textbook case: what customers click depends on what it shows them.
The compromise is a release gate. Each candidate is compared with the incumbent on a frozen, time-split test set. The crew adopts a policy modelled on the Atlas’s worked example, with a minimum improvement, a ceiling on how much any customer segment may lose and a response-time limit, and tunes the thresholds with the data team. In the first month the gate rejects two of five candidates. One of the rejections is a candidate whose overall score improved while results for new customers, a small slice, got visibly worse. Without the slice check it would have shipped.
Month two: the night the monitor lied
The crew sets up three levels of checks: service health, data health and prediction quality. One night the data checks go quiet and the dashboards stay a calm flat green. The on-call engineer, remembering the Atlas’s warning about push-based metrics that never forget, looks at the timestamp of the last push rather than the value. The statistics job had died eleven hours earlier and its last numbers were being shown as if fresh.
The fix is a rule: a stale statistics job pages the on-call. The Atlas’s reference implementation uses twenty-six hours as its threshold, and the crew picks a shorter one that suits a nightly cycle. The lesson the crew writes up is that a monitor needs a monitor.
Month two, week three: the rollback drill
Supporting a model means supporting the way back. The crew writes a one-page rollback note: move the production alias to the previous version, confirm that traffic follows, notify the owners on the consumer list. They also define a fallback for the search re-ranker that uses no model at all, plain ranking by recent popularity. Then they rehearse. The first drill takes forty minutes because the alias name differs between environments. The second takes four. The Atlas suggests drilling quarterly, and the crew books the dates in the client’s calendar, so the drill does not depend on anyone’s memory.
Month three: the handover that nearly broke
Halfway through the quarter, the retailer’s lead data scientist moves to another team. Under the old arrangement that departure spelled trouble; under the new one it becomes a checklist. The crew walks the replacement through the run records, the gate log and the rollback note, and asks her to perform one rebuild and one rollback drill under supervision. She finishes both in an afternoon. The crew counts that as the real test of the quarter, since a setup that only its authors can operate has not really been handed over.
The episode also surfaces an uncomfortable detail. The demand forecast depended on a holiday calendar that one person maintained by hand. The crew lists it as an input with an owner, adds a freshness check and notes the risk in the register. It is a small step, yet it echoes a lesson from the Atlas about a stale lookup table that, once refreshed, lifted a product metric by about two percent. Old inputs are quiet liabilities.
Quarter end: what the report shows
The crew’s end-of-quarter report is not a list of tickets closed. It lists artefacts: the consumer inventory, three reproducible runs, a gate policy with its decision log, a monitoring map with owners and a rehearsed rollback. It also records three things the crew deliberately did not do. It did not buy a platform, it did not automate promotion and it did not retrain the returns-risk model when its scores drifted, because the replay check showed the cause was a formatting difference between the training and live paths. Fixing the pipeline took an afternoon; a retrain would have taken a day and left the bug in place.
Why the story matters for support buyers
If you buy or sell managed support for machine-learning systems, the story suggests a few evidence-led questions. Does the provider ask who consumes the output? Can it show a rebuild of the live model? Does it have a written gate and a record of rejections? Has it rehearsed a rollback in the last quarter? These are cheaper to verify than any claim about uptime, and they tell you more.
For the teams staffing such work, two background reads from this site are useful: the walkthrough of the many job titles found around DevOps teams helps define who owns which part, and the piece on the principles that make software delivery work well supplies the delivery habits that models inherit.
A note on infrastructure
One more thing the crew learned: compute choices change the support burden. The Atlas mentions a Kubernetes queueing tool, Kueue, for managing GPU quotas, and notes that spot capacity on one cloud comes with roughly a two-minute reclaim warning given on a best-effort basis. It also says PyTorch’s distributed checkpointing can reshard at load time, which helps when a job resumes on a differently sized set of machines. For a support crew, the practical translation is simple: training jobs should survive interruption, and checkpoints are part of the runbook.