If someone else patches your servers, or you patch someone else’s, there is a good chance the arrangement rests on a sentence like “we keep everything up to date”. That sentence is easy to say and hard to audit. This checklist gives managed-service teams and their customers a shared list of questions to answer with evidence. Open a spreadsheet, tick the lines you can back with proof, and circle the ones you can only claim.

Before you start: agree on the words

An audit goes wrong when the two sides use the same word for different things. The Ops Atlas calls the subject a loose nickname rather than a formal field, and its scope list sits on the Atlas page for PatchOps. Behind it stands NIST’s published guidance, which strings the work into five stages ending with verification. Write the stage names on the audit sheet, because that last one is the part service reports most often skip.

Section A: scope and inventory

• Is there a list of every asset covered by the service, with an owner for each row?
• Does the list change automatically or by a defined procedure when assets are created or removed?
• Are firmware and third-party applications in scope, or only the operating system?
• Is every exclusion recorded alongside its justification and a date when it lapses?

The Atlas is direct about one point: relying on periodic scans to discover your estate is no longer a model to recommend, and passive scanning tends to under-report. Ask how the list is kept honest.

Section B: groups, rings and windows

• Are assets divided among a handful of maintenance groups that together take in nine tenths or more of the estate?
• Is there a first ring made of real production roles, not just idle test machines?
• Does each group have a named window, and is it outside business-critical hours?
• Is there a named person who can halt a rollout?

For orientation, the Atlas pictures the opening ring as a sliver of the fleet, followed by a somewhat bigger pilot, then a broad sweep up to the deadline, and finally the delicate systems one node at a time. A smaller estate should turn those proportions into actual machine counts, but keep the sequence.

Section C: gates and probes

• What condition lets a rollout move from one ring to the next?
• Do the health checks test what the system is for, such as completing a transaction, rather than only whether it answers?
• Does the process stop automatically when new incidents appear?

One common reference point is the default behaviour of a major endpoint management service, which the Atlas quotes: advancement requires an install failure rate no higher than two percent, at least ninety percent of devices reporting and no new incidents. You do not have to copy it, but you should have something equally explicit.

Section D: restarts and freezing

• How does the report treat machines that have installed an update but not restarted?
• Who decides restart timing, and do tool defaults override it?
• Is the tested package set frozen for each cycle, so that later rings install exactly what earlier rings validated?
• Are major version upgrades blocked unless deliberately scheduled?

Treat any answer of “the tool handles it” as a prompt to ask which setting, in which console, and who last reviewed it.

Section E: emergencies

• Is there a documented route for urgent fixes with named approvers?
• Does a freeze on ordinary changes still allow that route?
• Has the route been rehearsed, and was the time to reach ninety-five percent active recorded?
• Does the plan cover cases where no fix exists yet and only a mitigation is possible?

The Atlas, again echoing NIST, separates four situations: routine patching, emergency patching, emergency mitigation and assets that cannot be patched. A support contract should address all four, not just the first.

Section F: exceptions

• Does each exception name who owns it, why it exists, what protects the asset meanwhile and when it ends?
• Are expired exceptions flagged, or silently left out of the numbers?
• Do unpatchable assets live in their own group with extra safeguards?

Section G: measurement

Request the following figures for the previous quarter, together with the method used to calculate each.

1. Effective coverage at the deadline, with every in-scope asset in the divisor.
2. How long a fix takes to become active, shown both typically and for the slowest tenth.
3. Proportion of assets that have gone quiet.
4. Age of the oldest pending restart.
5. Failure rate per ring and patch-caused incidents.
6. Emergency lead time from the last drill.

If the figure given is a single “compliance percentage” with no definition, request the definition. One of the Atlas’s lab exercises has the same fifteen assets flatter themselves on a vendor screen and fall to just over half when recalculated, because lapsed exceptions were counted as fine and a stale agent went unnoticed.

Section H: the patch system itself

• Who has administrative access to the patching tool, and is that access reviewed?
• Are policy changes logged and approved?
• Do all content channels, including configuration or definition updates, pass through staged rollout?

The last bullet draws on a 2024 outage the Atlas discusses, in which a content update outside the customers’ ring policies caused widespread crashes. Check what your tools can push and whether each class of change honours your rings.

Section I: the customer’s side of the bargain

An audit that only questions the provider is half an audit. Customers contribute to failure too: they approve maintenance windows and then cancel them, forget to tell the provider about new servers, and keep legacy systems alive without budget for replacing them. List the commitments each side makes. Who notifies whom when an asset is added? Who signs off a window and how many times may it be postponed before the delay is recorded as an exception? What happens when a business freeze coincides with an urgent fix? Writing those answers into the service description turns a vague partnership into something you can measure.

It also helps to agree how disputes are settled. If the customer says an update broke an application and the provider says the application was already faulty, the ring logs, probe results and frozen package lists become the neutral record. Without them, the conversation devolves into opinion.

How to score it

Give each question a point only when evidence exists: a screenshot, a report, a log entry, a signed procedure. Anything that rests on “we believe” scores zero. Add the points by section and compare against the Atlas’s maturity ladder, which runs from ad hoc through scheduled, ringed and measured up to engineered. Most teams land between the first two levels at the first audit, and that is not a scandal; it is a baseline.

Connecting the audit to your wider service story

An audit sheet like this also feeds the proof customers want to see. Engineers building a public track record, as described in the guide to a standout engineer portfolio, can use the monthly one-page report and the drill notes as concrete artefacts. And the checks line up naturally with the ideas in the overview of foundations of dependable software delivery: small batches, fast feedback and shared ownership all show up in a ring-based patch process.

Common red flags during the audit

A few answers should make you slow down. If nobody can say how many assets are in scope, you are auditing a story. If the report contains one green percentage and no definition, request the working. If the provider cannot name the machines in the first ring, it probably does not have one. And if the reply to every question is that the tool does it automatically, ask to see the setting, the date it was last changed and the person who changed it.

What to do with the results

Pick the three lowest-scoring sections and set a thirty-day goal for each. Typical wins include a real first ring, an exception register with expiry dates and a coverage figure that divides by everything in scope. Re-run the audit quarterly. A support relationship that can survive this level of scrutiny tends to be one both sides are glad to keep.