The symptom: a change that looked flawless on the day it was written breaks something small within hours of release. A mis-set flag, a renamed variable that one script still uses, a cleanup that removed more than intended. Support teams see this pattern constantly. Carelessness is rarely to blame. The real cause is how our minds ordinarily read our own work, and the remedy is a next day review before the change goes out. Here is how to diagnose the problem and build the fix into your support routine.
Symptom: “I checked it and it was fine”
Nearly every post-change incident review includes a version of this sentence. The author did check, but the check was performed in the same sitting as the writing. Cause: the author sees the intention rather than the text in front of them. Familiar text slides past. Gaps are filled in automatically from memory.
Fix: separate making from checking in time. Finish the change, stop, and perform the check the next morning with the diff open in the review tool rather than the editor.
Symptom: the change does more than the description says
Support engineers dislike surprises, and unlisted side effects are the biggest ones. They creep in when a quick tidy-up is added on the way to the real fix.
Cause: scope drift during the working session.
Fix: in the morning read, go line by line and ask what each changed line has to do with the stated purpose. Anything unrelated moves to its own change. This tiny discipline makes rollbacks cleaner and reviews faster.
Symptom: it works for the author, not for anyone else
Your own environment contains assumptions: a locally installed tool, an environment variable, a path that only exists on your machine. They are invisible to you precisely because they are always there.
Cause: the author’s setup silently supplies missing dependencies.
Fix: the next day, run the change in a clean environment, or follow your own written instructions on a different machine. If a step requires knowledge that is not written down, add it to the documentation before release. Practical exercises such as those in this complete tutorial on setting up DevOps practice at home are a good way to rehearse working from scratch.
Symptom: no one knows how to undo it
When a change fails in production, the first question is how to go back. If the answer takes ten minutes of searching, the incident lasts ten minutes longer.
Cause: rollback was an afterthought in the original session.
Fix: make “how do I undo this?” a standing item in the morning checklist. Write the answer in the change description, in one or two sentences.
Symptom: the wording confuses the on-call person
Alert messages, runbook notes and commit descriptions are written by tired people for tired people. A phrase that seemed clear at the end of the day can look ambiguous at three in the morning.
Cause: the author knew the context and wrote for themselves.
Fix: re-read every message you wrote as if you had just been woken up and had no history with the system. Cut anything that relies on shared memory.
Symptom: the same kind of slip keeps returning
Some teams fix each incident individually and never notice the pattern. Three renamed flags in a quarter, or four changes with the wrong environment target, are signs that the process, not the individual, needs attention.
Cause: nothing records what the reviews catch, so the lessons stay in people’s heads and leave with them.
Fix: keep a short log of what each morning read finds, one line per change, with a tag such as scope, rollback, wording or environment. At the end of each month, scan the tags. The most common one becomes a new line on the checklist, or better, an automated check that stops the mistake before any human has to look. This turns scattered saves into steady improvement, and gives support leads something concrete to discuss in retrospectives rather than vague reminders to be more careful.
Symptom: customers learn of the problem before you do
When a customer reports an issue first, it usually means the change had no observable signal attached to it.
Cause: the author never asked how they would know the change had gone wrong.
Fix: during the morning read, name one signal that would change if the change misbehaves, such as an error rate, a queue length or a failed health check, and confirm it is visible somewhere the on-call person will actually look. If there is no such signal, add one, or at least plan to watch the system manually for an hour after release and note that in the description.
A short morning-after checklist for support work
- Does the description match the diff exactly?
- Can I run it from a clean environment?
- What is the first thing that would break, and who would notice?
- Is the rollback written down?
- Would the on-call person understand every message?
For skills behind the checklist, the article explaining why developers should learn DevOps shows how operational thinking changes the way people look at their own work.
A parallel from information services
Support is about reducing surprises for people who rely on you, and careful wording is part of that. IVFHospitalsNow, a directory for learning about fertility care, tells visitors it does not operate as a clinic. Saying up front what a service is not heads off misunderstanding, much as a clear change description heads off confusion on call.
When you cannot wait a night
Urgent fixes exist. In those cases, take the next best step: have a colleague read the change, wait even thirty minutes while doing something unrelated, and schedule the proper full read for the morning. Record that the full review is still pending so it is not forgotten.
The diagnosis in every case is the same: familiarity hides mistakes. The treatment is cheap, repeatable and kind to everyone on call. Try it on the next change you finish, and let the morning tell you what the evening missed.