{"id":5629,"date":"2026-11-19T15:50:00","date_gmt":"2026-11-19T15:50:00","guid":{"rendered":"https:\/\/www.devopssupport.in\/blog\/?p=5629"},"modified":"2026-11-19T15:50:00","modified_gmt":"2026-11-19T15:50:00","slug":"support-team-reused-runbooks-case-study","status":"publish","type":"post","link":"https:\/\/www.devopssupport.in\/blog\/support-team-reused-runbooks-case-study\/","title":{"rendered":"Case Study: A Support Team That Reused Its Best Incident Fixes"},"content":{"rendered":"<p>This account is invented, stitched together from situations that keep turning up in managed-support work. The team is a six-person group running support for a mid-sized online retailer&#8217;s platform: a handful of services, a message queue, two databases and a deployment pipeline. We follow them through one difficult quarter to see what happened when they stopped writing a fresh response for every incident and started reusing the responses that had already worked. Where we make claims about effects, we say how they were measured and what the limits of that measurement are.<\/p>\n<h2>Where they began<\/h2>\n<p>Going into the quarter, the team handled roughly the same kinds of incidents repeatedly: a queue backing up, a certificate nearing expiry, a deployment that needed rolling back, a database connection pool exhausted. Yet each time, whoever was on call wrote their own approach in the chat, improvised commands from memory and, afterwards, produced a post-incident note in whatever format they preferred.<\/p>\n<p>The team lead, a person who preferred measurement to impressions, asked the group to record two things for six weeks before changing anything: the time from alert to confirmed resolution, and the number of times someone asked in the channel &#8220;how did we fix this last time?&#8221;. Over that baseline period they logged forty-one incidents of which twenty-nine fell into five repeating categories. The question &#8220;how did we fix this last time?&#8221; appeared in the channel on a large share of them. These figures are the team&#8217;s own tallies and are not a benchmark; they simply gave the group something to compare against.<\/p>\n<h2>The intervention: one shelf, five categories<\/h2>\n<p>The group decided to do one narrow thing. For each of the five repeating categories they would find the best previous resolution, turn it into a short response card and store the cards together. Each card had four fields: how to recognise the situation, the first safe action, the check that confirms it worked and what to escalate if it did not.<\/p>\n<p>They deliberately did not try to document everything. The rule was that a card could only be created from a real, successful past resolution, never from a guess about what ought to work. That kept the shelf grounded in evidence. The principles behind this restraint echo what is covered in <a href=\"https:\/\/www.devopssupport.in\/blog\/devops-principles-for-effective-software-delivery\/\">a general article on the principles behind effective delivery<\/a>: small, repeatable steps, shared ownership and feedback from real outcomes.<\/p>\n<h2>What the first month looked like<\/h2>\n<p>Adoption was uneven. Two team members used the cards immediately. A third preferred to improvise and only opened a card after the fact. The lead did not insist; instead, at the weekly review, the group looked at each incident and asked whether a card existed, whether it was used and whether it was right.<\/p>\n<p>That last question produced the most useful findings. Three of the five cards needed corrections within the month. One referenced a command flag that had been renamed. One assumed a single database replica when there were now two. One had a confirmation step that checked the wrong dashboard panel. Because the cards were short and each had an owner, the corrections were made within a day. A rewritten-from-scratch response would have carried the same errors, but nobody would have been looking at them in one place.<\/p>\n<h2>The measurement, honestly reported<\/h2>\n<p>When the quarter closed, the team compared the incident categories that had cards with those that did not. For card-covered incidents, the median time from alert to confirmed resolution was lower than in the baseline period, and the &#8220;how did we fix this last time?&#8221; question became rare. For incidents without cards, times were roughly unchanged.<\/p>\n<p>We should be careful about what that shows. The comparison covers one team, one quarter and a modest number of incidents. Incident difficulty varies, the on-call roster changed in the middle of the period and people were aware that they were being observed, which can by itself improve performance. A reasonable reading is that the cards plausibly helped, especially in reducing repeated searching, but the data do not prove how much. The team&#8217;s own summary said as much, and that candour made the result more credible to the rest of the organisation, not less.<\/p>\n<h2>The unexpected effect: better postmortems<\/h2>\n<p>An effect the team had not planned for appeared in the post-incident notes. Since the cards defined the standard approach, the notes could concentrate on what was different about this occurrence. A short &#8220;deviation from the card&#8221; field replaced pages of narrative. Reviewers found it easier to spot patterns, such as a particular service repeatedly requiring the same exception, and the group used those patterns to decide which underlying fix to prioritise.<\/p>\n<p>This links to a broader theme in the support world, the move from repeated manual work towards codified, testable operations, which is discussed in <a href=\"https:\/\/www.devopssupport.in\/blog\/infrastructure-as-code-to-sre-why-operations-engineers-should-embrace-devops\/\">a piece on moving from infrastructure as code to site reliability engineering<\/a>. The cards were a modest first step along that road, a halfway point between memory and full automation.<\/p>\n<h2>Where reuse went wrong<\/h2>\n<p>A case study that reports only success is a poor one. In week nine, an on-call engineer applied the connection-pool card to an incident that looked identical on the dashboard. The card&#8217;s first safe action, restarting a worker group, made matters worse, because the underlying cause was a slow downstream service that the restart did not address. The card&#8217;s &#8220;how to recognise&#8221; field had been too loose.<\/p>\n<p>The team&#8217;s response is instructive. They did not abandon the shelf. They tightened the recognition field to include a check on downstream latency and added a line to the card: &#8220;if latency is elevated, do not restart; escalate&#8221;. The incident became a documented limit of the card. This is the central lesson of the case: reuse works when the reused piece carries its own boundaries, and when each failure is allowed to sharpen the boundary.<\/p>\n<h2>How the cards were written<\/h2>\n<p>Style turned out to matter as much as content. The team settled on cards that fit on one screen with no scrolling, written in short imperative sentences, with commands set apart from explanation. They avoided adjectives and avoided hedging in the action steps, because a person at three in the morning reads literally. Background reasoning, such as why a particular restart order is safe, went into a separate &#8220;why this works&#8221; paragraph at the bottom that nobody was expected to read during an incident but that reviewers valued during the weekly session. Splitting the two audiences, the responder and the reviewer, made the cards both quicker to use and easier to challenge.<\/p>\n<h2>What the team changed at the end<\/h2>\n<ul>\n<li>Every card gained a &#8220;last reviewed&#8221; date and a named owner.<\/li>\n<li>Cards unused for a full quarter were flagged for review or retirement.<\/li>\n<li>New cards were only created after the same incident type had occurred at least twice.<\/li>\n<li>The weekly review kept its three questions: was there a card, was it used, was it right?<\/li>\n<\/ul>\n<p>These are small governance habits, but they are what kept the shelf from decaying after the initial enthusiasm.<\/p>\n<h2>A note on where the team looked for ideas<\/h2>\n<p>The group borrowed some of its habits from outside the immediate field. One member regularly browsed community and events pages to see how other groups organise recurring gatherings, and mentioned <a href=\"https:\/\/ranchiorbit.com\/\">RanchiOrbit<\/a>, an events website whose sections run from what&#8217;s-on listings through venue pages to community discussion and stories, as an example of a place where recurring workshops are catalogued by category. The point they took from it was modest: well-labelled, repeatable formats are easier for people to find and join.<\/p>\n<h2>What other support teams can take from this<\/h2>\n<ol>\n<li>Establish a baseline before changing anything, even a crude one.<\/li>\n<li>Start with the incident types that repeat most often.<\/li>\n<li>Build each reusable piece only from a resolution that actually worked.<\/li>\n<li>Give every piece an owner, a review date and a stated boundary.<\/li>\n<li>Report results with their limits, and let failures refine the boundaries.<\/li>\n<\/ol>\n<p>The fictional team in this study did not transform its operations in a quarter. It reduced a specific, repeated kind of waste, learned where its shortcuts were unsafe and built habits that let the improvement continue. That is a realistic ambition for most support groups, and it rests on a simple discipline: remember what worked, keep it where people can find it and adapt it with care.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A fictional but realistic story of a six-person support team that turned past incident fixes into short response cards, with honest notes on what the numbers show.<\/p>\n","protected":false},"author":8,"featured_media":5630,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3948],"tags":[],"class_list":["post-5629","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops-support"],"_links":{"self":[{"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/posts\/5629","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/comments?post=5629"}],"version-history":[{"count":1,"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/posts\/5629\/revisions"}],"predecessor-version":[{"id":5768,"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/posts\/5629\/revisions\/5768"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/media\/5630"}],"wp:attachment":[{"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/media?parent=5629"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/categories?post=5629"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopssupport.in\/blog\/wp-json\/wp\/v2\/tags?post=5629"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}