The incident narrative: explain decisions with evidence

TL;DR: Explain an incident through user impact, your decisions and the evidence available when you acted. Distinguish mitigation from confirmed recovery, attribute other people's work accurately and show which follow-up was actually completed.

Build the account from the record

An incident narrative helps a reviewer understand how you reasoned under uncertainty. Start with an incident you actually handled and reconstruct it from material you are allowed to use: your notes, an approved postmortem or an anonymized timeline. Do not take confidential documents into an interview or manufacture details to fill a story template.

Name the service's useful operation before naming its infrastructure. “Customers could submit an order but could not see confirmation” gives a listener a consequence to follow. Then explain the part of the system necessary to understand it. A complete architecture history is rarely needed to assess the first recovery decision.

Separate four kinds of statement

The exercise is fictional. In your own account, preserve the distinctions even when the final diagnosis now seems obvious. Knowledge gained during the postmortem should not be presented as something you knew on the bridge call.

Walk through a decision that could have gone differently

Suppose the new version's requests have longer connection-holding times than comparable requests on the retained version. That observation strengthens the release hypothesis. It still leaves questions: are the request sizes comparable, did traffic shift between database shards, and can the old version safely read data written by the new one?

A useful explanation names the checks you performed before recommending a mitigation. If another responder verified schema compatibility, credit that person. State who approved the action and what you owned, such as collecting the cohort comparison or checking new requests afterward.

Restoring the previous application version may not clear existing sessions or repair persistent data. If latency remains high, verify that the rollback actually completed and inspect those remaining effects. A failed mitigation does not automatically disprove the release hypothesis. Explain the next observation that changed the investigation.

You do not need a wrong turn. If the first hypothesis was correct, show the evidence used to test it and the alternatives the observation made less likely. Adding a mistake to make the account sound authentic introduces false information rather than useful depth.

Define when impact ended

Detection, mitigation and recovery can have different timestamps. In the exercise, traffic routing finishes at 10:14 and fresh confirmations recover at 10:17. Earlier timed-out orders still require a reconciliation pass. Describe the restored request path and the unresolved business work separately; neither timestamp alone establishes that every affected customer is whole.

Use measured numbers where available, with their population and window. A percentage of failed requests is not automatically a percentage of affected customers, and the gap between those two is large enough to be worth working through once.

Continue the fictional exercise. The confirmation path takes about 400 requests a minute, and the window from 10:02 to 10:17 is 15 minutes, so roughly 400 × 15 = 6,000 requests were eligible. Of those, 2,280 failed, which is 2,280 / 6,000 = 0.38, or 38 percent. That figure is true and it is about requests.

Now count people. Those 2,280 failures came from 950 distinct customer identifiers, an average of 2,280 / 950 = 2.4 failed attempts each, because a customer who does not see a confirmation reloads the page. So describing the impact as "2,280 customers could not see their confirmation" overstates it by a factor of 2.4, and the error is in the direction that flatters the severity of your own incident.

Then count the people who were actually left without the thing they came for. Of the 950, suppose 620 succeeded on a later attempt inside the window, leaving 950 − 620 = 330 who did not. Three numbers, all measured, all true, describing different things: 2,280 failed requests, 950 affected customers, 330 customers who never got a confirmation during the incident.

Which of the three belongs in the account depends on what is being asked, and saying which one you are quoting is the part that matters. The 330 is the figure a business consequence rests on. The 2,280 is the figure an error-budget calculation uses. Quoting one and letting a listener assume the other is how an account becomes misleading without containing a false statement.

If exact figures are unavailable or confidential, explain the limitation and use a truthful qualitative description. Do not infer revenue loss from request volume without the relevant business evidence.

Make the follow-up testable

“Added monitoring” leaves the reader unable to assess the change. A stronger account describes an alert on connection wait for the affected operation, the failure fixture that triggers it and who owns its response. If the proposed detector would only have fired after customer reports, do not claim it would have prevented the original delay.

Keep action status accurate. A ticket created, a patch merged and a control operating in production are different milestones. State the one you reached and what remained. The postmortem guidance in the Google SRE Workbook provides background on learning from incidents and following through on actions; the example here is an independent practice scenario.

Prepare a spoken version and supporting detail

Start with a short account that covers the operation, the decision you owned and the recovery evidence. Retain a deeper explanation of the disputed mechanism for follow-up questions. Use the outage ownership question to rehearse the spoken answer and the incident handoff lesson to practise an account that another responder could act on.

Self-check

Here is a draft account of the exercise above, of the kind people give from memory. Read it and find everything in it that you could not defend under a follow-up question.

"We had an outage where the database ran out of connections. I spotted that it was the new release, rolled it back, and the graphs recovered at 10:14. It affected about 38 percent of customers. Afterwards I added monitoring so it can't happen again."

Work out what is wrong with each sentence before reading on. There are five distinct problems and one of them is not a factual error at all.

The first sentence states a diagnosis as the starting point. "The database ran out of connections" was a hypothesis at 10:02 and became a conclusion during the postmortem, and presenting it as the opening fact quietly claims you knew on the bridge call what you learned afterwards. The honest version leads with the observation and the user consequence: confirmation requests began timing out after 10:02, and customers could submit an order but not see confirmation.

"I spotted that it was the new release, rolled it back" compresses three things that were separate, and two of them were other people's. Somebody checked schema compatibility; an incident commander approved routing traffic; you collected the cohort comparison. Crediting those accurately is not modesty, it is the difference between an account a reviewer can follow and one that cannot survive "who else was on the call". This is also the problem that is not a factual error: every clause in it may be individually true while the whole sentence claims sole ownership by omission.

"The graphs recovered at 10:14" conflates the mitigation with the recovery. Routing finished at 10:14 and fresh confirmations recovered at 10:17, and orders that had already timed out still needed a reconciliation pass that no graph reflects. So the account should give the restored request path and the unresolved business work separately, and neither timestamp establishes that every affected customer is whole.

"About 38 percent of customers" is the arithmetic above, used on the wrong population. Thirty-eight percent was the share of requests that failed; the customer figure was 950, and the number left without a confirmation was 330. Quoting the request percentage against the word customers overstates the impact by the retry factor, and it is the kind of error a reviewer who has run an incident will spot immediately.

"Added monitoring so it can't happen again" fails on two counts. It is untestable as written, where the stronger version names an alert on connection wait for the affected operation, the fixture that triggers it, and who owns the response. And "can't happen again" is a claim about a class of failure that monitoring does not make; detection shortens an incident and does not prevent one. If the proposed detector would only have fired after customer reports, saying so is more credible than the claim.

The pattern across all five is a single habit: say what you observed, when you knew it, who decided, and what you verified, and keep those four apart even when the diagnosis now looks obvious. Next: rehearse the spoken version against the outage ownership question, and the incident handoff lesson practises an account another responder could act on.