Mitigate before you diagnose

TL;DR: Restore service with the fastest safe action available, then diagnose with the pressure off. Use a rehearsed rollback, failover, load-shedding rule or feature flag when its safety conditions hold. These actions can reduce impact before you know the root cause. Continue diagnosis during recovery where it helps, and complete causal analysis after immediate impact is controlled.

The two goals compete

An engineer looking at a broken system wants to understand it. That instinct is correct in every context except this one. While you read logs, users are still failing, and the error budget is still burning.

The discipline is to separate the questions. What stops this now is answerable in seconds from a short list. Why did this happen takes as long as it takes, and it can happen after the graphs recover.

The mitigation menu

These common mitigations may work before full root-cause analysis, but each requires enough diagnosis to establish its safety conditions:

• Roll back. If the timing correlates with a deploy and rollback is compatible with current state, reverse it. You do not need to know which line broke.
• Fail over. Move traffic to the healthy region, zone, or replica set.
• Shed load. Drop the lowest-value traffic so the highest-value traffic survives. Better than everything failing equally.
• Disable the feature. A flag is a mitigation, which is the strongest argument for having them.
• Scale up, when the constraint is capacity and capacity is available.
• Break the loop. Cascading failures need the retry storm or the feedback cycle interrupted, which is often a config change rather than a fix.

Keep these options in the runbook, with the conditions that make each safe.

When diagnosis has to come first

The rule is not absolute, and knowing the exceptions is what makes it a judgement rather than a slogan:

• Data corruption. Rolling back code while a bad writer keeps running can make the damage worse. Establish what is being written before you act.
• A rollback that is destructive. If the deploy removed a column or changed a stored format, old code may no longer work; reversing the migration may be destructive or impossible without a backup. That has to be known before the button is pressed.
• Actions that discard evidence. Restarting the process loses the state you needed. Capture a bounded diagnostic sample if it will not delay recovery or worsen pressure. A heap dump can pause the process, consume substantial disk and expose sensitive data; it is not a mandatory step before restarting.
• You cannot tell what is broken. If the mitigation would be a guess with real blast radius, spend two minutes narrowing rather than acting blindly.

What the delay actually costs

The instinct to diagnose first feels responsible, so it is worth pricing rather than arguing about. Take a service with a 99.9 percent availability objective over 30 days. The budget is 30 × 24 × 60 × 0.001 = 43.2 minutes of unavailability for the whole month.

Now run the same incident twice. A deploy at 14:00 breaks the service at 14:02. In the first version somebody checks rollback compatibility and rolls back, and service is restored at 14:07: five minutes, which is 5 / 43.2 ≈ 11.6 percent of the month's budget. In the second version the same engineer reads logs, finds the null dereference at 14:45 and ships a fix that is live at 14:52: fifty minutes, or 50 / 43.2 ≈ 116 percent of the budget.

The second incident consumed the entire month's allowance and then 16 percent more, in one morning, and the two engineers made the same diagnosis in the end. The difference is only when they made it. That is the whole argument, and it is an argument about arithmetic rather than about temperament.

Two things the calculation does not settle. It assumes the rollback was safe, and establishing that is itself a diagnostic step, which is why the exceptions below are part of the rule rather than a caveat to it. And it says nothing about the value of the eventual diagnosis, which is real and which the postmortem exists to capture; the claim is about sequence, not about whether understanding matters.

The habit that makes it possible

Mitigations have to be fast and safe before they are useful, which means they are built in advance, not found during the incident:

• rollback that is one command and does not depend on a rebuild
• feature flags on anything risky, with a documented owner
• a load-shedding path that has been exercised
• a failover that has been rehearsed rather than diagrammed

An organisation that cannot roll back in minutes will diagnose first every time, because it has no other option. That is the real reason this principle is about engineering rather than about discipline.

Write the stop condition before applying the mitigation

For a proposed traffic failover, record the destination's available capacity, data freshness and the signal that would stop the move. A region accepting TCP connections can still lack the current data or dependency access required for checkout. Shift a bounded cohort where the system permits it and verify the real operation.

For a feature disablement, confirm the flag reaches the relevant replicas and that cached configuration does not leave a mixed fleet indefinitely. Observe the affected customer cohort, not only the global average. Keep a timestamped record of the change so recovery can be compared with its actual application.

If the mitigation fails its stop condition, return to the known safe state when possible and choose the next action using the new evidence. Do not keep stacking changes without recording their effects. Parallel diagnosis is useful when another responder can do it without delaying containment.

Self-check

A deploy went out at 09:14. At 09:16 the error rate on checkout goes from near zero to 40 percent. You are on call, the service has a 99.9 percent monthly objective, and 12 minutes of budget have already been spent this month. Two facts emerge in the first minute: the release included a migration that dropped a column the previous binary still reads, and the new binary is writing to a new column that the previous one does not know about.

Decide whether to roll back, say what you would do instead and in what order, and work out what the delay costs while you decide. Work it through before reading on.

The budget arithmetic first, because it sets how much time you have to think. The month's allowance is 43.2 minutes and 12 are gone, leaving 31.2. At a 40 percent error rate the service is not fully down, so under a constant request-rate approximation for a request-based objective, you are burning roughly 0.4 equivalent minutes of budget per minute elapsed and have about 31.2 / 0.4 = 78 minutes before the budget is exhausted. That is enough time to act carefully and not enough to investigate first, which is the useful thing to know in minute one.

A binary rollback is not safe here, and this is the exception the page names rather than a reason to abandon the rule. The old binary reads a column that no longer exists, so rolling back replaces a 40 percent error rate with a service that cannot start or fails on every read of that column. Reversing the migration is worse: the new binary has been writing to the new column since 09:14, so recreating the old column leaves you with recent writes that only the new schema records and a restore point that predates them.

So the ordering is dictated by what is irreversible. Stop the damage from growing before repairing anything, which here means stopping new writes on the affected path: disable the feature by flag if one exists, or shed that traffic, so the divergence between the two schemas stops widening. That buys time without destroying evidence and without a schema change. Then fix forward rather than backward, because the forward direction is the one the data already supports: the smallest change that makes the new binary stop erroring is a far safer target than trying to reach a state the data has left behind. And before anything, capture what the current state is, since you will need to reconcile the rows written between 09:14 and now, and that reconciliation is a separate task with its own owner.

The general lesson is the one that makes this a judgement rather than a slogan. Mitigate first is a rule about sequence, and the sequence is chosen by reversibility rather than by speed. A rollback is usually the fastest safe action, which is why it is the default; when the deploy changed durable state, the fastest action and the safe one have come apart, and the rule still applies with a different action at the front of it.

Next: the diagnostic method covers the narrowing that establishes whether a mitigation is safe, and error budgets and burn rate is where the 43.2 minutes comes from.