Incidents

When production is on fire.

The Nature of Incidents

Production is broken. Users are affected. The pager goes off. Your heart rate spikes.

This is an incident. Not a bug to fix at your leisure. Not a task for the backlog. An emergency requiring immediate response.

How you handle incidents defines you as an engineer. Not because heroics matter, but because incidents reveal your training, your systems, and your character under pressure.

Name the Mode

I found a snake in or around a birdhouse once. I was doing routine work around the property, and snake removal was nowhere on the list until it was suddenly the whole list.

The tempting move was to treat it like any other cleanup job: reach in, get it out, done. But the unknown was alive and could move. I did not know exactly where it was, whether other openings existed, or where it would go if I disturbed the structure before I was ready. Disruption before containment could turn one known problem into several unknown ones.

So containment came first. Cut, pre-stage, seal, remove, dispose. Every other birdhouse got patched before removal began. The recovery path existed before anything was disturbed.

The lesson was not about snakes. It was that the work had changed mode, and I almost did not notice. Since then I sort this kind of work into three modes, each with its own questions:

Design

Make future change easier. What deserves to exist? What should it be called? Where are the boundaries?

Containment

Stop the current problem from spreading. What is the blast radius? What can move? What must be isolated first? What recovery path must exist before intervention?

Cleanup

Remove what no longer earns its space. What is obsolete? What temporary measure has outlived its purpose? (Technical Debt lives in this mode.)

An incident is containment. You don't refactor a snake; you contain it. A live incident is not a design problem: bound the unknown, reduce the blast radius, pre-stage recovery, then touch the system. Containment finishes before disruption starts. The elegant redesign can come later, and it will still be there when the system is stable.

The modes overlap in practice. Containment may require a small design decision; cleanup may expose an architectural problem. The taxonomy names the primary objective; it does not pretend the work fits in boxes. And temporary patches become architecture when forgotten, which is how containment work quietly turns into cleanup debt.

Stabilize Before You Diagnose

This is counterintuitive but critical. Your instinct is to understand what went wrong, find the root cause, fix it properly. But users are suffering now. Every minute you spend diagnosing is a minute they're down.

Get users back to a working state first. Roll back the bad deploy. Switch to the backup. Turn off the broken feature. Then, with users stable, figure out what happened.

The rollback decision has real teeth on both sides. Roll back when a recent change is the likely cause, the rollback is low-risk, and you can verify it worked. Don't roll back when migrations can't be undone, side effects are irreversible (payments processed, emails sent, external state changed), or the rollback itself risks a different outage.

The principle: When in doubt, choose the fastest reversible mitigation. Rollback is often that, but not always.

And communicate constantly while you work: impact, status, timeline, next update, every 15 to 30 minutes, even when nothing changed. Silence during an incident is terrifying; people imagine the worst, start poking around, and escalate unnecessarily. "We're investigating" is fine. Speculation is not. Document actions and timestamps as you go; the post-mortem will need them. (The Incident Update template holds the format.)

Latent vs Live

Investigation turns up hazards that are not the incident. Earlier, The Body Plan named the triage call for them: latent vs live. In incident terms, a hazard is live if a current path can reach it, and latent if it is reachable only through legacy or raw paths that nothing currently exercises.

On one system I reviewed, an update-all keyed on a nullable field would have collapsed to "update every null row" for records that never carry the field. Alarming on first read. But a model validation guarded the field's presence, so the hazard was reachable only through legacy or raw-write paths. Latent, not live. The guard that followed was defensive hardening, honestly framed, not an incident fix.

The call changes the urgency, the testing, and the story you tell the team. Calling latent hazards live spends incident urgency on hardening work. Calling live hazards latent leaves incident risk active.

Learn From Everything

The incident is over. Resolved means users are no longer affected; it does not mean the root cause is fixed forever. The follow-up work goes through a blameless post-mortem.^[1]

The goal is learning, not punishment. If people are afraid of blame, they'll hide information and you'll learn nothing. The question isn't "who screwed up?" It's "what allowed this to happen, and what changes so it doesn't again?" Work the timeline, the root cause and its contributing factors,^[2] how you detected it, and what would make detection and recovery faster. Then create the tickets, assign the owners, and track them: a post-mortem without follow-through is theater. When the same incident recurs, treat it as a follow-through question before a technical one; sometimes the fix genuinely takes longer than the interval between failures, but the burden of proof is on that claim.

You Fall to Your Training

In a crisis, you won't rise to the occasion. You'll fall to the level of your training. This is You Fight the Way You Train applied to incidents: know your systems, know your escalation paths, keep the runbooks current, and practice with drills before the real page arrives.

The military drills this relentlessly. Emergency procedures, response protocols, communication standards: all practiced until they're automatic. Not because anyone enjoys repetition, but because in an actual emergency, you won't have time to think. You'll do whatever you've done a hundred times in training. The purpose of drilling is to make the right response feel boring. When the actual crisis hits, you execute the same steps you've executed in every drill. Nothing special. Just the process.

Calm is contagious. So is panic. Breathe, speak slowly, follow the process. It exists for a reason.

The 2 AM Layer

This chapter is the doctrine. When the pager actually fires, use Quick Reference: Incidents: the priority order, the first ten minutes, the rollback checklist, the comms cadence, and the roles, as checkboxes. The response structure this chapter no longer surveys is exactly what the quick reference holds.

Stay calm. Stabilize first. Communicate always. Learn from everything. You fall to your training, so train well.