Appendix

Quick Reference: Incidents

Stabilize first. Communicate always. This is containment, not design.

The Priority Order

  1. Assess impact. Who's affected? How badly? Getting worse?
  2. Stabilize. Stop the bleeding. Fastest reversible mitigation.
  3. Communicate. Tell stakeholders what's happening.
  4. Diagnose. Now find the root cause. Not before.
  5. Resolve. Fix it properly. Verify before declaring victory.

First Ten Minutes

  • [ ] Breathe. Panic makes you stupid.
  • [ ] What's the signal? What's affected? How bad?
  • [ ] If it's real, declare it: "We have an incident. I'm investigating."
  • [ ] Get the right people. Don't be a hero.
  • [ ] Check the obvious: recent deploys, config changes, upstream dependencies.
  • [ ] Say something, even "we're aware and investigating."

The Rollback Decision

Roll back if:

  • [ ] A recent change is the likely cause
  • [ ] Rolling back is low-risk
  • [ ] You can verify it fixes the issue

Don't roll back if:

  • [ ] Data migrations can't be undone
  • [ ] Side effects are irreversible (payments, emails, external state)
  • [ ] Rollback risks a different outage

When in doubt: fastest reversible mitigation. Rollback is often that, not always.

Communicate Under Fire

  • [ ] Update every 15-30 minutes, even with nothing new
  • [ ] Impact, status, timeline, next update
  • [ ] No jargon. No speculation. "We're investigating" is fine.
  • [ ] Document actions and timestamps as you go
  • [ ] Use the Incident Update template

Roles (one person can wear all hats)

  • Incident Commander: owns decisions and cadence
  • Technical Lead: owns diagnosis and fix
  • Communications Lead: owns stakeholders and status page
  • Scribe: owns the timeline

Declaring Resolved

  • [ ] Immediate problem fixed
  • [ ] Users working normally
  • [ ] Metrics back to baseline
  • [ ] No further intervention needed

"Resolved" means users are no longer affected, not "root cause fixed forever."

After

  • [ ] Blameless post-mortem, scheduled immediately (template)
  • [ ] Systems, not individuals: what allowed this?
  • [ ] Action items with owners and deadlines
  • [ ] Same incident twice means you didn't follow through

Full doctrine: Incidents.