Incident response
Intermediate
8 min read
What to do when production breaks: stabilise first, communicate, then learn from it without blame.
Why it matters
Outages happen to every team. A calm, practised routine shortens them; a blameless review afterwards stops the same one happening again — and feeds the next plan.
Where it fits
What you will learn
- Follow the first steps when an alert fires.
- Prefer mitigation (rollback, switch off a feature) over a hurried fix.
- Write a short blameless postmortem.
Before you start
- Logs, metrics and alerts✓ Learned
During the incident
- Acknowledge
- Assess impact
- Mitigate
- Communicate
- Resolve
- One person coordinates; others investigate. Say who is doing what.
- Mitigate first: roll back the last release, switch off the new feature, add capacity. Find the root cause after users are fine.
- Post short status updates at regular times, even when nothing changed.
- Write down what you did and when — memory is unreliable at 3 a.m.
After the incident
- A timeline of what happened and when.
- The impact: who was affected, for how long.
- Why it happened — the conditions, not the person.
- A few concrete follow-up actions, each with an owner.
Tip: Being able to roll back in minutes is the best incident tool there is. Practise it before you need it.
Official documentation and references
Guides
- Managing Incidents (Site Reliability Engineering) ↗ Google SRE book
- Postmortem Culture: Learning from Failure ↗ Google SRE book
Learn next
- The software delivery lifecycle ›✓ Learned
Related topics
- Logs, metrics and alerts✓ Learned
- Backups and recovery✓ Learned
- The software delivery lifecycle✓ Learned
Done with this topic?
Reviewed 2026-10-23. Links go to the official documentation unless marked otherwise.