Blameless Postmortem Template

Blameless Postmortem Template

Incident Metadata

• Incident ID:
• Date/Time (UTC):
• Severity:
• Services affected:
• Incident Commander:

Summary

• What happened:
• Customer impact:
• Duration:

Timeline (UTC)

1. Detection:
2. Triage:
3. Mitigation:
4. Recovery:
5. Closure:

Root Cause Analysis

• Primary cause:
• Contributing factors:
• Why safeguards did not prevent it:

What Worked Well

What Didn’t Work

Action Items

1. Action:

• Owner:
• Due date:
• Validation method:

2. Action:

• Owner:
• Due date:
• Validation method:

Prevention & Hardening

• Guardrails to add or tighten:
• Alerting/SLO improvements:
• Runbook updates required:

AI Assistance Review

• Where AI helped:
• Where human judgment was critical:
• Any AI recommendation rejected and why:

Lab: Full Incident Lifecycle (24/7 SRE)

Lab: Full Incident Lifecycle (24/7 SRE)

Goal

Run one full lifecycle simulation:

• detect
• triage
• mitigate
• recover
• postmortem

Scenario Input

Use one recent controlled scenario (recommended from Chapter 11/12):

• backend crash/panic pattern, or
• elevated 5xx with recurring incidents

Step 1: Incident Declaration

Define:

• severity (SEV-1/SEV-2/SEV-3)
• blast radius
• IC and responder roles
• comms channel and update cadence

Step 2: Evidence Collection

Capture:

• symptom metrics
• representative trace(s)
• correlated log evidence
• guardian incident id (if available)

Step 3: Mitigation Decision

Choose one:

Quiz: Chapter 13 (24/7 Production SRE)

Quiz: Chapter 13 (24/7 Production SRE)

Questions

1. What is the first priority in the first minutes of an incident?

2. Which statement is correct?

• A) Decide mitigation first, collect evidence later.
• B) Collect evidence first, then choose mitigation.
• C) Wait for AI confidence to reach 100%.