Game Day Scorecard (Template)

Game Day Scorecard (Template)

• Date:
• Environment:
• Scenario:
• Driver:
• Incident Commander:
• Observer:

Detection

• First symptom timestamp:
• Detection signal:
• MTTD (minutes):

Triage

• Representative trace id:
• Correlated log evidence:
• Hypothesis quality (low/medium/high):

Recovery

• Mitigation applied:
• Recovery timestamp:
• MTTR (minutes):

Signal Quality

• Metrics usefulness (1-5):
• Traces usefulness (1-5):
• Logs usefulness (1-5):
• Alert noise assessment:

Outcome

• Blast radius within expectation: yes/no
• Guardrails respected: yes/no
• Follow-up hardening action:
• Owner:
• Due date:

Lab: Controlled Chaos with Safety Guardrails

Lab: Controlled Chaos with Safety Guardrails

Goal

Run one deterministic failure drill and one Chaos Monkey drill in develop:

• confirm detection
• run incident workflow
• verify recovery

Prerequisites

kubectl -n flux-system get kustomization chaos-monkey-develop
kubectl -n develop get deploy frontend backend
kubectl -n observability get prometheusrule backend-alerts backend-slo-rules

Step 1: Deterministic Drill (Backend 5xx)

Generate controlled 5xx from frontend Chaos page or directly:

kubectl -n develop port-forward svc/backend 8080:8080
for i in $(seq 1 40); do curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/status/500; done

Expected:

Quiz: Chapter 11 (Controlled Chaos)

Quiz: Chapter 11 (Controlled Chaos)

Questions

1. Why should chaos drills start in develop and not production?

2. Which CronJob field is the primary kill switch for Chaos Monkey?

3. In this repo, what target app labels are allowed for monkey pod deletion?

4. Which incident flow is required before mitigation decisions?

5. Which statement is correct?

• A) Chaos drills are successful without evidence if service recovers.
• B) Controlled chaos requires blast-radius limits and rollback path.
• C) Chaos Monkey should run in all namespaces for realistic behavior.