Incident Recovery Workflows
6-STEPThis guide details the closed-loop recovery state machine from initial error alert to verified restoration.
4 min read
•Production Verified
End-to-End Execution Sequence
Chronological Sequence: Webhook Ingest, Multi-Factor Bisection, Human Approval, Rollback, and Canary Verification
The 8-Factor Commit Blame Scorer
Every commit shipped in a broken release bundle is evaluated against the runtime error and stack trace using an 8-factor mathematical scoring model:
| Factor | Scoring Rule | Points Awarded |
|---|---|---|
| 1 | Exact File Match in Stack Trace | +40 pts |
| 2 | Line Number in Diff Hunk Range | +30 pts |
| 3 | Function / Symbol in Diff Patch | +25 pts |
| 4 | Symbol Mentioned in Commit Message | +25 pts |
| 5 | Same Directory Match | +10 pts |
| 6 | Error Token in Commit Message | +10 pts |
| 7 | Error Token in Diff Patch | +8 pts |
| 8 | Recency Decay (newer commits favored) | Up to +10 pts |
The 6 Preflight Rollback Safety Gates
Before an operator can approve a rollback, 6 safety assertions are evaluated:
| Check | Type | Safety Assertion |
|---|---|---|
| target_ready | Blocking | Target deployment on Vercel must be in READY state. |
| target_differs | Blocking | Target deployment ID and SHA must differ from current broken production. |
| target_not_known_bad | Blocking | Target deployment has not caused an earlier recorded incident. |
| no_drift | Blocking | Live production has not changed since the proposal was generated. |
| not_expired | Blocking | Proposal must be approved within its 15-minute Time-to-Live. |
| still_failing | Warning | Advisory check confirming service is still failing (turns yellow if recovered). |