[ FOR TEAMS SHIPPING ON VERCEL + GITHUB ]

Stop the spread.

Roll back with proof.

Firebreak finds the deploy that broke production, stages a preflight-checked rollback, and proves recovery with automated health probes.

LIVE INCIDENT STREAM
02:14:07ALERTcheckout /api/health 500 (Invariant failed: pricing)
02:14:09SUSPECT41ee4db refactor: support stacked discounts (+79)
02:14:41APPROVEDrollback approved by operator (ws_demo)
02:15:12VERIFIED3/3 healthy · 100% streak · recovered in 65s
[01]THE FIRST 20 MINUTES

The slowest part of an incident is the part a script could do.

When production fails at 3 AM, engineers burn twenty minutes asking manual questions before writing a single line of a fix.

FIREBREAK AUTOMATION PRINCIPLE

Collect evidence, pinpoint suspects, verify rollback safety, and stage recovery for a single operator confirmation.

02:14 → 02:18Which deploy is live in production right now?
Instant API lookup
02:18 → 02:24What code changed since the last known-healthy deploy?
Git compare diff
02:24 → 02:31Which commit touches the error location & stack trace?
Deterministic ranking
02:31 → 02:37Is it safe to roll back? Did another deploy sneak in?
Live preflight drift check
02:37 → 02:44Did the rollback actually recover users?
Automated probe streak
WITH FIREBREAK: 02:14 → 02:1565 SECONDS TO RECOVERY
[02]THE INCIDENT RESPONSE LOOP

One continuous loop. No panic.

From the first 500 error to verified recovery in under two minutes.

01

Signed alert intake

DETECT
PHASE 01/06

HMAC signed alert with replay defense.

HMAC-SHA256 signature verified
t: 1729174447
{
    "event": "alert.triggered",
    "serviceId": "svc_demo_shop",
    "status": 500,
    "error": "Pricing invariant failed: expected 6120, got NaN",
    "location": "lib/pricing.ts:47:12"
  }
Incident #1 opened · Triggering automated commit trace pipeline
02

Deterministic ranking

TRACE
PHASE 02/06

Commits ranked by trace & diff symbols.

Ranked suspects across 3 commits:GitHub API compare
RANK 141ee4db
+79 pts

refactor: support stacked discounts

file: lib/pricing.ts (+40)symbol: applyDiscount (+25)token: NaN (+10)recent (+4)
3ab9d04 chore: update banner copy+4 pts
109b8ce docs: update checklist0 pts
03

Preflight verified rollback

PROPOSE
PHASE 03/06

Zero drift & target readiness verified.

ROLLBACK PROPOSAL3/3 PREFLIGHT PASSED
CURRENT BROKEN41ee4dbstacked discounts
TARGET HEALTHY533a8aastable discount engine
Target deployment exists and ready in Vercel
No drift: live production matches broken commit
Health check confirms active 500 error
04

Human in the loop

APPROVE
PHASE 04/06

Explicit operator confirmation required.

Operator Confirmation Required

Firebreak never modifies production without human authorization.

Expires in 14m 32s
05

Atomic Vercel swap

EXECUTE
PHASE 05/06

Zero-downtime production alias swap.

Vercel Alias Mutation in Progress...id: act_01
POST https://api.vercel.com/v2/deployments/.../aliases
→ 200 OK: firebreak-demo.vercel.app reassigned
Previous: dpl_broken (41ee4db)Active: dpl_healthy (533a8aa)
06

Probe recovery streak

VERIFY
PHASE 06/06

Probes prove 3/3 healthy responses.

RECOVERY VERIFIED: 3/3 PROBES HEALTHYRESOLVED
PROBE 1200 OK34ms
PROBE 2200 OK28ms
PROBE 3200 OK31ms
Total incident duration: 65 seconds · Zero manual rollbacks.
[03]EVIDENCE, NOT GUESSES

Every suspect has receipts.

Deterministic static analysis scores suspect commits before AI ever sees them.

lib/pricing.ts(applyDiscount)
commit: 41ee4db
TypeError: Cannot read properties of undefined (reading 'reduce')
@@ -32,7 +32,9 @@ export function applyDiscount(totalCents, discount) {
if (!discount) return totalCents;
- const pct = Math.min(Math.max(discount.percent, 0), 100);
+ // Refactor: support stacked discount rules
+ const totalPct = discount.stacked.reduce((acc, d) => acc + d.percent, 0);
return Math.round(totalCents * (1 - totalPct / 100));
}
DETERMINISTIC SCORE
+79/ 100 PTS
file: lib/pricing.ts+40 pts
symbol: applyDiscount+25 pts
token: 'reduce'+10 pts
recency: newest deployment+4 pts
Deterministic: Identical stack traces produce identical scores.
ADVISORY · CLAUDE 3.5 SONNETRoot Cause Analysis & Suggested Fix

Commit 41ee4db assumed discount.stacked is always an array. When a single discount arrives, calling .reduce() throws a TypeError.

Suggested Patch:const totalPct = Array.isArray(discount?.stacked) ? discount.stacked.reduce((acc, d) => acc + d.percent, 0) : Math.min(Math.max(discount?.percent || 0, 0), 100);
[04]WORKS WHERE YOU WORK

Same loop, inside your MCP client.

Exposes a Model Context Protocol server powered by NitroStack. Ask Claude or Cursor to inspect suspects directly.

# Add Firebreak MCP to your Claude Code project
claude mcp add firebreak \
  --transport sse \
  --url http://localhost:4000/sse \
  --header "Authorization: Bearer fbk_your_api_key"
9 MCP TOOLS
list_incidents
get_incident
get_suspects
get_analysis
get_timeline
list_services
propose_rollback
get_action_status
approve_action
Read ToolsGuarded Mutations
[05]SECURITY & CONTROL

Safe by construction.

Four hard invariants enforced before any route mutation can touch production.

01 // APPROVAL GATEMANDATORY

Explicit Human Approval

Rollbacks never execute autonomously; an engineer must hold to authorize.

OPERATOR_GATE800ms HOLD
TIMEOUT: 15m LEASEIDEMPOTENT: 1 ROLLBACK
02 // PREFLIGHT INVARIANTZERO DRIFT

Deployment Drift Verification

Preflight checks abort immediately if live traffic has drifted from the incident deploy.

TARGET_DEPLOY41ee4db
LIVE_PROD_ALIAS41ee4db
INVARIANT MATCH (Δ 0ms)ABORT ON MISMATCH
03 // ENVELOPE ENCRYPTIONAES-256-GCM

AES-256-GCM Encryption

Provider tokens are sealed at rest with tenant-scoped authenticated data.

SEALED_TOKENvercel_tok_••••••••9c2e
CIPHER_SPECAES-256-GCM · 256-BIT
AAD: sha256(tenant_id)128-BIT TAG VERIFIED
04 // AUDIT LEDGERAPPEND-ONLY

Immutable Audit Timeline

Every alert, suspect ranking, and route mutation is cryptographically signed.

02:14:07ALERTcheckout 500 spike
02:14:09RANK41ee4db (+79 pts)
02:14:41APPROVEoperator:on-call
02:14:42SWAP→ 3ab9d04 [VERIFIED]
HASH: 0x8f2d...b41aAPPEND-ONLY SEAL
[06]FREQUENTLY ASKED QUESTIONS

Answers to common questions.

Everything you need to know about safety, permissions, and rollback mechanics.

No. Every rollback requires explicit human approval, and Firebreak validates live production for drift milliseconds before executing the alias mutation.
[ READY FOR PRODUCTION ]

Ship on Friday.

Sleep on Saturday.

Stop guessing during outages. Connect your GitHub repository and Vercel project in sixty seconds.