Launch-Week Runbook for New Web Feature

Here's the runbook. Print it landscape or as-is, tape it next to the monitors. ════════════════════════════════════════════════════════════ LAUNCH-WEEK RUNBOOK — [FEATURE NAME] Launch date: ______ Go/no-go at: ______ Owner: ______ ════════════════════════════════════════════════════════════ 1. PRE-LAUNCH CHECKLIST (T-3 days → T-0) ---------------------------------------- ☐ Feature flag in place, default OFF ☐ Rollback dry-run done: flag flip + code revert, both timed ☐ Dashboards live: error rate, p95 latency, throughput, conversion, signups, auth failures ☐ Alerts configured at thresholds below ☐ Load test passed: 3x expected peak, 30 min, zero errors ☐ DB migrations applied and backward-compatible ☐ Third-party deps checked: CDN, auth provider, payment API ☐ On-call roster published (names + phones) ☐ Comms templates drafted: internal, status page, customer ☐ Data backup/restore verified (if schema changes) ☐ Stakeholders notified of go/no-go time ☐ FINAL GO/NO-GO MEETING: T-2 HOURS 2. DAY-OF MONITORING ROTATION ---------------------------------------- Two-person shifts, 4 hours each: 08:00-12:00 | 12:00-16:00 | 16:00-20:00 20:00-24:00 | 00:00-04:00 | 04:00-08:00 Primary: watches dashboards, triages, declares incidents Secondary: backup; owns comms while Primary works the incident Cadence: - Dashboards: every 15 min - Alert page: every 5 min for first 2 hours after launch - First 2 hours: both people on, no meetings, no deploys Handoff (each shift change, 5 min): ☐ Review dashboards together ☐ Open incidents + current state ☐ Alert status (what fired, what was silenced) ☐ Notes / anything weird seen 3. COMMUNICATION TREE ---------------------------------------- IC (Incident Commander) ......... Primary on shift Deputy .......................... Secondary on shift Engineering lead (escalation) .. ______________ Product owner ................... ______________ Comms / status page owner ...... ______________ Support lead .................... ______________ Final escalation ................ CTO Escalation path: Primary -> Eng lead -> CTO Comms rules (clock starts at first alert): - 5 min: acknowledge alert in team channel - 15 min: internal status post (what/impact/who) - 30 min: update status page IF customer-facing impact - Every 30 min: refresh update during any incident - One voice: only IC or Comms owner posts externally - Support lead gets a heads-up before any customer sees it 4. ROLLBACK DECISION THRESHOLDS ---------------------------------------- ROLL BACK IMMEDIATELY on ANY ONE of these: - Error rate > 1% of requests for 5 min (baseline < 0.1%) - p95 latency > 2x baseline for 10 min - 5xx rate > 0.5% for 5 min - Throughput drop > 30% vs. same-time baseline for 10 min - Conversion/signups drop > 20% vs. 7-day avg for 30 min - ANY data corruption, duplicate writes, or P0 security issue -> instant rollback, no waiting on thresholds - 3+ distinct alert types firing at once Rollback procedure (timed): - Flip feature flag OFF ................. 2 min - If flag not enough: deploy previous release ................................ 10 min - Announce rollback ...................... 5 min - TOTAL TARGET: < 15 MINUTES 5. KILL CRITERIA — ABANDON THE LAUNCH ---------------------------------------- KILL = full stop. No relaunch until engineering sign-off. Triggered by ANY ONE: - Two rollbacks in the same launch window - Rollback fails: service not restored within 30 min - Data loss or corruption hitting real users - P0 security vulnerability found - Sustained outage > 45 min despite rollback - Root cause unknown after 2 hours of active work - Impact to > 5% of active users, or any named enterprise/critical account - Same failure mode repeats after two attempted fixes After a kill: - Incident report within 48 hours - Relaunch requires written go/no-go sign-off from engineering lead + product owner - No relaunch in the same week ════════════════════════════════════════════════════════════ RULE OF THUMB: When in doubt, roll back. A rollback is a Tuesday. A kill is a quarter. Don't let the second happen to avoid the first. ════════════════════════════════════════════════════════════ Fill in the blank names/feature before printing. If you want it as a file or a one-page PDF, say the word and I'll write it out.