How to Triage a Production Error

DIAGNOSING A PRODUCTION ERROR — REPEATABLE WORKFLOW 1) CAPTURE FIRST EVIDENCE (do this before touching anything) - Timestamp + exact user-facing error text (screenshot/log line, not paraphrase) - Request ID / trace ID / correlation ID — the single most valuable token - Affected scope: one user, one tenant, one region, or all traffic? (check error-rate dashboard) - Deploy state: which commit/version was live at the time; any deploy in the last 24h - Logs around the window: app logs, plus infra (DB, queue, proxy) — 5 min before/after - Config/schema drift: any feature flag flipped, migration run, or env change recently - Preserve the payload: capture request body/headers and response body verbatim Rule: you can always delete evidence later; you can never recreate it. Screenshot/export before restarting anything. 2) REDUCE TO A MINIMAL REPRODUCTION - Replay the exact request (curl with the captured payload) against staging - Bisect inputs: strip headers, params, and body fields one at a time until it stops failing - Vary environment, not code: same input against prod vs staging vs local - If it only fails in prod: suspect data (specific row/account), config, or scale — reproduce with a copy of the real record - If it's intermittent: look for a race, timeout, or resource exhaustion (threads, connections, memory) rather than a deterministic bug - Write the repro as a test (failing test = the bug, once it passes you're done) Goal: a single command or test that fails in <30 seconds on a dev machine. 3) CHOOSE THE SMALLEST SAFE FIX - Fix the root cause, not the symptom — but ship the smallest change that addresses it - Prefer: guard clause / input validation / retry-with-backoff / correct config value over rewriting a subsystem - Avoid: big refactors, dependency bumps, or "while I'm in here" changes during an incident - Match the fix to the evidence: if you can't explain WHY it fixes it, you haven't found the root cause - Add the regression test from step 2 alongside the fix — it must fail before and pass after - Get a second pair of eyes on the diff; production fixes deserve review 4) VERIFY - Run the regression test + full relevant test suite - Deploy behind a feature flag or to a canary/one instance first - Watch the same error-rate dashboard you used in step 1 — confirm the metric that spiked is the one that recovers - Check for NEW errors: the fix can shift the failure elsewhere - Verify the original affected user/tenant specifically, not just aggregate metrics - Leave the window open long enough to catch the slow path (cron, batch, low-traffic hours) COMPACT CHECKLIST [ ] Error text, timestamp, trace/request ID captured verbatim [ ] Scope known: one user vs all traffic [ ] Deploy/flag/config changes in last 24h reviewed [ ] Logs + payload preserved before any restart [ ] Minimal repro: single command/test failing <30s [ ] Root cause stated in one sentence, fix matches it [ ] Regression test added (fails before, passes after) [ ] Fix deployed to canary first [ ] Original metric recovered, no new errors, original user verified [ ] Postmortem note written (even 3 lines) while memory is fresh REALISTIC EXAMPLE Symptom: users report "Checkout failed" intermittently, ~2% of orders, no pattern. Evidence: trace ID shows a 504 from the payment service; error rate spikes at 14:00–15:00 UTC daily; deploy history shows a queue worker version bumped 3 days ago; the failing requests all carry a large cart (12+ items). Repro: curl with a 12-item cart payload against staging reproduces it; stripping items to 11 makes it pass. The payment service times out because the worker serializes the whole cart to the payment payload, and >11 items exceeds the provider's 60s timeout. Fix: smallest safe change — cap cart size at 11 items with a clear validation error, plus a retry-with-backoff on the payment call. One guard clause + one retry, no refactor. Verify: regression test with a 12-item cart fails before the fix, passes after; deploy to canary; error-rate dashboard for the payment 504 drops to zero; confirm the original reporter's account checks out successfully; watch for 24h to catch the batch path. Done. The whole loop should take minutes for a clear bug, a day for a nasty intermittent one — the discipline is the same either way.