Template
Incident postmortem template
A blameless postmortem in plain Markdown – summary, impact, timeline, root cause, and owned action items. Publish it to a URL in one click, no account.
- free – no account
- a URL in seconds
- yours if you claim it
Publishing this from your own stack? See display.dev for internal docs.
What it looks like published
Postmortem: checkout API outage, 18 July 2026
Severity: SEV-1 · Duration: 42 minutes (14:03–14:45 UTC) · Author: Marta K. · Status: Reviewed
Summary: A database migration added an index in a lock-taking mode, which blocked writes on the
orderstable. Checkout requests timed out for 42 minutes. Roughly 1,900 checkout attempts failed; no data was lost and no double charges occurred. The fix: kill the migration, rerun it as a non-blocking concurrent index build.
Impact
- 1,912 checkout attempts returned 504 between 14:03 and 14:45 UTC.
- Support received 31 tickets; status page updated at 14:19 (16 minutes in – too slow, see actions).
- No dropped orders after retry: 88% of affected customers completed checkout within an hour of recovery.
Timeline (UTC)
| Time | What happened |
|---|---|
| 14:01 | Deploy starts; migration add_orders_status_idx begins |
| 14:03 | Write latency on orders spikes; first 504s |
| 14:07 | Alert fires (p99 latency > 2 s for 3 min) |
| 14:12 | Incident declared, war room opened |
| 14:19 | Status page updated |
| 14:31 | Root cause identified: index build holding a write lock |
| 14:38 | Migration killed; lock released |
| 14:45 | Error rate back to baseline; incident downgraded |
| 15:20 | Index rebuilt concurrently without customer impact |
Root cause
The migration used the ORM's default index creation, which takes a table
lock for the duration of the build. On the 40 GB orders table the build
ran for 35+ minutes. Staging didn't catch it because staging's orders
table is 200 MB – the lock cleared in seconds there.
What went well
- The latency alert fired within 4 minutes of first impact.
- Killing the migration was a clean, rehearsed rollback path.
What went poorly
- 16 minutes to a status-page update; customers found out from timeouts.
- Migration review didn't flag the locking mode – no checklist item for it.
Action items
| Action | Owner | Due |
|---|---|---|
| Lint rule: index creation must use the concurrent mode | Marta | 25 Jul |
| Status-page update within 10 min is part of the SEV-1 runbook | Deniz | 25 Jul |
Staging seeds a production-scale orders table monthly | Ravi | 8 Aug |
Source – incident-postmortem.md
# Postmortem: checkout API outage, 18 July 2026
**Severity:** SEV-1 · **Duration:** 42 minutes (14:03–14:45 UTC) ·
**Author:** Marta K. · **Status:** Reviewed
> **Summary:** A database migration added an index in a lock-taking mode,
> which blocked writes on the `orders` table. Checkout requests timed out
> for 42 minutes. Roughly 1,900 checkout attempts failed; no data was lost
> and no double charges occurred. The fix: kill the migration, rerun it as
> a non-blocking concurrent index build.
## Impact
- 1,912 checkout attempts returned 504 between 14:03 and 14:45 UTC.
- Support received 31 tickets; status page updated at 14:19 (16 minutes in
– too slow, see actions).
- No dropped orders after retry: 88% of affected customers completed
checkout within an hour of recovery.
## Timeline (UTC)
| Time | What happened |
|---|---|
| 14:01 | Deploy starts; migration `add_orders_status_idx` begins |
| 14:03 | Write latency on `orders` spikes; first 504s |
| 14:07 | Alert fires (p99 latency > 2 s for 3 min) |
| 14:12 | Incident declared, war room opened |
| 14:19 | Status page updated |
| 14:31 | Root cause identified: index build holding a write lock |
| 14:38 | Migration killed; lock released |
| 14:45 | Error rate back to baseline; incident downgraded |
| 15:20 | Index rebuilt concurrently without customer impact |
## Root cause
The migration used the ORM's default index creation, which takes a table
lock for the duration of the build. On the 40 GB `orders` table the build
ran for 35+ minutes. Staging didn't catch it because staging's `orders`
table is 200 MB – the lock cleared in seconds there.
## What went well
- The latency alert fired within 4 minutes of first impact.
- Killing the migration was a clean, rehearsed rollback path.
## What went poorly
- 16 minutes to a status-page update; customers found out from timeouts.
- Migration review didn't flag the locking mode – no checklist item for it.
## Action items
| Action | Owner | Due |
|---|---|---|
| Lint rule: index creation must use the concurrent mode | Marta | 25 Jul |
| Status-page update within 10 min is part of the SEV-1 runbook | Deniz | 25 Jul |
| Staging seeds a production-scale `orders` table monthly | Ravi | 8 Aug |
What this template is for
The write-up that turns a bad afternoon into a better system. A summary a director can read in twenty seconds, a timeline in UTC, a root cause stated plainly, and action items that name an owner and a date – the two fields that decide whether a postmortem changes anything.
The structure is blameless by design: "what went well / what went poorly" describes the system and the process, not people.
How teams use it
- Drafted by an agent from the incident channel. Timestamps, alerts, and the deploy log are already machine-readable; the agent assembles the timeline, the responder corrects the narrative.
- Shared beyond the on-call rotation. Postmortems age well as institutional memory when they live at one URL your whole org can open behind its Google or Microsoft sign-in – not in a channel that scrolls away.
- Reviewed, then published as a version. The draft goes up, reviewers leave inline comments, the final version publishes to the same URL.
Research report template
An HTML research report – TL;DR, findings with confidence levels, a reliability-rated source sheet, and recommendations. Publish it to a URL in one click, no account.
HTMLStatus report template
A weekly status report in plain Markdown – TL;DR, progress, slips, the numbers that matter, and risks. Publish it to a URL in one click, no account.
MarkdownPublish your first artifact in 15 seconds.
Free tier. No credit card. One-time password auth on free, Google + Microsoft SSO on Pro (€49/month flat).