Engineering

Post Mortem Template for Incidents

A post-mortem is worth writing only if it changes something. This template keeps the timeline factual, pushes the whys to a cause the team can fix, and ends with action items that have owners, dates and a 30-day check. Copy it, or start from the postmortem example closest to your incident.

Updated 2026-10-27

Copy and adapt

The template

Incident post-mortemTemplate

Post-mortem: [Incident title], {{date}}

Incident: [ID] Severity: [SEV1 / SEV2 / SEV3] Status: [Draft / In review / Final] Owner: [name] Incident lead: [name] Review meeting: [YYYY-MM-DD] Related: [[Service name]]

Blameless: this document describes what the system and our process allowed, not who to blame.

Summary

[Three sentences: what broke, for how long and for whom, and what fixed it.]

Impact

Duration: [start] to [end] UTC ([total])
Who was affected: [number or percent of users, which features]
Detected by: [alert / customer report / employee], [minutes] after start
Support tickets: [number]
Data or money lost: [none, or the figure]

Timeline (UTC)

[HH:MM] [What happened] (source: [alert, chat, deploy log])
[HH:MM] [First response]
[HH:MM] [Mitigated]
[HH:MM] [Resolved]

Contributing factors

Trigger: [the change or event that started it]
Why it caused impact: [the condition that let the trigger hurt]
Why we did not catch it sooner: [the detection gap]
[Keep asking why until the answer is something the team can change]

What went well

[Item]

What went badly

[Item]

Where we got lucky

[Item that would have made it worse]

Action items

Prevent: [action] (@[owner]) @due(YYYY-MM-DD)
Detect: [action] (@[owner]) @due(YYYY-MM-DD)
Mitigate: [action] (@[owner]) @due(YYYY-MM-DD)

Follow-up check

Every action item closed or re-planned with a reason (@[owner]) @due([review date plus 30 days])

References

[Incident channel, dashboards, pull requests, status page post]

5 examples

Examples for different situations

The same structure, filled in for the situations people actually bring to it. Copy the one closest to yours.

1.SEV1: expired TLS certificate

When it fits: Use this when the outage was total, customers noticed before your monitoring did, and the real cause is a change made months earlier.

SEV1: expired TLS certificateExample 1

Post-mortem: API certificate expiry, 2026-11-03

Incident: INC-2291 Severity: SEV1 Status: Final Owner: Bianca Novak Incident lead: Desmond Clarke Review meeting: 2026-11-05 Related: [[orders-gateway]]

Blameless: this document describes what the system and our process allowed, not who to blame.

Summary

The TLS certificate on the API hostname expired at 13:00 UTC. Mobile apps and partner integrations could not connect for 47 minutes. A manually issued certificate restored service; automatic renewal is fixed.

Impact

Duration: 13:00 to 13:47 UTC (47 min)
Who was affected: all mobile users and 31 partner integrations; the website was unaffected (different certificate)
Detected by: customer reports on social media, 9 minutes after start
Support tickets: 412
Data or money lost: an estimated 1,900 orders not placed

Timeline (UTC)

13:00 Certificate expires (source: certificate transparency log)
13:09 First support ticket; 13:11 incident declared (chat)
13:18 Desmond identifies the expiry with an openssl check
13:41 New certificate issued manually, deployed to the gateway
13:47 Error rate back to baseline (dashboard)

Contributing factors

Trigger: the certificate reached its expiry date.
Why it caused impact: auto-renewal had failed silently since the August DNS migration removed the validation record.
Why we did not catch it sooner: the expiry alert checked the website hostname, not the API hostname. Renewal failures were logged but not alerted on.

What went well

Diagnosis took 7 minutes once the right person joined.

What went badly

9 minutes from start to declaration, driven by customers.
Nobody knew which DNS records the renewal depended on.

Where we got lucky

The website certificate was on a different renewal path and stayed up.

Action items

Prevent: restore the DNS validation record and test renewal end to end (@Desmond) @due(2026-11-06)
Detect: expiry alert for every public hostname at 21 and 7 days (@Bianca) @due(2026-11-10)
Detect: alert on any failed renewal attempt (@Bianca) @due(2026-11-10)
Process: DNS migration checklist lists records owned by other systems (@Desmond) @due(2026-11-20)

Follow-up check

All four closed or re-planned (@Bianca) @due(2026-12-05)

2.SEV2: config change that rate-limited customers

When it fits: Use this when a small change skipped review by design, the fix took minutes, and the action items are a disagreement about process rather than a code fix.

SEV2: config change that rate-limited customersExample 2

Post-mortem: public API rate limit set to 10 rps, 2026-11-12

Incident: INC-2307 Severity: SEV2 Status: In review Owner: Kofi Asante Incident lead: Ellie Brandt Review meeting: 2026-11-14 Related: [[Public API]]

Blameless: this document describes what the system and our process allowed, not who to blame.

Summary

A feature flag update set the default API rate limit to 10 requests per second instead of 100. For 2 hours 10 minutes, 18 percent of API customers received 429 errors. Reverting the flag fixed it.

Impact

Duration: 09:20 to 11:30 UTC (2 h 10 min)
Who was affected: 146 of 810 API customers, mostly those running nightly syncs
Detected by: 429 rate alert, 6 minutes after start
Support tickets: 58
Data or money lost: none; clients retried, two customers asked for SLA credit

Timeline (UTC)

09:20 Flag "api_limits_v2" updated in the flag console (audit log)
09:26 429 rate alert fires; on-call acknowledges (pager)
09:58 On-call rules out a deploy, no code changed (deploy log)
11:12 Ellie finds the flag change in the audit log
11:30 Flag reverted, 429 rate back to normal

Contributing factors

Trigger: the flag value was entered as 10 instead of 100.
Why it caused impact: flag changes apply to all customers instantly, with no staged rollout.
Why we did not catch it sooner: the runbook for 429 alerts checks deploys, not flag changes. It took 106 minutes to look in the right place.

What went well

The alert fired within 6 minutes.

What went badly

Flag changes are invisible in the deploy channel.

Where we got lucky

The change was made on a Thursday morning, not before a weekend.

Action items

Detect: post every flag change to the deploys channel (@Ellie) @due(2026-11-19)
Mitigate: add "check flag audit log" as step 2 of the 429 runbook (@Kofi) @due(2026-11-15)
Prevent: decide whether limit flags need a second approver. Product says it slows them down; open for the review meeting (@Kofi) @due(2026-11-14)

Follow-up check

Closed or re-planned (@Kofi) @due(2026-12-14)

3.SEV2: third-party payment provider outage

When it fits: Use this when the trigger was outside your control, and the honest finding is that your own fallback did not work when it was needed.

SEV2: third-party payment provider outageExample 3

Post-mortem: checkout failures during payment provider outage, 2026-11-21

Incident: INC-2318 Severity: SEV2 Status: Final Owner: Renata Silva Incident lead: Owen Tremblay Review meeting: 2026-11-24 Related: [[Checkout service]]

Blameless: this document describes what the system and our process allowed, not who to blame.

Summary

Our primary payment provider returned errors on 60 percent of charges for 35 minutes. Failover to the secondary provider failed because its API key had been rotated and not updated in our secrets store. Checkout recovered when the primary provider did.

Impact

Duration: 04:05 to 04:40 UTC (35 min)
Who was affected: about 620 checkout attempts failed
Detected by: checkout error alert, 3 minutes after start
Support tickets: 14
Data or money lost: roughly $21,000 in delayed orders; about a third were retried later

Timeline (UTC)

04:05 Provider status page shows degraded charges
04:08 Alert fires; Owen paged
04:15 Owen triggers failover to the secondary provider
04:16 Secondary returns 401 unauthorized for every request
04:40 Primary provider recovers

Contributing factors

Trigger: outage at the primary provider.
Why it caused impact: the secondary key was rotated on 2026-09-30 during a security review. The new key went to the password manager, not the secrets store.
Why we did not catch it sooner: failover has not been exercised since March. Nothing tests the secondary path.

What went well

Failover decision in 7 minutes, by the runbook.

What went badly

A fallback that had not been tested was treated as a fallback.

Where we got lucky

04:00 UTC is our lowest traffic hour.

Action items

Prevent: update the secondary key and add both keys to the rotation checklist (@Renata) @due(2026-11-25)
Detect: daily synthetic $1 authorization on the secondary provider (@Owen) @due(2026-12-05)
Process: quarterly failover drill on the calendar, first one in January (@Renata) @due(2026-12-12)

Follow-up check

Closed or re-planned (@Renata) @due(2026-12-24)

4.SEV3 near miss, short form

When it fits: Use this when nothing broke for customers, but it would have in a few hours, and you want a lightweight record instead of skipping the post-mortem.

SEV3 near miss, short formExample 4

Post-mortem: primary database disk at 94 percent, 2026-12-02

Incident: INC-2334 Severity: SEV3 (near miss) Status: Final Owner: Hana Kowalczyk Incident lead: Hana Kowalczyk Review meeting: async, comments by 2026-12-04 Related: [[Postgres primary]]

Summary

Disk usage on the primary database reached 94 percent at 02:10 UTC because a debug logging flag left on after a November investigation wrote 180 GB to an audit table. Hana turned the flag off and truncated the table. No customer impact.

Impact

Duration: none user-facing; estimated 5 hours to a full disk
Detected by: disk alert at 90 percent

Contributing factors

Trigger: debug flag enabled on 2026-11-18 and never turned off.
Why it nearly caused impact: debug flags have no expiry.

Where we got lucky

The 90 percent alert existed. It was added in July after a different near miss.

Action items

Prevent: debug flags require an expiry date and switch off automatically (@Hana) @due(2026-12-16)
Detect: alert on audit table growth over 10 GB per day (@Hana) @due(2026-12-09)

Follow-up check

Closed or re-planned (@Hana) @due(2027-01-04)

5.Non-infrastructure incident: email sent to unsubscribed users

When it fits: Use this when the incident crossed teams, legal or compliance has a stake, and the post-mortem has to be readable by people who do not work in engineering.

Non-infrastructure incident: email sent to unsubscribed usersExample 5

Post-mortem: newsletter sent to unsubscribed contacts, 2026-12-08

Incident: INC-2341 Severity: SEV2 Status: In review Owner: Lucia Romano (marketing ops) Incident lead: Ben Adeyemi (data) Review meeting: 2026-12-10 Related: [[Email platform]]

Blameless: this document describes what the system and our process allowed, not who to blame.

Summary

The December newsletter went to 12,400 contacts who had unsubscribed. A segment filter used "subscribed is not false", which included contacts whose status was empty after a CRM import. Sending stopped at 82 percent; an apology went out the same day.

Impact

Duration: send started 15:00 UTC, stopped 15:22 UTC
Who was affected: 12,400 unsubscribed contacts
Detected by: a colleague who had unsubscribed received it, 14 minutes after start
Support tickets: 230 replies, 9 formal complaints
Data or money lost: no data exposed; legal is assessing notification duties

Contributing factors

Trigger: the newsletter used a segment built in October, before the November CRM import.
Why it caused impact: the import left the subscription field empty for 12,400 records, and the filter treated empty as subscribed.
Why we did not catch it sooner: segment size jumped from 38,000 to 50,400 and nothing flagged the change.

What went badly

Nobody owned checking segment sizes before a send.

Action items

Prevent: set empty subscription fields to unsubscribed, and change the filter to "subscribed is true" (@Ben) @due(2026-12-11)
Detect: block sends when segment size moves more than 10 percent from last send (@Lucia) @due(2026-12-18)
Process: legal decision on regulator notification (@Lucia with counsel) @due(2026-12-12)

Follow-up check

Closed or re-planned (@Lucia) @due(2027-01-10)

What a post-mortem is for

Most postmortem templates agree on the sections: summary, impact, timeline, root cause, what went well and badly, action items. They agree less on the purpose, and the purpose decides how you fill the sections in. A post-mortem exists to change the system so the same failure is less likely, faster to detect, or smaller when it happens. A document that records the incident but changes nothing is a report, and reports are not read twice.

When to write it and how fast

Draft within 48 hours. The owner writes summary, impact and timeline while logs and memory are fresh.
Review within three to five working days. Responders correct the timeline and agree contributing factors and action items.
Final when every action item has an owner and a date. Not when the meeting ends.
Check after 30 days. The step every template leaves out. Open the post-mortem again and confirm each action item is closed or re-planned with a reason.

Build the timeline from sources, not memory

Start before the incident: the deploy, config change or expiry that set it up. Then detection, declaration, each hypothesis, mitigation and resolution. Use UTC and cite where each timestamp came from. Gaps in the timeline are findings. Thirty minutes between "alert fired" and "first action" is a detection or paging problem, whatever the trigger was.

How deep to go with the whys

Ask why until the answer is something the team can change, then stop. "The certificate expired" is not actionable. "Renewal failed silently after the DNS migration removed a record nobody knew was needed" gives three actions: restore the record, alert on failed renewals, and list cross-system records in the migration checklist. Going further ("why did we migrate DNS?") usually leaves the system and enters strategy, which belongs in a different meeting.

Most incidents have more than one contributing factor. Listing a trigger, a reason it caused impact, and a reason it was not caught sooner prevents the common failure of fixing only the trigger.

Making action items close

Action items are where postmortems quietly fail. They get filed in a tracker, compete with feature work, and are forgotten until the incident repeats. Four habits keep them alive:

One owner per item, by name. "Platform team" owns nothing.
A due date measured in days or weeks. Items with no date are wishes.
No more than five items. Ten items means none get done; pick the ones that change the most.
The 30-day follow-up is itself a task with the post-mortem owner's name on it.

Incident post-mortem or project post-mortem

"Post-mortem" covers two different documents. This page is the incident version: one failure, a timeline, contributing factors. If you are looking back on a project that shipped late or a launch that missed its goals, there is no single timeline to reconstruct. Use the lessons learned template for a project, or the sprint retrospective template for a regular team review.

Common mistakes

Naming a person as the root cause. The next person will make the same mistake if the system still allows it.
Fixing only the trigger. The expired certificate gets renewed; the missing alert stays missing.
Skipping near misses. They are the cheapest post-mortems you will ever run.
Not linking the runbook. If responders needed steps that were not written down, write them into a runbook as an action item.

Customization tips

Scale the document to the severity
A SEV1 gets every section and a meeting. A near miss gets summary, contributing factors, where you got lucky and two action items, reviewed async. A short post-mortem that gets written beats a long one that gets skipped.
Cite a source for every timeline entry
Pull timestamps from alerts, chat and deploy logs rather than memory. A source in brackets settles arguments about when something happened and keeps the timeline factual.
Label action items by type
Prevent, Detect, Mitigate and Process make gaps obvious. If every item is Prevent, nobody asked how to find the next one faster; if every item is Process, the system may not have changed at all.
Write for the person who joins next year
Spell out service names and acronyms. The most valuable reader of a post-mortem is the engineer who hits a similar problem later and searches for it.
Keep "where we got lucky"
It is the section people skip and the one that finds the next incident. Luck is an unowned control; naming it turns it into an action item.

In the app

How to do this in Scribelet

Open Desk Settings, then Templates, tap Add Template, paste the template above, choose the General category and set #postmortem as a default tag.
Link the affected service with [[orders-gateway]]. The service note's backlinks panel lists every post-mortem that touched it, which is the reading list for anyone joining the on-call rota.
Write action items as - [ ] Detect: expiry alert (@Bianca) @due(2026-11-10). They appear in the Tasks view until ticked, and a follow-up line with @remind(2026-12-05 10:00) sends you a push reminder for the 30-day check.
Attach dashboard screenshots and a PDF of the incident channel. Scribelet extracts the text from images and PDFs, so the error message in a screenshot is searchable.
Ask the AI chat "have we had certificate or DNS incidents before?" and it answers from past post-mortems with citations, which is the question a good incident review always asks.
With the GitHub connector, opt the post-mortem in from its Settings. Background verification (Pro) checks claims such as "renewal runs daily" against the repository and proposes a correction when the code says otherwise.

Templates, template variables and AI generation are documented in Note templates.

Frequently asked questions

How do you write a post-mortem report?

Within 48 hours of resolution, the owner drafts the summary, impact and a timeline built from logs and chat. Contributing factors and action items are completed in a review meeting with the responders. The draft becomes final once every action item has an owner and a date.

How do you run a post-mortem meeting?

Hold it within three to five working days, with the responders and the owners of affected systems. Read the timeline together first and correct it, then work through contributing factors, then agree action items with owners. The facilitator's main job is keeping the conversation on what the system allowed, not who made the call.

What does blameless mean in a post-mortem?

It means people are named for what they did and saw, never as the cause. "Human error" is not a root cause; the question is why the system made that error easy and its impact large. People who expect to be blamed leave details out, and the missing details are where the fix is.

What else is a post-mortem called?

Incident review, incident retrospective, post-incident review (PIR) and root cause analysis (RCA) are the common names. RCA is sometimes used for a narrower document that covers only the causal chain; a post-mortem also covers impact, response and follow-up.

When is a post-mortem required?

Most teams require one for every SEV1 and SEV2, for any data loss, and for any incident a customer escalated. Near misses are optional but cheap, and they are often more useful because nobody is defensive about them.

What is the difference between an incident post-mortem and a project post-mortem?

An incident post-mortem looks at one failure with a timeline and contributing factors. A project post-mortem looks back on weeks or months of work and asks what to repeat and what to change. For the second, use the lessons learned template.

Post-mortems that still get read when the next incident hits

Keep every post-mortem in Scribelet linked to the service it touched, track action items as tasks with a 30-day check, and ask your notes what happened last time before you repeat it. Encrypted, free to start.

Start your post-mortem

or get the app

Download on theApp Store

Free plan, no credit card. See pricing

Guide

Note Taking App for Developers: Markdown, MCP and Notes Checked Against Your Code

Developer notes fail in a specific way: they are right on the day they are written and quietly wrong six months later. This page says which tool fits which developer, what belongs in the repo instead, and a setup where notes are Markdown, your coding agent can read and write them over MCP, and runbooks get checked against the actual code.

Engineering

Runbook Template

A runbook is the page someone follows at 3 a.m. with no context. This template gives every step a literal command and the output to expect, and adds the two fields most templates skip: when the procedure was last verified, and what change would make it wrong.

Projects

Lessons Learned Template

Most lessons learned documents are written once and never read again. This template compares plan to actual, writes each lesson as an instruction for next time with its evidence, and ends by naming the checklist, template or estimate each lesson should change.

Meetings

Sprint Retrospective Template

A retrospective is only as good as what changes afterwards. This template opens by checking whether last retro's actions worked, grounds the discussion in sprint facts, and ends with at most two actions, each with an owner, a date and a sign that it worked.

Work

Decision Log Template

A decision log is a running record of what was decided, why, by whom, and what you expected to happen. This template adds the two fields most logs leave out: who disagreed, and a date to check whether the decision worked. Copy it for a team, or use the decision journal example for your own choices.

Product

Bug Report Template

A good bug report lets someone who has never seen the problem make it happen on their own machine. This template covers the fields every bug report template has (steps, expected, actual, environment, severity) and adds the ones most leave out: how often it reproduces, whether it is a regression, and the workaround.