Engineering

Runbook Template

A runbook is the page someone follows at 3 a.m. with no context. This template gives every step a literal command and the output to expect, and adds the two fields most templates skip: when the procedure was last verified, and what change would make it wrong.

Updated 2026-10-27

Copy and adapt

The template

RunbookTemplate

Runbook: [task or alert name]

Trigger: [the alert, symptom or schedule that means "open this page"] Service: [[Service name]] Owner: [team or rotation, not a person] Verified on: [YYYY-MM-DD] by [name], method: [executed in staging / game day / read-through only] Drift triggers: [renames, endpoints, flags and dashboards this procedure depends on] Blast radius: [what users notice if this goes wrong]

Preconditions

Access: [role, VPN, break-glass account]
Check first: [state that must be true before you change anything]

Steps

[Exact command or console path] Expected: [what healthy output looks like]
[Command] Expected: [observable result]
[Command] Expected: [observable result]

Verification

[Metric back to baseline, health check returning 200, queue depth falling]

Rollback

[How to undo each step that changed state, in reverse order]

Escalation

After [N] minutes without progress: [rotation or channel]
Then: [service owner rotation], then [incident commander]

Communication

Status update: "[one line you can paste into the status page or incident channel]"

Known failure modes

[Symptom] usually means [cause]. See [[Other runbook]].

Change log

{{date}}: [what changed and why]

5 examples

Examples for different situations

The same structure, filled in for the situations people actually bring to it. Copy the one closest to yours.

1.Alert response: queue consumer lag

When it fits: Use this when a paging alert fires for a backlog, the fix is usually a restart, and the risk is that someone restarts the wrong thing during a deploy.

Alert response: queue consumer lagExample 1

Runbook: ledger-writer consumer lag above 50,000

Trigger: PagerDuty alert LedgerWriterLagHigh (lag above 50,000 messages for 10 minutes) Service: [[ledger-writer]] Owner: Payments platform rotation Verified on: 2026-10-09 by Kwame Asante, method: executed in staging with a synthetic backlog of 80,000 messages Drift triggers: consumer group rename, move off the payments-prod cluster, change to the deploy tool's restart command Blast radius: merchant balances lag behind reality; payouts scheduled for 16:00 UTC can go out short

Preconditions

Access: payments-oncall role, production VPN
Check first: is a deploy of ledger-writer in progress? If yes, stop and ask in #payments-deploys before restarting.

Steps

kafka-consumer-groups --bootstrap-server kafka-1.payments-prod.internal:9092 --describe --group ledger-writer Expected: LAG per partition. If one partition holds most of the lag, go to Known failure modes.
shipctl restart ledger-writer --env prod --rolling Expected: "3/3 instances healthy" within 4 minutes.
Re-run step 1 after 5 minutes. Expected: total lag falling by at least 5,000 per minute.

Verification

Lag under 1,000 on the Ledger Writer dashboard, and the 16:00 payout job shows "inputs fresh".

Rollback

A restart has no rollback. If lag grows after the restart, do not restart again. Escalate.

Escalation

After 20 minutes without falling lag: Payments platform secondary
Then: Data infrastructure rotation (Kafka itself), then the incident commander

Communication

"Merchant balances are delayed by up to [N] minutes. Payouts are not affected yet. Next update in 30 minutes."

Known failure modes

One partition holding all the lag usually means a poison message. See [[ledger-writer: skip a poison message]].

Change log

2026-10-09: replaced the old kubectl rollout restart step after the move to shipctl.

2.Scheduled maintenance: TLS certificate rotation

When it fits: Use this when the task runs every 60 or 90 days, the person who did it last time has forgotten the order, and a mistake takes a public endpoint down.

Scheduled maintenance: TLS certificate rotationExample 2

Runbook: rotate the TLS certificate on the partner API gateway

Trigger: calendar reminder 14 days before expiry (current certificate expires 2026-12-03) Service: [[partner-gateway]] Owner: Edge team Verified on: 2026-09-04 by Bea Lindqvist, method: executed in production during the September rotation Drift triggers: move from nginx to the managed load balancer, change of certificate authority, new hostname added to the certificate Blast radius: 40 partner integrations fail TLS and stop sending orders

Preconditions

Access: edge-admin role, read access to the tls/partner-gateway secret
Check first: the new certificate covers both api.partners.lumenfreight.example and sandbox.partners.lumenfreight.example

Steps

openssl x509 -in new.crt -noout -subject -enddate -ext subjectAltName Expected: both hostnames listed, notAfter in March 2027.
vault kv put tls/partner-gateway cert=@new.crt key=@new.key Expected: "Success! Data written".
ssh edge-gw-1 'sudo nginx -t && sudo systemctl reload nginx', then the same on edge-gw-2 Expected: "syntax is ok" and "test is successful" on both hosts.

Verification

openssl s_client -connect api.partners.lumenfreight.example:443 -servername api.partners.lumenfreight.example </dev/null | openssl x509 -noout -enddate shows the new date on both hosts.

Rollback

vault kv rollback -version=[previous] tls/partner-gateway, then reload nginx on both hosts.

Escalation

Any partner reporting TLS errors within an hour: Edge team secondary, then the partner integrations lead

Known failure modes

Reload succeeds but clients still see the old date: one host was missed. Check edge-gw-2.

Change log

2026-09-04: added the sandbox hostname after it was missed in June.

3.Database failover drill with a failed step

When it fits: Use this when you exercise a procedure on purpose, a step does not do what the page says, and the runbook has to record that before anyone trusts it in a real outage.

Database failover drill with a failed stepExample 3

Runbook: planned switchover of the orders Postgres cluster

Trigger: quarterly game day, or primary host scheduled for maintenance Service: [[orders-db]] Owner: Data platform rotation Verified on: 2026-10-14 by Arjun Mehta and Hana Okafor, method: game day in production at 06:00 UTC. Step 4 failed, see change log. Drift triggers: Patroni upgrade, change of connection pooler, new read replica added Blast radius: checkout writes fail for the length of the switchover (target under 30 seconds)

Preconditions

Access: db-admin role, bastion host bastion-2
Check first: replication lag on the candidate under 1 MB, no migrations running

Steps

patronictl -c /etc/patroni/orders.yml list Expected: one Leader, two Replicas, all "running", lag 0.
patronictl -c /etc/patroni/orders.yml switchover --leader orders-db-a --candidate orders-db-b --force Expected: "Successfully switched over to orders-db-b".
Re-run step 1. Expected: orders-db-b is Leader.
Check that the pooler follows: psql -h orders-pool.internal -c 'select inet_server_addr()' Expected: the address of orders-db-b. On 2026-10-14 it still returned orders-db-a for 3 minutes.

Verification

Checkout error rate under 0.5 percent on the Orders dashboard within 2 minutes of step 2.

Rollback

Switch back: same command with --leader orders-db-b --candidate orders-db-a. Used on 2026-10-14 when step 4 failed.

Escalation

Error rate above 2 percent for 2 minutes: incident commander, declare an incident

Known failure modes

Pooler holds the old leader: PgBouncer is not reading the Patroni callback. Fix tracked in [[orders-db pooler callback]].

Change log

2026-10-14: step 4 added after the game day. Do not run this switchover outside a game day until the pooler fix ships (target 2026-11-07).

4.Production support: disk full on a build host

When it fits: Use this when the alert comes from a single host, anyone on the rotation might get it, and the tempting fix (delete whatever is biggest) can destroy something needed.

Production support: disk full on a build hostExample 4

Runbook: /var above 90 percent on CI build hosts

Trigger: alert CIHostDiskHigh on any ci-build-* host Service: [[CI build fleet]] Owner: Developer productivity rotation Verified on: 2026-10-21 by Oskar Brandt, method: executed on ci-build-07 during a real alert Drift triggers: change of container runtime, move of build caches off /var, new log shipping agent Blast radius: builds on that host fail with "no space left on device"; other hosts pick up the queue more slowly

Preconditions

Access: SSH via ci-bastion, sudo on build hosts
Check first: drain the host so new jobs stop landing on it: ciadm drain ci-build-07

Steps

df -h /var Expected: the usage the alert reported.
sudo du -xh /var --max-depth=2 | sort -h | tail -15 Expected: Docker images or journal logs at the top. Anything else, stop and escalate.
sudo docker system prune -af --filter "until=72h" Expected: "Total reclaimed space" of several GB.
sudo journalctl --vacuum-size=2G Expected: journal reduced to 2 GB.

Verification

df -h /var under 60 percent, then ciadm undrain ci-build-07 and one green build on the host.

Rollback

None needed for cache and logs. Never delete anything under /var/lib/ci/artifacts; it holds release builds still being signed.

Escalation

Usage back above 90 percent within a day: Developer productivity secondary, open a ticket to resize

Known failure modes

Prune frees almost nothing: a build left a running container. sudo docker ps and stop jobs older than 6 hours.

Change log

2026-10-21: added the artifacts warning after someone nearly deleted a signed release.

5.Access revocation when an engineer leaves

When it fits: Use this when the trigger is an HR date rather than an alert, several systems are involved, and a missed step leaves a live credential behind.

Access revocation when an engineer leavesExample 5

Runbook: revoke production access for a departing engineer

Trigger: offboarding ticket from People Ops, run on the person's last day at 17:00 local time Service: [[Production access]] Owner: Security engineering rotation Verified on: 2026-10-02 by Mireille Dupont, method: executed for a real departure and checked against the access audit Drift triggers: new SSO provider, new cloud account, shared secret added to the on-call vault Blast radius: a former employee keeps working credentials

Preconditions

Access: security-admin role in SSO and the cloud console
Check first: confirm the last day and username in the offboarding ticket, not from chat

Steps

Suspend the SSO account in the identity provider admin console (Users, select user, Suspend). Expected: status "Suspended", active sessions revoked.
aws iam list-access-keys --user-name dpark Expected: zero keys. If any exist: aws iam update-access-key --user-name dpark --access-key-id [id] --status Inactive
Remove the person's public key from the ops-ssh-keys repository and merge. Expected: CI pushes the new authorized_keys to all bastions within 10 minutes.
List the shared secrets the person could read in the on-call vault and open a rotation task for each. Expected: one task per secret, owner and due date set.

Verification

Next morning's access audit report shows no active credentials for the username.

Rollback

If the wrong account was suspended, reactivate it in the identity provider and tell the person directly.

Escalation

Any credential that cannot be revoked within an hour: Security engineering lead

Change log

2026-10-02: added step 4 after an audit found the on-call vault was missed in August.

How to fill in each field

The runbook guide on our blog covers what a runbook is and why runbooks drift. This page is the working copy. A few notes on the fields people get wrong:

Trigger. Copy the alert name exactly. If the runbook is for scheduled work, write the schedule and the date it is next due.
Owner. A team or rotation. A person's name is accurate until the reorg.
Verified on. A date, a name and a method. "Read it through" is a method, but say so, because it is the weakest one.
Drift triggers. The renames, endpoints, flags and dashboards the steps depend on. Three to five is normal.
Blast radius. One sentence on what users notice if this goes wrong. It tells the reader how careful to be and when to wake someone.
Steps. One action per step, with the literal command or console path and the expected output. If a step needs judgment, write the decision as a branch: "If X, go to Known failure modes."
Rollback. For every step that changes state, how to undo it. If a step cannot be undone, say so in bold before the step, not after.

Test the runbook before you need it

Reading a runbook finds typos. Running it finds the missing steps. The only verification that catches a step nobody wrote down is executing the procedure, so schedule it:

Staging run by a stranger. Ask someone who has never touched the service to follow the page literally in staging. Every question they ask is a gap in the document.
Game day. For failovers and restores, run the procedure in production in a planned window. The third example above shows what to write when a step fails: record it, use the rollback, and mark the runbook as unsafe until the fix ships.
Real incidents. After an incident that used the runbook, update it within a day while the differences are fresh. The post-mortem template has an action item slot for exactly this.

Keep it true after the next deploy

Runbooks describe live systems, and live systems ship every week. Two habits keep the page honest. First, update the verified-on date only after a real check, never after an edit. Second, when a change matches a drift trigger (a service rename, a new pooler, a moved dashboard), open the runbooks that list it before you merge. Searching your notes for the old name is the quickest way to find them.

When a different document fits

A runbook is for one task with one expected outcome. If the procedure is a business process that people follow during normal work, such as approving refunds or onboarding a vendor, use the SOP template. If the incident already happened and you are working out why, use the post-mortem template. If you are handing a whole service to another team, the handoff document template links to the runbooks rather than replacing them.

Common mistakes

Commands without expected output. The reader cannot tell whether a step worked, so they guess.
Links as steps. "See the dashboard" with a URL that moved in the last migration. Write what to look for on the dashboard, not only where it is.
One runbook per service. Ten pages that cover every alert are not readable in an incident. Split by alert.
Escalation to a person. That person is on leave, or left in the spring.
Phantom steps. A step that made sense two architectures ago and nobody dares delete. If you cannot say why a step exists, test the procedure without it in staging and remove it if nothing breaks.

Customization tips

One runbook per alert, not per service
Name the runbook after the alert that triggers it, word for word. The on-call engineer matches the page title to the alert title and opens one document, instead of scrolling a ten-page service guide for the relevant section.
Write the expected output, not just the command
A command without an expected result cannot tell anyone whether it worked. One line of "Expected: 3/3 instances healthy" is what lets a tired reader stop and escalate instead of improvising.
Separate verified on from last edited
Change the verified-on date only when someone ran or checked the procedure against the current system. Fixing a typo is an edit, not a verification, and a fresh edit date on an untested page makes people trust it more than they should.
Write drift triggers while you still remember them
When you write a step, note what would break it: a rename, a flag, a moved dashboard. When that change happens later, you know which runbook to open, because the runbook said so in advance.
Escalate to rotations, not names
People change teams and leave. A rotation or a channel survives that. The fifth example exists because a name-based list is exactly the kind of thing that goes stale quietly.
Drop sections for scheduled tasks
A certificate rotation or a monthly cleanup does not need a Communication section. Keep Trigger, Verified on, Steps, Verification and Rollback, and remove the rest rather than leaving empty headings.

In the app

How to do this in Scribelet

Open Desk Settings, then Templates, tap Add Template and paste the template above. Keep {{date}} on the Change log line (not on Verified on, which changes only after a real check) and add #runbook as a default tag so every runbook is one filter away.
Link the service with [[ledger-writer]]. The service note's backlinks panel then lists every runbook, post-mortem and design note that mentions it.
Connect the repository the runbook describes with the GitHub connector and opt the note in from its Settings. Background verification (Pro) compares the runbook's claims about services, flags and endpoints against your actual code and proposes a diff you accept or dismiss.
Add a task like - [ ] Re-verify in staging @remind(2027-01-14 10:00) at the bottom. It appears in the Tasks view and sends a push reminder in your local time, so the next verification is scheduled rather than hoped for.
Ask the AI chat "which runbooks mention the orders pooler?" and it answers from your notes with citations, which is faster than searching during an incident.
Agents can read runbooks too: add Scribelet's MCP server to Claude Code or Cursor and an agent can search and open the runbook for the alert it is investigating.

Templates, template variables and AI generation are documented in Note templates.

Frequently asked questions

What should a runbook template include?

At minimum: the trigger, preconditions, numbered steps with exact commands and expected output, a verification check, rollback, and an escalation path. The template above adds a verified-on date, drift triggers and blast radius, which are what tell a reader whether to trust the page.

What is the difference between a runbook and a playbook?

A runbook covers one specific task or alert and assumes almost no judgment: follow the steps. A playbook covers a whole class of incident and tells an incident commander how to coordinate it. The runbook guide on our blog compares runbooks, playbooks and SOPs side by side.

How do I write a runbook that works at 3 a.m.?

Write it for someone who has never seen the system: literal commands, expected output after each one, and a clear point at which to stop and escalate. Then have someone unfamiliar with the service follow it in staging and note every place it was wrong. That test finds more problems than any review.

How often should a runbook be updated?

Whenever something it depends on changes, which a calendar cannot tell you. Write drift triggers next to the steps, re-verify after any change that matches one, and run the procedure in a game day at least twice a year. A quarterly read-through on its own mostly refreshes the date.

Can I use this runbook template in Confluence, Word or a Git repository?

Yes. It is plain Markdown, so it pastes into Confluence, Google Docs or Word, and works as a .md file next to the code. The [[wikilinks]] and {{date}} variable are Scribelet features and show as plain text elsewhere.

Should runbooks be automated?

Automate the steps once the procedure is stable and needs no judgment, but keep the runbook. An automated script that calls a renamed endpoint fails just as hard as a stale page, and the runbook is where the expected behavior, rollback and escalation still live.

Runbooks that say when they stopped being true

Save this runbook template in Scribelet, link it to the service it covers, and connect the repository so verification can flag a renamed flag or service before the next page does. Free to start; background verification is part of Pro.

Set up your first runbook

or get the app

Download on theApp Store

Free plan, no credit card. See pricing

Guide

Note Taking App for Developers: Markdown, MCP and Notes Checked Against Your Code

Developer notes fail in a specific way: they are right on the day they are written and quietly wrong six months later. This page says which tool fits which developer, what belongs in the repo instead, and a setup where notes are Markdown, your coding agent can read and write them over MCP, and runbooks get checked against the actual code.

Article

Runbook

From the Scribelet blog

Work

SOP Template (Standard Operating Procedure)

An SOP is only useful if a person who has never done the task can follow it and get the same result. This template puts the trigger, the checks and the escalation path next to the steps, and gives every procedure an owner and a review date so it does not quietly go stale.

Engineering

Post Mortem Template for Incidents

A post-mortem is worth writing only if it changes something. This template keeps the timeline factual, pushes the whys to a cause the team can fix, and ends with action items that have owners, dates and a 30-day check. Copy it, or start from the postmortem example closest to your incident.

Work

Handoff Document Template

A handoff fails on the things nobody wrote down: the promise made in a hallway, the decision half made, the warning sign only the outgoing person knows. This handoff document template has a section for each, and ends with a read-back so you know the handover landed. Copy it, or start from the handover notes example closest to yours.

Engineering

RFC Template for Engineering Teams

An RFC (request for comments) is a short written proposal that asks reviewers to find the problems before the code does. This template adds what most RFC templates leave out: a named approver, a date comments close, and a Decision section, so the review ends instead of trailing off.