Your Runbook Was Written for the Engineer Who Doesn't Need It

It's 3:10 a.m. and the pager goes off: checkout-api p99 latency above threshold. The engineer on call joined the team four months ago. She opens the alert, finds a link to the runbook, and reads three paragraphs about how the checkout service was split out of the monolith in 2023. Then a line that says "check the usual dashboards." Then a command that refers to a cluster that no longer exists.

At 3:25 she gives up and pages the senior engineer who wrote it. He fixes it in eight minutes, because he didn't need the runbook. He never did. He wrote it for himself, from memory, the afternoon after the last incident, and it made perfect sense to him.

That is the most common runbook failure I see: the runbook was written by the person who will never need it, for a reader who doesn't exist. The cost doesn't show up as a documentation problem. It shows up as longer incidents, the same two people getting woken up every week, and senior engineers who start quietly looking for jobs with a lighter on-call load.

Why runbooks fail at the worst possible moment

Runbooks have a peculiar reader. They're tired, stressed, possibly unfamiliar with the service, and reading on a laptop balanced on their knees while a channel fills up with questions. Most runbooks are written as if for a calm colleague with an afternoon to spare. The usual problems:

• They're narratives, not procedures. History and architecture come first. The action the reader needs is buried in paragraph four.
• They assume context. "Restart the workers" assumes you know which workers, where, and how. "Check the usual dashboards" assumes you know which ones are usual.
• They describe the fix, not the diagnosis. The author knew the cause when he wrote it. The reader at 3 a.m. doesn't, so the runbook needs to help them figure out which situation they're in before telling them what to do.
• They rot silently. Hostnames, dashboard URLs, and commands change with every infrastructure migration. Nothing checks the runbook, so nobody notices until it fails during an incident.
• They don't say what's safe. A new engineer won't run a mitigation if they can't tell whether it might make things worse. So they escalate, which is the rational choice.

None of this is about writing talent. The person who knows the system best is the worst-placed person to notice what's missing, because his own knowledge fills every gap as he rereads it. It's the same problem behind new engineers learning the system by interrupting your senior engineers, except this time there's a pager attached.

What a runbook needs to answer, in order

A runbook should be organized around the questions the on-call engineer asks, in the order they ask them. For every alert that can page someone, that means:

1. What does this alert mean? One or two sentences, in plain terms. What is the system telling you, and what threshold was crossed?
2. How bad is it? Who is affected, and how urgently. "Customers can't complete checkout" is a different night from "a background report is running late."
3. How do I confirm it's real? The exact dashboard link, the exact query, and what normal looks like next to what broken looks like.
4. Which situation am I in? The two or three most likely causes, and how to tell them apart. This is the part most runbooks skip, and it's the part that saves the most time.
5. What can I safely do right now? Mitigations with exact, copy-pasteable commands, what each one affects, and how to undo it.
6. What should I not do? If restarting the database primary turns a slow night into a long outage, say so in bold, near the top.
7. When and how do I escalate? A time limit ("if this isn't improving within 20 minutes"), who to page, and what information to include when you do.

Here's a skeleton that follows that order. It's deliberately plain:

# Alert: checkout-api p99 latency high

**Means:** Checkout requests are taking more than 2s at p99 for 5+ minutes.
**Impact:** Customers see slow or failed checkouts. Treat as urgent.

## Confirm
- Dashboard: <direct link>
- Normal p99 is under 400ms. Check the error-rate panel on the same dashboard.

## Identify the cause
- Latency high, DB connections near the pool limit → see "Connection pool exhausted"
- Latency high, payment-provider timeouts in logs → see "Upstream provider slow"
- Neither → escalate

## Do NOT
- Do not restart the database primary. Page the database owner instead.

## Escalate
- No improvement in 20 minutes: page the checkout team secondary.
- Include: start time, what you've checked, what you've tried.

Notice what isn't there: history, architecture, or a design rationale. Those belong in the service's documentation, and the runbook can link to them. At 3 a.m. they're in the way.

Test runbooks with the person who didn't write them

A runbook that has only been read by its author hasn't been tested. The most useful review you can do costs about an hour: hand the runbook to the newest engineer in the on-call rotation, give them a scenario, and have them walk through it out loud while the author stays silent and takes notes. Every time the reader hesitates, asks a question, or goes looking for something, that's a gap. The author will be surprised by how many there are. Everyone always is.

If your team runs game days or failure drills, make "follow the runbook exactly as written" an explicit part of the exercise. You're testing the document as much as the system.

Make runbooks part of the incident, not an afterthought

Runbooks stay accurate only when updating them is part of the work everyone already does. A few habits that hold up:

• Link every paging alert to its runbook. If an alert can wake someone up and has no runbook, that's a gap worth tracking, and a good argument for whether the alert should page at all.
• Close every incident by asking what the runbook got wrong. Add it as a standing item in your post-incident review, with an owner and a due date.
• Keep runbooks next to the code or the alert definitions. When they live in the same repository and go through the same review, a changed hostname or renamed service is more likely to get caught in the same pull request.
• Put a "last verified" date at the top. It costs nothing and tells the reader how much to trust what follows.

Where to start this week

You don't need to rewrite every runbook. Pull the list of alerts that paged someone in the last quarter and sort by frequency. Take the top five. For each, check whether a runbook exists, whether its links and commands still work, and whether it answers the seven questions above. Then hand the worst one to your newest on-call engineer and watch them use it. That one hour will tell you more about your on-call health than any dashboard.

Need help with yours?

Good runbooks come from someone who can read the system like an engineer and write for the reader who's never seen it. That's the work I do. If your incidents keep escalating to the same few people, get in touch. I'm happy to review a handful of your runbooks and tell you what I'd change first.