← all posts
#operations#incident#datadog#postgresql

Why I write runbooks and playbooks

A runbook tells you how to do a known task. A playbook tells you how to think when you don't know what's wrong yet. I needed both.

A problem

For a long time, a lot of how our systems were operated lived in my head. How to fail over the database. How to restore a backup. Which logs to look at first when requests start failing.

That worked, until it didn’t. The first time I had to fail over a PostgreSQL primary by hand in the middle of the night, I noticed something uncomfortable: I wasn’t following a procedure, I was trying to remember one. Was the replica caught up? Which config did the applications read? Did I need to restart PgBouncer, or just reload it?

Nothing went badly wrong. But the recovery depended on me being awake, calm and not making a mistake. That’s a fragile thing to depend on.

So I started writing things down. Along the way I learned there are two different kinds of document, and that I needed both. Also it can help other members in my team can take a reference to fix something in 3 AM, also nowaday with AI especially Claude — it can help us to work as an on-call engineer.

Runbook vs playbook

Runbook Playbook
Answers How do I do this task? What do I do when this kind of thing happens?
Starts from a known task a symptom or situation
Shape numbered steps, exact commands decisions, questions, who to involve
Example “Promote a PostgreSQL replica” “The database is not accepting writes”

A runbook is for when you already know what to do. It should be boring: exact commands, what output to expect, and how to check each step worked.

A playbook is for when you don’t know yet. It doesn’t give you one answer. It gives you the order to look at things, and points you to the right runbook once you’ve found the cause.

alert: "writes failing"
   └─► playbook: database not accepting writes
         ├─ is the primary up?           → no  → runbook: fail over to a replica
         ├─ is the disk full?            → yes → runbook: free WAL / disk space
         └─ are connections exhausted?   → yes → runbook: check PgBouncer pools

What makes a runbook actually useful

I rewrote my first runbooks several times. These are the things that made the difference:

  1. Write it for someone tired. Assume the reader was asleep five minutes ago. No “obviously”, no skipped steps.
  2. Every step has a check. Not just patronictl switchover, but also “you should now see the new leader in patronictl list”. If the check fails, say what to do.
  3. Copy-pasteable commands. Real host names, real paths. If I have to edit a command at 3 AM, I’ll get it wrong.
  4. Say when not to do it. For example: don’t promote a replica that is far behind, or you lose data.
  5. Run it when nothing is broken. A runbook that has never been followed is a guess. I try out failover and restore on purpose, and fix the document every time it surprises me.

What makes a playbook useful

  • Start from what you can see: the alert, the error message, the graph. Not from the cause, because you don’t know the cause yet.
  • Cheapest checks first. Is it up? Is the disk full? Did someone just deploy?
  • Say when to escalate, and to whom.
  • End every branch in a runbook, or in “write one, we don’t have it yet”.

How I do it: every alert points to a runbook

Writing runbooks only helps if people can find them at 3 AM. So I connect them to the alerts.

The runbooks live in a Git repository, one Markdown file per situation, reviewed in pull requests like code:

runbooks/
├── README.md                  # index: which alert → which runbook
├── postgres/
│   ├── primary-down.md
│   ├── replication-lag.md
│   └── disk-full.md
├── api/
│   ├── high-error-rate.md
│   └── high-latency.md
└── playbooks/
    └── database-not-accepting-writes.md

Every Datadog monitor links to its runbook. The monitors are defined in Terraform, and the notification message always carries the link:

resource "datadog_monitor" "pg_replication_lag" {
  name  = "[postgres] replication lag is high on {{host.name}}"
  type  = "query alert"
  query = "avg(last_5m):avg:postgresql.replication_delay{env:prod} by {host} > 60"

  monitor_thresholds {
    warning  = 30
    critical = 60
  }

  message = <<-EOT
    Replication lag on {{host.name}} is {{value}}s.

    Runbook: https://github.com/<org>/runbooks/blob/main/postgres/replication-lag.md
    @slack-oncall
  EOT

  tags = ["service:postgres", "env:prod"]
}

When the alert fires, whoever is on call opens the notification, clicks the link, and follows the steps. Nobody has to remember where the document is, or that it exists.

A few rules I keep:

  • No runbook, no page. If I can’t say what someone should do when an alert fires, the alert is probably not worth waking anyone up for.
  • Alert on symptoms that users feel (errors, latency, a disk that will be full soon), not on every high CPU.
  • Warning goes to a channel, critical pages a person. If a warning never leads to any action, I delete it.
  • The incident isn’t closed until the runbook is updated with whatever surprised us.

The part I didn’t expect

Writing runbooks changed the systems themselves.

Once the database failover was written as ten careful steps, it was obvious that a machine should be doing them. That’s what pushed me to move the cluster to Patroni with automatic failover. The runbook didn’t go away. It got shorter, and it changed from “how to fail over” to “how to check that failover worked, and what to do if it didn’t”.

The same happened elsewhere. Writing down a step often made me notice that it could be automated, or should not exist at all.

What I took away

  • If knowledge only lives in my head, the system depends on me being available. That’s a single point of failure too.
  • A runbook is for doing. A playbook is for thinking. Mixing the two makes both worse.
  • After every incident, the last task is to update the runbook or playbook. That’s how the next one gets easier.

References:

Discussion

Enjoy!