~/notes/incident

#incident

Problems I ran into, what I tried, and what I learned along the way. Some are finished stories, some are notes I'm still adding to.

  1. 3,500 connections to PostgreSQL, and almost none of them working Our Tornado app hit PostgreSQL's connection limit. When I looked closer, most of those connections were just sitting there, idle or stuck in a transaction. #postgresql#python#tornado#incident
  2. Why I write runbooks and playbooks A runbook tells you how to do a known task. A playbook tells you how to think when you don't know what's wrong yet. I needed both. #operations#incident#datadog#postgresql