~/notes
Posts
Problems I ran into, what I tried, and what I learned along the way. Some are finished stories, some are notes I'm still adding to.
- 3,500 connections to PostgreSQL, and almost none of them working Our Tornado app hit PostgreSQL's connection limit. When I looked closer, most of those connections were just sitting there, idle or stuck in a transaction.
- Why I write runbooks and playbooks A runbook tells you how to do a known task. A playbook tells you how to think when you don't know what's wrong yet. I needed both.
- VACUUM in PostgreSQL If PostgreSQL never overwrites a row, where do all the old versions go? And why didn't my table shrink after I deleted most of it?
- Understanding MVCC in PostgreSQL What does transaction B see while transaction A is changing the same row? I finally sat down to find out what happens under a simple UPDATE.
- Deploying during business hours Our system takes payments all day, so every deploy meant downtime. Here's how we got to rolling deploys without Kubernetes, one question at a time.
- Why UFW doesn't block Docker ports I denied a port in UFW, but the container behind it was still reachable from the internet. It took me a while to stop blaming UFW and start reading iptables.
- A backup is only a hope until you restore it We had nightly backups that nobody had ever restored. Then someone asked what we'd get back if we lost the database at 2 PM.
- Hazelcast auto-discovery on Kubernetes Scale a Hazelcast deployment up and the new pods just join the cluster, with no member list anywhere. I liked that it worked. I didn't like not knowing why.
- Reading Spring Reactive I could read Mono and Flux, but I couldn't say what they actually bought anyone. So I sat down and figured out what "reactive" really changes.
- How Docker containers talk to each other How does a container reach another container, the host, or the internet underneath? What actually carries the packets?