← all posts
#deployment#traefik#docker#postgresql

Deploying during business hours

Our system takes payments all day, so every deploy meant downtime. Here's how we got to rolling deploys without Kubernetes, one question at a time.

A problem

Stores use our system all day, and every one of those days is full of payments. So the rule was simple: we only deploy outside business hours.

That works fine for planned releases. It doesn’t work for hotfixes. When something is broken now, you can’t wait until midnight, so we’d deploy in the middle of the day and accept a few seconds (sometimes more) of downtime. A cashier with a customer at the counter doesn’t care that it was “only a few seconds”. Annoying, right?

So the question for me and the team became: how do we deploy without anyone at the counter noticing?

Brainstorming

My first thought was Kubernetes. Rolling updates, readiness probes, all built in. But most of our system runs on bare metal and self-hosted servers with Docker. Moving everything to Kubernetes just to fix deployments would be a big project on its own.

Then again, Kubernetes isn’t magic. Rolling deployments are built from a few basic ideas, and I could rebuild those ideas myself. So I wrote down the questions I’d need to answer:

  1. How does the reverse proxy find the new version and start sending it traffic?
  2. How does the old version stop without dropping anything it’s in the middle of?
  3. What about transactions? This is a payment system. A request cut off halfway is not just an error page, it might be money.
  4. What about the database? The app can run two versions side by side. The database can’t.

OK, let’s go through them one by one.

1. Letting the proxy find new containers: from nginx to Traefik

We used to run nginx in front of the app, with the upstream servers written in its config:

upstream pos_api {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
}

That’s fine while nothing changes. But in a rolling deploy, containers come and go all the time. With nginx, every new container meant editing the config and reloading nginx, and doing it at exactly the right moment. That’s a lot of moving parts for a script to get right.

So we moved to Traefik. The feature that sold me is autodiscovery: Traefik watches the Docker API and builds its routing from container labels. A container starts, Traefik picks it up. A container stops, Traefik drops it. Nobody edits a config file.

docker run -d --name api-${VERSION} \
  --label traefik.enable=true \
  --label 'traefik.http.routers.api.rule=Host(`api.example.com`)' \
  --label traefik.http.services.api.loadbalancer.server.port=8080 \
  --health-cmd 'curl -fsS http://localhost:8080/health/ready' \
  --health-interval 5s --health-timeout 2s --health-retries 3 --health-start-period 20s \
  --stop-timeout 60 \
  registry.example.com/api:${VERSION}

The health check matters more than it looks. When a container has a health check, Traefik only sends it traffic once Docker reports it as healthy. So a new container that’s still starting up, or one that’s broken, never gets a single request.

With that, a deploy looks like this:

1. start the new version next to the old one     (old: serving, new: starting)
2. wait until the new one is healthy             (Traefik starts routing to it)
3. drain and stop the old one                    (Traefik stops routing to it)
4. if the new one never becomes healthy → stop it, keep the old one, fail the pipeline

Step 4 is the safety net. A bad release never replaces a good one, because it has to prove it’s healthy first.

2. Draining: stopping without dropping anything

Starting the new version was the easy part. Stopping the old one cleanly was where I learned the most.

When Docker stops a container, it sends SIGTERM, waits for the stop timeout (--stop-timeout), then sends SIGKILL. What happens in between is entirely up to the application. And when I actually tested it, some of our services just… exited immediately. Any request in flight was cut off. That was a fun afternoon.

What the app needs to do on SIGTERM:

  1. Stop accepting new requests.
  2. Finish the ones already in progress.
  3. Then exit.

In Spring Boot, for example, that’s two settings:

server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=30s

There’s a subtle race, though: the moment the app stops accepting connections, the proxy might still send it a request before it notices the container is going away. So we drain in two steps:

1. mark the old instance as "not ready"  → health check fails → Traefik removes it
2. wait a few seconds                     → no new requests arrive
3. send SIGTERM                           → it finishes what it has, then exits

And the stop timeout has to be longer than the slowest request you’re willing to wait for. Otherwise Docker sends SIGKILL right in the middle of it.

Transactions, and why idempotency matters

Draining handles the normal case: a request comes in, finishes, and the instance exits. But a payment system has to think about the abnormal case too. What if the process dies halfway anyway? A server crashes, an OOM kill, a request that takes longer than the grace period.

For database work alone, PostgreSQL protects us: a transaction that never commits is rolled back. Nothing half-written survives.

The real danger is when a request does two things in two different systems, like charging a card with the payment provider and saving the order in our database:

1. call the payment provider → card charged ✅
2. save the order in our DB  → process killed ❌

The customer has paid, but we have no order. The screen at the counter shows an error, the cashier tries again, and the customer is charged twice.

Drain timing alone can’t fully protect against this. Idempotency can. So, as part of this work, we went through every endpoint that changes something important and asked: if this request is sent twice, does something bad happen?

  • Payments carry an idempotency key, generated by the client once per attempt and sent to the payment provider. A retry with the same key returns the original result instead of charging again.
  • Our own write endpoints check for that key too, so a retried “create order” returns the existing order instead of creating a second one.
  • Background jobs either finish within the grace period or go back to the queue, and processing the same job twice must be safe.

Honestly, this checklist found more problems than the deploy work itself. Some of them were bugs waiting to happen with or without rolling deploys. A flaky network can cause a retry just as easily as a deploy can.

3. The database doesn’t roll

The app now rolls one instance at a time. But there’s only one database, which means for a few minutes the old version and the new version run against the same schema.

That’s where migrations can quietly break everything. Say the new version renames a column:

ALTER TABLE orders RENAME COLUMN total TO total_amount;

The migration runs, the new containers are happy, and every old container still running starts failing with column "total" does not exist, in the middle of a rolling deploy, during business hours. The exact thing we were trying to avoid.

The rule we follow now: every migration must work with both the old and the new version of the code. The pattern for that is expand, then contract:

release 1  expand    add total_amount (nullable); write to both total and total_amount
release 2  migrate   backfill old rows; read from total_amount
release 3  contract  stop writing total; then drop it

It’s slower. A column rename takes three releases instead of one. But no single release ever depends on the database and the code changing at the same moment.

A few more things we learned the hard way about migrations on a live database:

  • Migrations run before the rollout, as a separate step, so the schema is already “expanded” when the new code starts.
  • Watch the locks. CREATE INDEX blocks writes on the table while it runs. Use CREATE INDEX CONCURRENTLY instead.
  • Adding a NOT NULL column to a table that old code still inserts into will break those inserts. Add it as nullable (or with a default) first, and tighten it in a later release.
  • Set a lock_timeout for migrations, so a migration waiting on a lock fails fast instead of making every query behind it wait too.

What I took away

We never moved to Kubernetes. We didn’t need to. Each piece Kubernetes gives you (discovery, readiness, graceful termination) turned out to be something we could understand and build with the tools we already had.

But the bigger lesson wasn’t about tools. “Zero downtime” isn’t a feature you install. You get it by asking, honestly, what happens to this one request in flight while we deploy?, and then fixing every place where the answer was “it gets dropped” or “it gets charged twice”.

Now we deploy hotfixes in the middle of the day, and the cashiers don’t notice. That’s the whole point.

Discussion

Enjoy!