Hazelcast auto-discovery on Kubernetes

A problem

We were running a Spring application with Hazelcast embedded in it, as a distributed cache shared between instances. On Kubernetes, we scaled the deployment from 2 replicas to 4, and the logs of the new pods showed something like:

Members {size:4, ver:4} [
    Member [10.42.0.15]:5701 - 3f2a... this
    Member [10.42.1.22]:5701 - 9c1b...
    Member [10.42.2.31]:5701 - 77de...
    Member [10.42.0.16]:5701 - a0e4...
]

Four members, one cluster. The new pods had found the old ones and joined, all by themselves.

There was no list of IPs anywhere in our config. Pod IPs change every time a pod restarts anyway, so we couldn’t have written one even if we’d wanted to. So how did a brand-new pod know who to talk to?

I liked that it worked. I didn’t like that I couldn’t explain it. So I went digging.

Why the default doesn’t work here

Out of the box, Hazelcast finds other members using multicast: a new member shouts “is anyone out there?” to a multicast address on the local network, and existing members answer.

That’s fine on a flat office network. On Kubernetes it mostly doesn’t work. Most CNI plugins (the network layer of the cluster) don’t carry multicast traffic between pods, and many cloud networks block it entirely. So something else had to be doing the work.

The Kubernetes discovery plugin

Hazelcast has a Kubernetes discovery mode. Instead of shouting on the network, the new member asks Kubernetes itself: “which pods belong to my cluster?”

There are two ways it can ask.

Mode 1: the Kubernetes API

The member calls the Kubernetes API server and asks for the Endpoints of a Service, the list of pod IPs that Kubernetes already keeps up to date for that Service.

new pod starts
   │
   ├─► GET /api/v1/namespaces/my-ns/endpoints/hz-service   (Kubernetes API)
   │      ← [10.42.0.15, 10.42.1.22, 10.42.2.31]
   │
   └─► connect to each IP on port 5701, ask to join

For that, the pod’s service account needs permission to read those resources:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: hazelcast-discovery
rules:
  - apiGroups: [""]
    resources: ["endpoints", "pods", "services"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: hazelcast-discovery
subjects:
  - kind: ServiceAccount
    name: my-app
roleRef:
  kind: Role
  name: hazelcast-discovery
  apiGroup: rbac.authorization.k8s.io

This was the first thing I checked in our setup, and sure enough, the Role was there, added by someone long before me. Without it, the API call fails, and the member quietly forms a cluster of one. No crash, just four pods each holding their own separate cache. That’s a nasty failure, because everything looks fine.

Mode 2: DNS lookup

The other option needs no API access at all. You create a headless Service (clusterIP: None):

apiVersion: v1
kind: Service
metadata:
  name: hz-headless
spec:
  clusterIP: None
  selector:
    app: my-app
  ports:
    - name: hazelcast
      port: 5701

A normal Service gets one virtual IP that load-balances to the pods. A headless Service doesn’t. When you look it up in DNS, you get one A record per ready pod:

$ nslookup hz-headless.my-ns.svc.cluster.local
Name:    hz-headless.my-ns.svc.cluster.local
Address: 10.42.0.15
Address: 10.42.1.22
Address: 10.42.2.31

Hazelcast resolves that name and connects to every address it gets back.

hazelcast:
  network:
    join:
      multicast:
        enabled: false
      kubernetes:
        enabled: true
        service-dns: hz-headless.my-ns.svc.cluster.local

DNS mode is simpler (no RBAC), and API mode gives Hazelcast more information about the pods. Either way, the trick is the same: Kubernetes already knows where every pod is, so just ask it.

What happens after discovery

Finding each other is only the first step. Once a new member connects:

It asks to join, and the oldest member in the cluster (the one coordinating membership) accepts it and tells everyone the new member list.
Hazelcast splits data into 271 partitions by default, spread across members. When a new member arrives, some partitions migrate to it, so the data is rebalanced.
Each partition also has a backup copy on a different member, so losing one pod doesn’t lose its data.

That explained something else I’d noticed: right after scaling up, the logs were full of partition migration messages. The new pod wasn’t just joining. It was taking over its share of the data.

The things that can bite you

Digging into this, I found a few traps worth knowing:

Readiness and discovery depend on each other. Endpoints (and the headless Service’s DNS records) only include pods that are ready. If your readiness check says “ready only once the cluster has formed”, pods wait for each other forever. Setting publishNotReadyAddresses: true on the discovery Service breaks that loop.
Restarting too many pods at once loses data. If two pods holding a partition and its backup go down together, that partition is gone. Rolling updates should replace pods slowly, one at a time, giving partitions time to migrate. A PodDisruptionBudget helps here too.
Shut down gracefully. With hazelcast.shutdownhook.policy=GRACEFUL, a member hands its partitions over to others before leaving, instead of just disappearing.
Split brain. If the network splits the cluster in two, each half keeps working on its own, and they merge when the network heals. What happens to conflicting data then is decided by the merge policy, which is worth choosing on purpose rather than by default.

What I took away

The “magic” turned out to be a pretty simple idea: Kubernetes already tracks every pod, so Hazelcast doesn’t need its own discovery protocol. It just asks. Once I understood that, the failure modes made sense too: no permission to ask means no cluster, and no ready pods means nothing to find.