Resilience & recovery¶

How the cluster survives a node failure, and the playbook for when something breaks. The design goal is simple: any single node can die without taking down stored data or the public sites.

The setup¶

Two things used to be single points of failure — one box held all the data, and one box was the only public entry point. Both are now spread across three nodes.

Shared storage — GlusterFS¶

A replica-3 volume gv0: three copies of every file, one brick on each of nuc1 / nuc2 / nuc3. Mounted at /mnt/gluster on every node.
All persistent service state lives under it — the container registry, Portainer's database, and NPM's config + Let's Encrypt certificates — via docker bind-volumes pointing at /mnt/gluster/….
Write quorum is 2 of 3, so one brick node can be down and the volume stays read-write. It self-heals automatically when the node returns.

Ingress — floating VIP¶

A keepalived virtual IP 192.168.0.250 floats across nuc1 / nuc2 / nuc3 (priority: nuc1 highest). The router forwards public 80/443 to the VIP, not to any single node.
NPM (the reverse proxy) is unpinned and published via the swarm routing mesh, so whichever node holds the VIP can serve traffic. If that node dies, the VIP jumps to a healthy node and NPM keeps answering.

Off-cluster backup¶

Replica-3 protects against a node dying, but it is not a backup — a bad delete replicates instantly to all three copies. A separate box (.8, off-cluster) pulls a nightly encrypted restic backup of swarm state (registry, Portainer, NPM config + Let's Encrypt certs, and the swarm-raft cluster identity), with per-tag retention and an off-site mirror — the 3-2-1 safety net. It reads the live data from /mnt/gluster (and the raft dir on the Vbox1 manager).

Backups need watching

A backup that silently stops is worse than none — you think you're safe. This one once archived empty volumes for a week after the storage migration (its targets still pointed at emptied volume names) with nothing raising a flag. It's now checked from the outside every night: aos backup check confirms every restic tag has a recent, non-empty snapshot and emails on any problem.

Net effect

Lose any one node and: data stays online (2 remaining brick copies), and public sites stay online (VIP + NPM move to a survivor). Recovery is automatic and takes ~1–2 minutes.

If something breaks¶

Work top-down: confirm the symptom, then apply the matching fix. Most failures self-recover; these are for when they don't.

A node rebooted or is down¶

Swarm reschedules its services automatically. Check the fleet from any manager:

docker node ls          # all nodes Ready/Active? managers reachable?

Quorum: there are 3 managers; the cluster tolerates one manager down. Do not reboot a second manager while one is already down — that loses quorum and the swarm goes read-only.
After nodes come back, containers may bunch on one node. Rebalance them (skips pinned/global/ingress services automatically):

aos swarm rebalance      # or the swarm-rebalance skill

Public sites are down¶

The sites answer through the VIP → NPM. Check both:

# who currently holds the VIP?  (run on nuc1/2/3)
ip addr show eno1 | grep 192.168.0.250

# is NPM running, and does it serve each vhost?  (run on the node with NPM)
curl -sk --resolve ommen.it:443:127.0.0.1 https://ommen.it/ -o /dev/null -w '%{http_code}\n'

Testing TLS on the LAN

Always use --resolve host:443:127.0.0.1 so the request carries the right SNI. A plain curl https://127.0.0.1 (or -H Host:) sends SNI 127.0.0.1, which nginx rejects with an "unrecognized name" TLS alert — a false 000 that looks like an outage when nothing is wrong.

If NPM isn't running, redeploy the proxy stack (see the swarm repo infra/proxy). If the VIP is on a node whose mesh is broken, keepalived's health check should already have moved it — verify that node's overlay (next item).

502s, or DNS failures inside containers¶

Usually a stale overlay-network (vxlan) sandbox on one node after an unclean reboot. Fix it on the affected node:

sudo systemctl restart docker    # clears stale vxlan sandboxes

`/mnt/gluster` is empty or gives "Transport endpoint not connected"¶

The FUSE client mount dropped (often after a glusterd restart — restarting the daemon tears down every node's client mount). Remount on the affected node:

sudo umount /mnt/gluster 2>/dev/null; sudo mount -a
ls /mnt/gluster            # should show registry/ portainer/ npm-data/ …

Gluster shows pending heals or split-brain¶

Check, then let self-heal run (a returning node heals on its own):

sudo gluster volume heal gv0 info summary     # entries pending / in split-brain
sudo gluster volume status gv0                # all 3 bricks Online = Y?

Split-brain (the same file changed on two sides while they were partitioned) is rare with quorum-2 and needs manual resolution — pick the good copy with gluster volume heal gv0 split-brain latest-mtime <file>.

Registry pulls fail (new deploys can't get images)¶

The registry is still pinned to nuc1 and addressed at 192.168.0.133:5000 (baked into every node's daemon.json and image tags). Its data is safe on Gluster, but availability needs nuc1 up — bring nuc1 back and pulls resume. Already-running services are unaffected.

Recovering actual lost/corrupted data¶

Gluster won't save you from a bad delete — restore from the restic backup on .8. The pattern per volume: list snapshots, dump the tag's tar, ship it to the owning node, and load it into the live volume via a throwaway container.

# on .8 — what's there?
sudo -E env $(grep -v '^#' /opt/swarm-backup/env) restic snapshots --tag npm-data

# dump the latest npm-data snapshot and copy it to the NPM node
restic dump --tag npm-data latest npm-data.tar > /tmp/npm-data.tar
scp /tmp/npm-data.tar lommen@<npm-node>:/tmp/

# on that node — stop the service, repopulate the live volume, restart
docker service scale proxy_app=0
docker run --rm -v <live-npm-volume>:/dst -v /tmp/npm-data.tar:/in.tar alpine:3.20 \
    sh -c 'rm -rf /dst/* /dst/.[!.]* 2>/dev/null; tar -xf /in.tar -C /dst'
docker service scale proxy_app=1

Restore into the current target

Since the GlusterFS migration the live data is bind-mounted under /mnt/gluster/…, not the old per-node volume names. Confirm the current mount for the service before restoring (docker service inspect <svc>). Full step-by-step + disaster-recovery order is in the swarm repo (infra/backup/restore-howto.md), which carries the same caveat.

Growing the storage¶

If the bricks' disk fills up, move each brick onto a bigger disk — online, one node at a time, no service changes (the volume name and /mnt/gluster stay identical):

# on the node: add + format + mount a new disk (XFS), then:
sudo gluster volume replace-brick gv0 \
  <node>:/data/brick-gv0/brick <node>:/data/brick-gv0-new/brick commit force
sudo gluster volume heal gv0 info summary     # wait for 0 entries

One brick at a time

Never replace two bricks at once — quorum is 2/3, so a second brick offline loses redundancy. Finish and heal one node fully before starting the next.

To add total capacity via more nodes instead of bigger disks, add bricks in groups of three (add-brick → distributed-replicated) followed by a rebalance.