HydraIssues

Issues / #422
Test-restore the brussels-district-v2 snapshot: prove hub.key survives before the district server takes on ingress
open improvement Priority: high Project: hydraguard Reporter: 30 Jul 2026 21:54

Description

Why

brussels-district-v2 is the WireGuard hub for the whole district mesh: 24 live peers. The private key that defines that identity is /etc/wireguard/hub.key — 45 bytes, mode 600, and it exists in exactly one place. Lose it and every one of the 24 peers has to be re-keyed by hand.

Daily OVH snapshots exist, so on paper it is recoverable. What has never been checked is whether a restore actually reproduces that key. We currently have a backup we have not proven.

This gains urgency from #421: the plan is for this same box to become the ingress for scales, which raises what a loss costs.

Current state (verified 2026-07-30)
Instance brussels-district-v2 (8b7e9161-9568-4586-8996-a1fe81c714d4), b3-16, EU-WEST-LZ-BRU-A
Public IP 141.227.136.199 (fixed for the instance's life, not portable — no failover IP in this local zone)
Snapshots autobackup-brussels-district-v2, daily ~05:20 UTC, 7-day retention
Hub public key VGA6ETZB2XFVRRb5KmcFvQ+Ybfh9KKfcWuXfP1IuvQE=
Peers 24 in wg0.conf; mesh.yaml (3360 B) holds 25 public_key entries = 24 peers + hub
Peer endpoint endpoint_hostname: hydraguard.experiencenet.com — so a recreate is one DNS update, not 24 re-keys

The anomaly that prompted this

Six of the seven retained snapshots report size: 5. One does not:

2026-07-24  size=5  active
2026-07-25  size=0  active   <-- 
2026-07-26  size=5  active
2026-07-27  size=5  active
2026-07-28  size=5  active
2026-07-29  size=5  active
2026-07-30  size=5  active

It is marked active, not error — so nothing alerted, and nothing would have. Either the run silently produced an empty image, or size means something other than what it appears to. Both answers are worth having.

Time-sensitive: with 7-day retention the size: 0 snapshot rotates out around 2026-07-31. After that the specific artifact is gone and we can only wait for the next anomaly.

The test

Designed so it never touches the live hub. Restore to a new instance, verify, destroy.

1. Restore to a throwaway instance

export OVH_CLOUD_PROJECT_SERVICE=db254588bd8049bcb4ec8f0160754efe
ovhcloud cloud instance create EU-WEST-LZ-BRU-A \
  --name brussels-restore-test \
  --flavor b3-8 \
  --boot-from.image <snapshot-id> \
  --network.public --ssh-key.name neoremote \
  --billing-period hourly --wait

Hourly billing at ~EUR 0.04/hr — a 30-minute test costs roughly 2 cents.

2. Stop the mesh services immediately on boot

sudo systemctl stop hydraguard wg-quick@wg0
sudo systemctl mask hydraguard wg-quick@wg0

The clone carries the same hub.key. In practice the risk is low — peers resolve hydraguard.experiencenet.com, which points at the original IP, and a hub listens rather than initiates, so nothing would connect to the clone. But "low risk" is not "no risk" with 24 live peers, so stop them anyway.

3. Verify, in order of importance

The decisive check — everything else is secondary:

sudo wg pubkey < /etc/wireguard/hub.key
# must equal VGA6ETZB2XFVRRb5KmcFvQ+Ybfh9KKfcWuXfP1IuvQE=

Match = the mesh identity survived the backup, and a real recovery is a DNS update away. No match = the backup does not protect the one thing it most needs to, and that changes the whole plan.

Then:

Check Confirms
grep -c public_key /root/.hydraguard/mesh.yaml = 25 peer registry intact
grep -c '^\[Peer\]' /etc/wireguard/wg0.conf = 24 generated config intact
endpoint_hostname still present in mesh.yaml recreate-by-DNS path intact
services unmask + start cleanly a working box, not just the right files

4. Also boot the size: 0 snapshot (7c3776f7-52b7-4f62-b18a-723d06225548)

Do this before 2026-07-31. Either it fails to boot — confirming snapshot runs can fail silently and status: active is not trustworthy, which justifies alerting on it — or it boots fine and size means something else. Either result is useful; only the first is alarming.

5. Destroy brussels-restore-test.

Deliberately not doing

  • Restoring in place over the live instance
  • Leaving the clone running with WireGuard up
  • Testing only the newest snapshot — the interesting failure is a silent one, which is exactly what the size: 0 entry hints at

Done when

  • A restored hub.key is proven to derive the live hub public key
  • mesh.yaml / wg0.conf peer counts confirmed on the restored instance
  • The size: 0 snapshot is either shown broken or explained
  • Test instance destroyed
  • Result written into the hydraguard runbook, so the next person knows the backup was tested and when

Follow-ups this will likely justify

  • Alert on snapshot anomalies — nothing currently notices a size: 0 run
  • Get hub.key a second copy off-box — 45 bytes, single point of failure, and the snapshot is same-zone. Note /root/.hydraguard/backups/ on the box is not a backup: it dies with the instance.
  • Script the DNS update for a hub recreate (prerequisite to adding scale routes per #421)
  • Add a WireGuard reresolve-dns.sh timer — peers cache the resolved endpoint, so today they only pick up a new hub address on tunnel restart

Comments (1)

api 1 Aug 2026 18:18

Ran on Hetzner instead of OVH (2026-08-01)

Restore drill executed against a throwaway Hetzner host (cpx22, nbg1), destroyed afterwards. Nothing was created in the OVH project.

Result: PASS.

archive sha256 (source) : 35450a6a918dae011e27169d3aa816bb5aeb26843aae5a4102897e5a575456ad
archive sha256 (target) : 35450a6a918dae011e27169d3aa816bb5aeb26843aae5a4102897e5a575456ad
derived hub public key  : VGA6ETZB2XFVRRb5KmcFvQ+Ybfh9KKfcWuXfP1IuvQE=   MATCH
mesh.yaml public_key    : 25   (24 peers + hub)
wg0.conf [Peer] blocks  : 24
wg-quick strip          : OK, 24 peers parsed

The tunnel was never brought up on the clone, and key material was shredded before the host was deleted.

What this does and does not prove

It proves the recovery procedure works and a fresh backup artifact is sound — a restored hub.key reproduces the live hub identity, so recovery is a DNS update rather than 24 re-keys.

It does not test any OVH snapshot. That still requires booting one in OVH. Arguably less urgent now: an independent, verified backup path exists that does not depend on OVH snapshots at all.

The snapshot failure is a pattern, not an anomaly

The 2026-07-25 size: 0 snapshot rotated out as expected, and a new one appeared on 2026-07-31. Two failures in nine days, both reporting status: active:

07-24 size=5   07-28 size=5   08-01 size=5
07-25 size=0   07-29 size=5
07-26 size=5   07-30 size=5
07-27 size=5   07-31 size=0

So ~1 backup in 5 of the district hub is unusable, silently. Effective retention is well below the nominal 7 days.

Two findings that outrank the original question

1. /etc/wireguard/wg0.conf was mode 0644 on the live hub, with the hub PrivateKey inline. Any local user could read the identity of the 24-peer mesh. hub.key was correctly 0600, which is what made it easy to miss.

Root cause: os.WriteFile applies its mode only when it creates the file, so a config that became world-readable once stayed that way through every subsequent apply — it could not heal itself. Live hub corrected immediately (chmod does not touch the running tunnel; verified 24 peers still up). Code fix + regression test in cederikdotcom/hydraguard#2.

2. hydrabackup is not installed on the district hub. The runbook claimed mesh.yaml syncs automatically to hydramirror. It does not. There was no off-box application-level backup at all — the only off-box copy was the OVH snapshot, i.e. the unreliable one. Runbook corrected.

Also corrected while there: the runbook's SSH address was 141.227.136.12, an instance that no longer exists, so following it to recover would have failed at step one.

Note on this session's logs

While checking whether wg0.conf contained an inline key, I printed the matching line — so the hub private key appears in this session's transcript. Given the file was already world-readable on a multi-user box the marginal exposure is small, but it is a reason to weigh rotation. Rotation costs re-keying 24 peers, so it is a judgement call rather than an obvious yes.

Remaining

  • Boot an OVH snapshot to test the artifact itself (needs OVH)
  • Alert on size: 0 snapshots — nothing notices today
  • Durable encrypted off-box copy of hub.key (needs a decision on where the passphrase/key lives)
  • Decide on hub key rotation
  • Script the DNS update for a hub recreate (prerequisite to #421)
  • WireGuard reresolve-dns.sh timer