HydraIssues

## Why

`brussels-district-v2` is the WireGuard hub for the whole district mesh: **24 live peers**. The private key that defines that identity is `/etc/wireguard/hub.key` — **45 bytes, mode 600, and it exists in exactly one place**. Lose it and every one of the 24 peers has to be re-keyed by hand.

Daily OVH snapshots exist, so on paper it is recoverable. What has never been checked is whether a restore actually reproduces that key. We currently have a backup we have not proven.

This gains urgency from #421: the plan is for this same box to become the ingress for scales, which raises what a loss costs.

## Current state (verified 2026-07-30)

| | |
|---|---|
| Instance | `brussels-district-v2` (`8b7e9161-9568-4586-8996-a1fe81c714d4`), b3-16, `EU-WEST-LZ-BRU-A` |
| Public IP | `141.227.136.199` (fixed for the instance's life, **not** portable — no failover IP in this local zone) |
| Snapshots | `autobackup-brussels-district-v2`, daily ~05:20 UTC, 7-day retention |
| Hub public key | `VGA6ETZB2XFVRRb5KmcFvQ+Ybfh9KKfcWuXfP1IuvQE=` |
| Peers | 24 in `wg0.conf`; `mesh.yaml` (3360 B) holds 25 `public_key` entries = 24 peers + hub |
| Peer endpoint | `endpoint_hostname: hydraguard.experiencenet.com` — so a recreate is one DNS update, not 24 re-keys |

## The anomaly that prompted this

Six of the seven retained snapshots report `size: 5`. One does not:

```
2026-07-24 size=5 active
2026-07-25 size=0 active <--
2026-07-26 size=5 active
2026-07-27 size=5 active
2026-07-28 size=5 active
2026-07-29 size=5 active
2026-07-30 size=5 active
```

It is marked `active`, not `error` — so nothing alerted, and nothing would have. Either the run silently produced an empty image, or `size` means something other than what it appears to. Both answers are worth having.

**Time-sensitive:** with 7-day retention the `size: 0` snapshot rotates out around **2026-07-31**. After that the specific artifact is gone and we can only wait for the next anomaly.

## The test

Designed so it never touches the live hub. Restore to a **new** instance, verify, destroy.

### 1. Restore to a throwaway instance

```bash
export OVH_CLOUD_PROJECT_SERVICE=db254588bd8049bcb4ec8f0160754efe
ovhcloud cloud instance create EU-WEST-LZ-BRU-A \
--name brussels-restore-test \
--flavor b3-8 \
--boot-from.image <snapshot-id> \
--network.public --ssh-key.name neoremote \
--billing-period hourly --wait
```

Hourly billing at ~EUR 0.04/hr — a 30-minute test costs roughly **2 cents**.

### 2. Stop the mesh services immediately on boot

```bash
sudo systemctl stop hydraguard wg-quick@wg0
sudo systemctl mask hydraguard wg-quick@wg0
```

The clone carries the same `hub.key`. In practice the risk is low — peers resolve `hydraguard.experiencenet.com`, which points at the original IP, and a hub listens rather than initiates, so nothing would connect to the clone. But "low risk" is not "no risk" with 24 live peers, so stop them anyway.

### 3. Verify, in order of importance

**The decisive check** — everything else is secondary:

```bash
sudo wg pubkey < /etc/wireguard/hub.key
# must equal VGA6ETZB2XFVRRb5KmcFvQ+Ybfh9KKfcWuXfP1IuvQE=
```

Match = the mesh identity survived the backup, and a real recovery is a DNS update away. No match = the backup does not protect the one thing it most needs to, and that changes the whole plan.

Then:

| Check | Confirms |
|---|---|
| `grep -c public_key /root/.hydraguard/mesh.yaml` = 25 | peer registry intact |
| `grep -c '^\[Peer\]' /etc/wireguard/wg0.conf` = 24 | generated config intact |
| `endpoint_hostname` still present in `mesh.yaml` | recreate-by-DNS path intact |
| services unmask + start cleanly | a working box, not just the right files |

### 4. Also boot the `size: 0` snapshot (`7c3776f7-52b7-4f62-b18a-723d06225548`)

Do this **before 2026-07-31**. Either it fails to boot — confirming snapshot runs can fail silently and `status: active` is not trustworthy, which justifies alerting on it — or it boots fine and `size` means something else. Either result is useful; only the first is alarming.

### 5. Destroy `brussels-restore-test`.

## Deliberately not doing

- Restoring in place over the live instance
- Leaving the clone running with WireGuard up
- Testing only the newest snapshot — the interesting failure is a *silent* one, which is exactly what the `size: 0` entry hints at

## Done when

- [ ] A restored `hub.key` is proven to derive the live hub public key
- [ ] `mesh.yaml` / `wg0.conf` peer counts confirmed on the restored instance
- [ ] The `size: 0` snapshot is either shown broken or explained
- [ ] Test instance destroyed
- [ ] Result written into the hydraguard runbook, so the next person knows the backup was tested and when

## Follow-ups this will likely justify

- **Alert on snapshot anomalies** — nothing currently notices a `size: 0` run
- **Get `hub.key` a second copy off-box** — 45 bytes, single point of failure, and the snapshot is same-zone. Note `/root/.hydraguard/backups/` on the box is *not* a backup: it dies with the instance.
- **Script the DNS update** for a hub recreate (prerequisite to adding scale routes per #421)
- **Add a WireGuard `reresolve-dns.sh` timer** — peers cache the resolved endpoint, so today they only pick up a new hub address on tunnel restart