Pre-event work to unblock partner read access to Citymesh-managed MikroTik routers at cloud-seven and rupelmonde, so hydraneck can correlate venue network state during the 2026-05-22 → 2026-05-25 event.
First pass of this issue framed it as a public-internet API call with an IP whitelist (46.225.8.28). That's off-design. The documented Partner Network Design (last touched 2026-04-09) specifies that partner-managed routers join our hydraguard mesh as venue peers — the partner installs a WG peer profile we provide, no public-internet API surface required.
Corrected procedure (now documented in hydraguard/docs/runbooks/citymesh-venue.md and hydraneck/docs/runbooks/mesh-participation.md):
ssh ubuntu@141.227.136.199
hydraguard venue add cloud-seven --location bxl1 --guard citymesh
hydraguard venue add rupelmonde --location bxl1 --guard citymesh
hydraguard apply
hydraguard venue config cloud-seven > /tmp/cloud-seven.partner.conf
hydraguard venue config rupelmonde > /tmp/rupelmonde.partner.conf
Per the partner design's 'What HYDRA Provides' bundle:
.partner.conf files (peer profiles to install on the MikroTiks)hydraguard.experiencenet.com:51820/udp (their MikroTiks need outbound UDP/51820):443 preferred, legacy binary on :8729 acceptable fallback)hydraguard air add hydraneck
hydraguard apply
hydraguard air config hydraneck > /tmp/hydraneck.wg.conf
Install the resulting config on hydraneck.experiencenet.com:
ssh root@46.225.8.28
apt-get install -y wireguard
install -m 600 /path/to/hydraneck.wg.conf /etc/wireguard/wg0.conf
systemctl enable --now wg-quick@wg0
Once Citymesh confirms the peer is up and provides API creds, edit /root/.hydraneck/config.yaml:
venues:
cloud-seven:
mikrotik:
host: 10.10.X.1 # WG tunnel address — NOT public WAN
user: <readonly>
pass: <password>
transport: rest
rupelmonde:
mikrotik:
host: 10.10.Y.1
user: <readonly>
pass: <password>
transport: rest
The exact WG addresses are assigned by hydraguard apply — confirm via hydraguard status.
partner vs citymesh doc/code terminology drift — doc says partner, code requires citymesh. Use citymesh in CLI for now; rename tracked separately.hydraguard status shows handshake for both venues; ping <venue-LAN-gateway> succeeds.wg show shows hub handshake; ping 10.10.0.1 succeeds; curl -sk https://<venue-mikrotik-WG-IP>/rest/system/resource returns JSON.hydraneck scan cloud-seven and hydraneck scan rupelmonde populate clients + bandwidth.~/.claude/plans/meandering-trotting-mongoose.md~/.claude/plans/breezy-gliding-narwhal.mdCitymesh reported cloud-seven (10.10.2.1) and rupelmonde (10.10.3.1) cannot ping hydraneck (10.10.100.14), while mobile-kit (10.10.4.1) works.
Our side is not blocking. Full investigation:
ACCEPT for all wg0→wg0 traffic — no rules blocking inter-peer traffic10.10.0.0/16, 10.0.0.0/8 — covers all venue WG addressesThe mesh is bidirectional and routed correctly. If hydraneck's pings reach cloud-seven/rupelmonde and their ICMP replies traverse hub → hydraneck successfully, the path from venues to hydraneck is open end-to-end.
Root cause is on the MikroTik side at cloud-seven and rupelmonde. Likely:
10.10.0.0/16 via wg-interface — MikroTik does not auto-derive routes from AllowedIPs the way Linux does; the route must be added manually in /ip routeRecommended check for Citymesh: compare MikroTik routing table and firewall on cloud-seven/rupelmonde vs mobile-kit:
/ip route print — is there a route for 10.10.0.0/16 (or 10.10.100.0/24) via the WG interface on cloud-seven and rupelmonde?/ip firewall filter print — any rule dropping outgoing ICMP from 10.10.2.1 or 10.10.3.1?/ping 10.10.100.14 src-address=10.10.2.1 from cloud-seven MikroTik terminal to confirm source routingAll three venues now have WG handshakes and bidirectional routing confirmed:
| Venue | WG IP | Ping from hydraneck | Route status |
|---|---|---|---|
| cloud-seven | 10.10.2.1 |
20ms | route confirmed by Citymesh |
| rupelmonde | 10.10.3.1 |
32ms | route confirmed by Citymesh |
| mobile-kit | 10.10.4.1 |
48ms | route confirmed by Citymesh |
Root cause of initial ping failure was a missing explicit MikroTik route — MikroTik does not auto-derive routes from WireGuard AllowedIPs unlike Linux. Citymesh added 10.10.0.0/16 → WG interface on each device. Route for mobile-kit appeared to work earlier only because it was temporarily behind our own router (false positive).
Known constraint: Citymesh datacenter uses 10.10.8.0/23. Do not assign HYDRA WG addresses or source from that range — their management VPN more-specific route handles it on their side, but our side would lose replies.
Runbooks updated and pushed:
hydraguard/docs/runbooks/citymesh-venue.md — MikroTik routing requirement, 10.10.8.0/23 overlap, mobile-kit in reference table, asymmetric ping diagnostichydraneck/docs/runbooks/mesh-participation.md — MikroTik routing note in enrollment stepshydraneck/docs/runbooks/address-ranges.md — 10.10.8.0/23 overlap warningNext step: receive read-only MikroTik API credentials from Citymesh (cloud-seven + rupelmonde already sent; mobile-kit in mailbox per Citymesh message) and wire into /root/.hydraneck/config.yaml.
Credentials received and wired into hydraneck:
mobile-kit (bxl1, visit-flanders)10.10.4.1:8729, user hydra, protocol api (API-SSL)/root/.hydraneck/secrets.yaml under mobile-kit/mikrotik-mobile-kitConnectivity status:
10.10.4.1 from hydraneck: ✓ (ICMP works, WG tunnel confirmed)Root cause: On MikroTik, two layers must both allow the connection:
/ip firewall filter INPUT chain — the rule Citymesh added (from 10.10.100.14)/ip service settings — the API/API-SSL service must not restrict the allowed-from address in a way that blocks the WG interfaceICMP passing but all TCP failing is the classic symptom of the firewall filter rule allowing ICMP only (or a missing TCP rule), OR the service being bound to specific interfaces that exclude the WG interface.
Ask for Citymesh:
/ip service print — is API (port 8728) or API-SSL (port 8729) enabled and what is available-from?/ip firewall filter print chain=input — is there a rule allowing TCP from 10.10.100.14 to port 8728 or 8729?/tool telnet 10.10.100.14 from the MikroTik terminal to verify the WG route back to us works from their side1. Mobile-kit API — TCP ports blocked (Citymesh action needed)
WG tunnel up, ping 10.10.4.1 works. TCP 8729/8728/443 all timeout. MikroTik requires both layers: /ip firewall filter INPUT chain rule AND /ip service available-from settings. Asked Citymesh to verify both with /ip service print and /ip firewall filter print chain=input.
2. Cloud-seven + rupelmonde credentials — link expired
The secure link Citymesh sent with read-only API credentials has expired. Resend requested.
Email verstuurd naar Yentel met twee punten:
/ip service en /ip firewall filter chain=input te controleren op de mobile-kit MikroTikWachten op reactie Citymesh.
Root cause of username failure: config had hydra-read (dash) but actual MikroTik user is hydra_read (underscore). Fixed.
Config changes applied to hydraneck:
address: 10.10.2.1:8728, protocol: api, fixed username: hydra_readaddress: 10.10.3.1:8728, protocol: api, fixed username: hydra_readScan results:
mobile-kit still blocked: stored password does not match hydra user. Error code 6 from RouterOS (invalid user name or password). New password requested from Citymesh.
When Citymesh provides the new mobile-kit password, run:
curl -sk -X PUT https://hydraneck.experiencenet.com/admin/api/secrets/mobile-kit/mikrotik-mobile-kit \
-H "Authorization: Bearer <hydraneck-admin-token>" \
-H "Content-Type: application/json" \
-d '{"password":"<new-pw>"}'
hydraneck scan mobile-kit
(admin token is in /root/.hydraneck/config.yaml on hydraneck.experiencenet.com)
Open items:
Hub-side enrollment done. Both venues added as citymesh-guard peers in the Brussels hub mesh:
| Venue | WG addr | LAN CIDR | Status |
|---|---|---|---|
| cloud-seven | 10.10.2.1/32 |
10.0.2.0/24 |
added, awaiting Citymesh install |
| rupelmonde | 10.10.3.1/32 |
10.0.3.0/24 |
added, awaiting Citymesh install |
Reference table in hydraguard/docs/runbooks/citymesh-venue.md committed (7944a96).
hydraneck enrolled as air peer in the same mesh — air-hydraneck at 10.10.100.14. Handshake confirmed (~14ms RTT to hub over WG). Ready to scan the venues over the tunnel once Citymesh installs their side.
Email + both .partner.conf attachments sent to Citymesh today.
/root/.hydraneck/config.yaml updated with WG-side hostshydraneck scan cloud-sevenhydraneck scan rupelmondehydrarelease updater .backup filename chain) — done, shipped in hydrarelease v1.17.3hydraguard wrong systemd unit name) — done, shipped in hydraguard v1.10.9add subcommands corrupted pipeable config) — done, shipped in hydraguard v1.10.10Citymesh asked: Which IP should they allowlist in the MikroTik firewall rules to allow REST and binary API connections from hydraneck? They suspected it would be the WireGuard gateway IP but wanted confirmation, and asked whether there is a LAN range between the WG server and hydraneck.
Answer: allow 10.10.100.14/32 — hydraneck's WireGuard address.
Verified on the Brussels hub (141.227.136.199): no NAT in the POSTROUTING chain, pure routing between WG peers. This means the MikroTik sees hydraneck's own WG IP as the source — not the hub IP.
Ports to open inbound from 10.10.100.14:
No public internet IP needs to be whitelisted. There is no LAN range between the WG hub and hydraneck — hydraneck is an air peer directly in the mesh at 10.10.100.14; packets travel 10.10.100.14 → hub → 10.10.2.1 / 10.10.3.1 with source IP preserved.
Citymesh confirmed they will implement this week, expected Wednesday 2026-05-14 (venue IT staff are on an install tomorrow).
The mobile Hydra kit is also Citymesh-managed, so it follows the same enrollment path as cloud-seven and rupelmonde.
Hub-side enrollment done.
| Venue | WG addr | LAN CIDR | Guard | Status |
|---|---|---|---|---|
| cloud-seven | 10.10.2.1/32 |
10.0.2.0/24 |
citymesh | awaiting Citymesh install |
| rupelmonde | 10.10.3.1/32 |
10.0.3.0/24 |
citymesh | awaiting Citymesh install |
| mobile-kit | 10.10.4.1/32 |
10.0.4.0/24 |
citymesh | added today — awaiting Citymesh install |
Partner config saved to /tmp/mobile-kit.partner.conf on the Brussels hub (141.227.136.199). Needs to be included in the next handover email to Citymesh alongside the cloud-seven and rupelmonde configs (or as a follow-up if those are already sent).
/root/.hydraneck/config.yaml updated with WG-side hostshydraneck scan cloud-sevenhydraneck scan rupelmondehydraneck scan mobile-kitmobile-kit.partner.conf sent to Citymesh on 2026-05-13 (see #167). Awaiting acknowledgement.
Feedback citymesh: Rupelmonde VPN, firewall rules actief gezet (met user die je reeds had gekregen) maar daar kan ik niet pingen naar 10.10.100.14 desondanks dat de wireguard een allowed address 10.10.0.0/16 heeft staan, kan het zijn dat jouw kant nog iets blocked komende van 10.10.3.1 (rupelmonde) die wel werkt op mobilekit (10.10.4.1)?
Zelfde probleem voor cloud7 vanaf 10.10.2.1, daarmee kan ik niet naar 10.10.100.14
[14/5, 16:27] Yentel Hollebeke: De reden dat mobilekit werkte is omdat die achter jullie router zat
hij kon dus naar de hub via wan over jullie router, maar dat is niet de juiste weg als hij ergens anders zou staan.
Met wat finetuning van de routes (want wireguard routes worden niet automatisch toegevoegd aan de routing table) en te zeggen dat 10.10.100.14 over de wireguard moet gestuurd worden zou het moeten lukken
[14/5, 16:27] Yentel Hollebeke: Die zitten in je mailbox voor de dev kit (de andere 2 had je al) normaalgezien.
[14/5, 16:28] Yentel Hollebeke: Ik heb voorlopig wel enkel een route staan voor de 10.10.100.14 over wireguard, niet voor de volledige 10.10.0.0/16 range, maar kan dat op zich wel zo aanpassen.
Opgepast, die route overlapt met ons datacenter op 10.10.8.0/23 dus zolang daar niets in zit die belangrijk is langs jullie kant zou dat geen problemen mogen geven.
[14/5, 16:33] C: Miss best Dan een andere route zodat we niet botsen met jullie datacenter?
[14/5, 16:34] Yentel Hollebeke: Ik kan op zich perfect de 10.10.0.0/16 richting wireguard sturen
10.10.8.0/23 is kleiner & specifieker dus die gaat dan gewoon over onze eigen management vpn gaan.
Thing is gewoon dat als jij ooit iets zou sturen met source 10.10.8.123 dat hij daar nooit iets van terug zal krijgen. Dus zolang jij dat stuk ontwijkt is dit voor ons goed genoeg
[15/5, 12:46] Yentel Hollebeke: API, API-SSL en WWW-SSL poorten staan normaalgezien allemaal open, uit het hoofd 8728, 8729 en 443, allemaal tcp
[15/5, 12:46] Yentel Hollebeke: dit voor packets vanaf source 10.10.100.14, maar ik kijk straks nog even na
[15/5, 12:47] Yentel Hollebeke: zal ook nieuwe passwords genereren voor rupel & cloud7
Note dat de users daar hydra-read zijn en de mobilekit hydra is (in opzicht om daar later write op te zetten)
WG tunnel confirmed (hydraneck side):
ip route get 10.10.4.1 on hydraneck: routes via wg0 with source 10.10.100.14 — correct source IP matching the firewall rule Citymesh appliedhydraneck scan mobile-kit test run:
=== Scanning mobile-kit ===
Routers:
mikrotik-mobile-kit mikrotik unreachable (0 clients)
error: mikrotik dial 10.10.4.1:8729: i/o timeout
Our side is correct — scan targets 10.10.4.1:8729 out wg0 from 10.10.100.14. TCP still timing out, meaning Citymesh firewall rules on mobile-kit are not yet applied (Yentel said he would verify shortly).
Config fix — username mismatch:
Yentel confirmed cloud-seven and rupelmonde use user hydra-read. Config had hydraneck — corrected to hydra-read on both venues.
Remaining open items before first successful scan:
10.10.2.1:PORT) + protocol into hydraneck config10.10.3.1:PORT) + protocol into hydraneck confighydraneck scan cloud-seven greenhydraneck scan rupelmonde greenhydraneck scan mobile-kit greenCitymesh confirmed install on all three venues — WireGuard handshakes live as of 2026-05-15:
Bidirectional tunnels are up. Next step: receive read-only MikroTik API creds from Citymesh and update /root/.hydraneck/config.yaml with WG-side hosts to enable first scan.
Documentation for the operating mode has landed:
docs/runbooks/citymesh-venue.md(commit cd71790). Full procedure:hydraguard venue add --guard citymesh, peer profile export, handover bundle to Citymesh, verification, troubleshooting. Linked from the runbook.md guard-types table.docs/runbooks/mesh-participation.md(commit 491a7e5). One-time enrollment of hydraneck as a HydraGuard air peer, per-venue config pointing at WG addresses, graceful degradation, multi-district notes. Wired into the public docs nav under Operations.partnervscitymeshterminology drift (doc sayspartner, code requirescitymeshtoday).The operational ask to Citymesh changes from 'whitelist 46.225.8.28' to: 'install WG peer profiles on the MikroTik at cloud-seven and rupelmonde'. Procedure to generate those profiles is in the new citymesh-venue.md.