HydraIssues

Goal

Let one body machine host multiple concurrent streams. Each stream gets its own VDD virtual display and its own Sunshine instance. Native heads (hydraheadflatscreen, hydraheadipad) are the receivers. hydracluster becomes the slot allocation authority. hydraneckwebrtc is not in use and stays untouched.

PoC target: chunky-turnip-23 (node-b961f1c8, RTX A5000 24GB, parked in bxl1-test-2). The A5000 is a professional card with no NVENC session cap.

PoC test setup (decided)

• District: bxl1-test-2
• Body: chunky-turnip-23 (capacity 2)
• Test clients: the MacBook Air (hydraheadflatscreen, macOS) and the iPad (hydraheadipad), both placed in the bxl1-test-2 district
• Experience: Rupelmonde Castle Viewer on BOTH streams

Two consequences of this setup:

1. Same-experience concurrency is in scope, not deferred. hydrabody today dedupes launches by exe name (httpserver.go:226), stops and orphan-scans by exe name, and two UE instances share one GameUserSettings.ini. The PoC needs per-slot process identity (two instances of the same exe, tracked by PID per slot) and a window-placement answer per slot.
2. The iPad path is on the critical path. The moonlight-ios fork needs the sunshine_port glue and a VALID TestFlight build before the PoC run.

Design summary

Slot model. A body gets stream_capacity N (node field in nodes.yaml, default 1). Slot k = one VDD monitor + one Sunshine instance: own config dir, state, certs, apps.json, log, port family base 47989 + 1000*k. Slot 0 is byte-for-byte today's system (stock ports, ensure_primary, audio, interactive input). The VDD change is Monitors.Count=N on the SINGLE Root\MttVDD adapter. Never add a second adapter instance (that is the #428 mouse bug).

Addressing. The slot travels as additive fields, never inside stream_url (both heads strip ports from it today). headStreamConfig gains slot, sunshine_port, sunshine_web_port, mic_port. /api/v1/bodies/eligible gains capacity and slots_free; legacy stream_count becomes "1 if slot 0 is busy" so old heads stay correct. moonlight-qt is launched as stream <ip>:<port> <app> (the fork parses host:port). moonlight-ios gets sunshine_port injected with httpsPort auto-discovery. RTSP and media ports self-derive from serverinfo and RTSP SETUP.

Allocation. hydracluster owns it. Dynamic heads call POST /api/v1/bodies/{id}/slots/claim (head_id, experience, port capability) and get {slot, ports, claim_token, ttl 60s}. A claim converts to a session when the body reports that slot streaming; unclaimed leases expire. Claims must count toward advertised availability (slots_free and legacy stream_count), or the claim-to-stream window re-opens today's race. Occupancy rebuilds from body heartbeats after a cluster restart. Static bindings get a sticky head.HeadSlot.

Pairing. Mechanism unchanged, addressing port-parameterized. Each instance has its own server cert and UUID; hydrabody provisions the same district credentials into every instance web UI. Heads pair against the slot's ports and self-submit the PIN. Known fix needed: the Linux cert writer hardcodes localport/manualport 47989 and its hostname-fallback match can clobber a sibling slot's entry on first pair; match must become (host, port).

Lifecycle. hydraclusterapi grows additive types: BodyStatusRequest.Slots[]{slot, stream_status, streaming_client_uuid}, BodyStatusResponse.TerminateSlots[], BodyConfig.StreamCapacity. hydrabody maps prep-cmd hooks to slots via per-instance callback paths (stream/started?slot=k). The cluster session store rekeys to (bodyID, slot); watchdog, heartbeat reconciliation, and terminate become per-slot. Slot-less legacy stop calls must default to slot 0, never stop-all (today hydrabody /api/v1/stream/stop force-stops everything; the cluster fires it for every iPad stream stop).

v1 scope decisions

• Input: EVERY slot is interactive (decided 2026-08-18; non-interactive slots are not an option). Interactivity rides on native touch passthrough, not mouse: injected touch contacts route by coordinate to the window under them, so N touch streams on N VDD displays can coexist, while mouse streams fight over the one session cursor and focus. Consequences: (a) the moonlight-qt fork already sends LiSendTouchEvent (app/streaming/input/abstouch.cpp:175); (b) the iPad fork sends absolute MOUSE emulation (AbsoluteTouchHandler.m) and its vendored moonlight-common-c lacks LiSendTouchEvent, so it needs a moonlight-common-c update plus a touch-passthrough handler; (c) keyboard and simultaneous mouse stay per-body exclusive (one Windows session, one cursor); if that is ever needed on all slots, the fallback is per-app input injection via a UE plugin.
• Audio: default sink follows slot 0. Watch for audio bleed between slots in the PoC; define a mute policy if it fails the venue bar.
• Mic: one mic-enabled experience per body; hydravoice stays a single 47995 listener.
• Capacity stays 1 fleet-wide; only chunky-turnip-23 gets 2.
• hydraheadwebstream and hydraneckwebrtc are unaudited: keep capacity>1 bodies out of their reach until they learn to claim.
• Fix #137 (server-side clearing of stale head stream blocks) with or before this work; a stale block with slot fields points a head at a busy sibling instance.

Ordered plan

Input first: it is the only constraint with no configuration-level workaround, so retire it before any driver, firewall, or WDAC work. Steps 0a and 0b need no body changes and can run before everything else.

0a. Touch-only experience spike — PASSED 2026-08-19 (iPad, Rupelmonde Castle Viewer): driven entirely by touch on the real head; multi-finger works, pinch-zoom confirmed reaching the UE app. Build history: hydraheadipad v0.2.155 (VALID on TestFlight) implements real native touch passthrough. The fork submodule already vendored a LiSendTouchEvent-capable moonlight-common-c and routed all touches to NativeTouchHandler, but that handler was a mouse-emulation stub; it now sends per-touch LiSendTouchEvent DOWN/MOVE/UP/CANCEL frames with video-area-normalized coordinates (hydra-moonlight-ios e1ea4bb), with automatic absolute-mouse fallback on hosts without LI_FF_PEN_TOUCH_EVENTS. RUN: update the test iPad to v0.2.155, stream Rupelmonde Castle Viewer via the normal head flow, and drive it entirely by touch (orbit, pan, tap; no hover, no host cursor). Pass = the experience is fully driveable by touch. Multi-finger gestures now pass through for the first time; note any UE-side gesture gaps.
0b. Concurrent touch routing spike — PASSED 2026-08-19 on chunky-turnip-23 (run remotely via hydracluster exec plus an interactive scheduled task; harness kept at C:\HydraSpike\spike0b). Setup: desktop extended across the VDD monitor (DISPLAY9, primary) and the DP dongle (DISPLAY1), one logging window per display, two separate injector processes emulating two Sunshine instances. Results:

• With CreateSyntheticPointerDevice + InjectSyntheticPointerInput (the API modern Sunshine uses), two processes injected taps and drags concurrently with ZERO failures. Each window received its full WM_POINTER stream (78 and 70 events) with distinct system-assigned pointer ids and ZERO cross-display misrouting.
• Focus flapped between the two windows on taps, as predicted; delivery was unaffected.
• Caveat found and RESOLVED 2026-08-19: the LEGACY InjectTouchInput API does NOT support concurrent injection from two processes (second gets ERROR_INVALID_PARAMETER). Verified on the deployed C:\Sunshine\sunshine.exe binary: it references CreateSyntheticPointerDevice/InjectSyntheticPointerInput and contains NO InjectTouchInput/InitializeTouchInjection, so the fork already uses the concurrency-safe API. No Sunshine change needed for touch.
• Side findings: the body's desktop was NOT extended before the spike (single active display; extended via DisplaySwitch /extend, left extended), and hydrabody carried a stale "streaming" session with The Rupelmonde Experience running and zero connected clients — more evidence for the per-slot lifecycle work.

1. Hardware spike on chunky-turnip-23 (gates everything, needs physical access, no remote reboot): VDD Monitors.Count=2 on the one adapter; verify both monitors enumerate and mouse input survives (#428 check); copy sunshine.exe to a second path and confirm WDAC allows it; run instance 1 with port=48989; confirm serverinfo reports shifted HttpsPort, pairing works on 48989/48984/48990, RTSP and media ports negotiate; open the slot-1 firewall family; run two concurrent captures and check for DXGI capture loss when one stream starts while the other is live.
2. Same-experience and concurrent-input spike: launch Rupelmonde Castle Viewer twice in the one session, one window per VDD monitor (-WinX/-WinY placement, mind #372 GameUserSettings.ini persistence); confirm both render and both captures are correct. Then drive both with concurrent native touch injection (one client per display): verify each UE instance receives only its own touches, both stay responsive, focus flapping on tap is tolerable, and two Sunshine instances can both call InitializeTouchInjection. This test gates the all-slots-interactive requirement.
3. hydraclusterapi: additive Slots/TerminateSlots/StreamCapacity types; tag before dependents.
4. hydrabody: capacity-driven provisioning of N instances and N VDD monitors, per-slot callbacks and status, per-slot stop by PID, per-slot launch of the same exe, static firewall families, input-disable on slot 1. Capacity-1 path stays byte-identical.
5. hydracluster: StreamCapacity, per-slot session store and watchdog, claim endpoint with lease and policy checks, slot/port fields in headStreamConfig and eligible, per-slot terminate and PIN proxy.
6. hydraheadflatscreen (macOS first for the MacBook Air): thread the port family end to end (claim, pair, PIN, stream ip:port, cert writer port keys, probes, change-detection key).
7. hydraheadipad + moonlight-ios fork: sunshine_port in HydraPairSession/HydraStreamSession, claim call, moonlight-common-c update to a LiSendTouchEvent-capable version, touch-passthrough handler replacing absolute-mouse emulation, VALID TestFlight build plus smoke test.
8. PoC run: MacBook Air on slot 0 and iPad on slot 1, both streaming Rupelmonde Castle Viewer on chunky-turnip-23. Verify independent start and stop, watchdog teardown of one slot while the other keeps streaming, input policy, audio behavior, and monitoring visibility. Measure NVENC, VRAM, and NIC headroom to set stream_capacity policy.

Key risks (from adversarial review)

• VDD count=2 on one adapter is untested in this fleet; a bad VDD state on a body needs a site visit.
• WDAC may refuse the second sunshine.exe path; if the fork pins display_device.state or guards single-instance, a fork patch comes first.
• Shared Moonlight uniqueid plus auto-resume means claims are advisory; an old head that reaches a paired instance can hijack its stream.
• iPad reaches bodies via the mobilekit hydraneck and Citymesh-managed routers; the shifted port family must be verified end to end on that path.
• Two H.264 streams (the iPad forces H.264) at venue bitrates strain the encoder and NIC before the A5000 does; revisit bitrate policy before N>2.
• Monitoring (hydrastreamingmonitor, hydrabodystatus, hydranorthstar, admin UI) renders one stream per body and needs the per-slot reshape to stay truthful.

Full investigation dossiers (two ultracode runs, 16 agents, adversarially verified) are in the session workspace; ask Cederik for details.

Scenario: 3x mercator-talks on one body (added 2026-08-19)

Target: three concurrent mercator-talks sessions (portrait, mic-enabled) on chunky-turnip-23, three iPad heads on touch. Streaming, input, and GPU fit the existing design (capacity 3, three portrait VDD monitors, three Sunshine instances; measure VRAM in the PoC). Per-instance -UserDir isolates the shared Saved/GameUserSettings.ini between same-exe instances.

Audio is the gate, both directions: Mercator reads the DEFAULT recording device (CABLE Output, fed by hydravoice) and plays to the default render device, and Windows per-app routing cannot split three instances of one exe. Resolution:

• Issue #511 (project mercatortalks, for soulmade): add -MicDevice / -AudioOutDevice launch args to Mercator. This gates the scenario.
• Platform side (ours): VB-Cable plus Cable A/B on the body (three virtual mics), hydravoice gains configurable listen port + target cable and runs one instance per slot (per-slot mic_port is already in the design), heads relay mic RTP to the slot's mic port, Sunshine instance k captures instance k's render device via audio_sink.

Scenario: N Rupelmonde Castle Viewers, per-slot audio out (added 2026-08-19, near-term focus)

Rupelmonde is touch-only (input proven by spikes 0a/0b) and has audio out but no mic, so it needs no application change at all. Per-slot audio-out routing, platform-side only:

1. Per-slot exe identity: launch slot k's instance from a per-slot NTFS hardlink/junction path. Windows per-app audio routing keys on the app identity derived from the exe path, so distinct paths give distinct routing entries. Bonus: hydrabody's exe-name-based launch dedupe, stop, and orphan scanning become per-slot naturally, and a per-slot -UserDir isolates Saved/GameUserSettings.ini.
2. Three virtual cables (VB-Cable plus Cable A/B, same install the mic plan needs).
3. Route slot k's exe into cable k with SoundVolumeView /SetAppDefault (already deployed at C:\svv per the hydravoice runbook).
4. Sunshine instance k: audio_sink = cable k's render endpoint (loopback capture of only its own experience). hydrabody's audio picker filters virtual devices when writing audio_sink (audio_windows.go:32-70); the per-slot config path must override that filter.

Spike 1 additions: verify two hardlinked copies of one exe get SEPARATE per-app routing entries (AppId derivation assumption), and that WDAC (hash-based) accepts the hardlinked path.

Sequencing: this scenario has no external dependency; 3x mercator-talks additionally needs #511 (mic device selection in the app).

Mic via per-app capture routing (added 2026-08-19, likely obviates #511)

Windows per-app routing also covers CAPTURE devices (the Input column of App volume and device preferences; same persisted AppId-keyed mechanism, driven by SoundVolumeView /SetAppDefault). With per-slot exe hardlink identities, slot k's exe gets its default INPUT pinned to mic-cable k's Output side, and the app's "default mic" resolves per slot below the WASAPI surface, no app change. If spike 1 verifies this, #511 (Mercator -MicDevice/-AudioOutDevice args) becomes a fallback rather than a requirement.

Spike 1 additions: (a) per-app CAPTURE routing honors hardlinked-path identities like render does; (b) UE's audio capture follows the persisted per-app input override.

Cable inventory: a mic-enabled slot needs TWO cables (app render -> Sunshine loopback; hydravoice -> app mic). 3x mercator = 6 cables, beyond the VB-Cable family (CABLE/A/B = 3); use Virtual Audio Cable (VAC, many named cables) or VB-Cable plus VoiceMeeter VAIOs. Rupelmonde slots need one cable each.

2026-09-05: confirmed in production at sint-niklaas-tourism-office with THREE iPads on one body (cosmic-pretzel-98), 204 UUID mismatch log lines. Full evidence, mechanism and direction in #660.