HydraIssues

Native Head mic relay for kiosk experiences (Mercator Talks)
open feature Project: hydra-experiencenet Reporter: 9 Apr 2026 22:25

Description

## Context

Mercator Talks on a kiosk Mac Mini (Logitech Brio 4K) needs to capture the visitor's voice and send it to the Body's hydravoice so the UE experience can process it. Moonlight/Sunshine protocol is audio-unidirectional (no native mic passthrough). We need a separate RTP path, same as the browser streaming uses.

## Architecture

```
Browser path (existing):
Browser mic -> WebRTC Opus -> hydraneckwebrtc worker -> RTP UDP :47995 -> hydravoice

Native path (new):
Brio mic -> Qt Multimedia -> Opus encode -> RTP UDP :47995 -> hydravoice
```

hydravoice on the Body receives identical Opus RTP on port 47995 regardless of source. No Body-side changes.

## Qt app (HydraExperienceNet) changes

The Qt app owns the mic hardware (TCC permissions, C++ multimedia frameworks).

### 1. Add Qt Multimedia module
`app/app.pro`: `QT += multimedia`

### 2. New MicRelay class
`app/audio/micrelay.h` + `app/audio/micrelay.cpp`:
- QAudioSource: capture from default input (Brio mic), 48kHz stereo float32
- libopus encoder: encode 10ms frames (960 samples)
- QUdpSocket: send RTP packets (PT=111, SSRC=random) to body_ip:47995
- start(bodyIP) / stop() lifecycle

### 3. Local API endpoints
`app/api/localserver.cpp` on localhost:9741:
- POST /api/v1/mic/start {"body_ip": "10.10.100.42"} - starts mic relay
- POST /api/v1/mic/stop - stops mic relay

Body IP is only known after discovery+pairing, so hydraheadflatscreen controls mic relay via these endpoints after the stream is established.

### 4. Build dependency
- `brew install opus` on build machine
- Link against libopus (static for distribution)

## hydraheadflatscreen changes

Minimal orchestration:
- After Body discovered and stream started, if experience has enable_microphone: true:
POST localhost:9741/api/v1/mic/start with body_ip
- On stream end:
POST localhost:9741/api/v1/mic/stop
- EnableMicrophone field already exists in Experience Library responses

## RTP format (must match hydravoice)

- Payload Type: 111
- Codec: Opus, 48000 Hz, 2 channels (stereo)
- Frame duration: 10ms (960 samples)
- Standard 12-byte RTP header

## What stays unchanged

- hydravoice on Body: no changes
- hydraneckwebrtc browser mic: unchanged, parallel path
- VB-Cable on Body: unchanged
- Experience Library: enable_microphone already exists

## Verification

1. mercator-talks with enable_microphone: true in Experience Library
2. Kiosk Mac Mini with Logitech Brio
3. Select Mercator Talks, stream starts
4. Speak into Brio -> audio reaches UE on Body via hydravoice
5. Stream ends -> mic relay stops
6. Experiences without enable_microphone: no mic relay

## Prerequisites

- Issue #92: hydraheadflatscreen Experience Library integration (deployed upstream)
- Experience Library v0.10.3: venue filtering (deployed)
- HydraCluster v2.0.19: allowed_experiences + experience_library_url (deployed)