HydraIssues

Issues / #507
Single-app kiosk: video in the Qt scene graph, Linux first [M0-M4 delivered; remaining work -> #534]
closed improvement Project: hydraheadflatscreen Reporter: claude 19 Aug 2026 12:50

Description

Design plan for the TRUE single-app flatscreen kiosk: decoded video rendered inside the Qt Quick scene graph as a QML item, overlay and 3-dot menu as plain in-scene items above it, one window, one process. This is the iPad model. The owner chose this over the cheaper one-process-two-windows variant. This issue supersedes #506. The agent-side API design from #506 (GET /api/v1/stream/params, POST /api/v1/stream/state, the ready state, the pickup timeout, the liveness watchdog, and the per-OS useInProcessKioskStream gate) carries over UNCHANGED. See #506 sections 2.2, 2.3, and 3 for those contracts. Do not redesign them. Linux (omarchy) ships first. macOS stays inert behind the same double runtime gate.

1. What the code does today (verified in hydra-experiencenet master)

  • Session::exec() (app/streaming/session.cpp, ~line 2110) hijacks the Qt main thread: "Hijack this thread to be the SDL main thread. We have to do this because we want to suspend all Qt processing until the stream is over." It creates the SDL window, runs a blocking SDL event loop, and only returns at session end. main.cpp line 640 sets QSG_RENDER_LOOP=basic for exactly this reason. This blocking loop is the single structural obstacle. A QML VideoItem needs the Qt event loop and the scene graph ALIVE during the stream.
  • The decode pipeline is already window-independent up to the last step. FFmpegVideoDecoder runs its own pull-model decoder thread (decoderThreadProc in app/streaming/video/ffmpeg.cpp): LiWaitForNextVideoFrame, avcodec_send_packet/receive_frame, then m_Pacer->submitFrame(frame). Only Pacer and the frontend renderer touch the window.
  • The zero-copy path exists and is proven: VAAPIRenderer (backend) plus EGLRenderer (frontend). EglImageFactory::exportVAImages (app/streaming/video/ffmpeg-renderers/eglimagefactory.cpp line 265) is the exact primitive we need and it is ALREADY window-free: vaSyncSurface, vaExportSurfaceHandle(VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2), then eglCreateImage(EGL_LINUX_DMA_BUF_EXT) per layer. It needs only an EGLDisplay and an AVFrame. It closes the dmabuf fds after import and tracks EGLImage lifetime with an AVBufferRef attached to the frame.
  • VAAPIRenderer handles the Haswell i965 quirks we depend on: m_RequiresExplicitPixelFormat forces VASurfaceAttribPixelFormat NV12 on surfaces so vaExportSurfaceHandle works (vaapi.cpp line 995), and initializeEGL probes composed versus separate layer export and EGL import support for the format and modifier before committing (line 1079). Only openDisplay() needs an SDL window (to fetch the X11 display via SDL_GetWindowWMInfo); it already has a windowless vaGetDisplayDRM branch for KMSDRM.
  • Qt already renders through EGL on the kiosk: main.cpp line 601 sets SDL_HINT_VIDEO_X11_FORCE_EGL=1 and QT_XCB_GL_INTEGRATION=xcb_egl. So under QT_QPA_PLATFORM=xcb (forced by the agent, hydraheadflatscreen pkg/client/moonlight_linux.go moonlightEnv) the Qt scene graph GL context is an EGL context on the same Mesa driver and EGLDisplay class as the SDL stream window uses today. dmabuf import into the scene graph context is the same driver operation that works today in eglvid.cpp.
  • Frame pacing today on the Air: Pacer (pacer/pacer.cpp) has NO vsync source on X11 (only Windows DXGI and Wayland frame callbacks exist; the default case renders immediately). Pacing on the Air is really EGLRenderer's blocking SwapBuffers with swap interval 1 plus Pacer's queue-drop logic. There is no X11 pacer to replicate. The scene graph's own vsync-throttled render loop is a like-for-like replacement.
  • Audio is fully window-independent. SdlAudioRenderer (app/streaming/audio/renderers/sdlaud.cpp) initializes SDL_INIT_AUDIO itself, opens a device, and is driven by the moonlight-common-c audio thread through Session::arDecodeAndPlaySample. Nothing touches a window. Keep it byte for byte.
  • Gamepad without a window is already proven inside this app: SdlGamepadKeyNavigation (app/gui/sdlgamepadkeynavigation.cpp) runs SDL_INIT_GAMECONTROLLER with a QTimer poll inside the Qt event loop, no SDL window, today, on every platform.
  • Input during streams is SdlInputHandler (app/streaming/input/): SDL events to LiSend* calls. mouse.cpp line 201 sends LiSendMousePositionEvent scaled to the window size for absolute mode (the kiosk mode). keyboard.cpp maps SDL scancodes to Windows VK codes and builds the modifier byte. All Limelight input functions are connection-scoped, not window-scoped.
  • moonlight-common-c callbacks that assume the SDL loop: clConnectionTerminated pushes SDL_QUIT; clRumble and friends push SDL_USEREVENTs consumed by Session::exec(); the decoder thread pushes SDL_RENDER_DEVICE_RESET to request decoder recreation; Pacer main-thread mode pushes SDL_CODE_FRAME_READY. Every one of these needs a Qt-signal analog in scene mode.
  • The kiosk UI: KioskView.qml polls the agent (:9740) and shows a "Preparing stream" veil; the stream runs in a separate process today. StreamSegue.qml is the upstream in-process segue (used by the CLI path) and it hides the Qt window on connectionStarted because the video lives in the SDL window. StreamOverlay.qml is a separate always-on-top frameless Window. main.qml onClosing runs Qt.quit() (the Hyprland Super+W fix). LocalServer :9741 serves /probe, /window/hide, /window/show, /overlay/show.
  • Qt version: 6.10.2 (appimage.yml). Build system: qmake (app/app.pro).

2. Architecture decision: Session survives, gains scene mode

Decision: keep Session and add a runtime scene mode. Do NOT build a parallel streaming core.

Reasons: Session owns codec negotiation (SupportedVideoFormatList priority logic), validateLaunch warnings, audio setup and fallback, encryption flags, stats, and the connection lifecycle against moonlight-common-c. That logic is large, battle-tested, and identical for both modes. A parallel core would fork all of it. The parts of Session that are actually SDL-window-bound are localized: the exec() loop, window creation, SdlInputHandler, and callback routing. Those get a branch.

Scene mode shape (new code in new files wherever possible):

  • Session::initialize(qtWindow) stays. The hidden SDL test window used for decoder probing stays in scene mode for now (it is hidden, works under XWayland, and removing it is not on the critical path). SDL_INIT_VIDEO therefore stays initialized. Milestone 5 revisits this for Wayland-native.
  • Session::start() stays (async connection thread, LiStartConnection).
  • Session::exec() branches at the top: in scene mode it does NOT create an SDL window and does NOT loop. It creates the decoder (chooseDecoder with a null window and the scene renderer pair), creates the QuickInputHandler, connects Qt signals, and returns. The Qt event loop keeps running. Session teardown moves to a stop() slot that mirrors the DispatchDeferredCleanup block: destroy input handler, destroy decoder under m_DecoderLock, then DeferredSessionCleanupTask exactly as today (it already handles LiStopConnection and quitApp off-thread).
  • Callback re-routing in scene mode: clConnectionTerminated emits a queued Qt signal instead of SDL_PushEvent(SDL_QUIT). The decoder failure reset (SDL_RENDER_DEVICE_RESET) becomes a queued requestDecoderReset signal handled by a Session slot that replays today's reset block (destroy decoder, recreate, LiRequestIdrFrame, setHdrMode). clSetHdrMode already goes through m_DecoderLock and stays. Rumble, motion, LED, adaptive trigger callbacks are gamepad-only; in scene mode they no-op until gamepad streaming input lands (section 5).
  • Frame delivery: FFmpegVideoDecoder keeps its decoder thread and Pacer object untouched. The scene-mode frontend renderer (QuickSinkRenderer, new file) reports isRenderThreadSupported() false, so Pacer runs in main-thread mode and enqueues to m_RenderQueue. The one contained edit in pacer.cpp: in enqueueFrameForRenderingAndUnlock, when a sink callback is set, call it instead of SDL_PushEvent(SDL_CODE_FRAME_READY). The callback invokes VideoItem::update() through a queued connection. VideoItem calls Pacer::renderOnMainThread() equivalent from the scene graph sync phase, which dequeues and hands the AVFrame to the material. Frame free semantics keep Pacer's deferred-free pattern: the previous frame is freed only after the next one has rendered, and an EGL fence (same pattern as eglvid.cpp m_LastRenderSync) guards buffer reuse.
  • Resolution and mouse capture callbacks: drSetup only records dimensions and stays. There is no SDL mouse capture in scene mode; absolute mouse mode needs none (see section 5). Fullscreen toggling, updateOptimalWindowDisplayMode, and getWindowDimensions are SDL-path-only and are not called in scene mode; the kiosk window is already fullscreen.

Gating: scene mode is entered only from the new kiosk stream page, which is only reachable behind #506 Gate B (Qt.platform.os === "linux" in QML) after the agent reports ready behind Gate A (useInProcessKioskStream, per-OS files, zero darwin diff). The macOS DMG carries the new code as double-gated dead code, same argument and same reviewer checklist as #506 section 3.

3. Frame path on the Air (Haswell, i965, H.264 only, no Vulkan)

Constraint recap: Qt Quick RHI backend must be OpenGL on this hardware. Zero copy is VAAPI dmabuf export, EGLImage, GL texture, QSGTexture. Nothing may require Vulkan.

Preferred path (M2): separate-layer export, plain 2D textures, custom QSGMaterial.

  1. Decode: FFmpeg h264 hwaccel with AV_HWDEVICE_TYPE_VAAPI, extra_hw_frames = PACER_MAX_OUTSTANDING_FRAMES (already set in ffmpeg.cpp line 547). Backend renderer is VAAPIRenderer with one contained upstream edit: openDisplay() gains a windowless branch (params->window == nullptr) that opens the VA display via vaGetDisplayDRM on the render node (the KMSDRM branch already does exactly this; reuse it) or via the X11 display from QNativeInterface::QX11Application. The i965 explicit-pixel-format quirk and driver detection then work unchanged. Direct rendering (vaPutSurface) is reported unsupported in windowless mode so the decoder always pairs it with our frontend.
  2. Export: EglImageFactory::exportVAImages with VA_EXPORT_SURFACE_SEPARATE_LAYERS. For NV12 this yields two layers: DRM_FORMAT_R8 (Y) and DRM_FORMAT_GR88 (CbCr). vaSyncSurface inside exportVAImages is the decode-completion sync (explicit, no reliance on implicit dmabuf fencing for the read side).
  3. Import: eglCreateImage(EGL_LINUX_DMA_BUF_EXT) per layer against the scene graph's EGLDisplay, then glEGLImageTargetTexture2DOES with target GL_TEXTURE_2D. R8 and GR88 single-plane images bind as ORDINARY 2D textures. This is the key simplification: no samplerExternalOES anywhere, so the shader is expressible in Qt 6's qsb shader pipeline.
  4. Scene graph: VideoItem (QQuickItem, new file) with updatePaintNode() producing a QSGGeometryNode with a custom QSGMaterial. The material holds two QSGTextures wrapping the GL texture ids via QNativeInterface::QSGOpenGLTexture::fromNative(). The fragment shader (new .frag compiled with qsb) does the NV12 to RGB conversion using the premultiplied CSC constants from IFFmpegRenderer::getFramePremultipliedCscConstants and the chroma cositing offsets, both already computed per frame format in renderer.h. Limited and full range, BT.601 and BT.709 handled the same way eglvid.cpp does.
  5. Synchronization: after the scene graph frame that consumed frame N completes (QQuickWindow::frameSwapped or an EGL fence created in afterRendering), release frame N-1's AVFrame and destroy its EGLImages. The EglImageContext AVBufferRef mechanism already ties EGLImage lifetime to the AVFrame. Never free the AVFrame whose texture may still be referenced by the in-flight frame.
  6. Pacing: no Pacer vsync source (none exists on X11 today). The basic render loop plus swap interval 1 vsync-throttles rendering. Frame arrival triggers VideoItem::update(); at sync time the item latches the NEWEST queued frame and drops older ones, mirroring Pacer's drop policy and stats (pacerDroppedFrames, totalPacerTimeUs keep reporting).

Fallback A (if separate-layer import fails on i965): composed-layer export (single opaque EGLImage, needs GL_OES_EGL_image_external). External samplers do not fit the qsb pipeline, so this variant draws with raw GL instead: connect to QQuickWindow::beforeRendering and beforeRenderPassRecording, and render the video as an underlay with the exact GLES2 shader sources that eglvid.cpp already uses (egl.vert, egl_nv12.frag, egl_opaque.frag, shipped in the app resources today). QML items composite on top. The VideoItem then only reserves layout space and computes the viewport rectangle. This is the classic Qt "OpenGL underlay" pattern and works with the basic render loop.

Fallback B (always available, correctness backstop and M1 bring-up path): map and upload. av_hwframe_transfer_data to an NV12 CPU frame, glTexSubImage2D into two R8/RG8 textures, same QSGMaterial as the preferred path. Slower (one CPU copy per frame at 1080p60) but valid on any driver. This is the automatic runtime fallback whenever export or import fails, mirroring how the SDL path falls back from EGL to SDL readback today.

Decoder headroom note: whatever the path, the total number of AVFrames held outside the decoder (VideoItem current + previous + latch queue) must stay within PACER_MAX_OUTSTANDING_FRAMES (5), or the decoder stalls. The latch queue is capped at 3 like MAX_QUEUED_FRAMES.

4. On-device probes an implementer runs FIRST (M0, before any code)

Run on the reference Linux head (cranky-toaster-86) via the agent exec channel. All read-only.

  1. Existing proof from today's SDL path: grep the most recent stream session log for "Renderer 'EGL/GLES' with 'VAAPI' backend chosen" and the "Exporting composed layers" or "Exporting separate layers" line, plus "EGLImage pixel format". If today's streams already use the VAAPI backend with the EGL frontend, then vaExportSurfaceHandle, EGL_EXT_image_dma_buf_import, GL_OES_EGL_image, and fence sync are all PROVEN working on this exact driver and display stack, and the extension risk collapses to "same operations inside Qt's EGL context", which is the same Mesa code.
  2. vainfo: confirm i965 driver, H264 profiles (Main, High), and note the driver version.
  3. eglinfo: confirm EGL_EXT_image_dma_buf_import, EGL_EXT_image_dma_buf_import_modifiers (optional, factory tolerates absence), EGL_KHR_image_base, EGL_KHR_fence_sync.
  4. es2_info or glxinfo -B plus grep: GL_OES_EGL_image, GL_OES_EGL_image_external (for fallback A), GL_EXT_texture_rg or GLES3 (R8/RG8 sampling for the preferred path).
  5. QSG_INFO=1 run of the kiosk: record which RHI backend and GL flavor Qt picks (desktop GL versus GLES via forceGles in main.cpp line 583). If Qt lands on desktop GL and GL_OES_EGL_image is missing there, force GLES for the kiosk (QSurfaceFormat OpenGLES, one line behind the kiosk branch in main.cpp; Mesa i965 supports GLES3).
  6. Confirm QT_XCB_GL_INTEGRATION=xcb_egl took effect: the QSG_INFO log or eglGetCurrentDisplay validity inside a trivial beforeRendering hook.

Record all outputs in the issue before M1 starts. If probe 1 shows the SDL path is NOT using VAAPI plus EGL today, stop and investigate that first; the whole plan assumes this hardware does zero-copy today.

5. Input re-plumbing

New QuickInputHandler (new files, app/streaming/input/quick*.{h,cpp}) translating Qt events from the VideoItem to Limelight calls. The kiosk streams with absolute mouse mode (agent params, #506), which simplifies everything: no relative capture, no pointer lock, no warp.

  • Mouse: a MouseArea or direct QQuickItem event handlers on the VideoItem. Position events scale item-local coordinates into the video rectangle and call LiSendMousePositionEvent(x, y, videoW, videoH), the same math as mouse.cpp line 166-201 minus the SDL window query (the item knows its own size and letterbox rectangle). Buttons map 1:1 to LiSendMouseButtonEvent. Wheel: QWheelEvent angleDelta to LiSendHighResScrollEvent (120 units per notch, matching mouse.cpp line 254). Hover events must be enabled so motion without buttons streams (setAcceptHoverEvents true).
  • Keyboard: the VideoItem takes activeFocus while the stream page is current. QKeyEvent to LiSendKeyboardEvent2 needs a keycode map: on xcb and Wayland, QKeyEvent::nativeScanCode() is the evdev code plus 8. Build one static evdev-to-Windows-VK table (new file, adapted from keyboard.cpp's SDL-scancode switch; SDL scancodes are USB HID codes and SDL's own linux keymap table gives evdev to HID, so the composition is mechanical and testable). Modifier byte from QKeyEvent::modifiers() with the same MODIFIER_* mapping as keyboard.cpp line 204-218. Edge cases carried over: auto-repeat events forward as is (host expects repeats), modifier-only presses send their own VK, and a raiseAllKeys() equivalent fires on window deactivate and on stream end (track m_KeysDown like SdlInputHandler does). The special combos: keep Ctrl+Alt+Shift+Q (quit stream) for parity; the kiosk overlay is the primary exit.
  • Touch (future touch heads): Qt delivers QTouchEvent natively to QQuickItems. Phase now: compress single-finger touch to mouse position and button events (Qt does this automatically when the item does not accept touch). Native LiSendTouchEvent passthrough is a later, isolated addition to QuickInputHandler; the plan reserves the seam but does not build it.
  • Gamepad: SDL_INIT_GAMECONTROLLER stays for grid navigation (SdlGamepadKeyNavigation, unchanged, already windowless). Streaming gamepad input in scene mode is deferred: kiosk heads have no gamepads. When needed, a QTimer-driven SDL event pump feeding SdlInputHandler's controller methods is the shape (the polling pattern already exists in SdlGamepadKeyNavigation). Rumble and LED callbacks no-op until then.
  • SDL subsystems that remain in scene mode: TIMER (main.cpp init), AUDIO (audio renderer), GAMECONTROLLER (grid navigation), VIDEO (decoder probe test window only). The SDL stream window and its event loop are gone.

6. Audio

Confirmed windowless (section 1). Keep the SDL audio renderer unchanged. arInit, arDecodeAndPlaySample, arCleanup run on moonlight-common-c's audio thread and never touch the window or the Qt loop. muteOnFocusLoss is SDL-window-driven today; in scene mode bind it to QQuickWindow::activeChanged if we keep the behavior (kiosk is always focused; low priority).

7. Kiosk integration and UX

  • New KioskStreamPage.qml (new file) pushed onto the existing StackView in main.qml. Contents: VideoItem filling the page, the loading veil (reuse StreamSegue's dark backdrop and "Loading experience" label), the in-scene exit overlay: the 3-dot handle Rectangle and the "Exit experience" dropdown moved from StreamOverlay.qml into plain Items with z above the VideoItem. Hover, hit-testing, and animations are native QML. Exit calls session.triggerExitFromMenu() directly (Session already exposes it Q_INVOKABLE).
  • Upstream StreamSegue.qml stays untouched for the CLI and desktop paths. The kiosk page is a sibling, not a modification. StreamOverlay.qml stays for macOS until Phase 2; the Linux kiosk stops loading it.
  • Flow per #506 sections 2.1 to 2.3, unchanged: tile tap posts stream/start; agent runs discovery and native pairing; agent sets ready; KioskView (behind Gate B) fetches stream/params, applies them to StreamingPreferences verbatim, builds CliStartStream::Launcher(host, app, prefs) in process, and pushes KioskStreamPage instead of StreamSegue. The page pushes state transitions to POST /api/v1/stream/state (connectionStarted, sessionFinished, the 15 s re-assert timer). Mic relay lifecycle stays agent-owned, keyed off those pushes.
  • No window hiding: window.visible stays true throughout. The Session overlay-manager exit overlay (SDL-rendered) is disabled in scene mode; Session::showExitOverlay() becomes a no-op there since the QML overlay owns the job. The performance overlay (OverlayDebug) is re-homed: the stats text Session writes via OverlayManager is exposed to QML through a property (stringifyVideoStats already produces the string) and shown as a QML Text item toggled the same way.
  • Overlay hit-testing code in Session (isPointInExitOverlay, isPointInExitMenu) and the SDL_ttf overlay rendering are unused in scene mode. Leave them compiled (macOS still uses them); nothing to delete in this phase.
  • LocalServer :9741 on Linux: /window/hide, /window/show, /overlay/show become vestigial (the agent never calls them on the in-process path). Keep them serving 200 no-ops for compatibility; /probe stays live everywhere (discovery uses it). No code removal until macOS converges.
  • The quitting veil: same-process, same-window now. Popping the page after sessionFinished is atomic in the scene graph; a simple opacity transition on the page replaces the macOS Space choreography. No timers guarding compositor races on Linux.
  • Super+W (Hyprland closes the window): main.qml onClosing runs Qt.quit(). Mid-stream this now kills the single process; the agent watchdog (#506) sees the process gone within 2 s, cleans up state, and the main tick relaunches the kiosk. Acceptance: either graceful (preferred: onClosing during an active stream first calls session stop, then quits) or the watchdog path; a hung black screen or a leftover host app session is a FAIL. The --quit-app-after preference means the host app quits through DeferredSessionCleanupTask on the graceful path only; the watchdog path relies on the body-side idle timeout as today with a killed subprocess. Same tradeoff as #506 section 6, unchanged.
  • KioskView "Preparing stream" veil: stays for finding_body and pairing states; the handoff to the page happens at ready.

8. Upstream divergence containment (every touched file)

The fork tracks moonlight-qt. New behavior goes in new files; upstream files get minimal, branch-guarded edits.

New files (no upstream conflict surface):

  • app/streaming/video/quick/videoitem.h/.cpp (QQuickItem, material, latch queue)
  • app/streaming/video/quick/quicksinkrenderer.h/.cpp (IFFmpegRenderer frontend handing frames to the item)
  • app/streaming/video/quick/nv12material.vert/.frag plus qsb outputs
  • app/streaming/input/quickinput.h/.cpp and evdev-VK table
  • app/gui/KioskStreamPage.qml

Touched upstream files, with the nature of the edit:

  • app/streaming/session.h/.cpp: scene-mode flag, exec() early branch, stop() slot, callback routing signals. The largest edit; every hunk behind the scene-mode flag. This file is also the highest-churn file upstream; expect merge conflicts here at every upstream sync and budget for them (section 11, risk R2).
  • app/streaming/video/ffmpeg.cpp: frontend selection when params->window == nullptr picks QuickSinkRenderer; guarded.
  • app/streaming/video/ffmpeg-renderers/pacer/pacer.cpp/.h: sink callback instead of SDL_CODE_FRAME_READY when set; a few lines.
  • app/streaming/video/ffmpeg-renderers/vaapi.cpp/.h: windowless openDisplay branch; isDirectRenderingSupported false when windowless; small.
  • app/gui/main.qml: onClosing guard for graceful stop during scene-mode streams; a few lines.
  • app/gui/KioskView.qml: the ready branch, params fetch, launcher creation, page push, state pushes; all behind Gate B (this is kiosk-only QML already, low upstream risk).
  • app/main.cpp: register VideoItem QML type; optional GLES force in the kiosk branch.
  • app/app.pro: file list.

Not touched: eglimagefactory (consumed as is), eglvid, drm, plvk, sdlvid, audio, all other renderers, StreamSegue.qml, StreamOverlay.qml, moonlight-common-c.

9. Milestones, acceptance criteria, estimates

M0. Probes and baseline. 0.5 person-days.
Run section 4 probes on cranky-toaster-86; capture the SDL path's renderer log lines and a perf-overlay baseline (decode ms, queue ms, render ms, rendered FPS at 1080p60 H.264) plus process CPU (pidstat) and GPU (intel_gpu_top) during a 5-minute stream. This baseline is the M2 comparison target. Record everything in this issue.

M1. Proof: decoded frames inside a QQuickItem, copy path allowed. 4 to 5 person-days.
Scope: Session scene mode (exec branch, stop slot, terminated-signal routing), QuickSinkRenderer, VideoItem with the map-and-upload path (fallback B), KioskStreamPage reachable behind both gates, hardcoded or agent-supplied params. Accept: tile tap on the Air produces a playable stream rendered inside the Qt window; grid returns on exit via a temporary button; no SDL stream window exists (verify with xdotool or wmctrl: one window only); audio plays; app survives three consecutive stream cycles without restart.

M2. Zero copy, performance parity. 4 person-days.
Scope: windowless VAAPI backend init, separate-layer dmabuf export and EGLImage import, QSGMaterial NV12 shader, fence-based frame release, latch-queue pacing, fallback A implemented behind a runtime switch, automatic fallback B on import failure. Accept: log line proves dmabuf path active; no av_hwframe_transfer_data calls during steady state; perf overlay decode plus render times within 20 percent of the M0 SDL baseline and rendered FPS equal at 60; process CPU within 15 percent of baseline over a 5-minute stream; 30-minute stream with zero decoder resets.

M3. Input complete. 3 person-days.
Scope: QuickInputHandler mouse (absolute, hover, wheel), keyboard with the evdev-VK table, focus handling, raise-all-keys on deactivate and stream end, Ctrl+Alt+Shift+Q parity. Accept: full desktop interaction on the streamed experience (typing including shifted symbols and AltGr where the venue layout needs it, right-click, drag, scroll); a typing test streams every printable key correctly; holding a key repeats; no stuck modifiers after alt-tabbing the host app.

M4. Kiosk UX and agent flow. 3 person-days Qt side, plus the agent work carried over from #506 (about 3 person-days, its section 8 estimate for the Go side, unchanged).
Scope: in-scene overlay and 3-dot menu, exit flow through session.triggerExitFromMenu, state pushes and re-assert timer, agent params applied verbatim (orientation contract layer 1 stays agent-owned), Super+W behavior, quitting transition, perf overlay re-homed to QML. Accept: full visitor cycle on the Air with the agent driving; GET stream/params returns 1080x1920 for mercator-talks and the stream renders portrait with hydrabody layers 2 and 3 unchanged; Super+W mid-stream ends with grid back within one agent tick and no orphan host session; agent restart mid-stream re-learns streaming within 15 s; curl of stream/status walks idle, finding_body, pairing, ready, streaming, idle.

M5. Hardening, soak, testbook, Wayland experiment. 3 person-days.
Scope: kill -9 crash recovery drill (watchdog, body cache invalidation, relaunch), 24-hour soak with a visitor-cycle script, memory watch (EGLImage and AVFrame leak check across 500 cycles), testbook amendments (below), then the Wayland-native experiment: run the kiosk with QT_QPA_PLATFORM=wayland (no SDL stream window exists anymore; the decoder probe window and gamepad polling are the only SDL video users left), record what breaks, file findings as a separate issue. Wayland is an experiment gate, not an acceptance gate. Accept: soak passes with zero stuck screens and stable RSS; testbook merged; Wayland findings filed.

Testbook amendments (docs/testbooks/omarchy-head-e2e.md):

  • Section 4 (Streaming): replace the subprocess assertions with: agent log shows the in-process handoff, pgrep for a stream subprocess finds nothing, codec and renderer evidence (h264, VAAPI, dmabuf import line) read from the kiosk app log; add the portrait step (mercator-talks, params 1080x1920, portrait render, layers 2 and 3 untouched).
  • Section 5 (Kiosk lifecycle): add Super+W during stream, kill -9 during stream (watchdog within 2 s, relaunch within one tick, next tap re-runs discovery), exit-overlay cycle with agent status confirmed via curl, and the agent-restart re-assert check. Existing kiosk kill and kiosk-mode toggle steps rerun unchanged.
  • New section (State reporting): the full status walk plus the 15 s re-assert, as specified in #506 section 6.
  • Section 7 (Self-update chain): verify the same-version backfill plus forced refetch flow, used for every milestone rollout.

Total: 17.5 to 18.5 person-days Qt side, plus about 3 agent person-days carried from #506.

10. Risk register

R1. EGL and VAAPI capability gaps on Haswell i965. Likelihood low, impact high. The M0 probes retire this before any code: if today's SDL stream on the Air already runs VAAPI backend with EGL frontend (probe 1), the identical Mesa operations are proven. Residual risk is the import running inside Qt's context (different EGLConfig or GL flavor): mitigated by probe 5 and 6, by forcing GLES if needed, and by fallback A then B. Hard requirement set an implementer must see: EGL_EXT_image_dma_buf_import, EGL_KHR_image_base, GL_OES_EGL_image, and vaExportSurfaceHandle success; the modifiers extension and GL_OES_EGL_image_external are optional (degrade to no-modifier import and to fallback ordering).

R2. Session restructure regressions and upstream merge burden. Likelihood certain (merge burden), impact medium. session.cpp is upstream's highest-churn file. Mitigations: every scene-mode hunk behind one flag; new logic in new files; the SDL path stays byte-equivalent and is regression-tested on the desktop flow (start a normal stream from PcView) before every release; document in the repo a merge note listing the touched hunks. Accept the cost: this is the price of the true single-app model and the owner chose it knowingly.

R3. Qt 6 RHI shader constraints (no samplerExternalOES in qsb). Likelihood retired by design, impact medium. The preferred path avoids external samplers entirely via separate-layer R8/GR88 2D textures. If i965 refuses separate-layer import (probe or M2 evidence), fallback A (raw-GL underlay reusing the shipped eglvid GLES shaders) is immune to the qsb constraint.

R4. Latency or pacing regression versus the SDL renderer. Likelihood medium, impact medium. There is no X11 pacer today, so the bar is the blocking-swap SDL behavior. Measurement: perf overlay fields (totalPacerTimeUs becomes queue-latch time, totalRenderTimeUs, decode) against the M0 baseline, plus a 240 fps phone photon-to-photon measurement (tap on kiosk, photodiode-style frame count to on-screen reaction) on both builds. Mitigations: latch-newest drop policy, swap interval 1, cap latch queue at 3, and the basic render loop already forced. If the scene graph adds a frame of latency versus blocking swap, evaluate frameSwapped-driven release timing before accepting.

R5. Decoder reset and error paths in scene mode. Likelihood medium, impact medium. The SDL_RENDER_DEVICE_RESET replay path is rebuilt on signals; bugs here show up as permanent black video after a mid-stream failure. Mitigation: M2 acceptance includes a forced reset drill (kill the decode via VAAPI env fault injection or a synthetic requestDecoderReset) proving recovery, and the agent watchdog remains the backstop for a wedged kiosk.

R6. Input parity gaps (VK map, modifiers, layouts). Likelihood medium, impact low to medium. Mitigation: table-driven mapping with a unit test comparing against keyboard.cpp's switch for every SDL scancode both paths can express; M3's typing acceptance on the Air; venue keyboards are known hardware.

R7. XWayland single-window behavior. Likelihood low, impact low. The old two-window and SDL_RecreateWindow flicker classes disappear; the remaining risk is Hyprland fullscreen state on the one window during page transitions, covered by M4 visitor-cycle acceptance.

11. Rollout and rollback

Identical mechanics to #506 section 7, applied per milestone:

  1. Merge to hydra-experiencenet master. This alone ships nothing (DMG changes only at the next tag; appimage.yml auto-runs only on tags or workflow edits).
  2. Run appimage.yml workflow_dispatch with the CURRENT released version (same-version backfill). Zero macOS impact.
  3. On the Air: force the updater refetch (remove the managed binary at ~/.hydraheadflatscreen/bin/hydra-experiencenet and let QtAppUpdater reinstall, or the updater force path).
  4. The Air is kiosk_disabled between test windows; re-enable via POST /api/v1/nodes//kiosk-mode {"disabled":false} for each window, disable after.
  5. Agent changes (M4): normal hydrahead agent release. It auto-rolls to macOS and Windows agents, which is why Gate A lives in per-OS files with a zero darwin and windows diff, checkable at a glance in the release diff (reviewer checklist in #506 section 3 applies verbatim).
  6. Rollback per milestone: reinstall the previous AppImage from the release server at the managed path and hold the updater until a fixed backfill exists. Agent: re-tag previous or updater pin. Per-head operations, no fleet action.

macOS: remains untouched. The DMG that eventually rides a tag carries double-gated dead code only, behavior-identical by the #506 checklist, with the runtime proof step on the Visit Flanders test Mac mini before any tag. Flipping macOS to the in-scene model is a SEPARATE Phase decision with its own issue, its own soak, and its own point of no return at tag time. Nothing in this plan schedules it.

12. Relationship to #506

Carried over by reference, unchanged: agent state machine with ready (2.1), GET /api/v1/stream/params contract and StreamParams refactor (2.2), POST /api/v1/stream/state push model, re-assert timer, pickup timeout, liveness watchdog (2.3), subprocess fallback retention (2.4), double-gate mechanics and reviewer checklist (3), crash model and heartbeat kiosk-streaming value (4), rollout mechanics (7), agent effort (8).

Superseded and replaced: the two-window overlay model (#506 section 2.1 note), StreamSegue as the kiosk stream host, the retained SDL stream window, and the statement that a literal in-scene overlay is out of scope. This plan is that in-scene design.

Review (2026-08-19, fleet-side Claude, code-verified)

Code anchors checked and confirmed: main.cpp 601-602 puts SDL and Qt on the same Mesa EGL family under xcb; QSG_RENDER_LOOP=basic at main.cpp 640; EglImageFactory::exportVAImages and exportDRMImages are display-parameterized with format and modifier probing, a ready library surface; the pacer has no X11 vsync source, so the latency bar is blocking swap as stated. The R8 plus GR88 separate-layer path avoiding samplerExternalOES is the correct move for the qsb pipeline and the fallback ladder is real. Approved with the following additions.

FINDING 1 (add to M3 scope and acceptance): local cursor hiding. The SDL path hides the local cursor during streams; the host renders its own. A plain QQuickItem with hover events leaves the local cursor visible, giving a double cursor. Fix is one line (Qt.BlankCursor on the VideoItem while streaming, restored when the exit overlay is hovered or the page pops), but the M3 acceptance must name it: exactly one cursor visible during a stream.

FINDING 2 (add to section 7 and M4 or M5 acceptance): idle inhibition during in-process streams. Today keep-awake rides SDL_DisableScreenSaver on the stream window, which is gone in scene mode. The effective inhibitor on omarchy is the compositor rule idle_inhibit fullscreen on app-id com.moonlight_stream.Moonlight, which still matches the kiosk window and it is fullscreen, so coverage likely holds, but this must be stated and tested: a 10 minute untouched stream must not blank the display (DPMS-off already caused the grim hang incident; do not rediscover it here).

NIT: the nativeScanCode evdev-plus-8 rule is xcb-specific. Correct for the shipping path; the VK-table unit test should assert per platform so the M5 Wayland experiment does not silently inherit a wrong offset.

Estimate opinion: totals are honest for the happy path; the concentration of uncertainty is M2 on the legacy i965 driver, and the plan already absorbs it with pre-built fallbacks and the M0 probe gate. No structural changes requested.

M0 results (2026-08-19, cranky-toaster-86, gate PASSED with one amendment)

  • Probe 2: i965 2.4.5, H264 Main and High VLD. Mesa 26.1.7.
  • Probe 3: EGL_KHR_image_base, EGL_KHR_fence_sync, EGL_EXT_image_dma_buf_import_modifiers all present (modifiers implies the base import extension).
  • Probe 4: OpenGL ES 3.2 with GL_EXT_texture_rg and GL_OES_EGL_image_external.
  • Probe 1 (live stream): decoder used FFmpeg VAAPI; the driver exports DRM PRIME with COMPOSED layers only, NV12 with I915 Y-tiled modifier (0100000000000002). vaExportSurfaceHandle and dmabuf import are proven working on this exact stack.
  • Probes 5 and 6: Qt QRhi lands on desktop OpenGL 4.6 Compatibility via xcb_egl (context creation confirmed in the log). glxinfo shows 4 GL_OES_EGL_image family extensions on desktop GL.

AMENDMENT to section 3: i965 has no separate-layer export. The preferred two-plane QSGMaterial path stays alive by importing the R8 and GR88 planes from the COMPOSED dmabuf (one fd, per-plane offsets, modifiers extension present) instead of VA_EXPORT_SURFACE_SEPARATE_LAYERS. Runtime order on this driver: composed-fd two-plane import, then fallback A (external-sampler underlay, exactly today's proven eglvid operations, likely with the kiosk forced to GLES), then fallback B (map and upload). M1 uses fallback B and is unaffected. The force-GLES one-liner from section 4 probe 5 moves into M1 scope as a prepared option since Qt currently picks desktop GL.

M1 PASSED (2026-08-19, cranky-toaster-86)

Acceptance evidence: video rendered inside the Qt scene graph via the QuickSinkBridge copy path (owner witnessed the stream live and exited via the temporary button); three streams in one app process without restart (three sink-renderer activations in one log); exactly one visible OS window during streaming; zero stream subprocesses across all cycles; exit returns to the grid cleanly. Input is absent by M1 design (M3 scope). Audio not yet verified on this milestone (carry to M3/M4 check).

Six commits shipped during acceptance, all warm-process bugs the cold CLI never hits, plus one predicted risk:

  1. 0826b687 the M1 implementation (compiled first try).
  2. 49190f6b ComputerSeeker misses already-online computers (fires found immediately on warm state).
  3. a56ebb5e seeker warm path grants one poll round before releasing the polling ref (launcher depends on an in-flight round to deliver the app list).
  4. 97acdde2 CliStartStream checks the app list on entering seek-app state (event-only checking times out on an already-complete list).
  5. 55472f71 scene mode forces software decode for M1 (VAAPIRenderer fails hard on a null window; the integrator predicted this; M2 windowless VAAPI lifts it).
  6. 7c591f56 the unknown-decoder frontend factory is scene-gated (it bypassed createFrontendRenderer and paired software decoders with SdlRenderer).

Known cosmetics for M2: Pacer logs "Failed to get current display: invalid displayID" with a null window (harmless, refresh-rate default); audio thread priority warning under the user session. Next: M2 zero-copy dmabuf import per section 3 with the M0 composed-layer amendment.

M2 and M3 first hands-on (2026-08-19, owner on the Air)

Owner-validated on the combined M2+M3 build: touch interaction WORKS (mouse and trackpad arriving at the host as touch), the stream is smooth (zero-copy build; formal perf capture pending), the cursor hid as designed. Two findings:

  1. Cursor policy inverted (fixed, 8adadc31): with everything-as-touch the host renders no pointer, so blanking the local cursor left trackpad users pointing blind. The local cursor now always stays visible; review finding 1 applied only to the retired mouse-forwarding model.
  2. Precise taps on experience hotspots (the yellow ball markers in Rupelmonde) miss while drags work. Root cause on the BODY, not the head: cosmic-pretzel (tvl-body-one) carries THREE Virtual Display Driver instances (ROOT DISPLAY 0000/0001/0002), the documented issue #428 duplicate-VDD condition that breaks absolute input injection. Drags survive (relative), absolute taps land offset. Remedy per #428 when the owner clears it (may be deliberate #504 multi-stream state): disable duplicate VDD instances, delete display_device.state, restart Sunshine; or run M3 acceptance against a single-VDD body.

Remaining M2 acceptance items: zero-copy log-line and steady-state transfer check, CPU sampling (pidstat now installed), 30 minute reset-free stream. Remaining M3 acceptance: precise-tap verification after the body-side VDD cleanup, typing test.

Session close 2026-08-20: M1 through M3 delivered and owner-validated

Final state: touch interaction, smooth zero-copy stream, and the always-visible cursor all confirmed hands-on by the owner on cranky-toaster-86 streaming from cosmic-pretzel. The cosmic triple-VDD condition (#428) was cleaned (two duplicate instances removed, Sunshine restarted, single ROOT DISPLAY 0000 remains, no display_device.state existed). Kiosk mode re-disabled on the Air (workstation).

Residual: some clicking imprecision remains that the owner does not attribute to hydraheadflatscreen; candidates are experience-side hit logic or remaining body-side injection behavior. Park until M4 testing against agent-selected bodies; if it reproduces on a second body, open a dedicated issue with coordinates evidence (tap a known target, compare against the videoRect normalization).

Deployment state: master (2ac8aadc) carries M1+M2+M3 plus the cursor policy, all Linux-gated; the Air runs the backfilled v6.1.37 AppImage from that master. The macOS fleet DMG remains prior-tag; the gated code rides the next tag as dead code. CI hardening added along the way: apt fail-fast and a 25 minute job ceiling after three GitHub-hosted infra stalls in one day; moving this workflow onto own Linux builders (hydralinuxpipeline, #508) is the standing recommendation.

Also note: this delivers issue #111 (window consolidation) for Linux.

Remaining: M4 (agent stream params, in-scene 3-dot overlay replacing the temporary Exit button, body selection from the head's district), M5 (hardening, soak, perf capture with pidstat, Wayland experiment), macOS Phase 2 decision.