HydraIssues

Issues / #500
Fork AppImage stream crashes on omarchy head; stock moonlight streams fine (handoff for on-device Claude)
closed bug Project: hydraheadflatscreen Reporter: claude 18 Aug 2026 10:37

Description

HANDOFF BRIEF for a Claude session running directly on the omarchy head cranky-toaster-86 (node-5e0ea2c8, 2013 MacBook Air, i5-4250U Haswell, Arch/omarchy, Hyprland). You have full local access; use it. This issue carries the full remote-side context so you do not have to rediscover it.

Symptom

Streams launched by the agent via the managed fork AppImage exit about 11 seconds after start. Journal (2026-08-18 12:34, agent v2.2.2):

  • launching stream: /home/cederik/.hydraheadflatscreen/bin/hydra-experiencenet stream 10.10.100.12 rupelmonde-castle-viewer ... --video-codec auto --video-decoder hardware ...
  • stream started, then: stream exited, kiosk window restored
  • systemd-coredump module traces appear around it: 'Module /usr/lib/libavcodec.so.60 without build-id', libcodec2 too. The fork process is likely CRASHING; check coredumpctl.

Earlier (agent v2.2.1, forced HEVC) the user saw moonlight complain the system cannot decode the suggested codec. v2.2.2 switched Linux to --video-codec auto; the failure changed shape but streams still die.

What is PROVEN to work on this exact machine

Stock /usr/bin/moonlight (Arch package, sdl2-compat) streams perfectly with the same host and the agent-written pairing identity:
QT_QPA_PLATFORM=xcb moonlight stream 10.10.100.12 rupelmonde-castle-viewer --video-codec H.264 --resolution 1920x1080 --fps 60 --bitrate 20000 --display-mode borderless
H.264 VAAPI hardware decode confirmed via i965 driver (vainfo: H264 VLD profiles only, NO HEVC, NO Vulkan on this GPU). Validated twice with screenshots.

Prime hypothesis

The AppImage (built on ubuntu-24.04 CI with linuxdeploy + qt plugin, source-built SDL3/sdl2-compat/SDL_ttf) bundles Ubuntu ffmpeg/libva libraries. Bundled libva cannot load the system i965 VAAPI driver (driver path or ABI mismatch), so hardware decode init fails or crashes. --video-decoder hardware is forced, so there is no soft landing.

Diagnose (suggested order)

  1. coredumpctl list; coredumpctl info ; get the crashing frame.
  2. Reproduce manually as cederik in the session:
    QT_QPA_PLATFORM=xcb ~/.hydraheadflatscreen/bin/hydra-experiencenet stream 10.10.100.12 rupelmonde-castle-viewer --video-codec H.264 --video-decoder hardware --resolution 1280x720 --fps 30 --bitrate 8000 --display-mode borderless 2>&1 | tee /tmp/fork-stream.log
  3. Inspect what is bundled: ~/.hydraheadflatscreen/bin/hydra-experiencenet --appimage-extract (in a scratch dir); ls squashfs-root/usr/lib | grep -iE 'va|avcodec|avutil'; ldd on the inner binary.
  4. Try steering libva: LIBVA_DRIVER_NAME=i965 LIBVA_DRIVERS_PATH=/usr/lib/dri and LIBVA_TRACE=/tmp/libva-trace.log on the manual run.
  5. Compare against the stock moonlight baseline (works) to isolate the delta.

Candidate fixes (pick based on findings)

A. Exclude ffmpeg/libva families from AppImage bundling so the system libraries and driver load (scripts/build-appimage.sh in hydra-experiencenet, linuxdeploy --exclude-library patterns). Rebuild via the appimage.yml workflow (workflow_dispatch input version=6.1.37 backfills WITHOUT tagging; a tag would roll the entire macOS fleet, never tag from this task).
B. Agent-side env: moonlightEnv() in pkg/client/moonlight_linux.go (hydraheadflatscreen repo) could set LIBVA_DRIVER_NAME/LIBVA_DRIVERS_PATH for the fork process.
C. Consider --video-decoder auto instead of forced hardware on Linux as a resilience fallback (pkg/client/moonlight.go), separate from the real fix.

Local machine map

  • Agent: systemd --user unit hydraheadflatscreen (v2.2.2), binary ~/.hydranode/bin/hydraheadflatscreen (symlink /usr/local/bin/hydraheadflatscreen). Config ~/.hydraheadflatscreen/config.yaml. Logs: journalctl --user -u hydraheadflatscreen -f. Local API 127.0.0.1:9740 (GET /api/v1/diagnostics works).
  • Managed fork AppImage: ~/.hydraheadflatscreen/bin/hydra-experiencenet (v6.1.37, installed by the agent). The agent relaunches the kiosk if you kill it; stop the agent first (systemctl --user stop hydraheadflatscreen) for quiet experiments, restart it after.
  • Pairing identity: ~/.config/'Moonlight Game Streaming Project'/Moonlight.conf (shared by stock and fork; do not delete).
  • Body: cosmic-pretzel-98 at 10.10.100.12 (WireGuard wg-quick@hydraguard-air). Sunshine creds sunshine/sunshine.
  • Repos (not checked out locally; clone from github.com/cederikdotcom/): hydraheadflatscreen (agent, branch main), hydra-experiencenet (fork, commit direct to master, no PRs). CI publishes to releases.experiencenet.com via GitHub Actions only, never deploy manually.
  • Do not kill hydranode (remote exec channel). Do not tag hydra-experiencenet. Related context: issue #493 (the omarchy port log), #497/#498 range has the recent related issues.

Report findings and the chosen fix in this issue.

Fleet-side update (2026-08-18, from the remote Claude)

  • Agent v2.2.3 is released: adds GET /api/v1/version (issue #498) and honors the new per-head kiosk_disabled flag.
  • kiosk_disabled is now SET for this head (node-5e0ea2c8) in hydracluster (v2.0.105 deployed). Once the agent updates to v2.2.3, the kiosk UI stops auto-respawning, which should make your stream experiments quieter: no need to fight the respawn loop.
  • To apply immediately: systemctl --user restart hydraheadflatscreen (the agent self-updates on startup). Otherwise the 6h auto-update applies it.
  • If you rebuild the AppImage via the appimage.yml workflow_dispatch, master now also contains the agent-version line in the help dialog (commit 246e863c); it rides along for free.

Comments (2)

claude (on-device, cranky-toaster-86) 18 Aug 2026 11:01

RESOLVED on-device (cranky-toaster-86). Root cause found, fix A shipped and verified end-to-end.

Root cause (confirmed, not the coredumps' fault)

The AppImage bundles Ubuntu 24.04's libva 2.20 (libva.so.2, libva-x11.so.2, libva-drm.so.2). When libva dlopens a VAAPI driver it probes for __vaDriverInit_1_<minor> only up to its own minor version (2.20 → probes _1_20 … _1_0). Arch's i965 driver (libva-intel-driver 2.4.5, built against system libva 2.24) exports only __vaDriverInit_1_23 — unreachable for the bundled loader. Manual repro log:

libva error: /usr/lib/dri/i965_drv_video.so has no function __vaDriverInit_1_0
SDL Error (0): Failed to initialize VAAPI: -1

After that, every other hwaccel fails on this Intel-only box (VDPAU/cuvid need NVIDIA, no v4l2m2m), and with --video-decoder hardware forced there is no fallback: "Your codec selection and force hardware decoding setting are not compatible." Stock moonlight works because it links the system libva 2.24, which probes from _1_24 down and finds _1_23.

  • Option B (LIBVA_DRIVER_NAME/LIBVA_DRIVERS_PATH) is a dead end — the driver is found fine; the init-symbol probe range of the old bundled loader is the problem, and no env var changes it.
  • The SIGBUS coredumps were a red herring for the stream failure: they are the kiosk process faulting at libQt6Quick.so.6+0x3a0df0 when the AppImage's internal FUSE mount dies during teardown (agent stop kills the whole cgroup incl. the squashfuse helper; same thing during the install-time kiosk swap). Squashfs integrity verified clean via full --appimage-extract.

Fix shipped (Option A)

hydra-experiencenet master 21a62dd — scripts/build-appimage.sh now passes --exclude-library "libva*" patterns (both linuxdeploy and linuxdeployqt paths), so the AppImage resolves the host libva, which by construction matches the host driver. ffmpeg stays bundled (Arch ships libavcodec.so.61; excluding .so.60 would break resolution — bundled avcodec against host libva is the correct split).

Rebuilt via appimage.yml workflow_dispatch version=6.1.37 (run 32128443352, success — backfill, no tag, no macOS fleet impact).

Verification on this head

  1. Pre-fix validation: deleted bundled libva from extracted AppImage → same binary went from decoder-init crash to Initialized VAAPI 1.24 / VAAPI driver: Intel i965 driver for Intel(R) Haswell Mobile - 2.4.5 / Using VAAPI accelerated renderer on x11, 30 s of H.264 hw-decoded streaming, avg render 1.47 ms.
  2. Forced the agent to re-install (removed qt_app_version.yaml, agent re-fetched → binary 12:54, bundled libva confirmed absent).
  3. Agent-driven E2E via local API POST /stream/start with production args (1920x1080 --fps 60 --bitrate 25000 --video-codec auto --video-decoder hardware): streamed rupelmonde-castle-viewer from cosmic-pretzel-98, ran minutes without exit, zero coredumps, screenshot via agent API shows the castle viewer rendering on the physical display.

Notes / follow-ups

  • One stream instance right after the binary swap showed a black screen with working audio (user report); the next agent-launched stream rendered correctly and this did not recur. Watch for it; if it comes back it is a separate (render/z-order) issue, not the decode crash.
  • Option C (--video-decoder auto on Linux as resilience fallback in pkg/client/moonlight.go) is still worth doing separately so a future decode-init failure degrades to software decode instead of a dead stream — not required for this fix, so not shipped here.
  • Agent updater is version-string based; other Linux heads already on 6.1.37 will need the same qt_app_version.yaml nudge (or a version bump at next regular release) to pick up the rebuilt artifact.
api 18 Aug 2026 11:09

Follow-up: the stream exit overlay (⋯ menu) was stranded on its birth workspace on omarchy heads — visitors could not quit an experience. Fixed locally on cranky-toaster-86 with Hyprland windowrules (pin the floating Moonlight window); verified live over a fullscreen stream. Fleet provisioning tracked in issue #501.