2026-09-10: the Pearl-2 at cloud-seven (11.0.6.76, TSKB03679) pings fine and answers HEAD requests with the normal 401, but closes every GET request with no response. Confirmed from two machines (turbo-pancake-76 and cosmic-pretzel-98), so it affects all clients. Browsers show ERR_EMPTY_RESPONSE. The web UI worked on 2026-09-09, so the device degraded overnight. This is unrelated to the macOS Local Network permission from #657.
Action: graceful power-cycle of the Pearl on site (check for active recording/streaming first). Verify afterwards with GET http://11.0.6.76/ expecting 401.
Diagnosis procedure is in the hydraepiphan runbook (docs/runbooks/epiphan-pearl2-access.md). Venue: cloud-seven.
Follow-up 2026-09-10: after the first graceful reboot the web UI came up with an internal server error (500). A second restart via LONG-press (hard power-off), waiting, then powering on booted it clean. Verified: GET http://11.0.6.76/ returns the normal 401 login and the UI works after login on site. If the 500-after-boot recurs, do not keep rebooting: check free storage on the front screen, then Epiphan support / firmware reinstall. Runbook updated.
ROOT CAUSE FOUND 2026-09-10: the front-menu disk check fails and formatting is refused with: storage can not be formatted in state fsfail. The internal 512 GB SSD filesystem is flagged failed. This explains the whole chain: hung GETs, 500 after boot, repeated recovery by reboot. Next steps (in the runbook): 1) reboot and retry format from the WEB admin (Maintenance), not the front menu; 2) if refused, reinstall/update firmware from the admin panel (has cleared stuck storage states for other Pearl users); 3) if fsfail persists, the SSD is dying: download the diagnostics bundle and open an Epiphan support ticket for SSD replacement, stop rebooting; 4) stopgap: streaming works without the internal disk, and recording can go to a USB drive.
API investigation 2026-09-10 (admin creds provided, driven remotely via cluster exec from turbo-pancake-76): firmware is 4.24.5 (2026-07-09 build, current). REST API healthy: uptime, CPU temp 78C (threshold 90). Maintenance page shows: Storage is not available, permanent logs cannot be stored or downloaded. POST /admin/fsck_run.cgi (web-admin Check now) returns the SAME error: Storage cannot be formatted in state fsfail. So both the disk CHECK and format refuse; the OS cannot bring the volume up at all. VERDICT: internal 512 GB SSD failure. Next: Epiphan support ticket for SSD replacement. Good news: remote support + SSH tunnel to the Epiphan support server is already ENABLED on the unit, so support can inspect it directly. Stopgap: streaming unaffected, USB drive for recordings. Runbook updated with the API endpoints and the verdict.
Forwardable summary issue created as #686 (clean external write-up for Epiphan support, no internal details).
CONTINGENCY PLAN — Cloud Seven Epiphan Pearl-2 (TSKB03679) out of order (updated 2026-09-10)
The internal SSD has failed (state fsfail): recording is down and the disk cannot be checked or formatted. Streaming is NOT affected by the disk (encoder runs fine at 6.2 Mbps), but the YouTube livestream currently fails because the Pearl has no internet egress. New lead: Tom changed the Pearl's default gateway from 0.0.0.0 to 8.8.8.8 yesterday to make it work, which suggests the network setting reverts on reboot.
Workstreams (sub-tickets of this issue):
Immediate options to get a livestream back with audio (no SSD needed):
RESOLVED 2026-09-10: on-site power cycle of the Pearl-2 fixed it. Verified remotely from turbo-pancake-76 after the reboot: GET http://11.0.6.76/ returns 401 with the Pearl-2 login challenge again, for both plain and browser-shaped requests. Recovery procedure added to the hydraepiphan runbook. Closing.