HydraIssues

Body selection race condition: two heads can claim the same body simultaneously
open unclassified Project: hydracluster Reporter: claude-code 20 May 2026 23:35

Description

## Problem

When two heads query /api/v1/bodies/eligible at the same time, they receive identical eligible body lists with no reservation step. Both heads then attempt to connect to the same top-ranked body. Sunshine rejects the second connection, but hydracluster is not involved in that rejection -- it only learns about the stream via the next heartbeat.

This creates a real failure mode in 1:1 or low-body-count setups: the rejected head retries the same list, finds the body now streaming (stream_count=1), and has no fallback. The head gets stuck.

stream_count does not protect against this because it is read from the last-reported heartbeat, not updated atomically at connection time. There is a Reserve()/Release() mechanism in pkg/store/store.go:467 but it is only wired up for hydratestnode role reservations, not stream allocation.

## Affected scenario

Any deployment with two or more heads in the same district + owner -- including the bxl1-test setup where ipad-head and ipad-head-2 both target cosmic-pretzel-98.

## Suggested fixes (in order of preference)

### Option 1 -- Venue isolation (zero-code, works now)
Assign each iPad + body pair a unique venue name within the district. The quickroute partition puts same-venue bodies first