HydraIssues

Symptom

On cederikmini (node-38f14c8b, the shared Apple CI runner and the owner's desktop Mac), Bitwarden and iCloud repeatedly ask to unlock a keychain, and the macOS account password is rejected. Reported 2026-08-31.

Cause

The Apple release workflows create a throwaway signing keychain ci.keychain, make it the default keychain, and put it first in the user search list. They never restore the previous state. On a self-hosted runner that state is permanent, so from the last signed release onward every app that reads or writes a secret hits ci.keychain first. macOS then asks for the ci keychain password, and the account password correctly fails.

Observed state before the fix:

default keychain : /Users/cederik/Library/Keychains/ci.keychain-db
search list      : ci.keychain-db, System.keychain, login.keychain-db
ci.keychain-db   : locked, last written 2026-08-25 (last signed release)

Offending lines: hydraheadipad/.github/workflows/release.yml:95-96 and hydraheadmacos/.github/workflows/release.yml:68.

CORRECTION to the original filing

The first version of this issue said the hardcoded cikey was the export password of the distribution p12 and had to be rotated. That is wrong, and no rotation is needed. Both workflows build the p12 in the same step from the DISTRIBUTION_CERT_PEM and DISTRIBUTION_KEY_PEM secrets with -passout pass:cikey. cikey only ever protected a temporary file inside a single job. The real hygiene defect in that block is smaller and different: the job left /tmp/dist.p12, /tmp/inst.p12 and the plaintext /tmp/cert.pem and /tmp/key.pem on a shared desktop Mac after every release.

Fixed 2026-08-31

• hydraheadipad e6def28 (main), hydraheadmacos be5cc18 (master). Both pushed.
• Added a Restore login keychain step with if: always() to both release workflows: sets login.keychain-db back as the default, resets the user search list to login plus System, deletes ci.keychain, and removes the temporary PEM and p12 files.
• Replaced the literal cikey with a random per-run password from uuidgen, masked with ::add-mask::.
• Both workflows now delete-keychain ci.keychain before creating it, so a crashed previous run cannot poison the next one.
• Documented the shared-runner rule in the Signing section of both runbooks, with the manual repair commands.
• Machine repaired over hydracluster exec with no build in flight. cederikmini now has login.keychain-db as default and as the only user keychain, unlocked, no-timeout. The stale ci.keychain-db file is deleted.

Verified on cederikmini by running the new create/default/unlock/restore sequence end to end against a scratch keychain, then confirming the restore returns the exact expected state.

Remaining

ci.keychain is still a fixed name. cederikmini runs several runner instances (cederikmini, cederikmini-hydraheadflatscreen), so two signing jobs at once would still collide on that one keychain. A per-run name passed between steps through $GITHUB_ENV would close this. Left out of this change to keep the release path low risk; it needs a real signed release to test.

Verification at the next release

After the next signed release of each repo, on cederikmini:

security default-keychain        # must be login.keychain-db
security list-keychains -d user  # must not contain ci.keychain

and confirm Bitwarden and iCloud do not prompt.