HydraIssue Runbook

Overview

HydraIssue is an issue reporting microservice for the Hydra ecosystem. It provides a public web form for reporting issues, an API for programmatic access, and an admin dashboard for triage and management.

Infrastructure

• Server: issues.experiencenet.com
• Hetzner: hydraexperiencenet context
• Host: 78.47.174.83 (hydrastreamingmonitor), port 8085
• Config: /root/.hydraissue/config.yaml
• Data: /root/.hydraissue/data/
• Service: systemctl {start|stop|restart|status} hydraissue
• Logs: journalctl -u hydraissue -f

Health Check

curl https://issues.experiencenet.com/api/v1/health

API Endpoints

| Method | Path | Auth | Description | |--------|------|------|-------------| | GET /api/v1/health | none | Health check | | GET /uploads/{id}/{filename} | none | Serve uploaded attachment | | POST /api/v1/issues | Bearer (api) | Create issue | | GET /api/v1/issues | Bearer (api) | List issues (query: status, category, project, active_only) | | GET /api/v1/issues/{id} | Bearer (api) | Get single issue | | PATCH /api/v1/issues/{id} | Bearer (api) | Update issue (status, priority, category, plan, custom_fields) | | DELETE /api/v1/issues/{id} | Bearer (api) | Delete issue | | PUT /api/v1/issues/{id}/plan | Bearer (api) | Update plan text and status | | POST /api/v1/issues/{id}/comments | Bearer (api) | Add comment | | POST /api/v1/issues/{id}/nim-actions | Bearer (api) | Record agent action (name, action) | | GET /api/v1/stats | Bearer (api) | Aggregate statistics | | GET /api/v1/reports/quality | Bearer (api) | Quality review report (query: since, until, format) | | POST /api/v1/webhooks | Bearer (api) | Register a webhook | | GET /api/v1/webhooks | Bearer (api) | List registered webhooks | | DELETE /api/v1/webhooks/{id} | Bearer (api) | Unregister a webhook | | GET /api/v1/runbook | none | This runbook (markdown) |

Issue Lifecycle

Status

A single field tracking the full issue lifecycle from creation through implementation.

| Status | Meaning | Set by | |--------|---------|--------| | open | New issue, not yet groomed | Issue creation | | groomed | Triaged and ready for planning | groom-issues skill | | planning | Agent is writing an implementation plan | plan-issues skill | | proposed | Plan submitted for human review | plan-issues skill | | accepted | Plan approved, ready for implementation | Human via UI | | rejected | Plan rejected, parked until a human moves it back to groomed | Human via UI | | implementing | Agent is actively implementing the plan | next-plan skill | | done | Implementation is complete | next-plan skill | | closed | Issue closed manually (won't fix, duplicate, etc.) | Human via UI |

Transition rules: Terminal states (done, closed) cannot regress to earlier lifecycle states. For example, a done issue cannot be set back to groomed or open. Terminal states can only transition to each other (done ↔ closed). The API returns 409 Conflict when an invalid transition is attempted.

The status filter supports comma-separated values: ?status=open,groomed,planning

Stage (shorthand filter)

The stage query parameter provides semantic shortcuts for common agent queries. It resolves server-side to the appropriate status values, so skills don't need to hardcode statuses.

| Stage | Resolves to | Used by | |-------|------------|---------| | groomable | open | groom-issues | | plannable | groomed | plan-issues | | implementable | accepted | next-plan | | active | all except done, closed | dashboards |

Example: GET /api/v1/issues?stage=plannable

Priority

| Priority | Meaning | |----------|---------| | (unset) | Not yet triaged | | low | Nice to have, no urgency | | medium | Should be addressed in normal course | | high | Needs attention soon | | critical | Blocking or production-impacting |

Category

| Category | Meaning | |----------|---------| | bug | Something is broken | | feature | New functionality request | | improvement | Enhancement to existing functionality | | question | Clarification or support request | | unclassified | Default when no category specified |

Custom Fields

Issues support an optional custom_fields JSON object for project-specific metadata. Custom fields are stored as a free-form key-value map — no schema is enforced.

Via API

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title":"Example","description":"...","project":"nimsforest","custom_fields":{"environment":"staging","severity_score":8}}' \
  https://issues.experiencenet.com/api/v1/issues

Custom fields can also be set or replaced via PATCH /api/v1/issues/{id} with the custom_fields key.

Via Web Form

The public report form (/report) includes an optional "Custom fields" section where users can enter a JSON object.

Display

Custom fields render as key-value pairs on both the public issue view and the admin detail page.

How It Works

1. Issues arrive via the public web form (/report) or the API (POST /api/v1/issues)
2. Each issue is stored as an individual YAML file in ~/.hydraissue/data/issues/
3. A lightweight index at ~/.hydraissue/data/issues.yaml enables fast listing
4. Admins triage via the web dashboard (/admin) using session cookie auth
5. Agents interact via the API using bearer token auth
6. Plans can be attached to issues via PUT /api/v1/issues/{id}/plan — the status lifecycle (proposed → accepted → implementing → done) provides human-in-the-loop approval before agents execute

Webhooks

External services can register webhooks to receive issue events in real time. Registrations are dynamic — no config changes or restarts needed.

Register

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/webhooks/issues","events":["issue.created","issue.updated"],"projects":["nimsforest2"]}' \
  https://issues.experiencenet.com/api/v1/webhooks

• url (required): The endpoint to receive POST requests
• events (optional): Filter by event type. Omit to receive all events. Available: issue.created, issue.updated, issue.commented
• projects (optional): Filter by project name. Omit to receive events for all projects

List

curl -H "Authorization: Bearer $TOKEN" https://issues.experiencenet.com/api/v1/webhooks

Unregister

curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  https://issues.experiencenet.com/api/v1/webhooks/wh_1

Payload format

{
  "event": "issue.created",
  "timestamp": "2026-03-04T12:00:00Z",
  "data": { ... full issue object ... }
}

Delivery is fire-and-forget with 3 retries (1s, 2s backoff) and a 5s timeout per attempt.

Nim Actions

Agent actions are tracked via append-only nim_actions on each issue. This prevents duplicate processing and provides an audit trail.

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"nebula","action":"groomed"}' \
  https://issues.experiencenet.com/api/v1/issues/42/nim-actions

Common actions: groomed, planning, planned, implementing, implemented.

Attachments

Users can attach files (images, PDFs, logs, archives, etc.) when submitting issues via the web form.

• Storage: /root/.hydraissue/data/uploads/{issue_id}/{i}_{filename}
• Served at: GET /uploads/{id}/{filename} (public, no auth required)
• Limit: 50 MB per file (files exceeding this show an inline error on the form and are not added)
• Blocked types: .exe, .dll, .bat, .cmd, .ps1, .vbs, .msi, .dmg, .app, .scr, .com
• Filenames are sanitized to [a-zA-Z0-9\-_.] only; path traversal is rejected

To clean up orphaned upload directories:

ls /root/.hydraissue/data/uploads/
# Remove dirs for deleted issues manually if needed

Authentication

Two separate tokens configured in config.yaml:

• admin_token: Used for web dashboard login (session cookie)
• api_token: Used as Bearer token for API access

Auto-Update

HydraIssue auto-updates from releases.experiencenet.com every 6 hours via hydrarelease. Disabled in dev mode (--dev).

To check the current version:

hydraissue version

Troubleshooting

Service won't start

• Check config exists: cat /root/.hydraissue/config.yaml
• Check port conflict: ss -tlnp | grep 8085
• Check logs: journalctl -u hydraissue -f

Issues not showing up

• Web views default to active issues only (open/in-progress). Select "All statuses" in the dropdown to see closed/resolved issues.
• API: use ?active_only=true to filter, default shows all for backward compatibility.
• Verify data directory exists: ls /root/.hydraissue/data/issues/
• Check index file: cat /root/.hydraissue/data/issues.yaml

API returns 401

• Verify Bearer token matches api_token in config
• Admin endpoints require session cookie, not Bearer token

Backup & Recovery

What is backed up

Hetzner automated daily server snapshots are enabled on hydrastreamingmonitor (78.47.174.83), context hydraexperiencenet. Backup window: 02:00–06:00 UTC, 7-day retention.

The snapshot covers the entire server disk, including:

• /root/.hydraissue/data/issues/ — all issue YAML files
• /root/.hydraissue/data/uploads/ — file attachments
• /root/.hydraissue/data/issues.yaml — index
• /root/.hydraissue/config.yaml — service config

Restore procedure

1. In the Hetzner Cloud Console (hydraexperiencenet project), open Servers → hydrastreamingmonitor → Backups.
2. Identify the snapshot to restore from. Note: restoring overwrites the current disk — all changes since the snapshot are lost.
3. Power off the server, restore the snapshot, then power on.
4. Verify the service came back up:

   curl https://issues.experiencenet.com/api/v1/health
   

5. Spot-check recent issues to confirm data integrity:

   curl -H "Authorization: Bearer $TOKEN" https://issues.experiencenet.com/api/v1/issues?active_only=true
   

Manual data backup (without full restore)

If only the data directory needs to be recovered (e.g. accidental deletion), mount the snapshot as a temporary volume or copy files off it via the Hetzner rescue system, then rsync back to /root/.hydraissue/data/.