HydraIssue Runbook

Overview

HydraIssue is an issue reporting microservice for the Hydra ecosystem. It provides a public web form for reporting issues, an API for programmatic access, and an admin dashboard for triage and management.

Infrastructure

• Server: issues.experiencenet.com
• Hetzner: hydraexperiencenet context
• Host: 78.47.174.83 (hydrastreamingmonitor), port 8085
• Config: /root/.hydraissue/config.yaml
• Data: /root/.hydraissue/data/
• Service: systemctl {start|stop|restart|status} hydraissue
• Logs: journalctl -u hydraissue -f

Health Check

curl https://issues.experiencenet.com/api/v1/health

API Endpoints

| Method | Path | Auth | Description | |--------|------|------|-------------| | GET /api/v1/health | none | Health check | | GET /uploads/{id}/{filename} | none | Serve uploaded attachment | | POST /api/v1/issues | Bearer (api) | Create issue | | GET /api/v1/issues | Bearer (api) | List issues (query: status, category, project, active_only) | | GET /api/v1/issues/{id} | Bearer (api) | Get single issue | | PATCH /api/v1/issues/{id} | Bearer (api) | Update issue (status, priority, category, plan, custom_fields) | | DELETE /api/v1/issues/{id} | Bearer (api) | Delete issue | | PUT /api/v1/issues/{id}/plan | Bearer (api) | Update plan text and status | | POST /api/v1/issues/{id}/comments | Bearer (api) | Add comment | | POST /api/v1/issues/{id}/nim-actions | Bearer (api) | Record agent action (name, action) | | GET /api/v1/stats | Bearer (api) | Aggregate statistics | | GET /api/v1/reports/quality | Bearer (api) | Quality review report (query: since, until, format) | | POST /api/v1/webhooks | Bearer (api) | Register a webhook | | GET /api/v1/webhooks | Bearer (api) | List registered webhooks | | DELETE /api/v1/webhooks/{id} | Bearer (api) | Unregister a webhook | | GET /api/v1/runbook | none | This runbook (markdown) |

Admin Endpoints

Session cookie auth (/login with the admin token), not Bearer.

| Method | Path | Description | |--------|------|-------------| | GET /admin | Dashboard | | GET /admin/issues | Issue list | | GET /admin/issues/{id} | Issue detail | | POST /admin/issues/{id}/status | Update status | | POST /admin/issues/{id}/priority | Update priority | | POST /admin/issues/{id}/comment | Add comment | | POST /admin/issues/{id}/attachments | Upload attachments (multipart, field attachments) |

Issue Lifecycle

Status

A single field tracking the full issue lifecycle from creation through implementation.

| Status | Meaning | Set by | |--------|---------|--------| | open | New issue, not yet groomed | Issue creation | | groomed | Triaged and ready for planning | groom-issues skill | | planning | Agent is writing an implementation plan | plan-issues skill | | proposed | Plan submitted for human review | plan-issues skill | | accepted | Plan approved, ready for implementation | Human via UI | | rejected | Plan rejected, parked until a human moves it back to groomed | Human via UI | | implementing | Agent is actively implementing the plan | next-plan skill | | done | Implementation is complete | next-plan skill | | closed | Issue closed manually (won't fix, duplicate, etc.) | Human via UI |

Transition rules: Terminal states (done, closed) cannot regress to earlier lifecycle states. For example, a done issue cannot be set back to groomed or open. Terminal states can only transition to each other (done ↔ closed). The API returns 409 Conflict when an invalid transition is attempted.

The status filter supports comma-separated values: ?status=open,groomed,planning

Stage (shorthand filter)

The stage query parameter provides semantic shortcuts for common agent queries. It resolves server-side to the appropriate status values, so skills don't need to hardcode statuses.

| Stage | Resolves to | Used by | |-------|------------|---------| | groomable | open | groom-issues | | plannable | groomed | plan-issues | | implementable | accepted | next-plan | | active | all except done, closed | dashboards |

Example: GET /api/v1/issues?stage=plannable

Priority

| Priority | Meaning | |----------|---------| | (unset) | Not yet triaged | | low | Nice to have, no urgency | | medium | Should be addressed in normal course | | high | Needs attention soon | | critical | Blocking or production-impacting |

Category

| Category | Meaning | |----------|---------| | bug | Something is broken | | feature | New functionality request | | improvement | Enhancement to existing functionality | | question | Clarification or support request | | unclassified | Default when no category specified |

Custom Fields

Issues support an optional custom_fields JSON object for project-specific metadata. Custom fields are stored as a free-form key-value map — no schema is enforced.

Via API

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title":"Example","description":"...","project":"nimsforest","custom_fields":{"environment":"staging","severity_score":8}}' \
  https://issues.experiencenet.com/api/v1/issues

Custom fields can also be set or replaced via PATCH /api/v1/issues/{id} with the custom_fields key.

Via Web Form

The public report form (/report) includes an optional "Custom fields" section where users can enter a JSON object.

Display

Custom fields render as key-value pairs on both the public issue view and the admin detail page.

Attachments

Files live on disk under ~/.hydraissue/data/uploads/{issue_id}/, and the issue YAML holds the stored filenames. Stored names are prefixed with an index (0_notes.txt, 1_spec.md), so the same filename uploaded twice is kept twice rather than overwritten.

Uploading

| Where | Who | Notes | |-------|-----|-------| | /report | anyone | Optional attachments on the public report form | | POST /admin/issues/{id}/attachments | admin session | Upload form on the admin issue detail page; appends to the existing list |

Limits apply to both paths: 50 MB per file, and executable extensions (.exe, .dll, .bat, .cmd, .ps1, .vbs, .msi, .dmg, .app, .scr, .com) are rejected. Rejected or oversized files are skipped silently, and the rest of the upload still succeeds.

An admin upload updates the issue's updated_at and fires the issue.updated webhook.

There is no delete. To remove an attachment, delete the file under uploads/{issue_id}/ and edit the attachments list in the issue YAML.

How It Works

1. Issues arrive via the public web form (/report) or the API (POST /api/v1/issues)
2. Each issue is stored as an individual YAML file in ~/.hydraissue/data/issues/
3. A lightweight index at ~/.hydraissue/data/issues.yaml enables fast listing
4. Admins triage via the web dashboard (/admin) using session cookie auth
5. Agents interact via the API using bearer token auth
6. Plans can be attached to issues via PUT /api/v1/issues/{id}/plan — the status lifecycle (proposed → accepted → implementing → done) provides human-in-the-loop approval before agents execute

Webhooks

External services can register webhooks to receive issue events in real time. Registrations are dynamic — no config changes or restarts needed.

Register

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/webhooks/issues","events":["issue.created","issue.updated"],"projects":["nimsforest2"]}' \
  https://issues.experiencenet.com/api/v1/webhooks

• url (required): The endpoint to receive POST requests
• events (optional): Filter by event type. Omit to receive all events. Available: issue.created, issue.updated, issue.commented
• projects (optional): Filter by project name. Omit to receive events for all projects

List

curl -H "Authorization: Bearer $TOKEN" https://issues.experiencenet.com/api/v1/webhooks

Unregister

curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  https://issues.experiencenet.com/api/v1/webhooks/wh_1

Payload format

{
  "event": "issue.created",
  "timestamp": "2026-03-04T12:00:00Z",
  "data": { ... full issue object ... }
}

Delivery is fire-and-forget with 3 retries (1s, 2s backoff) and a 5s timeout per attempt.

Nim Actions

Agent actions are tracked via append-only nim_actions on each issue. This prevents duplicate processing and provides an audit trail.

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"nebula","action":"groomed"}' \
  https://issues.experiencenet.com/api/v1/issues/42/nim-actions

Common actions: groomed, planning, planned, implementing, implemented.

Attachments

Users can attach files (images, PDFs, logs, archives, etc.) when submitting issues via the web form.

• Storage: /root/.hydraissue/data/uploads/{issue_id}/{i}_{filename}
• Served at: GET /uploads/{id}/{filename} (public, no auth required)
• Limit: 50 MB per file (files exceeding this show an inline error on the form and are not added)
• Blocked types: .exe, .dll, .bat, .cmd, .ps1, .vbs, .msi, .dmg, .app, .scr, .com
• Filenames are sanitized to [a-zA-Z0-9\-_.] only; path traversal is rejected

To clean up orphaned upload directories:

ls /root/.hydraissue/data/uploads/
# Remove dirs for deleted issues manually if needed

Authentication

Two separate tokens configured in config.yaml:

• admin_token: Used for web dashboard login (session cookie)
• api_token: Used as Bearer token for API access

Auto-Update

HydraIssue auto-updates from releases.experiencenet.com every 6 hours via hydrarelease. Disabled in dev mode (--dev).

To check the current version:

hydraissue version

Troubleshooting

Service won't start

• Check config exists: cat /root/.hydraissue/config.yaml
• Check port conflict: ss -tlnp | grep 8085
• Check logs: journalctl -u hydraissue -f

Issues not showing up

• Web views default to active issues only (open/in-progress). Select "All statuses" in the dropdown to see closed/resolved issues.
• API: use ?active_only=true to filter, default shows all for backward compatibility.
• Verify data directory exists: ls /root/.hydraissue/data/issues/
• Check index file: cat /root/.hydraissue/data/issues.yaml

API returns 401

• Verify Bearer token matches api_token in config
• Admin endpoints require session cookie, not Bearer token

Backup & Recovery

What is backed up

Hetzner automated daily server snapshots are enabled on hydrastreamingmonitor (78.47.174.83), context hydraexperiencenet. Backup window: 02:00–06:00 UTC, 7-day retention.

The snapshot covers the entire server disk, including:

• /root/.hydraissue/data/issues/ — all issue YAML files
• /root/.hydraissue/data/uploads/ — file attachments
• /root/.hydraissue/data/issues.yaml — index
• /root/.hydraissue/config.yaml — service config

Restore procedure

1. In the Hetzner Cloud Console (hydraexperiencenet project), open Servers → hydrastreamingmonitor → Backups.
2. Identify the snapshot to restore from. Note: restoring overwrites the current disk — all changes since the snapshot are lost.
3. Power off the server, restore the snapshot, then power on.
4. Verify the service came back up:

   curl https://issues.experiencenet.com/api/v1/health
   

5. Spot-check recent issues to confirm data integrity:

   curl -H "Authorization: Bearer $TOKEN" https://issues.experiencenet.com/api/v1/issues?active_only=true
   

Manual data backup (without full restore)

If only the data directory needs to be recovered (e.g. accidental deletion), mount the snapshot as a temporary volume or copy files off it via the Hetzner rescue system, then rsync back to /root/.hydraissue/data/.