HydraIssues

HydraIssue Runbook

Overview

HydraIssue is an issue reporting microservice for the Hydra ecosystem. It provides a public web form for reporting issues, an API for programmatic access, and an admin dashboard for triage and management.

Infrastructure

Health Check

curl https://issues.experiencenet.com/api/v1/health

API Endpoints

| Method | Path | Auth | Description | |--------|------|------|-------------| | GET /api/v1/health | none | Health check | | GET /uploads/{id}/{filename} | none | Serve uploaded attachment | | POST /api/v1/issues | Bearer (api) | Create issue | | GET /api/v1/issues | Bearer (api) | List issues (query: status, category, project, active_only) | | GET /api/v1/issues/{id} | Bearer (api) | Get single issue | | PATCH /api/v1/issues/{id} | Bearer (api) | Update issue (status, priority, category, plan, parent, custom_fields) | | DELETE /api/v1/issues/{id} | Bearer (api) | Delete issue | | PUT /api/v1/issues/{id}/plan | Bearer (api) | Update plan text and status | | POST /api/v1/issues/{id}/comments | Bearer (api) | Add comment | | POST /api/v1/issues/{id}/nim-actions | Bearer (api) | Record agent action (name, action) | | GET /api/v1/stats | Bearer (api) | Aggregate statistics | | GET /api/v1/reports/quality | Bearer (api) | Quality review report (query: since, until, format) | | POST /api/v1/webhooks | Bearer (api) | Register a webhook | | GET /api/v1/webhooks | Bearer (api) | List registered webhooks | | DELETE /api/v1/webhooks/{id} | Bearer (api) | Unregister a webhook | | GET /api/v1/runbook | none | This runbook (markdown) |

Admin Endpoints

Session cookie auth (/login with the admin token), not Bearer.

| Method | Path | Description | |--------|------|-------------| | GET /admin | Dashboard | | GET /admin/issues | Issue list | | GET /admin/issues/{id} | Issue detail | | POST /admin/issues/{id}/status | Update status | | POST /admin/issues/{id}/priority | Update priority | | POST /admin/issues/{id}/parent | Set or clear the parent issue (form field parent, blank clears) | | POST /admin/issues/{id}/comment | Add comment | | POST /admin/issues/{id}/attachments | Upload attachments (multipart, field attachments) |

Issue Lifecycle

Status

A single field tracking the full issue lifecycle from creation through implementation.

| Status | Meaning | Set by | |--------|---------|--------| | open | New issue, not yet groomed | Issue creation | | groomed | Triaged and ready for planning | groom-issues skill | | planning | Agent is writing an implementation plan | plan-issues skill | | proposed | Plan submitted for human review | plan-issues skill | | accepted | Plan approved, ready for implementation | Human via UI | | rejected | Plan rejected, parked until a human moves it back to groomed | Human via UI | | implementing | Agent is actively implementing the plan | next-plan skill | | done | Implementation is complete | next-plan skill | | closed | Issue closed manually (won't fix, duplicate, etc.) | Human via UI |

Transition rules: Terminal states (done, closed) cannot regress to earlier lifecycle states. For example, a done issue cannot be set back to groomed or open. Terminal states can only transition to each other (done ↔ closed). The API returns 409 Conflict when an invalid transition is attempted.

The status filter supports comma-separated values: ?status=open,groomed,planning

Stage (shorthand filter)

The stage query parameter provides semantic shortcuts for common agent queries. It resolves server-side to the appropriate status values, so skills don't need to hardcode statuses.

| Stage | Resolves to | Used by | |-------|------------|---------| | groomable | open | groom-issues | | plannable | groomed | plan-issues | | implementable | accepted | next-plan | | active | all except done, closed | dashboards |

Example: GET /api/v1/issues?stage=plannable

Priority

| Priority | Meaning | |----------|---------| | (unset) | Not yet triaged | | low | Nice to have, no urgency | | medium | Should be addressed in normal course | | high | Needs attention soon | | critical | Blocking or production-impacting |

Category

| Category | Meaning | |----------|---------| | bug | Something is broken | | feature | New functionality request | | improvement | Enhancement to existing functionality | | question | Clarification or support request | | unclassified | Default when no category specified |

Issue Text Rendering

Descriptions, plans and comments are rendered as GitHub-flavoured markdown (since v1.12.0). Headings, lists, tables, task lists, code spans, fenced blocks, blockquotes and strikethrough all work. Bare URLs are linkified.

Issue references

#123 becomes a link to issue 123. Write references bare, no markdown link syntax needed:

Blocked on #554, superseded by #116.

The link target follows the page you are on: a reference on /issues/{id} links to the public view, the same reference on /admin/issues/{id} links to the admin view, so an operator clicking through a master tracker keeps the status and priority controls.

Not treated as references:

| Input | Why | |-------|-----| | `#554` or a fenced block | Inline parsers do not run inside code | | ## Purpose | Consumed by the heading parser first | | #12ab34 | A trailing letter disqualifies the run | | #000 | Id 0 is rejected | | More than six digits | Keeps hex colours in prose from linking |

Issue titles are not rendered as markdown, so a #123 in a title stays plain text.

Line breaks

Hard wraps are on: a single newline renders as a line break. This keeps the several hundred plain-text issues written before v1.12.0 readable instead of collapsing them into run-on paragraphs.

Raw HTML is stripped

POST /report is public and unauthenticated, so issue text is untrusted. Raw HTML in a description is not passed through to the page. Use markdown, not HTML.

Parent Issues

An issue can reference one parent issue. This turns the parent into a master issue with a sub-issue list.

Custom Fields

Issues support an optional custom_fields JSON object for project-specific metadata. Custom fields are stored as a free-form key-value map — no schema is enforced.

Via API

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title":"Example","description":"...","project":"nimsforest","custom_fields":{"environment":"staging","severity_score":8}}' \
  https://issues.experiencenet.com/api/v1/issues

Custom fields can also be set or replaced via PATCH /api/v1/issues/{id} with the custom_fields key.

Via Web Form

The public report form (/report) includes an optional "Custom fields" section where users can enter a JSON object.

Display

Custom fields render as key-value pairs on both the public issue view and the admin detail page.

Attachments

Files live on disk under ~/.hydraissue/data/uploads/{issue_id}/, and the issue YAML holds the stored filenames. Stored names are prefixed with an index (0_notes.txt, 1_spec.md), so the same filename uploaded twice is kept twice rather than overwritten.

Uploading

| Where | Who | Notes | |-------|-----|-------| | /report | anyone | Optional attachments on the public report form | | POST /admin/issues/{id}/attachments | admin session | Upload form on the admin issue detail page; appends to the existing list |

Limits apply to both paths: 50 MB per file, and executable extensions (.exe, .dll, .bat, .cmd, .ps1, .vbs, .msi, .dmg, .app, .scr, .com) are rejected. Rejected or oversized files are skipped silently, and the rest of the upload still succeeds.

An admin upload updates the issue's updated_at and fires the issue.updated webhook.

There is no delete. To remove an attachment, delete the file under uploads/{issue_id}/ and edit the attachments list in the issue YAML.

How It Works

  1. Issues arrive via the public web form (/report) or the API (POST /api/v1/issues)
  2. Each issue is stored as an individual YAML file in ~/.hydraissue/data/issues/
  3. A lightweight index at ~/.hydraissue/data/issues.yaml enables fast listing
  4. Admins triage via the web dashboard (/admin) using session cookie auth
  5. Agents interact via the API using bearer token auth
  6. Plans can be attached to issues via PUT /api/v1/issues/{id}/plan — the status lifecycle (proposed → accepted → implementing → done) provides human-in-the-loop approval before agents execute

Webhooks

External services can register webhooks to receive issue events in real time. Registrations are dynamic — no config changes or restarts needed.

Register

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/webhooks/issues","events":["issue.created","issue.updated"],"projects":["nimsforest2"]}' \
  https://issues.experiencenet.com/api/v1/webhooks

List

curl -H "Authorization: Bearer $TOKEN" https://issues.experiencenet.com/api/v1/webhooks

Unregister

curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  https://issues.experiencenet.com/api/v1/webhooks/wh_1

Payload format

{
  "event": "issue.created",
  "timestamp": "2026-03-04T12:00:00Z",
  "data": { ... full issue object ... }
}

Delivery is fire-and-forget with 3 retries (1s, 2s backoff) and a 5s timeout per attempt.

Nim Actions

Agent actions are tracked via append-only nim_actions on each issue. This prevents duplicate processing and provides an audit trail.

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"nebula","action":"groomed"}' \
  https://issues.experiencenet.com/api/v1/issues/42/nim-actions

Common actions: groomed, planning, planned, implementing, implemented.

Attachments

Users can attach files (images, PDFs, logs, archives, etc.) when submitting issues via the web form.

To clean up orphaned upload directories:

ls /root/.hydraissue/data/uploads/
# Remove dirs for deleted issues manually if needed

Authentication

Two separate tokens configured in config.yaml:

Auto-Update

HydraIssue auto-updates from releases.experiencenet.com every 6 hours via hydrarelease. Disabled in dev mode (--dev).

To check the current version:

hydraissue version

Troubleshooting

Service won't start

Issues not showing up

API returns 401

Backup & Recovery

What is backed up

Hetzner automated daily server snapshots are enabled on hydrastreamingmonitor (78.47.174.83), context hydraexperiencenet. Backup window: 02:00–06:00 UTC, 7-day retention.

The snapshot covers the entire server disk, including:

Restore procedure

  1. In the Hetzner Cloud Console (hydraexperiencenet project), open Servers → hydrastreamingmonitor → Backups.
  2. Identify the snapshot to restore from. Note: restoring overwrites the current disk — all changes since the snapshot are lost.
  3. Power off the server, restore the snapshot, then power on.
  4. Verify the service came back up: 
    curl https://issues.experiencenet.com/api/v1/health
  5. Spot-check recent issues to confirm data integrity: 
    curl -H "Authorization: Bearer $TOKEN" https://issues.experiencenet.com/api/v1/issues?active_only=true

Manual data backup (without full restore)

If only the data directory needs to be recovered (e.g. accidental deletion), mount the snapshot as a temporary volume or copy files off it via the Hetzner rescue system, then rsync back to /root/.hydraissue/data/.