# alert-triage-adapter The far end of alertstore's agent webhook: it receives one alert and turns it into a usher session, then gets out of the way. ``` alertstore ──POST /dispatch──▶ adapter ──POST /api/sessions──▶ usher ▲ │ agent works └──────── agent PATCHes its verdict back ──────────────────┘ ``` It never waits for the agent and never reads its output. The agent reports its own result straight back to alertstore, so this process holds no state and can be restarted mid-flight without losing a run. Nothing here can ack, resolve or assign an alert, and neither can the agent — the report endpoint is the only write either of them has. Neither `/dispatch` nor the report endpoint is authenticated. Both are reachable by anything that can reach them, which is a deliberate choice for the experiment and not a property to rely on: keep them off any address outside the cluster. One usher call does it. `POST /api/sessions` carries `initial_message`, so creating the session and asking the question are the same request, and `auto_approve`, without which a permission prompt would wait for a human who is not there. ## Try it with no deployment and no alertstore Three terminals: ```bash make sink # stands in for alertstore's callback make run # the adapter make e2e # sends one alert shaped like a real dispatch ``` A run that reaches the sink has proven everything except alertstore itself: the dispatch shape, the prompt, whether the agent reports back at all, and whether the verdict it chose is one alertstore accepts. The sink applies the same checks alertstore does and says out loud when a report would have been refused. Against a real alert — pass the `alert` object from `GET /alerts/`: ```bash ./hack/fake-dispatch.sh --alert-file ./alert.json ``` ## Deployment There is no image, CI workflow or manifest for this yet — deliberately, until the experiment it exists for has an answer. Today it is two files copied onto RAP's persistent volume. Build a static binary here and copy it plus the prompt: ```bash make build # → output/bin/alert-triage-adapter (linux/amd64, static) kubectl cp ../../../output/bin/alert-triage-adapter :/home/dev/triage/ kubectl cp prompt.md :/home/dev/triage/ ``` usher listens on `127.0.0.1:3001` inside the container (nginx on `:3000` is the browser surface), so the defaults already point at it and nothing needs to be exposed: ```bash cd /home/dev/triage USHER_PASSWORD=... \ USHER_PUBLIC_URL=https:// \ AGENT=runbook-triage \ ./alert-triage-adapter ``` Building here rather than on the box is not just convenience: RAP installs Go from Debian, which may be older than this module's `go` directive, so `go run` there can refuse outright. (`GOTOOLCHAIN=auto` fetches a newer toolchain if the pod has egress.) When this does get an image, the open question is `prompt.md`. Baking it in costs the edit-in-place loop that makes tuning cheap; a ConfigMap keeps it, at the price of a mount. That is a decision to make once the experiment says how often the prompt actually changes. It lives here, under `resilience-agents-platform/`, because RAP is the runtime it drives: the adapter is useless without a usher to start sessions in, and the two are deployed and reasoned about together. Whether it is baked into the RAP image is a separate question, still open. Today it is not: it is built as a static binary and run by hand off the persistent `$HOME`, because the prompt is the part that needs the most iteration and rebuilding a container with Node, podman and code-server in it to retune a paragraph is the wrong loop. `prompt.md` is re-read on every dispatch, so editing it in place does not even need a restart. Whatever the image ends up doing with the binary, that property is the one worth keeping — see the note on ConfigMaps under Deployment. ## Point alertstore at it Insert one routing row, and set alertstore's callback base URL: ```sql INSERT INTO agent_subscription (id, name, endpoint, filter, enabled, created_by) VALUES ( UUID(), 'runbook-triage', 'http://:8899/dispatch', '{"alertname": ["UnhealthyPod"]}', -- start with one alertname, never {} 1, 'you@pingcap.com' ); ``` `ALERTSTORE_AGENT_CALLBACK_BASE_URL` (set in `tidbcloud/app-delivery`) is alertstore's own address as the agent can reach it. With it empty alertstore does not dispatch at all — deliberately, so a half-configured environment drops nothing silently rather than dead-lettering every alert until somebody notices. An empty filter (`{}`) is refused by alertstore: it would match every owned alert, and until these rows are written through a validating API the commonest way to end up with one is a misspelled field name the JSON decoder drops in silence. ## Configuration Every flag also reads the environment variable named beside it. `-h` prints this list with the defaults in force. | Flag | Env | Default | | |---|---|---|---| | `-addr` | `ADDR` | `:8899` | where alertstore POSTs | | `-usher-url` | `USHER_URL` | `http://127.0.0.1:3001` | usher as this process reaches it | | `-usher-public-url` | `USHER_PUBLIC_URL` | `-usher-url` | usher as a human clicking `session_ref` reaches it | | `-usher-password` | `USHER_PASSWORD` | — | empty means usher runs without one | | `-prompt` | `PROMPT` | `prompt.md` | re-read per dispatch | | `-cwd` | `CWD` | — | working directory; empty defers to the agent profile | | `-agent` / `-backend` / `-model` | … | usher's defaults | | | `-proxy-url` | `PROXY_URL` | `http://127.0.0.1:8990` | datasource proxy, quoted into the prompt | usher authenticates people, not machines: a password and a cookie MAC'd over its hash, with no bearer token to hold. The adapter logs in the way the web UI does and refreshes the cookie when a request comes back bounced. ### Two files, two lifetimes `prompt.md` is rendered per dispatch and carries one alert: the payload, the four steps, the output format, where to report. `workspace-AGENTS.md` is everything true of every alert on that box — which skills are installed, that queries go through the proxy, where the runbooks are checked out, what the agent may not do. The second one is **not loaded from this repository**. It is archived here for review and history; to use it, copy it into the agent's working directory on the RAP box as `AGENTS.md`: ```bash cp workspace-AGENTS.md /home/dev/workspace/cloud-staging/AGENTS.md ``` It is deliberately not named `AGENTS.md` here, so that agents working on this repository do not pick up instructions written for a different machine. The split is one question: would this still hold at turn twenty, after a human has taken the session over and asked about something else? The initial message scrolls away; the workspace file does not. ### Prefer an agent profile over these flags `-cwd`, `-backend` and `-model` all override the usher agent profile named by `-agent`, because usher takes the request's value whenever it is set. So leave them empty and put the settings in the profile, where they sit next to `append_system_prompt` and `auto_approve` and belong to the agent rather than to this process: ```bash curl -X POST http://127.0.0.1:3001/api/agents -H 'Content-Type: application/json' -d '{ "name": "runbook-triage", "cwd": "/home/dev/work/triage", "backend": "claude", "auto_approve": true, "append_system_prompt": "..." }' ``` The working directory matters more than it looks: the agent finds its skills in `/.claude/skills/` and the datasource-proxy address in `/.env`. Point it somewhere without those and the agent can no longer read alertstore back — and the only symptom is a report saying it could not get the current state. ## Failure behaviour The status codes are chosen against alertstore's retry taxonomy and should not be changed casually — there are tests pinning them: - **400** — a payload that will not parse on the eighth attempt either. alertstore treats 4xx as permanent and stops. - **502** — usher is down or refused. alertstore retries with backoff, which is exactly the transient case that exists for. - **500** — the prompt file is unreadable. This deployment is broken rather than the request, and the file may come back, so a retry is right. - **202** — the session started. It says nothing about whether the agent will succeed; that arrives later on the callback, or does not, and a run that never reports stays visible in alertstore as `dispatched`. A dispatch is idempotent per run: redelivering one — because the response was lost, or because starting a session outlasted alertstore's timeout — returns the session already running rather than starting a second one. Without that, two agents investigate the same alert and report against the same run, and since alertstore's report merge takes the last non-empty value per field, whichever finishes second silently overwrites the other's verdict. The dedup table is in memory, so one window remains: if the adapter is restarted between starting a session and answering, the retry that follows starts a second one. Narrow enough to accept for now — closing it means writing a marker per run to the volume.