# AI Triage Monitor Design ## Chosen approach Use a persistent Python watcher with an embedded SQLite state store. It keeps the current standard-library-only implementation, makes restarts recoverable, and avoids the scheduling delay and state handoff problems of chaining long GitHub Actions jobs. GitHub Actions remains the monitored system, not the host of the production polling loop. Two alternatives were considered. A five-minute scheduled scanner is operationally simple but scheduled events can be delayed and durable state would depend on a particular self-hosted runner or another service. A GitHub webhook is lower latency but requires a public endpoint, signature validation, and new service operations. Neither is justified for this polling-focused iteration. ## Components - `config.py` and `models.py`: replace dispatch task fields with monitor target fields and add monitor/state/retry settings. - `github.py`: list completed runs for a workflow with branch, creation-range, and pagination bounds. For a job log, validate the HTTPS redirect, issue an unauthenticated `HEAD` against the signed plain-text object, and request an explicit final byte range bounded by `max_log_bytes`. - `state.py`: own the SQLite schema, migrations, per-target baseline, event claims, stage transitions, retry timestamps, and terminal retention. - `monitor.py`: run one scan cycle or the continuous loop, isolate target failures, coordinate state claims, and handle shutdown. - `orchestrator.py`: triage one already-discovered run/job. It extracts evidence once from the final `max_log_bytes` of the job log, so an actionable failure near the end cannot prematurely discard earlier step context inside the same bounded window. It no longer dispatches or waits. It writes the report before notification and can resume notification without rerunning AI. - `evidence.py`: in addition to the fixed diagnostic fields, discover ID-shaped business metadata dynamically from URL path collections, query parameters, nested JSON bodies, and raw key/value evidence. Canonicalize spelling variants, merge repeated values, remove redundant prefixed aliases, and reject masked, credential, and internal correlation IDs under strict count/size budgets. - `docs/knowledge/triage-experience-rules.md`: maintain reviewed historical patterns with explicit triggers, exclusions, classification, responsibility, and next actions. The file is loaded through the bounded ACP knowledge path. - `report.py`: accept a caller-selected event directory rather than overwriting one global report. - `api_routing.py`: load and validate the maintained API-path-to-Team TOML, match only normalized URL paths, prefer the most specific rule, and return no decision for unmatched or conflicting rules so AI routing remains the safe fallback. - `oncall.py`: discover the current Team catalog, read a selected Team's live primary duty engineers, cache short-lived assignments, and never perform schedule writes. - `feishu.py`: render a bounded Card 2.0 notification with highlighted diagnosis, columns, Team/current-oncall or Case Owner mention, evidence, recommendations, and GitHub job links. - `cli.py`: expose `watch`, `watch --once`, `watch --dry-run`, and `validate`. ## Discovery and bootstrap Each target has an initialization timestamp in SQLite. A scan requests completed runs whose `created_at` falls inside the configured maximum run-age window, paginating newest to oldest within a strict page cap. The watcher filters on `updated_at >= initialized_at`; this ignores completed history but includes a run that was already active when the watcher started and completed later. The lookback must be longer than the longest monitored workflow duration plus expected GitHub scheduling/API delay. If a page cap is exhausted before reaching the lookback boundary, the cycle records an explicit target error instead of silently advancing state. Abnormal runs are expanded into current-attempt jobs. Each abnormal job is inserted with a unique event key. SQLite `INSERT ... ON CONFLICT` makes discovery idempotent. Jobs that GitHub reports as successful, neutral, or skipped are ignored. If an abnormal run exposes no abnormal job, a synthetic run-level event is created so the failure is not lost. ## Event state machine `discovered -> triaging -> report_ready -> grouped notifying -> notified` Recoverable failures move the event to `retry_wait` with the resume stage, attempt count, sanitized error, and next retry timestamp. A lease owner and lease expiry allow another watcher process to recover work after a crash. Terminal state is `notified`; optional `abandoned` is used only after a configured retry limit and still retains its report/error. AI is invoked only during `triaging`. Once `report_ready` is stored, notification retries load the saved redacted payload and never invoke AI again. Notification waits until all sibling jobs in one target/workflow/run/attempt are `report_ready` or terminal, then groups incident records by shared component or shared problem fingerprint. State transactions never wrap GitHub, ACP, filesystem, or Feishu network calls. Job-log collection is single-stage. The GitHub API first supplies a temporary HTTPS redirect to a plain-text object. The client reads only object metadata, computes the explicit final range from its length, and accepts the response only when its status and `Content-Range` prove that the requested bytes were returned. The range contains at most `max_log_bytes` and is the only job-log payload used for evidence extraction before the single AI invocation. The normal AI gate requires an exact allowed scenario step whose conclusion is `failure`. One controlled advisory exception accepts an exact allowed step whose conclusion is `cancelled` and whose extracted primary error contains `signal: interrupt`. AI may use the historical rule catalog to explain the interruption, while the orchestrator overwrites category, owner type, Team, and mention fields with the infrastructure-safe values. This prevents model output from turning a cancellation into a product or case assignment. ## Notification semantics The Feishu custom-bot webhook identifies the configured destination group. In a topic-form group, a new root message is displayed as a topic. At startup, the watcher reads current Teams from the Oncall API and validates every Team key in the maintained path mapping against that catalog. For an eligible product failure, the request URL path first goes through the deterministic mapping. The most specific matching rule fixes the Team; when no unambiguous rule matches, the watcher supplies current Team keys and display names to the AI as a strict product-routing catalog. For the resulting product Team it reads the live primary duty engineer and uses Feishu app credentials to resolve that email to an open ID. Configured case and infrastructure owner emails are resolved at startup. An unknown component or an unresolved product owner falls back to the configured Case Owner, while a known component still attempts live oncall resolution first. The Card 2.0 root message highlights the diagnosis, uses columns for component/scope/Team/owner, includes bounded evidence and recommendations, and contains direct GitHub Actions links. Partial validated AI output retains its available fields and marks missing fields in gray. A large group renders two detailed failures and at most three additional job links while retaining the total failure count; empty evidence bullets and recommendation sections are omitted. Internal event IDs, confidence, category, and grouping fingerprints are retained in reports/state but excluded from the user-facing card. Each failed-task block places the bounded related-resource metadata mapping before potentially large request/response details so the IDs remain visible when the card text budget is tight. Custom webhooks do not provide an idempotency key. The watcher therefore provides at-least-once delivery: confirmed success is never resent, definite failure is retried, and an uncertain timeout may duplicate a message. The stable event ID remains in the stored report for correlation. ## Failure isolation A target scan failure records sanitized target health and does not block other targets. An event failure affects only that event. GitHub rate-limit and transient server responses use bounded retries; poll cycles add jittered backoff after target-wide failures. Missing logs that prevent proving the allowed scenario boundary use the deterministic infrastructure gate. Invalid or unavailable AI for an eligible scenario failure uses a case-owner fallback and cannot block report creation or notification. Feishu failure preserves `report_ready` state and retries notification only. ## Deployment The watcher runs under systemd on the target EC2 host with durable SQLite and report storage, a protected environment file, and non-interactive ACP credentials. The included unit treats `ai_triage/` as the package root; the deployment process resolves that repository-relative layout to host-local paths without recording machine-specific paths in repository documentation. The GitHub integration remains read-only and does not dispatch downstream workflows. Python output is unbuffered so every completed scan cycle is immediately visible in the system journal. The ACP boundary applies separate limits to the complete JSON-RPC stream and the validated assistant answer. OpenCode can emit protocol progress messages well before the answer, so using the answer-size limit for the whole stream would reject otherwise valid classifications. ## Testing strategy Tests use fake GitHub, ACP, clocks, sleeps, and Feishu transports plus temporary SQLite databases. Contract tests cover URL/query construction and response parsing. State tests exercise uniqueness, leases, crash recovery, retry resumption, and retention. Monitor tests prove bootstrap, new failure detection, reruns, per-target isolation, bounded pagination, graceful shutdown, and no repeated AI/notification after restart. Existing redaction, evidence, ACP, and classification tests remain regression coverage.