# AI Triage Monitor Requirements

## Goal

Replace the dispatch-and-wait controller with a persistent monitor that polls
configured GitHub Actions workflows, triages newly observed failed jobs, and
sends grouped diagnostic topics to a configured Feishu topic group.

## Functional requirements

1. Run as a Python 3.11+ CLI in either continuous mode or a single-cycle mode suitable for local validation.
2. Poll every enabled workflow target at a configurable interval without dispatching, rerunning, cancelling, or otherwise mutating GitHub Actions.
3. Query completed workflow runs through the GitHub REST API with bounded pagination and a configurable creation-time lookback.
4. Treat `failure`, `cancelled`, `timed_out`, `action_required`, and `stale` as abnormal conclusions. Ignore `success`, `neutral`, and `skipped`.
5. On first startup, establish a baseline that ignores already-completed historical failures. A run that is still active at startup and becomes abnormal later must be processed.
6. Process each failed job once per workflow run attempt. A failed rerun is a new event even when it reuses the workflow run ID.
7. Persist discovery, triage, report, notification, retry, and terminal state in SQLite so a process restart resumes work without rescanning from scratch.
8. Reuse the existing deterministic failure gate, bounded job-log collection, evidence extraction, AI triage validation, owner routing, redaction, and safe AI fallback.
   Preserve validated fields from a non-empty partial AI object, record its
   missing fields, and render those gaps in gray. Invalid or unavailable AI for
   an eligible scenario failure defaults to `case_error`; only failures outside
   the scenario execution boundary or confirmed GitHub Actions runner/platform
   failures are `infrastructure_error`.
   Fetch one configurable final byte range of at most `max_log_bytes` from the
   job log and use that single bounded payload for evidence extraction and the
   one AI invocation. Do not use a smaller initial tail or a complete-log
   fallback.
   Dynamically extract related business-resource IDs from the relevant request
   URL, JSON request/response bodies, and failure text into a bounded
   ID-name-to-values mapping. Do not depend on a fixed resource-ID inventory.
   Exclude credential, masked, trace, event, job, run, case, and execution IDs.
   Load a reviewed historical triage-rule file into every eligible AI request.
   A cancelled allowed scenario step containing `signal: interrupt` may use AI
   for a user-facing explanation, but its final category remains
   `infrastructure_error`, its product/case routing remains disabled, and its
   Feishu header remains orange.
9. Write reports under a unique workflow/run-attempt directory so later failures never overwrite earlier evidence.
10. Within one workflow run attempt, merge failures into one Feishu root topic
    when they share the same stable component or the same stable problem
    fingerprint. Send unrelated failures as separate root topics. Each Card 2.0
    message includes highlighted diagnosis, multi-column metadata, useful
    evidence, a prioritized related-resource metadata section, recommendations,
    GitHub job links, and an `@owner` mention when resolvable. It excludes event
    IDs, confidence, internal category values, and grouping fingerprints. A
    grouped message shows at most two detailed failures and three additional
    failed-job links, while preserving the total failure scope. Empty evidence
    bullets and empty recommendation sections are omitted.
11. For `product_error`, first route by the maintained API-path-to-Team mapping.
    A matching rule deterministically overrides AI Team selection; an unmatched
    or ambiguous path falls back to AI selection constrained to the current Team
    catalog returned by the read-only Oncall API. Resolve the selected Team's
    live primary oncall and mention those users in Feishu. For `case_error`,
    label and mention the configured Case Owner. If the component is unknown or
    neither live oncall nor a configured product owner can be resolved, route to
    the configured Case Owner instead of the infrastructure owner.
12. Retry recoverable discovery, triage, and notification failures with bounded exponential backoff. A failed notification must not erase the report or force a second AI triage.
13. Handle `SIGINT` and `SIGTERM` by finishing the current state transaction, stopping before the next poll, and returning a deterministic exit code.
14. Provide a dry-run mode that validates configuration and shows monitored workflows without GitHub, AI, state, report, or Feishu side effects.

## State and delivery semantics

- Event identity is `(target_id, run_id, run_attempt, job_id)`.
- SQLite uniqueness prevents duplicate processing by normal restarts or overlapping watcher processes that share the same state database.
- Notification delivery is at least once. A timeout after Feishu accepts a
  request but before the watcher receives the response can produce a duplicate;
  the stored report and GitHub job links provide correlation without exposing
  the internal event ID in the user-facing card.
- A notification batch is eligible only after every discovered sibling event in
  the same target/workflow/run/attempt has reached notification-ready or a
  terminal state.
- State retention is configurable and applies only to terminal rows older than the retention window.

## Reliability and safety requirements

- Use only Python standard-library runtime dependencies.
- Use GitHub Actions read permission only. Remove workflow-dispatch permissions, inputs, and code paths.
- Bound the run lookback, pages per target, job-log bytes, evidence characters,
  validated AI output, the full ACP protocol stream, HTTP response sizes, and
  retry delays. The ACP stream limit must be independent from the final answer
  limit because agent progress messages can be substantially larger.
- A ranged job-log response must prove the requested `Content-Range`. Missing
  length metadata, unsupported ranges, or a server returning the full file for
  a partial request must fail the bounded log-read stage rather than silently
  analyze the wrong byte window.
- Do not persist raw logs, resolved secrets, GitHub tokens, Feishu webhook URLs, app secrets, tenant tokens, or ACP credentials.
- Preserve the infrastructure classification gate: only configured scenario step failures may become product or case errors.
- Bound related metadata field count, values per field, per-value size, and
  total character budget before AI, persistence, or notification.
- Historical rules must define positive evidence guards and exclusions. Direct
  request, response, assertion, or `do action failed` evidence takes precedence
  over a later cancellation marker.
- A GitHub, Oncall, or Feishu outage must leave retryable state and must not corrupt completed reports.

## Out of scope

- GitHub webhooks or any inbound public service.
- Dispatching or scheduling downstream test workflows.
- Exactly-once delivery from a Feishu custom-bot webhook.
- Replying inside a pre-existing Feishu thread.
- A web UI or monitoring dashboard.

## Acceptance criteria

- A fresh state database ignores historical completed failures and later detects a newly failed run.
- Restarting against the same database does not repeat AI triage or a successful notification.
- A new failed attempt for an existing run is triaged and notified.
- Multiple configured workflows are scanned independently; failure of one target does not block healthy targets.
- Reports are unique and contain redacted evidence, classification, recommendations, owner, run/job links, and the stable event ID.
- Same-component or same-problem failures in one run produce one notification;
  unrelated failures produce separate notifications.
- User-facing cards contain all displayed failed-job URLs and omit the internal
  event ID, confidence, raw category, and problem fingerprint.
- Partial AI cards preserve validated fields, mark missing fields in gray, and
  never replace the available diagnosis with the generic unavailable fallback.
- Large grouped cards show two detailed failures and at most three additional
  job links, preserve the total failure count, and contain no empty bullet or
  recommendation list.
- API Team mapping rules match paths currently present in
  `scenario_testcases/*.yaml`; mapped paths choose the configured Team and
  unmapped paths preserve AI-based routing.
- A cancelled `Run scenario cases` step whose log contains `signal: interrupt`
  is explained as a sibling-failure interruption, remains an infrastructure
  warning, and produces an orange card without product Team or Case Owner
  routing.
- Evidence extraction uses one request for the final `max_log_bytes` of the job
  log. It does not stop at a smaller actionable tail or issue a separate
  complete-log fallback, and it invokes AI at most once.
- Arbitrary ID-shaped resource keys such as `resourcePoolId`, `restoreId`,
  `changefeedId`, or snake-case equivalents are preserved as related metadata;
  multiple values are de-duplicated, organization/tenant aliases are merged,
  sensitive/internal IDs are excluded, and the Feishu card displays the ID
  list directly below the testcase link before large response bodies.
- Unit tests cover configuration, pagination and filtering, bootstrap behavior, state transitions, deduplication, reruns, retries, graceful shutdown, report paths, and end-to-end polling with fakes.
- The full unit suite, compile check, packaging check, workflow lint, config validation, dry run, and secret scan pass without live external calls.