# AI Triage Monitor for GitHub Actions

This package continuously polls configured GitHub Actions workflows, detects newly failed jobs, extracts bounded failure evidence, classifies each failure with a local ACP agent, writes redacted reports, and sends grouped diagnostic topics to a configured Feishu topic group.

The monitor is read-only toward GitHub Actions. It does not dispatch, rerun,
cancel, or schedule downstream workflows. The production deployment runs as
the dedicated `ai-triage.service` described below.

## Behavior

- Polls every enabled `[[targets]]` workflow by workflow file, optional branch, completed status, and a bounded creation-time lookback.
- Treats `failure`, `cancelled`, `timed_out`, `action_required`, and `stale` as abnormal. It ignores `success`, `neutral`, and `skipped`.
- Creates a first-start baseline and ignores failures that were already completed. A workflow that was active before startup and fails afterward is processed.
- Uses `(target_id, run_id, run_attempt, job_id)` as the stable event identity. A failed rerun is therefore a new event.
- Persists target baselines, event stages, leases, redacted result payloads, retries, and terminal state in SQLite.
- Retries only the failed stage. A Feishu retry never reruns AI triage or rewrites a valid report.
- Writes each result under `TARGET/run-RUN_ID-attempt-N/job-JOB_ID/`, so later failures cannot overwrite earlier evidence.
- Reads at most the final `limits.max_log_bytes` of each job log in one bounded
  byte-range request. This keeps the latest failure while preserving up to the
  full configured context window before it, including scenario step markers.
- Waits for every discovered job in one workflow run attempt to reach the
  notification boundary, then merges failures that share a stable component or
  problem fingerprint. Unrelated failures become separate topics.
- Sends Feishu Card 2.0 topics with a de-duplicated diagnosis, objective
  request/response evidence, dynamically extracted related resource IDs, failed
  GHA links, and direct links to detected `scenario_testcases/*.yaml` files in
  `tidbcloud/APITestGen`.
- For a product error, first matches the request URL path against the maintained
  `routing/api-team-map.toml`. A hit fixes the Team deterministically; a miss
  keeps the existing AI selection from the current Oncall Team catalog. It then
  reads and mentions the selected Team's live primary oncall. For a case error,
  it labels and mentions the configured `Case Owner`.
- Requires every user-facing AI conclusion, evidence statement, root-cause
  explanation, and recommendation to use Simplified Chinese. English-only AI
  output is rejected and replaced with a conservative Chinese fallback.
- Handles `SIGINT` and `SIGTERM` between poll cycles and restores recoverable leased state after a crash.

## Classification gate

Only an exact configured scenario step, initially `Run scenario cases`, can become `product_error` or `case_error`. Every setup, dependency, cancellation, timeout, summary-only, missing-log, and run-level failure is forced to `infrastructure_error`. A cancelled scenario step containing `signal: interrupt` may use AI to explain the historical failure pattern, but the orchestrator still locks its category to infrastructure and its Card 2.0 header to orange. Invalid or unavailable AI output for an eligible scenario failure produces a conservative `case_error` fallback and does not block reporting.

For eligible scenario failures, the report includes the scenario YAML, case,
operation, order, request method and URL, request body, expected result, actual
status and response, trace ID, dynamically named related resource IDs, evidence
gaps, analysis, recommendations, and owner when the log contains them. Related
metadata is represented as an ID-name-to-values mapping rather than a fixed
field list, so cases can expose IDs such as cluster, resource pool, organization,
backup, restore, import, branch, or changefeed IDs. Organization and tenant ID
spellings are treated as aliases and rendered once under the organization key.

## Requirements

- Python 3.11 or newer.
- A durable local filesystem for the SQLite state database and reports.
- A GitHub token with read access to Actions in the monitored repository.
- OpenCode with `opencode acp` and non-interactive provider authentication.
- A Feishu custom-bot webhook created in the destination topic group.
- A Feishu app ID/secret with user-ID lookup permission when an owner is
  configured by email. The app's availability range must include those owners.
- An Oncall service API key with `oncall:read`, supplied through
  `ONCALL_API_KEY`.

Runtime code uses only the Python standard library.

## Configuration

Use [config.example.toml](config.example.toml) as the inert field reference.
Create `ai_triage/config.toml` only as a local deployment file. It is ignored by
Git because it may contain deployment targets, owner identities, and other
environment-specific information. Likewise, keep the local OpenCode provider
configuration in the ignored `ai_triage/opencode.json`; never commit either
file. Secret values still belong in environment variables, not TOML or JSON.

Important monitor fields:

- `poll_interval_seconds`: delay between completed scan cycles.
- `lookback_seconds`: creation-time window queried from GitHub; make it longer than the longest workflow duration plus expected API delay.
- `max_pages_per_target`: hard pagination bound. Reaching it is an explicit target error, never silent truncation.
- `state_path`: SQLite database path. A relative path is resolved next to the TOML file.
- `lease_seconds`: crash-recovery lease; validation requires it to cover the ACP timeout plus the bounded GitHub/network retry budget.
- `retry_base_seconds`, `retry_max_seconds`, and `max_attempts`: bounded stage retry policy.
- `retention_days`: terminal event retention; validation requires it to cover the full lookback window.
- `max_log_bytes`: maximum bytes requested from the end of the redirected
  plain-text job log for evidence extraction.
- `max_agent_output_chars`: maximum validated assistant-answer size.
- `max_acp_protocol_bytes`: separate bound for the full ACP JSON-RPC stream,
  including OpenCode progress and reasoning updates before the final answer.

`agent.prompt_path` defines the classification contract.
`agent.knowledge_paths` is a bounded list of additional curated Markdown files
that are read into the same redacted prompt. The committed knowledge package
summarizes the scenario E2E framework without feeding full scenario inventories,
generated artifacts, raw logs, or credentials to the agent.
`docs/knowledge/triage-experience-rules.md` is the reviewed historical-rule
catalog. Each rule records evidence guards, exclusions, classification,
responsibility, and the recommended investigation order. Direct request,
response, assertion, and `do action failed` evidence always takes precedence
over a historical rule.

`routing.api_team_map_path` points to the maintained API-path-to-Team TOML
mapping. Patterns match only the URL path: hosts and query strings are ignored.
When multiple patterns match, the most specific pattern wins. Ambiguous or
unmapped paths deliberately fall back to the existing AI routing logic. The
mapping uses stable Team keys from the live Oncall catalog and startup fails
when a configured Team no longer exists. Rules are derived from the current
`scenario_testcases/*.yaml`; `tests/test_api_routing.py` prevents stale rules
that no longer match any testcase path.

Each `[[targets]]` entry defines `id`, `enabled`, `workflow`, optional `branch`, and `allowed_scenario_steps`. The monitor never accepts downstream workflow inputs or credentials.

`github.token_env`, `feishu.webhook_env`, `feishu.app_id_env`,
`feishu.app_secret_env`, and `oncall.api_key_env` are environment-variable
names, never literal credentials. `oncall.base_url` defaults to
`https://oncall.pingcap.net`; Team discovery and current-duty resolution use
read-only endpoints. Configured owner values may be a Feishu email, an
`ou_...` open ID, or a display-only label. Product oncall emails are resolved
to Feishu open IDs on demand and cached; case and infrastructure emails are
resolved at startup.

## Local validation

Run these commands from the Git repository root. The committed example is
disabled and safe for validation:

```bash
PYTHONPATH=ai_triage/src python3.11 -m unittest discover -s ai_triage/tests -v
python3.11 -m compileall -q ai_triage/src ai_triage/tests
PYTHONPATH=ai_triage/src python3.11 -m ai_triage validate \
  --config ai_triage/config.example.toml
PYTHONPATH=ai_triage/src python3.11 -m ai_triage watch \
  --config ai_triage/config.example.toml \
  --dry-run
```

Dry-run validates the prompt and configuration and prints enabled targets, paths, interval, and lookback. It does not read credentials or touch GitHub, SQLite, ACP, reports, or Feishu.

## Running continuously

Create the ignored local files from the repository root, then adjust them for
the deployment:

```bash
cp ai_triage/config.example.toml ai_triage/config.toml
```

Run the watcher under a process manager on a host with durable storage. All
paths below are relative to the Git repository root:

```bash
export GITHUB_TOKEN='...'
export AI_TRIAGE_FEISHU_WEBHOOK='...'
export LARK_APP_ID='...'
export LARK_APP_SECRET='...'
export ONCALL_API_KEY='...'

PYTHONPATH=ai_triage/src python3.11 -m ai_triage watch \
  --config ai_triage/config.toml \
  --state-path ai_triage/ai-triage-state/state.sqlite3 \
  --output-dir ai_triage/ai-triage-results
```

The process manager owns restart and host-level health. Multiple watcher processes may share one SQLite database; leases prevent normal duplicate work. They must also share the same report filesystem.

The repository includes [deploy/ai-triage.service](deploy/ai-triage.service) as
a deployment template. Resolve its installation paths from the repository
layout during deployment; do not copy host-specific checkout paths into
documentation or commit the environment file, `config.toml`, or
`opencode.json`.

Use `--once` for a single read-only scan. A cycle exits `0` when every target and event stage succeeds, `1` when the cycle records isolated target/event errors, and `2` for configuration or startup failure.

## Feishu topic-group delivery

A custom-bot webhook is bound to exactly one destination group. In a
topic-form group, each grouped notification is sent as a new root message and
appears as a topic. Failures in one workflow run attempt share a topic when they
have the same component or the same problem fingerprint; otherwise they are
sent as separate topics.

The notification uses Feishu Card 2.0 with a highlighted diagnosis, a
multi-column summary, Team/current-oncall or configured Case Owner mention,
bounded request/response evidence,
related resource metadata immediately below the testcase link, recommendations,
per-job GitHub Actions links, an emoji title, and an `open_url` button. Partial
validated AI output keeps its available fields and marks missing fields in gray.
Large groups show two detailed failures and at most three additional job links;
empty evidence bullets and empty recommendation sections are omitted. It
deliberately omits event IDs, confidence, internal category values, and grouping
fingerprints.

Webhook delivery is at least once. Confirmed success is never resent and
definite failures are retried. A network timeout after Feishu accepts a
message but before the client receives the response can produce a duplicate.
The stored report and GitHub job link provide the audit trail. This integration
creates root topics; it does not reply inside a pre-existing topic.

## Deployment model

The repository does not provide a GitHub Actions workflow for running the
monitor. Validate and operate it through the CLI commands above. Production
polling belongs under a normal process manager, which provides clear restart
behavior and durable-state ownership without tying the service lifecycle to a
finite GitHub Actions job.

## Security boundaries

- Raw job logs stay in memory, are byte-bounded, and are never persisted or sent to ACP.
- Extracted evidence is redacted before ACP, SQLite, reports, and Feishu.
- Related metadata extraction is bounded and accepts only ID-shaped keys from
  the relevant request URL, JSON bodies, and failure text. Credential IDs,
  masked values, trace/event/job/run/case IDs, and redundant prefixed aliases
  are excluded.
- Redaction covers authorization headers, cookies, tokens, passwords, credential-like fields, DSNs, SAS query fields, secret URL parameters, and email addresses.
- ACP receives no filesystem capabilities, every permission request is rejected, controller credentials are removed from its environment, and timeout cleanup terminates its process group.
- GitHub log redirects use a separate HTTPS request without the GitHub authorization header.
- Notification errors never include the webhook URL.
- Oncall API errors never include the bearer key; only Team catalog and current
  duty GET endpoints are used.

The design, requirements, and execution plan are under [docs/specs/ai-triage-monitor](docs/specs/ai-triage-monitor/requirements.md).