# Agent Evidence File Channel Design and Plan ## Summary Make files the only diagnosis evidence channel for agents. `collect_evidence.py` should use stdout only for execution status and paths, while Python writes the agent-readable summary, final diagnosis report, and categorized evidence files. The core goal is to avoid two competing evidence sources: - stdout text summary - JSON files on disk After this change, the agent reads one file: ```text /evidence_summary_for_agent.json ``` All categorized evidence artifacts are written under: ```text /evidence/ ``` ## Current State `collect_evidence.py` currently has a mixed model: - stdout prints a human-readable diagnosis summary via `format_summary(...)`. - stderr prints status/progress, including `Evidence files saved to /`. - `summary/evidence_summary.json` contains compact/core evidence. - `metrics/evidence_metrics.json` still contains metric chart payloads. - `logs/evidence_logs.json` still contains raw log entries. - `queries/evidence_queries.json` contains slow query, TopSQL, and decoded plan evidence. - `diagnosis_report.json` is still expected to be written by the agent, then hydrated by backend code. This creates ambiguity: the agent can reason from stdout, from summary files, or from a hydrated report. Those paths can drift. ## Target Design ### Agent Contract stdout is not an evidence channel. It is only a command protocol: ```json { "status": "ok", "run_dir": "", "agent_summary_path": "/evidence_summary_for_agent.json", "diagnosis_report_path": "/diagnosis_report.json", "errors": [] } ``` On failure, stdout should still be parseable JSON: ```json { "status": "error", "run_dir": "", "agent_summary_path": "", "diagnosis_report_path": "", "error": "" } ``` stderr remains for progress, warnings, and non-structured diagnostics. ### File Layout New diagnosis run layout: ```text / evidence_summary_for_agent.json diagnosis_report.json evidence/ metrics.json workload.json resource_control.json queries.json logs.json tikv_thread_cpu.json links.json gate_checks.json collection_errors.json ``` The agent reads only: ```text /evidence_summary_for_agent.json ``` The agent must not read category files during normal diagnosis unless explicitly asked to inspect a specific evidence category. ### Agent Summary Schema `evidence_summary_for_agent.json` contains everything required by `tidb-clinic-diagnosis/SKILL.md`: ```json { "metadata": { "cluster_id": "", "time_range": { "start": "", "end": "" }, "collected_at": "", "run_dir": "" }, "metrics": [], "workload": [], "resource_control": {}, "tikv_thread_cpu": {}, "queries": { "slow_queries": [], "topsql": [], "decoded_plan": {} }, "logs": { "log_counts": [], "warn_count": 0, "loki_url": "" }, "gate_checks": { "infra": "", "hotspot": "" }, "links": {}, "errors": [] } ``` ### Diagnosis Report Ownership `diagnosis_report.json` is written by Python, not by the agent. The Python-generated report should reuse the same core evidence object and add report-oriented fields when available: - `type: "diagnosis_report"` - `metadata` - `evidence` - `resource_control` - `gate_checks` - `links` - optional analysis placeholders or script-derived verdicts The agent can still produce natural-language conclusions in chat, but the file artifact is controlled by the Python script so schema and file placement remain stable. ## Evidence Classification ### Diagnostic Data These belong in `evidence_summary_for_agent.json` and categorized files under `evidence/`: - compact metrics: name, instance, peak, mean, quota, utilization, unit, severity, promql, link - workload: statement count paired with affected rows by SQL type - resource control: RU usage, request p90, available RU - slow queries - TopSQL - decoded plan and full SQL - stats info, backoff, indexes used, process keys, total keys, timing fields - log occurrence counts - gate checks and links - collection errors ### Detail Data These should not be written as default artifacts for agent diagnosis: - metric raw `values` - metric raw label/meta dumps beyond the compact identity fields - `chart_data.series` - full log rows - raw Loki entries If a future debugging mode needs raw data, it should be behind an explicit flag such as `--debug-raw-output`, and its output must not be referenced by the normal agent workflow. ## Metrics Query Strategy Metric collection may continue to use Prometheus range queries internally, because peak/mean/min/severity often require a time window. The important rule is: - query range data in memory - aggregate in Python - write only compact diagnostic rows - do not write raw values or chart payloads This avoids complex PromQL rewrites while still preventing raw metric data from entering files read by agents. ## Implementation Plan ### 1. Build a Single Evidence Object Refactor `collect_evidence.py` so one Python object is the source of truth: ```text agent_summary = build_agent_summary(data, infra_verdict, hotspot_verdict, ...) ``` Use this object for: - stdout path response metadata - `/evidence_summary_for_agent.json` - `/diagnosis_report.json` - categorized files under `/evidence/` Do not separately assemble stdout text and file JSON from different code paths. ### 2. Replace Raw Sidecar Writes Stop writing these default files: ```text metrics/evidence_metrics.json logs/evidence_logs.json metrics/tikv-thread-cpu.json summary/evidence_summary.json queries/evidence_queries.json ``` Write new categorized diagnostic files: ```text evidence/metrics.json evidence/workload.json evidence/resource_control.json evidence/queries.json evidence/logs.json evidence/tikv_thread_cpu.json evidence/links.json evidence/gate_checks.json evidence/collection_errors.json ``` `evidence/queries.json` should include: - `slow_queries` - `topsql` - `decoded_plan` `evidence/logs.json` should include only: - `log_counts` - `warn_count` - `loki_url` ### 3. Change stdout Behavior Remove normal `print(format_summary(...))` from `main()`. stdout should print only the command result JSON: ```json { "status": "ok", "run_dir": "...", "agent_summary_path": ".../evidence_summary_for_agent.json", "diagnosis_report_path": ".../diagnosis_report.json", "errors": [] } ``` Keep progress and reminders on stderr. ### 4. Update Skill Guidance Update `agentkit/skills/builtin/tidb-clinic-diagnosis/SKILL.md`: - The script writes structured evidence files. - stdout is only used to locate `agent_summary_path`. - The agent must read `evidence_summary_for_agent.json` for reasoning. - The agent must not read `evidence/*.json` during normal diagnosis unless explicitly asked. - The agent must not write `diagnosis_report.json`; Python writes it. ### 5. Update Backend and Frontend Fallbacks Backend hydration should prefer: ```text evidence_summary_for_agent.json ``` Then fall back to old paths for historical workspaces: ```text summary/evidence_summary.json queries/evidence_queries.json logs/evidence_logs.json metrics/evidence_metrics.json ``` Frontend evidence picker should prefer: ```text evidence_summary_for_agent.json diagnosis_report.json evidence/queries.json evidence/metrics.json evidence/logs.json ``` Old path fallback can remain read-only for historical runs. ## Test Plan - CLI stdout test: - stdout is valid JSON. - stdout contains `status`, `run_dir`, `agent_summary_path`, `diagnosis_report_path`. - stdout does not contain diagnosis section text such as `A1`, `TopSQL`, `decoded_plan`, or `full_sql`. - File layout test: - `evidence_summary_for_agent.json` exists at run root. - `diagnosis_report.json` is written by Python. - `evidence/*.json` category files exist. - old raw files are not written by default. - Evidence content tests: - metrics contain compact rows and no `chart_data`, `values`, or raw series. - logs contain `log_counts` and no raw `entries`. - queries include slow queries, TopSQL, `full_sql`, and `decoded_plan`. - workload pairs `statement_rate_*` with `affected_rows_rate_*`. - resource control includes RU usage, request p90, and available RU summaries. - Backend compatibility tests: - hydration reads `evidence_summary_for_agent.json`. - old summary/sidecar paths still hydrate historical workspaces. - raw metric/log fallback does not leak raw values or entries into hydrated reports. - Frontend tests or checks: - evidence picker prefers the new root agent summary. - old evidence paths still display when present. ## Assumptions - `full_sql`, `decoded_plan`, slow query, and TopSQL are core diagnostic data and must remain agent-readable. - Raw metric values and full log rows are not needed for normal diagnosis. - Python is the owner of structured artifact files; the agent uses them for reasoning and may produce narrative conclusions in chat. - Historical workspace compatibility is read-only fallback; new runs should use only the new layout.