# Evidence Collection Reference Standalone reference only. Multi-agent collection follows `evidence-agent.md`, including sweep-without-upfront-skill-reads and targeted mode. The examples below do not add preflight reads, authorize analyst collection, or require a sweep for ad-hoc requests. Paths to scripts and their behavior are unchanged. ## Input Format The evidence is provided as a structured summary with these sections: - **A1. Infrastructure Metrics**: EBS throttle counts, disk IO utilization - **A2. Application Metrics**: TiDB P99 latency, TiKV gRPC P99, CPU by instance, TiDB GC metrics, component count/restarts, worker memory - **A2b. Workload Metrics**: statement count paired with affected rows by SQL type - **A2c. Resource Control**: RU usage, resource-manager request p90, available RU - **A3. Slow Queries**: Dedicated/Premium top digests by full-window exec_count; BYOC latency-ranked samples labeled with sample_count/sample_avg_time, paired with full-window Statements Summary for workload ranking - **A3b. Targeted decoded plan**: warehouse Slow Query plan on supported non-BYOC tiers, or BYOC live Statements `plans` + `plan-detail` evidence - **A4. TopSQL**: TiDB and TiKV CPU attribution by SQL digest - **A5. Cluster Events**: Activity history during the incident window - **A6. Logs**: ERROR/FATAL log occurrence counts and WARN count - **Gate Check Results**: Infra gate, log gate, and hotspot detection verdicts ### Checkpoint — KB Step A (blocking — before Step B / `collect_evidence.py`) Automated and fast paths often skip KB and jump to evidence collection; **Premium** clusters then query TiKV/PD with the wrong `cluster_id`. **After** cluster metadata exists and **before** KB Step B / `collect_evidence.py` or any PromQL/metrics calls: 1. `read("skills/builtin/tidb-knowledge-base/knowledge/tier-architecture.json")` (if not already loaded this session). 2. Map `cluster_type` / `cluster_type_v2` / `parent_id` / `keyspace_id` to the correct tier entry and **apply `metric_routing`** (no special tag or fixed-format line required in the user-visible reply — internal consistency matters). **Hard rule:** If you have not done step 1–2, you have **not** completed the workflow; go back before KB Step B and evidence collection. **Do not** treat stderr hints from `collect_evidence.py` as a substitute for Step A. ### Checkpoint — KB Step B (blocking — after Step A, before Step 0 / `collect_evidence.py`) **Before** running `collect_evidence.py`, any PromQL calls, or ranking TopSQL/SlowQuery suspects: 1. `read("skills/builtin/tidb-knowledge-base/knowledge/incorrect-conclusions.json")`. 2. Note relevant `entries[]` (`wrong_pattern`, `corrective_checks`, `conclusion_rule`) and apply them **during** evidence collection and interpretation — not only when writing the conclusion. **Hard rule:** If Step B was skipped, **STOP** — do not attribute CPU or root cause from TopSQL/SlowQuery until the pitfall catalog is loaded. Only after Step A **and** Step B, run `collect_evidence.py` / clinic metrics. ## Knowledge Files Load ONLY the relevant file(s) based on cluster type: - Shared metrics: `tidb.json`, `node-info.json`, `pd.json`, `tiflash.json` - Starter/Essential: `*starter_essential*.json` - Premium: `*premium*.json` - Dedicated: `*dedicated*.json` - BYOC: shared/component files plus the BYOC routing entry in `tier-architecture.json` ``` read("skills/tidb-clinic-diagnosis/knowledge/filename.json") ``` ### Step 0: Evidence Collection and Gate Checks **Prerequisite:** **KB Step A and Step B** (checkpoints above) are satisfied for this cluster/time range. Do not run the shell commands below until they are. Before starting analysis, collect structured evidence and run gate checks. This step ensures you have complete data (A1-A4) and automated hotspot detection (including CPU saturation at >80% utilization). Before running `collect_evidence.py`, determine the **final canonical time window** — never guess timestamps or run collection with placeholders. Window selection priority: (1) user-provided fault time in the conversation, (2) workspace `Time Range` metadata, (3) JIRA issue / alert creation time. The script accepts 10-digit epoch seconds and timezone-bearing ISO datetime strings; **prefer ISO 8601 with timezone** (e.g. `2026-03-09T04:14:00Z`) so the original year/time stays visible in tool traces and no epoch is hand-calculated. Never pass a datetime without timezone — normalize it first with the `time-conversion` skill: ```bash python3 .opencode/skills/builtin/time-conversion/scripts/time_convert.py dt-to-ts \ --dt '2026-03-09 12:14:00' --assume-tz Asia/Shanghai --field utc_iso ``` Run the collector once `cluster_id`, `start_time`, and `end_time` are final. It normalizes times to epoch seconds, prints the canonical UTC ISO + epoch window on stderr, chooses the workspace evidence root, and writes the evidence files before returning. One workspace has one current evidence store — repeated runs merge/upsert into it. Run `collect_evidence.py -h` for supported time formats and concrete `time-conversion` examples. ### Collection Consistency Rule `run_dir` is the workspace evidence root. Do not infer correctness from the directory name. - If you later discover that `start_time` / `end_time` were wrong, rerun `collect_evidence.py` with the corrected window. - The corrected run merges into the same current evidence files. - Known risk: repeated collection in the same workspace can mix old time ranges, clusters, or previous report analysis with newer evidence. This is currently accepted as low probability; do not manually edit evidence files to compensate. ```bash PYTHONPATH=.opencode/skills/builtin/clinic-cli/scripts \ python3 .opencode/skills/builtin/tidb-clinic-diagnosis/scripts/collect_evidence.py \ 2026-03-09T04:14:00Z 2026-03-09T05:14:00Z [--parent-id ] # 10-digit epoch seconds are also accepted, but ISO 8601 is preferred: python3 .opencode/skills/builtin/tidb-clinic-diagnosis/scripts/collect_evidence.py \ 1773029640 1773033240 [--parent-id ] ``` The script writes structured evidence files and prints a JSON command result to stdout. stdout is **only** used to locate evidence file paths — it is not an evidence channel. **`--parent-id`** (Premium clusters only): Pass the Resource Pool `parent_id` from workspace context so that component health queries (component count, restarts, worker memory) are scoped to the shared pool where TiKV/PD/worker pods actually run. Without this flag on Premium clusters, worker metrics will return empty and the diagnosis will miss OOM / capacity failures. For BYOC, do not pass the physical parent as a Premium-style metric scope. The collector detects logical and physical BYOC IDs automatically, keeps cluster-store metrics on the requested ID, and scopes exact Kubernetes/node metrics to `tidb{physical_cluster_id}`. Slow Query, TopSQL, logs, and targeted plan evidence use customer-side O11y routes. Warehouse SQL and LogStats are not fallbacks; unsupported warehouse evidence is marked explicitly. After an impactful digest is selected, the collector persists Statements `plans` and `plan-detail` operations under `queries.json.statements` with `source: "statement"`. stdout JSON on success (on failure, `status` is `"error"` with an `error` message and a `hint` pointing at `collect_evidence.py -h`): ```json { "status": "ok", "run_dir": "", "report_path": "/workspace_evidence_report.json", "agent_summary_txt_path": "/evidence_summary_for_agent.txt", "agent_summary_path": "/evidence_summary_for_agent.json", "workspace_evidence_report_path": "/workspace_evidence_report.json", "raw_dir": "/raw", "errors": [] } ``` stderr contains progress messages and warnings (not structured evidence). Evidence is written into the workspace evidence root returned by stdout: ```text /evidences/ evidence_summary_for_agent.txt evidence_summary_for_agent.json workspace_evidence_report.json metrics.json workload.json resource_control.json component_health.json queries.json logs.json tikv_thread_cpu.json links.json gate_checks.json collection_errors.json raw/ # verbatim upstream API responses (see below) manifest.jsonl metrics/ logs/ queries/ events/ meta/ ``` Repeated collection reuses the same directory instead of creating a new opaque numeric path and merges into the current evidence files. `raw/` archives every successful upstream API response verbatim — full PromQL matrices for all instances (not just the top-N kept in `metrics.json`), LogStats/Loki payloads, SQL result rows, activity events, and cluster metadata. `raw/manifest.jsonl` is an append-only index with one JSON line per file (`file`, `kind`, `label`, `query_preview`, `scope`, `start`, `end`, `bytes`). It exists for reproducibility, timeline reconstruction, cross-signal correlation, and verifying a suspicious aggregate. Log/query/event/meta payloads are small and line-oriented — read or grep them directly. Metrics matrices are large — do **not** read them into context directly; use the bounded reader to filter and downsample: ```bash python3 .opencode/skills/builtin/tidb-clinic-diagnosis/scripts/read_raw_evidence.py --list python3 .opencode/skills/builtin/tidb-clinic-diagnosis/scripts/read_raw_evidence.py \ --kind metrics --label-contains cpu --series-contains db-tidb-0 \ --start --end --max-series 3 --max-points 60 ``` Disable archiving with `--no-raw` or `COLLECT_EVIDENCE_NO_RAW=1`. The script prints `Evidence files saved to /` on stderr. **Read `evidence_summary_for_agent.txt` for all diagnostic reasoning.** It is a compact, token-efficient plain-text file: a section index with line numbers (read specific sections via offset + limit), key findings, grouped TSV metrics (each promql/link appears once per group header), workload, resource control, component health, queries (slow queries + TopSQL + decoded plan as an indented text tree), log occurrence counts, gate checks, and links. Fall back to `evidence_summary_for_agent.json` only when the text summary lacks a required field; `workspace_evidence_report.json` is the structured report written by Python. **Do NOT edit evidence/report files.** `collect_evidence.py` owns those writes. Do not write `analysis_patch.json`, run report merge helpers, manually edit `workspace_evidence_report.json`, or reason about UI key-evidence persistence; those are not agent actions in this skill. Other evidence files (`metrics.json` raw metric values/chart payloads, `component_health.json` counts/restarts/worker memory, `logs.json` log occurrence counts, `tikv_thread_cpu.json` thread-level detail) can be large — read them targeted: `head -N` / parse + slice, `grep -m N `, or a short Python one-liner to filter or aggregate before reasoning. For metric chart datapoints, resolve `metric_group_id` from `workspace_evidence_report.json` / analysis `metric_ref`, filter `metrics.json.metrics[]` to matching rows, and read only their `chart_data.series`; if a row has no `chart_data`, use its `chart_data_ref` instead of re-querying Clinic. ### Collect Evidence Failure Rule If `collect_evidence.py` times out, exits non-zero, or stdout `status` is `"error"`, you MUST NOT proceed to diagnosis reasoning. After a successful run, parse the stdout JSON for `agent_summary_txt_path` and confirm `evidence_summary_for_agent.txt` exists and is readable; if missing, treat the collection as failed and rerun. Never reconstruct diagnosis evidence manually or write raw evidence files from agent tool output. The script prints a stderr reminder about KB alignment but does **not** perform tier routing. Use its structured evidence + gate check verdicts as your evidence data, then proceed to Node C. **DO NOT skip this step** — without structured evidence and gate checks you will miss critical signals like CPU saturation hotspots. ## Data Access for Additional Queries If you need additional metrics not in the pre-collected evidence, you can query directly: | Data Type | Endpoint | Method | |-----------|----------|--------| | Metrics (PromQL range) | `/data-proxy/metrics/${CLUSTER_ID}/api/v1/query_range` | GET | | Metrics (instant) | `/data-proxy/metrics/${CLUSTER_ID}/api/v1/query` | GET | | Series discovery | `/data-proxy/metrics/${CLUSTER_ID}/api/v1/series` | GET | | Slowlog SQL | `/data-proxy/query` | POST | | TopSQL SQL | `/data-proxy/query` | POST | | Loki logs | `/data-proxy/loki/${CLUSTER_ID}/api/v1/query_range` | GET | Load credentials first: ```bash python3 .opencode/skills/builtin/clinic-cli/scripts/clinic_cli.py --help ```