# TiKV Thread CPU Evidence Collection Plan

## Summary

Fix the diagnosis workflow gap observed in workspace
`ws-1776220777461-75f939e68` / `O11Y-2396397`: after `collect_evidence.py`
identified TiKV CPU saturation, the agent still needed a separate manual
query for `tikv_thread_cpu_seconds_total`.

The manual follow-up path is fragile. In this workspace it produced a failed
tool call with `JSONDecodeError` after trying to parse an empty response. The
thread-level evidence should be collected by `collect_evidence.py` whenever
TiKV CPU hotspot candidates are present.

## Key Changes

- Update `agentkit/skills/builtin/tidb-clinic-diagnosis/scripts/collect_evidence.py`.
- Add TiKV hotspot instance selection using existing `cpu` and `cpu_quota`
  evidence.
- Select TiKV instances with peak utilization greater than or equal to 80%.
- If no TiKV instance reaches 80% but TiKV CPU data exists, collect thread CPU
  for the top TiKV CPU instances so relative hotspots still have supporting
  evidence.
- Query thread CPU with PromQL based on
  `tikv_thread_cpu_seconds_total`, grouped by `instance` and `name`.
- Classify dominant thread groups into diagnosis-friendly buckets such as
  `grpc`, `cop`, `raftstore`, `apply`, `sched`, `unified_read`, `rocksdb`,
  `background`, and `other`.
- Save the derived output to `<run_dir>/metrics/tikv-thread-cpu.json`.
- Add a compact `TiKV_THREAD_CPU:` section to the stdout summary.
- Treat empty Prometheus responses, API errors, and malformed structures as
  recoverable collection errors recorded in `data["errors"]`; do not raise
  `JSONDecodeError`.
- Update `agentkit/skills/builtin/tidb-clinic-diagnosis/SKILL.md` so Node D
  uses the generated thread CPU sidecar when present and only performs manual
  follow-up when the sidecar is missing or explicitly empty.

## Test Plan

- Add unit coverage with a fake `ClinicAPI` where TiKV CPU utilization is above
  80%; verify thread CPU collection is triggered.
- Add unit coverage where `query_range` returns an API error; verify the script
  records the error and continues.
- Add unit coverage where `query_range` returns empty data; verify no
  `JSONDecodeError` is raised.
- Verify `save_evidence_files` writes `metrics/tikv-thread-cpu.json` under the
  same run directory as the other evidence sidecars.
- Run a regression collection for an O11Y-2396397-like TiKV CPU alert and
  confirm the first collection pass includes thread CPU evidence without a
  manual follow-up command.

## Assumptions

- The diagnosis flow still requires KB Step A before evidence collection, so
  metric routing is decided before `collect_evidence.py` runs.
- This plan does not redesign tier routing inside `collect_evidence.py`.
- Thread CPU evidence is diagnostic support data; it should not block the
  entire evidence collection if unavailable.