Evaluation Run Detail Report
Run ID: evalrun-adhoc-evidence-demo-merged-20260722
Suite: adhoc-evidence-demo @ 126ce520d38c

Run Metadata
- execution_status: completed
- comparison_status: review_only
- trigger_mode: manual_merge
- schedule_date: -
- started_at: 2026-07-21T10:05:47.564612+00:00
- finished_at: 2026-07-22T01:58:35.778942+00:00

Runtime Metadata
- agent_type: opencode
- resolved_model: chat/claude-standard/high
- promote_eligibility: not_evaluated
- baseline_id: -

Key Metrics
- counts: cases=12 pass=6 fail=6 invalid=0
- score: avg=79.58 scored_cases=12/12
- run_duration: 952m48s
- case_duration: avg=0m53s stddev=0m21s measured_cases=12/12
- token_usage: input_avg=1 input_stddev=0 output_avg=659.50 output_stddev=685.87 reasoning_avg=0 reasoning_stddev=0 cache_read_avg=33747.17 cache_read_stddev=14407.51 cache_write_avg=3821.83 cache_write_stddev=5024.20 cached_total_avg=37569 cached_total_stddev=15963.52 cost_avg=$0.2213 cost_stddev=$0.1103 measured_cases=12/12

Case Details

Case: classify-selected-stores-as-tikv-or-tiflash
- execution_status: completed
- verdict: pass
- score: current=100 baseline=- diff=-
- failure_code: -
- invalid_subtype: -
- workspace: ws-eval-1784628733310-411e36ce
- time: started=2026-07-21T10:12:11.440809+00:00 finished=2026-07-21T10:12:53.109813+00:00 duration=0m42s
- token usage: input=1 output=212 reasoning=0 cache_read=40738 cache_write=455 cached_total=41193 cost=$0.2336
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=80/80 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Raw Output
<<<
`ad-hoc store-roles` 是匹配这个需求的 recipe，直接调用。从返回数据中提取目标 store 的 component 字段，结果如下（`46869168` 未在数据中出现，需确认）：
- `13` → tikv ✓
- `25` → tikv ✓  
- `49` → tikv ✓
- `67` → 需检查（数据中未直接看到）
- `1014458590` → tikv ✓
- `46869168` → 未出现

让我快速从完整输出文件中确认 `67` 和 `46869168`：所有 6 个 store 均已找到，结果齐全。

```json
{
  "cluster_id": "10518506906906971664",
  "evidence_sufficient": true,
  "stores": [
    {
      "store": "13",
      "component": "tikv"
    },
    {
      "store": "25",
      "component": "tiflash"
    },
    {
      "store": "49",
      "component": "tikv"
    },
    {
      "store": "67",
      "component": "tiflash"
    },
    {
      "store": "1014458590",
      "component": "tikv"
    },
    {
      "store": "46869168",
      "component": "tiflash"
    }
  ]
}
```
>>>

Case: compaction-signals-on-db-tikv-9-in-saas01-during-latest-window
- execution_status: completed
- verdict: fail
- score: current=69 baseline=- diff=-
- failure_code: critical_evidence_partial
- invalid_subtype: -
- workspace: ws-eval-1784628774908-142a0c7d
- time: started=2026-07-21T10:12:53.120581+00:00 finished=2026-07-21T10:13:58.358302+00:00 duration=1m05s
- token usage: input=1 output=308 reasoning=0 cache_read=58175 cache_write=12042 cached_total=70217 cost=$0.4370
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=74/80 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Judging Conditions
- critical evidence: impact=6 desc=Evidence must not use a broad instance regex that can pull db-tikv-90/91/92 into db-tikv-9 analysis.
Bucket Failure Details
- critical evidence: partial score=74/80
-   - broad_instance_regex_absent earned=0/6 desc=Evidence must not use a broad instance regex that can pull db-tikv-90/91/92 into db-tikv-9 analysis. rules=artifact_json/json_values_not_contains_any/matched=no/actual=tikv_engine_pending_compaction_bytes{instance=~"db-tikv-9.*"}
Raw Output
<<<
Now I have the PromQL for the metrics I need. Let me check the metrics subskill for L0 files:I have the key PromQL patterns. Now I'll fetch the four required metrics for `db-tikv-9` using the clinic-cli:
- `tikv_engine_pending_compaction_bytes` (for `write` and `lock` CF)  
- `tikv_engine_num_files_at_level` where level=0 (for `write` and `lock` CF)Both metrics fetched. Now let me extract the per-CF max values from the persisted evidence:All data is in the `series_summaries`. I can read the exact values for `db-tikv-9` directly from the output:

**`db-tikv-9` — pending compaction bytes (max):**
- `write` CF: **17,669,078,025** bytes
- `lock` CF: **34,343,645** bytes

**`db-tikv-9` — L0 files (max):**
- `write` CF: **8**
- `lock` CF: **4**

```json
{
  "cluster_id": "10989049060142230334",
  "tikv_node": "db-tikv-9",
  "window": {
    "start": "2026-07-21T03:00:00Z",
    "end": "2026-07-21T03:05:00Z"
  },
  "evidence_sufficient": true,
  "signals": {
    "write_cf_pending_compaction_bytes_max": 17669078025,
    "lock_cf_pending_compaction_bytes_max": 34343645,
    "write_cf_l0_files_max": 8,
    "lock_cf_l0_files_max": 4
  }
}
```
>>>

Case: map-selected-tikv-pods-to-worker-nodes-in-saas01
- execution_status: completed
- verdict: pass
- score: current=100 baseline=- diff=-
- failure_code: -
- invalid_subtype: -
- workspace: ws-eval-1784628840145-28bfed9c
- time: started=2026-07-21T10:13:58.366557+00:00 finished=2026-07-21T10:14:46.807774+00:00 duration=0m48s
- token usage: input=1 output=266 reasoning=0 cache_read=37043 cache_write=577 cached_total=37620 cost=$0.2163
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=80/80 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Raw Output
<<<
`kube_pod_info` 包含 pod 到 node 的映射，直接用 PromQL 查询。数据返回了 13 条 series，其中部分来自其他集群的 namespace（不是 `tidb10989049060142230334`）。需要按 namespace 过滤出目标集群的记录。四个 pod 全部命中，证据充分。

```json
{
  "cluster_id": "10989049060142230334",
  "snapshot_time": "2026-07-21T10:04:00Z",
  "evidence_sufficient": true,
  "pod_mappings": [
    {
      "tikv_pod": "db-tikv-0",
      "worker_node": "ip-172-26-139-120.ec2.internal"
    },
    {
      "tikv_pod": "db-tikv-35",
      "worker_node": "ip-172-26-11-236.ec2.internal"
    },
    {
      "tikv_pod": "db-tikv-69",
      "worker_node": "ip-172-26-68-211.ec2.internal"
    },
    {
      "tikv_pod": "db-tikv-144",
      "worker_node": "ip-172-26-77-187.ec2.internal"
    }
  ]
}
```
>>>

Case: new-slow-query-digests-between-adjacent-windows-in-saas01
- execution_status: completed
- verdict: pass
- score: current=100 baseline=- diff=-
- failure_code: -
- invalid_subtype: -
- workspace: ws-eval-1784628544729-860b0978
- time: started=2026-07-21T10:09:02.938727+00:00 finished=2026-07-21T10:10:26.666724+00:00 duration=1m24s
- token usage: input=1 output=1357 reasoning=0 cache_read=31801 cache_write=1940 cached_total=33741 cost=$0.2566
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=80/80 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Raw Output
<<<
The recipe uses `exec_count` ordering but I need raw digest-level counts from slow query logs. Let me check the `sql` subcommand help and the data-proxy subskill to understand the right query shape.Now I have all I need. I'll use the `sql` subcommand with `slow_query_logs` for both windows. The date partition is `20260721`, and I need to use epoch-second `time` predicates. Let me run both queries in parallel.Both queries returned 10 rows. The summary only shows 3 samples each — I need the full rows. Let me read the persisted evidence file to get all 10 rows for each window.I have all the data. Now let me build the final JSON output with the proper fields, SQL truncated to 200 chars, and sorted by occurrence_count DESC then digest lexicographic.```json
{
  "cluster_id": "10989049060142230334",
  "comparison": {
    "previous_window": {
      "start": "2026-07-21T02:50:00Z",
      "end": "2026-07-21T03:00:00Z"
    },
    "current_window": {
      "start": "2026-07-21T03:00:00Z",
      "end": "2026-07-21T03:10:00Z"
    }
  },
  "evidence_sufficient": true,
  "new_sql_digests": [
    {
      "sql_digest": "2d142ccf344650a9226ea89d072cdae8eeb850fff82d96e8bf0406cdf52ee69f",
      "sql_statement": "SELECT `dh_active_receive_logs`.`id`,`dh_active_receive_logs`.`activeType`,`dh_active_receive_logs`.`useridx`,`dh_active_receive_logs`.`username`,`dh_active_receive_logs`.`account_type`,`dh_active_rec",
      "occurrence_count": 883
    },
    {
      "sql_digest": "82e12f641a02f5f1b77c6e122f529f4b3585021f9ac0f23a6577dc8c0a9bd3bf",
      "sql_statement": "/* job 42/270 */ UPDATE `dh_app_5073`.`dh_active_user_coupon` SET `is_delete`=1, `status`=3, `update_time`=1784602824, `expire_time`=IF((`expire_time` = 0), 1784602824, `expire_time`), `front_remark`=",
      "occurrence_count": 129
    },
    {
      "sql_digest": "9a79d3bceafdec75e6c59ed6687bdb7c2993a525a13cbfc134306732bc254685",
      "sql_statement": "SELECT `id` FROM `dh_active_receive_logs` WHERE `activeid` IN (?) AND usercurrency = ? AND activetype = ? AND (update_time >= ? and update_time <= ?) AND site_code IN (?) AND log_category in (?) AND l",
      "occurrence_count": 126
    },
    {
      "sql_digest": "18576e217f5f93fa2a934f84290447caa4b783cab493503c4ad71c912ea16930",
      "sql_statement": "SELECT * FROM `dh_audit_record` WHERE (user_id = ? and site_code = ?) AND status in (?,?,?) AND charge_time < ? ORDER BY charge_time asc,id asc [arguments: (677577915, \"5073\", 1, 2, 7, \"2026-07-21 11:",
      "occurrence_count": 124
    },
    {
      "sql_digest": "7d75919b6fcc6bbbc8516539ac4a165887c4a9765b9e986314ef43ab77267874",
      "sql_statement": "SELECT ? as create_ymd,t2.register_currency as currency,IFNULL(t2.club_id,0) as club_id,count(distinct t1.useridx) as login_persons FROM dh_user_action_logs as t1 left join dh_account_basic as t2 on t",
      "occurrence_count": 95
    },
    {
      "sql_digest": "67a28431a20b995bf91fa37e2691be34a00d1c353e1f0efc9c67e418dd499878",
      "sql_statement": "SELECT t1.register_currency as currency,t1.regpkgid as channel_id,ifnull(t2.parent_cids, '[]') as parent_cids,t3.register_os_type,t0.valid_bet FROM dh_user_day_report t0 left join dh_account_basic as ",
      "occurrence_count": 94
    },
    {
      "sql_digest": "4afa34fab28500dd78a9d5a531dcf93b2c8f36f525d970c99028638c243f2c3b",
      "sql_statement": "SELECT a.id,a.site_code,a.useridx,a.username,a.register_currency currency,a.realname_secret,a.account_type ,a.user_status,a.loan_status,a.game_gold,a.member_level,a.vip_level, a.register_time,a.regpkg",
      "occurrence_count": 88
    },
    {
      "sql_digest": "5ed3c66a6505fd76ff485d8829bf90185120209653739e7ed473e676d622ce5b",
      "sql_statement": "SELECT '699' as site_code,20260720 as create_ymd,case when t2.register_os_type in (2,11,12,13,20) then 1 when t2.register_os_type in (1,9,10,14,18,19) then 2 else 3 end as kind,case when t2.register_o",
      "occurrence_count": 74
    }
  ]
}
```
>>>

Case: representative-business-slow-queries-in-saas01-during-latest-window
- execution_status: completed
- verdict: fail
- score: current=44 baseline=- diff=-
- failure_code: critical_evidence_partial
- invalid_subtype: -
- workspace: ws-eval-1784628501401-3950606f
- time: started=2026-07-21T10:08:19.748257+00:00 finished=2026-07-21T10:09:02.928312+00:00 duration=0m43s
- token usage: input=1 output=469 reasoning=0 cache_read=21713 cache_write=16875 cached_total=38588 cost=$0.1778
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=32/80 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Judging Conditions
- critical evidence: impact=24 desc=Output JSON digest ordering matches the runtime reference.
- critical evidence: impact=12 desc=Output JSON avg_time_s values match the runtime reference.
- critical evidence: impact=12 desc=Output JSON exec_count values match the runtime reference.
Bucket Failure Details
- critical evidence: partial score=32/80
-   - slow_query_digests_match_reference earned=0/24 desc=Output JSON digest ordering matches the runtime reference. rules=output_json/json_array_matches_reference/matched=no/actual=[]
-   - slow_query_exec_count_match_reference earned=0/12 desc=Output JSON exec_count values match the runtime reference. rules=output_json/json_array_matches_reference/matched=no/actual=[]
-   - slow_query_avg_time_match_reference earned=0/12 desc=Output JSON avg_time_s values match the runtime reference. rules=output_json/json_array_matches_reference/matched=no/actual=[]
Raw Output
<<<
从返回的 20 条记录中，过滤掉事务控制语句（`COMMIT` — digest `9505cacb...`、`START TRANSACTION` — digest `9a4170a0...`），剩余业务查询按 `exec_count` 降序排列，取前 3：

1. `2d142ccf...` — exec_count 404，avg 2.549 s
2. `7d75919b...` — exec_count 95，avg 1.553 s
3. `4afa34fa...` — exec_count 71，avg 1.290 s

```json
{
  "cluster_id": "10989049060142230334",
  "window": {
    "start": "2026-07-21T03:00:00Z",
    "end": "2026-07-21T03:05:00Z"
  },
  "evidence_sufficient": true,
  "slow_queries": [
    {
      "digest": "2d142ccf344650a9226ea89d072cdae8eeb850fff82d96e8bf0406cdf52ee69f",
      "exec_count": 404,
      "avg_time_s": 2.549
    },
    {
      "digest": "7d75919b6fcc6bbbc8516539ac4a165887c4a9765b9e986314ef43ab77267874",
      "exec_count": 95,
      "avg_time_s": 1.553
    },
    {
      "digest": "4afa34fab28500dd78a9d5a531dcf93b2c8f36f525d970c99028638c243f2c3b",
      "exec_count": 71,
      "avg_time_s": 1.290
    }
  ]
}
```
>>>

Case: single-node-cpu-core-scope-compare-during-incident-window
- execution_status: completed
- verdict: fail
- score: current=62 baseline=- diff=-
- failure_code: critical_evidence_partial
- invalid_subtype: -
- workspace: ws-eval-1784628888955-ff42d19f
- time: started=2026-07-21T10:14:46.816835+00:00 finished=2026-07-21T10:15:56.796798+00:00 duration=1m10s
- token usage: input=1 output=338 reasoning=0 cache_read=35243 cache_write=879 cached_total=36122 cost=$0.2537
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=47/79 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Judging Conditions
- critical evidence: impact=25 desc=Output JSON windows numeric fields match the runtime reference.
- critical evidence: impact=5 desc=Evidence includes TiKV CPU core and quota metrics for this request. snippet=_group_id": "tikv_cpu_win1", "name": "tikv_cpu_db-tikv-9_win1", "promql": "rate(process_cpu_seconds_total{instance=~\"db-tikv-9.*\"}[2m])", "unit": "cores",...
- critical evidence: impact=4 desc=Evidence shows the CPU query used the 1-minute rate window rather than a wider window. snippet=etric_group_id": "tikv_cpu_win1", "name": "tikv_cpu_db-tikv-9_win1", "promql": "rate(process_cpu_seconds_total{instance=~\"db-tikv-9.*\"}[2m])", "unit": "cor...
Bucket Failure Details
- critical evidence: partial score=47/79
-   - cpu_metric_evidence_present earned=0/5 desc=Evidence includes TiKV CPU core and quota metrics for this request. snippet=_group_id": "tikv_cpu_win1", "name": "tikv_cpu_db-tikv-9_win1", "promql": "ra... rules=artifact_json/json_text_contains_any/matched=yes/actual=_group_id": "tikv_cpu_win1", "name": "tikv_cpu_db-tikv-9_win1", "promql": "rate(process_cpu_seconds_total{instance=~\"db-tikv-9.*\"}[2m])", "unit": "cores", "summary": {"point_count": 2; artifact_json/json_text_contains_any/matched=no/actual=[]
-   - cpu_1m_rate_window_evidence_present earned=0/4 desc=Evidence shows the CPU query used the 1-minute rate window rather than a wider window. snippet=etric_group_id": "tikv_cpu_win1", "name": "tikv_cpu_db-tikv-9_win1", "promql"... rules=artifact_json/json_text_contains_any/matched=yes/actual=etric_group_id": "tikv_cpu_win1", "name": "tikv_cpu_db-tikv-9_win1", "promql": "rate(process_cpu_seconds_total{instance=~\"db-tikv-9.*\"}[2m])", "unit": "cores", "summary": {"point_count": 2; artifact_json/json_text_contains_any/matched=no/actual=[]
-   - windows_numeric_fields_match_reference earned=0/25 desc=Output JSON windows numeric fields match the runtime reference. rules=output_json/json_array_matches_reference/matched=no/actual=[]
Raw Output
<<<
Now I need to get cluster metadata to find the CPU spec for `db-tikv-9`, then query CPU metrics for each of the 3 windows. Let me first get the cluster topology to know the CPU spec, then run the CPU queries for all 3 windows.TiKV CPU spec is 32 cores. Now query CPU for each of the 3 windows. I need `process_cpu_seconds_total` rate for `db-tikv-9`. Let me run all 3 windows in parallel.The summary stats show min/max/avg across all 4 series (the regex `db-tikv-9.*` may match multiple instances). I need to confirm these are all for the single `db-tikv-9` node and get per-instance series. Let me read the metrics.json to extract the actual series data for each window, specifically for `db-tikv-9`.The regex `db-tikv-9.*` matched 4 instances: `db-tikv-9`, `db-tikv-90`, `db-tikv-91`, `db-tikv-92`. I need to isolate only `db-tikv-9` (exact match). Let me compute avg/peak from the `db-tikv-9` series only.All 3 windows have complete data (61 points each) for the exact `db-tikv-9` instance. CPU spec is 32 cores confirmed from cluster topology.

```json
{
  "cluster_id": "10989049060142230334",
  "node": "db-tikv-9",
  "cpu_spec_cores": 32.0000,
  "evidence_sufficient": true,
  "windows": [
    {
      "start": "2026-07-21T03:00:00Z",
      "end": "2026-07-21T03:30:00Z",
      "avg_cpu_cores": 15.4052,
      "peak_cpu_cores": 18.4583
    },
    {
      "start": "2026-07-20T03:00:00Z",
      "end": "2026-07-20T03:30:00Z",
      "avg_cpu_cores": 15.3339,
      "peak_cpu_cores": 17.7250
    },
    {
      "start": "2026-07-19T03:00:00Z",
      "end": "2026-07-19T03:30:00Z",
      "avg_cpu_cores": 14.7867,
      "peak_cpu_cores": 17.8750
    }
  ]
}
```
>>>

Case: tikv-read-iops-full-ranking-across-two-clusters-during-incident-window
- execution_status: completed
- verdict: fail
- score: current=69 baseline=- diff=-
- failure_code: critical_evidence_partial
- invalid_subtype: -
- workspace: ws-eval-1784685385429-feac804f
- time: started=2026-07-22T01:56:23.749249+00:00 finished=2026-07-22T01:57:46.470130+00:00 duration=1m23s
- token usage: input=1 output=2668 reasoning=0 cache_read=66363 cache_write=4235 cached_total=70598 cost=$0.4449
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=59/80 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Judging Conditions
- critical evidence: impact=12 desc=Output JSON grouped per-cluster node ranking matches the runtime reference ordering and values.
Bucket Failure Details
- critical evidence: partial score=59/80
-   - full_node_ranking_matches_runtime_reference earned=0/12 desc=Output JSON grouped per-cluster node ranking matches the runtime reference ordering and values. rules=output_json/json_path_equals_reference/matched=no/actual=[]
Raw Output
<<<
这是 ad-hoc 数据查询，直接用 `ad-hoc topsql-links` recipe 来查 TiKV 读盘 IOPS。先确认两个集群的 help，再并行对两个集群执行查询。`--top-n` 默认值不明，但用户要全部节点。并行对两个集群执行，设置足够大的 `--top-n` 确保返回所有 TiKV 节点：注意：recipe 返回的是 `avg_read_iops`（窗口内的平均值），但用户要求的是 `max 1m read IOPS`。从 `top_series` 里每个节点都有 `max` 字段，那才是窗口内的峰值。现在用 Python 从两个结果中提取 `top_series[].max` 并按要求排序。```json
{
  "window": {
    "start": "2026-07-21T03:00:00Z",
    "end": "2026-07-21T03:05:00Z"
  },
  "mapping_strategy": "node_filter",
  "evidence_sufficient": true,
  "nodes_by_cluster": [
    {
      "cluster_id": "10518506906906971664",
      "nodes": [
        ["db-tikv-52", 10683.7],
        ["db-tikv-20", 10664.55],
        ["db-tikv-48", 10135.37],
        ["db-tikv-26", 9419.62],
        ["db-tikv-18", 9303.28],
        ["db-tikv-39", 8893.55],
        ["db-tikv-38", 8851.43],
        ["db-tikv-92", 8797.17],
        ["db-tikv-49", 8723.3],
        ["db-tikv-44", 8388.88],
        ["db-tikv-14", 8306.1],
        ["db-tikv-30", 8119.2],
        ["db-tikv-37", 8065.78],
        ["db-tikv-32", 7916.2],
        ["db-tikv-34", 7883.2],
        ["db-tikv-33", 7756.07],
        ["db-tikv-11", 7724.05],
        ["db-tikv-23", 7678.87],
        ["db-tikv-27", 7623.0],
        ["db-tikv-21", 7590.38],
        ["db-tikv-17", 7467.12],
        ["db-tikv-42", 7424.28],
        ["db-tikv-47", 7413.13],
        ["db-tikv-5", 7382.37],
        ["db-tikv-36", 7340.72],
        ["db-tikv-10", 7283.27],
        ["db-tikv-13", 7278.05],
        ["db-tikv-22", 7160.97],
        ["db-tikv-2", 7141.38],
        ["db-tikv-9", 7126.55],
        ["db-tikv-15", 7081.88],
        ["db-tikv-4", 6932.85],
        ["db-tikv-45", 6842.83],
        ["db-tikv-28", 6831.3],
        ["db-tikv-24", 6784.08],
        ["db-tikv-31", 6776.3],
        ["db-tikv-0", 6759.92],
        ["db-tikv-8", 6729.55],
        ["db-tikv-46", 6686.05],
        ["db-tikv-7", 6625.55],
        ["db-tikv-90", 6615.78],
        ["db-tikv-3", 6607.72],
        ["db-tikv-12", 6575.62],
        ["db-tikv-51", 6568.77],
        ["db-tikv-6", 6468.45],
        ["db-tikv-25", 6462.75],
        ["db-tikv-40", 6438.92],
        ["db-tikv-43", 6349.63],
        ["db-tikv-35", 6347.53],
        ["db-tikv-16", 6158.25],
        ["db-tikv-50", 6099.52],
        ["db-tikv-91", 5994.03],
        ["db-tikv-41", 5935.6],
        ["db-tikv-19", 5925.85],
        ["db-tikv-1", 5825.03],
        ["db-tikv-53", 5643.53],
        ["db-tikv-29", 4752.35]
      ]
    },
    {
      "cluster_id": "10989049060142230334",
      "nodes": [
        ["db-tikv-54", 9094.63],
        ["db-tikv-10", 8542.23],
        ["db-tikv-56", 8377.47],
        ["db-tikv-77", 8000.03],
        ["db-tikv-41", 7907.48],
        ["db-tikv-8", 7877.38],
        ["db-tikv-13", 7842.7],
        ["db-tikv-50", 7811.23],
        ["db-tikv-82", 7773.32],
        ["db-tikv-62", 7746.7],
        ["db-tikv-135", 7735.95],
        ["db-tikv-17", 7726.52],
        ["db-tikv-36", 7659.83],
        ["db-tikv-59", 7558.15],
        ["db-tikv-72", 7530.15],
        ["db-tikv-35", 7410.77],
        ["db-tikv-58", 7389.95],
        ["db-tikv-84", 7345.3],
        ["db-tikv-85", 7342.38],
        ["db-tikv-42", 7332.85],
        ["db-tikv-22", 7278.28],
        ["db-tikv-64", 7251.15],
        ["db-tikv-88", 7164.03],
        ["db-tikv-4", 7145.42],
        ["db-tikv-49", 7088.0],
        ["db-tikv-74", 7082.75],
        ["db-tikv-7", 7072.12],
        ["db-tikv-90", 7035.38],
        ["db-tikv-140", 7025.78],
        ["db-tikv-71", 7018.05],
        ["db-tikv-92", 7004.3],
        ["db-tikv-3", 6966.1],
        ["db-tikv-24", 6941.18],
        ["db-tikv-1", 6851.03],
        ["db-tikv-30", 6800.22],
        ["db-tikv-45", 6799.02],
        ["db-tikv-81", 6792.12],
        ["db-tikv-26", 6773.5],
        ["db-tikv-66", 6743.42],
        ["db-tikv-53", 6733.42],
        ["db-tikv-32", 6710.9],
        ["db-tikv-16", 6673.3],
        ["db-tikv-46", 6623.4],
        ["db-tikv-61", 6577.9],
        ["db-tikv-68", 6546.25],
        ["db-tikv-139", 6529.65],
        ["db-tikv-136", 6526.43],
        ["db-tikv-137", 6411.28],
        ["db-tikv-38", 6399.87],
        ["db-tikv-86", 6346.25],
        ["db-tikv-37", 6317.48],
        ["db-tikv-9", 6309.88],
        ["db-tikv-14", 6273.97],
        ["db-tikv-40", 6246.08],
        ["db-tikv-20", 6242.32],
        ["db-tikv-28", 6240.82],
        ["db-tikv-44", 6237.32],
        ["db-tikv-132", 6236.58],
        ["db-tikv-0", 6236.32],
        ["db-tikv-79", 6195.92],
        ["db-tikv-89", 6139.35],
        ["db-tikv-75", 6104.98],
        ["db-tikv-34", 6104.87],
        ["db-tikv-39", 6007.17],
        ["db-tikv-11", 5978.45],
        ["db-tikv-51", 5967.68],
        ["db-tikv-27", 5914.35],
        ["db-tikv-80", 5896.65],
        ["db-tikv-141", 5852.4],
        ["db-tikv-63", 5810.45],
        ["db-tikv-43", 5771.52],
        ["db-tikv-143", 5738.43],
        ["db-tikv-73", 5719.1],
        ["db-tikv-31", 5692.72],
        ["db-tikv-70", 5664.43],
        ["db-tikv-52", 5604.72],
        ["db-tikv-76", 5599.98],
        ["db-tikv-65", 5585.27],
        ["db-tikv-133", 5554.07],
        ["db-tikv-55", 5477.42],
        ["db-tikv-69", 5410.67],
        ["db-tikv-18", 5408.55],
        ["db-tikv-19", 5407.62],
        ["db-tikv-134", 5354.37],
        ["db-tikv-2", 5332.1],
        ["db-tikv-23", 5324.37],
        ["db-tikv-25", 5297.93],
        ["db-tikv-29", 5243.08],
        ["db-tikv-21", 5230.35],
        ["db-tikv-83", 5228.07],
        ["db-tikv-12", 5141.32],
        ["db-tikv-15", 5110.55],
        ["db-tikv-48", 5104.35],
        ["db-tikv-33", 5064.15],
        ["db-tikv-47", 5060.12],
        ["db-tikv-60", 5049.9],
        ["db-tikv-138", 4979.48],
        ["db-tikv-91", 4959.72],
        ["db-tikv-57", 4854.58],
        ["db-tikv-87", 4699.2]
      ]
    }
  ]
}
```
>>>

Case: tikv-read-iops-ranking-during-incident-window
- execution_status: completed
- verdict: fail
- score: current=69 baseline=- diff=-
- failure_code: critical_evidence_partial
- invalid_subtype: -
- workspace: ws-eval-1784628349150-c92b148e
- time: started=2026-07-21T10:05:47.582113+00:00 finished=2026-07-21T10:06:15.126521+00:00 duration=0m28s
- token usage: input=1 output=228 reasoning=0 cache_read=21712 cache_write=1657 cached_total=23369 cost=$0.1179
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=63/80 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Judging Conditions
- critical evidence: impact=10 desc=Output JSON top nodes and average read IOPS match the runtime reference ordering and values.
Bucket Failure Details
- critical evidence: partial score=63/80
-   - top_nodes_match_runtime_reference earned=0/10 desc=Output JSON top nodes and average read IOPS match the runtime reference ordering and values. rules=output_json/json_array_matches_reference/matched=no/actual=[]
Raw Output
<<<
根据技能路由表，`ad-hoc topsql-links` 可以按 read IOPS 排名 TiKV 节点。先查看该 recipe 的帮助，然后执行。```json
{
  "cluster_id": "10518506906906971664",
  "window": {
    "start": "2026-07-21T03:00:00Z",
    "end": "2026-07-21T03:05:00Z"
  },
  "mapping_strategy": "node_filter",
  "evidence_sufficient": true,
  "top_nodes_by_avg_read_iops": [
    {
      "tikv_node": "db-tikv-48",
      "avg_read_iops": 8288.71
    },
    {
      "tikv_node": "db-tikv-20",
      "avg_read_iops": 7857.46
    },
    {
      "tikv_node": "db-tikv-52",
      "avg_read_iops": 7615.07
    }
  ]
}
```
>>>

Case: top-cpu-overload-nodes-and-log-patterns-in-saas01-during-latest-window
- execution_status: completed
- verdict: pass
- score: current=100 baseline=- diff=-
- failure_code: -
- invalid_subtype: -
- workspace: ws-eval-1784628628244-6b238c69
- time: started=2026-07-21T10:10:26.675833+00:00 finished=2026-07-21T10:11:37.304355+00:00 duration=1m11s
- token usage: input=1 output=809 reasoning=0 cache_read=22173 cache_write=4453 cached_total=26626 cost=$0.1387
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=80/80 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Raw Output
<<<
The recipe `ad-hoc cpu-overload-logs` is exactly what's needed here. Let me run it directly.注意到 tidb 节点名带有 `-ac-2006050899364941824` 后缀（这是 nodepool 标识，不是 K8s pod 名的标准形式）。需确认 pod 名是否即为带后缀的完整名称，由于 metrics 里的 `pod` label 就是这个值，结果直接使用即可。

```json
{
  "cluster_id": "10989049060142230334",
  "window": {
    "start": "2026-07-21T03:00:00Z",
    "end": "2026-07-21T03:05:00Z"
  },
  "evidence_sufficient": true,
  "ranked_nodes": [
    {
      "component": "tidb",
      "rank": 1,
      "node": "db-tidb-3-ac-2006050899364941824",
      "avg_cpu_overload": 0.3979,
      "error_warn_log_count": 0
    },
    {
      "component": "tidb",
      "rank": 2,
      "node": "db-tidb-17-ac-2006050899364941824",
      "avg_cpu_overload": 0.3425,
      "error_warn_log_count": 0
    },
    {
      "component": "tidb",
      "rank": 3,
      "node": "db-tidb-10-ac-2006050899364941824",
      "avg_cpu_overload": 0.3415,
      "error_warn_log_count": 0
    },
    {
      "component": "tikv",
      "rank": 1,
      "node": "db-tikv-30",
      "avg_cpu_overload": 0.5474,
      "error_warn_log_count": 0
    },
    {
      "component": "tikv",
      "rank": 2,
      "node": "db-tikv-44",
      "avg_cpu_overload": 0.5411,
      "error_warn_log_count": 0
    },
    {
      "component": "tikv",
      "rank": 3,
      "node": "db-tikv-138",
      "avg_cpu_overload": 0.5360,
      "error_warn_log_count": 0
    },
    {
      "component": "pd",
      "rank": 1,
      "node": "db-pd-0",
      "avg_cpu_overload": 0.1995,
      "error_warn_log_count": 285
    },
    {
      "component": "pd",
      "rank": 2,
      "node": "db-pd-1",
      "avg_cpu_overload": 0.0520,
      "error_warn_log_count": 1
    },
    {
      "component": "pd",
      "rank": 3,
      "node": "db-pd-2",
      "avg_cpu_overload": 0.0493,
      "error_warn_log_count": 2
    }
  ]
}
```
>>>

Case: top-tidb-p99-latency-nodes-in-saas01-during-latest-window
- execution_status: completed
- verdict: pass
- score: current=100 baseline=- diff=-
- failure_code: -
- invalid_subtype: -
- workspace: ws-eval-1784685468326-179011a2
- time: started=2026-07-22T01:57:46.479228+00:00 finished=2026-07-22T01:58:35.767877+00:00 duration=0m49s
- token usage: input=1 output=612 reasoning=0 cache_read=26578 cache_write=987 cached_total=27565 cost=$0.1456
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=80/80 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Raw Output
<<<
This is a `tidb-p99-breakdown` ad-hoc recipe. Let me check the recipe help first, then run it.Recipe returned all required fields with `evidence_sufficient=true`. Here is the result:

```json
{
  "cluster_id": "10989049060142230334",
  "window": {
    "start": "2026-07-21T03:00:00Z",
    "end": "2026-07-21T03:05:00Z"
  },
  "evidence_sufficient": true,
  "cluster_avg_tikv_grpc_request_handle_p99_s": 0.013188,
  "cluster_avg_pd_tso_request_handle_p99_s": 0.000495,
  "top_nodes_by_avg_p99_latency": [
    {
      "node": "db-tidb-88",
      "avg_p99_latency_s": 0.056485,
      "avg_plan_build_p99_s": 0.012286,
      "avg_distributed_execution_p99_s": 0.050407,
      "avg_kv_request_p99_s": 0.013411,
      "avg_get_tso_wait_p99_s": 0.002630,
      "avg_gc_stw_pause_p99_s": 0.023696
    },
    {
      "node": "db-tidb-4",
      "avg_p99_latency_s": 0.056273,
      "avg_plan_build_p99_s": 0.013471,
      "avg_distributed_execution_p99_s": 0.051131,
      "avg_kv_request_p99_s": 0.013630,
      "avg_get_tso_wait_p99_s": 0.002163,
      "avg_gc_stw_pause_p99_s": 0.038845
    },
    {
      "node": "db-tidb-82",
      "avg_p99_latency_s": 0.056241,
      "avg_plan_build_p99_s": 0.013116,
      "avg_distributed_execution_p99_s": 0.053487,
      "avg_kv_request_p99_s": 0.013686,
      "avg_get_tso_wait_p99_s": 0.002666,
      "avg_gc_stw_pause_p99_s": 0.010284
    }
  ]
}
```
>>>

Case: topsql-links-for-hottest-two-tikv-nodes-during-incident-window
- execution_status: completed
- verdict: fail
- score: current=69 baseline=- diff=-
- failure_code: critical_evidence_partial
- invalid_subtype: -
- workspace: ws-eval-1784628476449-a1d374b1
- time: started=2026-07-21T10:07:54.729254+00:00 finished=2026-07-21T10:08:19.744162+00:00 duration=0m25s
- token usage: input=1 output=510 reasoning=0 cache_read=21737 cache_write=1268 cached_total=23005 cost=$0.1202
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=55/80 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=0/0 baseline=- diff=-
Judging Conditions
- critical evidence: impact=10 desc=Output JSON TopSQL URLs match the runtime reference semantics for cluster, window, and instance.
- critical evidence: impact=7 desc=Output JSON node ordering matches the runtime reference.
Bucket Failure Details
- critical evidence: partial score=55/80
-   - topsql_link_nodes_match_runtime_reference earned=0/7 desc=Output JSON node ordering matches the runtime reference. rules=output_json/json_array_matches_reference/matched=no/actual=[]
-   - topsql_urls_match_runtime_reference earned=0/10 desc=Output JSON TopSQL URLs match the runtime reference semantics for cluster, window, and instance. rules=output_json/topsql_url_semantically_matches_reference/matched=no/actual=[]
Raw Output
<<<
`ad-hoc topsql-links` 正好对应这个场景，直接跑。```json
{
  "cluster_id": "10518506906906971664",
  "window": {
    "start": "2026-07-21T03:00:00Z",
    "end": "2026-07-21T03:05:00Z"
  },
  "mapping_strategy": "node_filter",
  "evidence_sufficient": true,
  "topsql_links": [
    {
      "tikv_node": "db-tikv-48",
      "url": "https://clinic.pingcap.com/portal/dashboard/cloud/ngm.html?clusterId=10518506906906971664&provider=aws&region=us-east-1&orgName=Buffet%27s%20Org&orgId=1372813089209061633&projectId=1372813089454544954&clusterName=prod-us-saas-gosaas-region-tidb-v85-02&deployType=dedicated&version=v8.5.4#/topsql?from=1784602800&to=1784603100&instance=db-tikv-48.db-tikv-peer.tidb10518506906906971664.svc%3A20160"
    },
    {
      "tikv_node": "db-tikv-20",
      "url": "https://clinic.pingcap.com/portal/dashboard/cloud/ngm.html?clusterId=10518506906906971664&provider=aws&region=us-east-1&orgName=Buffet%27s%20Org&orgId=1372813089209061633&projectId=1372813089454544954&clusterName=prod-us-saas-gosaas-region-tidb-v85-02&deployType=dedicated&version=v8.5.4#/topsql?from=1784602800&to=1784603100&instance=db-tikv-20.db-tikv-peer.tidb10518506906906971664.svc%3A20160"
    }
  ]
}
```
>>>

Case: ttl-row-processing-during-sql-tail-latency-incident-window
- execution_status: completed
- verdict: pass
- score: current=73 baseline=- diff=-
- failure_code: -
- invalid_subtype: -
- workspace: ws-eval-1784628958524-c18f0927
- time: started=2026-07-21T10:15:56.813701+00:00 finished=2026-07-21T10:16:20.208421+00:00 duration=0m23s
- token usage: input=1 output=137 reasoning=0 cache_read=21690 cache_write=494 cached_total=22184 cost=$0.1136
Bucket Scores vs Baseline
- workflow: current=0/0 baseline=- diff=-
- evidence collection: current=5/5 baseline=- diff=-
- critical evidence: current=0/0 baseline=- diff=-
- evidence routing: current=0/0 baseline=- diff=-
- diagnosis: current=35/50 baseline=- diff=-
Raw Output
<<<
The `ad-hoc ttl-rows` recipe is exactly the right path. Let me run it directly.```json
{
  "cluster_id": "10989049060142230334",
  "window": {
    "start": "2026-07-21T03:00:00Z",
    "end": "2026-07-21T03:05:00Z"
  },
  "evidence_sufficient": true,
  "ttl_activity_present": false,
  "signals": {
    "ttl_insert_rows_delta": 0,
    "ttl_processed_expired_rows_delta": 0
  }
}
```
>>>