Evaluation Run Detail Report Run ID: evalrun-adhoc-evidence-premium-demo-merged-20260721-afternoon Suite: adhoc-evidence-premium-demo @ 2214c0c56eb6 Run Metadata - execution_status: completed - comparison_status: review_only - trigger_mode: manual_merge - schedule_date: - - started_at: 2026-07-21T07:00:41.746701+00:00 - finished_at: 2026-07-21T09:17:56.585121+00:00 Runtime Metadata - agent_type: opencode - resolved_model: chat/claude-standard/high - promote_eligibility: not_evaluated - baseline_id: - Key Metrics - counts: cases=10 pass=5 fail=5 invalid=0 - score: avg=64.40 scored_cases=10/10 - run_duration: 2h17m15s - case_duration: merged_from_multiple_runs - token_usage: input_total=10 output_total=4818 reasoning_total=0 cache_read_total=370192 cache_write_total=25196 cached_total=395388 cost_total=$3.3198 Source Runs - evalrun-dca9124193b04b2c873027cd9fb2ca6f - evalrun-9f9830380cfa4796ad7e093d293528af - evalrun-a6921afa263e4be1a7c1bdcc298bbe72 Case Details Case: classify-selected-stores-as-tikv-or-tiflash-in-premium-cluster - source_run: evalrun-dca9124193b04b2c873027cd9fb2ca6f - execution_status: completed - verdict: fail - score: current=16 baseline=- diff=- - failure_code: critical_evidence_partial - invalid_subtype: - - workspace: ws-eval-1784617243353-88cd4ec7 - time: started=2026-07-21T07:00:41.746701+00:00 finished=2026-07-21T07:01:35.700266+00:00 duration=0m54s - token usage: input=1 output=331 reasoning=0 cache_read=22809 cache_write=5061 cached_total=27870 cost=$0.1531 Bucket Scores vs Baseline - workflow: current=0/10 baseline=- diff=- - evidence collection: current=5/5 baseline=- diff=- - critical evidence: current=11/80 baseline=- diff=- - evidence routing: current=0/5 baseline=- diff=- - diagnosis: current=0/50 baseline=- diff=- Raw Output <<< Store count returned 0 — the Premium cluster may route through a parent/resource-pool. Let me get the cluster metadata to find the parent cluster.This is a Premium cluster with `parentID: 2026601847170531328`. I need to query store roles on the parent cluster.注意：用户请求的 store `"4"` 在父集群数据中未找到（找到的是 `"5"`，不是 `"4"`），`"805763"` 也未找到（找到的是 `"805762"`）。结果按用户给定顺序输出，缺失的 store 证据不足。 不过仔细对照用户列表与返回数据： - `"194"` → tikv ✓ - `"805763"` → 未找到（最近的是 `805762`，但 ID 不同） - `"1115"` → tikv ✓ - `"4"` → 未找到（有 `"5"` 和 `"1"` 等，没有 `"4"`） - `"1114"` → tikv ✓ - `"184"` → tiflash ✓ `"805763"` 和 `"4"` 不在返回结果中，证据不足以确认其类型，需将 `evidence_sufficient` 设为 `false`。 ```json { "cluster_id": "10530333593343689954", "evidence_sufficient": false, "stores": [] } ``` >>> Case: new-slow-query-digests-between-adjacent-windows-in-premium-cluster - source_run: evalrun-dca9124193b04b2c873027cd9fb2ca6f - execution_status: completed - verdict: pass - score: current=85 baseline=- diff=- - failure_code: - - invalid_subtype: - - workspace: ws-eval-1784617297384-4da98606 - time: started=2026-07-21T07:01:35.712182+00:00 finished=2026-07-21T07:03:40.630841+00:00 duration=2m05s - token usage: input=1 output=2288 reasoning=0 cache_read=21854 cache_write=9145 cached_total=30999 cost=$0.1790 Bucket Scores vs Baseline - workflow: current=0/10 baseline=- diff=- - evidence collection: current=5/5 baseline=- diff=- - critical evidence: current=80/80 baseline=- diff=- - evidence routing: current=0/5 baseline=- diff=- - diagnosis: current=0/50 baseline=- diff=- Raw Output <<< The recipe uses `exec_count` ordering and aggregates by digest — perfect. Let me run both windows in parallel.两窗口数据均已取回。现在做对比： **Previous top10 digests（按 exec_count 排序）：** 1. `6243b62cf59d883f0bf3f3f943ca108355b33f7aa8db3c5d2ca23d2a9cc4940d` (62) 2. `d61abc214847b7aeb2de7fb205343d153517065f19109612836802574604fe6c` (62) 3. `0d1a468cd9b29441a4714ce448e141c49fc67b065405cede39555c1633f3b086` (59) 4. `d3a331c39f5c3d0e4442f854ef6b23739023718970c86d064f267a3b87528129` (52) 5. `a46254ac8d0d4e725f900443ea8abea91591509bce8af4a124b5a901c3c8b4f0` (35) 6. `fba5144eb034c3c24fc77a6b2bf1514fb8bfdc4756b01b7aa1d0d1820a87703e` (27) 7. `08975adb0ae99089ef88a60a548864a9e3bb6c175bafaae8fd98df7f755f6451` (26) 8. `e963d32fce590067bb97303e8a99b6ae9d660b6bfad4dc0ad7611bf46453bef0` (7) 9. `9505cacb7c710ed17125fcc6cb3669e8ddca6c8cd8af6a31f6b3cd64604c3098` (5) 10. `7383428efdc2e7f142bf2aa6883b3b28c610d6013d0ddf693e0b75b9ce18de67` (2) **Current top10 digests：** `992377525bc78cf77741e870a6a509f866749e322c5baf19d55703fc26970577`、`bda1f10c99c3bfba07501f9579aeedaff16fcde326a3e99d75dd75adfd880ca6`、`d94be824b6c9bd25dd280afec2e0dda0e5b6b8f38d0f912d085a02f137f3130a`、`004e25591e3f89e48a9f851d43b1bb0d82d6dd7195e7300ba814a7d7aad5f8ba`、`a9cde48f6a24501b29025d1bcaea103777ac4429ba05a921cbe341de425df1e7`、`a46254ac8d0d4e725f900443ea8abea91591509bce8af4a124b5a901c3c8b4f0`、`4e9e2c20c79b6a52fc2c41dccdfb6009285959e518449ed759769fb7726f07dc`、`2d8a2f5b6ca9720258c125633d77e2c85c6586e38e115a662588c9263c82a6c3` 其中 `a46254ac...` 在 previous top10 中存在（第5位），其余7个均为新增，按 `occurrence_count` 降序、`sql_digest` 字典序排序： 1. `992377525...` 30 次 2. `bda1f10c...` 22 次 3. `d94be824...` 5 次 4. `004e2559...` 4 次 5. `a9cde48f...` 1 次 → 字典序比较 `a9` vs `4e` vs `2d` 6. `4e9e2c20...` 1 次 7. `2d8a2f5b...` 1 次 occurrence_count=1 的三个按 digest 字典序：`2d8a2f5b` < `4e9e2c20` < `a9cde48f` ```json { "cluster_id": "10748116073341772391", "comparison": { "previous_window": { "start": "2026-07-21T02:50:00Z", "end": "2026-07-21T03:00:00Z" }, "current_window": { "start": "2026-07-21T03:00:00Z", "end": "2026-07-21T03:10:00Z" } }, "evidence_sufficient": true, "new_sql_digests": [ { "sql_digest": "992377525bc78cf77741e870a6a509f866749e322c5baf19d55703fc26970577", "sql_statement": "SELECT LOW_PRIORITY SQL_NO_CACHE `id` FROM `c66_seamless_others`.`t_seamless_log_aqua` WHERE `id` > 1152921504665110825 AND `id` < 2305843009250084684 AND `created_time` < FROM_UNIXTIME(177933", "occurrence_count": 30 }, { "sql_digest": "bda1f10c99c3bfba07501f9579aeedaff16fcde326a3e99d75dd75adfd880ca6", "sql_statement": "SELECT LOW_PRIORITY SQL_NO_CACHE `id` FROM `c66_seamless_others`.`t_seamless_log_inbetween` WHERE `id` > 4899916394593024407 AND `id` < 6341068275349384987 AND `created_time` < FROM_UNIXTIME(177", "occurrence_count": 22 }, { "sql_digest": "d94be824b6c9bd25dd280afec2e0dda0e5b6b8f38d0f912d085a02f137f3130a", "sql_statement": "SELECT LOW_PRIORITY SQL_NO_CACHE `id` FROM `c66_seamless_others`.`t_seamless_log_inbetween` WHERE `id` >= 4899916394580972940 AND `id` < 6341068275349384987 AND `created_time` < FROM_UNIXTIME(17", "occurrence_count": 5 }, { "sql_digest": "004e25591e3f89e48a9f851d43b1bb0d82d6dd7195e7300ba814a7d7aad5f8ba", "sql_statement": "SELECT LOW_PRIORITY SQL_NO_CACHE `id` FROM `c66_seamless_others`.`t_seamless_log_aqua` WHERE `id` > 8070450532306169480 AND `created_time` < FROM_UNIXTIME(1779332471) ORDER BY `id` ASC LIMIT 500;", "occurrence_count": 4 }, { "sql_digest": "2d8a2f5b6ca9720258c125633d77e2c85c6586e38e115a662588c9263c82a6c3", "sql_statement": "SELECT LOW_PRIORITY SQL_NO_CACHE `id` FROM `c66_seamless_others`.`t_seamless_log_aqua` WHERE `id` >= 3458764513864257921 AND `id` < 4611686018471382252 AND `created_time` < FROM_UNIXTIME(177993247", "occurrence_count": 1 }, { "sql_digest": "4e9e2c20c79b6a52fc2c41dccdfb6009285959e518449ed759769fb7726f07dc", "sql_statement": "SELECT LOW_PRIORITY SQL_NO_CACHE `id` FROM `c66_seamless_others`.`t_seamless_log_aqua` WHERE `id` >= 8070450532289975596 AND `created_time` < FROM_UNIXTIME(1779332471) ORDER BY `id` ASC LIMIT 500;", "occurrence_count": 1 }, { "sql_digest": "a9cde48f6a24501b29025d1bcaea103777ac4429ba05a921cbe341de425df1e7", "sql_statement": "SELECT LOW_PRIORITY SQL_NO_CACHE `id` FROM `c66_seamless_others`.`t_seamless_log_inbetween` WHERE `id` < 1441151880767086829 AND `created_time` < FROM_UNIXTIME(1779332464) ORDER BY `id` ASC LIMIT 50", "occurrence_count": 1 } ] } ``` >>> Case: premium-keyspace-storage-query-semantics - source_run: evalrun-dca9124193b04b2c873027cd9fb2ca6f - execution_status: completed - verdict: fail - score: current=56 baseline=- diff=- - failure_code: critical_evidence_partial - invalid_subtype: - - workspace: ws-eval-1784617422348-71a05571 - time: started=2026-07-21T07:03:40.640888+00:00 finished=2026-07-21T07:05:49.850286+00:00 duration=2m09s - token usage: input=1 output=340 reasoning=0 cache_read=33893 cache_write=391 cached_total=34284 cost=$0.3270 Bucket Scores vs Baseline - workflow: current=0/10 baseline=- diff=- - evidence collection: current=5/5 baseline=- diff=- - critical evidence: current=30/80 baseline=- diff=- - evidence routing: current=0/5 baseline=- diff=- - diagnosis: current=21/50 baseline=- diff=- Raw Output <<< This is a Premium cluster. Per tier-architecture, TiKV metrics use `parent_id` as `cluster_id`. I need to query `tikv_store_size_bytes{type="used"}` using `parent_id=2026948324023603200` to discover the label set, then identify which label represents keyspace. Let me first check the metrics subskill for Premium TiKV query guidance, then run the query.Zero series with instance filter — the instance label value might not match exactly. Let me try without the instance filter first to see all label names, and also try with a broader query.1 series returned. Let me read the metrics.json to see the full label set of this series.The `--inline-chart-data` flag was used but series labels are empty in the row. The raw chart data with per-series labels might be in the file differently. Let me check the raw structure more carefully.I can see the label `keyspace_name` in the metric. Now the result only returns one series on a different instance (`db-125d9e7b-tikv-uinnfa`) but not the target `db-125d9e7b-tikv-pmbces`. The target instance may have data earlier in the window. Let me query across the full window and filter for the target instance using the `instance` or `pod` label.The target instance `pmbces` doesn't appear at the window end but `uinnfa` does. Let me query the full window with a broader step to see all series across the entire window, including all instances with the `keyspace_name` label.Only one series (`db-125d9e7b-tikv-uinnfa`) exists in the entire window. The target `db-125d9e7b-tikv-pmbces` has no data in `tikv_store_size_bytes{type="used"}` during this window. However, the question says to count distinct `keyspace_name` values for the `target_tikv_instance` at the end of the window. The metric aggregates by `keyspace_name` — there might be multiple series per instance if different keyspaces are present. Let me now query specifically with a `by(keyspace_name)` grouping to see if multiple keyspace values exist on the `pmbces` instance, perhaps using a different time range approach. Let me also check if `pmbces` appears at the start of the window.Now I have all the information needed. Let me summarize the findings: 1. **Keyspace label**: `keyspace_name` — discovered directly from the metric's label set. 2. **Target instance `db-125d9e7b-tikv-pmbces`**: No data series for this specific pod in the observation window. The only active TiKV instance with `tikv_store_size_bytes{type="used"}` data is `db-125d9e7b-tikv-uinnfa`, which has 1 distinct `keyspace_name` value (`SYSTEM`). 3. The `count by(keyspace_name)` at the window end shows **1** distinct value globally across all instances in this pool. Since `db-125d9e7b-tikv-pmbces` returns 0 series in the window, it has no data — the evidence is insufficient to count its distinct keyspace values directly. However, the broader pool shows 1 distinct `keyspace_name` value. The target instance has **no direct metric data** in this window. ```json { "target_metric": "tikv_store_size_bytes", "window": { "start": "2026-07-21T03:00:00Z", "end": "2026-07-21T03:05:00Z" }, "evidence_sufficient": false, "clusters": [] } ``` >>> Case: read-pool-cpu-scope-compare-in-premium-cluster-during-latest-window - source_run: evalrun-dca9124193b04b2c873027cd9fb2ca6f - execution_status: completed - verdict: pass - score: current=100 baseline=- diff=- - failure_code: - - invalid_subtype: - - workspace: ws-eval-1784617551544-6c16bf5b - time: started=2026-07-21T07:05:49.863040+00:00 finished=2026-07-21T07:09:01.070381+00:00 duration=3m11s - token usage: input=1 output=173 reasoning=0 cache_read=72152 cache_write=2099 cached_total=74251 cost=$0.7376 Bucket Scores vs Baseline - workflow: current=0/10 baseline=- diff=- - evidence collection: current=5/5 baseline=- diff=- - critical evidence: current=80/80 baseline=- diff=- - evidence routing: current=0/5 baseline=- diff=- - diagnosis: current=35/50 baseline=- diff=- Raw Output <<< For a Premium cluster, TiKV metrics use `parent_id` as the `cluster_id`. Let me first get the cluster metadata to find the `parent_id` (metrics_cluster_id).`parent_id` = `2026948324023603200`. Now query `tikv_unified_read_pool_cpu` using the parent_id as the metrics cluster_id.Now let me look at the metrics subskill to understand the correct `tikv_unified_read_pool_cpu` PromQL:Now I have the PromQL for `tikv_unified_read_pool_cpu`. The metric uses `tikv_thread_cpu_seconds_total` with `name=~"unified-read.*"`. Let me check the metrics subskill for the exact query, then execute the metrics call:Now I have everything I need. The `tikv_unified_read_pool_cpu` metric is derived from `tikv_thread_cpu_seconds_total` with `name=~"unified-read.*"`. Let me now execute the metrics query via `clinic_cli.py metrics`:The query with `instance` label returned 0 series. The Premium cluster's TiKV metrics may use `pod` label. Let me try querying with a join against `kube_pod_info` to get pod names, or try using the `pod` label directly:The metric exists. The issue is the label filter. Let me check what labels are available:The metric uses `instance` label (which equals `pod`). The thread name is `name`. I need to filter `name=~"unified-read.*"`. The issue before was that the `{name=~"unified-read.*"}` label selector produced no result. Let me check what unified-read thread names exist:The thread name pattern is `unified_read_pool_*` (with underscores, not hyphens). The correct filter is `name=~"unified-read.*"` doesn't match — it should be `name=~"unified_read_pool.*"`. Now let me run the actual query:Now I have data. Let me compute the peak per-instance and cluster total:```json { "cluster_id": "10748116073341772391", "metrics_cluster_id": "2026948324023603200", "window": { "start": "2026-07-20T07:10:00Z", "end": "2026-07-20T08:00:00Z" }, "evidence_sufficient": true, "peak_instance_node": "db-125d9e7b-tikv-3xh796", "peak_instance_read_pool_cpu_cores": 7.2665, "cluster_total_peak_read_pool_cpu_cores": 29.7647 } ``` >>> Case: representative-business-slow-queries-in-premium-cluster-during-latest-window - source_run: evalrun-a6921afa263e4be1a7c1bdcc298bbe72 - execution_status: completed - verdict: pass - score: current=100 baseline=- diff=- - failure_code: - - invalid_subtype: - - workspace: ws-eval-1784625405607-0089da6b - time: started=2026-07-21T09:16:44.120974+00:00 finished=2026-07-21T09:17:56.585121+00:00 duration=1m12s - token usage: input=1 output=463 reasoning=0 cache_read=21725 cache_write=4221 cached_total=25946 cost=$0.1304 Bucket Scores vs Baseline - workflow: current=0/0 baseline=- diff=- - evidence collection: current=5/5 baseline=- diff=- - critical evidence: current=80/80 baseline=- diff=- - evidence routing: current=0/0 baseline=- diff=- - diagnosis: current=0/0 baseline=- diff=- Raw Output <<< 所有 8 条记录均为业务查询（`SELECT`），无事务控制语句（COMMIT/BEGIN/ROLLBACK 等），按 `exec_count` 降序取前 3，精确到小数点后 3 位： - `992377525…` exec_count=90，avg_query_time≈0.41354s - `bda1f10c…` exec_count=66，avg_query_time≈0.71446s - `d94be824…` exec_count=15，avg_query_time≈0.44922s ```json { "cluster_id": "10748116073341772391", "window": { "start": "2026-07-21T03:00:00Z", "end": "2026-07-21T03:05:00Z" }, "evidence_sufficient": true, "slow_queries": [ { "digest": "992377525bc78cf77741e870a6a509f866749e322c5baf19d55703fc26970577", "exec_count": 90, "avg_time_s": 0.414 }, { "digest": "bda1f10c99c3bfba07501f9579aeedaff16fcde326a3e99d75dd75adfd880ca6", "exec_count": 66, "avg_time_s": 0.714 }, { "digest": "d94be824b6c9bd25dd280afec2e0dda0e5b6b8f38d0f912d085a02f137f3130a", "exec_count": 15, "avg_time_s": 0.449 } ] } ``` >>> Case: single-node-cpu-core-scope-compare-in-premium-cluster-during-incident-window - source_run: evalrun-dca9124193b04b2c873027cd9fb2ca6f - execution_status: completed - verdict: fail - score: current=56 baseline=- diff=- - failure_code: critical_evidence_partial - invalid_subtype: - - workspace: ws-eval-1784617798848-469bdb0f - time: started=2026-07-21T07:09:57.218883+00:00 finished=2026-07-21T07:13:07.936104+00:00 duration=3m11s - token usage: input=1 output=379 reasoning=0 cache_read=47112 cache_write=907 cached_total=48019 cost=$0.4925 Bucket Scores vs Baseline - workflow: current=0/10 baseline=- diff=- - evidence collection: current=5/5 baseline=- diff=- - critical evidence: current=51/79 baseline=- diff=- - evidence routing: current=0/5 baseline=- diff=- - diagnosis: current=0/50 baseline=- diff=- Raw Output <<< Now I need to get the metrics CLI help and then query CPU metrics for the specific node. Let me first check the metrics subcommand help and also get cluster metadata to find CPU spec.This is a Premium cluster with a `parentID`. I need to query the parent cluster for topology/component metadata. Let me check the parent cluster and also start querying the CPU metrics for the specific node in all three windows simultaneously.The `pod` label filter returned 0 series. For Premium clusters, the node label may be `instance` or `kubernetes_pod_name`. Let me check what labels are available by querying without the pod filter, and also try `instance` label.The child cluster returns 0 series. For Premium clusters, metrics are routed to the parent/resource-pool cluster. Let me query via the parent cluster ID.The tier architecture confirms: for Premium clusters, TiKV metrics use **`parent_id`** as the cluster_id. The parent_id is `2026948324023603200`. The previous query to the parent returned 0 series too. Let me try with `node_cpu_seconds_total` (node-level metric) and also check what label format is used for the specific node pod.There are 42 series with `process_cpu_seconds_total` on the parent. Now I need to find the instance label for `db-125d9e7b-tikv-6diswl`. Let me read the metrics.json to see the instance labels.Now I can see the label format. The instance label uses `pod` or `instance`. The node name is `db-125d9e7b-tikv-6diswl`. Let me check which pods match TiKV and find this specific one.`db-125d9e7b-tikv-6diswl` is confirmed present. Now I need to: 1. Query CPU for this specific pod across the 3 windows with `--inline-chart-data` 2. Get the CPU spec cores via cluster metadata Let me query all 3 windows and also get kube CPU limit for this node.All 3 windows have data (61 points each, 1 series). The CPU limit query returned 0 — I need to try a different approach for the spec. Let me read the raw datapoints from metrics.json and also try other ways to get CPU spec.`tikv_server_cpu_cores_quota` returns **31.0** consistently. However the user asked for CPU spec cores — this is the TiKV-reported CPU quota. Let me also check the `ad-hoc cpu-overload-logs` to see if there's a cleaner way to get node-level spec, but the `tikv_server_cpu_cores_quota` value of 31 is the authoritative TiKV-reported core quota. Now let me verify and compile the final result.All data is confirmed. The `tikv_server_cpu_cores_quota` is consistently **31** (not 32 — this is the actual TiKV-reported core quota for this pod). The CPU usage values are in cores (from `rate(process_cpu_seconds_total[2m])`). ```json { "cluster_id": "10748116073341772391", "node": "db-125d9e7b-tikv-6diswl", "cpu_spec_cores": 31.0000, "evidence_sufficient": true, "windows": [ { "start": "2026-07-21T03:00:00Z", "end": "2026-07-21T03:30:00Z", "avg_cpu_cores": 5.7784, "peak_cpu_cores": 6.5417 }, { "start": "2026-07-20T03:00:00Z", "end": "2026-07-20T03:30:00Z", "avg_cpu_cores": 4.1805, "peak_cpu_cores": 4.4833 }, { "start": "2026-07-19T03:00:00Z", "end": "2026-07-19T03:30:00Z", "avg_cpu_cores": 4.9258, "peak_cpu_cores": 5.2167 } ] } ``` >>> Case: throttle-refill-rate-verification-after-change-window - source_run: evalrun-9f9830380cfa4796ad7e093d293528af - execution_status: completed - verdict: pass - score: current=100 baseline=- diff=- - failure_code: - - invalid_subtype: - - workspace: ws-eval-1784624523538-753849df - time: started=2026-07-21T09:02:02.019350+00:00 finished=2026-07-21T09:06:31.350636+00:00 duration=4m29s - token usage: input=1 output=330 reasoning=0 cache_read=82624 cache_write=1297 cached_total=83921 cost=$0.9426 Bucket Scores vs Baseline - workflow: current=0/10 baseline=- diff=- - evidence collection: current=5/5 baseline=- diff=- - critical evidence: current=82/82 baseline=- diff=- - evidence routing: current=0/5 baseline=- diff=- - diagnosis: current=35/50 baseline=- diff=- Raw Output <<< Now let me check the metrics subskill for TiKV resource control metrics:No resource control metrics in the pre-cataloged docs. Let me check the metrics help and query for the relevant TiKV resource control metric directly:That metric name didn't match. Let me try other likely metric names:Let me look at TiKV resource control metrics more carefully from the known metric schema and try a broader search:I found the key metric: `tikv_resource_control_priority_quota_limit`. Let me also check for the specific refill rate metric for the low-priority DAG throttle:Now let me look at the `tikv_resource_control_priority_quota_limit` metric in detail and search for any metrics specifically related to the refill rate for low-priority DAG:Still empty. This is a Premium cluster — let me check the cluster metadata and understand its topology first to determine the correct metric routing:This is a Premium resource pool cluster (`deployTypeV2: "premium resource pool"`) with no topology available. The key metadata: `parentID` is empty, which means it's the resource pool itself. Let me check the tidb-knowledge-base for Premium metric routing rules:Critical finding: for TiKV metrics on a Premium cluster, I need to use the `parent_id` (resource pool ID), not the premium cluster_id. The metadata shows `parentID: ""` which means this cluster IS the resource pool. Let me verify this and check TiKV metrics using the cluster's own ID since it's the resource pool:Excellent! Now I have confirmation: - `2026948324023603200` is the **resource pool** (`premium resource pool`) - `10748116073341772391` (`prod-c66-seamless-others`) is a child Premium cluster with `parent_id=2026948324023603200` According to tier-architecture, for TiKV metrics in Premium, I need to use the **parent_id** (resource pool ID) as the `cluster_id`. Since the resource pool itself IS `2026948324023603200`, TiKV metrics should route to this same ID. The metrics returning empty may be because the metric name is wrong. Let me try broader PromQL with wildcard matching:The key note from the SUBSKILL: **"IMPORTANT: `tidb_cluster_id` is auto-injected for non-starter/essential clusters. If you suspect the metric is from shared Resource Pool components (Premium/BYOC), and the query returns no data, try adding `{tidb_cluster_id="{parentID}"}` manually."** Since `2026948324023603200` is the resource pool itself (parentID is empty), TiKV metrics routing for this pool's cluster_id should work. But the metrics keep returning empty. Let me try using the Python ClinicAPI directly to do series discovery and check what resource control metrics actually exist:TiKV metrics do exist for this cluster. Now let me try discovering `tikv_resource_control_priority_quota_limit` with a prefix search:Excellent! Found it! The exact metric is `tikv_cpu_throttle_low_priority_dag_refill_rate_us_per_interval`. Now let me query this metric over the window to observe the actual values:The observed refill rate is consistently **992,000 µs/interval** across all 24 TiKV instances throughout the entire window. Now let me also check the global refill rate to understand what the expected value should be for ratio=0.5:Now I have all the data needed. Let me compute the expected value: - Global refill rate = **2,480,000 µs/interval** (this is the total CPU capacity in µs per interval available to TiKV) - `low-priority-dag-global-ratio = 0.5` means the low-priority DAG bucket should get 0.5 × 2,480,000 = **1,240,000 µs/interval** - Observed refill rate = **992,000 µs/interval** (constant throughout entire window, including after the 22:00 UTC change time) The observed value of 992,000 does NOT match the expected 1,240,000 for ratio=0.5. The config change did not take effect on TiKV. ```json { "cluster_id": "2026948324023603200", "window": { "start": "2026-07-06T21:50:00Z", "end": "2026-07-07T01:00:00Z" }, "requested_ratio": 0.5, "expected_refill_rate_us_per_interval": 1240000, "observed_refill_rate_us_per_interval": 992000, "applied": false, "evidence_sufficient": true } ``` >>> Case: top-cpu-overload-nodes-and-log-patterns-in-premium-cluster-during-latest-window - source_run: evalrun-dca9124193b04b2c873027cd9fb2ca6f - execution_status: completed - verdict: fail - score: current=21 baseline=- diff=- - failure_code: critical_evidence_partial - invalid_subtype: - - workspace: ws-eval-1784618267343-4a2d5ff3 - time: started=2026-07-21T07:17:45.834947+00:00 finished=2026-07-21T07:19:07.721911+00:00 duration=1m22s - token usage: input=1 output=139 reasoning=0 cache_read=23065 cache_write=919 cached_total=23984 cost=$0.1207 Bucket Scores vs Baseline - workflow: current=0/10 baseline=- diff=- - evidence collection: current=5/5 baseline=- diff=- - critical evidence: current=16/80 baseline=- diff=- - evidence routing: current=0/5 baseline=- diff=- - diagnosis: current=0/50 baseline=- diff=- Raw Output <<< The `ad-hoc cpu-overload-logs` recipe is exactly what I need. Let me run it directly.The recipe returned `evidence_sufficient=false` with 0 observed nodes for all components — no metrics data was returned for this cluster in this window. The recipe is authoritative on this. Here is the result: ```json { "cluster_id": "10748116073341772391", "window": { "start": "2026-07-21T03:00:00Z", "end": "2026-07-21T03:05:00Z" }, "evidence_sufficient": false, "ranked_nodes": [] } ``` >>> Case: top-tidb-p99-latency-nodes-in-premium-cluster-during-latest-window - source_run: evalrun-dca9124193b04b2c873027cd9fb2ca6f - execution_status: completed - verdict: fail - score: current=40 baseline=- diff=- - failure_code: critical_evidence_partial - invalid_subtype: - - workspace: ws-eval-1784618349255-46bfa596 - time: started=2026-07-21T07:19:07.729852+00:00 finished=2026-07-21T07:19:53.343202+00:00 duration=0m46s - token usage: input=1 output=235 reasoning=0 cache_read=23274 cache_write=659 cached_total=23933 cost=$0.1233 Bucket Scores vs Baseline - workflow: current=0/10 baseline=- diff=- - evidence collection: current=5/5 baseline=- diff=- - critical evidence: current=35/80 baseline=- diff=- - evidence routing: current=0/5 baseline=- diff=- - diagnosis: current=0/50 baseline=- diff=- Raw Output <<< This is an ad-hoc data lookup — the `ad-hoc tidb-p99-breakdown` recipe is exactly right for this. Let me run it directly.根据 `ad-hoc tidb-p99-breakdown` 配方的权威返回结果，所有关键信号均缺失（主延迟指标、5 个节点级补充信号、2 个集群级信号全部 missing）。按规则，证据不足不可猜测。 ```json { "cluster_id": "10748116073341772391", "window": { "start": "2026-07-21T03:00:00Z", "end": "2026-07-21T03:05:00Z" }, "evidence_sufficient": false, "cluster_avg_tikv_grpc_request_handle_p99_s": null, "cluster_avg_pd_tso_request_handle_p99_s": null, "top_nodes_by_avg_p99_latency": [] } ``` >>> Case: ttl-row-processing-in-premium-cluster-during-sql-tail-latency-incident-window - source_run: evalrun-dca9124193b04b2c873027cd9fb2ca6f - execution_status: completed - verdict: pass - score: current=70 baseline=- diff=- - failure_code: - - invalid_subtype: - - workspace: ws-eval-1784618394922-ee5c111b - time: started=2026-07-21T07:19:53.351337+00:00 finished=2026-07-21T07:20:19.551588+00:00 duration=0m26s - token usage: input=1 output=140 reasoning=0 cache_read=21684 cache_write=497 cached_total=22181 cost=$0.1136 Bucket Scores vs Baseline - workflow: current=0/10 baseline=- diff=- - evidence collection: current=5/5 baseline=- diff=- - critical evidence: current=0/30 baseline=- diff=- - evidence routing: current=0/5 baseline=- diff=- - diagnosis: current=35/50 baseline=- diff=- Raw Output <<< `ad-hoc ttl-rows` is the exact recipe for this request. Running it now.```json { "cluster_id": "10748116073341772391", "window": { "start": "2026-07-21T03:00:00Z", "end": "2026-07-21T03:05:00Z" }, "evidence_sufficient": true, "ttl_activity_present": true, "signals": { "ttl_insert_rows_delta": 1754729, "ttl_processed_expired_rows_delta": 567330 } } ``` >>>