--- name: tikv-slow-diagnosis description: | Diagnose TiKV-side latency regressions for Jira incidents or direct cluster investigations. Use when the user asks to run `tikv-slow-diagnosis analyze {jira-issue-id}`, or wants to determine whether slowness is really on TiKV, whether it is on the read path or write path, which TiKV stage is slow first, which metrics prove it, or why TiKV leader drops happened and whether they were caused by PD scheduling, gRPC heartbeat delay, raftstore slowdown, network jitter, or node-level stalls, whether short-lived store-local latency spikes point to TiKV IO jitter on raft engine, RocksDB KV, or node / cloud-disk layers, whether transport latency comes from TiDB <-> TiKV or TiKV <-> TiKV network issues, or which exact flow-control / `ServerIsBusy` mechanism is rejecting or throttling work, or why hot-leader rebalance / scatter is slow because snapshot generation, delete-range backlog, or shared worker-pool pressure is blocking progress. --- # tikv-slow-diagnosis Use this skill to diagnose TiKV slowdowns with a metric-driven workflow instead of symptom guessing. This skill also owns dedicated TiKV leader-drop, IO-jitter, network-issues, flow-control, apply-wait, and snapshot-backlog sub-capabilities. ## Quick Routing Cheatsheet Use this table as the first routing pass before reading deeper branch logic. | Primary signal | Activate first | |---|---| | `leader drop`, `leader missing`, suspicious re-election, uncertain PD-driven transfer versus follower re-election | `references/leader-drop.md` | | One-store or few-store short-lived tails, stable workload, disk jitter suspicion | `references/io-jitter.md` | | `KV Request duration` and `RPC Layer Latency` rise together, `tikvRPC` backoff, reconnect / `unreachable` signals | `references/network-issues.md` | | `ServerIsBusy`, `deadline exceeded`, throttle, discard, read-pool rejection | `references/flow-control.md` | | `apply wait` is high, slow log points to `apply_batch_wait`, or the same hot region keeps growing and splitting | `references/apply-wait.md` | | `scatter` slow, `hot leader` persists, snapshot generate wait or delete-range backlog | `references/snapshot-backlog.md` | | Slow log or `EXPLAIN ANALYZE` already provides TiKV execution detail | `references/slow-log-execution-detail.md` | | Need TiKV-vs-network split, health check, gRPC, flow control, snapshot, or Resource Control cross-checks | `references/cross-cutting.md` | | `Prewrite` / `Commit` / async write / raftstore side is slow | `references/write-path.md` | | `BatchGet` / `Cop` / unified read pool / coprocessor side is slow | `references/read-path.md` | ## Skill Layers Keep the ownership boundaries clear when updating this skill: - `SKILL.md` owns entry routing, the top-level workflow, output contract, and global guardrails - `references/cross-cutting.md` owns shared checks used by multiple branches - branch references own branch-specific proof logic and wording constraints - `references/read-path.md` and `references/write-path.md` own deeper path-stage decomposition - `references/apply-wait.md` owns hotspot-region-driven `apply wait` long-tail diagnosis - `references/skill-evolution.md` owns maintainer-facing evaluator feedback and promotion policy ## Branch Activation Summary Use this summary only for first-pass activation. Detailed branch logic stays in the dedicated mode sections and companion files below. - activate `leader-drop` when the symptom is about leader movement, `leader missing`, or uncertainty between PD-driven transfer and follower-side re-election - activate `io-jitter` when one or a few TiKV stores show short-lived long tails and workload growth does not already explain them - activate `network-issues` when `KV Request duration` and `RPC Layer Latency` rise together, or transport / reconnect signals dominate - activate `flow-control` when `ServerIsBusy`, throttling, rejection, or `deadline exceeded` is the visible symptom - activate `apply-wait` when a write-path tail is dominated by `apply wait`, `apply_batch_wait`, or one / a few regions repeatedly growing and splitting - activate `snapshot-backlog` when `scatter`, `hot leader`, or rebalance progress is blocked by snapshot generation / apply progress - activate `read-path` or `write-path` only after the latency-domain split and path classification are done ## What This Skill Must Answer ### Global Questions Every diagnosis must explicitly answer: 1. Is the slowdown really on TiKV, or is it upstream in TiDB gRPC or network? 2. Is the issue cluster-wide, limited to some stores, or limited to a hotspot / long-tail subset? 3. Is the dominant path read, write, or both? 4. Which stage becomes slow first? 5. Which exact metrics support the conclusion? 6. Which evidence is still missing before claiming a final root cause? 7. If a TiDB slow log or `EXPLAIN ANALYZE` execution detail is already available, does it actually justify entering TiKV-side diagnosis, or should this skill exit early? ### Branch Add-On Questions When `leader-drop` is active, also answer: 7. Was the leader movement PD-driven, or was it follower-side re-election after heartbeat delay? 8. If it was not PD-driven, which signal appeared first: TiDB RPC layer latency, TiKV gRPC stress, raftstore slowdown, network jitter, or host-level memory / kernel stall? 9. What evidence rules out `evict-slow-store` before naming another mechanism? When `io-jitter` is active, also answer: 10. Has workload volume, request mix, or hotspot distribution actually changed in the same window? 11. Which store or instance is the outlier, and is the asymmetry strong enough to call it store-local? 12. Does the evidence point first to raft-engine IO, RocksDB-KV IO, or node / cloud-disk latency? 13. Which tail quantile proves the problem, and what node-level evidence is still missing? When `network-issues` is active, also answer: 14. Which hop is affected: TiDB <-> TiKV, TiKV <-> TiKV, TiDB <-> PD, or PD <-> PD? 15. Is it network interruption, latency jitter, or bandwidth saturation? 16. Which metrics prove transport time, not TiKV local processing, is the main contributor? 17. What evidence rules out gRPC thread exhaustion or host stalls being mistaken for network failure? When `flow-control` is active, also answer: 18. Which exact `ServerIsBusy`, throttle, or rejection mechanism fired first? 19. Is the protection on the write path, read path, or both? 20. Which lower-layer bottleneck triggered the protection? 21. Which threshold was crossed, or which short spike effectively crossed it even if a gauge missed it? When `apply-wait` is active, also answer: 22. Is the tail concentrated on one or a few hotspot regions, or broad across many regions? 23. Which region IDs are responsible first, and do slow log and TiKV log point to the same regions? 24. Did region size growth and repeated split directly participate in the mechanism? 25. Is `apply wait` the first bottleneck, or only the visible fallout of snapshot / ingest or another lower-layer write-path issue? When `snapshot-backlog` is active, also answer: 26. Is the problem really leader imbalance, or is leader movement blocked by snapshot generation / apply progress? 27. Which stage is slow first: snapshot generate wait, snapshot read IO, snapshot send, or downstream apply, and are `pending delete range`, shared worker pools, or RocksDB / disk wait blocking progress? Do not write conclusions like `maybe TiKV`, `probably disk`, or `seems like CPU`. Every claim must map to concrete metrics and observed trends. ## Entry Contract There are only two operational entry modes: - Jira incident entry - direct cluster or direct evidence entry Both must converge into the same Step 0 entry gate before deep diagnosis starts. ### Jira Incident Entry Use this path when the user runs `tikv-slow-diagnosis analyze {jira-issue-id}`. 1. Resolve the Jira cloud ID with Atlassian MCP. 2. Fetch the issue first. 3. Extract: - issue summary - issue description - cluster ID from the configured Jira `Cluster ID` field if present - clinic / grafana / cluster links if present in the issue description 4. If cluster ID is missing but a Clinic link, Grafana link, or explicit cluster link exists in the issue body: - continue with link-first diagnosis - extract the time window from the link if possible 5. If cluster ID is missing and no usable link exists: - do not stop immediately - run a text-first first-pass split for TiKV-side versus upstream, read versus write, and obvious branch selection such as `network-issues`, `flow-control`, or `snapshot-backlog` - say explicitly which evidence is still missing before deep metric proof 6. Use the issue description as the initial incident context, and treat comments as hints only after the initial split is done. 7. Then enter Step 0. ### Direct Cluster Or Direct Evidence Entry Use this path when the user already provides a cluster ID, cluster link, metrics window, slow log, or `EXPLAIN ANALYZE` detail. 1. Resolve cluster metadata first when a cluster identifier or cluster link is available. 2. Confirm cluster type and deploy mode before selecting metrics. 3. If slow log or `EXPLAIN ANALYZE` is already available, use it as the preferred Step 0 input instead of opening generic panels first. 4. Then enter Step 0. ## Leader Drop Mode ### When To Activate Switch into this mode when the primary symptom is TiKV `leader drop`, even if write latency is also high. ### Goal The goal in this mode is different from generic slow diagnosis: - the main output must explain the `leader drop` mechanism first - `commit log duration`, `apply duration`, or `RPC latency` are supporting evidence, not the headline root cause by themselves - do not treat `slow store` and `evict-slow-store` as interchangeable ### Common Signals Direct mechanism split: - PD-driven leader movement: - `evict-leader` during upgrade / rolling restart - `evict-slow-store` after slow-score reaches 100 and PD really schedules eviction - follower-side re-election after heartbeat delay: - gRPC thread busy or stuck - raftstore slowdown - network jitter - host-level memory stall or swap ### Guardrails - do not write `evict-slow-store` unless both the slow-store score and PD operator / scheduler evidence support it - do not write `commit log duration high caused leader drop`; explain the re-election mechanism first - when comparing onset order, use `15s`-aligned metrics if available instead of 1-minute smoothing - if a suspected metric rises after `leader drop` starts, treat it as amplification or fallout, not the first cause ## IO Jitter Mode ### When To Activate Switch into this mode when the primary symptom is a short-lived latency spike on one or a few TiKV stores and disk tail latency is plausible. ### Common Signals - workload is roughly flat, but slow SQL or KV latency tails appear suddenly - one store is a clear outlier in `KV Request duration by store` - `p99` is too smooth, while `p999`, `p99999`, or `max` exposes a burst - write-path or read-path metrics spike only on a subset of stores - cloud disk or EBS is involved ### Goal The goal in this mode is: - rule out business growth, workload-shape change, and hotspot movement first - identify the slow TiKV store before going deep into a path-specific stage - separate `raft-engine jitter`, `RocksDB-KV jitter`, and `node / cloud-disk jitter` - say clearly whether the conclusion is only `suspected IO jitter` or has node / disk corroboration ### Guardrails - do not write `IO jitter` before checking workload growth and hotspot changes in the same window - if many stores slow together, investigate workload-wide or broader infra causes before naming disk jitter - when `p99` is too smooth, escalate from `0.99` to `0.999`, then `0.99999`, then `1` - do not use average-only node disk panels to clear or confirm disk jitter - if evidence stops at store-local latency tails without node / disk proof, write `suspected IO jitter`, not `confirmed disk fault` ## Network Issue Mode ### When To Activate Switch into this mode when transport delay between TiDB, TiKV, or PD looks more important than TiKV local processing time. ### Common Signals - `TiDB -> KV Request duration by store/type` rises together with `TiDB -> KV Request -> RPC Layer Latency` - TiDB slow logs show `tikvRPC` backoff, abnormal `lock_rpc`, or suspicious RPC time - `Batch Receive Average Duration`, `Batch Client Unavailable Duration`, or `Wait Connection Establish Duration` rises - `TiKV -> Error -> Server report failures` shows `unreachable` - `Commit log duration` rises well above `Append log duration` while append / apply stay relatively normal - node network allowance, retransmit, or drop signals rise ### Goal The goal in this mode is: - identify the affected hop first - separate interruption, jitter, and bandwidth saturation - avoid mislabeling gRPC thread starvation or host stalls as network faults - say explicitly when the evidence points to an upstream non-TiKV hop such as TiDB <-> PD or PD <-> PD ### Guardrails - if `RPC Layer Latency` is flat while TiKV internal stages rise, do not blame network first - do not treat `unreachable` alone as proof of a network fault; gRPC exhaustion or stuck threads can mimic it - if `Commit log duration` rises, compare it with `Append log duration` before calling it network - if the evidence points to TiDB <-> PD or PD <-> PD, say clearly that the root cause is upstream of TiKV ## Flow Control Mode ### When To Activate Switch into this mode when the primary symptom is `ServerIsBusy`, `deadline exceeded`, explicit throttling, or read/write admission rejection. ### Common Signals - TiDB `KV Errors`, `KV Backoff OPS`, or `KV Backoff Duration` grouped by type show `server_is_busy`, `deadline_exceeded`, or similar reasons - TiKV `Server is busy`, `Throttle duration`, or `Scheduler discard ratio` rises - coprocessor errors show `max_pending_tasks_exceeded`, `memory_quota_exceeded`, or `deadline_exceeded` - `Scheduler writing bytes`, `Compaction pending bytes`, `Flow controller factors`, `Running tasks`, or `raftstore` channel-full signals look suspicious ### Goal The goal in this mode is: - identify the exact protection mechanism first - separate write flow control from read flow control - find the lower-layer bottleneck that caused the protection to fire - distinguish a real threshold crossing from a short spike that gauges may miss ### Guardrails - do not stop at `ServerIsBusy`; it is a protection or symptom, not the final root cause - prefer TiDB or TiKV logs for the `reason` field when available - gauge-style factor metrics can miss short spikes, so use supporting metrics and higher quantiles when needed - increasing thresholds is mitigation only; it does not prove the root cause is solved ## Snapshot Backlog Mode ### When To Activate Switch into this mode when `scatter`, `hot leader`, or rebalance progress is slow because snapshot work may be blocked. ### Common Signals - scatter or rebalance runs for a long time and is interrupted before converging - one hot store remains hot because region movement cannot finish - `snapshot generate wait`, snapshot duration, or pending delete-range work rises on one store ### Goal The goal in this mode is to explain why balancing did not progress and which snapshot stage was actually blocked first. ### Guardrails - do not stop at `hot leader`; explain why balancing did not progress - do not recommend `evict leader` or more scatter as root cause mitigation unless snapshot progress is healthy - if the evidence only proves snapshot backlog but not the lower layer, say `snapshot progress blocked` and read `references/snapshot-backlog.md` ## Data Sources Use these skills and sources: - `clinic-api` for cluster metadata - `o11y-auth` when authentication is needed - `o11y-data` or `o11y-metrics-api` for metrics Credential convention on this workstation: - check `~/.env` first - then repository-local `.env` - then current shell environment Resolve metadata from Clinic first and collect at least: - `clusterID` - `tenantID` - `clusterProviderName` - `clusterRegionName` - `deployType` - `clusterDeployTypeV2` if present - `parentID` for Premium when present Resolve O11Y targets only after metadata is known. ## Top-Level Workflow Contract Use this as the stable orchestration layer for future updates. 1. Entry gate - Required input: Jira incident text, direct cluster input, or slow-log / `EXPLAIN ANALYZE` detail - Decision: whether TiKV-side diagnosis should even start - Output: `entry_surface`, `entry_gate_result`, and any early-exit reason 2. TiKV-vs-upstream split - Required input: slow-log execution detail or first-pass TiDB-facing metrics - Decision: `tikv_local`, `transport_or_grpc`, `upstream_non_tikv`, or `unclear` - Output: the current dominant latency domain plus missing evidence 3. Scope and path classification - Required input: KV request types, per-store scope, and health-check context - Decision: `read`, `write`, or `both`, plus `cluster_wide`, `subset`, `hotspot`, or `long_tail` - Output: active path and scope labels 4. Branch activation - Required input: symptom pattern plus the initial classification - Decision: which specialized branches must be activated, such as `leader-drop`, `io-jitter`, `network-issues`, `flow-control`, or `snapshot-backlog` - Output: ordered companion-file read list 5. Earliest proven bottleneck - Required input: branch-specific evidence - Decision: the earliest stage that is actually proven slow, or whether evidence is still incomplete - Output: `first_proven_stage`, ruled-out alternatives, and unresolved evidence gaps 6. Final output - Required input: the structured findings from steps 1-5 - Decision: final wording and confidence level - Output: the diagnosis contract defined under `Output Format` ## Intermediate Diagnosis State Keep the following working state stable across future updates. This is the recommended internal schema even when the final answer is prose. ```json { "entry_surface": "jira_incident | direct_cluster | slow_log | explain_analyze | mixed", "entry_gate_result": "enter_tikv | exit_upstream | unclear", "latency_domain": "tikv_local | transport_or_grpc | upstream_non_tikv | unclear", "path": "read | write | both | unclear", "scope": "cluster_wide | subset | hotspot | long_tail | unclear", "active_modes": [], "first_proven_stage": "", "key_metrics": [], "ruled_out_explanations": [], "missing_evidence": [], "confidence": "confirmed | probable | suspected | blocked" } ``` Field intent: - `entry_surface`: where the diagnosis started - `entry_gate_result`: whether this skill should continue or exit early - `latency_domain`: whether dominant time is inside TiKV or outside it - `path`: read, write, both, or still unclear - `scope`: fleet-wide, subset, hotspot, or long-tail - `active_modes`: specialized branches activated after the initial split - `first_proven_stage`: earliest bottleneck stage actually supported by evidence - `key_metrics`: exact metrics or slow-log fields used to support the conclusion - `ruled_out_explanations`: alternatives explicitly rejected by evidence - `missing_evidence`: blockers that prevent overclaiming - `confidence`: wording guardrail for final output ## Core Workflow ### Step 0: Use Slow Log Or Plan Execution Detail As The Entry Gate If the user already provides a TiDB slow log, statement summary row, or `EXPLAIN ANALYZE` execution info, parse that before opening metric panels. Supported entry surfaces: - TiDB slow log statement fields such as `Process_time`, `Wait_time`, `Backoff_time`, `Request_count`, `Process_keys`, `Total_keys` - slow-log embedded TiKV RPC blocks such as `Slowest_prewrite_rpc_detail`, `Commit_primary_rpc_detail`, or `slowest_rpc` - slow-log embedded write-detail fields such as `store_batch_wait`, `persist_log`, `commit_log`, `apply_batch_wait`, `sync_log`, or `write_wal` - `EXPLAIN ANALYZE` `cop_task: {...}` on root nodes - `EXPLAIN ANALYZE` `tikv_task:{...}`, `scan_detail:{...}`, `time_detail:{...}` on `cop[tikv]` nodes - `EXPLAIN ANALYZE` `commit_txn:{...}`, `slowest_prewrite_rpc:{...}`, `commit_primary_rpc:{...}`, or `lock_keys:{...}` for write DML Primary enter-or-exit rule: - continue into `tikv-slow-diagnosis` only when the provided execution detail already shows TiKV-local wall time is dominant - if `tikv_wall_time` is absent, tiny, or clearly not the dominant component, exit this skill and route to upstream TiDB, transport, lock-wait, or PD diagnosis instead Treat `tikv_wall_time` as dominant only when one of these is true: - an embedded TiKV RPC block shows `time_detail: {tikv_wall_time: ...}` and it is the largest TiKV-local time field in that block - the same RPC block has `total: ...`, and `tikv_wall_time` is at least the majority of that RPC total - a `cop[tikv]` node shows `time_detail:{..., tikv_wall_time: ...}` together with non-trivial `tikv_task` / `scan_detail`, proving the slow stage is inside TiKV rather than before TiKV Exit this skill early when any of these patterns dominates instead: - `Backoff_time`, `Prewrite_Backoff_types`, `Commit_Backoff_types`, or `tikvRPC` backoff dominates while `tikv_wall_time` is small or absent - `Resolve_lock_time`, `lock_rpc`, or pessimistic-lock wait dominates - `Get_commit_ts_time`, `Wait_prewrite_binlog_time`, or PD-facing wait is the main cost - root-node `cop_task` or SQL `Query_time` is high, but `cop[tikv]` node details do not show large `tikv_wall_time` When execution detail already gives the split, use it to pre-classify: - read-side TiKV local time: `cop[tikv]` with `tikv_task`, `scan_detail`, `time_detail` - write-side TiKV local time: `slowest_prewrite_rpc`, `commit_primary_rpc`, or `slowest_rpc` with `write_detail` - transport or upstream suspicion: high root `cop_task` / RPC total but weak embedded `tikv_wall_time` Additional slow-log-first rules: - if slow-log or execution-detail fields already show a clear slow TiKV stage, use that stage as the initial diagnosis anchor before opening generic panels - if `store_batch_wait`, `persist_log`, `commit_log`, `sync_log`, or `write_wal` shows extreme tails, do not let calm average host metrics clear the TiKV path - if one incident contains multiple distinct windows, classify each window separately before merging any conclusion; mixed TiKV-side and TiDB-side windows must remain separate - if a later comment adds a new slow-log file for a different date range, treat it as a potentially separate diagnosis track rather than automatic confirmation of the original window Read `references/slow-log-execution-detail.md` whenever this step is used. ### Step 1: Confirm The Problem Is On TiKV Start from TiDB-facing metrics. Quantile-first rules: - use `p99` first - use `p999` for sparse, store-local, or obvious long-tail issues - use `p9999` when tails are extreme - use `p99999` or `max` when slow-log evidence already proves a long-tail stage but `p99` and `p999` still look too smooth - do not use averages as the primary reason to declare TiKV healthy Tail-escalation rule: - if slow logs show a clear long-tail stage such as `prewrite`, `store_batch_wait`, `persist_log`, `commit_log`, or `sync_log`, then Clinic or Grafana metric reading must escalate from `p99` to `p999`, then `p99999`, then `max` before concluding that the TiKV stage is not slow - if average disk or network latency is calm while slow-log tail fields are extreme, treat averages as weak counterevidence only Required first panels: - `TiDB -> KV Request OPS` - `TiDB -> KV Request duration by store/type` - `TiDB -> KV Request -> RPC Layer Latency` From the first pass, answer: - which KV request types are slow - whether all stores are slow or only some stores - whether latency increases inside TiKV or before TiKV - whether the main problem is real user-visible latency or only CPU / hotspot / rebalance pressure without clear latency impact TiKV-side confirmation rule: - if `KV Request duration` rises but `RPC Layer Latency` does not rise with it, continue into TiKV diagnosis - if both rise together, suspect network or TiDB-side gRPC first, then read `references/network-issues.md` before jumping into deep TiKV internals CPU-spike boundary rule: - if the primary symptom is only `CPU spike`, first check whether latency, timeout, or user-visible slow SQL also rose - compare thread-CPU composition and workload change before concluding backup, checksum, or TiKV-local bottlenecks - if foreground workload or query mix changed more than backup / checksum threads did, or the issue is mainly underprovisioning without clear slow-path evidence, do not label backup as the root cause For fast triage, reduce request types to: - read: `BatchGet`, `Cop` - write: `Prewrite`, `Commit` ### Step 2: Run A Basic Health Check Before deep path analysis, classify health check metrics into **critical** and **major** severity. **Must scan every metric in both lists.** Do not skip any metric because you already checked a different severity level. #### Critical These metrics indicate active errors or resource exhaustion. **Any non-zero value must be explained.** | Metric / Query | What it means | |---|---| | `sum by (instance) (rate(tikv_scheduler_too_busy_total{$filter_labels}[30s]))` | Scheduler write-pool rejected requests | | `sum by (instance, type) (rate(tikv_channel_full_total{$filter_labels}[30s]))` | Raft message channel full | | `avg by (instance, db) (tikv_engine_write_stall{$filter_labels})` | RocksDB write stall active | | `sum by (instance) (rate(tikv_raftstore_store_write_msg_block_wait_duration_seconds_count{$filter_labels}[30s]))` | Store write message blocked | | `sum by (instance, type) (tikv_raftstore_process_busy{$filter_labels})` | Raftstore or apply process busy | | `sum by (instance, type) (rate(tikv_server_report_failure_msg_total{$filter_labels}[30s]))` | Store reports peer unreachable or not_leader | | `sum by (instance) (rate(tikv_critical_error_total{$filter_labels}[30s]))` | Any critical TiKV error fired | | `tikv_raftstore_leader_missing{$filter_labels}` | Current leader-less regions | | `tikv_rocksdb_damaged_files{$filter_labels}` or `tikv_rocksdb_damaged_files_deleted{$filter_labels}` | SST file corruption or auto-deleted | | `sum by (instance) (rate(tikv_server_raft_append_rejects_total{$filter_labels}[30s]))` | Raft log append rejected | #### Major These metrics indicate degraded behavior or increased error rates. **Escalate to critical if any series exceeds 1/s rate for more than 1 minute.** | Metric / Query | What it means | |---|---| | `sum by (instance, status) (rate(tikv_storage_engine_async_request_total{status!="success", $filter_labels}[30s]))` | Storage async request errors | | `sum by (instance, stage) (rate(tikv_scheduler_stage_total{stage=~"snapshot_err|prepare_write_err", $filter_labels}[30s]))` | Scheduler stage errors | | `sum by (instance, type) (rate(tikv_coprocessor_request_error{$filter_labels}[30s]))` | Coprocessor execution errors | | `sum by (instance, type) (rate(tikv_grpc_msg_fail_total{$filter_labels}[30s]))` | gRPC transport failures | | `sum by (instance) (delta(tikv_raftstore_region_count{type="leader", $filter_labels}[30s]))` | Leader movements | Also check node-level resource availability before deep TiKV analysis: - **CPU**: `process_cpu_seconds_total{$filter_labels}` vs `tikv_server_cpu_cores_quota{$filter_labels}` — compare utilization against quota - **Memory**: `process_resident_memory_bytes{$filter_labels}` vs `tikv_server_memory_quota_bytes{$filter_labels}` — compare usage against quota (note: `tikv_server_memory_quota_bytes` requires TiKV v7.5+) - **Disk**: `tikv_store_size_bytes{$filter_labels, type="available"}` vs `{type="capacity"}` — disk usage percentage (verify label value `"available"` matches your TiKV version; some older versions use `"disk_available"`) - **Restart**: `delta(process_start_time_seconds{$filter_labels}[5m]) > 0` — detect TiKV restarts in the problem window - **Busy threads**: `tikv_thread_cpu_seconds_total{$filter_labels}` grouped by `name` — identify threads with >80% utilization - `PD -> Abnormal stores -> Slow stores` - `PD -> Abnormal stores -> Lowspace stores` - `PD -> Abnormal stores -> Others stores` - Node exporter IO / network metrics when TiKV store-local issues are suspected Rules: - query every metric with `{$filter_labels}` using the same `$k8s_cluster`, `$tidb_cluster`, `$instance` filter pattern that the dashboard panels use - scan all relevant series, not just the first few returned lines - summarize abnormal series count and top offenders - for **critical** metrics: any non-zero series must be quoted in the diagnosis output with its value, instance, and time window - for **major** metrics: if the rate exceeds 1/s, treat it the same as critical for that window - if only some TiKV are slow, node / infra checks are mandatory - if the spike is short-lived, compare per-store tails with higher quantiles or `max` instead of relying on `p99` alone Leader-drop add-on checks: - `delta(tikv_raftstore_region_count{type="leader"}[30s])` — leader count change per 30s - `tikv_raftstore_leader_missing{$filter_labels}` — current leader-less regions - `tikv_raftstore_slow_score{$filter_labels}` — PD slow store score per instance/type - PD operator / scheduler evidence when `evict-leader` or `evict-slow-store` is suspected IO-jitter add-on checks: > Note: queries prefixed with `tidb_tikvclient_*` come from the TiDB datasource; all other queries use the TiKV datasource. - `histogram_quantile(0.99, sum(rate(tidb_tikvclient_request_seconds_bucket{$filter_labels}[30s])) by (le, store))` — KV request duration p99 by store - `histogram_quantile(0.99, sum(rate(tikv_grpc_msg_duration_seconds_bucket{type!="kv_gc", $filter_labels}[30s])) by (le, instance))` — gRPC message duration p99 by instance - `histogram_quantile(0.99, sum(rate(tikv_raftstore_append_log_duration_seconds_bucket{$filter_labels}[30s])) by (instance, le))` — append log duration p99 per server - `histogram_quantile(0.99, sum(rate(tikv_raftstore_apply_log_duration_seconds_bucket{$filter_labels}[30s])) by (instance, le))` — apply log duration p99 per server - `tikv_raftstore_slow_score{$filter_labels}` — PD slow store score per instance/type Network add-on checks: > Note: `tidb_tikvclient_*` queries come from the TiDB datasource; `node_*` from the node_exporter; all others from the TiKV datasource. - `tidb_tikvclient_batch_client_unavailable_seconds_sum{$filter_labels} / tidb_tikvclient_batch_client_unavailable_seconds_count{$filter_labels}` — batch receive average duration - `tidb_tikvclient_batch_client_unavailable_seconds{$filter_labels}` — batch client unavailable duration - `tidb_tikvclient_batch_client_wait_connection_establish_seconds{$filter_labels}` — wait connection establish duration - TiDB slow log `tikvRPC` backoff or suspicious RPC duration - `tikv_server_report_failure_msg_total{$filter_labels}` — server reports peer unreachable or not_leader - `histogram_quantile(0.99, sum(rate(tikv_raftstore_commit_log_duration_seconds_bucket{$filter_labels}[30s])) by (instance, le))` — commit log duration p99 per server - `histogram_quantile(0.99, sum(rate(tikv_raftstore_append_log_duration_seconds_bucket{$filter_labels}[30s])) by (instance, le))` — append log duration p99 per server - `rate(node_network_receive_drop_total{$filter_labels}[30s])` / `rate(node_network_transmit_drop_total{$filter_labels}[30s])` — node network drops Flow-control add-on checks: > Note: `tidb_tikvclient_*` queries come from the TiDB datasource; all others from the TiKV datasource. - `rate(tidb_tikvclient_region_err_total{$filter_labels}[30s])` — TiClient region error OPS - `rate(tidb_tikvclient_backoff_seconds_count{$filter_labels}[30s])` — KV backoff OPS - `tidb_tikvclient_backoff_seconds_sum{$filter_labels}` — KV backoff duration - `tikv_scheduler_too_busy_total{$filter_labels}` — scheduler write-pool rejected requests (Server is busy) - `tikv_scheduler_throttle_duration_seconds_bucket{$filter_labels}` — throttle duration - `tikv_scheduler_discard_ratio{$filter_labels}` — scheduler discard ratio by type - `histogram_quantile(0.99, sum(rate(tikv_yatp_pool_schedule_wait_duration_bucket{name=~"unified-read.*", $filter_labels}[30s])) by (le, instance))` — unified read pool wait duration p99 - `tikv_unified_read_pool_running_tasks{$filter_labels}` — unified read pool running tasks Snapshot-backlog add-on checks: - PD operator / scheduler evidence showing scatter / rebalance progress exists - `histogram_quantile(0.99, sum(rate(tikv_storage_engine_async_request_duration_seconds_bucket{type="snapshot", $filter_labels}[30s])) by (instance, le))` — storage async snapshot duration p99 by instance - `histogram_quantile(0.99, sum(rate(tikv_raftstore_snapshot_generation_wait_duration_seconds_bucket{$filter_labels}[30s])) by (instance, le))` — snapshot generation wait p99 per instance - `histogram_quantile(0.99, sum(rate(tikv_raftstore_snapshot_apply_wait_duration_seconds_bucket{$filter_labels}[30s])) by (instance, le))` — snapshot apply wait p99 per instance - `tikv_thread_cpu_seconds_total{name=~"raftstore.*", $filter_labels}` — raftstore thread CPU - region-worker or equivalent worker-pool panels - per-store disk read / IO wait asymmetry for the outlier store ### Step 3: Classify Path And Scope Always classify both dimensions: Path: - read-dominant - write-dominant - both Scope: - cluster-wide - subset of stores - hotspot / imbalance - long-tail on a small subset of requests IO-jitter scope note: - if only one or a few stores show a short burst while workload volume is flat, keep the analysis per store and do not average the fleet together Network scope note: - if the latency is broad and `RPC Layer Latency` rises across many stores at once, check shared transport or upstream hops before diving into one TiKV Use: - `KV Request duration by store/type` - gRPC latency by instance - `tikv_thread_cpu_seconds_total by (instance, name)` - request-count or QPS imbalance by store / instance Read-side signals: - slow `BatchGet` or `Cop` - high coprocessor duration - high unified read pool wait or overload - high snapshot duration Write-side signals: - slow `Prewrite` or `Commit` - high scheduler command duration - high async write duration - high raftstore / commit / apply duration - flow control or `ServerIsBusy` ### Step 4: Read Only The Relevant Companion Files Read only the files needed for the classified path. Ownership rule: - the main skill owns orchestration and stop conditions - companion files own branch-specific proof logic - when a rule only applies inside one branch, prefer putting it in that branch reference instead of repeating it in `SKILL.md` | Situation | Read | |---|---| | User already provided slow log or `EXPLAIN ANALYZE` execution detail and you need to decide whether TiKV is really the bottleneck | `references/slow-log-execution-detail.md` | | Primary symptom is `leader drop` or suspected re-election | `references/leader-drop.md` | | Short-lived single-store or few-store tails suggest cloud-disk / IO jitter | `references/io-jitter.md` | | `tikvRPC` backoff, batch-client reconnect, `unreachable`, or `commit log >> append log` suggests transport issues | `references/network-issues.md` | | `ServerIsBusy`, `deadline exceeded`, throttle, discard, or read-pool rejection is prominent | `references/flow-control.md` | | `apply wait` is prominent, slow log exposes `apply_batch_wait`, or one / a few regions repeatedly split while carrying most slow writes | `references/apply-wait.md` | | `scatter` is slow, `hot leader` persists, or snapshot generation / delete-range backlog blocks rebalance | `references/snapshot-backlog.md` | | Need TiKV-vs-network split, basic health check, gRPC, flow control, snapshot, or Resource Control | `references/cross-cutting.md` | | `Prewrite` / `Commit` / `async write` / raftstore side is slow | `references/write-path.md` | | `BatchGet` / `Cop` / unified read pool / coprocessor side is slow | `references/read-path.md` | Combination rules: - if slow log or `EXPLAIN ANALYZE` already includes `time_detail`, `scan_detail`, `tikv_task`, or embedded TiKV RPC details, read `references/slow-log-execution-detail.md` first - if `leader-drop` is active and write-path tails are also present, add `references/write-path.md` - if `leader-drop` is active and TiDB RPC / gRPC suspicion is also present, add `references/cross-cutting.md` - if `io-jitter` is active and the first strong evidence is on write-path metrics, add `references/write-path.md` - if `io-jitter` is active and the first strong evidence is on read-path metrics, add `references/read-path.md` - if `network-issues` is active because `RPC Layer Latency` rises together with KV latency, keep `references/network-issues.md` ahead of deeper TiKV-local files - if `flow-control` is active, treat it as a protection branch and keep tracing into the lower-layer read or write path as needed - if `apply-wait` is active because the same hotspot region dominates slow `commit` or `prewrite`, read `references/apply-wait.md` before generic `references/write-path.md` - if `apply-wait` is active and snapshot or ingest is clearly amplifying the tail, add `references/snapshot-backlog.md` - if `apply-wait` is active but region concentration is weak, fall back to `references/write-path.md` - if `snapshot-backlog` is active and snapshot duration looks suspicious but the lower layer is unclear, add `references/cross-cutting.md` - if both read and write are slow, read `references/cross-cutting.md` first before going deeper into either path ### Step 5: Stop At The Earliest Proven Bottleneck Do not stop at a symptom if a downstream metric clearly explains it. Rules: - treat flow control as a symptom, not the final root cause - treat high gRPC duration as either downstream slowness or reply-path slowness until proven otherwise - if `cop wait` and `unified read pool schedule wait` rise together while `handle` / `task exec` stay flat, stop at `unified_read_pool_queueing` or `localized_read_pool_saturation` instead of continuing into deeper coprocessor execution blame - do not upgrade `high gRPC coprocessor duration` into `gRPC is the first bottleneck` when read-pool queueing already explains the onset order - if slow log or execution detail already proves `tikv_wall_time` is not dominant, stop and exit this skill instead of forcing a TiKV-local root cause - if transport metrics already explain the latency, stop before blaming a deeper TiKV local stage - for write path, investigate raftstore in reverse order to avoid blaming wait metrics that are only queue fallout - if evidence is incomplete, say that it is incomplete ## Output Format The final diagnosis should be organized around these five points: 1. Whether the issue is really on TiKV and whether a workload spike, hotspot, or workload-shape change exists in the same time window 2. Whether the issue affects all TiKV, part of the fleet, or only outlier stores / regions 3. Whether there is an obvious resource bottleneck 4. Whether the issue is on the read path, write path, or both 5. Which specific stage is slow first Also include: - exact metrics used - exact slow-log fields or `EXPLAIN ANALYZE` fragments used, if this diagnosis started from execution detail - key evidence - alternative explanations already ruled out - missing evidence - mitigation or next-step suggestions that match the proven bottleneck When listing `missing evidence`, explicitly call out standard access blockers such as: - Clinic link exists but `data_status != 100`, so metrics or bundle evidence is not ready yet - Grafana link uses only a relative time window such as `now-3h` / `now`, so the historical evidence window is not reproducible yet Leader-drop mode must add: - whether the direct mechanism was PD-driven transfer or follower-side re-election - which exact evidence rules out `evict-slow-store` if that path is rejected - which signal appeared first in the narrow onset window - which later metrics are only secondary amplification IO-jitter mode must add: - whether workload growth or hotspot change was ruled out - which exact store or instance is the outlier - which quantile exposed the tail - whether the best evidence points to raft-engine, RocksDB-KV, or node / cloud-disk layer - whether node-level proof exists or the conclusion is still only `suspected IO jitter` Network mode must add: - which hop is affected - whether the issue is interruption, jitter, or bandwidth saturation - which metric proves transport time is dominant - what evidence rules out gRPC stall or TiKV local bottlenecks Flow-control mode must add: - which exact rejection or throttle mechanism fired - whether it is write-side or read-side - which metric or log proves the threshold crossing - which lower-layer bottleneck triggered the protection Apply-wait mode must add: - whether the long tail is concentrated on one or a few hotspot regions - which slow-log fields tie the SQL digest or DML family to those region IDs - the size-check and split timeline for the hotspot regions - whether the hotspot remained on the same table / index family after split - whether `apply wait` is the direct bottleneck or is being amplified by snapshot / ingest side effects Read-path dominant cases must add: - whether the earliest proven issue is `queueing before execution`, `snapshot acquisition`, or `handler execution` - whether `gRPC coprocessor duration` is the primary bottleneck or only the visible fallout of downstream queueing - whether the pressure is `localized_read_pool_saturation` or true instance-wide CPU saturation - when the first bottleneck is queueing before execution, prefer the canonical label `unified_read_pool_queueing` - the minimal evidence bundle: `cop wait`, `schedule wait`, `running tasks`, `time used by level`, `handle / task exec`, and optional `read-pool CPU` versus CPU quota Snapshot-backlog mode must add: - whether balancing was blocked by snapshot progress rather than PD scheduling itself - which stage was slow first: generate, read IO, send, or apply - whether delete-range backlog or shared worker-pool pressure is suspected, and what evidence is still missing before naming the lower-layer root cause ## Skill Evolution Maintainer-facing evaluator feedback policy and promotion rules live in: - `references/skill-evolution.md` Keep that policy separate from the operational diagnosis flow so normal usage stays focused on routing, evidence, and output quality. ## Guardrails - Do not invent cluster IDs, regions, providers, deploy types, application UUIDs, or infra UUIDs before resolving metadata. - Do not declare TiKV healthy from averages alone; use quantiles first. - Do not start deep TiKV diagnosis until the split between TiKV-side latency and RPC / network latency is made. - Treat hotspot, workload-shape change, and imbalance as first-class possibilities before jumping to disk or CPU bottlenecks. - Prefer per-store or per-thread evidence over cluster averages when only a subset is slow. - If a version-specific bug or metric bias is involved, name the affected versions explicitly. - If the evidence only proves correlation but not causation, say so. - In leader-drop cases, do not equate `node became slow` with `PD evict-slow-store`; PD scheduler evidence is required. - In IO-jitter cases, do not treat average `Disk wait latency` as decisive proof; long-tail issues often need higher quantiles, `max`, or cloud-disk evidence. - In network cases, do not label the issue as TiKV local slowness if the dominant time is on transport or PD-client paths. - In flow-control cases, do not recommend threshold tuning as the main RCA unless the triggering bottleneck is also explained. - If the issue body contains a Clinic or Grafana link but you did not actually consume it, list that link-derived evidence as `missing evidence`; do not write the output as if metric proof already exists. - Treat `Clinic data_status != 100` as a standard `missing evidence` reason, not as proof that the cluster had no issue. - Treat Grafana links that only contain relative windows such as `from=now-3h&to=now` as a standard `missing evidence` reason until an absolute historical window is supplied. - Do not let L1 or ticket-body hypotheses such as `backup`, `checksum`, or `DDL` become the replay-confirmed root cause until thread CPU, workload change, and latency impact are checked.