---
name: diagnosis-overview
description: |
  Investigation-level discipline for TiDB Cloud diagnosis: the claim-verification
  guide mapping claim types (query slowness, CPU, OOM, write latency, errors,
  replication lag) to their direct measures and verification method. Load before
  planning an investigation or verifying any user-asserted premise, collector
  hint, or hypothesis. Pairs with tidb-clinic-diagnosis (component analysis) and
  tidb-knowledge-base (tier routing, pitfall catalog); it does not replace them.
---

# Diagnosis Overview

The core loop of every investigation: **collect evidence → surface anomalies → reason from anomalies to cause**. This skill carries the discipline that keeps that loop honest.

## Overview Comparisons

### Comparable Peers and Nodegroup Boundaries

Use nodegroup labels supplied by the evidence; do not infer membership from node
names. Compare like-for-like peers within the same nodegroup and analyze groups
separately before a cluster-wide summary. A cross-group comparison must state
capacity, quota, and workload/routing differences before treating a difference
as anomalous; different groups are not interchangeable peers. If membership or
comparability is unknown, record that limitation rather than declaring a hotspot.
Use "non-default node group (ac-xxx)", not "expansion group", for ac groups.

The first qualitative read of any slowness investigation is two per-entity latency ladders, before any component dive:

1. **TiDB query latency by instance** — `tidb_server_handle_query_duration_seconds` p99 per TiDB instance, vs peers and baseline. Locates whether slowness is cluster-wide or instance-local at the SQL layer.
2. **TiKV gRPC latency by store** — `tikv_grpc_msg_duration_seconds` p99 per store, vs peers and baseline. Locates whether the storage tier is implicated and on which store.

The sweep collects both and flags skew via the `tidb_latency_skew_present` / `store_latency_skew_present` anomaly hints. Read these two ladders first: they decide the tier (SQL vs storage) and the entity (which instance/store) that the rest of the plan investigates. CPU, compaction, and other signals are mechanism candidates ranked after the latency ladders place the problem.

## Resource Cross-Check

Once a component's latency stands above its peers, compare its resource-class metrics against the same peers in the same window — the resource that is anomalous on the same entity and timeline is the leading mechanism candidate:

| Resource | What to compare (per instance/store, vs peers + baseline) |
|---|---|
| CPU | process CPU vs quota; TiKV thread-pool CPU (readpool, apply, raftstore) |
| Disk IO | IO utilization, read/write latency and throughput; cloud-volume throttling (EBS exceeded-IOPS) |
| Memory | usage vs limit, heap growth, OOM/restart events |
| Network | per-store RPC network latency (`tidb_tikvclient_rpc_net_latency_seconds`) |

Reading the result:

- A resource anomalous on the same entity and timeline → mechanism candidate (disk IO saturated → storage path; CPU saturated → compute path); descend that branch of the latency tree below.
- No resource anomaly → the slowdown is a software path: write stall / flow control, latch/lock waits, scheduling — check the throttle-path signals.

Finding the metric: sweep evidence first (`metrics.json`, raw matrices), then the latency-breakdown reference below. If the metric you need is in neither — or the component has no recipe at all (TiCDC, TiFlash, PD internals, ...) — follow `references/metric-discovery.md`: one lookup in the PSC panel knowledge base lists the ready-to-use PromQL, then execute through `clinic-cli` or a targeted collection. Never guess metric names one call at a time, and never invent PromQL when a panel exists.

## Latency Breakdown

Once the ladders place the problem, walk the latency decomposition tree top-down: measure each stage for the affected entity vs peers and baseline, descend into the stage that dominates, and stop at the first stage that explains the delta.

1. **TiDB e2e** = token wait + parse + compile + execute. Parse/compile dominating → SQL text / plan-cache side; execute dominating → descend.
2. **Execute** = TSO wait + KV requests. TSO wait high → PD path; KV request high → descend.
3. **KV request** = batch-client wait + encode + TiKV gRPC + network. A gap between TiDB-side request duration and TiKV-side gRPC duration → batching or network, not TiKV. gRPC high on one store → descend into that store.
4. **TiKV gRPC** by request type: reads = latch/scheduler wait + snapshot + seek/block read; coprocessor = wait (snapshot/schedule) + handler build + handle; writes = latch wait + reads + async write.
5. **Async write** = propose (wait + send) → commit = max(persist raft log, replicate to followers) → apply (wait + RocksDB write). Write stall / flow control shows up as apply-side `write_scheduling_flushes_compactions_time` and apply/persist wait growth — this is where background debt becomes foreground latency.

Exact stage→metric mapping, formulas, and PromQL templates: `references/latency-breakdown-metrics.md` — read it on demand when walking the tree, not by default.

## Evidence-Weighing Principles

Apply these before promoting any primary cause; they encode recurring misattribution patterns.

- **Capacity differential**: when comparable workload runs against reduced effective capacity — fewer TiDB nodes than before, a smaller nodegroup saturated while a larger peer nodegroup stays normal under similar load, or a before/after scale event — treat `resource_control_or_capacity` as the leading candidate. Query-level signals on the saturated side (TopSQL rankings, plan cache miss or hit=0, GC pressure, TiKV read hotspots) are usually existing workload carried by the reduced capacity; promote one to primary only when the evidence shows it changed independently of the differential (for example a new dominant digest that alone explains most of the incident load and is absent from the baseline). State which query-level signals you evaluated and why you rejected them as differentiators, and support the classification with at least two capacity-related `signal_tags` using `baseline=peer_nodes` or `baseline=before_after_event`.
- **Access-path deficiency**: when expensive-query or slow-query evidence, plan/access-path bottleneck signals, and storage-side hotspot impact form a consistent chain, treat `sql_plan_or_query` as the leading candidate and record the hotspot impact in `candidates`. Describe the mechanism the evidence actually shows (large scan, full scan, ineffective index) rather than a generic label. Never attribute to statistics staleness or a missing ANALYZE without direct evidence — record it in `rejected_classes` with the reason when rejected. Prefer access-path remediation (index, composite index, SQL binding) before secondary tuning.
- **Rank is not cause**: a TopSQL digest's rank alone does not make it the primary cause. Promote a digest only when it independently explains most of the symptom. Many modest contributors plus a capacity/configuration differential → prefer the differential as primary and record `sql_plan_or_query` in `rejected_classes` with the reason no single SQL explains the differential better.
- **Capacity vs SQL tie-breaker**: when `resource_control_or_capacity` and `sql_plan_or_query` are close, choose the class whose evidence directly explains the incident timeline and cross-instance differential. Capacity stays primary while the SQL signals only describe the workload carried by reduced capacity; SQL becomes primary only when a query or access-path change independently explains the before/after delta.
- **Mechanism beats pressure**: profile, stack, source-code issue, or known-fixed-version evidence is mechanism evidence. When it explains the user-visible latency path, rank it above generic workload or capacity pressure; keep workload/capacity as trigger or amplifier unless it better explains the full timeline on its own.
- **Prefer the narrower mechanism**: store-local stall indicators (slow score, RPC latency skew, warning logs) establish the affected component and timing. Use them as primary only when they are the most specific available mechanism; otherwise prefer the narrower engine, thread-pool, scheduling, or storage mechanism evidenced by the same timeline.
- **Overlap is not cause**: a cluster event, restart, scale-out, CDC leader message, or PD scheduling log is not the primary root cause merely because it overlaps the incident window — it must explain the primary symptom better than the query, lock, CPU, and plan evidence. When TopSQL/SlowQuery/decoded_plan and gate checks identify a dominant SQL or lock path, overlapping events are secondary unless metrics prove they initiated that exact path.
- **CPU hotspot chain**: for TiDB CPU hotspots, prioritize `single TiDB CPU saturation -> dominant TopSQL or SlowQuery digest -> decoded_plan/statistics/access-path mechanism`; a scale event or restart is not primary when a dominant SQL path explains most of the CPU.
- **Lock path first for tail latency**: with lock-contention evidence (`pessimistic lock retry limit reached` logs, lock manager metrics, a dominant point-write path), prioritize the application write/lock-contention path; CDC `not_leader` or PD/keyspace messages are supporting or triggering context unless the evidence proves no dominant write path exists.
- **Degrade honestly**: if the handoff says decoded plan is missing, present the access-path mechanism as "likely" / "most consistent with", not proven. Still classify `sql_plan_or_query` when expensive-query logs plus slow-query/TopSQL and TiKV hotspot evidence form a complete chain, listing decoded plan / index usage as missing evidence for higher confidence.
- **Rejection needs its own dimension too**: clearing a candidate is a claim and carries the same burden as promoting one. Reject a component only on direct data-path evidence in its own dimension over the full incident window — the error logs and latency of the operation that supposedly works (write/produce/apply, client-library errors), fetched without keyword filters tuned to a different hypothesis. A fast control-plane probe (metadata/describe/ping) exercises a different code path than the failing data path; an empty filtered or time-sampled slice only proves the fetch did not cover it; and a dimension whose collection failed wholesale (a metric query returned empty, a whole panel group errored out with HTTP failures or tenant-token errors, logs for that path were never fetched) was never observed at all. None of these clear anything — mark the candidate unverified, not rejected, and never let the class you happened to collect win by default over a class whose dimension you could not see. When the missing dimension is load-bearing, one more targeted collection beats concluding around the hole.
- **Background debt hurts through a throttle path**: background pressure (compaction backlog, GC debt, ingest) does not slow foreground queries by itself — it becomes user-visible latency only through the throttle mechanism it triggers: RocksDB write stalls, TiKV flow control / admission control, scheduler backpressure ("server is busy"). When a single store is slow and such debt is elevated, verify the throttle path on that store (write stall metrics, scheduler too-busy / throttle counters, flow-control gauges, "server is busy" or write-stall log lines — fetch them via a targeted collection if absent from the sweep) and report it as the primary path, keeping the debt itself as a surface signal. Remediation targets the flow-control path (admission control, throttling, stall thresholds), not the debt alone.

## Claim Verification Guide

Any claim — from the user, from the collector handoff, or your own hypothesis — must be verified in its own metric dimension against peers and a baseline window before it is used. A proxy dimension can corroborate a verified claim but can never confirm or refute one. Map the claim to its direct measure:

| Claim | Direct measure (per entity, vs peers + baseline window) |
|---|---|
| "queries slower" / latency on a TiKV store | `tikv_grpc_msg_duration_seconds` p99 by instance; store slow score; slow-query duration attributed to that store |
| "queries slower" on a TiDB instance | `tidb_server_handle_query_duration_seconds` p99 by instance |
| high CPU | process CPU by instance vs quota / utilization |
| OOM / memory pressure | memory usage and heap metrics; OOM and restart logs |
| writes slow | apply/commit duration, raft propose wait, disk IO time |
| errors / failures | failed-query rate; error log counts backed by raw samples |
| replication lag | CDC / changefeed lag metrics |

If the direct measure is missing from the compacted summary, fetch it from the full raw matrices (`read_raw_evidence.py` under `tidb-clinic-diagnosis/scripts/`) or a targeted collection — absence from a top-N view is not normality; a compacted view only proves the row was cut.

Record the outcome per claim in the `premise_verification` field of the analyst artifact: `claim`, `dimension` checked, `method`, `verdict` (`confirmed` / `refuted` / `unverifiable`), and the `comparison` numbers. A refuted claim is a finding — report it and re-plan around what actually is anomalous.

## Conclusion Contract

Every conclusion — the analyst's or the investigator's direct one — must pass these gates before it ships. Stopping short of any is the most common way a technically-correct investigation still produces a weak diagnosis. Run them in order: weighing selects the primary before you name its mechanism, never the reverse — deep-diving the first mechanism you can collect is how a real-but-secondary cause gets promoted over the true primary.

1. **Weighing selects the primary class first.** Score the candidate classes and let the Evidence-Weighing Principles pick the primary; per candidate, `evidence_for`/`evidence_against` cite the principle and the differential that decided it ("capacity differential: plan-cache hit=0 is carried workload, not a new mechanism"). A prominent or well-evidenced secondary signal — plan-cache stats, a genuine missing index, a co-occurring event, background debt, load-balancer rebalancing — becomes primary only through the principles, never because it stood out most or because its dimension was the one you happened to collect. When a competing class's evidence dimension was never observable — a metric query returned empty, a whole panel group failed to collect (mass HTTP errors, tenant-token failures), logs for that path were not fetched — you cannot rule that class out by its absence: cap your `classification_confidence`, keep it a live candidate, record the unobserved dimension in `missing_evidence`, and prefer one more targeted collection over concluding around the hole. A failed or empty collection is not a negative finding (pitfall catalog `ic-006`).
2. **Name the terminal mechanism of the selected primary.** Once weighing has chosen the primary class, state the root cause as the narrowest named mechanism path the evidence supports — "compaction backlog → RocksDB write stall → unified read pool queueing on store-9", not "TiKV is slow" (a component) or "compaction pressure" (a pressure). Walk the chain until the next step would need evidence you do not have, then say exactly which evidence would extend it. Do not deep-dive the mechanism of a class weighing did not select as primary. When profile or stack evidence exists (heap profile, goroutine stacks, panic trace) for the selected path, the dominant path it shows must appear in the chain — a profile left uninterpreted is a gate failure, not an optional extra.
3. **Correlate with known issues.** When the mechanism carries an error signature, panic stack, or version-change context (the `known_issue_correlation_context_present` hint, "fixed in" events, recent upgrades), match the signature against known issues — via `github-issue-search` in a targeted collection when local evidence alone cannot name it — and cite the issue id and fixed version in the conclusion. An upgrade recommendation states the version relation explicitly (running vX.Y < fixed vX.Z).
4. **Remediation covers four horizons.** Immediate mitigation; root-cause fix (with the fixed version when a known issue matched); prevention for the failure mode the incident exposed (capacity protection, resource control, limits) whenever the incident showed an unprotected path; and a follow-up investigation item for every unresolved thread in `missing_evidence` — an unexplained early signal (a worker disappearing, an unexplained restart) gets a named next step, not silence.

## Evidence-To-Output Materialization

Reasoning that stays only in scratch text does not count. Before finalizing, map every load-bearing evidence class into the structured report and the final answer:

- **Critical evidence hints:** if a hint changed the diagnosis, cite the underlying artifact in `key_evidence` and carry it into the causal chain. If the hint was investigated and rejected, explain the direct evidence that rejected it. If it was not verifiable, mark the exact missing evidence.
- **Profile / stack / known issue:** profile and stack evidence must be interpreted into a dominant path, not merely mentioned. If it correlates with a known issue, name the issue id when available, state the fixed-version relationship, and make the upgrade or fixed-version action explicit.
- **Capacity differential:** when node count, worker count, quota, resource group, or nodegroup capacity changed, state the capacity delta as the mechanism. TopSQL rank, plan-cache hit=0, GC, or hotspots are secondary unless they independently explain the before/after delta.
- **Plan / access-path shift:** when plan evidence exists, say whether the mechanism is plan shift, full/large scan, ineffective index, missing composite index, or another access-path problem. "SQL was slow" is a symptom, not a root-cause mechanism.
- **Background debt / TiKV stall:** pending compaction, slow score, and warning logs place impact and timing. A root cause must connect them to the throttle path that creates foreground latency: RocksDB write stall, flow control / admission control, scheduler backpressure, read pool queueing, unified pool saturation, or the missing evidence that would prove that link.
- **TiCDC pipeline:** separate source/puller health from downstream pipeline and sink health. If source and puller are healthy while lag persists, report the downstream TiCDC pipeline after puller and name sink / Kafka / proxy / broker / TLS follow-up.
- **Remediation:** every primary mechanism needs at least one immediate mitigation and one durable fix/prevention item. Capacity or worker-loss incidents require capacity protection recommendations; unexplained worker disappearance, restart, or early event requires a named follow-up.

## Extending This Guide

New claim types get a new table row here; observed misdiagnosis patterns belong in `tidb-knowledge-base/knowledge/incorrect-conclusions.json` — not in agent prompts.