---
name: tidb-latency-diagnosis
description: Diagnose TiDB query, transaction, connection, storage, network, PD, and TSO latency incidents with a narrow evidence-first workflow. Use when a TiDB support case reports latency spikes, QPS drops, slow requests, slow connections, storage jitter, PD/TSO waits, or component-to-component network delay and still needs cluster-side evidence. A supplied screenshot, trace, or log that names a latency mechanism selects the first evidence branch; it does not make the skill unnecessary. Do not use for a self-contained question fully answered by a supplied execution plan or artifact without provider queries, or for non-latency OOM, scaling, merge-region, TiCDC, Kill/Cancel, or transaction-flow reference questions.
---

# TiDB Latency Diagnosis

Use this skill to choose one latency branch, rank direct evidence, and produce a
bounded diagnosis. Provider routing and authentication remain owned by the
runtime and provider-specific skills.

## Boundaries

- Keep all access read-only. Never restart, scale, kill, cancel, change
  configuration, add or remove schedulers, execute DDL/DML, or perform a
  production operation.
- Do not replace Investibook, Clinic, Wetech, or Manus routing. Never fall back
  to another account or environment when the selected provider is unavailable.
- Do not copy historical ticket bodies, customer identifiers, screenshots,
  human replies, bug catalogs, or case IDs into the diagnosis.
- Do not infer current behavior from product source paths, old dashboard names,
  or an unverified configuration default.
- If a supplied execution plan, profile, trace, log excerpt, screenshot, or
  normalized summary fully answers the question without provider queries, use
  it directly and do not load a generic latency reference merely to add more
  context.
- If an artifact names a direct wait or latency mechanism but cluster-side
  confirmation is still needed, use that mechanism to select the branch. The
  first provider query batch must confirm it before collecting generic health
  or baseline metrics.

## Historical Case Freshness

Use an explicit incident time when reliable; otherwise use Jira creation time.

- `<= 90 days`: a case may be a trusted candidate example only when product,
  deployment type, version behavior, symptom, and evidence path are compatible.
- `> 90 days and <= 365 days`: use only to generate a hypothesis or identify a
  missing check. It cannot prove the current root cause, threshold, default,
  workaround, fixed version, or product behavior.
- `> 365 days`: discard it. Do not load, cite, summarize, or use it.
- Version incompatibility overrides freshness. Treat a recent case about
  behavior fixed or removed for the target version as non-transferable.
- Validate version-dependent statements against current official documentation
  or release notes. Recency does not replace version verification.

## Select One Branch

Load at most one reference for the initial diagnosis:

- Storage or one slow TiKV/TiFlash entity: read
  `references/storage-latency.md`.
- PD wait, StartTS, TSO RPC/wait, transaction-start delay, or PD-side latency:
  read `references/pd-tso-latency.md`.
- Connection establishment, RPC/network delay, disconnect, timeout, or
  bandwidth symptom: read `references/connection-network-latency.md`.

If the strongest direct evidence belongs to another mechanism, stop using the
initial branch and switch once. Do not load all references as a checklist.

## Evidence Order

1. Read available ticket artifacts and normalized diagnostic summaries.
2. Start with the customer-visible result metric and identify the entity that
   is actually abnormal plus the exact incident samples. Do not choose the
   entity from CPU or workload rank alone.
3. Identify the most direct time-aligned signal for the customer-visible
   latency on that same entity.
   Wait, backoff, contention, queue, throttle, retry, or spill evidence outranks
   generic CPU, GC, P99, or utilization.
4. When provider evidence is required, make the first query batch no more than
   three checks: the direct mechanism, its server/client counterpart, and only
   the strongest alternative. Do not query generic CPU, GC, overall SQL P99, or
   broad TopSQL before this batch.
5. Use one matching reference to define that first batch. Provider-specific
   diagnosis skills must preserve this order rather than replace it with a
   broad sweep.
6. Compare the same entity and time window with peers or a nearby normal
   baseline.
7. Record missing and failed dimensions as `unverified`, never as healthy. If a
   direct metric is unavailable, record that gap before broadening the search.

Lock the incident entity and time before mechanism exploration. If a reliable
direct-result artifact supplies both, reuse that tuple and do not replace it
with a wider query, dashboard legend, window maximum, CPU rank, or workload
rank. If only the event time is supplied, query the direct result grouped by
entity in a narrow window, locate the peak or change point, and choose the
entity at that same sample or within one query step. If neither is supplied,
discover the direct result interval before selecting its aligned entity.

Provider evidence must remain progressive. Use compact command output and the
compact report for reasoning; keep raw metrics, TopSQL, Statements, Slow Query,
plan, and log payloads out of context by default. If one field is load-bearing,
run one narrow extraction by exact entity, Digest, field, or time slice instead
of reading the full raw artifact. Use TopSQL only for workload attribution or
one explicit load-bearing workload gap. Before an allowed TopSQL query, select
exactly one affected instance from the entities aligned to the direct result
sample for that component and request only its Top 10 Digests.

Do not promote a root cause from a single proxy metric. Require two independent
observed evidence classes for a confident conclusion; otherwise label the
diagnosis preliminary.

For transient backoff, disconnect, unreachable, leader-change, or Region
movement incidents, do not finalize from a single latency branch. Before
causal synthesis, record a three-row fault-domain matrix for
`host_process`, `component_network`, and `scheduler_leadership`. Give each row
`supported`, `contradicted`, or `unverified`, with the affected entity, aligned
window, direct evidence, strongest counter-evidence, and one bounded next
check. Generic reachability does not prove process health; a proposed network
cause does not prove the component path; leader movement is an outcome until
direct scheduling, election, term, or Region evidence establishes its trigger.
Never omit an unverified row or promote evidence from one domain into another.

## Causal Alignment

For every load-bearing result and candidate mechanism, retain the metric or
artifact, entity labels, sample timestamp, query step or interval, and value.
Validate the mechanism against the same entity and exact sample timestamp. If
scrape times differ, state an alignment tolerance no wider than one query step,
or use a directly overlapping incident interval.

Window maxima, averages, percentile summaries, compact previews, and top-N
ranks locate candidates but do not prove concurrency. Never join maxima from
different timestamps into one causal chain. Never join different instances or
components without independently observing the dependency path.

TopSQL cumulative CPU rank, statement count, or an `ANALYZE` entry in the
window does not establish workload causality. Require instance-scoped workload
evidence aligned with the incident samples, or a direct trace or log connecting
that workload to the observed wait path.

For a write-path mechanism objective, complete the aligned scheduler command
and Commit/Prewrite latch chain and check one strongest storage alternative
before finalizing. Compare each stage with the end-to-end result. Treat CPU as
pressure context only, and do not run TopSQL, Statements, or Slow Query merely
to complete a table.

When only aggregate evidence is available, request one bounded per-point,
per-instance series for the affected result and leading mechanism. If the
provider cannot return it, mark the causal conclusion `provisional`, state the
missing aligned series, and do not substitute a broader generic sweep.

## Stop Condition

Stop collecting and write a bounded preliminary conclusion when all are true:

- a direct artifact or metric accounts for a material part of the visible
  latency;
- its client/server counterpart confirms the path or narrows the boundary; and
- the strongest alternative has been checked or named as one exact unverified
  gap.

At that point, do not run a metric-name inventory, generic resource sweep,
TopSQL, Slow Query, or extra baseline queries merely to fill a completeness
table. Continue only when one named check can overturn the leading conclusion.

## Output Contract

Keep the customer-facing draft concise. Put the following internal structure in
the detailed diagnosis:

1. **Current conclusion** — one sentence with the confirmed boundary.
2. **Direct evidence** — the strongest time-aligned fact first.
3. **Causal path** — one evidence-supported chain to the visible symptom.
4. **Ruled out / unverified** — distinguish negative evidence from missing
   evidence.
5. **Next check** — only the smallest check that can change the conclusion.

Do not output a full troubleshooting tree or historical case list.