---
name: root-cause-classification
description: Classify TiDB Cloud diagnosis evidence into stable root-cause families, symptom tags, and schema-compatible fault classes before final diagnosis synthesis.
---

# Purpose

Use this subskill after evidence collection and before writing the final root cause.
It turns `critical_evidence_hints`, `signal_tags`, TopSQL/SlowQuery, metrics, logs,
and component health evidence into a stable classification that downstream diagnosis
can use for wording, remediation, and specialist-skill routing.

This subskill does not collect evidence and does not override raw evidence. It only
classifies already collected evidence.

# Inputs

- `critical_evidence_hints` from `evidences/agent-runs/evidence-agent.json` or
  `evidences/evidence_summary_for_agent.json`.
- `signal_tags` derived by `diagnosis-analyst`.
- Evidence excerpts from `evidences/evidence_summary_for_agent.json` or `.txt`.
- Optional raw artifacts only when the summary is internally inconsistent.

# Symptom Tags

Use symptom tags to describe observed user-facing or metric-facing signals. Symptoms
are not root causes by themselves.

Allowed symptom tags:

- `high_cpu`
- `slow_query`
- `tikv_hotspot`
- `oom_signal`
- `gc_signal`
- `lock_wait`
- `cdc_lag`
- `pd_tso_wait`
- `io_saturation`
- `capacity_pressure`
- `network_latency`
- `panic_or_restart`

# Standard Root-Cause Classes

Score the top 2-3 plausible standard root-cause classes before selecting the final
action-driving primary class. Each candidate must include confidence, role, evidence
for, and evidence against. Prefer the class that best explains why the primary symptom
started or became user visible, but preserve strong contributing classes instead of
collapsing them into prose. Do not classify a symptom such as high CPU as the root
cause.

The final diagnosis still needs one action-driving primary class for remediation and
schema compatibility, but the classification decision must remain auditable through
the candidate list.

After selecting the primary class, read the matching family subskill before final
diagnosis synthesis.

| Standard class | Matching family subskill | Use when the primary mechanism is... | Strong evidence | Specialist routing |
|---|---|---|---|
| `sql_plan_or_query` | `sql-plan-or-query/SUBSKILL.md` | a specific SQL/query workload, inefficient plan, missing access path, full/large scan, plan shift, or query-driven resource pressure | SlowQuery, TopSQL, decoded plan, est/act divergence, missing index, plan shift, SQL binding evidence | `tidb-query-tuning-autox`, `tidb-query-tuning` |
| `transaction_or_lock_contention` | `transaction-or-lock-contention/SUBSKILL.md` | lock contention, transaction conflict, stale/follower read anomaly, or commit/prewrite/lock-resolution path | deadlock, lock wait timeout, pessimistic lock retry, write conflict, `PessimisticLockNotFound`, transaction backoff | `transaction-oncall-handbook` |
| `tikv_hotspot_or_coprocessor` | `tikv-hotspot-or-coprocessor/SUBSKILL.md` | store-local TiKV read/write/coprocessor/unified-read hotspot not better explained by a single SQL plan | TiKV CPU skew, TiKV thread CPU, gRPC P99 on same instance, region/key hotspot, TopSQL-TiKV attribution | `tikv-performance`, `tikv-slow-diagnosis` |
| `storage_or_io_bottleneck` | `storage-or-io-bottleneck/SUBSKILL.md` | device/storage engine bottleneck, IOPS/throughput ceiling, RocksDB write stall, raft/apply/propose latency, or disk exhaustion | disk util, EBS/NVMe exceeded IOPS, write stall, RocksDB/raft latency, slow score, full disk | `tikv-storage`, `tikv-performance` |
| `memory_or_oom_pressure` | `memory-or-oom-pressure/SUBSKILL.md` | memory exhaustion, memory quota pressure, heap limit, OOM kill, or memory tracker failure | OOM kill/restart, heap near limit, memory alarms, large `max_mem`, worker memory, memory tracker stack | `tikv-memory`, `tidb-perf-diagnosis` |
| `gc_or_allocator_pressure` | `gc-or-allocator-pressure/SUBSKILL.md` | allocation churn or Go GC overhead is the main latency/CPU amplifier or root mechanism | `go_gc_duration_seconds`, `go_gc_pauses_seconds`, GC count/sum/bucket, allocator-heavy profile, STW/GC pause spike | `tidb-perf-diagnosis`, continuous profiling |
| `resource_control_or_capacity` | `resource-control-or-capacity/SUBSKILL.md` | insufficient effective capacity, RU/resource-group throttling, worker/coprocessor capacity loss, scale-in/node-count gap, or quota bottleneck | node count, quota, RU, resource manager queue time, worker count delta, scale event, peer/baseline capacity comparison | `pd-resource-control`, `tikv-scale`, `autoscaler-dashboard` |
| `pd_tso_or_scheduling` | `pd-tso-or-scheduling/SUBSKILL.md` | PD/TSO, placement, scheduling, region health, ETCD, or PD operational path | TSO wait, PD leader CPU, scheduler/region checker signals, placement rule changes, region heartbeat, ETCD capacity | `pd-oncall-handbook` |
| `cdc_or_replication_lag` | `cdc-or-replication-lag/SUBSKILL.md` | TiCDC replication lag, changefeed failure, owner/maintainer/dispatcher/sink bottleneck | checkpoint/resolved-ts lag, changefeed state, owner/maintainer metrics, processor/sink errors, downstream latency | `ticdc-health-inspection`, `tikv-cdc` |
| `network_or_connectivity` | `network-or-connectivity/SUBSKILL.md` | network/connectivity failure or communication bottleneck is independent and earlier than component/query pressure | network-lag alerts, connection reset bursts, gRPC errors/latency not explained by TiKV load, packet loss, cross-zone degradation | component-specific skill plus Loki/metrics |
| `component_failure_or_bug` | `component-failure-or-bug/SUBSKILL.md` | component panic, restart, process crash, known bug, or version-specific defect is the best primary explanation | panic stack, restart count, OOM kill, known issue match, fixed release, repeated fatal/error signature | `tikv-panic`, `github-issue-search`, component skill |

# Compatibility Mapping

`diagnosis-analyst-output.schema.json` currently requires `fault_classification.primary_class`
to use the legacy enum. Preserve compatibility by mapping the standard class as follows:

| Standard class | Compatible `primary_class` |
|---|---|
| `sql_plan_or_query` | `sql_plan_or_query` |
| `transaction_or_lock_contention` | `lock_or_transaction_contention` |
| `tikv_hotspot_or_coprocessor` | `tikv_read_hotspot` |
| `storage_or_io_bottleneck` | `storage_or_infra_io` |
| `memory_or_oom_pressure` | `tidb_cpu_gc_or_memory` |
| `gc_or_allocator_pressure` | `tidb_cpu_gc_or_memory` |
| `resource_control_or_capacity` | `resource_control_or_capacity` |
| `pd_tso_or_scheduling` | `service_unavailability_or_worker_failure` unless schema is extended |
| `cdc_or_replication_lag` | `service_unavailability_or_worker_failure` unless schema is extended |
| `network_or_connectivity` | `service_unavailability_or_worker_failure` unless schema is extended |
| `component_failure_or_bug` | `component_restart_or_panic` |

When using a compatibility fallback such as `service_unavailability_or_worker_failure`,
store the more precise standard class in `fault_classification.standard_root_cause_class`.
Candidate `class` values should use the standard class names from this subskill. The
schema-compatible `primary_class` may use the legacy enum from the mapping above.

# Classification Rules

1. Score candidates before selecting the primary. Keep the strongest 2-3 classes in
   `fault_classification.candidates`, sorted by confidence. Use `role=primary` for the
   action-driving class, `role=contributing` when the class materially amplifies the
   incident, `role=secondary` for timeline-aligned but non-driving signals, and
   `role=rejected` only for plausible classes that the evidence rules out.
2. Separate symptom from cause. `high_cpu`, `slow_query`, and `tikv_hotspot` can be
   symptom tags even when the root cause is capacity, SQL plan, IO, or lock contention.
3. Prefer causality over severity. A severe downstream TiKV hotspot is secondary if a
   specific SQL plan or capacity loss explains why the hotspot formed.
4. Prefer specific root-cause classes over broad availability classes. Use
   `service_unavailability_or_worker_failure` only as a compatibility mapping or when
   no more specific schema class exists.
5. A query is primary only when it is leading, dominant, and timeline-aligned. Otherwise
   keep it as evidence or a contributor.
6. GC and OOM are distinct:
   - Use `memory_or_oom_pressure` for heap/quota/OOM kill/process failure.
   - Use `gc_or_allocator_pressure` for allocation churn, GC pause/frequency, or
     profile-confirmed allocator overhead.
7. For TiDB high CPU, always check GC signals. If GC is abnormal but not primary, mark
   `gc_signal` and add `gc_or_allocator_pressure` as contributing or secondary.
8. For recoverable panic caused by query timeout/kill, classify the alert as
   `component_failure_or_bug` only when the panic/known bug itself is the action-driving
   root cause. Otherwise classify the upstream SQL/IO/capacity path as primary and keep
   `panic_or_restart` as a symptom tag.

# Candidate Scoring Guidance

Use confidence as a probability-like score from 0.0 to 1.0:

- `0.80-1.00`: direct, timeline-aligned evidence and no stronger competing mechanism.
- `0.60-0.79`: strong evidence, but one important artifact is missing or another class
  explains part of the chain.
- `0.35-0.59`: plausible contributor or secondary signal, but not enough to drive the
  incident alone.
- `<0.35`: plausible false lead; put it in `rejected_classes` when it could otherwise
  mislead the diagnosis.

Use the candidate comparison to prevent common misroutes:

- Capacity evidence, query evidence, and downstream hotspot/IO evidence often coexist.
  When comparing `resource_control_or_capacity` with `sql_plan_or_query`, choose the class
  that best explains the incident timeline and cross-instance differential. Capacity should
  remain primary when query signals only describe the workload being carried by reduced
  effective capacity; SQL should become primary when a query, plan, or access-path change
  independently explains the before/after delta.
- TiCDC checkpoint lag plus Kafka `EOF`, `broken pipe`, `closeNotify`, Sarama broker
  metadata errors, or Kafka proxy connection errors should keep `cdc_or_replication_lag`
  as the primary class and `network_or_connectivity` as a likely contributor. Lock
  resolver logs are secondary unless they explain the sink-stage stall better than
  the Kafka errors.
- Coprocessor worker count loss, disappeared workers, a remaining overloaded worker,
  restart/OOM loop, QPS collapse, and request backlog should score
  `resource_control_or_capacity` or `component_failure_or_bug` above
  `sql_plan_or_query`, even when slow SQL exists downstream.
- Node-count reduction, fewer TiDB nodes, scale-in, or strong nodegroup capacity
  comparison should score `resource_control_or_capacity` above `sql_plan_or_query`
  unless one new SQL workload independently explains the before/after delta.
- Profile, stack, source-code issue, or known-fixed-version evidence is mechanism evidence.
  When it explains the user-visible latency path, score the corresponding root-cause family
  above generic workload or capacity explanations. Keep workload/capacity as trigger or
  amplifier unless it better explains the full timeline on its own.
- Store-local stall indicators such as slow score, RPC latency skew, or warning logs establish
  affected component and timing. Use them as primary only when they are the most specific
  available mechanism; otherwise prefer the narrower engine, thread-pool, scheduling, or
  storage mechanism evidenced by the same timeline.

# Family Subskill Loading Limit

After scoring candidates, read `root-cause-classification/SUBSKILL.md`, the primary
family subskill, and at most one contributing family subskill with confidence >= 0.60.
Do not read every family subskill. If candidates are close, use the tie-breakers above
instead of broad skill loading.

# Output Shape

Write the classification inside `diagnosis-analyst.json` under `fault_classification`.
The existing required fields must remain present. Add these optional fields:

```json
{
  "primary_class": "sql_plan_or_query",
  "standard_root_cause_class": "sql_plan_or_query",
  "symptom_tags": ["slow_query", "tikv_hotspot"],
  "candidates": [
    {
      "class": "sql_plan_or_query",
      "confidence": 0.86,
      "role": "primary",
      "evidence_for": ["slow query log identifies dominant digest", "decoded plan shows large scan"],
      "evidence_against": []
    },
    {
      "class": "tikv_hotspot_or_coprocessor",
      "confidence": 0.68,
      "role": "contributing",
      "evidence_for": ["hot TiKV gRPC P99 and unified read pressure"],
      "evidence_against": ["hotspot follows query timeline"]
    }
  ],
  "contributing_root_cause_classes": ["tikv_hotspot_or_coprocessor"],
  "ruled_out_root_cause_classes": ["resource_control_or_capacity"],
  "rejected_classes": [
    {
      "class": "resource_control_or_capacity",
      "confidence": 0.18,
      "evidence_for": [],
      "evidence_against": ["no node count reduction or RU throttling evidence"]
    }
  ],
  "classification_confidence": "high",
  "missing_evidence_for_higher_confidence": [],
  "specialist_skill_routing": ["tidb-query-tuning-autox", "tidb-query-tuning"],
  "final_response_anchors": ["slow query log", "execution plan changed"]
}
```

Also keep matching `signal_tags` with evidence refs so the classification can be audited.

# Final Response Anchors

Use classification-specific wording anchors in the final response when evidence supports
the class:

- `resource_control_or_capacity`: say `node count reduction`, `fewer TiDB nodes`, or
  `capacity reduction` when node-count/capacity comparison is the primary mechanism.
- `sql_plan_or_query`: cite `slow query log` or `expensive query log evidence`, plus
  TopSQL/decoded-plan evidence when available.
- `tikv_hotspot_or_coprocessor`: name the hot TiKV instance and thread pool or TopSQL-TiKV
  driver.
- `storage_or_io_bottleneck`: distinguish workload-induced IO saturation from independent
  infrastructure failure.
- `gc_or_allocator_pressure`: state whether GC is primary, contributing, or secondary;
  recommend continuous profiling when abnormal.
- `component_failure_or_bug`: distinguish process crash/restart, recoverable panic,
  known bug, and false alarm. For benign recoverable timeout panics, say no TiDB fix is
  needed for the panic itself if the evidence supports that conclusion.