# General Diagnosis Fallback

This is a library of optional methods, not a second orchestration workflow.
Read only the section needed after the selected family/specialist leaves a
specific reasoning question uncovered. A family with no external specialist
does not automatically require this reference.

## Scope and Precedence

- Record why this reference is needed and which section was read in
  `quality_checks`; include its path in `selected_diagnosis_skills`.
- The agent prompt owns collection, delegation, canonical time window, and
  artifact writing. Methods below that call for more data describe a gap to
  send to evidence-agent, never permission for analyst collection.
- KB guardrails, diagnosis-overview, and specialist evidence/validation gates
  take precedence. These historical checklists are relevant to the named
  component and claim only, not mandatory gates for unrelated diagnoses.
- Do not force a CPU/profile/TopSQL loop for a transaction, CDC, wrong-result,
  or other mechanism already covered by its specialist.
- Missing observations remain unknown; out-of-window evidence remains context.
  Reading more methods cannot manufacture missing incident evidence.
- A missing required specialist is a visible loading gap, not a fallback trigger.
- Finish through the analyst's normal report writer; this reference does not
  define a separate prose report or authorize source-code investigation.

## Nodegroup Boundary Rules for TiDB Instance Comparison

Apply the mandatory Comparable Peers and Nodegroup Boundaries rules in
`diagnosis-overview/SKILL.md`. They apply whether or not this fallback is read.

### Node C: App-layer Triage (database time model)

**REASONING GUARD — Internal operations vs application queries:**
Do not attribute an internal operation from SlowQuery count or an empty digest alone.
Compare application and internal-work candidates using claim-appropriate evidence.
Neither an internal operation nor application workload has unconditional priority:
a verified mechanism can explain heap growth, GC CPU, or failures independently
of TopSQL rank. Apply the KB and diagnosis-overview weighing rules.

**Common traps to avoid:**
- **SlowQuery count bias**: ANALYZE often dominates by count but uses little CPU. Always compare CPU time in TopSQL, not execution count in SlowQuery.
- **Empty digest (`""`) in TopSQL**: Aggregates many internal operations. Do NOT attribute to ANALYZE unless the digest explicitly matches.
- **TiKV query hotspot**: If TopSQL-TiKV shows a single query consuming >3x CPU on one instance, that is a candidate to verify against the same-instance timeline, not proof of the driver.

Use a holistic top-down approach (database time model):
- Is bottleneck inside or outside the cluster?
- If inside: which component? TiDB / TiKV / PD?
- Is there a single slow instance impacting the whole system?

Read `performance-overview-dedicated.json` or `performance_overview_starter-essential-premium.json` according to cluster type.

**For NextGen/TiDB-X (starter/essential/premium) only:**
1. Analyze RCU usage: Read (1 storage read = 0.475 RU; 64 KiB = 1 RU), Write (1 write = 2.55 RU; 2 KiB = 1 RU)
2. Check throttling by Max RCU via `resource_manager_*` metrics or slow query `total time queued by RC`
3. Common issues: TiProxy bottleneck (Essential), Cop worker auto-scale limits (Premium)

### Node D: Component Deep-dive

**If TiDB is bottleneck:**
- The evidence data already groups TiDB instances by nodegroup — use only within-group peers for comparison
- Per-instance outlier: check `tidb query latency by instance` for one instance much worse than peers in the **same nodegroup**
- High resource utilization (cpu, memory), unbalanced connections
- For high TiDB CPU, **must inspect TiDB GC metrics** from `TIDB_GC_CHECK` (`go_gc_duration_seconds`, `go_gc_pauses_seconds`, and related series). If GC duration, pause latency, or frequency is abnormal, explicitly tell the user and recommend continuous profiling to confirm the responsible function/package and allocation path.
- High parse/compile overhead, High TSO wait (>20% of execute time)
- Further check: tidb.json sections (Query Summary, Query Detail, Server, Transaction, Executor, Distsql, KV Errors, KV Request, PD Client, etc.)

**MANDATORY validation order for TiDB CPU issues:**
1. **Same-node time comparison first**: compare the same TiDB instance in abnormal vs normal window
2. **Same-Nodegroup peer comparison second**: compare only with peers in the same Nodegroup
3. **Cross-day same-clock-time comparison third**: compare with previous normal day(s) to rule out periodic business patterns

If you skip step 1 or step 3 and directly conclude "traffic increased", "query load increased", or "load imbalance is root cause", the conclusion is incomplete.

**Cross-day baseline rule:**
- For incidents lasting more than 2 hours, or for obviously periodic workloads, you **must** compare the same clock-time window against at least one previous normal day
- Prefer at least 2 previous days when available
- If "previous days at the same time had no issue", treat cross-day comparison as required evidence, not optional supporting context

**Optional deepening for an unresolved TiDB CPU mechanism:**
Use the relevant checks below when existing family/specialist analysis cannot
explain CPU. They are not a second full-collection prerequisite. If a necessary
artifact is absent, return the precise evidence request through the normal
supplemental-pass contract rather than starting a new collection loop.

0. Review collected CPU/GC metrics, query history, plans, TopSQL, statement summary, and slow query samples as relevant to the hypothesis; use CPU profiles when available. Missing artifacts limit the corresponding claims.
1. Define both the problematic time window and a comparable non-problematic baseline window. Prefer the same time-of-day pattern.
2. For Dedicated/Premium, compare incident-window and baseline CPU profiles when profile bytes are available. For BYOC, use profiling inventory only; do not require profile downloads or flame graphs.
3. Work backward from the hot stacks. Infer which query or plan patterns could produce them. Consider optimizer, executor, compiler, GC, memory tracking, range building, and internal SQL paths instead of assuming all CPU comes from user SQL.
4. Check TopSQL, statement summary, and slow query records in the same window. Collect candidate SQLs together with their plans, and keep internal SQL in the candidate set until it is ruled out.
5. Build the smallest candidate query set that still explains the CPU symptom well. Use statistical validation, such as correlation analysis, principal-component-style reduction, variance contribution, or combinational reasoning, to find the minimum SQL combination that remains highly correlated with the incident signal.
6. Compare candidate queries against the observed symptom shape, not just top-N totals. Check correlation, periodicity, dispersion, and confidence across time slices. CPU spikes may align better with GC pressure, compile/plan-building time, or execution hot loops than with raw query count.
7. Require mutual validation across at least two independent sources. For Dedicated/Premium, include flame graphs when profiles are available; for BYOC, validate with CPU/GC, TopSQL, Statements Summary, Slow Query samples, and plan evidence.
8. If the candidate set still does not explain the symptom, report the remaining hypotheses and the precise missing comparison. Any supplemental collection is owned by evidence-agent and bounded by the investigator's normal pass limit.

**Decision rule for the next action after high TiDB CPU analysis:**
- If a small candidate SQL set explains the CPU symptom and the plan is obviously inefficient, route to query tuning / plan inspection.
- If the hot stacks point to plan building or plan instability, inspect plans and bindings before blaming execution.
- If the evidence points to stale or suspicious estimates, route to statistics diagnosis.
- If internal SQL, GC, compiler, memory tracking, or range building explains the symptom better than user SQL, investigate the internal path first.
- If the stacks and evidence still do not fit known workload behavior after the expanded profile/baseline loop, treat it as a product bug candidate and prepare a high-quality bug report with the retained evidence set.
- Do not present a final root cause unless the conclusion is supported by statistical validation and at least two independent evidence sources. BYOC profiling inventory is capability evidence, not a substitute for profile bytes.

**If TiKV is bottleneck — resource-first layered diagnosis:**

1. **CPU saturation check (absolute threshold — MUST check first)**:
   The gate check reports `cpu_saturation` when any instance's peak CPU exceeds 80% of its quota.
   An instance at >80% utilization IS a hotspot signal, even if its ratio to peers is below 2x.
   If `cpu_saturation` appears in the Hotspot verdict, treat that instance as the primary hotspot candidate.
2. **Instance identification (relative threshold)**: From the evidence, identify which TiKV instance(s) are outliers (CPU/memory/IOPS >2x peers). This catches imbalance even when no single instance is saturated.
3. **Thread-level classification** (if you need more data, query `sum(rate(tikv_thread_cpu_seconds_total{component="tikv"}[1m])) by (instance, name)` on the hotspot instance):
   - `grpc` threads hot → gRPC hotspot
   - `cop` / `coprocessor` threads hot → read hotspot (coprocessor)
   - `raftstore` threads hot → raft/write hotspot
   - `sched` threads hot → transaction contention
   - `unified-read` threads hot → point-get / short-scan read hotspot
4. **TopSQL attribution**: Use TopSQL TiKV only after confirming the same TiKV hotspot instance has a fast CPU rise during the incident window. The query with highest `cpu_pct` is only a root cause candidate when it dominates the hotspot instance CPU and aligns with the incident timeline.

**CRITICAL: Distinguish "Slow" vs "Hot":**
- P99 latency high ≠ CPU hotspot. ANALYZE may show P99 = 10s but only 2% of total CPU.
- A frequent SELECT with P99 = 200ms running 100x/sec may consume 55% of CPU.
- Use TopSQL `cpu_pct` for attribution, NOT P99 latency or SlowQuery count.

**CPU attribution:** Use same-instance TopSQL CPU measurements with timeline and
mechanism evidence. A small percentage weakens a CPU-dominance claim; it does not
rule out latency, lock, memory, or IO mechanisms. Absence from a top-N view is
not proof of absence.

**REASONING GUARD — `statement_count` must be paired with `affected_rows`:**
For `DELETE` / `UPDATE` / `INSERT ... ON DUPLICATE KEY` / batch DML style workloads, execution count alone is **not** enough to prove workload growth or CPU causality.

- You **must** check both:
  - statement execution rate (`statement count`, QPS, or execution frequency)
  - actual work done (`affected_rows`, scanned rows, processed keys, total keys, scan keys, KV requests)
- If `statement count` rises sharply but `affected_rows` stays flat or very low, treat this as a **high-overhead / low-row-effect** pattern, not as proof of larger business data volume
- In that case, prioritize investigating framework overhead, empty scans, repeated retries, plan inefficiency, or per-statement fixed cost
- Do **not** say "business traffic increased" from statement count alone

**REASONING GUARD — prefer same-node time comparison before cross-node attribution:**
If one TiDB instance or one Nodegroup is suspected, compare the **same instance** during:
- incident window vs nearby normal window on the same day
- incident day vs previous normal day(s) at the same clock time

This check has higher priority than cross-node comparison because it removes routing / workload-placement bias.

**REASONING GUARD — sustained issue vs short spike:**
Do not infer the root cause from a few 1-minute CPU spikes alone.

- If CPU is reported as "持续升高 / periodic high", validate that the suspect metric also shows a sustained or repeatedly aligned pattern across the broader incident window
- Always compare at least two granularities when possible:
  - 1-minute view: identify sharp spike correlation
  - 10-minute or broader window: verify whether the correlation is sustained enough to explain the incident
- If a metric only matches a few spikes but does not explain the prolonged high-CPU window, it is **not sufficient** as the primary root cause

**DML workload interpretation rule:**
- For `DELETE` / `UPDATE` / write-heavy framework SQL, always record:
  - `statement_count`
  - `affected_rows`
  - `process_keys` / `total_keys` if available
  - CPU / memory allocation signal
- If `statement_count` rises by tens of times but `affected_rows` stays nearly flat, describe it as:
  - "execution overhead increased"
  - or "high-frequency low-row-effect DML"
- Do **not** describe it as "more rows were deleted/updated" unless affected_rows or scan work also increased materially

**SQL mechanism routing:** Read `sql-plan-or-query/SUBSKILL.md` and its Specialist Loading Contract; the SQL gate has one source there. Do not use this general reference to bypass AutoX.

**If PD is bottleneck:** High CPU/memory, slow TSO, slow region heartbeat. Check pd.json.

**If TiFlash is involved:** High CPU/memory/IO. Check tiflash.json.

**If Premium coprocessor-worker or tikv-worker is suspected:**

Premium coprocessor-worker and tikv-worker pods are queried via `parent_id`, not `cluster_id`.
When coprocessor P99 latency or TiKV gRPC latency spikes, check **A2 component health** before
query-level attribution. Interpret component count drops as scale-down/crash, restart counts as
process restarts, and worker memory peaks plus series gaps as likely OOM. If the data is empty on
a Premium cluster, request a targeted evidence pass to verify `parent_id` scope. If worker loss/restarts align with
latency or QPS impact, treat capacity failure/OOM cascade as a root-cause candidate before blaming SQL.

### Node E: Cross-check, Synthesize, and Conclude

**PRE-CHECK GATE (must pass before writing any conclusion):**
1. If a concrete SQL was identified in Node D → did you evaluate the SQL specialist candidate routes and causal checks? If no route passed, record why it is affected workload rather than a primary or material contributing mechanism.
2. If a SQL specialist route passed → did you load `tidb-query-tuning-autox` and record its mechanism classification plus validation level? If NO → **STOP**, load it from `.opencode/.all-skills/` now.
3. **KB pitfall gate:** Did you already complete **KB Step B** at analysis start (`read("skills/builtin/tidb-knowledge-base/knowledge/incorrect-conclusions.json")` before evidence interpretation)? If your thesis changed materially since then, **re-read** the file and confirm you still satisfy `conclusion_rule` and `corrective_checks` for any matched entry. If unsafe → **STOP**, apply checks and revise.

Only after ALL gates pass, proceed to cross-reference validation below.

**Cross-reference validation (MANDATORY):**

Verify evidence from **at least 2 independent data sources**:
- [ ] **Metrics evidence**: At least one metric supports root cause
- [ ] **Query evidence**: claim-appropriate runtime evidence identifies the SQL; plan/schema/statistics evidence is present for any operator or optimizer-change claim
- [ ] **Log/Event evidence**: Supports conclusion OR explicitly rules out alternatives

**Additional cross-checks:**
1. **Infra verdict from gate check** — is it incorporated? If EBS throttle was severe, evaluate its causal path and timing; missing EBS data remains unknown.
2. **Timeline alignment** — do infra, app metrics, and slow queries correlate temporally?
3. **TopSQL cross-validation** — if attributing CPU hotspot to a SQL, verify same-instance CPU share and incident alignment
4. **Recommendation direction** — use the AutoX evidence-backed direction only after a SQL specialist route passes; do not force an optimizer action for `non_optimizer_bottleneck` or incomplete plan/schema evidence
5. **Same-node time comparison check** — did you compare affected TiDB instance(s) against their own normal window before using cross-node evidence?
6. **Cross-day same-clock-time check** — for long or periodic incidents, did you compare against at least one previous normal day?
7. **DML interpretation check** — if a DML statement was blamed, did you verify `affected_rows` and/or scan work, instead of using statement count alone?
8. **TiDB high CPU GC check** — if TiDB CPU was high, did you inspect `TIDB_GC_CHECK` and report abnormal `go_gc_duration_seconds` / `go_gc_pauses_seconds` findings? If GC is abnormal, did you recommend continuous profiling?
9. **TiDB high CPU workflow check** — if TiDB CPU was high, did you complete the incident-vs-baseline CPU profile comparison, collect TopSQL + statement summary + slow query evidence, and reduce the candidate SQL/internal-operation set before choosing the next action?
10. **Statistical and multi-source validation check** — if TiDB CPU was high, did you validate the suspected cause statistically and cross-check at least two independent sources? Require flame-graph agreement only when the tier supports downloaded profiles; for BYOC, use CPU/GC, TopSQL, Statements Summary, Slow Query, and plan evidence.

If the answer to item 5 is "no", the TiDB attribution is incomplete; add same-node abnormal-vs-normal comparison first.
If the answer to item 6 is "no" for a long or periodic incident, the baseline is incomplete; add cross-day comparison first.
If the answer to item 7 is "no" for a blamed DML workload, the SQL interpretation is incomplete; verify `affected_rows` / scan work first.
If the answer to item 8 is "no" for a TiDB high CPU case, the CPU attribution is incomplete; inspect GC metrics before concluding.
For items 9-10, identify which missing comparison is necessary for the proposed
mechanism. Do not require unrelated profile/query checks once the specialist's
evidence contract is satisfied; otherwise report the gap and bound confidence.
11. **Query-level mechanism analysis** (only if a SQL specialist route passed):
   - Did you complete the bounded AutoX classification from the available query evidence?
   - If a decoded or statement plan is available, name the load-bearing operator and runtime values.
   - If plan/schema/statistics evidence is unavailable, record the validation level and evidence gap; do not invent a specific index, hint, or Binding.
12. **TopSQL cross-validation for SQL CPU attribution:**
   - Verify the affected instance, CPU share, and incident timeline in underlying evidence.
   - Missing from TopSQL top-N means coverage is insufficient, not that the operation is absent.
   - Do not use latency alone to prove CPU dominance, or CPU share alone to reject non-CPU mechanisms.
13. **Recommendation precision** — if a SQL specialist route passed:
   - Did you load `tidb-query-tuning-autox` skill?
   - Does the recommendation match the classified direction: Binding, Index, TiFlash/MPP, non-optimizer investigation, or no optimizer action?
   - Are concrete optimizer changes supported by plan plus schema/index/statistics evidence?
   If not, **STOP and reduce the recommendation to the supported validation level**.
14. **ANALYZE recommendation precision** — recommend ANALYZE only when runtime `estRows`/`actRows` divergence and statistics evidence support it. State the divergence, affected table/columns, and validation limit explicitly.