# Expected Diagnosis: GH-229

## Root Cause

TiDB CPU spike on 2026-04-08 was caused by TiDB node count reduction from 27 (13 original + 14 auto-scaler) to 18 (4 original + 14 auto-scaler), while total QPS remained stable (~80K vs ~73K). This increased per-node load by ~65% (2,714/s → 4,474/s). The application uses prepare:exec:close=1:1:1 pattern where every execution triggers full plan compilation (`generateNewPlan`), so higher per-node QPS directly increases compile CPU. Plan cache hit=0 is expected and identical on both days — it is NOT the root cause.

## Evidence Chain

1. **Cluster topology**: Apr 7 had 27 TiDB instances, Apr 8 had 18 (9 original nodes removed)
2. **Per-node load**: StmtExecute rate per node increased from ~2,714/s to ~4,474/s (+65%)
3. **Plan cache metrics**: Miss rate = 99.6% on BOTH days — no change, not the differentiator
4. **StmtPrepare:StmtExecute:StmtClose = 1:1:1**: Confirms prepare-once-execute-once pattern; plan cache cannot help
5. **CPU profile**: `generateNewPlan` at 37.85% is expected under this pattern; it's amplified by higher per-node QPS

## Causality

TiDB scale-in (27→18 nodes) → per-node StmtExecute load +65% → more generateNewPlan calls per node → more malloc/GC per node → CPU spike on remaining nodes

## Severity

Medium — capacity planning issue, not a system bug.

## Recommendations

1. **Scale TiDB back to previous node count** or ensure auto-scaler compensates for original node removal
2. **Do NOT attempt to "fix" plan cache** — hit=0 is expected for prepare:exec:close=1:1:1 pattern
3. **Consider connection pooling with prepare-once-execute-many** if the application can change its usage pattern (long-term optimization)

## Scoring Rubric

> Checklist for regression testing. The Agent must satisfy these to pass.

- [ ] Check StmtPrepare:StmtExecute:StmtClose ratio and identify 1:1:1 pattern
- [ ] Recognize plan cache hit=0 as expected for prepare-once-execute-once pattern
- [ ] NOT blame plan cache miss as root cause
- [ ] Compare plan cache miss rate between problem day and baseline — confirm it's the same on both days
- [ ] Identify cluster topology change (node count reduction) as the actual differentiator
- [ ] Calculate per-node load increase and correlate with CPU spike
- [ ] Recommend capacity adjustment, not plan cache tuning