# Expected Diagnosis: TCID-12692

## Root Cause

Cluster-wide TiDB CPU saturation during 2026-05-12 20:00–21:00 JST driven by a peak SELECT workload surge (~515K stmt/s). All 43+ `db`-nodegroup TiDB nodes simultaneously hit 80-88% of their 64-core CPU quota, and all resource groups exhausted their RU budgets (available_ru=0). TiDB autoscaled from 39→44 nodes with 10 pod restarts, and ProxySQL's traffic shift to MySQL generated connection kill events. A `db-tikv-26` read hotspot (unified_read = 32.1 cores) is a downstream consequence of the concentrated read pressure.

## Evidence Chain

1. **TiDB CPU metrics**: 25 of 43 `db`-nodegroup nodes simultaneously at 80-88% of 64-core quota; `ac-*` nodes at 3% (separate routing nodegroup)
2. **Resource group RU**: All 10 resource groups show `available_ru=0.0` (full throughput throttle); peak rg_item at 1.96M RU/s
3. **Component health / Activity Hub**: TiDB count 39→44 (3 scale-out transitions), 10 TiDB pod restarts; `-kill` events = ProxySQL closing idle connections
4. **SlowQuery**: `coupon_attached_items` JOIN (1.87M execs, avg 0.47s, max 3.1s) and `items` table queries (1.14M execs, max 943s) dominating the window
5. **TiKV hotspot**: `db-tikv-26` process CPU 44.15 cores (69%), `unified_read` thread = 32.1 cores — read concentration on one node
6. **TopSQL**: Unavailable (GCS Delta Lake path error) — treat as data pipeline limitation, not diagnostic failure

## Causality

```
High SELECT load surge (~515K stmt/s peak)
  → TiDB CPU saturation (25+ nodes @ 80-88% of 64-core quota)
  → All RU groups exhausted (available_ru = 0)
  → Query latency spike (avg 0.4-0.5s, max 943s)
  → TiDB autoscale (39→44) + pod restarts (10 instances)
  → Connection kills (ProxySQL traffic shift) / Events OPM spike
  → Service errors 20:34–20:57 JST
  → TiKV db-tikv-26 read hotspot (downstream effect of concentrated reads)
```

## Severity

High — cluster-wide capacity saturation caused service errors for 23+ minutes; proactive query optimization and capacity planning required.

## Recommendations

1. **ProxySQL gradual drain**: Implement gradual traffic drain (not abrupt kill-all) when shifting TiDB→MySQL to avoid active connection errors
2. **RU Burst configuration**: Review resource group RU limits; if all groups simultaneously hit 0, consider Burst RU settings to absorb load spikes
3. **TiDB capacity**: Evaluate permanent scale-out (current 44-node max) or larger instance size to handle peak load without saturation
4. **TiKV hotspot investigation**: Run `SHOW TABLE REGIONS` on tables accessed by `coupon_attached_items` and `items` queries; check for uneven region distribution causing `db-tikv-26` hotspot
5. **Query optimization (when TopSQL available)**: Review execution plans for `coupon_attached_items` JOIN (1.87M/hr, max 3.1s) and `items` table queries (max 943s) for index gaps

## Scoring Rubric

> Checklist for regression testing. The agent must satisfy all items to pass.

- [ ] Identifies cluster-wide TiDB CPU saturation (not single-node imbalance) as primary root cause
- [ ] Reports `available_ru=0` across all resource groups as independent saturation signal
- [ ] Explains "kill events" as ProxySQL connection termination during traffic shift (not anomalous kills)
- [ ] Identifies `db-tikv-26` as read hotspot (unified_read dominant thread)
- [ ] Correctly states NO TiDB CPU imbalance among `db`-nodegroup nodes (uniformly saturated)
- [ ] Notes TopSQL unavailability (Delta Lake path error) if applicable, without treating it as a diagnosis failure
- [ ] Reports TiDB scale-out from 39→44 with 10 pod restarts from Activity Hub
- [ ] Provides actionable recommendations (at least: ProxySQL gradual drain, RU burst config, TiKV hotspot investigation)