# Expected Diagnosis: O11Y-2612493

## Root Cause

6 consecutive TiKV scale-out batches (36→54 nodes, ~every 5–8 minutes) between 08:08–09:30 UTC triggered repeated PD region balance storms. After the 3rd batch completed (08:39:16 UTC), accumulated region migration pressure drove error 1317 "Query execution was interrupted" across 36+ TiDB nodes, causing the error rate to exceed 4.5× the 5-minute baseline and trigger the alert at 08:41:00 UTC. TiDB CPU elevation and Resource Control RU exhaustion were amplifying factors, not root causes. Alert auto-recovered at 09:12:20 UTC.

## Evidence Chain

1. **Activity Hub events** — 6 `ModifyClusterSize` events (batches of +3 TiKV nodes each) between 08:08–09:30 UTC; 3rd batch completed at 08:39:16, alert fired at 08:41:00
2. **Loki logs** — `[executor:1317]` / `[tikv:1317]` "Query execution was interrupted" errors across 36+ TiDB instances (users ms_item_select_v2, ms_user_select_v2); correlated with `tidb_server_execute_error_total{type=unknown}` surge
3. **Metrics** — TiDB CPU saturation: db-tidb-962 89%, db-tidb-965 86%, db-tidb-970 85% (side effect of old nodes bearing full region load during rebalance); TiKV gRPC P99 hotspots on db-tikv-13/24/537 (2.1–3.0× peers)
4. **Resource control** — rg_item, rg_mercari, rg_user available_ru min=0 (RU exhaustion amplified failure probability during scheduling pressure)
5. **EBS disk** — disk_io_util max=0.00 (infrastructure clear; storage is not a factor)
6. **TopSQL** — unavailable (Delta Lake path missing, HTTP 500); not a diagnosis gap

## Causality

TiKV scale-out batches (6 rounds) → PD initiates region balance (leader + replica migration) → KV requests to migrating regions interrupted with error 1317 → TiDB executor layer returns `type=unknown` errors → error rate 4.5× baseline → alert triggered. TiDB CPU high + RU exhaustion are amplifying factors, not independent root causes.

## Severity

Major (alert auto-recovered in ~31 minutes; no data loss reported)

## Recommendations

1. **Scale-out procedure**: Wait for PD balance scheduler activity to approach zero (observe `pd_schedule_store_regions_event` or region peer count stabilize) before starting the next batch
2. **During scale-out**: Temporarily reduce `region-schedule-limit` to throttle migration rate and reduce request interruption probability
3. **Resource groups**: Configure `BURSTABLE` mode or raise RU upper limits for rg_item and rg_mercari to preserve headroom during scheduling pressure
4. **Post-incident**: Check db-tidb-2 / db-tidb-970 restart logs to confirm whether restarts were OOM or scale-related

## Scoring Rubric

> Checklist for regression testing. The Agent must satisfy these to pass.

- [ ] Identifies TiKV scale-out (36→54 nodes, 6 batches) as the root cause trigger
- [ ] References Activity Hub `ModifyClusterSize` events as primary evidence for scale-out timeline
- [ ] Identifies error 1317 "Query execution was interrupted" in logs as the mechanism linking scale-out to error rate increase
- [ ] Notes that TiDB CPU elevation is a side effect/amplifying factor, not the root cause
- [ ] Identifies Resource Control RU exhaustion (available_ru min=0) as an amplifying factor
- [ ] Confirms EBS disk_io_util ≈ 0 (infrastructure clear)
- [ ] Notes TopSQL unavailable due to Delta Lake path missing (data pipeline gap, not agent gap)
- [ ] Recommends waiting for PD region balance completion between scale-out batches