# Expected Diagnosis: O11Y-2530675

## Root Cause

TiDB P99 latency spike caused by TiKV infrastructure saturation: EBS IOPS throttling on multiple TiKV volumes (up to 40% throttle on db-tikv-16/db-tikv-4/db-tikv-8) combined with a severe gRPC hotspot on db-tikv-9 (P99 104.86s vs peer average 6s, 17.5x ratio). Business traffic peaked at ~3x normal volume, pushing disk IO over capacity. Query wait_time (231s) >> process_time (10s) confirms queries were queuing on IO/resources, not inefficient execution. SQL statistics divergence and lock contention are secondary symptoms of the infrastructure pressure.

## Evidence Chain

1. **Metrics**: TiDB P99 latency peak 81–219s; TiKV gRPC P99 on db-tikv-9 = 104.86s vs peers ~6s (17.5x ratio); EBS IOPS throttle up to 40% on 5 TiKV volumes; business traffic 3x normal (Select/Update/Transaction peaks)
2. **TopSQL TiKV**: Single SQL (digest 470AA562) heavily distributed across 26 TiKV nodes, indicating table-wide read pressure on dh_active_receive_logs
3. **SlowQuery**: COMMIT avg_time 480s (18,089 executions), SELECT dh_active_receive_logs avg 119s (15,337 executions) — wait_time=231.7s >> process_time=10.1s confirms queuing not computation
4. **Events**: TiKV node count 26→25 (1 node unavailable), TiCDC restarted twice — secondary effects of disk IO pressure

## Causality

Business traffic peak (3x normal) → high-frequency SELECT on dh_active_receive_logs across 26 TiKV nodes → EBS IOPS saturation (5 volumes throttled up to 40%) → queries queue waiting for IO → wait_time 231s >> process_time 10s → db-tikv-9 accumulates disproportionate gRPC backlog (P99 104s, 17.5x hotspot) → write transactions (COMMIT/UPDATE) blocked by IO contention → TiDB P99 latency spikes to 81–219s

## Severity

High — event self-recovered but root cause (IOPS under-provisioning + read hotspot on db-tikv-9) persists; next traffic spike will reproduce.

## Recommendations

1. **P0**: Investigate and address EBS IOPS capacity on db-tikv-16, db-tikv-4, db-tikv-8 (consider provisioned IOPS upgrade or volume resize)
2. **P1**: Investigate db-tikv-9 read hotspot — check if dh_active_receive_logs data is unevenly distributed (hot region), consider split/scatter
3. **P2**: ANALYZE TABLE dh_active_receive_logs to fix statistics divergence (site_code Selection divergence 29.4x, reduces unnecessary plan instability)
4. **P3**: Consider TiFlash offload for high-frequency SELECT queries on dh_active_receive_logs during peak traffic

## Scoring Rubric

> Checklist for regression testing. The Agent must satisfy these to pass.

- [ ] Identifies db-tikv-9 as a gRPC hotspot with quantified ratio (≥10x vs peers)
- [ ] Identifies EBS IOPS throttling on multiple TiKV volumes with percentage values
- [ ] Notes that wait_time >> process_time (e.g., 231s vs 10s) indicating IO/resource queuing
- [ ] Identifies dh_active_receive_logs as the primary hot table from TopSQL TiKV
- [ ] Lists COMMIT slow queries (avg ~480s) as evidence of lock/IO contention
- [ ] Primary recommendation targets TiKV infrastructure (IOPS or hotspot), not SQL statistics
- [ ] Provides causal chain linking traffic peak → IOPS throttle → wait_time spike → latency