# Expected Diagnosis: TCOC-5387

## Root Cause

TiProxy bug [tiproxy/issues/1172](https://github.com/pingcap/tiproxy/issues/1172) caused by simultaneous restart of all 9 TiProxy pods. After restart, `tiproxy-6` was rescheduled to the IP formerly held by `tiproxy-0` (Zone A zonal owner). Due to DNS confusion at restart, other TiProxy nodes cached stale long connections to `tiproxy-6` thinking it was `tiproxy-0`. Since `tiproxy-6` is not a zone owner, it returns empty CPU metrics for Zone A TiDB instances, causing affected TiProxy nodes to treat Zone A TiDB as fully saturated and stop routing there — concentrating all traffic on the remaining 14 TiDB nodes.

## Evidence Chain

1. **TiDB CPU metrics**: 14 TiDB nodes in non-Zone-A groups at 67–75% utilization; Zone A node (`db-tidb-0-ac`) near 0% — signature of routing exclusion, not SQL hotspot
2. **TopSQL**: SQL digest CPU% distributed evenly across hot nodes (~1.2–1.4% per digest per node) — no single SQL dominates, confirming the cause is connection routing, not query pattern
3. **TiKV hotspot**: `db-tikv-2` gRPC P99 = 282ms (9× peer average); `db-tikv-0` unified read = 10.26 cores — downstream cascade from concentrated TiDB load
4. **Jira engineer analysis** (Ming Zhang, 2026-06-11/18): Confirmed TiProxy bug mechanism with exact pod IP mapping and DNS confusion timeline
5. **GC/heap evidence**: GC P99 pause ≤97ms, duration rate <0.002 s/s — GC pressure ruled out as independent cause

## Causality

Simultaneous TiProxy pod restart → pods rescheduled to different K8s nodes (tiproxy-6 gets tiproxy-0's IP 10.251.5.152) → DNS chaos during restart, some TiProxy nodes cache stale connection to 10.251.5.152 → tiproxy-6 (non-owner) returns empty Zone A metrics → affected TiProxy nodes treat Zone A TiDB CPU as "fully loaded" → zero connections routed to Zone A TiDB (`db-tidb-0-ac`) → 14 remaining TiDB nodes absorb all traffic at 67–75% CPU → TiKV coprocessor hotspot and disk IO saturation as secondary downstream effects.

## Severity

High — persistent CPU imbalance degrading 14 of 15 TiDB nodes; Zone A TiDB node receiving near-zero traffic while others are overloaded; slow TiProxy memory leak (rate unknown).

## Recommendations

1. **P0**: Restart `tiproxy-6` to force DNS re-resolution, restoring Zone A routing
2. **P0**: Set `balance.policy="connection"` as immediate mitigation to bypass CPU-metric-based routing
3. **P1**: Upgrade TiProxy to a version containing the tiproxy/issues/1172 fix
4. **P1**: Change future TiProxy maintenance to rolling restarts (not simultaneous) to avoid DNS state confusion
5. **P2**: Collect `process_resident_memory_bytes{component='tiproxy'}` over multiple days to quantify memory leak rate and assess OOM risk
6. **P2**: After fix, verify TiKV hotspot (db-tikv-2 gRPC P99, db-tikv-0 CPU) dissipates as routing recovers

## Scoring Rubric

> Checklist for regression testing. The Agent must satisfy these to pass.

- [ ] Identifies TiProxy bug (tiproxy/issues/1172 or equivalent description of DNS confusion/stale connection) as primary root cause
- [ ] Identifies Zone A TiDB node (`db-tidb-0-ac` or "Zone A" or "ac zone") as the excluded node with near-0% CPU
- [ ] Explains the mechanism: simultaneous TiProxy restart + pod IP reassignment + stale long connection + empty CPU metrics from non-owner
- [ ] Identifies TiKV hotspot (db-tikv-2 or gRPC P99 spike) as secondary/downstream effect of routing concentration
- [ ] Uses TopSQL even distribution as evidence that the cause is routing (not SQL hotspot)
- [ ] Recommends restarting tiproxy-6 (or the affected TiProxy node) to force DNS re-resolution
- [ ] Recommends `balance.policy="connection"` as mitigation
- [ ] Explicitly rules out GC pressure / SQL hotspot as independent root causes