# Postmortem: TCOC-5387

> Workspace: `ws-1782112035290-90355e8ee`
> Date: 2026-06-24
> Reviewer: evolver-bot

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | TCOC-5387 |
| Alert Type | TiDB CPU imbalance + TiProxy memory leak |
| Cluster ID | 10947714666900611238 (wiz2-prd-tidb11, Cygames) |
| Cluster Type | dedicated |
| TiDB Version | v8.5.4-cygames |
| Time Window | 2026-06-11T08:00:00+08:00 – 2026-06-11T12:00:00+08:00 |

**Issue Description**: 15 TiDB nodes show severe CPU imbalance (14 nodes at 67–75%, 1 node in Zone A near 0%), coexisting with TiProxy memory leak — both traced to TiProxy bug tiproxy/issues/1172.

## Actual Root Cause

> Human-confirmed root cause from Jira resolution by Ming Zhang (2026-06-11/18).

TiProxy bug [tiproxy/issues/1172](https://github.com/pingcap/tiproxy/issues/1172): When all 9 TiProxy pods were restarted simultaneously, pods were rescheduled to different K8s nodes. `tiproxy-6` was assigned the IP formerly held by `tiproxy-0` (10.251.5.152). During the DNS chaos at restart, some TiProxy nodes resolved `tiproxy-0` (the Zone A zonal owner) to 10.251.5.152 and cached that long connection — never re-resolving DNS. Since `tiproxy-6` is not a zone owner, it returns empty CPU metrics for Zone A TiDB instances. The affected TiProxy nodes interpreted empty metrics as "CPU fully saturated" and stopped routing connections to Zone A TiDB nodes, causing all traffic to concentrate on the remaining 14 TiDB nodes. The same bug also causes a slow memory leak on affected TiProxy nodes.

## Agent Diagnosis Summary

> What the Agent concluded (single unguided run via auto-chat, 0 human interventions).

The agent correctly identified the TiProxy bug (tiproxy/issues/1172) as the primary root cause without any human intervention. It reconstructed the exact mechanism: simultaneous TiProxy restart → pod IP shuffle → DNS confusion → stale long connections to wrong zonal owner → empty CPU metrics → Zone A TiDB nodes excluded from routing → traffic concentration on 14 remaining nodes. The agent correctly identified `tiproxy-6` receiving connections intended for `tiproxy-0`, the downstream TiKV hotspot (db-tikv-2 gRPC P99 = 282ms, 9x peer average) and disk IO saturation as secondary effects. Excluded GC pressure, SQL hotspot, and ANALYZE mis-attribution with explicit evidence. Recommendations were accurate: restart tiproxy-6, set `balance.policy="connection"`, upgrade TiProxy.

**Notable gap**: The agent noted it could not directly verify with `tiproxy_session_total`/`tiproxy_balance_events` metrics (not currently collected by collect_evidence.py), but correctly used Jira engineer analysis as substitute evidence, maintaining high confidence in the diagnosis.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 5 | Exact match with engineer's confirmed root cause (tiproxy/issues/1172, DNS confusion, zone owner stale connection) |
| Evidence Completeness | 4 | Excellent use of metrics (CPU imbalance, TopSQL, decoded plan), TiKV hotspot quantified; TiProxy session metrics not collected (gap noted) |
| Causal Chain | 5 | Complete A→B→C chain: simultaneous restart → DNS chaos → stale connection → empty metrics → routing exclusion → CPU concentration |
| Actionability | 5 | Specific P0/P1/P2 recommendations with exact configuration (balance.policy="connection"), component name (tiproxy-6), and GitHub issue reference |
| Efficiency | 5 | 0 human interventions; auto-chat single pass; evidence collector ran in 83.9s |
| **Overall** | **4.8** | |

## Human Guidance Log

No human interventions required. The agent reached the correct diagnosis in a single unguided pass.

## Skill Deficiencies

### ISS-0111: TiProxy session/balance metrics not collected by collect_evidence.py

- **Type**: TEMPLATE
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: collect_evidence.py does not collect `tiproxy_session_total` or `tiproxy_balance_events` metrics
- **Expected behavior**: For clusters with TiProxy, these metrics should be collected to directly confirm routing imbalance and zone owner connection state
- **Suggested fix**: Add TiProxy-specific metric collection to collect_evidence.py when TiProxy instances are detected in cluster topology
- **Priority**: P3-low
- **Status**: open

## Key Queries

### TiProxy session distribution (should be added to collect_evidence.py)

```bash
# tiproxy_session_total — confirm Zone A TiDB receives zero sessions
clinic_cli metrics --cluster-id 10947714666900611238 \
  --promql 'sum by (instance) (tiproxy_session_total{component="tiproxy"})' \
  --start 1781136000 --end 1781150400
```

### Zone-aware routing confirmation

```bash
# tiproxy_balance_events — confirm route exclusions
clinic_cli metrics --cluster-id 10947714666900611238 \
  --promql 'rate(tiproxy_balance_events_total[5m])' \
  --start 1781136000 --end 1781150400
```

## Lessons Learned

- TiProxy simultaneous pod restart (vs rolling restart) is a known footgun — the SKILL.md should note that when TiDB CPU is severely imbalanced with one zone near 0%, TiProxy zonal owner DNS confusion is a likely cause.
- Absence of a single data source (TiProxy metrics) does not block high-confidence diagnosis when Jira engineer analysis is available as corroborating evidence.
- The agent correctly applied the "exclude ANALYZE mis-attribution (ic-001)" and "stale stats vs missing index (ic-002)" pitfall catalog — knowledge base is working as intended for this case type.