# Postmortem: TCID-12692

> Workspace: `ws-1778810697094-1dff52c74` (local replay)
> Original prod workspace: `ws-1778738824385-fdab142dc`
> Date: 2026-05-15
> Reviewer: claude-code

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | TCID-12692 |
| Alert Type | Customer-reported incident: query latency spike + kill events |
| Cluster ID | 10123293495242577103 |
| Cluster Type | dedicated (GCP, asia-northeast1) |
| TiDB Version | v8.5.3 |
| Time Window | 2026-05-12 20:00–21:00 JST (11:00–12:00 UTC) |

**Issue Description**: Customer (Japan) reported 15–30x query slowdown on specific TiDB nodes at 20:23 JST, kill events across almost all TiDB nodes with limited CPU reduction, and disruption coinciding with a ProxySQL traffic shift from TiDB to MySQL.

## Actual Root Cause

> From user feedback questions in GitHub issues #528 and #529 (TCID-12692 part.2).

Customer experienced a cluster-wide CPU saturation event on 2026-05-12 20:00–21:00 JST. All 43+ TiDB nodes in the `db` nodegroup simultaneously reached 80-88% CPU utilization driven by a peak SELECT workload surge (~515K stmt/s). All resource groups exhausted their RU budgets (available_ru=0). TiDB autoscaled from 39→44 nodes with 10 pod restarts. The "kill events" correspond to ProxySQL closing idle TiDB connections when shifting traffic to MySQL. Certain node queries (963, 964, 967, etc.) appeared 15-30x slower because they hit the capacity ceiling, not because of node-specific issues.

## Agent Diagnosis Summary

The agent correctly diagnosed the root cause as cluster-wide TiDB CPU saturation driven by peak SELECT load. Key findings:

- **Correctly identified**: 25 of 43 `db`-nodegroup TiDB nodes simultaneously at 80-88% of their 64-core quota
- **Correctly identified**: All 10 resource groups with `available_ru=0` (full RU exhaustion)
- **Correctly identified**: TiDB scale-out 39→44 with 10 pod restarts as platform response
- **Correctly identified**: `db-tikv-26` read hotspot (unified_read = 32.1 cores, gRPC P99 = 3.6x peers)
- **Correctly identified**: ProxySQL traffic shift → connection kills → Events OPM spike
- **Correctly addressed user Q1**: No TiDB CPU imbalance — all `db` nodes uniformly saturated; `ac-*` nodes at 3% by design (different routing)
- **Correctly addressed user Q2**: Events OPM spike = scale-out events + pod restarts + connection kills
- **TopSQL unavailable**: GCS Delta Lake path error (ISS-0057 known limitation)

The agent demonstrated good reasoning quality despite TopSQL unavailability, cross-referencing SlowQuery and metrics to support the workload-surge hypothesis.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 5 | Correct: cluster-wide CPU saturation + RU exhaustion from SELECT surge |
| Evidence Completeness | 4 | All major data sources queried; TopSQL blocked by Delta Lake path error (known ISS-0057); no decoded_plan analysis |
| Causal Chain | 5 | Clear SELECT surge → CPU saturation → RU exhaustion → autoscale + pod restarts → kill events |
| Actionability | 4 | Good recommendations for ProxySQL gradual drain, RU burst config, TiKV hotspot investigation; missing SQL optimization detail (TopSQL unavailable) |
| Efficiency | 5 | Single-pass diagnosis via collect_evidence.py; 0 wasted tool calls; no human interventions needed |
| **Overall** | **4.6** | Strong diagnosis despite TopSQL data unavailability |

## Human Guidance Log

No human interventions were required. The agent completed the diagnosis in a single pass.

## Skill Deficiencies

### ISS-0057: TopSQL unavailable for GCS clusters (Delta Lake path error)

- **Type**: TOOLING
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: collect_evidence.py A4 returns HTTP 500 for GCS clusters (Delta Lake path not found)
- **Expected behavior**: Agent should clearly report "TopSQL unavailable (data pipeline issue)" and proceed with higher confidence on SlowQuery evidence
- **Suggested fix**: See ISS-0057/plan.md
- **Priority**: P2-medium
- **Status**: open

## Key Queries

### Activity Hub events (correct: verified scale-out + kill events)

```bash
# From collect_evidence.py A5 — cluster events query
# Returns TiDB node count change 39→44, 10 TiDB restarts, kill events
```

### SlowQuery top digests

```bash
# From collect_evidence.py A3 — slow_query_logs
# coupon_attached_items JOIN: 1.87M execs, avg 0.47s, max 3.1s
# items table query: 1.14M execs, avg 0.41s, max 943s
```

## Lessons Learned

- **RU exhaustion is a strong saturation signal**: When all resource groups show `available_ru=0`, this independently confirms workload has hit capacity ceiling, even without TopSQL attribution
- **`ac-*` nodegroup routing**: In Japan Premium clusters, `ac-*` nodes are a separate routing nodegroup (likely for auditlog or other dedicated traffic) and should not be counted in CPU saturation analysis of main workload
- **ProxySQL traffic shift → kill events**: The `-kill` pattern in Events OPM is expected behavior when ProxySQL closes idle TiDB connections during traffic shift; this is not an anomaly
- **TopSQL Delta Lake unavailability (ISS-0057)**: GCS Japan clusters commonly have Delta Lake path errors; agents should treat SlowQuery as primary evidence when TopSQL is unavailable and communicate the limitation clearly