# Postmortem: TCOC-5382

> Workspace: `ws-1781054808089-764584b34`
> Date: 2026-06-10
> Reviewer: claude-code

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | TCOC-5382 |
| Alert Type | Connection timeout (user-reported) |
| Cluster ID | 10123293495242577103 |
| Cluster Type | dedicated |
| TiDB Version | v8.5.3 |
| Time Window | 2026-06-07 12:50–14:30 JST (03:50–05:30 UTC) |
| Customer | Mercari |
| Environment | prod |

**Issue Description**: Three connection timeout windows on Mercari's `main` cluster — 13:01, 13:26, and 14:21 JST. ProxySQL observed ECONNRESET(104) and ECONNREFUSED(111) errors, Shunning cascade, and "Too many connections" for downstream clients.

## Actual Root Cause

> Based on workspace diagnosis + Jira comments (TCID-12774, TCID-12722 noted in ticket)

Three independent fault windows with distinct root causes:

**Fault 1 (13:01~02 JST — no restart):** `ModifyClusterSize` scale-out event expanded DefaultGroup TiDB nodes 39→41, started 12:53 JST, completed 13:08 JST. Fault window fell at minute 8 of pod initialization — new pods not yet ready caused ProxySQL ECONNRESET(104) Shunning cascade.

**Fault 2 & 3 (13:26 + 14:21 JST — tidb-2946, tidb-2954 restart):** RU saturation (all 10 resource groups `available_ru = 0`, 1.74M RU/s peak) + TiKV db-tikv-15 hotspot (7.8x gRPC P99) + TiDB CPU 72-77% caused mass slow queries to exceed `max_execution_time`. Simultaneous SQLKiller kill waves across 10+ nodes triggered recoverable panics on two nodes (db-tidb-2946 at 13:26, db-tidb-2954 at 14:21), causing ECONNREFUSED(111). TopSQL unavailable (Delta Lake GCS error); diagnosis based on panic logs + SlowQuery + RU metrics.

## Agent Diagnosis Summary

The agent ran `tidb-perf-diagnosis` autonomously (no human interventions after initial workspace creation). It:

1. Fetched TCOC-5382 Jira issue + 2 comments for context
2. Ran evidence collection (~658s): collected metrics, component health, resource control, slow queries, logs, events across 15 artifacts
3. Ran diagnosis analysis: identified 3 fault windows, confirmed panic logs for windows 2/3, linked scale-out event to window 1, identified RU saturation + TiKV hotspot + CPU pressure as compounding root causes

**Final verdict**: 置信度 HIGH. Correct identification of all three root causes with evidence chains.

**Gaps identified by --check:**
- `slowquery`: Not explicitly called out in --check mapping (slow queries were actually collected via collect_evidence.py)
- `decoded_plan`: Top query decoded plan was captured and analyzed in evidence collection
- `complete_sql`: Recommendations had incomplete SQL statements (e.g., `SPLIT TABLE coupon_attached_items ...`)
- No logs/events marked in data_coverage (likely because the new tidb-perf-diagnosis skill emits different tool call labels than the checker expects)

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 5 | Correctly identified all 3 fault windows with distinct root causes; no hallucinated data |
| Evidence Completeness | 4 | Metrics/logs/events/SlowQuery/component-health collected; TopSQL unavailable (GCS error, documented); TiDB latency metrics hit vmstorage sample limit (documented) |
| Causal Chain | 5 | RU saturation → queries hit max_execution_time → SQLKiller kill wave → panic on 2 nodes → ECONNREFUSED chain is clean and multi-source validated |
| Actionability | 4 | P0/P1/P2 prioritized action table; SPLIT TABLE and index recommendations specific; continuous profiling suggestion warranted. SQL statements incomplete (missing exact range syntax) |
| Efficiency | 5 | 0 human interventions; 8 tool calls total (including 2 todo-list calls); autonomous end-to-end diagnosis |
| **Overall** | **4.6** | |

## Human Guidance Log

No human interventions. Fully autonomous diagnosis.

## Skill Deficiencies

No new skill deficiencies identified. Known gaps:

- **ISS-0057**: TopSQL unavailable on GCP Japan clusters (Delta Lake path not found) — confirmed again
- **ISS-0089**: vmstorage sample limit on large clusters (34+ TiDB nodes) — TiDB latency metrics exceeded limit
- **ISS-0102**: Agent responded in Chinese despite likely Japanese-preferred context — TCOC-5382 ticket is in Japanese, diagnosis output is in Chinese; this is a new occurrence of the language mismatch issue. User satisfaction impact: medium (internal TCOC user).

## Data Source Coverage

- [x] Metrics (CPU, gRPC P99, GC pause, RU available/consumed)
- [x] TopSQL TiDB — unavailable (Delta Lake GCS error, documented in evidence)
- [x] TopSQL TiKV — unavailable (same GCS error)
- [x] SlowQuery logs (10 digests captured, top decoded)
- [x] Loki error logs (PANIC + SQLKiller entries confirmed)
- [x] Cluster events / Activity Hub (scale-out ModifyClusterSize confirmed)
- [x] Component health (restart counts, runtime)
- [x] Resource control (RU available/consumed per resource group)

## Key Queries

### Check SQLKiller mass-kill event
```bash
# LogQL — look for cluster-wide simultaneous kill wave
{cluster_id="10123293495242577103", component="tidb"} |~ "sqlkiller|executor:1317|kill initiated"
```

### Check RU saturation
```bash
# Prometheus — resource group available RU
min by (name) (tidb_resource_group_query_total{cluster_id="10123293495242577103"})
```

### Check Activity Hub for scale-out
```bash
curl "$CLINIC_API/clusters/10123293495242577103/events?start=1780800000&end=1780810000" \
  -H "Authorization: Bearer $TOKEN"
```

## Lessons Learned

1. **RU saturation is a causal amplifier, not just a symptom**: When resource groups exhaust token buckets, queries queue for tokens before executing, directly extending execution time and pushing borderline queries over `max_execution_time`. This is the mechanism linking workload surge to mass kill waves.
2. **Scale-out during peak load creates a ECONNRESET window**: New TiDB pods joining a pool during pod initialization (before they're ready) will cause ECONNRESET for any ProxySQL connections attempted to those pods. This is a known risk of scale-out during active traffic.
3. **Recoverable panics from SQLKiller are a product behavior, not a bug**: `[executor:1317]` panics in recoverable goroutines are expected when `max_execution_time` fires during certain goroutine states. The fix is to prevent queries from hitting the timeout, not to suppress panics.
4. **Autonomous multi-agent diagnosis is viable for complex multi-window cases**: The tidb-perf-diagnosis skill handled a 3-window complex case with no human interventions, correctly attributing each window to a different root cause.