# Expected Diagnosis: TCOC-5382

## Root Cause

Three independent fault windows on 2026-06-07 JST on Mercari's `main` cluster (Dedicated v8.5.3, cluster 10123293495242577103):

- **Window 1 (13:01~02 JST)**: ModifyClusterSize scale-out (TiDB 39→41 nodes) started 12:53 JST; new pods not yet ready caused ECONNRESET(104) cascade on ProxySQL. No TiDB restart in this window.
- **Windows 2 & 3 (13:26 + 14:21 JST)**: RU saturation (all resource groups `available_ru = 0`) + TiKV db-tikv-15 hotspot (7.8x gRPC P99) + TiDB CPU 72-77% drove mass slow queries over `max_execution_time`; SQLKiller cluster-wide kill wave triggered recoverable panics on db-tidb-2946 (13:26) and db-tidb-2954 (14:21), causing ECONNREFUSED(111).

## Evidence Chain

1. Activity Hub `ModifyClusterSize` event: TiDB DefaultGroup 39→41, start epoch 1780804429 (~12:53 JST), complete 1780805298 (~13:08 JST)
2. Loki logs: `db-tidb-2946` and `db-tidb-2954` show `panic: [executor:1317]Query execution was interrupted` immediately before their restart timestamps
3. Loki logs: 10+ TiDB nodes simultaneously showing `sqlkiller 'kill initiated'` WARN + `command dispatched failed [executor:1317]` ERROR
4. Resource control metrics: all 10 resource groups `available_ru` min = mean = 0 during incident; `rg_item` peak = 1.74M RU/s
5. Clinic metrics: db-tikv-15 gRPC P99 = 14ms vs fleet average 2ms (7.8x ratio); decoded plan `wait_time=0.606s >> process_time=0.023s` confirms TiKV waiting is dominant
6. Component health: db-tidb-2946 restart_count > 0 matches fault window 2; db-tidb-2954 matches fault window 3

## Causality

Window 1: Scale-out 39→41 → new pods initializing → not ready → ECONNRESET(104) on ProxySQL → Shunning cascade

Windows 2+3: RU saturation (token bucket exhausted) + TiKV db-tikv-15 hotspot → query execution time extends → hits `max_execution_time` → SQLKiller cluster-wide kill wave → recoverable panic on 2 nodes → ECONNREFUSED(111)

## Severity

High — three connection timeout windows totaling ~15 minutes of customer impact; each window isolated to 1 ProxySQL slave

## Recommendations

1. Raise resource group RU quotas (rg_item at 1.74M RU/s peak → increase capacity or scale cluster)
2. Review `max_execution_time` configuration — consider per-resource-group tiered timeouts to prevent synchronized kill waves
3. Configure ProxySQL for graceful ECONNRESET retry/backoff during scale-out operations
4. Split `coupon_attached_items` table by `item_id` range to eliminate db-tikv-15 hotspot
5. Optimize `ratings WHERE user_id` (max 10.3s) and `transaction_evidences WHERE item_id IN (...)` (max 9.9s) queries with appropriate indexes
6. Enable continuous profiling on high-CPU TiDB nodes to characterize heap allocation patterns

## Scoring Rubric

> Checklist for regression testing. The Agent must satisfy ALL of these to pass.

- [ ] Identify the ModifyClusterSize scale-out event (39→41 TiDB nodes) as the cause of fault window 1 (13:01 JST)
- [ ] Identify that fault window 1 had no TiDB restart (ECONNRESET only, not ECONNREFUSED)
- [ ] Identify PANIC logs for db-tidb-2946 and db-tidb-2954 as the cause of fault windows 2 and 3
- [ ] Classify the panic as `[executor:1317]` (query execution interrupted / max_execution_time) — a recoverable panic, not a bug
- [ ] Identify RU saturation (available_ru = 0 for all resource groups) as a compounding causal factor
- [ ] Identify TiKV db-tikv-15 as a hotspot (gRPC P99 significantly higher than fleet average)
- [ ] Note that TopSQL was unavailable (Delta Lake GCS error) without treating it as a diagnostic blocker
- [ ] Provide P0 recommendation to address RU saturation (increase quotas or scale cluster)