# Expected Diagnosis: TCOC-5140

## Root Cause

Latency spikes were caused by a human-initiated `ScaleResourcePool` operation (TiKV CPU 16→32 cores upgrade) at 10:47:23 UTC+8 by `debin.jia@pingcap.com`. K8s rolling restart of TiKV pods caused Raft leader migration storms, with 3 latency spikes corresponding to 4 nodes going offline sequentially. Two of the later nodes had prestop hook failures (leader count didn't decrease), worsening the impact. This was a planned operation, not a failure.

## Evidence Chain

1. **Activity Hub events**: `ScaleResourcePool` event at 10:47:23 UTC+8 by debin.jia — TiKV CPU upgrade
2. **TiKV logs**: All 4 nodes received `signal 15` (SIGTERM) — graceful termination, not OOM kill
3. **Metrics**: TiDB P99 latency stable at 13-15ms before 10:47, spiked to 88s at 10:52 after first 2 nodes went offline
4. **Memory**: 66-68GB on 128GiB nodes is 51% utilization — normal operating range, not OOM
5. **Prestop hook failures**: tikv-3ijqoq and tikv-3xh796 had "leader count does not desc in 10 times" errors

## Causality

Human ScaleResourcePool operation → K8s rolling restart with SIGTERM → TiKV pods terminate sequentially → Raft leader migration (delayed by prestop hook failures on 2 nodes) → pessimistic lock wait / latch wait spikes → TiDB P99 query latency spikes (88s / 36s / 64s)

## Severity

Medium — planned operation impact, not a system failure. Impact duration ~10 minutes.

## Recommendations

1. **Increase prestop hook timeout or improve leader transfer strategy** — 2 of 4 nodes had prestop hook failures, causing uncontrolled leader migration
2. **Ensure adequate node spacing in rolling restarts** — first 2 nodes went offline within 2min20s, doubling the leader migration impact
3. **Review TiKV memory configuration** — 51% memory utilization is normal but leaves limited headroom; not the cause of this incident

## Scoring Rubric

> Checklist for regression testing. The Agent must satisfy these to pass.

- [ ] Query Activity Hub events and find the ScaleResourcePool operation
- [ ] Check TiKV termination logs and identify SIGTERM (signal 15), not SIGKILL
- [ ] NOT conclude OOM as root cause — memory drop is Pod replacement, not OOM kill
- [ ] Correctly attribute the restarts to the human scaling operation
- [ ] Identify the 3 latency spikes and correlate each with specific TiKV node(s) going offline
- [ ] Note prestop hook failures on tikv-3ijqoq and tikv-3xh796
- [ ] Verify pre-restart latency was normal (13-15ms P99)