# Expected Diagnosis: TCOC-5019

## Root Cause

TiDB `v8.5.4` on Premium cluster `prod-c66-seamless-others` hit the known TiDB bug [pingcap/tidb#66096](https://github.com/pingcap/tidb/issues/66096), an infinite-loop/high-allocation issue in `splitKeyRangesByBuckets`. During the incident, heavy `IndexLookUp` and table-reader workloads from `t_seamless_log_tiptop` queries/deletes produced large handle/range sets. With TiKV Region Bucket splitting enabled, TiDB repeatedly called `RegionCache.SplitKeyRangesByBuckets`, `LocationKeyRanges.splitKeyRangesByBuckets`, and `KeyRanges.Slice`, retaining several GB of heap on multiple TiDB instances.

The heap bloat drove Go GC pressure and long-tail latency. The visible symptom was P999 SQL Duration spike and TiDB CPU saturation, with GC pause P99 around `117ms` on high-CPU TiDB instances.

## Evidence Chain

1. **Cluster and routing**: Premium cluster `prod-c66-seamless-others`, cluster_id `10748116073341772391`, parent/resource pool id `2026948324023603200`, TiDB `v8.5.4`, AliCloud `ap-southeast-1`. The diagnosis must use Premium routing: TiDB metrics use `cluster_id`, while TiKV/PD/TiFlash/node and continuous profiling use `parent_id`.
2. **Incident window**: Beijing time `2026-03-31 18:00-20:11`, corresponding to Unix `1774951200-1774959060`.
3. **Latency/CPU signal**: P999/P99 SQL duration spikes, with `tidb-krx4fy` reaching about `3.51s` P99 and about `27/31` CPU cores (`87%`). Several TiDB nodes were above 80% CPU.
4. **GC signal**: High-CPU TiDB instances show Go GC pause P99 around `117ms`, so the analysis must inspect heap profile rather than stopping at slow queries.
5. **Slow query signal**: Top slow query digest `908792e63b56548d90d46345e2077cecc168cbb3a2f42baf6ef5e0a159930a05` is `DELETE FROM t_seamless_log_tiptop WHERE id BETWEEN ? AND ? AND created_time ...`, with about `4,978` executions, average `1.26s`, and max `19.39s`.
6. **Plan/statistics signal**: The decoded plan for the top DELETE uses `TableRangeScan` on `t_seamless_log_tiptop`, estRows about `18,772,224` vs actRows `5,000` (about `3754x` divergence), `process_keys` about `8.8M`, `write_keys` about `30,000`, and `lock_keys` about `10,000`. This is an important trigger/workload signal, but not the final root cause by itself.
7. **Logs**: TiDB logs contain many `read packet timeout, close this connection`; TiKV/PD logs do not show a stronger primary failure such as OOM/panic as the root cause.
8. **Heap profile collection**: The diagnosis should download heap profiles for all seven TiDB instances at a finished continuous profiling snapshot around `ts=1774955400` (about `2026-03-31 18:50` Beijing), because pod-name to IP mapping is not directly available.
9. **Heap profile totals**: Heap inuse_space shows abnormal TiDB instances: `172.22.132.165` about `8.42GB`, `172.22.164.237` about `7.59GB`, and `172.22.132.163` about `4.71GB`; normal instances are around `0.9-1.0GB`.
10. **Heap profile stack**: The dominant memory chain is `IndexLookUpExecutor.startTableWorker -> tableWorker.pickAndExecTask/executeTask -> IndexLookUpExecutor.buildTableReader -> dataReaderBuilder.buildTableReaderFromHandles -> distsql.Select -> CopClient.Send -> BuildCopIterator -> buildCopTasks -> RegionCache.SplitKeyRangesByBuckets -> LocationKeyRanges.splitKeyRangesByBuckets -> KeyRanges.Slice`.
11. **Allocation attribution**: `splitKeyRangesByBuckets` plus `KeyRanges.Slice` account for roughly `4GB` of the `8.42GB` heap on the worst instance, which is not normal transient memory.
12. **Known bug correlation**: Searching GitHub for `splitKeyRangesByBuckets` must find `pingcap/tidb#66096` and the fix PR `#66070` / cherry-pick `#66135`. The bug says an abnormal range start key can cause an infinite loop in `splitKeyRangesByBuckets`, causing high GC pressure and possible OOM.
13. **Version impact**: The current cluster is `v8.5.4`, which predates the `2026-02-07` fix and does not contain it; v8.5.6 or an appropriate nextgen build after the fix contains the mitigation.

## Causality

Slow `t_seamless_log_tiptop` SELECT/DELETE workload with large scans and IndexLookUp/table-reader handle sets -> TiDB splits key ranges by TiKV buckets -> bug #66096 in `splitKeyRangesByBuckets` can loop or repeatedly slice ranges when range start key is smaller than region start key -> `KeyRanges.Slice` retains GBs of heap on several TiDB instances -> Go GC pause and CPU pressure rise -> P999 SQL Duration spikes and connection read timeouts appear.

## Severity

High. The incident affected a Premium production cluster, several TiDB nodes exceeded 80% CPU, P999/P99 SQL latency spiked to seconds, and heap on multiple TiDB instances reached several GB. No conclusive OOM/panic was observed in the captured logs.

## Recommendations

1. **Primary fix**: Upgrade TiDB to a build that includes the fix for `pingcap/tidb#66096`, preferably `v8.5.6+` or the relevant nextgen build after `2026-02-07`.
2. **Short-term mitigation**: Consider disabling TiKV Region Bucket splitting (`split.enable-region-bucket = false`) to avoid the problematic `splitKeyRangesByBuckets` path while validating impact.
3. **Workload mitigation**: Reduce the `t_seamless_log_tiptop` scan/IndexLookUp pressure by batching deletes/selects more narrowly and avoiding very large intermediate handle sets.
4. **Plan/statistics follow-up**: Refresh and validate statistics for `t_seamless_log_tiptop`, but do not treat stale stats as the only root cause.
5. **Monitoring**: Add alerts for TiDB heap inuse > `4GB`, GC pause P99 > `100ms`, and P999 SQL Duration spikes. Capture heap profiles when GC pressure appears.

## Scoring Rubric

> Checklist for regression testing. The Agent must satisfy these to pass.

- [ ] Establish Premium cluster routing correctly: TiDB metrics by `cluster_id`, non-TiDB/profiling by `parent_id`.
- [ ] Establish the incident window `2026-03-31 18:00-20:11 UTC+8` / `1774951200-1774959060`.
- [ ] Identify P999/P99 SQL duration spike, `tidb-krx4fy` as the worst latency instance, and multiple TiDB nodes above 80% CPU.
- [ ] Check Go GC metrics/pause because high TiDB CPU is present; explicitly report GC pause P99 around `117ms`.
- [ ] Use slow query and decoded plan evidence for `t_seamless_log_tiptop`, including digest `908792e63b56548d90d46345e2077cecc168cbb3a2f42baf6ef5e0a159930a05`, `estRows` vs `actRows` divergence, and large `process_keys`.
- [ ] Download/analyze heap profiles for all TiDB instances at an incident snapshot instead of only the highest-CPU pod.
- [ ] Report heap inuse_space outliers: `172.22.132.165` about `8.42GB`, `172.22.164.237` about `7.59GB`, and `172.22.132.163` about `4.71GB`.
- [ ] Identify the heap allocation chain through `IndexLookUpExecutor`, `buildTableReaderFromHandles`, `RegionCache.SplitKeyRangesByBuckets`, `LocationKeyRanges.splitKeyRangesByBuckets`, and `KeyRanges.Slice`.
- [ ] State that `splitKeyRangesByBuckets` + `KeyRanges.Slice` account for roughly `4GB` of the worst instance heap.
- [ ] Search/correlate GitHub issues and identify `pingcap/tidb#66096` plus fix PR `#66070` / cherry-pick `#66135`.
- [ ] Correctly state that v8.5.4 does not contain the fix and recommend v8.5.6+ or an appropriate nextgen build.
- [ ] Recommend short-term bucket mitigation and workload batching, while distinguishing stale stats from the final known-bug root cause.