# Postmortem: TCOC-5019

> Workspace: `ws-1776244241408-1ad2616f8`
> Source bundle: `~/Downloads/1776245477.zip`
> Date: 2026-04-15
> Reviewer: huyibin

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | TCOC-5019 |
| Alert Type | P999 SQL Duration spike |
| Cluster Name | prod-c66-seamless-others |
| Cluster ID | 10748116073341772391 |
| Parent ID | 2026948324023603200 |
| Cluster Type | premium |
| TiDB Version | v8.5.4 |
| Provider / Region | alicloud / ap-southeast-1 |
| Time Window | 2026-03-31 18:00 - 20:11 Beijing Time |

**Issue Description**: A Premium cluster had second-level SQL long-tail latency and TiDB CPU/GC pressure. Heap profiling showed several TiDB instances retaining multi-GB heap in the Region Bucket range-splitting path.

## Actual Root Cause

The cluster hit TiDB known bug `pingcap/tidb#66096` in `splitKeyRangesByBuckets`. Under heavy `IndexLookUp`/table-reader workloads from `t_seamless_log_tiptop` queries and deletes, TiDB split large key-range sets by TiKV buckets. The buggy path could repeatedly slice key ranges and retain huge amounts of heap when the range start key was smaller than the region start key.

The captured heap profiles showed `RegionCache.SplitKeyRangesByBuckets -> LocationKeyRanges.splitKeyRangesByBuckets -> KeyRanges.Slice` dominating memory, with the largest TiDB heap around `8.42GB` and about `4GB` attributed to that path. This caused GC pressure and contributed to the P999 SQL Duration spike. The cluster was on `v8.5.4`, which does not include the fix.

## Agent Diagnosis Summary

The agent first performed a normal TCOC evidence collection using `tidb-clinic-diagnosis`. It correctly handled Premium routing, converted the UTC+8 incident window, collected metrics/slow queries/logs, and found:

- `tidb-krx4fy` had P99 SQL latency around `3.51s` and CPU around `87%`
- multiple TiDB nodes exceeded 80% CPU
- Go GC pause P99 reached about `117ms`
- top slow query was a `DELETE FROM t_seamless_log_tiptop ...`, with `4978` executions and max time `19.39s`
- decoded plan had severe est/act divergence and large scan/write/lock work

After the user asked for heap profile analysis, the agent downloaded heap profiles for all seven TiDB instances, compared inuse_space, identified three multi-GB heap outliers, and traced the dominant memory chain to `splitKeyRangesByBuckets` and `KeyRanges.Slice`.

After the user asked whether this was a known issue, the agent searched GitHub and found `pingcap/tidb#66096`, the fix PR `#66070`, cherry-pick `#66135`, and concluded that `v8.5.4` lacked the fix.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 5 | Correctly reached known bug #66096 and tied it to heap profile evidence |
| Evidence Completeness | 4 | Good metrics, slow query, decoded plan, logs, heap profile, and GitHub correlation; pod-name/IP mapping was indirect |
| Causal Chain | 5 | Clear chain from IndexLookUp/range splitting to heap bloat, GC pressure, and P999 latency |
| Actionability | 5 | Upgrade path, bucket mitigation, workload batching, stats follow-up, and monitoring |
| Efficiency | 4 | 40 recorded tool calls; some IP mapping exploration was necessary for Premium profiling |
| **Overall** | **4.6** | |

## Human Guidance Log

### Intervention 1: Collect Basic Diagnosis

- **Round**: #1
- **User said**: "故障时间： 2026-03-31 ~18:00-20:11 (UTC+8) 现象 P999 SQL Duration 飙升，帮我采集下基本诊断信息"
- **Agent's mistake**: None; this was the initial task.
- **Correct behavior**: Use Premium metric routing, collect metrics/slow queries/logs/events, and summarize initial signals.
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: FLOW

### Intervention 2: Inspect Heap Profile After GC Signal

- **Round**: #2
- **User said**: "好的，看到 gc 压力大，帮忙看下 heap profile ，看看有没有发现"
- **Agent's mistake**: The first report identified GC pressure but did not automatically fetch heap profiles.
- **Correct behavior**: When high TiDB CPU and GC pause are abnormal, automatically inspect heap profiles for affected TiDB instances.
- **Skill**: `clinic-api`
- **Deficiency type**: FLOW

### Intervention 3: Search Known Issues For Abnormal Heap Stack

- **Round**: #3
- **User said**: "splitKeyRangesByBuckets + KeyRanges.Slice 合计占 8.42GB 中的 ~4GB，感觉不太正常，能搜下相应的 github issue 看有没有发现吗"
- **Agent's mistake**: The heap report identified the abnormal stack but did not immediately search upstream known issues.
- **Correct behavior**: For suspicious TiDB internal stacks, search GitHub issues/PRs and correlate fix versions before finalizing recommendations.
- **Skill**: `github-issue-search`
- **Deficiency type**: CROSS-REF

## Skill Deficiencies

### D1: High TiDB CPU Plus GC Requires Heap Profile

- **Type**: FLOW
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Basic diagnosis can stop at GC pause metrics and slow query evidence.
- **Expected behavior**: If high TiDB CPU and GC pause are both abnormal, collect or request heap profiles and identify allocation sources.
- **Suggested fix**: Add a mandatory heap-profile branch when `go_gc_*` metrics are abnormal during TiDB CPU incidents.
- **Priority**: P1-high
- **Status**: open

### D2: Premium Continuous Profiling Routing Is Easy To Get Wrong

- **Type**: KNOWLEDGE
- **Skill**: `clinic-api`
- **Current behavior**: The agent had to explore metadata fields and manually use parent/resource pool id for NGM profiling calls.
- **Expected behavior**: For Premium clusters, continuous profiling should use `parent_id` as `x-cluster-id`, `x-deploy-type: premium`, provider/region headers, and usually no project id.
- **Suggested fix**: Add a Premium ConProf header recipe to the clinic-api skill.
- **Priority**: P1-high
- **Status**: open

### D3: Heap Stack Should Be Cross-Referenced With Known Bugs

- **Type**: CROSS-REF
- **Skill**: `github-issue-search`
- **Current behavior**: GitHub issue search happened only after user prompt.
- **Expected behavior**: If a TiDB internal function dominates GB-level heap, search upstream issues and fix PRs automatically.
- **Suggested fix**: Add a rule to search `pingcap/tidb` for dominant TiDB function names from CPU/heap profiles.
- **Priority**: P2-medium
- **Status**: open

### D4: Distinguish Workload Trigger From Product Bug

- **Type**: REASONING
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Initial evidence could overemphasize stale stats and slow DELETE as the root cause.
- **Expected behavior**: Treat stale stats/large scans as workload triggers, then verify whether the profile indicates a TiDB product bug or abnormal runtime path.
- **Suggested fix**: In final reports, separate "trigger workload", "product/runtime failure mode", and "customer-visible symptom".
- **Priority**: P2-medium
- **Status**: open

## Key Queries

### Evidence Collection

```bash
python3 .opencode/skills/builtin/tidb-clinic-diagnosis/scripts/collect_evidence.py \
  10748116073341772391 1774951200 1774959060 \
  --output-dir clinic-downloads/2026-03-31/diagnosis-001
```

### Premium Continuous Profiling Group List

```bash
curl -s -G "${CLINIC_ENDPOINT}/ngm/api/v1/continuous_profiling/group_profiles" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -H "x-cluster-id: 2026948324023603200" \
  -H "x-org-id: 1372813089209289635" \
  -H "x-deploy-type: premium" \
  -H "x-provider: alicloud" \
  -H "x-region: ap-southeast-1" \
  --data-urlencode "begin_time=1774951200" \
  --data-urlencode "end_time=1774959060"
```

### Heap Profile Detail At Incident Snapshot

```bash
curl -s -G "${CLINIC_ENDPOINT}/ngm/api/v1/continuous_profiling/group_profile/detail" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -H "x-cluster-id: 2026948324023603200" \
  -H "x-org-id: 1372813089209289635" \
  -H "x-deploy-type: premium" \
  -H "x-provider: alicloud" \
  -H "x-region: ap-southeast-1" \
  --data-urlencode "ts=1774955400"
```

### Heap Analysis

```bash
go tool pprof -top -inuse_space raw/heap/pb/heap-172.22.132.165_10080.pb
go tool pprof -cum -inuse_space raw/heap/pb/heap-172.22.132.165_10080.pb
```

## Lessons Learned

- In TiDB CPU incidents, abnormal GC metrics are not an endpoint; they are a pointer to heap profiles and allocation callers.
- For Premium clusters, profiling uses resource-pool routing, so the agent must not assume the same headers as Dedicated clusters.
- A slow query with stale stats can be the trigger, while the actual root cause is a TiDB internal bug exposed by that workload.