# Postmortem: TCOC-5140

> Workspace: `ws-1776646913272-b0d2b9b99`
> Date: 2026-04-20
> Reviewer: claude-code
> Source: GitHub issue #218

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | TCOC-5140 |
| Alert Type | Latency spike (user-reported) |
| Cluster ID | 10748116073341772391 |
| Cluster Type | premium |
| TiDB Version | v8.5.4 |
| Time Window | 2026-04-16 10:51 - 10:59 UTC+8 |
| Environment | dev |
| User Feedback | "Incorrect OOM without verifying" |

**Issue Description**: Agent incorrectly diagnosed TiKV node restarts as OOM kills without verifying the termination signal. The actual cause was a human-initiated ScaleResourcePool operation (CPU upgrade) triggering K8s rolling restarts with SIGTERM.

## Actual Root Cause

> From workspace conversation — confirmed via Activity Hub events after human guidance.

`debin.jia@pingcap.com` triggered a `ScaleResourcePool` operation at 10:47:23 UTC+8, upgrading TiKV CPU from 16 to 32 cores. K8s performed rolling restarts of TiKV pods (SIGTERM, not OOM kill), causing 3 latency spikes as Raft leaders migrated:
- Spike 1 (88s): tikv-0tx048 + tikv-14rj46 simultaneously offline
- Spike 2 (36s): tikv-3ijqoq offline (prestop hook failed)
- Spike 3 (64s): tikv-3xh796 offline (prestop hook failed)

Memory drop from 66-68GB to 4-8GB was Pod replacement, not OOM. 128GiB nodes running at 63-68GB is normal operating range.

## Agent Diagnosis Summary

> First unguided diagnosis (round 0):

The agent collected evidence (metrics, TopSQL, SlowQuery, logs) and identified:
- 4 TiKV nodes restarted during the window
- Memory dropped from 66-68GB to 4-8GB after restart
- TiKV pessimistic lock P99 = 140s, latch wait P99 = 20s

**Concluded**: OOM Killed — "4个节点内存均处于 65-68 GB 高位，重启瞬间降至 4-8 GB，这是 OOM Killed 的典型特征"

**This was incorrect**. The agent:
1. Did not check termination signals (SIGTERM vs SIGKILL)
2. Did not check Activity Hub for human operations
3. Did not consider that 66GB on a 128GiB node is normal (51% utilization)
4. Assumed memory drop = OOM without verification

## Human Guidance Log

### Intervention 1: Investigate restart cause
- **Round**: #2
- **User said**: "调查4个重启节点的重启原因，以及重启前有什么异常"
- **Agent's mistake**: Only concluded OOM from memory patterns, didn't look at process termination signals
- **Correct behavior**: Check SIGTERM/SIGKILL logs, check K8s events, check OOM killer logs
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: FLOW

### Intervention 2: 5-hour memory baseline
- **Round**: #5
- **User said**: "如果怀疑是 OOM，那么在重启发生前5个小时内存使用情况如何"
- **Agent's mistake**: Concluded OOM without checking if memory was at a stable baseline
- **Correct behavior**: Should have proactively checked whether memory level was normal for the node spec
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: REASONING

### Intervention 3: Check termination type
- **Round**: #7
- **User said**: "能看到这几个 tikv 节点是被动终止进程，还是有收到信号有序退出的吗"
- **Agent's mistake**: Never checked SIGTERM vs SIGKILL — the most basic OOM verification
- **Correct behavior**: For any suspected OOM, first verify: was it SIGKILL (OOM) or SIGTERM (graceful)?
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: FLOW

### Intervention 4: Check for human operations
- **Round**: #9
- **User said**: "这个过程有可能是人工干预/操作导致的吗？"
- **Agent's mistake**: Never queried Activity Hub events for human operations
- **Correct behavior**: For any node restart, check cluster events/Activity Hub for recent operations (scaling, upgrades, maintenance)
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: FLOW

### Intervention 5-12: Deep dive guidance
- **Rounds**: #11-24
- User guided detailed analysis of pre-restart latency, specific node behavior (tikv-6diswl), and profiling bug investigation
- Agent performed well on these directed tasks

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 2 | Initial OOM conclusion was wrong; correct root cause (ScaleResourcePool) only after 4 interventions |
| Evidence Completeness | 3 | Metrics/TopSQL/logs collected, but missed Activity Hub events and termination signal verification |
| Causal Chain | 3 | Final chain is excellent after guidance, but initial chain was wrong (OOM → latency) |
| Actionability | 4 | Final recommendations are specific and actionable (prestop hook tuning, profiling bug report) |
| Efficiency | 2 | 12 human interventions; agent didn't self-verify OOM hypothesis |
| **Overall** | **2.8** | |

## Skill Deficiencies

### ISS-0038: Node restart diagnosis should verify termination signal before concluding OOM

- **Type**: FLOW
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent sees memory drop after restart and concludes OOM without checking SIGTERM/SIGKILL
- **Expected behavior**: Check process termination logs (signal 15 = SIGTERM = graceful, signal 9 = SIGKILL = OOM) before any OOM conclusion
- **Suggested fix**: Add mandatory verification step in SKILL.md: "If suspecting OOM, verify: (1) SIGKILL in logs? (2) OOM killer messages? (3) Memory relative to node spec?"
- **Priority**: P1-high
- **Status**: open

### ISS-0039: Agent should check Activity Hub events for node restarts

- **Type**: FLOW
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent analyzes metrics/logs but never checks Activity Hub for human operations (scaling, upgrades, maintenance)
- **Expected behavior**: For any node restart scenario, check cluster events first — human operations are the most common cause of planned restarts
- **Suggested fix**: Add to Node C routing: "If nodes restarted → query Activity Hub events for ScaleResourcePool/UpgradeCluster/MaintenanceWindow operations within ±30min"
- **Priority**: P1-high
- **Status**: open

## Data Source Coverage

- [x] Metrics (CPU, memory, gRPC latency, latch wait)
- [x] TopSQL TiDB
- [x] TopSQL TiKV
- [x] SlowQuery logs
- [x] Loki error logs
- [ ] Activity Hub events (only checked after human prompt in round 4)
- [x] Continuous Profiling (heap profile found, CPU profile missing due to bug)

## Key Queries

### Check TiKV termination signal
```bash
# Query Loki for signal/SIGTERM/SIGKILL in TiKV logs
{cluster_id="CLUSTER_ID", component="tikv"} |~ "signal|SIGTERM|SIGKILL|oom"
```

### Check Activity Hub for operations
```bash
# Query Activity Hub events around the restart window
curl -sS "$CLINIC_API/clusters/$CLUSTER_ID/events?start=$START&end=$END" \
  -H "Authorization: Bearer $TOKEN"
```

## Lessons Learned

1. **OOM is a hypothesis, not a conclusion from memory patterns alone** — memory dropping after restart can be Pod replacement (K8s rolling restart), not OOM. Must verify termination signal.
2. **Activity Hub events should be checked early for restart scenarios** — human operations (scaling, upgrades) are the most common cause of planned restarts. This was only discovered after 4 human interventions.
3. **Memory utilization context matters** — 66GB on a 128GiB node is 51%, which is normal. Agent should compare memory against node spec before concluding OOM.
4. **Agent excels at directed investigation** — once pointed in the right direction (rounds 5-12), the agent produced excellent detailed analysis (3-spike correlation, profiling bug discovery, tikv/tikv#15760).
5. **Profiling bug discovery was a bonus** — human-guided deep dive on tikv-6diswl uncovered that TiKV v8.5.4 likely still has tikv/tikv#17234 (CPU profiling blocked by heap profiling), with no fix cherry-picked to release-8.5.