# ISS-0038: Node restart 诊断应验证终止信号后再判定 OOM

## Background

In workspace `ws-1776646913272-b0d2b9b99` (TCOC-5140), the agent observed 4 TiKV nodes restart with memory dropping from 66-68GB to 4-8GB. The agent immediately concluded OOM Killed without checking:
1. Whether the process received SIGKILL (OOM) or SIGTERM (graceful termination)
2. Whether there were OOM killer logs in the kernel/dmesg
3. Whether 66GB on a 128GiB node (51% utilization) was actually abnormal

The user had to explicitly ask "能看到是被动终止还是有序退出" (intervention #3) before the agent checked termination signals and found `signal 15` (SIGTERM) — proving it was NOT OOM.

## Observed Behavior

Agent's reasoning chain:
```
Memory before restart: 66-68 GB
Memory after restart: 4-8 GB
→ "Memory drop = OOM Killed 的典型特征"
```

This is a common false positive. Memory dropping after restart can also indicate:
- K8s Pod replacement (rolling restart/upgrade) — new Pod starts with empty cache
- K8s eviction (memory pressure) — SIGTERM, not SIGKILL
- Manual Pod deletion
- Node hardware failure + Pod rescheduling

## Root Cause

The `tidb-clinic-diagnosis` SKILL.md has no OOM verification checklist. The agent treats "memory drops after restart" as sufficient evidence for OOM, when it's actually just one of several possible explanations.

## Fix Plan

### Step 1: Add OOM verification checklist to SKILL.md

In Node C (analysis routing), add a mandatory verification for OOM diagnosis:

```markdown
### OOM Verification Checklist (MANDATORY before concluding OOM)

If you suspect OOM on a TiKV/TiDB/PD node, verify ALL of the following:

1. **Termination signal**: Query Loki for `signal` in the node's logs
   - `signal 9` (SIGKILL) → likely OOM Killer
   - `signal 15` (SIGTERM) → graceful termination (NOT OOM)
   - No signal log → check if logs were truncated by crash

2. **OOM killer evidence**: Search for `oom_kill` or `Out of memory` in kernel logs
   - If found → confirm OOM
   - If not found → OOM not confirmed

3. **Memory context**: Compare memory usage against node spec
   - Usage < 80% of spec → NOT OOM (normal operating range)
   - Usage > 90% of spec → possible OOM, continue investigation
   - Usage ≈ 100% of spec → likely OOM

4. **Activity Hub check**: Query cluster events for recent operations
   - ScaleResourcePool / UpgradeCluster / Maintenance → planned restart, not OOM

Only conclude OOM if items 1+2 confirm AND item 4 rules out planned operations.
```

### Step 2: Add incorrect-conclusion entry

```json
{
  "id": "ic-004",
  "trigger": "Agent concludes OOM based solely on memory drop after restart",
  "correction": "Memory dropping after restart can be Pod replacement (rolling upgrade), not OOM. Must verify: (1) SIGKILL vs SIGTERM in logs, (2) OOM killer messages, (3) memory relative to node spec, (4) Activity Hub for planned operations."
}
```

### Verification

1. Replay TCOC-5140 and verify:
   - Agent checks termination signal before concluding OOM
   - Agent finds SIGTERM and does NOT conclude OOM
   - Agent checks Activity Hub and finds ScaleResourcePool operation
2. Replay TCOC-5011 (actual OOM case) and verify:
   - Agent still correctly identifies real OOM with SIGKILL evidence