# Postmortem: GH-501

> Workspace: `ws-1778579944582-3e343f5af`
> Date: 2026-05-13
> Reviewer: evolver-bot

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | GH-501 |
| Alert Type | Performance degradation / latency spike |
| Cluster ID | (dedicated, us-east-1, 32-core TiKV nodes) |
| Cluster Type | dedicated |
| TiDB Version | (unknown, not captured) |
| Time Window | 2026-05-12 17:30 UTC+8 onwards |

**Issue Description**: TiKV read hotspot driving unified_read CPU saturation and gRPC latency spike; agent made three diagnostic errors requiring user correction.

## Actual Root Cause

Per conversation: TiKV read hotspot driven by two high-frequency large-scan SQLs (digest F2C6... on dh_account_ext_parentinfo three-table LEFT JOIN, and digest 470AA5... on dh_active_receive_logs IndexLookUp). The concurrent `ModifyClusterVolume` (IOPS adjustment, not topology change) completing at ~16:05 UTC+8 was unrelated to the 17:30 latency event.

## Agent Diagnosis Summary

The agent eventually reached the correct root cause (TiKV read hotspot / SQL-driven CPU saturation) after three user interventions:
1. Intervention 1: Agent mentally computed epoch timestamps instead of using a script, getting 17:25 instead of 16:05 — causing it to wrongly link the ModifyClusterVolume event to the 17:30 incident.
2. Intervention 2: Agent confused `ModifyClusterVolume` (IOPS adjustment) with `ModifyClusterSize` (topology change).
3. Intervention 3: Agent misread `component_health` `tidb: 12→1` / `tikv: 27→11` as actual node down events — actually Prometheus window truncation artifact at 11:30 UTC query boundary.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 3 | Correct final conclusion, but required 3 corrections to get there |
| Evidence Completeness | 4 | Queried metrics, TopSQL, activity events — good coverage |
| Causal Chain | 3 | Chain presented correctly after corrections, but intermediate steps had logic errors |
| Actionability | 3 | Provided SQL-level recommendations after corrections |
| Efficiency | 2 | 18 tool calls, 3 human interventions; epoch mis-calculation caused wasted analysis round |
| **Overall** | **3.0** | |

## Human Guidance Log

### Intervention 1: Epoch timestamp mental calculation error

- **Round**: #6
- **User said**: "cluster modify size 时间行为的误判，不是扩缩绒，是调整 IOPS; modify size event 时间的误判"
- **Agent's mistake**: Manually computed `1778573148` epoch → "17:25 UTC+8" (correct is 16:05 UTC+8). This caused agent to wrongly connect ModifyClusterVolume to the 17:30 latency spike.
- **Correct behavior**: Must call the time-conversion script for all epoch→datetime conversions. Rule already documented in SKILL.md: "绝不手算 epoch".
- **Skill**: `tidb-perf-diagnosis`
- **Deficiency type**: KNOWLEDGE / REASONING

### Intervention 2: ModifyClusterVolume vs ModifyClusterSize confusion

- **Round**: #6
- **User said**: "不是扩缩绒，是调整 IOPS"
- **Agent's mistake**: Called the event `ModifyClusterSize` (topology/node count change) in analysis when the activity event was `ModifyClusterVolume` (EBS IOPS adjustment). These have fundamentally different impacts.
- **Correct behavior**: Read the event type string exactly from the Activity Hub API response, never paraphrase. `ModifyClusterVolume` = volume/IOPS config change, `ModifyClusterSize` = node count change.
- **Skill**: `tidb-perf-diagnosis`
- **Deficiency type**: KNOWLEDGE

### Intervention 3: component_health monitoring truncation misread

- **Round**: #8+
- **User said**: "component health 因为数据采集错误，导致的误判"
- **Agent's mistake**: Interpreted `tidb: 12→1`, `tikv: 27→11` in component_health as actual node failures/scale-down, and spent analysis time on this false signal.
- **Correct behavior**: When the query window end time exceeds the Prometheus data retention boundary, the last samples are sparse. `component_count.last` near the window end will appear to drop even when no nodes actually went down. Agent should check if the window endpoint is near "now" or a data boundary before interpreting component count drops as real events.
- **Skill**: `tidb-perf-diagnosis`
- **Deficiency type**: KNOWLEDGE

## Skill Deficiencies

### ISS-NEW-A: component_health count drop near window boundary is monitoring artifact, not real event

- **Type**: KNOWLEDGE
- **Skill**: `tidb-perf-diagnosis`
- **Current behavior**: Agent treats `component_count.last < component_count.first` as evidence of node failure/scale-down.
- **Expected behavior**: Agent must check whether the query window endpoint is near the current time or a data retention boundary. If `last` sample count drops near the window end, note it as a possible monitoring truncation artifact rather than an actual topology change. Validate against Activity Hub events before concluding.
- **Suggested fix**: Add note to SKILL.md: "component_health count drop near window boundary is a Prometheus sparse-sample artifact, not a real node loss. Validate with Activity Hub before treating as topology change."
- **Priority**: P1-high
- **Status**: open

### ISS-NEW-B: ModifyClusterVolume vs ModifyClusterSize must not be conflated

- **Type**: KNOWLEDGE
- **Skill**: `tidb-perf-diagnosis`
- **Current behavior**: Agent paraphrases activity event types, confusing ModifyClusterVolume (IOPS/volume config) with ModifyClusterSize (node count).
- **Expected behavior**: Quote event type verbatim from API. Know that `ModifyClusterVolume` = EBS config only, no rolling restart; `ModifyClusterSize` = node count change, triggers rolling restart.
- **Suggested fix**: Add to SKILL.md Activity Hub interpretation section: event type taxonomy distinguishing volume vs size vs scaling events and their expected cluster impact.
- **Priority**: P2-medium
- **Status**: open

## Key Queries

### Activity Hub event timing validation

```bash
# Always use time-conversion script, never mental math
python3 /path/to/time_conversion.py --epoch 1778573148 --tz UTC+8
# Expected: 2026-05-12 16:05:48 CST
```

## Lessons Learned

- Epoch timestamp mental calculation is forbidden — always use the script. One wrong mental conversion cascades into multiple wrong analytical conclusions.
- `component_health` drops at the query window boundary are Prometheus artifacts, not real events. Check Activity Hub before acting on them.
- Read activity event type strings verbatim — `ModifyClusterVolume` ≠ `ModifyClusterSize`.