# Postmortem: O11Y-2429368

> Workspace: `lark-ws-3f41b760-4a79-4ce7-b523-ac731d3098ee`
> Date: 2026-04-20
> Reviewer: claude-code
> Source: GitHub issue #217

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | O11Y-2429368 |
| Alert Type | TiDBUnexpectedPanicInRecoverableGoroutine |
| Cluster ID | 10518506906906971664 |
| Cluster Type | dedicated |
| TiDB Version | v8.5.4 |
| Time Window | 2026-04-19 21:50 UTC |
| Environment | prod |
| User Feedback | "问题诊断不对，1. 实际有 panic 2. 但是 panic 来自于用户请求主动超时中断" |

**Issue Description**: Panic alert fired but agent failed to find the panic log, then misclassified the alert as FAKE ALARM. The actual panic is `[executor:3024]Query execution was interrupted, maximum statement execution time exceeded` — a normal, expected behavior when user queries exceed `max_execution_time`, not a bug.

## Actual Root Cause

> From user feedback in GitHub issue #217 title.

The panic is real and comes from TiDB's recoverable goroutine panic handler when a query exceeds `max_execution_time`. The error `[executor:3024]Query execution was interrupted, maximum statement execution time exceeded` is **expected behavior** — TiDB panics in the recoverable goroutine to interrupt the query, then recovers. This is not a bug and requires no fix. The correct diagnosis should acknowledge the panic exists, identify it as a timeout-interrupt pattern, and classify it as normal/expected.

## Agent Diagnosis Summary

> Auto-diagnosis via Lark integration (0 human interventions, ~15 tool calls inferred from content).

The agent:

1. Read issue metadata and executed KB Step A (dedicated cluster routing) and Step B (incorrect conclusions prevention)
2. Collected evidence: metrics, TopSQL (TiDB + TiKV), SlowQuery, execution plans
3. Attempted Loki log query but **got zero results** — reported "未发现 panic 相关日志条目"
4. Concluded the panic alert was a **FAKE ALARM** ("Jira 已标为 FAKE ALARM，符合此判断")
5. Pivoted to analyzing TiKV CPU hotspot from TopSQL — found a JOIN query on `dh_active_bet_total` / `dh_account_ext_inviterinfo` with severe statistics divergence (4714x / 10045x)
6. Provided recommendations for ANALYZE + missing index on `(inviter_useridx, site_code)`

**Critical failures:**
- **Failed to find the panic log**: The panic `[executor:3024]` exists but agent's Loki query returned nothing. Likely wrong query filter or time window.
- **Wrong conclusion**: Classified as FAKE ALARM when the panic is real (just expected/normal).
- **No panic classification logic**: Even if the log was found, agent has no knowledge to distinguish normal panics (timeout interrupts) from abnormal panics (bugs).
- **Went off-topic**: Spent most of the diagnosis on an unrelated TiKV performance issue instead of addressing the panic alert.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 1 | Concluded FAKE ALARM; actual panic exists and is normal timeout-interrupt. Completely wrong. |
| Evidence Completeness | 2 | TopSQL/metrics collected, but Loki logs failed (the most critical data source for panic alerts) |
| Causal Chain | 1 | No causal chain for the panic. The TiKV performance chain is unrelated to the alert. |
| Actionability | 2 | TiKV recommendations are valid but address the wrong problem. No action for the actual panic. |
| Efficiency | 3 | Autonomous completion (0 human interventions), but wasted effort on wrong analysis path |
| **Overall** | **1.8** | |

## Human Guidance Log

No human interventions (auto-diagnosis via Lark). The diagnosis completed autonomously but reached the wrong conclusion.

## Skill Deficiencies

### ISS-0035: Agent does not distinguish normal panics from abnormal panics

- **Type**: KNOWLEDGE
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent treats all panics equally — either as serious bugs or dismisses them as fake alarms
- **Expected behavior**: Agent should match panic messages against known normal patterns (e.g., `[executor:3024]` timeout interrupt) and classify accordingly
- **Suggested fix**: Add a known normal panic pattern list to SKILL.md or incorrect-conclusions knowledge base
- **Priority**: P1-high
- **Status**: open

### ISS-0036: Loki log query for panic alerts returns empty results

- **Type**: TOOLING
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent's Loki query for panic logs returned zero results despite panic actually existing
- **Expected behavior**: Loki query should find the panic log entry within the alert time window
- **Suggested fix**: Investigate Loki query filter/time window construction; ensure panic-related queries use broader time range and correct log level filters
- **Priority**: P1-high
- **Status**: open

## Data Source Coverage

- [x] Metrics (via collect_evidence.py)
- [x] TopSQL TiDB
- [x] TopSQL TiKV
- [x] SlowQuery logs
- [x] Decoded execution plans
- [ ] Loki error logs (query returned empty — **critical miss for panic alert**)
- [ ] Cluster events

## Lessons Learned

1. **Panic alerts MUST find the panic log** — if Loki returns empty for a panic alert, the agent should broaden the time window, try alternative queries, or flag "unable to verify" rather than concluding FAKE ALARM
2. **Panic classification is essential** — TiDB has multiple "normal panic" patterns (timeout interrupt, kill query, etc.) that are expected behavior. Agent needs a pattern list to avoid misclassifying these as bugs or dismissing them as fake
3. **Alert type should drive analysis priority** — for `TiDBUnexpectedPanicInRecoverableGoroutine`, the #1 priority is finding and classifying the panic, not analyzing unrelated TiKV performance metrics
4. **Don't pivot away from the core question** — agent spent most effort on TiKV CPU hotspot analysis, which is useful context but doesn't answer "what caused the panic alert"