# Postmortem: O11Y-2298309

> Workspace: `ws-1775048289893-544adbfbf`
> Date: 2026-04-07
> Reviewer: Claude Code (auto-replayer Mode D)

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | O11Y-2298309 |
| Alert Type | TiDB Pod Panic Restart |
| Cluster ID | 10766841556523235152 |
| Cluster Type | dedicated |
| TiDB Version | v8.5.2 |
| Time Window | 2026-04-01 11:35:48 UTC |

**Issue Description**: db-tidb-60 pod panic and restart caused by a known TiDB bug (GitHub #63956) triggered when KILL QUERY is executed on an idle transaction.

## Actual Root Cause

> Human-confirmed root cause from Jira resolution, oncall handoff, or expert analysis.

Known TiDB bug (GitHub Issue #63956): when a remote TiDB instance sends a KILL QUERY command to a connection holding an idle transaction, the transaction rollback triggers `memory.(*Tracker).Consume()` on an already-terminated session, causing a panic. Fixed in PR #63992 (recover panic) and PR #64001 (root fix: reset memory hook before rollback). Both cherry-picked to release-8.5 in Nov/Dec 2025, but v8.5.2 (released 2025-06-04) predates the fix.

## Agent Diagnosis Summary

> What the Agent concluded (final answer from unguided first attempt).

The agent correctly identified the panic as caused by GitHub Issue #63956. It traced the full chain: remote KILL command from 10.250.159.158 -> connection 2107655910 in idle transaction -> transaction rollback cleanup -> `memory.(*Tracker).Consume()` panic on terminated session. The agent found both fix PRs (#63992, #64001), verified their cherry-pick status to release-8.5, and confirmed v8.5.2 does not contain the fix. Recommended upgrading to v8.5.3+.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 5 | Correctly identified GitHub #63956; matched panic stack, trigger conditions, and affected version |
| Evidence Completeness | 5 | Logs -> panic stack -> GitHub issue -> PRs -> cherry-pick status -> version timeline; all relevant evidence gathered |
| Causal Chain | 5 | Clear chain: remote KILL -> txn rollback -> memory tracker Consume() -> panic; included reproduction steps |
| Actionability | 5 | Specific recommendation: upgrade to v8.5.3+; short-term mitigation (monitor KILL frequency); all PR links provided |
| Efficiency | 5 | 23 tool calls, all completed, 0 stuck, 0 human interventions; no wasted calls |
| **Overall** | **5.0** | |

## Human Guidance Log

> Every point where the reviewer had to manually steer the Agent.

No human interventions required. The agent completed the full analysis autonomously.

## Skill Deficiencies

> Actionable improvement items extracted from the guidance log.

No deficiencies identified in this case. The agent correctly:
- Chose log analysis over performance metrics (appropriate for panic/crash scenario)
- Found and correlated the GitHub issue
- Traced fix PRs and cherry-pick status
- Verified version timeline

## Key Queries

> Correct queries that the Agent should have used (for skill template reference).

### Loki error log query (TiDB panic)

The agent queried TiDB error/fatal logs around the restart time and extracted the panic stack trace from db-tidb-60 logs. This is the correct approach for pod restart investigations.

### GitHub issue search

The agent searched the tidb repo for "memory tracker panic" issues, found #63956, then traced PRs #63992 and #64001 including their cherry-pick status to the release-8.5 branch.

## Lessons Learned

> General insights beyond specific skill fixes.

- For panic/restart cases, log analysis + GitHub issue correlation is more valuable than performance metrics (TopSQL, SlowQuery). The agent correctly skipped irrelevant data sources.
- Checking fix cherry-pick status against the cluster's actual version is critical for determining if the issue is a known-but-unpatched bug.
- This case demonstrates the agent can handle non-performance issues (crash bugs) end-to-end without human intervention.