# ISS-0035: Agent 未区分正常 panic（超时中断）和异常 panic（bug）

## Background

User `nizheming@pingcap.com` reported in GitHub issue #217 that workspace `lark-ws-3f41b760-4a79-4ce7-b523-ac731d3098ee` (O11Y-2429368) diagnosed a panic alert incorrectly. The agent concluded the alert was a FAKE ALARM, but the actual panic exists:

```
[2026/04/19 21:50:00.323 +00:00] [ERROR] [misc.go:95] ["panic in the recoverable goroutine"]
[r="[executor:3024]Query execution was interrupted, maximum statement execution time exceeded"]
```

The user states: "This panic is normal, not a bug." The panic occurs when a query exceeds `max_execution_time` — TiDB intentionally panics in a recoverable goroutine to interrupt the query, then recovers automatically.

## Known Normal Panic Patterns in TiDB

| Error Code | Message Pattern | Cause | Normal? |
|------------|----------------|-------|---------|
| `executor:3024` | `Query execution was interrupted, maximum statement execution time exceeded` | User query exceeds `max_execution_time` setting | Yes — timeout enforcement |
| `executor:???` | `Query execution was interrupted` (KILL QUERY) | Another session issued KILL QUERY | Yes — intentional kill |
| N/A | `runtime error: index out of range` in recoverable goroutine | Potential bug | No — investigate |
| N/A | Panic in `memory.Tracker.Consume` (known bug #63956) | Bug in TiDB v8.5.2 | No — known bug |

## Root Cause

The `tidb-clinic-diagnosis` SKILL.md has no guidance on panic classification. The agent treats panic alerts as binary: either a bug (serious) or fake alarm (no panic at all). It lacks the concept of "real panic but expected behavior."

## Fix Plan

### Step 1: Add panic classification knowledge to SKILL.md

In the Node C (analysis routing) section, add a panic classification step:

```markdown
### Panic Alert Analysis (TiDBUnexpectedPanicInRecoverableGoroutine)

When analyzing panic alerts, classify the panic before determining severity:

**Step 1: Find the panic log**
Query Loki for the panic log entry within ±2h of the alert time. Search for:
- `panic in the recoverable goroutine`
- The specific error message in the panic

**Step 2: Classify the panic**
Match the panic message against known normal patterns:

| Pattern | Classification |
|---------|---------------|
| `[executor:3024]Query execution was interrupted, maximum statement execution time exceeded` | NORMAL — query timeout interrupt |
| `Query execution was interrupted` (from KILL QUERY) | NORMAL — intentional session kill |
| Other patterns not in the list | ABNORMAL — investigate further |

**Step 3: Report based on classification**
- NORMAL: "The panic is real but expected behavior (timeout interrupt). No fix needed. Optionally investigate why the query timed out."
- ABNORMAL: Proceed with full root cause analysis of the panic.
```

### Step 2: Add to incorrect-conclusions knowledge base

In `incorrect-conclusions.json` (ic-XXX), add:

```json
{
  "id": "ic-003",
  "trigger": "Agent concludes panic alert is FAKE ALARM without finding the actual panic log",
  "correction": "A panic alert should not be dismissed as FAKE ALARM unless Loki confirms zero panic entries in ±2h window. If panic is found, classify it (normal timeout vs. abnormal bug) before determining severity."
}
```

### Step 3: Verification

1. Replay O11Y-2429368 and verify:
   - Agent finds the panic log
   - Agent classifies `[executor:3024]` as normal timeout interrupt
   - Agent does NOT conclude FAKE ALARM
   - Agent recommends no action for the panic itself
2. Replay O11Y-2298309 and verify:
   - Agent still correctly identifies the `memory.Tracker.Consume` panic as a known bug (#63956)
   - The normal panic filter does NOT suppress this abnormal panic