# ISS-0036: Loki 日志查询对 panic 告警返回空结果

## Background

In workspace `lark-ws-3f41b760-4a79-4ce7-b523-ac731d3098ee` (O11Y-2429368), the agent queried Loki for panic logs but got zero results. The agent reported: "故障时间窗口内（±60min）的 Loki 日志中未发现 panic 相关日志条目。日志系统中 error/warn/info 计数均为 0。"

However, the user confirmed the panic log exists:
```
[2026/04/19 21:50:00.323 +00:00] [ERROR] [misc.go:95] ["panic in the recoverable goroutine"]
[r="[executor:3024]Query execution was interrupted, maximum statement execution time exceeded"]
```

## Possible Causes

1. **Time window too narrow**: The agent may have used a ±30min or ±60min window around the alert time. The panic log timestamp (21:50:00) should be within range, but if the alert metadata time is off, the window might miss it.

2. **Wrong LogQL query**: The Loki query might filter by log level or component incorrectly. For example:
   - Filtering `level="error"` but the panic log uses a different level field
   - Filtering by wrong component (e.g., `tikv` instead of `tidb`)
   - Using wrong label matchers for the cluster

3. **Clinic Loki data gap**: The Clinic platform may not have ingested logs for this time window on this cluster. This is an infrastructure issue outside agent control.

4. **Query string too specific**: If the agent searches for exact phrases like `"panic"` in structured log fields rather than using regex or broader patterns.

## Investigation Steps

### Step 1: Examine collect_evidence.py Loki query construction

Check how `collect_evidence.py` (A6 section) constructs the LogQL query for panic alerts:
- What label matchers are used? (`cluster_id`, `component`, `level`)
- What line filter is applied? (`|= "panic"`, `|~ "panic|fatal"`)
- What time range is used? (alert_time ± how many minutes)

### Step 2: Test with broader query

Manually test a broader Loki query against the Clinic API:
```
{cluster_id="10518506906906971664", component="tidb"} |~ "panic|executor:3024"
```
with time range 2026-04-19T20:00:00Z to 2026-04-19T23:00:00Z (±3h).

### Step 3: Compare with working panic case

In O11Y-2298309 (known working panic case), check if Loki queries succeeded. Compare the query construction between that case and this one to identify differences.

## Fix Plan

### Option A: Broaden default Loki query for panic alerts

In `collect_evidence.py`, when the alert type contains "Panic":
1. Use ±2h time window (instead of default ±1h)
2. Use broader line filter: `|~ "panic|PANIC|fatal|FATAL"`
3. Include all TiDB components: `component=~"tidb|tidb-.*"`
4. If first query returns empty, retry with ±4h window

### Option B: Add Loki query retry logic

In the agent's Loki query flow:
1. First attempt: standard query with ±1h window
2. If empty AND alert type is panic: retry with ±3h window and relaxed filters
3. If still empty: report "Unable to retrieve panic logs from Loki — data may not be available for this time window" instead of concluding FAKE ALARM

### Verification

1. After fix, replay O11Y-2429368 and verify Loki query returns the panic log
2. If Loki data is genuinely unavailable, verify the agent reports "unable to verify" rather than "FAKE ALARM"