# Postmortem: O11Y-2489480

> Workspace: `ws-1777512849128-e855d536a`
> Date: 2026-05-01
> Reviewer: claude-code

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | O11Y-2489480 |
| Alert Type | TiDBComponentPodRestartsForJapan |
| Cluster ID | 10766841556523235152 |
| Cluster Type | Dedicated |
| TiDB Version | Unknown |
| Time Window | 2026-04-30 00:28:00 - 00:33:00 UTC |
| Region | gcp-asia-northeast1 (Japan) |
| Tenant ID | 1372813089209272332 |

**Issue Description**: Single TiDB pod `db-tidb-76` restarted at 00:32:10 UTC. Alert auto-recovered. Root cause investigation needed to confirm whether restart was OOM Kill, panic, or other.

## Actual Root Cause

> From workspace conversation — K8s events and Loki logs were inaccessible (Access Denied / 404), so root cause could not be fully confirmed. Best-fit hypothesis based on available signals:

**Most likely cause: Kubernetes cgroup OOM Kill** — the container memory limit was hit by a transient RSS spike, triggering K8s to terminate the process. Key evidence:
- RSS stable at ~4.7 GB for 50+ minutes before restart
- Go GC frequency surged 6x (0.017/s → 0.1/s) in the last 4 minutes before restart — classic signal of external memory pressure
- `tidb_server_memory_quota` = 26.6 GB (app-level quota not exceeded)
- `tidb_server_panic_total` = 0 (no Go panic)
- Loki logs returned 404 (tooling issue); K8s events returned Access Denied (tooling issue)

Both tooling failures prevented positive confirmation of OOM Kill from K8s events.

## Agent Diagnosis Summary

The agent correctly:
1. Confirmed restart time via `process_start_time_seconds` jump (Apr 21 → Apr 30 00:32:10 UTC)
2. Ruled out Go panic (`panic_total` = 0)
3. Ruled out application-level OOM (RSS 4.7 GB vs 26.6 GB quota = 18%)
4. Identified GC frequency surge as the key differentiating signal (6x spike)
5. Stated conclusion with appropriate uncertainty ("most likely K8s OOM Kill")
6. Provided actionable P1 recommendations (`kubectl describe pod`, check container limits)

The agent **could not**:
- Confirm from K8s events (Access Denied)
- Get Loki logs (404 error)
- Check TopSQL (Delta Lake path invalid for this cluster)
- Get Activity Hub events for any operational context

Note: TopSQL unavailability and Loki/Events tooling failures are data/access issues, not agent reasoning failures.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 4 | Correct hypothesis (K8s OOM Kill) with appropriate uncertainty; missed confirmation due to tooling failures |
| Evidence Completeness | 3 | Got core metrics; TopSQL unavailable (data pipeline); Loki/events tooling failures |
| Causal Chain | 4 | Clear GC spike → K8s OOM Kill chain even without confirmation data |
| Actionability | 5 | Specific P1/P2 actions including `kubectl describe pod` and memory limit comparison |
| Efficiency | 3 | 36 tool calls; multiple retries of collect_evidence; 1 failed Loki tool call; redundant tooling |
| **Overall** | **3.8** | Good diagnosis under adverse tooling conditions |

## Human Guidance Log

No manual human interventions. The agent ran autonomously from initial trigger to final diagnosis.

## Skill Deficiencies

### ISS-0036 (existing): Loki endpoint returns 404 for this cluster

- **Type**: TOOLING
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Loki query returns `404 page not found`
- **Expected behavior**: Should return logs or a meaningful error (empty result with count=0)
- **Suggested fix**: Verify Loki endpoint URL construction for GCP Japan region clusters; check if API key has Loki access
- **Priority**: P1-high
- **Status**: open (already tracked as ISS-0036)

### NEW: Activity Hub API returns HTML Access Denied for pod restart events

- **Type**: TOOLING
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: `Query cluster events for OOM or restart records` returns HTML "Access Denied" page instead of JSON
- **Expected behavior**: Should return K8s pod events or a JSON error
- **Suggested fix**: Check if the Activity Hub/events endpoint requires a different auth scope or cluster-level permission for Japan region dedicated clusters
- **Priority**: P1-high
- **Status**: open (see ISS-0054)

### ISS-0039 (existing): Activity Hub not proactively checked for node restart scenarios

- **Type**: FLOW
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent checks metrics/Loki but doesn't prioritize Activity Hub for pod restart alerts
- **Expected behavior**: Pod restart alert → Activity Hub first check (may reveal ScaleResourcePool, rolling upgrade, etc.)
- **Priority**: P1-high
- **Status**: open (already tracked as ISS-0039)

## Key Queries

### Confirm restart via process_start_time

```promql
process_start_time_seconds{job="tidb", instance="db-tidb-76"}
```

Detects: timestamp jump = process restart event.

### GC frequency as K8s OOM indicator

```promql
rate(go_gc_duration_seconds_count{job="tidb", instance="db-tidb-76"}[1m])
```

Key finding: 6x surge in last 4 minutes before restart = classic external OOM kill signal.

### RSS vs Go heap divergence

```promql
# RSS (container-level):
process_resident_memory_bytes{job="tidb", instance="db-tidb-76"}

# Go heap in-use:
go_memstats_heap_inuse_bytes{job="tidb", instance="db-tidb-76"}
```

Key finding: RSS stable at 4.7 GB but Go heap fluctuated wildly (2-4 GB), suggesting transient allocation spike hit container limit.

## Lessons Learned

1. **GC frequency surge is a reliable K8s OOM Kill proxy**: When Loki/K8s events are inaccessible, 6x GC frequency spike + heap drop → OOM Kill is a strong inference even without direct confirmation.

2. **TopSQL unavailability is data-pipeline-specific**: Japan region GCS Delta Lake paths may not have TopSQL data. Agent should not penalize itself for API 500 on invalid Delta Lake paths.

3. **Access Denied on cluster events = coverage gap**: The inability to access K8s events is a systematic tooling gap. In pod restart cases, K8s events would directly show OOMKilled status and reason. Fix is either auth scope expansion or alternative API path.

4. **Appropriate uncertainty is correct**: The agent's "most likely K8s OOM Kill" framing with P1 verification action (`kubectl describe pod`) is the right response when confirmation data is unavailable.