# Expected Diagnosis: O11Y-2489480

> Ground truth for regression testing
> Case type: Single TiDB pod restart — K8s OOM Kill (unconfirmed due to tooling gaps)
> Alert: TiDBComponentPodRestartsForJapan / cluster gcp-asia-northeast1

## Root Cause

**Single TiDB pod `db-tidb-76` was OOM-killed by Kubernetes cgroup** at 2026-04-30 00:32:10 UTC. The container memory limit was exceeded by a transient RSS allocation spike. Application-level OOM (`tidb_server_memory_quota`) was not involved — RSS (~4.7 GB) was well below the 26.6 GB app quota. K8s events and Loki logs were inaccessible (tooling gaps), so OOM Kill is the best-fit hypothesis confirmed by the GC frequency surge pattern.

## Evidence Chain

1. **Restart confirmed**: `process_start_time_seconds` for `db-tidb-76` jumped from 2026-04-21 23:00:53 → 2026-04-30 00:32:10 UTC — confirms process fully restarted
2. **Go panic ruled out**: `tidb_server_panic_total` = 0, no increment around restart time
3. **App-level OOM ruled out**: `tidb_server_memory_quota` = 26.6 GB; RSS stable at 4.6–4.9 GB (18% of quota)
4. **GC surge as OOM indicator**: GC frequency rose from baseline 0.017/s → 0.1/s (6x) in the 4 minutes before restart; Go heap in-use dropped to 0.31 GB at restart — characteristic of external SIGKILL under memory pressure
5. **K8s events**: Access Denied (tooling gap — cannot directly confirm OOMKilled status)
6. **Loki logs**: 404 (tooling gap — cannot get container logs confirming OOM)

## Causality

```
Go heap fluctuating 2–4 GB (transient allocation spikes)
  → Transient RSS peak exceeds container memory limit (cgroup)
    → Kubernetes OOMKiller sends SIGKILL to db-tidb-76
      → GC frequency spikes 6x in final 4 minutes (runtime memory pressure)
        → Process restart at 00:32:10 UTC
          → Alert TiDBComponentPodRestartsForJapan fires at 00:33:00 UTC
```

## Severity

**Medium** — Alert auto-recovered; only 1 of 24 TiDB nodes affected; cluster continued serving traffic.

## Recommendations

1. Run `kubectl describe pod db-tidb-76 -n tidb10766841556523235152` to confirm `OOMKilled` reason and memory limit
2. Compare container memory limit vs RSS peak (~4.9 GB) — if limit < 8 GB, increase it or reduce `server-memory-quota`
3. Investigate if any slow query (e.g., `select * from account_presents` with IndexLookUp scanning 2.61M key rows) caused transient memory spike
4. Fix Loki endpoint for Japan GCP region to restore log access for future pod restart investigations

---

## Scoring Rubric

> The agent MUST satisfy ALL items below to pass regression.

### Restart Confirmation (2 items)

- [ ] **Confirm restart time**: State that `db-tidb-76` restarted at approximately 2026-04-30 00:32 UTC (use `process_start_time_seconds` or equivalent)
- [ ] **Restart type**: State that this is a process restart (not just alert noise), confirmed by process start time jump

### Ruling Out Alternatives (2 items)

- [ ] **Rule out Go panic**: State that `tidb_server_panic_total` = 0 (no panic), or that no panic evidence found
- [ ] **Rule out app-level OOM**: State that TiDB `server-memory-quota` (or equivalent) was NOT exceeded — RSS was well below quota

### Root Cause Identification (2 items)

- [ ] **Identify K8s OOM Kill hypothesis**: State that K8s cgroup OOM Kill is the most likely cause (even if unconfirmed due to missing events/logs)
- [ ] **GC frequency surge**: Identify the GC frequency spike (≥3x increase) in the minutes before restart as a supporting signal

### Evidence Quality (2 items)

- [ ] **Memory stability context**: Report that RSS was stable at ~4.7 GB (not trending up sharply) — i.e., this was a transient spike, not gradual exhaustion
- [ ] **Acknowledge tooling gaps**: Note that K8s events and/or Loki logs were unavailable (Access Denied / 404), explaining why OOM Kill cannot be directly confirmed

### Recommendations (2 items)

- [ ] **Verification action**: Recommend checking K8s pod description (`kubectl describe pod`) for OOMKilled status and memory limit
- [ ] **Container limit check**: Recommend comparing container memory limit against RSS peak and TiDB `server-memory-quota` setting

---

**Total Rubric Items**: 10

**Pass Criteria**: 8/10 (80%) — Tooling gap items (#7 and #8) are credit but not required for pass

**Regression Status**: First baseline run (postmortem only; no regression run yet)

---

## Notes

- **Tooling gaps in this case**: TopSQL (Delta Lake path invalid for Japan GCP), Loki (404), K8s Events (Access Denied) — these are environment issues, not agent failures. Regression should not penalize missing TopSQL/Loki data.
- **Key differentiator**: GC frequency surge is the diagnostic signal that distinguishes K8s OOM Kill from graceful restart / rolling upgrade. Agent must use this signal.
- **ISS-0036**: Loki 404 for this cluster is a known tracked issue.
- **ISS-0054**: Activity Hub Access Denied for Japan dedicated clusters is a new issue.