# Postmortem: O11Y-2233230

> Workspace: `ws-1774410786847-0ad377763`
> Date: 2026-05-09
> Reviewer: claude-code

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | O11Y-2233230 |
| Alert Type | TiDB P999 Latency |
| Cluster ID | Unknown |
| Cluster Type | Dedicated |
| TiDB Version | Unknown |
| Time Window | 2026-03-21 02:37:00 - 03:37:00 UTC |

**Issue Description**: TiDB P999 query latency spiked during the alert window. Root cause: compound — EBS IOPS throttling (infrastructure) + high-frequency inefficient slow queries (application).

## Actual Root Cause

> From workspace conversation — no Jira resolution or expert confirmation available; inferred from agent diagnosis.

**Compound root cause**: EBS IOPS throttling on 37 volumes (>20%, peak 38.9%) degraded TiKV write path, while a high-frequency inefficient query (`SELECT * FROM dh_active_receive_logs`) with 11,420 executions in 30 minutes (avg latency 1.53s) amplified the pressure. Commit time spiked 3-4x and P999 latency crossed alert threshold.

## Agent Diagnosis Summary

The agent correctly:
1. Identified EBS IOPS throttling as infrastructure root cause (37 volumes >20%, peak 38.9%)
2. Identified top slow query: `SELECT * FROM dh_active_receive_logs` with 56 fields, 11,420 executions, avg latency 1.53s
3. Confirmed Commit time spike (120→570s/min) tied to TiKV write path slowdown
4. Proposed appropriate indexes and SELECT optimization

The agent **skipped**:
- TopSQL TiDB/TiKV data (tool calls show MCP metrics only, no collect_evidence.py / TopSQL)
- Loki log queries (no log evidence collected)
- Decoded query plan analysis (no `decoded_plan` fetched)
- Activity Hub events check

Note: The workspace used an older skill flow (pre-collect_evidence.py). The agent used a mix of MCP metric tools and clinic-api directly.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 4 | Correctly identified EBS throttling + slow query combo; causal chain partially shown |
| Evidence Completeness | 3 | Got EBS metrics, DB time model, slow queries; skipped TopSQL, Loki, decoded plan |
| Causal Chain | 3 | "EBS throttle → write latency → P999 spike" stated but not fully quantified |
| Actionability | 4 | Specific index DDL and SELECT optimization provided |
| Efficiency | 3 | 21 tool calls; auth path discovery overhead; no TopSQL/log collection |
| **Overall** | **3.4** | |

## Human Guidance Log

No manual human interventions recorded. The agent ran autonomously.

## Skill Deficiencies

### ISS-0005 / ISS-0006 (pre-existing): TopSQL not queried

- **Type**: FLOW / TEMPLATE
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent used MCP metrics and clinic-api directly without collect_evidence.py; no TopSQL data collected
- **Expected behavior**: TopSQL TiDB and TiKV should both be queried as mandatory steps
- **Priority**: P1-high
- **Status**: fixed in collect_evidence.py (pre-existing, already tracked)

### ISS-0008 (pre-existing): No decoded_plan analysis

- **Type**: TEMPLATE
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Slow queries fetched but no decoded_plan extracted
- **Expected behavior**: Top slow query digests should have decoded_plan fetched and analyzed
- **Priority**: P1-high
- **Status**: fixed in collect_evidence.py A3b (pre-existing, already tracked)

### ISS-0004 (pre-existing): Loki logs not queried

- **Type**: FLOW
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: No log queries performed
- **Expected behavior**: Error/Fatal logs should be collected as mandatory step
- **Priority**: P2-medium
- **Status**: fixed in collect_evidence.py A6 (pre-existing, already tracked)

## Key Queries

### EBS IOPS Throttling

```bash
# clinic-api EBS throttling query
# Returns: total_volumes, severe_gt20pct, moderate_5_20pct, peak_throttle_pct
```

### DB Time Model

```promql
# Commit time spike: baseline ~120-160s/min → peak ~450-570s/min (3-4x)
# SELECT queries: no data collected (TopSQL gap)
```

## Lessons Learned

1. **Pre-collect_evidence.py diagnosis quality is lower**: Workspaces run before collect_evidence.py was introduced miss TopSQL, decoded_plan, and Loki — the current skill is significantly better.

2. **EBS throttling + slow query compound**: This is the second case (after O11Y-2190994) showing the same pattern — EBS throttling amplifies the impact of otherwise-manageable slow queries. A compound root cause template would improve future diagnoses.

3. **Write vs read dominant diagnosis matters**: O11Y-2233230 was write-dominant (Commit spike 3-4x), while O11Y-2190994 was read-dominant (SELECT +1253%). The same EBS throttling root cause manifests differently depending on workload.