# Postmortem: O11Y-2377229

> Workspace: `ws-1776244750109-a1746115d`
> Date: 2026-04-15
> Reviewer: Claude Code

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | O11Y-2377229 |
| Alert Type | TiKV CPU High / Performance Degradation |
| Cluster ID | 10989049060142230334 |
| Cluster Type | Dedicated |
| TiDB Version | Unknown |
| Time Window | 2026-04-11 01:40:29 - 02:40:29 (UTC) |

**Issue Description**: TiKV CPU saturation (3 instances >80%) caused by combination of EBS IOPS throttling and high-frequency migration queries.

## Actual Root Cause

> To be confirmed from Jira resolution or oncall handoff.

**Primary**: EBS IOPS throttling affecting 8 volumes (>20% throttle rate), with peak at 26% on vol-0fa0db5d9244aa480.

**Secondary**: High-frequency SELECT queries on migration tables consuming significant TiKV CPU:
- SQL digest 5FE6D785: 12.8% TiKV CPU
- SQL digest EE719BD1: 10.0% TiKV CPU  
- SQL digest 37705ED2: 9.3% TiKV CPU

**Contributing factor**: ResolveLock RPC overhead (54% of query time = 622.8ms / 1150ms total) indicating transaction conflicts or stale locks.

## Agent Diagnosis Summary

The agent successfully identified the root cause in the first attempt without human intervention:

**Root Cause Identified**:
1. EBS IOPS throttling as primary infrastructure bottleneck
2. Migration workload queries as application-level contributor
3. TiKV CPU hotspots across 3 instances

**Evidence Collected**:
- ✅ Metrics: EBS IOPS, TiKV CPU saturation
- ✅ TopSQL: Both TiDB and TiKV side attribution
- ✅ SlowQuery: Execution statistics for hot queries
- ✅ Execution Plans: Decoded plans with actual vs estimated rows
- ❌ Logs: Not examined (agent determined unnecessary)
- ❌ Events: Not examined

**Recommendations Provided** (prioritized):
1. Fix EBS IOPS throttling (increase provisioned IOPS or migrate to gp3)
2. Optimize migration queries (add WHERE filters, pagination, create index on `migration.table_name`)
3. Investigate ResolveLock overhead (check for long transactions, lock contention, GC safepoint)
4. Monitor TiDB GC metrics (P99 GC pause at 0.111s)
5. Do NOT run ANALYZE (stats divergence only 1.57x, below 2x threshold)

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 5 | Correctly identified dual root cause (infrastructure + workload) |
| Evidence Completeness | 4 | Got all critical data sources; skipped logs (acceptable) |
| Causal Chain | 5 | Clear linkage: EBS throttling → I/O latency → TiKV CPU → query slowdown |
| Actionability | 5 | Prioritized recommendations with specific actions (index creation, IOPS tuning) |
| Efficiency | 5 | Clean execution, no wasted effort, appropriate data source selection |
| **Overall** | **4.8** | Excellent unguided diagnosis |

## Human Guidance Log

No human interventions required. The agent completed the diagnosis autonomously.

## Skill Deficiencies

None identified. The diagnosis was successful without any skill gaps or deficiencies.

## Key Queries

### TopSQL TiKV CPU Attribution

Agent successfully queried TopSQL for TiKV-side CPU attribution to identify hot SQL digests.

### Slow Query with Execution Plan

Agent retrieved decoded execution plans showing:
- Index scan: `idx_table_name_page_size`
- Actual rows: 45,076 vs Estimated: 28,736 (1.57x divergence)
- ResolveLock RPC: 622.8ms (54% of total)

### EBS Metrics

Agent identified 8 volumes with >20% IOPS throttling, correctly attributing this as the primary infrastructure bottleneck.

## Lessons Learned

### Positive Outcomes

1. **Dual-layer diagnosis**: Agent correctly identified both infrastructure (EBS) and application (queries) contributors, showing good systems thinking.

2. **Evidence-based prioritization**: Recommendations were properly prioritized based on impact (fix infrastructure bottleneck first, then optimize queries).

3. **Correct negative recommendation**: Agent explicitly stated NOT to run ANALYZE because stats divergence (1.57x) was below threshold, demonstrating understanding of when NOT to take action.

4. **Efficient data source selection**: Agent did not waste time on logs/events when metrics and query data were sufficient.

### Observations

1. **ResolveLock analysis**: Agent identified the high ResolveLock overhead but didn't deeply investigate the cause (long transactions, lock contention, GC lag). This is acceptable for initial diagnosis but could be enhanced in follow-up analysis.

2. **Missing metadata**: Cluster metadata (TiDB version, cluster type) was not properly populated in the workspace. This is a data pipeline issue, not an agent issue.

3. **Logs skipped**: Agent did not examine logs. This was appropriate for this case (infrastructure bottleneck), but in other cases logs might reveal critical error patterns.