# Postmortem: O11Y-2530675

> Workspace: `ws-1778205637705-1d3a5a116`
> Date: 2026-05-08
> Reviewer: evolver-bot

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | O11Y-2530675 |
| Alert Type | TiDB P95/P99 Latency Spike |
| Cluster ID | 10698648422638660035 |
| Cluster Type | dedicated |
| TiDB Version | v8.5.4 |
| Time Window | 2026-05-08 00:26 ~ 02:06 UTC |

**Issue Description**: Business traffic spike triggered EBS IOPS throttling on multiple TiKV volumes plus a gRPC hotspot on db-tikv-9, causing P99 latency to spike to 81–219s and lock-based write contention in COMMIT/UPDATE transactions.

## Actual Root Cause

> Human-confirmed root cause from user feedback (GitHub issue #415).

User reported "no need diagnose query, TiKV has problem" — indicating the primary issue was TiKV-level: EBS IOPS throttling across 5 TiKV volumes (up to 40% throttle) and a severe gRPC hotspot on db-tikv-9 (P99 104.86s vs peer average 6s, 17.5x ratio). The query-level statistics divergence and lock contention are secondary symptoms of the underlying TiKV infrastructure pressure, not the root cause requiring immediate action.

## Agent Diagnosis Summary

The agent correctly identified all key signals: db-tikv-9 gRPC hotspot (P99 104.86s, 17.5x ratio), EBS IOPS throttling on 5 TiKV volumes (highest 40%), COMMIT slow queries (avg 480s, 18,089 count), and the main SQL's wait_time (231s) >> process_time (10s). The causal chain correctly traced business traffic peak → query accumulation → EBS IOPS throttle → TiKV hotspot → latency spike.

However, the agent's recommendations were **ranked incorrectly**: P0 recommendation was "ANALYZE TABLE" (statistics refresh) rather than addressing TiKV infrastructure issues. The user's core concern was TiKV-level problems, but the agent front-loaded SQL-level fixes (ANALYZE, SQL Binding) and only listed EBS IOPS investigation as P1. This mismatch between agent priority ordering and actual root cause severity led to user dissatisfaction.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 4 | Correctly identified all signals; TiKV hotspot + EBS IOPS noted, but SQL stats issue over-weighted |
| Evidence Completeness | 5 | Queried metrics, TopSQL TiDB/TiKV, SlowQuery, decoded_plan, logs, events — all sources covered |
| Causal Chain | 4 | Chain is mostly correct but conflates contributing factors (stats divergence) with root cause (IOPS throttle + TiKV hotspot) |
| Actionability | 3 | P0 recommendation (ANALYZE) is valid but misplaced; real P0 should be TiKV IOPS/hotspot investigation |
| Efficiency | 5 | 13 tool calls, 0 human interventions, all completed — very clean execution |
| **Overall** | **4.2** | |

## Human Guidance Log

No human interventions during this diagnosis run. User dissatisfaction expressed via the `/issue` command after reviewing the output.

## Skill Deficiencies

### SD-1: Recommendation priority does not match root cause severity

- **Type**: REASONING
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent ranks ANALYZE TABLE as P0 recommendation when statistics divergence ≥ 5x is found, even when TiKV infrastructure issues (IOPS throttling, gRPC hotspot) are also present and more severe.
- **Expected behavior**: When both SQL-level issues (stats divergence) and TiKV infrastructure issues (EBS IOPS throttle, node hotspot) are present, infrastructure issues that caused the wait_time spike should be ranked higher (P0/P1) than statistics-level fixes (P1/P2).
- **Suggested fix**: Add a priority tiebreaker rule in Node D: if wait_time >> process_time AND infrastructure bottleneck (IOPS throttle > 20% or gRPC hotspot ratio > 5x) is present, promote infrastructure fix to P0 ahead of statistics/plan fixes.
- **Priority**: P1-high
- **Status**: open

### SD-2: Agent should explicitly state when query issues are secondary to infrastructure pressure

- **Type**: KNOWLEDGE
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Agent presents query-level analysis (stats divergence, execution plan) at the same level as infrastructure analysis, without clearly stating the causal hierarchy.
- **Expected behavior**: When high wait_time (>>process_time) is detected alongside IOPS throttle, the diagnosis should state: "Query execution plan issues are secondary symptoms; the primary bottleneck is TiKV IOPS saturation causing queries to queue."
- **Suggested fix**: Add a Node C/D rule: if wait_time > 10x process_time and IOPS throttle > 20%, prefix the summary with "Primary bottleneck: infrastructure (IOPS/node hotspot). Query optimization is secondary."
- **Priority**: P2-medium
- **Status**: open

## Key Queries

### Evidence collection
```bash
# collect_evidence.py successfully ran for cluster 10698648422638660035
# Time window: 2026-05-08 00:30:00 ~ 02:00:00 UTC
# Key finding: db-tikv-9 gRPC P99=104.858s (17.5x vs peers), EBS IOPS throttle 40%
```

## Lessons Learned

- When `wait_time >> process_time`, the bottleneck is queuing (IO/lock/resource), not query efficiency. Recommendations should prioritize the queuing root cause.
- EBS IOPS throttle + node-level gRPC hotspot are more urgent infrastructure signals than statistics divergence, even if the latter has a clear numeric threshold (29.4x divergence).
- The recommendation section needs a priority re-ranking step: assess severity of infrastructure vs. query issues and sort accordingly.