# Postmortem: O11Y-2612493

> Workspace: `ws-1783591129535-db196fe8d`
> Date: 2026-07-13
> Reviewer: evolver-bot

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | O11Y-2612493 |
| Alert Type | RapidErrorRateIncreaseOver450ForJapan |
| Cluster ID | 10123293495242577103 |
| Cluster Type | dedicated |
| TiDB Version | unknown |
| Time Window | 2026-07-09 08:41:00 UTC – 2026-07-09 09:12:20 UTC |

**Issue Description**: Cluster error rate in Japan (GCP asia-northeast1) exceeded 4.5× the 5-minute baseline, triggered by 6 consecutive TiKV scale-out batches (36→54 nodes) causing PD region balance storms.

## Actual Root Cause

> Human-confirmed root cause from Jira resolution, oncall handoff, or expert analysis.

The Jira issue has only two automated comments: "notified ... by lark" and "Alert auto-recovered, no firing alert." No human diagnosis or confirmed root cause is present in Jira. The agent's diagnosis is the only analysis available.

Based on Activity Hub events and Loki logs: 6 consecutive TiKV scale-out batches (36→54 nodes) between 08:08–09:30 UTC triggered repeated PD region balance storms. After the 3rd batch completed at 08:39:16, accumulated region migrations drove error 1317 ("Query execution was interrupted") across 36+ TiDB nodes, exceeding the 4.5× threshold and triggering the alert. Resource groups (rg_item, rg_mercari, rg_user) with available_ru min=0 amplified request failures. Alert auto-recovered at 09:12:20 UTC.

## Agent Diagnosis Summary

> What the Agent concluded (final answer from unguided first attempt).

The agent correctly identified the root cause: 6 consecutive TiKV scale-out batches (36→54 nodes) triggered PD region balance storms, causing error 1317 "Query execution was interrupted" on 36+ TiDB nodes simultaneously. The agent accurately noted: TiDB CPU elevation was a side effect (not root cause), resource control RU exhaustion amplified failures, and TopSQL data was unavailable (Delta Lake path missing for this cluster). The diagnosis noted `db-tidb-2` / `db-tidb-970` restart signals but recommended checking restart logs separately. The causal chain, timeline reconstruction from Activity Hub, and recommendations were all well-structured.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 5 | Correctly identified scale-out → region balance → error 1317 chain |
| Evidence Completeness | 4 | Metrics, logs, Activity Hub, slow queries all queried; TopSQL unavailable (Delta Lake 500) — data pipeline issue not agent gap |
| Causal Chain | 5 | Clear timeline: each scale-out batch → PD balance → KV interrupt errors; excluded CPU/GC/disk as false candidates |
| Actionability | 5 | Specific: wait for region balance before next batch, lower region-schedule-limit, configure BURSTABLE RGs |
| Efficiency | 5 | 33 tool calls, 0 human interventions, 1 failed tool (write_analyst_report.py missing — fallback succeeded) |
| **Overall** | **4.8** | |

## Human Guidance Log

No human interventions were needed. The agent completed the diagnosis in a single unguided pass.

## Skill Deficiencies

### SD-1: write_analyst_report.py missing from workspace skills

- **Type**: TOOLING
- **Skill**: `tidb-perf-diagnosis`
- **Current behavior**: Agent calls `/data/workspaces/.../scripts/write_analyst_report.py` which does not exist, causing tool failure (exit 2)
- **Expected behavior**: Fallback path handles the merge correctly and succeeded; but the primary path should not reference a non-existent script
- **Suggested fix**: Verify that `write_analyst_report.py` is deployed with the skill, or update the primary merge path to use the fallback script directly
- **Priority**: P2-medium
- **Status**: open

### SD-2: TopSQL unavailable with no user-visible warning in diagnosis output

- **Type**: UX
- **Skill**: `tidb-perf-diagnosis`
- **Current behavior**: TopSQL fails with HTTP 500 (Delta Lake path missing); agent notes it only at the end as "TopSQL 数据无法获取"
- **Expected behavior**: Prominently warn at the start of the diagnosis when TopSQL is unavailable so the user knows the analysis lacks SQL attribution evidence
- **Suggested fix**: In diagnosis output, when TopSQL errors are in warnings, add a bold note at the start: "⚠️ TopSQL data unavailable for this cluster — SQL attribution limited to slow query logs"
- **Priority**: P2-medium
- **Status**: open (existing ISS-0057)

## Key Queries

### Activity Hub Events

Activity Hub correctly captured all 6 scale-out batches via `ModifyClusterSize` events, providing the definitive timeline.

### Loki Logs

Loki returned `[executor:1317]` / `[tikv:1317]` "Query execution was interrupted" log samples across 36+ TiDB instances, confirming the error type.

## Lessons Learned

- Scale-out operations in Japan (GCP asia-northeast1) trigger PD region rebalancing that can exceed error-rate alert thresholds; planning should include inter-batch wait periods
- This cluster does not have TopSQL data in Clinic (Delta Lake path missing) — this is a Clinic data pipeline gap, not a diagnosis gap; ISS-0057 tracks this
- Resource Control `available_ru min=0` is an amplifying factor in high-PD-scheduling-pressure scenarios; monitoring RU headroom during scale operations is advisable