# Postmortem: TCOC-5011

> Workspace: `ws-1775032909072-7b63e5f16`
> Date: 2026-04-01
> Reviewer: lingtao.li

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | TCOC-5011 |
| Alert Type | Cluster performance degradation |
| Cluster Type | dedicated |
| Time Window | 2026-03-31 02:35 - 03:24 UTC (BJ 10:35 - 11:24) |

**Issue Description**: Full cluster outage (~50 min) caused by 3 of 4 coprocessor TiKV workers going offline simultaneously, leading to cascading overload on the remaining single worker.

## Actual Root Cause

> Human-confirmed root cause from workspace diagnosis with human guidance.

At 02:35 UTC, 3 out of 4 coprocessor TiKV workers went offline simultaneously (cause unclear — possibly HPA miscalculation or node eviction). The single remaining worker was unable to handle the full cluster coprocessor load, causing request queue buildup, coprocessor P99 latency spiking to 100s+, TiKV gRPC blocking, TiDB query timeouts, and QPS collapse from 40K to 334. The remaining worker repeatedly OOM'd and restarted. Recovery occurred at 03:24 when new workers launched.

## Agent Diagnosis Summary

> What the Agent concluded after 8 human interventions.

The agent eventually identified the coprocessor worker failure cascade as root cause, with detailed timeline reconstruction. However, it required significant human steering:
- Initial time window was wrong (based on Jira creation time instead of actual incident time)
- Did not proactively check node health/uptime — had to be prompted
- Did not check coprocessor worker uptime specifically — had to be prompted
- Did not check logs until prompted

Final diagnosis quality was high once guided to the right data: correct root cause, detailed timeline, worker resource analysis, and actionable recommendations.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 4 | Correct root cause identified, but only with heavy guidance |
| Evidence Completeness | 3 | Missed TopSQL (TiDB+TiKV), decoded_plan; good on metrics/logs/events |
| Causal Chain | 5 | Excellent timeline reconstruction: worker offline -> overload -> cascade -> recovery |
| Actionability | 4 | Good recommendations (minReplicas, HPA tuning, monitoring), but no specific SQL/config |
| Efficiency | 2 | 54 tool calls, 8 human interventions — very inefficient unguided |
| **Overall** | **3.6** | |

## Human Guidance Log

> Every point where the reviewer had to manually steer the Agent.

### Intervention 1: Wrong time window

- **Round**: #2
- **User said**: "现根据metric确定故障时间"
- **Agent's mistake**: Was about to use Jira creation time as incident time
- **Correct behavior**: Should query metrics first to identify actual incident window
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: FLOW

### Intervention 2: Time window too late

- **Round**: #4
- **User said**: "应该是更早的10点左右"
- **Agent's mistake**: Identified wrong time window from metrics
- **Correct behavior**: Should correlate multiple metric signals to pinpoint start
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: REASONING

### Intervention 3: Jira creation time irrelevant

- **Round**: #6
- **User said**: "你不需要考虑jira创建时间"
- **Agent's mistake**: Was still anchoring on Jira creation timestamp
- **Correct behavior**: Use metric anomaly detection, not Jira metadata for time window
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: FLOW

### Intervention 4: Confirm time window

- **Round**: #8
- **User said**: "对的，北京时间10:00-10:30之间"
- **Agent's mistake**: Needed confirmation
- **Correct behavior**: Should be confident after metric analysis
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: REASONING

### Intervention 5: Use diagnosis skill

- **Round**: #10
- **User said**: "确定了时间后，我们用tidb-clinic-diagnosis分析下原因吧"
- **Agent's mistake**: Didn't automatically invoke the diagnosis skill
- **Correct behavior**: Should follow diagnosis flow after time window established
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: FLOW

### Intervention 6: Check node health

- **Round**: #12
- **User said**: "你是不是应该查下节点健康状态，比如uptime"
- **Agent's mistake**: Didn't check node health/uptime proactively
- **Correct behavior**: Node health check should be part of standard diagnosis flow
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: FLOW

### Intervention 7: Check coprocessor worker uptime

- **Round**: #14
- **User said**: "你没有差copressor worker的uptime吧"
- **Agent's mistake**: Only checked TiDB/TiKV uptime, missed coprocessor workers
- **Correct behavior**: Should check ALL component uptimes including coprocessor workers
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: KNOWLEDGE

### Intervention 8: Check logs

- **Round**: #16
- **User said**: "查下日志吧"
- **Agent's mistake**: Didn't query logs proactively
- **Correct behavior**: Loki log query should be standard step in diagnosis flow
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: FLOW

## Skill Deficiencies

> Actionable improvement items extracted from the guidance log.

### D1: Time window determination relies on Jira metadata

- **Type**: FLOW
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Uses Jira creation/started time as incident time anchor
- **Expected behavior**: Should query metrics first to detect anomaly window, then validate against Jira time
- **Suggested fix**: Add a "metric-based time window detection" step before A0 that queries P99 latency and QPS to find the actual spike window
- **Priority**: P1-high
- **Status**: open

### D2: No proactive node/component health check

- **Type**: FLOW
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Checks metrics, TopSQL, SlowQuery but not component uptime/restarts
- **Expected behavior**: Should check uptime/restart count for ALL components (TiDB, TiKV, PD, coprocessor workers) as a standard step
- **Suggested fix**: Add "Node B: Component Health Check" step that queries `process_start_time_seconds` for all components
- **Priority**: P1-high
- **Status**: open

### D3: Coprocessor worker not in diagnostic scope

- **Type**: KNOWLEDGE
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Only considers TiDB, TiKV, PD as diagnostic targets
- **Expected behavior**: Should include coprocessor TiKV workers, TiFlash, and other auxiliary components
- **Suggested fix**: Add coprocessor worker metrics to the diagnostic checklist (worker count, uptime, memory usage)
- **Priority**: P1-high
- **Status**: open

### D4: Logs not queried automatically

- **Type**: FLOW
- **Skill**: `tidb-clinic-diagnosis`
- **Current behavior**: Loki log query only happens when prompted
- **Expected behavior**: Error/Fatal log extraction should be a mandatory step
- **Suggested fix**: Add mandatory Loki log step to Node B or Node C in the diagnosis flow
- **Priority**: P2-medium
- **Status**: open

## Key Queries

> Correct queries that the Agent should have used.

### Coprocessor worker process start time (detect restarts)

```
process_start_time_seconds{job=~".*coprocessor.*"}
```

### Worker memory usage

```
process_resident_memory_bytes{job=~".*coprocessor.*"}
```

### Worker count over time

```
count(up{job=~".*coprocessor.*"})
```

## Lessons Learned

> General insights beyond specific skill fixes.

- Coprocessor worker failures are a distinct failure mode not covered by current diagnosis flow — the skill assumes TiDB/TiKV/PD are the only components that matter
- Time window determination is the #1 source of wasted effort — getting it wrong cascades into querying wrong data and wrong conclusions
- For "cluster performance degradation" alerts, component health (uptime/restarts) should be checked BEFORE diving into query-level analysis
- The agent needed 8 interventions for what should be a 0-2 intervention diagnosis — this case type (infrastructure failure vs query issue) needs a different diagnostic branch