# Postmortem: TCID-12625

> Workspace: `ws-1778484823635-24419a844`
> Date: 2026-05-13
> Reviewer: evolver-bot

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | TCID-12625 |
| Alert Type | TSO latency / cluster performance degradation |
| Cluster ID | (unknown — workspace empty, no conversation data) |
| Cluster Type | (unknown) |
| TiDB Version | (unknown) |
| Time Window | Around 21:27 JST 2025-04-20 |

**Issue Description**: User filed three quality complaints against the agent diagnosis: wrong EBS volume type (gp2 vs gp3), unsubstantiated network anomaly inference from customer comment, and etcd disk I/O stall claim with no evidence.

## Actual Root Cause

Unknown — workspace has no conversation messages (empty). Root cause derived from user feedback titles only.

## Agent Diagnosis Summary

Unable to evaluate — workspace `ws-1778484823635-24419a844` is empty (0 messages). The diagnosis may have been performed in a different session or the messages were not persisted. Three user feedback issues were filed via `/issue` command:

1. **#494**: Agent claimed "PD instance volumes are gp2" — user corrected that all PD volumes are gp3.
2. **#495**: Agent stated "development and production clusters experienced the same delay at 21:27 JST proving a shared infrastructure cause" — user says this was inferred from customer comment, not verified in Clinic network metrics.
3. **#496**: Agent concluded "Intermittent etcd disk I/O stalls on the PD leader and follower nodes caused TSO slowdown" — no evidence from application logs or metrics was provided.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | N/A | Cannot evaluate — no conversation data |
| Evidence Completeness | 1 | Three facts stated without evidence |
| Causal Chain | N/A | Cannot evaluate |
| Actionability | N/A | Cannot evaluate |
| Efficiency | N/A | Cannot evaluate |
| **Overall** | **N/A** | |

## Human Guidance Log

No conversation available. Issues derived from user feedback titles only.

## Skill Deficiencies

### ISS-NEW-C: Agent states EBS volume type (gp2/gp3) without evidence from cluster metadata

- **Type**: REASONING
- **Skill**: `tidb-perf-diagnosis`
- **Current behavior**: Agent asserts EBS volume type in diagnosis without querying or verifying from cluster metadata API.
- **Expected behavior**: EBS volume type (gp2/gp3) must be read from `cluster_meta.json` or TiDB Cloud node API, never assumed. The default changed from gp2 to gp3 in recent TiDB Cloud versions.
- **Suggested fix**: Add pre-diagnosis check: query cluster node volume type from metadata before making storage-related claims. Add note in SKILL.md: "Never assume EBS type — verify from cluster_meta or TiDB Cloud API."
- **Priority**: P1-high
- **Status**: open

### ISS-NEW-D: Agent infers "shared infrastructure cause" from customer comment without metric verification

- **Type**: REASONING
- **Skill**: `tidb-perf-diagnosis`
- **Current behavior**: Agent uses customer-provided timing narrative to assert correlated events (e.g., "both clusters affected simultaneously → shared infrastructure cause") without querying network metrics.
- **Expected behavior**: Correlation claims must be verified from objective data (Clinic metrics, network latency graphs) not from customer comments. Customer comments provide hypotheses to test, not evidence.
- **Suggested fix**: Add rule to SKILL.md Node D: "Customer-reported observations are hypotheses. To assert 'shared infrastructure cause', query network metrics for both clusters in the same time window and show correlated anomalies."
- **Priority**: P1-high
- **Status**: open

### ISS-NEW-E: Agent concludes "etcd disk I/O stall" without supporting log or metric evidence

- **Type**: REASONING
- **Skill**: `tidb-perf-diagnosis`
- **Current behavior**: Agent presents infrastructure root causes (etcd disk I/O stall, PD TSO slowdown) as conclusions without showing the evidence chain (PD logs, etcd latency metrics, fsync latency).
- **Expected behavior**: etcd/disk I/O stall claims require: (1) PD/etcd log entries showing slow fsync or disk latency warnings, (2) disk latency metrics (p99 write latency > threshold), or (3) etcd raft log evidence. Without ≥1 of these, the claim is inadmissible.
- **Suggested fix**: Add to SKILL.md: "To conclude etcd disk I/O stall: must show PD logs with 'slow' or 'fsync' entries, OR disk write latency > 50ms p99. Stating this without evidence is not permitted."
- **Priority**: P1-high
- **Status**: open

## Key Queries

### Verify EBS volume type

```bash
# From cluster_meta.json (collected by collect_evidence.py A0)
cat cluster_meta.json | jq '.nodes[] | {name, volume_type, volume_size}'
# Or via TiDB Cloud API
# GET /api/v1beta/projects/{project_id}/clusters/{cluster_id}/components
```

### PD etcd disk stall evidence

```bash
# Query PD logs for fsync/disk latency warnings
# In collect_evidence.py A6: search for 'slow' or 'fsync' in PD component logs
# Disk write latency metric: tikv_disk_latency or pd_disk_latency via clinic metrics API
```

## Lessons Learned

- EBS volume type (gp2/gp3) must be verified from metadata — never assume. gp3 is now default in most TiDB Cloud dedicated clusters.
- Customer comments are hypotheses, not evidence. Network anomaly claims require Clinic network metrics.
- Infrastructure root cause assertions (etcd disk I/O stall) require concrete log or metric evidence before being stated as conclusions.