# Postmortem: GH-264 (Cross-Workspace Inconsistency — Same Ticket, Different Conclusions)

> Workspace A: `lark-ws-5c83e277-065b-4f29-8c06-033756674f83` (2026-04-21)
> Workspace B: `lark-ws-a7da5d80-9dce-456b-b947-627f49eb03df` (2026-04-20)
> Date: 2026-04-21
> Reviewer: claude-code
> Source: GitHub issue #264

## Issue Summary

| Field | Value |
|-------|-------|
| Ticket ID | O11Y-2426677 |
| Alert Type | Expensive query (100+ queries > 60s in 1 min) |
| Cluster Type | dedicated |
| Time Window | 2026-04-19 09:01 UTC+8 |
| Environment | prod |
| User Feedback | "两个 workspace 对同一问题的结论不一致" |

**Issue Description**: Two independent Lark bot sessions diagnosed the same O11Y-2426677 expensive query alert but reached **different conclusions**. Workspace B correctly identified the missing index on `dh_account_basic.last_update_timestamp` (confirmed by L3 expert), while Workspace A attributed the problem to multiple hot SQL patterns and recommended ANALYZE as P0 instead of adding an index.

## Ground Truth (L3 Expert)

The L3 expert confirmed:
- **Root cause**: Missing index on `dh_account_basic.last_update_timestamp` causing IndexFullScan of ~52M rows
- **Fix**: Add composite index `(last_update_timestamp, birthday_secret)` + SQL binding

## Workspace A Diagnosis (Incorrect)

> 4 messages, 0 human interventions. Auto-diagnosis via Lark bot.

The agent:
1. Collected metrics: TiKV gRPC P99, CPU, EBS IOPS — same data as WS-B
2. Identified hot nodes: db-tikv-67 (90% CPU), db-tikv-74 (77% CPU) — correct
3. Found multiple hot SQL via TopSQL TiKV:
   - **Focused on**: `dh_message_setting` JOIN (db-tikv-30, cpu_pct=65.4%)
   - **Focused on**: `dh_pay_order_success_user_summary` (divergence 140x)
   - **Mentioned but did not deep-dive**: `dh_account_basic` (digest `9C0FA76B`, db-tikv-67/74)
4. **Concluded**: Multiple high-frequency read SQL causing TiKV CPU saturation
5. **P0 recommendation**: ANALYZE TABLE (statistics fix), not add index

**Critical error**: The agent saw `dh_account_basic` appearing on the exact hot nodes (db-tikv-67/74) but prioritized a different SQL on a different node (db-tikv-30) as the "main hotspot". It failed to correlate hot-node evidence with the correct query.

## Workspace B Diagnosis (Correct)

> 3 messages, 1 human intervention. Auto-diagnosis via Lark bot.

The agent:
1. Same metrics collection — identified same hot nodes
2. TopSQL TiKV: directly found `dh_account_basic` as #1 by cpu_ms
3. Deep execution plan analysis: IndexFullScan on wrong index, actRows=52M vs estRows=690K (76x)
4. **Concluded**: Missing index on `last_update_timestamp` → IndexFullScan → hot TiKV
5. **P0 recommendation**: Add index on `(last_update_timestamp)` or `(last_update_timestamp, birthday_secret)`
6. Human feedback: "expensive query 是要找 tidb 对应日志的，并且去掉 analyze 相关 query"

**What WS-B got right that WS-A missed**:
- Correctly identified `dh_account_basic` as THE query causing the hot nodes
- Full execution plan analysis with IndexFullScan root cause
- Actionable P0: add index (matches L3 expert recommendation)

## Why the Conclusions Diverged

The divergence stems from **how the agent ranked TopSQL results**:

| TopSQL Metric | WS-A Prioritized | WS-B Prioritized |
|---|---|---|
| cpu_pct (%) on single node | `dh_message_setting` on db-tikv-30 (65.4%) | — |
| cpu_ms (total) across nodes | `dh_pay_order_success_user_summary` (3.2M cpu_ms) | `dh_account_basic` (highest cpu_ms) |
| Hot node correlation | Did NOT correlate db-tikv-67/74 → dh_account_basic | DID correlate |

**Root cause of inconsistency**: The agent does not have a clear rule to **cross-reference hot-node identity with per-query per-node TopSQL data**. WS-A found the right hot nodes but attributed them to the wrong SQL.

## Scores (Workspace A — the new/incorrect one)

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 2 | Identified hot TiKV nodes correctly but attributed to wrong SQL; missed the actual root cause query |
| Evidence Completeness | 3 | Same data sources as WS-B, but failed to deep-dive the correct SQL's execution plan |
| Causal Chain | 2 | Chain exists but connects wrong cause to correct symptoms |
| Actionability | 2 | P0 is ANALYZE (palliative), not add index (structural fix); would not resolve the issue |
| Efficiency | 3 | No human intervention needed, but reached wrong conclusion autonomously |
| **Overall** | **2.4** | |

## Human Guidance Log

No human intervention in WS-A. The diagnosis was fully autonomous and reached the wrong conclusion.

## Data Source Coverage (Workspace A)

- [x] Metrics (TiKV CPU, gRPC latency, EBS IOPS)
- [x] TopSQL TiKV (multiple digests analyzed)
- [ ] TopSQL TiDB
- [ ] SlowQuery logs / expensive query logs (MISSED — same gap as WS-B)
- [x] Execution plan analysis (but on wrong SQL — dh_pay_order_success_user_summary instead of dh_account_basic)
- [x] Logs (checked, none found)
- [x] Component health (no restarts)

## Lessons Learned

1. **Hot-node attribution is critical**: When TopSQL shows multiple hot SQL, the agent MUST cross-reference which SQL is running on the hot nodes (db-tikv-67/74 in this case), not just pick the highest cpu_pct on any node. The SQL with 65.4% on db-tikv-30 is a different hotspot from the one causing db-tikv-67/74 saturation.

2. **Diagnosis non-determinism is a trust issue**: Users expect the same input to produce the same diagnosis. When two workspace sessions reach different conclusions, it erodes trust in the system. The agent needs deterministic ranking rules for TopSQL analysis.

3. **Both workspaces missed slow query logs**: Neither WS-A nor WS-B queried TiDB slow query logs, which would have directly shown the expensive queries. This confirms ISS-0043 (expensive query alert should use slow query logs as primary data source).

4. **L3 expert confirms composite index**: The optimal fix is `(last_update_timestamp, birthday_secret)` composite index + SQL binding, not just a single-column index. WS-B was close (suggested both options) but WS-A missed entirely.