# Postmortem: O11Y-2392287

> Workspace: `ws-1776139348995-a7101b3a3`
> Date: 2026-04-16
> Reviewer: claude-code

## Issue Summary

| Field | Value |
|-------|-------|
| Issue ID | O11Y-2392287 |
| Alert Type | TiDBExpensiveQueryCntGt100For1m |
| Cluster ID | 10754652599988043412 |
| Cluster Type | dedicated |
| TiDB Version | v8.5.4 |
| Time Window | 2026-04-13 ~20:16 UTC (alert) / 2026-04-14 04:17 UTC (this issue) |

**Issue Description**: TiDBExpensiveQueryCntGt100For1m alert fired on a dedicated cluster (prod-us-saas-gosaas-region-tidb-v85-03). Alert auto-recovered; Jira status "Created" with no explicit root cause in ticket.

## Actual Root Cause

> Human-confirmed root cause from Jira resolution, oncall handoff, or expert analysis.

Not confirmed in Jira — ticket status is "Created" with note "alert has auto-recovered". Based on the agent's correlation analysis: expensive queries (>100 per minute) caused TiKV gRPC P99 latency spikes (db-tikv-28: 9.2s, db-tikv-6: 2.9s), coinciding with NetworkLagIncrease alerts. The TiDBExpensiveQueryCntGt100For1m fired twice within ~1 hour (19:26 and 20:16 UTC on 2026-04-13), indicating a sustained expensive query workload. TiFlash jitter and TiKV high memory were pre-existing background conditions.

## Agent Diagnosis Summary

> What the Agent concluded (final answer from unguided first attempt).

The workspace used **auto-diagnosis** (triggered automatically when Jira issue data was loaded, without user input). The auto-diagnosis ran `collect_evidence.py` and loaded `tidb-knowledge-base` + `tidb-clinic-diagnosis` skills to collect evidence. However, the auto-diagnosis results are **not captured in the messages API** — they were consumed by the system silently.

The only visible agent outputs in the conversation are:
1. **Issue info card** (message 1): parsed Jira issue details, cluster metadata
2. **Alert correlation** (message 4): searched for other alerts in the same time window, identified 19 alerts, correlated NetworkLagIncrease + TiFlash jitter + TiKV memory patterns with the expensive query event
3. **Problem analysis** (message 6): identified 3 tooling issues encountered during the session

The agent **did not produce a formal diagnosis with root cause, evidence chain, and recommendations** in the visible conversation. The auto-diagnosis ran in the background but its output is not accessible via the messages API.

## Scores

| Dimension | Score (1-5) | Notes |
|-----------|-------------|-------|
| Root Cause Accuracy | 1 | No formal root cause diagnosis in visible messages; auto-diagnosis not captured |
| Evidence Completeness | 1 | No TopSQL, SlowQuery, metrics, logs, or events queried in visible conversation |
| Causal Chain | 2 | Alert correlation in msg 4 shows some causal reasoning (expensive query -> TiKV latency -> network lag) |
| Actionability | 1 | No specific recommendations provided |
| Efficiency | 2 | 4 tool calls in visible session; 1 failed (Jira API v2 410); auto-diagnosis ran but wasn't captured |
| **Overall** | **1.4** | |

## Human Guidance Log

> Every point where the reviewer had to manually steer the Agent.

### Intervention 1: Request alert correlation

- **Round**: #2 (message 2)
- **User said**: "这个集群在这个时间点还有什么别的告警么？"
- **Agent's mistake**: Auto-diagnosis ran but user had no visibility; agent didn't proactively correlate alerts
- **Correct behavior**: After auto-diagnosis, agent should present findings including related alerts
- **Skill**: `tidb-clinic-diagnosis`
- **Deficiency type**: FLOW

### Intervention 2: Nudge for delayed response

- **Round**: #3 (message 3)
- **User said**: "我的问题结果呢？"
- **Agent's mistake**: Jira API v2 failed silently, no progress feedback to user
- **Correct behavior**: Should immediately inform user of tool failure and retry strategy
- **Skill**: `jira`
- **Deficiency type**: UX

### Intervention 3: Request problem analysis

- **Round**: #5 (message 5)
- **User said**: "分析下这一次分析里碰到的问题？"
- **Agent's mistake**: N/A — user proactively asked for meta-analysis
- **Correct behavior**: Agent should self-report issues without being asked
- **Skill**: general
- **Deficiency type**: UX

### Intervention 4: Clarify auto-diagnosis trigger

- **Round**: #7 (message 7)
- **User said**: "第一次是系统自带有 user input 么？"
- **Agent's mistake**: Auto-diagnosed without user consent
- **Correct behavior**: Should not auto-trigger diagnosis without explicit user request
- **Skill**: `investibook-core`
- **Deficiency type**: FLOW

### Intervention 5: Disable auto-diagnosis

- **Round**: #9 (message 9)
- **User said**: "不需要自动触发 诊断行为"
- **Agent's mistake**: Auto-diagnosis was the default behavior
- **Correct behavior**: Diagnosis should only start on explicit user request
- **Skill**: `investibook-core`
- **Deficiency type**: FLOW

## Skill Deficiencies

> Actionable improvement items extracted from the guidance log.

### SD-1: Jira API v2 endpoint deprecated (HTTP 410)

- **Type**: TOOLING
- **Skill**: `jira`
- **Current behavior**: `jira_api.py` uses `GET /rest/api/2/search` which returns HTTP 410
- **Expected behavior**: Should use `POST /rest/api/3/search/jql` with `fields` array in body
- **Suggested fix**: Update `.opencode/skills/builtin/jira/scripts/jira_api.py` `search-issues` endpoint from v2 to v3; pass `fields` in POST body
- **Priority**: P0-critical
- **Status**: open

### SD-2: Jira v3 response field parsing failure

- **Type**: TOOLING
- **Skill**: `jira`
- **Current behavior**: v3 response returns only `id` in issues array without explicit `fields` parameter; agent gets `?` for all fields
- **Expected behavior**: Request should include `"fields": ["key","summary","created","status",...]` in POST body
- **Suggested fix**: Ensure `--fields` parameter is passed in both query params and POST body `fields` array
- **Priority**: P0-critical
- **Status**: open

### SD-3: Auto-diagnosis triggers without user consent

- **Type**: FLOW
- **Skill**: `investibook-core`
- **Current behavior**: When Jira issue data is loaded, diagnosis is auto-triggered without user prompt
- **Expected behavior**: Diagnosis should only begin when user explicitly requests it
- **Suggested fix**: Remove or gate the auto-diagnosis trigger behind a user preference / explicit opt-in
- **Priority**: P1-high
- **Status**: open

### SD-4: No progress feedback on tool failure

- **Type**: UX
- **Skill**: `jira`
- **Current behavior**: When Jira search-issues fails (HTTP 410), agent silently retries with alternative approach; user waits without feedback
- **Expected behavior**: Agent should immediately inform user of failure and explain retry strategy
- **Suggested fix**: Add error-reporting step in tool failure handler that sends user-facing message before retrying
- **Priority**: P2-medium
- **Status**: open

### SD-5: Auto-diagnosis output not captured in messages API

- **Type**: FLOW
- **Skill**: `investibook-core`
- **Current behavior**: Auto-diagnosis runs via `collect_evidence.py` but results are not visible in `/api/workspace/{id}/messages`
- **Expected behavior**: All diagnosis output (including auto-triggered) should be captured in messages for auditability
- **Suggested fix**: Ensure auto-diagnosis writes its output as assistant messages to the workspace conversation
- **Priority**: P1-high
- **Status**: open

## Key Queries

> Correct queries that the Agent should have used (for skill template reference).

### Jira v3 search (working)

```bash
curl -s -X POST "https://YOUR_JIRA/rest/api/3/search/jql" \
  -H "Authorization: Bearer $JIRA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "jql": "project = O11Y AND text ~ \"10754652599988043412\" ORDER BY created DESC",
    "maxResults": 50,
    "fields": ["key", "summary", "created", "status", "priority", "components"]
  }'
```

## Lessons Learned

> General insights beyond specific skill fixes.

- **Auto-diagnosis without consent is a UX anti-pattern**: Users expect to control when diagnosis starts. Auto-triggering creates confusion about what the agent did and why.
- **Silent tool failures degrade trust**: When a core tool (Jira search) fails, the agent must immediately communicate the failure and its workaround plan. Silent retries make users feel ignored.
- **Auto-diagnosis output must be auditable**: If the system auto-runs a diagnosis, the full output must be captured in the conversation messages. Otherwise, postmortem and regression workflows cannot evaluate the diagnosis quality.
- **Jira API migration is overdue**: Atlassian deprecated API v2; the Jira skill must migrate to v3 to remain functional. This is a P0 blocking issue.
- **This workspace is not suitable for regression testing**: Since the auto-diagnosis output is not captured in messages, the workspace cannot be evaluated against expected.md. A re-run with explicit user-triggered diagnosis would be needed for regression.