---
name: replayer
description: |
  Run postmortems and regressions on TCOC/O11Y incident diagnoses through Investibook.
  Postmortem: extract lessons from an existing workspace. Regression: replay a case and
  verify the agent still produces correct results. This is the only way to systematically
  evaluate and improve diagnosis skills — without it, skill changes are blind guesses.
---

# Auto Replayer Skill

Replay a TCOC/O11Y incident through Investibook and produce a postmortem or regression evaluation.

## Core Capabilities

1. **Postmortem** — Generate a postmortem report from an existing workspace's diagnosis. No re-run needed. **Default: Mode D** (most workspaces run on remote environments).

2. **Regression** — Run (or re-run) a diagnosis session on a known case, then evaluate quality against `expected.md`. Used to verify skill changes haven't caused regressions. **Default: Mode F** (interactive replay with automated follow-ups).

## Prerequisites

- Valid auth token (see [api-cookbook.md](references/api-cookbook.md) Phase 1)
- For local modes (A/B/C/F): Investibook backend running at `http://localhost:8000`
- For remote modes (D/E): `investibook` CLI with a configured profile

## Usage Modes

| Mode | Capability | Environment | Description |
|------|-----------|-------------|-------------|
| **D** | Postmortem | Remote | Generate postmortem from an existing remote workspace |
| **B** | Postmortem | Local | Generate postmortem from an existing local workspace |
| **F** | Regression | Local | Interactive regression with automated follow-ups (recommended) |
| **A** | Regression | Local | Full regression: create workspace + run diagnosis + evaluate |
| **C** | Regression | Local | Re-run diagnosis on existing workspace + evaluate |
| **E** | Regression | Remote | Full regression on remote environment |

### Mode D: Postmortem Remote Workspace (default for postmortem)

```
Postmortem remote workspace ws-xxx
```

Executes: Auth (remote) -> Extract results -> Evaluate -> Generate postmortem + expected.md

### Mode F: Interactive Regression (default for regression)

```
Regress O11Y-2328599
Regress workspace ws-xxx
```

Executes: Auth -> Create (if new) -> Interactive diagnosis -> Check -> Rubric verify -> Generate report

### Mode A: Full Regression (new case, non-interactive)

```
Replay O11Y-2328599
```

Executes: Auth -> Create workspace -> Run diagnosis -> Extract -> Evaluate -> Generate

### Mode B: Postmortem Local Workspace

```
Postmortem workspace ws-xxx
```

Executes: Auth (local) -> Extract results -> Evaluate -> Generate postmortem + expected.md

### Mode C: Re-run on Existing Workspace

```
Re-run diagnosis on workspace ws-xxx
```

Executes: Auth -> Clear messages -> Run diagnosis -> Extract -> Evaluate -> Generate

### Mode E: Full Regression on Remote

```
Replay O11Y-2328599 on remote
```

Same as Mode A but uses remote environment.

---

## Execution Phases

### Phase 0: Discover Workspaces from GitHub Issues (optional entry point)

Instead of being given a workspace ID directly, replayer can start from GitHub issues created by the `/issue` command. These issues have labels `diagnosis, auto-created` and contain workspace URLs in their body.

**Step 1: Fetch candidate issues**

```bash
curl -sS -G "https://api.github.com/search/issues" \
  --data-urlencode "q=repo:${REPO} is:issue is:open label:diagnosis label:auto-created -label:replayer-triaged" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Accept: application/vnd.github+json"
```

The `-label:replayer-triaged` filter excludes issues filed by Phase 8, preventing loops.

**Step 2: Extract workspace info from each issue**

Parse the issue body for:
- **Workspace ID**: from the `**Workspace**: [ws-xxx](url)` line
- **Environment**: `dev.investibook.com` → dev, `investibook.com` → prod
- **Ticket ID**: from the issue title `[Investibook] {ticket_id}: ...`
- **User feedback**: the text after `{ticket_id}:` in the title — this is the user's original complaint

**IMPORTANT — title analysis is always required.** The issue title contains the user's original problem report. This feedback must ALWAYS be triaged into `issues.yaml` (Phase 7), regardless of whether the workspace has conversation data. Title-derived issues and workspace-derived issues are **additive**:

- Workspace has conversation → postmortem (Phase 1-8) **AND** triage user feedback from title
- Workspace is empty or conversation unrelated → triage user feedback from title only (Phase 7 → Phase 8)

Example: issue #99 title says *"Get slow query plan details for top digest" 失败*. The postmortem found 3 issues from the conversation (ISS-0024/0025/0026), but the user's specific complaint about "plan details failing" should also be registered as an issue or matched against existing ones.

**Step 3: Decide whether to pull workspace messages**

Analyze the title content to decide the execution path:

- **Title describes a non-diagnosis problem** (UI bug, UX issue, feature request, config, display) → skip workspace messages entirely, go directly to Phase 7 with title-based triage. Examples: "前端刷新会导致chat message显示丢失", "Download 按钮无反应", "输入法回车问题"
- **Title describes a diagnosis quality problem** (wrong root cause, missed analysis, tool failure during diagnosis) → pull workspace messages and run full postmortem (Phase 1-8). Examples: "诊断不对", "Get slow query plan details 失败", "Incorrect OOM without verifying"

This avoids wasting time pulling and analyzing workspace conversations when the user's issue is clearly about the product itself, not diagnosis quality.

For each workspace that needs postmortem, execute Phase 1 → Phase 7 → Phase 8 as normal (Mode D for remote workspaces).

**Step 4: Link back to the source GitHub issue**

After Phase 8 files any new issues, add a comment on the original `/issue` GitHub issue summarizing the postmortem outcome:

```
Replayer postmortem completed.
- Diagnosis quality: {score}
- Issues found: ISS-NNNN (#url), ISS-MMMM (#url)
- User feedback triaged: ISS-XXXX (#url)
- Postmortem: cases/{case_id}/postmortem.md
```

> Phase 0 is optional. You can still start replayer with a workspace ID directly — Phase 0 just automates the discovery step.

### Phase 1: Authenticate

Two tokens are needed:

**1. Investibook API token** (`TOKEN` + `BASE_URL`) — for workspace/messages API:
- **Local**: Create dev token via Python one-liner (see [api-cookbook.md](references/api-cookbook.md) Phase 1)
- **Remote**: Use `investibook` CLI profile (`investibook profile use <name>`)

**2. GitHub token** (`GITHUB_TOKEN`) — for Phase 0 (read issues) and Phase 8 (create issues):
- **Primary**: read from local file `tools/replayer/.github_token`
- **Fallback**: env var `GITHUB_TOKEN` or `GH_TOKEN`
- **Last resort**: obtain from backend pod: `kubectl exec -n investibook deploy/backend -- printenv SKILL_GITHUB_TOKEN` (requires `aws sso login --profile sso`)
- Repo target: `GITHUB_ISSUE_REPO` defaults to `tidbcloud/investibook`

```bash
GITHUB_TOKEN=$(cat tools/replayer/.github_token 2>/dev/null || echo "${GITHUB_TOKEN:-$GH_TOKEN}")
```

If the GitHub token is unavailable after all attempts, skip Phase 0/8 and proceed with postmortem only. Do NOT ask the user for the token repeatedly.

### Phase 2: Create Workspace (Mode A/E only)

Skip for modes B/C/D/F-with-existing-ws. See [api-cookbook.md](references/api-cookbook.md) Phase 2.

Creates workspace, adds Jira source, waits for source preparation.

### Phase 3: Run Diagnosis

> Skip for Mode B and D (using existing conversation).
> For Mode C: clear messages first with `DELETE /api/workspace/$WS_ID/messages`.

#### Non-interactive (Mode A/C/E)

Single SSE stream call with `--max-time 600`. See [api-cookbook.md](references/api-cookbook.md) Phase 3.

#### Interactive (Mode F) — Recommended

Uses `replay.sh` as the execution layer. This is the recommended mode because it catches gaps early and sends targeted follow-ups.

**Step 1: Start diagnosis**

```bash
./tools/replayer/replay.sh <ws_id>
# Or for a new case:
./tools/replayer/replay.sh <ws_id> --create <issue_id>
```

**Step 2: Check diagnosis completeness**

```bash
./tools/replayer/replay.sh <ws_id> --check
```

Outputs structured JSON with `data_coverage`, `quality`, `gaps`, and `all_passed`.

**Step 3: Evaluate against expected.md (MANDATORY)**

> `--check` only performs generic quality checks (data coverage, has_root_cause, etc.).
> It does NOT verify case-specific correctness. You MUST perform this step.

Read `cases/{issue_id}/expected.md` and verify each scoring rubric item against the agent's diagnosis output:

1. Read the agent's final diagnosis text (from workspace messages API)
2. For **each** rubric checkbox in expected.md, check if the agent's output satisfies it
3. Record the result: `{passed}/{total}` rubric items

A regression **passes** only when ALL rubric items are satisfied. If any rubric item fails, it is a gap that may require a follow-up.

**Step 4: Send follow-up if needed (max 3 rounds)**

```bash
./tools/replayer/replay.sh <ws_id> --followup "Your follow-up message here"
```

The `--followup` mode sends the message, waits for completion, then runs `--check` again.

**Step 5: Repeat or proceed**

- All rubric items pass → Phase 4
- Max follow-ups (3) reached → Phase 4 (evaluate as-is, note failed items)
- Otherwise → back to Step 3

**Decision flow:**

```
start → wait → --check → rubric check → all pass? ── yes ──→ Phase 4
                                            │
                                            no
                                            │
                                   --followup (round 1)
                                            │
                                  --check → rubric check → all pass? ── yes ──→ Phase 4
                                                              │
                                                              no
                                                              │
                                                     --followup (round 2)
                                                              │
                                                    --check → rubric check → all pass? ── yes ──→ Phase 4
                                                                                │
                                                                                no
                                                                                │
                                                                       --followup (round 3, final)
                                                                                │
                                                                            Phase 4
```

**Follow-up guidelines:**

- Reference the specific gap from `--check` output or failed rubric item
- Be directive — tell the agent exactly what to do
- Include correct field/column names (e.g., "use `decoded_plan`, not `plan`")
- One gap per follow-up for clarity

### Phase 4: Extract Results

Read workspace messages and extract tool calls + diagnosis text. See [api-cookbook.md](references/api-cookbook.md) Phase 4 for the Python extraction script.

### Phase 5: Evaluate

Score the diagnosis using the evaluation framework. See [evaluation-guide.md](references/evaluation-guide.md) for:
- Data source coverage checklist
- Quality dimensions (1-5 scoring)
- Cross-reference validation
- Known deficiency checks (D1-D10)

### Phase 6: Generate Artifacts

Create two files in `cases/{issue_id}/`:

**`postmortem.md`** — Copy from `cases/TEMPLATE.md` and fill in:
- Issue summary, actual root cause, agent diagnosis summary
- Scores from Phase 5, human guidance log, skill deficiencies
- See [evaluation-guide.md](references/evaluation-guide.md) for template structure

**`expected.md`** — Ground truth for regression testing:
- Root cause, evidence chain, causality, recommendations
- **Scoring rubric** — checklist items the agent must satisfy to pass regression
- Base on **actual root cause** (Jira/oncall/expert), NOT agent output
- See [evaluation-guide.md](references/evaluation-guide.md) for template structure

**`cases.yaml`** — Add entry for the new case.

> `postmortem.md` and `expected.md` MUST be separate files. Regression primarily uses `expected.md`.

### Phase 7: Issue Triage

Scan the workspace conversation for **non-diagnosis issues** (tooling bugs, UX gaps, skill flow defects, wrong templates, etc.) and register them in [`issues/issues.yaml`](issues/issues.yaml).

**Step 1: Identify issues**

Review workspace messages for:
- Tool call failures (HTTP errors, wrong API versions, missing fields)
- Agent confusion or wasted tool calls (path discovery, wrong column names)
- Missing user feedback (silent retries, no error reporting)
- Reasoning gaps (premature conclusions, no cross-reference)
- Flow gaps (skipped steps, wrong routing)

**Step 2: Dedup against existing issues**

Read `tools/replayer/issues/issues.yaml` and check each discovered issue:
- Match by `title` similarity or `affects` (same skill + file)
- If match found: append this case ID to `found_in`, update `status` if needed
- If no match: assign the next `ISS-NNNN` ID
- **Anti-loop**: Skip any issue that has a `github_issue` field with label `replayer-triaged` — these are issues filed by Phase 8 itself. Do not re-analyze or re-file them.

**Step 3: Write new entries**

For each new issue, fill all fields:

```yaml
- id: ISS-NNNN
  title: Short description
  type: FLOW | TEMPLATE | KNOWLEDGE | REASONING | CROSS-REF | TOOLING | UX
  severity: P0-critical | P1-high | P2-medium | P3-low
  status: open
  affects:
    - skill: skill-name
      file: relative/path/to/file
  description: What goes wrong
  fix_plan: ISS-NNNN/plan.md
  found_in: [CASE-ID]
```

**Step 4: Write detailed plan (for non-trivial issues)**

For issues that need detailed context (tooling bugs, multi-step fixes), create `issues/ISS-NNNN/plan.md` with:
- Full background: what workspace, what the agent was doing, what went wrong
- Observed errors: exact error messages, tool call sequence, response snippets
- Root cause analysis: why it happened, what code is involved
- Fix plan: concrete steps with code snippets
- Verification: how to confirm the fix works

Set `fix_plan: ISS-NNNN/plan.md` in the yaml entry. The goal is that a fixing agent can read the plan and implement the fix **without needing to re-pull the original workspace conversation**.

> Issues that are already tracked in postmortem.md "Skill Deficiencies" sections should also be in `issues/issues.yaml`. The yaml is the **single source of truth** for cross-case dedup.

### Phase 8: File Issues to GitHub

After Phase 7 writes issues to `issues/issues.yaml`, file them as GitHub issues so humans can review, prioritize, and approve fixes.

**Scope**: Only non-diagnosis issues (type: TOOLING, FLOW, UX, TEMPLATE, etc.) that have a `fix_plan` with actionable steps. Skip diagnosis-quality issues (REASONING, KNOWLEDGE, CROSS-REF) unless they have a concrete code fix.

**Step 1: Select unfiled issues**

Read `issues/issues.yaml`. Select entries where:
- `status: open`
- `github_issue` field is absent (not yet filed)
- Has a `fix_plan` pointing to a `plan.md` file

**Step 2: Build GitHub issue content**

For each selected issue, create a GitHub issue with:

- **Title**: `[ISS-NNNN] {title}`
- **Labels**: `replayer-triaged`, severity (e.g. `P1-high`)
- **Body**:
  ```
  ## {title}

  | Field | Value |
  |-------|-------|
  | ID | ISS-NNNN |
  | Type | {type} |
  | Severity | {severity} |
  | Affects | {skill} — {file} |
  | Found in | {found_in cases} |
  | Workspace | {link to workspace where issue was discovered} |

  ## Description

  {description from issues.yaml}

  ## Fix Plan

  {full content of ISS-NNNN/plan.md}

  ---
  _Auto-triaged by replayer Phase 8. Label: `replayer-triaged`._
  ```

**Step 3: Check remote for duplicates**

Before creating, search GitHub for an existing issue with the same `[ISS-NNNN]` prefix:

```bash
curl -sS -G "https://api.github.com/search/issues" \
  --data-urlencode "q=repo:${REPO} is:issue [ISS-NNNN] in:title" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Accept: application/vnd.github+json"
```

If `total_count > 0`, the issue already exists remotely — write its URL back to `github_issue` in yaml and skip creation. This handles cases where yaml was not committed after a previous Phase 8 run.

**Step 4: Create via GitHub API**

```bash
TOKEN="${GITHUB_TOKEN:-$GH_TOKEN}"
REPO="${GITHUB_ISSUE_REPO:-tidbcloud/investibook}"

curl -sS -X POST "https://api.github.com/repos/${REPO}/issues" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  -d '{
    "title": "[ISS-NNNN] ...",
    "body": "...",
    "labels": ["replayer-triaged", "P1-high"]
  }'
```

**Step 5: Write back to issues.yaml**

On success, add `github_issue: <url>` to the yaml entry. This prevents duplicate filing on subsequent runs.

```yaml
- id: ISS-0021
  title: Jira search-issues API 返回 HTTP 410（v2 已废弃）
  github_issue: https://github.com/tidbcloud/investibook/issues/42
  # ... rest of fields
```

**Anti-loop rule**: The `replayer-triaged` label distinguishes replayer-filed issues from user-filed `/issue` issues (which use `diagnosis, auto-created` labels). Replayer must never pull `replayer-triaged` issues back into its own analysis pipeline.

---

## CLI Reference: replay.sh

| Command | Description |
|---------|-------------|
| `replay.sh <ws_id>` | Rerun: clear messages, send diagnosis prompt, wait, evaluate |
| `replay.sh <ws_id> --eval-only` | Evaluate existing messages (human-readable) |
| `replay.sh <ws_id> --check` | Structured checklist (JSON for agent consumption) |
| `replay.sh <ws_id> --followup "<msg>"` | Send follow-up, wait, then --check |
| `replay.sh <ws_id> --create <issue_id>` | Create workspace + add source + prepare + run |
| `replay.sh --profile dev <ws_id> --check` | Use CLI profile for remote environment |

### --check output format

```json
{
  "workspace_id": "ws-xxx",
  "message_count": 5,
  "human_interventions": 1,
  "tool_calls": {"total": 36, "completed": 36, "stuck": 0},
  "data_coverage": {
    "metrics": true, "topsql_tidb": true, "topsql_tikv": true,
    "slowquery": true, "decoded_plan": true, "logs": true, "events": true
  },
  "quality": {
    "has_root_cause": true, "has_evidence": true, "has_recommendation": true,
    "has_specific_table": true, "has_complete_sql": true, "no_stuck_tools": true
  },
  "gaps": [],
  "all_passed": true,
  "final_text_length": 4521
}
```

### --profile flag

Reads `api` (base URL) and `token` from `~/.investibook/config.json` profiles. Enables `--check` and `--eval-only` on remote workspaces without a local backend.

---

## Reference Files

Load these when you need detailed code or templates:

| File | Contents | When to load |
|------|----------|--------------|
| [api-cookbook.md](references/api-cookbook.md) | Auth, workspace creation, diagnosis, extraction API calls | Phase 1-4 execution |
| [evaluation-guide.md](references/evaluation-guide.md) | Scoring templates, checklists, postmortem/expected.md structure | Phase 5-6 evaluation |
| [issues/issues.yaml](issues/issues.yaml) | Global issue registry with dedup, fix plans, cross-case tracking | Phase 7 issue triage |
| [troubleshooting.md](references/troubleshooting.md) | Auto-chat hangs, "agent is busy" errors, tips | When things go wrong |

---

## Troubleshooting

See [troubleshooting.md](references/troubleshooting.md) for common issues:
- Auto-chat hangs (status="running" indefinitely)
- "Agent is busy with a previous request"
- Recovery steps (stop auto-chat, kill process, restart backend, new workspace)