# TCOC Postmortem SOP

> Standard operating procedure for reviewing TCOC / O11Y tickets with Investibook.
> Each postmortem produces a structured evaluation stored under `tools/replayer/cases/{issue_id}/`.

## Purpose

1. Evaluate Investibook's diagnostic quality against real incidents
2. Identify concrete skill deficiencies with actionable improvement items
3. Build the Auto Replayer validation set (same `expected.md` serves as regression baseline)

## Environment Setup

### Prerequisites

- Investibook repository cloned
- Python 3.12+, uv, Node.js 18+, pnpm installed
- PingCAP AWS SSO account

### 1. AWS Bedrock Configuration

Investibook uses OpenCode + AWS Bedrock to call Claude models. Configure the Bedrock bearer token:

```bash
# Option 1: Environment variables (recommended)
export AWS_BEARER_TOKEN_BEDROCK="<your-token>"
export AWS_REGION="us-east-1"

# Option 2: AWS SSO login for temporary credentials
aws sso login --profile sso
```

OpenCode model config lives at `~/.config/opencode/opencode.jsonc`, defaulting to Claude Opus 4.6 via Bedrock:

```jsonc
{
  "model": "chat/arn:aws:bedrock:us-east-1:...:application-inference-profile/...",
  "provider": {
    "chat": {
      "npm": "@ai-sdk/amazon-bedrock",
      "options": { "region": "us-east-1" }
    }
  }
}
```

### 2. Backend Environment Variables

```bash
cd backend

# First time: create .env.local from template
make setup-env

# Edit .env.local:
#   AUTH0_CLIENT_ID    — frontend SSO (public value, see .env.example)
#   AUTH0_DOMAIN       — backend SSO token verification
#   ADMIN_EMAILS       — admin email allowlist

# Edit .env (loaded by python-dotenv automatically):
#   INVESTIBOOK_JIRA_BASE_URL
#   INVESTIBOOK_JIRA_USERNAME
#   INVESTIBOOK_JIRA_API_TOKEN
#   INVESTIBOOK_CLINIC_ENDPOINT
#   INVESTIBOOK_CLINIC_API_KEY
```

`INVESTIBOOK_CLINIC_API_KEY` is injected as `CLINIC_API_KEY` into the agent subprocess at startup. Skills reference it via `${CLINIC_API_KEY}` in curl commands.

### 3. Install Dependencies and Start

```bash
# From project root
make sync-skills   # Sync skill files
make install       # Install frontend + backend deps

# Terminal 1: Start backend (port 8000)
make api

# Terminal 2: Start frontend (port 3000)
make web
```

### 4. Verify

```bash
# Backend health check
curl -s http://localhost:8000/api/config | python3 -m json.tool

# If no Auth0, create a local dev user
cd backend && .venv/bin/python3 -c "
from investibook.db.queries import _db_init, _db_get_or_create_user, _db_create_user_session, _hash_password, _db_connect, _db_lock
_db_init()
user = _db_get_or_create_user('dev@localhost')
pw_hash = _hash_password('dev123')
with _db_lock:
    conn = _db_connect()
    conn.execute('UPDATE users SET password_hash = ? WHERE id = ?', (pw_hash, user['id']))
    conn.commit()
token = _db_create_user_session(user['id'])
print(f'Login: dev@localhost / dev123')
print(f'Token: {token}')
"
```

## Workflow

### Step 0: Use the Auto Replayer Skill (recommended)

If you are working in Claude Code, use `tools/replayer/SKILL.md` to automate Steps 1-3:

```
Use the auto-replayer skill to run a postmortem on O11Y-2328599
```

The skill will: create workspace -> add Jira source -> wait for data -> send diagnostic prompt -> wait for completion -> extract tool calls and diagnosis text -> generate initial evaluation from template.

Manual guidance (Step 2) and subsequent steps still require human involvement.

### Step 1: Run the diagnosis

**Option A: Web UI**

Open `http://localhost:3000`, log in, create a workspace, add a Jira source, and send:

```
Use tidb-clinic-diagnosis to analyze this alert. Provide root cause, evidence chain, and recommendations.
```

**Option B: API (scriptable)**

```bash
TOKEN="<your-token>"  # From login endpoint or dev user script above

# 1. Create workspace
WS_ID=$(curl -s -X POST http://localhost:8000/api/workspaces \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "O11Y-2328599 postmortem"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")

# 2. Add Jira source
curl -s -X POST "http://localhost:8000/api/workspace/$WS_ID/sources" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"type":"jira_issue","name":"O11Y-2328599","issueId":"O11Y-2328599","status":"loading"}'

# 3. Prepare data (fetch Jira + cluster metadata)
TASK_ID=$(curl -s -X POST "http://localhost:8000/api/sources/prepare" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"workspace_id\":\"$WS_ID\",\"issue_id\":\"O11Y-2328599\"}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['task_id'])")

# 4. Wait for preparation
while true; do
  STATUS=$(curl -s "http://localhost:8000/api/sources/prepare/$TASK_ID" \
    -H "Authorization: Bearer $TOKEN" \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['status'])")
  [ "$STATUS" = "completed" ] && break
  [ "$STATUS" = "error" ] && echo "Prepare failed" && exit 1
  sleep 3
done

# 5. Send diagnostic prompt (SSE stream, 10 min max)
curl -s -N --max-time 600 \
  "http://localhost:8000/api/chat/stream?message=Use%20tidb-clinic-diagnosis%20to%20analyze%20this%20alert.&workspace_id=$WS_ID&token=$TOKEN" \
  > /tmp/replay-$WS_ID.txt

# 6. Read full messages
curl -s "http://localhost:8000/api/workspace/$WS_ID/messages" \
  -H "Authorization: Bearer $TOKEN" > /tmp/messages-$WS_ID.json
```

Keep the workspace shared so others can review.

### Step 2: Guide the Agent through deep analysis

The Agent's first-round diagnosis is usually incomplete. Intervene in the conversation to guide it toward a thorough diagnosis:

1. **Check data source coverage**: Did the Agent query Metrics, TopSQL, SlowQuery, and Loki logs? Explicitly prompt it to query any missing sources.
2. **Correct misattribution**: If the Agent attributes root cause to a secondary factor (e.g., ANALYZE), steer it toward the primary workload driver.
3. **Require cross-validation**: Conclusions from a single data source are unreliable -- prompt the Agent to verify with other sources.
4. **Push for quantitative analysis**: Guide the Agent from qualitative descriptions ("uses full table scan") to quantitative analysis ("table C took 15 minutes, 83% of total time").
5. **Confirm actionable recommendations**: The final diagnosis must include specific optimization advice (e.g., which index to add, which config to change).

> All human interventions in this step must be documented in detail in Step 4, as input for skill improvements.

### Step 3: Evaluate final diagnosis

Compare the Agent's guided final diagnosis against the actual resolution (from Jira comments, oncall handoff, or your own expert knowledge). Score each dimension 1-5:

| Dimension | 1 (Fail) | 3 (Partial) | 5 (Excellent) |
|-----------|----------|-------------|---------------|
| **Root Cause Accuracy** | Wrong root cause | Identified a contributing factor but missed the primary cause | Correctly identified the primary root cause |
| **Evidence Completeness** | Missing critical data sources (e.g., no TopSQL, no SlowQuery) | Collected most data but missed one key source | All relevant data sources queried and analyzed |
| **Causal Chain** | No causal reasoning; just listed findings | Some causal links but gaps in the chain | Clear A->B->C chain from infra to user-visible symptom |
| **Actionability** | No actionable recommendations | Generic recommendations (e.g., "add an index") | Specific, implementable recommendations with SQL/config |
| **Efficiency** | >30 tool calls, many retries/dead ends | 15-30 tool calls, some wasted effort | <15 tool calls, focused and systematic |

### Step 4: Document human guidance

List every point from Step 2 where you had to manually steer the Agent. For each intervention:

- **What you said** (the prompt)
- **What Agent was doing wrong** (the mistake)
- **What it should have done** (the correct behavior)
- **Which skill is responsible** (tidb-clinic-diagnosis / clinic-api / system prompt / other)

### Step 5: Identify skill deficiencies

Classify each deficiency:

| Type | Description | Example |
|------|-------------|---------|
| **FLOW** | Missing or incorrect step in the diagnostic workflow | TopSQL not queried as part of standard flow |
| **TEMPLATE** | Wrong or incomplete query template | TopSQL SQL using non-existent columns |
| **KNOWLEDGE** | Missing schema/API knowledge | Not knowing `timestamps` field in TopSQL table |
| **REASONING** | Premature convergence or wrong analytical pattern | Focusing on table A when table C had 83% of execution time |
| **CROSS-REF** | Failed to cross-reference data sources | Had TopSQL data but didn't correlate with SlowQuery |

### Step 6: Deduplicate against the deficiency backlog

Before writing the postmortem, check the **Skill Deficiency Backlog** table in `docs/plans/2026-03-04-backlog.md` (under Milestone #2) for existing deficiencies. For each deficiency found in Step 5:

- **If it already exists in the backlog**: Do NOT create a duplicate entry. Instead, add this case's issue_id to the existing item's `Reproduced in` list, confirming it as a systemic issue.
- **If it is new**: Assign it the next available ID (e.g., `D10`) and add it to the backlog.

This prevents multiple reviewers from filing overlapping improvement items and ensures cross-case visibility.

### Step 7: Write the postmortem file

Create `tools/replayer/cases/{issue_id}/postmortem.md` using the template below.
Create `tools/replayer/cases/{issue_id}/expected.md` with the correct diagnosis (for replayer baseline).
Add an entry to `tools/replayer/cases.yaml`.

### Step 8: File skill improvement PRs

For each deficiency, either:
- Fix the skill directly and reference the postmortem
- Or create a GitHub issue tagged `skill-improvement` with the deficiency details

---

## Postmortem Template

The canonical template is at `tools/replayer/cases/TEMPLATE.md`. Copy it to create new postmortems.

```
tools/replayer/cases/{issue_id}/
  postmortem.md   # Full analysis (from template)
  expected.md     # Correct diagnosis (replayer baseline)
```

## cases.yaml Format

```yaml
cases:
  - issue_id: "O11Y-2328599"
    prompt: "Use tidb-clinic-diagnosis to analyze this alert. Provide root cause, evidence chain, and recommendations."
    cluster_type: "dedicated"
    tags: ["expensive-query", "slow-query", "hotspot"]
    date_added: "2026-04-07"
```