# Evaluation Guide

> Reference material for the replayer skill. Load this when scoring a diagnosis.

## Data Source Coverage Checklist

Check which data sources the agent queried (from Phase 4 tool list):

- [ ] Metrics (EBS IOPS, CPU, latency, DB time)
- [ ] TopSQL TiDB component
- [ ] TopSQL TiKV component
- [ ] SlowQuery logs
- [ ] Loki error logs
- [ ] Cluster events (Activity Hub)

## Quality Dimensions (score 1-5)

| Dimension | Score | Notes |
|-----------|-------|-------|
| Root Cause Accuracy | {1-5} | Did the agent identify the correct primary root cause? |
| Evidence Completeness | {1-5} | Were all relevant data sources queried? |
| Causal Chain | {1-5} | Is there a clear A->B->C chain? |
| Actionability | {1-5} | Are recommendations specific (SQL/config)? |
| Efficiency | {1-5} | {N} tool calls, {N} human interventions |
| **Overall** | **{avg}** | |

## Cross-Reference Validation

A diagnosis is only valid with evidence from 2+ independent sources:

- [ ] Metrics evidence supports root cause
- [ ] Query evidence (TopSQL/SlowQuery) confirms which SQL
- [ ] Log/Event evidence supports OR rules out alternatives

## Known Deficiency Checks (D1-D10)

Verify previously-identified issues are resolved:

| Check | Pass/Fail | Notes |
|-------|-----------|-------|
| D1: TopSQL both components queried | | |
| D2: Correct column names (timestamps, not timestamp) | | |
| D3: Hotspot instance identified if present | | |
| D5: Cross-reference validation performed | | |
| D8: ANALYZE/DDL not blamed as primary cause | | |
| D10: Structured output format | | |

## Postmortem Template Structure

When generating a postmortem, use this structure (see `cases/TEMPLATE.md` for the full template):

1. **Issue Summary** — table with issue ID, alert type, cluster ID, version, time window
2. **Actual Root Cause** — from Jira comments, oncall handoff, or expert analysis (NOT agent output)
3. **Agent Diagnosis Summary** — what the agent concluded, what it missed
4. **Scores** — the quality dimension table filled in
5. **Human Guidance Log** — each intervention with round, user said, agent's mistake, correct behavior, deficiency type
6. **Skill Deficiencies** — each with type, skill, current/expected behavior, suggested fix, priority
7. **Key Queries** — the actual working API calls that produced results
8. **Lessons Learned** — key takeaways for skill improvement

## Expected.md Template Structure

```markdown
# Expected Diagnosis: {ISSUE_ID}

## Root Cause
{2-3 sentences describing the confirmed root cause}

## Evidence Chain
1. {data source + what it shows}
2. ...

## Causality
{A -> B -> C chain from root cause to observed symptoms}

## Severity
{Critical / High / Medium / Low}

## Recommendations
1. {recommendation}
2. ...

## Scoring Rubric
> Checklist for regression testing. The Agent must satisfy these to pass.
- [ ] {rubric item 1}
- [ ] {rubric item 2}
```

The `expected.md` is the primary input for regression evaluation. Keep it concise and focused on what the agent should find. Base it on **actual root cause** (from Jira, oncall, or expert), NOT the agent's output.

## cases.yaml Entry

```yaml
  - issue_id: "{ISSUE_ID}"
    prompt: "Use tidb-clinic-diagnosis to analyze this alert. Provide root cause, evidence chain, and recommendations."
    cluster_type: "{dedicated/premium/starter}"
    tidb_version: "{version}"
    tags: ["{tag1}", "{tag2}"]
    expected_root_cause: "{one-line summary}"
    date_added: "{YYYY-MM-DD}"
    reviewer: "{your name}"
```