# Regression Registry

This directory contains the formal regression registry used by the current eval
runner.

- `cases/` contains case-owned runtime input plus judge assets
- `suites/` contains suite definitions
- `baselines/` contains repo-versioned active baseline manifests per suite

This registry is intentionally separate from legacy workspace postmortem and
replayer metadata.

## Execution Model

Active regression execution is `live eval`:

- fresh workspace per case
- blocking ACP execution
- live Clinic/Jira/ACP calls

Historical frozen replay paths are not the active registry contract for this
tree.

## Case Layout

Each case lives under:

```text
evaluation/regression/cases/<case_id>/
  case.yaml
  judge.yaml
  expected.md
```

Roles:

- `case.yaml`
  - run-visible case input
  - prompt, environment, time window, execution config
- `judge.yaml`
  - judge-only policy and ground truth
- `expected.md`
  - human-readable expected diagnosis notes

## Suite Layout

Suites live under:

```text
evaluation/regression/suites/<suite_id>.yaml
```

Repo baselines live under:

```text
evaluation/regression/baselines/<suite_id>.yaml
```

A suite defines:

- suite id and title
- active/inactive status
- baseline eligibility
- included case ids
- execution policy metadata

A baseline manifest records:

- the promoted source `eval_run`
- the exact `suite_version`
- per-case `case_version` and verdict snapshot
- operator metadata such as `created_by` and `notes`

Promoting a baseline also exports a repo-owned runtime snapshot under:

```text
evaluation/regression/baselines/runs/<eval_run_id>/
```

The export is intentionally filtered. It keeps comparison-relevant text/json
artifacts and excludes transient local runtime state.

## Naming and Sanitization

Case ids and directory names must be anonymized.

- Do not use Jira issue ids in case ids or paths
- Do not encode the final diagnosis or root cause in case ids or paths
- Prefer symptom-oriented names that describe the observed phenomenon

Examples:

- `component-restart-during-incident-window`
- `service-degradation-during-incident-window`
- `tail-latency-surge-during-incident-window`

## Prompt Contract

`case.yaml.prompt.text` should follow a live-like no-source contract.

- Put the user-visible diagnosis request directly in `prompt.text`
- Include the minimum execution context in the message itself:
  - `cluster_id`
  - diagnosis time window
  - observed symptom
  - requested task
- Do not rely on the runner to prepend another cluster/time template
- Do not include Jira ids, known root causes, GitHub issue numbers, fix versions,
  or other answer-revealing hints

## Judge Contract

The current judge is layered. Case-level policy lives in `judge.yaml`, not in a
case-specific hardcoded branch.

Current bucket families:

- `contract`
- `execution_contract`
- `workflow`
- `evidence_collection`
- `critical_evidence`
- `evidence_routing`
- `diagnosis`

See `evaluation/judge-pipeline-explainer.zh.md` for the implementation details.

Common judge policy sections include:

- `judge.contract`
- `judge.execution_contract`
- `judge.workflow`
- `judge.evidence_collection`
- `judge.critical_evidence`
- `judge.evidence_routing`
- `judge.diagnosis`

## Verdict Precedence

The current overall verdict precedence is:

| Precedence | Condition | Overall Verdict | Primary Failure Code |
| --- | --- | --- | --- |
| 1 | `contract = fail` | `invalid` | `contract_failed` |
| 2 | `execution_contract = fail` | `invalid` | `execution_contract_failed` |
| 3 | `workflow in {partial, fail}` | `invalid` | `workflow_incomplete` |
| 4 | `evidence_collection = fail` | `invalid` | `evidence_collection_incomplete` |
| 5 | `evidence_collection = partial` | `fail` | `evidence_collection_partial` |
| 6 | `critical_evidence = fail` | `fail` | `critical_evidence_missing` |
| 7 | `critical_evidence = partial` | `fail` | `critical_evidence_partial` |
| 8 | `evidence_routing = fail` | `fail` | `evidence_routing_failed` |
| 9 | `evidence_routing = partial` | `fail` | `evidence_routing_partial` |
| 10 | `diagnosis = fail` | `fail` | `diagnosis_critical_gate_failed` |
| 11 | all buckets `pass` | `pass` | `null` |

Notes:

- Evaluate rows from top to bottom; the first matching row wins
- `partial` must never be promoted to overall `pass`
- `invalid` is reserved for non-comparable runs
- `fail` is reserved for comparable runs that miss the required bar

## Validation and Inspection

Useful entrypoints:

- `backend/investibook/scripts/eval.py`
- `backend/investibook/scripts/inspect_eval_suite.py`

Common commands:

```bash
cd /Users/huyibin/Documents/dev/investibook/backend
uv run python investibook/scripts/eval.py list
uv run python investibook/scripts/eval.py --suite core-daily-demo --execution-mode dry-run
uv run python investibook/scripts/inspect_eval_suite.py
```

The current add/update authoring workflow is also documented in:

- `evaluation/regression/add-case-skill/SKILL.md`