# Evaluation

This directory is the current entrypoint for Investibook evaluation runtime and
operator-facing usage.

For related documents:

- Current implementation overview: `evaluation/current-implementation.zh.md`
- Evidence-only scoring design: `evaluation/evidence-only-evaluation.zh.md`
- Registry and case authoring: `evaluation/regression/README.md`
- Judge implementation: `evaluation/judge-pipeline-explainer.zh.md`

## Canonical Entrypoint

The canonical local CLI entrypoint is:

- `backend/investibook/scripts/eval.py`

This wrapper bootstraps local runtime defaults before importing the eval
modules. It is the recommended entrypoint for local list/run usage.

Other wrappers such as `backend/investibook/scripts/run_eval_suite.py` and
`backend/investibook/scripts/inspect_eval_suite.py` are thin adapters over the
core modules and are not the primary operator entrypoint.

## Registry Layout

- Suites live under `evaluation/regression/suites/`
- Cases live under `evaluation/regression/cases/`
- Repo-versioned baselines live under `evaluation/regression/baselines/`

To inspect the currently active suites, use:

```bash
cd /Users/huyibin/Documents/dev/investibook/backend
uv run python investibook/scripts/eval.py list
```

The repo currently includes the active suite `core-daily-demo`:

- Suite definition: `evaluation/regression/suites/core-daily-demo.yaml`

## Common Usage

Run commands from the `backend/` directory.

List suites:

```bash
cd /Users/huyibin/Documents/dev/investibook/backend
uv run python investibook/scripts/eval.py list
```

Run a suite:

```bash
cd /Users/huyibin/Documents/dev/investibook/backend
uv run python investibook/scripts/eval.py --suite core-daily-demo
```

Run selected cases:

```bash
cd /Users/huyibin/Documents/dev/investibook/backend
uv run python investibook/scripts/eval.py \
  --suite core-daily-demo \
  --case-id service-degradation-during-incident-window \
  --case-id panic-in-recoverable-goroutine
```

Promote a completed run into the repo baseline manifest:

```bash
cd /Users/huyibin/Documents/dev/investibook/backend
uv run python investibook/scripts/eval.py \
  promote \
  --run-id evalrun-xxxxxxxx \
  --created-by you@example.com \
  --notes "accept current behavior"
```

This promotion also exports a repo-committable runtime snapshot under:

- `evaluation/regression/baselines/runs/<eval_run_id>/`

Run dry-run skeleton only:

```bash
cd /Users/huyibin/Documents/dev/investibook/backend
uv run python investibook/scripts/eval.py \
  --suite core-daily-demo \
  --execution-mode dry-run
```

Print JSON:

```bash
cd /Users/huyibin/Documents/dev/investibook/backend
uv run python investibook/scripts/eval.py \
  --suite core-daily-demo \
  --json
```

Adjust timeouts:

```bash
cd /Users/huyibin/Documents/dev/investibook/backend
uv run python investibook/scripts/eval.py \
  --suite core-daily-demo \
  --timeout-s 900 \
  --idle-timeout-s 300
```

## Current CLI Behavior

The user-facing CLI defaults are defined in `backend/investibook/core/eval/eval_cli.py`:

- `command=run` when omitted
- `mode=manual`
- `execution-mode=acp`
- `user-email=eval@localhost`
- `timeout-s=900`
- `idle-timeout-s=300`

The runner supports:

- `execution-mode=dry-run`
- `execution-mode=acp`

`dry-run` persists run and case-run records without starting ACP execution.
`acp` creates a fresh workspace and runs live Clinic/Jira/ACP execution.

## Runtime Model

The suite runner implementation lives in `backend/investibook/core/eval/eval_runner.py`.

At a high level it:

1. Loads and validates the suite registry.
2. Creates a fresh no-source workspace per case.
3. Runs ACP in live mode when `execution-mode=acp`.
4. Mirrors workspace artifacts into `backend/data/eval_audits/`.
5. Persists `eval_runs`, `eval_case_runs`, and `eval_judgment` records.
6. Writes run-level text reports.

## Output Locations

Run-level reports are written to:

- `backend/data/eval_audits/<eval_run_id>/report.txt`
- `backend/data/eval_audits/<eval_run_id>/report_detail.txt`

Case execution artifacts are mirrored under the corresponding run audit tree and
workspace audit directories under `backend/data/eval_audits/`.

The active repo baseline manifest is written to:

- `evaluation/regression/baselines/<suite_id>.yaml`

The manifest also records runtime metadata for the promoted run, including:

- `agent_type`
- `resolved_model`

The exported runtime snapshot includes:

- run-level `report.txt` and `report_detail.txt`
- per-case `case_result.yaml`
- per-case `output.txt`
- per-case `judgment.json`
- per-case `.opencode/telemetry.log`
- per-case `semantic_context_v1.json` and `semantic_packet_v1.json` when present
- per-case `evidences/` text/json artifacts

It intentionally excludes transient workspace/runtime files such as `.opencode/`,
download caches, and arbitrary binary files under local data directories.
The exception is the case-local `.opencode/telemetry.log`, which is exported
because deterministic workflow judging depends on it.

You can validate re-judgeability with:

```bash
cd /Users/huyibin/Documents/dev/investibook/backend
uv run python investibook/scripts/eval.py \
  rejudge-baseline-case \
  --run-id evalrun-xxxxxxxx \
  --case-id service-degradation-during-incident-window
```

The text reports include:

- run status and comparison status
- repo baseline comparison summary when a matching baseline manifest exists
- case pass/fail/invalid counts
- total and average scores
- timing
- token usage
- cache token usage
- total cost

When a repo baseline is available for the same `suite_version`, the reports also
include case-level baseline diff signals such as `regressed`, `improved`,
`unchanged`, `new`, and `missing`.

Baseline comparison is only executed when the current run and the baseline have
the same `agent_type` and the same non-empty `resolved_model`. Otherwise the
comparison is reported as `skipped`.

## Environment Bootstrap

`backend/investibook/scripts/eval.py` bootstraps local runtime before importing
the eval modules. It:

- loads `backend/.env.local`
- loads `backend/.env`
- prepends `$HOME/.local/bin` to `PATH` when missing
- sets `UV_CACHE_DIR=/tmp/uv-cache` when unset
- prepends the repo `backend/` root to `PYTHONPATH` when needed

`.env.local` takes precedence over `.env` because it is loaded first and later
loads do not override existing values.

## Environment Variables

Important variables for local eval usage:

### Agent runtime

- `ACP_AGENT_TYPE`
  - Optional.
  - Supported values in current code: `opencode`, `claude`.
  - Current default is `opencode`.
- `ACP_AGENT_BIN`
  - Optional absolute path to the ACP binary when it is not on `PATH`.

### Live data access

- `INVESTIBOOK_CLINIC_API_KEY`
- `INVESTIBOOK_JIRA_USERNAME`
- `INVESTIBOOK_JIRA_API_TOKEN`

### Semantic shadow evaluation

- `SEMANTIC_EVAL_PROVIDER`
  - Current default: `opencode`
  - Also supports `heuristic`
- `SEMANTIC_EVAL_OPENCODE_BIN`
  - Optional override for the binary used by semantic evaluation
- `INVESTIBOOK_SEMANTIC_EVAL_OPENCODE_MODEL`
  - Optional model override for semantic evaluation
  - When unset, semantic evaluation falls back to `ACP_MODEL_STANDARD`

### Optional admin and MCP configuration

- `ADMIN_EMAILS`
- `DEVOPS_BASE_URL`
- `DEVOPS_API_KEY`

## Semantic Evaluation

The current runner always performs deterministic judgment first.

When a case defines supported semantic diagnosis checks, the runner also writes:

- `semantic_context_v1.json`
- `semantic_packet_v1.json`

These are shadow artifacts produced by `backend/investibook/core/eval/semantic_evaluator.py`.
They supplement the deterministic pipeline; they do not replace the registry,
runner, or artifact contract.

## Practical Notes

- Direct CLI eval runs do not require the FastAPI server to be started first.
- Local eval currently assumes a working ACP runtime and valid live-data credentials.
- If a run ends with `acp_bootstrap_timeout`, `acp_error`, or `agent_execution_timeout`,
  inspect the run audit directory under `backend/data/eval_audits/` and the
  corresponding workspace under `backend/data/workspaces/`.