# Evaluation Judge Pipeline Explainer

This document explains the current evaluation judge pipeline implemented in [backend/investibook/core/eval/eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:1).

It focuses on four questions:

- What are the judge steps?
- What inputs does each step use?
- How does each step decide pass/fail/partial?
- What is the practical meaning of each step?

## Scope

This document describes the current implementation behind `judge_case_output()` rather than the aspirational design.

The main focus is the deterministic judgment pipeline, with the current semantic
shadow artifact path noted where relevant.

Entry point:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:24)

Artifact resolution support:

- [eval_artifacts.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_artifacts.py:1)

Schema normalization and enum constraints:

- [eval_contracts.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_contracts.py:1)
- [eval_registry.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_registry.py:382)

Semantic shadow artifact generation:

- [semantic_evaluator.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/semantic_evaluator.py:1)

## High-Level Flow

The judge pipeline runs in this order:

1. Resolve the run bundle.
2. Extract facts from trace, artifacts, evidence files, output text, and analysis patch.
3. Evaluate `contract`.
4. Evaluate `workflow`.
5. Evaluate `execution_contract`.
6. Evaluate `evidence_collection`.
7. Evaluate `critical_evidence`.
8. Evaluate `evidence_routing`.
9. Evaluate `diagnosis` quality.
10. Apply verdict precedence to produce `pass` / `fail` / `invalid`.

The orchestration lives in [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:35).

Additional note:

- The runner executes deterministic judgment first.
- When a case defines supported semantic diagnosis checks, the runner also writes
  `semantic_context_v1.json` and `semantic_packet_v1.json`.
- The semantic shadow is considered usable only when `semantic_packet_v1.json`
  has `status == ok`.
- Missing provider, provider failure, timeout, or unsupported-provider paths
  still leave shadow artifacts, but they do not pollute the deterministic primary score.
- Those semantic files are currently shadow artifacts that supplement diagnosis
  evaluation context; they do not replace the deterministic pipeline described here.

## Step 0: Run Bundle Resolution

### Inputs

- `audit_dir`
- `workspace_id` -> `workspace_dir`
- expected cluster id from case metadata
- allowed time windows from case metadata

Source:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:43)
- [eval_artifacts.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_artifacts.py:112)

### What it does

It locates a usable run bundle containing:

- manifest
- diagnosis report
- evidence directory
- optional analysis patch
- optional summary text/json
- optional logs directory
- optional telemetry file

Resolved bundle fields are defined in [ResolvedRunBundle](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_artifacts.py:10).

### Decision method

- If no valid candidate bundle can be resolved, the whole judgment becomes `invalid`.
- If multiple valid candidates exist and cannot be unambiguously selected, the whole judgment becomes `invalid`.
- Otherwise, resolution succeeds and the chosen bundle is used for the rest of the pipeline.

Failure path:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:339)

### Practical meaning

This step answers: "Are we even judging the right run with the minimum required files?"

Without this gate, later checks could compare the wrong eval run or missing evidence and produce misleading results.

## Step 1: Fact Extraction

The judge does not reason directly over raw files in every bucket. It first extracts normalized fact sets.

### 1.1 Trace Facts

Source:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:1145)

Inputs:

- diagnosis report JSON
- manifest JSON
- telemetry text
- case metadata: expected cluster id and allowed time window

Produced facts:

- `window.*`
- `cluster.*`
- `workflow.*`
- `artifacts.*`

Examples:

- `window.matched_allowed_window`
- `cluster.matched`
- `workflow.architecture_context_loaded`
- `artifacts.analysis_patch_present`

How it works:

- Time window comes from diagnosis report metadata, with manifest fallback.
- Cluster id comes from diagnosis report metadata, with manifest fallback.
- Workflow milestones are inferred from telemetry text and bundle presence.

Practical meaning:

- Trace facts capture whether the run aligns with the intended case and whether the agent executed the expected operational steps.

### 1.2 Evidence Facts

Source:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:1304)

Inputs:

- `evidence/collection_errors.json`
- `evidence/gate_checks.json`
- `evidence/log_stats.json`
- `evidence/queries.json`
- diagnosis report JSON
- summary text
- supplemental log files under `logs_dir`

Produced facts:

- artifact bundle presence
- channel coverage
- channel-attempt / artifact-persistence consistency
- panic / kill-query signals
- simple infra/hotspot indicators
- extracted metric/query evidence for later routing checks

Examples:

- `core_evidence_bundle_present`
- `logs_channel_observed`
- `query_or_metric_channel_observed`
- `events_collection_attempted`
- `events_output_file_present`
- `events_attempted_implies_artifact_present`
- `panic_signal_artifact_present`
- `kill_query_signal_present`
- `metric_evidence`
- `query_evidence`

Practical meaning:

- Evidence facts answer: "Did the run collect the expected kinds of evidence, and what usable signal classes are present?"

### 1.3 Output Facts

Source:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:2170)

Input:

- final `output_text` emitted by the model/agent

Produced facts:

- booleans derived from keyword/regex heuristics over the final answer

Examples:

- `mentions_v8_5_2`
- `diagnosis_kill_query_chain_asserted`
- `diagnosis_cpu_hotspot_asserted`
- `fake_alarm_primary_asserted`
- `missing_index_primary_root_cause_asserted`

Practical meaning:

- Output facts are the bridge between free-form final answer text and deterministic scoring.

### 1.4 Routing Facts

Source:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:1985)

Inputs:

- `analysis_patch.json`
- extracted metric evidence
- extracted query evidence
- channel-observed flags for logs and events

Produced facts:

- patch presence and JSON validity
- item shape validity
- whether key/supporting evidence exists
- whether metric refs are resolvable
- whether query refs are resolvable
- whether log/event references are resolvable

Examples:

- `analysis_patch_present`
- `analysis_items_shape_valid`
- `analysis_metric_refs_required`
- `analysis_metric_refs_resolvable`
- `analysis_query_refs_resolvable`

Practical meaning:

- Routing facts evaluate whether structured evidence references in the analysis patch point to real collected evidence, rather than being made up or stale.

## Step 2: Contract

Source:

- evaluation: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:426)
- normalized schema: [eval_contracts.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_contracts.py:532)

### Inputs

- trace facts only

Typical checks:

- `time_window_match`
- `cluster_scope_match`

### Decision method

- A check evaluates its rules with `all_of` or `any_of`.
- Bucket status is `fail` if any `required` check fails.
- Otherwise bucket status is `pass`.

### Practical meaning

This is the comparability gate.

It answers: "Is this run about the same cluster and same incident window as the case?"

If not, the run is not comparable and should not be scored as a meaningful diagnostic result.

## Step 3: Workflow

Source:

- evaluation: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:512)

### Inputs

- `trace["workflow"]`
- workflow spec from `judge.workflow`

Typical checks:

- `architecture_context_loaded`
- `misdiagnosis_guardrails_loaded`
- `diagnosis_run_initialized`
- `evidence_collection_completed`
- optional checks like `summary_read_completed`

### Decision method

- Each check becomes a scored item with weight.
- Score uses required checks only via `_required_scores()`.
- Status logic:
  - `pass`: no required failures
  - `partial`: required failures exist, but the run clearly progressed
  - `fail`: required failures exist and the run did not meaningfully progress

Source:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:543)

### Practical meaning

This step answers: "Did the agent actually perform the intended diagnosis workflow?"

It is about execution progress, not diagnosis correctness.

## Step 4: Execution Contract

Source:

- evaluation: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:1868)

### Inputs

- workflow check pass/fail facts
- artifact contract facts
- channel contract facts

Those inputs are built from:

- workflow bucket output
- trace artifacts
- evidence facts

Helper builders:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:1844)

### Typical checks

- `required_workflow_present`
- `required_artifacts_present`
- `preferred_channels_minimum_observed`
- `preferred_channels_all_observed`

### Decision method

- `fail`: any required check fails
- `partial`: all required checks pass, but some optional checks fail
- `pass`: all configured checks pass

### Practical meaning

This is the minimum viability gate for scoring.

It answers: "Was enough of the intended workflow, artifact set, and evidence-channel coverage present to make the run judgeable?"

Compared with `workflow`, this is less about process milestones and more about minimum usable execution/output contract.

## Step 5: Evidence Collection

Source:

- evaluation: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:661)

### Inputs

- evidence facts
- artifact facts
- channel facts

### Typical checks

- `required_artifacts_present`
- `preferred_channels_minimum_observed`
- `preferred_channels_all_observed`
- `attempted_events_collection_persisted`

### Decision method

- Uses weighted checks.
- `fail`: any required check fails
- `partial`: required checks pass but optional checks fail
- `pass`: all checks pass

### Practical meaning

This step measures coverage quality, not root-cause correctness.

It answers: "Did the run collect enough evidence, and did it cover the preferred channels well enough?"

Typical use:

- hard-gate "at least one preferred channel exists"
- reward "all preferred channels were observed"
- soft-deduct "events query was attempted but no `events.json` / report `evidence.events` artifact was persisted"

Special note:

- `attempted_events_collection_persisted` is intentionally a soft consistency check.
- If telemetry shows an Activity Hub events query was attempted, the run should persist an events artifact even when the result is empty.
- Failing this check reduces the `evidence_collection` bucket score, but does not by itself change the bucket status to `fail` or make the overall run `invalid`.

## Step 6: Critical Evidence

Source:

- evaluation: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:1347)

### Inputs

- raw log documents from `logs_dir`
- selected JSON artifacts such as:
  - `component_health.json`
  - `gate_checks.json`
  - `metrics.json`
  - `queries.json`
  - restart metric files
  - summary JSON

Artifact loading:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:1520)

### Match methods

- `keyword_any`
- `json_list_nonempty`
- `json_text_contains_any`
- `numeric_series_any_gt`

### Decision method

- Each check gets a weight.
- Each check evaluates its rules under `all_of` or `any_of`.
- Bucket status:
  - `pass`: all checks pass
  - `partial`: some checks pass
  - `fail`: no checks pass

### Practical meaning

This is the objective-evidence layer.

It answers: "Did the collected artifacts actually contain the critical symptom/mechanism evidence that a good diagnosis should have surfaced?"

This bucket is intentionally independent of the model's final wording. It anchors the case in observable evidence.

## Step 7: Evidence Routing

Source:

- evaluation: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:872)

### Inputs

- routing facts extracted from `analysis_patch.json`
- metric/query evidence
- channel availability for logs/events

### What it checks

- patch presence
- patch JSON validity
- item shape validity
- presence of key evidence
- metric ref resolvability
- query ref resolvability
- log/event ref resolvability

### Decision method

- `fail`: any required check fails
- `partial`: required checks pass but optional checks fail
- `pass`: all checks pass

The final score for this bucket currently uses required-check raw score normalized to 10.

Source:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:154)

### Practical meaning

This answers: "Is the structured evidence map internally consistent and resolvable against the actual collected evidence?"

It protects against analysis patches that reference nonexistent metrics, queries, logs, or malformed items.

## Step 8: Diagnosis Quality

The diagnosis bucket is split into two layers.

### 8.1 Hard Checks

Source:

- evaluation: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:467)

Inputs:

- output facts
- raw output text

Decision method:

- Each check is boolean.
- Check passes under `all_of` or `any_of`.
- Overall diagnosis quality status becomes `pass` only if all hard checks pass.

Scoring:

- hard checks are normalized into an 18-point bucket

Practical meaning:

- Hard checks encode must-have conclusions and must-avoid misdiagnoses.
- This is the "critical rubric" for diagnosis quality.

### 8.2 Supporting Checks

Source:

- evaluation: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:399)

Inputs:

- output facts
- raw output text

Decision method:

- Weighted checks under `all_of` / `any_of`
- score normalized to configured supporting weight

Practical meaning:

- Supporting checks reward explanation quality, mechanism reconstruction, comparison, mitigation advice, and follow-up guidance.
- They enrich quality scoring but do not override failed hard checks.

## Step 9: Verdict Precedence

Source:

- [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:173)

Verdict precedence is explicit:

1. `contract fail` -> `invalid`
2. `execution_contract fail` -> `invalid`
3. `workflow fail/partial` -> `invalid`
4. `evidence_collection fail` -> `invalid`
5. `evidence_collection partial` -> `fail`
6. `critical_evidence fail/partial` -> `fail`
7. `evidence_routing fail/partial` -> `fail`
8. all above pass and diagnosis hard checks pass -> `pass`
9. otherwise -> `fail`

### Practical meaning

This ordering separates three classes of outcomes:

- `invalid`
  The run is not comparable or not complete enough to judge fairly.

- `fail`
  The run is comparable, but evidence quality or diagnosis quality is insufficient.

- `pass`
  The run is comparable, evidence is adequate, and diagnosis quality clears the critical bar.

## Input Source Map

The table below summarizes the main inputs by bucket.

| Bucket | Main inputs | Origin |
|---|---|---|
| artifact resolution | audit/workspace directories, manifest, report, evidence dir | file-system layout |
| contract | trace facts | diagnosis report, manifest, case metadata |
| workflow | workflow facts | telemetry text, bundle presence |
| execution_contract | workflow/artifact/channel facts | workflow result, trace artifacts, evidence facts |
| evidence_collection | evidence/artifact/channel facts | evidence JSONs, logs, report |
| critical_evidence | logs and artifact JSON documents | logs dir, evidence JSONs, metrics files, summary JSON |
| evidence_routing | routing facts | analysis patch + collected evidence references |
| diagnosis | output facts + output text | final answer text |

## What Each Layer Protects Against

- artifact resolution
  Protects against judging the wrong run or an incomplete run directory.

- contract
  Protects against cluster/window mismatch.

- workflow
  Protects against runs that did not actually execute the intended diagnosis flow.

- execution_contract
  Protects against scoring runs that do not meet minimum workflow/artifact/channel viability.

- evidence_collection
  Protects against shallow or incomplete evidence coverage.

- critical_evidence
  Protects against diagnoses that ignore the key objective signals in collected evidence.

- evidence_routing
  Protects against malformed or unresolvable structured evidence references.

- diagnosis hard checks
  Protects against wrong root cause, missing must-have conclusions, and classic misdiagnoses.

- diagnosis supporting checks
  Rewards deeper explanation and better remediation guidance.

## Current Caveats

- `json_text_contains_any` in `critical_evidence` currently searches the serialized whole JSON document rather than a path-scoped subfield.
- `ticket_context_channel_observed` is currently hardcoded `False` in evidence extraction.
- output facts are heuristic keyword/regex detectors; they are deterministic but not semantic parsers.
- evidence routing quality depends on the structure and reference style of `analysis_patch.json`.

Relevant implementation references:

- evidence extraction: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:1304)
- output extraction: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:2170)
- routing extraction: [eval_judge.py](/Users/huyibin/Documents/dev/investibook/backend/investibook/core/eval/eval_judge.py:1985)