# Search Quality Evaluation Plan

Date: 2026-03-22
Status: Draft (Normative for roadmap gate references)
Owner: Semantic Search / Query Broker / Testing

## 1. Scope

This document defines the benchmark assets, judging rules, metrics, and evidence requirements used to close roadmap search-quality gates for semantic and hybrid retrieval.

Normative usage:
- `docs/governance/roadmap.md` Phase-1 and later roadmap gates may reference this document as the authoritative evaluation rule set.
- This document governs retrieval quality evaluation only.
- It does not redefine online latency, throughput, or compatibility conformance suites.

## 2. Evaluation Assets

### 2.1 Benchmark Bundle

Every milestone- or checkpoint-closing run MUST publish one immutable benchmark bundle containing:
- corpus snapshot identifier
- embedding model / analyzer / rerank configuration identifiers
- route / planner coverage declaration for retrieval-affecting routes and planning modes exercised by the benchmark
- retrieval-affecting feature-toggle summary (for example hybrid-planning mode, rerank enablement, or route-selection policy)
- query-set version
- ground-truth judgment version
- evaluation harness version
- result artifact manifest

The benchmark bundle MUST be versioned and replayable.

### 2.2 Query Cohorts

The benchmark query set MUST include at least the following cohorts:
- semantic-only cohort
- hybrid cohort (`metadata predicates + semantic retrieval`)
- hard-negative hybrid cohort (metadata filters that remove many near neighbors)

Minimum query counts per milestone- or checkpoint-closing run:
- semantic-only: `>= 400`
- hybrid: `>= 400`
- hard-negative hybrid: `>= 200`

Phase-1 gate closure uses the semantic-only and hybrid cohorts as normative gate cohorts.
Phase-3 gate closure uses the semantic-only, hybrid, and hard-negative hybrid cohorts as normative gate cohorts.

## 3. Ground-Truth Rules

Ground truth MUST be frozen before a milestone- or checkpoint-closing run starts.

Admitted relevance sources:
- manually labeled judgments
- curated product/task gold sets
- deterministic task-derived positives reviewed by humans

The same query MUST NOT mix incomparable judgment schemes inside one milestone- or checkpoint-closing benchmark bundle.

Every judged query MUST have:
- one stable query identifier
- one stable cohort label
- one stable set of relevant document identifiers

## 4. Metrics

Normative metrics:
- `Recall@10`
- `Recall@50`
- `nDCG@10`

Roadmap gate metric:
- `Recall@10`

Reporting rules:
- report metrics separately for the semantic-only cohort and the hybrid cohort
- report the hard-negative hybrid cohort separately; do not merge it into the primary gate cohorts
- publish both per-cohort metrics and weighted aggregate metrics

## 5. Gate Thresholds

### 5.1 Phase-1 Gate

Phase-1 roadmap search-quality gate is passed only when:
- semantic-only cohort `Recall@10 > 90%`
- hybrid cohort `Recall@10 > 90%`

### 5.2 Phase-3 GA-Readiness Gate

Phase-3 roadmap search-quality gate is passed only when:
- semantic-only cohort `Recall@10 > 90%`
- hybrid cohort `Recall@10 > 90%`
- hard-negative hybrid cohort `Recall@10 > 85%`
- unless a new benchmark bundle is published under Section `6`, no gate cohort regresses by more than `2` percentage points versus the most recent accepted milestone- or checkpoint-closing bundle

### 5.3 Phase-4 Post-GA Expansion Continuation Gate

When a later milestone or checkpoint closes retrieval-affecting post-GA expansion work (for example rerank or new compatibility routes), that gate passes the search-quality continuation gate only when:
- the milestone or checkpoint publishes a fresh benchmark bundle whenever the expansion changes retrieval-affecting route coverage, planner mode, rerank policy, analyzer policy, embedding policy, or other benchmark-relevant retrieval behavior;
- semantic-only cohort `Recall@10 > 90%`
- hybrid cohort `Recall@10 > 90%`
- hard-negative hybrid cohort `Recall@10 > 85%`
- unless the fresh benchmark bundle intentionally re-baselines under Section `6`, no previously gate-bearing cohort regresses by more than `2` percentage points versus the most recent accepted milestone- or checkpoint-closing bundle.

### 5.4 Reporting-Only Metrics

For Phase-1 closure, the following metrics are mandatory evidence but not standalone roadmap pass/fail criteria:
- semantic-only `Recall@50`
- hybrid `Recall@50`
- semantic-only `nDCG@10`
- hybrid `nDCG@10`
- hard-negative hybrid cohort metrics

## 6. Freshness and Change Control

Benchmark refresh policy:
- milestone- or checkpoint-closing runs MUST use a frozen corpus and judgment bundle
- the bundle MAY be refreshed between milestones or checkpoints, but not during a closing run
- any refresh MUST increment the benchmark bundle version and publish a change summary

If a corpus, embedding model, analyzer, rerank policy, retrieval route set, planner mode, or other retrieval-affecting policy changes materially, the next milestone- or checkpoint-closing run MUST publish a new benchmark bundle rather than silently reusing old judgments.

### 6.1 Benchmark Reuse Boundary by Roadmap Family

| Benchmark Bundle Accepted At | Admissible Reuse | Mandatory Freshness Boundary | Explicit Non-Substitution Rule |
|---|---|---|---|
| Phase-1 closing run | later audit context or non-gate comparison only | any material retrieval-affecting change before the next gate-bearing milestone | does not close later Phase-3 or Phase-4B gates by itself |
| Phase-3 closing run | same-date `external GA minimum-bar` and `platform qualification` closeout records may both reference it when the accepted verdict records name the same bundle explicitly | any retrieval-affecting route/planner/rerank/analyzer/embedding change before closeout requires a fresh bundle | does not replace the Phase-3 workload bundle or the October single-collection package |
| Phase-4B closing run | later audit context and explicit re-baseline comparison only | any later retrieval-affecting expansion change | does not replace the post-GA workload/evidence package |

## 7. Evidence Artifacts

Every milestone- or checkpoint-closing run MUST publish:
- benchmark bundle manifest
- raw ranked-result outputs
- computed metrics table
- query-level failure slices for misses in the gate cohorts
- environment/configuration summary
- accepted closeout record cross-reference when the roadmap gate also requires one

The roadmap gate is not auditable without these artifacts.