# Metadata Search Reader

Status: Frozen architecture note
Owner: Search Platform Architecture

## 1. Scope

This document defines the execution design of `Metadata Search Reader`, the stateless query-plane component that evaluates metadata/text retrieval work under one broker-pinned snapshot.

This document is subordinate to the canonical owner stack:
- machine-readable contract: `api/proto/searchfabric/v1/search_fabric.proto`
- internal owner file for this boundary: `api/proto/searchfabric/v1/metadata_search_reader.proto`
- public query semantics: `docs/specs/core/search-fabric-api.md`
- canonical error registry: `docs/specs/core/error-codes.md`
- system topology and state ownership: `docs/specs/core/search-fabric-architecture.md`

It must not redefine:
- public `QueryRequest` / `QueryHit` / `AggregationResponse` semantics,
- query admission, `safe_ts`, `effective_read_ts`, or continuation-token semantics,
- manifest publication or schema-serving readiness ownership,
- write-path ordering, durability, or `MutationAck` behavior,
- semantic/vector retrieval behavior owned by `Semantic Search Reader`.

## 2. Canonical Position in the System

`Metadata Search Reader` sits on the query plane behind `Query Broker`:

```text
Query Gateway
  -> Query Broker
       -> Metadata Search Reader
            -> Local Cache
            -> Object Store
            -> NRT Delta Store (Metadata Search)
```

The component is the execution engine for:
- scalar predicate evaluation,
- text retrieval,
- metadata-side candidate generation,
- metadata-side count/estimate work,
- shard-local partial aggregation work,
- broker-requested metadata materialization.

It is not:
- a public RPC surface,
- a second snapshot-admission authority,
- a semantic/vector execution engine,
- a final aggregation owner,
- a manifest or token owner.

Primary source anchors:
- `docs/specs/core/search-fabric-architecture.md` Section `7`
- `docs/architecture/query-broker.md`
- `docs/architecture/query-gateway.md`
- `docs/architecture/nrt-delta-store.md`
- `api/proto/searchfabric/v1/metadata_search_reader.proto`
- `api/proto/searchfabric/v1/query_broker.proto`

## 3. Design Anchors

The design below is constrained by the following checked-out facts:

1. Read query execution never bypasses `Query Broker`; admitted execution always flows `Query Gateway -> Query Broker -> Metadata Search Reader`.
2. `Query Gateway` owns consistency admission and derives or validates the execution snapshot; `Query Broker` pins the immutable query lineage and committed manifest.
3. `Metadata Search Reader` executes only against the broker-admitted snapshot boundary:
   - exact caller `read_ts` only when strict execution was admitted as-is,
   - otherwise admitted `effective_read_ts`,
   - continuation/search-context execution reuses token-bound snapshot state rather than recomputing from the newest readiness frontier.
4. Reader-visible state is the merge of:
   - immutable metadata-search artifacts selected by the pinned committed manifest,
   - publish-admitted metadata deltas from `NRT Delta Store (Metadata Search)`.
5. Row visibility remains MVCC-based: only versions with `commit_ts <= effective_read_ts` and not tombstoned by a winning delete at `commit_ts <= effective_read_ts` may contribute to results.
6. Metadata-side aggregation pushdown is shard-local only; final public `AggregationResponse` trees remain broker-owned.
7. Reader outputs remain broker-facing execution products such as candidates, counts, semantic prefilter payloads, partial aggregations, and optional materialized hit fragments; public response reconstruction remains outside this component.

## 4. Responsibilities and Non-Responsibilities

### 4.1 Responsibilities

`Metadata Search Reader` is responsible for:

1. Executing broker-dispatched metadata/text subplans against one immutable snapshot.
2. Reading the pinned committed manifest and its referenced metadata-search objects through disposable local cache layers.
3. Applying metadata/text predicates and text ranking over immutable artifacts plus publish-admitted NRT metadata deltas.
4. Enforcing same-snapshot MVCC visibility on metadata-side row versions.
5. Producing broker-consumable outputs such as:
   - `(doc_id, commit_ts, metadata score/features)` candidates,
   - `COUNT` / `ESTIMATE` results,
   - semantic prefilter payloads,
   - partial aggregation states,
   - optional materialized hit fragments when broker requests metadata-side materialization.
6. Preserving deterministic ordering and deterministic field/dictionary alignment inside its internal response carriers.
7. Remaining stateless for correctness: restart may lose cache, but must not lose the ability to execute against manifests and S3 truth.

### 4.2 Non-Responsibilities

`Metadata Search Reader` is not responsible for:
- parsing Elasticsearch / Turbopuffer / SQL source syntax,
- collection-name resolution or catalog admission,
- determining `safe_ts` or waiting for visibility,
- validating public continuation/search-context tokens,
- publishing manifests or advancing watermarks,
- final fusion across metadata and semantic lanes,
- final aggregation reduction,
- relational overlay ownership,
- write-lane OCC or current-version authority.

## 5. Runtime Placement and Request Model

### 5.1 Upstream / Downstream Boundary

```text
Query Broker
  -> Metadata Search Reader
       -> pinned committed manifest objects
       -> metadata-search NRT deltas
       -> local cache / object store fetches
  <- Metadata Search Reader response
```

The reader consumes broker-owned dispatch context that already fixes:
- collection identity,
- applied schema lineage,
- pinned committed manifest identity,
- admitted `effective_read_ts`,
- one metadata physical subplan,
- aggregation/materialization intent.

Because these inputs are already fixed upstream, the reader must not reopen:
- query-admission choices,
- serving-generation selection,
- consistency-mode reinterpretation,
- token legality.

### 5.2 Reader RPC Shape

The current internal contract is split deliberately:
- service surface: `api/proto/searchfabric/v1/metadata_search_reader.proto`
- shared admitted-snapshot / broker-reader execution carriers: `api/proto/searchfabric/v1/query_broker.proto`

The service boundary remains:
- `MetadataSearchReaderService.ExecuteSearch(MetadataSearchRequest) returns (MetadataSearchResponse)`

That split keeps the metadata-reader RPC owner local to the component while reusing broker-shared execution carriers that still belong to the broker planning/execution model. The boundary remains internal-only and subordinate to the public query contract.

## 6. State Model

### 6.1 Immutable Truth + Short-Lived Delta Overlay

For one admitted subplan, the reader reconstructs metadata state from two sources:

```text
visible_metadata_snapshot
  = pinned committed manifest metadata artifacts
    + publish-admitted metadata NRT delta window
```

The immutable side typically includes manifest-selected metadata-search artifacts such as:
- inverted-index/postings data,
- term dictionaries,
- fast-field / columnar metadata,
- stored-field or field-value materialization assets,
- visibility/tombstone sidecars,
- any manifest-declared supporting metadata needed by the chosen subplan.

The mutable side is limited to `NRT Delta Store (Metadata Search)` and remains bounded by the flush window described in `docs/architecture/nrt-delta-store.md`.

### 6.2 Identity and Versioning

Metadata-side execution uses canonical collection-scoped identity:
- row identity: `(namespace_id, collection_id, doc_id)`
- visible-version order: `commit_ts`, then canonical tie-breaks `(source_partition, source_epoch, source_seq)` when needed upstream

Reader-local ordinals, shard IDs, segment IDs, or cache keys are implementation details only. They must not become cross-request correctness identity.

## 7. Snapshot and Visibility Rules

### 7.1 Reader-Local Invariant

`Metadata Search Reader` must evaluate every candidate under the broker-admitted snapshot boundary:

```text
visible(row_version) =
  pinned_committed_manifest_contains_base_state
  AND row_version.commit_ts <= effective_read_ts
  AND no later winning tombstone with commit_ts <= effective_read_ts hides the row
```

The exact storage representation may differ between immutable artifacts and NRT, but the visibility predicate must stay equivalent.

### 7.2 What the Reader Must Not Do

The reader must not:
- compute a fresh `safe_ts`,
- treat raw request `read_ts` as the execution boundary when broker supplied `effective_read_ts`,
- read unpublished metadata deltas,
- widen visibility beyond the pinned committed-manifest frontier,
- downgrade strict/bounded execution into best-effort locally.

If the requested snapshot was not admissible, that failure should already have been handled by `Query Gateway` / `Query Broker` using canonical `SF_VISIBILITY_TIMEOUT` behavior; the reader does not own that branch.

## 8. Execution Pipeline

A production-quality reader pipeline should be understood as the following sequence:

```text
broker request
  -> validate internal request coherence
  -> open pinned manifest-backed metadata plan inputs
  -> acquire cache/object handles
  -> execute metadata/text operators
  -> apply MVCC visibility and tombstones
  -> merge publish-admitted metadata NRT deltas
  -> produce candidates / counts / partial aggs / materialized hits
  -> return broker-facing response
```

### 8.1 Internal Request Validation

Before heavy work begins, the reader should validate internal coherence of the broker dispatch, including:
- collection identity presence,
- manifest pin identity/checksum presence,
- positive `effective_read_ts`,
- subplan/operator compatibility,
- aggregation/materialization flags consistent with the chosen plan.

This is internal contract validation, not public query validation.

### 8.2 Immutable Artifact Open

The reader opens the exact pinned committed manifest selected by broker and resolves the metadata-search artifacts referenced by that manifest. Head advancement after the broker pinned the manifest must not affect this execution.

### 8.3 Cache / Object Access

Local cache exists only as a latency optimization. Correctness must remain reconstructible from:
- the pinned manifest,
- referenced immutable objects,
- the publish-admitted NRT metadata window.

A cache miss may increase latency, but it must not change results.

### 8.4 Operator Evaluation

The reader executes metadata/text operators such as:
- structured predicate evaluation,
- text clause evaluation,
- metadata-only filter execution,
- count/estimate probes,
- aggregation partial collection,
- metadata-side hit materialization requested by broker.

The precise local data structures are implementation choices; the semantic result must remain subordinate to the canonical query contract.

### 8.5 MVCC + NRT Merge

After or during immutable-artifact evaluation, the reader merges publish-admitted NRT metadata deltas under the same `effective_read_ts` and committed-manifest frontier. This merge must preserve the same winner selection the system already uses elsewhere for metadata state.

### 8.6 Broker-Facing Response Construction

The response returned to broker may contain, depending on the plan:
- candidate rows (`doc_id`, `commit_ts`, score/features),
- `local_total_hits`,
- metadata-produced semantic prefilter payload,
- shard-local partial aggregation states,
- response-local dictionaries needed to decode any materialized hit fragments.

The reader must not return a final public `AggregationResponse` tree or final hybrid ranking order; those remain broker-owned.

## 9. Query Shapes Served by Metadata Search Reader

### 9.1 Metadata-Only Retrieval

When the admitted plan is metadata-only, the reader can serve:
- scalar filter queries,
- full-text queries,
- mixed metadata + text queries,
- filter-only execution,
- count/estimate probes,
- metadata-driven aggregation partials.

In this shape the reader may be the only data reader engaged by broker.

### 9.2 Metadata-Assisted Hybrid Retrieval

For hybrid plans the reader commonly serves one of two broker-visible roles:

1. return metadata candidates directly for later fusion with semantic candidates,
2. produce a semantic prefilter payload that narrows semantic execution.

The checked-out architecture explicitly allows metadata output to become:
- an exact `doc_id` set for highly selective filters,
- a compressed payload such as aligned bitmap/Bloom-style filter material for broader filters.

This output is optimization-only. Final eligibility still remains governed by the admitted canonical global filter and broker-owned fusion/recheck steps.

### 9.3 Selector Materialization Reuse

By-query write lanes may reuse broker snapshot execution to materialize deterministic selector results. In those cases the metadata reader still remains read-only; it simply contributes deterministic `doc_id` / `version_commit_ts` data to broker so the ingest path can later emit ordered mutations.

## 10. Aggregation Boundary

The reader may execute shard-local metadata aggregation work, but only within a strict boundary:
- reader returns partial aggregation states,
- broker performs final reduction into the public response shape,
- semantic/vector-dominant facet reduction remains broker-local in the active revision.

This boundary matters because it keeps:
- public aggregation semantics centralized,
- cross-lane fusion and final bucket ordering centralized,
- metadata reader focused on local scan/filter/partial-reduce execution.

## 11. NRT Integration

### 11.1 Purpose

`NRT Delta Store` lets streaming writes become query-visible after committed-manifest publication without waiting for immutable micro-segment flush. `Metadata Search Reader` is one of the consumers of that state.

### 11.2 Reader Contract with NRT

The reader must consult metadata NRT only when all of the following hold:
- the data belongs to the metadata modality,
- the write scope is admitted under the pinned committed-manifest frontier,
- the row version satisfies `commit_ts <= effective_read_ts`.

The reader must not treat NRT as:
- an independent publish surface,
- a best-effort side channel,
- a replacement for immutable metadata artifacts.

### 11.3 LWW and Tombstones

When immutable metadata artifacts and metadata NRT both contain versions for the same `doc_id`, the visible winner must follow canonical version ordering. Tombstones participate in the same rule; they are not an auxiliary filter that may lag behind version choice.

## 12. Storage and Cache Design Guidance

### 12.1 Immutable Object Usage

The core architecture already fixes that durable truth lives in object storage and manifests. For metadata execution this means the reader should expect immutable object-backed metadata-search bundles and sidecars selected by the pinned manifest rather than mutable local index ownership.

### 12.2 Cache Guidance

Reasonable cache contents include immutable, rebuildable artifacts such as:
- manifest blobs,
- postings blocks,
- dictionaries,
- fast-field pages,
- bitmap sidecars,
- stored-field chunks,
- plan-friendly index headers.

Reasonable cache properties:
- disposable,
- bounded,
- manifest-aware,
- safe to invalidate or lose on restart.

The reader must not require:
- durable local cache state for correctness,
- cache-global coordination to preserve query semantics,
- cache residency as a precondition for continuation correctness.

### 12.3 Prefetch and Warmup

Prefetch, hedging, and warmup are acceptable implementation techniques, but they are performance policies only. They must not create a hidden correctness dependency or alter canonical results.

## 13. Failure Semantics and Recovery

### 13.1 Failure Ownership

Public query/readiness failures such as `SF_VISIBILITY_TIMEOUT` are owned upstream by gateway/broker admission. The metadata reader mainly exposes execution failures through the internal broker boundary using canonical `ErrorStatus` projection.

Typical reader-local failure families include:
- inaccessible or mismatched pinned manifest/object inputs,
- corrupted or unreadable metadata-search artifacts,
- internal execution bugs,
- local resource exhaustion that prevents completing the dispatched subplan.

Exact public error-code mapping remains owned by the canonical API/error stack and the broker/gateway surfaces that project these failures outward.

### 13.2 Restart Behavior

The reader is stateless for correctness. On restart it may lose:
- manifest/object caches,
- local file handles,
- warmed dictionaries/bitmaps,
- execution-local heuristics.

After restart it must still be able to serve the same request shape by reopening the pinned manifest and referenced objects plus any still-retained NRT state admitted under the committed-manifest frontier.

### 13.3 Delayed GC Dependency

Continuation correctness depends on the pinned manifest and its referenced immutable objects remaining readable for at least the maximum admitted token lifetime. The reader relies on that system-wide delayed-GC rule but does not own it.

## 14. Operational Constraints

### 14.1 Multi-Tenant Isolation

Because the system targets extremely high collection cardinality, metadata-reader implementation should assume:
- bounded per-request memory,
- bounded intermediate candidate buffers,
- bounded aggregation partial state,
- cooperative cancellation when broker/request deadlines expire.

### 14.2 Observability Guidance

Useful reader-local telemetry includes:
- immutable-object cache hit/miss behavior,
- pinned-manifest open latency,
- metadata operator latency by plan phase,
- NRT merge participation rate,
- partial-aggregation input/output counts,
- candidate counts before/after visibility filtering,
- cancellation and resource-shedding counters.

These are operational signals only; they do not change the public contract.

## 15. Open Implementation Choices Kept Outside Canonical Semantics

The checked-out owner stack leaves several implementation choices intentionally open, provided the external semantics above are preserved. Examples include:
- exact internal library/layout choices for metadata/text indexing,
- exact cache admission/eviction algorithm,
- exact postings/dictionary/stored-field object bundling,
- exact concurrency model inside one reader process,
- exact prefilter payload encoding choice when it remains semantically equivalent to the broker contract.

Any chosen implementation must still preserve:
- pinned-manifest execution,
- admitted-snapshot visibility,
- metadata-only shard-local aggregation boundary,
- publish-admitted NRT merge semantics,
- deterministic broker-facing outputs.