# Metadata Stream Writer

Status: Frozen architecture note
Owner: Search Platform Architecture

## 1. Scope

This document explains the execution design of `Metadata Stream Writer` (MSW), the metadata/text streaming writer named by the canonical architecture.

This document is subordinate to the canonical owner stack:
- machine-readable contract: `api/proto/searchfabric/v1/search_fabric.proto`
- internal owner file for this boundary: `api/proto/searchfabric/v1/metadata_stream_writer.proto`
- ingest/query semantics: `docs/specs/core/search-fabric-api.md`
- system topology and state ownership: `docs/specs/core/search-fabric-architecture.md`
- canonical error registry: `docs/specs/core/error-codes.md`

It must not redefine public wire semantics, storage identity, or error semantics owned by those documents.

## 2. Canonical Position in the Write Path

MSW is one write-plane executor in the canonical path:

```text
Ingest Gateway
  -> Ingest Broker
  -> Write Planner
  -> Commit Coordinator
  -> Metadata Stream Writer
  -> Visibility Barrier
  -> Manifest Publisher
```

MSW receives already-admitted metadata/text work after:
- request authz, schema, truth-mode, and replay checks are complete,
- Flow A keeps source `commit_ts`,
- Flow B has already received canonical `commit_ts` injection from `Commit Coordinator`,
- canonical batch and stream identity have already been fixed by the admitted `MutationBatch`.

MSW does not:
- accept source-facing API requests directly,
- invent a second durable stream identity,
- mutate manifests,
- define visibility boundaries,
- own background compaction, repair, or garbage collection.

## 3. Contract Binding

MSW must execute against the canonical ingest carriers rather than writer-local envelopes:
- batch scope: `MutationBatch`
- row scope: `RowMutation`
- write policy: `IngestPolicy`
- progress/reporting: `MutationAck.success.applied_watermarks`
- stream identity: `MutationBatch.stream_identity`
- replay identity: `batch_id` plus `batch_fingerprint`

Durable object identity must remain derived from the canonical stream tuple and collection scope:
- `(namespace_id, collection_id, source_partition, source_epoch)` for WAL/checkpoint identity
- canonical manifest lineage for visibility publication

Writer-local scheduling or cache affinity may use internal shard assignment, but that assignment is observational only. It must not appear in durable object identity, replay identity, or recovery semantics.

## 4. Responsibilities

MSW is responsible for metadata/text streaming execution only:

1. Consume commit-ordered metadata/text mutations dispatched by `Write Planner`.
2. Append metadata-search WAL records.
3. Materialize committed-frontier unflushed execution state in `NRT Delta Store (Metadata Search)`.
4. Seal immutable WAL chunks under bounded size/op/age policy.
5. Persist WAL chunks to object storage and report durable completion.
6. Flush immutable metadata micro-segments to object storage.
7. Emit completion/progress information needed by `Visibility Barrier` and downstream ACK reconstruction.

MSW is not responsible for:
- semantic/vector execution,
- public compatibility response reconstruction,
- manifest CAS publication,
- catalog head advancement,
- lifecycle transition execution,
- compaction rewrites,
- GC,
- reader-driven online repair flows.

## 5. Execution Model

### 5.1 Input Ordering

MSW executes mutations in the canonical admitted order:
- primary key: `commit_ts`
- tie-breaks: `source_partition`, `source_epoch`, `source_seq`

That ordering is already defined by the canonical ingest contract and must not be replaced by writer-local arbitration rules.

### 5.2 WAL and Delta Application

For each admitted metadata/text mutation scope, MSW:
- appends a WAL representation suitable for replay within the canonical anchored replay window,
- mutates `NRT Delta Store (Metadata Search)` so query serving can merge publish-admitted unflushed metadata deltas without waiting for segment flush,
- prepares immutable micro-segment outputs for later manifest publication.

The WAL is object-chunked, not append-log emulation. Chunk seal policy may use bounded thresholds such as:
- `max_chunk_bytes`
- `max_chunk_ops`
- `max_chunk_age_ms`

Those thresholds are execution tuning only. They do not change the canonical replay or visibility contract.

### 5.3 Durability Boundary

MSW may report durable completion only after:
1. the sealed WAL chunk has been persisted successfully to object storage, and
2. the corresponding durable replay anchor/checkpoint state has been persisted.

This is the canonical WAL-durable boundary used by `DURABILITY_POLICY_WAL_CHUNK_DURABLE`.

### 5.4 Visibility Boundary

MSW does not publish query visibility by itself.

Read visibility is reached only after:
1. every touched modality reports the required durable completion,
2. `Visibility Barrier` emits a publish-ready intent,
3. `Manifest Publisher` writes `manifest_vN+1`,
4. `Manifest Publisher` successfully advances `manifest_committed_head.json`.

Therefore MSW must treat `NRT Delta Store (Metadata Search)` as read-path execution state eligible for downstream publish evaluation under the committed-manifest frontier, not as a local watermark transition or independent public visibility boundary.

## 6. Watermarks and Progress Reporting

MSW must use canonical progress vocabulary.

For metadata execution, the relevant reported frontier is:
- `applied_ts_metadata_search`

Collection-visible safety is derived downstream as:
- `safe_ts = min(present enabled modality applied ts)`

The public/readiness watermark contract remains downstream and collection-scoped:
- `EngineWatermarks` returns every modality enabled on the target collection, even when the latest admitted write touched only metadata search and left another enabled modality unchanged
- writer-local completion signals and `ProgressMark.watermarks` are narrower modality-scoped advance inputs, so omission means "this mark does not advance that modality", not "the collection lacks that modality"

MSW must not redefine those concepts as writer-local public semantics such as `applied_ts_nrt` or `applied_ts_durable`.

Related durable-manifest metadata remains canonical:
- `applied_ts_*`
- `cdc_resolved_ts`
- `wal_replay_start_anchor`
- `wal_durable_anchor`

## 7. Scheduling and Affinity

MSW may use collection affinity to improve cache locality for:
- schema caches,
- tokenizer/analyzer state,
- in-memory batching,
- local flush buffers.

Reasonable implementation techniques include:
- consistent hashing on `(namespace_id, collection_id)`,
- virtual nodes for long-tail balance,
- dedicated pools for whale tenants.

These are scheduler concerns only. They must not:
- create collection-local durable key namespaces that bypass canonical storage layout,
- change WAL/chunk identity,
- require `msw_instance_id` in canonical S3 object identity,
- enable cross-collection shared WAL objects that break canonical replay and recovery boundaries.

## 8. Storage and Spill Behavior

Local NVMe may be used only as ephemeral execution spill/cache:
- absorb temporary object-store latency,
- bound memory pressure during flush preparation,
- stage local files before immutable object upload.

Because S3 is the only durable source of truth:
- local spill state is disposable,
- recovery must rely on canonical WAL/manifests/anchors,
- no local spill artifact may become an independent durability authority.

Backpressure policy may escalate by memory or spill pressure, but external failure projection must still follow canonical ingest/error semantics.

## 9. Failure Handling

MSW failure handling must preserve owner boundaries:
- WAL-durable retry and replay remain bounded by canonical replay anchors.
- Write-side `wait_for_visible` deadline expiry is not a writer-defined visibility timeout; it remains a stream-level ingest failure projected as retryable `SF_BACKEND_UNAVAILABLE`.
- Query/read-side snapshot waiting is the only domain that uses `SF_VISIBILITY_TIMEOUT`.
- Heavy rebuild, compaction rewrite, and object reclamation remain background-plane responsibilities.

Crash recovery must be pointer-driven:
- load the committed manifest lineage,
- load `wal_replay_start_anchor` and `wal_durable_anchor`,
- replay only the canonical retained WAL window,
- ignore orphan/unanchored tails.

Implementation tuning may still vary for chunk seal thresholds, memory-to-spill transition thresholds, metadata micro-segment flush sizing, tokenizer/analyzer cache sizing, and fair scheduling under whale and long-tail coexistence, but any such choice must preserve the canonical ingest, replay, durability, and visibility contracts described above.