# Semantic Stream Writer

Status: Frozen architecture note
Owner: Search Platform Architecture

## 1. Scope

This document explains the execution design of `Semantic Stream Writer` (SSW), the semantic/vector streaming writer named by the canonical architecture.

This document is subordinate to the canonical owner stack:
- machine-readable contract: `api/proto/searchfabric/v1/search_fabric.proto`
- internal owner file for this boundary: `api/proto/searchfabric/v1/semantic_stream_writer.proto`
- ingest/query semantics: `docs/specs/core/search-fabric-api.md`
- system topology and state ownership: `docs/specs/core/search-fabric-architecture.md`
- canonical error registry: `docs/specs/core/error-codes.md`

It must not redefine public wire semantics, visibility boundaries, storage authority, or canonical vector-schema semantics owned by those documents.

## 2. Canonical Position in the Write Path

SSW is one write-plane executor in the canonical path:

```text
Ingest Gateway
  -> Ingest Broker
  -> Write Planner
  -> Commit Coordinator
  -> Semantic Stream Writer
  -> Visibility Barrier
  -> Manifest Publisher
```

SSW receives already-admitted semantic work after:
- request authz, truth-mode, schema, and replay checks are complete,
- `Write Planner` has already decided that the admitted batch touches semantic modality,
- Flow A keeps source `commit_ts`,
- Flow B has already received canonical `commit_ts` injection from `Commit Coordinator`,
- canonical batch and stream identity have already been fixed by the admitted `MutationBatch`.

SSW does not:
- accept source-facing API requests directly,
- invent a second durable stream identity,
- redefine `PatchPayload` or other public mutation semantics,
- define public visibility boundaries,
- mutate manifests,
- own background compaction, repair, or garbage collection.

## 3. Contract Binding

SSW must execute against the canonical ingest carriers rather than writer-local envelopes:
- batch scope: `MutationBatch`
- row scope: `RowMutation`
- write policy: `IngestPolicy`
- progress/reporting: `MutationAck.success.applied_watermarks`
- stream identity: `MutationBatch.stream_identity`
- replay identity: `batch_id` plus `batch_fingerprint`

Durable object identity must remain derived from canonical collection scope plus canonical stream tuple:
- `(namespace_id, collection_id, source_partition, source_epoch)` for WAL/checkpoint identity
- canonical manifest lineage for visibility publication

Writer-local shard routing, thread assignment, queue IDs, and cache affinity are observational only. They must not appear in durable object identity, replay identity, or recovery semantics.

## 4. Responsibilities

SSW is responsible for semantic/vector streaming execution only:

1. Consume commit-ordered semantic mutations dispatched by `Write Planner`.
2. Append semantic WAL records.
3. Materialize committed-frontier unflushed execution state in `NRT Delta Store (Semantic Search)`.
4. Maintain mutable semantic in-memory execution state needed for low-latency incremental ANN updates.
5. Seal immutable WAL chunks under bounded size/op/age policy.
6. Persist WAL chunks to object storage and report durable completion.
7. Flush immutable semantic micro-segments to object storage.
8. Emit completion/progress information needed by `Visibility Barrier` and downstream ACK reconstruction.

SSW is not responsible for:
- metadata/text execution,
- public compatibility response reconstruction,
- manifest CAS publication,
- catalog head advancement,
- lifecycle transition execution,
- compaction rewrites,
- GC,
- public readiness or query-routing semantics.

## 5. Execution Model

### 5.1 Input Ordering

SSW executes semantic mutations in the canonical admitted order:
- primary key: `commit_ts`
- tie-breaks: `source_partition`, `source_epoch`, `source_seq`

That ordering is already defined by the canonical ingest contract and must not be replaced by writer-local arbitration rules.

### 5.2 Semantic WAL and Delta Application

For each admitted semantic mutation scope, SSW:
- appends a WAL representation suitable for replay within the canonical anchored replay window,
- mutates `NRT Delta Store (Semantic Search)` so query serving can merge publish-admitted unflushed semantic deltas without waiting for segment flush,
- updates mutable semantic in-memory state needed by the active incremental ANN structure,
- prepares immutable semantic segment outputs for later manifest publication.

The WAL is object-chunked, not append-log emulation. Chunk seal policy may use bounded thresholds such as:
- `max_chunk_bytes`
- `max_chunk_ops`
- `max_chunk_age_ms`

Those thresholds are execution tuning only. They do not change the canonical replay, durability, or visibility contract.

### 5.3 Semantic Execution Details

SSW may use semantic-writer-specific execution techniques such as:
- incremental graph maintenance for mutable ANN frontier state,
- semantic payload fingerprinting to detect no-op semantic updates on already routed semantic work,
- segment-local quantization/re-encoding policy,
- short-lived neighbor/topology reuse inside the current mutable execution window,
- modality-local batching and flush preparation tuned for semantic payload size.

Those techniques are implementation details only. They must remain subordinate to the public vector schema contract:
- `distance_metric`,
- `element_type`,
- `VectorSearchTuningSpec`,
- `VectorRetrievalIntent`.

SSW must not assume one universal normalization, quantization, or fidelity policy across all collections unless that policy is already implied by the admitted canonical schema.

### 5.4 Durability Boundary

SSW may report durable completion only after:
1. the sealed WAL chunk has been persisted successfully to object storage, and
2. the corresponding durable replay anchor/checkpoint state has been persisted.

This is the canonical WAL-durable boundary used by `DURABILITY_POLICY_WAL_CHUNK_DURABLE`.

### 5.5 Visibility Boundary

SSW does not publish query visibility by itself.

Read visibility is reached only after:
1. every touched modality reports the required durable completion,
2. `Visibility Barrier` emits a publish-ready intent,
3. `Manifest Publisher` writes `manifest_vN+1`,
4. `Manifest Publisher` successfully advances `manifest_committed_head.json`.

Therefore SSW must treat `NRT Delta Store (Semantic Search)` as query-serving execution state that becomes eligible for downstream visibility evaluation under the committed-manifest frontier, not as a writer-local public visibility boundary.

## 6. Watermarks and Progress Reporting

SSW must use canonical progress vocabulary.

For semantic execution, the relevant reported frontier is:
- `applied_ts_semantic_search`

Collection-visible safety is derived downstream as:
- `safe_ts = min(present enabled modality applied ts)`

The public/readiness watermark contract remains downstream and collection-scoped:
- `EngineWatermarks` returns every modality enabled on the target collection, even when the latest admitted write touched only semantic search and left another enabled modality unchanged
- writer-local completion signals and `ProgressMark.watermarks` are narrower modality-scoped advance inputs, so omission means "this mark does not advance that modality", not "the collection lacks that modality"

SSW must not redefine those concepts as writer-local public semantics such as `applied_ts_nrt`, `applied_ts_graph`, or `applied_ts_durable`.

Related durable-manifest metadata remains canonical:
- `applied_ts_*`
- `cdc_resolved_ts`
- `wal_replay_start_anchor`
- `wal_durable_anchor`

## 7. Scheduling and Affinity

SSW may use collection affinity to improve cache locality for:
- vector-schema caches,
- quantization/runtime tuning state,
- mutable graph state,
- in-memory batching,
- local flush buffers.

Reasonable implementation techniques include:
- consistent hashing on `(namespace_id, collection_id)`,
- virtual nodes for long-tail balance,
- dedicated pools for whale tenants.

These are scheduler concerns only. They must not:
- create collection-local durable key namespaces that bypass canonical storage layout,
- change WAL/chunk identity,
- require `ssw_instance_id` in canonical object identity,
- introduce cross-collection shared immutable objects unless the canonical manifest/lifecycle/GC model is extended to own that coupling explicitly.

## 8. Storage and Spill Behavior

Local NVMe may be used only as ephemeral execution spill/cache:
- absorb temporary object-store latency,
- bound memory pressure during flush preparation,
- stage local files before immutable object upload,
- keep short-lived mutable semantic frontier state that can be reconstructed from canonical durable state.

Because S3 plus canonical manifest/WAL anchors remain the only durable source of truth:
- local spill state is disposable,
- recovery must rely on canonical WAL/manifests/anchors,
- no local spill artifact may become an independent durability authority.

Backpressure policy may escalate by memory or spill pressure, but external failure projection must still follow canonical ingest/error semantics.

## 9. Failure Handling

SSW failure handling must preserve owner boundaries:
- WAL-durable retry and replay remain bounded by canonical replay anchors.
- Write-side `wait_for_visible` deadline expiry is not a writer-defined visibility timeout; it remains a stream-level ingest failure projected as retryable `SF_BACKEND_UNAVAILABLE`.
- Query/read-side snapshot waiting is the only domain that uses `SF_VISIBILITY_TIMEOUT`.
- Heavy rebuild, compaction rewrite, and object reclamation remain background-plane responsibilities.

Crash recovery must be pointer-driven:
- load the committed manifest lineage,
- load `wal_replay_start_anchor` and `wal_durable_anchor`,
- replay only the canonical retained WAL window,
- ignore orphan/unanchored tails.

Any internal replay warmup or local mutable-state rebuild may inform broker/reader-local health or scheduling, but it must not redefine public readiness state or gateway-visible query-routing semantics on its own.

Implementation tuning may still vary for chunk seal thresholds, mutable-frontier memory budgeting, semantic micro-segment flush sizing, graph-maintenance heuristics, quantization strategy selection, and fair scheduling under whale and long-tail coexistence, but any such choice must preserve the canonical ingest, replay, durability, visibility, and vector-schema contracts described above.