# NRT Delta Store Status: Frozen architecture note Owner: Search Platform Architecture ## 1. Scope This document defines the execution design of `NRT Delta Store`, the short-lived stateful delta tier used by query serving to read committed-but-not-yet-flushed data without waiting for immutable segment flush. This document is subordinate to the canonical owner stack: - machine-readable contract: `api/proto/searchfabric/v1/search_fabric.proto` - internal owner file for this boundary: `api/proto/searchfabric/v1/nrt_delta_store.proto` - ingest/query semantics: `docs/specs/core/search-fabric-api.md` - system topology and state ownership: `docs/specs/core/search-fabric-architecture.md` - canonical error registry: `docs/specs/core/error-codes.md` It must not redefine: - public `MutationBatch` / `MutationAck` semantics, - query-read consistency semantics (`safe_ts`, `effective_read_ts`, continuation snapshot binding), - manifest publication semantics, - WAL anchor ownership (`wal_replay_start_anchor`, `wal_durable_anchor`), - snapshot-sync session lifecycle semantics. ## 2. Canonical Position in the System `NRT Delta Store` sits between streaming writers and query readers: ```text Commit Coordinator -> Metadata Stream Writer -> NRT Delta Store (Metadata Search) -> Semantic Stream Writer -> NRT Delta Store (Semantic Search) Query Broker -> Metadata Search Reader ----\ -> same-snapshot merge -> Semantic Search Reader ----/ ^ | committed manifest frontier ``` It is a shared write/read execution substrate, not a public API surface and not a second visibility authority. `NRT Delta Store` exists so: - streaming writes can become query-visible after committed-manifest publish without waiting for micro-segment flush, - readers can reconstruct one admitted snapshot from immutable objects plus unflushed deltas, - recovery can rebuild only a bounded unflushed window from canonical WAL anchors instead of replaying the full collection history. It does not: - accept source-facing requests, - allocate `commit_ts`, - publish manifests, - decide query admission, - own batch-writer execution, - own snapshot-import upload/cutover flows, - replace immutable segment storage as durable truth. ## 3. Design Anchors The design below is constrained by the following canonical facts: 1. `INGEST_MODE_STREAMING` is the online path that uses stream writers plus `NRT Delta Store`; `INGEST_MODE_BATCH` is a distinct batch lane and does not define an NRT early-visibility surface. 2. Readers may consult NRT only under the same committed-manifest publish frontier; NRT is not an independent publish surface. 3. Query visibility is evaluated against the admitted execution snapshot: - strict and bounded-staleness first-page reads use admitted `effective_read_ts = min(read_ts, safe_ts)` for readiness evaluation, - best-effort first-page reads normally execute at `effective_read_ts = safe_ts`, - continuation reuses token-bound `effective_read_ts` rather than recomputing from the newest `safe_ts`. 4. Query-level snapshot isolation remains: only versions with `commit_ts <= effective_read_ts` and not tombstoned by `commit_ts <= effective_read_ts` are visible. 5. Collection-visible safety remains `safe_ts = min(present enabled modality applied ts)`, emitted only over enabled modalities. 6. WAL replay reconstructs unflushed delta state only inside anchored replay bounds defined by published manifest metadata. Primary source anchors: - `api/proto/searchfabric/v1/search_fabric.proto` - `docs/specs/core/search-fabric-api.md` - `docs/specs/core/search-fabric-architecture.md` - `docs/architecture/commit-coordinator.md` - `docs/architecture/metadata-stream-writer.md` - `docs/architecture/semantic-stream-writer.md` ## 4. Responsibilities `NRT Delta Store` is responsible for: 1. Holding a bounded unflushed delta window for streaming-ingested collection state. 2. Exposing modality-scoped read-path state to metadata and semantic readers. 3. Preserving canonical winner order for unflushed row versions: - `commit_ts` - `source_partition` - `source_epoch` - `source_seq` 4. Participating in downstream visibility/readiness by serving state that corresponds to modality `applied_ts_*` frontiers. 5. Supporting flush handoff so immutable micro-segment production can retire old unflushed state after manifest publish. 6. Supporting restart recovery by rebuilding only the anchored unflushed window from canonical WAL/manifests. `NRT Delta Store` is not responsible for: - schema/latest-generation admission with CCS, - batch-lane immutable artifact build, - public ACK timing, - manifest mutation or CAS publication, - durable replay-anchor movement, - background compaction or GC, - query result ranking/planning. ## 5. State Model ### 5.1 Collection-Scoped, Modality-Scoped Delta Views The store is collection-scoped and modality-scoped: - `NRT Delta Store (Metadata Search)` - `NRT Delta Store (Semantic Search)` Each view keeps commit-ordered unflushed row effects keyed by collection identity and document identity: - `(namespace_id, collection_id, doc_id)` Implementation may physically shard or partition this state, but those local shard identities are observational only. They must not become durable object identity or public replay identity. ### 5.2 Row-Version Semantics For one `doc_id`, the effective unflushed winner remains the canonical last-write-wins winner already defined by admitted write order: 1. highest `commit_ts` 2. canonical tie-breaks: `source_partition`, `source_epoch`, `source_seq` The store therefore needs enough per-row metadata to let readers and recovery logic answer: - which unflushed version is the canonical winner at one admitted snapshot, - whether the row is live or deleted at that snapshot, - which modalities have unflushed state for that row. Delete effects must participate in the same snapshot rule as row updates. They are not a separate relaxed-visibility channel. ### 5.3 Watermark Participation `NRT Delta Store` does not publish `safe_ts` by itself, but it is one substrate from which stream-writer completion contributes to: - `applied_ts_metadata_search` - `applied_ts_semantic_search` - optional `applied_ts_metadata_aux` when that modality exists elsewhere in the collection topology Public/readback `EngineWatermarks` remain collection-scoped downstream products. NRT must therefore avoid inventing writer-local public concepts such as: - `nrt_safe_ts` - `delta_visible_ts` - `flush_safe_ts` ## 6. Write-Side Interaction Model ### 6.1 Streaming Lane Only `NRT Delta Store` is fed by: - `Metadata Stream Writer` - `Semantic Stream Writer` It is not the execution substrate for: - `Metadata Batch Writer` - `Semantic Batch Writer` - `Snapshot Import Worker` That separation is important: - streaming lane needs bounded mutable state so writes can become query-visible before flush, - batch lane prepares immutable publish candidates and stays query-invisible until manifest publication, - snapshot import is session/chunk/cutover based rather than `MutationAck`/per-entry NRT state based. ### 6.2 Admitted Ordering and Apply Discipline Writers must mutate NRT only after admitted ordering is fixed by the canonical ingest path. NRT therefore assumes: - Flow A keeps source `commit_ts`, - Flow B receives canonical `commit_ts` injection before persistence/apply ordering, - `MutationBatch.stream_identity` and canonical source ordering are already fixed upstream. NRT must not contain a second ordering authority that re-arbitrates row winners using local arrival order, worker identity, or local shard ownership. ### 6.3 Durability and Visibility Boundaries NRT mutation alone does not create public durability or visibility. For streaming execution: - `durability_policy=wal_chunk_durable` remains owned by WAL persistence plus anchor persistence, - `visibility_policy=wait_for_visible` remains owned by `Visibility Barrier` plus `Manifest Publisher`, - readers may only consult the NRT window that is admitted under the committed-manifest publish frontier. That means NRT is best understood as query-serving execution state whose contents become readable only when downstream publication has admitted the corresponding write scope. ## 7. Read-Side Snapshot Reconstruction ### 7.1 Same-Snapshot Merge Rule Readers reconstruct one admitted snapshot from two sources: - immutable segment state selected by the committed manifest, - unflushed deltas from `NRT Delta Store` Both sources must be evaluated under the same admitted snapshot boundary. For NRT this means: - include only deltas with `commit_ts <= effective_read_ts`, - exclude rows tombstoned by `commit_ts <= effective_read_ts`, - use the same committed-manifest frontier as the immutable segment reader. NRT lookup must never be keyed to raw request `read_ts` alone, because: - best-effort first-page execution commonly uses `read_ts = 0` but `effective_read_ts = safe_ts`, - continuation uses token-bound `effective_read_ts`, - projected overlay-bearing best-effort execution can use `effective_read_ts = snapshot_read_ts`. ### 7.2 Reader Contract Metadata and semantic readers may use different modality-local execution structures, but they must both observe the same collection snapshot contract: ```text visible(row_version) = published_under_committed_manifest_frontier AND row_version.commit_ts <= effective_read_ts AND no tombstone with commit_ts <= effective_read_ts wins for the same doc_id ``` Any modality-local acceleration structure must remain subordinate to this predicate. ### 7.3 Hybrid Read Coordination Hybrid reads rely on the collection-level `safe_ts` rule: ```text safe_ts = min(present enabled modality applied ts) ``` Because of that rule: - metadata NRT cannot claim a stronger public frontier than semantic NRT for hybrid strict visibility, - semantic NRT cannot advance hybrid strict visibility if metadata is behind, - untouched enabled modalities still matter to `safe_ts` even when the latest write advanced only one modality. This is why the store is split by modality but coordinated by collection-level watermark semantics upstream. ## 8. Flush, Retirement, and Retention ### 8.1 Flush Handoff Streaming writers periodically flush immutable micro-segments to object storage. NRT participates in that handoff as follows: 1. the unflushed window remains query-readable while the flush candidate is being built, 2. downstream publish records the new immutable objects in a new manifest version, 3. committed-pointer CAS makes that manifest reader-visible, 4. only after publication may the corresponding NRT window be retired. Retiring NRT state before committed-manifest publish would break same-snapshot reconstruction for reads that still depend on unflushed deltas. ### 8.2 Bounded Retention Retention is intentionally short: - roughly flush-window scale rather than full-history scale, - bounded by published manifest progress and WAL replay anchors, - constrained by memory and spill budgets. The store must not grow into a second long-lived storage tier. If pressure rises, the system should prefer: - faster flush handoff, - stricter backpressure, - bounded spill, rather than unbounded delta retention. ### 8.3 Batch-Lane Exclusion Batch writers publish immutable outputs directly and do not rely on NRT retention. NRT retirement logic therefore must not wait on batch-lane internal chunk progress or invent mixed streaming/batch retirement semantics. ## 9. Recovery Model Recovery is pointer-driven and bounded: 1. load the reader-visible committed manifest lineage, 2. read `wal_replay_start_anchor` and `wal_durable_anchor`, 3. replay only the retained WAL window needed to reconstruct unflushed deltas, 4. ignore orphan or unanchored tails, 5. rebuild modality-scoped NRT state from canonical replay order. Consequences: - local RAM and local spill are disposable, - NRT is reconstructable rather than independently durable, - incomplete or unpublished flush work must not become visible on restart, - committed-manifest truth wins over any local cache residue. The recovery objective is not instant warm state for every collection. It is correctness first, then bounded incremental warmup. ## 10. Memory, Spill, and Collection Churn `NRT Delta Store` is a stateful hot-path component and must operate under explicit resource discipline. Allowed implementation techniques include: - collection-affine in-memory partitions, - bounded local spill/NVMe staging, - eviction or hydration of cold collection windows, - per-tenant or per-collection memory budgets, - backpressure when flush lag or replay backlog exceeds SLO. Those techniques remain implementation details only. They must preserve: - same-snapshot read correctness, - canonical winner ordering, - pointer-driven recovery, - absence of any local-only visibility truth. In particular: - cold-collection hydration must rebuild from canonical manifest/WAL state, not from speculative local caches, - eviction must never discard still-needed published-frontier deltas without first ensuring they are recoverable from the anchored replay window, - spill files must be safe to delete after crash. ## 11. Operational Signals The component should expose implementation telemetry that supports the canonical system SLOs, including: - per-collection and per-modality unflushed window size, - flush lag relative to published frontier, - replay backlog within anchored window, - `safe_ts` lag and modality frontier skew, - memory pressure and spill utilization, - hydration/rehydration rate for cold collections. These are operational signals only. They do not introduce new public API semantics or new error families. Implementation tuning may still vary for in-memory layout, tombstone-heavy retirement heuristics, collection hydration and eviction policy, spill thresholds, fairness under sustained flush delay, and metadata-aux retention reuse, but any such choice must preserve the canonical ingest, visibility, watermark, manifest, and recovery contracts described above.