# Commit Coordinator Date: 2026-03-23 Status: Frozen architecture note Authors: Search Platform Architecture Component: Write Plane (Ordering and Watermark Authority) Scale target: 1B collections | shared Flow-A / Flow-B commit ordering ## 1. Scope This document defines the component-level architecture of `Commit Coordinator`, the stateful high-availability write-plane authority that sits after `Write Planner` and before modality writers, `Visibility Barrier`, and `Manifest Publisher`. It applies to canonical write traffic that has already passed upstream admission: - Flow A: `TiCDC -> CDC Adapter -> Ingest Gateway -> Ingest Broker -> Write Planner -> Commit Coordinator` - Flow B: Search-Fabric-owned write gateways -> `Ingest Gateway -> Ingest Broker -> Write Planner -> Commit Coordinator` The canonical contract stack remains: - `api/proto/searchfabric/v1/search_fabric.proto` - internal owner file for this boundary: `api/proto/searchfabric/v1/commit_coordinator.proto` - `docs/specs/core/search-fabric-api.md` - `docs/specs/core/error-codes.md` - `docs/specs/core/search-fabric-architecture.md` This document does not redefine public `MutationBatch`, `MutationAck`, `RowMutation.commit_ts`, or `safe_ts` semantics. It explains how the coordinator should realize shared timestamp ownership, deterministic ordering, watermark tracking, and recovery at production scale while remaining subordinate to the checked-out canonical owners. Out of scope: - public authentication, schema admission, and route normalization owned by `Ingest Gateway` - Flow-A source extraction, checkpointing, and DDL fence orchestration owned by `CDC Adapter` - frame building and planner-side modality selection owned by `Ingest Broker` and `Write Planner` - manifest mutation and committed-pointer CAS owned by `Manifest Publisher` - public ACK translation owned by `Ingest Gateway` ## 2. Role and Boundary `Commit Coordinator` is the write-plane authority for three things: - preserve Flow-A source `commit_ts` - inject Flow-B canonical `commit_ts` - maintain collection-local modality progress used to derive `safe_ts` It is intentionally stateful. Unlike `Ingest Broker`, which is a disposable memory stabilizer, the coordinator owns replay-critical ordering state and fencing state for active write shards. Its responsibilities are: - receive planner-classified canonical write work on a collection-local lane - preserve Flow-A timestamps and inject Flow-B timestamps from shared TiDB TSO - enforce deterministic same-`commit_ts` ordering with the canonical tie-break tuple (`source_partition`, `source_epoch`, `source_seq`) - split internal work below the public batch boundary only when required by checked-out canonical semantics - register row-bearing write intents with `Visibility Barrier` - maintain per-collection `local_max_commit_ts`, `applied_ts_*`, and derived `safe_ts` - surface lag and pressure signals to the broader admission-control loop - rebuild replay/watermark state after failover using durable anchors Its non-responsibilities are: - validating collection ownership, lifecycle, or active schema against CCS - inventing a second public write envelope - mutating manifests directly - becoming a second durable source of truth outside manifest anchors and the existing canonical replay contract High-level data path: ```text Write Planner -> Commit Coordinator -> TSO Batcher / Collection Lane State -> Metadata Stream Writer -> Semantic Stream Writer -> Metadata Batch Writer -> Semantic Batch Writer -> Visibility Barrier -> Manifest Publisher ``` Important boundary: - public `MutationBatch` remains the admission and ACK unit - coordinator-private `BatchSlice` is an internal execution unit only - `BatchSlice` exists because the checked-out public contract allows one canonical batch to contain multiple Flow-A row `commit_ts` values ## 3. Design Rules 1. Canonical-contract first: - Flow A keeps source `RowMutation.commit_ts` - Flow B sends `RowMutation.commit_ts=0` - only `Commit Coordinator` injects Flow-B timestamps before persistence/apply ordering 2. One global TSO domain: - injected and preserved timestamps all live on the shared TiDB TSO axis - whenever lag is compared to millisecond SLOs, implementations must use `tso_physical_ms(ts)` 3. No new public batch semantics: - coordinator may split internal execution into `BatchSlice` units - it must not rewrite `batch_id`, `batch_fingerprint`, `stream_identity`, or source ordering 4. Marks remain sequencing fences: - `ProgressMark` and `BarrierMark` stay in the same collection lane - they do not become synthetic row-mutation write intents - they must never be reordered ahead of earlier admitted row work 5. Collection-local affinity is mandatory: - all work for one `(namespace_id, collection_id)` should land on the same active coordinator shard while the shard is healthy - correctness may survive failover, but healthy-path routing should avoid cross-shard timestamp races 6. Fail closed on authority uncertainty: - if the coordinator cannot obtain fenced TSO ranges, restore active lane state, or trust the current leader epoch, it must stop admitting fresh work rather than speculating ## 4. Commit Lifecycle ### 4.1 Collection-Local Ordering Lane Each active collection is handled on one logical coordinator lane. Recommended lane state: ```text CollectionLaneState = local_max_commit_ts + applied_ts_metadata_search + applied_ts_semantic_search + optional applied_ts_metadata_aux + pending_row_slices + last_access_unix_ms + hot-tier residency metadata ``` The lane is the serialization point for: - Flow-A timestamp preservation - Flow-B timestamp injection - same-collection same-`commit_ts` tie-breaking - watermark advancement `Write Planner` still decides which modalities a batch touches. `Commit Coordinator` does not redo that classification. It takes the planner output, finalizes row-order timestamps, and turns row-bearing work into barrier-trackable write intents. ### 4.2 Flow A Preservation, Flow B Injection, and the Monotonicity Guard Flow handling is intentionally asymmetric: - Flow A (`TiCDC`): - row mutations arrive with authoritative source `commit_ts` - coordinator performs pass-through validation and advances `local_max_commit_ts` - Flow B (native search APIs and TiDB remote DML): - row mutations arrive with `commit_ts=0` - coordinator obtains TSO values from an asynchronous local batcher - all row mutations in one Flow-B row slice receive the same injected `commit_ts` To prevent a Flow-B write from being assigned a timestamp older than an already processed Flow-A event on the same collection lane, the injected timestamp is: ```text commit_ts_injected = max(local_max_commit_ts + 1, next_batched_tso) ``` Recommended implementation: - one shard-local `tokio::sync::mpsc` request channel gathers TSO demand from collection lanes - the batcher fetches a TSO range in one TiDB RPC - the batcher hands out monotonic values from that local range - after assignment, the lane advances `local_max_commit_ts` This keeps public Flow-B semantics unchanged while avoiding one TiDB TSO RTT per small write. ### 4.3 Deterministic Ordering and Internal `BatchSlice` The checked-out public contract allows one canonical `MutationBatch` to carry multiple Flow-A row `commit_ts` values. That means an internal carrier keyed only by whole-batch identity plus one `commit_ts` would be ambiguous. The coordinator therefore uses one internal `BatchSlice` as the execution and barrier-registration unit: - slice scope is always one original canonical batch - slices preserve original `entry_index` and `source_seq` ranges - every row-bearing slice has exactly one authoritative `commit_ts` Recommended slicing rules: - Flow-B row-bearing batches normally map to one row slice because the coordinator injects one `commit_ts` per admitted row slice - Flow-A batches are split only when contiguous row regions carry different `commit_ts` - `ProgressMark` and `BarrierMark` force slice closure but remain coordinator-local fence/progress events rather than barrier-tracked row writes Conflict resolution remains canonical: - highest `commit_ts` wins - same-`commit_ts` conflicts use canonical tuple order: - `source_partition` - `source_epoch` - `source_seq` - source kind (`TiCDC` vs native-search API, streaming vs batch writer) is not a tie-break dimension ### 4.4 Writer Fanout and Intent Registration For each row-bearing slice: 1. coordinator finalizes the authoritative `commit_ts` 2. coordinator constructs one internal `WriteIntent` 3. coordinator registers that intent with `Visibility Barrier` 4. coordinator dispatches the slice to the planner-selected touched modalities The intent contains only the information needed for cross-modality durability aggregation: - original batch identity - slice entry/source-seq range - authoritative `commit_ts` - touched modalities - fencing epoch - visibility/durability policy copied from the canonical batch This keeps the barrier contract narrow. `Visibility Barrier` does not need the full row payload to wait for durable completion. ## 5. Watermark Tracking and 1B-Collection State Management ### 5.1 Collection Progress Model The coordinator maintains per-collection modality progress: - `applied_ts_metadata_search` - `applied_ts_semantic_search` - optional `applied_ts_metadata_aux` Public/readback projection rule: - `EngineWatermarks` returns every modality enabled on the target collection after deriving the current collection frontier - `ProgressWatermarks` is narrower: omission means "this progress mark does not advance that modality" Derived rule remains canonical: ```text safe_ts = min(present enabled modality applied ts) ``` `safe_ts` is never caller-supplied. `ProgressMark` remains a per-modality progress input only. Mark handling rules: - `ProgressMark` may advance modality progress for the collection lane - `BarrierMark` is a sequencing fence and drain point - neither mark type creates a synthetic row-bearing publish intent ### 5.2 Hierarchical State Tiers The raw per-collection state footprint is too large for an always-hot single-node map at `1B` collection scale. The coordinator should therefore use three state tiers: ```text Hot tier (RAM) active collections touched in the recent window -> sharded map + LRU/TinyLFU-style eviction metadata Warm tier (local NVMe / RocksDB) recently evicted lane state -> low-latency reactivation without S3 round trip Cold tier (S3 / committed manifests) authoritative reconstruction inputs -> latest manifest watermarks + WAL anchors ``` Recommended hot-tier contents: - `local_max_commit_ts` - modality `applied_ts_*` - wheel-slot pointer / eviction metadata - pending intent count - last access timestamp Recommended warm-tier contents: - the same lane state serialized in a compact form - enough recovery metadata to re-open a collection lane without consulting all historical WAL objects Cold-tier reconstruction inputs remain the existing authoritative ones: - the latest committed manifest lineage - `wal_replay_start_anchor` - `wal_durable_anchor` The coordinator does not introduce a second S3 truth source for visibility. ### 5.3 Timer-Wheel Acceleration and Time Compression A coordinator shard still needs fast answers to questions such as: - which active collections are oldest / most lagged - which hot entries should be evicted - which lanes crossed the lag SLO Full scans over every active collection do not scale cleanly. The recommended accelerator is a shard-local hierarchical timer wheel (HTW): - slots are keyed by physical TSO millisecond windows - each hot collection stores a pointer to its current slot - slot aggregates are used for lag scanning, eviction, and oldest-active detection Important boundary: - the HTW is an accelerator, not the source of truth for per-collection `safe_ts` - authoritative per-collection watermarks still live in the collection lane state To reduce memory cost further, timestamp fields should be stored as deltas from a shard-local base TSO: - hot-tier representation may keep small deltas for the common case - warm-tier serialization may use varint/delta encoding - periodic rebasing is allowed as a local implementation detail ### 5.4 Visibility Barrier Linkage and ACK Gates `Visibility Barrier` keeps one pending-write aggregation record per row-bearing `WriteIntent`. Recommended barrier state: - `write_intent_id` - touched modality set - completed modality set - per-modality durable anchors - latest derived watermarks after the final modality completes Completion rules: - metadata-only writes wait only for metadata search durability - semantic-only writes wait only for semantic search durability - mixed writes wait for every touched modality - optional `metadata_aux` participates only when the collection/write path actually touches it Once the final touched modality reports durable completion: - barrier derives the new collection watermarks - barrier emits one publish-ready intent to `Manifest Publisher` - `Manifest Publisher` still owns `manifest_vN+1` allocation and committed-pointer CAS ACK timing remains the public contract: | Policy combination | Internal gate | |---|---| | `durability=INGEST_ACK`, `visibility=ASYNC` | coordinator admitted and dispatched the work | | `durability=WAL_CHUNK_DURABLE`, `visibility=ASYNC` | touched modalities reached durable anchors | | `visibility=WAIT_FOR_VISIBLE` | manifest publish completed after barrier aggregation | ## 6. Backpressure and CCS Linkage ### 6.1 Lag Detection The coordinator should monitor lag on the same physical-ms basis used elsewhere in the checked-out architecture: ```text lag_ms = tso_physical_ms(latest_committable_ts) - tso_physical_ms(safe_ts) ``` This signal is meaningful both per collection and per coordinator shard. If lag exceeds the configured SLO for a sustained window, the coordinator should emit an internal pressure signal to the CCS-owned admission-control loop. Recommended downstream behavior: - CCS tightens or withholds additional ingest lease issuance for the affected namespace / dimension - upstream gateways surface the canonical public rejection as `429 SF_RATE_LIMITED` - required public detail keys remain the existing canonical ones: - `retry_after_ms` - `throttled_dimension` - `server_limit` This document does not introduce a new public lag-specific error code. ### 6.2 Pressure Sources and Classification Typical coordinator pressure sources: - TSO service RTT expansion or temporary unavailability - hot-tier memory pressure - writer queue saturation - long-tail durability lag in one modality - manifest publish backlog Internal classification should stay compatible with the checked-out public error families: - adaptive shedding / bounded queue saturation -> `SF_RATE_LIMITED` - missing required dependency such as TSO or durable warm-tier state -> retryable backend/catalog unavailable path, depending on the failing authority The coordinator itself is not the public error owner. `Ingest Gateway` remains responsible for projecting canonical public failures. ## 7. Failure Recovery and HA ### 7.1 Fenced Leadership Each coordinator shard runs under leader election with a fenced `coordinator_epoch`. Required properties: - only the active leader may allocate Flow-B timestamps - every registered write intent carries the leader epoch that created it - stale leaders cannot continue durable state mutation after failover The coordinator must restore its last issued logical watermark before admitting fresh Flow-B work so duplicate timestamp allocation does not occur across failover. ### 7.2 Durable Anchors and Replay Window Recovery Recovery uses the existing anchored replay model: - modality writers publish durable anchors for committed WAL/object state - manifests persist `wal_replay_start_anchor` and `wal_durable_anchor` - restart replays only the anchored window needed to rebuild unflushed state Operational rule: - orphan WAL tails without authoritative anchor coverage are ignored - replay relies on canonical entry/batch identity, not on best-effort local buffers Exactly-once logical effects come from the combination of: - canonical replay identity - fenced coordinator leadership - writer durable anchors - manifest-committed visibility boundary ### 7.3 Lazy Loading and Fast Start A full eager reload of all collection lanes is not acceptable at `1B` collection scale. Recommended restart sequence: 1. restore shard routing and leader epoch 2. reload pending intents and the minimal last-issued timestamp watermark 3. reopen only collections that still have active WAL tails or pending publish work 4. lazily load other collection lane state on first new traffic This keeps failover bounded even when total catalog cardinality is far larger than the active write set. ### 7.4 Failure Cases 1. Leader failover during normal Flow-B traffic: - new leader restores last issued watermark and pending intents - fresh Flow-B admission remains paused until restore is complete 2. Writer durable completion arrives after leader change: - barrier uses fenced intent epoch and idempotent slice identity to reject zombie completion or replay it safely 3. Barrier/publisher lag after durable writers succeeded: - write remains durable but not yet visible - `visibility=WAIT_FOR_VISIBLE` ACK continues waiting on the publish fence - async visibility callers may already have received the earlier durability-gated ACK ## 8. Suggested Runtime Decomposition Recommended Rust-side module split: - `collection_lane_router` - shard ownership, sticky routing, lane lifecycle - `tso_batcher` - batched TiDB TSO acquisition and local range distribution - `lane_state_store` - hot-tier cache, warm-tier RocksDB state, cold reconstruction loader - `batch_slice_builder` - Flow-A split logic, Flow-B injected-timestamp assignment, mark fence handling - `intent_registry` - internal write-intent creation and epoch stamping - `watermark_tracker` - `applied_ts_*`, `safe_ts`, HTW maintenance, lag scanning - `recovery` - manifest-anchor reload, pending-intent replay, epoch restore ## 9. Performance and Concurrency Model Recommended implementation choices: - shard by collection hash so each collection has one active ordering lane - run one lightweight async task per active hot collection only when it has pending work - batch TSO demand per coordinator shard rather than per collection - use bounded channels between coordinator, writers, and barrier so pressure is explicit - co-locate `Commit Coordinator` and `Visibility Barrier` in the same control bundle when possible to reduce intent-registration RTT, while keeping logical ownership separate Tuning directions: - shorter TSO batches reduce injected-timestamp wait but increase TiDB RPC pressure - larger hot-tier windows reduce warm-tier churn but increase RAM cost - tighter barrier/buffer bounds protect tail latency but increase adaptive shedding ## 10. Observability Recommended metrics: - Flow-B TSO batch size, RTT, and local range exhaustion rate - per-collection and per-shard `safe_ts` lag in both raw TSO units and physical milliseconds - hot-tier hit/miss, warm-tier hit/miss, and collection reactivation latency - pending write-intent count and oldest pending intent age - barrier wait time by modality combination - manifest-ready queue depth - leader-epoch changes and recovery duration Recommended log/tracing fields: - `namespace_id` - `collection_id` - `batch_id` - `source_partition` - `source_epoch` - `first_source_seq` - `last_source_seq` - `commit_ts` - `coordinator_epoch` - `lag_ms` These fields are sufficient to correlate write ordering, replay, publish delay, and recovery without inventing a second public protocol vocabulary.