# Unified Search Platform: Technical Design Document Date: 2026-02-16 Status: Frozen (Normative architecture) Authors: Search Platform Architecture ## 1. Executive Summary This document defines Unified Search Platform, whose core architecture layer is Search Fabric, to serve 1B collections (logical tenancy model: 1 user typically maps to 1 isolated collection) over 5 exabytes of raw vector + metadata data. This is a high-level architecture design; normative API design and wire-protocol ownership live in `docs/specs/core/search-fabric-api.md`, while this document may summarize those contracts only insofar as they are needed to explain architectural boundaries and state-flow invariants. Primary goal: maximize storage efficiency by making S3 the only durable source of truth and avoiding replicated full datasets on local NVMe. Core architecture decisions: - Loosely coupled federation with TiDB while keeping independent storage/serving domains. - CQRS split by responsibility: command ingress can come from TiDB/TiKV or native search APIs (governed by collection ownership mode), while query model serving remains Search-Fabric-optimized in decoupled search services. - Shared TiDB Timestamp Oracle (TSO) for globally ordered writes and snapshot reads (`read_ts`). - Search Fabric is a first-class serving layer with native read and write endpoints. - Four query-serving services on the hot path: - Query Gateway (stateless edge): public query/readiness/statistics surface, gateway normalization, and cost-aware admission. - Query Broker (stateless): query planning + result fusion (request-level retrieval groups use canonical fusion semantics; broker-local heuristics are limited to internally synthesized mixed-modality plans). - Metadata Search Reader (stateless compute): metadata + text retrieval. - Semantic Search Reader (stateless compute): semantic vector retrieval. - Query API compatibility profiles are first-class: - Elasticsearch-style native search APIs. - Turbopuffer-style native query APIs. - Unified write-ingress infrastructure with two source channels: - Flow A: TiDB -> TiCDC -> CDC Adapter -> Ingest Gateway. - Flow B: Search-Fabric-owned write gateways (Elasticsearch/Turbopuffer native APIs and TiDB federated SQL remote-write gateway) -> Ingest Gateway. - Near-real-time reads are supported through a short-lived strongly consistent delta tier (`NRT Delta Store`) that readers may consult under the same admitted published frontier, so visibility does not wait for segment flush. - Background-heavy tasks (compaction and garbage collection) run in an isolated Background Compute Plane to protect ingest/query latency. - Query-side TiDB integration is intentionally constrained to two mechanisms only: - Search-Fabric-side auxiliary indexes accelerate flexible queries over combinations of structured fields, especially ad-hoc predicate sets that are impractical to cover with fixed secondary indexes. - Search Fabric collections exposed to TiDB as federated tables for federated SQL. - Unified MVCC model across metadata search and semantic search data with snapshot isolation at `read_ts`. - Streaming writers (WAL-based) and batch writers (offline SST/segment build) coexisting on the same S3 object namespace, with conflict policy: highest `commit_ts` wins, and same-`commit_ts` conflicts follow canonical mutation order (`source_partition`, `source_epoch`, `source_seq`). This design keeps query-serving compute stateless, constrains durable state to immutable S3 objects plus the explicitly declared stateful control/NRT components, and scales through object-based LSM compaction and per-tenant isolation manifests. ## 2. Goals, Non-Goals, and Constraints ### 2.1 Goals - Serve hybrid queries (vector + text + metadata) with predictable latency at hyperscale. - Make serving performance goals first-class design contracts with explicit, measurable targets for latency, throughput, and freshness. - Support strict snapshot reads (`read_ts`) with global ordering via TiDB TSO. - Preserve logical tenant isolation (`namespace_id/collection_id`) for 1B collections. - Minimize storage duplication via index-as-storage. - Support both low-latency streaming writes and high-throughput batch ingestion. - Provide first-class native retrieval APIs for both read and write paths. - Support popular search query/write API styles (Elasticsearch and Turbopuffer) as compatibility subsets. - Expose native search write APIs for non-TiDB users while reusing the same ingestion/commit infrastructure for TiCDC-derived data. - For MySQL compatibility profiles, keep normative applicability on collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL` unless a phase-scoped addendum explicitly expands scope per `docs/governance/roadmap.md` Section `3.1`. ### 2.2 Non-Goals - Full OLTP semantics replacement for TiDB. - Cross-tenant transactional joins inside retrieval engine. - Dependence on always-hot NVMe replicas for correctness. - Normative API design, request/response schemas, and protocol details as source-of-truth definitions (see `docs/specs/core/search-fabric-api.md`). ### 2.3 Hard Constraints - Data scale: 5 EB raw vectors + metadata/text. - Collection scale: 1B isolated collections under multi-tenant governance. - S3 is the sole durable source of truth. - NVMe is ephemeral cache only. ### 2.4 Architectural Stance: CQRS - This system is a strict responsibility-split CQRS architecture: - Command side: command ingress is selected by the authoritative control-plane `collection_truth_mode`; `COLLECTION_TRUTH_MODE_EXTERNAL` collections admit TiDB/TiKV-derived projection through `TiCDC -> CDC Adapter`, while `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` collections admit Search-Fabric-owned write gateways through `Ingest Gateway` (native search APIs and TiDB federated SQL remote-write gateway). Accepted commands are normalized into one deterministic canonical mutation order stream: (`commit_ts`, `source_partition`, `source_epoch`, `source_seq`). - Query side: Retrieval domain is the probabilistic read model optimized for relevance and Top-K latency. - Query model is populated by projection pipelines: - Flow A (`TiCDC -> CDC Adapter`) for ledger-derived changes. - Flow B (Search-Fabric-owned write gateways, including native search APIs and TiDB federated SQL remote DML) for retrieval-owned data paths requiring lower-latency indexing. - Both flows are normalized into one internal write envelope and processed by the same ingest/commit pipeline. - CQRS contract: - command truth is immutable from the query side. - query side may be stale, but staleness is explicit and bounded by `safe_ts`/SLO. - global ordering is preserved by shared TSO (`commit_ts`, `read_ts`). CQRS consequence: - Do not introduce cross-system 2PC between TiDB and retrieval storage. - Correctness is maintained through idempotent projection, replayability, and timestamp-based visibility, not distributed locking. ### 2.5 TiDB Integration Boundary (Query Direction) In query direction, integration between retrieval and TiDB is restricted to exactly two patterns: 1. Auxiliary index acceleration: - TiDB SQL queries can use auxiliary indexes maintained in retrieval to accelerate flexible multi-field predicate combinations that are not practical with fixed TiDB secondary indexes. 2. External table federation: - Retrieval-managed collections can be registered as TiDB federated tables and queried through federated SQL plans. Anything beyond these two patterns is out of scope for TiDB-query integration. Compatibility scope note: - MySQL compatibility profile commitments are normative for collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`. - `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` collections are architecture-supported in this TDD, but are out of MySQL compatibility profile commitments unless explicitly declared by phase-scoped conformance addendum per `docs/governance/roadmap.md` Section `3.1`. - Elasticsearch compatibility profile commitments are normative for collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`. - For collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`, Elasticsearch-style write APIs are out of scope by default and MUST be rejected with `409 SF_OWNERSHIP_CONFLICT` unless a phase-scoped conformance addendum explicitly expands scope per `docs/governance/roadmap.md` Section `3.1`. - For collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`, Elasticsearch-style read/query APIs are also outside default profile-conformance scope; if exposed as convenience routing they are non-normative and MUST NOT be counted in profile/phase acceptance unless explicitly included by a phase-scoped conformance addendum per `docs/governance/roadmap.md` Section `3.1`. - TiDB enhancement profile commitments are normative for collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL` unless a phase-scoped conformance addendum explicitly expands scope per `docs/governance/roadmap.md` Section `3.1`. - `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` is architecture-supported in this TDD for federated reads plus the active statement-scoped TiDB remote-DML subset, but remains outside full TiDB enhancement profile parity unless a phase-scoped conformance addendum explicitly expands scope per `docs/governance/roadmap.md` Section `3.1`. - Canonical `SF_*` error-code definitions are centralized in `docs/specs/core/error-codes.md`; this TDD only references those canonical codes in flow contracts. ### 2.6 Terminology Glossary (Quota, Admission, and Throttling) - `Quota`: long-horizon resource budget and hard limits per tenant/collection/tier (for example cost, rate, retention, and capacity envelopes). - `RCU Admission`: query-path pre-execution gate that estimates read cost in `RCU` and decides allow/reject/explicit-partial-execution before running the query. It MUST NOT silently weaken strict-mode semantics; any non-strict degradation must be surfaced explicitly in the response/fallback contract. - `Tenant/API Throttling`: gateway-level runtime rate/cost enforcement (distributed token buckets, admission shedding, and runtime cost-governor pre-emption), surfaced as `429 SF_RATE_LIMITED` when that control loop rejects or aborts work. Query Broker MAY still perform shard-local abort/pre-emption for already-admitted work to protect hot shards, but that is not a second public quota authority. - `S3 Throttling`: object-store-side protection feedback (for example `503 Slow Down` and per-prefix throughput limits) handled by internal backoff/retry/pacing; this is not tenant/API throttling. ## 3. System Architecture ### 3.1 High-Level Topology ```text Public/API Edge (external-ingress view; trusted internal query callers also call Query Gateway directly under trusted service identity; the Query Gateway component hosts `QueryGatewayService` plus `RelationalQueryService`, and trusted callers use one of those service surfaces according to route) Query Clients -----------------------------------------------> Unified Edge/Auth Ingress -> Query Gateway component (`QueryGatewayService` + `RelationalQueryService`) -> Query Broker -> Metadata Search Reader -> Semantic Search Reader Trusted Internal Query Callers (for example TiDB Planner/Executor) --> Query Gateway component (`QueryGatewayService` + `RelationalQueryService`) Control Clients ---------------------------------------------> Unified Edge/Auth Ingress ----------------> Catalog Control Service (public catalog/control API owner) Control Clients ---------------------------------------------> Unified Edge/Auth Ingress ----------------> IdAllocatorService (public allocator API owner) IdAllocatorService ------------------------------------------> Catalog Control Service (private admission-state reads via machine-readable owner `catalog_control_private.proto`: lifecycle/truth-mode/doc-id policy/schema-generation fence) Query Gateway component -------------------------------------> Catalog Control Service (internal descriptor/name-resolution reads + private CCS admission-state reads via machine-readable owner `catalog_control_private.proto` + CCS-owned `GetCollectionStatistics` helper path) Query Gateway component -------------------------------------> Query Broker (query execution + broker-owned readiness lane) Query Broker ------------------------------------------------> Query Gateway component (same internal stream terminal receipt by `stream_attempt_id`; no reverse callback service) Query Broker ------------------------------------------------> Catalog Control Service (private serving-readiness lookup) Metadata Search Reader + Semantic Search Reader -> Local Cache -> Object Store Metadata Search Reader + Semantic Search Reader -> NRT Delta Store (Metadata Search/Semantic Search unflushed deltas, commit-ordered) Ingest Plane (Write Control + Execution) Write Clients -----------------------------------------------> Unified Edge/Auth Ingress -> Ingest Gateway -> Ingest Broker -> Write Planner TiDB DDL / Schema Manager ------------------------------------> Catalog Control Service (`PutNamespaceMetadata` / `PutCollectionMetadata` / `PutCollectionSchema` / `DropCollection`) TiDB/TiKV + TSO + TiCDC -> CDC Adapter -> Ingest Gateway TiDB Federated SQL Write Gateway -----------------------------> Ingest Gateway Ingest Gateway ---------------------------------------------> Catalog Control Service (ownership/schema/policy check) Ingest Gateway ---------------------------------------------> IdAllocatorService (pre-ingest create/insert identity resolution on allocator-admitted collections) Ingest Gateway ---------------------------------------------> Current-State Reader (private point-read/OCC; machine-readable owner `current_state_reader.proto`) Ingest Gateway ---------------------------------------------> Query Broker (private selector materialization only) Write Planner -> Commit Coordinator (preserve Flow-A `commit_ts`, inject Flow-B `commit_ts`, finalize write order) Commit Coordinator -> Metadata Stream Writer -> NRT Delta Store (Metadata Search) Commit Coordinator -> Semantic Stream Writer -> NRT Delta Store (Semantic Search) Commit Coordinator -> Metadata Batch Writer Commit Coordinator -> Semantic Batch Writer Metadata/Semantic Stream and Batch Writers -> Object Store (SST/Segments/Bitmaps) Metadata/Semantic Stream and Batch Writers -> Visibility Barrier -> Manifest Publisher (committed-pointer visibility publish) Manifest Publisher -> Catalog Control Service (advance manifest head pointer) Background Compute Plane (Compaction + GC) Background Compute Coordinator -> Compaction Workers -> Object Store -> Background Compute Coordinator -> Manifest Publisher (CAS swap) Background Compute Coordinator <-------------------------------- Catalog Control Service (collection lifecycle and policy watch) Background Compute Coordinator -> GC Workers -> Object Store (DeleteObjects, throttled/backoff) ``` Read query execution never bypasses `Query Broker`: whether admission entered through `QueryGatewayService` (`ExecuteQuery` / `StreamQuery`) or `RelationalQueryService` (`ExecuteProjectedQueryOrFallback` / `ExecutePrimaryQuery`), admitted execution flows `Query Gateway component -> Query Broker -> Metadata Search Reader/Semantic Search Reader`. `GetReadiness` also enters through `Query Gateway component -> Query Broker` and joins CCS only on the broker-owned serving-readiness lane; the direct `Query Gateway component -> Catalog Control Service` edge covers internal descriptor/name-resolution reads, private CCS admission-state reads, and the CCS-owned `GetCollectionStatistics` helper path. TiDB production aux-index acceleration also enters through Query Gateway, but it uses a TiDB-scoped fast path that either returns an explicit native-plan fallback directive or dispatches admitted execution into the same `Query Broker` path. Only data-plane document mutation traffic goes through `Ingest Gateway`, `CDC Adapter`, `Ingest Broker`, and `Write Planner`. External/public APIs reach service owners through a unified edge/auth ingress that terminates authn/authz, routing, audit, and rate-governor policy without changing canonical service ownership. Public catalog reads and public catalog/schema/lifecycle mutations belong directly to `Catalog Control Service`; allocator-backed document-ID issuance belongs directly to `IdAllocatorService`; `QueryGatewayService` and `IngestGatewayService` are not public facades for those RPCs. Catalog/schema/lifecycle mutations use `Catalog Control Service` directly and are not part of the ingest lane. Allocator-backed document-ID issuance is a pre-ingest control-plane step: create/insert callers on collections whose `DocIDMapping.generation_policy=DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL` first resolve collection identity, then call `IdAllocatorService.AllocateIds` with one source-ordered mixed-mode allocation plan (`plan.items[]`) before constructing canonical `RowMutation.doc_id` values; omitted-ID rows lower to `issue`, explicit-ID rows lower to `explicit_doc_id`, patch/delete/update-style routes remain explicit-identity-only, and ingest never performs hidden server-side allocation. For Flow-A DDL synchronization, a distinct TiDB DDL/schema-sync actor owns the control-plane branch with the canonical caller shape (`CLIENT_KIND_TIDB_DDL_ADAPTER`): first-time `COLLECTION_TRUTH_MODE_EXTERNAL` identity bootstrap uses `CatalogControlService.PutNamespaceMetadata` / `PutCollectionMetadata`, search-affecting schema evolution calls `CatalogControlService.PutCollectionSchema` and allows CDC Adapter to resume after convergence, and destructive DDL calls `CatalogControlService.DropCollection` and requires CDC Adapter to detach/terminate the affected worker rather than resume on a tombstoned lineage. `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` federated namespace/collection identity bootstrap remains a separate control-plane provisioning workflow using `CLIENT_KIND_CONTROL_ADMIN` rather than TiDB hot-path allocation behavior. Flow A (TiCDC) and Flow B (Search-Fabric-owned write gateways, including native search APIs and TiDB federated SQL remote DML) are parallel ingress channels that converge at `Ingest Gateway`, then split by `Write Planner` into metadata search/semantic search execution paths. `Write Planner` hands each admitted write scope to `Commit Coordinator` before writer persistence/apply ordering: Flow A preserves source `commit_ts`, while Flow B receives canonical TSO injection there. For strict or bounded-staleness reads, readers merge two sources at one admitted snapshot boundary: exact caller-requested `read_ts` for admitted strict reads, or admitted `effective_read_ts` for bounded-staleness reads. The two sources are immutable S3-backed segments plus modality-scoped `NRT Delta Store` entries not yet flushed into immutable segments, but admitted only under the same published committed-manifest frontier. Global end-to-end topology (ASCII; external ingress emphasized; trusted internal query callers such as TiDB Planner/Executor also call Query Gateway directly under service identity; the Query Gateway component hosts `QueryGatewayService` plus `RelationalQueryService`, and trusted callers use one of those service surfaces according to route): ```text +--------------------------------------------------------------------------------+ | Control Plane (Stateful) | | Catalog Control Service | IdAllocatorService | Commit Coordinator | Visibility | | | | | Barrier | | Manifest Publisher (CAS + head advance) | +--------------------------------+-----------------------------------------------+ | v +----------------------+ +---------------------------+ +-------------------+ +------------------+ | Query Clients | ---> | Unified Edge/Auth Ingress| ---> | Query Gateway | ---> | Query Broker | +----------------------+ +---------------------------+ | Component | +-----+--------+---+ +----------------------+ | (`QueryGateway- | | | | Trusted Internal | ----------------------------------------> | Service` + | | | | Query Callers | | `Relational- | | | | (for example TiDB | | QueryService`) | | | | Planner/Executor) | +---------+---------+ | | +----------------------+ | | | | internal descriptor /| | | helper facts | | v v v +---------------------------+ +---------------------+ +---------------------+ | Catalog Control Service | | Metadata Search | | Semantic Search | | public catalog/control | | Reader | | Reader | | API owner | +----------+----------+ +----------+----------+ +-------------+-------------+ \ / ^ \ / | v v +----------------------+ +---------------------------+ | Control Clients | ---> | Unified Edge/Auth Ingress| -----> IdAllocatorService +----------------------+ +---------------------------+ | | private CCS admission-state reads | (`generation_policy`, | `allocator_admission_policy`, | `schema_generation_token`) v +---------------------------+ | Catalog Control Service | | private admission-state | | owner for allocator | +---------------------------+ +---------------------+ | NRT Delta Store | | (Metadata/Semantic) | +----------+----------+ | v +---------------------+ | Local Cache | +----------+----------+ | v +---------------------+ | Object Store (S3) | +---------------------+ +----------------------+ +---------------------------+ +-------------------+ +------------------+ +---------------------+ | Write Clients | ---> | Unified Edge/Auth Ingress| ---> | Ingest Gateway | ---> | Ingest Broker | ---> | Write Planner | +----------------------+ +---------------------------+ +----+--------+-----+ +------------------+ +----------+----------+ ^ | \ | | \ private selector materialization +----------------------+ | +---------------------------------------------> Query Broker | TiDB DDL / Schema |------------------------------------------+ | ownership/schema/lifecycle check (control) | Manager |---------------------------------------------------> Catalog Control Service +----------------------+ +----------------------+ TiCDC Flow | | TiDB/TiKV + TSO |-> CDC Adapter ---------------------------> Ingest Gateway +----------------------+ | +-----------------------------------------------> IdAllocatorService | allocator-backed create/insert identity +-----------------------------------------------> Catalog Control Service (private schema/descriptor refresh + fence-observation reads only; not `PutCollectionSchema` / `DropCollection`) v +---------------------+ ---> | Commit Coordinator | | preserve Flow A / | | inject Flow B TSO | +-----+-----------+---+ | | +-------+ +--------+ | | v v +----------------+ +----------------+ | Metadata/Sem | | Metadata/Sem | | Stream Writers | | Batch Writers | +--------+-------+ +--------+-------+ | | +--------------+--------------+ | v +---------------------+ | Object Store (S3) | +----------+----------+ | writer durable completion v +---------------------+ | Visibility Barrier | +----------+----------+ | v +---------------------+ | Manifest Publisher | +----------+----------+ | v Catalog Control Service Write commit control: Write Planner -> Commit Coordinator -> Metadata/Sem Stream Writers + Metadata/Sem Batch Writers -> Visibility Barrier -> Manifest Publisher -> Catalog Control Service Background Compute Plane: Background Compute Coordinator -> Compaction Workers -> Object Store -> Background Compute Coordinator -> Manifest Publisher -> Catalog Control Service Background Compute Coordinator -> GC Workers -> Object Store CommitSnapshotSync owner -> Manifest Publisher -> Catalog Control Service ``` Detailed ingest topology rule: Flow-A `TiCDC -> CDC Adapter` reaches `Ingest Gateway` for canonical mutation ingest and MAY use private read-only `Catalog Control Service` refresh/fence-observation paths; `Current-State Reader` remains a Flow-B point-read/OCC helper and is not on the CDC path, and Flow-A does not make `PutCollectionSchema` / `DropCollection` calls itself. ### 3.2 Service Responsibilities Query Broker (stateless): - Interprets admitted canonical query requests and performs physical planning over that already-lowered query intent. - Selects execution strategy (semantic-only, metadata-search-only, or mixed). - Dispatches sub-queries on one shared admitted snapshot boundary. - Resolves query-serving schema lineage against CCS convergence state: execution pins the latest `READY` generation, while a newer active generation that is still `APPLYING` remains control-plane-visible but not yet query-serving-admissible. - Resolves the pinned execution manifest through the private committed-manifest pointer object (`manifest_committed_head.json`); CCS `manifest_head_version` remains derivative governance metadata and is not a Query Broker runtime dependency in the current revision. - Pins `(committed_manifest_version, manifest_checksum_or_etag)` for the full request lifetime and for any derived continuation/search-context chain so one logical query never mixes manifest generations. - Validates ordinary query-page tokens, composite-page tokens, and `search_context.token` locally from their self-contained signed payloads rather than asking CCS to resolve page or PIT state. - Mints keep-alive refresh tokens from the admitted search-context lineage without introducing a separate PIT registry or close-tombstone store in the active revision. - Mints ordinary page, composite page, and search-context continuation tokens from the admitted execution lineage and enforces `max_page_depth` deterministically from the token-bound page ordinal before planning resumed work. - Token payload shape, legality, keep-alive/natural-expiry semantics, and other wire-contract details are owned by `docs/specs/core/search-fabric-api.md`; architecture only depends on the consequence that no separate PIT/page registry is required. - Chooses filtered-ANN execution plans using selectivity-aware CBO when predicates must run before ANN. - Applies the admitted canonical fusion strategy for the active retrieval plan, then performs any admitted broker-local aggregation reduction before final page slicing/token minting. Request-level `RetrievalGroup.fusion_strategy=UNSPECIFIED` still means canonical `WEIGHTED_SUM`; when the broker internally synthesizes separate metadata-search and semantic-search subplans from one admitted retrieval intent, it MAY use broker-local heuristics such as Reciprocal Rank Fusion (RRF) only as physical-plan choices that do not replace request-level fusion semantics. - Enforces tenant/collection isolation and query budget limits. - MAY serve private selector-materialization requests for Flow-B by-query lanes through a dedicated internal selector-materialization RPC, but MUST NOT be the write-authoritative source for single-document current-version/OCC decisions. - Emits final streamed-query execution receipts on the same internal broker stream surface rather than through a reverse callback service: `QueryBrokerStreamEvent.public_frame` carries the public frame contract, `QueryBrokerStreamEvent.terminal_receipt` carries exactly one final `QueryBrokerStreamExecutionReceipt` per admitted attempt, and Query Gateway applies deterministic fallback reconciliation only when the internal stream ends before that terminal receipt is observed. The receipt remains stage-aware: attempts that already established `PinnedSnapshot` use the full `receipt` branch carrying `QueryBrokerExecutionReceipt`, while fatal attempts classified before pinning completes use the explicit `early_failure` branch. TiDB DDL / Schema Manager (external control-plane caller): - Detects search-affecting TiDB DDL and classifies it into schema-evolution vs destructive-drop control branches. - Calls `CatalogControlService.PutCollectionSchema` for search-affecting schema-evolution DDL and `CatalogControlService.DropCollection` for destructive DDL that removes the Search Fabric collection lineage, always with the canonical TiDB DDL caller role. - Owns DDL session-context capture and schema-sync replay inputs required by the canonical API contract. - Captures the replay-critical schema-admission context at DDL admission time and persists it into `CollectionSchemaDefinition.replay_context` on `PutCollectionSchema`. In the current revision this typed carrier holds `sql_mode` and `time_zone`; parser/tokenizer semantics must already be canonicalized into typed schema fields, while source-round-trip charset/collation metadata remains supplemental `FieldSchema.options` / `CollectionSettings.options` data. - Is distinct from CDC Adapter: it advances control-plane schema lineage, while CDC Adapter fences data emission around that convergence boundary. Current-State Reader (private, stateless or tightly coupled internal service; machine-readable owner `api/proto/searchfabric/v1/current_state_reader.proto`): - Serves write-authoritative single-document current-state/current-version lookups for Flow-B point-read/OCC routes. - Returns the current row existence/version marker needed to derive deterministic `expected_commit_ts` and other prewrite route decisions; authoritative source response classification still comes from post-apply ACK outcomes. - Is not a public query API and does not inherit query-plane stale-snapshot semantics. Metadata Search Reader (stateless): - Executes metadata predicates + text ranking from Tantivy-on-S3 segments. - Applies MVCC visibility masking using external visibility bitmaps. - Merges publish-admitted deltas from `NRT Delta Store (Metadata Search)` for `commit_ts <= effective_read_ts` before final top-K return. - Returns `(doc_id, score_metadata_search, commit_ts)` candidates. Semantic Search Reader (stateless): - Runs ANN retrieval on SPFresh graph with RaBitQ compressed vectors. - Applies `effective_read_ts` visibility filtering and tombstone checks. - Applies optional pre-filter masks (PK list / Bloom filter / bitmap) from Query Broker plans. - Merges publish-admitted deltas from `NRT Delta Store (Semantic Search)` so semantic reads do not wait for segment flush. - Returns `(doc_id, score_semantic, commit_ts)` candidates. Ingest Gateway (stateless): - Exposes native search write APIs to external users (Elasticsearch-style and Turbopuffer-style profiles). - Accepts internal TiCDC-derived events via CDC Adapter. - Normalizes all writes into a canonical internal write envelope. - Resolves external route names through typed collection catalog flow before any downstream write/control RPC: - Elasticsearch `{index}` and Turbopuffer route namespace token `{namespace}` first lower to `CollectionNameBinding{namespace_id, collection_name}` inside the already resolved canonical namespace/account scope. - Gateway then uses `GetCollectionMetadata.lookup.name_binding` and `PutCollectionMetadata.name_binding` to resolve or ensure the authoritative `CollectionRef`. - Once resolved, downstream schema, lifecycle, ingest, and snapshot RPCs operate on `CollectionRef`; raw external route tokens do not bypass catalog resolution. - When an admitted create/insert compatibility/profile path targets a `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` collection whose `DocIDMapping.generation_policy=DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL` and whose `CollectionSettings.allocator_admission_policy` publishes `caller_profile=ALLOCATOR_CALLER_PROFILE_ES_WRITE_GATEWAY_ONLY` plus positive `max_plan_items_per_request` / `max_forward_reservation_jump` ceilings, the gateway MUST call `IdAllocatorService.AllocateIds` after collection resolution and before canonical mutation construction so the emitted `MutationBatch` still carries explicit `RowMutation.doc_id` values. - The gateway lowers the admitted source batch into one ordered allocator plan so omitted source-side IDs use `issue` items and caller-supplied create/insert IDs use `explicit_doc_id` items under one replay/schema-fence unit and one authoritative reservation space. - Patch/delete/update-style routes still require explicit identity and MUST NOT call allocator. - The returned allocator-side `schema_generation_token` and signed `allocator_receipt` are carried into the follow-on canonical `MutationBatch`; the receipt lets ingest validate that allocator-backed explicit `RowMutation.doc_id` values came from allocator authority rather than caller injection, and the schema token remains subject to ordinary ingest revalidation. - Routes normalized writes into shared write-control pipeline (`Ingest Broker -> Write Planner`) using shared policy and quota controls. - Enforces mandatory tenant-level write throttling via distributed token buckets. Ingest Broker (write-plane component): - Receives normalized write events from Ingest Gateway. - Performs bounded buffering/admission, coalesces compatible canonical `MutationBatch` payloads into broker-private collection-local dispatch frames, and schedules those frames with namespace-aware fairness before dispatch. - Does not participate in query/read execution path. Write Planner (write-plane control): - Receives broker-private dispatch frames from Ingest Broker while preserving the enclosed original canonical batch boundaries. - Expands each original batch into metadata search/semantic search tasks according to payload modality (`metadata/text` vs `vector`). - Selects execution class per task (`streaming` or `batch`) under shared policy. - Dispatches to `Metadata/Semantic Stream Writer` or `Metadata/Semantic Batch Writer` with shared canonical stream/batch identity (`stream_identity`, `batch_id`, `source_seq`) and `commit_ts` context, and completion still fans back one canonical `MutationAck` per original batch. Metadata Stream Writer (write-plane executor): - Applies interactive metadata/text mutations. - Appends metadata search WAL and writes commit-ordered deltas to `NRT Delta Store (Metadata Search)`. - Seals metadata search WAL chunks by policy and uploads immutable chunk objects to Object Store. - Flushes metadata search micro-segments to Object Store. Semantic Stream Writer (write-plane executor): - Applies interactive vector/tombstone mutations. - Appends semantic WAL and writes commit-ordered deltas to `NRT Delta Store (Semantic Search)`. - Seals semantic WAL chunks by policy and uploads immutable chunk objects to Object Store. - Flushes semantic micro-segments to Object Store. Metadata Batch Writer (write-plane executor): - Builds metadata search immutable segments offline from already-admitted metadata batch slices while preserving original canonical batch lineage for downstream ACK reconstruction. - Uploads metadata search artifacts/range metadata to Object Store. Semantic Batch Writer (write-plane executor): - Builds semantic immutable segments/quantized blocks offline from already-admitted semantic batch slices while preserving original canonical batch lineage for downstream ACK reconstruction. - Uploads semantic artifacts/range metadata to Object Store. Query Gateway (stateless): - The Query Gateway component hosts two public query-serving service surfaces: - `QueryGatewayService` for canonical query/readiness/statistics/search-context flows, including Elasticsearch-style and Turbopuffer-style compatibility callers on the native-search edge. - `RelationalQueryService` for trusted relational projected-query and primary-query execution paths that need relational-engine-private execution context. Admission is capability-based; TiDB is the current deployment mapping. - The component therefore owns the public query-serving surfaces `ExecuteQuery`, `StreamQuery`, `GetReadiness`, `GetCollectionStatistics`, and the trusted relational `ExecuteProjectedQueryOrFallback` / `ExecutePrimaryQuery` helpers. - Consumes authenticated caller/scope context from `Unified Edge/Auth Ingress` for external REST/compatibility traffic; it does not own external bearer/API-key termination, edge-mTLS termination, or rate-governor policy. - Treats `StreamQuery` as a dedicated hit-streaming surface: aggregation-bearing requests stay on unary `ExecuteQuery`, while `StreamQuery` is reserved for hit export / transport chunking and reconnect. - Lowers source DSL/options into canonical query semantics and trusted relational route admission inputs; shared query clients dispatch canonical `QueryRequest`, trusted relational callers dispatch route wrappers (`ProjectedQueryRequest` / `PrimaryQueryRequest`) whose inner `query` remains canonical, projected-route fallback selection is resolved at Query Gateway before broker dispatch, and Query Broker owns physical planning/execution strategy selection only after that admission/fallback split is complete. - Resolves Elasticsearch `{index}` and Turbopuffer route namespace token `{namespace}` through the same typed collection catalog flow (`CollectionNameBinding` -> `GetCollectionMetadata`) before issuing query/readback RPCs on a concrete `CollectionRef`. - Resolves/propagates `read_ts` and read mode (`strict`, `bounded-staleness`, `best-effort`). - Performs only pre-admission continuation/search-context coherence checks at the edge; authoritative page/search-context token interpretation, immutable snapshot reconstruction, keep-alive token minting, and request-lifetime manifest pinning remain Query Broker responsibilities. - Dispatches normalized plans to Query Broker and returns profile-shaped responses. - Receives final streamed-query execution receipts from Query Broker on the same internal broker stream surface, reconciles them by `stream_attempt_id`, and uses them for post-stream lease reconciliation and estimator tuning; that receipt may end in internal outcome `ok`, `partial`, canonical `error`, or internal-only `cancelled`, and its execution-context branch is stage-aware (`receipt` after truthful pinning, `early_failure` when the fatal outcome was classified before `PinnedSnapshot` existed). - If a stream attempt reaches a locally observable terminal state (terminal success/error frame consumed, client-side termination classified, or a receipt-only pre-first-frame broker failure is observed) and the internal broker stream ends before the terminal receipt is observed, Gateway MUST run deterministic fallback reconciliation (pessimistic lease release, mark estimator sample incomplete/non-authoritative, emit metric/alert). - Returns explicit caller-native fallback directives only on the dedicated `RelationalQueryService.ExecuteProjectedQueryOrFallback` path; shared `QueryGatewayService` outcomes and `RelationalQueryService.ExecutePrimaryQuery` remain success/error only. - Serves `GetReadiness` by fronting query-serving watermarks plus CCS serving-generation readiness facts; external callers do not bypass Query Gateway to call CCS for this helper RPC. - Serves `GetCollectionStatistics` by fronting the CCS-owned optimizer-statistics hot projection/snapshot family for the current query-serving `latest READY` lineage; external callers do not treat CCS as a second public query helper endpoint for this capability. - Enforces cost-aware query admission (RCU-based) and adaptive load shedding under backend pressure. Commit Coordinator (stateful, HA): - Preserves Flow-A source `commit_ts`, injects Flow-B canonical `commit_ts` when request payload carries `0`, and finalizes write ordering across both flows. - Tracks per-collection modality progress (`applied_ts_metadata_search`, `applied_ts_semantic_search`, and optional `applied_ts_metadata_aux`) for visibility control. - Persists WAL durable anchors used as replay upper bounds during failover recovery. Visibility Barrier (stateful, HA): - Waits for all modalities touched by a write before publish eligibility. - Supports modality-selective fast path (metadata-search-only or semantic-only writes do not wait on untouched modality). - Emits per-write publish-ready intents to `Manifest Publisher`; it does not mutate manifests directly. Manifest Publisher (stateful, HA): - Accepts publish-ready intents from write path and compaction path and serializes per-collection manifest version progression. - Writes immutable manifest version `N+1` and then CAS-advances the private committed-manifest pointer object `manifest_committed_head.json`; that committed-pointer CAS is the atomic visibility boundary. - Micro-batches manifest updates to bound single-collection control-plane PUT rates. - Persists snapshot progress metadata (`applied_ts_*`, `cdc_resolved_ts`, `wal_replay_start_anchor`, `wal_durable_anchor`) in published manifests. - Preserves rollback safety by keeping previous manifest pointer valid until publish commit succeeds. Catalog Control Service (stateful, HA): - Owns namespace descriptors (policy/quota/audit bindings, lifecycle state) and collection descriptors (ownership mode, schema generation, collection-policy bindings, lifecycle state). - Owns the private quota-allocation/refill surface used by query/write gateways for distributed token-bucket leasing. - Owns namespace-scoped durable quota state and lease-epoch invalidation rules so recovery can revoke stale gateway leases without inventing a second quota authority. - Allocates stable `namespace_id` / `collection_id` in the control plane before descriptor-object publication; allocated IDs are durable control metadata, not TiDB-generated incidental values. - Serves linearizable namespace metadata create/update/ensure plus authoritative public-read flows for namespace provisioning/admin clients (`PutNamespaceMetadata`, `ListNamespaces`, `GetNamespaceMetadata`); individual catalog-read requests observe the latest authoritative online projection, while multi-page catalog traversals remain keyset-based per-page linearizable reads rather than whole-list snapshot reads unless a future revision adds explicit list-snapshot fencing; public mutation success is returned only after the authoritative online read projection for those public flows has been updated. - For first-time `COLLECTION_TRUTH_MODE_EXTERNAL` identity bootstrap, TiDB DDL/bootstrap is the canonical caller of `PutNamespaceMetadata` / `PutCollectionMetadata` with `CLIENT_KIND_TIDB_DDL_ADAPTER`; `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` namespace/collection provisioning remains a separate `CLIENT_KIND_CONTROL_ADMIN` workflow, while gateway-managed native-search ensure/apply flows use the gateway caller kinds only where the canonical API explicitly admits them. - Serves the canonical public collection metadata/schema read surface used by external-metadata parity and ordinary control-plane readback; any public control-plane mutation whose success payload is observable through that surface MUST finish authoritative online projection before success is returned. - Serves a private CCS admission-state and quota-leasing model for internal components that need non-public admission/cache/lease facts; the machine-readable owner for this seam is `api/proto/searchfabric/v1/catalog_control_private.proto`, and it remains intentionally outside the canonical public protobuf surface. - Maintains the authoritative active schema pointer per collection for collection-scoped schema readback and ownership validation. - Maintains serving-convergence state per collection so query-serving components can resolve the latest `READY` generation independently from a newer active generation that is still `APPLYING`. - Maintains per-collection manifest head pointer (`manifest_head_version`) as private derived control metadata, not as an extra field on the public `CollectionDescriptor` / `NamespaceDescriptor` payloads. - Applies manifest head updates with CAS precondition (`expected_head_version -> new_head_version`) after successful committed-pointer CAS. - Stores append-only catalog command/outcome objects plus deterministic replay-lookup objects in S3 for audit and deterministic replay; the terminal command/outcome object is the commit-authoritative durable truth, replay-lookup objects are deterministic derivative indexes, and any TiDB-side replay index is a rebuildable online projection rather than an independent durable truth source. - Owns descriptor enrichment projectors for public typed metadata (`approx_stats`, `index`, `completeness`) and the corresponding freshness budgets. - Owns optimizer-statistics snapshot production for `GetCollectionStatistics`, including immutable snapshot persistence, hot projection refresh, `stats_commit_ts` derivation, staleness evaluation, and recovery/rebuild. - Owns lifecycle-policy materialization into rebuildable tier-window projections used for background transition scheduling, retention-deadline tracking, and namespace aggregate freshness. - Does not own query search contexts; PIT/search-context semantics remain fully token-driven inside the stateless query plane. - Runs reconciliation tasks to detect and repair lagging catalog head pointers against committed manifest versions. IdAllocatorService (stateful, HA): - Owns canonical pre-ingest create/insert document-identity authority for collections whose schema chooses `DocIDMapping.generation_policy=DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`. - Depends on the private CCS admission-state read model for authoritative collection identity, lifecycle, truth mode, `DocIDMapping.generation_policy`, full `CollectionSettings.allocator_admission_policy` (`caller_profile`, `max_plan_items_per_request`, `max_forward_reservation_jump`), and the active write-admission `schema_generation_token`, but remains a separate bounded context because allocator throughput, hot-key isolation, replay state, and failure modes differ from catalog metadata mutation. - Fresh allocation against an existing collection whose authoritative allocator admission facts are unavailable/incomplete fails closed as `SF_CATALOG_UNAVAILABLE`; the missing-first-generation branch reuses the canonical schema-bootstrap diagnostics (`catalog_unavailable_reason=schema_bootstrap`, `schema_bootstrap_state=no_active_schema_generation`) and the allocator typed carrier `ErrorStatus.context.allocator.catalog_unavailable.schema_bootstrap`. - Admits only resolved stable `CollectionRef`; it never scopes allocator state by mutable names and therefore avoids same-name ABA coupling with catalog lifecycle. - Persists private per-collection allocator state plus replay truth independently from CCS descriptor rows; CCS remains authoritative for whether future allocation is still allowed, while allocator durable state remains authoritative for the collection-local dense-burn identity space and replay truth. - `AllocateIds` executes one ordered mixed-mode allocation plan per admitted request so omitted create/insert identity and caller-supplied explicit create/insert identity share one replay-governed apply unit, one returned `schema_generation_token`, one signed `allocator_receipt`, and one collection-local dense-burn reservation space; allocator replay identity is keyed on authenticated caller identity + `client_id` + `client_kind` + `rpc_method` + `CollectionRef` + `idempotency_key`. - Explicit reservation with `doc_id >= next_doc_id` and within the admitted forward-reservation window advances allocator watermark durably during allocator admission itself, every skipped lower `doc_id < next_doc_id` becomes a permanently burned gap, over-window jumps fail as `SF_TOO_MANY_OPERATIONS`, and later ingest failure burns the reservation rather than rolling allocator state back, so no post-commit callback path exists. - Allocator request-item ceilings (`max_plan_items_per_request`), forward-reservation windows (`max_forward_reservation_jump`), and identity-space exhaustion are explicit fail-closed admission guardrails rather than implementation-defined overload behavior, and the first two are discoverable through `CollectionSettings.allocator_admission_policy`. - Does not allocate IDs inside `IngestGatewayService` or `StreamMutations`; its sole job is to let callers construct explicit canonical `RowMutation.doc_id` values before the ingest-plane contract begins. NRT Delta Store (Metadata Search/Semantic Search, stateful, short-lived): - Stores commit-ordered unflushed deltas separately for metadata search and semantic search paths keyed by `(namespace_id, collection_id, doc_id)`. - Provides a strongly consistent read-path substrate for deltas not yet flushed into immutable segments. - Is not an independent publish surface: readers may consult it only while reconstructing an admitted snapshot under the same committed-manifest publish frontier. - Retention is short (flush-window scale) and bounded by checkpointed manifest progress. Background Compute Coordinator (stateful, HA): - Subscribes to manifest evolution, computes compaction/GC priorities, and schedules background jobs. - Schedules lifecycle tier-transition and lifecycle-retention execution for collection windows whose admitted `LifecyclePolicy` deadlines are due. - Enforces isolation budgets so compaction/GC cannot starve ingest/query planes. Compaction Workers / GC Workers (stateless fleet): - Compaction workers rewrite immutable objects and return replacement facts to Background Compute Coordinator; workers do not execute committed-manifest CAS directly. - Lifecycle transition workers and compaction workers become query-visible only after Manifest Publisher performs the committed-manifest CAS cutover on the accepted replacement set. - GC workers perform safe, retention-aware object deletion with S3-aware rate control. ### 3.3 Modular Deployment and Composition Profiles Design principle: - Keep one logical architecture with stable module boundaries, then choose deployment granularity per environment. - Build-time and deploy-time composition MUST NOT change correctness semantics (`commit_ts` ordering, `safe_ts` visibility, idempotency, ownership guardrails). Cloud profile (service-split, elastic-first): - Each module can run as an independent service/process: - `Query Gateway`, `Query Broker`, `Metadata Search Reader`, `Semantic Search Reader` - `Current-State Reader`, `Ingest Gateway`, `Ingest Broker`, `Write Planner`, writer executors - `Commit Coordinator`, `Visibility Barrier`, `Manifest Publisher`, `Catalog Control Service`, `IdAllocatorService` - `Background Compute Coordinator`, `Compaction Workers`, `GC Workers` - Modules scale independently by workload shape (query QPS, write QPS, background debt). - Failure domains are narrower; rollout and autoscaling policies can be tuned per module. Private profile (process-composed, operations-first): - Multiple modules may run inside the same process binary to reduce service count and operational overhead. - Recommended composition presets: - `query_bundle`: `Query Gateway + Query Broker + Metadata Search Reader + Semantic Search Reader` - `ingest_bundle`: `Current-State Reader + Ingest Gateway + Ingest Broker + Write Planner + Metadata/Semantic Stream Writers + Metadata/Semantic Batch Writers` - `control_bundle`: `Commit Coordinator + Visibility Barrier + Manifest Publisher + Catalog Control Service + IdAllocatorService` - `background_bundle`: `Background Compute Coordinator + Compaction Workers + GC Workers` - Small private deployments MAY use one all-in-one process with isolated thread pools/quotas per bundle. Cross-profile invariants: - Same canonical module interfaces and envelope schemas, regardless of in-process vs RPC invocation. - Same authn/authz, quota, and admission checks at module boundaries. - Same observability dimensions and SLO accounting per logical module, even when co-located in one process. - Profile switch (split <-> composed) MUST NOT require data-format migration. ## 4. Data Model and Partitioning ### 4.1 Logical Isolation: Namespace Domain with Multi-Collection Catalog - Search Fabric logical data organization is strictly two-level: `namespace -> collection`. - `namespace_id` and `collection_id` are mandatory in all keys and manifests. - Each `namespace_id` is modeled as a logical catalog, authz, and policy domain managed by `Catalog Control Service`; it participates in the shared global TSO-backed `commit_ts` domain rather than defining its own timestamp space. - All collections under the same `namespace_id` share that global `commit_ts` domain while keeping collection-level manifests and policy controls; namespace scoping is an ownership/filtering boundary, not an independent time abstraction. - Isolation controls: - query authorization bound to `namespace_id/collection_id`. - namespace-scoped quota and rate-limit policy, plus per-collection retention policy. - per-collection snapshot reads are resolved against the shared global `commit_ts` domain after namespace/collection scope filtering. ### 4.2 Physical Layout (Storage-Efficient and Prefix-Scalable) S3 keyspace is split into independent domains: - Data path (high-throughput immutable payloads): - `s3://bucket/data/{shard_hash}/{namespace_id}/{collection_id}/{object_type}/{object_id}` - `shard_hash` is high-entropy (`hash(namespace_id + collection_id + object_id) % N`) and can scale from `N=256` to `N=1024+` for whale tenants. - WAL path (sealed chunk objects, no append semantics): - `s3://bucket/wal/{hash(namespace_id + collection_id)[0:3]}/{yyyy-mm-dd}/{namespace_id}/{collection_id}/partition={base64url(source_partition)}/epoch={source_epoch}/chunk_{chunk_seq}_{min_commit_ts}_{max_commit_ts}.wal` - WAL checkpoint path (compact replay anchors): - `s3://bucket/wal-checkpoints/{hash(namespace_id + collection_id)[0:2]}/{namespace_id}/{collection_id}/partition={base64url(source_partition)}/epoch={source_epoch}/checkpoint_{max_source_seq}_{max_commit_ts}.json` - Control path (deterministic manifests/catalogs): - `s3://bucket/manifests/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/manifest_v{version}.json` - `s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/namespace_descriptor_v{namespace_revision}.json` - `s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/namespace_quota_state.json` - `s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/collection_descriptor_v{descriptor_revision}.json` - `s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/schema_generation_v{schema_version}_{schema_generation_token}.json` - `s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/optimizer_statistics_head.json` - `s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/optimizer_statistics_v{applied_schema_generation_token}_{stats_commit_ts}.json` - `s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/collection_tombstone_v{drop_revision}.json` - `s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/manifest_committed_head.json` - `s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/manifest_head.json` - `s3://bucket/catalog-log/{hash(command_identity)[0:2]}/{command_identity}.json` - `s3://bucket/catalog-replay-transport/{hash(transport_lookup_key)[0:2]}/{transport_lookup_key}.json` - `s3://bucket/catalog-replay-business/{command_kind}/{hash(business_lookup_key)[0:2]}/{business_lookup_key}.json` - `s3://bucket/catalog-recovery-segments/segment_{recovery_segment_epoch}.json.zst` - `s3://bucket/catalog-allocator/namespace-name/{hash(namespace_name)[0:2]}/{namespace_name}.json` - `s3://bucket/catalog-allocator/collection-name/{hash(namespace_id)[0:2]}/{namespace_id}/{hash(collection_name)[0:2]}/{collection_name}.json` - `s3://bucket/catalog-checkpoints/checkpoint_{checkpoint_epoch}.manifest.json` - `s3://bucket/catalog-checkpoints/shard={checkpoint_shard_id}/checkpoint_{checkpoint_epoch}.json.zst` Naming rules: - `namespace_revision` is the monotonic namespace-descriptor revision assigned by `Catalog Control Service` for namespace metadata mutation history. - `descriptor_revision` is the monotonic collection-descriptor revision assigned by `Catalog Control Service` for collection metadata mutation history. - `schema_version` is the public row-level schema lineage number, while `schema_generation_token` is the active fence token for that admitted generation. - all admitted `upsert` / `patch` rows inside one batch MUST share one uniform non-zero `schema_version`. - `delete`, `progress`, and `barrier` entries do not carry row-level `schema_version`; progress-only / barrier-only batches carry only `schema_generation_token` as lineage state. - externally supplied `schema_generation_token` fence values are capped at 64 Unicode scalar values and are preserved exactly for CAS/replay identity rather than truncated/hashed during persistence. - one immutable optimizer-statistics snapshot is durably identified within one collection by the composite tuple (`applied_schema_generation_token`, `stats_commit_ts`). - `stats_commit_ts` is the committed collection-state frontier covered by one immutable optimizer-statistics snapshot and is not durably unique by itself across schema lineages. - `drop_revision` is the monotonic collection-drop/tombstone revision that reserves dropped collection identity until reclamation is safe. - `namespace_name_allocation_epoch` is the private monotonic namespace-name reservation epoch; it changes only when a tombstoned namespace name is released and later reallocated. - `collection_name_allocation_epoch` is the private monotonic collection-name reservation epoch within one namespace; it changes only when a tombstoned collection name is released and later reallocated. - `command_identity` is the deterministic replay/audit object identity for one CCS mutation or internal control intent. - `transport_lookup_key` is the deterministic transport-replay lookup identity for one CCS mutation or internal control intent. - `business_lookup_key` is the deterministic business-replay lookup identity only for CCS mutations/control intents that define semantic replay beyond transport dedupe (`PutCollectionSchema`, `DropCollection`, and internal `head_advance_intent` in this revision). - `head_advance_intent` uses the same tuple for deterministic internal transport replay and business replay in this revision; CCS does not define a second distinct transport-only identity for that command class. - `recovery_segment_epoch` is the monotonic ordered recovery-segment index for compact disaster-recovery replay; each segment references a contiguous append-ordered committed command set and its applied-ts range. - canonical catalog names (`namespace_name`, `collection_name`) are valid UTF-8 Unicode scalar sequences submitted in NFC form, are limited to 128 Unicode scalar values, reject control characters, bidirectional formatting/isolate controls, and default-ignorable format characters, and use the canonical NFC string under binary collation for equality, uniqueness, business-replay identity where such an identity exists, prefix filtering, and stable ordering. Routing rules: - Data plane never relies on `LIST`; all objects are discovered by manifest pointers. - WAL replay discovery is pointer-driven via manifest/checkpoint anchors; full-prefix WAL scans are forbidden in recovery path. - WAL/chunk/checkpoint identity MUST be derived directly from the canonical stream tuple `(namespace_id, collection_id, source_partition, source_epoch)`; opaque writer-local `stream_id` aliases are forbidden. - `source_partition` path materialization MUST use a stable reversible encoding (for example unpadded base64url over the raw canonical string) so recovery tooling can reconstruct the exact stream identity. - WAL chunk seal policy is bounded by size/time/op thresholds to avoid tiny-object explosion. - Manifest updates are micro-batched to keep single-collection control-plane PUT rates bounded. - `Catalog Control Service` reads and writes control objects by deterministic key; catalog scans rely on known descriptor/head pointers, not prefix enumeration. - CCS replay recovery and duplicate/conflict detection are pointer-driven through `catalog-replay-transport` for all mutating commands and through `catalog-replay-business` only for commands that define a business replay family; prefix scans over `catalog-log` are forbidden on steady-state replay paths, and ordinary disaster recovery starts from the latest compacted CCS recovery checkpoint plus post-checkpoint ordered `catalog-recovery-segments` rather than from whole-history scans. - CCS recovery MUST materialize descriptor/schema/tombstone/head state only from control objects referenced by `APPLIED` command/outcome records; unreferenced control objects are orphan/uncommitted state and MUST be ignored or quarantined. - CCS same-name ABA protection is durably anchored by canonical S3 allocator-state objects for namespace and collection names; TiDB allocator tables are rebuildable online projections of that durable allocator state. - Small collections can still be aggressively consolidated within their own collection-scoped object sets, while object IDs remain entropy-sharded to avoid prefix hotspots. ### 4.3 Index-as-Storage No duplicate row-store copy is maintained. - Metadata Search path: Tantivy stored fields + postings + fast fields are authoritative for metadata/text. - Semantic path: vector segment payload + graph + quantized codes are authoritative for vectors. - Recovery rebuilds serving state from manifests and immutable objects in S3. ### 4.4 Schema Management and Evolution Each collection tracks schema metadata in manifests (`schema_version`, field descriptors, analyzer/vector config hashes). Index sort contract (Elasticsearch `index.sort.*` equivalent): - Schema may include `IndexSortSpec` to declare physical sort layout for metadata/full-text components. - `index.sort.field` / `index.sort.order` / `index.sort.mode` / `index.sort.missing` are normalized into `IndexSortSpec.fields[]` at schema publish time. - `IndexSortSpec` is valid only for metadata-bearing indexes (`fulltext`, `metadata_aux`, `hybrid`); applying it to pure vector indexes is rejected. - If `index_sort` is specified, `IndexSortSpec.fields[]` must be non-empty; field order defines lexicographic sort precedence. - Changing `IndexSortSpec` between schema generations is a rebuild-class change; in-place mutation is not allowed. - Query output ordering still follows `sort_keys` from the query contract; index sort is a physical-layout optimization and must not alter logical ordering. Compatibility rules: - Additive field changes (new metadata/text field): - accepted without rewriting old segments. - query engine materializes default/null values for older segments missing that field. - Compatible widening changes (for approved scalar types): - allowed via deterministic coercion rules captured in schema metadata. - Incompatible changes (e.g., vector dimension change, incompatible analyzer change): - require new schema generation and dual-read/controlled cutover. - old and new generations are isolated by manifest pointers until migration completes. Compaction behavior: - Compaction workers perform schema upcasting when merging mixed-version segments. - Upcasting is deterministic and must preserve snapshot semantics at `read_ts`. Lifecycle tier/window realization: - When a collection admits `CollectionSettings.lifecycle_policy`, the physical data model becomes window-aware: - every committed manifest entry for a lifecycle-managed partition/window carries its window bounds plus effective `DataTier`, - referenced metadata-search and semantic objects inherit that manifest-published tier identity for query pruning and background transition planning. - Resource governance is three-axis and orthogonal: - temporal axis: `LifecyclePolicy` + manifest-published `DataTier` decide row/window placement and retention horizon, - structural axis: canonical field-level `FieldAccessIntent` expresses cache/latency priority for a field family, - quality axis: canonical vector-level `VectorRetrievalIntent` expresses fidelity/compression/residual-preservation policy for vector retrieval. - Execution nodes combine those axes rather than collapsing them into one overloaded storage knob: - temporal placement decides where data lives, - structural intent biases cache residency and prefetch priority, - quality intent biases quantization format, residual preservation, and exact-rerank eligibility. - Effective defaults remain deterministic throughout the fleet: - absent or `UNSPECIFIED` structural intent resolves to effective `FREQUENT`, - absent or `UNSPECIFIED` vector quality intent resolves to effective `STANDARD`. - Query-time `target_tiers[]` pruning reads those manifest-published `DataTier` annotations directly from the committed snapshot chosen by `Query Broker`; query hot paths do not call CCS to resolve tier membership. - CCS maintains a rebuildable hot projection of lifecycle windows keyed by `(namespace_id, collection_id, window_start_ms, window_end_ms)`: - source of truth is still the admitted lifecycle policy plus committed manifests, - hot projection exists to drive transition scheduling, retention-deadline checks, and namespace-level aggregates without forcing steady-state manifest rescans. - Background lifecycle execution is manifest-first: - when a window crosses a transition deadline, workers rewrite or relocate the affected objects into the target tier, - the new tier becomes query-visible only after manifest CAS publishes the updated object references and tier annotation, - `retention_ms` marks when a lifecycle-managed window becomes reclaimable, but physical deletion still waits until manifests stop referencing the retired objects. - Background compaction may also realize quality-axis evolution without changing the public API: - lower-fidelity intents may switch to denser quantization when policy allows, - `HIGH_FIDELITY` vectors remain eligible for residual-preserving layouts needed by exact rerank or audit-grade replay, - when a window ages into colder tiers, compaction may drop residual payloads only when the effective quality intent allows that downgrade. - Lifecycle transitions and quality rewrites are therefore coupled by policy, not by ownership: - moving a window from `HOT` to `WARM` or `COLD` MAY change the physical encoding, - but the transition itself MUST NOT weaken the effective vector quality contract; only a schema generation whose admitted effective quality intent allows residual dropping may do so. - Recovery rebuilds lifecycle-window hot state from committed manifests plus the admitted collection lifecycle policy; there is no second handwritten lifecycle truth store. ### 4.5 Document Identity and Cross-Index Alignment - Canonical document identity is collection-scoped `doc_id` (`uint64` / 64-bit unsigned integer). - Metadata Search and semantic search internal structures align on the same `doc_id` domain: - postings/fast-field sidecars, vector graph entries, tombstone bitmaps, NRT deltas, and replay/LWW filters are keyed by `(namespace_id, collection_id, doc_id)`. - API/profile-level `_id`/`id` values are first treated as wire-level lexical tokens, then losslessly parsed to canonical `uint64` `doc_id` before writer dispatch; non-lossless/out-of-range values are rejected (`400 SF_INVALID_DOC_ID`). - `DocIDMapping` is authoritative for source-field typing, immutability/uniqueness constraints, and active identity-generation policy. - Active-profile canonical admission currently allows only `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_UINT64`. - For collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`: - `DocIDMapping.generation_policy` is production-closed to `DOC_ID_GENERATION_POLICY_CALLER_REQUIRED`. - external-truth / CDC paths are identity-supplying, not allocator-backed. - collection schema MUST expose one canonical `doc_id` source column with `BIGINT UNSIGNED` and `NOT NULL` semantics. - that column MUST be `PRIMARY KEY` or `UNIQUE NOT NULL`. - the `doc_id` column is immutable after insert; CDC events violating this rule are rejected as schema/contract errors. - For collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`: - canonical row identity remains collection-scoped `uint64` `doc_id`, - TiDB is only a remote query/write gateway for that identity domain, - and TiKV does not carry a duplicate authoritative row copy or a second `doc_id` truth source for the federated collection. - schema may use either `DOC_ID_GENERATION_POLICY_CALLER_REQUIRED` or `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`. - when `generation_policy=DOC_ID_GENERATION_POLICY_CALLER_REQUIRED`, every write route stays explicit-identity-only and `CollectionSettings.allocator_admission_policy` MUST be absent. - when `generation_policy=DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`, `CollectionSettings.allocator_admission_policy` MUST be present and publish explicit `caller_profile=ALLOCATOR_CALLER_PROFILE_ES_WRITE_GATEWAY_ONLY` plus positive `max_plan_items_per_request` / positive `max_forward_reservation_jump` guardrail ceilings; future caller expansion requires a later canonical contract update. - on allocator-optional collections, both omitted-ID create/insert and caller-supplied explicit create/insert resolve through one ordered `IdAllocatorService.AllocateIds` plan before canonical mutation construction so allocator-issued and explicit create/insert IDs share one collection-local reservation space and one schema-fence result, and the follow-on canonical `MutationBatch` carries the signed allocator receipt returned by that plan. - patch/delete/update-style routes remain explicit-identity-only even on allocator-optional collections; allocator authority is not a generic ingest-time fallback. ## 5. Consistency and MVCC Model ### 5.1 Timestamping - All committed writes participate in one globally monotonic `commit_ts` domain backed by shared TiDB TSO: - Flow A (`TiCDC`) preserves source `commit_ts`. - Flow B (Search-Fabric-owned write gateways, including native search APIs and TiDB federated SQL remote DML) obtains canonical `commit_ts` injection from `Commit Coordinator` before writer persistence/apply ordering. - Each `namespace_id` observes writes from the shared global `commit_ts` domain through namespace ownership/filtering only; this is not an independent namespace-local time domain. - Reads carry `read_ts`. - Whenever a TSO-based timestamp is compared to a wall-clock budget/SLO expressed in milliseconds, implementations MUST first project the TSO to its physical-millisecond component (`tso_physical_ms(ts)`); raw TSO integer subtraction is not a valid millisecond comparison. - Query-level snapshot isolation rule: - only versions where `commit_ts <= effective_read_ts` and not tombstoned by `commit_ts <= effective_read_ts`. ### 5.2 Safe Visibility for Hybrid Reads Each subsystem publishes `applied_ts` per collection: - `applied_ts_metadata_search` - `applied_ts_semantic_search` - optional `applied_ts_metadata_aux` with explicit presence semantics (absent when the metadata-aux modality does not exist; not a `0` sentinel) Public/readback watermark carriers expose only enabled collection modalities: - metadata-search frontier is present only when the collection exposes metadata-search state - semantic-search frontier is present only when the collection exposes semantic-search state - metadata-aux frontier is present only when the collection exposes metadata-aux state - disabled/structurally absent modalities are omitted rather than represented by synthetic copied or zero-valued frontiers Hybrid strict snapshot bound: - `safe_ts = min(present enabled modality applied ts)` Read modes: - strict: - compute `effective_read_ts = min(read_ts, safe_ts)` internally for readiness evaluation. - use the remaining transport deadline budget as the upper bound for visibility waiting and keep rechecking fresh readiness/watermarks until `safe_ts >= read_ts` or the deadline expires. - `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`: if the deadline expires before `safe_ts >= read_ts`, TiDB must fallback to its native plan; production hot-path callers receive this through `RelationalQueryService.ExecuteProjectedQueryOrFallback` rather than through a third outcome on shared `QueryGatewayService`. - `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`: if the deadline expires before `safe_ts >= read_ts`, fail closed with `SF_VISIBILITY_TIMEOUT`. - bounded-staleness: - compute `effective_read_ts = min(read_ts, safe_ts)` and use the remaining transport deadline budget as the upper bound for visibility waiting. - when current lag exceeds the caller/configured bound, keep rechecking fresh readiness/watermarks until lag enters the admitted bound or the deadline expires. - millisecond lag budget is evaluated as `tso_physical_ms(read_ts) - tso_physical_ms(effective_read_ts) <= max_staleness_ms`. - if the deadline expires before lag enters the admitted bound, the external behavior split matches strict mode: `COLLECTION_TRUTH_MODE_EXTERNAL` fallback (surfaced explicitly on the TiDB fast path), `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` fail-closed. - best-effort: - request carries `read_ts=0`. - first-page execution normally uses current `effective_read_ts = safe_ts`. - projected-route overlay-bearing execution is the explicit exception to that default rule: when a trusted `ProjectedQueryRequest` carries a validated non-empty overlay, gateway materializes broker dispatch with `effective_read_ts = snapshot_read_ts` even though shared `read_ts` remains `0`; see Section `9.1`. - continuation execution reuses token-bound `effective_read_ts` under the canonical pagination contract rather than recomputing from the latest `safe_ts`. - may return partial/stale warnings, but never masquerades as strict visibility. ## 6. Data Ingestion and Unified Write Paths ### 6.1 Unified Ingestion Flow Overview (Search APIs + TiCDC) ```text Flow B (Search-Native APIs): External Clients (Elasticsearch/Turbopuffer style writes) -> Ingest Gateway -> Ingest Broker Flow A (TiCDC): TiDB/TiKV + TSO + TiCDC -> CDC Adapter -> Ingest Gateway -> Ingest Broker Common downstream: Ingest Broker -> Write Planner Write Planner -> Commit Coordinator (preserve Flow-A `commit_ts`, inject Flow-B `commit_ts`, finalize write order) Commit Coordinator -> Metadata Stream Writer (RT/WAL) -> NRT Delta Store (Metadata Search) -> Semantic Stream Writer (RT/WAL) -> NRT Delta Store (Semantic Search) -> Metadata Batch Writer (Offline) -> Semantic Batch Writer (Offline) Metadata/Semantic Stream and Batch Writers -> Object Store (SST/Segments/Bitmaps) Metadata/Semantic Stream and Batch Writers -> Visibility Barrier -> Manifest Publisher (micro-batched CAS updates) ``` Flow A and Flow B are parallel ingress channels. They are not a single serial chain. `NRT Delta Store (Metadata Search/Semantic Search)` closes the real-time gap without waiting for S3 segment flush, but it does not bypass the committed-manifest publish boundary. ### 6.2 Ingest Gateway (Public + Internal) - Provides user-facing native search write APIs beyond TiDB integration. - Supports compatibility profiles: - Elasticsearch-style APIs (`index`, `update`, `delete`, `_bulk`) as a compatibility subset. - Turbopuffer-style APIs (`upsert`, `delete`, vector+metadata patch, batch ingest) as a compatibility subset. - Accepts TiCDC-derived row changes from CDC Adapter as internal write events. - Enforces auth, schema validation, quota, and tenancy checks before routing. - Resolves external route names through typed collection catalog operations before any downstream control-plane or data-plane call: - Elasticsearch `{index}` and Turbopuffer route namespace token `{namespace}` lower to `CollectionNameBinding{namespace_id, collection_name}` within the already resolved canonical namespace/account scope. - gateway uses `GetCollectionMetadata.lookup.name_binding` to resolve existing collections and `PutCollectionMetadata.name_binding` when it must ensure collection identity before schema or ingest admission. - if gateway caches `CollectionNameBinding` results locally, both positive and negative cache entries MUST carry the namespace-scoped `collection_name_binding_revision`; any revision change invalidates the cached result before reuse. - write-route `encryption` intent is collection metadata, not row payload: first allocation MUST carry explicit encryption through `PutCollectionMetadata`, and existing collections treat route-supplied encryption as validation-only equality against the authoritative descriptor. - once that flow returns an authoritative `CollectionRef`, all downstream `PutCollectionSchema`, `DropCollection`, snapshot, and ingest RPCs use that `CollectionRef`. - Resolves write-admission state from `Catalog Control Service` before write admission: - public typed readback (`GetCollectionMetadata` / `GetCollectionSchema`) remains the source for public descriptor/schema payloads, - private CCS admission-state read model supplies internal-only facts such as collection lifecycle state, descriptor revision/etag, namespace-scoped collection-name-binding revision/etag for positive/negative cache invalidation, and serving-generation readiness needed to resolve the latest query-serving-admissible `READY` schema generation. - for lifecycle-enabled collections, that same private admission-state surface must also expose the concrete event-time carry contract needed on the hot path: authoritative `time_partition_field` identity/type plus the rule that online `upsert` and `patch` must carry a non-null payload value for that field so late/future-arrival checks remain payload-driven with no read-before-write dependency. - Uses short-lived local admission caches with version/etag checks against that private admission-state model; cache miss or stale version triggers re-fetch. - For admitted read-modify-write and selector-driven write routes, splits internal prewrite reads by consistency need: - point-read/OCC routes use `Current-State Reader` to derive current row existence/version and produce `expected_commit_ts`, - selector routes use `Query Broker` snapshot semantics to materialize deterministic `doc_id` order plus selected current-version markers and enforce `effective_max_result_window` / batch-budget probe clamps before mutation emission. - Existence-sensitive source response reconstruction for single-document `index` / `delete` does not rely on a gateway-local prewrite existence guess; gateways consume authoritative post-apply result kinds from ingest ACK item outcomes. - Source by-query conflict policies (for example Elasticsearch `conflicts=abort|proceed`) must map onto explicit canonical batch execution modes before mutation emission; the gateway must not leave that branch as implementation-defined behavior. - Undeclared fields no longer trigger gateway-owned schema inference or replay; write admission stays bound to the currently admitted schema snapshot, and schema expansion requires an explicit control-plane mutation before retry. - Enforces mandatory tenant-level throttling with distributed token buckets (write-capacity units, not only request count). - Uses local in-memory buckets with asynchronous refill/prefetch from a private CCS-owned quota-allocation surface to avoid critical-path remote RTT; there is no separate quota authority component in this design. - Rejects oversized batch payloads pre-flight when unrolled write cost exceeds hard per-request tenant limits. - Routes accepted requests into shared write-control pipeline; `Write Planner` performs final streaming/batch + metadata search/semantic search dispatch. ### 6.3 Canonical Write Envelope and Normalization All inbound sources are normalized to the proto-shaped ingest contract in `api/proto/searchfabric/v1/search_fabric.proto`. This architecture document does not define a parallel ad hoc write-envelope struct. Architectural invariants of that canonical envelope: - one admitted write batch carries explicit collection scope, explicit stream identity, explicit ingest policy, and an explicit schema-generation fence; write lanes must not infer those from transport context or source-specific defaults. - one admitted ordered entry stream mixes only the canonical entry kinds (`row mutation`, `progress mark`, `barrier mark`) under one deterministic sequencing model. - replay identity, conflict resolution, and retry slicing are defined on the canonical batch/entry stream rather than on source-ecosystem payload shapes. - the Flow-A / Flow-B `commit_ts` split remains part of the canonical contract: - Flow A / CDC preserves source `RowMutation.commit_ts`. - Flow B / native-search gateways send `RowMutation.commit_ts=0`, then `Commit Coordinator` injects canonical TSO before persistence/apply ordering. - durability/visibility intent, undeclared-field behavior, continue-on-error semantics, and physical ingest-lane intent are all expressed through the canonical ingest-policy surface rather than through gateway-local booleans or source-specific side channels. - `batch_fingerprint` stability is defined on canonical business content and therefore remains invariant across transport retries and Flow-B runtime `commit_ts` injection. Non-envelope internal execution metadata such as source ecosystem label, modality routing, WAL chunk IDs, and writer-local tracing/correlation remain downstream implementation details. They MUST NOT be described as fields of the canonical admitted mutation envelope. ### 6.4 Streaming Writers (Real-Time) - `Write Planner` dispatches streaming events to `Metadata Stream Writer`, `Semantic Stream Writer`, or both based on `modality_mask`. - Metadata Stream Writer appends metadata search WAL and mutates `NRT Delta Store (Metadata Search)`. - Semantic Stream Writer appends semantic WAL and mutates `NRT Delta Store (Semantic Search)`. - Both writers execute under shared canonical stream/batch identity and the proto-defined `RowMutation.commit_ts` contract: Flow A preserves source commit timestamps, while Flow B receives injected canonical TSO from `Commit Coordinator` before persistence/apply ordering; optional local correlation IDs are observational only. - Writers buffer append records and seal immutable WAL chunks by bounded policy (for example: `max_chunk_bytes`, `max_chunk_ops`, `max_chunk_age_ms`). - A chunk is considered durable only after S3 PUT success and checkpoint/anchor persistence. - `durability_policy=ingest_ack`: ACK once request is accepted by ingest control path and enqueued to writer buffers. - `durability_policy=wal_chunk_durable`: ACK only after corresponding WAL chunk is durably persisted and anchored. - `visibility_policy=wait_for_visible`: ACK after `Visibility Barrier + Manifest Publisher` completes (durable + query-visible). - Guarantees query execution can merge committed deltas (`commit_ts <= effective_read_ts`) without waiting for S3 micro-segment flush once the committed-manifest publish boundary has admitted that write. - Periodically flushes metadata search/semantic search immutable micro-segments to S3. - `Visibility Barrier` emits publish-ready intent only after all touched modalities of a write are durable; `Manifest Publisher` remains the sole manifest-mutating actor. - `MutationAck.success.applied_watermarks` and readiness readback report the full derived frontier for every enabled collection modality after apply/publish gating; a metadata-only or semantic-only write may leave another enabled modality unchanged, but it is still reported rather than omitted. Ack semantics clarification: - `durability_policy` sets the durability gate for ACK (receive-only vs WAL-durable). - `ingest_ack` means the request is accepted into the ingest control path and buffered; durability is not guaranteed yet. - `wal_chunk_durable` means the WAL chunk is persisted to S3 and anchored before ACK. - `visibility_policy` sets the visibility gate for ACK. - `wait_for_visible` means ACK only after the visibility barrier confirms publish eligibility and `Manifest Publisher` completes manifest CAS publish, making the write query-visible. - `async` means the ACK does not wait for visibility, and only follows the `durability_policy` gate. - Effective ACK timing is the stricter of the two gates when both are specified. - If the write-side `wait_for_visible` budget expires before publish completes, that failure stays on the ingest stream surface as retryable `SF_BACKEND_UNAVAILABLE`; `SF_VISIBILITY_TIMEOUT` is reserved for read/query-side visibility waiting. ### 6.5 Batch Writers (Batch) - `Write Planner` dispatches normalized bulk chunks to `Metadata Batch Writer`, `Semantic Batch Writer`, or both. - Metadata Batch Writer builds metadata search immutable segments offline from already-admitted `MutationBatch` work. - Semantic Batch Writer builds semantic immutable segments/quantized payloads offline from already-admitted `MutationBatch` work. - Writers emit artifacts directly to S3 and register produced key-ranges + `max_commit_ts` before publish. - Any internal chunking, spill, or batching remains subordinate to the original canonical batch identity: batch writers must preserve canonical winner order `(commit_ts, source_partition, source_epoch, source_seq)` and still allow downstream reconstruction of exactly one `MutationAck` per original admitted batch. - In the current revision, `INGEST_MODE_BATCH` is admitted only with `durability_policy=ingest_ack` plus `visibility_policy=wait_for_visible`; the batch lane does not define a separate public pre-visibility durable ACK anchor. - Internal chunking is an execution detail only; public visibility must still respect the original admitted batch atomicity contract. - under `ATOMICITY_POLICY_ALL_OR_NOTHING`, batch writers may stage/build artifacts incrementally but MUST defer reader-visible publish until the entire original batch has succeeded. - under `ATOMICITY_POLICY_FAIL_FAST_PARTIAL`, any incremental publish must remain a canonical accepted prefix only; later chunks must not publish ahead of unresolved earlier positions. - `Visibility Barrier` and `Manifest Publisher` therefore operate on a publish scope that remains subordinate to the original admitted batch atomicity unit rather than on arbitrary writer-local chunk IDs. - batch-lane policy / atomicity invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-INGEST-001`. Snapshot sync session lifecycle and chunk upload protocol are defined in `docs/specs/core/search-fabric-api.md`; offline snapshot-import execution belongs to Section `6.10` and `docs/architecture/snapshot-import-worker.md`, not to the `MutationBatch` batch-writer boundary described here. Contract details are defined in `docs/specs/core/search-fabric-api.md`. ### 6.6 Conflict Resolution: Commit-TS First with Canonical Same-TS Order Conflict policy for overlapping key-ranges: - Batch preflight and publish checks compare produced key-range coverage plus relevant `max_commit_ts` against the latest already-published conflicting coverage for those ranges. - On overlap: - the conflicting internal batch build unit is rejected from publish for the affected ranges. - implementation may rebuild or retry that internal work unit, but any retry remains subordinate to the same original admitted `MutationBatch` lineage rather than to a writer-owned snapshot session. - At merge time, winner rule is deterministic: - if same `doc_id` exists in both paths within a modality, choose highest `commit_ts` first (LWW by commit time). - if `commit_ts` ties, choose canonical order by `source_partition`, then `source_epoch`, then `source_seq`. - source kind (`streaming` vs `batch`, `TiCDC` vs native-search API) is not an independent tie-break dimension. Operational mechanism: - Maintain per-collection conflict index: - key-range -> latest already-published conflicting frontier metadata needed for deterministic batch publish checks. - Batch publish for covered ranges requires the produced coverage metadata submitted by the batch writer to remain compatible with that already-published frontier under the canonical winner rule above. - Otherwise the affected internal batch build unit is rejected from publish and must be rebuilt or retried without weakening the original admitted batch atomicity or ACK contract. ### 6.7 Idempotency and Exactly-Once Effects To convert TiCDC at-least-once delivery into exactly-once logical effects for retrieval: 1. Canonical stream-scoped identity: - Entry-level replay identity follows the canonical API contract: - `ROW_MUTATION`: `(namespace_id, collection_id, source_partition, source_epoch, doc_id, commit_ts, source_seq)`. - `PROGRESS_MARK` / `BARRIER_MARK`: `(namespace_id, collection_id, source_partition, source_epoch, entry_kind, source_seq)`. - Batch-level replay identity follows `(namespace_id, collection_id, source_partition, batch_id)` plus `batch_fingerprint`. - Flow-B stateless gateways derive retry-stable `source_partition`, fix `source_epoch=1`, and assign 1-based `source_seq` array order exactly as defined in `docs/specs/core/search-fabric-api.md`. 2. Hot-path replay acceleration: - Metadata and Semantic Stream Writers MAY maintain short-lived per-collection replay/LWW caches (for example latest applied cursor per `doc_id`) in memory/NVMe to short-circuit obviously old/replayed events. - These caches are performance optimizations only. They MUST NOT weaken or replace the canonical entry/batch idempotency contract above, and they MUST remain safe under same-`commit_ts` multi-entry ordering and failover retransmit. 3. Manifest-level durable checkpoint and replay anchors: - Every manifest publish includes `cdc_resolved_ts` plus WAL anchors (`wal_replay_start_anchor`, `wal_durable_anchor`). - On failover, CDC Adapter/writers resume replay from anchors instead of scanning WAL prefixes. - WAL chunks beyond durable anchors are treated as uncommitted-orphan data and excluded from replay. 4. Storage-level idempotent operations: - Delete/tombstone application on Roaring bitmap sidecars is set-based (`add(doc_id)`), naturally idempotent. - Final post-fusion dedup by `doc_id` is request-scoped: Query Broker collapses duplicate top-level `doc_id` candidates only when the admitted canonical request sets `search_payload.dedup_by_doc_id=true`. CDC stream ordering, framing, and idempotency key details are defined in `docs/specs/core/search-fabric-api.md`; this document only states the architectural principles. Contract details are defined in `docs/specs/core/search-fabric-api.md`. ### 6.8 Source Truth Model and Guardrails Collection truth is modeled by one authoritative control-plane field per collection: - `collection_truth_mode`: `COLLECTION_TRUTH_MODE_EXTERNAL` | `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` Allowed values in this design: 1. `COLLECTION_TRUTH_MODE_EXTERNAL` (mandatory for Section `11.2`): - canonical mutations MUST come from `TiDB -> TiCDC -> CDC Adapter`. - direct native search API writes to the same collection are rejected (`409 SF_OWNERSHIP_CONFLICT`). 2. `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` (mandatory for Section `11.3` in the active federated model): - Search Fabric is the command-truth owner for the collection. - command ingress MAY come from native search gateways or from TiDB federated SQL remote DML through `Ingest Gateway`. - TiKV does not store federated-table row truth for this collection, and TiCDC ingestion for the collection is disabled. - TiDB direct-write behavior is remote DML against Search Fabric, not a TiKV-backed local OLTP write. Any unspecified or unsupported truth-mode value is rejected at catalog configuration publish time by `Catalog Control Service`. ### 6.9 Stateful Collaboration Workflow (Catalog + Commit + Visibility + Manifest) Normal write path (`visibility_policy=wait_for_visible`): 1. `Ingest Gateway` resolves write admission against two CCS surfaces before any canonical write is admitted: - public typed descriptor/schema readback supplies the authoritative external descriptor/schema payloads used for public contract alignment; - private CCS admission-state supplies the hot-path admission facts that are intentionally not public wire fields, including lifecycle admission state, descriptor validator/etag, `collection_name_binding_revision`, resolved `collection_truth_mode`, active schema pointer, latest query-serving `READY` generation facts, the effective query-window bound used for query admission, and any lifecycle event-time carry contract required for online ingest. 2. `Commit Coordinator` validates/preserves Flow-A `commit_ts`, injects Flow-B `commit_ts` where needed, and establishes deterministic canonical write order. 3. touched stream writers persist modality artifacts plus durable WAL anchors, while touched batch writers persist immutable batch artifacts plus publish-coverage metadata for the same admitted write scope. 4. `Visibility Barrier` waits until all touched modalities report durable completion for the same canonical admitted write scope (stream identity + batch/entry membership + `commit_ts`). 5. `Manifest Publisher` writes immutable `manifest_vN+1`, then CAS-advances the private committed-manifest pointer object `manifest_committed_head.json` from `N -> N+1`; success of that committed-pointer CAS defines the atomic read-model visibility boundary. 6. `Manifest Publisher` sends `Catalog Control Service` a head-advance intent (`namespace_id`, `collection_id`, `expected_head=N`, `new_head=N+1`, manifest checksum/etag`) so CCS can refresh its derivative fast pointer and governance metadata. 7. `Catalog Control Service` applies CAS update of `manifest_head.json`, appends audit record, and acknowledges head advancement. Critical invariant: - the committed-manifest pointer CAS on `manifest_committed_head.json` is the visibility/commit boundary seen by readers. - immutable `manifest_vN+1` must already exist before that pointer can advance. - catalog head advancement is derivative metadata for fast-path lag detection and governance workflows. Failure handling between stateful services: - If step 5 (immutable manifest write or committed-pointer CAS) fails: - `Manifest Publisher` returns retryable failure; catalog head is not advanced. - writers keep idempotent artifacts; subsequent retry republishes the same logical write set and only the first successful committed-pointer CAS makes it visible. - If step 5 succeeds but step 7 (catalog head CAS) fails: - data is committed and query-safe because readers resolve through `manifest_committed_head.json`. - collection may temporarily serve CCS head `N` while the committed pointer already exposes `N+1`. - reconciliation loop in `Catalog Control Service` detects `head < committed_manifest_head` and repairs CAS update asynchronously. - If `Catalog Control Service` is unavailable at step 1: - `Ingest Gateway` fails closed for writes requiring schema/truth-mode validation (`503 SF_CATALOG_UNAVAILABLE`). - query/write admission paths that require authoritative CCS facts fail closed until fresh revision/etag-validated admission state is re-established; bounded-time stale-descriptor reuse is not a canonical fallback contract in this design. ### 6.10 TiCDC High-Lag Degradation: Offline Batch Behind Canonical Snapshot RPCs (Phase-Gated) When TiCDC lag exceeds configured guardrails under extreme bulk-import windows, platform MAY enable a controlled offline-import execution path for designated tenants/jobs, but the cross-boundary protocol remains the canonical snapshot-sync lifecycle only: - public/cross-boundary contract: - `StartSnapshotSync` creates the fenced snapshot session bound to `CollectionRef`, `snapshot_ts`, `schema_version`, and `schema_generation_token`. - `UploadSnapshotChunks` remains the only admitted upload/session-progress RPC boundary for snapshot data exchange. - `CommitSnapshotSync` remains the only admitted cutover/visibility commit boundary. - snapshot session binder / resume invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-SNAPSHOT-001`. - internal execution detail behind that contract: - offline workers MAY build Tantivy/SPFresh-compatible immutable artifacts out of band and MAY stage objects directly in object storage under prefixes owned by the admitted snapshot session. - session-owned snapshot workers, rather than `MutationBatch` batch writers, MAY use external sort, range partitioning, or staged object assembly because the public contract on this path is chunk/session-based (`SnapshotChunkAck`, `CommitSnapshotSync`) rather than `MutationAck`-based. - such staging is internal execution/persistence behavior owned by snapshot workers and `Manifest Publisher`; it is not a second public protocol and it MUST NOT require callers to submit signed manifest intents directly. - safety constraints: - this path is explicitly non-realtime and mutually exclusive with conflicting online stream windows for the same key ranges. - visibility becomes effective only after successful canonical snapshot commit/cutover, not after raw object upload. - replay/idempotency remains anchored on snapshot session identity, chunk acknowledgements, and commit validation rather than on ad hoc manifest-intent APIs. - governance: - feature is phase-gated and disabled by default in normal online operation. - SLO/runbook must include explicit rollback path and lag-recovery handoff back to TiCDC online flow. Detailed execution guidance for this internal path lives in `docs/architecture/snapshot-import-worker.md`. ## 7. Metadata Search Reader Deep Dive (Tantivy on S3 + MVCC) Metadata Search Reader handles all fact-based retrieval, including exact-match lookup over structured attributes (for example tags/dates) and full-text search over unstructured text through tokenized inverted indexes. ### 7.1 Tantivy-on-S3 - Tantivy segments remain immutable, uploaded as object bundles: - postings - term dictionary - fast fields - stored fields - Segment manifests map collection snapshots to segment lists. ### 7.2 MVCC Problem with Immutable Segments Tantivy supports immutable segments and logical deletions, but does not natively maintain full snapshot visibility across arbitrary external `read_ts` over long history at this scale. ### 7.3 Visibility Bitmap Adaptation We store external sidecar visibility objects per segment using Roaring Bitmaps. For each segment `seg_X`: - `seg_X.visible_base.rbm`: document ordinals visible at segment publish checkpoint. - `seg_X.obsolete_until_.rbm`: cumulative bitmap of doc ordinals made obsolete by updates/deletes up to that `ts_bucket`. - `seg_X.delta_.rbm` (optional): short-window delta for fine-grained `read_ts` between checkpoints. Query-time visibility at `read_ts`: 1. Load closest checkpoint bucket `b <= read_ts`. 2. Candidate docs from Tantivy query are ANDed with: - `visible_base` - NOT(`obsolete_until_b`) 3. If exact `read_ts` is between buckets, apply in-memory or short-window delta masks. Effective visible set: - `visible_docs(read_ts) = visible_base AND NOT(obsolete_docs(read_ts))` Why this works: - Segment immutability is preserved. - MVCC masking is externalized and compact (Roaring). - Old segment data is reused until background compaction rewrites cleaner segments. ### 7.4 Metadata Search Query Execution Path - Metadata predicate planning prefers inverted-index set algebra first: - `AND`/must predicates -> postings intersection. - `OR`/should predicates -> postings union. - For predicate fields not covered by suitable inverted indexes, fallback path is: - retrieval auxiliary index evaluation for structured predicate combinations; - bitmap/Bloom runtime filters when available; - fast-field filtering or bounded scans as last resort. - Execute BM25/term queries. - Apply visibility bitmap mask. - Merge publish-admitted metadata search deltas from `NRT Delta Store (Metadata Search)` under the same admitted snapshot boundary. - Return top-K metadata search candidates with `doc_id` and scoring features. ## 8. Semantic Search Reader Deep Dive (SPFresh + RaBitQ) ### 8.1 SPFresh for Incremental ANN - Graph-based ANN structure optimized for frequent incremental updates. - Supports append/update/delete without full offline rebuild. - Mutable frontier on ingest path, immutable graph snapshots in S3. ### 8.2 RaBitQ for Storage Efficiency - Binary quantization without training stage. - Reduces vector footprint and memory bandwidth. - Enables large candidate scans with lower I/O and cache pressure. ### 8.3 Semantic Data Layout For each semantic segment object: - quantized vector codes (RaBitQ) - optional residual/full-precision blocks for rerank - SPFresh graph adjacency/layers - doc-key mapping and commit metadata - tombstone metadata (or external bitmap sidecars) ### 8.4 Semantic Search Query Execution Path 1. Retrieve candidate nodes using SPFresh traversal over quantized representations. 2. Apply optional Query Broker pre-filter payload (bitmap, Bloom filter, or explicit PK set). 3. Filter by `effective_read_ts` visibility and tombstones. 4. Merge publish-admitted unflushed semantic deltas from `NRT Delta Store (Semantic Search)` under the same admitted snapshot boundary. 5. Optional rerank using higher-fidelity vector payload. 6. Return top-K semantic search candidates. ## 9. Hybrid Query Planning and Fusion ### 9.1 Query Broker Planning Given one admitted broker-dispatch query after any projected-route fallback selection has already completed, Query Broker receives either a shared canonical `QueryRequest` or a trusted relational route wrapper (`ProjectedQueryRequest` / `PrimaryQueryRequest`) whose inner `query` remains the canonical semantic input: - treat `collection`, resolved descriptor `collection_truth_mode`, and `consistency` as already-normalized routing/fencing inputs. - interpret `search_payload.global_filter` as the global hard filter and `search_payload.root_retrieval` as the single canonical retrieval root. - use `search_payload.global_candidate_topk` and `fetch_spec.limit` as distinct planning inputs: candidate generation budget vs final materialization budget. - when `search_payload.root_retrieval` is absent, execute the request as filter-only / aggregation-only evaluation over `search_payload.global_filter` (or `MatchAll` when that filter is also absent). - enforce one shared admitted snapshot boundary across all subplans: exact `consistency.read_ts` only when admitted as-is, otherwise the admitted `effective_read_ts` (including continuation/search-context token-bound snapshots and the projected-route best-effort overlay exception where gateway has already materialized `effective_read_ts = snapshot_read_ts`). ### 9.2 Broker-Local Fusion Heuristics for Internally Synthesized Mixed-Modality Plans For candidate doc `d`, one admissible broker-local heuristic is Reciprocal Rank Fusion (RRF): - `RRF(d) = sum_i 1 / (k + rank_i(d))` Where `i` spans independently produced metadata-search and semantic-search rank lists. Scope note: - this section describes a broker-local physical heuristic for internally synthesized mixed-modality plans. - it does not override the canonical request-level `RetrievalGroup.fusion_strategy` contract, where `UNSPECIFIED` falls back to `WEIGHTED_SUM` and explicit request payloads may choose `RRF` or `DIS_MAX`. - if the admitted request already carries an explicit retrieval tree/fusion declaration, Query Broker MUST preserve request-level fusion semantics and MUST NOT substitute a broker-local heuristic in its place. Post-fusion steps: - if `search_payload.dedup_by_doc_id=true`, collapse duplicate top-level `doc_id` candidates before pagination. - apply final `search_payload.global_filter` validation as the authoritative gate; clause-level pre-filter payloads are optimization-only and cannot relax eligibility. - optional second-stage rerank (`fetch_spec.rescore`) over a bounded Top-K window. - rerank evolution target is cross-encoder integration (for example BGE reranker class) with strict latency guardrails and deterministic tie-break (`doc_id asc`) on equal rerank scores. ### 9.3 Pre-Filtered ANN Execution Plan (Metadata Search -> Semantic Search) For strict filter-first queries, Query Broker uses a selectivity-aware CBO instead of a fixed execution path: 1. Selectivity estimation: - Query Broker requests metadata-search-side `ESTIMATE/COUNT` under the same admitted snapshot boundary. 2. Path A: high selectivity (small candidate set, e.g., < 10,000): - Metadata Search Reader returns explicit PK set. - Semantic Search Reader bypasses ANN graph traversal and performs RaBitQ-assisted brute-force distance on that bounded PK set. - This avoids masked-graph disconnect penalties and yields deterministic recall for filtered candidates. 3. Path B: low selectivity (large candidate set): - Metadata Search Reader returns compressed filter payload (Bloom filter or aligned bitmap), not raw PK list. - Semantic Search Reader runs masked ANN traversal and only evaluates nodes passing filter checks. - Query Broker performs final exact post-filter before response to eliminate Bloom false positives. Guardrail for both Path A and Path B: - Query Broker MUST enforce final `search_payload.global_filter` equivalence before emit/paginate; any pre-filtered ANN path is a physical optimization and MUST NOT widen final result eligibility. Optional optimization (recommended): - Co-compaction aligns metadata search/semantic search internal ordinals within the same segment bundle so metadata search filters can be transferred as Roaring bitmaps without PK string translation. ## 10. Storage Architecture: S3-Native LSM + NVMe Cache ### 10.1 S3-Native LSM - All durable data is immutable objects in S3: - sealed WAL chunks - WAL checkpoints/anchors - SSTables (metadata/key version index) - Tantivy metadata search segments - semantic vector/graph segments - visibility/tombstone bitmaps - manifests/catalogs - WAL is object-chunked (not append-log emulation), which bounds small-object fanout and keeps durability semantics explicit. - Compaction is object rewrite + manifest swap. - Heavy rewrite/delete operations are prohibited on ingest/query plane nodes and must run only in the isolated Background Compute Plane. ### 10.2 Manifest-Driven Atomicity - New object sets are published by writing immutable manifest version `N+1`, then compare-and-swap advancing the private committed-manifest pointer object `manifest_committed_head.json`. - Manifest JSON format is pinned in `docs/specs/core/manifest-format.md` (segment/object pointers, schema fence metadata, `applied_ts_*`, `cdc_resolved_ts`, `wal_replay_start_anchor`, and `wal_durable_anchor`). - Manifest writes are micro-batched per collection to bound S3 control-plane mutation rates. - Readers resolve `manifest_committed_head.json`, pin `(committed_manifest_version, manifest_checksum_or_etag)` for request lifetime, and reuse that pinned tuple for any continuation/search-context chain derived from the same logical query. - Rollback is pointer rollback to previous manifest if publish fails before commit. - After successful committed-pointer CAS, `Catalog Control Service` advances derivative `manifest_head_version` using CAS (`N -> N+1`) as post-commit metadata update. - If head advancement is delayed, readers still resolve the newest committed manifest through `manifest_committed_head.json`; CCS head freshness affects only fast-path lag detection/governance, not correctness. - Durability and visibility boundaries are separated: - WAL durable anchor confirms crash-recoverable durability. - committed-manifest-pointer CAS confirms read-model visibility boundary. ### 10.3 NVMe Cache Architecture and Affinity Routing - Query Broker uses consistent hashing on `(namespace_id, collection_id)` so requests land on a deterministic Reader subset. - Cache affinity maximizes reuse and avoids cluster-wide cache thrashing under 1B-collection cardinality. - Reader nodes tail manifest changes and asynchronously prefetch new segments/bitmaps before they are queried. - Cold-start readers enter a warmup phase and are marked ready only after mandatory hot objects are present. - Cache is disposable; on miss, readers fall back to S3/object manifests. ### 10.4 Cost-Aware Eviction, Tiering, and Local GC - Tiered placement: - RAM-pinned: roaring bitmaps, term dictionaries, ANN entry layers. - NVMe/mmap: postings lists, vector residual payloads, large graph blocks. - Admission/eviction uses cost-aware scoring, not pure LRU: - `score = (access_frequency * s3_fetch_latency_ms) / object_size_mb` - TinyLFU-style admission avoids one-off large scans polluting cache. - Local cached files are actively purged by manifest diffing once objects are no longer referenced. ### 10.5 Background Compaction Execution Model Trigger policies are workload-aware (not time-only): - Read amplification trigger: compaction when query fan-in crosses configured segment-count threshold. - Space amplification trigger: compaction when obsolete/tombstone density exceeds threshold (for example, 30 percent). - Small-object consolidation trigger: merge tiny fragmented objects within one collection-scoped object set to reduce S3 GET cost. Execution model: - Compaction workers read source objects from S3, rewrite merged targets, then attempt CAS manifest swap. - If concurrent writes advanced during long compaction, workers apply delta tombstones before final CAS. - Background queues are CPU/I/O budgeted separately from ingest/query pools to prevent latency regression. - Vector rewrite policy is governed by the effective three-axis tuple `(DataTier, FieldAccessIntent, VectorRetrievalIntent)` rather than by tier alone: - `HIGH_FIDELITY`: - compaction MUST preserve the payload required for exact rerank/audit-grade replay, - first-stage ANN/index encoding MAY still be quantized when policy allows, but a rerank-capable residual/raw payload MUST remain referenced by the committed manifest, - colder tiers MAY relocate that protected payload to slower objects, but MUST NOT drop it. - `STANDARD`: - compaction MAY rewrite between admitted balanced layouts (`NONE`, `INT8`, `BINARY`, or successor equivalents) according to service policy, - residual payload MAY be preserved in hot/warm windows and MAY be dropped in colder tiers when recall targets for the effective profile remain satisfiable without exact rerank. - `ECONOMY`: - compaction MAY choose the densest admitted quantization layout for the current service profile, - residual/raw rerank payload is optional and MAY be dropped at any tier. - Structural intent influences how aggressively those rewritten artifacts are kept hot after publication: - `CRITICAL` biases NVMe/RAM residency and prefetch for the surviving vector/index blocks, - `ARCHIVAL` allows the same admitted encoding to remain object-store-first even when the row/window is still in a hotter temporal tier. - Rewrites MUST be monotonic with respect to the admitted schema generation: - compaction MAY realize a cheaper layout only if that layout still satisfies the generation's effective quality contract, - compaction MUST NOT manufacture a stronger contract than the admitted generation and then expose it as if schema had changed, - any rewrite that would require changing the effective quality contract belongs to a fenced schema evolution, not an opportunistic background rewrite. ### 10.6 Catalog-Control State Model Catalog records are control-plane authoritative metadata, not payload storage: - namespace catalog record: - stable `namespace_id`, namespace lifecycle state, policy profile, quota class, audit config. - routing ownership: - first allocation resolves on the namespace-name lane because no stable identity exists yet, - once `namespace_id` exists, namespace descriptor mutation and namespace lifecycle transitions serialize on the namespace-owned control lane, - the namespace-name lane remains responsible for first allocation, recreate arbitration, and the final namespace-name reservation release after lifecycle GC has established that the tombstoned lineage is no longer active. - private reservation/cache state: - `namespace_name_allocation_epoch` for ABA-safe name reuse after lifecycle GC release, - `collection_name_binding_revision` for namespace-scoped invalidation of positive and negative `CollectionNameBinding` caches. - namespace-scoped quota state and `quota_epoch` for the private CCS-owned query/write lease-refill surface. - internal lifecycle states in this revision: - `ACTIVE`: namespace is listed/readable, successful public namespace descriptors project the `active` lifecycle branch, and ordinary namespace metadata mutation plus child collection admission are admitted. - `QUIESCING`: namespace remains listed/readable, successful public namespace descriptors project the `quiescing` lifecycle branch, `PutNamespaceMetadata` updates to existing namespace state remain admitted, but child collection allocation MUST be rejected. - `TOMBSTONED`: namespace is omitted from `ListNamespaces`, `GetNamespaceMetadata` fails closed as `SF_RESOURCE_NOT_FOUND`, same-name recreation remains blocked until internal lifecycle GC releases the reservation, child collection admission MUST fail closed, and this state never appears in a successful public namespace descriptor. - transitions into `QUIESCING` / `TOMBSTONED` are private CCS/admin workflows outside the current public mutation surface and are serialized on the namespace-owned control lane together with post-allocation namespace mutation; final namespace-name reservation release happens later on the namespace-name lane after lifecycle GC confirms the release preconditions. - transition into `TOMBSTONED` is admitted only after every child collection is already in a terminal hidden state, so namespace tombstoning cannot strand publicly visible descendant collections. - typed namespace descriptor projection: - `lifecycle` is the CCS-owned readable-state projection of authoritative namespace lifecycle: `ACTIVE` and `QUIESCING` map directly, while `TOMBSTONED` is not projected because tombstoned namespaces are omitted/not-found on successful public reads. - `encryption` is synchronous descriptor state written by namespace mutation. - `approx_stats` and `index` are CCS-owned typed metadata projections for namespace readback. - `completeness` is a CCS-owned derived public projection computed from the currently authoritative lagging namespace descriptor fields (`approx_stats`, `index`) and the canonical required-field set; it is not an independently authored mutable state record. - missing `encryption` is not a publication-lag branch in this design; successful descriptor reads must already carry the synchronous namespace encryption descriptor. - namespaces with zero active child collections MUST synchronously publish the zero-child aggregate plus `index=up_to_date` so successful public descriptors do not remain artificially incomplete. - exact field presence, missing-field encoding, and fail-closed public-read behavior are owned by the canonical API contract, not this architecture document. - collection catalog record: - stable `collection_id`, authoritative `collection_truth_mode`, active schema-generation pointer, admitted lifecycle/retention policy, manifest head pointer, and private `collection_name_allocation_epoch` for ABA-safe same-name recreation after tombstone GC release. - routing ownership: - first allocation resolves on the collection-name lane because no stable identity exists yet, - once `CollectionNameBinding{namespace_id, collection_name}` resolves to an existing `CollectionRef`, descriptor mutation, schema mutation, and drop serialize on that collection-owned control lane, - the collection-name lane remains responsible for first allocation, recreate arbitration, pre-identity resolution, and the final collection-name reservation release after tombstone GC has established that the dropped lineage is no longer active. - private admission-state projection: - internal collection lifecycle states in this revision: - `ACTIVE`: collection is eligible for `ListCollections`, `GetCollectionMetadata`, `GetCollectionSchema`, and write/schema admission subject to the ordinary ownership and fence checks. - `TOMBSTONED`: collection is omitted from `ListCollections`, `GetCollectionMetadata` / `GetCollectionSchema` fail closed as `SF_RESOURCE_NOT_FOUND`, new write/schema admission fails closed, and same-name recreation remains blocked until tombstone GC releases the reservation. - implementations may keep stricter transient admin substates internally, but the private admission-state read model consumed by gateways MUST collapse them into deterministic admission semantics rather than exposing ad hoc state names. - descriptor revision / etag used by internal gateway cache validation and invalidation. - query-admission fragments such as the effective query-window bound consumed by Query Gateway and other read-path admission checks; when `serving_readiness.state=READY`, that bound MUST be a positive resolved value, and when readiness is `SCHEMA_BOOTSTRAP` or `SCHEMA_CONVERGENCE`, the private carrier MUST return explicit sentinel `0` because no READY serving generation is query-admissible yet. - write-admission fragments such as the active write-admission `schema_generation_token`, `DocIDMapping.generation_policy`, and `CollectionSettings.allocator_admission_policy` (`caller_profile`, `max_plan_items_per_request`, `max_forward_reservation_jump`) consumed by Ingest Gateway and `IdAllocatorService`. - write-admission fragments such as, for lifecycle-enabled collections, the exact payload-carried `time_partition_field` contract (`field identity`, admitted type, and required non-null carry on online `upsert` / `patch`). - pre-schema bootstrap rule: - collection identity may exist before the first schema generation is admitted. - while no active schema generation exists yet, serving-side schema/readiness/statistics operations and query execution follow the canonical schema-bootstrap-unavailable branch owned by the API and error-registry documents, not a generic not-found path. - serving-generation rule: - public active-schema readback may expose a newer active generation while it is still `APPLYING`. - query serving, readiness resolution, and public optimizer-statistics serving MUST continue to use the latest `READY` generation until convergence reaches `READY`. - continuation/search-context validation uses the token-bound serving lineage as long as that lineage remains retained and query-admissible; later `READY` promotions do not invalidate already-issued query tokens by themselves. - if an active schema lineage exists but no `READY` generation exists yet, serving-side query/readiness/statistics behavior follows the canonical schema-convergence-unavailable branch owned by the API and error-registry documents. - collection-name reservation release is a private durable control fact: - tombstone GC eventually records collection-name reservation release through an internal CCS control intent/outcome before same-name recreation becomes admissible again, - recovery must therefore distinguish `still tombstoned` from `tombstone reservation already released`. - typed collection descriptor projection: - public typed collection descriptors are served only for lifecycle state `ACTIVE`. - `encryption` is synchronous descriptor state written by collection mutation. - `approx_stats` and `index` are CCS-owned typed metadata projections for collection readback. - `completeness` is a CCS-owned derived public projection computed from the currently authoritative lagging collection descriptor fields (`approx_stats`, `index`) and the canonical required-field set; it is not an independently authored mutable state record. - missing `encryption` is not a publication-lag branch in this design; successful descriptor reads must already carry the synchronous collection encryption descriptor. - descriptor completeness remains fail-closed for metadata-parity consumers; exact required-field legality and missing-field encoding are owned by the canonical API contract, not this architecture document. - schema generation record: - field descriptors, analyzer/hash config, vector dimension/model metadata, compatibility policy, convergence state, and control-signal evidence used to move the generation through `APPLYING` / `READY` / `DRIFTED` / `ERROR`. - optimizer statistics snapshot record: - CCS owns immutable per-collection optimizer-statistics snapshots plus a rebuildable hot projection used to satisfy `GetCollectionStatistics` without mandatory steady-state scans of historical objects. - snapshots carry `collected_at_unix_ms` freshness clock as persisted optimizer-statistics freshness state for successful statistics reads. - durable snapshot identity is the composite tuple (`applied_schema_generation_token`, `stats_commit_ts`); `stats_commit_ts` alone is not sufficient because schema-lineage promotion can require a new zero-row snapshot without advancing the user-data frontier. - snapshot selection is always against the current query-serving `latest READY` lineage, not merely the newest active schema lineage. - CCS must publish an initial zero-row snapshot when a lineage first becomes query-serving-admissible. - optimizer-statistics lineage/freshness invariants for this architecture note are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-QUERY-003`. - exact request legality, freshness-budget miss behavior, tuple-family admissibility, and response ordering are canonical API/error-registry concerns and are intentionally not redefined here. - lifecycle window record: - one rebuildable per-window projection keyed by `(namespace_id, collection_id, window_start_ms, window_end_ms)`. - derived from the admitted collection `LifecyclePolicy` plus the latest committed manifests for that collection. - records effective `DataTier`, the manifest version that published that tier, next transition deadline, and lifecycle retention deadline. - drives background lifecycle scheduling and namespace aggregate freshness, but query-time `target_tiers[]` pruning still reads manifest-published tier metadata directly from the pinned query snapshot rather than consulting this CCS projection. - collection tombstone record: - dropped collection identity, drop revision, dropped-at timestamp, name-reservation horizon, and GC eligibility. - manifest head record: - current head version, expected previous head, last update epoch, and reconciler status (`IDLE`, `REPAIR_PENDING`, `REPAIRING`, `ERROR`). Boundary clarification: - public catalog read RPC payloads use the canonical protobuf descriptor messages (`NamespaceDescriptor`, `CollectionDescriptor`) defined by the API contract. - descriptor-level `approx_stats` is a lightweight public metadata projection, while `GetCollectionStatistics` reads the CCS-owned optimizer-statistics snapshot family; the two products are intentionally distinct and MUST NOT be collapsed into one weaker store. - public namespace descriptor mutation uses `PutNamespaceMetadata`; public collection descriptor mutation uses `PutCollectionMetadata`. - internal catalog/control records may contain additional state such as manifest-head tracking and reconciler bookkeeping, but that state is private control metadata rather than part of the public descriptor wire shape. Consistency contract: - Catalog writes use CAS with monotonic version checks. - Descriptor/schema updates that affect admission must complete before admitting writes against the new config. - Manifest head lag is tolerated (bounded) and repaired by reconciler; manifest commit correctness remains primary. ## 11. TiDB Query Integration (Bounded Scope) ### 11.1 Boundary Statement In query direction, retrieval integrates with TiDB in exactly two ways: 1. Auxiliary index acceleration for TiDB SQL. 2. External table federation from retrieval into TiDB. No other query-side coupling is part of this design. ### 11.2 Auxiliary Index Acceleration for TiDB SQL Purpose: - accelerate flexible multi-field structured predicates where fixed TiDB secondary indexes are ineffective or too expensive to enumerate, especially when query fields are not fully covered by retrieval inverted indexes. Model: - Retrieval planner first attempts metadata inverted-index intersection/union. - For predicates not fully satisfiable by inverted-index set algebra, retrieval falls back to auxiliary index structures keyed for ad-hoc combinations. - Retrieval maintains auxiliary index structures keyed for ad-hoc predicate combinations. - TiDB planner can delegate qualifying predicate evaluation/top-K candidate generation to retrieval. - Retrieval returns candidate `doc_id` values (and optional scores/features) to TiDB for final relational operations. Write ownership contract: - Tables participating in auxiliary index acceleration MUST target collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`. - Table schema MUST expose canonical `doc_id` source as: - `BIGINT UNSIGNED PRIMARY KEY`, or - `BIGINT UNSIGNED UNIQUE NOT NULL` column, - and must satisfy the active `DocIDMapping` contract (current implementation: `SCALAR_FIELD_TYPE_UINT64` only). - Data mutations for these collections are accepted only from `TiDB -> TiCDC`. - Native search write APIs targeting these collections MUST be rejected with `409 SF_OWNERSHIP_CONFLICT`. Nested data-model boundary for this mode: - `NestedConfig` / canonical structured carriers (`StructDocument`, `ObjectArrayValue`) are part of the shared collection data model and are most commonly exercised on `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` write routes. - In `COLLECTION_TRUTH_MODE_EXTERNAL`, `TiCDC -> CDC Adapter` is treated as row-oriented relational change propagation; cross-table real-time assembly into deep nested document arrays is an unsupported anti-pattern in this revision. - If relational child-table acceleration is required for aux-index workloads, it must use flat/indexable relational projections in current phases, with parent-child join style extensions deferred to future dedicated capability work. Execution flow: 1. TiDB planner identifies query fragments eligible for retrieval auxiliary index acceleration. 2. Planner obtains/propagates `read_ts`. 3. TiDB canonicalizes the remote predicate fragment into `RelationalQueryService.ExecuteProjectedQueryOrFallback`. 4. Query Gateway evaluates canonical malformed/error branches first, then TiDB-owned fallback branches (stale strict, lag-budget breach, non-covering projection) without requiring a separate external `GetReadiness` round trip. 5. If the gateway returns `fallback`, TiDB executes its native plan under the same snapshot target. 6. If the gateway returns `success`, Query Broker executes under the admitted snapshot boundary and returns candidate set. 7. TiDB completes remaining SQL operators (joins/aggregations/projections). ### 11.3 External Table Federation Expose retrieval-managed collections as TiDB federated tables, e.g.: - `external_hybrid_index(namespace_id, collection_id, doc_id, text, metadata_json)` `score` is not durable collection schema in the canonical contract. If TiDB exposes ranking output for federated retrieval queries, it SHOULD do so as a virtual query projection column rather than a persisted external-table field. Likewise, row-version metadata is exposed canonically as query-time `QueryHit.version_commit_ts`, not as durable collection schema. If TiDB chooses to expose `commit_ts`/version metadata to SQL consumers, it SHOULD do so as a virtual projected column derived from the query result rather than as a persisted external-table field. Vector payloads or handles are likewise query-time projection concerns, not durable canonical collection-schema columns. If TiDB needs vector-return or vector-handle semantics for federated reads, it SHOULD expose them as virtual projected columns derived from the query result rather than as persisted external-table fields. Benefits: - SQL-native access path for retrieval-resident datasets. - federated queries across TiDB OLTP tables and retrieval collections. - snapshot alignment through propagated `read_ts`. Search-Fabric-truth federated collection contract: 1. `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`: - Search Fabric is the command-truth owner for the collection. - TiKV stores no federated-table row truth for this collection. - command ingress MAY come from native search APIs or from TiDB federated SQL remote DML through `Ingest Gateway`. - TiDB federated reads execute directly against Search Fabric at query time; TiDB does not maintain a mirrored TiKV row set for the collection. - TiCDC ingestion for the collection is disabled. - TiDB direct writes in this mode are remote DML and therefore do not imply TiKV-backed local transaction semantics. ### 11.4 Pushdown Boundaries and Anti-Scan Guardrails Whitelist pushdown (only): - vector distance ordering with bounded Top-K (`ORDER BY vec_distance(...) LIMIT K`) - text matching (`MATCH ... AGAINST` equivalent forms) - partial aggregate pushdown with `GROUP BY` for supported functions (`SUM`, `COUNT`, `MIN`, `MAX`, `AVG`) under strict limit and guardrails - sargable scalar predicates with ordered execution preference: - metadata inverted-index intersections/unions first (`=`, `<`, `>`, `IN`, `AND`, `OR`, `IS NULL`) when index coverage exists; - auxiliary index or bitmap/Bloom runtime filters when inverted coverage is partial; - indexed fast-fields as bounded fallback path. Readiness/fallback behavior by resolved route plus `collection_truth_mode` is defined in `docs/specs/core/search-fabric-api.md`; this document only states the architectural boundaries. Contract details are defined in `docs/specs/core/search-fabric-api.md`. Hard guardrails: - Every pushdown fragment must carry deterministic `LIMIT K` (platform cap for TiDB pushdown paths, for example `K <= 10000`) or highly selective point conditions. - External API profiles may enforce stricter Top-K limits at gateway level (for example Elasticsearch compatibility profile `Top-K <= 1000`); profile-level limits are authoritative for those APIs. - Unbounded scans against retrieval tier are rejected with fatal planner/runtime error. Runtime filter support: - large `IN`-list pushdown for index-join style execution. - serialized Bloom filter pushdown for hash-join pruning. Fallback: - Unsupported operators (unsupported aggregate functions or `GROUP BY` forms, window functions, complex string functions, unrestricted `LIKE`) are evaluated in TiDB on truncated result sets. - strict and bounded-staleness reads propagate caller `read_ts` into retrieval RPC payloads for snapshot alignment. - best-effort reads MUST send `read_ts=0`. - shared-query best-effort execution uses retrieval `safe_ts`. - projected-route overlay-bearing best-effort execution is the explicit exception: trusted callers still send `read_ts=0`, but gateway must materialize `effective_read_ts = snapshot_read_ts` for the broker dispatch rather than synthesizing an unrelated snapshot timestamp; see Section `9.1`. ### 11.5 CQRS Consistency Contract (Command vs Query) Command ownership: - For collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`, business-critical ACID writes are committed in TiDB first. - TiDB commit order is the canonical event order for downstream projections on these collections. - For collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`, command ingress is Ingest Gateway; admitted callers may be native search APIs or TiDB federated SQL remote-write gateway, and neither TiKV nor TiCDC is the command-truth owner. Projection semantics: - Search-Fabric-owned write gateways (native search APIs and TiDB federated SQL remote DML) are normalized into the same internal envelope and pipeline. - Change propagation to retrieval indexes is asynchronous but ordered by `commit_ts`. - Exactly-once logical effects are enforced via canonical stream/batch idempotency keys, replay-safe retransmit handling, and fail-fast authoritative Flow-A ingestion (`continue_on_error=false` on the CDC-authoritative lane) when aux-index CDC projection is active. - Per-collection progress is tracked by `applied_ts_*`; query serving safety bound is `safe_ts`. - Cross-modal alignment key is canonical `uint64` `doc_id` for both metadata search and semantic search projections. Read semantics exposed to callers are defined in Section `5.2 Safe Visibility for Hybrid Reads` plus the canonical API consistency contract. For this TiDB-boundary section, only the integration-specific consequence is restated here: - TiDB `aux_index` callers surface strict / bounded-staleness miss handling through the deterministic fallback branch on `RelationalQueryService.ExecuteProjectedQueryOrFallback`. - TiDB federated-table callers use `RelationalQueryService.ExecutePrimaryQuery` and therefore inherit the same fail-closed read semantics defined in Section `5.2` and the canonical API contract, with no TiDB-specific fallback outcome and no TiKV-backed snapshot-overlay exception in the active Search-Fabric-truth model. TiDB SQL immediate-visibility contract (MySQL Profile 3 path): - Aux-index path: - applies to trusted TiDB query routes that carry TiDB-private delta overlays, - after TiDB acknowledges a write success, TiDB SQL reads expose read-your-write through a merged view of the latest retrieval-visible committed version plus TiDB-visible incremental changes after that version. - Federated-native path: - TiDB direct remote DML waits for Search Fabric visibility publication before ACK, - subsequent TiDB federated reads observe read-your-write from the committed remote Search Fabric view rather than from a TiKV-backed overlay. - This is a TiDB-side visibility contract on the trusted TiDB query surface; public/shared retrieval APIs remain governed by `safe_ts`-bounded asynchronous projection semantics above. Failure and recovery model in CQRS terms: - Projection workers (CDC Adapter / native ingestion) are replayable from checkpoints. - Manifest publish is the atomic read-model version boundary. - On restart, rebuilding from manifest + WAL/checkpoint re-establishes query model without command-side ambiguity. Operational SLOs specific to CQRS: - projection lag checks that are evaluated against time budgets MUST use TSO physical-ms projection (`tso_physical_ms(commit_ts) - tso_physical_ms(applied_ts)`), not raw TSO subtraction. - alerting is tied to `safe_ts` lag and replay backlog growth. ### 11.6 Partitioned-Table Support Design (Phase-4 TiDB Enhancement Path) Scope: - This section defines the architecture-level contract for partitioned-table support committed in the Phase-4 TiDB enhancement track. - Normative applicability is `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`. - External MySQL-facing compatibility semantics for this capability are owned by `docs/compatibility/mysql-fulltext.md` Profile 3; `docs/compatibility/tidb-enhancement.md` owns delivery/readiness, while this section owns only the internal architecture contract. Data and identity model: - Canonical identity remains collection-scoped `uint64` `doc_id`; partition is routing metadata, not identity replacement. - TiCDC payloads carry partition provenance (`partition_id` or equivalent physical partition handle) and are normalized into the canonical write envelope with stable `doc_id`. Ingestion and indexing: - `CDC Adapter` and `Ingest Gateway` MUST resolve partition-aware descriptors from `Catalog Control Service` before admission. - Metadata search indexes and semantic indexes MAY be physically partition-sharded, but manifest/catalog state MUST expose one logical collection snapshot boundary per `read_ts`. - Partition-level artifacts are published under deterministic partition-aware object prefixes and referenced through collection-level manifest metadata. Query and snapshot semantics: - Planner performs partition pruning first when predicates/partition metadata allow deterministic elimination. - Query API SHOULD carry explicit partition-pruning hints (`SearchPayload.routing.routing_keys` and/or `SearchPayload.routing.partition_filter`) so gateway/planner can skip irrelevant object ranges before S3 segment reads. - `partition_filter` references only declared partition keys; malformed/non-partition pushdown input is rejected fail-closed at admission. - For a fixed admitted snapshot boundary for one `(namespace_id, collection_id)` query execution, results MUST be snapshot-consistent across all touched partitions under the same `commit_ts` visibility rules. - `safe_ts` evaluation for partitioned tables is computed over all touched partition shards in both metadata and semantic paths. DDL and lifecycle: - Partition add/drop/reorg operations are schema-generation events managed by `Catalog Control Service`. - Migration between partition layouts uses dual-generation manifests with controlled cutover; old-generation partition artifacts remain query-safe until GC eligibility. Recovery and correctness: - Replay anchors/checkpoints MUST be partition-aware to support partial-partition recovery without violating collection-level ordering guarantees. - Conflict resolution remains `commit_ts`-first, but replay/idempotency acceptance MUST continue to use the canonical stream/batch identity contract; partition relocation MUST NOT create duplicate visible versions or invalidate replay-safe retransmit. ## 12. Reliability, Recovery, and Operations ### 12.1 Failure Domains - Query/ingest execution nodes are durability-stateless and can be replaced without data loss; short-lived NRT/replay caches are ephemeral and reconstructable. - Control-plane services (`Commit Coordinator`, `Visibility Barrier`, `Manifest Publisher`, `Catalog Control Service`) rely on replicated durable state and fenced leadership. - Durable failure boundary for long-lived system truth is S3 + manifest/catalog metadata. ### 12.2 Recovery - On restart, services reload latest committed manifest versions. - `Manifest Publisher` and `Query Broker` discover the reader-visible committed manifest through `manifest_committed_head.json`; CCS `manifest_head_version` is a derivative cache/governance pointer rather than the authoritative visibility boundary. - Recovery loads `wal_replay_start_anchor` and `wal_durable_anchor` to determine replay window. - WAL replay reconstructs unflushed delta states only within anchored replay bounds. - WAL chunks not referenced by durable anchors are ignored as orphan/uncommitted tails. - Incomplete publishes are ignored unless the committed-manifest pointer CAS exists. - PIT/search-context continuation remains recoverable after restart because the immutable snapshot pin is fully self-contained inside the signed token; there is no separate registry state to rebuild. - Ordinary query-page continuation also remains recoverable after restart because the signed page token is self-contained and carries the pinned snapshot tuple, token-bound page ordinal, and the last-emitted canonical sort cursor; there is no separate page-session registry to rebuild. ### 12.3 Stateful Service Failure Recovery and Reconciliation Stateful service set: - `Commit Coordinator` - `Visibility Barrier` - `Manifest Publisher` - `Catalog Control Service` - `IdAllocatorService` - `Background Compute Coordinator` Shared recovery principles: - leader election with fencing tokens/epochs; stale leaders cannot commit control mutations. - idempotent retry for all cross-service commands (`publish_intent`, `head_advance_intent`, `reconcile_scan`). - stateful services do not all share one identical durable-authority pattern; each service owner document MUST define its own minimum authoritative recovery set and any rebuildable derivative artifacts explicitly. - commit-authoritative command/outcome logs in S3 are the authoritative recovery pattern for CCS/manifest-related control paths that define them as such; they are not an implicit universal model for every stateful service. Service-specific recovery authority matrix in the current revision: | Stateful service | Minimum authoritative recovery set | Notes | | --- | --- | --- | | `Commit Coordinator` | coordinator durable checkpoint / logical watermark state | New leader restores the last committed logical watermark from durable checkpointed state before issuing more `commit_ts`. | | `Visibility Barrier` | durable barrier log / pending-write state | Recovery replays pending write states and reissues publish intent idempotently. | | `Manifest Publisher` | immutable manifest lineage + `manifest_committed_head.json` + publisher post-commit task state | Visibility truth follows the committed-manifest pointer, not CCS head freshness. | | `Catalog Control Service` | CCS recovery checkpoint + ordered `catalog-recovery-segments` + authoritative `APPLIED` command/outcome records + namespace quota state + committed-manifest-pointer validation inputs | Public descriptors, schema generations, optimizer-statistics snapshots, tombstones, replay indexes, and derivative head state are rebuilt/materialized from this CCS-owned authority set. | | `IdAllocatorService` | allocator-owned `state/...` objects + immutable `replay-success/...` objects | Optional `failure-audit/...`, `recovery-segments/...`, CCS-side exports, and TiDB-side exports are non-authoritative. | | `Background Compute Coordinator` | active manifest lineage + CCS lifecycle/tombstone scheduling state | Recovery is rebuild-only in the current revision: coordinator-owned queues, leases, debt scores, and task/fence records are disposable derivatives rather than durable authority, and duplicate background work must remain harmless through worker revalidation plus owner-side idempotent handoff. | Failure cases and required behavior: 1. `Commit Coordinator` leader failover: - new leader restores last committed logical watermark from durable checkpoint. - duplicate `commit_ts` allocation is prevented by fenced epoch and monotonic watermark restore. 2. `Visibility Barrier` crash after writer durable ACK but before publish intent: - barrier replays pending write states from durable barrier log and reissues publish intent idempotently. - duplicate intents are harmless because `Manifest Publisher` enforces per-collection CAS progression. 3. `Manifest Publisher` crash between manifest CAS success and catalog head update: - on restart, publisher reloads `manifest_committed_head.json`, replays unresolved post-commit tasks, and re-sends `head_advance_intent` when committed pointer version is ahead of CCS head. - if `manifest_committed_head.json` points at `N+1` while CCS head is still `N`, readers continue to pin `N+1` and CCS reconciliation advances only the derivative head pointer. - query correctness remains based on the committed-manifest pointer, not on immediate CCS head freshness. 4. `Catalog Control Service` crash or partition: - admission traffic that cannot validate ownership/schema against fresh authoritative CCS state is rejected fail-closed; implementations must not substitute TTL-only stale-descriptor reuse for revision/etag validation in the canonical contract. - after recovery, reconciler loads the latest compacted CCS recovery checkpoint, replays only post-checkpoint ordered `catalog-recovery-segments`, materializes all CCS-owned control objects and online projections referenced by `APPLIED` command/outcome records (including namespace descriptors, collection descriptors, schema generations, optimizer-statistics snapshots and head state, tombstones, replay indexes, namespace quota state, and manifest-head state), restores committed-manifest-pointer state from checkpointed state plus `manifest_committed_head.json` validated against committed manifest lineage, and then repairs `manifest_head_version` gaps against that committed-manifest pointer. - after recovery, reconciler also restores namespace quota state from `namespace_quota_state.json`, bumps `quota_epoch` when stale in-flight gateway leases cannot be trusted, and forces fresh gateway lease acquisition before ordinary throttling decisions resume. - reconciler emits audit events for every repaired head transition. 5. `IdAllocatorService` leader failover or partition: - fresh allocation traffic remains fail-closed until the active allocator owner for the affected collection shard has rebuilt authoritative in-memory state from allocator-owned durable objects (`state/...` and `replay-success/...`) under a new fenced allocator epoch. - allocator recovery restores watermark state, replay-success lookup state, and any leased-range ownership facts from allocator-owned durable authority; CCS does not reconstruct allocator replay truth or allocator watermark history on its behalf. - once allocator-owned state is rebuilt, the recovered owner re-reads CCS admission facts for current lifecycle / truth-mode / policy / schema-generation admissibility before serving fresh `AllocateIds`; identical replay hits remain satisfiable from restored allocator replay truth even when CCS now rejects fresh allocation. - any optional CCS-side or TiDB-side allocator admission/export projections may lag recovery, but they are not allowed to become the replay or watermark source of truth. 6. `Background Compute Coordinator` acting on stale head: - background tasks must verify target manifest version exists and is active before compaction/GC actions. - GC uses active-manifest reachability checks, never catalog head alone, to avoid premature deletion. Operational runbook gates: - page if `catalog_head_lag_versions > threshold` or lag duration exceeds SLO. - page if unresolved `manifest_committed && head_not_advanced` backlog grows. - block destructive maintenance when reconciliation backlog is above safety threshold. ### 12.4 Consistency Guardrails - Per-collection `safe_ts` monitoring. - request-level admission lag alarms for strict/bounded reads when `tso_physical_ms(requested_read_ts) - tso_physical_ms(effective_read_ts)` exceeds the configured millisecond SLO. - per-collection freshness alarms when `now_ms - tso_physical_ms(safe_ts)` exceeds the configured millisecond SLO. - conflict retry metrics for bulk overlap. ### 12.5 Observability Key metrics: - ingest qps, WAL chunk seal latency, WAL chunk PUT latency, flush throughput - metadata search/semantic search applied_ts and safe_ts lag (tracked in both raw TSO units for ordering diagnostics and physical-ms projection for time-budget SLOs) - WAL replay backlog (chunks/bytes) and replay catch-up lag - bitmap mask hit ratio and filter cost - S3 GET/PUT latency and cache hit rate - RRF fusion latency and recall-at-K - catalog descriptor read latency/error rate and admission-cache revalidation failure count - `manifest_committed_not_head_advanced` backlog size/age - catalog reconciliation sweep throughput and repair count ### 12.6 Serving Performance Targets (Initial) Serving-first targets (explicit in technical design and directly verifiable in release gates): - Query latency: - metadata search latency p99: <= 50 ms - semantic search latency p95: <= 200 ms - hybrid query latency p95 (K<=100): <= 500 ms - Query throughput: - hybrid search throughput (sustained): QPS >= 5000 under the Phase-3 single-collection quantitative envelope. - Write-to-query freshness and visibility: - native write-to-search visibility p99 (strict mode): <= 2 s - `safe_ts` lag p99 under steady load: <= 3 s - TiCDC end-to-end replication lag p99: <= 5 s - Write durability: - write durability ack p99 (interactive, `durability=wal_chunk_durable`): <= 150 ms - Control-plane convergence: - catalog head lag repair p99 (after manifest commit): <= 5 s - Milestone gate precedence: for release go/no-go, quantitative acceptance follows `docs/governance/roadmap.md` Section `6` gates; this SLO list is an operational baseline and MUST NOT relax those roadmap gates. ### 12.7 Asynchronous GC and S3 Safety Controls Two-phase GC: 1. Logical dereference: - After compaction manifest swap, old objects become unreferenced but are not deleted immediately. 2. Physical sweep: - GC workers compare active manifests with physical objects and delete only safe candidates. Delete horizon conditions (both required): - inactive in latest manifests. - `evict_commit_ts < global_safe_ts - retention_window`. - if the collection admits lifecycle retention, the affected lifecycle window/object set is also beyond its `LifecyclePolicy.retention_ms` deadline as published in the current committed manifest/window projection. S3 protection rules: - GC deletes use batched `DeleteObjects` calls. - Worker delete rate is token-bucket limited and adaptive to S3 `503 Slow Down` responses. - On S3 throttling, workers back off exponentially and preserve ingest/query S3 budgets first. ### 12.8 Stateful Control-Plane Simulation Framework (Phase-4 Verification) Purpose: - Verify scale and correctness of key stateful components that are hard to scale horizontally without requiring full physical-scale payload. Simulation scope (must include): - `Commit Coordinator` - `Visibility Barrier` - `Manifest Publisher` - `Catalog Control Service` - `IdAllocatorService` - `Background Compute Coordinator` Modeling approach: - Deterministic event-driven state-machine simulation with virtual time. - Cardinality-first load model: - projects `1B`-collection/namespace control metadata cardinality; - uses sampled payload volume while preserving production-like event shape (write skew, burstiness, replay backlog, compaction debt). - Trace-calibrated parameters from production/pre-GA-Readiness telemetry (`commit_ts` rate, publish fan-in, head-lag distribution, replay throughput). Fault matrix (minimum): - leader failover storms and epoch fencing races; - network partitions between stateful services; - duplicate/reordered/lost retry intents (`publish_intent`, `head_advance_intent`, `reconcile_scan`); - WAL replay spikes and delayed durable-anchor advancement; - S3 throttling/backpressure waves affecting publish and GC chains. Required invariants: - `commit_ts` monotonicity and no duplicate allocator issue/reservation outcomes across epochs or replay recovery. - Manifest CAS progression is monotonic (`N -> N+1`) with no split-brain visibility. - Catalog head never exceeds committed manifest lineage and converges within SLO after transient failures. - Replay and intent retries remain idempotent (no double-apply visibility effects). - GC never deletes objects reachable from any active manifest chain. Phase-4 gate outputs: - Invariant pass/fail report for each scenario. - Control-plane saturation curves (queue depth, lag, retry amplification, failover recovery time). - Capacity recommendation for shard/node envelope and hot-spot guardrails. ## 13. Multi-Tenant Isolation, Quota, and QoS - Every API call carries tenant auth context and strict `namespace_id/collection_id` ACL checks. - Object key prefixes and KMS keys can be tenant-scoped. - Query admission is cost-aware (RCU-based), not request-count-only. - One factorized RCU model governs public admission: ```text RCU_estimated = Base + ScanWork + ProjectionWork + AggregationWork + VectorWork ``` Where: - `Base` covers gateway normalization, authz, route resolution, broker dispatch, and fixed per-request overhead. - `ScanWork = W_scan * estimated_rows_scanned * TierMultiplier(max touched DataTier)`. - `ProjectionWork = W_proj * projected_field_units * StructuralMultiplier(weighted projected FieldAccessIntent set)`. - `AggregationWork = W_agg * lowered_aggregation_nodes`. - `VectorWork = sum over vector clauses(W_vec * effective_query_dimensions * clause_topk * TierMultiplier(target tier set) * QualityMultiplier(effective VectorRetrievalIntent))`. - `TierMultiplier`, `StructuralMultiplier`, and `QualityMultiplier` are service-policy coefficient tables rather than public wire fields, but they MUST be derived from the admitted three-axis schema contract rather than ad hoc backend heuristics. - Query Gateway owns the estimated-RCU decision and lease spend using this model; Query Broker consumes the admitted envelope plus the effective axis tuple for execution, but does not become a second public quota authority. - Query and write gateways both enforce mandatory per-tenant distributed token buckets. - Refill/leasing authority for those token buckets is a private CCS-owned quota-allocation surface; deployments MUST NOT invent a second independent quota authority. - If a gateway cannot establish or refresh a lease because that quota-allocation authority is unavailable, admission MUST fail as retryable backend/dependency unavailability rather than as tenant/API throttling; only real budget exhaustion or policy shedding maps to `429 SF_RATE_LIMITED`. - Tenant/API throttling and S3 throttling are distinct control loops: tenant/API throttling maps to API-level `429`; S3 throttling is internal object-store feedback handled by retries/backoff. - Batch APIs (`_msearch`, bulk query envelopes) are unrolled and rejected pre-flight if estimated cost exceeds tenant hard limits. - Adaptive load shedding is automatic under backend pressure: - return `429` + `Retry-After` for lower tiers. - strict mode MUST NOT silently downgrade to best-effort. - requests exceeding effective `K`/window/payload guardrails MUST fail closed with canonical limit errors instead of silently shrinking result cardinality. - Tenant tiering policy routes whale tenants to dedicated resource pools while long-tail tenants remain in shared pools. - Query Broker MAY still compute a runtime cost signal from actual CPU, S3/NVMe IO, bytes scanned, vector rerank work, and overlay merge work; it may pre-empt already-admitted work only when runtime cost materially overruns the admitted envelope or shard-local safety limits, and such pre-emption still projects as canonical `SF_RATE_LIMITED`. ## 14. Performance and Capacity Strategy ### 14.1 Scale Strategy - Shard by `(namespace_id, collection_id, key_hash)` and time windows. - Separate data plane (query + ingest) from control/background compute planes with isolated node pools and quotas. - Route whale tenants to dedicated query/ingest pools; keep long-tail tenants in shared pools with strict quotas. - Manifest-driven compaction scheduling and GC execution are asynchronous and S3-rate-aware. - Support two deployment profiles with the same module boundaries: cloud service-split for elastic scaling, private process-composed for simpler operations. ### 14.2 Cost Controls - Quantization-first semantic storage (RaBitQ) to shrink vector footprint. - Bitmap sidecars to avoid full segment rewrites on each update. - Consolidate tiny collections within collection-scoped object sets to reduce object-count overhead. ### 14.3 Capacity Envelope (Planning Baseline) Given 5 EB raw input: - Semantic vectors: - RaBitQ binary codes plus metadata search residual payload target 4-10x shrink versus raw float vectors. - Metadata Search (metadata/text): - compression plus immutable segment packing target 1.5-3x shrink depending on token entropy. - Sidecar overhead: - visibility/tombstone bitmaps budget 3-8 percent of metadata search segment bytes for high-churn collections. Operational planning note: - capacity model should be tracked as `raw_bytes -> indexed_bytes -> durable_s3_bytes -> hot_cache_bytes` per tenant tier, then fed into compaction and cache admission policies. ## 15. Trade-Offs ### 15.1 Chosen Trade-Offs 1. S3 as source of truth vs local replicated NVMe: - pros: major storage efficiency and simpler durability model. - cons: higher remote I/O sensitivity, stronger cache dependency. 2. Immutable segment model vs in-place updates: - pros: strong crash safety, append-friendly throughput. - cons: compaction complexity and delayed reclamation. 3. External visibility bitmaps vs rewriting Tantivy segments on updates: - pros: preserves immutability, reduces rewrite frequency. - cons: extra query-time bitmap operations and sidecar lifecycle management. 4. Commit-ts-first conflict policy with canonical same-ts order: - pros: preserves MVCC/LWW correctness while keeping deterministic same-ts ordering without source-kind special cases. - cons: batch jobs may still retry when overlapping hot ranges with recent streaming writes. 5. Decoupled Search Fabric layer vs embedding directly in TiDB storage: - pros: independent scalability and modality-specific optimization. - cons: federation complexity and remote execution planning cost. 6. Compatibility API layer (Elasticsearch/Turbopuffer read+write profiles) vs single proprietary API: - pros: faster ecosystem adoption and easier migration from existing search stacks. - cons: translation-layer maintenance, semantic-gap handling, and compatibility test burden. 7. Bounded TiDB query integration (aux-index + federated-table access only) vs broad deep coupling: - pros: clearer ownership boundary, lower coupling risk, and easier independent evolution. - cons: some advanced cross-system optimizations are intentionally out of scope. ### 15.2 Risks and Mitigations - Risk: bitmap sidecar explosion for high-churn collections. - Mitigation: checkpoint bucket compaction and TTL-based coalescing. - Risk: S3 tail latency impacts p99 query times. - Mitigation: aggressive NVMe caching, prefetch, and admission control. - Risk: WAL chunk object amplification drives S3 PUT cost and control-plane pressure under ultra-high write QPS. - Mitigation: adaptive chunk seal thresholds (`bytes/ops/age`), per-collection stream multiplexing, and periodic WAL checkpoint compaction. - Risk: safe_ts lag under heavy semantic indexing load. - Mitigation: adaptive ingest throttling and separate semantic pipeline backpressure. - Risk: S3 per-prefix throttling (`503 Slow Down`) under bursty tenants. - Mitigation: entropy-sharded keyspace, WAL stream shard scaling, control/data path split, adaptive API backoff, and batched deletes. - Risk: background compute bursts degrade online latency. - Mitigation: strict plane isolation with independent CPU/I/O budgets and backpressure-aware schedulers. ## 16. Rollout Plan Detailed rollout planning is maintained in `docs/governance/roadmap.md`. `docs/governance/roadmap.md` is the canonical source of truth for phase sequencing and scope.