# Catalog Control Service Date: 2026-03-13 Status: Frozen architecture note Authors: Search Platform Architecture ## 1. Scope This document defines the component-level architecture of `Catalog Control Service` (CCS), the stateful control-plane authority for: - namespace descriptor mutation and readback - collection descriptor mutation and readback - schema generation mutation and convergence status - lifecycle-policy materialization for tier-window scheduling and retention tracking - optimizer-statistics snapshot production and hot serving support for `GetCollectionStatistics` - private quota-allocation/refill authority for query/write gateway leasing - collection drop / tombstone lifecycle - collection manifest-head advancement and repair The canonical public contract remains: - `api/proto/searchfabric/v1/search_fabric.proto` - `docs/specs/core/search-fabric-api.md` - `docs/specs/core/error-codes.md` This document does not redefine public RPC semantics. It explains how CCS realizes them at production scale while keeping S3 as the durable source of truth for control objects and replay state. ## 2. Boundary ### 2.1 Public Control Surface CCS serves these canonical RPC families: - namespace metadata create/update/ensure: `PutNamespaceMetadata` - namespace catalog reads: `ListNamespaces`, `GetNamespaceMetadata` - collection metadata create/update/ensure: `PutCollectionMetadata` - collection catalog reads: `ListCollections`, `GetCollectionMetadata` - schema mutation and readback: `PutCollectionSchema`, `GetCollectionSchema`, `GetSchemaSyncStatus` - collection lifecycle delete: `DropCollection` Caller-ownership matrix for identity-bearing public control paths: | Scenario | Canonical caller | RPCs | Required `client_kind` | Ownership notes | | --- | --- | --- | --- | --- | | `COLLECTION_TRUTH_MODE_EXTERNAL` first-time bootstrap | TiDB DDL/bootstrap actor | `PutNamespaceMetadata`, `PutCollectionMetadata`, then later `PutCollectionSchema` / `DropCollection` as needed | `CLIENT_KIND_TIDB_DDL_ADAPTER` | TiDB owns first-create namespace/collection identity ensure for aux-index lineages; TiDB query/statistics/CDC hot paths consume resolved IDs only and MUST NOT allocate identity. | | `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` / federated provisioning | direct control-plane provisioner/admin | `PutNamespaceMetadata`, `PutCollectionMetadata` | `CLIENT_KIND_CONTROL_ADMIN` | Namespace/collection identity and encryption policy are provisioned ahead of TiDB traffic; TiDB callers fail closed on missing identity rather than allocating replacements. | | gateway-managed native-search ensure/apply (where the canonical API admits it) | Elasticsearch/Turbopuffer gateway control path | `GetCollectionMetadata`, `PutCollectionMetadata`, `PutCollectionSchema` | gateway caller kind (`CLIENT_KIND_ES_*` / `CLIENT_KIND_TP_*`) | Gateways may resolve or ensure Search-Fabric-truth collection identity inside their own namespace/account mapping, but they do not own TiDB aux-index bootstrap or TiDB federated provisioning. | ### 2.2 Private Control State CCS also owns private control metadata. Unless stated otherwise, this state is intentionally not projected into public descriptor payloads: - internal namespace lifecycle control state beyond the public readable-state projection (`TOMBSTONED`, lifecycle transition machinery, lifecycle GC horizon, reservation-release workflow state) - namespace-scoped quota state, active lease projections, and `quota_epoch` - per-collection committed-manifest pointer cache/repair state - per-collection `manifest_head_version` - manifest-head repair state - lifecycle-window hot projections and transition scheduling state - namespace policy/quota/audit bindings plus collection policy bindings - tombstone and GC scheduling state - replay/outcome index projection - descriptor, schema, and tombstone revision counters `NamespaceDescriptor.lifecycle` is the only public readable-state projection of namespace lifecycle in this revision: successful public namespace descriptors expose exactly one branch (`active` or `quiescing`), while `TOMBSTONED` and the internal transition/GC machinery remain private CCS control state. `manifest_committed_head` and `manifest_head_version` remain private control metadata. They MUST NOT appear as fields on public `NamespaceDescriptor` or `CollectionDescriptor`. Query search-context/PIT lifecycle is out of scope for CCS and remains fully token-driven inside the stateless query plane. ## 3. Request Routing and Cache Ownership ### 3.1 Routing Keys CCS uses four deterministic ownership lanes: 1. Namespace-name lane: - `PutNamespaceMetadata` requests route by `namespace_name` before a `namespace_id` exists. - namespace create/ensure is serialized on that lane. - once `namespace_name` resolves to an already allocated namespace lineage, the request MUST pivot onto the namespace-id lane before the authoritative mutation apply phase; the namespace-name lane is allocation/recreate arbitration only after first allocation. - `namespace_name_reservation_release_intent` is serialized on this lane because it releases the same `namespace_name` reservation that namespace create/recreate competes on. 2. Namespace-id lane: - `namespace_lifecycle_transition_intent` routes by `namespace_id` once stable identity already exists. - `GetNamespaceMetadata` routes by resolved namespace identity: direct `namespace_id` stays on this lane, while `lookup.namespace_name` first resolves on the namespace-name lane and then pivots to this lane before descriptor readback. - post-allocation `PutNamespaceMetadata` updates also serialize on that lane. - namespace-internal lifecycle progression is serialized on that lane after allocation. - the private quota-allocation/refill surface is also namespace-owned: requests serialize by `namespace_id`, while `api_plane` and `throttled_dimension` act as deterministic subkeys inside that namespace-owned quota state. 3. Collection-name lane: - `PutCollectionMetadata` and `GetCollectionMetadata.lookup.name_binding` route by `(namespace_id, collection_name)` before a `collection_id` exists. - once `CollectionNameBinding{namespace_id, collection_name}` resolves to an existing active lineage, `PutCollectionMetadata` MUST pivot onto the collection-id lane before the authoritative mutation apply phase; the collection-name lane is allocation/recreate arbitration plus pre-identity resolution only after first allocation. - internal `collection_name_reservation_release_intent` is serialized on this lane because it releases the same `(namespace_id, collection_name)` reservation that create/ensure and ABA checks depend on. 4. Collection-id lane: - `CollectionRef`-scoped requests (`GetCollectionMetadata.lookup.collection`, `PutCollectionSchema`, `DropCollection`, `GetCollectionSchema`, `GetSchemaSyncStatus`) route by `(namespace_id, collection_id)`. - post-resolution `PutCollectionMetadata` updates also serialize on that lane. - internal `head_advance_intent` is also serialized on this lane. This avoids the ambiguity in trying to hash all requests by `collection_id` when identity allocation may not have happened yet. Catalog-name inputs that participate in those lanes are admitted only after canonical validation: - valid UTF-8 Unicode scalar sequence - Unicode NFC - maximum 128 Unicode scalar values - reject control characters, bidirectional formatting/isolate controls, and default-ignorable format characters - no silent normalize-on-write Externally supplied request/replay identity strings that CCS persists or binds into replay/index state are also canonically bounded: - `RequestMeta.idempotency_key`: non-empty when required, maximum 128 Unicode scalar values - `RequestMeta.correlation_id`: optional, but when present it must be non-empty and maximum 128 Unicode scalar values - `RequestMeta.client_id`: non-empty when required, maximum 256 Unicode scalar values - `RequestMeta.principal_subject`: optional, but when present it must be non-empty and maximum 256 Unicode scalar values - overlength inputs are rejected as `SF_BAD_REQUEST`; CCS persistence never truncates or hashes identity-bearing strings before replay checks or public-read materialization ### 3.2 Cache Model - Gateways MAY keep short-lived L1 caches for namespace and collection descriptors. - Gateways MAY keep `CollectionNameBinding` cache entries only when each cached result, whether positive or negative, is bound to the namespace-scoped `collection_name_binding_revision` returned by the private CCS admission-state read model. - a cached positive `CollectionNameBinding -> CollectionRef` mapping becomes invalid as soon as the bound revision changes; TTL alone is not a sufficient ABA-safety rule. - CCS nodes keep local L2 caches keyed by: - `namespace_name` - `namespace_id` - `(namespace_id, collection_name)` - `(namespace_id, collection_id)` - `command_identity` - cache misses are coalesced with singleflight-style request suppression. - cache entries are disposable. - relational projection state is the authoritative online serving view. - S3 control objects and S3 command/outcome objects are the authoritative durable source. Public descriptor fields returned by CCS MUST come from authoritative CCS state. Gateways MUST NOT synthesize typed metadata such as `index`, `completeness`, or `effective_max_result_window`. ### 3.3 Stable Identity Allocation Stable identity allocation is a CCS responsibility, not a TiDB incidental side effect. Caller ownership decides who is allowed to request first allocation for a given integration path, but CCS remains the sole allocator of the resulting stable IDs. Allocation rules: - first-create `PutNamespaceMetadata` allocates `namespace_id` on the namespace-name lane before descriptor-object publication. - first-create `PutCollectionMetadata` allocates `collection_id` on the collection-name lane before descriptor-object publication. - CCS allocates these IDs from the shared TSO domain: - one freshly reserved TSO value becomes the opaque stable ID, - gaps are allowed, - replay of the same admitted create command MUST reuse the original allocated ID rather than allocating a second one. Why this model: - the ID exists before the durable descriptor-object write, so S3 control-object keys and success payloads can use it immediately. - the ID is frozen into durable descriptor and replay objects, so TiDB rebuild does not redefine identity. Relational projection tables therefore materialize already allocated IDs. They are not the source of truth for ID generation. ### 3.4 Name Allocation Epochs Stable IDs alone are insufficient for ABA-safe name reuse once lifecycle/tombstone GC eventually releases a reserved name. Allocation-epoch rules: - namespace names carry a private monotonic `namespace_name_allocation_epoch`. - collection names carry a private monotonic `collection_name_allocation_epoch` scoped to one `namespace_id`. - first admitted create for a never-before-materialized reservation starts at epoch `1`. - ordinary update/ensure of the same active lineage reuses the current epoch. - tombstoning retains the reservation and its current epoch until lifecycle/tombstone GC releases that name. - same-name recreation after reservation release MUST allocate a fresh epoch greater than the previous one. - resolve phase chooses the current reservation epoch (existing active/tombstoned lineage) or allocates a fresh one for a newly admitted create before writing durable control state. - CCS persists the most recent allocated epoch in a tiny durable allocator state even after reservation release, so later same-name recreation never has to scan `catalog-log` just to choose the next epoch. These epochs are private control metadata. They do not appear on public descriptor payloads. In this revision they protect ABA-safe lineage allocation and auditability, but metadata mutations no longer derive a separate business replay identity from them. ## 4. Mutation and Replay Model ### 4.1 Command Families CCS handles two replay families: 1. Public RPC mutations: - `PutNamespaceMetadata` - `PutCollectionMetadata` - `PutCollectionSchema` - `DropCollection` These commands carry canonical `RequestMeta`, including replay-bearing `idempotency_key`, optional `correlation_id`, and `client_kind`, because that is part of the public contract. 2. Internal control intents: - `head_advance_intent` from `Manifest Publisher` - `namespace_lifecycle_transition_intent` from private CCS/admin lifecycle workflows - `namespace_name_reservation_release_intent` from private lifecycle GC once a tombstoned namespace name becomes releasable - `collection_name_reservation_release_intent` from private tombstone GC once a tombstoned collection name becomes releasable again These intents do not carry `RequestMeta.idempotency_key`. They use deterministic internal intent identity derived from their intent tuple. Public RPC replay and internal intent replay MUST NOT be conflated into one undocumented "idempotency key for everything" model. Exception for the private quota-allocation/refill surface: - quota lease/refill requests are high-churn internal coordination traffic, not public catalog mutations. - they therefore do not use the `catalog-log` / `catalog-replay-*` command family defined for public metadata/schema/lifecycle mutations. - instead, they use namespace-owned quota state, deterministic `lease_request_id` idempotency inside that state, and `quota_epoch` invalidation semantics defined in Section `5.10`. ### 4.2 Command Identity Every CCS mutation or internal intent resolves one deterministic `command_identity`. Public RPC command identity: - derived from `(command_kind, authenticated_client_identity, client_id, scope, idempotency_key)` - this is the transport/network replay identity - caller-supplied `principal_subject` remains audit/authz context only and MUST NOT participate in replay-key derivation - caller-supplied `correlation_id` remains audit/tracing metadata only and MUST NOT participate in replay-key derivation Business replay identity by command class: - `PutCollectionSchema`: `(namespace_id, collection_id, schema_version, fence.schema_generation_token)` - `DropCollection`: `(namespace_id, collection_id, drop fence tuple when present)` or `(namespace_id, collection_id, "__unconditional_drop__")` when the request omits `fence` - `head_advance_intent`: `(namespace_id, collection_id, expected_head_version, new_head_version, manifest_checksum_or_etag)` Deterministic internal intent identity by command class: - `head_advance_intent`: `(namespace_id, collection_id, expected_head_version, new_head_version, manifest_checksum_or_etag)` - `namespace_lifecycle_transition_intent`: `(namespace_id, target_lifecycle_state, expected_namespace_revision)` - `namespace_name_reservation_release_intent`: `(namespace_name, releasing_namespace_name_allocation_epoch, release_cutoff_unix_ms)` - `collection_name_reservation_release_intent`: `(namespace_id, collection_name, releasing_collection_name_allocation_epoch, release_cutoff_unix_ms)` The public contract's replay semantics are implemented by combining: - transport replay identity for duplicate RPC delivery on every mutating command - business replay identity for semantic duplicate/conflict detection only on commands that define one (`PutCollectionSchema`, `DropCollection`, `head_advance_intent`) - deterministic internal intent identity for replay-safe de-duplication of private namespace/collection lifecycle workflows that do not define a separate business replay family ### 4.3 Durable Replay State Replay state itself is part of the durable control-plane truth. It MUST therefore survive relational projection loss. CCS persists replay truth in four durable object families: ```text s3://bucket/catalog-log/{hash(command_identity)[0:2]}/{command_identity}.json s3://bucket/catalog-replay-transport/{hash(transport_lookup_key)[0:2]}/{transport_lookup_key}.json s3://bucket/catalog-replay-business/{command_kind}/{hash(business_lookup_key)[0:2]}/{business_lookup_key}.json s3://bucket/catalog-recovery-segments/segment_{recovery_segment_epoch}.json.zst ``` Object roles: - `catalog-log/.../{command_identity}.json` is the immutable command/outcome record and remains the durable replay truth. - `catalog-replay-transport/.../{transport_lookup_key}.json` is the durable pointer object used to answer transport replay lookups without scanning; it is a deterministic derivative of the command/outcome record rather than an independent source of truth. - `catalog-replay-business/.../{business_lookup_key}.json` is emitted only for commands that define a business replay identity; when present, it is the durable pointer object used to answer business replay lookups without scanning and is a deterministic derivative of the command/outcome record rather than an independent source of truth. - `catalog-recovery-segments/segment_{recovery_segment_epoch}.json.zst` is the durable ordered recovery index for disaster recovery; each segment contains an append-ordered list of committed command references plus its covered applied-ts range, and lets CCS discover the exact post-checkpoint replay set without scanning hash-sharded `catalog-log` prefixes. Lookup objects carry: - the lookup key they represent, - referenced `command_identity`, - `command_kind`, - minimal scope metadata needed for validation/debugging, - creation/apply timestamps. Each command/outcome object MUST persist at least: - `command_identity` - `transport_lookup_key` - `business_lookup_key` when the command class defines a business replay family - `command_kind` - `replay_family` (`PUBLIC_RPC` or `INTERNAL_INTENT`) - scope keys (`namespace_name`, `namespace_id`, `collection_name`, `collection_id`) - name-allocation epoch fields when applicable: - `namespace_name_allocation_epoch` - `collection_name_allocation_epoch` - public-RPC metadata when applicable: - `idempotency_key` - `correlation_id` - `authenticated_client_identity_hash` - `principal_subject` - `client_id` - `client_kind` - canonical payload fingerprint - business replay key fields: - `schema_version` when applicable - fence tuple when applicable - head-advance CAS tuple when applicable, including `manifest_checksum_or_etag` - deterministic internal-intent key fields when applicable: - `target_lifecycle_state` - `expected_namespace_revision` - `releasing_namespace_name_allocation_epoch` - `releasing_collection_name_allocation_epoch` - `release_cutoff_unix_ms` - referenced durable control-object keys written by the command - terminal outcome payload: - original canonical success payload for replay-stable success branches - or canonical error classification for durable failure branches - terminal lifecycle state (`APPLIED` or `FAILED`) - replay classification (`ORIGINAL`, `REPLAY_SAME_PAYLOAD`, `REPLAY_CONFLICT`) - creation/apply timestamps The S3 command/outcome object is the durable replay truth and commit-authoritative source for recovery. Replay-lookup objects and any TiDB replay index are rebuildable derivative indexes only. ### 4.4 Three-Phase Apply Pattern All command classes, including `DropCollection`, follow one logical pattern: 1. Resolve phase: - validate caller authority and command-family rules - derive transport replay identity and business replay identity when the command class defines one - for name-scoped metadata mutations, resolve/create on the name lane first and then pivot the already allocated lineage onto the stable-id owner lane before authoritative apply whenever the target identity already exists - check rebuildable replay index and local cache for fast-path duplicate detection - if fast-path cache/index is inconclusive, consult the deterministic S3 replay-lookup object first, then dereference the referenced command/outcome object before admitting a conflicting replay branch 2. Durable-object phase: - write or update the target S3 control object(s) - write the immutable S3 command/outcome object that captures the original admitted outcome - publish deterministic `catalog-replay-transport` objects for every mutating command and `catalog-replay-business` objects only for command classes that define business replay identity - internal-intent durable acceptance is established at the end of this phase, but public control-plane RPC success ACK MUST NOT be returned yet; public mutation ACK waits for the materialize phase so subsequent public catalog reads observe the authoritative post-mutation state - any control object not referenced by an `APPLIED` command/outcome record is orphan/uncommitted state and MUST be ignored or quarantined during recovery 3. Materialize phase: - update relational projection state used for online serving - update relational replay-index projection - return success ACK for public control-plane RPCs only after the authoritative online serving projection needed by the public read surface has been updated If CCS crashes after target control-object write but before the `APPLIED` command/outcome object exists, the target control object is orphan/uncommitted and MUST NOT be materialized as committed state during recovery. If CCS crashes after the command/outcome object exists but before replay-lookup publication or relational projection materialization finishes, recovery MUST rebuild the missing replay-lookup objects and finish the missing public-read projection work from the command/outcome record before replay-sensitive traffic or public success acknowledgment resumes. Relational-backend durability is not required to preserve replay correctness, but completion of the authoritative online projection is required before a public mutation is considered successfully acknowledged. ### 4.5 Public Replay Rules `PutNamespaceMetadata` and `PutCollectionMetadata`: - same transport replay identity + same canonical payload MUST return the original success payload verbatim, not merely the same status enum - same transport replay identity + different payload MUST fail closed as `SF_IDEMPOTENCY_CONFLICT` - metadata mutations do not define a separate business replay identity in this revision; a different `idempotency_key` is an ordinary update/ensure command against current state, not business replay - when the target lineage already exists, current-state means the authoritative stable-id-owned namespace/collection lineage resolved during the name-lane phase; a conflicting lifecycle or descriptor transition discovered during the stable-id-owned apply phase rejects the update as `SF_VERSION_CONFLICT` with the canonical lifecycle-conflict detail keys - post-resolution namespace/collection metadata-update races reuse the canonical lifecycle-conflict detail family rather than inventing an implementation-local diagnostic shape: - resolved collection-lineage race -> `reservation_scope=resolved_collection_lineage` - resolved namespace-lineage race -> `reservation_scope=resolved_namespace_lineage` - metadata-mutation success / lifecycle-conflict invariants for this component note are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-CATALOG-002` - create branches that allocate a new namespace or collection MUST carry explicit `encryption` - update branches MAY omit `encryption` only to preserve the existing descriptor value unchanged `PutCollectionSchema`: - transport replay identity handles duplicate RPC delivery - business replay identity handles schema-semantics replay - same transport replay identity + same canonical schema payload MUST return the original success payload verbatim - same transport replay identity + different payload MUST fail closed as `SF_IDEMPOTENCY_CONFLICT` - same business replay identity under a different `idempotency_key` + same canonical schema payload MUST return replay-safe success according to the originally admitted schema outcome - same business replay identity under a different `idempotency_key` + different payload MUST fail closed as `SF_IDEMPOTENCY_CONFLICT` or `SF_VERSION_CONFLICT` by canonical precedence `DropCollection`: - same transport replay identity after successful drop returns transport `OK` `DropCollectionSuccessResponse.status=already_dropped` - durable tombstone state, not TiDB row survival alone, is the authoritative basis for the `already_dropped` success branch - same business replay identity under a different `idempotency_key` after a successful drop also returns transport `OK` `DropCollectionSuccessResponse.status=already_dropped` - a fresh request against a still-tombstoned collection returns transport `OK` `DropCollectionSuccessResponse.status=already_dropped` - a request against a never-existent or fully GC-released collection fails closed as `SF_RESOURCE_NOT_FOUND` `namespace_lifecycle_transition_intent`: - private namespace lifecycle workflows are durably replayable commands rather than in-memory admin toggles - a transition to `QUIESCING` or `TOMBSTONED` records the target lifecycle state and the referenced namespace descriptor revision in the immutable command/outcome log - transition to `TOMBSTONED` is admitted only after every child collection in that namespace is already in a terminal hidden state, so later public collection reads do not need a second parent-lifecycle filter to hide descendants `namespace_name_reservation_release_intent`: - lifecycle GC releases a tombstoned namespace name only after the recorded reservation horizon has passed and the tombstoned namespace has no remaining child collections or unresolved cleanup dependencies - the immutable command/outcome record is the durable truth for the reservation-release event, while the canonical S3 allocator-state object keeps the last allocated epoch for future anti-ABA recreation; relational allocator tables remain online projections of that durable allocator state `collection_name_reservation_release_intent`: - tombstone GC releases a tombstoned collection name only after the recorded reservation horizon has passed and all referenced cleanup preconditions have succeeded - the immutable command/outcome record is the durable truth for the reservation-release event, while the canonical S3 allocator-state object keeps the last allocated epoch for future anti-ABA recreation; relational allocator tables remain online projections of that durable allocator state Internal `head_advance_intent`: - deterministic internal transport replay identity is identical to the business replay tuple in this revision: `(namespace_id, collection_id, expected_head_version, new_head_version, manifest_checksum_or_etag)` - duplicate head-advance intents are harmless when the CAS tuple already matches durable state ## 5. Relational Projection State ### 5.1 Design Rule The online relational projection state MUST be sufficient to serve the current public contract without gateway synthesis and without mandatory S3 reads on steady-state paths. Fields may be: - materialized directly, or - deterministically derived from materialized records For every public field, CCS MUST document which of those two categories applies. The SQL tables in this section are online projections. They are rebuildable from S3 control objects plus S3 command/outcome objects. Backend contract: - correctness does not depend on TiDB-specific SQL features; it depends on a transactional relational store that can provide: - atomic single-row and small multi-row mutations on one ownership lane, - unique constraints for name bindings and replay lookup keys, - deterministic binary ordering/equality for canonical NFC UTF-8 names, - JSON or equivalent opaque structured-document columns for typed descriptor/state payloads, - keyset pagination over stable ordered keys, - idempotent insert/update patterns for rebuildable projection refresh. - the durable source of truth remains S3 control objects plus S3 command/outcome objects; the relational backend is an online serving projection and a low-latency lookup index, not the durability anchor. - the shared TSO domain used for stable ID allocation and TiDB-facing `ddl_commit_ts` / `applied_commit_ts` semantics is an integration dependency of CCS and surrounding Search Fabric flows, not a requirement that the relational projection backend itself be TiDB. Backend profiles: - ordinary MySQL is functionally viable for the current design because this section does not rely on TiDB-only correctness features; the current DDL already sits close to a MySQL-compatible profile. - TiDB remains a strong default profile because it shares the wider TiDB integration environment, already speaks the required MySQL-family DDL, and can serve as the online projection store without changing the control-plane semantics. - PostgreSQL is also viable, but it requires a separate SQL profile for types, collations, and write patterns; those adjustments are documented below and do not change the canonical API or S3 durability model. Fixed-width string columns in these projections are normative realizations of the canonical API contract rather than local truncation policies: - catalog names and catalog-name prefixes are capped at 128 Unicode scalar values - `RequestMeta.idempotency_key` and `RequestMeta.correlation_id` are capped at 128 Unicode scalar values - `RequestMeta.client_id` and `RequestMeta.principal_subject` are capped at 256 Unicode scalar values - `SchemaFencePrecondition.schema_generation_token` is capped at 64 Unicode scalar values - CCS rejects overlength input before projection materialization; it never truncates or hashes identity-bearing strings to fit these columns SQL-profile notes for the table sketches in this section: - the inline DDL below is a MySQL-family profile intended to be directly realizable on TiDB and ordinary MySQL with minimal translation. - TiDB-only table comments such as `/*T![clustered_index] CLUSTERED */` are physical-layout hints, not semantic requirements. - PostgreSQL implementations should map: - `BIGINT UNSIGNED` -> `BIGINT` plus non-negative `CHECK` constraints or a dedicated positive-integer domain, - inline `ENUM(...)` columns -> either PostgreSQL enum types or `TEXT` plus `CHECK` constraints, - `JSON` -> `JSONB`, - `utf8mb4_bin` binary collation requirements -> a deterministic bytewise text ordering profile such as `COLLATE "C"` on canonical NFC text keys. - PostgreSQL write paths should use native `INSERT ... ON CONFLICT DO UPDATE` patterns where MySQL/TiDB profiles use insert-or-update logic, and should prefer a partial unique index for nullable `business_lookup_key` (`WHERE business_lookup_key IS NOT NULL`) rather than depending on backend-specific null-handling details. - no backend in this section is assumed to provide FULLTEXT, vector, or TiDB coprocessor features for CCS correctness; CCS stores metadata/control projections only. ### 5.2 Namespace Table ```sql CREATE TABLE namespaces ( namespace_id BIGINT UNSIGNED NOT NULL, namespace_name VARCHAR(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, namespace_name_allocation_epoch BIGINT UNSIGNED NOT NULL, namespace_revision BIGINT UNSIGNED NOT NULL, lifecycle_state ENUM('ACTIVE', 'QUIESCING', 'TOMBSTONED') NOT NULL, lifecycle_updated_at_unix_ms BIGINT NOT NULL, lifecycle_gc_after_unix_ms BIGINT, collection_name_binding_revision BIGINT UNSIGNED NOT NULL, options_json JSON, encryption_json JSON NOT NULL, approx_stats_json JSON, index_json JSON, internal_policy_state_json JSON NOT NULL, created_at_unix_ms BIGINT NOT NULL, updated_at_unix_ms BIGINT NOT NULL, PRIMARY KEY (namespace_id) /*T![clustered_index] CLUSTERED */, UNIQUE KEY uk_namespace_name (namespace_name) ); ``` Materialized public payload sources: - `NamespaceDescriptor.namespace_id` <- `namespace_id` - `namespace_name` <- `namespace_name` - timestamps <- `created_at_unix_ms`, `updated_at_unix_ms` - `lifecycle` <- readable projection of `lifecycle_state` (`ACTIVE -> active`, `QUIESCING -> quiescing`; `TOMBSTONED` has no successful public descriptor projection because the namespace is omitted/not-found) - `options`, `encryption`, `approx_stats`, `index` <- typed JSON columns above - `completeness` <- derived at read time from authoritative lagging descriptor fields (`approx_stats_json`, `index_json`) plus the canonical required-field set Typed-field publication and completeness derivation rule: - CCS does not persist an independent namespace public-completeness state in this design; successful public readback derives `DescriptorCompleteness` from the authoritative typed namespace descriptor fields that back that same row. - Let the canonical required-field set be `{approx_stats, index}`. - The derived missing-field set is the unique byte-ascending subset of those canonical names whose authoritative typed source field is currently absent from the online read projection. - When the derived missing-field set is empty, public readback MUST emit `completeness.ready`. - When the derived missing-field set is non-empty, public readback MUST emit `completeness.publishing` and return that exact ordered missing-field set. - `encryption_json` is synchronous authoritative descriptor state and successful public namespace descriptor reads MUST project it; implementations MUST NOT map missing authoritative encryption to `completeness.publishing`. - `approx_stats_json` MAY be null only while namespace descriptor publication is still incomplete. - when `approx_stats_json` is null, the derived `completeness` state MUST be `PUBLISHING` and the derived missing-field set MUST include `approx_stats`. - `index_json` MAY be null only while namespace descriptor publication is still incomplete. - when `index_json` is null, the derived `completeness` state MUST be `PUBLISHING` and the derived missing-field set MUST include `index`. - derived `completeness.ready` forbids null `approx_stats_json` and null `index_json`. Private-only state: - `namespace_name_allocation_epoch` - `lifecycle_updated_at_unix_ms` - `lifecycle_gc_after_unix_ms` - `collection_name_binding_revision` - quota class, audit config, and policy profile under `internal_policy_state_json` Namespace lifecycle model: - `ACTIVE` - `PutNamespaceMetadata` create/update/ensure is admitted normally. - `ListNamespaces` includes the namespace. - `GetNamespaceMetadata` returns the authoritative `NamespaceDescriptor`. - successful public namespace descriptors carry the `active` lifecycle branch. - child collection create/update admission proceeds subject to collection-level checks. - `QUIESCING` - `PutNamespaceMetadata` may still update existing namespace metadata. - `ListNamespaces` continues to include the namespace for discoverability and stable namespace-name continuity. - `GetNamespaceMetadata` continues to return the authoritative `NamespaceDescriptor`. - successful public namespace descriptors carry the `quiescing` lifecycle branch. - `PutCollectionMetadata` that would allocate a new collection in the namespace is rejected fail-closed as `SF_VERSION_CONFLICT` until the namespace returns to `ACTIVE`; the response MUST carry the canonical lifecycle-conflict detail keys. - `TOMBSTONED` - `PutNamespaceMetadata` by the same `namespace_name` is rejected as `SF_VERSION_CONFLICT` to prevent namespace-name ABA until `lifecycle_gc_after_unix_ms` passes and private lifecycle GC releases the reservation; the response MUST carry the canonical lifecycle-conflict detail keys. - `ListNamespaces` excludes the namespace. - `GetNamespaceMetadata` fails closed as `SF_RESOURCE_NOT_FOUND`. - `NamespaceDescriptor.lifecycle` never exposes this state because tombstoned namespaces have no successful public descriptor readback. - child collection admission fails closed as `SF_VERSION_CONFLICT`; the response MUST carry the canonical lifecycle-conflict detail keys. Lifecycle transitions into `QUIESCING` and `TOMBSTONED` are private CCS/admin workflows outside the current public mutation surface and are executed through replayable internal intents recorded in `catalog-log`. Successful public `ListNamespaces` / `GetNamespaceMetadata` readback projects readable-state lifecycle explicitly: `NamespaceDescriptor.lifecycle` mirrors authoritative `lifecycle_state` through exactly one branch (`active` or `quiescing`) and never exposes `TOMBSTONED` because tombstoned namespaces are omitted/not-found. A transition to `TOMBSTONED` is admitted only after every child collection is already in a terminal hidden state, so the namespace tombstone cannot leave publicly visible descendants behind. When the namespace has zero active child collections, namespace create seeds `approx_stats_json` to the zero-child aggregate and `index_json` to `up_to_date`, so the derived public completeness state is `READY`. Every later transition that leaves the namespace with zero active child collections MUST synchronously recompute/publish the same zero-child aggregate and `index_json=up_to_date` before ACK or before serving a later successful namespace read. `collection_name_binding_revision` increments on every active collection-name binding add/remove/reservation-release event under that namespace. When lifecycle GC eventually releases a tombstoned namespace name, it executes `namespace_name_reservation_release_intent`, may remove the namespace row, and retains the allocator-state epoch in `namespace_name_epochs`. Private admission-state contract for collection-name resolution (machine-readable owner: `api/proto/searchfabric/v1/catalog_control_private.proto`): - positive `CollectionNameBinding{namespace_id, collection_name}` hits return the resolved `CollectionRef` plus the current `collection_name_binding_revision` - negative misses return the same current `collection_name_binding_revision` - gateway positive and negative caches are valid only while the bound revision remains unchanged Private admission-state contract for collection-scoped internal reads (machine-readable owner: `api/proto/searchfabric/v1/catalog_control_private.proto`): - CCS exposes an internal collection admission-state snapshot keyed by `CollectionRef` for gateway/query-serving consumers; this surface is private and is not projected into the public protobuf API. - successful reads MUST return at least: - lifecycle admission state collapsed to the deterministic public-facing branches used by internal consumers (`ACTIVE` or `TOMBSTONED` in this revision), - `descriptor_revision` as the canonical collection-descriptor cache validator, - an equivalent opaque descriptor etag/cache token that MUST change whenever `descriptor_revision` changes, - the current namespace-scoped `collection_name_binding_revision`, - the authoritative `collection_truth_mode`, - the authoritative `DocIDMapping.generation_policy`, - the active schema lineage pointer (`active_schema_version`, `active_schema_generation_token`) when one exists, - the query-serving readiness facts needed to resolve the latest admitted `READY` generation (`ready_generation_version`, `ready_generation_token`, or deterministic bootstrap/convergence state when no `READY` generation exists yet), - query-admission fragments derived from the active descriptor/schema such as the resolved effective query-window bound, which is admission-critical and therefore must not rely on gateway-local defaulting; this field MUST be `> 0` when a `READY` serving generation exists and MUST be explicit sentinel `0` during bootstrap/convergence when no `READY` serving generation is query-admissible, - allocator-admission fragments such as `CollectionSettings.allocator_admission_policy` whenever `DocIDMapping.generation_policy=DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`, - write-admission fragments such as the exact payload-carried event-time contract required on the hot path for lifecycle-enabled collections: `time_partition_field` identity, admitted type, and the rule that online `upsert` / `patch` must carry a non-null value for that field before late/future-arrival checks can pass. - bootstrap/convergence branches MUST be explicit: - no active schema lineage -> `schema_bootstrap_state=no_active_schema_generation`, - active lineage exists but no `READY` serving generation -> `schema_convergence_state=no_ready_serving_generation`. - cache-validation rules: - internal consumers MAY cache positive admission-state snapshots only while both the descriptor validator (`descriptor_revision` or its equivalent opaque etag) and the bound `collection_name_binding_revision` remain unchanged, - a change to either validator invalidates the cached snapshot before reuse, - implementations MUST NOT require hidden gateway-local state beyond this admission-state snapshot plus the canonical public query/control requests. ### 5.3 Collection Table ```sql CREATE TABLE collections ( namespace_id BIGINT UNSIGNED NOT NULL, collection_id BIGINT UNSIGNED NOT NULL, collection_name VARCHAR(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, collection_name_allocation_epoch BIGINT UNSIGNED NOT NULL, descriptor_revision BIGINT UNSIGNED NOT NULL, active_schema_version BIGINT UNSIGNED, active_schema_generation_token VARCHAR(64), collection_truth_mode INT, lifecycle_state ENUM('ACTIVE', 'TOMBSTONED') NOT NULL, drop_revision BIGINT UNSIGNED, dropped_at_unix_ms BIGINT, gc_after_unix_ms BIGINT, committed_manifest_version BIGINT UNSIGNED NOT NULL DEFAULT 0, committed_manifest_checksum_or_etag VARCHAR(255), options_json JSON, encryption_json JSON NOT NULL, approx_stats_json JSON, index_json JSON, manifest_head_version BIGINT UNSIGNED NOT NULL DEFAULT 0, expected_previous_head BIGINT NOT NULL DEFAULT -1, manifest_head_last_update_epoch BIGINT UNSIGNED NOT NULL DEFAULT 0, manifest_head_reconciler_status ENUM('IDLE', 'REPAIR_PENDING', 'REPAIRING', 'ERROR') NOT NULL DEFAULT 'IDLE', internal_policy_state_json JSON NOT NULL, created_at_unix_ms BIGINT NOT NULL, updated_at_unix_ms BIGINT NOT NULL, PRIMARY KEY (namespace_id, collection_id) /*T![clustered_index] CLUSTERED */, UNIQUE KEY uk_namespace_collection_name (namespace_id, collection_name) ); ``` Materialized public payload sources for active collections: - `CollectionDescriptor.collection` <- `(namespace_id, collection_id)` - `collection_name` <- `collection_name` - timestamps <- `created_at_unix_ms`, `updated_at_unix_ms` - `options`, `encryption`, `approx_stats`, `index` <- typed JSON columns above - `completeness` <- derived at read time from authoritative lagging descriptor fields (`approx_stats_json`, `index_json`) plus the canonical required-field set Typed-field publication and completeness derivation rule: - CCS does not persist an independent collection public-completeness state in this design; successful public readback derives `DescriptorCompleteness` from the authoritative typed collection descriptor fields that back that same row. - Let the canonical required-field set be `{approx_stats, index}`. - The derived missing-field set is the unique byte-ascending subset of those canonical names whose authoritative typed source field is currently absent from the online read projection. - When the derived missing-field set is empty, public readback MUST emit `completeness.ready`. - When the derived missing-field set is non-empty, public readback MUST emit `completeness.publishing` and return that exact ordered missing-field set. - `encryption_json` is synchronous authoritative descriptor state and successful public collection descriptor reads MUST project it; implementations MUST NOT map missing authoritative encryption to `completeness.publishing`. - `approx_stats_json` MAY be null only while descriptor publication is still incomplete. - when `approx_stats_json` is null, the derived `completeness` state MUST be `PUBLISHING` and the derived missing-field set MUST include `approx_stats`. - `index_json` MAY be null only while descriptor publication is still incomplete. - when `index_json` is null, the derived `completeness` state MUST be `PUBLISHING` and the derived missing-field set MUST include `index`. - derived `completeness.ready` forbids null `approx_stats_json` and null `index_json`. Private-only state: - `collection_name_allocation_epoch` - `active_schema_version` - `active_schema_generation_token` - `collection_truth_mode` - `lifecycle_state` - `drop_revision` - `dropped_at_unix_ms` - `gc_after_unix_ms` - `committed_manifest_version` - `committed_manifest_checksum_or_etag` - `manifest_head_version` - `expected_previous_head` - `manifest_head_last_update_epoch` - `manifest_head_reconciler_status` - internal collection policy bindings (`internal_policy_state_json` on the collection row does not own namespace quota authority) Serving rules: - `ListCollections` and `GetCollectionMetadata` expose only `lifecycle_state='ACTIVE'`. - `GetCollectionSchema` reads only through the active collection row; `lifecycle_state='TOMBSTONED'` therefore fails closed as `SF_RESOURCE_NOT_FOUND` rather than surfacing stale pre-drop schema state. - CCS read models for `GetCollectionSchema`, `GetSchemaSyncStatus`, and the downstream query-serving helpers that consume CCS schema/serving facts MUST fail closed against an `ACTIVE` collection with no admitted schema lineage yet as `SF_CATALOG_UNAVAILABLE` with canonical schema-bootstrap details. - query execution resolves against the latest `READY` serving generation rather than blindly following the active schema pointer: - if the collection is `ACTIVE` but still has no admitted schema lineage, query execution itself also fails closed as `SF_CATALOG_UNAVAILABLE` with canonical schema-bootstrap details; bootstrap is not a generic not-ready or not-found branch. - if the active generation is `READY`, query-serving-admissible generation equals the active generation. - if the active generation is `APPLYING`, `DRIFTED`, or `ERROR`, query serving continues to use the most recent `READY` generation when one exists. - if an active schema lineage exists but no `READY` generation exists yet, query execution and the downstream query-serving helper owners that consume CCS serving facts (`Query Gateway` / `Query Broker` for `GetReadiness`, and `Query Gateway` fronting CCS-owned optimizer statistics for `GetCollectionStatistics`) MUST fail closed as `SF_CATALOG_UNAVAILABLE` with canonical details `catalog_unavailable_reason=schema_convergence` and `schema_convergence_state=no_ready_serving_generation`. - `lifecycle_state='TOMBSTONED'` rows remain materialized until tombstone GC completes so CCS can reject ABA-style recreation safely. ### 5.4 Name Allocation Epoch State Canonical allocator durability lives in S3 allocator-state objects: ```text s3://bucket/catalog-allocator/namespace-name/{hash(namespace_name)[0:2]}/{namespace_name}.json s3://bucket/catalog-allocator/collection-name/{hash(namespace_id)[0:2]}/{namespace_id}/{hash(collection_name)[0:2]}/{collection_name}.json ``` Allocator object payload: - canonical name scope (`namespace_name` or `(namespace_id, collection_name)`) - `last_allocated_epoch` - `updated_at_unix_ms` These S3 allocator objects are the durable anti-ABA source of truth. The relational allocator tables below are authoritative online projections of that durable state and MUST be rebuilt from it during recovery. The private name-allocation epochs remain durable even after a name reservation is released. ```sql CREATE TABLE namespace_name_epochs ( namespace_name VARCHAR(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, last_allocated_epoch BIGINT UNSIGNED NOT NULL, PRIMARY KEY (namespace_name) /*T![clustered_index] CLUSTERED */ ); CREATE TABLE collection_name_epochs ( namespace_id BIGINT UNSIGNED NOT NULL, collection_name VARCHAR(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, last_allocated_epoch BIGINT UNSIGNED NOT NULL, PRIMARY KEY (namespace_id, collection_name) /*T![clustered_index] CLUSTERED */ ); ``` Rules: - first admitted create initializes `last_allocated_epoch=1`. - same-name recreation after reservation release uses `last_allocated_epoch + 1`. - lifecycle/tombstone GC MUST NOT delete these allocator rows merely because the active reservation was released. - these rows are private anti-ABA allocator-state projections only; public catalog reads never surface them. ### 5.5 Schema Generation Table ```sql CREATE TABLE schema_generations ( namespace_id BIGINT UNSIGNED NOT NULL, collection_id BIGINT UNSIGNED NOT NULL, schema_version BIGINT UNSIGNED NOT NULL, schema_generation_token VARCHAR(64) NOT NULL, collection_truth_mode INT NOT NULL, schema_definition_json JSON NOT NULL, ddl_commit_ts BIGINT UNSIGNED NOT NULL DEFAULT 0, applied_commit_ts BIGINT UNSIGNED NOT NULL, active_schema_update_policy INT NOT NULL, effective_max_result_window INT UNSIGNED NOT NULL, sync_state ENUM('APPLYING', 'READY', 'DRIFTED', 'ERROR') NOT NULL, drift_reason VARCHAR(255), error_reason VARCHAR(255), created_at_unix_ms BIGINT NOT NULL, updated_at_unix_ms BIGINT NOT NULL, PRIMARY KEY (namespace_id, collection_id, schema_version) ); ``` Materialized public payload sources: - `CollectionSchemaDescriptor.definition` <- `collection_truth_mode`, `schema_definition_json` - `applied_schema_version` <- `schema_version` - `applied_schema_generation_token` <- `schema_generation_token` - `applied_commit_ts` <- `applied_commit_ts` - `active_schema_update_policy` <- `active_schema_update_policy` - `effective_max_result_window` <- `effective_max_result_window` - `GetSchemaSyncStatus.state` <- `sync_state` + `drift_reason` + `error_reason` Pointer rules: - `collections.active_schema_version` + `collections.active_schema_generation_token` identify the authoritative active generation for: - `GetCollectionSchema` - ordinary ownership validation - write admission that depends on the current active schema contract - `collections.collection_truth_mode` is the materialized authoritative truth mode for the active generation. - `GetCollectionSchema` dereferences the active pointer and returns that row. - `GetSchemaSyncStatus` reads the same active generation row and returns its convergence state. - query-serving components must resolve execution against serving readiness rather than against the active pointer alone; the active pointer is authoritative for control-plane readback/admission, but the latest `READY` generation is authoritative for query execution. ### 5.6 Replay Index Projection Table ```sql CREATE TABLE catalog_replay_index ( command_identity CHAR(64) NOT NULL, transport_lookup_key CHAR(64) NOT NULL, business_lookup_key CHAR(64), command_kind ENUM( 'PUT_NAMESPACE_METADATA', 'PUT_COLLECTION_METADATA', 'PUT_COLLECTION_SCHEMA', 'DROP_COLLECTION', 'HEAD_ADVANCE', 'NAMESPACE_LIFECYCLE_TRANSITION', 'NAMESPACE_NAME_RESERVATION_RELEASE', 'COLLECTION_NAME_RESERVATION_RELEASE' ) NOT NULL, replay_family ENUM('PUBLIC_RPC', 'INTERNAL_INTENT') NOT NULL, idempotency_key VARCHAR(128), correlation_id VARCHAR(128), authenticated_client_identity_hash CHAR(64), principal_subject VARCHAR(256), client_id VARCHAR(256), namespace_name VARCHAR(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin, namespace_name_allocation_epoch BIGINT UNSIGNED, namespace_id BIGINT UNSIGNED, collection_name VARCHAR(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin, collection_name_allocation_epoch BIGINT UNSIGNED, collection_id BIGINT UNSIGNED, schema_version BIGINT UNSIGNED, fence_expected_prev_schema_version BIGINT UNSIGNED, fence_schema_generation_token VARCHAR(64), fence_enforce_monotonic_apply BOOLEAN, expected_head_version BIGINT, new_head_version BIGINT, manifest_checksum_or_etag VARCHAR(255), target_lifecycle_state ENUM('QUIESCING', 'TOMBSTONED'), expected_namespace_revision BIGINT UNSIGNED, release_cutoff_unix_ms BIGINT, payload_fingerprint CHAR(64) NOT NULL, terminal_state ENUM('APPLIED', 'FAILED') NOT NULL, replay_classification ENUM('ORIGINAL', 'REPLAY_SAME_PAYLOAD', 'REPLAY_CONFLICT') NOT NULL, original_outcome_json JSON NOT NULL, command_object_key VARCHAR(512) NOT NULL, created_ts BIGINT NOT NULL, applied_ts BIGINT, PRIMARY KEY (command_identity), UNIQUE KEY uk_transport_lookup (transport_lookup_key), UNIQUE KEY uk_business_lookup (command_kind, business_lookup_key) ); ``` This table is not an independent durable ledger. It is a rebuildable lookup/index projection of S3 command/outcome objects for low-latency replay checks and reconciliation scans. Lookup-key rules: - `transport_lookup_key` is the canonical hash of the transport replay identity: - public RPCs hash `(authenticated_client_identity, client_id, rpc_method, scope, idempotency_key)`, - internal intents hash the deterministic internal intent identity. - `business_lookup_key` is populated only for command classes with business replay identity: - `PutCollectionSchema` -> `(namespace_id, collection_id, schema_version, fence.schema_generation_token)` - `DropCollection` -> `(namespace_id, collection_id, drop fence tuple when present)` or `(namespace_id, collection_id, "__unconditional_drop__")` when `fence` is absent - `HEAD_ADVANCE` -> `(namespace_id, collection_id, expected_head_version, new_head_version, manifest_checksum_or_etag)` - `PutNamespaceMetadata`, `PutCollectionMetadata`, `NAMESPACE_LIFECYCLE_TRANSITION`, `NAMESPACE_NAME_RESERVATION_RELEASE`, and `COLLECTION_NAME_RESERVATION_RELEASE` leave `business_lookup_key` null because they do not define a separate business replay family in this revision. - internal lifecycle/release intents still project their deterministic internal key fields (`target_lifecycle_state`, `expected_namespace_revision`, release epoch/cutoff fields) so low-latency replay checks and rebuild scans can reconstruct the same intent identity without inventing a business-replay family. ### 5.7 Descriptor Enrichment Ownership and Freshness Public typed descriptor enrichment is CCS-owned descriptor readback state. It MUST NOT be synthesized by gateways or readers. `encryption`, `approx_stats`, and `index` are authoritative typed descriptor fields. `completeness` is a derived public projection over the lagging descriptor fields (`approx_stats`, `index`) rather than an independently materialized mutable truth. Producer and freshness rules: - `collection.encryption` - producer: `PutCollectionMetadata` - refresh: synchronous on admitted metadata mutation - allowed staleness: none on successful CCS reads - absence rule: successful descriptor reads MUST NOT omit `encryption`; implementations MUST NOT project missing authoritative collection encryption through `completeness.publishing` - `collection.approx_stats` - producer: CCS collection-stats projector fed by manifest-head advancement, compaction, and GC events - refresh: asynchronous after committed collection-state changes - allowed staleness: may lag the latest committed manifest head by one successful projector/reconciler cycle - absence rule: when not yet published, successful descriptor reads MUST expose `completeness.publishing` and `missing_fields[]` MUST include `approx_stats`; `READY` descriptors MUST NOT omit `approx_stats` - `collection.index` - producer: CCS collection-index projector fed by schema-convergence and indexing-backlog signals - refresh: asynchronous after backlog/convergence changes - allowed staleness: may lag the latest active-generation backlog by one successful projector/reconciler cycle - absence rule: when not yet published, successful descriptor reads MUST expose `completeness.publishing` and `missing_fields[]` MUST include `index`; `READY` descriptors MUST NOT omit `index` - `collection.completeness` - producer: no separate materializer; CCS derives it during descriptor readback from the authoritative lagging collection descriptor fields and the canonical required-field set - refresh: recomputed on every successful read from the current typed descriptor state whenever required typed descriptor fields change or enrichment projectors complete - allowed staleness: none on successful CCS reads - collection completeness derivation rule: - `READY` requires collection `approx_stats` and `index`, - `PUBLISHING` is mandatory whenever `approx_stats` or `index` is not yet materialized, - public `publishing.missing_fields[]` MUST enumerate every still-missing required typed collection field in canonical byte-ascending order. - `namespace.encryption` - producer: `PutNamespaceMetadata` - refresh: synchronous on admitted metadata mutation - allowed staleness: none on successful CCS reads - absence rule: successful descriptor reads MUST NOT omit `encryption`; implementations MUST NOT project missing authoritative namespace encryption through `completeness.publishing` - `namespace.approx_stats` - producer: `PutNamespaceMetadata` synchronously seeds the zero-child aggregate on namespace creation, CCS synchronously republishes that same zero-child aggregate on every later transition back to zero active child collections, and CCS namespace aggregator maintains the non-zero-child aggregate between those synchronous zero-child transitions - refresh: synchronous on namespace creation with zero active child collections and synchronous on every later transition to zero active child collections; otherwise asynchronous after child-collection stat changes while at least one active child collection remains - allowed staleness: none for any zero-active-child state; otherwise may lag child collection projections by one successful namespace aggregation cycle - absence rule: when not yet published, successful descriptor reads MUST expose `completeness.publishing` and `missing_fields[]` MUST include `approx_stats`; `READY` descriptors MUST NOT omit `approx_stats` - `namespace.index` - producer: `PutNamespaceMetadata` synchronously seeds `up_to_date` when the namespace has zero active child collections, and CCS synchronously re-publishes `up_to_date` on every later transition to zero active child collections; CCS namespace aggregator maintains the non-zero-child aggregate between those synchronous zero-child transitions - semantics: - namespaces with zero active child collections are `up_to_date` - `up_to_date` iff every active child collection is `up_to_date` - `updating.unindexed_bytes` is the sum of active child-collection backlog - refresh: synchronous on namespace creation with zero active child collections and synchronous on every later transition to zero active child collections; otherwise asynchronous after child-collection index-state changes while at least one active child collection remains - allowed staleness: none for any zero-active-child state; otherwise may lag child collection projections by one successful namespace aggregation cycle - absence rule: when not yet published, successful descriptor reads MUST expose `completeness.publishing` and `missing_fields[]` MUST include `index`; `READY` descriptors MUST NOT omit `index` - `namespace.completeness` - producer: no separate materializer; CCS derives it during descriptor readback from the authoritative lagging namespace descriptor fields and the canonical required-field set - refresh: recomputed on every successful read from the current typed namespace descriptor state whenever required typed descriptor fields change or namespace enrichment projectors complete - allowed staleness: none on successful CCS reads - namespace completeness derivation rule: - `READY` requires namespace `approx_stats` and `index`, - `PUBLISHING` is mandatory whenever `approx_stats` or `index` is not yet materialized, - public `publishing.missing_fields[]` MUST enumerate every still-missing required typed namespace field in canonical byte-ascending order. - For both collection and namespace descriptors, public `publishing.missing_fields[]` plus fail-closed `missing_descriptor_fields` diagnostics MUST be derived directly from the current authoritative lagging descriptor fields. The derived ordered set MUST use only the canonical required-field names `approx_stats` and `index`, MUST keep them unique, and MUST be emitted in byte-ascending order. - descriptor completeness / missing-fields invariants for this component note are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-CATALOG-001`. ### 5.8 Optimizer Statistics Snapshot Ownership `GetCollectionStatistics` is backed by a CCS-owned optimizer-statistics snapshot family, not by descriptor-level `approx_stats`. Durable object path: ```text s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/optimizer_statistics_head.json s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/optimizer_statistics_v{applied_schema_generation_token}_{stats_commit_ts}.json ``` Hot projection tables: ```sql CREATE TABLE collection_statistics_heads ( namespace_id BIGINT UNSIGNED NOT NULL, collection_id BIGINT UNSIGNED NOT NULL, latest_applied_schema_version BIGINT UNSIGNED NOT NULL, latest_applied_schema_generation_token VARCHAR(64) NOT NULL, latest_stats_commit_ts BIGINT UNSIGNED NOT NULL, latest_collected_at_unix_ms BIGINT NOT NULL, PRIMARY KEY (namespace_id, collection_id) /*T![clustered_index] CLUSTERED */ ); CREATE TABLE collection_statistics_snapshots ( namespace_id BIGINT UNSIGNED NOT NULL, collection_id BIGINT UNSIGNED NOT NULL, stats_commit_ts BIGINT UNSIGNED NOT NULL, applied_schema_version BIGINT UNSIGNED NOT NULL, applied_schema_generation_token VARCHAR(64) NOT NULL, collected_at_unix_ms BIGINT NOT NULL, snapshot_json JSON NOT NULL, PRIMARY KEY (namespace_id, collection_id, applied_schema_generation_token, stats_commit_ts) /*T![clustered_index] CLUSTERED */ ); ``` Ownership and derivation rules: - producer: CCS optimizer-statistics projector - source inputs: committed manifest state, compaction outputs, tombstone/GC state, and future explicit analyze triggers - immutable snapshot durable identity is the collection-scoped composite tuple (`applied_schema_generation_token`, `stats_commit_ts`); `stats_commit_ts` is the logical commit frontier covered by the snapshot, not the wall-clock publication time, and is not sufficient by itself to distinguish schema-only lineage promotions - `collected_at_unix_ms` is the freshness timestamp used to enforce `GetCollectionStatistics.max_staleness_ms` - `collection_statistics_heads` points at the newest admitted snapshot available for the current query-serving `latest READY` lineage through the authoritative tuple `(latest_applied_schema_generation_token, latest_stats_commit_ts)` - `optimizer_statistics_head.json` is the durable pointer to the newest admitted snapshot for that same query-serving lineage and mirrors the authoritative `(latest_applied_schema_version, latest_applied_schema_generation_token, latest_stats_commit_ts, latest_collected_at_unix_ms)` tuple used to rebuild `collection_statistics_heads` - every snapshot MUST also carry the schema lineage it was computed against: - `applied_schema_version` - `applied_schema_generation_token` - once any schema generation becomes query-serving-admissible, CCS MUST publish an initial zero-row optimizer-statistics snapshot as part of the same materialization wave that makes that lineage the current `latest READY` generation, so the RPC has a canonical base snapshot even before user data appears - later schema generations MUST each publish their own zero-row optimizer-statistics snapshot before CCS is allowed to promote that lineage to the current query-serving `latest READY` generation; that publish step MUST allocate a new composite durable identity even when no user-data commit frontier changed - optimizer-statistics lineage/freshness invariants for this component note are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-QUERY-003` - `snapshot_json` is the hot online projection used to serve `GetCollectionStatistics` without mandatory S3 reads; it must carry: - `applied_schema_version` - `applied_schema_generation_token` - `collected_at_unix_ms` - `collection_stats`, - `field_stats` in canonical `field_name` byte-ascending order, - the declared supported extended-stat tuple family for that snapshot, - and the tuple-keyed `extended_stats` entries for exactly that declared family in canonical tuple-identity order. - tuple identity is the exact ordered canonical field-path tuple. - request admission follows the canonical API contract: - CCS first resolves the current query-serving `latest READY` lineage, then selects one admitted statistics snapshot from that lineage that satisfies the freshness rule, then validates requested fields and tuples against that selected snapshot lineage - `target_fields=[]` is the collection-only fast path and therefore requires `extended_requests[]` to be empty, - non-empty `extended_requests[]` with `target_fields=[]` is rejected as `SF_BAD_REQUEST` rather than ignored or partially served. Freshness and recovery rules: - `max_staleness_ms=0` requires serving the latest admitted snapshot referenced by `collection_statistics_heads` for the current query-serving lineage - `max_staleness_ms>0` requires selecting the newest admitted snapshot for that same lineage whose `collected_at_unix_ms` satisfies `now_unix_ms - collected_at_unix_ms <= max_staleness_ms` - if no admitted snapshot satisfies that budget, CCS MUST fail the RPC closed as `SF_CATALOG_UNAVAILABLE` with canonical details `catalog_unavailable_reason=statistics_freshness`, `max_staleness_ms`, `latest_stats_commit_ts`, and `latest_stats_collected_at_unix_ms` - if the collection has no admitted schema lineage yet, the RPC fails with canonical schema-bootstrap details instead of fabricating an empty statistics snapshot - if the collection has an admitted lineage but no query-serving-admissible `READY` generation yet, the RPC fails with canonical schema-convergence details instead of serving a newer non-serving lineage - if a requested field path is not declared in the selected snapshot lineage, CCS MUST fail the request closed as `SF_BAD_REQUEST` rather than silently switching lineage or synthesizing an empty stat branch - if `extended_requests[]` names a tuple outside the declared family stored in the selected snapshot, CCS MUST reject the request as `SF_BAD_REQUEST`; the active revision does not perform on-demand computation or silent omission for unsupported tuples - `optimizer_statistics_head.json` and the immutable snapshot objects are the durable truth; both TiDB statistics tables are rebuildable hot projections - recovery rebuilds `collection_statistics_heads` from `optimizer_statistics_head.json` and then restores the referenced immutable `optimizer_statistics_v{applied_schema_generation_token}_{stats_commit_ts}.json` objects into `collection_statistics_snapshots` as needed ### 5.9 Lifecycle Tier-Window Projection and Retention Ownership Lifecycle policy is admitted through schema/control APIs, but CCS owns the rebuildable hot projection that turns that policy into due transition work, retention deadlines, and namespace aggregate inputs. Source-of-truth rule: - durable truth is the admitted `CollectionSettings.lifecycle_policy` plus the latest committed manifests for the collection. - CCS hot lifecycle tables are rebuildable online projections; query serving MUST NOT treat them as a second source of truth for `target_tiers[]` pruning. Hot projection table: ```sql CREATE TABLE collection_lifecycle_windows ( namespace_id BIGINT UNSIGNED NOT NULL, collection_id BIGINT UNSIGNED NOT NULL, window_start_ms BIGINT NOT NULL, window_end_ms BIGINT NOT NULL, effective_tier ENUM('HOT', 'WARM', 'COLD') NOT NULL, published_manifest_version BIGINT UNSIGNED NOT NULL, next_transition_at_unix_ms BIGINT, retention_deadline_unix_ms BIGINT, window_state ENUM('LIVE', 'TRANSITIONING', 'RECLAIMABLE') NOT NULL, PRIMARY KEY (namespace_id, collection_id, window_start_ms, window_end_ms) /*T![clustered_index] CLUSTERED */ ); ``` Ownership and derivation rules: - producer: CCS lifecycle projector over committed manifests plus the admitted collection lifecycle policy. - each row represents one lifecycle-managed time window for one collection. - `effective_tier` is the currently committed query-visible `DataTier` for that window. - `published_manifest_version` identifies the manifest version that made the current tier/object set visible. - `next_transition_at_unix_ms` is computed from the window boundary plus the admitted monotonic `transitions[]` ladder. - `retention_deadline_unix_ms` is computed from the window boundary plus admitted `retention_ms`; null means no lifecycle reclamation deadline is configured. - `window_state='RECLAIMABLE'` means lifecycle policy says the window is beyond its retention deadline, but physical deletion still waits for committed manifests to stop referencing the retired objects and for ordinary S3 GC safety checks to pass. Execution rules: - query-time `target_tiers[]` pruning reads manifest-published tier metadata directly from the pinned committed manifest and MUST NOT call CCS per request. - CCS lifecycle projection exists to drive background scheduling, namespace aggregates, and recovery, not to become a query-serving dependency. - tier transitions are manifest-first: - background workers rewrite/move the affected window's objects, - new object references and new `effective_tier` become visible only after committed-manifest CAS, - CCS updates `collection_lifecycle_windows` only after that manifest advancement succeeds. - retention ownership is lifecycle-aware: - generic GC rules are necessary but not sufficient, - lifecycle-managed windows MUST NOT be physically reclaimed before `retention_deadline_unix_ms`, - once that deadline has passed, reclamation still waits for manifest dereference and normal GC safety checks. Recovery rules: - CCS rebuilds `collection_lifecycle_windows` by replaying admitted lifecycle policy from schema state and walking the latest committed manifest for each lifecycle-managed collection. - missing relational lifecycle rows do not change query truth because manifests remain authoritative for query pruning. ### 5.10 Private Quota Allocation Surface CCS owns the only private quota-allocation/refill authority used by `Query Gateway` and `Ingest Gateway` for distributed token-bucket leasing. This surface is intentionally internal-only and is not part of the public protobuf contract; its machine-readable owner is `api/proto/searchfabric/v1/catalog_control_private.proto`. Scope and ownership rules: - consumers: `Query Gateway` and `Ingest Gateway` only - lease scope key: `(namespace_id, api_plane, throttled_dimension)` - `namespace_id` is the routing owner and serializes on the namespace-id lane - `api_plane` is `QUERY` or `WRITE` - `throttled_dimension` names the service-static budget dimension for that plane (for example `query_rcu`, `query_qps`, `ingest_bytes`, `ingest_entries`) - local gateway token buckets are caches of CCS-issued leases; they are not durable truth Durable object path: ```text s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/namespace_quota_state.json ``` Hot projection tables: ```sql CREATE TABLE namespace_quota_state ( namespace_id BIGINT UNSIGNED NOT NULL, quota_epoch BIGINT UNSIGNED NOT NULL, quota_state_json JSON NOT NULL, last_refill_at_unix_ms BIGINT NOT NULL, updated_at_unix_ms BIGINT NOT NULL, PRIMARY KEY (namespace_id) /*T![clustered_index] CLUSTERED */ ); CREATE TABLE namespace_quota_leases ( namespace_id BIGINT UNSIGNED NOT NULL, api_plane ENUM('QUERY', 'WRITE') NOT NULL, throttled_dimension VARCHAR(64) NOT NULL, lease_holder_id VARCHAR(128) NOT NULL, lease_request_id VARCHAR(128) NOT NULL, granted_units BIGINT UNSIGNED NOT NULL, lease_epoch BIGINT UNSIGNED NOT NULL, expires_at_unix_ms BIGINT NOT NULL, terminal_state ENUM('ACTIVE', 'RETURNED', 'EXPIRED') NOT NULL, updated_at_unix_ms BIGINT NOT NULL, PRIMARY KEY (namespace_id, api_plane, throttled_dimension, lease_holder_id, lease_request_id) /*T![clustered_index] CLUSTERED */ ); ``` Durability and projection rules: - `namespace_quota_state.json` is the durable truth for namespace-scoped quota policy, refill watermark, currently leaseable budget, and `quota_epoch` - `namespace_quota_state` is the rebuildable online projection of that durable truth - `namespace_quota_leases` is the online projection of short-lived active leases; it is authoritative for steady-state fast checks but is rebuildable from durable quota state plus lease invalidation policy - gateways MUST treat lease expiry and `quota_epoch` mismatch as immediate invalidation and reacquire before spending more local bucket units Private lease contract: - each lease/refill request carries at least: - `namespace_id` - `api_plane` - `throttled_dimension` - `lease_holder_id` - `lease_request_id` - requested units - caller-observed `quota_epoch` - successful response returns: - granted units - `lease_epoch` (equal to current namespace `quota_epoch`) - `expires_at_unix_ms` - same `(namespace_id, api_plane, throttled_dimension, lease_holder_id, lease_request_id)` + same payload MUST replay the original grant deterministically - same key + different payload MUST be rejected as a private idempotency conflict - explicit early return is optional; lease expiry is always authoritative and reclaimable by CCS sweeper logic Recovery and invalidation rules: - recovery rebuilds `namespace_quota_state` from `namespace_quota_state.json` - if recovery cannot trust in-flight gateway lease state, CCS MUST bump `quota_epoch`, republish `namespace_quota_state.json`, and rebuild `namespace_quota_leases` as empty or fully expired - gateways holding leases from an older epoch MUST stop using them and reacquire from CCS before serving ordinary traffic - this epoch-bump rule is the canonical stale-lease revocation mechanism after relational projection loss or CCS failover - there is no second quota authority and no gateway-local recovery path that may continue spending stale leases across a `quota_epoch` change ## 6. Drop Tombstone Model ### 6.1 Durable Tombstone Object Every successful `DropCollection` writes: - a tombstone control object - a matching S3 command/outcome object ```text s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/collection_tombstone_v{drop_revision}.json ``` The tombstone object MUST carry at least: - `namespace_id` - `collection_id` - `collection_name` - `collection_name_allocation_epoch` - `drop_revision` - `dropped_at_unix_ms` - admitted drop fence snapshot when present - `gc_after_unix_ms` - manifest-head / descriptor revision references needed for audit and delayed cleanup ### 6.2 Behavioral Rules - After tombstone publication becomes active, `GetCollectionMetadata` and `GetCollectionSchema` for that collection fail closed as `SF_RESOURCE_NOT_FOUND`. - After tombstone publication becomes active, `GetSchemaSyncStatus` for that collection also fails closed as `SF_RESOURCE_NOT_FOUND`. - `ListCollections` excludes tombstoned collections. - replay of the same successful drop returns transport `OK` `DropCollectionSuccessResponse.status=already_dropped`. - recreating the same `(namespace_id, collection_name)` before tombstone GC completes is rejected as `SF_VERSION_CONFLICT` to prevent ABA races, and the response MUST carry the canonical lifecycle-conflict detail keys. - physical cleanup of obsolete schema objects, manifest lineage, and the tombstoned row happens only after `gc_after_unix_ms` and all referenced background cleanup preconditions are satisfied. - when that release point is reached, CCS records `collection_name_reservation_release_intent` before removing the online tombstoned reservation state; the anti-ABA allocator state in `collection_name_epochs` remains. ## 7. Schema Convergence State Machine `GetSchemaSyncStatus` is a serving-convergence contract, not a bare metadata-write acknowledgement. CCS uses four public states: - `applying` - schema mutation is accepted - serving fleet has not yet converged on the target schema generation - `ready` - schema generation is the active serving generation - required control metadata and serving-side convergence conditions are satisfied - `drifted` - CCS has detected divergence between expected control state and serving/materialized state - repair is still possible without declaring terminal failure - `error` - schema transition cannot progress automatically - operator or higher-level recovery action is required The state machine is driven by explicit CCS-owned signals: - `schema_object_persisted` - emitted when the target schema-generation control object and command/outcome object are durably written - effect: generation enters `APPLYING` and becomes the collection's active schema pointer for control-plane reads/admission - `serving_generation_ready` - emitted when the serving fleet has acknowledged readiness for the active generation - effect: generation enters `READY` - `drift_detected` - emitted by reconciler comparison between expected active-generation state and observed serving/materialized state - effect: generation enters `DRIFTED` - `repair_failed` - emitted when automatic repair exhausts policy or detects a terminal inconsistency - effect: generation enters `ERROR` - `repair_succeeded` - emitted when a drifted generation is brought back into the expected serving state - effect: generation re-enters `READY` State-transition rules: - `PutCollectionSchema` writes the new generation row, updates the collection's active schema pointer, and sets `sync_state='APPLYING'`. - `GetCollectionSchema` always reads through the collection's active schema pointer, even while the generation is `APPLYING`. - if the collection is still `ACTIVE` but has no active schema pointer yet, `GetCollectionSchema`, `GetSchemaSyncStatus`, and the downstream query-serving helper owners that depend on CCS schema/serving facts all fail closed as `SF_CATALOG_UNAVAILABLE` with canonical schema-bootstrap details. - the same no-active-schema bootstrap branch also applies to query execution itself; CCS read models and serving facts MUST support that canonical fail-closed branch rather than inventing a generic not-ready or `SF_RESOURCE_NOT_FOUND` branch. - `GetSchemaSyncStatus` reports the convergence state of that same active generation. - query-serving components MUST execute the latest `READY` generation: - if a previous `READY` generation exists, query serving keeps using it until `serving_generation_ready` promotes the active generation to `READY`. - if no previous `READY` generation exists yet but an active schema pointer does exist, query serving and the downstream query-serving helper owners that consume CCS serving facts MUST fail closed as `SF_CATALOG_UNAVAILABLE` with canonical schema-convergence details rather than speculating on the `APPLYING` generation. - if a query explicitly fences on `expected_schema_generation_token`, that token must equal the resolved query-serving generation; a token naming the newer active-but-not-`READY` generation fails closed as `SF_VERSION_CONFLICT`. ## 8. Control Objects in S3 ```text s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/namespace_descriptor_v{namespace_revision}.json s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/namespace_quota_state.json s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/collection_descriptor_v{descriptor_revision}.json s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/schema_generation_v{schema_version}_{schema_generation_token}.json s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/optimizer_statistics_head.json s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/optimizer_statistics_v{applied_schema_generation_token}_{stats_commit_ts}.json s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/collection_tombstone_v{drop_revision}.json s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/manifest_committed_head.json s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/manifest_head.json s3://bucket/catalog-log/{hash(command_identity)[0:2]}/{command_identity}.json s3://bucket/catalog-replay-transport/{hash(transport_lookup_key)[0:2]}/{transport_lookup_key}.json s3://bucket/catalog-replay-business/{command_kind}/{hash(business_lookup_key)[0:2]}/{business_lookup_key}.json s3://bucket/catalog-recovery-segments/segment_{recovery_segment_epoch}.json.zst s3://bucket/catalog-allocator/namespace-name/{hash(namespace_name)[0:2]}/{namespace_name}.json s3://bucket/catalog-allocator/collection-name/{hash(namespace_id)[0:2]}/{namespace_id}/{hash(collection_name)[0:2]}/{collection_name}.json s3://bucket/catalog-checkpoints/checkpoint_{checkpoint_epoch}.manifest.json s3://bucket/catalog-checkpoints/shard={checkpoint_shard_id}/checkpoint_{checkpoint_epoch}.json.zst ``` Version axes: - `namespace_revision`: monotonic namespace-descriptor revision - `descriptor_revision`: monotonic collection-descriptor revision - `schema_version`: canonical public schema lineage version - `schema_generation_token`: canonical fencing token for that schema generation - `optimizer_statistics_head.json`: private pointer to the newest admitted optimizer-statistics snapshot on the current query-serving `latest READY` lineage for one collection - `stats_commit_ts`: logical commit frontier covered by one immutable optimizer-statistics snapshot - optimizer-statistics immutable snapshot identity: collection-scoped composite tuple (`applied_schema_generation_token`, `stats_commit_ts`); schema-only serving-lineage promotion may therefore mint a new snapshot identity without advancing the user-data frontier - `drop_revision`: monotonic collection-drop/tombstone revision - `namespace_name_allocation_epoch`: private monotonic namespace-name reservation epoch for ABA-safe same-name recreation - `collection_name_allocation_epoch`: private monotonic collection-name reservation epoch for ABA-safe same-name recreation within one namespace - `recovery_segment_epoch`: monotonic ordered recovery-segment index used to discover the post-checkpoint replay set without scanning hash-sharded `catalog-log` prefixes - `command_identity`: deterministic replay/audit object identity for one CCS mutation or internal control intent - `transport_lookup_key`: deterministic replay lookup key for transport-level duplicate detection - `business_lookup_key`: deterministic replay lookup key for business-semantic duplicate/conflict detection when the command class defines such a key `manifest_committed_head.json` payload shape: - `committed_manifest_version` - `previous_committed_manifest_version` - `manifest_checksum_or_etag` - `publisher_epoch` - `updated_at_unix_ms` `manifest_head.json` payload shape: - `manifest_head_version` - `expected_previous_head` - `last_update_epoch` - `reconciler_status` Compacted CCS recovery checkpoint objects: - checkpoint manifest: - `checkpoint_epoch` - `checkpoint_upper_bound_applied_ts` - `checkpoint_upper_bound_recovery_segment_epoch` - `checkpoint_shard_count` - `checkpoint_shards[]` - checkpoint shard payload: - latest committed namespace rows - latest committed namespace quota-state rows - latest committed collection rows - latest committed schema-generation rows - latest committed tombstone rows - latest committed replay-index rows - latest committed allocator-state rows projected from canonical S3 allocator objects - latest committed `manifest_committed_head` / `manifest_head` rows Checkpoint rules: - CCS emits periodic compacted recovery checkpoints at a bounded cadence. - each checkpoint shard is a compacted committed-state snapshot, not an append log. - ordinary CCS disaster recovery MUST start from the latest complete checkpoint manifest and replay only post-checkpoint `catalog-recovery-segments`. These names are deterministic and sufficient for: - disaster recovery replay - reconciler repair - safe garbage collection of superseded control objects ## 9. Reconciliation ### 9.1 Manifest-Head Repair Manifest publish visibility is anchored by `manifest_committed_head.json`, not by CCS `manifest_head_version`. Query Broker manifest-resolution algorithm: - read `manifest_committed_head.json` through object-store cache or a co-located private cache - pin `(committed_manifest_version, manifest_checksum_or_etag)` for the full request lifetime and any derived continuation/search-context chain - treat CCS `manifest_head_version` as private derivative governance metadata, not as a Query Broker runtime input - if operators later observe `manifest_head_version < committed_manifest_version`, Query Broker still executes against the committed pointer version and ignores the stale CCS derivative head If `manifest_committed_head.json` is `N+1` but `collections.manifest_head_version` remains `N`, CCS: - marks `manifest_head_reconciler_status='REPAIR_PENDING'` - detects the gap from `manifest_committed_head.json`, the S3 command/outcome objects, and control objects - advances through `REPAIRING` - retries the CAS head update - updates `manifest_head_last_update_epoch` on each admitted head transition or repair attempt - returns to `manifest_head_reconciler_status='IDLE'` once the repair converges, or `ERROR` if operator intervention is required - preserves the old head until the CAS advance succeeds ### 9.2 Replay Lookup and Index Repair If replay-lookup objects or relational replay-index rows are missing or stale: - CCS loads the latest complete compacted recovery checkpoint, then loads only post-checkpoint ordered `catalog-recovery-segments/segment_{recovery_segment_epoch}.json.zst` - CCS dereferences the segment-listed `catalog-log/{hash(command_identity)[0:2]}/{command_identity}.json` command/outcome objects in segment order - CCS republishes any missing `catalog-replay-transport/...` objects for every mutating command and any missing `catalog-replay-business/...` objects only for command classes whose command/outcome records carry `business_lookup_key` - CCS rebuilds relational replay-index rows from those replayed command/outcome objects - CCS also repairs any missing optimizer-statistics hot projection rows from `optimizer_statistics_head.json` plus the referenced immutable `optimizer_statistics_v{applied_schema_generation_token}_{stats_commit_ts}.json` objects before statistics-serving traffic resumes - `terminal_state` and `replay_classification` remain distinct axes - replay conflict is represented as: - `terminal_state='FAILED'` - `replay_classification='REPLAY_CONFLICT'` ### 9.3 Tombstone Repair If a tombstone object exists but the relational projection backend still shows the collection as active: - CCS projects `lifecycle_state='TOMBSTONED'` - restores `drop_revision`, `dropped_at_unix_ms`, and `gc_after_unix_ms` - prevents stale list/get responses and blocks ABA recreation until tombstone GC ## 10. Scale and Operational Notes ### 10.1 Whale Namespace Provisioning Namespace provisioning is anchored on `PutNamespaceMetadata`. If the namespace is admitted with a private whale-tier policy binding, CCS MAY asynchronously pre-split TiKV ranges for the namespace after successful namespace materialization. That private tiering decision is internal CCS state and does not need to appear in the public `NamespaceDescriptor` shape. ### 10.2 Prefix-Safe Pagination Control-plane list tokens are opaque, but the online execution model is deterministic keyset pagination. Canonical catalog-name rules for this execution path: - `namespace_name`, `collection_name`, and the corresponding prefix filters are admitted only as canonical NFC UTF-8 strings of at most 128 Unicode scalar values with no silent normalize-on-write. - CCS rejects invalid UTF-8, Unicode control characters, bidirectional formatting/isolate control characters, and default-ignorable format characters as malformed catalog-name input. - the relational projection backend MUST use deterministic binary comparison semantics for catalog names, not locale-aware / case-insensitive / accent-insensitive comparison. - on TiDB/MySQL profiles, this means `utf8mb4` binary collation (`utf8mb4_bin`). - on PostgreSQL profiles, this means a bytewise/deterministic collation profile such as `COLLATE "C"` on already validated canonical NFC text. - stable ordering, equality, uniqueness, replay-name identity, prefix filtering, and page-token last-seen keys all operate on those canonical NFC UTF-8 byte sequences. - empty prefix means no prefix range predicate. Illustrative `ListCollections` query shapes: ```sql -- branch A: empty prefix SELECT * FROM collections WHERE namespace_id = :namespace_id AND lifecycle_state = 'ACTIVE' AND collection_name > :last_seen_name ORDER BY collection_name ASC LIMIT :effective_page_size; -- branch B: non-empty canonical NFC prefix SELECT * FROM collections WHERE namespace_id = :namespace_id AND lifecycle_state = 'ACTIVE' AND collection_name >= :prefix AND collection_name < :prefix_successor AND collection_name > :last_seen_name ORDER BY collection_name ASC LIMIT :effective_page_size; ``` For non-empty prefixes, `prefix_successor` is the smallest canonical UTF-8 byte string strictly greater than all canonical names whose NFC UTF-8 bytes share the requested prefix bytes. Under a backend's required binary-collation profile, this is equivalent to the bytewise lexicographic upper bound for that prefix. The opaque token MUST bind: - RPC method - server-authenticated caller identity - admitted `client_id` - admitted `client_kind` - `principal_subject` when present on the bound request - list scope - literal prefix after canonical NFC validation - effective page size - last-seen canonical sort key Catalog page tokens are part of the same unified token family used by query page/search-context/composite-page tokens: - they MUST use the same shared signed binary wrapper format, explicit `key_id`, TTL policy, and key-rotation overlap window, - they are not a CCS-local ad hoc token format, - and successful list responses with non-empty `next_page.token` MUST also return the corresponding `next_page.expires_at_unix_ms`. Traversal semantics: - `ListCollections` and `ListNamespaces` use authoritative keyset pagination, not a snapshot-pinned list session. - each page is served from the latest authoritative relational online projection at that request boundary. - opaque tokens do not bind a catalog snapshot revision or catalog-read timestamp in this revision. - concurrent create/drop/recreate activity between pages may therefore change which rows appear on later pages of the same logical traversal. ## 11. Recovery If relational projection state is lost, CCS rebuilds from commit-authoritative S3 command/outcome objects plus referenced S3 control objects: - CCS first loads the latest complete compacted recovery checkpoint manifest and its shard objects to establish the committed base state at `checkpoint_upper_bound_applied_ts`. - CCS then loads only `catalog-recovery-segments/segment_{recovery_segment_epoch}.json.zst` objects whose segment epoch is greater than `checkpoint_upper_bound_recovery_segment_epoch`, and replays the referenced `catalog-log/{hash(command_identity)[0:2]}/{command_identity}.json` command/outcome records in segment order. - CCS republishes any missing replay-lookup objects from the recorded `transport_lookup_key` and, when present on the command/outcome record, `business_lookup_key` before replay-sensitive traffic resumes. - namespace descriptors rebuild only from `namespace_descriptor_v{namespace_revision}.json` objects referenced by `APPLIED` command/outcome records. - namespace quota state rebuilds from `namespace_quota_state.json`, with `namespace_quota_state` restored from that durable object and `namespace_quota_leases` rebuilt conservatively under the current `quota_epoch` - private namespace lifecycle state and reservation-release outcomes rebuild from the corresponding internal namespace lifecycle command/outcome records, with namespace rows materialized only from the latest committed lifecycle transition still in force. - collection descriptors rebuild only from `collection_descriptor_v{descriptor_revision}.json` objects referenced by `APPLIED` command/outcome records. - schema generations rebuild only from `schema_generation_v{schema_version}_{schema_generation_token}.json` objects referenced by `APPLIED` command/outcome records. - optimizer-statistics head state rebuilds from `optimizer_statistics_head.json`, and optimizer-statistics snapshots rebuild from the referenced `optimizer_statistics_v{applied_schema_generation_token}_{stats_commit_ts}.json` objects, with `collection_statistics_heads` recomputed from that head state. - lifecycle-window hot state rebuilds from admitted collection lifecycle policy plus the latest committed manifests for lifecycle-managed collections, with `collection_lifecycle_windows` recomputed from that authoritative pair rather than from stale relational rows. - tombstone state rebuilds from `collection_tombstone_v{drop_revision}.json` objects referenced by `APPLIED` command/outcome records, then applies any later committed `collection_name_reservation_release_intent` outcomes so released names do not reappear as still-reserved tombstones after recovery. - allocator-state rows rebuild from canonical S3 allocator objects before same-name create/recreate admission resumes. - replay index rows rebuild from the same replayed `catalog-log/{hash(command_identity)[0:2]}/{command_identity}.json` records. - `manifest_committed_head` state rebuilds from checkpoint state plus `manifest_committed_head.json`, with the pointer object validated against committed manifest lineage before CCS resumes ordinary traffic. - manifest-head state rebuilds from `manifest_committed_head` plus repaired CAS advancement, never from orphan control objects alone. - when quota recovery cannot safely preserve previously granted gateway leases, CCS MUST increment `quota_epoch` during rebuild so stale local gateway buckets fail closed and reacquire. During recovery, CCS MAY temporarily gate mutation admission and serve only validated read paths according to higher-level readiness policy. It MUST NOT guess missing typed descriptor fields, missing tombstone state, or missing replay outcomes.