# Why Catalog Control Service Matters for Scale, Availability, and Durable Persistence

In a large search platform, the control plane is easy to underestimate.

Most teams focus on query latency, indexing throughput, or storage cost first. Those matter. But once a system has to support many namespaces, many collections, rolling schema evolution, and continuous background publishing, the harder question becomes: what keeps the catalog itself correct while the rest of the system keeps moving?

That is the role of Search Fabric's `Catalog Control Service` (CCS).

CCS is the control-plane authority for namespace metadata, collection metadata, schema generation lifecycle, collection deletion, and catalog readback. It is also the part of the system that makes those operations durable, replay-safe, and recoverable at production scale. The public API is intentionally compact: customers see a small set of gRPC methods for creating and updating namespaces and collections, listing and reading them back, evolving schemas, checking schema sync status, and dropping collections. Behind that API, CCS is designed to answer a much more demanding question: how do you let the catalog evolve continuously without making control metadata fragile?

For architects evaluating this module, three properties matter most:

| Concern | CCS design choice | Customer outcome |
| --- | --- | --- |
| Extensibility | Contract-first APIs, stable IDs, typed descriptors, schema-generation lineage | Catalog features can evolve without redefining identity or overloading ad hoc metadata |
| High availability | Fast online projections in TiDB, deterministic replay, projection-gated ACKs, cache-affine ownership, repairable derivative state | Control reads stay efficient, and failures do not force the system to guess catalog truth |
| Durable persistence | Immutable S3 command/outcome records, durable control objects, tombstones, allocator state, checkpoints | Catalog truth survives TiDB loss and can be rebuilt without redefining committed state |

## Why the control plane has to be designed differently

The central CCS design choice is a deliberate separation between online serving state and durable control truth.

Steady-state public reads are served from TiDB materialized projections. That keeps common metadata operations fast and avoids mandatory object-store reads on every `GetCollectionMetadata`, `ListCollections`, or schema status query. But TiDB is not treated as the final source of truth for committed control state. Durable truth lives in S3 control objects and immutable S3 command/outcome records.

That split is important because it avoids a common failure mode in distributed control planes: the online database becomes both the serving cache and the only durable history. In that model, fast reads are easy, but rebuilding correctness after partial failure is hard. CCS does the opposite. It uses TiDB for efficient online serving, while keeping the authoritative catalog history in durable object storage. The result is a control plane that can scale for everyday reads without tying long-term correctness to the survival of one serving projection.

This is also an example of what an architect usually wants from infrastructure reuse: CCS does not try to reinvent a metadata database from scratch. It meta-builds on top of TiDB's existing horizontal scalability and high-availability behavior by treating TiDB as the online materialized serving layer for control metadata, then adds the control-plane semantics that TiDB alone does not provide, such as replay lineage, tombstone lifecycle, allocator safety, and checkpoint-based catalog recovery. In other words, the service amplifies TiDB's strengths instead of bypassing them.

## Scalability starts with deterministic ownership

At scale, control-plane correctness usually fails at contention boundaries, not at the happy path.

CCS addresses that by routing work through deterministic ownership lanes. Namespace creation is serialized on a namespace-name lane before a stable `namespace_id` exists. Collection creation is serialized on a collection-name lane before a stable `collection_id` exists. Once identity exists, lifecycle and schema operations move to ID-based lanes.

That sounds like an implementation detail, but it has direct business value. It means the platform does not rely on ambiguous "who owns this mutation?" logic when multiple create, update, drop, or recreate requests hit the same catalog name. The routing model defines clear serialization points for the exact keys that matter.

The same idea appears in catalog pagination. CCS uses deterministic keyset pagination rather than snapshot-pinned list sessions. Tokens bind the caller identity, scope, prefix, page size, and last-seen key, and they are governed by the same signed token model used elsewhere in the platform. This keeps catalog traversal scalable without turning listing into a long-lived server-side session problem.

Operationally, this routing model also maps well to a consistent-hash placement strategy. Because ownership is deterministic and node-local caches are disposable, the platform can preserve lane-to-node affinity across ordinary scale-out events instead of redistributing hot metadata paths randomly. That improves cache hit rate under expansion, because only a fraction of ownership lanes move when the ring changes.

The same property helps during failure handling. If a node disappears, the consistent-hash reassignment naturally promotes a replacement owner for the affected lanes, and that replacement can repopulate its caches from the authoritative TiDB projection and the durable S3 control truth. The cache is therefore recoverable automatically, without turning cache loss into catalog ambiguity.

## Extensibility comes from stable identity, not from flexible strings

Many control planes become difficult to evolve because identity and meaning are mixed together. A collection name becomes the durable primary key. A schema update rewrites semantics in place. Optional metadata grows into an untyped catch-all.

CCS is explicitly designed to avoid that trap.

First, stable namespace and collection IDs are allocated before descriptor publication and persisted into durable control objects. That means identity does not depend on the incidental lifetime of a TiDB row, and replay of an admitted create reuses the same ID instead of allocating a new one.

Second, same-name recreation is guarded by durable allocation epochs. A name can be released and later reused, but the lineage is still distinguishable. That matters when applications want deletion and recreation to be operationally safe rather than "probably fine."

Third, the public descriptors are typed. Encryption, indexing state, and approximate statistics are explicit fields. For collections, descriptor completeness is explicit too. If a descriptor is still being enriched, CCS says so directly instead of silently omitting fields and forcing clients to guess whether the omission is intentional, temporary, or erroneous.

This is a better extensibility model for enterprise platforms. It gives the product room to add richer control-plane capabilities without asking architects to reverse-engineer semantics out of loosely structured metadata blobs.

## Schema evolution is treated as a control problem, not just a write

Schema change is where many distributed systems reveal whether their control plane is truly designed or merely assembled.

CCS models schema evolution as explicit generation lineage. Each admitted schema generation carries a `schema_version`, a `schema_generation_token`, the active integration mode, the write mode, and a convergence state. The public surface exposes both the active generation for control-plane readback and the serving readiness of that generation.

The most important design choice here is that the latest active generation is not automatically the latest serving generation. Query serving resolves against the latest `READY` generation. If a newer generation has been accepted into the control plane but is still `APPLYING`, `DRIFTED`, or `ERROR`, the system does not pretend that it is ready just because the metadata write succeeded.

For architects, that distinction matters. It means schema rollout can progress without creating a false sense of readiness. `GetSchemaSyncStatus` tells you where the active generation stands. `GetReadiness` tells you what query serving can actually execute against. That is exactly the kind of contract boundary a cautious technical buyer expects in a production control plane.

## High availability depends on replay-safe mutation semantics

Control-plane availability is not only about having multiple nodes. It is also about surviving duplicates, retries, partial failures, and restarts without producing conflicting state.

CCS treats replay semantics as a first-class part of the architecture. Public mutations use transport replay identity through `RequestMeta.idempotency_key`. Schema changes and drops also have business-level replay rules where needed. Internal control intents use deterministic internal identities. The durable replay truth is recorded in immutable S3 command/outcome objects, while replay lookup objects and TiDB replay indexes are treated as rebuildable accelerators.

This matters because it lets the system recover from partial failure without turning replay into guesswork. The control plane can answer, with durable evidence, whether a request was the original command, the same request delivered again, or a conflicting attempt under the same logical identity.

Just as importantly, CCS does not acknowledge a public mutation as successful immediately after writing a durable object. It acknowledges success only after the authoritative online public-read projection has been updated. That reduces a class of control-plane anomalies that architects often encounter in weaker systems: the write succeeds, but the next read still sees the old state because the read model is lagging in an undefined way.

This is another place where CCS benefits from building on TiDB rather than around it. TiDB already gives the system a horizontally scalable, highly available serving substrate for the materialized catalog view. CCS uses that substrate as the fast read plane, then adds deterministic ownership, replay repair, and durable recovery above it. The result is not a second metadata stack competing with TiDB, but a control-plane architecture that compounds TiDB's existing scalability with stronger catalog semantics.

## Persistence is built around committed truth, not around optimistic repair

If the architect question is "what survives a database loss?", CCS gives a concrete answer.

Committed control truth survives in S3:

- namespace and collection descriptor objects
- schema generation objects
- collection tombstone objects
- immutable command/outcome records
- replay lookup derivatives
- allocator-state objects for name reuse safety
- periodic compacted recovery checkpoints
- ordered post-checkpoint recovery segments

That layout is more than archival storage. It defines the recovery contract. If TiDB materialized state is lost, CCS rebuilds from the latest complete checkpoint and replays only post-checkpoint recovery segments. It does not need to rediscover truth by scanning an unbounded history blindly, and it does not need to infer committed state from whatever happened to survive in an online table.

This is especially relevant for architects who care about data persistence in the broader operational sense: not just whether bytes are stored durably, but whether the system can reconstruct authoritative catalog state with the same identities, tombstones, replay outcomes, and schema lineage after a failure.

## Delete and recreate are handled with audit-grade discipline

Collection deletion is another area where enterprise control planes are often weaker than they appear.

In CCS, a successful drop is not just a row removal. It writes a durable tombstone object and a matching command/outcome record. The collection disappears from list and get paths, but the tombstone remains until garbage-collection rules and cleanup preconditions are satisfied. Same-name recreation is blocked until that reservation is safely released, and allocator state persists beyond release so later recreation still has an ABA-safe lineage boundary.

That model is operationally conservative in the right way. It avoids accidental identity reuse, preserves auditability, and keeps replay behavior deterministic even when architects automate create-drop-recreate workflows.

## Derivative state can lag without making the system incorrect

One of the strongest parts of the CCS design is that it distinguishes committed state from derivative state.

For example, manifest visibility is anchored by `manifest_committed_head.json`. If derivative head metadata inside CCS lags, the system treats that lag as a repair problem, not as a reason to forget what was actually committed. Query-time resolution follows the committed pointer, while CCS repairs the lagging derivative head asynchronously.

This is the right availability posture for a large system. It allows some internal projections to be repaired without turning those projections into correctness bottlenecks.

## What this means for an architect

The practical conclusion is straightforward: CCS is not just an admin API layer. It is the mechanism that makes catalog change safe at scale.

If your organization expects many namespaces, continuous collection lifecycle changes, rolling schema evolution, strict replay behavior, and recovery that preserves identity and history, this design is materially stronger than a conventional metadata service backed only by an online relational table.

Its core advantages are structural:

- fast public control reads from online projections
- durable catalog truth independent of those projections
- explicit lineage for IDs, schemas, tombstones, and replay outcomes
- deterministic pagination and name handling for large catalogs
- bounded disaster recovery through checkpoints and ordered recovery segments

That combination is why CCS is a scalability feature, not only a governance feature. It lets the search platform grow in namespace count, collection count, operational concurrency, and schema churn without making the catalog itself a weak point.