# CDC Adapter Date: 2026-03-16 Status: Frozen architecture note Authors: Search Platform Architecture ## 1. Scope This document defines the Flow-A `TiCDC -> CDC Adapter -> IngestGatewayService.StreamMutations` realization for Search Fabric. Flow-A applicability is catalog-authoritative, not adapter-local: - admitted collection truth mode is only: - `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL` - `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC` is out of Flow-A scope and must not accept TiCDC-driven mutation ingest. Canonical ownership remains unchanged: - machine-readable wire contract: `api/proto/searchfabric/v1/search_fabric.proto` - normative API semantics: `docs/specs/core/search-fabric-api.md` - canonical error semantics: `docs/specs/core/error-codes.md` - global runtime architecture: `docs/specs/core/search-fabric-architecture.md` This document does not redefine public protobuf fields or canonical `SF_*` mappings. When it uses phrases such as “adapter carries”, “adapter tags”, or “adapter marks”, that means one of: - an existing canonical protobuf field, - adapter-private transport metadata on the internal gRPC hop, - adapter-private quarantine/audit state, - or a clearly marked future protocol follow-up. Out of scope: - Flow-B Search-Fabric-truth write admission (owned by `docs/architecture/ingest-gateway.md`) - TiDB federated-table remote-DML lowering/execution (owned by `docs/architecture/tidb-search-fabric-integration.md`) - TiDB planner/executor query behavior (owned by `docs/architecture/tidb-search-fabric-integration.md`) - public control-plane schema semantics (owned by `CatalogControlService`) ## 2. Role and Runtime Shape The CDC Adapter is the source-side normalization and reliability boundary between TiCDC changefeeds and canonical Search Fabric mutation ingest. Its responsibilities are: - subscribe to TiCDC row and DDL events - resolve authoritative collection/schema identity before emit - normalize source rows into canonical `MutationBatch` / `MutationEntry` - preserve TiCDC ordering, replay, and checkpoint semantics - consume the canonical DDL-effect classification produced by the separate TiDB DDL/schema-sync actor and react to it on the Flow-A data path - stop only at search-affecting DDL boundaries and resume or terminate only after the canonical control-plane branch converges - provide source-side quarantine, shadow, sampling, masking, and audit hooks without redefining public ingest semantics Its non-responsibilities are: - computing collection `safe_ts` - inventing schema truth outside `CatalogControlService` - weakening canonical validation or error selection - bypassing Ingest Plane idempotency, fencing, or ACK rules Recommended runtime decomposition: ```text TiCDC Subscriptions -> Adapter Manager -> Collection Router -> Schema Resolver -> Transformer / Validator -> Batch Builder -> Dispatch Scheduler -> StreamMutations -> Checkpoint Store Side channels: -> Quarantine Zone -> Shadow Evaluation Store -> Metrics / Tracing / Audit Log ``` Core components: - `AdapterManager`: process supervisor, liveness monitor, worker recycling, config rollout. - `SubscriptionWorker`: owns one TiCDC subscription shard and emits ordered row / DDL / resolved-ts events. - `SchemaResolver`: caches `CollectionRef`, active `schema_generation_token`, effective schema metadata, and source supplemental metadata such as collation hints. - `Transformer`: performs type mapping, null encoding, PK extraction, masking, JSON normalization, vector validation, and provenance stamping. - `BatchBuilder`: groups events into single-stream batches that share one `schema_generation_token` and, for any `upsert` / `patch` rows they contain, one uniform row-level `schema_version`, then builds canonical `MutationBatch` payloads. - `DispatchScheduler`: applies RU gating, fair queueing, priority, backpressure, retry, and graceful drain behavior. - `CheckpointStore`: persists last successful ACK cursor, last acknowledged watermark, optional last sent watermark for observability, DDL barrier state, and recovery metadata. - `CollectionFenceCoordinator`: tracks collection-scoped DDL fence epochs, active source partitions, per-partition pre-DDL barrier completion, and the resume/detach decision after control-plane convergence. ## 3. Design Rules 1. Contract first: - the adapter must emit only canonical `MutationBatch`, `MutationEntry`, `RowMutation`, `ProgressMark`, and `BarrierMark` payloads. - adapter-private features must not invent a parallel public wire shape. 2. Source truth first: - Flow-A owns `RowMutation.commit_ts`; the adapter must forward TiCDC commit timestamps as authoritative source ordering data. 3. Fence before emit: - every emitted `MutationBatch` must carry the schema-generation token for that batch's schema-generation cohort; in Flow-A it may be active or historical within the canonical commit-ts validity boundary. - the adapter must stop at DDL boundaries rather than racing incompatible rows across a schema cut. 4. Server-derived visibility: - the adapter may forward `resolved_ts`. - the adapter must never calculate or send aggregate `safe_ts`. 5. No silent lossy coercion: - decimal, JSON, vector, null, and composite-key handling must remain reversible or fail closed. 6. Source-side safety features must be explicit: - quarantine, masking, sampling, shadow, and canary modes are opt-in controls with deterministic observability. - canonical production Flow-A projection must remain lossless: it must not silently quarantine invalid rows and continue advertising normal freshness/readiness as if projection completeness were intact. ## 4. Data Contract & Transformation ### 4.1 Data Type Mapping & Null Handling The adapter normalizes TiCDC row columns into canonical `FieldData` carriers using the collection schema currently admitted by Search Fabric. Mapping baseline: | TiDB source family | Canonical carrier | Adapter rule | |---|---|---| | signed integer types | `AtomicValue.int64_value` / `SCALAR_FIELD_TYPE_INT64` | preserve exact signed range; overflow is malformed | | unsigned integer types | `AtomicValue.uint64_value` / `SCALAR_FIELD_TYPE_UINT64` | preserve exact unsigned range; required for active-profile canonical `doc_id` | | `FLOAT` | `AtomicValue.float_value` / `SCALAR_FIELD_TYPE_FLOAT` | preserve 32-bit payload; hashing uses canonical IEEE 754 rules | | `DOUBLE` | `AtomicValue.double_value` / `SCALAR_FIELD_TYPE_DOUBLE` | preserve 64-bit payload; hashing uses canonical IEEE 754 rules | | `DECIMAL` / `NUMERIC` | `AtomicValue.string_value` under a schema field declared to preserve exact decimal text | never downcast to float/double unless an explicit collection schema already chose a lossy floating field | | `TIMESTAMP` / `DATETIME` / time-partition columns | `AtomicValue.int64_value` / `SCALAR_FIELD_TYPE_TIMESTAMP` | encode Unix microseconds exactly | | `JSON` | canonical JSON text in `AtomicValue.string_value`, or `bytes_value` when the owning field schema chooses raw bytes | canonicalization occurs before hashing, quarantine, and shadow compare | | text / varchar / char / enum / set | `AtomicValue.string_value` / `SCALAR_FIELD_TYPE_STRING` | preserve NFC-normalized source text where collection policy requires it | | binary / varbinary / blob | `AtomicValue.bytes_value` / `SCALAR_FIELD_TYPE_BYTES` | preserve raw bytes | | vector payloads | `FieldData.vector_value` | dimension and finiteness checks are mandatory | | one-dimensional arrays | `FieldData.list_value` | element type must be homogeneous | | object / nested payloads | `FieldData.object_array_value` | active public payload contract admits only depth `<= 1`; deeper source shapes must be rejected or quarantined before public payload emission | Null handling rules: - explicit NULL must be encoded as `FieldData.atomic_value.null_value = NULL_VALUE_NULL`. - missing and null must stay distinct: - in `patch`, missing means “leave unchanged”; explicit null means “clear”. - in `upsert`, missing means the field is absent in the new full-row state; explicit null means the field is present with null value. - the adapter must never collapse “column absent from source image”, “column present with SQL NULL”, and “field intentionally omitted by patch semantics” into one representation. Implementation notes: - the current canonical protobuf already satisfies the “missing vs null” requirement through `oneof` presence plus `NULL_VALUE_NULL`; no parallel wrapper-type contract should be introduced in this architecture document. - if TiCDC omits unchanged columns in update events, the adapter must recover the correct patch/upsert semantics from the upstream event shape before building canonical `FieldData`. - Flow-A has no lifecycle-enabled collection branch in the current revision: - lifecycle policy is admitted only for `COLLECTION_TRUTH_MODE_SEARCH_FABRIC`, - this document's Flow-A scope is restricted to `COLLECTION_TRUTH_MODE_EXTERNAL`, - so CDC Adapter must treat lifecycle-specific event-time carry rules as out of scope rather than trying to realize them locally on external-truth collections. - if a future canonical API/proto revision ever admits lifecycle policy on external-truth collections, the required event-time carry contract must be added there first and only then reflected in this architecture note. ### 4.2 Primary Key Extraction & Normalization The adapter must derive one stable business identity per source row before `RowMutation.doc_id` is emitted. Current Flow-A admission boundary: 1. the collection must already declare one canonical `DocIDMapping.source_field` whose type is admitted as `SCALAR_FIELD_TYPE_UINT64`, 2. the source row must carry that canonical `uint64` `doc_id` value losslessly in the row image presented to the adapter, 3. if that canonical `uint64` identity is absent, non-lossless, or otherwise unavailable in the source row, the adapter must reject or quarantine the row before `MutationBatch` emission. Composite-key normalization: - the adapter's source-side business-key normal form is the ordered concatenation of normalized key segments. - separator is logically `|`, but the implementation must use an escaping-safe form such as length-prefixed segments before final render so embedded separators cannot create ambiguity. - the business-key normal form is used for: - quarantine records, - shadow compare, - dedupe diagnostics, - and recovery/audit trails. Current canonical-contract constraint: - the active public Search Fabric profile still admits only `DocumentID.id_value.u64` for hot-path canonical row identity. - direct emission of composite string `_id` values into `RowMutation.doc_id` is therefore out of current canonical scope. - Flow-A does not define a surrogate-id bridge, PK/UK projection fallback, or composite-key-to-`uint64` mapping layer in the active revision. - source tables that do not already expose the required canonical `uint64` `doc_id` source column are outside the current Flow-A contract and must be rejected or quarantined rather than adapted implicitly. Collation rule: - collection-specific source collation semantics used for PK/UK normalization must be read from the schema's supplemental source metadata, because charset/collation are not canonical typed schema fields. - the adapter must apply the same source collation normalization on every retry and worker restart. - canonical catalog identity ordering remains binary/NFC, but that catalog-name rule must not be confused with source SQL key comparison rules. ### 4.3 Heterogeneous Data Safeguard Canonical JSON: - object keys must be normalized before hashing or shadow comparison. - key ordering must be deterministic and byte-stable. - insignificant formatting must be removed. - maximum source-side JSON nesting depth is 16; rows exceeding that depth must be rejected or quarantined before public payload emission to avoid recursive-stack and pathological-shape failures. Vector integrity: - vector dimensions must be checked against the declared `VectorTypeSpec.dimensions`. - all float-bearing vector branches must reject `NaN` and `Inf`. - `f16_vector` decoding must verify even byte length and little-endian binary16 framing before any hash or dimension computation. - quantized payload branches must still validate logical dimension count exactly. Quarantine escape hatch: - `allow_quarantine=false`: - invalid rows fail the affected batch by the normal canonical path. - `allow_quarantine=true`: - the adapter diverts invalid source rows to a Quarantine Zone before `MutationBatch` emission. - this mode is reserved for shadow validation, pre-production burn-in, or explicitly non-authoritative remediation lanes. - it MUST NOT be enabled for a production Flow-A lineage whose public freshness/readiness signals are expected to represent a lossless TiCDC projection. - when enabled in a non-authoritative lane, operators MUST treat that lane as degraded and MUST NOT interpret its normal checkpoint/freshness advancement as proof of projection completeness. - quarantine records must contain at least: - collection identity, - business key / `doc_id`, - TiCDC position, - `commit_ts`, - schema token/version seen by the adapter, - validation error code, - adapter version, - logic epoch, - raw source payload snapshot. Important boundary: - the current canonical public ingest contract explicitly forbids hidden gateway-side quarantine/infer/replay for undeclared-field handling. - this design therefore keeps quarantine as a source-side adapter feature, not an Ingest Plane reinterpretation of `UNDECLARED_FIELD_POLICY`. - for the authoritative production Flow-A path, the recommended operating mode remains `allow_quarantine=false` so canonical freshness/readiness semantics are not decoupled from projection completeness. ## 5. Protocol & Consistency ### 5.1 Watermark Propagation & `safe_ts` Authoritative rule: - the adapter forwards TiCDC `resolved_ts` as the authoritative Flow-A progress watermark. - the adapter may emit `ProgressMark.watermarks` only as per-modality progress inputs when such modality-specific progress is available. - the adapter must not compute, aggregate, or transmit collection `safe_ts`. - authoritative progress confirmation exists only after `MutationAck.success`; the returned `acked_cursor` plus `applied_watermarks` define the applied recovery boundary, not local send timing. Monotonicity shield: - each worker persists `last_acked_watermark` as the authoritative replay-suppression boundary and MAY keep `last_sent_watermark` only for in-process observability and duplicate coalescing. - any incoming `resolved_ts` older than `last_acked_watermark` is discarded and counted as a source regression anomaly. - duplicate `resolved_ts` equal to `last_acked_watermark` may be coalesced but must not regress checkpoints. - duplicate `resolved_ts` equal to `last_sent_watermark` but newer than `last_acked_watermark` may be coalesced only while the corresponding ACK is still outstanding. Watermark emission policy: - watermarks must be emitted only after all earlier row mutations for the same stream scope are durably staged for dispatch. - on successful `MutationAck.success`, workers advance `last_acked_watermark` only from that acknowledged/applied boundary and persist the returned `acked_cursor` plus `applied_watermarks` as authoritative recovery state. - a worker restart must reload the last acknowledged cursor and last acknowledged watermark before resuming source consumption; `last_sent_watermark` may be restored for logs/metrics only and MUST NOT suppress replay after restart. - CDC replay-watermark invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-CDC-001`. ### 5.2 Deterministic Hashing (IEEE 754) The adapter must produce stable `batch_fingerprint` inputs across languages and CPU architectures. Rules: - normalize floats/doubles to canonical IEEE 754 bit patterns before hashing. - serialize those normalized bit patterns as little-endian bytes before feeding the hash implementation. - normalize `-0.0` to `+0.0`. - collapse all NaN payloads to one canonical NaN representation per width. - canonicalize map keys in byte-ascending order before hashing. - exclude transport and runtime envelope fields already excluded by the canonical contract, including `RowMutation.commit_ts`. The same normalization rules apply to: - scalar floating fields, - vector payloads, - JSON numbers once converted into canonical source-side numeric forms, - sampled shadow/canary checksums. ### 5.3 Protocol Addendum Checksum formatting: - all textual checksums and hashes must be lowercase hexadecimal without prefixes. - the adapter must reject `0x` forms, uppercase forms, and width-mismatched forms before forwarding. Identifier limits: - `RequestMeta.idempotency_key` must be capped at 128 Unicode scalar values. - `RequestMeta.client_id` must be capped at 256 Unicode scalar values. - `RequestMeta.client_kind` must be set explicitly to `CLIENT_KIND_TIDB_CDC_ADAPTER`. - truncation, hashing, or silent normalization of those identity-bearing strings is forbidden. Current-proto fit: - row-level schema cohorting in the current protobuf is realized here as: - every admitted batch carries one `schema_generation_token` as the batch-level schema-generation cohort, - every data-bearing batch that contains `upsert` / `patch` rows keeps those rows in one uniform row-level `schema_version`, - every `upsert` / `patch` row inside that batch carries the same `schema_version`, - the batch itself carries the schema-generation token for that schema-generation cohort on every admitted batch, including progress-only or barrier-only batches, - `delete`, `progress`, and `barrier` entries do not carry a row-level `schema_version`, - progress-only / barrier-only batches are therefore fenced only by `schema_generation_token`, not by a row-level `schema_version`, - no public batch-level `schema_version` field exists or is implied in this document. ## 6. Reliability & Lifecycle ### 6.1 DDL Boundary Control DDL classification authority: - the semantic classification of one TiDB DDL boundary (`non-search-affecting`, `search-affecting schema-evolution`, or `search-affecting destructive drop`) is owned by the separate TiDB DDL/schema-sync actor defined in `docs/specs/core/search-fabric-architecture.md`. - CDC Adapter consumes that already-classified decision together with the converged control-plane result; it does not independently redefine or override DDL semantic class on the Flow-A path. Adapter reaction by canonical DDL class: - non-search-affecting DDL: - MUST NOT force a full Flow-A drain/fence for the collection. - adapter may refresh supplemental metadata asynchronously or ignore the event when it provably cannot affect canonical Search Fabric schema/ownership semantics. - search-affecting schema-evolution DDL: - `CollectionFenceCoordinator` opens one collection-scoped DDL fence epoch keyed by the affected `CollectionRef` and DDL boundary, - no worker may emit post-DDL rows for that collection once the fence epoch is open, - every active source partition for the collection must drain pre-DDL rows, emit its final pre-DDL `BarrierMark`, and wait for the corresponding successful ACK, - a partition may be excluded from the final resume quorum only when the coordinator can prove it is quiescent, detached, or no longer owns rows for that collection at this boundary, - adapter then waits for the separate TiDB DDL/schema-sync control-plane actor to apply `PutCollectionSchema`, including the replay-critical schema-admission context persisted into `CollectionSchemaDefinition.replay_context`, and for that schema apply to converge to the new active generation, - resume is collection-wide, not worker-local: it is allowed only after the coordinator has both: - collected successful final pre-DDL barrier completion from every active partition or a proof of quiescence/detach for the excluded ones, - observed control-plane convergence to the new generation, - after that collection-wide decision, workers refresh `schema_generation_token`, effective schema metadata, and source supplemental metadata including the persisted DDL session-context subset, then resume rows strictly after the DDL boundary. - search-affecting destructive DDL: - the same collection-scoped fence epoch is opened and all active partitions must reach the final pre-DDL cut or be proven quiescent/detached, - adapter then waits for the separate TiDB DDL/schema-sync control-plane actor to apply canonical `DropCollection` semantics for the affected `CollectionRef`, - after successful drop convergence, the coordinator must publish a collection-wide detach decision rather than a resume decision, - after that detach decision, workers must not perform schema refresh plus resume; they terminate or detach the affected collection worker because the collection lineage is tombstoned. Failure policy: - if search-affecting schema or drop convergence fails, keep the collection worker fenced and retry control-plane refresh rather than leaking rows across the version boundary. - if the Ingest Plane returns `invalidated_at_commit_ts`, split the stream at that boundary and replay from the rejected cursor. ### 6.2 Retries, Idempotency & Checkpoints Retry identity: - same-producer retransmit MUST preserve `stream_identity`, `RequestMeta.idempotency_key`, `batch_id`, and `batch_fingerprint` for exact retry of the same admitted payload. - failover retransmit MUST preserve `source_partition`, `batch_id`, and `batch_fingerprint` but increment `source_epoch`; because canonical batch content changes, failover retransmit MUST use a new `RequestMeta.idempotency_key`. - sliced retries after first rejection must use a new `batch_id`. Checkpoint advancement: - TiCDC checkpoint may advance only after a successful `MutationAck`. - the durable checkpoint record must include: - acknowledged cursor, - resume token and expiry, - last acknowledged source watermark (`last_acked_watermark`), - last returned Search Fabric `applied_watermarks`, - optional `last_sent_watermark` for observability only; it MUST NOT be used as a replay-suppression boundary, - schema generation token, - effective schema lineage state for the acknowledged prefix (`schema_generation_token` always, uniform row-level `schema_version` when present), - quarantine side effects committed before the checkpoint. - authoritative Flow-A keeps `MutationBatch.policy.continue_on_error=false`; partial-accept CDC batches are not checkpoint-safe for completeness semantics. - in the recommended production Flow-A mode (`allow_quarantine=false`), successful checkpoint advancement therefore continues to imply lossless projection progress for that acknowledged prefix. - any explicitly non-authoritative lane that enables `allow_quarantine=true` must treat checkpoint advancement as transport/apply progress only and must not reuse it as a public completeness/freshness guarantee. Double-ACK alignment: - “source ACK” means the adapter has accepted and normalized the source event. - “Search Fabric ACK” means `MutationAck.success` confirms canonical admit/apply progress. - only the second ACK may move the durable TiCDC checkpoint. Recovery path: - on restart, workers rebuild state from the durable checkpoint, not from in-memory buffers. - replay begins from the last acknowledged cursor or server-provided restart cursor. - replay suppression compares incoming `resolved_ts` only against `last_acked_watermark`; pre-crash sent-but-unacknowledged watermarks must remain replay-eligible until acknowledged. - replay must preserve source ordering and batch fingerprint determinism. ### 6.3 Graceful Shutdown & Liveness Drain on `SIGTERM`: - stop accepting new TiCDC events, - stop the subscriber cleanly, - flush buffered batches, - wait for final ACK or checkpoint-safe timeout, - persist the final local checkpoint, - exit only after all owned collection workers reach `DRAINED` or `FORCED_ABORT`. Worker liveness: - each worker publishes a heartbeat at least every 5 seconds. - `AdapterManager` forcefully recycles any worker that fails to update heartbeat for more than 30 seconds. - recycled workers rehydrate from checkpoint; in-memory buffers from the dead worker are discarded unless already checkpointed. ## 7. Resource & Traffic Management ### 7.1 PD Resource Manager Integration RU binding: - an adapter deployment may bind to a configured TiDB Resource Group to pace source-side extraction work. - any such RU accounting is a local upstream pacing signal, not a second Search Fabric quota authority. Backpressure behavior: - when local RU budget is exhausted, the scheduler should slow source pull, queue growth, or batch construction for the affected resource group before additional dispatch work is prepared. - source-side local queues may continue only up to a bounded watermark; beyond that, the adapter must slow subscription pull or block batch construction. - RU starvation must surface through dedicated metrics and tracing events rather than appearing as generic latency. - canonical Search Fabric admission authority still remains the CCS-owned quota/throttling model; adapter-local RU pacing must not redefine server-side `SF_RATE_LIMITED` semantics. ### 7.2 Memory Guardrails `MemoryController` is a process-wide singleton. Recommended thresholds: - below 80% heap: normal batching. - at or above 80% heap: - increase `linger_ms`, - decrease `max_entries_per_batch`, - shrink per-collection in-memory queue budgets, - prefer flushing partially full batches. - at or above 90% heap: - stop admitting new large-value rows into memory, - force flush and drain, - apply local backpressure to TiCDC subscribers. The controller must react to: - raw heap usage, - vector payload bytes, - JSON expansion ratio, - quarantine spill backlog, - pending shadow-mode duplication overhead. ### 7.3 Fair Queuing & Priority Fairness: - the demultiplexer must use Weighted Round-Robin across collection queues. - default weights should prevent “elephant” collections from monopolizing worker time. - recovery traffic may not starve realtime traffic. Priority classes: - `PRIORITY_REALTIME`: steady-state online CDC for freshness-sensitive collections. - `PRIORITY_RECOVERY`: replay, catch-up, backfill, or restart recovery traffic. Current-contract fit: - priority is adapter-internal scheduler metadata or private gRPC metadata on the internal hop. - it is not a public protobuf enum in the current canonical `MutationBatch`. ## 8. Observability & Quality Assurance ### 8.1 Metrics & Tracing Metrics aggregation: - default Prometheus labels must aggregate at namespace level. - collection-level drill-down must be disabled by default and enabled only for targeted debugging. Required metric families: - source lag and watermark: - `cdc_adapter_resolved_ts_lag_ms` - `cdc_adapter_watermark_regression_total` - dispatch and ACK: - `cdc_adapter_batch_build_total` - `cdc_adapter_batch_retry_total` - `cdc_adapter_ack_latency_ms` - `cdc_adapter_checkpoint_advance_total` - validation and safety: - `cdc_adapter_quarantine_total` - `cdc_adapter_vector_reject_total` - `cdc_adapter_json_depth_reject_total` - `cdc_adapter_shadow_hash_diff_rate` - resource control: - `cdc_adapter_ru_wait_ms` - `cdc_adapter_heap_pressure_ratio` Tracing: - every batch span must include TiDB `commit_ts`, source partition identity, schema token, and batch fingerprint. - DDL boundary spans must show flush start, barrier ACK, schema refresh, and resume. ### 8.2 Data Sampling & Lineage Probabilistic checksum sampling: - sample 0.01% of rows after transformation but before dispatch. - compute a source-side checksum over: - business key, - transformed row payload, - uniform row-level `schema_version` when present, - schema generation token, - logic epoch. - send sampled checksums to an async verification sink consumed by an adapter-owned comparison job. Provenance: - every emitted batch must be associated with: - `_adapter_version` - `logic_epoch` - transformation policy revision - masking policy revision - in the current contract this provenance should travel through adapter-private audit metadata or explicit collection-reserved fields only when the collection schema opts in. - the adapter must not silently invent new public document fields on collections that did not declare them. ## 9. Security & Validation ### 9.1 Security & Credential Management mTLS: - all gRPC communication from CDC Adapter to Search Fabric must use mutual TLS. - service identity must bind to the allowed namespace/collection scope before any batch dispatch. Caller identity: - every batch dispatch must carry canonical `RequestMeta` caller identity, including stable `client_id` plus `client_kind=CLIENT_KIND_TIDB_CDC_ADAPTER`. - the active canonical contract relies on mTLS caller binding plus `RequestMeta`; this document does not introduce a second mandatory per-batch credential family. - if a deployment adds extra private credentials on the internal hop, that remains deployment-private hardening and must not be treated as a canonical public or cross-component contract in this document. PII masking: - transformer hooks must support rule-based masking before data leaves the TiDB trust zone. - masking rules should support deterministic hashing, tokenization, and allow-list pass-through by field path. - masking must happen before: - batch fingerprint construction, - shadow compare, - sampling checksum, - quarantine persistence outside the trust boundary. ### 9.2 Shadow Mode & Canary Validation Shadow ingestion: - the adapter supports an adapter-private shadow evaluation mode for canary analysis. - in the active revision, shadow mode is adapter-local offline comparison only: it MUST NOT depend on Search Fabric ingest-plane admission, validation-only execution, or a hidden non-committing internal endpoint. - the adapter may persist shadow samples, transformed payloads, and comparison digests to adapter-owned audit storage and compare them asynchronously against production-path outcomes, but it must not model shadow mode as a second Search Fabric write path. Canary analysis: - run shadow and production logic in parallel over the same source sample or tenant slice. - compute: - hash-diff rate, - reject-code drift, - schema-token mismatch rate, - quarantine delta rate, - latency overhead. - promote a new `logic_epoch` only when shadow metrics remain inside the configured promotion budget for a full observation window. ## 10. Required Contract Follow-Ups The requested design includes several behaviors that exceed the current public protobuf surface. This document keeps them explicit instead of redefining canonical semantics silently. Items that need either adapter-private realization or later protocol evolution: - composite-key concatenated `_id` as a public Search Fabric `doc_id` - batch-level public `schema_version` - public priority tag on `MutationBatch` - public provenance fields such as `_adapter_version` and `logic_epoch` - public per-batch access-token field Current recommendation: - keep those concerns in adapter-private metadata, adapter-owned offline audit/compare flows, control-plane configuration, or explicit follow-up protocol proposals until the canonical owners decide to promote them into `search_fabric.proto`.