# Embedded Mini-CDC Design for Emulator ## 1. Goal Build an embedded Mini-CDC runtime for emulator usage that serves as the TiDB/TiCDC/TiKV implementation of the emulator's internal CDC upstream adapter contract while keeping only the minimum production-relevant CDC path: - pull TiKV KV changes for selected tables, - decode KV into DML row callbacks, - fetch schema/DDL timeline and emit ordered DDL callbacks, - reuse existing TiCDC/TiDB/TiKV code paths. Positioning and implementation contract: 1. Mini-CDC is an **internal component inside emulator**, not an external standalone service, and it implements the `schema_aware` CDC upstream adapter profile owned by `docs/specs/core/emulator.md`. 2. Mini-CDC is implemented in **Golang** (same language/runtime family as emulator for direct module reuse). 3. Reuse-first rule: prefer directly calling validated TiDB/TiCDC modules (`logpuller`, `schemastore`, `mounter`, codec helpers) instead of re-implementing equivalent logic. 4. Language boundary is strict: components outside emulator domain are implemented in **Rust**. Non-goals: - HA, distributed scheduling, and standalone sorter/event-store, - standalone Mini-CDC service process independent from emulator, - global total ordering across all tables, - backward compatibility with older internal drafts, - dynamic table add/remove during runtime (v1 uses fixed table set from `StartConfig.TableIDs`). ## 2. Requirement Fit and Review Verdict Verdict: **feasible, production-viable, and still minimal**. Requirement mapping: - Minimal end-to-end CDC path: satisfied by `SubscriptionClient + SchemaStore + Mounter + callback adapter`. - Real protocol/codec reuse: satisfied by existing production modules: - `logservice/logpuller/subscription_client.go` - `logservice/schemastore/schema_store.go` - `pkg/common/event/mounter.go` - Structured DML/DDL callbacks: satisfied with a thin callback API layer. Reuse policy (normative): 1. CDC pull, schema timeline, and row decode hot paths SHOULD stay on reused TiCDC/TiDB implementations. 2. Mini-CDC-specific code should focus on adapter/fencing/lifecycle glue and must avoid replacing proven upstream logic unless required by explicit contract gaps. 3. Any unavoidable divergence from reused upstream behavior must be documented and covered by deterministic fail-fast rules. Key strengths: - High fidelity to TiCDC runtime behavior without re-implementing low-level protocol details. - Small operational surface and clear lifecycle boundaries. - Hot path can stay allocation-aware with chunk reuse and lazy column extraction. Inherited risks that must be explicit: - Schemastore persistence defaults are weak (`DisableWAL`, `NoSync`) and DDL write errors are not strongly propagated. - Schemastore warm restart cache is currently disabled (local DB directory cleanup). - Trigger-DDL fetch has error paths where snapshot handles are not closed before return. - App-context dependency wiring is panic-sensitive on missing/mismatched services. - App-context is a global singleton; concurrent Mini-CDC instances in one process will conflict. - Path-local ordering only; no global cross-table total DML order. - These limitations are inherited from reused TiCDC modules; Mini-CDC documents them and fails fast on detected anomalies but cannot fully eliminate them without upstream changes. ## 3. Source-Verified Hard Contracts ### 3.1 Subscription callback contract - `consumeKVEvents(raw, wake) bool` semantics are strict: - `false`: synchronous completion path, - `true`: asynchronous completion path, and this path is blocked until `Wake(path)` is issued. - References: - `logservice/logpuller/subscription_client.go:351-359` - `logservice/logpuller/region_event_handler.go:141-161` - `utils/dynstream/interfaces.go:95-99` - `utils/dynstream/interfaces.go:169-171` Mini-CDC rule: - Use synchronous mode only (`return false`). - Never call `wake` in synchronous mode. ### 3.2 Payload ownership and mutation - Raw KV bytes are borrowed slices and may alias reusable cache storage. - Decode path normalizes key in place (`RemoveKeyspacePrefix` mutates `raw.Key`). - `GetTableInfo` returns a shared `*TableInfo` pointer from versioned store. - Some `TableInfo` accessors return internal references (columns/maps/rowcodec metadata). - DDL events carry rich influenced-table metadata (`NeedDroppedTables`, `NeedAddedTables`, `UpdatedSchemas`, `TableNameChange`, `MultipleTableInfos`) that should not be projected away in Mini-CDC callback boundary. - Trigger-DDL build path actually fills these influenced-table fields in schemastore conversion logic; Mini-CDC should preserve the built `DDLEvent` object instead of re-projecting it. - References: - `logservice/logpuller/region_event_handler.go:265-273` - `logservice/logpuller/subscription_client.go:148-153` - `pkg/common/event/mounter.go:74-80` - `logservice/schemastore/multi_version.go:120-131` - `pkg/common/table_info.go:186-207` - `pkg/common/table_info.go:364-371` - `pkg/common/event/ddl_event.go:34-83` - `logservice/schemastore/persist_storage_ddl_handlers.go:1667-1747` - `logservice/schemastore/persist_storage_ddl_handlers.go:2513-2696` Mini-CDC rules: - Treat callback payload as borrowed unless explicitly copied. - Deep-copy only for retention beyond callback lifetime. - Use normalized key as canonical row key after decode. - Treat `RowChanged.TableInfo` as read-only; never mutate it or references returned by its accessors. - Treat DDL callback payload as read-only and callback-lifetime borrowed; if retained beyond callback, copy required fields. - Keep full DDL event semantics instead of flattening away influenced-table metadata. - Emulator integration must enforce the callback-lifetime borrow rule; long-term retention requires explicit copying. Ownership summary (callback boundary): | Field | Ownership | Copy rule | |---|---|---| | `RawKVEntry.Key/Value/OldValue` | borrowed, may alias cache | copy if retained beyond callback | | `RowChanged.PreRow/Row` | borrowed chunk view | copy selected columns if retained | | `RowChanged.RowKey` | event-owned copy | copy again if retained long-term | | `RowChanged.TableInfo` | shared read-only pointer | copy metadata if mutation/long retention needed | | `DDLOutput.Event` | read-only, callback-lifetime borrowed | copy required fields if retained | ### 3.3 Decode precondition - `DecodeToChunk` calls `DecodeRowKey` before table-prefix skip logic. - Feeding non-record-key spans can fail decode. - Subscription span keys should be memcomparable-encoded table record range keys. - References: - `pkg/common/event/mounter.go:74-80` - `pkg/common/span_op.go:42-54` - `pkg/common/span_op.go:237-240` - `logservice/schemastore/ddl_job_fetcher.go:270-289` - `logservice/eventstore/event_store.go:1408-1411` Mini-CDC rule: - Build subscription spans with `common.TableIDToComparableSpan(keyspaceID, tableID)`. - If constructing manually, apply `common.ToComparableKey(...)` to both start/end keys returned by `common.GetKeyspaceTableRange(...)`. ### 3.4 Schema lookup blocking behavior - `GetTableInfo` and `RegisterTable` both use an internal non-cancelable wait loop for resolved-ts readiness. - References: - `logservice/schemastore/schema_store.go:207-217` - `logservice/schemastore/schema_store.go:326-334` - `logservice/schemastore/schema_store.go:351-363` Mini-CDC rule: - Pre-gate with `GetTableDDLEventState` before invoking these APIs. - Do not rely on timeout-wrapper goroutines around the blocking call itself. ### 3.5 Trigger DDL API constraints - Trigger DDL fetch window is `(start, end]`. - `limit > 0` is mandatory. - Cursor semantics: - full batch: `next = events[limit-1].FinishedTs` - partial batch: `next` follows resolved-ts snapshot rules. - References: - `logservice/schemastore/schema_store.go:407-433` - `logservice/schemastore/persist_storage.go:488-547` ### 3.6 Inherited DDL durability limitation - Persistent storage runs with `DisableWAL: true`; DDL writes use `NoSync`. - DDL write failures are weakly surfaced to resolved-ts progression path. - Practical implication: DDL persistence can be lossy after crashes and some write failures may not surface; Mini-CDC must treat any detected inconsistency as fatal and avoid “best-effort continue” behavior. - References: - `logservice/schemastore/persist_storage.go:116-120` - `logservice/schemastore/persist_storage.go:722-765` - `logservice/schemastore/disk_format.go:518-519` - `logservice/schemastore/schema_store.go:150-160` Mini-CDC rule: - Fail fast on anomaly signals; do not continue silently after DDL inconsistency. ### 3.7 Strict app-context dependency constraints - `GetService` is a direct type assertion and may panic. - Runtime dependencies are: - `NewSubscriptionClient`: requires `RegionCache` and `DefaultPDClock`. - `schemastore.New`: requires `DefaultPDClock`. - `RegisterKeyspace`: requires `KeyspaceManager` and `SubscriptionClient`. - References: - `pkg/common/context/app_context.go:61-64` - `logservice/logpuller/subscription_client.go:239-240` - `logservice/schemastore/schema_store.go:242` - `logservice/schemastore/schema_store.go:451` - `logservice/schemastore/schema_store.go:488` - `pkg/keyspace/keyspace_manager.go:95-97` ### 3.8 Keyspace registration invariant - Any schema API call (`GetTableDDLEventState`, `RegisterTable`, `GetTableInfo`, `FetchTable*`) before explicit `RegisterKeyspace` success is forbidden in Mini-CDC. - Reason: current schemastore has a lazy fallback path that may auto-call `RegisterKeyspace(context.Background(), ...)`, which is not acceptable for deterministic lifecycle control. - References: - `logservice/schemastore/schema_store.go:254-265` - `logservice/schemastore/schema_store.go:366-421` ### 3.9 Single-active-instance process invariant - App-context service registry is process-global singleton state. - `SetService` writes into shared global map keys with no Mini-CDC namespace isolation. - References: - `pkg/common/context/app_context.go:20-24` - `pkg/common/context/app_context.go:49-64` Mini-CDC rule: - Run at most one active Mini-CDC instance per process. ### 3.10 RegisterKeyspace context lifetime contract - `RegisterKeyspace(ctx, ...)` binds keyspace workers and the resolved-ts updater loop to the provided `ctx`. - If that context is canceled early, resolved-ts progression can stop, and later schema calls that wait on resolved-ts can block indefinitely. - References: - `logservice/schemastore/schema_store.go:499-517` - `logservice/schemastore/schema_store.go:207-217` Mini-CDC rule: - Always call `RegisterKeyspace` with the runtime root context (`runCtx`) that lives until shutdown. - Never use a short timeout context for `RegisterKeyspace`. ### 3.11 Unsubscribe and path-removal delivery boundary - Unsubscribe is asynchronous; actual path cleanup occurs later in `onTableDrained`. - Dynamic-stream `RemovePath` semantics explicitly allow dropping already queued events for that path. - References: - `logservice/logpuller/subscription_client.go:383-386` - `logservice/logpuller/subscription_client.go:489-501` - `utils/dynstream/interfaces.go:184-185` Mini-CDC rule: - Keep v1 fixed-table-set design (no runtime table add/remove). - During shutdown, do not rely on post-remove queue drain guarantees. ## 4. Minimal Architecture ```text +--------------------------------------+ | Mini-CDC Runtime | | | | +-------------------------------+ | TiKV CDC stream ->|->| SubscriptionClient |---+--> resolvedTsByTable | | (one record-key span/table) | | | +---------------+---------------+ | | | raw KV batch | | v | | +-------------------------------+ | | | DML Processor | | | | schema gate + mounter decode |---+--> OnRowChanged | +-------------------------------+ | | | | +-------------------------------+ | | | DDL Poller | | | | FetchTriggerDDLEvents + fence |---+--> OnDDL | +-------------------------------+ | +-------------------+------------------+ | v +----------------------+ | SchemaStore | +----------------------+ ``` Component boundary: 1. Mini-CDC runs in the same process as emulator and satisfies the generic CDC upstream adapter contract defined by the emulator spec before lowering into emulator core state paths. 2. Mini-CDC does not expose an independent gRPC service surface. 3. When emulator runs in standalone mode or TiDB-embedded mode, Mini-CDC remains an internal component with unchanged contracts. 4. Query/ingest/control/stateful/background production components remain outside Mini-CDC scope and are implemented in Rust. ## 5. Ordering and Consistency Model ### 5.1 DML ordering scope - Ordering is guaranteed per dynamic-stream path (table subscription), not globally across all tables. - References: - `utils/dynstream/interfaces.go:30-33` - `utils/dynstream/interfaces.go:151-154` Exposed guarantee: - Same-path (table path) callback processing is sequential. - Timestamp priority is best-effort, not a strict per-table commit-ts monotonic guarantee. - No cross-table total DML ordering guarantee. ### 5.2 DML versus DDL ordering - Event-service reference behavior uses schema at `commitTs-1`, then merges DDL/DML by timestamp rules. - References: - `pkg/eventservice/event_scanner.go:212-223` - `pkg/eventservice/event_scanner.go:509-537` - `logservice/schemastore/types.go:135-136` Mini-CDC ordering rules: - Emit a DDL callback only when `ddl.FinishedTs <= ddlFenceTs`. - `ddlFenceTs = min(resolvedTsByTable[t] for all configured tables t)`. - Same-ts rule: for any timestamp `ts`, emit all DML callbacks at `commitTs == ts` before any DDL callback with `FinishedTs == ts`. ### 5.3 Fixed-table-set DDL fence (v1) - Mini-CDC v1 tracks a fixed table set from `StartConfig.TableIDs`. - DDL polling emits nothing until every configured table has initial resolved progress. - Runtime does not support table add/remove in v1. Rationale: - Keeps runtime minimal and deterministic. - Avoids unsubscribe/drain lifecycle complexity in hot correctness logic. ### 5.4 Checkpoint model - Emit checkpoint only after resolved progress changes. - `GlobalResolvedTs = min(resolvedTsByTable[t] for all configured tables t)`. - `TableResolvedTs` is a snapshot copy (do not expose internal mutable map). - Coalesce checkpoint callbacks with a small minimum interval (for example 25-100ms) and always emit the latest monotonic snapshot to cap hot-path map-copy/callback overhead. - Checkpoint callbacks are monotonic; regressions are fail-fast anomalies. - Reference (coalescing precedent in eventservice): - `pkg/eventservice/event_broker.go:47` - `pkg/eventservice/event_broker.go:333-364` ### 5.5 Search Fabric Emulator Normalization Bridge Mini-CDC is per-table-path and callback-oriented. The embedded emulator ingest contract is collection-oriented and requires canonical `(source_partition, source_epoch, source_seq)` plus collection-level progress semantics. The bridge between them is normative: 1. `source_partition` derivation: - one Mini-CDC table path maps to exactly one canonical `source_partition`. - when `KeyspaceID` is configured, derive `source_partition = "keyspace:/table:"`. - otherwise derive `source_partition = "table:"`. - if the source table is a physical partition, use the physical partition/table identifier in `source_partition`; parent logical table identity remains schema metadata, not stream identity. 2. `source_epoch` derivation: - initialize `source_epoch=1` when the Mini-CDC instance finishes startup and establishes its fixed subscription set. - increment `source_epoch` only when the adapter restarts, re-subscribes, or otherwise resets per-partition sequence state. - ordinary DDL, resolved-ts advancement, and checkpoint emission MUST NOT bump `source_epoch`. 3. `source_seq` derivation: - maintain one strictly increasing 1-based sequence counter per `source_partition`. - callback order is input observation order only; it is not, by itself, the final canonical ingest order when callbacks arrive non-monotonically by `commit_ts`. - adapter MUST maintain a small per-partition reorder buffer keyed by `(commit_ts, raw_callback_ordinal)`. - reorder-buffer capacity MUST be explicitly bounded (`max_buffered_unsealed_rows` and/or `max_buffered_unsealed_bytes` under runtime policy). - a buffered row becomes sealed and eligible for flush only when one of the following happens: - checkpoint seal: latest `OnCheckpoint.TableResolvedTs` for that partition is `>= row.commit_ts`; - DDL seal: `OnDDL(ts)` arrives for the same table-path, and the adapter first flushes every buffered row with `commit_ts <= ts` before applying the DDL. This relies on the Mini-CDC same-ts rule that no later DML callback for `commit_ts <= ts` may arrive after `OnDDL(ts)` for that table-path. - no other batch-cut heuristic is allowed to assign `source_seq` early; timer-based flush, byte-budget flush, or callback-count flush may emit only already sealed rows. - if the next callback would exceed the reorder-buffer bound and no buffered row is newly sealable, the adapter MAY apply only a short bounded backpressure window while waiting for checkpoint/DDL seal; it MUST NOT emit unsealed rows, assign speculative `source_seq`, or silently drop buffered entries. - if seal still does not arrive before the bounded backpressure window expires, the adapter MUST fail fast and stop the subscription/runtime with reorder-buffer overflow. - timer-based flush, byte-budget flush, and callback-count flush remain restricted to already sealed rows even under pressure; unsealed rows stay buffered until sealed or terminal overflow. - because the upstream consume bridge has no error return, overflow handling is terminal failure after the bounded backpressure window rather than indefinite callback-time blocking. - assign `source_seq` in the final emitted canonical order for that `source_partition`, not in raw callback arrival order. - for rows sharing the same `commit_ts`, preserve raw callback order as the stable intra-timestamp tie-break before assigning `source_seq`. - canonical `MutationBatch` is single-partition by contract, so rows from different `source_partition` values MUST be split into different admitted batches. 4. Row-event normalization: - `OnRowChanged` preserves source `commit_ts` and lowers one row event to one canonical `ROW_MUTATION`. - `schemaTs = commitTs - 1` lookup result determines the schema generation lineage used for canonical `schema_generation_token`. - if a DDL fence would change `schema_generation_token`, the adapter MUST cut the current batch at that boundary and start a new batch under the new generation token. 5. DDL normalization: - `OnDDL` is a control-plane event, not a stream-epoch reset. - same-ts rule still applies: all DML at `ts` are normalized and admitted before the DDL at `ts` advances schema generation. - the canonical control-plane bridge for one admitted DDL event is one complete `PutCollectionSchemaRequest`, not just a fence tuple: - `meta.idempotency_key` MUST be non-empty and replay-stable for the DDL callback identity, - `meta.client_id` MUST identify the TiDB DDL adapter instance, - `meta.client_kind=CLIENT_KIND_TIDB_DDL_ADAPTER`, - `scope.collection=`, - `scope.ddl_commit_ts=DDLEvent.FinishedTs`, - `schema_version` MUST come from authoritative DDL/catalog lineage, - `fence.expected_prev_schema_version`, `fence.schema_generation_token`, and `fence.enforce_monotonic_apply=true` MUST come from the resolved pre-DDL lineage snapshot, - `definition` MUST be the full canonical post-DDL schema snapshot for the affected collection, - `update_policy` MUST be explicit and MUST NOT be `SCHEMA_UPDATE_POLICY_UNSPECIFIED`, - if the DDL creates the first schema generation for a collection that already has catalog identity but no prior schema lineage, the request MUST use the full canonical zero-state tuple (`schema_version=1`, `fence.expected_prev_schema_version=0`, `fence.schema_generation_token="__initial__"`, `update_policy=SCHEMA_UPDATE_POLICY_INITIAL_CREATION`, `fence.enforce_monotonic_apply=true`), - `update_policy` MUST be explicit and classified by canonical schema-diff rules (`INITIAL_CREATION` / `ADDITIVE` / rebuild classes as applicable). - replayed DDL callbacks MUST materialize the same canonical request fields above so DDL replay stays aligned with the canonical control-plane contract. - after DDL apply, subsequent row batches for affected tables MUST use the new schema-generation token and keep the same `source_partition`. 6. Checkpoint/progress normalization: - `OnCheckpoint.TableResolvedTs` updates per-partition resolved progress only; it does not write `safe_ts` directly. - for each collection/modality fed by Mini-CDC, emulator derives collection-level `applied_ts` as the minimum resolved ts across all mapped source partitions for that collection. - emulator then recomputes `safe_ts = min(enabled modality applied_ts, including metadata_aux only when that optional modality is present)` using the same readiness rules as the canonical emulator contract. - checkpoint regressions, partition-progress loss, reorder-buffer overflow/backpressure-timeout, or direct `safe_ts` override through the Mini-CDC bridge are fail-fast anomalies. ## 6. DML Pipeline For each `RawKVEntry`: 1. Determine `schemaTs = commitTs - 1`. 2. Use per-table `schemaReadyTs` cache: - if `schemaTs <= schemaReadyTs`, skip readiness polling, - otherwise poll `GetTableDDLEventState` with bounded backoff until `ResolvedTs >= schemaTs`, then update `schemaReadyTs`. 3. Read schema via `GetTableInfo(keyspaceMeta, tableID, schemaTs)`. 4. Decode using `Mounter.DecodeToChunk`. 5. Iterate rows with `GetNextRow` and invoke `OnRowChanged`. 6. Advance table resolved-ts only after all DML callbacks up to that progress point are completed successfully. References: - `pkg/eventservice/event_scanner.go:212-223` - `pkg/eventservice/event_scanner.go:509-537` - `pkg/common/event/mounter.go:72-153` - `pkg/common/event/dml_event.go:521-607` - `pkg/common/event/dml_event.go:665-716` Additional invariants: - Update semantics require old value (`ExtraOp_ReadOldValue` path). - Missing old value for update-classified traffic is treated as fail-fast anomaly. - Start boundary in COMMIT path is exclusive in practice: `commitTs <= startTs` events are dropped. - Ordering anomalies at/behind resolved-ts boundary are process-fatal in current logpuller (`log.Fatal` paths). - Since `consumeKVEvents` has no error return, callback failure must trigger terminal runtime shutdown. - References: - `logservice/logpuller/region_request_worker.go:431` - `pkg/common/kv_entry.go:68-76` - `logservice/logpuller/region_event_handler.go:321-324` - `logservice/logpuller/region_event_handler.go:292-300` - `logservice/logpuller/region_event_handler.go:329-337` ## 7. DDL Pipeline Polling loop: 1. `events, next, err := FetchTableTriggerDDLEvents(..., cursor, limit)`. 2. Validate `limit > 0` at config load; runtime should never pass invalid limit. 3. Compute `ddlFenceTs = min(resolvedTsByTable over configured fixed table set)`. 4. Emit in order while `event.FinishedTs <= ddlFenceTs`. - Emit fetched `commonEvent.DDLEvent` as read-only callback payload; avoid lossy field projection. - If callback API takes `*DDLEvent`, pass `&events[i]` with index iteration (`for i := range events`) to avoid range-variable-address bugs. 5. Cursor update rules: - per emitted event: `cursor = event.FinishedTs`, - only when all fetched events are consumed and `next <= ddlFenceTs`, allow `cursor = next`, - otherwise keep `cursor` at last emitted value (no skip). 6. If no event is emitted because of the fence, wait for new table-resolved progress (or sleep for a bounded poll interval) before the next fetch. This preserves no-skip semantics when fetched batches cross the local DML fence. ## 8. Runtime Lifecycle ### 8.1 Startup order (strict dependency-safe) Start config must pass hard validation before wiring: - `DDLPollLimit > 0` (`FetchTableTriggerDDLEvents` uses `limit` as slice cap/index). - `TableIDs` must be non-empty, unique, and each `> 0`. - In next-gen mode, `KeyspaceID != 0` for keyspace table-range construction. - References: - `logservice/schemastore/schema_store.go:425-427` - `logservice/schemastore/persist_storage.go:500` - `logservice/logpuller/subscription_client.go:360-362` - `pkg/common/span_op.go:118-121` 1. Create PD client. 2. Create PD API client and set `appctx.PDAPIClient`. 3. Create region cache and set `appctx.RegionCache`. 4. Create/start PD clock and set `appctx.DefaultPDClock`. 5. Align global server security config with Mini-CDC credentials. 6. Create keyspace manager and set `appctx.KeyspaceManager`. 7. Create lock resolver (non-nil). 8. Create schema store and set `appctx.SchemaStore`. 9. Create subscription client and set `appctx.SubscriptionClient`. 10. Start `SubscriptionClient.Run(runCtx)`. 11. Explicitly call `RegisterKeyspace(runCtx, keyspaceMeta)` and require success before any schema API call. 12. For each tracked table: - wait `GetTableDDLEventState.ResolvedTs >= startTs`, - call `RegisterTable`, - immediately probe `GetTableInfo(..., tableID, startTs)` and fail startup on error (including "no version found" / deleted table). 13. Build table spans via `common.TableIDToComparableSpan(...)` and subscribe. 14. Wait all configured tables have initial resolved progress, then enable DDL polling. 15. Start DML and DDL loops. Critical notes: - `pd == nil` puts subscription client in test-mode no-op and is invalid in real runtime (`logservice/logpuller/subscription_client.go:443-445`). - `RegisterTable` with `startTs < gcTs` is a hard startup error (`logservice/schemastore/persist_storage.go:340-345`). - `RegisterTable` may still succeed even when table info history is effectively empty at `startTs`; the startup probe via `GetTableInfo` catches this early (`logservice/schemastore/persist_storage.go:549-570`, `logservice/schemastore/multi_version.go:103-130`). - Single process should run only one active Mini-CDC instance because `appctx` is global singleton state. - Do not ignore `Run` goroutine errors. - `schemastore.New` depends on `DefaultPDClock` service (`logservice/schemastore/schema_store.go:242`). - `RegisterKeyspace` depends on both `KeyspaceManager` and `SubscriptionClient` services (`logservice/schemastore/schema_store.go:451,488`). - Do not rely on lazy `RegisterKeyspace(context.Background(), ...)` fallback (`logservice/schemastore/schema_store.go:262-265`). - `RegisterKeyspace` must use long-lived `runCtx`; a short-lived context can silently stop schema progression (`logservice/schemastore/schema_store.go:505-517`). ### 8.2 Shutdown order 1. Cancel root context. 2. Stop DML/DDL loops and wait workers. 3. Wait `SubscriptionClient.Run` return. 4. Close subscription client. 5. Close remaining resources (`SchemaStore`, keyspace manager, region cache, PD clock, PD client) in host-runtime-defined order. References: - `logservice/logpuller/subscription_client.go:466-471` - `logservice/schemastore/schema_store.go:302-313` - `logservice/schemastore/ddl_job_fetcher.go:85-103` - `logservice/schemastore/schema_store.go:505-517` - `pkg/keyspace/keyspace_manager.go:132-141` Notes: - Hard contract is only `cancel run ctx -> wait Run return -> Close` for subscription client. - `SchemaStore.Close` currently closes keyspace data storage; it is not a full explicit stop API for DDL fetcher workers. - `ddlJobFetcher` has no dedicated `Close`/`Stop` API in current implementation, so context cancellation is the stop mechanism. ## 9. Failure Policy Mini-CDC is fail-fast with single owner error propagation. Stop runtime on: - callback error (`OnRowChanged` / `OnDDL` / `OnCheckpoint`), - deterministic GC-boundary trigger-DDL fetch errors (for example `startTs < gcTs`), - repeated trigger-DDL fetch failures (non-deterministic/transient class), - cursor regression or ordering anomaly, - logpuller process-fatal ordering anomalies (`log.Fatal` paths), - schema-readiness deadline violation, - initial table-progress deadline violation before enabling DDL polling, - reorder-buffer overflow or bounded-backpressure timeout while waiting for seal of unsealed rows, - startup config validation failure (`KeyspaceID`/`TableIDs`/`DDLPollLimit`), - startup table-schema probe failure (`GetTableInfo(..., startTs)`), - startup dependency/miswiring error. Reason: infra CDC path must prefer explicit stop over silent divergence. Inherited limitation to document explicitly: - Some DDL persistence anomalies can be silent because write errors are weakly propagated (for example ignored return values in the write/apply path). Fail-fast applies to detected anomalies, but detection is not complete. - Trigger-DDL fetch has error branches that can return without closing snapshot handles (`logservice/schemastore/persist_storage.go:523,527,535,541`), so repeated retry-on-error loops can accumulate resource pressure. Prefer immediate-stop on deterministic correctness errors. - These limitations are not fully removable within Mini-CDC without changing schemastore internals; the design therefore treats them as explicit operational risks. ## 10. Performance and Allocation Rules Hot-path rules: - Synchronous consume callback only (`return false`). - Synchronous callback must avoid network I/O, disk I/O, and unbounded waits; the only allowed wait is the bounded reorder-buffer backpressure window while waiting for a seal, after which runtime must fail fast. - Synchronous callback must be CPU-bound and non-blocking (no network I/O, disk I/O, or long waits). - Reuse per-worker chunk and decoder state; avoid per-row object churn. - Avoid eager full-row materialization; extract only requested columns lazily. - Keep payload borrowed unless retention is required. Known hotspots in current reused code: - v2 decoder allocation per decode call: `pkg/common/event/chunk.go:68-78` - v1 `colID2CutPos` map rebuild per decode: `pkg/common/event/chunk.go:88-93` - key copy per decoded physical row: `pkg/common/event/dml_event.go:595-599` Practical optimization boundary: - Cache decoder/mapping by `(tableID, tableInfoVersion)` per worker. - Preserve ownership safety; do not remove copy boundaries without redesign. ## 11. Callback API Sketch ```go type StartConfig struct { KeyspaceID uint32 StartTs uint64 // Physical table IDs only (include partition IDs for partitioned tables). // Must be non-empty, unique, and each > 0. TableIDs []int64 // Must be > 0. DDLPollLimit int } type RowChanged struct { TableID int64 CommitTs uint64 RowType common.RowType PreRow chunk.Row // borrowed view, callback-lifetime only Row chunk.Row // borrowed view, callback-lifetime only RowKey []byte // event-owned copy; copy again if retained long-term TableInfo *common.TableInfo // shared read-only pointer; copy selected metadata if mutation/long retention is needed } type DDLOutput struct { // Read-only pointer with full DDL semantics. // Callback-lifetime borrowed object; copy required fields if retained. // Emit via index-based address (&events[i]) if sourced from []DDLEvent. // (NeedDroppedTables, NeedAddedTables, UpdatedSchemas, TableNameChange, MultipleTableInfos, etc.). Event *commonEvent.DDLEvent } type Checkpoint struct { GlobalResolvedTs uint64 TableResolvedTs map[int64]uint64 } type Callback interface { OnRowChanged(ctx context.Context, ev RowChanged) error OnDDL(ctx context.Context, ev DDLOutput) error OnCheckpoint(ctx context.Context, cp Checkpoint) error } ``` ## 12. Validation Plan Unit tests: - fixed-table-set fence correctness (`min(resolvedTsByTable)`), - cursor no-skip behavior at fence boundary, - startup config validation (`DDLPollLimit`, `TableIDs`, next-gen `KeyspaceID`), - per-table-path -> canonical `source_partition` / `source_seq` normalization, - reorder-buffer seal behavior (`checkpoint` / `OnDDL`) and stable intra-`commit_ts` tie-break, - bounded backpressure on full unsealed reorder buffer and fail-fast overflow when seal does not arrive in time, - no early emission of unsealed rows under timer/byte-budget/count pressure, - checkpoint -> collection `applied_ts` / `safe_ts` derivation bridge, - startup table-schema probe behavior (`GetTableInfo(..., startTs)` success/error), - startup/readiness gate for `RegisterTable` and `GetTableInfo`, - callback-side `TableInfo` read-only ownership contract, - DDL callback metadata fidelity for multi-table / rename / schema-move DDLs (no lossy flattening), - DDL callback pointer/lifetime ownership contract (read-only, callback-lifetime borrow, copy-on-retain), - update old-value invariant behavior, - start-boundary drop behavior (`commitTs <= startTs`), - callback fatal-stop behavior in synchronous consume bridge, - checkpoint monotonicity. Integration tests (real TiDB/TiKV/TiCDC components): - same-ts DML/DDL ordering behavior, - GC lower-bound startup failure handling, - callback fail-fast and shutdown drain behavior, - partitioned table coverage with physical table IDs. ## 13. Deferred Improvements (Out of Current Minimal Scope) - Optional durable schemastore mode (WAL/fsync controls). - Better error propagation from DDL persistence path. - Optional cross-table sorter when global order becomes a hard requirement. This design intentionally keeps the runtime small while staying aligned with current TiCDC source semantics and production-grade correctness expectations.