--- title: Outbox Pattern Support for TiCDC feature_id: ticdc-outbox-protocol owner: Benjamin Meadowcroft status: draft target_release: Summer 2026 created: 2026-03-16 last_updated: 2026-03-25 --- # Scenarios This document describes operator and system workflows for the Outbox protocol. Scenario narratives are maintained here; see [PRD](./PRD.md#user--operator-scenarios) for indexed references. --- ## SCN-001 Emit Event from Outbox Table Actor: Application + TiCDC Description: Core outbox flow: a transactional outbox row is committed and emitted downstream. Preconditions: - A changefeed is running with `protocol=outbox-json`. - The outbox table is included in the changefeed. - Key and payload columns are configured. Steps: 1. The application writes a business row and a corresponding outbox row in the same transaction. 2. The transaction commits successfully. 3. TiCDC reads the committed outbox row. 4. TiCDC emits a downstream message using configured key and payload columns. Expected Result: - The committed outbox row is published with expected key and payload. - Emission reflects the transaction commit boundary. Validates: FR-001, FR-002, FR-003 --- ## SCN-002 Topic Dispatch Using Column Value Actor: Application + TiCDC Description: Validate topic routing when topic name comes from a column value. Preconditions: - A changefeed is running with `protocol=outbox-json`. - Dispatcher configuration uses `{column:}`. - Outbox rows include a valid topic column value. Steps: 1. The application inserts an outbox row with a topic-designating column. 2. The transaction commits and becomes visible to TiCDC. 3. TiCDC reads the topic column while preparing the event. 4. TiCDC emits to the derived topic. Expected Result: - Event is routed to the expected topic derived from the row value. Validates: FR-004 --- ## SCN-003 Custom Message Header Propagation Actor: Application + TiCDC Description: Verify metadata from outbox columns flows into message headers. Preconditions: - A changefeed is running with `protocol=outbox-json`. - Header mappings are configured. Steps: 1. The application writes an outbox row with payload data and header columns. 2. The transaction commits. 3. TiCDC applies configured header mappings. 4. TiCDC emits the message with headers. Expected Result: - Downstream messages include configured headers from outbox row columns. Validates: FR-005 --- ## SCN-004 High-Throughput Outbox Event Pipeline Actor: Data Platform Engineer Description: Validate sustained outbox emission under steady high-volume load. Preconditions: - A changefeed is running with `protocol=outbox-json`. - Downstream system is healthy. Steps: 1. Workload inserts a sustained stream of outbox transactions. 2. TiCDC reads committed outbox rows continuously. 3. TiCDC emits messages continuously. 4. Lag and throughput are observed. Expected Result: - Emission remains continuous without process interruption. - Operational lag stays within target bounds. Validates: FR-001, FR-002, FR-003 --- ## SCN-005 Multiple Partitions Receiving Events Actor: Data Platform Engineer Description: Validate partitioned fan-out behavior for outbox emissions. Preconditions: - A changefeed is running with `protocol=outbox-json`. - Target topic has multiple partitions. Steps: 1. Committed outbox rows include keys with varied values. 2. TiCDC emits messages for those rows. 3. The downstream system assigns partitions by key/topic behavior. Expected Result: - Events distribute across multiple partitions consistently with routing and key strategy. Validates: FR-001, FR-002, FR-003, FR-004 --- ## SCN-006 TiCDC Restart During Outbox Processing Actor: Operator Description: Validate recovery behavior after processor restart. Preconditions: - A changefeed is running with `protocol=outbox-json`. - Outbox transactions are being committed. Steps: 1. Operator restarts TiCDC while processing committed outbox rows. 2. TiCDC resumes from persisted changefeed state. 3. Operator validates resumed delivery. Expected Result: - No committed events are lost. - Delivery resumes from the correct checkpoint. Validates: FR-001, INV-003 --- ## SCN-007 Kafka Temporarily Unavailable Actor: Operator Description: Validate behavior when downstream availability is disrupted and restored. Preconditions: - A changefeed is running with `protocol=outbox-json`. - Kafka is temporarily unreachable during processing. Steps: 1. TiCDC emits outbox events successfully. 2. Downstream Kafka becomes unavailable. 3. Committed outbox rows accumulate at TiCDC boundary. 4. Kafka recovers and TiCDC resumes emission. Expected Result: - No silent discards. - Emission resumes after recovery. Validates: FR-001, INV-004 --- ## SCN-008 Incorrect Dispatcher or Invalid Topic Input Actor: Operator Description: Validate detection and recovery for routing misconfiguration and invalid topic input. Preconditions: - A dispatcher is configured for column-based routing. - Outbox rows may include invalid routing values. Steps: 1. Commit outbox rows with incorrect routing configuration and/or invalid topic values. 2. Observe TiCDC emission and changefeed diagnostics. 3. Correct dispatcher configuration. 4. Re-run representative outbox rows. Expected Result: - Invalid routing is observable. - Misrouted events are not silently accepted as correct. - Corrected configuration restores expected routing behavior. Validates: FR-004, INV-004 --- ## SCN-009 Migration from Existing CDC Pipeline to outbox-json Actor: Migration Engineer Description: Validate staged migration from an existing outbox CDC style to TiCDC outbox-json. Preconditions: - Existing outbox schema and consumer contracts are documented. - Downstream consumers accept comparison validation. Steps: 1. Map source contract to TiCDC `outbox-json` protocol mapping. 2. Stage protocol migration in controlled environment. 3. Emit representative outbox traffic and compare message semantics. 4. Complete controlled rollout with checkpoints and acceptance checks. Expected Result: - Event shape, routing, and header behavior remain compatible enough for safe downstream adoption. Validates: FR-001, FR-002, FR-003, FR-004, FR-005 --- ## SCN-010 Configure a New Outbox Changefeed Actor: Data Platform Engineer Description: Validate first-time setup of an outbox-enabled changefeed. Preconditions: - A Kafka downstream is reachable. - An outbox table schema and required field mappings are known. - The operator has permission to create or update a changefeed. Steps: 1. The operator configures a new changefeed with `protocol=outbox-json`. 2. The operator sets the key, payload, and optional header mappings for the outbox table. 3. The operator starts the changefeed and emits a representative test outbox row. 4. The operator validates the emitted event in the downstream system. Expected Result: - The changefeed starts successfully with the declared outbox mappings. - The first test event is emitted with the expected key, payload, and optional headers. Validates: FR-001, FR-002, FR-003, FR-005 --- ## SCN-011 Null Key Column Actor: Application + Operator Description: Validate behavior when a configured key column is null. Preconditions: - A changefeed is running with `protocol=outbox-json`. - Key column is configured and a null key case can be injected. Steps: 1. Commit an outbox row with `NULL` in the key column. 2. Observe TiCDC emission and downstream behavior. 3. Confirm validation or handling path. Expected Result: - Null key conditions are handled visibly and consistently. Validates: FR-001, FR-002 --- ## SCN-012 Oversized Payload Actor: Application + Operator Description: Validate oversized payload handling and observability. Preconditions: - A changefeed is running with `protocol=outbox-json`. - Payload column can be populated with large payload values. Steps: 1. Commit an outbox row with oversized payload data. 2. Observe emission, lag, and downstream handling. 3. Capture operational signals and failure surfaces. Expected Result: - Large payload behavior is observable: the operator can confirm whether delivery succeeded or failed. - No silent truncation or silent corruption. - If the Kafka sink is configured with a large-message handling mode (e.g., compression or claim-check), that mode's behavior applies consistently with other TiCDC protocols. Validates: FR-001, FR-003 --- ## SCN-013 Disable or Migrate Away from outbox-json Actor: Operator Description: Validate rollback or decommission behavior when an operator disables an outbox changefeed or migrates away from `outbox-json`. Preconditions: - An outbox-enabled changefeed is running and has emitted production-like traffic. - The operator has a documented rollback or replacement plan. - Downstream consumers can confirm whether expected events have been delivered before cutover. Steps: 1. The operator pauses or drains the active outbox changefeed according to the documented cutover procedure. 2. The operator confirms the downstream system has received the expected committed events before decommission. 3. The operator disables the `outbox-json` configuration or replaces it with the intended successor workflow. 4. The operator validates that no additional unexpected events are emitted after decommission. Expected Result: - The operator can execute a controlled decommission or migration workflow using documented steps. - No previously observed committed event is silently lost during the transition. - The final state of the changefeed is observable to the operator. Validates: FR-001, INV-003, INV-004 ---