# Merged Engine Maintainer Guide

## Scope and assumptions
- This document explains the non-obvious design and invariants of merged_engine.
- It assumes familiarity with the storage-node Raft log structure and rfengine concepts.
- It focuses on how we merge WAL from multiple stores into one synthetic store.
- It avoids restating trivial config fields and struct definitions.

## Purpose and mental model
- Merged_engine is a local, synthetic store used by replication_worker.
- It replays upstream Raft logs into a single rfengine+kvengine instance.
- Each region in the merged store has exactly one peer whose ID equals the
  region ID and whose store ID equals `merged_store_id`.
- It is not a Raft participant; it is a deterministic replay engine.
- Commit decisions are inferred by observing the same log index from multiple
  upstream stores (quorum heuristic).

## High-level architecture
- Rfengine stores the merged Raft log and region state.
- Kvengine stores the applied data (the snapshot visible to CDC clients).
- A manifest tracks store progress, keyspace states, and uncommitted entries.
- Preprocessors translate upstream Raft entries into merged-store semantics.
- Appliers reuse rfstore apply logic to update kvengine.

## Key data structures and why they exist
### StoreProgress
- Tracks `(epoch, offset)` per upstream store to ensure WAL is applied in order.
- Enforces strict monotonicity when ingesting WAL chunks.

### RegionProgress
- Tracks per-region log entries, commit index, and synced index.
- `entries` contains a `RaftLogOpWithCounter` per log index.
- The counter increments when the same log index is seen from multiple stores.
- When the counter reaches quorum, `commit_index` can advance.
- `synced_index` is the last log index actually applied to kvengine.

### TruncatedIndex
- Records the upstream truncated index plus the commit index that justified it.
- Prevents unsafe truncation before merged_engine has applied enough data.
- Required because we cannot derive truncation safety from kvengine alone when
  shards are not loaded.

### Manifest
- Persists store progress, keyspace states, and uncommitted entries.
- Stores `synced_target_ts` so WAL target selection can continue correctly.
- Uses a checksum to guard against partial writes.
- `uncommitted_entries` allows recovery of in-flight logs after restart.

### Replay-local bootstrap state
- `InstallableRegionBootstrap` caches live-bootstrap state for ordinary
  first-seen learner peers that are still moving into `target_stores`; those
  bootstraps may be installed directly from origin WAL.
- Any collected bootstrap whose `shard_meta.parent.is_some()` is cleared
  instead of cached. Replay does not install that origin payload directly.
- When replay installs a cached live bootstrap for a keyspace that is already
  active in kvengine, it also reloads that shard into kvengine immediately so
  the same sync round can continue apply instead of treating the region as
  outside the loaded keyspace set.

### newly_bootstrapped_regions
- `MergedEngine` maintains a `Vec<u64>` that accumulates region IDs each time
  `install_installable_region_bootstrap` succeeds.
- `take_newly_bootstrapped_regions` drains and returns the vec via `mem::take`.
- Callers (replication_worker) drain this after each successful `sync_merged`
  to immediately report the regions to rep-pd, rather than waiting for the next
  periodic `report_regions_loop` iteration (which can be minutes away).
- On `sync_merged` error, the vec is NOT drained; IDs accumulate and are
  reported on the next successful call. This is safe because bootstrap install
  writes are applied in-memory (via `ctx.raft.apply`) before `Ok(true)` is
  returned, so the region state is readable regardless of whether the overall
  `sync_merged` ultimately persisted to disk.

### Manifest persistence timing
- `update_wal` mutates progress in memory but does not persist the manifest.
- The manifest is persisted at the end of `sync_merged` after apply completes.
- A crash before `sync_merged` simply discards in-memory WAL progress and
  triggers a fresh fetch on restart, which is safe.

### Preprocessor
- Rewrites Raft entries so region metadata matches the merged store.
- Tracks pending merge state and other apply-time metadata.
- Decrypts per-shard encryption keys so the applier can read old values.

## WAL ingestion (update_wal)
### Ordering and validation
- `update_wal` compares incoming `(epoch, start_off)` with stored progress.
- A mismatch is treated as fatal because it implies WAL reordering.
- The method updates store progress only after all batches are ingested.
- `update_wal` decodes and buffers all write batches before mutating progress so
  decode errors (and injected failpoints) do not leave a partially-updated
  in-memory state.

### Failpoints (test-only)
- With the `failpoints` feature enabled, `merged_engine::update_wal_err` injects
  an error after WAL decoding and before applying, to exercise caller retry and
  atomicity logic.

### Per-batch processing
- WAL chunks are decoded into rfengine `WriteBatch` instances.
- For each region in the batch:
  - If the region is tombstone or uninitialized, it is skipped.
  - A new `RegionProgress` is created when the region is first seen.
  - The log entry is inserted or updated in the region's `entries` map.
  - The entry counter may advance the region commit index.

### Why we track entry counters
- The merged store has no Raft quorum of its own.
- We approximate commit by observing the same log index from multiple stores.
- The counter provides a cheap quorum signal without building a new consensus.

### Truncated index handling
- Upstream truncate indices are captured when available in a write batch.
- Truncated indices are ignored if the peer was restored from snapshot.
- Missing truncated-state metadata in a live bootstrap batch falls back to
  `RAFT_INIT_LOG_INDEX`; current rfengine batches do not guarantee an explicit
  truncated state for newly observed peers.
- The commit index from the same store is recorded to validate truncation.

## WAL rotation
- `rotate_wal` increments the epoch only when current progress exactly matches
  the rotation boundary.
- This preserves strict ordering across epochs and avoids gaps.

## Sync pipeline (sync_merged)
### Overview
- `sync_merged` applies committed entries from updated regions.
- It is the only path that advances `synced_index` and persists progress.
- It batches Raft state writes to bound latency and memory.

### Region processing loop
- Updated regions are placed in a queue and processed iteratively.
- Regions may be postponed when merge dependencies are not ready.
- Postponed regions are re-queued to the next sync cycle.

### Preprocessing and admin commands
- Each entry is rewritten by `update_entry` to match merged-store metadata.
- `prepare_merge` and `commit_merge` entries are rewritten with single-peer
  region meta.
- `commit_merge` is postponed until the source region's merge state is ready.
- When replaying a parent split, rfstore preprocess may detect that a child
  peer already has newer local metadata. In that case it records the child
  shard ID in `PendingSplit`, and kvengine replay skips materializing that
  stale child shard.
- Pending merge states are tracked across regions to enforce ordering.
- Commit-merge source shard destruction is scheduled only when preprocessing
  emits `ApplyMsg::PrepareCommitMerge` for the current entry. If preprocessing
  skips the admin request (e.g. epoch mismatch), no destruction is scheduled.
- When `commit_merge` is accepted, merged_engine emits
  `RegionChangeEvent::Destroy` for the source region immediately as a logical
  topology change. Physical shard destruction may still be delayed by
  dependents.

### Applying entries to kvengine
- The preprocessor builds a `PreprocessContext` and generates apply messages.
- `sync_merged` uses `PreprocessScene::MergedEngine { .. }`, so merged_engine
  still preprocesses committed `restore_shard` entries while native BR
  keyspace restore rejects them in its `PreprocessScene::Restore`.
- Each region has an `Applier` that uses rfstore logic in replication mode.
- The replay kvengine always uses the Crossbeam MemTable because it must accept
  every large entry that an online Crossbeam-enabled source cluster accepted.
- Apply is skipped when the shard is not loaded (region outside keyspace set).
- The applier may pause; prepared messages are buffered and replayed later.

### Why we buffer prepared messages
- Some apply flows require interleaving region-level messages (e.g. merges).
- The router delivers these messages asynchronously.
- Buffering avoids deadlocks when another region's messages arrive first.
- Shared rfstore prepared-message types now also carry txn-file
  `apply_generation`; merged_engine threads that field through unchanged even
  though its appliers are currently long-lived, which keeps replay/test paths
  aligned with online rfstore semantics.

### Persisting Raft state
- Each region's raft state is updated as entries are processed.
- Batched `WriteBatch` objects are written to rfengine once size thresholds
  are exceeded.

## Progress updates and truncation
### Updating in-memory progress
- After syncing, entries at or below `synced_index` are dropped from memory.
- Uncommitted entries are retained for the next sync round.

### Truncation rules
- Truncation is allowed only when `synced_index` has passed the commit index
  that justified the upstream truncation.
- Regions with dependents are not truncated to avoid breaking parent recovery.
- If truncation is skipped because dependents still exist, the region is
  re-queued so truncation is retried in later `sync_merged` rounds even
  without new WAL for that region.
- Truncation respects the persisted log index and never truncates beyond it.

### Why dependents block truncation
- Parent regions can depend on child logs during merges or split recovery.
- Removing logs too early can break merge replay or recovery correctness.

## Region destruction
- Tombstone regions are removed from raft state and kvengine shards.
- If a region still has dependents, destruction is delayed.
- A delayed region is destroyed once its dependents are removed.
- Destroyed regions are marked as `TRUNCATE_ALL_INDEX` to prevent reuse.

### Admin-only unsafe recover clear
- `unsafe_recover.rs` contains the admin-only clear path used by
  replication_worker `POST /cdc/unsafe_recover/clear`.
- `unsafe_recover_check_clear_regions` is stricter than ordinary
  `destroy_regions()`: every dependent of every requested region must also be
  in the explicit request set, otherwise the whole request is rejected before
  any local state is mutated.
- `unsafe_recover_execute_clear_regions` intentionally copies the normal
  destroy body instead of changing `destroy_regions()`. This keeps the safe
  replay path unchanged while allowing admin clear to synchronously remove the
  full requested set.
- Admin clear destroys the explicit request one region at a time in the
  caller-visible order instead of reordering it dependents-first. If the
  process does fail partway through that rare admin flow, restart rebuilds the
  dependency graph from replay/bootstrap state before apply resumes.
- When a cleared child removes the last dependency edge from a parent that is
  already tombstoned in `delay_destroy_regions`, the parent is immediately
  pulled into the same destroy pass. This mirrors normal dependency-unblock
  behavior and avoids delayed tombstone parents getting stuck forever.
- Admin clear drops the entire `region_progresses` entry instead of leaving a
  tombstoned replay progress behind. This removes stale in-memory log state and
  manifest `uncommitted_entries`, and lets later WAL recreate the region if the
  same region ID moves back into the included store set.
- Admin clear still truncates local rfengine state to `TRUNCATE_ALL_INDEX`, but
  merged-engine treats that marker as "no reusable bootstrap" for live replay
  re-entry. If the keyspace is still active locally, the returning live
  bootstrap also reloads the kv shard so replay resumes without re-adding the
  keyspace. Ordinary destroy paths remain unchanged: they keep the region
  progress tombstoned and therefore still block reuse in the safe path.

## Recovery flows
### Recovery from backup
- Used when the manifest has no stored progress (fresh or wiped state).
- Bootstrap treats the local merged rfengine as disposable input:
  - If `<local_dir>/merged` exists, it is reset before opening rfengine to avoid
    stale peers from previous runs (see issue #4818).
  - The first observed stale directory is preserved as `<local_dir>/merged.stale`
    for troubleshooting. If `merged.stale` already exists, we keep it and
    remove the current `merged` directory.
- The latest backup meta is fetched from object storage.
- Per-store rfengine instances are restored from snapshot + WAL chunks.
- For each region:
  - Region state and raft state are loaded from the origin store.
  - Logs from the origin are collected into `RegionProgress`.
  - Commit index is inferred from counters across stores.
  - The origin selected for copying engine meta/state prefers higher
    preprocessing progress (`last_preprocessed_index`), to enusre that
    shard version will not go backward.
- A merged raft log is built and written into the local rfengine.
- The original per-store raft directories are deleted afterward.

### Recovery from merged rfengine
- Used when the manifest already has store progress.
- Region state and raft state are read from merged rfengine.
- `RegionProgress` is rebuilt from raft state and uncommitted entries.
- If commit index exceeds synced index, the missing entries are fetched so that
  `sync_merged` can continue safely.
- Regions that still lack merged bootstrap are rediscovered from subsequent WAL
  replay rather than from a dedicated manifest record.
- Tombstone regions are recorded for destruction during startup.

### Why uncommitted entries are persisted
- The worker may crash after ingesting WAL but before applying it.
- Persisting uncommitted entries preserves quorum counters and ordering.
- This avoids rewinding commit decisions after restart.

## Keyspace and shard loading
- Startup shard loading is controlled by `MergedEngineInitOpts::load_shards_mode`
  (used by replication_worker lazy shard loading).
  - `LoadShardsMode::EagerAll`: kvengine loads shards for all enabled keyspaces
    on startup.
  - `LoadShardsMode::Lazy { pinned_policy: Arc<dyn PinnedShardPolicy> }`:
    kvengine loads a bounded shard subset (e.g. TiDB meta shards for
    replication_worker) on startup, while keeping other shards unloaded.
  - If no snapshot matches the policy, kvengine remains empty on startup.
  - Snapshot iteration still runs to populate the rfengine dependency graph and
    to perform the same stale/removed peer cleanup as the eager path.
  - Tombstone regions are then destroyed as usual, and truncation/destroy still
    respects `raft.has_dependents`.
- Shard metas are built by iterating snapshots via the recover handler.
- Tombstone regions are skipped at load time and destroyed later.
- Meta packer exposes a compact keeper; replication_worker can pause compaction
  during shard preparation to avoid races with load.

## Encryption handling
- Shard meta includes encrypted data keys; these are decrypted by the
  preprocessor at startup.
- The applier uses the decrypted key when applying entries.
- This ensures old-value reads and writes use consistent encryption state.

## Invariants and failure modes
- Store progress must be monotonic; mismatches indicate WAL corruption.
- `commit_index` must never be less than `synced_index`.
- Tombstone or uninitialized regions are never applied to kvengine.
- `peer_is_restored_from_snapshot` avoids trusting truncated indices from
  snapshot-restored peers.

## Interfaces used by replication_worker
- `update_wal` ingests WAL bytes for a store and epoch.
- `rotate_wal` advances the store epoch after a full epoch is applied.
- `sync_merged` applies committed entries and persists the manifest.
- `collect_store_progresses` collects store progresses into a caller-provided
  vector to drive WAL fetching without extra allocations.
- `MergedEngineInitOpts` controls how shards are loaded on startup.
- `load_shards` and `load_keyspace_shard_metas` manage keyspace shard loading
  and shard meta iteration.
- `load_region_shard` loads a single region shard into kvengine using the
  latest shard meta persisted in merged rfengine (used by lazy shard loading).
- `register_region_change_observer` allows replication_worker to observe merged
  region create/update/destroy events via `CoprocessorHost`; destroy for merge
  sources is emitted at commit-merge acceptance (logical destroy), not delayed
  physical cleanup.
- `get_*` helpers expose store progress, synced target ts, and keyspace regions.
- `unsafe_recover_check_clear_regions` and
  `unsafe_recover_execute_clear_regions` implement the admin-only stale-region
  destroy flow without changing the normal replay destroy path.

## Maintenance tips
- If sync stalls with postponed regions, inspect pending merge states and
  dependent counts for those regions.
- If manifest appears corrupted, verify checksum failures and check for
  partial writes or disk issues.
- If commit index never advances, verify that WAL logs from multiple stores
  are being ingested (quorum counter logic).