# Online Snapshot Branch Design ## Status This document describes the current design and implementation contracts. The existing native BR restore pipeline already reconstructs schema at the restore timestamp and safely retains or resets derived metadata; the online snapshot source reuses those guarantees. Rollout still requires end-to-end, failpoint, performance, deployment-admission, and observability follow-up work. ## Summary This document describes a fast keyspace branch mechanism for Cloud Storage Engine. It captures the currently applied state directly from the leader of each Region in the source keyspace, reuses the immutable files referenced by those snapshots, and then enters the existing native BR keyspace restore pipeline at the lock-resolution stage. The online snapshot source replaces the expensive source-side reconstruction stage in the current PITR-based branch path: ```text Current path: Instant backup -> Load and replay rfengine/WAL state -> Extract keyspace shards New path: Ensure source keyspace Region boundaries -> Install the source RestoreShard fence -> Choose branch_ts and register its GC safe point -> Establish the attempt deadline -> Install source schedule=deny -> Snapshot operators, scan source Regions, drain relevant operators, and rescan -> Install target schedule=deny guard -> Split/place hidden target Regions once under both guards -> Establish the existing backup-ts barrier on source Stores -> Export a stable KVEngine snapshot from each source Region leader ``` The online path reuses the native BR restore pipeline from lock resolution onward, while threading the online attempt deadline and target scheduling guard through the shared phases and adding final split/placement reconciliation. After all target Regions have been restored, it performs a metadata-only source dump and retains only files that are no longer referenced by the source. The shared restore pipeline already selects schema at the restore timestamp and conservatively retains or resets vector and FTS metadata. The online snapshot source reuses those guarantees instead of implementing parallel restore semantics: ```text ResolveLocks -> FlushShards -> TruncateTs -> Reconcile missing capture-time split boundaries and placement -> RestoreShard -> Selective RetainFiles ``` For each one-to-one source/target Region pair, `tikv-api` attempts to place target voter peers on the same Stores as the source voter peers before invoking `RestoreShard`. Placement alignment is not required for correctness, but it can turn most S3 downloads during restore into local file hits. ## Terminology - **Source keyspace**: the keyspace from which the branch is created. - **Target keyspace**: the new keyspace that receives the branched data. - **`tikv-api`**: the deployed component that executes the current PITR/restore-keyspace tasks. In this design, it also owns online branch task state, source-capture orchestration, target placement, restore, retain, and finalization. The deployment control plane owns target creation, isolation, and publication. - **Source Region**: a Region that covers the source keyspace. - **Target Region**: a Region that covers the corresponding range in the target keyspace. - **Inner range**: a key range with the API v2 keyspace prefix removed. - **branch_ts**: the single logical timestamp represented by a completed branch. - **Attempt**: one complete `tikv-api` execution from selecting `branch_ts` through successful finalization. Snapshots, the prepared Store set, the temporary engine, and range progress are not persisted as recoverable state. The deployment control plane may publish the target only after the attempt reports success. - **Capture time**: the physical time at which one Region snapshot is created. Different Regions may have different capture times. - **Completeness barrier**: proof that every transaction whose final `commit_ts <= branch_ts` is represented at capture time by authoritative outcome evidence: a commit/rollback write record, or the complete async-commit lock set from which the outcome is derived. It also prevents a new outcome at or below the boundary from being established after the barrier. This design uses TiKV's existing Store-level backup-ts tracking to provide this proof. - **Branch deadline**: one in-memory attempt deadline created after the source `RestoreShard` fence, `branch_ts`, and GC safe point, but before stable source discovery and target preparation, and checked through final `RetainFiles`. Cancellable Store requests and target split batches are capped by its remaining duration. Some long-running shared restore phases currently use cooperative phase/batch-boundary checks and can return after the deadline; they must still prevent the expired attempt from reporting success. - **Target reference set (`R_target`)**: all DFS file references in the final target `RestoreShard` ChangeSets, deduplicated by `(file_type, file_id)`. - **Live source reference set (`S_source`)**: the current file-reference set obtained and validated through metadata-only dumps from source Region leaders after the target has been fully restored. - **Target scheduling guard**: a temporary PD Region Label rule covering the entire target keyspace range. It uses `schedule=deny` to reduce background scheduler and merge movement from guarded placement revalidation through the last `RestoreShard`. ## Background and Motivation The current branch operation is effectively a cross-keyspace PITR restore. It is general and supports historical recovery points, but it performs unnecessary work when a user only needs a branch near the current time: - create or locate an instant backup; - load the rfengine state of each Store; - replay WAL to the backup boundary; - extract the requested keyspace from the reconstructed Store state; - process historical Region topology before entering the normal keyspace restore path. Cloud Storage Engine already stores row, blob, columnar, vector, and FTS files as immutable files in DFS. Keys in KVEngine files use inner keys, so the same immutable file can be referenced by a shard in another API v2 keyspace when range and schema metadata are rewritten correctly. Existing native BR already reuses original file IDs when constructing the target `ShardMeta`. The new design uses these properties to obtain live shard state directly from source leaders, avoiding backup/WAL reconstruction while continuing to reuse the existing restore pipeline. ## Goals - Create a branch of a multi-Region API v2 keyspace at a near-current `branch_ts`. - Reuse the existing backup-ts commit/apply barrier while keeping the source keyspace online for reads and writes during capture. - Reuse immutable DFS file IDs instead of copying the entire source keyspace. - Reuse native BR lock resolution, flush, truncation, Region alignment, `RestoreShard`, retry, and file-retention infrastructure. The online-specific differential retain operates only on file types for which the shared pipeline has complete referenced-file enumeration. - After the target has taken ownership of its files, obtain current referenced-file metadata from source leaders and execute DFS retain only for `R_target - S_source`, avoiding tagging requests proportional to the total size of the source keyspace. - Keep the target keyspace invisible to users until every target Region has been restored. - Align target peer placement with source peer placement to improve local file reuse during `RestoreShard`. - Retry safely from the beginning after a failure or interruption without exposing a partially restored target. - Define explicit limits for concurrency, memory, S3 reads, and placement operators. ## Non-Goals - PITR to an arbitrary historical timestamp. The first version selects a near-current `branch_ts` at the start of each attempt. - Cross-cluster or cross-DFS branches. - Re-encrypting data into a different encryption domain. - Branching the default keyspace. - Changing the current native BR flush/truncate order. - Fixing shared PITR restore issues such as schema, vector, or FTS only in the online source adapter. - Treating Region placement alignment as a correctness prerequisite. - Persisting a snapshot manifest, recovering an interrupted phase, or resuming from partially completed Region ranges. ## Architecture ### High-Level Flow ```text source keyspace | ensure source keyspace Region boundaries | install the source RestoreShard fence | select branch_ts T | register GC safe point and branch job state | establish the attempt deadline | install source schedule=deny | snapshot operators, scan, drain relevant operators, and rescan | install target schedule=deny guard | split target at source inner boundaries | align target peers to source Stores | prepare source Stores and wait for backup-ts barrier T | export snapshots from Region leaders | validate exact snapshot coverage of the keyspace | build a temporary multi-shard KVEngine | existing ResolveLocks -> Flush -> Truncate | final split/placement reconciliation -> RestoreShard -> selective RetainFiles | validate and finalize the target keyspace | control plane publishes the isolated target ``` ### Component Boundary The implementation adds an online snapshot source adapter and connects it to the existing shared restore pipeline: ```text PITR source backup + rfengine/WAL replay + shard extraction \ -> BackupCluster / shared restore pipeline / Online snapshot source Region scan + barrier-protected leader snapshots ``` The online source must produce the same logical inputs required by `BackupCluster`: source shard metadata, stable immutable-file references, complete memtable contents, Region metadata required by lock resolution, and a unified `truncate_ts`. The online source has a dedicated constructor for the temporary restore KVEngine rather than passing through the packed-backup constructor. It loads immutable files that may contain locks before installing captured memtables, then reuses the semantics of `LockResolver`, flush, truncation, target alignment, and `RestoreShard`. A near-current request relies on deployment enablement to provide a uniformly supported Store version. The current worker validates capture encoding when it receives each response and treats an unsupported response as an attempt failure; it does not perform a separate all-Store capability preflight. A failed task retries from the beginning with a new `branch_ts`; a terminal failure returns a structured error. ## Branch Semantics The final target represents committed data visible in the source at one unique `branch_ts = T`. Let `S_i(t_i)` be the complete MVCC snapshot of Region `i` captured at physical time `t_i`. The `t_i` values do not have to be equal. The final result is: ```text Branch(T) = Union_i TruncateT(ResolveLocks(S_i(t_i))) ``` This result is correct if and only if all of the following conditions hold: 1. Every `t_i` occurs after the Store that provides that leader snapshot has completed the backup-ts completeness barrier for `T`. 2. Every snapshot contains all immutable files and all memtable versions required by its range. 3. The captured ranges form an exact, gap-free, non-overlapping cover of the source keyspace. 4. Lock resolution sees every captured shard in the same temporary engine. 5. Lock resolution runs before flush and truncation. 6. Truncation removes every version newer than `T` from managed CFs. 7. The final `RetainFiles` completes within the branch deadline, which must remain safely below the deployment's shortest DFS physical-deletion grace period. 8. The target remains unreadable and unwritable until every Region has been restored. 9. From the selection of `T` until final retention completes, the source must reject `RestoreShard`, because that operation has no timestamp that can be ordered relative to `T`. 10. Target metadata uses the schema file valid at `T`, rather than unconditionally using the newest schema file at capture time. ### Non-MVCC Shard State An online snapshot also captures shard metadata such as schema, delete-prefix, compaction, and storage-class state. This section covers concurrent state changes relevant to online capture. Shared restore guarantees involving schema files, vector indexes, and FTS are covered separately below. - **`RestoreShard`**: this operation has no timestamp that can be ordered relative to `T`. Before selecting `T`, `tikv-api` must install a `RestoreShard` fence for the source keyspace, wait for already-issued operations to complete, and keep the fence until final selective retention succeeds or the attempt terminates. In addition to protecting `Branch(T)`, extending the fence ensures that every immutable file ID returned by the second `file_refs` dump has been referenced continuously by the source since the first capture, instead of being removed and then reintroduced by `RestoreShard`. New `RestoreShard` operations must be rejected while the fence is active. If the fence cannot be proven effective, the attempt fails. - **Delete-prefix**: the service GC safe point for the source keyspace at `T` protects this operation. A dropped table executes physical delete-prefix only after the relevant versions have passed the GC safe point. While the service safe point remains fixed at `T`, it cannot delete data required by the branch at `T`. For data older than `T` that has already satisfied the GC condition, executing delete-prefix before or after capture does not change the logical result visible at `T`. - **Compaction and storage-class transition**: these operations change only the physical organization or storage tier of the same logical data. They do not change keys, MVCC versions, or visibility. A snapshot may use either equivalent representation, provided that it references a complete and readable file set. ### Backup-ts Completeness Barrier Allocating `T` from PD does not prove that every Region has applied all data with `commit_ts <= T`. A transaction or 1PC request may already have obtained a timestamp below `T` while its Region command is still in the network or Raft apply path. The online branch reuses the Store-level barrier from native BR: ```text update_max_ts(T) -> replace_backup_ts(T) -> wait_old_backup_ts_released(T) -> create a stable SnapAccess ``` Before checking its commit timestamp, a transaction obtains a reference-counted `TrackedBackupTs`. The max-ts and backup-ts checks push normal 2PC commit, async-commit, and 1PC paths to the new boundary. A transaction already allowed to commit before `T` retains the old marker until the Raft command that makes its outcome authoritative has been applied. For normal 2PC and 1PC this produces the write record; for async commit, successful prewrite applies the complete lock state whose `min_commit_ts` values determine the final commit timestamp, even if the primary write record has not been materialized yet. Waiting for every old marker to be released closes the window in which either form of outcome evidence could otherwise be missed between timestamp selection and Raft apply. `tikv-api` lazily prepares only the current leader Store. Before accepting a snapshot, it must install `T` on that Store, wait for old transactions to finish applying, and then create a stable `SnapAccess`. After the barrier succeeds, `tikv-api` records the Store within the attempt. If leadership transfers to another Store that was already known when the fence was acquired, it prepares that Store before retrying capture. This also covers a peer added during capture on an initially known Store and then elected leader. A newly registered Store instead triggers the conservative whole-attempt retry described below. `replace_backup_ts` advances a Store's backup-ts boundary monotonically. After a Store successfully completes the wait for `T`, a later, larger backup-ts can only strengthen the condition; it cannot invalidate the completed barrier. The prepared Store set recorded by `tikv-api` is therefore sufficient for subsequent capture. The branch API defines `storage.check_backup_ts` and `backup_ts = T` in the request as required capabilities. A failed `update_max_ts`, a wait timeout, or TTL-based removal of an old marker fails the current capture. `tikv-api` records every TiKV Store ID known to PD, including tombstones, and installs the source `RestoreShard` fence on every current non-tombstone Store. This fenced Store set is immutable for the attempt. Encountering a leader outside it causes the Store-side capture or file-reference endpoint to reject the request, and rescanning the complete Store ID set after capture and after final retention also catches a new Store that never becomes a leader. A retry installs the fence on that Store before selecting its new `T`. Each fenced Store rejects new requests for the keyspace and drains requests that already entered. Renewal may only extend the same owner's unexpired fence. A Store restart or lease gap cannot be accepted as a fresh successful install; the attempt fails. ## Restore Correctness Guarantees Reused from Existing PITR Online snapshot branch changes only how source state is acquired: live leader snapshots replace instant backup, rfengine/WAL replay, and shard extraction. Starting from `ResolveLocks` in the temporary KVEngine, the online and current PITR sources share the same flush, truncation, metadata rewrite, `RestoreShard`, and retain pipeline. The base restore pipeline already provides the following guarantees. Future extensions must remain in that shared pipeline so both source adapters receive the same semantics. The online task fails if the shared pipeline cannot meet a required guarantee for its input metadata. ### Schema File Visibility at the Restore Timestamp A schema version number cannot be compared directly with a TSO, and `truncate_with_ts` does not automatically roll a schema file back to the version valid at restore timestamp `T`. The shared restore pipeline uses authoritative schema history and DDL commit timestamps to find the schema version valid at `T`, regardless of whether a shard comes from PITR reconstruction or online capture. If an input shard carries a schema file newer than `T`, the shared logic selects the corresponding older immutable schema file and then performs the target keyspace ID and restore-version rewrites. Restore hard-fails if that version cannot be found or verified. ### Vector Index Retain or Reset at the Restore Timestamp The shared pipeline retains a vector index only when its schema definition matches schema-at-`T` and every source vector file is at or before `T`. Otherwise it resets that logical index for rebuild. The online source uses this same retain-or-reset decision. ### FTS File Enumeration and Retention For FTS, the shared restore pipeline completely enumerates and protects every referenced packed and dedicated file. It retains FTS metadata as a shard-wide unit only when tracked definitions match schema-at-`T` and all source FTS files are at or before `T`; otherwise it resets the derived state for rebuild. Online selective retain changes only the retain candidates in the shared `R_target`. The second source dump uses the same file-type enumeration to compute `R_target - S_source`; it does not define a separate FTS inventory. ### Shared Implementation and Validation Boundary - Schema selection, vector retain/reset, and target referenced-file enumeration for every file type live in the shared native BR restore pipeline, not in the online source adapter. - Capability and admission must distinguish the file types supported by the shared pipeline. Online-specific capability describes only snapshot encoding, the barrier, and the `file_refs` differential-retain support. - The same schema/vector/FTS fixtures must enter the shared pipeline through both the PITR and online sources, verifying equivalent target metadata, visible data, and retained files. - A schema newer than `T` selects the older file valid at `T`; both paths hard-fail when schema history is missing. - Vector fixtures must contain index entries on both sides of `T` and verify the retain-or-reset/rebuild strategy. - FTS fixtures must verify that ChangeSet enumeration, full retain, and online selective retain cover the same complete set of FTS file types. ## Snapshot Capture ### Source Region Discovery `tikv-api` scans the source keyspace through PD and records the following for each candidate Region: - Region ID and epoch; - outer range and inner range; - current leader; - voter peer Store IDs; - optional learner/TiFlash peers, which do not participate in locality alignment. Because source and target use different API v2 keyspace prefixes, range matching must use inner ranges. Both ends of the source keyspace must be Region boundaries. The MVP splits a Region that crosses a keyspace boundary before capture. The client uses `online-snapshot-capture-concurrency` to bound concurrent Region capture requests globally. Each TiKV Store independently limits dump work with a CPU-derived semaphore and a per-response byte limit; there is no separate configurable client-side per-Store capture limit. After capture completes, snapshots are sorted by inner start key and validated: ```text first.start == source_keyspace.inner_start last.end == source_keyspace.inner_end snapshot[i].end == snapshot[i + 1].start ``` Any gap or overlap fails the current attempt. ### Topology Changes A leader transfer is handled by refreshing the leader and retrying. If the new leader is on an initially fenced Store that has not been prepared, it must first complete the backup-ts barrier at `T`. A leader on a Store added after fence acquisition fails the attempt immediately because activity before a late fence could not be excluded. An epoch mismatch triggers a rescan of the complete source keyspace. A split-only refinement restarts the entire capture; merge or another non-split topology change fails the attempt. Source capture installs a key-range `schedule=deny` lease before discovery. Because the label does not cancel an operator that PD already accepted, `tikv-api` first snapshots the active operator list and then scans the source Regions. It waits only for merge or peer-movement operators whose Region IDs occur in that scan; operators for unrelated keyspaces, leader-only operators, and split operators do not block capture. After every matching operator finishes, it rescans the complete source keyspace. The source lease remains active while the hidden target is split and placed and while source snapshots are captured. The label prevents new automatic topology operators from entering the range under the no-manual-operator contract, while normal Region split handling remains active. ### Online Snapshot APIs The online capture endpoint reuses the essential snapshot primitives behind `dump_kvengine_snapshot`: - validates the leader and Region version; - creates a stable `SnapAccess`; - serializes the snapshot ChangeSet; - serializes every MVCC version and tombstone from all memtable CFs. It intentionally does not enter the legacy range, lock-check, or snapshot-cache path. The branch path uses the dedicated `GET /kvengine/online-snapshot/capture/` endpoint. Successful prepare and fence operations use the HTTP status as their complete acknowledgement and return an empty body. Capture does not echo source Store ID, `branch_ts`, or sequences in response headers. Consumers validate the structured payload's source Region ID/version/range, paired data sequence, and V3 memtable format. The complete source Region, leader, and voter metadata comes from the protected PD scan and is paired with the response by `tikv-api`. The payload contains: - the source Region ID, version, and range identity; - complete WRITE_CF, LOCK_CF, and EXTRA_CF memtable data; - txn-file references and the chunk IDs required to load them; - every immutable file ID referenced by the snapshot. The capture endpoint continues to return structured `NotLeader` and `EpochNotMatch` errors. The legacy snapshot-cache endpoint may continue to return `Locked`. A branch-capable response must capture complete LOCK_CF state so that, after all snapshots enter the temporary engine, the existing global `LockResolver` can process cross-Region transactions. The backup-ts barrier does not require all secondary locks to disappear or every async-commit transaction to have materialized its primary write record. A committed async transaction may instead be represented by its captured primary lock and complete secondary lock set. The existing offline `LockResolver` follows the primary's secondary list and uses the maximum `min_commit_ts` to derive the final commit timestamp; it consults write records when a lock is absent. It subsequently commits or rolls back the captured locks. A lock with `start_ts > T` is rolled back only in the branch; this does not affect the independent outcome of the transaction in the source keyspace. The separate `GET /kvengine/online-snapshot/file-refs/` endpoint handles the second source dump after target restore. Its response is metadata rather than a restore snapshot. It validates the current leader and Region version and obtains persisted file metadata from a stable `SnapAccess`, but it does not reinstall the backup-ts barrier, inspect locks, or serialize memtable rows. A `file_refs` response contains: - source Region ID, version, range, `meta_sequence`, and `write_sequence`; - immutable files referenced by current snapshot metadata, including SST, blob, columnar, schema, vector, and FTS files; - the complete `TXN_FILE_REF` shard property, from which `tikv-api` extracts and deduplicates currently referenced txn chunks with `TxnFileRefPropertyHelper::get_all_chunk_ids()`; - a capability/version indicating that the response is valid only for referenced-file enumeration and cannot be passed to a normal snapshot restore consumer. Txn-file retain enumeration does not require memtable rows. Txn-file references in the memtable encoding reconstruct range-filtered transaction content, while the `TXN_FILE_REF` shard property is the source of truth for enumerating the shard's current txn-chunk lifecycle references. A normal snapshot ChangeSet copies only the subset of shard properties required for restore and does not guarantee a complete `TXN_FILE_REF`. The file-reference endpoint must read and return that property explicitly without changing the format or semantics of the legacy snapshot response. The endpoint enumerates snapshot metadata and the `TXN_FILE_REF` property from one stable `SnapAccess`. The first version does not cache or reuse the file-reference set from the initial capture. ### Complete Memtable Format Online capture uses the dedicated versioned V3 encoding. Unlike the legacy WRITE_CF-only snapshot format, V3 serializes every CF, every version visible at the captured apply boundary, raw value metadata, and tombstones: ```text MemTableSnapshotV3 format_version cf_count repeated { cf_id rows { inner_key, meta, user_meta, version, value } } txn_file_refs ``` Rows contain tombstones and every captured MVCC version, rather than only values visible at `branch_ts`. For `LOCK_CF`, versions newer than the captured apply boundary are excluded because the writable memtable can be shared with concurrent apply. LOCK_CF tombstones cannot be omitted. For example, a source shard may contain: ```text old SST: a lock new memtable: a tombstone that deletes the lock ``` If the tombstone is not exported, the old lock becomes visible again in the target. Restore reconstructs the complete memtable and preserves the shared pipeline order: ```text ResolveLocks -> FlushShards -> TruncateTs ``` ## Lifetime Constraints ### Logical MVCC Retention Before capture begins, `tikv-api` registers a service GC safe point for the source keyspace at `branch_ts` and continuously refreshes its TTL until the branch completes or terminates. This prevents the GC safe point from passing the branch point while different Regions are captured at different physical times. ### Branch Deadline Let `G_min` be the shortest physical-deletion grace period across all referenced DFS file types and supported backends. Correctness requires: ```text branch_max_duration + deletion_safety_margin < G_min ``` The deployment is responsible for configuring the DFS deletion grace so that this inequality holds before enabling the feature. The worker does not inspect backend grace-period configuration when admitting a job. The first version relies on the existing deletion grace and does not add a per-job DFS pin or lease. The deadline is created after the source `RestoreShard` fence, `branch_ts`, and source GC safe point are established, before source scheduling-guard installation, operator draining, target preparation, or the first accepted `SnapAccess`. It is intended to cover target split/placement, capture, lock resolution, flush, truncation, every `RestoreShard` retry, the second metadata-only source dump, and final retention. Cancellable Store requests and target split batches are bounded by the remaining time. Other shared restore operations currently check the deadline cooperatively at phase or batch boundaries, so an executing operation can return after the nominal deadline. Such a result must not allow the expired attempt to report success. If the process stops, the task becomes `Error` and does not resume with the old deadline. Once expiration is observed, the attempt discards all captured metadata and range progress. The next retry selects a new near-current `T` and rescans and recaptures every source Region. Final retention must still complete before publication: the normal fast path retains `R_target - S_source`; if `S_source` cannot be obtained or validated, it safely retains all of `R_target`. A file already missing before expiration is observed is a hard failure and reports a violated DFS lifecycle invariant. ## Temporary Restore Engine All captured shards must be installed in the same temporary multi-shard KVEngine. A transaction's primary and secondaries may span Regions. Resolving locks independently per Region could observe inconsistent transaction results from different physical capture points and materialize only part of a transaction. The temporary engine preserves existing semantics and executes: 1. Load captured shard metadata and memtables. 2. Resolve txn-file locks and normal locks across all shards at `branch_ts`. 3. Flush source memtables and lock-resolution writes to DFS. 4. Run `truncate_with_ts` on files whose `max_ts > branch_ts`. 5. Apply the generated ChangeSets back to the temporary shard metadata. Snapshot preparation and lock resolution may still load DFS files into the worker's local storage. The design removes backup/WAL reconstruction; it does not make the shared restore pipeline metadata-only. ## Target Region Preparation ### Range Alignment To maximize file locality before paying the source-capture cost, the first attempt splits the newly created empty target keyspace at every source Region inner boundary before capture. A whole-task retry may instead see data left by a prior failed hidden attempt; it reconciles that current topology and later overwrites every target range. An existing target-only boundary is allowed; each target Region must overlap exactly one source Region: ```text source [source_prefix + a, source_prefix + m) target [target_prefix + a, target_prefix + m) ``` The target uses its own Region ID, peer IDs, epoch, and keyspace prefix. Existing native BR metadata rewrite constructs a new target `ShardMeta`, rewrites schema metadata, and carries only files overlapping the target range. The MVP requires each target Region to have one unambiguous source voter Store set. Capture-time source splits are tolerated: the shared restore pipeline performs a final target split and placement reconciliation under the already-installed target guard before `RestoreShard`. Target boundary reconciliation intentionally occurs at two points: one guarded pass before capture and the shared pre-restore pass that incorporates source splits observed during capture. The target guard is installed before the first target mutation, so there is no guard-installation race to repair. Every pass derives split keys only for source boundaries missing from the current target scan. When no boundary is missing, it sends no split request. These are idempotent reconciliation checks, not repeated splits of boundaries that already exist. Unlike ordinary native BR restore, online snapshot reconciliation does not best-effort scatter coarse-split Regions. Every target split phase is followed by exact source-voter placement, so scatter would create unrelated PD operators that the placement phase must wait for and then replace. ### Peer Placement Alignment Before invoking `RestoreShard`, `tikv-api` compares the voter Store sets: ```text source voter stores = {s1, s2, s3} target voter stores = {t1, t2, t3} ``` It then uses PD to move the empty target Region to: ```text target voter stores == source voter stores ``` Peer IDs are not reused; only Store placement is aligned. Adding and removing target peers is inexpensive on the initial empty target. It can cost more on a whole-task retry whose hidden target contains data from a prior partial restore. `tikv-api` must: 1. install one `schedule=deny` guard for the entire target keyspace before its first mutation; 2. submit bounded-concurrency PD move/change-peer operators while the target is hidden; 3. validate voter roles and Store IDs from observed Region metadata; while they do not match, wait for any active operator before submitting another one; leader alignment is requested but is best effort; 4. capture the source while continuously renewing both guards; 5. reconcile any capture-time split refinement under the target guard; 6. immediately before `RestoreShard`, query the current target Region and require the expected voter roles and Store IDs again; use its current leader even if preferred-leader alignment did not converge. PD already implements the operator HTTP API required to schedule a Region onto a target Store set. `tikv-api` creates a `transfer-region` operator through `POST /pd/api/v1/operators`: ```json { "name": "transfer-region", "region_id": 100, "to_store_ids": [1, 2, 3], "peer_roles": ["leader", "follower", "follower"] } ``` `peer_roles` is always provided explicitly so the request works when placement rules are enabled. The source leader Store is assigned `leader` and the other Stores are assigned `follower`. PD's `CreateMoveRegionOperator` generates the add-learner, promote, transfer-leader, and remove-peer steps. The implementation requests this leader alignment whenever voter movement requires a `transfer-region`, but convergence and final safety checks require only the desired voter Store set. A different current leader is safe because every target voter has the required source-file locality or can use the normal DFS load path. TiKV only needs a typed `transfer_region` POST wrapper in `PdControl`. Before and after submission, it reuses the existing `get_operators()` call (`GET /pd/api/v1/operators?object=1`) to check whether the Region still has an active operator. The observed voter Store set is the source of truth for convergence even while an operator remains active. If the voters do not match, any active operator suppresses a conflicting submission until the next reconciliation round. The preferred leader is best effort and never gates convergence. On `ErrAddOperator`, it first rereads active operators and topology, then waits or retries subject to the placement-phase timeout capped by the remaining attempt time. `tikv-api` does not cancel PD operators. If a Region already has an active operator and its voters do not match, it waits for completion and then rereads the topology. If the operator remains active while placement is still non-converged when the placement-phase timeout expires, the attempt fails. If no operator remains, a non-converged voter Store set is instead accounted in `U`. A whole-task retry always starts from the current topology. Placement checks and operator submissions run with bounded concurrency across target Regions. `online-snapshot-placement-concurrency` defaults to 64; each round still uses one shared active-operator snapshot and preserves the per-Region submission backoff. Placement alignment does not affect data correctness, but unaligned peers increase DFS downloads and local storage consumption. Define: ```text U = Sum_r |target_voter_stores(r) - source_voter_stores(r)| ``` `U` counts target voter peers that remain unaligned across all one-to-one Region pairs after placement retries finish. Repeated operator attempts for the same peer count once. The calculation uses final Store sets, not the accumulated number of operator failures. The MVP uses a configurable `max_unaligned_voter_peers` with a default of 10: - **`U = 0`**: all peers are aligned; continue restore with full expected local file reuse. - **`0 < U <= max_unaligned_voter_peers`**: continue with the final placement; unaligned peers load missing files from DFS. - **`U > max_unaligned_voter_peers`**: fail the attempt before any `RestoreShard`; the target remains hidden. If an unavailable source Store affects many peers, `U` exceeds the limit and the branch fails before causing large-scale DFS downloads. After all target Regions complete their placement attempts and final-state validation, `tikv-api` computes `U`. The first `RestoreShard` starts only after the unaligned-peer check passes. There is no aggregate missing-byte or PD available-space admission check: txn and schema files can be shared by many Shards, so summing per-Shard estimates can materially overstate physical space. Existing RestoreShard preparation and Store/follower disk checks remain authoritative and can fail the hidden target after some Regions have already completed. The target scheduling guard reuses native BR's key-range Region Label. The rule covers the entire target keyspace, sets `schedule=deny` with a finite TTL, and is refreshed periodically by `tikv-api`. The guard reduces background scheduler and merge movement; Region version, voter Store, and current-leader validation still runs before `RestoreShard`. Each attempt creates one target-keyspace-level guard after the initial target split and placement pass. The guard remains active through the guarded placement revalidation, capture, final reconciliation, and successful `RestoreShard` of the last target Region, after which renewal stops. The current implementation uses a one-minute TTL and refreshes every 20 seconds. The rule expires automatically after renewal stops or the process exits. An attempt that fails or is canceled before the last `RestoreShard` also stops renewal. ### Placement Reuse Boundary Placement alignment improves hit rate only for source immutable files that still exist locally on the target Store and whose file IDs have not changed. The following data may still require download: - new L0 files produced by flushing captured memtables; - new files produced by `truncate_with_ts`; - source files already evicted or transitioned by storage-class policy; - files represented only as IA metadata on the source peer; - derived files explicitly reset or rebuilt during restore. Source Regions may resume normal balancing after capture. Immutable files on an old Store are usually retained until the local file-GC delay expires. `RestoreShard` can still reuse files not yet collected, while the existing DFS load path fetches missing files. After the last `RestoreShard` succeeds, target-guard renewal stops and target Regions resume normal balancing when the rule's TTL expires. New peers prepare files through the normal KVEngine path. The attempt still observes the branch deadline cooperatively until final retention completes. ## Restore and Publication During restore, the deployment control plane keeps the target keyspace administratively hidden and rejects user traffic. `RestoreShard` replaces an entire target shard; writes accepted before restore would be silently lost. TiKV does not currently reject target requests according to branch job state. The MVP therefore reuses the existing branch-restore contract with the higher-level deployment control plane. The control plane creates a new empty target, keeps it locked across failed and retried attempts, and prevents it from being routed, receiving access credentials, or accepting user or administrative writes. `tikv-api` treats these guarantees as trusted preconditions and does not independently acquire or validate target isolation. The control plane invokes restore only while those guarantees hold and publishes the target only after `tikv-api` reports successful finalization. Any direct caller of the restore-keyspace API must provide the same lifecycle guarantees. For every current target Region, `tikv-api`: 1. constructs a `RestoreShard` ChangeSet using the target Region ID, version, range, and rewritten keyspace metadata; 2. queries PD for the target Region's current leader; 3. sends the request to that leader's status endpoint; 4. retries `NotLeader` and transient PD errors; 5. realigns and regenerates the ChangeSet after a Region version mismatch; 6. records the restored inner range in in-memory attempt progress for coverage validation and metrics. For a failed `RestoreShard` request, online restore records the target Region's ID, version, `conf_ver`, and range. A retry tolerates an automatic split only when the current descendants exactly cover that failed range and all retain the recorded `conf_ver`; it then rebuilds snapshots for the descendants. A `conf_ver` change or a Region crossing the recorded boundary indicates peer membership change or merge and fails the attempt. Once a parent request has succeeded, its inner range remains successful: a later split only creates descendants inside an already-covered range and does not invalidate that success. The leader proposes the ChangeSet through Raft, and every target voter prepares the referenced files. In local-storage mode, each Store opens an existing local file by global file ID and downloads it from DFS when missing. The system has no atomic multi-Region `RestoreShard`. The deployment control plane provides user-visible atomicity by keeping the target hidden while `tikv-api` performs the restore and finalization: ```text restore all target ranges -> validate exact coverage and restore version -> metadata-only dump of current source file references -> retain R_target - S_source -> report successful finalization to the control plane -> control plane publishes or enables the target keyspace ``` Finalization also reuses native BR handling for the target restore version, resolved timestamp, and keyspace GC state. Before publication, target GC must not begin at a timestamp that would immediately invalidate the branch point. ### Final Selective Retain Current native BR calls DFS `retain_file` for every file in `R_target`. The S3 backend performs at least one tagging GET per file and also performs a copy if the file has already been logically deleted. The current implementation batches 1,024 files and waits one second before each subsequent batch. For a keyspace with many files, a full retain causes long fixed latency and many DFS requests even when the source still uses most files. After the target has taken ownership, the online branch selects retain candidates as follows: 1. Enumerate all supported file types from the final target shard metadata and deduplicate them by `(file_type, file_id)` to produce `R_target`. This must use final metadata after resolve, flush, truncation, range trimming, and schema repair, not the original capture metadata. 2. Rescan the source's current Region topology and invoke the file-reference endpoint by leader Store. Every response validates leader, Region ID/version, range, and the body-level response version. All accepted ranges must form an exact cover of the source keyspace. 3. Merge and deduplicate current source references by file type to produce `S_source`. The current implementation materializes the complete `S_source` hash set and then filters `R_target`; streaming source IDs directly out of `R_target` is a possible memory optimization, not the current behavior. 4. Pass only `R_target - S_source` to the existing `retain_sst_files`/DFS retain framework. Files created by flush, truncation, selection of an older schema, or derived-file reset do not appear in `S_source` and are therefore retained. 5. A timeout, `NotLeader`, `EpochNotMatch`, coverage/capability validation failure, or property parse failure must not use an unverified or stale set to reduce retain candidates. `tikv-api` may retry the second dump. If it cannot finish within the deadline, it safely retains all of `R_target`. If full retain also cannot complete within the deadline, the attempt fails and the target is not published. After this set-difference step, the shared retain path skips files created in the most recent 15 minutes. Production DFS GC enforces a minimum 30-minute start-time safety interval, so the retain skip window is one half of that minimum and the GC side remains the longer protection window. Correctness of this optimization depends on the second dump occurring after all target `RestoreShard` operations and coverage validation have completed. For any `f in R_target`: - If the source no longer references `f` when its source Region is dumped, then `f in R_target - S_source`, and final retain clears any existing logical-deletion state. - If the source still references `f`, the source protects it at that moment and the target has already taken ownership of the file. If the source later removes the reference through compaction, split/merge, or storage-class adjustment, the target is already another valid referrer, so there is no interval in which neither source nor target references the file. The second dumps for individual Regions need not occur at one global physical instant, but they must come from current active leaders. Stale followers or unverified old Region incarnations cannot contribute references. The source `RestoreShard` fence remains active through final retention. Together with the requirement that immutable file IDs cannot be removed and later reintroduced by another operation, this keeps the file-reference lifecycle monotonic. Existing DFS deletion grace continues to cover the second dump, full-retain fallback, and publication. `file_refs` enumeration and DFS GC must share one authoritative definition of a referenced file. Every file included in `S_source` must also be considered in use by source-side DFS GC. Otherwise, skipping retain has no lifecycle protection and selective retain cannot be enabled. Selective retain applies to every file type with complete referenced-file enumeration, including TxnChunk files obtained from the `TXN_FILE_REF` property. Schema, vector, and FTS follow the shared guarantees defined in "Restore Correctness Guarantees Reused from Existing PITR." Once the shared pipeline enables a file type, the same set-difference protocol applies. A metadata-only dump that skips memtables does not justify unconditionally retaining every TxnChunk or other completely enumerated file type. After a failed or canceled attempt, the control plane never publishes the target, and `tikv-api` retains no recoverable range progress. A retry must select a new `branch_ts`, recapture every source Region, rebuild the temporary engine, and restore every target range. It cannot skip ranges completed by the previous attempt. The control plane keeps the partially restored target hidden while the next attempt overwrites it with a complete `RestoreShard` pass. On permanent cancellation, the control plane deletes or reclaims the hidden target. ## Branch Task Record and Whole-Task Retry Online branch is triggered explicitly through the existing restore-keyspace endpoint: ```text PUT /api/v1/restore_keyspace/{restore_id} ?cluster_id={cluster_id} &keyspace={target_keyspace} &source_keyspace={source_keyspace} &restore_mode=online_snapshot ``` This mode requires different source and target keyspaces and rejects `backup_id`, `backup_name`, and `point_in_time`. Missing backup parameters alone never imply an online branch; without `restore_mode=online_snapshot`, the existing normal/PITR parsing rules remain in effect. Like a normal restore-keyspace task, `tikv-api` persists only task inputs and startup information, with semantics equivalent to the existing `RestoreTaskMeta`: ```text branch_job_id source_keyspace_id target_keyspace_id request_options start_time ``` The state model reuses the simple restore-task states: ```text Pending Init Running Succeed Error ``` `state`, current `branch_ts`, deadline, phase progress, snapshot objects, source Region inventory, prepared Store set, fence lease, target scheduling-guard renewal state, temporary KVEngine, target range mapping, restored ranges, created file IDs, `R_target`, and `S_source` exist only in current-process memory and are not persisted. The GC safe point, `RestoreShard` fence, and target scheduling guard use attempt-scoped leases/rules with TTLs and expire naturally after a process interruption. When a persisted task record is loaded after an incomplete execution, the task is marked `Error("interrupted")`, like the current restore-keyspace task. An explicit retry reuses the task ID, transitions `Error -> Init -> Running`, and runs from the beginning. Each retry allocates a new `branch_ts` for the near-current branch, and a successful response reports the timestamp actually used. Online tasks do not support requests that require the original fixed timestamp. ## Failure Handling ### Source Failures - **NotLeader**: refresh the leader and retry the same Region/version. - **EpochNotMatch**: rescan the complete source keyspace. If the new topology is a split-only refinement of the protected source cover, discard all partial capture results and recapture every source Region in the same attempt with the same `branch_ts`. A merge or another non-split topology change fails the attempt; a whole-task retry then starts with a new `branch_ts`. - **Backup-ts barrier failure**: reject the capture if `update_max_ts` fails, waiting for an old marker times out, or an old marker is discarded by TTL. - **Source `RestoreShard` fence failure**: fail the attempt if the fence cannot be installed, renewed, or validated, or if an operation observed during draining cannot be attributed to the period before the fence. - **Incomplete lock-state capture**: reject a legacy response that does not contain the complete `LOCK_CF` state required by branch mode. - **Unavailable source Store**: wait for a Raft election and retry from the new leader. Do not capture from a stale follower. - **Branch deadline exceeded**: fail the attempt. A retry starts by selecting a new `branch_ts` and cannot reuse captured metadata. ### Materialization Failures - Keep the target hidden. - Discard the attempt's capture objects, temporary engine, and in-memory progress without creating a durable capture manifest. - Reuse existing structured native BR errors for lock resolution, flush, and truncation. - If a source file is found missing before the deadline, fail the whole attempt and report a violated DFS lifecycle invariant. Do not continue with partial shard metadata. - If the second `file_refs` dump cannot be validated completely, discard the collected `S_source` and retain all of `R_target`. Partial or stale responses must not reduce retain candidates. ### Placement Failures - A target Region may have only one active operator. If its observed voters do not match, `tikv-api` waits for the operator to finish, rereads the topology, and does not cancel it. The observed voter Store set alone defines convergence. - If an active operator has not finished while placement is still non-converged by the placement-phase timeout, capped by the remaining attempt time, the attempt fails. If no operator remains, final voter Store mismatches contribute to `U`. - Reread the Region and leader after every operator, then validate Store state and voter topology. - Obey placement rules, Store labels, disk state, and replica count. - After operator retries are exhausted, calculate `U` from the final target voter Store sets, not from the number of failed requests. If `U` exceeds `max_unaligned_voter_peers`, terminate the attempt before any `RestoreShard`. - If `U` remains within the threshold, continue with the final placement. Existing restore-size calibration throttles downloads, while Store/follower disk checks remain the capacity authority during RestoreShard. ### Restore Failures - **NotLeader**: query PD again and retry. - **Region version mismatch**: for a failed request, reread and realign target Regions within the current attempt. Split descendants are accepted only when they exactly cover the failed range with the same `conf_ver`; rebuild their snapshots and retry. A merge, coverage change, or `conf_ver` change terminates the attempt. A Region that split only after its parent request succeeded needs no retry because the successful inner range is already complete. - **Store disk full**: keep online placement unchanged and retry within the existing backoff/deadline. If the error remains terminal, fail the hidden target without a global capacity precheck. - **Terminal batch error**: stop queued RestoreShard requests from starting, keep the scheduling lease active, and drain requests already in flight before reporting task failure. - **Partial success**: keep the target hidden and do not recover range progress after the attempt fails. The next attempt starts with source capture and restores every target range again only after the previous batch is quiescent. ## Performance Model Define: - `R`: number of source Regions; - `M`: total bytes in captured memtables and txn files; - `F_T`: total size of immutable files with `max_ts > branch_ts` that require truncation rewrite; - `N_source`: number of current source file references enumerated by the second metadata-only dump; - `N_retain`: `|R_target - S_source|`, the number of deduplicated files passed to DFS retain; - `L(f, s)`: 1 if target Store `s` is missing file `f`, otherwise 0; - `P(r)`: voter Stores of target Region `r`. Approximate online branch cost: ```text capture CPU/network = O(referenced file count + M) flush DFS writes = O(M) truncate I/O = O(F_T) target DFS reads = Sum_r Sum_f-in-r Sum_s-in-P(r) L(f, s) * size(f) final metadata dump = O(N_source) in-memory metadata serialization/network DFS retain requests = O(N_retain) control-plane work = O(R + placement operators) ``` When target peers align with source peers, most existing source immutable files satisfy `L(f, s) = 0`. Target replicas still need to load newly flushed or truncated files. Without selective retain, the final DFS request count is `O(|R_target|)`. With the second source dump, requests are proportional only to files removed from the source during the branch or newly created for the target. The metadata dump still scans source file references linearly, but accesses only TiKV in-memory metadata and transfers structured IDs; it does not issue a per-file DFS GET. Capturing all Regions in parallel immediately after obtaining `branch_ts` reduces data created after `branch_ts` and lowers `F_T`. If capture is slow, compaction may combine a small amount of new data with a large amount of old data in one large file, causing truncation to rewrite the whole file. ## Resource and Admission Control The first implementation reuses existing native BR restore throttling and adds: - `online-snapshot-capture-concurrency`, default 32, for concurrent Region capture requests; completed payloads remain resident until materialization, so this controls source/network pressure rather than aggregate worker memory; - `online-snapshot-placement-concurrency`, default 64, for concurrent target Region metadata checks and placement-operator submissions; - at most 32 concurrent `file_refs` and other small source control requests in `tikv-api`; - online dump concurrency of half the available CPU cores on each TiKV Store, rounded down with a minimum of one, and a 512 MiB hard limit per response; - build online snapshot payloads on the blocking pool so CPU-bound serialization does not occupy status-server async workers; - `online-snapshot-max-unaligned-voter-peers`, default 10; - existing restore-size calibration and Store/follower disk checks during RestoreShard, without an aggregate missing-byte or PD available-space admission check; - `online-snapshot-attempt-timeout`, default six hours. Aggregate missing-byte admission is intentionally omitted because txn and schema files can be shared by many Shards and repeated per-Shard accounting can substantially exceed physical usage. Local hits, final downloads, and disk rejection remain handled by the existing RestoreShard prepare/calibration path. A shared client-side per-Store limit across source capture, target placement, and restore preparation is a possible follow-up. The current implementation uses the global client limits above and independent Store-side admission for online dumps. `LockResolver` does not run on online source Stores. It processes captured shards in `tikv-api`'s temporary local KVEngine, so its CPU, memory, local-disk, and DFS load count against the `tikv-api`/native BR worker resource budget rather than the source dump semaphore. ## Implementation Touchpoints and Ownership The implementation remains close to existing interfaces: - `components/cloud_server/src/status_server/mod.rs` - Reuses the existing `update_max_ts`, `replace_backup_ts`, and `wait_old_backup_ts_released` sequence for branch capture. Successful return from all three completes Store preparation; timeout, TTL removal of an old marker, or `update_max_ts` failure is a hard failure. - Exposes a separate `GET /kvengine/online-snapshot/capture/` endpoint with a full-CF V3 memtable response. Capture identity and sequences are validated from the request and structured payload without echo headers. - Exposes a separate `GET /kvengine/online-snapshot/file-refs/` endpoint. It reuses leader/version and stable-`SnapAccess` validation, skips the backup-ts barrier, lock scan, and memtable serialization, and returns current typed file references. - Makes `file_refs` and `/kvengine/files`/DFS GC share one shard-level authoritative file enumeration. The typed response only adds file types to the same result and does not maintain a separate list that can omit new file types. - Uses separate server-side dump and memory semaphores. Requests wait for admission permits, and a response above the 512 MiB limit returns `PAYLOAD_TOO_LARGE`. - Verifies the keyspace fence installed by `tikv-api` before the `restore_shard` endpoint accepts a request for the source keyspace. Renewal cannot silently recreate a missing or expired fence after a Store restart or lease gap. - Keeps leader, version, and stable-`SnapAccess` validation in each online snapshot request. Legacy lock validation remains inside the legacy snapshot-cache request. - `components/kvengine/src/read.rs` and `components/kvengine/src/shard.rs` - Provide a versioned full-CF memtable serializer/deserializer without changing the legacy snapshot-cache format. - Provide a narrow coherent-shard-state interface for the file-reference endpoint. It completely enumerates snapshot immutable files by type and `TXN_FILE_REF` chunk IDs from shard properties without reading file contents or serializing memtable rows. - `components/native_br/src/restore_keyspace.rs` - Owns the online snapshot source adapter and `tikv-api` branch phases. - Reuses `LockResolver`, `flush_shards`, `truncate_ts`, target alignment, target snapshot generation, restore retry, schema selection, vector retain/reset, and derived-file enumeration from the shared PITR/native BR path. - Reuses `pause_merge_and_scheduler_during_restore` with one `schedule=deny` rule covering the entire target keyspace from guarded placement revalidation through the last `RestoreShard`. The initial target split and placement pass occurs before this guard is installed. Renewal stops after the last `RestoreShard` succeeds or the attempt terminates. - Separates placement and restore into global phases. After placement settles for every target Region, it calculates final `U` and starts RestoreShard only after `max_unaligned_voter_peers` passes. Existing Store/follower checks during RestoreShard remain the capacity authority. - On a terminal RestoreShard error, stops queued requests, drains every in-flight request under the scheduling lease, and only then reports the attempt as failed. - After every target range has been restored, builds deduplicated `R_target`, performs the second source `file_refs` dump, validates exact coverage, and passes only `R_target - S_source` to the existing retain framework. On any validation failure, it discards partial results and retains all target references. - `components/native_br/src/lock.rs` - Allows captured static Region metadata to satisfy recovery apply because the online source does not create per-Store rfengine instances. - Keeps lock resolution global rather than adding a Region-local lock resolver. - `components/pd_client/src/pd_control.rs` - Provides a typed `transfer_region` wrapper for the existing `POST /pd/api/v1/operators`, serializing `region_id`, `to_store_ids`, and explicit `peer_roles`. Placement reuses the existing `get_operators()` while waiting; no PD server change is required. - `components/cloud_worker/src/native_br.rs` - Persists only the task record, like a normal restore-keyspace task, and exposes create, status, whole-task retry, and cancellation. The deployment control plane consumes the successful task status before publishing the target. It does not recover an interrupted phase or range progress. The corresponding durable contracts are also recorded in the relevant component `MAINTAINER_GUIDE.md` files. ## Planned Metrics The complete metric set below is not yet implemented. Add low-cardinality job metrics for: - branch duration by phase and outcome; - source Regions discovered, captured, and retried; - backup-ts barrier wait duration and hard failures; - snapshot metadata and memtable bytes; - second `file_refs` dump duration, response bytes, Region retries, and full-retain fallback count; - `R_target`, `S_source` intersection, and actual retained files/bytes by file type; - branch deadline remaining at every phase boundary and deadline failures; - referenced-file-missing failures before the deadline; - lock count and lock-resolution duration; - flush input/output bytes; - truncation input/output bytes and rewritten-file count; - aligned target Region count and placement duration; - aligned/unaligned voter peers after placement and failures caused by the unaligned-peer limit; - target files/bytes found locally and loaded from DFS; - `RestoreShard` retries by cause; - hidden-target age and incomplete restored ranges. Metrics must not use keyspace ID, Region ID, branch job ID, or Store ID as labels. These identifiers belong only in structured logs. ## Security and Compatibility - Source and target must be non-default API v2 keyspaces. - Source and target must belong to the same cluster and DFS namespace. - Target shards must be able to decrypt shared files. Reject branches across encryption domains. - Schema, vector, FTS, and other derived-index types use shared PITR/native BR restore capabilities. Online admission rejects a type until the shared pipeline meets its timestamp and retention correctness requirements. - TiFlash learners do not participate in source/target voter placement matching. - The feature is enabled only after every participating Store runs a version with full-CF memtable encoding, hard-failure backup-ts barrier semantics, and metadata-only `file_refs`. The current worker validates capture/file-reference response formats when each response is decoded instead of running a separate all-Store capability preflight. - The online endpoints do not change the existing snapshot-cache request format. ## Rollout Scope ### Correctness and Restore - same cluster, DFS, and encryption domain; - non-default API v2 keyspaces; - a newly created, empty, and hidden target keyspace; - near-current `branch_ts` only; - reuse of the existing backup-ts barrier with strict failure handling; - exact one-to-one source/target inner ranges; - full-CF `MemTableSnapshotV3`; - one temporary multi-shard KVEngine; - unchanged native BR resolve, flush, row/columnar truncation, and restore pipeline; - a source `RestoreShard` fence; - reuse of existing shared PITR/native BR support for schema-at-`T`, vector retain/reset, and FTS enumeration/retention; - a metadata-only `file_refs` dump after full target restore, selective retain by file type, and full retain after validation failure; - admission rejection of derived file types not declared supported by the shared pipeline; - persistence of only the normal task record, with whole-task retry after failure or interruption; - bounded parallel capture and restore; - strict coverage validation and control-plane-owned atomic publication. ### Placement-Aware Restore - typed `transfer-region` wrapper over the existing PD operator HTTP API; - target voter Store alignment; - best-effort target leader alignment; - one temporary scheduling guard covering the complete target keyspace, renewed only through the last successful `RestoreShard`; - recalibration of local/missing files after placement; - a default limit of 10 final unaligned voter peers, with Store/follower disk checks performed during RestoreShard. ## Testing ### Unit Tests - encode/decode `MemTableSnapshotV3` for every CF and tombstones; - reject unsupported snapshot encodings or inconsistent branch timestamps; - detect range-coverage gaps and overlaps; - verify that online target split reconciliation does not scatter coarse Regions; - map inner ranges under different keyspace prefixes; - mark an interrupted task `Error` after loading its task record, and verify the retry transition `Error -> Init -> Running`; - compare source/target voter Store sets; - calculate `U` from final voter Store sets, deduplicate operator retries, and cover `max_unaligned_voter_peers` boundaries; - verify that the file-reference endpoint performs neither lock scan nor memtable serialization and that a normal snapshot consumer rejects the response; - extract and deduplicate typed file IDs from snapshot metadata and the `TXN_FILE_REF` property; - verify that `file_refs` and `/kvengine/files` produce identical shard-level GC reference sets for every enabled file type; any difference invalidates `S_source` and triggers full retain; - calculate `R_target - S_source`, and restore the candidate set to complete `R_target` after any invalid or partial response. ### Integration Tests - create a branch of an idle multi-Region keyspace and compare all data at `branch_ts`; - create a branch while writes continue on both sides of `branch_ts`; - delay commit apply before and after backup-ts replacement and verify that capture waits; - reject capture after a backup-ts wait timeout or stale-marker removal; - cover cross-Region transactions whose primary and secondaries are in different Regions; - cover pessimistic, async-commit, 1PC, shared-lock, and txn-file transactions; - verify that a memtable lock tombstone hides an older SST lock; - transfer a source leader during capture; - add a Store during capture, optionally transfer leadership to it, and verify that Store-set validation fails the attempt before restore; - split or merge source Regions during capture; - verify that the source `RestoreShard` fence rejects new operations, drains already-issued operations, and remains active through final retention; - execute delete-prefix before and after `T` and verify the result visible at `T` is unchanged; - run compaction or a storage-class transition during capture and verify branch data is unchanged; - split a target Region, transfer its leader, or change its epoch during restore; - cover fully successful target peer placement and 1 to 10 final unaligned peers proceeding with final placement; - fail before `RestoreShard` with 11 final unaligned peers; - allow an unavailable source Store that affects a few peers, and fail when it affects enough peers to exceed the limit; - inject Store/follower disk rejection after placement and verify that the target remains hidden and whole-task retry begins only after the previous RestoreShard batch is quiescent; - wait for and rescan topology around a pre-existing active placement operator; if it exceeds the placement-phase timeout, fail and calculate placement from current topology on the next retry; - verify local file hits when target peers match source peers; - verify DFS loading when target peers do not match source peers; - balance a source peer away after alignment and verify local hits during the local file-GC delay and DFS loading after the file is collected; - stop renewing the keyspace-level scheduling guard after the last target `RestoreShard` succeeds and verify that normal balancing after rule TTL expiry does not change branch data; - interrupt the worker during placement/restore without recovering guard state and verify that the unrenewed rule expires by TTL; - truncate a file containing versions on both sides of `branch_ts`; - perform a fresh second source dump when metadata is unchanged and issue no retain for file IDs that remain referenced; - compact captured files out of the source before the second dump and retain only files that disappeared from source references; - remove a file from the source after the second dump and verify that the restored target reference continues to protect the skipped file; - skip retain for a TxnChunk still referenced by the source through `TXN_FILE_REF` and retain a removed chunk; - on leader transfer, Region version change, timeout, missing capability, or malformed property during the second dump, reject partial `S_source` and verify retry or full retain; - after partial `RestoreShard` success, recapture and restore every range in the next attempt; - after deadline expiry or worker restart during capture/materialization, retry the whole task from a new `branch_ts`; - delete a DFS file early and verify that the lifecycle invariant violation is detected and reported; - keep the target inaccessible until every range completes. Failpoint tests must live in `tests/cloud_engine_failpoints/` because failpoints are process-global. ## Decision Summary Online snapshot branch targets near-current branching. Deployment enablement is responsible for the uniform-version and DFS-grace prerequisites; the worker validates response formats as it executes and enforces the attempt deadline cooperatively. The MVP obtains source state from leader snapshots and reuses native BR from `ResolveLocks` onward. Placement tolerates at most 10 unaligned voter peers. Final retain processes only `R_target - S_source`, falling back to all of `R_target` after validation failure. The existing shared restore pipeline provides schema-at-`T`, vector retain/reset, and FTS enumeration/retention.