# native_br Maintainer Guide This document is for maintainers of the `components/native_br` crate. It explains the complex flows, invariants, and design tradeoffs that are not obvious from reading the code alone. It intentionally skips trivial field lists and one-to-one code mappings. ## Scope and responsibilities native_br provides native backup and restore logic used by the cloud storage engine and related tooling. It covers: - Lightweight (incremental) backup orchestration. - Full restore (storage-node + PD metadata) from lightweight backups. - Keyspace-level restore with range alignment and region reshaping. - WAL capture and replay, including online chunk fallback. - Archive packaging of cold backup data for low-cost storage. - Restore throttling and data-size calibration. It does not implement the low-level storage engines themselves. Instead, it orchestrates rfengine, kvengine, PD/etcd, and S3-compatible object storage via HTTP and gRPC entrypoints. ## Big picture: data and control flow ``` +-----------------+ +------------------+ | PD / etcd | | Storage nodes | | (cluster meta) | | (rfengine, kv) | +--------+--------+ +--------+---------+ | | | lightweight backup RPC | /rfengine/backup + flush | | runtime DFS worker control | (meta, GC safepoint) | v v +----+-------------------------------+ | S3-compatible object storage | | - backup/*.meta | | - store_backup/... (rlog/WAL data) | | - archive/* (packaged cold data) | +----+-------------------------------+ ^ | | restore (WAL + snapshots) | | + keyspace restore HTTP | /restore-shard | v +--------+---------+ +-------+---------+ | PD / etcd | | Storage nodes | | (restore meta) | | (apply snapshot)| +------------------+ +-----------------+ ``` Two paths dominate maintenance work: - **Lightweight backup** produces metadata and WAL/rlog data for each store. - **Keyspace restore** reconstructs a target keyspace by replaying WAL, aligning backup shards to live regions, and restoring snapshots. ## Glossary and mental model The terms below are used consistently across modules. They are the minimum set you need to keep a correct mental model. ### Time markers - **backup_ts**: The timestamp at which a backup is taken. This is the upper bound of data included in the backup. - **safe_ts**: The GC safe point at backup time. A restore should not use data older than this without extra checks. - **truncate_ts**: The target time for PITR-like restore. Must satisfy `safe_ts <= truncate_ts <= backup_ts`. ### Data artifacts - **Snapshot meta / rlog**: rfengine snapshot metadata and raft log payload that bootstrap the raft engine during restore. - **WAL chunk**: A compressed segment of WAL with `[start_off, end_off)`. WAL chunks are stored per epoch and may be incomplete in the last epoch. - **Online WAL chunk**: A tail segment fetched directly from a CSE node when the latest WAL chunks have not reached object storage yet. ### Topology and range concepts - **Region**: PD-level region metadata (`metapb::Region`). - **Shard**: kvengine/rfengine unit (mapped 1:1 to region in restore logic). - **Keyspace range**: The byte range for a keyspace, including the prefix. - **Inner key**: The logical key range inside a keyspace prefix. Alignment logic uses inner keys to compare ranges between keyspaces. ### Backup metadata - **ClusterBackupMeta**: Top-level metadata for a backup. Includes backup_ts, safe_ts, store list, and keyspace metadata. - **StoreBackupMeta**: Per-store metadata (snapshot epoch, WAL offsets, wal_chunks list, and per-keyspace sizes). ## External systems and APIs native_br depends on several external APIs. The interfaces and failure modes shape many design choices. - **PD (gRPC)** - Get TSO, safe point, cluster ID, store list, region info. - Control endpoints for splitting and scattering regions. - Keyspace metadata and placement rules through PD control. - **etcd (gRPC)** - Used for backing up keyspace metadata (raw key-value pairs). - Used for restoring PD state in a fresh cluster. - **Store status HTTP** - `/config`: enable or disable lightweight backup with the `rfengine.lightweight-backup` field. - `GET /rfengine/dfs_worker`: poll the normal DFS worker. - There is no cloud-worker cluster proxy for these two store APIs; external control planes fan out and aggregate them directly. - `/rfengine/backup`: capture a store-local lightweight-backup WAL boundary. - `/rfengine/flush_wal`: make prior captured WAL boundaries durable before cluster metadata publication. - `/rfengine/wal_chunk`: fetch online WAL chunk (tail of epoch). - `/restore-shard`: restore snapshot data to a region. - `/kvengine/files_with_type`: list file IDs by type for throttling. - `/tiflash/sync-region/keyspace/`: check TiFlash replica status. - **S3-compatible object storage** - Stores backup meta, WAL chunks, and archive packages. - Accessed through `kvengine::dfs::S3Fs` with optional object cache. ## Module map (what is complex vs obvious) You will spend most of your time in a few modules. This is the only "map" section; details are elsewhere. - `backup.rs` and `backup_worker.rs`: Complex orchestration, safe point updates, and batching logic. - `restore_keyspace.rs`: The most complex flow: shard selection, preprocess, lock resolution, alignment, snapshot generation, and restore retry. - `common.rs` and `wal.rs`: WAL chunk collection, integrity checks, online chunk fallback, and WAL assembly into a replayable stream. - `archive.rs`: Packaging and index format for cold backups and file restoration. - `lock.rs`: Lock resolution and transaction status reconstruction for correctness. - `limiter.rs` and `tikv.rs`: Throughput limiter and store file existence cache. Simple or mechanical modules (`metrics.rs`, `error.rs`) are not expanded beyond their interactions with complex flows. ### Backup request ownership invariant The current deployment routes cluster backup requests to one designated cloud worker. Its `BackupWorker` / `BackupRunner` owns periodic and instant backups. Temporary DFS-worker enablement still uses the ordinary instant-backup path and therefore enters the same `BackupRunner` queue; there is no separate on-demand backup entry point. The worker protects second-based backup names within that owner. If backup routing is changed to allow multiple owners, equivalent cluster-wide coordination is required first. ## Lightweight backup: correctness and operational behavior Lightweight backup is the "source of truth" for restore flows in this crate. It is intentionally conservative in time and topology handling to avoid silent data loss. ### High-level sequence ``` backup entry -> backup_stores -> /rfengine/backup \-> backup_pd_keyspace_meta (etcd) \-> write ClusterBackupMeta to S3 ``` Key design choices: - **Backup_ts from PD**: The backup timestamp is anchored on PD's TSO so all stores share a consistent time boundary. - **GC protection follows the entry point**: - `backup_cluster` advances the shared native-BR service safepoint only after a successful backup. - A periodic `BackupWorker` attempts to pin that shared service at initialization and advances it after successful batches. When periodic backup is disabled, instant requests do not manage the shared service safepoint. - **Incremental keyspace meta**: PD keyspace metadata is backed up incrementally by revision to avoid full scans and preserve history. ### backup_cluster and store fan-out `backup_cluster` drives the core flow and writes the meta file. Important logic that often surprises maintainers: - **Tolerated store errors** are allowed to a configured limit. The backup meta records tolerated stores to inform later restore logic. - **Missing stores are detected after fan-out** by comparing PD store list with the stores recorded in the meta. `backup_stores` handles concurrency and error tolerance: - Each store is contacted via `/rfengine/backup`. - Errors are aggregated; if missing stores are within tolerance, the backup proceeds and records them. - On retry, the missing stores list is narrowed to improve success rate. Why tolerate store errors at all? - Lightweight backup is often run frequently. - A single store can be transiently unavailable. - It is better to succeed with metadata that declares partial coverage than to fail and block all backup progress. A caller that temporarily enables DFS workers uses the same ordinary instant backup and `/rfengine/backup` fan-out. The cloud-worker instant-backup endpoint issues the existing cluster WAL Flush before returning; native_br has no dedicated on-demand path or per-store flush mode. ### backup_pd_keyspace_meta and etcd behavior Keyspace metadata is stored in PD's etcd space. native_br intentionally backs up **raw key-value pairs**, not parsed data: - Parsing PD's internal formats would require tight coupling to PD versions. - Raw KV preserves fidelity and makes restore safe across versions. The backup path has two collection modes. The default mode scans etcd directly. When `backup_pd_meta_watch_cache_enabled` is enabled for the worker, `BackupRunner` owns a long-lived PD metadata watch cache and asks it for a complete snapshot before falling back to the scan path. The cache: - Bootstraps from the latest backup meta when it has `keyspace_meta` and a non-zero `meta_revision`; otherwise it takes a full etcd snapshot at one fixed revision. - Watches all etcd keys on one stream, but applies only put/delete events whose keys are under the same raw prefixes as the scan path. - Tracks the resume-safe revision, not merely the response header revision: event batches advance to the maximum delivered event modification revision, while empty progress responses advance to the header revision. - Requests watch progress for each backup snapshot request; if progress cannot be proven before the configured timeout, the backup falls back to scanning. - Marks itself stale and resyncs on compacted/canceled watch, stream EOF, or stream errors. Stream EOF/errors restart from the local snapshot revision. A compacted watch cannot restart from that old revision, so it first performs an incremental scan with `min_mod_revision` from the local snapshot revision and then watches from the scan revision. Scan-mode incremental behavior is driven by `meta_revision`: - Only keys with modification revision >= last revision are included. - Deleted keys are not special-cased because keyspace meta is append-only in practice for this deployment model. Why use several explicit PD paths? - Keyspace metadata spans several logical features (rules, labels, groups). - The code enumerates known prefixes to ensure completeness without relying on external PD helpers. ### Backup naming and conflict avoidance Incremental backup files are named by UTC second. `BackupWorker` serializes periodic and instant requests and retries a batch when its name matches the last successful backup. Client-composed temporary enablement uses this same queue. If you see repeated "backup_ts conflict" logs, the batching logic is working as intended and intentionally requeues those requests. ### Backup worker batching and delay The worker coordinates instant and periodic requests: - **batch_interval** controls when a batch is collected. - **backup_delay** defers instant and periodic batch execution to allow WAL flush to complete. - **periodic_backup_interval** injects periodic requests into the queue. Why introduce a delay at all? - The backup RPC captures a boundary, but the latest WAL chunks may still be in flight to object storage. - A short delay reduces the risk of having to fetch online WAL for the last epoch, which would slow restore. ### Failure classification and retriable errors `backup_store` treats rfengine DFS worker health specially. If the store reports a DFS worker unhealthy, the error is surfaced so that higher-level logic can decide to tolerate or retry. In the worker: - Errors are returned to all batched requests. - The worker itself keeps running; it is meant to be long-lived. ## Offline packed backup `PackContext::create_from_config`, used by the CLI and worker, resolves the requested keyspace from `ClusterBackupMeta` without contacting live PD and accepts only keyspaces whose backed-up state is `ENABLED`. The lower-level `create_from_env` constructor relies on its caller to provide an offline client and a validated keyspace. New manifests retain the raw backed-up PD keyspace metadata while continuing to populate the legacy ID and name fields. Bulk metadata lookup for descriptive listing skips records that cannot be parsed and returns their count; exact-target lookup remains fail-closed. Offline file IDs come from a local TSO. New KVEngine files are written to the `new_tables` overlay, leaving source backup objects unchanged. For normal backups, packing disables online WAL fallback and aborts before publishing the manifest when DFS does not cover the backup offset. Archive-backed packing relies on the archive index's completeness invariant. ## Full restore: store and PD recovery Full restore is less frequent but has strict correctness requirements. It rehydrates the raft engine, replays WAL, and reconstructs PD metadata. ### Store ID remapping The restore path always assumes a new cluster. Store IDs in the backup must be remapped to avoid collisions: - The new base is derived from the **latest** backup's alloc_id. - A user-provided delta is added to avoid conflicts with post-backup stores. - The mapping is deterministic: sorted store IDs map to increasing IDs. Why is this needed? - PD's alloc_id is monotonically increasing. - A restored cluster should not reuse IDs that might still exist in PD or in object storage, or it can accidentally reuse old artifacts. ### setup_raft_engine in full restore `restore_tikv` calls `setup_raft_engine` with `full_restore = true`: - The rfengine snapshot (meta + rlog) is restored first. - WAL for each epoch after the snapshot is replayed. - Store and region metadata are rewritten with the new store IDs. Key correctness checks: - The snapshot epoch used is the latest valid snapshot before backup. - WAL replay is bounded by backup_offset to avoid reading beyond backup_ts. ### PD metadata restore `restore_pd` is intended for a **fresh** PD cluster. It uses etcd transactions to guarantee atomic bootstrap: - Writes cluster_id, alloc_id, and cluster meta in a single txn. - Uses `create_revision == 0` to ensure the PD cluster is empty. After bootstrap, keyspace metadata is written in batches because etcd lacks a true bulk put API. Design rationale: - The check protects against accidental restore into a live PD cluster. - The new alloc_id is offset beyond all known store IDs to avoid reuse. ## Keyspace restore: the most complex path Keyspace restore is the main reason this crate exists. It reconstructs a **subset** of the data by keyspace range, and can optionally restore into a different target keyspace. ### Restore pipeline (major steps) The steps are reported through `RestoreStep` in strict order. The core loop is: ``` Init -> LoadBackupMeta -> ExtractBackupShards -> ResolveLocks -> FlushShards -> TruncateTs -> SplitRegions -> AlignRegions -> RestoreSnapshotsToServers -> RetainSstFiles -> Finalize ``` Each step is idempotent or retried with backoff. Only a few steps can safely fail fast (e.g., invalid backup range). ### In-place vs cross-keyspace restore Two modes are supported: - **In-place**: restore into the same keyspace ID. - **Cross-keyspace**: restore into a different keyspace ID. Cross-keyspace restore rewrites the keyspace prefix in range boundaries and shard metadata. It is intentionally blocked for the default keyspace to avoid catastrophic misrouting of data. Source and target names are resolved with `PdClient::load_keyspace`, which rejects returned metadata with a different name before its ID is exposed. This keeps restore independent of the PD HTTP control endpoint; `PdControl` remains responsible for operations that only have a REST control-plane API, such as region and placement changes. For metadata-based CMEK, restore outside the same-cluster in-place path republishes the source encryption meta under a new file ID allocated by the target PD before sending snapshots. The keyspace lineage and key material are unchanged; only the current meta file ID advances. This keeps the restored file ID newer than metadata previously allocated by the target PD, which preserves TiFlash's monotonic file-ID expectation when an empty target was initialized with CMEK before restore. Target lifecycle and encryption-mode compatibility remain control-plane responsibilities. After snapshots are retained, the existing restore flow synchronizes the target CMEK config from the republished metadata. ### BackupCluster: shared state and invariants `BackupCluster` is the workhorse that owns: - A set of rfengine instances per store. - A shared kvengine instance used for lock resolution and flushing. - Shard metadata (range, files, raft state, and properties). - Derived sets (shards needing flush, truncate, or columnar reset). Important invariants: - `sorted_shards` must cover the keyspace range without gaps. - Full-range reconstruction excludes only peers whose persisted `RegionLocalState` is `Tombstone`; transitional states remain eligible so split and merge preprocessing can recover a continuous view. - If persisted full-range peers are observed but tombstone filtering removes every eligible peer across all stores, reconstruction fails with an integrity error rather than treating the backup as a genuinely empty cluster. - Full-range coverage verification returns a structured integrity error. Endpoint mismatches, gaps, or overlaps must abort reconstruction rather than publishing a partial view. - `raw_metas` is consumed when initializing kvengine and must be reconstructed if the keyspace changes. - `shards_need_flush` and `shards_need_truncate` are computed only after shards are loaded and raft state is known. - The master key is resolved once for a restore path when `BackupCluster` is built. If the caller already has a resolved key, `BackupCluster` reuses it; otherwise it decrypts from `restore_conf.security`. - After `BackupCluster` is built, the cached key is reused for WAL scanning and kvengine setup. Do not issue a second KMS decrypt from later helper paths. ### Shard selection and leader choice Shard selection uses rfengine state directly rather than PD metadata. Reasons: - Backup is taken from store-local rfengine snapshots. - PD metadata may be stale or missing in a partial restore scenario. Leader selection uses raft state: - The leader is chosen as the peer with max `(term, last_index)`. - Store ID is a deterministic tie-breaker to avoid retry loops. - If the chosen leader's commit index is behind its last index, commit is forced forward to allow preprocessing. Why force commit forward? - In tolerated-error scenarios, the true leader may be missing. - The most up-to-date follower is still a safe replay source when its last index is committed in practice. ### Preprocessing raft logs Preprocessing applies committed raft entries to rebuild in-memory metadata that is not fully represented in the snapshot: - `preprocess_shard` replays raft entries from `data_sequence + 1` to `commit + 1`. - The resulting meta may include split or merge changes. - If a split is detected (range changed), the shard set is reloaded to avoid complex in-place reconciliation. - `restore_shard` is intentionally rejected in `PreprocessScene::Restore`. It is not a transactional record and carries no timestamp, so offline replay cannot tell whether it should survive the later `truncate_ts` step or be truncated away. Rejecting it is safer than replaying a restore barrier across an unknown PITR boundary. This is intentionally conservative: - Splits and merges are rare at the exact boundary of backup. - Reloading provides a consistent view at the cost of extra I/O. - Rejecting `restore_shard` is preferred over continuing with a partially preprocessed view; callers should retry with another backup. ### Overlapping shards and the intact chain Shard ranges can overlap when followers lag during split/merge. The restore logic models shards as edges in a DAG: - Nodes are inner start/end keys. - A valid restore path is a chain from keyspace start to end. `find_intact_shards` walks this DAG to find a continuous coverage. If no chain exists, restore fails early. This avoids restoring a keyspace with gaps or overlaps. ### Lock resolution and transaction correctness After shard extraction, **all locks are resolved** before truncation. This is crucial and easy to get wrong: - Lock CF cannot be truncated by timestamp. - A committed primary key without its secondary locks can be compacted away after restore, causing data loss. Lock resolution has these paths: - **Normal locks** in LOCK CF are committed or rolled back. - **Txn file locks** are handled using `TxnFileRef` and custom logs. - **Shared-lock wrappers** in normal LOCK CF are deleted as lock cleanup. native_br does not reconstruct lock-only `WriteType::Lock` records or rollback metadata for shared-lock holders. This intentionally differs from online commit/resolve behavior: shared locks carry no user value, and native_br prioritizes restored user-visible data plus a clean LOCK CF. Transaction status checks are expensive and cached: - `TxnStatus` caches commit_ts per transaction using a DashMap. The cache key stays at `start_ts`: it stores transaction outcome only, not the final write shape of each scanned lock. - It reads primary lock and writes using `CloudReader` on snapshots. - For async commit, it inspects secondary locks to compute commit_ts. Committed-lock materialization for ordinary `txn_types::Lock` values is a separate step from status lookup: - Once restore proves a lock is committed, it must still validate `commit_ts > start_ts` and `commit_ts >= min_commit_ts`. - Restore mirrors online commit semantics for the final write record. In particular, a committed leftover pessimistic lock is restored as `WriteType::Lock`, not rejected or silently dropped. - Restore also preserves the commit metadata that still matters offline: `last_change_ts`, `versions_to_last_change`, `txn_source`, and the overlapped-rollback marker when the lock already recorded that `commit_ts`. Concurrency pitfalls: - `apply_custom_log_in_recover` conflicts with `load_unloaded_tables`. - `ApplyLocks` enforces a per-shard mutex to prevent races. Why use CloudReader instead of direct engine reads? - It respects MVCC semantics and provides uniform access to locks and commit records across sharded storage. ### Flushing memtables and meta application Some shards require flush because: - Their raft last index exceeds snapshot data_sequence. - They are derived from a parent snapshot (initial flush needed). Flush logic: - Each shard is flushed via kvengine to persist memtables. - `MetaApplier` listens for change sets and applies them to shard meta. - After flush, shard metadata is taken back from the applier. Why a separate MetaApplier thread? - The kvengine emits change sets asynchronously during flush. - The applier serializes metadata updates and captures errors. ### Truncation to truncate_ts Truncation is applied to shards where `max_ts >= truncate_ts`: - It uses kvengine's `truncate_with_ts` and applies the returned change set to shard metadata. - This is the main mechanism enabling PITR within a keyspace. Truncation happens after lock resolution to avoid dropping locks that are needed to resolve committed transactions. ### Online snapshot source `online_snapshot.rs` supplies a live-source adapter for near-current branches. It records every source Region's identity, range, and version before selecting `branch_ts`, registers a GC safe point at that timestamp, and holds a source-keyspace `schedule=deny` lease while it discovers the source topology, prepares the empty target, and captures every source Region from its leader. Installing the rule does not cancel operators that PD already accepted: the adapter snapshots active operators before its first Region scan, waits only for merge or peer-movement operators whose Region IDs occur in that scan, and then rescans the source keyspace. Unrelated-keyspace, leader-only, and split operators do not block capture. After that stable scan, the target's `schedule=deny` lease is installed before a lightweight plan splits and places the empty target once under both scheduling guards. Only then does the adapter prepare current leader Stores with the existing backup-ts barrier and begin capture. A leader transfer lazily prepares the new leader Store before retrying capture. A Region split discards the partial capture and re-captures the refined Region set at the same `branch_ts`; a same-layout Region version change is also re-captured so the attempt can finish its cleanup and report the conflict during the final `file_refs` validation. Every capture-time split transition must follow RFStore's batch-split invariant: splitting one Region into N Regions increments every child version by N-1 and keeps the parent ID on the rightmost child. The final `file_refs` validation compares its observation with the pre-branch topology, so it accepts at most one split generation for an initial Region; a child splitting again retries the attempt. Prepare success is determined only by the HTTP status. Capture validation uses the Region/version/range, paired data sequence, and V3 memtable format carried by the payload rather than echoed response headers. `online-snapshot-capture-concurrency` limits only concurrent Region payload capture; completed payloads are still retained until the temporary engine is materialized, so worker memory sizing remains an independent concern. Captured full-CF memtables and immutable-file metadata are loaded into one temporary KVEngine. Online restore constructs that engine directly rather than through the packed-backup constructor: packed inputs have already resolved locks, while online restore must load every L0/lock-CF file before running `LockResolver`. The captured snapshot's `TXN_FILE_REF` property reloads unresolved txn-file locks, while committed txn files attached to WRITE_CF memtables remain in the V3 memtable payload. Captured memtables are installed directly in KVEngine shards; their shard metadata remains owned by `MetaApplier` until the shared flush completes. From `LockResolver` onward, the normal flush, truncation, final target reconciliation, `RestoreShard`, and retain pipeline is reused. The final reconciliation is normally a no-op, but it splits and replaces descendants if the source topology refined during capture. One in-memory hard deadline covers pre-capture target preparation through final retention; it is never persisted. The final source `file_refs` phase uses a five-minute cooperative deadline and a thirty-second timeout per Store request. The online adapter does not clear or rebuild schema, vector, or FTS metadata; timestamp correctness for those structures remains in the restore path shared with PITR. Every cross-keyspace restore synchronizes the target PD encryption config to the complete restored source mode, including legacy and plaintext, so future target files remain readable with the restored shard properties. In-place Normal/PITR restore keeps its established behavior of patching the target only for exported metadata encryption. Before capture, online restore installs the target scheduling guard and then attempts once to align target voters with source voter Stores under both guards, tolerating a configured number of unaligned voters. Before the first `RestoreShard`, it reconciles capture-time source splits under the target guard. Region metadata checks and placement submissions use bounded concurrency; `online-snapshot-placement-concurrency` defaults to 64. Each placement round shares one active-operator snapshot and retains per-Region submission backoff. The observed voter Store set alone defines convergence; while it does not match, any active operator suppresses a conflicting submission. Preferred leader placement remains best effort. A target-keyspace `schedule=deny` lease is held from the pre-capture reconciliation through the last successful `RestoreShard`. A transient renewal failure is tolerated only until the last conservatively acknowledged TTL expires; crossing that deadline is terminal for the attempt. Each online `RestoreShard` also fences the target Region's `conf_ver` at the leader and rejects an already pending configuration change. Version-only changes are reconciled only when the current topology is a split-only refinement of the failed request range; merges and configuration changes fail the attempt. Disk-full retries keep placement unchanged instead of scattering Regions under this fence. After target RestoreShard, a metadata-only source dump enumerates current typed file references, including txn chunks from `TXN_FILE_REF`. The retain filter processes only `R_target - S_source`; any dump or coverage-validation error aborts the attempt before file retention because a complete leader-local file-reference observation is required. After validating every response against its scanned `(range, id, version)` tuple, the filter compares that topology with the pre-`branch_ts` baseline before retaining files. Each Region's successful stable `file_refs` access is its own final observation boundary; a later change to that Region is outside the check. One transition that exactly follows the batch-split invariant is accepted; continuous splits are outside the supported model and are not guaranteed to be detected. RestoreShard atomically adds an extra increment to the affected Region version; leader transfers do not, so rolling TiKV restarts and valid splits are ignored while a RestoreShard or merge observed by the affected Region check makes the attempt return an error. Final retain skips files whose TSO-based file IDs are less than fifteen minutes old. Production DFS GC enforces a thirty-minute minimum start-time safety interval, leaving another fifteen minutes for clock drift and operational delay. Test-only DFS GC profiles may use a shorter interval. ### Region pre-splitting and scatter Before snapshot restore, target regions are split to match backup shards. Why this is needed: - Restoring a large shard into a large region is expensive and can overload a single store. - Region splits improve parallelism and reduce per-region restore cost. The logic uses two levels: - **Coarse split** at a configurable factor to avoid too many splits. - **Fine split** for the remaining boundaries. Scatter is best-effort: - Failure to scatter is not fatal but may lead to temporary hotspots. - Online snapshot restore disables this coarse scatter because each split pass is followed by exact source-voter placement; scatter operators would only conflict with and delay that placement. ### Alignment of backup shards to target regions Alignment maps backup shards onto the **current** target regions: - Both are sorted by start key. - Inner-key comparison handles cross-keyspace restore correctly. - A region may map to multiple shards, or a shard may cover multiple regions. Alignment is repeated on retry because region layout can change between attempts (splits, merges, leadership changes). ### Snapshot generation and schema handling For each target region, a new `ShardMeta` is constructed: - File lists are filtered by overlap with the target range. - Properties are merged with a helper to keep metadata consistent. - The schema version effective at `truncate_ts` is read from the backup keyspace's TiDB meta KV using MVCC snapshot reads. - If that version matches the backup schema file with the largest file ID, the file is reused after rewriting `restore_version` and target keyspace ID. - If the versions differ, the schema is fully rebuilt from the backup meta KV at `truncate_ts`. - One prepared schema file is shared by all target regions. Why rewrite schema files? - Schema versioning is used to guard columnar and indexing metadata. - A restore must produce the schema view at `truncate_ts`, not the newer view represented by the backup-time schema file. Columnar and vector index metadata: - Columnar L0 lists are filtered to only include files in range. - Vector index files are grouped by table/column/index ID. - Missing columnar or vector files can trigger a reset to avoid restoring inconsistent metadata. - PiTR decides derived-metadata retention before truncate rewrites allocate new file IDs. A Shard keeps columnar metadata only when its columnar table IDs match schema-at-T and every source columnar file ID is at or before `truncate_ts`; otherwise columnar, FTS, and vector metadata are reset together. - With reusable columnar state, FTS is retained as a Shard-wide unit only when its tracked definitions match schema-at-T and all source FTS files are at or before `truncate_ts`. Vector retention is decided independently for each logical `(table_id, index_id, column_id)` index. - A target Region may reuse a range-filtered subset of one backup Shard. Derived metadata is conservatively reset when it spans multiple backup Shards, because their coverage watermarks cannot be combined here without reconstructing pending work. ### Snapshot restore requests Each snapshot is sent to the **current leader** of the region: - The region's epoch is verified before sending. - The restore request is retried until timeout. - Disk-full errors are detected and trigger region scatter + retry. Concurrency is controlled by a semaphore to limit in-flight requests. This prevents excessive memory and network usage. If one request returns a terminal error, the batch prevents queued requests from starting but does not abort requests that may already have reached a Store. It keeps the scheduling lease active and drains every spawned task before returning the error, so a persisted terminal restore state means no local batch task from the failed attempt is still running. ### Throughput throttling (optional) If enabled, the restore process estimates the size of each snapshot and uses a limiter to throttle total restore throughput. Two estimation paths exist: - **Local estimate**: sum sizes from snapshot metadata. - **Calibrated estimate**: consult CSE nodes for missing files when total size exceeds a threshold. Calibration is designed to be conservative: - If a store does not respond, all files are considered missing. - This slows down restore, which is safer than overloading a cluster. ### Retaining restored files in object storage After a successful ordinary restore, all referenced SST/metadata files are explicitly "retained" in object storage. Online snapshot restore may first validate source Region versions and subtract files that are already referenced by the source keyspace. This is a guard against external cleanup policies that might otherwise delete files still needed by CSE nodes. The newly written schema-at-T file is retained explicitly because it is not part of the source shard file inventory. ## WAL collection and replay WAL handling is one of the most subtle parts of native_br. It must bridge object storage, online store state, and rfengine replay without gaps or overlaps. ### WAL epochs and chunk integrity WAL data is organized by epoch. For each epoch, WAL chunks are expected to be contiguous: - Each chunk has `[start_off, end_off)`. - The last chunk is marked with `last = true`. `collect_wal_chunk_metas` enforces integrity: - If `end_off == u64::MAX`, the last chunk must exist. - If `end_off == 0`, continuity is required but "last" is not. Why two modes? - The last epoch may still be writing to DFS. - Earlier epochs should be complete and verified strictly. ### Online WAL chunk fallback The most error-prone case is the **last epoch**: - DFS may not yet contain the tail of WAL. - The restore still needs a complete WAL to replay up to backup_ts. The fallback strategy is: 1. List chunk files from DFS. 2. If the last offset is short, fetch the tail directly from the store via `/rfengine/wal_chunk`. 3. If the store reports "epoch overwritten", wait for DFS to catch up and retry using DFS data only. This handles the race between async WAL writer and DFS worker. The design intentionally prefers DFS when possible to avoid stale or overwritten online WAL. ### WAL assembly: memory vs local files WAL chunks can be assembled in multiple modes: - **In-memory** for normal restore. - **Local files** when `lower_memory` is enabled. - **Local chunks with cache hook** when object cache is used. The assembly path is selected by `assemble_wal_chunks` based on: - Whether chunks are memory or local files. - Whether object cache is enabled (for decompressed cache mode). Why support local-file mode? - Some restores require replaying large amounts of WAL. - Local files reduce memory usage at the cost of disk I/O. ### WAL readers and partial range reads `LocalWalChunksReader` provides a streaming view of WAL: - It lazily loads and decompresses each chunk. - It supports reading a partial range `[start, end)`. - Chunks are closed and released as the reader moves forward. This is critical for large restores where WAL does not fit in memory. ### Replay into rfengine Replay uses the rfengine API: - The WAL reader is passed to `replay_wal_file`. - For the last epoch, replay is truncated at `backup_offset`. - Earlier epochs replay the full WAL range. Full restore uses `full_restore = true`. Keyspace restore uses `full_restore = false`, which allows rfengine to apply keyspace-aware replay when supported (keyspace IDs are provided during lightweight restore). ## Archive packaging for cold backups Archive support exists to reduce object count and move cold backup data to lower-cost storage classes. It is intentionally separated from the main backup flow. ### Design overview The archive process: - Iterates daily backups in order. - Looks ahead to the next real backup when a day has no backup meta; the legacy `skip-no-meta-days` setting is ignored. - Computes **deleted files** between consecutive days. - Writes deleted files and WAL/rlog into compact packages. - Emits an index describing object addresses. Empty archive input is a no-op, not an archive failure: - Missing archive indexes from the requested start date only means the archiver should locate the first incremental backup. - If no incremental backup exists from that date, the archive command succeeds without writing packages. - `BackupEmptyForKeyspace(0)` during full-range archiving means there is no cluster data to archive for that backup date. The archiver keeps scanning later dates instead of ending the whole archive window. Other keyspace-empty errors remain restore errors. - An archive candidate is considered to have cluster data only when its collected file map is non-empty. Empty candidates are kept as scan state but never write archive packages or indexes. - A full-range candidate that returns `RejectReplayRestore` or `ShardRangeIntegrityError` is skipped at the calendar-date level. The last usable backup remains the comparison base, no archive package or index is written for the rejected candidate, and the existing next-backup lookup selects a later date. All other candidate errors remain fatal and do not advance archive scan state. Why archive deleted files only? - Unchanged files remain referenced by newer backups. - Archiving only deleted files minimizes storage and avoids duplication. ### Archive index format and object addresses Archive objects are addressed by `(package_id, offset, length)`. The index supports two versions; v2 is the current format: - v1: meta address + table file addresses. - v2: meta address + per-store rlog/WAL addresses + table file addresses. Store metadata includes: - Snapshot meta and rlog addresses. - A list of WAL epochs and chunk addresses. Why store WAL separately? - WAL is needed for replay and must be accessed by epoch. - Keeping WAL addresses grouped enables streaming restore. ### ArchiveWriter behavior `ArchiveWriter` appends objects into a buffer until it reaches `max_archive_file_size`, then rotates to a new package: - The first object is always the cluster backup meta. - Each store's rlog/WAL data is appended before table files. - Packages are stored in a cold storage class when supported. Rotation is **not** optional: - Keeping packages reasonably sized avoids huge object downloads during restore and fits within S3 request limits. ### ArchiveReader behavior `ArchiveReader` builds an index map from the start date onward: - The start date's index provides the meta address and store list. - Later indices may provide more recent table file addresses. Restoration uses the map as a lookup table: - If a file is missing from the normal backup path, restore from archive. - This is used in keyspace restore when older backups were archived. - Archive inventories exclude columnar, vector, and FTS files. Restore still fetches all required row/schema/transaction files and resets the affected Shard's complete derived state for rebuilding. Columnar files are special: - If columnar or vector index files are missing, restore resets columnar metadata to avoid inconsistent state. ## Throughput limiter and store file cache Throttling is optional but useful for large restores. It keeps restore throughput under a configured maximum. ### Local size estimation `ThroughputLimiter::estimate_snapshot_size_locally`: - Sums L0, table, and blob sizes from snapshot metadata. - For IA storage class, estimates table size by meta overhead only. This is intentionally approximate. Accuracy is improved by calibration when enabled. ### Calibration from CSE nodes When the total estimated size exceeds a threshold: - The restore queries CSE nodes for existing SST files. - Only **missing** files are counted toward restore size. `StoresFiles` caches store responses with a TTL. If a store is unreachable: - All files are treated as missing. - The limiter slows down restore (safe but conservative). Why tolerate missing store responses? - The restore can continue even if one replica is down. - Over-throttling is safer than under-throttling. ## Metrics that matter in practice This crate exposes a small set of metrics that reflect real issues. Backup: - `native_br_backup_success`, `native_br_backup_error`. - `native_br_backup_missing_commit_record`: backup consistency warning. - `native_br_backup_batch_size`: batch sizes in backup worker. Archive: - `native_br_archive_candidate_skipped_total{reason=...}`: candidates skipped after reconstruction proves they are unsafe to use as an archive boundary. Restore: - `native_br_restore_error{type=...}`: categorized restore failures. - `native_br_restore_pending_data_size`: throttler queue size. - `native_br_restored_data_size`, `native_br_restored_kv_size`. WAL: - `native_br_restore_rfengine_wal_epoch_overwritten_error`. Interpretation hints: - A rising WAL epoch overwritten count usually means DFS is lagging. ## Operational guidance and failure modes This section is intentionally opinionated. It is where most maintenance mistakes happen. ### Time boundaries and safety - Always validate `truncate_ts` against `safe_ts` and `backup_ts`. - For archive-based restore, `truncate_ts` is forced to `backup_ts`. - Avoid restoring data earlier than safe_ts unless you fully control GC configuration and can prove safety. ### Tolerated errors during restore `RestoreConfig.tolerate_err` and `strict_tolerate` are dangerous knobs: - Tolerance is only safe for errors that do not affect data completeness (e.g., transient store unavailability during WAL fetch). - If backup already tolerated errors, restore refuses to tolerate more. When in doubt, keep tolerance at zero and fail fast. ### Lower-memory mode and WAL cache `lower_memory` and `cache_for_decompressed_wal_chunks` are useful for large restores but change performance characteristics: - Lower memory mode writes WAL chunks to local disk. - Decompressed cache can increase CPU efficiency but uses memory. Always consider local disk capacity when enabling these options. ### Region split strategy `coarse_split_regions_factor` trades off: - Fewer splits (faster pre-split) vs larger restore units. - More splits (slower pre-split) vs better restore parallelism. Use higher values for large keyspaces to avoid split storms. ### TiFlash replica removal In-place keyspace restore removes TiFlash replicas first: - TiFlash may lag or keep stale data. - The restore waits for TiFlash to acknowledge removal before proceeding. - If PD reports no TiFlash stores, restore skips TiFlash placement-rule HTTP calls entirely. If this step times out, investigate TiFlash availability and placement rule configurations. ## Debugging and observability ### Step reporting `step!` and `step_error!` emit structured log lines. Set `STEP_TO_STDOUT` to mirror these to stdout for CLI tooling. Use this when debugging interactive restore flows. ### Failpoints Failpoints exist in backup store paths (e.g., `native_br::backup_store`). Use them to simulate store failures or slow responses. ### Logs and crash analysis The repository-level tip applies here too: - Use `LOG_FILE=/tmp/test.log` to capture panics. - Some panic hooks may call `_exit(1)` without flushing logs. ## Testing and validation This crate has unit tests and depends heavily on integration tests. Suggested workflows: - `cargo test -p native_br` for unit tests. - `cargo test -p native_br --features failpoints` for failpoint coverage. - `cargo test -p tests --test cloud_engine` for integration flows. When changing restore logic, also consider running: - Targeted nextest runs for keyspace restore tests. ## Appendix: data layout in object storage This is a conceptual layout. Do not treat it as a contract for external tools. ``` /backup//.meta /store_backup//... (rfengine data) /archive/index/.idx /archive/package//.pack ``` Paths are constructed by `S3Fs` and helpers in `backup.rs` and `archive.rs`. ## Appendix: error taxonomy (what to expect) A few error classes are particularly important: - **TopoChanged / MetaNotFound**: typically requires full backup. - **WalChunkIntegrityError**: often a DFS lag or partial upload. - **RegionVerNotMatch / RegionNotFoundOrNoLeader**: retriable, caused by region movement during restore. - **StoreDiskFull**: triggers region scatter and backoff. Errors are deliberately bubbled with context so callers can choose whether to retry, tolerate, or abort. ## Appendix: invariants to protect when changing code If you change any of these, re-evaluate correctness carefully: - A backup must not advance GC safe point before it is fully written. - WAL replay for the last epoch must not exceed backup_offset. - A restore must not mix shards from different keyspace ranges. - Lock resolution must happen before truncate_ts is applied. - Shard ranges must cover the keyspace range without gaps. - Restored schema files must represent the backup meta KV snapshot at `truncate_ts` and use the target keyspace ID. Violating these invariants can lead to silent data loss.