# kvengine Maintainer Guide This doc records non-obvious design choices, invariants, and workflows. It intentionally: - Prioritize relatively complex logic, explain why it is designed this way. - Skip content that can be easily find in the code. - Keep concise, explicit, while readable. Status: describes current behavior as implemented in `components/kvengine`. If you change a design invariant, update this doc in the same patch. Terminology note: CSE is maintained as a separate project from TiKV. New CSE-owned code, comments, and documentation should refer to the product as CSE. Keep `TiKV` only when naming inherited upstream APIs, crates or binaries, compatibility behavior, historical sources, or external TiKV-specific concepts. ## Contents 1. Scope And Reading Strategy 2. System Context And Responsibility Boundaries 3. Core Concepts And Invariants 4. Key Encoding, Ranges, And Versioning 5. Engine And Shard Model 6. Write Path And Property Semantics 7. MemTable Lifecycle And Flush Scheduling 8. Read Path And Snapshot Semantics 9. Iterators And Multi-Source Merging 10. LSM Layout And File Types 11. Blob Storage Path 12. ChangeSet: Deterministic Metadata Updates 13. Recovery And Startup 14. Flush Implementation Details 15. Compaction Scheduler And Priorities 16. Row-Store Compaction Details 17. Columnar Pipeline And Schema Files 18. Columnar Compaction And Major Compaction 19. Vector Index Pipeline 20. Full-Text Search (FTS) Pipeline 21. TxnFile And Lock Handling 22. Split, Prepare-Merge, Commit-Merge 23. Storage Class And Infrequent-Access (IA) Tier 24. DFS Layer, Local Files, And Caching 25. Resource Control And Flow Throttling 26. Observability And Debug Aids 27. Common Failure Modes And Pitfalls 28. Maintainer Playbook For Changes 29. Glossary ## 1. Scope And Reading Strategy - Focus: maintainer work (correctness, performance, recovery, extensions). - Explains design intent (why), not just behavior (what). - Use it to locate invariants and avoid regressions. - Source is ground truth for anything not captured here. Suggested reading order: - 2-5: architecture + invariants. - 6-9: data plane flows. - 12-16: metadata, flush, compaction. - 17-20: analytical + vector features. - 22-24: lifecycle + tiering. ## 2. System Context And Responsibility Boundaries kvengine sits between Raft (rfstore) and storage (DFS + local cache). It is the state machine for committed Raft commands. Key boundaries: - rfstore owns consensus and replication. - kvengine owns deterministic application of committed changes and background IO. - DFS provides durable storage; kvengine decides what to load and when. - Metadata changes must be replicated via Raft to keep replicas consistent. Why this matters: - Background work (flush/compaction) cannot be applied unilaterally. - LSM changes must be captured as ChangeSets and applied on all replicas in the same order. - Heavy work (flush/compaction/conversion) can run on one node while others replay the result. ## 3. Core Concepts And Invariants These invariants recur in correctness fixes. Violating one can cause subtle replica divergence. ### 3.1 Shard As The Unit Of Consistency - Shard is an independent LSM tree for a key range. - Shard corresponds to a Raft region in rfstore. - Shard metadata is versioned (shard id + shard ver). - Shards can be split or merged, but files must remain range-consistent. ### 3.2 Deterministic Metadata Updates - LSM structure changes are captured as ChangeSets and proposed through Raft. - Applying the same ChangeSet on all replicas must yield the same metadata. - Any background worker output that mutates files must produce a ChangeSet. ### 3.3 Versioning And Sequencing There are multiple version counters that must stay consistent: - `write_sequence`: applied Raft index for WriteBatch. - `meta_seq`: applied Raft index for ChangeSet. - `base_version`: per-shard offset for `SnapVersion`; it can be rebased when a shard incarnation changes: - Split (original shard id): unchanged. - Split (new shard id): `base_version = parent.base_version + split_sequence` so SnapVersion stays monotonic even when the new shard starts from `initial_seq`. - Commit-merge: rebase to the max memtable SnapVersion from source/target (note: `source.sequence` is the `ChangeSet.sequence` carried in `source_meta` (source shard's applied index after `prepare_merge`), not `data_sequence`): `base_version = max(source.base_version + source.sequence, target.base_version + merge_sequence) - merge_sequence`. - Restore-shard: leader adjusts the incoming snapshot `base_version` to avoid regressing current memtable SnapVersion: `base_version = max(old.base_version + old.write_sequence, snapshot.base_version + snapshot.data_sequence) - old.write_sequence`. - `SnapVersion`: `base_version + sequence` (a single u64 used as the memtable snapshot version and the file watermark). - Memtables use `write_sequence`, persisted files use `data_sequence`. - `meta_seq` can lag `write_sequence` because ChangeSets are applied by background workers. - `data_sequence` in ShardMeta tracks the Raft index of data persisted in the latest L0 file. Rules of thumb: - Memtable snapshot version must monotonically increase per shard. - ChangeSet with sequence <= meta_seq must be treated as duplicate. - L0 tables created from a memtable must carry the same SnapVersion. ### 3.4 Properties Are Part Of State - Shard properties are not just metadata; they can alter read/write behavior. - Some properties must be persisted before advancing data sequence. - Some properties are intentionally skipped during initial flush. - Property handling is a common source of divergence after restore/split/merge. - Encryption properties are single-mode today: - legacy regions use `ENCRYPTION_KEY` - CMEK/keyspace-aware regions use `ENCRYPTION_META_KEY` - mixed shard properties are something we avoid; if they appear unexpectedly, the runtime prefers `ENCRYPTION_META_KEY` and warns - the process latch is only a best-effort signal for enabling encryption on new keyspaces; existing shard reads still follow shard properties - that signal comes from loaded shard properties or, if startup sees only plain shards, from a real-vendor global master key that seeds legacy mode ## 4. Key Encoding, Ranges, And Versioning ### 4.1 Outer Key vs Inner Key - kvengine stores `InnerKey` derived from outer keys. - In API v2, outer keys include a keyspace prefix; `InnerKey` strips it for internal ordering. - Range end keys are exclusive; keyspace-boundary end keys need special handling. - `txn_types::Key::from_raw` is for raw keys at TiKV public interfaces. In API v2 that means the outer key, including the keyspace prefix when one exists. Do not feed `InnerKey`, `OwnedInnerKey`, kvengine iterator keys, or buffers known to come from `InnerKey::from_inner_buf` into `Key::from_raw` unless the caller has first reconstructed the correct outer key. - Crossing the kvengine boundary must be explicit: either stay in the `InnerKey` domain for LSM/table operations, or convert back to the outer-key domain before using APIs that compare with region/downstream ranges, `ObservedRange`, CDC event keys, or other TiKV-facing `Key` values. Why: - Keeps LSM ordering stable and compatible with TiDB table key encoding; - Range bounds drive overlap checks and file selection. ### 4.2 Range Representation - `ShardRange` stores outer start and exclusive outer end. - `inner_key_off` is preserved to handle mixed keyspace boundaries. - Global end key uses a sentinel for empty/end-of-keyspace. ### 4.3 MVCC Metadata - kvengine uses three CFs: write, lock, and extra. - Managed CFs (write/extra) require non-zero versions; lock CF uses version 0 in WriteBatch and is rewritten to SnapVersion. - `UserMeta` encodes (start_ts, commit_ts). - Extra CF stores auxiliary transaction status keys and other side data. - Extra CF txn-status key writer uses TiKV-style descending timestamp encoding `raw_key + BigEndian(!start_ts)`. - Readers remain backward-compatible with the legacy `reverse_bits` format for historical data written before the encoding switch. - Value includes meta, user_meta_len, version, user_meta, and value bytes. Why: - Keeps MVCC semantics compatible with TiKV while enabling multi-format storage and remote processing; - Separating lock/write enables efficient lock conflict checks. ## 5. Engine And Shard Model ### 5.1 EngineCore Responsibilities - Owns global caches (block cache, fd cache, value cache). - Owns DFS handle and compaction client. - Owns shard registry and keyspace -> shard mapping. - Runs background workers: flush, compaction, free-mem. ### 5.2 Shard Structure - `Shard` contains range, version, and in-memory state. - `ShardData` holds memtables, L0 tables, L1+ tables, columnar levels, vector indexes, FTS levels, and blob table mapping. - `ShardMeta` is persisted metadata used at startup and recovery. Why separate ShardData and ShardMeta: - ShardData: live in-memory state (handles, caches). - ShardMeta: compact persisted view for recovery/snapshots. - Apply ChangeSets to ShardData; reconstruct ShardMeta as needed. ### 5.3 Active vs Inactive Shards - Inactive shards are ignored by flush and compaction. - During split/merge, shards may temporarily be inactive or parent-only. - `initial_flushed` indicates whether the shard owns its own L0 tables. Why: - Split/merge reuse files; - Initial flush makes each shard self-contained; - Pausing background work avoids building files on stale versions. ### 5.4 Pending Operations And Shard Properties - Shard pending operations track delayed actions such as delete prefixes, truncate_ts, trim_over_bound, manual major compaction, storage class changes, and GC of lock or extra CF files. - Pending ops are derived from shard properties and updated on apply. - They drive compaction priority selection and background tasks. Why: - Decouples Raft apply from background scheduling; - Keeps per-shard decision state explicit and restart-persistent. ## 6. Write Path And Property Semantics ### 6.1 WriteBatch Semantics - WriteBatch contains per-CF write batches and a property map. - Write path runs after Raft commit, not before. - Each entry has InnerKey, meta flags, user_meta, and version. Important version rules: - Managed CFs (write/extra) require non-zero versions. - Non-managed CF (lock) uses version 0 in WriteBatch and is rewritten to SnapVersion. - Write path fills SnapVersion for non-managed entries when needed. Why: - Makes MVCC consistency independent of read/compaction order; - Lets non-managed CF participate in the same LSM structure. ### 6.2 Property Updates During Write Some properties affect behavior and need special handling: - `DEL_PREFIXES_KEY` influences destroy-range behavior. - `TRIM_OVER_BOUND` triggers compaction trimming at range boundaries. - `MANUAL_MAJOR_COMPACTION` schedules a major compaction. - `TXN_FILE_REF` updates lock txn files and commit/rollback tracking. Key behavior: - Properties can be duplicated during restore or recovery. - Write path checks duplicates but still runs deterministic side effects (e.g. memtable switches). - Some properties are tracked as "must persist" before advancing sequence. Why: - Property side effects must be deterministic across peers; - Skipping duplicates can diverge memtable switching and cause version mismatches. ### 6.3 Write Sequence And MemTable Switching - Write sequence updates after each WriteBatch. - Memtable switch triggers when: - The current memtable exceeds `max_mem_table_size`. - A property requires a switch (for example, delete range overlap). - Split/merge forces a switch to isolate old data. Side effects of switching: - Old memtable becomes immutable. - New writable memtable is pushed to the front. - Shard triggers a flush for the immutable one. ### 6.4 TxnFileRef Handling During Write - Txn file refs are stored in the `TXN_FILE_REF` property. - Live rfstore apply can attach a prepared `TxnFile` directly to the `WriteBatch`; `Engine::write()` rebases it to the final `SnapVersion` and consumes it without another `TxnChunkManager` lookup. - Recovery, native BR, merged-engine replay, and direct callers can still fall back to loading txn files via `TxnChunkManager`. - Locks are merged into shard state; commit entries go into memtables. - `TxnFileLocks` stores `start_ts -> lock_value_prefix` so shard metadata keeps only the lock payload prefix and avoids duplicating full user keys. - For a live txn-file lock, `TxnFileLocks` must keep the latest applied `lock_value_prefix` for that `start_ts`. Heartbeat and `check_txn_status` updates are overwrite operations, not write-once metadata. - `TxnFileLocks::get_lock_infos` and `TxnFileLocks::get_lock_infos_by_start_tss` reconstruct `LockInfo` on demand from metadata plus a caller-provided raw key for routing. - `TxnFileLocks::get_key_errors` is derived from the same reconstructed `LockInfo`, so rfstore split checks and background resolve use one source of truth. Why: - Store large lock payloads outside the main LSM; - Keep metadata-backed resolve paths consistent with the latest live lock state; - Prevent lock-heavy workloads from bloating memtables/L0 files. ### 6.5 DeletePrefixes And Destroy Range - DeletePrefixes represent prefix range tombstones. - They are stored in shard properties and merged across split/merge. - When a new delete prefix overlaps the writable memtable, the write path forces a memtable switch to isolate old data. - Destroy range compaction consumes delete prefixes and materializes deletions by removing/compacting affected files. Why: - Prefix tombstones are cheap to replicate through Raft vs large delete batches; - Memtable switch ensures the delete prefix applies deterministically across replicas, even if in-memory state differs. ## 7. MemTable Lifecycle And Flush Scheduling ### 7.1 MemTable List And Snapshot Versions - Shard maintains a list of memtables, newest first. - Each memtable has a SnapVersion assigned when it becomes immutable. - SnapVersion ensures flush results match metadata. Invariant: - Last (oldest) memtable SnapVersion must match the L0 SnapVersion flushed from it. Memtable storage invariants: - `kvengine.enable-crossbeam-memtable` selects the production shard backend at startup. Crossbeam is the default; set the option to false to use the original Arena skiplist. Every new, split, or switched memtable inherits the selected backend. Code paths without a production shard configuration, including remote memtable reconstruction, also default to Crossbeam. - Arena memtables retain the original 16 MiB encoded-value limit. Raft entries above 16 MiB are accepted only when the Crossbeam backend is enabled. - In the Crossbeam backend, versions for one key form an immutable newest-first chain. Its atomic pointer owns the published head's `Arc`; each replacement owns the preceding head before publication. This keeps reads lock-free without `ArcSwap`'s deferred-reference cleanup on the serialized writer. `store_head` must remain protected by the per-skiplist write mutex, and readers must acquire an owning `Arc` through `load_head`. - Crossbeam writes to one skiplist are serialized; reads and iterators remain concurrent. Point reads and iterators own the value nodes backing their raw `Value` views, so a removed chain can be reclaimed after its active readers release it. Uniquely owned version tails are detached iteratively during destruction to avoid recursive drops for hot keys with deep version chains. - Leaving the writable slot permanently seals every CF in a memtable generation under the existing Crossbeam writer locks. Production flush uses a dedicated forward iterator that borrows map entries and immutable version nodes without per-key or per-version `Arc` operations. The normal concurrent iterator remains owning, and writing a sealed Crossbeam memtable is a bug. - The owning Crossbeam iterator entry extends the dependency's borrow lifetime. Its wrapper owns the same skiplist and explicitly drops the entry first. Its `Send` proof relies on the pinned Crossbeam revision keeping only an atomic node reference, while navigation and release pin the calling thread. Any Crossbeam revision change must re-audit `Entry`/`RefEntry`, navigation, release, and drop order, then rerun the forward/reverse deletion and cross-thread iterator tests. - Write batches keep legal Arena-sized values in the reusable packed buffer. Values whose encoded size exceeds the Arena limit use one dedicated `Bytes` allocation; Crossbeam value nodes clone that backing without copying the payload, and batch reset releases its reference. Dedicated backing is per value so one surviving entry cannot retain unrelated deleted values. - `MemTable::size()` remains cumulative submitted bytes and controls switching and flush scheduling. Memory limiters use the separate `memory_usage()` value: Arena preserves its existing accounting, while Crossbeam tracks keys, values, and stable node overhead until active readers and epoch reclamation release them, so deleted churn does not remain charged until the whole memtable drops. ### 7.2 Forced Switch And Initial Flush - Split and merge require a forced memtable switch, even if empty. - Ensures the new shard owns a clean data boundary. - Initial flush uses a dedicated ChangeSet type and sets shard metadata. ### 7.3 Flush Scheduling - Background flush worker consumes FlushMsg events. - Flush tasks are tagged with shard id/ver and range. - Flush uses encryption key from shard state if present. - CMEK/keyspace-aware flush tasks are enqueued before resolving a pending encryption key. If the key is unavailable, the flush worker delays retry instead of failing before the task reaches the retry queue. Why: - Flush is IO-heavy; - Keep it off the raft apply path; - A single worker reduces ordering complexity and metadata races. ## 8. Read Path And Snapshot Semantics ### 8.1 SnapAccess - `SnapAccess` is the primary read facade for a shard snapshot. - Captures memtables, L0 tables, L1+ tables, and schema files. - Can be constructed from a ChangeSet (for remote coprocessor usage). ### 8.2 Read Timestamp Selection - If a read_ts is provided, use it directly. - For managed CFs, a managed safe ts can be injected. - For non-managed CFs, use the shard memtable SnapVersion. Why: - Non-managed CFs must stay consistent with the snapshot that produced them. ### 8.3 Point Get Order The read order is deterministic: 1. Lock txn files (lock CF only). 2. Memtables (newest to oldest). 3. L0 tables (newest to oldest). 4. L1+ levels (level order, non-overlapping). Ensures the latest visible version is returned. ### 8.4 Timestamp-Pruned EXTRA_CF Scans - EXTRA_CF conflict scans may prune immutable memtables, L0 tables, and level tables whose max timestamp cannot contain a conflicting write. - The writable memtable must always be included even if its current `data_max_ts` is below the pruning threshold, because writers insert entries before publishing the updated max timestamp. - Key-bound pruning for EXTRA_CF uses the timestamp-free raw key prefix so all status records for the same user key remain visible. - `storage.extra-cf-scan-mode = "legacy"` restores the pre-pruning conflict check: one ordered scan over all sources, with self-rollback detection inline. `pruned` and the test-only `both` mode use the physical-gap strategy heuristic. When a scan is needed, `both` runs full-source and pruned-source scans with identical conflict parameters and compares their results. - The runtime scan mode is owned by each Storage scheduler, captured once per command, and passed explicitly to readers that perform EXTRA_CF conflict checks. It must not use process-global state because tests and embedded users can run multiple Storage instances with different modes concurrently. Why: - Dataset pruning reduces conflict-check scan cost without changing MVCC visibility. - The writable-memtable exception avoids a read/write race where a concurrent reader could observe a stale max timestamp and skip a newly inserted status record. ### 8.5 Blob Reference Resolution - Values may be stored as BlobRef in the main LSM. - Resolve BlobRef via (file id, offset). - Reads may prefetch blob data to reduce tail latency. - Metadata-only probes such as `SnapAccess::has_newer` must not resolve BlobRef payloads. Coprocessor cache admission only needs to know whether a newer version exists and must not add object-store IO or failures for an invisible value. Why: - Blob separation reduces large-value write amplification; - Also allows different storage classes for large values. ### 8.6 Value Cache - Optional cache stores latest visible values per (shard, keyspace, key). - Cache entries include a valid SnapVersion. - On shard change, cache entries are validated via a read. Why: - Avoids repeated L0/L1+ traversals for hot keys; - Validation prevents stale reads after memtable or L0 changes. ### 8.7 Snapshot Construction For Remote Reads - Remote reads can build a snapshot from ChangeSet bytes + memtable data. - Remote ChangeSet snapshots carry storage-class properties so worker-side preparation preserves IA/Auto-IA read and throttling policy. - Memtable data supports multiple formats (V1/V2/V3) for compatibility. - V2 encodes skiplist data + txn file refs for lock visibility. - Skiplist serialization filters WRITE_CF versions newer than the request timestamp. It also reports whether such a version was skipped so the origin TiKV can veto remote coprocessor cache admission; the worker cannot recover evidence that was omitted from its reconstructed snapshot. - Marshaled remote-read snapshots intentionally leave both `Snapshot.base_version` and `Snapshot.data_sequence` unset. These fields describe persisted shard snapshot metadata, while this ChangeSet contains a range-selected read view and `SnapAccess` cannot report an accurate persisted `data_sequence`. Do not encode the captured in-memory `write_sequence` into either field. The origin TiKV retains `SnapAccess`, combines origin and worker cache-admission evidence, and stamps remote coprocessor `cache_last_version` after worker execution instead. - V3 is reserved for online snapshot branch capture. It serializes every CF, every version visible at the captured apply boundary, raw value metadata, and tombstones, then appends WRITE_CF txn file refs. Because `LOCK_CF` uses physical `SnapVersion`s and the writable memtable remains shared, V3 filters lock versions newer than the captured `base_version + write_sequence` while retaining older versions of the same key. Unresolved txn-file locks remain separate from memtables, so online snapshot metadata derives `TXN_FILE_REF` from the same captured `ShardData` and recovery reloads those lock files before lock resolution. Restore insertion must preserve tombstones so an old immutable value cannot reappear. Reconstruction uses the default Crossbeam backend for every CF, so large captured values remain supported and all CFs retain one backend. - Online metadata uses a dedicated marshal path that records the captured `base_version`, the same `write_sequence` in `Snapshot.data_sequence`, the shard `TERM_KEY` needed by restore-side lock resolution, and range-filtered pending delete prefixes. Legacy marshal intentionally leaves the data sequence unset. - Full-CF capture exposes the cumulative memtable skiplist KV bytes as a cheap, conservative input to Store-side memory admission. It excludes txn-file contents because the payload carries only their references. - `SnapAccess::get_all_files_with_type` is the authoritative shard-level referenced-file enumeration used by metadata-only online dumps. Txn chunks are enumerated directly from the captured `ShardData`: unresolved lock txn files and WRITE_CF memtable txn files. Neither snapshot construction nor file enumeration reparses the `TXN_FILE_REF` property. New immutable file types must be added there and to DFS GC together. - Acquire a MemoryLimiter guard before loading tables to avoid OOM. - With IA enabled, relax table size estimation to avoid double-counting. Why: - Remote coprocessor requests must be self-contained; - Worker reconstruction must not depend on the omitted persisted-version fields for remote-read correctness; - Coprocessor cache validation tokens must come from the origin snapshot, not the worker reconstruction; - Memory limits prevent a single remote read from exhausting the process. ## 9. Iterators And Multi-Source Merging ### 9.1 Iterator Hierarchy - TableIterator scans one SSTable (or L0 sub-table). - ConcatIterator stitches files in a level that are already non-overlapping. - MergeIterator merges multiple sorted iterators into one stream. ### 9.2 Version Iteration - `next()` advances to the next key. - `next_version()` walks older versions of the same key. - Read path uses `seek_to_version` to find the visible version. Why: - Explicit version iteration avoids accidentally skipping MVCC data; - Iterators are reused across row, columnar, and vector pipelines. ### 9.3 Async vs Sync Iterators - Some tables are async-backed (IA or remote). - Iterator traits provide sync and async variants for seek/next. - Call sites must respect `is_next_sync` and `is_next_version_sync`. - WRITE_CF delta scans group memtable and L0 iterators behind one `is_always_sync` merge child, then merge that child with stable-level iterators in the outer async merge. Why: - Async is needed when reads can block on remote fetches; - Dual-mode traits reuse the same logic in both paths. - Grouping synchronous delta-scan sources keeps the outer merge heap bounded by the number of stable levels plus one, instead of the number of memtables, L0s, and stable levels combined. ## 10. LSM Layout And File Types ### 10.1 Levels And CF Layout - There are three CFs: write, lock, extra. - Each CF has a configured number of levels (CF_LEVELS). - L0 files overlap by range; L1+ files are non-overlapping per level. Why: - L0 overlap allows fast flush without immediate reorganization; - L1+ non-overlap enables efficient range scans and compaction decisions. ### 10.2 L0 File Format - An L0 file can contain multiple CFs in a single physical file. - Special L0 format is used for "split L0" where only write CF is present. - L0 footer stores SnapVersion and CF offsets. - Opening a multi-CF L0 validates the minimum outer metadata size, CF count, nondecreasing CF offsets, and each offset against the start of the outer offset table before any SST segment length is computed. Why: - Multi-CF L0 reduces file count and keeps flush atomic; - Split L0 reduces read overhead when only write CF is populated. ### 10.3 L1+ SSTable Format Each SSTable contains: - Data blocks (sorted key/value entries). - Optional old-version data blocks for MVCC history. - Index blocks for data and old data. - BinaryFuse8 filter for key existence checks. - Properties and footer with offsets and magic number. - `entries` and `tombs` properties describe only latest-version data-block entries; old-version data is accounted separately through `old_entries` and old data block bytes. Why: - Separating old-version blocks reduces read amplification for latest reads; - Filter and block cache allow high QPS on read-heavy workloads. Approximate range half-split selection uses only current WRITE SST metadata. It first chooses the middle distinct SST start key strictly inside the requested range. If none exists, it reads the current index of the overlapping loaded SST with the largest `kv_size` and chooses the middle block start key in the range. The fallback revalidates the cached keyspace ID against both outer range bounds before converting an inner candidate back to an outer key; ranges spanning multiple keyspaces are skipped because inner keys do not retain their source keyspace. The index load is cache-neutral on a miss and validates the footer section order against the current SST segment before allocating. SST indexes also require strictly increasing block keys before binary search. Related unloaded SSTs, including multi-CF L0 files whose CF is not recorded separately, index errors, or a range without an internal boundary produce no candidate; row data is never scanned. Auto-IA validates the same footer bounds before reading a property from a standalone SST metadata file. ### 10.4 File Types kvengine uses multiple file types with different lifecycles: - SST (row-store data). - Blob (large values). - Columnar (analytical storage). - Vector index (ANN search). - FTS (derived fulltext index). - Schema file (table schema for columnar/vector). - Txn chunk (lock data segments). Why: - Separate file types allow independent caching and storage-class policies; - Enables different compaction and GC strategies per format. ## 11. Blob Storage Path ### 11.1 BlobRef In LSM - Values above a threshold are stored in BlobTable. - Main LSM stores a BlobRef pointing to (file id, offset, length). - Blob data is fetched on demand during reads. ### 11.2 Blob Table Build - Blob tables are created during compaction or flush when values exceed size. - L0 and L1+ tables reference blob entries for large values. - Blob table size contributes to shard estimated size and GC decisions. Why: - Large values dominate compaction cost when embedded in SSTs; - Blob tables reduce write amplification and support tiering policies. ## 12. ChangeSet: Deterministic Metadata Updates ### 12.1 What A ChangeSet Represents A ChangeSet is a Raft-replicated description of a metadata mutation. Examples include: - Flush (memtable -> L0). - Compaction (L0/Ln -> new Ln). - Destroy range or trim over bound. - Initial flush after split/merge. - Ingest files (bulk import). - Columnar compaction, vector index update, or FTS index update. - Schema updates or storage class changes. ### 12.2 Apply Path Invariants - ChangeSets are applied in order of Raft sequence. - Duplicate sequences are ignored. - Shard version must match, otherwise apply is rejected. - After apply, shard states are refreshed and meta_seq is updated. - Apply reloads the shard pointer because split/merge can replace it. Why: - Applying the same ChangeSet on every replica is the only way to keep LSM metadata consistent in a distributed system. ### 12.3 ChangeSet Materialization - ChangeSets are prepared by loading referenced files from DFS. - `prepare_change_set` resolves file ids and loads local handles. - When prepared, the ChangeSet contains actual table objects. - PrepareType can limit loading to row or columnar files. - Table filters can skip files during restore or partial recovery. - `PrepareOpts::io_type` defaults to `Other` and is passed explicitly through asynchronous table, schema, and transaction-chunk loading. It labels remote-read metrics and synchronous local OS I/O; rate-limiter scheduling remains unchanged. - Thread-local I/O guards stay within the synchronous I/O scope. Why: - Isolates DFS latency/encryption from the apply path; - Enables file filtering for restore/partial recovery. ### 12.4 Ingest And Restore ChangeSets - IngestFiles introduces externally built tables (bulk import). - An ingest id property deduplicates repeated ingestion. - Legacy BR conversion sets the ingest id property and does not run the load-data apply-time overlap check. Level selection uses one applied `ShardData` snapshot for both WRITE_CF L0 and L1+ ranges; without an L0 overlap, ingest stays at the topmost WRITE_CF L1+ level to avoid racing with compaction. Raft preprocess can expose an L0 before async apply installs it in `ShardData`, so conversion waits for that boundary. On timeout, it falls back to one `ShardMeta` snapshot and conservatively uses all-CF physical L0 ranges because L0 metadata does not retain CF ranges. Before apply, rfstore revalidates same-level WRITE_CF overlap so a compaction that raced with conversion cannot violate L1+ ordering. - Load-data omits the ingest id property. Its WRITE_CF level selector ignores L1+ files from other CFs, and rfstore revalidates same-level WRITE_CF overlap when the committed ChangeSet is preprocessed. - Ingest updates can adjust `columnar_table_ids` to force re-derivation of columnar data for that table id. - RestoreShard ChangeSets rebuild shard state from a snapshot and advance the shard version to invalidate earlier ChangeSets. Why: - Bulk ingestion must be idempotent under Raft retries; - Restore must reset both data and metadata in a single deterministic step. ## 13. Recovery And Startup ### 13.1 Metadata Loading - Engine startup reads shard metadata from a MetaIterator. - ShardMeta is reconstructed from ChangeSets recorded in meta storage. - Track blacklisted files to avoid reloading known-bad files. ### 13.2 Loading Parent Shards - During recovery, parent shards are loaded if a child references them. - Keep parent shards only long enough to seed child data. - Parent memtables can be shared to avoid data loss during split recovery. Why: - Split recovery needs parent data until children have their own L0 files; - Prevents missing data on restart in split-in-progress scenarios. ### 13.3 Concurrency Controls - Recovery uses a token bucket (RECOVERY_CONCURRENCY). - DFS loading is globally limited by DfsLoadLimiter. - Each request also has per-request concurrency limits. Why: - Recovery can spawn many DFS reads; - Concurrency limits prevent OOM; - Defaults scale with CPU cores. ### 13.4 Meta Pack And Tiny Metadata - Engine can consume a MetaPack that stores "tiny" SST metadata. - Tiny metadata includes footer, properties, and optional IA segment offsets. - When available, it avoids full footer reads during startup and preparation. Why: - DFS round trips dominate startup with many small files; - Tiny metadata keeps recovery fast without loading full table data. ## 14. Flush Implementation Details ### 14.1 Normal Flush - Normal flush converts one immutable memtable into one or more L0 tables. - L0 can be split into multiple files if configured. - Flush emits a ChangeSet with L0Create entries and properties. - For CMEK/keyspace-aware shards, builders may run only after the task resolves to a Ready key or Disabled state. Pending state must not be treated as Disabled, because that would write plaintext files for encrypted shards. Property handling during flush: - Carry only "need flush" properties. - Filter TXN_FILE_REF to remove finished txn files. - Skip TERM_KEY in initial flush to preserve sequencing. ### 14.2 Initial Flush - Initial flush is for split/merge. - Sets shard range bounds and base_version in metadata. - Creates L0 files representing a clean shard snapshot. ### 14.3 Apply Flush Consistency Checks - Apply flush verifies last memtable SnapVersion matches L0 SnapVersion. - Mismatch is fatal to avoid silent divergence. - After apply, the flushed memtable is freed asynchronously. Why: - Memtable version mismatch implies out-of-order ChangeSet apply or wrong memtable switching. ## 15. Compaction Scheduler And Priorities ### 15.1 Compaction Runner - Compaction is driven by a background runner consuming CompactMsg. - Messages include: Compact, Finish, Applied, Clear, Pause, UnblockKeyspace. - Compaction priority is tracked in shard state. Why: - A dedicated runner serializes decisions and avoids conflicting file selection. ### 15.2 Priority Types CompactionPriority includes: - L0 compaction (score-based). - L1+ compaction (per CF and level). - Major compaction (manual or automatic). - Destroy range, truncate_ts, trim_over_bound. - GC of lock files or extra CF files. - L0 to columnar conversion. - Columnar L0/L1/major/clear. - Vector index update (incremental or rebuild). - FTS maintenance (incremental build, new index build, etc). ### 15.3 Scoring And Ordering - Each priority has a score used for ordering. - Some operations are treated as max priority (destroy range, truncate_ts). - Columnar, vector, and FTS priorities can override row priorities. Why: - Overbound data and range cleanup must preempt size-based compaction; - Columnar, vector, and FTS features must stay consistent with row data. ### 15.4 Safe Point Integration - Compaction uses GC safe point to drop old versions. - Prefer API v2 safe points; fallback to v1 when allowed. - Keyspace id is derived from shard range prefix. Why: - GC must respect transactional visibility across keyspaces; - Safe point is globally coordinated but applied locally during compaction. ### 15.5 Remote Compaction - Compaction can be offloaded to a remote compactor service. - Request is JSON-serialized and includes a required protocol version. - Request carries exported encryption key material when needed. - Response is a ChangeSet applied via the normal Raft path. Fallback behavior: - If no remote compactor is configured, compaction runs locally. - Incompatible remote compactor can trigger a local fallback (if enabled). - Temporary failures trigger retries with exponential backoff. - Non-permanent remote endpoints can be removed after repeated failures. - `CompactRunner`'s `num_compactors` limit applies to the spawned compaction task, regardless of whether that task executes remotely or falls back to local compaction. `Clear` can make a task's result obsolete, but it must not release that task's running slot until the spawned future finishes. Why: - Remote compaction reduces compute pressure on CSE nodes; - ChangeSet boundary keeps metadata deterministic even when work is remote. - Fallback must preserve progress without bypassing the existing compaction concurrency limit. ## 16. Row-Store Compaction Details ### 16.1 L0 Compaction - L0 files overlap; compaction merges them into level 1. - Selection considers overlap size and L0 pressure. - Input size is used to schedule file id allocation. Why: - L0 overlap leads to high read amplification; - L0 compaction is the primary driver for read performance stability. ### 16.2 L1+ Compaction - L1+ files are non-overlapping per level. - Compaction selects a level and merges with overlapping next level files. - Base size and level ratios determine pressure. Why: - L1+ compaction keeps size ratios stable and controls amplification. ### 16.3 Major Compaction - Major compaction merges all levels into a target level. - Can be triggered manually via property. - Used for aggressive GC and reclaiming dead versions. ### 16.4 GC Major Compaction Triggers - Major compaction may trigger from tombstones or old-version data. - Both trigger families use WRITE_CF level 2+ stats and require L2+ `max_ts` to be at or below the GC safe point. - rfstore skips automatic GC compaction when kvengine available disk space is below 5% of capacity, because compaction can create replacement local files before local file GC removes obsolete ones. - Tombstone GC uses tombstone count and tombstone ratio thresholds. - Old-version-data GC uses compressed old data block bytes from the SST footer. It requires both `kvengine.compaction-old-data-size` and `kvengine.compaction-old-data-ratio` to pass. - `kvengine.compaction-old-data-size = 0B` disables the old-data trigger. Why: - Tombstone ratio captures delete-heavy garbage; - Old-version data bytes capture MVCC history that no longer has tombstones but still needs a major compaction to reclaim space. ### 16.5 Trim And Truncate Operations - `truncate_ts` removes data newer than a threshold (restore use-case). - `trim_over_bound` removes data outside shard range. - Both are implemented as specialized compactions to keep metadata consistent. ## 17. Columnar Pipeline And Schema Files ### 17.1 SchemaFile Role - SchemaFile stores table schemas used for columnar and vector building. - Standalone file with checksum and format version. - SchemaFile includes per-table and per-partition storage class specs. - `schema_from_table_info` is the shared TiDB `TableInfo` conversion path for schema manager updates and restore-time schema reconstruction. Why: - Columnar conversion needs schema to interpret row data; - Schema can change independently of data files. ### 17.2 Columnar File Layout - Columnar files store data in column packs. - Each pack is a compressed batch of values. - Columnar file properties persist `origin_fid`, the file ID used to derive encrypted pack IVs. Readers fall back to the current physical file ID for legacy files that predate this property. - Special columns exist for handle and MVCC version. - Metadata section maps tables and columns to pack offsets. - Table metadata can carry pack-level MVCC flags for delete markers and repeated handles. Readers must ignore missing or malformed flags and fall back to the normal handle/version path. - Pack clean read is only valid for L2 scans that do not need merge, do not project handle/version, and fully cover clean packs with a visible table/file max version. - Pack clean read must materialize segments lazily and release each drained pack segment immediately. Prebuilding the whole scan range can retain one pack buffer per segment and inflate request RSS for large scans. - Pack clean read reuses one clean reader and one bounded MVCC reader across segments, rebinding them as the scan moves across packs/files instead of constructing a fresh segment reader every time. - Dummy handle/version buffers used by clean read are internal row-count placeholders. They must never be passed into MVCC filtering or returned for projected handle/version columns. Callers that use internal block handles for scan progress must disable pack clean read. Why: - Columnar layout allows efficient scans and predicate pushdown; - Packs enable vectorized processing and compression efficiency. ### 17.3 Unconverted L0 Tracking - Shards track `unconverted_l0s` for row files not yet converted. - Keep tracking new flushed L0s as long as `columnar_table_ids` is non-empty, even if the schema file is temporarily cleared during schema refresh or non-overlap handling. - Conversion is triggered as a compaction priority when schema is ready. - Columnar conversion validates row data against schema and will fail with `SchemaOutOfDate` (and wait for schema refresh) if the schema is too old to interpret the row correctly (for example, row contains `NULL` for a column marked `NOT NULL` in the local schema). Why: - Decoupling conversion keeps the write path fast; - Tracking unconverted L0 avoids losing visibility in analytical queries. ### 17.4 Schema Updates And Columnar Table Ids - Schema updates arrive as UpdateSchemaMeta ChangeSets. - ShardData keeps `schema_version` and `restore_version` even if the schema file is temporarily unavailable. - `columnar_table_ids` tracks tables that have completed columnar major compaction, and influences conversion, vector index, and FTS scheduling. Why: - Schema changes must be applied deterministically to derived data; - Keeping explicit table id sets prevents partial columnar state. ## 18. Columnar Compaction And Major Compaction ### 18.1 Columnar L0 and L1 Compaction - Columnar L0 files are ordered by SnapVersion (newest first). - Columnar L1 files are also ordered by SnapVersion to preserve MVCC view. - Columnar L2 files are ordered by key range to support non-overlapping scans. - Compaction merges packs and rebuilds metadata indexes. ### 18.2 Columnar Major From SST - Columnar major from SST is used when table id sets need add/clear actions or when a manual major compaction is requested from row data. - Table ids to add or clear are computed from schema changes. - Schema file version is validated to avoid rebuilding with stale schema. - Row-version pruning by `safe_ts` is applied on the merged MVCC stream before row decode in the row-to-columnar reader path, so versions older than the first `version <= safe_ts` boundary (for the same handle, across all inputs) are skipped instead of being schema-checked. Why: - Columnar files must reflect the latest schema; - Rebuild from SST avoids compounding errors from incremental conversion. ### 18.3 Columnar Major From Columnar - Purely columnar major compaction merges columnar files without row data. - Used for space reclamation when schema is stable. ### 18.4 Clear Columnar - Clearing columnar removes all columnar files and resets L2 snap version. - Used when columnar data is corrupted or intentionally dropped. Why: - Columnar data is derived; - It can be rebuilt from row data when needed. ## 19. Vector Index Pipeline ### 19.1 Vector Index Files - Each vector index file is tied to a table id, index id, and column id. - Files are ordered by SnapVersion and then file id. - Index data is stored in a usearch format with metadata sections. ### 19.2 Update Triggers - Vector index updates are scheduled via compaction priority. - Updates can be incremental or full rebuild. - Schema version changes can force rebuilds. Why: - Vector indices are derived from columnar data and must track MVCC visibility. - Rebuild is necessary when schema or vector column definition changes. ### 19.3 MVCC And Deletes - Vector index includes versions and nulls bitmap to represent deletions. - SnapVersion tracks the visibility horizon for the index. Why: - Vector results must match MVCC visibility of row data. ## 20. Full-Text Search (FTS) Pipeline ### 20.1 File and Level Structure - FtsPackedFile for mix-table data. - FtsDedicatedFile for single-table data. - Both FtsPackedFile and FtsDedicatedFile are organized by "LogicalPartition" (table id + index id + optional user key). - Fts Levels: - L0 (PackedFile): comes from columnar L0 files → index quickly - L1 (PackedFile): merged from multiple FTS L0 files → read fewer files - L2 (DedicatedFile): target at single logical partition, optimize for read, use compaction similar as Tantivy (size tiered) - Brute-force is used where index structures are not yet available. ### 20.2 Tracking And Watermarks - tracked_indexes: set of (table id, index id). - Used in read path to select indexed reads and to resolve the physical layout for indexed and brute-force execution. All FTS queries require a resolved tracked layout; an untracked index is not executable. - Used in compact path - know whether index need to build incrementally, and whether index need to build from full historical data, and what index need to clear. - New tracked indexes choose their field layout when tracked metadata is first built from `FullTextIndexDef.initial_field_layout`. Schema conversion marks TiDB's legacy `parser_type`-only definitions as `FtsFieldLayoutLegacyBody`, while definitions with canonical `index_fields` use `FtsFieldLayoutIndexFieldOrdinalString`, including a single column with one parser. Both forms keep normalized `FullTextIndexFieldDef` entries. Old schema files without an initial layout safely default to legacy. - After tracked metadata is written, that recorded layout becomes the source of truth for both read and write paths; the schema's initial layout no longer affects that index. - Legacy tracked metadata may contain only the physical `"body"` layout and omit index-field definitions. Match-expression reads hydrate that missing logical field definition from the current table schema before resolving `(column id, analyzer)` references; non-empty tracked field definitions remain authoritative and are validated against the schema. - Remote compaction requests from older binaries may omit tracked-layout metadata entirely. In that compatibility path, the worker must reconstruct legacy single-field `"body"` layout from schema definitions so old requests continue to build the same physical index format. - For `FtsFieldLayoutIndexFieldOrdinalString`, tracked metadata preserves the complete `FullTextIndexDef.index_fields` definitions and their order. Tantivy field names are `field_idx.to_string()`, and Tantivy field ids are the corresponding positions in that ordered list. A column ID may appear more than once when the schema assigns multiple analyzers to the same column; each occurrence remains a distinct physical field by ordinal. - Tracked `index_fields` are physical index-format metadata. Their order, analyzer options, and BM25 settings are immutable for a (table id, index id), and stable-index builders and delta writers must use the same tracked values. Field-local tokenizer names intentionally omit analyzer parameters, so a schema mismatch must fail or clear and rebuild the index under new metadata; postings produced by incompatible analyzers must never be merged. - `l0_snap_version`: a monotonic watermark for "indexed columnar L0", only effective for tracked indexes. - Used in read path - know what data is indexed - Used in compact path - know what columnar L0 need to index incrementally. - `pending_columnar_l0_ids` marks columnar L0s that are explicitly not covered (mainly after merge). - L2 DedicatedFile `snap_version`: a monotonic ordering key used by FTS read path for MVCC visibility checks across L2 files. - Allocated when emitting a new L2 segment from L0/L1 merge. - Allocated when emitting a new L2 segment from L2 merge compaction. - Must be preserved for rewrites that do not introduce newer MVCC versions (trim_over_bound / destroy_range / truncate_ts). - L2 merge compaction must merge the file with the largest snap_version in the LP, because the output file receives a fresh snap_version and becomes the newest file for that LP. Invariants: - Watermark separates historical build from incremental maintenance; - Pending ids prevent incorrect coverage assumptions after merge. - Columnar L0 compaction must not compact away columnar L0 files that are not tracked by FTS yet (newer than watermark or explicitly pending), because the incremental indexer and read fallback rely on those L0 files. - Scored searches aggregate BM25 document frequencies per complete Tantivy term and token totals per physical field across every selected FTS source before comparing or pruning cross-source results. ### 20.3 `safe_ts` Pruning And Full-L2 Compaction - FTS merges use `safe_ts` to prune old MVCC versions and delete markers. - When a logical partition (LP) already has L2 files, incremental pipelines may keep tombstones below `safe_ts` temporarily to avoid resurrecting older live versions that still exist in other L2 files. - Tombstones below `safe_ts` are reclaimed by a *full L2 compaction* that merges all L2 files of an LP; it can be triggered by `fts_build_options.l2_full_compact_dead_ratio` and `fts_build_options.l2_full_compact_dead_bytes`. ### 20.4 Query Evaluation And Brute-Force Fallback - A resolved `clara_fts::Query` owns the physical Tantivy schema, tokenizer registry, source-column-to-field bindings, and the fields and terms needed by BM25. Indexed search and brute-force fallback therefore use the same lowered Tantivy query and analyzer configuration. - `CloudColumnarReaders` shares only the immutable request-level `FtsQueryInfo`. `SnapAccessCore::new_fts_reader` resolves that request into a fresh, complete `QueryCore` for each concrete table/shard reader. Clones inside that reader's indexed/fallback chain deliberately share the same lazily prepared BM25 statistics. Queries and prepared BM25 statistics must not be shared across table readers or shard snapshots. - Whole-table fallback builds the complete query from the tracked index's physical layout even when columnar reads are disabled or the indexed reader cannot be constructed. Legacy and match-expression queries both require a resolved tracked layout; an unresolved layout is an error and must never be inferred from the number of fields in the current schema. The legacy layout is exactly one text `"body"` field using a standard or multilingual analyzer. Ordinal layout supports one or more physical fields and is required for multi-field definitions. - Brute-force readers materialize a Tantivy document from every source column referenced by the resolved query. One source column may populate multiple physical fields when the index defines multiple analyzers for that column. Source columns omitted from the TableScan projection recover their `ColumnInfo` only from the complete table schema. Match-expression requests identify source fields in their expression metadata and may leave the legacy `FtsQueryInfo.columns` field empty; request-level columns must not replace physical schema metadata. - `BruteScoredSearcher` prepares each document once with Tantivy's single-document preparer. The prepared document supplies both evaluator input and per-field token/term statistics, so the read path must not tokenize the source text independently. - When indexed shard statistics have been prepared, fallback rows use those aggregate BM25 statistics and remain score-compatible with indexed hits. Otherwise, the single-document evaluator uses valid empty BM25 statistics; scores do not depend on other fallback rows or block boundaries. - In conditional fallback, a non-NULL score means the indexed path already evaluated the row. Such rows are never re-evaluated and are retained regardless of the score's numeric value; in particular, zero is a valid filter-mode match score. Only rows with a NULL score invoke the single-document evaluator, and rows that remain unmatched are filtered. Consequently, query kinds unsupported by Tantivy's single-document evaluator fail only when an actual fallback row is encountered. - The conditional reader's physical input contains only source columns referenced by the resolved query expression, with their original table nullability, plus a nullable internal score. TableScan columns used only for projection must not be read before filtering. - Query source columns are consumed and discarded by the conditional reader. Its output contains only handle/version and, for scored queries, a non-NULL score. `FtsJoinReader` materializes every final projected physical column by exact handle/version lookup in row storage. Indexed source placeholders are therefore ignored values and never use NULL as a control signal. ## 21. TxnFile And Lock Handling ### 21.1 Why TxnFile Exists - Lock data can be large (for example, secondary locks). - Storing lock payloads directly in the LSM inflates memtables and L0 files. - TxnFile stores lock data in separate chunked files. ### 21.2 TxnFile Structure - TxnFile is identified by (shard_id, shard_ver, start_ts). - Contains a list of TxnChunks, each with an index over key ranges. - TxnCtx records the key bounds and metadata used for reads. ### 21.3 TxnChunkManager - Manages local or in-memory storage of txn chunks. - Loads txn chunks from DFS when not cached locally. - Maintains a GC worker when chunks are in-memory. - `Shard.unapplied_pending_files` tracks not-yet-applied txn-file chunk ids by raft log sequence while rfstore is waiting for async preparation or apply. Why: - Lock data is transient and does not need full SST semantics; - Separate management allows specialized retention policies. Cleanup rules that matter to correctness: - Normal apply progress can clean txn-file pending records by watermark because the applier has already proven older log indexes are obsolete. - When an entire applier lifetime is abandoned (for example re-registration after snapshot reset), rfstore may bulk-remove all txn-file pending records for that old lifetime. ### 21.4 Lock Reads - Read path checks txn files for lock CF before memtables. - This ensures lock visibility is correct even when lock data is offloaded. ## 22. Split, Prepare-Merge, Commit-Merge ### 22.1 Split Flow - rfstore issues a split ChangeSet after Raft commit. - kvengine creates new shards for each split range. - Merged-engine replay may pass a small skip list of child shard IDs whose rfstore metadata write was intentionally skipped; kvengine must not materialize those stale child shards during `Engine::split`. - New shards share parent files until initial flush completes. - Split metadata must stay in exactly one encryption format per request. The code intentionally avoids mixing legacy-key and CMEK-metadata split payloads instead of building a combined compatibility bundle. Key steps: 1. Switch memtables and clear pending background tasks. 2. Build new shard ranges and inherit relevant properties. 3. Split memtables, L0s, L1+ tables, blobs, columnar files, vector indexes. Split/retain FTS levels and files by range overlap. 4. Trigger initial flush to make each shard independent. 5. Filter columnar table ids by table id range derived from the shard bounds. 6. Keep only columnar and vector index files that overlap the new range. Keep only FTS files that overlap the new range and tracked tables. Why: - Sharing files avoids expensive re-compaction during split; - Initial flush ensures future compactions are shard-local. ### 22.2 Merge Preconditions `check_merge` enforces constraints before merge: - Both source and target must be initial flushed. - No txn file locks in source or target. - No unconverted L0s (columnar consistency). - Consistent encryption keys if in same keyspace. - Storage class specs must be compatible. Why: - Merging with pending lock data or unconverted L0s can lose visibility. - Encryption and storage class mismatches can break keyspace isolation. ### 22.3 Prepare Merge - `prepare_merge` switches memtables and creates a new shard version. - New shard inherits parent id to enable initial flush. - Background tasks are cleared to avoid using stale versions. - `new_shard_version` must carry forward cached store-usage summaries when it reuses the same persisted files; prepare-merge does not recompute shard stats before the new version becomes visible to store-level usage collection. ### 22.4 Commit Merge - Merge combines source and target shard data and properties. - Data can be cleared for source or target when keyspace boundaries differ. - Range bounds are updated to cover both shards. - Base version is advanced to be higher than both source and target. Why: - When inner key offsets differ (different keyspaces), old data can violate ordering; - Clearing prevents corruption; - Advancing base version prevents SnapVersion regressions. ### 22.5 Rollback Merge - If merge is rolled back, a new shard version is created. - Shard is marked initial flushed to continue normal operation. ## 23. Storage Class And Infrequent-Access (IA) Tier ### 23.1 StorageClassSpec - StorageClassSpec describes tiering rules per table or partition. - Encoded in schema metadata and persisted as a shard property. - Shard pending ops track the current storage class spec. ### 23.2 IA Files - IA files keep only metadata locally; data blocks are fetched on demand. - IA files are built for SST, columnar, vector, blob, and FTS file types. - Segment offsets are used to align remote fetches to block boundaries. Why: - Reduces local disk footprint and suits object storage backends; - Preserves correctness while trading latency for cost. ### 23.3 IaAutoFile And Transitions - IaAutoFile can transition between storage classes based on rules. - Decision uses timestamps and configured transit rules. - Transition logic is designed to be safe for concurrent reads. ### 23.4 IA Manager - IaManager tracks cached segments using an S3-FIFO policy. - `dynamic-capacity` preserves the original queue-capacity modes. When enabled, periodic IA data-size reports derive the main target from data size and `cache-cap-to-total-data-size-ratio`, then scale the small target proportionally. When disabled, reports ignore data size and ratio and use the configured memory and disk capacities as the queue targets. Cloud worker does not run the reporting task and therefore keeps its fixed startup caps. - Online memory cap, disk cap, `cache-cap-to-total-data-size-ratio`, and `cache-capacity-shrink-step` updates publish one immutable desired configuration snapshot. They do not immediately change live queue caps, grow memory admission, schedule a trim, or persist the IA manifest. The periodic data-size reporter loads one complete snapshot and is the sole runtime writer that performs those effects. Multiple updates before a report coalesce into the latest snapshot without a capacity-state mutex. - Capacity growth reaches the latest target in one report. Capacity shrink moves the main live cap toward the latest target by at most `cache-capacity-shrink-step` per report; the default is `4GiB`. The small queue uses the same step scaled by the configured small/main maximum-cap ratio, with a one-segment minimum so it always makes progress. Repeated reports converge to the latest target and each intermediate live-cap change bounds the work submitted by the existing full trim. - The ratio-derived main queue target is rounded down to a `1 GiB` boundary before the existing segment-count floor and disk-cap ceiling are applied. Startup and online updates both reject non-finite or negative ratios. Online ratio updates are retained but remain inert while `dynamic-capacity` is disabled. - Online memory-cap updates must be at least `64 * segment-size`, matching the dynamic small-queue floor. Startup keeps the existing behavior, and a disk-only update does not revalidate an unchanged memory cap. - Online cap updates require IA to be enabled at startup; they cannot enable a disabled IA manager. - A new remote segment larger than the live small queue is rejected before its remote read starts. A read that already passed this check is allowed to finish across an applied online shrink; S3-FIFO then trims it against the new live cap. Existing and restart-restored main-store segments remain readable and are governed by normal main-queue trimming. - Remote-read memory admission starts at the configured small queue cap plus the larger of one full small queue or the fixed concurrent-read buffer. The extra queue-sized headroom guarantees that one largest legal item can finish loading and trigger eviction even when the small queue is full. Admission grows when the periodic reporter applies a memory-cap increase but remains at its high-water mark after decreases; active S3-FIFO trimming enforces the lower live cache cap without revoking in-use permits. - IA remote reads and local segment-cache writes have independent byte-rate limiters. They protect object storage and local disk without throttling non-IA DFS loads. In TiKV server, these two limiter fields are online configurable through the `kvengine.ia` config manager: `kvengine.ia.remote-read-max-throughput` and `kvengine.ia.local-write-max-throughput`. Each cap is per TiKV process, defaults to `500MiB/s`, and can be set to `0` for unlimited throughput. The shared limiter refill period is configured at startup through `kvengine.ia.io-rate-limiter-refill-period`; it defaults to `100ms` and is not online configurable. Set it to `10ms` to keep the old hard-coded burst window. - Storage-class transitions have a separate per-process limiter under `kvengine.ia`: `ia-to-standard-transition-max-regions` bounds concurrent regions, and `ia-to-standard-transition-wait-timeout` is a stuck-region guard for each admitted region. Normal completion is reported back to the rfstore schema worker by the apply side after the storage-class change set is applied. IA to Standard materialization additionally uses `ia-to-standard-transition-max-throughput` to cap remote full-file reads. Region admission is owned by the single-threaded rfstore schema worker; kvengine stores the dynamic cap and applies the file-level remote-read limiter. The remote-read limiter is charged at file granularity using DFS file size before the full-file read starts. If the size lookup fails, the transition falls back to unbounded loading for that file instead of failing the whole reload. - Stores a local "meta file" for quick access to table metadata. Why: - Segment-level caching enables consistent performance without keeping full files locally. ### 23.5 Storage Class Updates - Storage class changes are applied via a ChangeSet with `STORAGE_CLASS_KEY`. - ChangeSet can include reloaded L1+ tables with updated storage class. - If the new spec is unspecified, the property is cleared and only reloads are applied. Why: - Storage class changes are external to data correctness but must remain consistent across replicas and restarts. ## 24. DFS Layer, Local Files, And Caching ### 24.1 DFS Abstraction - Dfs trait provides create/read/remove and runtime access. - Local and in-memory DFS implementations are available. - S3-backed DFS supports storage class operations. - `MultiPartUploader` owns DFS multipart upload state, bounds concurrent part uploads, and aborts unfinished uploads on drop. Multipart DFS primitives are async; the uploader keeps a synchronous buffering API for callers that only feed bytes and later hand off completion ownership. - S3 retry delays use one bounded exponential-backoff helper. It saturates defensively so malformed settings cannot overflow the retry calculation. - S3 multipart upload uses the same AWS CRC32 checksum representation as single-object uploads. Uploaded part metadata carries the optional checksum through to the complete request. ### 24.2 File Naming And Directories - File ids are allocated by an IdAllocator shared across components. - Local directories can be multiple; files are distributed by hash. - Global file lock prevents multiple engines from opening the same dir. - rfstore fail-fast tracks local-disk forward progress per registered local dir, not as one global kvengine signal. - Only local temp-file publish completions are allowed to feed that signal today, including `LocalFs::create` and temp-file publish flows used by prepare and IA helpers after `rename`. - Publish progress from subdirectories is attributed to the longest registered local-dir ancestor, so IA paths under `/ia/...` still refresh the correct local-dir signal without masking sibling local dirs. - The signal intentionally represents "same-dir kvengine local temp-file publish is still making forward progress", not "foreground WriteBatch apply succeeded". Why: - Distribution across dirs reduces single-disk contention; - File locking avoids destructive concurrent usage. - Per-dir progress avoids letting writes on one mounted disk mask a hang on a different kvengine-mounted disk. ### 24.3 File Caches - BlockCache caches data blocks, index blocks, and BinaryFuse8 filters for SST and L0 tables. - FdCache caches open file descriptors. - ColumnarMetaCache caches columnar metadata sections. - Vector and FTS caches reduce repeated IO for derived index reads. - Meta file cache reduces repeated reads of table metadata. Why: - Caches reduce object storage latency and local disk churn; - Metadata caching is critical for IA and remote reads. ### 24.4 File Blacklist - Files can be marked as blacklisted by the metadata source. - Blacklisted file ids are tracked by the engine and consulted during load. - This prevents repeated attempts to load known-bad files. ## 25. Resource Control And Flow Throttling ### 25.1 Store And Region Limiters - Store limiter throttles write throughput based on memtable usage. - Region limiter controls per-shard memtable and L0 growth. - Submitted memtable size and estimated live memory usage are separate stats; store/region memory limiters and the memtable-memory gauge use live usage. - Throttling uses soft and hard limits with a dynamic speed limit. Why: - Throttling prevents memory spikes and reduces tail latency; - Avoids compaction backlog explosions during heavy writes. ### 25.2 DFS Load Limiter - DFS loading is bounded by a global semaphore. - Per-request concurrency also limits file load parallelism. - Required full-file buffers use the configured quota and may wait. Opportunistic `/dfs` cache responses use separate, non-blocking admission: a bounded allowance uses current required reservations, while larger cache reservations preserve the complete future required quota in the process headroom check. - Callers hold the byte reservation for the source buffer's lifetime. `/dfs` responses transfer it to the HTTP body so completion or drop releases it. - DFS read throughput can be capped via `dfs.max-read-throughput` (implemented in body streaming loops). - The limiter is per-DFS-client-instance (shared by all callers using the same `Arc`), not globally enforced across the entire process if multiple DFS clients are constructed. - `RemoteCachedDfs` treats admission HTTP 503 as a cache miss and falls back to its inner DFS. - IA has separate remote-read and local-write throughput limiters under `ia`. Use those when the goal is to protect online traffic by slowing IA without slowing non-IA DFS materialization. - IA to Standard transition reads use an extra file-level limiter before DFS load memory is acquired, so throttling does not hold DFS load memory while sleeping. - The transition limiter returns a remote-read charge. Commit the charge only after the remote load succeeds; dropping an uncommitted charge refunds the consumed limiter bytes. - Currently only S3 backend is supported. ### 25.3 Low Space Threshold - Engine tracks available space and can reject operations when below the threshold. - Engine also exposes `is_critical_low_space()` for the fixed 5% guard used by automatic GC compaction; it reads capacity from `tikv_util::sys::disk` and uses the engine's latest available-space snapshot. ## 26. Observability And Debug Aids ### 26.1 Metrics - Compaction trigger and result counters track scheduler behavior. - Block-cache lookup and miss counters cover data blocks, filters, and indexes. Quick-cache probes increment lookup once, backing initialization increments miss once, and disabled caches increment neither, so interval deltas preserve `0 <= miss <= lookup`. - Value cache hit/fill metrics indicate caching efficiency. - IA metrics track sync reads and cache behavior. - Stats APIs provide per-shard and per-level size information. - `Engine::get_all_shard_stats()` is a full diagnostic walk: it clones shard state and recomputes per-file/per-level stats on demand. Do not use it in periodic heartbeat or metrics paths. - Periodic store-usage collection should use the lightweight cached stats path (`collect_store_usage_stats`), which reuses summaries refreshed by `Shard::refresh_states()` and only reads live memtable state on demand. ### 26.2 Debug Features - `debug-trace-mem-table` logs memtable switches and writes. - Failpoints exist for flush and compaction paths. - Meta pack counters report usage of tiny metadata optimization. ### 26.3 Diagnostic Tips - When investigating compaction issues, check shard compaction priority. - For inconsistent reads, compare memtable SnapVersion vs L0 SnapVersion. - For columnar issues, verify schema file version and columnar table ids. ## 27. Common Failure Modes And Pitfalls - Memtable version mismatch during flush apply indicates missed switch. - Applying ChangeSet out of order can silently drop files. - Columnar conversion without schema file leads to inconsistent analytics. - Merging shards with unconverted L0s risks losing columnar visibility. - Using different encryption keys within a keyspace breaks merge safety. - Clearing data on merge is required when keyspace boundaries differ. - Loading files without respecting DfsLoadLimiter can trigger OOM. - Skipping TXN_FILE_REF persistence can break lock visibility. - Mixing inner and outer key domains can make range filters and lock accounting disagree. A common red flag is `Key::from_raw(&key)` where `key` is an `InnerKey` from a kvengine iterator or txn-file path. It compiles because `InnerKey` derefs to `[u8]`, but the result is an outer/raw `Key` built from stripped inner bytes. ## 28. Maintainer Playbook For Changes ### 28.1 Adding A New File Type - Extend FileType enum and suffix mapping. - Implement metadata parsing for table footer and properties. - Ensure ChangeSet preparation and apply can load and install the file. - Add IA and storage class handling if the file can be tiered. ### 28.2 Extending ChangeSet - Add protobuf fields in kvenginepb. - Update prepare_change_set to load referenced files. - Update apply_change_set with sequencing and state refresh logic. - Update ShardMeta reconstruction and persistence. ### 28.3 Modifying Compaction - Update CompactionPriority scoring and ordering. - Ensure input selection avoids overlap and preserves ordering. - Include safe point handling where MVCC data is removed. - Add metrics for trigger and result. ### 28.4 Changing Columnar, Vector, Or FTS Logic - Ensure schema version gating is correct. - Update snapshot and restore handling to keep derived data consistent. - Validate table id range calculations for split/merge. - For FTS, preserve watermark/pending-id invariants and coordinate with columnar compaction priorities. ### 28.5 Storage Class And IA Changes - Keep StorageClassSpec and pending ops consistent. - Ensure transitions are idempotent and safe for concurrent reads. - Update stats and metrics to reflect new tiers. ### 28.6 Changing Key Encoding, TxnFile, Or CDC Boundary Logic - Review every conversion between `InnerKey` and `txn_types::Key`. Treat `Key::from_raw` near `InnerKey`, `OwnedInnerKey`, `Iterator::key()`, txn-file iterators, or keyspace-prefix code as suspicious until proven safe. - For CDC and downstream-facing paths, event rows and lock accounting must use TiKV raw outer keys; in API v2, those keys include the keyspace prefix. If an input path starts from `InnerKey`, reconstruct the correct outer key before updating `Delegate.lock_tracker`, building event rows, or comparing with observed ranges. - Prefer explicit helper names that state the domain, for example `*_outer_key`, `*_inner_key`, or `build_*_event_key`, over generic local names like `key` when both domains are present in one function. - When changing key APIs, prefer semantic newtypes or explicit conversion helpers over generic byte-slice constructors. Avoid adding implicit conversions, such as `Deref`, to invariant-bearing key wrappers unless the wrapper is truly byte-transparent in every caller context. - Mechanical review recipe: - `rg -n "Key::from_raw\\(" components src tests` - `rg -n "Key::from_raw\\(&?key\\)|Key::from_raw\\(.*iter\\.key\\(\\)" components src tests` - `rg -n "use .*InnerKey|InnerKey::from_inner_buf|OwnedInnerKey|Iterator::key" ` - If `ast-grep` is available: `sg -l rust -p 'Key::from_raw(&$KEY)' components src tests` - When a suspicious site is intentional, leave a short comment explaining which domain the bytes are in and why `Key::from_raw` is correct there. Otherwise, reconstruct the outer key before calling `Key::from_raw`, or keep the code in the `InnerKey` domain. ## 29. Glossary - ChangeSet: Raft-replicated metadata mutation describing LSM changes. - Shard: LSM tree bound to a key range and a Raft region. - SnapVersion: `base_version + data_sequence` (u64 watermark) tagging file snapshots. - L0: Level 0 SSTs, overlapping by key range. - L1+: Non-overlapping SST levels. - Columnar: Derived analytic storage built from row data. - Vector index: Derived ANN index built from columnar data. - IA: Infrequent-access tier using metadata-only local files. - TxnFile: Lock data stored outside the main LSM tree.