# File Based Transaction

**File Based Transaction** is a feature which pre-build all data of a transaction as one or more files (i.e. **txn file**). Then the **"txn files"** are written to **Cloud Storage Engine** as a whole (through **DFS**). This approach will greatly reduce overhead compared to writing row by row.

This document outlines the critical code path associated with the **File Based Transaction** feature.

## Write Flow

### Clients

On transaction committing, clients (e.g. TiDB) submit all data of the transaction to **tikv-worker**. Data is separated to files (called **txn chunk**) according to size. For each **txn chunk**, **tikv-worker** will return an unique **txn chunk id**. See [`chunkWriterClient.buildChunk`](https://github.com/tikv/client-go/blob/074830f9cb5b9735fd425ce45e8a90f69f917186/txnkv/transaction/txn_file.go#L887).

After all **txn chunks** are built, clients commit the transaction by 2PC. The **txn chunk ids** will be submit to **Cloud Storage Engine** in prewrite request. See [tidbcloud/pull/26](https://github.com/tidbcloud/kvproto/pull/26).

### `tikv-worker`

**tikv-worker** receive transaction data, and build **txn chunks**. See [`cloud_worker::txn_chunk::create_txn_chunk`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/cloud_worker/src/txn_chunk.rs#L117).

### Cloud Storage Engine

#### Scheduler

* Prewrite
  
  On received prewrite request of transaction with **txn file**, **scheduler** will first **prepare** the txn chunks (pull data files from **DFS**, see [`Scheduler::build_and_scheduler_txn_file_cmd`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/src/storage/txn/scheduler.rs#L1758)). Then check conflict with existing latches and locks of other transactions (see [`GlobalLatches::acquire`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/src/storage/txn/region_latch.rs#L31) and [`TxnFileCommand::process_prewrite`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/src/storage/txn/commands/txn_file.rs#L313)). If no conflict is found, **schduler** constructs and propose the `TxnFileRef` (see **Data Structures** chapter below) carried by `CustomRequest` (see [`modifies_to_requests`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/cloud_server/src/raftkv.rs#L654)).

  New txn-file prewrites are also bounded by `storage.max-txn-file-locks-per-region`.
  The scheduler does a fast-path check on the applied region snapshot before chunk preparation, and
  `TxnFileCommand::process_prewrite` rechecks on the later execution snapshot to close concurrent
  admission races. Same-transaction updates for an already-visible txn-file lock remain allowed.

* Other Commands

  Process of other commands are similar to normal transactions. They all execute the procedures of transaction, then modify and propose the `TxnFileRef` if necessary. See `process_xxx` of [`TxnFileCommand`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/src/storage/txn/commands/txn_file.rs#L33).

  Non-prewrite txn-file commands keep the txn-file metadata observed when the command is built as an explicit optional snapshot. The snapshot may start as a live lock and later become finalized commit/rollback metadata in the same command. If the build snapshot and process snapshot disagree, the command returns a retryable region error before proposing any write: `EpochNotMatch` for shard-version changes, otherwise `StaleCommand`. If both snapshots miss the txn-file lock, existing missing-lock transaction behavior is preserved and no synthetic whole-region `TxnFileRef` is persisted.

  The scheduler still treats missing-lock txn-file commands as write commands through a scheduler-only latch marker. That marker serializes txn-file commands with one another but does not block normal per-key commands because it has no known key coverage and is never written to raft or kvengine metadata.

#### Preprocess

During preprocess, `ShardMeta` will trace alive transactions (neither committed nor rollback), which is used to block the region to be splited or merged as source. The reason of blocking is that the latch of **txn file** in **scheduler** is of region granularity (for scalability), so the region id must not be changed before transaction is committed or rollback.

Alive transactions are kept in [`ShardMeta::txn_file_locks`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/meta.rs#L42), and persisted by property [`TXN_FILE_LOCKS`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/shard.rs#L119).
For every live txn-file lock keyed by `start_ts`, `ShardMeta::txn_file_locks` must keep the latest applied `lock_val_prefix`. Heartbeat and `check_txn_status` updates therefore overwrite the previous prefix so split checks and active resolve reconstruct the current TTL and `min_commit_ts`.
Leader maintenance ticks proactively schedule active resolve for old-enough
txn-file locks from this metadata, using the preprocessed region start key only
for region routing. Split request paths themselves still reject immediately and
do not enqueue background resolve work. There is no per-region cooldown after a
finished attempt; retries are paced by `maintenance_tick_interval` plus the
existing worker/admission/concurrency limits.

After preprocessed, the `TxnFileRef` will be prepared (see [`Applier::handle_prepare_txn_file`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/rfstore/src/store/apply.rs#L1623)) and ready for applying.

See [`preprocess_txn_file_ref`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/rfstore/src/store/peer.rs#L2355) and [`ShardMeta::merge_txn_file_ref`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/meta.rs#L1010).

#### Apply

During apply, for primary region, first write corresponding record to `EXTRA_CF` (see [`Applier::exec_txn_file_ref`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/rfstore/src/store/apply.rs#L503)). Then pass the `TxnFileRef` to **kvengine** through property of `TXN_FILE_REF` in write batch.

Process of applying in **kvengine**:

1. Merge `TxnFileRef` to currenty property of `TXN_FILE_REF`.
  
   Value of the property is a collection of `TxnFileRef` of transactions. The "Merge" action will overwrite `TxnFileRef` value of the same `start_ts`, or add to the collection if not existed. See [`Engine::merge_txn_file_ref`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/write.rs#L313).
  
   The property of `TXN_FILE_REF` is mainly for persistence, which will be written to `ShardMeta` along with mem table flush. On recovery, **kvengine** will recover [`ShardData::lock_txn_files`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/shard.rs#L965) from value of this property.

2. Merge `TxnFileRef` to `ShardData::lock_txn_files`.

   Value of `ShardData::lock_txn_files` is a collection of `TxnFile` of alive transactions. It's used to check conflict or return locks of the transactions.

   The `TxnFile` (see **Data Structures** below) is loaded by [`TxnChunkManager`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/txn_chunk_manager.rs#L33) according to `TxnFileRef`.

   The "Merge" is similar with merge of property `TXN_FILE_REF`, overwrite value for the same `start_ts`, or add if not existed.

   See [`Engine::merge_lock_txn_files`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/write.rs#L326).

3. Write to Mem Table

   If the transaction is committed, the `TxnFile` will be added to current writable mem table. As each mem table can have only one `TxnFile`, the writable mem table will be switched immediately.

   See [`Engine::write_txn_file_ref`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/write.rs#L272).

4. Flush

   After switched, data of **txn file** will be flush to `L0` during flush process. Then the lifetime of the **txn file** is ended.

## Read Flow

Keys of alive transactions or all key-values of committed transactions can be retrieved by table iterator with `cf = LOCK_CF` or `cf = WRITE_CF`, respectively. See [`SnapAccess::new_table_iterator`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/read.rs#L558).

Note that for alive transactions, table iterator only return keys of locks. Other fields of locks can be acquired by [`TxnFile::get_lock_val_prefix()`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/table/txn_file.rs#L185). `ShardMeta::txn_file_locks` keeps the same latest lock prefix per `start_ts`, and [`TxnFileLocks::get_key_errors`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/util.rs#L270) reconstructs `LockInfo` from that metadata.

## Data Structures

### TxnFileRef

[`TxnFileRef`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvenginepb/src/changeset.proto#L155) is the **reference** to **txn file**, which contains reference of data (i.e. id list of **txn chunks**) and meta data of the transaction.

### TxnFile

[`TxnFile`](https://github.com/tidbcloud/cloud-storage-engine/blob/32a1e0037a7da11604ec4dfeba5f434f58d902bb/components/kvengine/src/table/txn_file.rs#L130) is the memory object of a **txn file**, which contains meta data and all data of the transaction.