# `TiDB Lightning` Architecture

![`smart`](TiDBLightningArchitecture_aa_003_20221102.png)

-------------------------------------------------------------
Before importing data, `TiDB Lightning` switches the `TiKV` cluster to `import mode` to optimize write efficiency and stop automatic compression.

`TiDB Lightning` creates the schema and tables in the target database and acquires their metadata.

Each table is split into contiguous chunks, so that data from large tables (larger than 200 GB) can be imported incrementally in parallel.

`TiDB Lightning` prepares an "`engine file`" for data engine for each block to handle `key-value` pairs. Data engine is an engine for sorting actual row data.

`TiDB Lightning` then reads the `SQL` dump concurrently, converts the data source into `key-value` pairs with the same encoding as `TiDB`, and then writes the sorted `key-value` pairs to the local temporary storage file.

When an engine file has finished writing data, `TiDB Lightning` starts splitting and scheduling the target `TiKV` cluster data, and then imports the data into the `TiKV` cluster.
- There are two types of engine files: the `data engine` and the `index engine`, each corresponding to two `key-value` pairs: row data and secondary indexes. 
- Normally row data is sorted, while secondary indexes are unordered. Therefore, `data engine` files are uploaded as soon as the corresponding block is written, while all `index engine` files are imported only after all blocks of the entire table have been encoded.

After all the engine files associated with the entire table are imported, `TiDB Lightning` compares the checksum of the local data source and the downstream cluster to ensure that the imported data is intact, and then `TiDB` analyzes the newly added data to optimize future operations. Also, `TiDB Lightning` adjusts the `AUTO_INCREMENT` value to prevent conflicts when data is added later.

The `AUTO_INCREMENT ID` for a table is derived from the upper bound estimation of the number of rows, which is proportional to the total size of the table data file. As a result, the final self-incrementing `ID` is usually much larger than the actual number of rows. This is normal because self-incrementing `ID` is not necessarily assigned continuously in `TiDB`.

After all the steps are completed, `TiDB Lightning` automatically switches `TiKV` back to `normal mode`, after which the `TiDB` cluster can provide external services normally.