## 模块 2：并行将数据导入 TiDB + 在本练习中，你将使用 `TiDB Lightning` 并行地将多个表导入 TiDB。下图展示了如何使用 `TiDB Lightning` 并行地将分片数据库和表导入 TiDB。在我们的示例中，`my_table_1` 包含所有 ID 为奇数的记录，而 `my_table_2` 包含所有 ID 为偶数的记录 ![60%](https://lab-static.pingcap.com/quick-demo/TiDBLightningLabToplogy_aa_003_20220725.png) ### 步骤 1. 列出数据文件目录的文件 ```bash ls ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-Data-into-TiDB-in-Parallel/my_table_1 ``` ```output metadata my_db.my_table_1.000000000.sql my_db.my_table_1-schema.sql my_db-schema-create.sql ``` ### 步骤 2. 检查数据源文件 + 查看 `my_db-schema-create.sql`, `my_db.my_table_1-schema.sql` 和 `my_db.my_table_1.000000000.sql` 文件，了解它们的结构。 ```bash cat ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-Data-into-TiDB-in-Parallel/my_table_1/my_db-schema-create.sql ``` ```output /*!40101 SET NAMES binary*/; CREATE DATABASE `my_db` /*!40100 DEFAULT CHARACTER SET latin1 */; ``` ```bash cat ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-Data-into-TiDB-in-Parallel/my_table_1/my_db.my_table_1-schema.sql ``` ```output /*!40101 SET NAMES binary*/; CREATE TABLE `my_table_1` ( `id` int(11) NOT NULL, `name` varchar(32) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1; ``` ```bash cat ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-Data-into-TiDB-in-Parallel/my_table_1/my_db.my_table_1.000000000.sql ``` ```output /*!40101 SET NAMES binary*/; > INSERT INTO `my_table_1` VALUES (1,'tom'), (3,'frank'); ... ``` ### 步骤 3. 打开一个新终端（终端 2） + 点击这里 - [终端 2](ssh://monitor1-ssh#2) 打开一个新的终端，连接到为你的实践实验分配的虚拟机。 ### 步骤 4. 在终端 2 中，登录到 `PD1` ```bash ./ssh-to-pd1.sh ``` ```output ... ... ``` ### 步骤 5. 在终端 2 中，解压数据 ```bash unzip -u stage/tidb-admin-dataset-v2.zip -d stage/ ``` ```bash ls ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-Data-into-TiDB-in-Parallel/my_table_2 ``` ```output metadata my_db.my_table_2.000000000.sql my_db.my_table_2-schema.sql my_db-schema-create.sql ``` ### 步骤 6. 在终端 2 中，检查数据文件 ```bash cat ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-Data-into-TiDB-in-Parallel/my_table_2/my_db-schema-create.sql ``` ```output /*!40101 SET NAMES binary*/; CREATE DATABASE `my_db` /*!40100 DEFAULT CHARACTER SET latin1 */; ``` ```bash cat ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-Data-into-TiDB-in-Parallel/my_table_2/my_db.my_table_2-schema.sql ``` ```output /*!40101 SET NAMES binary*/; CREATE TABLE `my_table_2` ( `id` int(11) NOT NULL, `name` varchar(32) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1; ``` ```bash cat ~/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-Data-into-TiDB-in-Parallel/my_table_2/my_db.my_table_2.000000000.sql ``` ```output /*!40101 SET NAMES binary*/; INSERT INTO `my_table_2` VALUES (2,'jack'), (4,'tony'); ... ``` ### 步骤 7. 在终端 1 中，查看 `solution-lightning-p1.toml` + 在 [终端 1](ssh://monitor1-ssh#1) 中，下面显示的 IP 地址可能会与你的不同。 ```bash cat solution-lightning-p1.toml ``` ```output [lightning] # The concurrency number of data. It is set to the number of logical CPU # cores by default. When deploying together with other components, you can # set it to 75% of the size of logical CPU cores to limit the CPU usage. # region-concurrency = 1 # Logging level = "info" file = "tidb-lightning-p1.log" [tikv-importer] # Whether to allow importing data to tables with data. The default value is `false`. # When you use parallel import mode, you must set it to `true`, because multiple TiDB Lightning instances are importing the same table at the same time. incremental-import = true # Sets the backend to the "local" mode. backend = "local" # Sets the directory of temporary local storage. sorted-kv-dir = "/tmp/sorted-kv-dir" [mydumper] # Local source data directory data-source-dir = "/home/ec2-user/stage/TiDB-Administration-exercise-data-v2/exercise-lightning/Importing-Data-into-TiDB-in-Parallel/my_table_1" # Only import tables if these wildcard rules are matched. See the corresponding section for details. filter = ['*.*', '!mysql.*', '!sys.*', '!INFORMATION_SCHEMA.*', '!PERFORMANCE_SCHEMA.*', '!METRICS_SCHEMA.*', '!INSPECTION_SCHEMA.*'] [[routes]] schema-pattern = "my_db" table-pattern = "my_table_*" target-schema = "my_db" target-table = "my_table" [tidb] # Configuration of any TiDB server from the cluster. host = "" port = 4000 user = "ltask" password = "q1w2e3R4_" # Table schema information is fetched from TiDB via this status-port. status-port = 10080 # Address of any PD server from the cluster. pd-addr = "10.90.2.84:2379" ``` ### 步骤 8. 在终端 2 中，安装 `TiUP` 和 `Lightning` - 切换到 [终端 2](ssh://monitor1-ssh#2) 中执行以下命令。 ```bash ./01-install-tiup.sh ``` ```bash source .bash_profile ``` ```bash tiup mirror set tidb-community-server-7.5.1-linux-amd64 tiup install tidb-lightning ``` ### 步骤 9. 在终端 1 中，将 `solution-lightning-p2.toml` 文件复制到 `pd1` - 切换到 [终端 1](ssh://monitor1-ssh#1) 中执行以下命令。 ```bash scp -i ~/.ssh/pe-class-key-${LAB:REGION_NAME}.pem solution-lightning-p2.toml ec2-user@${HOST_PD1_PRIVATE_IP}:/home/ec2-user/ ``` ```output solution-lightning-p2.toml 100% 720 678.5KB/s 00:00 ``` ### 步骤 10. 在终端 1 中，连接到 TiDB 并删除 `my_db` 数据库 ```bash mysql -h ${HOST_DB1_PRIVATE_IP} --port 4000 -u root ``` ```bash DROP DATABASE IF EXISTS my_db; ``` ```output mysql> DROP DATABASE IF EXISTS my_db; ``` ```bash EXIT ``` ```output mysql> EXIT; Bye ``` ### 步骤 11. 在终端 1 中，使用 Lightning 导入数据 + 使用 `nohup` 命令启动 `tidb-lightning` 进程（按 Enter 让 `tidb-lighting` 在后台运行）。 + 请同时执行步骤 11 和步骤 12。 ```bash nohup tiup tidb-lightning -config solution-lightning-p1.toml > nohup.out & ``` ```output [1] 4256 # nohup: ignoring input and redirecting stderr to stdout ... [1]+ Done nohup tidb-lightning -config solution-lightning-p1.toml > nohup ``` ### 步骤 12. 在终端 2 中，使用 Lightning 导入数据 - 切换到 [终端 2](ssh://monitor1-ssh#2) 中执行以下命令。 ```bash nohup .tiup/bin/tiup tidb-lightning -config solution-lightning-p2.toml > nohup.out & ``` ```output [1] 4256 # nohup: ignoring input and redirecting stderr to stdout ... [1]+ Done nohup tidb-lightning -config solution-lightning-p2.toml > nohup ``` ### 步骤 13. 在两个终端中，查看 Lightning 的输出 + 在[终端 1](ssh://monitor1-ssh#1)中，查看 `nohup.out` 文件的内容，以验证 `TiDB Lightning` 已成功退出。 ，然后输入 `:q!` 退出终端： ```bash cat nohup.out ``` ```output +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | # | CHECK ITEM | TYPE | PASSED | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 1 | Source csv files size is proper | performance | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 2 | checkpoints are valid | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 3 | table schemas are valid | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 4 | Cluster is available | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 5 | Lightning has the correct storage permission | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 6 |sorted-kv-dir:/tmp/sorted-kv-dir and data-source-dir:/home/ec2-user/sample/sql/my_table_1 are in the same disk, may slow down performance | performance | false | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 7 | local disk resources are rich, estimate sorted data size 85B, local available is 5.149GiB | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 8 | Cluster available is rich, available is 54.46GiB, we need 255B | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 9 | Cluster doesn't have too many empty regions | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 10 | Cluster region distribution is balanced | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ tidb lightning exit successfully ``` + 如果发生任何错误，查看 `tidb-lightning-p1.log` 文件的内容，以确定错误原因。 ```bash cat tidb-lightning-p1.log ``` + 你可能需要在再次导入数据之前删除 lightning 检查点。以下命令假设检查点存储在 `/tmp/` 中。 ```bash rm /tmp/tidb_lightning_checkpoint.pb ``` + 在[终端 2](ssh://monitor1-ssh#2)中，查看 `nohup.out` 文件的内容，以验证 `TiDB Lightning` 已成功退出。 ```bash cat nohup.out ``` ```output Verbose debug logs will be written to tidb-lightning-p2.log +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | # | CHECK ITEM | TYPE | PASSED | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 1 | Source csv files size is proper | performance | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 2 | checkpoints are valid | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 3 | table schemas are valid | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 4 | Cluster is available | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 5 | Lightning has the correct storage permission | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 6 |sorted-kv-dir:/tmp/sorted-kv-dir and data-source-dir:/home/ec2-user/sample/sql/my_table_2 are in the same disk, may slow down performance | performance | false | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 7 | local disk resources are rich, estimate sorted data size 85B, local available is 7.173GiB | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 8 | Cluster available is rich, available is 54.46GiB, we need 255B | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 9 | Cluster doesn't have too many empty regions | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ | 10 | Cluster region distribution is balanced | critical | true | +----+----------------------------------------------------------------------------------------------------------------------------------+-------------+--------+ tidb lightning exit successfully ``` ### 步骤 14. 在终端 1 中，连接到 TiDB 并查询 `emp.sal` table ```bash mysql -h ${HOST_DB1_PRIVATE_IP} --port 4000 -u root ``` ```bash SELECT COUNT(*) FROM my_db.my_table; ``` ```output mysql> SELECT COUNT(*) FROM my_db.my_table; +----------+ | COUNT(*) | +----------+ | 40156 | +----------+ 1 row in set (0.01 sec) ```