## Module 1: Initialization of TiDB Cluster and data ### Step 1. Connect to the TiUP control machine + Click here - [Terminal 1](ssh://monitor1-ssh#1) to open a new terminal connected to the virtual machine assigned for your hands-on lab. ### Step 2. Check the status of the TiDB cluster + Verify that a TiDB cluster with TiProxy is up and running, and there are 1 `tidb` and 3 `tikv` servers in the cluster. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.74.236.1:2379/dashboard Grafana URL: http://10.90.4.196:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.196:9093 alertmanager 10.90.4.196 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.196:3000 grafana 10.90.4.196 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.178:2379 pd 10.90.2.178 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.196:9090 prometheus 10.90.4.196 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.140:4000 tidb 10.90.1.140 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.169:20160 tikv 10.90.1.169 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.88:20160 tikv 10.90.1.88 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.99:20160 tikv 10.90.1.99 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.196:6000 tiproxy 10.90.4.196 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 9 ``` > [!TIP] > Please note! If Module 1 is enabled, it is recommended that there should not be too long of a time interval (more than 10 minutes) between each step from the beginning to Step 7 of Module 1. As this may cause the PD component to automatically adjust data distribution, which could prevent achieving the desired results. ### Step 3. Create the sysbench configuration ```bash cat << EOF > sysbench_config mysql-host=${HOST_MONITOR1_PRIVATE_IP} mysql-port=6000 mysql-user=root mysql-password= mysql-db=sbtest time=120 threads=14 report-interval=5 EOF cat sysbench_config ``` ```output mysql-host=${HOST_MONITOR1_PRIVATE_IP} mysql-port=6000 mysql-user=root mysql-password= mysql-db=sbtest time=120 threads=14 report-interval=5 ``` ### Step 4. Connect to the TiDB cluster, create an empty database `sbtest` ```bash mysql -h ${HOST_MONITOR1_PRIVATE_IP} -P 6000 -uroot ``` ```output Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 0 Server version: 8.0.11-TiDB-v${LAB:TIDB_VERSION} TiDB Server (Apache License 2.0) Community Edition, MySQL 8.0 compatible Copyright (c) 2000, 2023, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> ``` ```sql CREATE DATABASE sbtest; EXIT; ``` ```output Query OK, 0 rows affected (0.11 sec) Bye ``` ### Step 5. Prepare schema data + Prepare the `sysbench` data, it will take around 5 to 10 minutes to finish. While waiting for the data import process to complete, you can proceed to the next step to learn about the relevant monitoring metrics that will be used in the subsequent steps. ```bash sysbench --config-file=sysbench_config oltp_read_only --tables=10 --table-size=3000000 prepare ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest2'... Creating table 'sbtest8'... Creating table 'sbtest5'... Creating table 'sbtest6'... Creating table 'sbtest7'... Creating table 'sbtest9'... Creating table 'sbtest1'... Creating table 'sbtest4'... Creating table 'sbtest3'... Creating table 'sbtest10'... Inserting 3000000 records into 'sbtest10' Inserting 3000000 records into 'sbtest1' Inserting 3000000 records into 'sbtest5' Inserting 3000000 records into 'sbtest7' Inserting 3000000 records into 'sbtest4' Inserting 3000000 records into 'sbtest8' Inserting 3000000 records into 'sbtest2' Inserting 3000000 records into 'sbtest6' Inserting 3000000 records into 'sbtest3' Inserting 3000000 records into 'sbtest9' ``` ### Step 6. Learn about monitoring related to TiKV scaling out/in + This step will introduce the monitoring metrics related to TiKV scaling out/in. If participants are already familiar with these monitoring metrics, please skip to Step 7. Log in to the [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000). Participants are encouraged to keep the monitoring pages open from this step and set the monitoring time range to the last 5 or 15 minutes, while changing the refresh rate to 5 seconds. ![70%](https://lab-static.pingcap.com/images/2025/7/24/9c787410ab425dd7eb7e7297cc46711ec63ccc22.png) + To understand the total data volume of the TiDB cluster, log in to the [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000), go to `Home`, then select `tidb-demo`, and subsequently choose `tidb-demo-PD`. In the upper section of `tidb-demo-PD`, you will see `Current storage size`, which indicates the total data volume of the TiDB cluster, `Normal Stores`, which indicates the current number of TiKV nodes, `Number of Regions`, which shows the current number of Leaders, and `Current peer count`, which represents the total number of regions. ![70%](https://lab-static.pingcap.com/images/2025/7/24/f5789ae1bf0d0012aadb2efd25ebb019aced7755.png) + To monitor the progress of scaling out/in, in the upper section of `tidb-demo-PD`, you can see the following information: `Online store progress` indicates the progress of the balance after the current scaling-out command is executed. `Offline store progress` indicates the progress of the balance after the current scaling-in command is executed. `Left time` represents the estimated completion time for the balance during the current scaling operation. `Current scaling spee` indicates the rate of balance during the scaling operation, measured in MB/s. ![70%](https://lab-static.pingcap.com/images/2025/7/24/33ca440cdffeacf0f0c761c01569128f4095568b.png) + To observe the relevant settings for scaling in and out in PD, go to the lower section of `tidb-demo-PD` and open the `Cluster` tab. You will see that the `PD scheduler config` panel lists the settings related to scheduling in the PD component, while the `Store limit` panel displays the Store limit settings for all TiKV nodes. ![70%](https://lab-static.pingcap.com/images/2025/7/24/abe91e672688c726ab01f31d5a328ea16c519d59.png) + To monitor the efficiency of the PD component in generating scheduling commands, go to the lower section of `tidb-demo-PD` and open the `Operator` tab. You will see that the `Scheduler operator create` panel displays the efficiency of operator generation by the PD component, while the `Scheduler operator finish` panel shows the efficiency of the execution of the operators generated by the PD component. ![demo-013-metrics-04.png](https://lab-static.pingcap.com/images/2025/7/24/166bae9064df01fac2ea5ebcbab5da006b303d0f.png) + To monitor the schedules activated by the PD component, go to the lower section of `tidb-demo-PD` and open the `Scheduler` tab. You will see that the `Scheduler` panel displays the currently active schedulers in the PD component. ![70%](https://lab-static.pingcap.com/images/2025/7/24/0dc48adba12cd619c1504728f9e4af549ddbd5a5.png) + To monitor the data distribution in the TiKV cluster, go to the lower section of `tidb-demo-PD` and open the `Statistics - balance` tab. You will see that the `Store available` panel indicates the available data volume for each TiKV node, while the `Store available ratio` panel shows the percentage of available storage space for each TiKV node. ![70%](https://lab-static.pingcap.com/images/2025/7/24/8f9c5b2cd165337e7b1685e8a905bc0e32ddf951.png) + To monitor the IO load of the TiKV cluster, log in to [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000), navigate to `Home`, then select `tidb-demo`, and subsequently choose `tidb-demo-TiKV-Details`. Open the `Cluster` tab, where you will find the `CPU` panel displaying the CPU usage for each TiKV node, and the `IO utilization` panel showing the IO usage for each node. ![70%](https://lab-static.pingcap.com/images/2025/7/24/4731b3d2183b8d351eae72b449cb547bf23f52ec.png) ### Step 7. Config TiDB Cluster + Wait for the data import in `Step 5` to complete, and then check the scheduler configuration of the PD component. The first execution will take 1 to 2 minutes because it needs to download the `PD ctl` component. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 scheduler show ``` ```output The component `ctl` version v${LAB:TIDB_VERSION} is not installed; downloading from repository. Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 scheduler show [ "evict-slow-store-scheduler", "balance-region-scheduler", "balance-leader-scheduler", "balance-hot-region-scheduler" ] ``` + Delete the `balance-region-scheduler` from the PD component to prepare for the following module. The reason for removing the `balance-region-scheduler` will be explained in the next step. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 scheduler remove balance-region-scheduler ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 scheduler remove balance-region-scheduler Success! ``` + Confirm that the `balance-region-scheduler` has been deleted from the PD component. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 scheduler show ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.3.229:2379 scheduler show [ "balance-leader-scheduler", "balance-hot-region-scheduler", "evict-slow-store-scheduler" ] ``` + Log in to [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000), navigate to `Home`, then select `tidb-demo`, and subsequently choose `tidb-demo-PD`. In the lower section of `tidb-demo-PD`, open the `Scheduler` tab,where you will see that the `balance-region-scheduler` has been removed from the PD component. ![85%](https://lab-static.pingcap.com/images/2025/7/24/d39ff1e27d81d847844986d80fba45691a8036db.png) + Change the `region-schedule-limit` variable of the PD component to `0` in preparation for the following module. We will explain why the `region-schedule-limit` variable should be set to `0` in the next step. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 config set region-schedule-limit 0 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 config set region-schedule-limit 0 Success! ``` + Confirm that the `region-schedule-limit` variable has been updated to `0` in the PD component. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 config show ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 config show { "replication": { "enable-placement-rules": "true", "enable-placement-rules-cache": "false", "isolation-level": "", "location-labels": "", "max-replicas": 3, "strictly-match-label": "false" }, "schedule": { "enable-cross-table-merge": "true", "enable-diagnostic": "true", "enable-heartbeat-breakdown-metrics": "true", "enable-heartbeat-concurrent-runner": "true", "enable-joint-consensus": "true", "enable-tikv-split-region": "true", "enable-witness": "false", "high-space-ratio": 0.7, "hot-region-cache-hits-threshold": 3, "hot-region-schedule-limit": 4, "hot-regions-reserved-days": 7, "hot-regions-write-interval": "10m0s", "leader-schedule-limit": 4, "leader-schedule-policy": "count", "low-space-ratio": 0.8, "max-merge-region-keys": 540000, "max-merge-region-size": 54, "max-movable-hot-peer-size": 512, "max-pending-peer-count": 64, "max-snapshot-count": 64, "max-store-down-time": "30m0s", "max-store-preparing-time": "48h0m0s", "merge-schedule-limit": 8, "patrol-region-interval": "10ms", "patrol-region-worker-count": 1, "region-schedule-limit": 0, "region-score-formula-version": "v2", "replica-schedule-limit": 64, "slow-store-evicting-affected-store-ratio-threshold": 0.3, "split-merge-interval": "1h0m0s", "store-limit-version": "v1", "switch-witness-interval": "1h0m0s", "tolerant-size-ratio": 0, "witness-schedule-limit": 4 } } ``` + Return to the [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000)'s `tidb-demo-PD` section. In the lower section of `tidb-demo-PD`, open the `Operator` tab, where you will see that there are currently no operators generated in the `Scheduler operator create` panel and the `Scheduler operator finish` panel. ![85%](https://lab-static.pingcap.com/images/2025/7/24/41b41866585731f9bd597a4e19180e3465da3fff.png) + Set the `store limit` for all TiKV nodes to `1` in preparation for the following steps. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit all 1 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION} /ctl pd -u http://10.90.2.178:2379 store limit all 1 Success! Set store limit successfully. ``` + Confirm that all items of the `store limit` for all TiKV nodes have been set to `1`. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit { "1": { "add-peer": 1, "remove-peer": 1 }, "4": { "add-peer": 1, "remove-peer": 1 }, "7": { "add-peer": 1, "remove-peer": 1 } } ``` + Return to the `tidb-demo-PD` section of [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000). In the lower section of `tidb-demo-PD`, open the `Cluster` tab, where you can see that in the `PD scheduler config` panel, `region-schedule-limit` is set to `0`, and in the `Store limit` panel, all items are set to `1`. ![85%](https://lab-static.pingcap.com/images/2025/7/24/95665cbee2ec67dd9104e8508364a2379b9d6ead.png) + At this point, the configuration and data for the TiDB cluster have been prepared, and you can begin the scaling out/in. ### Step 8. Explanation of "Removing the `balance-region-scheduler` from the PD component, setting the `region-schedule-limit` variable of the PD component to `0`, and modifying the `Store limit` to `1`" + This section uses the Scaling out of TiKV as an example to illustrate how PD and TiKV cooperate to maintain a balanced distribution of data during this process. In the diagram below, all TiKV nodes and the Leader role's Region replicas send relevant information to the PD component by heartbeat, allowing PD to understand the distribution of data across the various TiKV nodes. ![85%](https://lab-static.pingcap.com/images/2025/7/24/aba924da1fff5a1a82ae02d05393a50a4358b9d6.png) + When a new TiKV node joins, the PD component receives its heartbeat information. Once the PD component determines that the current data distribution is unbalanced, its `balance-region-scheduler` component will create multiple `balance-region-operator` which is one kind of opertors, each containing specific steps for achieving data balance. These steps will be sent to the relevant Leader role's Region replicas, and then the corresponding Region Group will begin the process of balancing. ![85%](https://lab-static.pingcap.com/images/2025/7/24/52c6ded417e9c8bcbae6621023f4b39349c3b5a9.png) + In the previous step, after removing the `balance-region-scheduler` from the PD component, even if the PD component detects an unbalanced data distribution, it will not be able to issue a `balance-region-operator`. Therefore, after scaling out, it will be necessary to add the `balance-region-scheduler` for data balancing to commence. + The `region-schedule-limit` indicates the number of various operators that the PD component can generate at the same time. If it is set to `0`, the PD component will also be unable to generate `balance-region-operator`. It must be set to a value greater than `0` for the data balancing operation to be activated. + The `Store limit` defines the maximum number of Operator operations executed by TiKV per minute as directed by PD, thereby allowing for adjustment of the speed of data balancing actions. + In the following modules and steps, phenomena resulting from "removing the `balance-region-scheduler` from the PD component, setting the `region-schedule-limit` variable of the PD component to `0`, and modifying the `Store limit` to `1`" will be observed, and corresponding adjustments will be made. + For more information on `region-schedule-limit`, please refer to: + For more information on `Store limit`, please refer to: + If the PD's `balance-region-scheduler` component successfully creates a `balance-region-operator`, the first step will notify the selected Leader role's Region replicas to copy the data and transfer it to the new TiKV node (TiKV Node 4), where a replica of the Learner role will be established. ![85%](https://lab-static.pingcap.com/images/2025/7/24/e7874791484e5f440de4f9a5e9763043c23399f9.png) + Next, the PD component's `balance-region-operator` will proceed to the second step, switching the roles of the Learner and Follower replicas within the same Raft Group. As shown, the Region3 replica in the newly added TiKV node (TiKV Node 4) becomes a Follower, while the Region3 replica in the existing TiKV node (TiKV Node 1) becomes a Learner. ![85%](https://lab-static.pingcap.com/images/2025/7/24/223b9ba88e98391b1cd31e419b7558c78d75a576.png) + Then, the PD component's `balance-region-operator` will move on to the third step, removing the Learner role replica from the Raft Group. ![85%](https://lab-static.pingcap.com/images/2025/7/24/6e66f0bc2fd1eddfe263bf1f7800da9fbc543c99.png) + Finally, with the collaboration of the PD component and TiKV component, the data balancing operation after the scaling out is completed. ![85%](https://lab-static.pingcap.com/images/2025/7/24/f7bfe57d10aaaee6ffb7a3e6f1f6467f42c71514.png) + For the scaling in operation, the migration principle of Region replicas within the same Raft Group is the same as that of the scaling out. The difference is that the operator name issued by the PD component is `replace-rule-offline-peer` or `replace-rule-offline-leader-peer`. For more detailed content, please refer to the course materials.