## Module 3: Enable balancing for the TiKV cluster and adjust its rate

+ In this module, participants will address the issue of the TiKV cluster not initiating balancing after scaling out in the previous module and will adjust its rate.

### Step 1. Investigate whether the balance scheduling feature is enabled in PD and enable it if necessary

+ Add the `balance-region-scheduler component` to PD.

   ```bash
   ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 scheduler show
   ```

   ```output
   Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 scheduler show
   [
   "evict-slow-store-scheduler",
   "balance-leader-scheduler",
   "balance-hot-region-scheduler"
   ]
   ```

   ```bash
   ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 scheduler add balance-region-scheduler
   ```

   ```output
   Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 scheduler add balance-region-scheduler
   Success! The scheduler is created.
   ```

   ```bash
   ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 scheduler show
   ```

   ```output
   Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 scheduler show
   [
   "balance-region-scheduler",
   "balance-leader-scheduler",
   "balance-hot-region-scheduler",
   "evict-slow-store-scheduler"
   ]
   ```

+ Log in to [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000), go to `Home`, then select `tidb-demo`, and subsequently choose `tidb-demo-PD`. In the lower section of `tidb-demo-PD`, open the `Scheduler` tab, where you can see that the `balance-region-scheduler` component has been added to PD.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/2ad5a2ececf1dc8cac303e828270f34539539187.png)

+ Wait for 30 seconds, and in the upper section of `tidb-demo-PD`, you can see that `Online store progress` is still `No Data`, `Left time` is `N/A`, and `Current scaling speed` is `0 MB/s`. This indicates that balancing has not yet started, so further investigation will continue.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/adb4348802beb9d2722289e55e99d477906e6bc8.png)

### Step 2. Investigate whether `region-schedule-limit` is set to `0` and make the necessary adjustments

+ In the lower section of `tidb-demo-PD`, open the `Operator` tab, where you can see that both the `Scheduler operator create` panel and the `Scheduler operator finish` panel show that no operators for `balance-region-scheduler` have been created or executed.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/30ff56c740955c7475e0c10380d510e33a371e11.png)

+ Adjust the `region-schedule-limit` to `2048`.

   ```bash
   ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 config set region-schedule-limit 2048
   ```

   ```output
   Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 config set region-schedule-limit 2048
   Success!
   ```

+ Return to the [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000) for `tidb-demo-PD`, wait for 30 seconds, and in the lower section of `tidb-demo-PD`, open the `Operator` tab. You will see that both the `Scheduler operator create` panel and the `Scheduler operator finish` panel show that operators for `balance-region-scheduler` have been created and executed.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/cd10d370eb1503fb94dbf7f852b70901556ab7e8.png)

+ Wait, and in the upper section of `tidb-demo-PD`, you can see that `Online store progress`, `Left time`, and `Current scaling speed` all have values. This indicates that balancing has started, but the rate of balance is currently very slow, and `Left time` shows that it will take a long time to complete. Next, it is necessary to increase the balance rate.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/19b35aa528cf6ce53e18c83c652246dd38d60444.png)

+ For the reasons why data balancing is activated after adding the `balance-region-scheduler` and setting the `region-schedule-limit`, please refer to Step 8 of Module 1.

+ In the `Cluster` tab of `tidb-demo-TiKV-Details`, you can see that the IO utilization of each TiKV node is currently quite low.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/c42203151614b13996cf5d789006b2f35e48ba10.png)

### Step 3. Speed up the Balance after TiKV cluster scaling out

+ In the upper section of `tidb-demo-PD`, wait until the `Online store progress` exceeds 10% (Note: It is recommended to initiate this step when `Online store progress` is between 10% and 20%, as exceeding 20% may not allow enough time for the subsequent steps). Then, set the `store limit` for all TiKV nodes from 1 to 5 to increase the rate at which TiKV executes balancing.

   ```bash
   ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit all 5
   ```

   ```output
   Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit all 5
   Success! Set store limit successfully.
   ```

+ Confirm that the `store limit` for all TiKV nodes has been set to `5`.

   ```bash
   ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit
   ```

   ```output
   Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit
   {
      "1": {
         "add-peer": 5,
         "remove-peer": 5
      },
      "4": {
         "add-peer": 5,
         "remove-peer": 5
      },
      "576": {
         "add-peer": 5,
         "remove-peer": 5
      },
      "7": {
         "add-peer": 5,
         "remove-peer": 5
      }
   }
   ```

+ Return to the [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000) for `tidb-demo-PD`, wait for 10 seconds, and in the `Operator` tab of `tidb-demo-PD`, you will see that the rate of `balance-region-scheduler` in both the `Scheduler operator create` panel and the `Scheduler operator finish` panel begins to rise. This is due to the increased efficiency of TiKV in executing operators.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/cfac45c51fe095c95ee4be3df34f923994e4ec52.png)

+ In the `Cluster` tab of `tidb-demo-TiKV-Details`, you can see that the IO utilization of each TiKV node is starting to rise. This indicates that the newly added TiKV nodes and the nodes selected as sources will allocate IO resources to balancing, which may impact the existing business load.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/c6d214e68fd635c0438e7915dc837424814fd5cd.png)

+ Wait and observe the upper section of `tidb-demo-PD`. You will see that the percentage of `Online store progress` is rising more rapidly, the value of `Left time` is decreasing, and the `Current scaling speed` is increasing. This indicates that the increase in store limit can accelerate the rate of balancing.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/2e3208e8936fd688605f28771aa8346ee0b4ed23.png)

### Step 4. Slow down the Balance after TiKV cluster scaling out

+ In the upper section of `tidb-demo-PD`, wait until the `Online store progress` exceeds 50% (Note: It is recommended to initiate this step when `Online store progress` is between 50% and 60%, as exceeding 60% may not allow enough time for the subsequent steps). Then, begin to reduce the rate of balancing.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/1bf8559bc784ca742d1fec69d1e80f509b3a145f.png)

+ Set the `store limit` for all TiKV nodes to `1` to reduce the rate at which TiKV performs balancing.

   ```bash
   ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit all 1
   ```

   ```output
   Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit all 1
   Success! Set store limit successfully.
   ```

+ Confirm that the `store limit` for all TiKV nodes has been successfully set to `1`.

   ```bash
   ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit
   ```

   ```output
   Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit
   {
      "1": {
         "add-peer": 1,
         "remove-peer": 1
      },
      "4": {
         "add-peer": 1,
         "remove-peer": 1
      },
      "576": {
         "add-peer": 1,
         "remove-peer": 1
      },
      "7": {
         "add-peer": 1,
         "remove-peer": 1
      }
   }
   ```

+ Return to the [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000) for `tidb-demo-PD`, wait for 10 seconds, and in the `Operator` tab of `tidb-demo-PD`, you will see that the rate of `balance-region-scheduler` in both the `Scheduler operator create` panel and the `Scheduler operator finish` panel begins to decline. This is due to the decreased efficiency of TiKV in executing operators.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/247a958930cd5b9cb9c197cd48a45d6609cc2c6b.png)

+ In the `Cluster` tab of `tidb-demo-TiKV-Details`, you can see that the `IO utilization` of each TiKV node is starting to decline. This indicates that the newly added TiKV nodes and the nodes selected as sources will use fewer IO resources for balancing, which may help avoid impacting existing business loads. However, the balancing rate after the scaling out will become slower.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/27fd326a8002e4739f98bbdba4e5a5c538fb5666.png)

+ Wait and observe the upper section of `tidb-demo-PD`. You will notice that the rate of increase in the percentage of `Online store progress` begins to slow down, the value of `Left time` increases, and the `Current scaling speed` starts to decline. This indicates that lowering the store limit can reduce the rate of balancing.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/2c3f49858b070480f48a2cab770bae758732cdc1.png)

### Step 5. Wait for the balancing process to complete after the scale out of the TiKV cluster

+ Continue to wait and observe the changes in `Online store progress`, `Left time`, and `Current scaling speed` in the upper section of `tidb-demo-PD` until `Online store progress` shows `No data`, indicating that the scaling out has officially completed.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/d744c40ef5e03fe774d2abb27ec1145153fd65ed.png)

+ You can also try adjusting the `store limit` to a larger value, such as `20`, to speed up the balancing process.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/800a2d65b07bf6bb389bc4361446833640c98b22.png)

Finally, when `Online store progress` shows `No data`, it indicates that the scaling out has completed.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/60f7c026e5b42076f2d766b1f8a3656d55f476d5.png)

+ In the `Cluster` tab of `tidb-demo-TiKV-Details`, locate the `IO utilization` panel and expand the time range to cover the entire scaling out process. You will notice that the load conditions of the TiKV cluster vary according to the `store limit` value.

  ![70%](https://lab-static.pingcap.com/images/2025/7/24/8420e7c08ed1d71db8bd7cd9475d4b1834da1474.png)