## Module 4: Scale In The TiKV cluster + In this module, participants will manually remove one TiKV node from the TiKV cluster to perform scaling in. Subsequently, participants will control the rate of scaling in in the next module. ### Step 1. Set the `store limit` for all TiKV nodes to `1` + Switch to [Terminal 1](ssh://monitor1-ssh#1)，and set the `store limit` for all TiKV nodes to `1`. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit all 1 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit all 1 Success! Set store limit successfully. ``` + Confirm that the `store limit` for all items on all TiKV nodes has been set to `1`. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit { "1": { "add-peer": 1, "remove-peer": 1 }, "4": { "add-peer": 1, "remove-peer": 1 }, "576": { "add-peer": 1, "remove-peer": 1 }, "7": { "add-peer": 1, "remove-peer": 1 } } ``` ### Step 2. Remove one TiKV server from the cluster + On [Terminal 1](ssh://monitor1-ssh#1), scale in the storage layer by removing one random TiKV server. ```bash RANDOM_TIKV_NODE=`~/.tiup/bin/tiup cluster display tidb-demo | grep 20160 | sed 's/ .*$//' | head -n 1` ~/.tiup/bin/tiup cluster scale-in tidb-demo --node ${RANDOM_TIKV_NODE} -y ``` ```output ... Scaled cluster `tidb-demo` in successfully ``` + Destroy the removed TiKV node. ```bash ~/.tiup/bin/tiup cluster prune tidb-demo -y ``` ```output ... Destroy success ``` + Confirm the status of the TiDB cluster. At this point, you will find that the deleted TiKV node is in a `Pending Offline` state because not all regions on that node have been removed. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.74.236.1:2379/dashboard Grafana URL: http://10.90.4.196:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.196:9093 alertmanager 10.90.4.196 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.196:3000 grafana 10.90.4.196 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.178:2379 pd 10.90.2.178 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.196:9090 prometheus 10.90.4.196 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.140:4000 tidb 10.90.1.140 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.169:20160 tikv 10.90.1.169 20160/20180 linux/x86_64 Pending Offline /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.192:20160 tikv 10.90.1.192 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.88:20160 tikv 10.90.1.88 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.99:20160 tikv 10.90.1.99 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.196:6000 tiproxy 10.90.4.196 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 10 ``` + Return to the [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000) for `tidb-demo-PD`. In the upper part of `tidb-demo-PD`, you can see values for `Offline store progress`, `Left time`, and `Current scaling speed`. This indicates that the scaling in process is ongoing, and the next step is to increase the scaling in rate. ![70%](https://lab-static.pingcap.com/images/2025/7/24/651768f831fc8f0850c53a533ab4529e3908197f.png) ### Step 3. Speed up the scaling in of TiKV cluster + In the upper part of `tidb-demo-PD`, wait until the `Offline store progress` exceeds 10% (Note: It is recommended to initiate this step when `Offline store progress` is between 10% and 20%, as exceeding 20% may not leave enough time for subsequent steps). Then, set the `store limit` for all TiKV nodes to `5` to increase the rate at which TiKV executes Operators. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit all 5 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit all 5 Success! Set store limit successfully. ``` + Confirm that the `store limit` for all items on all TiKV nodes has been set to `5`. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit { "1": { "add-peer": 5, "remove-peer": 5 }, "4": { "add-peer": 5, "remove-peer": 5 }, "576": { "add-peer": 5, "remove-peer": 5 }, "7": { "add-peer": 5, "remove-peer": 5 } } ``` + Wait for 10 seconds. In the `Operator` tab of `tidb-demo-PD`, you will see that the rates for `replace-rule-offline-peer` and `replace-rule-offline-leader-peer` in the `Scheduler operator create` and `Scheduler operator finish` panels start to rise. This is because the efficiency of TiKV executing Operators has increased. Note that these two Operators play a major role in balancing during the scaling in process initiated by PD. ![70%](https://lab-static.pingcap.com/images/2025/7/24/90df54268d6aa8fa0c85a20c8baada857e155668.png) + In the `Cluster` tab of `tidb-demo-TiKV-Details`, you can see that the IO utilization of each TiKV node starts to rise. This indicates that the newly added TiKV nodes and the selected source TiKV nodes are utilizing IO resources for the scaling in process, which may impact the existing business workload. ![70%](https://lab-static.pingcap.com/images/2025/7/24/b8369458e4f86946c674b3645dccebf788208b19.png) + Wait and observe that in the upper part of `tidb-demo-PD`, the percentage of `Offline store progress` is rising more quickly, the value of `Left time` is decreasing, and the `Current scaling speed` is increasing. This indicates that increasing the store limit can accelerate the scaling in rate. ![70%](https://lab-static.pingcap.com/images/2025/7/24/43df4c4872c1c59b15f220118a4e42f865d76e62.png) ### Step 4. Slow down the scaling in of TiKV cluster + In the upper part of `tidb-demo-PD`, wait until the `Offline store progress` exceeds 50% (Note: It is recommended to initiate this step when `Offline store progress` is between 50% and 60%, as exceeding 60% may not leave enough time for subsequent steps). Then, set the `store limit` for all TiKV nodes to `1` to reduce the rate at which TiKV executes Operators. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit all 1 ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit all 1 Success! Set store limit successfully. ``` + Confirm that all items of the `store limit` for all TiKV nodes have been set to `1`. ```bash ~/.tiup/bin/tiup ctl:v${LAB:TIDB_VERSION} pd -u http://${HOST_PD1_PRIVATE_IP}:2379 store limit ``` ```output Starting component ctl: /home/ec2-user/.tiup/components/ctl/v${LAB:TIDB_VERSION}/ctl pd -u http://10.90.2.178:2379 store limit { "1": { "add-peer": 1, "remove-peer": 1 }, "4": { "add-peer": 1, "remove-peer": 1 }, "576": { "add-peer": 1, "remove-peer": 1 }, "7": { "add-peer": 1, "remove-peer": 1 } } ``` + Return to the [Grafana Dashboard](http://${LAB:HOST_MONITOR1_PUBLIC_IP}:3000) for `tidb-demo-PD`. Wait for 10 seconds, and in the `Operator` tab of `tidb-demo-PD`, you will see that the rates for `replace-rule-offline-peer` and `replace-rule-offline-leader-peer` in the `Scheduler operator create` and `Scheduler operator finish` panels start to decline. This is because the efficiency of TiKV executing Operators has decreased. ![70%](https://lab-static.pingcap.com/images/2025/7/24/406021a71460034ad668fcf176f750098ed8175b.png) + In the `Cluster` tab of `tidb-demo-TiKV-Details`, you can see that the IO utilization of each TiKV node starts to decline. This indicates that the TiKV cluster will use a smaller amount of IO resources for scaling in, which may help avoid impacting existing business workloads, but the scaling in process will become slower. ![70%](https://lab-static.pingcap.com/images/2025/7/24/214390b37261a4bf2e4dacca120f9382a7dad6f7.png) + Wait and observe that in the upper part of `tidb-demo-PD`, the rate of increase for the `Online store progress` percentage starts to slow down, the value of `Left time` increases, and the `Current scaling speed` begins to decline. This indicates that lowering the store limit can reduce the rate of scaling in. ![70%](https://lab-static.pingcap.com/images/2025/7/24/e31ebaa56b7d09b7720cd0cd9123c91550186a90.png) ### Step 5. Wait for the balancing process of the TiKV cluster scaling in complete, and then proceed to destroy the node + Continue to wait and observe the changes in `Offline store progress`, `Left time`, and `Current scaling speed` in the upper part of `tidb-demo-PD` until `Offline store progress` shows `No data`, indicating that the scaling in process is officially complete. Note: If participants feel that the scaling in is too slow, they can try adjusting the `store limit` to `20` to expedite the completion of the scaling operation. ![70%](https://lab-static.pingcap.com/images/2025/7/24/3b911baaaa83ac8dbb10dee3181fa6613f721bf9.png) + When the `Offline store progress` reaches above 90%, the `Current scaling speed` continues to decrease. ![70%](https://lab-static.pingcap.com/images/2025/7/24/32f4cef10c31bc4456d01881deeec463eb6153e6.png) + When the `Offline store progress` shows `No Data`, it indicates that the balancing stage of the scaling in process is complete. ![70%](https://lab-static.pingcap.com/images/2025/7/24/127c33db5b21e0c16ee8d35d248dbb880e7be75d.png) + In the lower part of `tidb-demo-PD` under the `Statistics - balance` tab, you can see the `Store available` panel and the `Store available ratio` panel, which indicate the data distribution of the TiKV cluster after scaling in. ![70%](https://lab-static.pingcap.com/images/2025/7/24/4a584f550935b2f764f27aac55d44c25942bbcc1.png) + In the `Cluster` tab of `tidb-demo-TiKV-Details`, locate the `IO utilization` panel and expand the time range to cover the entire scaling in process. You will observe that the load conditions of the TiKV cluster vary according to the `store limit` value. ![70%](https://lab-static.pingcap.com/images/2025/7/24/2e6a23f3e643340053b5b2ae9db2fe3181b9fb92.png) + Confirm the status of the TiDB cluster. At this point, you will notice that the deleted TiKV nodes transition from the `Pending Offline` state to the `Tombstone` state, indicating that the regions on them have been migrated, and you can proceed with the `prune` operation. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.74.236.1:2379/dashboard Grafana URL: http://10.90.4.196:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.196:9093 alertmanager 10.90.4.196 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.196:3000 grafana 10.90.4.196 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.178:2379 pd 10.90.2.178 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.196:9090 prometheus 10.90.4.196 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.140:4000 tidb 10.90.1.140 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.169:20160 tikv 10.90.1.169 20160/20180 linux/x86_64 Tombstone /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.192:20160 tikv 10.90.1.192 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.88:20160 tikv 10.90.1.88 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.99:20160 tikv 10.90.1.99 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.196:6000 tiproxy 10.90.4.196 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 10 There are some nodes can be pruned: Nodes: [10.90.1.169:20160] You can destroy them with the command: `tiup cluster prune tidb-demo` ``` + Destroy the removed TiKV node. ```bash ~/.tiup/bin/tiup cluster prune tidb-demo -y ``` ```output ... Destroy success ``` + Verify there are 3 `tikv` servers in the TiDB cluster now, you scaled the storage layer size from 4 to 3. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.74.236.1:2379/dashboard Grafana URL: http://10.90.4.196:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.196:9093 alertmanager 10.90.4.196 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.196:3000 grafana 10.90.4.196 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.2.178:2379 pd 10.90.2.178 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.196:9090 prometheus 10.90.4.196 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.140:4000 tidb 10.90.1.140 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.192:20160 tikv 10.90.1.192 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.88:20160 tikv 10.90.1.88 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.99:20160 tikv 10.90.1.99 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.196:6000 tiproxy 10.90.4.196 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 9 ```