## Module 1: Scale Out the SQL Layer and the Storage Layer ### Step 1. Connect to the TiUP control machine + Click [Terminal 1](ssh://monitor1-ssh#1) to open a new terminal connected to the virtual machine assigned for your hands-on lab. ### Step 2. Check the initial topology of the existing TiDB cluster + Verify that a TiDB cluster with TiProxy is up and running, and there are 1 `tidb` and 3 `tikv` servers in the cluster. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output {2,14-18} Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.91.126.63:2379/dashboard Grafana URL: http://10.90.4.204:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.204:9093 alertmanager 10.90.4.204 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.204:3000 grafana 10.90.4.204 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.89:2379 pd 10.90.3.89 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.204:9090 prometheus 10.90.4.204 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.74:4000 tidb 10.90.1.74 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.103:20160 tikv 10.90.1.103 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.30:20160 tikv 10.90.1.30 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.34:20160 tikv 10.90.1.34 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.204:6000 tiproxy 10.90.4.204 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 9 ``` ### Step 3. Run the test workload - Run the `sysbench` test workload, it takes 30 minutes to complete. Keep it running, do NOT wait for it to complete. ```bash source ~/hosts-env.sh mysql -h ${HOST_MONITOR1_PRIVATE_IP} -P 6000 -uroot < [!NOTE] > If you want to scale out the storage (TiKV) layer instead, you can skip ahead to Step 6. - Click [Terminal 2](ssh://monitor1-ssh#2), and run the following command to scale out one TiDB server for the cluster. ```bash source ~/hosts-env.sh ~/.tiup/bin/tiup cluster scale-out tidb-demo scale-out-tidb-2.yaml \ --user ec2-user \ -i ~/.ssh/pe-class-key-${REGION_NAME}.pem \ -y ``` ```output ... Scaled cluster `tidb-demo` out successfully ``` ### Step 5. Check the topology of the existing TiDB cluster - Verify there are two `tidb` servers in the TiDB cluster now. You had scaled the SQL layer size from 1 to 2. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output {2,14,15} Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://54.148.94.44:2379/dashboard Grafana URL: http://10.90.4.172:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.172:9093 alertmanager 10.90.4.172 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.172:3000 grafana 10.90.4.172 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.139:2379 pd 10.90.3.139 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.172:9090 prometheus 10.90.4.172 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.2.230:4000 tidb 10.90.2.230 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.75:4000 tidb 10.90.2.75 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.3.102:20160 tikv 10.90.3.102 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.171:20160 tikv 10.90.3.171 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.56:20160 tikv 10.90.3.56 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.172:6000 tiproxy 10.90.4.172 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 10 ``` ### Step 6. Scale out the storage layer by adding a TiKV server + Stay in [Terminal 2](ssh://monitor1-ssh#2). Scale out the storage layer by adding 1 TiKV server. ```bash source ~/hosts-env.sh ~/.tiup/bin/tiup cluster scale-out tidb-demo scale-out-tikv.yaml \ --user ec2-user \ -i ~/.ssh/pe-class-key-${REGION_NAME}.pem \ -y ``` ```output ... Scaled cluster `tidb-demo` out successfully ``` ### Step 7. Check the cluster topology - Verify there are 4 `tikv` servers in the TiDB cluster now, you scaled TiKV layer size from 3 to 4. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output {2,16-19} Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.91.126.63:2379/dashboard Grafana URL: http://10.90.4.204:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.204:9093 alertmanager 10.90.4.204 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.204:3000 grafana 10.90.4.204 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.89:2379 pd 10.90.3.89 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.204:9090 prometheus 10.90.4.204 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.48:4000 tidb 10.90.1.48 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.74:4000 tidb 10.90.1.74 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.103:20160 tikv 10.90.1.103 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.245:20160 tikv 10.90.1.245 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.30:20160 tikv 10.90.1.30 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.34:20160 tikv 10.90.1.34 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.204:6000 tiproxy 10.90.4.204 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 11 ``` ### Step 8. Scale in the storage layer by removing a TiKV server - On [Terminal 2](ssh://monitor1-ssh#2), scale in the storage layer by removing one TiKV server. ```bash source ~/hosts-env.sh ~/.tiup/bin/tiup cluster scale-in tidb-demo \ --node ${HOST_KV4_PRIVATE_IP}:20160 -y ``` ```output ... Scaled cluster `tidb-demo` in successfully ``` - Destroy the removed TiKV node. ```bash ~/.tiup/bin/tiup cluster prune tidb-demo -y ``` ```output ... Destroy success ``` ### Step 9. Check the cluster topology - Verify there are 3 `tikv` servers in the TiDB cluster now, you scaled the storage layer size from 4 to 3. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output {2,16-18} Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://54.185.182.184:2379/dashboard Grafana URL: http://10.90.4.216:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.216:9093 alertmanager 10.90.4.216 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.216:3000 grafana 10.90.4.216 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.191:2379 pd 10.90.3.191 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.216:9090 prometheus 10.90.4.216 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.16:4000 tidb 10.90.1.16 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.51:4000 tidb 10.90.1.51 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.3.213:20160 tikv 10.90.3.213 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.40:20160 tikv 10.90.3.40 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.71:20160 tikv 10.90.3.71 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.216:6000 tiproxy 10.90.4.216 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 10 ``` ### Step 10. Scale in the SQL layer by removing a TiDB server + On [Terminal 2](ssh://monitor1-ssh#2), scale in the SQL layer by removing one TiDB server. ```bash source ~/hosts-env.sh ~/.tiup/bin/tiup cluster scale-in tidb-demo \ --node ${HOST_DB2_PRIVATE_IP}:4000 -y ``` ```output ... Scaled cluster `tidb-demo` in successfully ``` ### Step 11. Check TiDB cluster topology - Verify there is one `tidb` server in the TiDB cluster now, you scaled the SQL layer size from 2 to 1. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output {2,15-17} Cluster type: tidb Cluster name: tidb-demo Cluster version: v${LAB:TIDB_VERSION} Deploy user: ec2-user SSH type: builtin Dashboard URL: http://54.185.182.184:2379/dashboard Grafana URL: http://10.90.4.216:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.216:9093 alertmanager 10.90.4.216 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.216:3000 grafana 10.90.4.216 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.191:2379 pd 10.90.3.191 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.216:9090 prometheus 10.90.4.216 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.16:4000 tidb 10.90.1.16 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.3.112:20160 tikv 10.90.3.112 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.213:20160 tikv 10.90.3.213 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.40:20160 tikv 10.90.3.40 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.216:6000 tiproxy 10.90.4.216 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 9 ``` ### Step 12. Cleanup - Throughout the entire scale-in/out workflow, the testing workload in [Terminal 1](ssh://monitor1-ssh#1) remains uninterrupted. ```output ... [ 100s ] thds: 14 tps: 1225.19 qps: 19613.31 (r/w/o: 17162.93/0.00/2450.39) lat (ms,95%): 13.22 err/s: 0.00 reconn/s: 0.00 [ 105s ] thds: 14 tps: 1220.81 qps: 19528.54 (r/w/o: 17086.92/0.00/2441.62) lat (ms,95%): 13.22 err/s: 0.00 reconn/s: 0.00 [ 110s ] thds: 14 tps: 1221.82 qps: 19549.49 (r/w/o: 17105.66/0.00/2443.84) lat (ms,95%): 13.22 err/s: 0.00 reconn/s: 0.00 ... ``` - Click the `Terminate` button on the left to end the lab and release the resources. For a deeper understanding of TiDB horizontal scaling, explore the labs [Database Seamless Scalability](/labs/demo_009) and [Database Scalability Deep Dive - Best Practice during SQL Layer Scaling In](/labs/demo_016).