--- title: "TiDB Cluster Scaling In and Scaling Out" version: "v2" status: "GA" description: "This hands-on lab demonstrates how to scale a TiDB cluster in and out. Scaling in and out refers to adding or removing nodes from a TiDB cluster to adjust its capacity. Scaling in and out is a common operation in a cloud environment where workloads can change rapidly. This lab shows you how to scale a TiDB cluster in and out using TiUP, a powerful cluster management tool for TiDB." --- # TiDB Cluster Scaling In and Scaling Out ## Overview TiDB uses a distributed architecture design that separates computing (SQL layer) from storage (TiKV layer), making it easy to scale horizontally. You can add more nodes to a TiDB cluster or remove existing nodes to change its capacity and performance without causing significant downtime or interruption to your application. ![56%](https://lab-static.pingcap.com/quick-demo/demo_009_en.png) ### Duration + This lab should take you approximately 60 minutes to complete. ### Tasks + Apply the test workload by using `sysbench`. + Observe the TiDB performance metrics from Grafana Dashboard. + Horizontally scale the SQL layer of the TiDB. + Horizontally scale the Storage layer of the TiDB. > [!TIP] > Click **Start** to launch the lab environment. > Once the environment is ready, select **Next** on the bottom toolbar to advance to **Module 1**. > And, follow the step-by-step lab guide to connect to the hands-on environment and begin your tasks. ## Module 1: Verify The Provisioned Environment And Set Up The Schema ### Step 1. Connect to the TiUP control machine + Click here - [Terminal 1](ssh://monitor1-ssh#1) to open a new terminal connected to the virtual machine assigned for your hands-on lab. + The banner after you logged in: ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### Step 2. Check the status of the TiDB cluster + Verify that a TiDB cluster with TiProxy is up and running, and there are 1 `tidb` and 3 `tikv` servers in the cluster. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v8.1.0 Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.91.126.63:2379/dashboard Grafana URL: http://10.90.4.204:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.204:9093 alertmanager 10.90.4.204 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.204:3000 grafana 10.90.4.204 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.89:2379 pd 10.90.3.89 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.204:9090 prometheus 10.90.4.204 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.74:4000 tidb 10.90.1.74 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.103:20160 tikv 10.90.1.103 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.30:20160 tikv 10.90.1.30 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.34:20160 tikv 10.90.1.34 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.204:6000 tiproxy 10.90.4.204 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 9 ``` ### Step 3. Create the sysbench configuration ```bash cat << EOF > sysbench_config mysql-host=${HOST_MONITOR1_PRIVATE_IP} mysql-port=6000 mysql-user=root mysql-password= mysql-db=sbtest time=120 threads=14 report-interval=5 EOF cat sysbench_config ``` ```output mysql-host=10.90.4.204 mysql-port=6000 mysql-user=root mysql-password= mysql-db=sbtest time=120 threads=14 report-interval=5 ``` ### Step 4. Connect to the TiDB cluster, create an empty database `sbtest` ```bash mysql -h ${HOST_MONITOR1_PRIVATE_IP} -P 6000 -uroot ``` ```output Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 0 Server version: 8.0.11-TiDB-v8.1.0 TiDB Server (Apache License 2.0) Community Edition, MySQL 8.0 compatible Copyright (c) 2000, 2023, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> ``` ```sql CREATE DATABASE sbtest; EXIT; ``` ```output Query OK, 0 rows affected (0.11 sec) Bye ``` ### Step 5. Prepare schema data + Prepare the `sysbench` data, and wait for it's completion. It will take around 3 minutes to finish. ```bash sysbench --config-file=sysbench_config oltp_read_only --tables=6 --table-size=1000000 prepare ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest4'... Creating table 'sbtest1'... Creating table 'sbtest3'... Creating table 'sbtest2'... Creating table 'sbtest6'... Creating table 'sbtest5'... Inserting 1000000 records into 'sbtest6' Inserting 1000000 records into 'sbtest5' Inserting 1000000 records into 'sbtest4' Inserting 1000000 records into 'sbtest3' Inserting 1000000 records into 'sbtest1' Inserting 1000000 records into 'sbtest2' Creating a secondary index on 'sbtest6'... Creating a secondary index on 'sbtest1'... Creating a secondary index on 'sbtest4'... Creating a secondary index on 'sbtest2'... Creating a secondary index on 'sbtest5'... Creating a secondary index on 'sbtest3'... ``` ## Module 2: Run Workload And Observe The Baseline ### Step 1. Run the test workload + Run the `sysbench` test workload, it takes 2 minutes to complete. And then, observe the `TiDB CPU` metrics from the Grafana Dashboard. The default login username and password are both `admin`. + Start `sysbench` workload. ```bash sysbench \ --config-file=sysbench_config oltp_read_only \ --tables=6 --table-size=1000000 run ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 14 Report intermediate results every 5 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! [ 5s ] thds: 14 tps: 1106.04 qps: 17712.85 (r/w/o: 15498.17/0.00/2214.68) lat (ms,95%): 14.73 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 14 tps: 1134.45 qps: 18164.46 (r/w/o: 15895.35/0.00/2269.11) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 15s ] thds: 14 tps: 1127.79 qps: 18036.91 (r/w/o: 15781.72/0.00/2255.19) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 20s ] thds: 14 tps: 1123.20 qps: 17972.54 (r/w/o: 15725.75/0.00/2246.79) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 25s ] thds: 14 tps: 1118.78 qps: 17901.07 (r/w/o: 15663.52/0.00/2237.56) lat (ms,95%): 14.46 err/s: 0.00 reconn/s: 0.00 [ 30s ] thds: 14 tps: 1121.02 qps: 17935.94 (r/w/o: 15693.90/0.00/2242.04) lat (ms,95%): 14.73 err/s: 0.00 reconn/s: 0.00 [ 35s ] thds: 14 tps: 1121.80 qps: 17946.19 (r/w/o: 15702.99/0.00/2243.20) lat (ms,95%): 14.46 err/s: 0.00 reconn/s: 0.00 [ 40s ] thds: 14 tps: 1132.40 qps: 18117.60 (r/w/o: 15852.60/0.00/2265.00) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 45s ] thds: 14 tps: 1125.60 qps: 18013.78 (r/w/o: 15762.38/0.00/2251.40) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 50s ] thds: 14 tps: 1127.81 qps: 18038.28 (r/w/o: 15782.87/0.00/2255.41) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 55s ] thds: 14 tps: 1138.60 qps: 18227.42 (r/w/o: 15950.02/0.00/2277.40) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 60s ] thds: 14 tps: 1132.79 qps: 18117.01 (r/w/o: 15851.64/0.00/2265.38) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 65s ] thds: 14 tps: 1125.40 qps: 18010.87 (r/w/o: 15760.06/0.00/2250.81) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 70s ] thds: 14 tps: 1129.00 qps: 18057.99 (r/w/o: 15799.99/0.00/2258.00) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 75s ] thds: 14 tps: 1121.60 qps: 17951.22 (r/w/o: 15708.02/0.00/2243.20) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 80s ] thds: 14 tps: 1127.20 qps: 18032.66 (r/w/o: 15778.25/0.00/2254.41) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 85s ] thds: 14 tps: 1127.99 qps: 18047.72 (r/w/o: 15791.93/0.00/2255.79) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 90s ] thds: 14 tps: 1124.00 qps: 17982.04 (r/w/o: 15733.83/0.00/2248.20) lat (ms,95%): 14.46 err/s: 0.00 reconn/s: 0.00 [ 95s ] thds: 14 tps: 1131.20 qps: 18100.34 (r/w/o: 15837.94/0.00/2262.39) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 100s ] thds: 14 tps: 1133.19 qps: 18137.49 (r/w/o: 15870.90/0.00/2266.59) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 105s ] thds: 14 tps: 1132.61 qps: 18120.69 (r/w/o: 15855.48/0.00/2265.21) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 110s ] thds: 14 tps: 1127.39 qps: 18036.31 (r/w/o: 15781.72/0.00/2254.59) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 115s ] thds: 14 tps: 1121.41 qps: 17944.32 (r/w/o: 15701.70/0.00/2242.61) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 120s ] thds: 14 tps: 1126.40 qps: 18023.22 (r/w/o: 15770.02/0.00/2253.20) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 1892856 write: 0 other: 270408 total: 2163264 transactions: 135204 (1126.59 per sec.) queries: 2163264 (18025.46 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 120.0103s total number of events: 135204 Latency (ms): min: 9.71 avg: 12.42 max: 226.04 95th percentile: 14.21 sum: 1679909.05 Threads fairness: events (avg/stddev): 9657.4286/184.44 execution time (avg/stddev): 119.9935/0.00 ``` ### Step 2. Observe the baseline + Log into Grafana Dashboard with username - admin and password - admin, go to `Home`, and then `tidb-demo`, and then `tidb-demo-Performance-Overview`. Under the `Performance Overview`, scroll down to the `TiDB CPU` panel, note the metrics `sysbench` workload had been generated. ![56](https://lab-static.pingcap.com/quick-demo/demo-009-status-01.png) ## Module 3: Scale Out The SQL Layer + Scale out the SQL layer by adding one TiDB server to the cluster. Run the same workload again and verify that the `TiDB CPU` consumption decreases proportionally compared to the previous step. ### Step 1. Verify the scaling out configuration for TiDB server ```bash cat scale-out-tidb.yaml ``` ```output tidb_servers: - host: 10.90.1.48 port: 4000 status_port: 10080 deploy_dir: /tidb-deploy/tidb-4000 log_dir: /tidb-deploy/tidb-4000/log ``` ### Step 2. Add one TiDB server to the existing cluster ```bash ~/.tiup/bin/tiup cluster scale-out tidb-demo scale-out-tidb.yaml \ --user ec2-user \ -i ~/.ssh/pe-class-key-oregon.pem \ -y ``` ```output ... Scaled cluster `tidb-demo` out successfully ``` ### Step 3. Check the TiDB cluster status + Verify there are two `tidb` servers in the TiDB cluster now. You had scaled the SQL layer size from 1 to 2. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v8.1.0 Deploy user: ec2-user SSH type: builtin Dashboard URL: http://54.148.94.44:2379/dashboard Grafana URL: http://10.90.4.172:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.172:9093 alertmanager 10.90.4.172 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.172:3000 grafana 10.90.4.172 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.139:2379 pd 10.90.3.139 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.172:9090 prometheus 10.90.4.172 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.2.230:4000 tidb 10.90.2.230 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.75:4000 tidb 10.90.2.75 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.3.102:20160 tikv 10.90.3.102 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.171:20160 tikv 10.90.3.171 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.56:20160 tikv 10.90.3.56 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.172:6000 tiproxy 10.90.4.172 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 10 ``` ### Step 4. Run the same `sysbench` test workload again. It takes 2 minutes to complete. ```bash sysbench \ --config-file=sysbench_config oltp_read_only \ --tables=6 \ --table-size=1000000 run ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 14 Report intermediate results every 5 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! [ 5s ] thds: 14 tps: 1138.22 qps: 18237.17 (r/w/o: 15958.53/0.00/2278.65) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 14 tps: 1128.66 qps: 18053.90 (r/w/o: 15796.19/0.00/2257.71) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 15s ] thds: 14 tps: 1146.20 qps: 18341.64 (r/w/o: 16049.03/0.00/2292.60) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 20s ] thds: 14 tps: 1133.40 qps: 18125.81 (r/w/o: 15859.41/0.00/2266.40) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 25s ] thds: 14 tps: 1142.40 qps: 18288.81 (r/w/o: 16004.01/0.00/2284.80) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 30s ] thds: 14 tps: 1138.60 qps: 18210.79 (r/w/o: 15933.19/0.00/2277.60) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 35s ] thds: 14 tps: 1147.39 qps: 18363.10 (r/w/o: 16068.31/0.00/2294.79) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 40s ] thds: 14 tps: 1138.80 qps: 18220.87 (r/w/o: 15943.26/0.00/2277.61) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 45s ] thds: 14 tps: 1137.80 qps: 18204.20 (r/w/o: 15928.80/0.00/2275.40) lat (ms,95%): 14.21 err/s: 0.00 reconn/s: 0.00 [ 50s ] thds: 14 tps: 1149.21 qps: 18380.90 (r/w/o: 16082.68/0.00/2298.21) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 55s ] thds: 14 tps: 1138.80 qps: 18227.03 (r/w/o: 15949.23/0.00/2277.80) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 60s ] thds: 14 tps: 1144.20 qps: 18305.17 (r/w/o: 16016.77/0.00/2288.40) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 65s ] thds: 14 tps: 1150.61 qps: 18409.75 (r/w/o: 16108.93/0.00/2300.82) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 70s ] thds: 14 tps: 1146.38 qps: 18348.26 (r/w/o: 16054.91/0.00/2293.36) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 75s ] thds: 14 tps: 1146.61 qps: 18336.94 (r/w/o: 16043.92/0.00/2293.02) lat (ms,95%): 13.70 err/s: 0.00 reconn/s: 0.00 [ 80s ] thds: 14 tps: 1140.80 qps: 18264.44 (r/w/o: 15982.63/0.00/2281.80) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 85s ] thds: 14 tps: 1140.60 qps: 18242.54 (r/w/o: 15961.55/0.00/2280.99) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 90s ] thds: 14 tps: 1141.40 qps: 18260.07 (r/w/o: 15977.06/0.00/2283.01) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 95s ] thds: 14 tps: 1148.80 qps: 18390.78 (r/w/o: 16093.18/0.00/2297.60) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 100s ] thds: 14 tps: 1136.80 qps: 18181.97 (r/w/o: 15908.38/0.00/2273.60) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 105s ] thds: 14 tps: 1146.60 qps: 18341.07 (r/w/o: 16048.46/0.00/2292.61) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 110s ] thds: 14 tps: 1142.60 qps: 18286.77 (r/w/o: 16001.37/0.00/2285.40) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 [ 115s ] thds: 14 tps: 1145.40 qps: 18321.19 (r/w/o: 16030.00/0.00/2291.20) lat (ms,95%): 13.70 err/s: 0.00 reconn/s: 0.00 [ 120s ] thds: 14 tps: 1147.79 qps: 18365.31 (r/w/o: 16069.92/0.00/2295.39) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 1919484 write: 0 other: 274212 total: 2193696 transactions: 137106 (1142.42 per sec.) queries: 2193696 (18278.68 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 120.0126s total number of events: 137106 Latency (ms): min: 9.26 avg: 12.25 max: 224.92 95th percentile: 13.95 sum: 1679939.18 Threads fairness: events (avg/stddev): 9793.2857/474.34 execution time (avg/stddev): 119.9957/0.00 ``` ### Step 5. Observe the changed metrics + Observe the `TiDB CPU` metric from Grafana Dashboard again, click on the `avg` ledger on the panel, the average `TiDB CPU` consumption decreases proportionally compared to the previous step because of we now have 2 `tidb` servers at the SQL layer, as a result, the average CPU decreased to 50% of the previous configuration which has 1 `tidb` server only. ![56](https://lab-static.pingcap.com/quick-demo/demo-009-status-02.png) ## Module 4: Scale Out The Storage Layer + Scale out the storage layer while the `sysbench` workload is running. ### Step 1. Open a new terminal (Terminal 2) and log into monitor1_address + Click here - [Terminal 2](ssh://monitor1-ssh#2) to open a new terminal connected to the virtual machine assigned for your hands-on lab. + The banner after you logged in: ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### Step 2. Run test workload + Run `sysbench` workload in [Terminal 2](ssh://monitor1-ssh#2). While the workload is running, proceed to the next step, do not wait for the workload to complete. ```bash sysbench \ --config-file=sysbench_config oltp_read_only \ --tables=6 \ --table-size=1000000 run ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 14 Report intermediate results every 5 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! ... ``` ### Step 3. Add one TiKV server to the cluster + Switch back to [Terminal 1](ssh://monitor1-ssh#1). Scale out the storage layer by adding 1 TiKV server. ```bash cat scale-out-tikv.yaml ``` ```output tikv_servers: - host: 10.90.1.245 port: 20160 status_port: 20180 ``` ```bash ~/.tiup/bin/tiup cluster scale-out tidb-demo scale-out-tikv.yaml --user ec2-user -i ~/.ssh/pe-class-key-oregon.pem -y ``` ```output ... Scaled cluster `tidb-demo` out successfully ``` ### Step 4. Check TiDB cluster status + Verify there are 4 `tikv` servers in the TiDB cluster now, you scaled TiKV layer size from 3 to 4. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v8.1.0 Deploy user: ec2-user SSH type: builtin Dashboard URL: http://35.91.126.63:2379/dashboard Grafana URL: http://10.90.4.204:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.204:9093 alertmanager 10.90.4.204 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.204:3000 grafana 10.90.4.204 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.89:2379 pd 10.90.3.89 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.204:9090 prometheus 10.90.4.204 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.48:4000 tidb 10.90.1.48 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.74:4000 tidb 10.90.1.74 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.1.103:20160 tikv 10.90.1.103 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.245:20160 tikv 10.90.1.245 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.30:20160 tikv 10.90.1.30 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.34:20160 tikv 10.90.1.34 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.204:6000 tiproxy 10.90.4.204 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 11 ``` ## Module 5: Scale In The SQL Layer + Scale in the SQL layer while the `sysbench` workload is running. ### Step 1. Run test workload + Switch to [Terminal 2](ssh://monitor1-ssh#2) and run the `sysbench` workload again. While the workload is running, proceed to the next step, do not wait for the workload to complete. ```bash sysbench \ --config-file=sysbench_config oltp_read_only \ --tables=6 \ --table-size=1000000 run ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 14 Report intermediate results every 5 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! ... ``` ### Step 2. Remove one TiDB server from the cluster + On [Terminal 1](ssh://monitor1-ssh#1), scale in the SQL layer by removing one random TiDB server. ```bash RANDOM_TIDB_NODE=`~/.tiup/bin/tiup cluster display tidb-demo | grep 4000 | sed 's/ .*$//' | head -n 1` ~/.tiup/bin/tiup cluster scale-in tidb-demo --node ${RANDOM_TIDB_NODE} -y ``` ```output ... Scaled cluster `tidb-demo` in successfully ``` ### Step 4. Check TiDB cluster status + Verify there is one `tidb` server in the TiDB cluster now, you scaled the SQL layer size from 2 to 1. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v8.1.0 Deploy user: ec2-user SSH type: builtin Dashboard URL: http://54.185.182.184:2379/dashboard Grafana URL: http://10.90.4.216:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.216:9093 alertmanager 10.90.4.216 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.216:3000 grafana 10.90.4.216 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.191:2379 pd 10.90.3.191 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.216:9090 prometheus 10.90.4.216 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.16:4000 tidb 10.90.1.16 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.3.112:20160 tikv 10.90.3.112 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.213:20160 tikv 10.90.3.213 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.40:20160 tikv 10.90.3.40 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.71:20160 tikv 10.90.3.71 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.216:6000 tiproxy 10.90.4.216 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 10 ``` ## Module 6: Scale In The Storage Layer + Scale in the storage layer while the `sysbench` workload is running. ### Step 1. Run test workload + Switch to [Terminal 2](ssh://monitor1-ssh#2) and run the `sysbench` workload again. While the workload is running, proceed to the next step, do not wait for the workload to complete. ```bash sysbench \ --config-file=sysbench_config oltp_read_only \ --tables=6 \ --table-size=1000000 run ``` ```output sysbench 1.0.17 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 14 Report intermediate results every 5 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! ... ``` ### Step 2. Remove one TiKV server from the cluster + On [Terminal 1](ssh://monitor1-ssh#1), scale in the storage layer by removing one random TiKV server. ```bash RANDOM_TIKV_NODE=`~/.tiup/bin/tiup cluster display tidb-demo | grep 20160 | sed 's/ .*$//' | head -n 1` ~/.tiup/bin/tiup cluster scale-in tidb-demo --node ${RANDOM_TIKV_NODE} -y ``` ```output ... Scaled cluster `tidb-demo` in successfully ``` + Destroy the removed TiKV node. ```bash ~/.tiup/bin/tiup cluster prune tidb-demo -y ``` ```output ... Destroy success ``` ### Step 3. Check TiDB cluster status + Verify there are 3 `tikv` servers in the TiDB cluster now, you scaled the storage layer size from 4 to 3. ```bash ~/.tiup/bin/tiup cluster display tidb-demo ``` ```output Cluster type: tidb Cluster name: tidb-demo Cluster version: v8.1.0 Deploy user: ec2-user SSH type: builtin Dashboard URL: http://54.185.182.184:2379/dashboard Grafana URL: http://10.90.4.216:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.216:9093 alertmanager 10.90.4.216 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.216:3000 grafana 10.90.4.216 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.3.191:2379 pd 10.90.3.191 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.216:9090 prometheus 10.90.4.216 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.16:4000 tidb 10.90.1.16 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.3.213:20160 tikv 10.90.3.213 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.40:20160 tikv 10.90.3.40 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.71:20160 tikv 10.90.3.71 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.216:6000 tiproxy 10.90.4.216 6000/3080 linux/x86_64 Up - /tidb-deploy/tiproxy-6000 Total nodes: 9 ``` ## Conclusion You can scale out/in the SQL layer (TiDB servers) and the storage layer (TiKV servers) independently based on your needs. The scaling process remains seamless for application operations and maintenance teams, especially when a load balancer is positioned in front of the TiDB cluster. In our lab, TiProxy effectively fulfills this role. During the process of the horizontal scaling your application keeps running. ```hbsptform { "ui": { "titleLevel": 3 }, "portalId": "4466002", "formId": "89fa8368-eba1-4b84-b91e-eb9eb6613ddf", "region": "na1", "sfdcCampaignId": "701RC00000VqCODYA3" } ```