--- title: "Maintaining a TiDB Cluster" version: "v2" status: "GA" description: "In this hands-on lab, you will learn how to operate a TiDB cluster. Specifically, we will cover how to start/stop/scale-in/scale-out a TiDB cluster. This lab is a practical workshop that allows you to learn how to operate a TiDB cluster." --- # Lab 4: Maintaining a TiDB Cluster ## Overview >[!NOTE] > This hands-on lab is also included in the [TiDB Certification Training: TiDB Administration Using TiUP](https://www.pingcap.com/education/course-tidb-database-administration-for-self-hosted-deployments/) course. In this lab, you perform a few tasks to maintain a TiDB cluster, including upgrading, scale-in and scale-out, monitoring, as well as using placement rules. The diagram below illustrates the deployment topology of the TiDB cluster at its initial status. ![60%](https://lab-static.pingcap.com/quick-demo/TiDBClusterDeploymentTopologyExercise_aa_007_20220914.png) > [!TIP] > Click **Start** to launch the lab environment. > Once the environment is ready, select **Next** on the bottom toolbar to advance to **Module 1**. > And, follow the step-by-step lab guide to connect to the hands-on environment and begin your tasks. ## Module 1: Upgrading the TiDB from v7.5.0 to v7.5.1 + Before upgrading a TiDB cluster in production environment, you need to review compatibility changes, upgrade TiUP or TiUP offline mirror, edit TiUP topology configuration file, check the DDL and backup status of the cluster, and check the health status of the current cluster. Please refer to our [documentation](https://docs.pingcap.com/tidb/v7.5/upgrade-tidb-using-tiup) for more details ### Step 1. Log into the remote Linux + Click here - [Terminal 1](ssh://monitor1-ssh#1) to open a new terminal connected to the virtual machine assigned for your hands-on lab. + The banner after you logged in: ```output , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ ``` ### Step 2. Check the version of the TiDB cluster ```bash tiup cluster display tidb-test ``` ```output ... Cluster version: v7.5.0 ... ``` ### Step 3. Change TiUP local mirror to version 7.5.1 ```bash tiup mirror set tidb-community-server-7.5.1-linux-amd64 ``` ```output Successfully set mirror to /home/ec2-user/tidb-community-server-7.5.1-linux-amd64 ``` ### Step 4. Upgrade the TiDB cluster + Upgrade of the existing TiDB v7.5.0 cluster to TiDB v7.5.1 ```bash tiup cluster upgrade tidb-test 7.5.1 ``` ```output Starting component `cluster`: /home/ec2-user/.tiup/components/cluster/v1.10.1/tiup-cluster upgrade tidb-test v7.5.0 This operation will upgrade tidb v7.5.0 cluster tidb-test to v7.5.1. Do you want to continue? [y/N]:(default=N) y ... Upgraded cluster `tidb-test` successfully ``` ### Step 5. Check the version of the TiDB cluster again ```bash tiup cluster display tidb-test ``` ```output tiup is checking updates for component cluster ... Starting component `cluster`: /home/ec2-user/.tiup/components/cluster/v1.10.1/tiup-cluster display tidb-test Cluster type: tidb Cluster name: tidb-test Cluster version: v7.5.1 ... ``` ## Module 2: Scaling out a TiKV Cluster ### Step 1. Read `solution-scale-out-tikv.yaml` file. ```bash cat solution-scale-out-tikv.yaml ``` ```output tikv_servers: - host: 10.90.4.93 ssh_port: 22 port: 20160 status_port: 20180 deploy_dir: /tidb-deploy/tikv-20160 data_dir: /tidb-data/tikv-20160 log_dir: /tidb-deploy/tikv-20160/log config: server.labels: region: seattle zone: az4 storage-performance: high ``` ### Step 2. Scale out the TiKV cluster ```bash tiup cluster scale-out tidb-test --user ec2-user -i ~/.ssh/pe-class-key-oregon.pem solution-scale-out-tikv.yaml ``` ```output ... Do you want to continue? [y/N]: (default=N) y ... Scaled cluster `tidb-test` out successfully ``` ### Step 3. Check the status of the TiDB cluster + Check the status of the deployed TiDB cluster and the new TiKV node (IP address: `HOST_MONITOR1_PRIVATE_IP`). There are 4 TiKV nodes now. ```bash tiup cluster display tidb-test ``` ```output tiup is checking updates for component cluster ... Starting component `cluster`: /home/ec2-user/.tiup/components/cluster/v1.12.1/tiup-cluster display tidb-test Cluster type: tidb Cluster name: tidb-test Cluster version: v7.5.1 Deploy user: tidb SSH type: builtin Dashboard URL: http://10.90.1.122:2379/dashboard Grafana URL: http://10.90.4.214:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.214:9093 alertmanager 10.90.4.214 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.214:3000 grafana 10.90.4.214 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.122:2379 pd 10.90.1.122 2379/2380 linux/x86_64 Up|L|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.2.243:2379 pd 10.90.2.243 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.3.25:2379 pd 10.90.3.25 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.214:9090 prometheus 10.90.4.214 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.133:4000 tidb 10.90.1.133 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.92:4000 tidb 10.90.2.92 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.4.103:9000 tiflash 10.90.4.103 9000/8123/3930/20170/20292/8234 linux/x86_64 Up /tidb-data/tiflash-9000 /tidb-deploy/tiflash-9000 10.90.1.98:20160 tikv 10.90.1.98 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.2.235:20160 tikv 10.90.2.235 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.148:20160 tikv 10.90.3.148 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.4.214:20160 tikv 10.90.4.214 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 Total nodes: 13 ``` ## Module 3: Scaling in a TiKV Cluster ### Step 1. Scale in the TiKV cluster ```bash tiup cluster scale-in tidb-test --node ${HOST_MONITOR1_PRIVATE_IP}:20160 ``` ```output ... Do you want to continue? [y/N]:(default=N) y The component `[tikv]` will become tombstone, maybe exists in several minutes or hours, after that you can use the prune command to clean it Do you want to continue? [y/N]:(default=N) y ... Scaled cluster `tidb-test` in successfully ``` ### Step 2. Check the status of the TiDB cluster + The status of the TiKV node (IP address: `HOST_MONITOR1_PRIVATE_IP`) should be changed to `Tombstone` or `Pending Offline`. ```bash tiup cluster display tidb-test ``` ```output tiup is checking updates for component cluster ... ... ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- ... 10.90.4.214:20160 tikv 10.90.4.214 20160/20180 linux/x86_64 Tombstone /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 Total nodes: 13 There are some nodes can be pruned: Nodes: [10.90.4.214:20160] You can destroy them with the command: `tiup cluster prune tidb-test` ``` ### Step 3. Prune the TiDB cluster + If the status of your scaled in TiKV is `Tombstone`, you can prune this TiKV node to clean up the data. ```bash tiup cluster prune tidb-test ``` ```output ... Do you confirm this action? [y/N]:(default=N) y ... Destroy success ``` ### Step 4. Check the status of the TiDB cluster ```bash tiup cluster display tidb-test ``` ```output tiup is checking updates for component cluster ... Starting component `cluster`: /home/ec2-user/.tiup/components/cluster/v1.12.1/tiup-cluster display tidb-test Cluster type: tidb Cluster name: tidb-test Cluster version: v7.5.1 Deploy user: tidb SSH type: builtin Dashboard URL: http://10.90.1.122:2379/dashboard Grafana URL: http://10.90.4.214:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.214:9093 alertmanager 10.90.4.214 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.214:3000 grafana 10.90.4.214 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.122:2379 pd 10.90.1.122 2379/2380 linux/x86_64 Up|L|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.2.243:2379 pd 10.90.2.243 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.3.25:2379 pd 10.90.3.25 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.214:9090 prometheus 10.90.4.214 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.133:4000 tidb 10.90.1.133 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.92:4000 tidb 10.90.2.92 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.4.103:9000 tiflash 10.90.4.103 9000/8123/3930/20170/20292/8234 linux/x86_64 Up /tidb-data/tiflash-9000 /tidb-deploy/tiflash-9000 10.90.1.98:20160 tikv 10.90.1.98 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.2.235:20160 tikv 10.90.2.235 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.148:20160 tikv 10.90.3.148 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 Total nodes: 12 ``` ## Module 4: Implementing Hot and Cold Data Storage Tiering ### Overview + In this exercise, you will gain experience with TiDB's Placement Rules in SQL, a feature that allows you to specify where data is stored in a TiDB using SQL statements. You practice using the placement rules to separate cold data to `lower-cost` storage (or low performance node) and hot data to `higher-cost` storage (or high performance node) with table partitions to improve cost efficiency. ![40%](https://lab-static.pingcap.com/quick-demo/PlacementRuleColdHot_aa_001_20230422.png) ### Step 1. Scale out the TiKV cluster + Scale out the TiKV cluster to include three extra TiKV nodes with low performance storage. In this experiment, we just put an extra TiKV instance on each EC2, where the TiKV nodes run on. We pretend three new TiKV nodes have low performance storage, where three existing TiKV nodes have high performance storage. ```bash tiup cluster scale-out tidb-test solution-topology-placement-rule.yaml --yes --user ec2-user -i ~/.ssh/pe-class-key-oregon.pem ``` ```output ... Scaled cluster `tidb-test` out successfully ``` ### Step 2. Check the status of the TiDB cluster ```bash tiup cluster display tidb-test ``` ```output ... Cluster type: tidb Cluster name: tidb-test Cluster version: v7.5.1 Deploy user: tidb SSH type: builtin Dashboard URL: http://10.90.2.59:2379/dashboard Grafana URL: http://10.90.4.204:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.204:9093 alertmanager 10.90.4.204 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.204:3000 grafana 10.90.4.204 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.22:2379 pd 10.90.1.22 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.2.59:2379 pd 10.90.2.59 2379/2380 linux/x86_64 Up|L|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.3.126:2379 pd 10.90.3.126 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.204:9090 prometheus 10.90.4.204 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.123:4000 tidb 10.90.1.123 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.193:4000 tidb 10.90.2.193 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.4.100:9000 tiflash 10.90.4.100 9000/8123/3930/20170/20292/8234 linux/x86_64 Up /tidb-data/tiflash-9000 /tidb-deploy/tiflash-9000 10.90.1.45:20160 tikv 10.90.1.45 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.45:20161 tikv 10.90.1.45 20161/20181 linux/x86_64 Up /tidb-data/tikv-20161 /tidb-deploy/tikv-20161 10.90.2.251:20160 tikv 10.90.2.251 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.2.251:20161 tikv 10.90.2.251 20161/20181 linux/x86_64 Up /tidb-data/tikv-20161 /tidb-deploy/tikv-20161 10.90.3.130:20160 tikv 10.90.3.130 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.130:20161 tikv 10.90.3.130 20161/20181 linux/x86_64 Up /tidb-data/tikv-20161 /tidb-deploy/tikv-20161 Total nodes: 15 ``` ### Step 3. Create placement policies + Create placement policies for designating hot data to TiKV nodes with high performance storage, and cold data to TiKV nodes with low performance storage. + Connect to TiDB server as root. ```bash ./connect-db1.sh ``` ```output tidb:db1> ``` + Verify the available labels for placement rules. Pay attention to the label with key `storage-performance`, it's safe to ignore `engine`, `zone`, and `region` for now. ```sql SHOW PLACEMENT LABELS; ``` ```output tidb:db1> SHOW PLACEMENT LABELS; +---------------------+-----------------------+ | Key | Values | +---------------------+-----------------------+ | engine | ["tiflash"] | | region | ["Seattle"] | | storage-performance | ["high", "low"] | | zone | ["az1", "az2", "az3"] | +---------------------+-----------------------+ 4 rows in set (0.00 sec) ``` + Check the available labels and their corresponding TiKV stores. Note that the `store_id` column represents the TiKV store ID, and the `label` column indicates the labeled key-values for each TiKV store. ```sql SELECT store_id, address, store_state_name, label FROM information_schema.tikv_store_status; ``` ```output tidb:db1> SELECT store_id, address, store_state_name, label -> FROM information_schema.tikv_store_status; +----------+-------------------+------------------+--------------------------------------------------------------------------------------------------------------------------+ | store_id | address | store_state_name | label | +----------+-------------------+------------------+--------------------------------------------------------------------------------------------------------------------------+ | 1 | 10.90.3.130:20160 | Up | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "high"}, {"key": "zone", "value": "az3"}]| | 2 | 10.90.2.251:20160 | Up | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "high"}, {"key": "zone", "value": "az2"}]| | 3 | 10.90.1.45:20160 | Up | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "high"}, {"key": "zone", "value": "az1"}]| | 98 | 10.90.4.100:3930 | Up | [{"key": "engine", "value": "tiflash"}] | | 186 | 10.90.2.251:20161 | Up | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "low"}, {"key": "zone", "value": "az2"}] | | 187 | 10.90.1.45:20161 | Up | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "low"}, {"key": "zone", "value": "az1"}] | | 188 | 10.90.3.130:20161 | Up | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "low"}, {"key": "zone", "value": "az3"}] | +----------+-------------------+------------------+--------------------------------------------------------------------------------------------------------------------------+ 7 rows in set (0.01 sec) ``` + Create a placement policy for hot data, mapping it to the label key-value pair: `(storage-performance, high)`: ```sql DROP PLACEMENT POLICY IF EXISTS hot; ``` ```output tidb:db1> DROP PLACEMENT POLICY IF EXISTS hot; Query OK, 0 rows affected, 1 warning (0.00 sec) ``` ```sql CREATE PLACEMENT POLICY hot CONSTRAINTS="[+storage-performance=high]"; ``` ```output tidb:db1> CREATE PLACEMENT POLICY hot -> CONSTRAINTS="[+storage-performance=high]"; Query OK, 0 rows affected (0.07 sec) ``` + Create a placement policy for cold data, mapping it to the label key-value pair: `(storage-performance, low)`: ```sql DROP PLACEMENT POLICY IF EXISTS cold; ``` ```output tidb:db1> DROP PLACEMENT POLICY IF EXISTS cold; Query OK, 0 rows affected, 1 warning (0.01 sec) ``` ```sql CREATE PLACEMENT POLICY cold CONSTRAINTS="[+storage-performance=low]"; ``` ```output tidb:db1> CREATE PLACEMENT POLICY cold -> CONSTRAINTS="[+storage-performance=low]"; Query OK, 0 rows affected (0.07 sec) ``` + Verify the constraint definitions of created placement policies, `hot` and `cold`: ```sql SELECT policy_name, constraints FROM information_schema.placement_policies; ``` ```output tidb:db1> SELECT policy_name, constraints FROM information_schema.placement_policies; +-------------+-----------------------------+ | policy_name | constraints | +-------------+-----------------------------+ | hot | [+storage-performance=high] | | cold | [+storage-performance=low] | +-------------+-----------------------------+ 2 rows in set (0.00 sec) ``` ### Step 4. Create two tables + Create two range-partitioned tables utilizing placement policies in the previous step, one with hot-cold partitions and the other with cold-only partitions, to maximize the storage cost efficiency. + Create a partitioned table `test.orders` using `hot` and `cold` placement policies. Note that orders with an `order_date` before 2010 and between 2010 and 2015 (exclusive) are marked as `cold`, while those with an `order_date` after 2015 are marked as `hot` which mapped to the `MAXVALUE` partition: ```sql DROP TABLE IF EXISTS test.orders; ``` ```output tidb:db1> DROP TABLE IF EXISTS test.orders; Query OK, 0 rows affected, 1 warning (0.00 sec) ``` ```sql CREATE TABLE test.orders (order_id INT, item VARCHAR(50), order_date DATE, PRIMARY KEY (order_date, order_id)) PARTITION BY RANGE( YEAR(order_date) ) ( PARTITION p10 VALUES LESS THAN (2010) PLACEMENT POLICY=cold, PARTITION p15 VALUES LESS THAN (2015) PLACEMENT POLICY=cold, PARTITION pm VALUES LESS THAN MAXVALUE PLACEMENT POLICY=hot ); ``` ```output tidb:db1> CREATE TABLE test.orders (order_id INT, item VARCHAR(50), order_date DATE, PRIMARY KEY (order_date, order_id)) -> PARTITION BY RANGE( YEAR(order_date) ) ( -> PARTITION p10 VALUES LESS THAN (2010) PLACEMENT POLICY=cold, -> PARTITION p15 VALUES LESS THAN (2015) PLACEMENT POLICY=cold, -> PARTITION pm VALUES LESS THAN MAXVALUE PLACEMENT POLICY=hot -> ); Query OK, 0 rows affected (0.34 sec) ``` + Create a partitioned table `test.orders_log` using the `cold` placement policy only for all its partitions: ```sql DROP TABLE IF EXISTS test.orders_log; ``` ```output tidb:db1> DROP TABLE IF EXISTS test.orders_log; Query OK, 0 rows affected, 1 warning (0.00 sec) ``` ```sql CREATE TABLE test.orders_log (order_id INT, item VARCHAR(50), order_date DATE, PRIMARY KEY (order_date, order_id)) PARTITION BY RANGE( YEAR(order_date) ) ( PARTITION p10 VALUES LESS THAN (2010) PLACEMENT POLICY=cold, PARTITION p15 VALUES LESS THAN (2015) PLACEMENT POLICY=cold, PARTITION pm VALUES LESS THAN MAXVALUE PLACEMENT POLICY=cold ); ``` ```output tidb:db1> CREATE TABLE test.orders_log (order_id INT, item VARCHAR(50), order_date DATE, PRIMARY KEY (order_date, order_id)) -> PARTITION BY RANGE( YEAR(order_date) ) ( -> PARTITION p10 VALUES LESS THAN (2010) PLACEMENT POLICY=cold, -> PARTITION p15 VALUES LESS THAN (2015) PLACEMENT POLICY=cold, -> PARTITION pm VALUES LESS THAN MAXVALUE PLACEMENT POLICY=cold -> ); Query OK, 0 rows affected (0.38 sec) ``` ### Step 5. Verify that the placement rules had taken effect + Show TiKV region information about the hot-cold table `test.orders`. Pay attention to the `REGION_ID` with the highest `START_KEY`, it's the `MAXVALUE` partition. In the example output, the `REGION_ID` is 264 (note that you might have different numbering for region id and start keys in your environment). ```sql SHOW TABLE test.orders REGIONS\G ``` ```output tidb:db1> SHOW TABLE test.orders REGIONS\G *************************** 1. row *************************** REGION_ID: 256 START_KEY: t_83_ END_KEY: t_84_ LEADER_ID: 259 LEADER_STORE_ID: 188 PEERS: 259, 269, 271 SCATTERING: 0 WRITTEN_BYTES: 136 READ_BYTES: 0 APPROXIMATE_SIZE(MB): 1 APPROXIMATE_KEYS: 0 SCHEDULING_CONSTRAINTS: CONSTRAINTS="[+storage-performance=low]" SCHEDULING_STATE: SCHEDULED *************************** 2. row *************************** REGION_ID: 260 START_KEY: t_84_ END_KEY: t_85_ LEADER_ID: 263 LEADER_STORE_ID: 188 PEERS: 263, 270, 272 SCATTERING: 0 WRITTEN_BYTES: 136 READ_BYTES: 0 APPROXIMATE_SIZE(MB): 1 APPROXIMATE_KEYS: 0 SCHEDULING_CONSTRAINTS: CONSTRAINTS="[+storage-performance=low]" SCHEDULING_STATE: SCHEDULED *************************** 3. row *************************** REGION_ID: 264 START_KEY: t_85_ END_KEY: t_86_ LEADER_ID: 266 LEADER_STORE_ID: 3 PEERS: 265, 266, 268 SCATTERING: 0 WRITTEN_BYTES: 0 READ_BYTES: 0 APPROXIMATE_SIZE(MB): 1 APPROXIMATE_KEYS: 0 SCHEDULING_CONSTRAINTS: CONSTRAINTS="[+storage-performance=high]" SCHEDULING_STATE: SCHEDULED 3 rows in set (0.02 sec) ``` + Verify the regions and the TiKV stores mapping for table `test.orders` is aligned with the placement policy. According to the placement rules, `REGION_ID` 264's region should be mapped to TiKV node with `storage-performance:high`, whereas the other two regions, 256 and 260 should be mapped to TiKV nodes with `storage-performance:low`: ```sql SELECT kvrs.region_id, kvss.store_id, kvrp.is_leader, kvss.label, kvss.start_ts FROM information_schema.tikv_region_status kvrs JOIN information_schema.tikv_region_peers kvrp ON kvrs.region_id = kvrp.region_id JOIN information_schema.tikv_store_status kvss ON kvrp.store_id = kvss.store_id JOIN information_schema.tables t ON t.table_name = kvrs.table_name WHERE t.table_schema='test' AND t.table_name='orders' AND kvrs.is_index = 0 AND kvrp.is_leader = 1; ``` ```output tidb:db1> SELECT kvrs.region_id, kvss.store_id, kvrp.is_leader, kvss.label, kvss.start_ts -> FROM information_schema.tikv_region_status kvrs -> JOIN information_schema.tikv_region_peers kvrp -> ON kvrs.region_id = kvrp.region_id -> JOIN information_schema.tikv_store_status kvss -> ON kvrp.store_id = kvss.store_id -> JOIN information_schema.tables t -> ON t.table_name = kvrs.table_name -> WHERE t.table_schema='test' -> AND t.table_name='orders' -> AND kvrs.is_index = 0 -> AND kvrp.is_leader = 1; +-----------+----------+-----------+--------------------------------------------------------------------------------------------------------------------------+---------------------+ | region_id | store_id | is_leader | label | start_ts | +-----------+----------+-----------+--------------------------------------------------------------------------------------------------------------------------+---------------------+ | 264 | 3 | 1 | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "high"}, {"key": "zone", "value": "az1"}]| 2023-06-08 06:47:58 | | 256 | 188 | 1 | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "low"}, {"key": "zone", "value": "az3"}] | 2023-06-08 06:49:39 | | 260 | 188 | 1 | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "low"}, {"key": "zone", "value": "az3"}] | 2023-06-08 06:49:39 | +-----------+----------+-----------+--------------------------------------------------------------------------------------------------------------------------+---------------------+ 3 rows in set (0.01 sec) ``` + Show TiKV region information about the `COLD` only table `test.orders_log`. Pay attention to the `REGION_ID` for all regions, in the example output, they are 277, 281 and 285 (note that you might have different numbering for region id and start keys in your environment): ```sql SHOW TABLE test.orders_log REGIONS\G ``` ```output tidb:db1> SHOW TABLE test.orders_log REGIONS\G *************************** 1. row *************************** REGION_ID: 277 START_KEY: t_88_ END_KEY: t_89_ LEADER_ID: 280 LEADER_STORE_ID: 188 PEERS: 280, 289, 292 SCATTERING: 0 WRITTEN_BYTES: 136 READ_BYTES: 0 APPROXIMATE_SIZE(MB): 1 APPROXIMATE_KEYS: 0 SCHEDULING_CONSTRAINTS: CONSTRAINTS="[+storage-performance=low]" SCHEDULING_STATE: SCHEDULED *************************** 2. row *************************** REGION_ID: 281 START_KEY: t_89_ END_KEY: t_90_ LEADER_ID: 284 LEADER_STORE_ID: 188 PEERS: 284, 291, 294 SCATTERING: 0 WRITTEN_BYTES: 136 READ_BYTES: 0 APPROXIMATE_SIZE(MB): 1 APPROXIMATE_KEYS: 0 SCHEDULING_CONSTRAINTS: CONSTRAINTS="[+storage-performance=low]" SCHEDULING_STATE: SCHEDULED *************************** 3. row *************************** REGION_ID: 285 START_KEY: t_90_ END_KEY: t_91_ LEADER_ID: 288 LEADER_STORE_ID: 188 PEERS: 288, 290, 293 SCATTERING: 0 WRITTEN_BYTES: 136 READ_BYTES: 0 APPROXIMATE_SIZE(MB): 1 APPROXIMATE_KEYS: 0 SCHEDULING_CONSTRAINTS: CONSTRAINTS="[+storage-performance=low]" SCHEDULING_STATE: SCHEDULED 3 rows in set (0.02 sec) ``` + Verify the regions and the TiKV stores mapping for table `test.orders_log` is aligned with the placement policy. According to the placement rules, all regions should be mapped to TiKV nodes with `storage-performance:low`. ```sql SELECT kvrs.region_id, kvss.store_id, kvrp.is_leader, kvss.label, kvss.start_ts FROM information_schema.tikv_region_status kvrs JOIN information_schema.tikv_region_peers kvrp ON kvrs.region_id = kvrp.region_id JOIN information_schema.tikv_store_status kvss ON kvrp.store_id = kvss.store_id JOIN information_schema.tables t ON t.table_name = kvrs.table_name WHERE t.table_schema='test' AND t.table_name='orders_log' AND kvrs.is_index = 0 AND kvrp.is_leader = 1; ``` ```output tidb:db1> SELECT kvrs.region_id, kvss.store_id, kvrp.is_leader, kvss.label, kvss.start_ts -> FROM information_schema.tikv_region_status kvrs -> JOIN information_schema.tikv_region_peers kvrp -> ON kvrs.region_id = kvrp.region_id -> JOIN information_schema.tikv_store_status kvss -> ON kvrp.store_id = kvss.store_id -> JOIN information_schema.tables t -> ON t.table_name = kvrs.table_name -> WHERE t.table_schema='test' -> AND t.table_name='orders_log' -> AND kvrs.is_index = 0 -> AND kvrp.is_leader = 1; +-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------+---------------------+ | region_id | store_id | is_leader | label | start_ts | +-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------+---------------------+ | 281 | 188 | 1 | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "low"}, {"key": "zone", "value": "az3"}]| 2023-06-08 06:49:39 | | 285 | 188 | 1 | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "low"}, {"key": "zone", "value": "az3"}]| 2023-06-08 06:49:39 | | 277 | 188 | 1 | [{"key": "region", "value": "Seattle"}, {"key": "storage-performance", "value": "low"}, {"key": "zone", "value": "az3"}]| 2023-06-08 06:49:39 | +-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------+---------------------+ 3 rows in set (0.01 sec) ``` + Run a query trace to verify the partition pruning for table `test.orders`. The query scan on the cold storage is anticipated, as you are querying data between 2010 and 2015. From the output, observe that the region with region ID 260 is scanned, which is on `storage-performance:low` labeled TiKV node: ```bash TRACE SELECT item, order_date FROM test.orders WHERE order_date BETWEEN STR_TO_DATE('01/01/2013', '%m/%d/%Y') AND STR_TO_DATE('12/31/2013', '%m/%d/%Y'); ``` ```output tidb:db1> TRACE -> SELECT item, order_date FROM test.orders -> WHERE order_date BETWEEN STR_TO_DATE('01/01/2013', '%m/%d/%Y') AND STR_TO_DATE('12/31/2013', '%m/%d/%Y'); +----------------------------------------------------------------------+-----------------+------------+ | operation | startTS | duration | +----------------------------------------------------------------------+-----------------+------------+ | trace | 07:21:19.269181 | 3.05361ms | | ├─session.ExecuteStmt | 07:21:19.269186 | 993.479µs | | │ ├─executor.Compile | 07:21:19.269203 | 325.317µs | | │ └─session.runStmt | 07:21:19.269551 | 604.178µs | | │ └─TableReaderExecutor.Open | 07:21:19.270058 | 64.011µs | | │ └─distsql.Select | 07:21:19.270078 | 31.097µs | | │ └─regionRequest.SendReqCtx | 07:21:19.270210 | 1.92593ms | | │ └─rpcClient.SendRequest, region ID: 260, type: Cop | 07:21:19.270231 | 1.894279ms | | │ └─tikv.RPC | 07:21:19.270232 | 638.566µs | | │ ├─tikv.Wait | 07:21:19.270232 | 36.129µs | | │ │ └─tikv.GetSnapshot | 07:21:19.270232 | 36.129µs | | │ └─tikv.Process | 07:21:19.270268 | 9.578µs | | │ └─tikv.RocksDBBlockRead | 07:21:19.270268 | 9.578µs | | └─*executor.TableReaderExecutor.Next | 07:21:19.270188 | 2.011021ms | +----------------------------------------------------------------------+-----------------+------------+ 14 rows in set (0.00 sec) ``` + Run a query trace to verify the partition pruning for table `test.orders` again. The query scan on the hot storage is anticipated, as you are querying data after 2015. From the output, observe that the region with region ID 264 is scanned, which is on `storage-performance:high` labeled TiKV node. ```sql TRACE SELECT item, order_date FROM test.orders WHERE order_date BETWEEN STR_TO_DATE('01/01/2018', '%m/%d/%Y') AND STR_TO_DATE('12/31/2020', '%m/%d/%Y'); ``` ```output tidb:db1> TRACE -> SELECT item, order_date FROM test.orders -> WHERE order_date BETWEEN STR_TO_DATE('01/01/2018', '%m/%d/%Y') AND STR_TO_DATE('12/31/2020', '%m/%d/%Y'); +----------------------------------------------------------------------+-----------------+------------+ | operation | startTS | duration | +----------------------------------------------------------------------+-----------------+------------+ | trace | 07:22:01.845194 | 3.520554ms | | ├─session.ExecuteStmt | 07:22:01.845198 | 1.770717ms | | │ ├─executor.Compile | 07:22:01.845213 | 316.007µs | | │ └─session.runStmt | 07:22:01.845548 | 1.403662ms | | │ └─TableReaderExecutor.Open | 07:22:01.846018 | 896.463µs | | │ └─distsql.Select | 07:22:01.846035 | 864.376µs | | │ ├─loadRegion | 07:22:01.846045 | 813.222µs | | │ └─regionRequest.SendReqCtx | 07:22:01.847019 | 1.598742ms | | │ └─rpcClient.SendRequest, region ID: 264, type: Cop | 07:22:01.847036 | 1.569261ms | | │ └─tikv.RPC | 07:22:01.847037 | 1.176324ms | | │ └─tikv.Wait | 07:22:01.847037 | 1ms | | │ └─tikv.GetSnapshot | 07:22:01.847037 | 995.958µs | | └─*executor.TableReaderExecutor.Next | 07:22:01.846982 | 1.689167ms | +----------------------------------------------------------------------+-----------------+------------+ 13 rows in set (0.01 sec) ``` + If you are interested, you can try similar queries against the table `test.orders_log`, which can only be landed on cold TiKV nodes. ### Step 6. Drop the tables and placement policies ```sql DROP TABLE IF EXISTS test.orders; DROP TABLE IF EXISTS test.orders_log; DROP PLACEMENT POLICY IF EXISTS hot; DROP PLACEMENT POLICY IF EXISTS cold; ``` ```output Query OK, 0 rows affected (0.52 sec) Query OK, 0 rows affected (0.52 sec) Query OK, 0 rows affected (0.53 sec) Query OK, 0 rows affected (0.52 sec) ``` ### Step 7. Exit the TiDB connection, and scale in the TiKV nodes ```sql EXIT ``` ```output tidb:db1> EXIT Bye ``` ```bash tiup cluster scale-in tidb-test --node ${HOST_KV1_PRIVATE_IP}:20161 --yes ``` ```output ... Scaled cluster `tidb-test` in successfully ``` ```bash tiup cluster scale-in tidb-test --node ${HOST_KV2_PRIVATE_IP}:20161 --yes ``` ```output ... Scaled cluster `tidb-test` in successfully ``` ```bash tiup cluster scale-in tidb-test --node ${HOST_KV3_PRIVATE_IP}:20161 --yes ``` ```output ... Scaled cluster `tidb-test` in successfully ``` + The dropped `tikv` will become tombstone in several minutes or hours, after that you can use the prune command to clean it. If you don't want to wait, you may use `--force` in the scale in command. However, you may see abnormalities in monitoring service and dashboard. ```bash tiup cluster display tidb-test ``` ```output ... Cluster type: tidb Cluster name: tidb-test Cluster version: v7.5.0 Deploy user: tidb SSH type: builtin Dashboard URL: http://10.90.2.59:2379/dashboard Grafana URL: http://10.90.4.204:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.14:9093 alertmanager 10.90.4.14 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.14:3000 grafana 10.90.4.14 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.254:2379 pd 10.90.1.254 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.2.60:2379 pd 10.90.2.60 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.3.58:2379 pd 10.90.3.58 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.14:9090 prometheus 10.90.4.14 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.2.231:4000 tidb 10.90.2.231 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.3.141:4000 tidb 10.90.3.141 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.4.207:9000 tiflash 10.90.4.207 9000/8123/3930/20170/20292/8234 linux/x86_64 Up /tidb-data/tiflash-9000 /tidb-deploy/tiflash-9000 10.90.1.71:20160 tikv 10.90.1.71 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.71:20161 tikv 10.90.1.71 20161/20181 linux/x86_64 Pending Offline /tidb-data/tikv-20161 /tidb-deploy/tikv-20161 10.90.2.109:20160 tikv 10.90.2.109 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.2.109:20161 tikv 10.90.2.109 20161/20181 linux/x86_64 Pending Offline /tidb-data/tikv-20161 /tidb-deploy/tikv-20161 10.90.3.221:20160 tikv 10.90.3.221 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.221:20161 tikv 10.90.3.221 20161/20181 linux/x86_64 Pending Offline /tidb-data/tikv-20161 /tidb-deploy/tikv-20161 Total nodes: 15 ``` ### Step 8. Prune the scaled-in TiKV + You may clean the TiKV after its status turns into `Tombstone`, which may take several minutes or hours. ```bash tiup cluster display tidb-test ``` ```output tiup is checking updates for component cluster ... Starting component `cluster`: /home/ec2-user/.tiup/components/cluster/v1.13.1/tiup-cluster display tidb-test Cluster type: tidb Cluster name: tidb-test Cluster version: v7.5.1 Deploy user: ec2-user SSH type: builtin Dashboard URL: http://18.141.56.57:2379/dashboard Grafana URL: http://10.90.4.14:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.14:9093 alertmanager 10.90.4.14 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.14:3000 grafana 10.90.4.14 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.254:2379 pd 10.90.1.254 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.2.60:2379 pd 10.90.2.60 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.3.58:2379 pd 10.90.3.58 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.14:9090 prometheus 10.90.4.14 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.2.231:4000 tidb 10.90.2.231 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.3.141:4000 tidb 10.90.3.141 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.4.207:9000 tiflash 10.90.4.207 9000/8123/3930/20170/20292/8234 linux/x86_64 Up /tidb-data/tiflash-9000 /tidb-deploy/tiflash-9000 10.90.1.71:20160 tikv 10.90.1.71 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.1.71:20161 tikv 10.90.1.71 20161/20181 linux/x86_64 Tombstone /tidb-data/tikv-20161 /tidb-deploy/tikv-20161 10.90.2.109:20160 tikv 10.90.2.109 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.2.109:20161 tikv 10.90.2.109 20161/20181 linux/x86_64 Tombstone /tidb-data/tikv-20161 /tidb-deploy/tikv-20161 10.90.3.221:20160 tikv 10.90.3.221 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.221:20161 tikv 10.90.3.221 20161/20181 linux/x86_64 Tombstone /tidb-data/tikv-20161 /tidb-deploy/tikv-20161 Total nodes: 15 There are some nodes can be pruned: Nodes: [10.90.3.221:20161 10.90.1.71:20161 10.90.2.109:20161] You can destroy them with the command: `tiup cluster prune tidb-test` ``` ```bash tiup cluster prune tidb-test --yes ``` ```output ... Destroy success ``` ## Module 5: Monitor TiDB Using the Prometheus and Grafana Monitoring Tools ### Step 1. Check the status of the TiDB cluster ```bash tiup cluster display tidb-test ``` ```output tiup is checking updates for component cluster ... Starting component `cluster`: /home/ec2-user/.tiup/components/cluster/v1.12.1/tiup-cluster display tidb-test Cluster type: tidb Cluster name: tidb-test Cluster version: v7.5.1 Deploy user: tidb SSH type: builtin Dashboard URL: http://10.90.1.157:2379/dashboard Grafana URL: http://10.90.4.229:3000 ID Role Host Ports OS/Arch Status Data Dir Deploy Dir -- ---- ---- ----- ------- ------ -------- ---------- 10.90.4.229:9093 alertmanager 10.90.4.229 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093 10.90.4.229:3000 grafana 10.90.4.229 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000 10.90.1.157:2379 pd 10.90.1.157 2379/2380 linux/x86_64 Up|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.2.207:2379 pd 10.90.2.207 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.3.169:2379 pd 10.90.3.169 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379 10.90.4.229:9090 prometheus 10.90.4.229 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090 10.90.1.127:4000 tidb 10.90.1.127 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.2.227:4000 tidb 10.90.2.227 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000 10.90.4.33:9000 tiflash 10.90.4.33 9000/8123/3930/20170/20292/8234 linux/x86_64 Up /tidb-data/tiflash-9000 /tidb-deploy/tiflash-9000 10.90.1.137:20160 tikv 10.90.1.137 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.2.24:20160 tikv 10.90.2.24 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 10.90.3.198:20160 tikv 10.90.3.198 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160 Total nodes: 12 ``` ### Step 2. Login to Grafana Dashboard + Launch a browser and enter the following URL for `Prometheus` and `Grafana` monitoring: `http://${HOST_MONITOR1_PUBLIC_IP}:3000`. (if you don't know how to get the `${HOST_MONITOR1_PUBLIC_IP}` please ask your instructor for help). Enter `admin` in the `Email or username` field. Enter `admin` in the `Password` field. ![60%](https://lab-static.pingcap.com/quick-demo/GrafanaSignIn_aa_001_20220623.png) ### Step 3. Explore the Grafana Dashboard + On the `Search` page, expand the menu with the name as the `tidb-test`, and then click `tidb-test-Overview`. ![60%](https://lab-static.pingcap.com/quick-demo/GrafanaOverview01_aa_001_20220623.png) ![60%](https://lab-static.pingcap.com/quick-demo/GrafanaOverview02_aa_001_20220623.png) + Expand the `System-Info` menu of `tidb-test-Overview`, you can view the CPU and memory configuration, CPU, memory, IO, and network usage for each node, etc. ![60%](https://lab-static.pingcap.com/quick-demo/SystemInfo_en_001_20220621.png) + Expand the `Service Port Status` of `tidb-test-Overview`, you can view the online nodes number of each service. ![60%](https://lab-static.pingcap.com/quick-demo/ServicePortStatus_en_001_20220621.png) + Expand the PD monitoring page of `tidb-test-Overview`, you can get an overview of the component PD status from the PD dashboard, where the key metrics are displayed. ![60%](https://lab-static.pingcap.com/quick-demo/MonitoringIndicatorsForPD_en_001_20220621.png) + Expand the TiDB monitoring page of `tidb-test-Overview`, you can get an overview of the QPS, TPS, response delay, which serves as the monitoring information to be displayed or reported. ![60%](https://lab-static.pingcap.com/quick-demo/MonitoringIndicatorsForTiDBServer_en_001_20220621.png) + Expand the TiKV monitoring page of `tidb-test-Overview`, you can get an overview of the component TiKV status, where the key metrics are displayed. According to the Performance Map, you can check whether the status of the cluster is as expected. ![60%](https://lab-static.pingcap.com/quick-demo/MonitoringIndicatorsForTiKV_en_001_20220621.png) ## Module 6: Monitoring TiDB by TiDB Dashboard ### Step 1. Login to TiDB Dashboard + Open [TiDB PD Dashboard](http://${LAB:HOST_PD1_PUBLIC_IP}:2379/dashboard), enter `root` as the username and log in. This is also the `root` user of TiDB. ![60%](https://lab-static.pingcap.com/quick-demo/TiDBDashboard_aa_001_20220621.png) ### Step 2. Explore the TiDB Dashboard + Expand the `Overview` page of `TiDB Dashboard`, you can get an overview of the TiDB cluster's queries per second (QPS), execution time, the types of SQL statements that consume the most resources, and other overview information. ![60%](https://lab-static.pingcap.com/quick-demo/DashboardMonitoringMetrics_en_001_20220622.png) + Expand the `Cluster Info` page of `TiDB Dashboard`, you can get an overview of the running status of TiDB, TiKV, PD, TiFlash components in the entire cluster and the running status of the host on which these components are located. If there are TiKV nodes still pending offline, you may see the status of those TiKV as `leaving`. ![60%](https://lab-static.pingcap.com/quick-demo/DashboardClusterInformation_en_001_20220622.png) ## End of Exercises