# Lab 5: Monitoring TiDB
### Overview
In this exercise, you monitor a running TiDB cluster using Prometheus and Grafana for monitoring and visualization purposes. The diagram below illustrates the deployment topology of the TiDB cluster.
### Notes
+ Click the private key file [pe-class-key-oregon.pem](lab-controller/class_303_get_ec2_key_for_lab.php?region=oregon) to download it. It is required to log into the Control Machine for this demo.
+ Commands and SQL statements presented on this page are clickable for easy copy and paste.
+ The required IP addresses for specific commands will be substituted with the actual ones automatically once you have set up the lab environment after Step 1.
+ Your environment may be different with these exercises, including but not limited to IP, port, username, password and directory. Example output is displayed in a gray background in this guide.
+ The `$` in your terminal window is an OS prompt. The `tidb>` or `mysql>` in your terminal window is SQL command prompt.
### Assumptions
+ You have downloaded and secured the TiUP Control Machine private key (for example the file is saved under `~/.ssh/` folder locally): `chmod 400 [~/.ssh/pe-class-key-oregon.pem](lab-controller/class_303_get_ec2_key_for_lab.php?region=oregon)`
+ Windows users please refer to this [guide](https://github.com/pingcap/tidb-course-201-lab/blob/master/README_303_v65_en_class_setup_guide_for_student.md) for private key preparation.
## Exercise 1
### Monitor TiDB Using the Prometheus and Grafana Monitoring Tools.
### Overview
+ In this exercise, you monitor a running TiDB cluster using Prometheus and Grafana for monitoring and visualization purposes.
### Duration
+ This exercise should take you approximately 25 minutes to complete.
### Tasks
+ Open a web browser and navigate to the Grafana Dashboard web interface.
+ Check CPU, memory, IO, and network usage for each node under the `System-Info` menu of `tidb-test-Overview`.
+ Check the number of online nodes under the `Service Port Status` of `tidb-test-Overview`.
+ Check the PD status from the PD dashboard under `tidb-test-Overview`.
+ Check the QPS, TPS, response delay, which serves as the monitoring information under the PD monitoring page of `tidb-test-Overview`.
+ Check the status of TiKV under the TiKV monitoring page of `tidb-test-Overview`.
### Solution
1. Login as user ec2-user on monitor1_address. Enter the following command at the terminal prompt and receive the results shown. If you do not possess the credentials to connect, kindly review the Assumptions section under the OVERVIEW tab. The returned message may be slightly different from what is displayed here. If prompted, answer yes.
```command
$ ssh -i ~/.ssh/pe-class-key-oregon.pem ec2-user@monitor1_address
```
```output
, #_
~\_ ####_ Amazon Linux 2
~~ \_#####\
~~ \###|
~~ \#/ ___
~~ V~' '->
~~~ / A newer version of Amazon Linux is available!
~~._. _/
_/ _/ Amazon Linux 2023, GA and supported until 2028-03-15.
_/m/' https://aws.amazon.com/linux/amazon-linux-2023/
```
2. Check the status of the deployed TiDB cluster and view the node ID information. Enter the following command at the terminal prompt and receive the results shown:
```command
$ tiup cluster display tidb-test
```
```output
tiup is checking updates for component cluster ...
Starting component `cluster`: /home/ec2-user/.tiup/components/cluster/v1.12.1/tiup-cluster display tidb-test
Cluster type: tidb
Cluster name: tidb-test
Cluster version: v6.5.0
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://10.90.1.157:2379/dashboard
Grafana URL: http://10.90.4.229:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
10.90.4.229:9093 alertmanager 10.90.4.229 9093/9094 linux/x86_64 Up /tidb-data/alertmanager-9093 /tidb-deploy/alertmanager-9093
10.90.4.229:3000 grafana 10.90.4.229 3000 linux/x86_64 Up - /tidb-deploy/grafana-3000
10.90.1.157:2379 pd 10.90.1.157 2379/2380 linux/x86_64 Up|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379
10.90.2.207:2379 pd 10.90.2.207 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379
10.90.3.169:2379 pd 10.90.3.169 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
10.90.4.229:9090 prometheus 10.90.4.229 9090/12020 linux/x86_64 Up /tidb-data/prometheus-9090 /tidb-deploy/prometheus-9090
10.90.1.127:4000 tidb 10.90.1.127 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
10.90.2.227:4000 tidb 10.90.2.227 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
10.90.4.33:9000 tiflash 10.90.4.33 9000/8123/3930/20170/20292/8234 linux/x86_64 Up /tidb-data/tiflash-9000 /tidb-deploy/tiflash-9000
10.90.1.137:20160 tikv 10.90.1.137 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
10.90.2.24:20160 tikv 10.90.2.24 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
10.90.3.198:20160 tikv 10.90.3.198 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
Total nodes: 12
```
3. Launch a browser and enter the following URL for `Prometheus` and `Grafana` monitoring: `http://${HOST_MONITOR1_PUBLIC_IP}:3000`. (if you don't know how to get the `${HOST_MONITOR1_PUBLIC_IP}` please ask your instructor for help). Enter `admin` in the `Email or username` field. Enter `admin` in the `Password` field.
4. On the `Search` page, expand the menu with the name as the `tidb-test`, and then click `tidb-test-Overview`.
5. Expand the `System-Info` menu of `tidb-test-Overview`, you can view the CPU and memory configuration, CPU, memory, IO, and network usage for each node, etc.
6. Expand the `Service Port Status` of `tidb-test-Overview`, you can view the online nodes number of each service.
7. Expand the PD monitoring page of `tidb-test-Overview`, you can get an overview of the component PD status from the PD dashboard, where the key metrics are displayed.
8. Expand the TiDB monitoring page of `tidb-test-Overview`, you can get an overview of the QPS, TPS, response delay, which serves as the monitoring information to be displayed or reported.
9. Expand the TiKV monitoring page of `tidb-test-Overview`, you can get an overview of the component TiKV status, where the key metrics are displayed. According to the Performance Map, you can check whether the status of the cluster is as expected.
## Exercise 2
### Monitoring TiDB by `TiDB Dashboard`
### Overview
+ In this exercise, you monitor a running TiDB instance by `TiDB Dashboard`.
### Duration
+ This exercise should take you approximately 15 minutes to complete.
### Tasks
+ Launch a browser and go to the webpage for `TiDB Dashboard`.
+ Check QPS, execution time, and the types of SQL statements that consume the most resources under the `Overview` page of `TiDB Dashboard`.
+ Check the running status of TiDB, TiKV, PD, and TiFlash under the `Cluster Info` page of `TiDB Dashboard`.
### Solution
1. Launch a browser and enter the following URL for `TiDB Dashboard`: `http://${HOST_PD1_PUBLIC_IP}:2379/dashboard`. (if you don't know how to get the `${HOST_PD1_PUBLIC_IP}` please ask your instructor for help). Enter `root` in the `Username`. This is also the `root` user of TiDB.
2. Expand the `Overview` page of `TiDB Dashboard`, you can get an overview of the TiDB cluster's queries per second (QPS), execution time, the types of SQL statements that consume the most resources, and other overview information.
3. Expand the `Cluster Info` page of `TiDB Dashboard`, you can get an overview of the running status of TiDB, TiKV, PD, TiFlash components in the entire cluster and the running status of the host on which these components are located.
## End of Exercises