# Custom deployed Metrics data collection tool ## User scenario The user's Prometheus is self-deployed/binary deployed, without using TiUP or operator. ## How to use - TLS enabled cluster Deployment Instructions & Note: - The server that will deploy the clinic tool must be able to communicate with the PD and Prometheus of the TiDB cluster - If TiUP cannot be deployed, please send the operating system type of the Server to a technical support engineer to obtain the corresponding executable file Deploy TiUP 1. Download and install TiUP . ```{.bash .copyable} curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh ``` 2. Declare global environment variables. After TiUP installation is complete, it will prompt for the absolute path of the corresponding profile file. Before executing the following source command, you need to follow the Profile actual location and source the file. ```bash source ${your_shell_profile} ``` ### Install the Diag collection tool Install the diag command, the latest version is installed by default. ```bash tiup install diag ``` Check the installed version of diag to make sure the version number > = 0.7.1. ```bash tiup diag --version ``` ### Collecting data The diag utility supports custom collection times and metrics required to filter TiDB by metrics name: ```{.bash .copyable} tiup diag util metricdump --name ttt --pd="[127.0.0.1:2379](https://www.google.com/url?q=http://127.0.0.1:2379/&sa=D&source=editors&ust=1654220190110669&usg=AOvVaw2rqM0Rvhxv-QVMNlmYuWJy)" --prometheus="[127.0.0.1:9090](https://www.google.com/url?q=http://127.0.0.1:9090/&sa=D&source=editors&ust=1654220190111099&usg=AOvVaw1tAleAJE-7I2286X4D_eP_)" -f="-4h" -t="-2h" --metricsfilter="node,process,probe,TiDB,process,go\_,tikv,pd,grpc,etcd,tiflash,binlog,ticdc,br,lightning,net_conntrack,os_fd,scrape,unistore,up"  \ --ca-file ~/.tiup/storage/cluster/clusters/ttt/tls/ca.crt \ --cert-file ~/.tiup/storage/cluster/clusters/ttt/tls/client.crt \ --key-file ~/.tiup/storage/cluster/clusters/ttt/tls/client.pem ``` Note: 1. If you use an executable (not using TiUP), you can execute it directly through ./diag util metricdump E.g. TiUP installs binary under TiUP main folder: ubuntu@ip-172-31-32-194:~/.tiup/components/diag/v0.7.1 2. If your cluster does not leverage TLS encryption then you can skip the options from “--ca-file” Sample output: ![Diag Clinic Output](/hc/article_attachments/6585798800151/image2.png) Parameter description: - \-F/--from : Specifies the starting point of the collection time. If this parameter is not specified, the default starting point is 2 hours before the current time. To modify the time zone, use the \-f = "12:30 +0800" syntax. If no time zone information is specified in this parameter, such as \+ 0800 , the default time zone is UTC. The format -f = "-4h" is also supported, specifying that the data collection start time is 4 hours ago. - \-T/--to : Specifies the end point of the collection time. If this parameter is not specified, the default end point is the current moment. To modify the time zone, use the \-f = "12:30 +0800" syntax. If no time zone information is specified in this parameter, such as \+ 0800 , the default time zone is UTC. The format -t = "-4h" is also supported, specifying that the end time of data collection is 4 hours ago. - \--Metricsfilter: The prefix of the Prome Metrics name to be collected, multiple prefixes are separated by commas. #### Dashboard and Metrics prefix correspondence table If you only need to collect the metrics required by a certain dashboard, you can refer to the following table | Dashboard name | Metrics filter configuration | Approximate number of metrics | |-----------------|--------------------------------------------------------------------------------------------------------------|-------------------------------| | Node explorer | node,process | 300+ | | Black explorer | probe | 3 | | Timb | tidb,process,go_,pd_client | 292 | | TIKV -Details | tikv,process,node_disk_io_time_seconds_total | 359 | | PD | pd,process,go,grpc,etcd | 255 | | TiFlash-Summary | tiflash | 433 | | ticdc | process,go,ticdc,pd_cluster_tso,grpc_client,pd_schedule,tikv_thread_cpu_seconds_total,tikv_cdc,tikv_grpc_msg | 86 | | binlog | binlog,tidb_server_critical_error_total,go_ | 30 | ### View data locally in the cluster After the data collection is completed, it will prompt the location where the data collection is stored as follows: Dumping metrics Query server 127.0.0.1:9090 ... Done Collected data are stored in `/Users//diag-fS0dC5M0Ltm` Data In the collection file, - cluster.json records the clusterid, cluster name, time interval for data packet collection, and commands used during collection - Under the monitor folder path there is alert information and the original data source information of all metrics. These json files are compressed and cannot be viewed directly. ​​ ### Upload data to server for diagnostics After data collection is completed, TSE should help NA customers to upload data to cloud server hosted in North America. Below steps need to be followed: 1. Associate your account to clinic server by using work email login 2. Choose or create a cluster for the customer (if customer account has not yet been setup, please contact management team) 3. Create an access token for upload[\[a\]](#cmnt1) (Need admin privilege) Get Access Token by clicking “Get Access Token For Diag Tool” button ![Get Access Token](/hc/article_attachments/6585798801687/image1.png) 4. Run below command on your workstation to upload the diagnostic folder (uncompressed) onto the clinic server (using a test server created for Pinterest as example): 1. tiup diag config clinic.token ${token-value} 2. tiup diag upload ${filepath} --endpoint=[http://clinic-prod-ingress-1108543381.us-west-2.elb.amazonaws.com:8085](https://www.google.com/url?q=http://clinic-prod-ingress-1108543381.us-west-2.elb.amazonaws.com:8085/&sa=D&source=editors&ust=1654220190134262&usg=AOvVaw2d8-7UKW29gI1oBJt-4pWA)