# Monitoring stack
The ducument will introduce the current monitoring stack, How to extension and add new scrape targets, dashboards and alerts.

- [Aggregate prometheus of multiple clusters](https://docs.google.com/document/d/1TP-Hoj-FC-DJdTbFQLrEhCy85ii0Nwn72JKMJyfovD4/edit#heading=h.efxhx09hxyyo)

## Overview

- Base
    - **[Prometheus-operator](https://github.com/prometheus-operator/prometheus-operator)** creates/configures/manages Prometheus clusters atop Kubernetes
    - **[Prometheus](https://github.com/prometheus/prometheus)** is a monitoring system and series system, understanding the basic concepts of [prometheus](https://prometheus.io/docs/introduction/overview/) is very important for you to continue learn for DBaaS monitoring stack
    - **[Kube State Metrics](https://github.com/kubernetes/kube-state-metrics)** will expose all k8s object metrics, it's very important for k8s monitoring stack, we use it to monitor resource quota and workload condition. we will use resource quota and kubelet metrics to add dashboards for resource usage and add alerts for `resourceovercommit`. workload condition will used for add alerts, eg: pod crashlookbackoff
    - **[Node Exporter](https://github.com/prometheus/node_exporter)** for machine metrics, eg: filesystem, network, load1
    - **[Alert Manager](https://github.com/prometheus/alertmanager)** handles alerts sent by client applications such as the Prometheus server. we will only deploy alert manager in base cluster, it is a central alert manager. all prometheus or thanos will send alerts to this alert manager
    - **[Thanos](https://github.com/thanos-io/thanos)** is highly available Prometheus setup with long term storage capabilities. you can see our [design](https://docs.google.com/document/d/1TP-Hoj-FC-DJdTbFQLrEhCy85ii0Nwn72JKMJyfovD4/edit#) when introduce Thanos into DBaaS
        - **Thanos Receiver** used for receiving the metrics sent by prometheus and send long term metrics to S3
        - **Thanos Query** used for querying metrics, it provides a global view for all metrics
        - **Thanos Rules** used for adding alerts, currently we add alerts for the shoot clusters because we don't want to modify the gardener code
    - **[Dead Mans Switch](https://github.com/pingcap/dead-mans-switch)** is a bypass monitoring prober
    - **[Blackbox Exporter](https://github.com/prometheus/blackbox_exporter)** is a Blackbox prober exporter, we use it do DNS and http probe for tidbcloud.com
    - **[Grafana](https://github.com/grafana/grafana)** used for visualization of logs and monitoring
- Seed
    - Prometheus-operator
    - Prometheus(managed by Prometheus-operator, deployed in monitoring namespace)
    - Kube State Metrics
    - Node Exporter
    - **Prometheus**(managed by gardener, deployed with controlplane) will aggreate all TiDB(prometheus federated) and shoot metrics, the prometheus fetch metrics via VPN. you can see [detail](https://docs.google.com/document/d/1TP-Hoj-FC-DJdTbFQLrEhCy85ii0Nwn72JKMJyfovD4/edit#) info why use prometheus federated.
- Shoot
    - Kube State Metrics
    - Node Exporter
    - **[TiDB Monitor](https://docs.pingcap.com/tidb-in-kubernetes/stable/monitor-using-tidbmonitor)** is a CRD managed by tidb-operator.

## How to add new scrape target
When you add a new component, you may have some metrics that need to be collected or you want to relabel the current metrics. you can get detail information for prometheus [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config). 

Currently we use prometheus-operator to manage `scrape_config` in base and seed clusters. It's a `ServiceMonitor` CRD object. But in the shoot cluster, it is managed by gardener, gardener uses configmap to manage `scrape_config`. 

- Base & Seed
    - [central and billing service monitor](https://github.com/tidbcloud/dbaas/blob/1ca0f859792da1a6dbf3d5015724efa571c8d44d/deploy/aws/index.ts#L1322-L1364)
    - [cloud quota monitoring service monitor](https://github.com/tidbcloud/infra/blob/9201235d53fef0463200080ad6c4e437656ff5ed/deploy/monitoring/quotas.ts#L344-L368)
- Shoot
    - [TiDB cluster prometheus scrape_config](https://github.com/pingcap/gardener/blob/a12d8830fe1d404e744a6b4f07b9216295a0fa93/charts/seed-monitoring/charts/core/charts/prometheus/templates/config.yaml#L77-L102)


## How to add new alerts
The [prometheus document](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) introduced how to add new alert rules and alert rule attributes.

- Base & Seed
    - [central and billing alert rules](https://github.com/tidbcloud/infra/blob/dec464072fdbdd3aa91ee7cf1147a2612294a701/deploy/monitoring/manifest/dbaas/dbaas-alerts.yaml)
    - [cloud quota alert rules](https://github.com/tidbcloud/infra/blob/9201235d53fef0463200080ad6c4e437656ff5ed/deploy/monitoring/quotas.ts#L370-L412)
    - [kubernetes cluster alert rules](https://github.com/tidbcloud/infra/blob/9201235d53fef0463200080ad6c4e437656ff5ed/deploy/monitoring/manifest/prometheus/prometheus-rules.yaml#L1-L1758)
- Shoot
    - [TiDB alert rules](https://github.com/tidbcloud/infra/blob/9201235d53fef0463200080ad6c4e437656ff5ed/deploy/monitoring/manifest/thanos/thanos-rule-configmap.yaml#L1-L619)

We have some alerts added on Thanos rule, the reason is we do not want to modify gardener code. another reason TiDB can be deployed cross-region in the future. one TiDB cluster might be deployed on multiple k8s clusters. So we can only get complete metrics from thanos.

## How to add new dashboards
We use grafana to visualize metrics and logs. The grafana provides a global view for our monitoring.

```
--
|-- Alerts (promethus alerts metrics, it's used to view alert history)
|-- Capacity Planning
    |-- Cloud Quota (AWS and GCP Cloud Quota metrics)
    |-- Node Quota
    |...
|-- DBaaS
    |-- central
    |-- billing
    |-- Logging (Quickly view and search central and biling logs)
    |-- central-grpc-server (The dashboards contains central server request RED(Rate、Error、Duration) metrics)
|-- Ingress (Kubernetes nginx ingress controller metrics)
|-- Kubernetes (All kubernetes cluster dashboards)
    |-- Kubernetes/Compute Resources/Namespace (you can get pod resource usage from the dashboard)
    |-- Kubernetes/USE Method/Node (cpu、memory、disk...)
    |...
|-- Logs (Used for searching all kubernetes cluster logs)
|-- TiDB (All TiDB cluster dashboards, central TiDB and TiDB cluster in the shoot cluster)
```

Currently, we use configmap to manage Kubernetes dashboards, it is generated by [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus). Other dashboards are managed by files. The reason is the size of the dashboard file exceeds the limit of the configmap.

- [add logging dashboards for quick search](https://github.com/tidbcloud/infra/pull/37)

### Add grafana datasource

Current datasource
 - prometheus: all components metrics in the base cluster, eg: central、billing
 - tidb-cluster & thanos: base and seed and shoot cluster metrics
 - loki: base and seed and shoot cluster logs

If you want to add a new datasource, you can edit [datasource file](https://github.com/tidbcloud/infra/blob/dec464072fdbdd3aa91ee7cf1147a2612294a701/deploy/monitoring/manifest/grafana/grafana-dashboardDatasources.yaml)

### Add grafana folder

If you want to add a new grafana folder, you can edit [folder file](https://github.com/tidbcloud/infra/blob/dec464072fdbdd3aa91ee7cf1147a2612294a701/deploy/monitoring/manifest/grafana/grafana-dashboardSources.yaml)

Below is the kubernetes directory, the dashboards in `options.path` will be displayed in the folder `TiDB`.
```json
{
   "folder":"TiDB",
   "name":"2",
   "options":{
      "path":"/grafana-dashboard-definitions/2"
   },
   "orgId":1,
   "type":"file"
}
```

### Add dashboards to the specific folder

Add dashboards via [Dockerfile](https://github.com/tidbcloud/infra/blob/dec464072fdbdd3aa91ee7cf1147a2612294a701/deploy/monitoring/manifest/grafana/Dockerfile#L9)

If you want add dashboards to TiDB, you need to add dashboards file to `deploy/monitoring/manifest/grafana/tidb-cluster-dashboards/` directory.

Finally execute `deploy/monitoring/manifest/grafana/tidb-cluster-dashboards/update-grafana-version.sh` to modify grafana image version.

## grafana backup and restore

backup
```
kubectl cp monitoring/grafana-legecy-5cc9686866-k5hzd:/var/lib/grafana/ ./grafana
```

restore
```
kubectl cp ./grafana monitoring/grafana-749cd5bd94-cs4cp:/var/lib/
```