# Setting Up the Alarm System for the TiDB Database

__*TiDB alarm system*__
__*TiDB Warning Level*__
__*TiDB Emergency Level*__
__*TiDB Severity Level*__
__*TiDB Warning Level*__

+ Warning levels
| Extent | Description |
| :-- | :-- |
| Urgent Level | Highest severity, service unavailable, usually due to service outage or node failure, requiring immediate human intervention |
| Severity Level | Service availability is declining and requires users to pay close attention to abnormal indicators |
| Warning Levels | Alerts for a Problem or Error |

+ metrics
  + `PD_cluster_offline_tikv_nums`

+ Alarm rules:
  + `sum(pd_cluster_status{type="store_down_count"}) > 0`

+ Description of the rules:
  + PD did not receive a TiKV heartbeat for a long time (default configuration is 30 minutes).
-------------------------------------------------------------
For the monitoring method of Prometheus+Grafana, the alarm items can be divided into three categories: emergency level, severity level, and warning level according to the severity level from high to low. This rating applies to the alarm items of each of the following components.

| Extent | Description |
| :-- | :-- |
| Urgent Level | Highest severity, service unavailable, usually due to service outage or node failure, requiring immediate human intervention |
| Severity Level | Service availability is declining and requires users to pay close attention to abnormal indicators |
| Warning Levels | Alerts for a Problem or Error |

Level indicates the highest severity of the emergency level, the service is unavailable. Usually due to service outage or node failure, manual intervention is required at this time. Severity level service availability is reduced, and the user needs to pay close attention to the alert level of the anomaly indicator warning level to a problem or error

 Prometheus + Grafana defines the alert rules and levels according to the preset rules. The rules mentioned above are configured in the monitoring server's /home/tidb/deploy/conf. There are 12 rule configuration files, which can be distinguished by name. We can make related configurations on the web page board.

How do we set up an alarm? For example, if the PD does not receive a TiKV heartbeat for a long time, then we need to trigger an emergency level alarm. The PD*cluster*offline*tikv*star1nums emergency level alarm item has been preset in the alarm system. The contents are as follows:

- Alarm rules:
sum (pd*cluster*Status {type="Store*Down*Count "}) > 0

- Description of the rules:
PD did not receive a TiKV heartbeat for a long time (default configuration is 30 minutes).

- Treatment method:
  - Check that the TiKV process is healthy, the network is isolated, and the load is excessive, and recover service as much as possible.
  - If it is determined that TiKV cannot be recover, it can be offline.

For information on other alarm items, please refer to: https://docs.pingcap.com/zh/tidb/stable/alert-rules

Reference documentation: https://docs.pingcap.com/zh/tidb/stable/alert-rules