groups: # Be cautious to change the group name unless you know what you are doing - name: instance-customer rules: - alert: QPSSuddenDrop expr: |- 1 - sum by (cluster_id) (rate(tidb_executor_statement_total{component="tidb"}[2m])) / min by (cluster_id) ( avg_over_time(sum by (cluster_id) (rate(tidb_executor_statement_total{component="tidb"}[2m]))[1h:]) or avg_over_time(sum by (component, cluster_id) (rate(tidb_executor_statement_total{component="tidb"}[2m]))[2h:]) ) > 0.2 for: 1m labels: tier: next-gen severity: critical component: tidb visibility: external ruleID: "100" canEdit: "all" status: disabled monitorTarget: "QPS of past 2 minutes" operator: "has dropped by more than {thresholdPercentage}%" description: "QPS suddenly dropped by {thresholdPercentage}%" message: | The QPS to cluster TiDBClusterName has dropped by more than {thresholdPercentage}% for {triggerInterval} minutes. Suggested actions: 1. Check if it's caused by increasing query latency. 2. Check if your application is healthy. If this drop is intentional, please ignore this alert. If not, and you cannot identify the root cause, please contact Support immediately. annotations: description: "QPS suddenly dropped by {thresholdPercentage}%" message: | The QPS to cluster TiDBClusterName has dropped by more than {thresholdPercentage}% for {triggerInterval} minutes. Suggested actions: 1. Check if it's caused by increasing query latency. 2. Check if your application is healthy. If this drop is intentional, please ignore this alert. If not, and you cannot identify the root cause, please contact Support immediately. - alert: TiDBQueryP95LatencyIncreased expr: |- histogram_quantile(0.95, sum(rate(tidb_server_handle_query_duration_seconds_bucket{component="tidb", sql_type!="internal"}[1m])) by (le, cluster_id)) *1000 > 200 for: 5m labels: tier: next-gen severity: critical component: tidb visibility: external ruleID: "101" canEdit: "all" status: disabled monitorTarget: "Query P95 Latency within the past 1 minute" operator: "has exceeded {threshold} ms" description: "Query P95 latency exceeded {threshold} ms" message: | Query P95 Latency of cluster TiDBClusterName within the past 1 minute has exceeded {threshold}ms for {triggerInterval} minutes. Recommended Actions: 1. Investigate Slow Queries: Go to Slow Query page. Filter by the recent time range to find newly appeared or slower-running queries. 2. Review Recent Changes: Was there a recent application deployment, schema change, or data import job that could explain the change in traffic pattern? If you cannot identify the root cause, please contact Support immediately. annotations: description: "Query P95 latency exceeded {threshold} ms" message: | Query P95 Latency of cluster TiDBClusterName within the past 1 minute has exceeded {threshold}ms for {triggerInterval} minutes. Recommended Actions: 1. Investigate Slow Queries: Go to Slow Query page. Filter by the recent time range to find newly appeared or slower-running queries. 2. Review Recent Changes: Was there a recent application deployment, schema change, or data import job that could explain the change in traffic pattern? If you cannot identify the root cause, please contact Support immediately. - alert: TiDBQueryP99LatencyIncreased expr: |- histogram_quantile(0.99, sum(rate(tidb_server_handle_query_duration_seconds_bucket{component="tidb", sql_type!="internal"}[1m])) by (le, cluster_id)) *1000 > 200 for: 5m labels: tier: next-gen severity: critical component: tidb visibility: external ruleID: "102" canEdit: "all" status: disabled monitorTarget: "Query P99 Latency within the past 1 minute" operator: "has exceeded {threshold} ms" description: "Query P99 latency exceeded {threshold} ms" message: | Query P99 Latency of cluster TiDBClusterName within the past 1 minute has exceeded {threshold}ms for {triggerInterval} minutes. Recommended Actions: 1. Investigate Slow Queries: Go to Slow Query page. Filter by the recent time range to find newly appeared or slower-running queries. 2. Review Recent Changes: Was there a recent application deployment, schema change, or data import job that could explain the change in traffic pattern? If you cannot identify the root cause, please contact Support immediately. annotations: description: "Query P99 latency exceeded {threshold} ms" message: | Query P99 Latency of cluster TiDBClusterName within the past 1 minute has exceeded {threshold}ms for {triggerInterval} minutes. Recommended Actions: 1. Investigate Slow Queries: Go to Slow Query page. Filter by the recent time range to find newly appeared or slower-running queries. 2. Review Recent Changes: Was there a recent application deployment, schema change, or data import job that could explain the change in traffic pattern? If you cannot identify the root cause, please contact Support immediately. - alert: RequestErrorRateIncreased expr: |- ( sum(increase(tidb_server_execute_error_total{component="tidb"}[5m])) by (cluster_id) / sum(increase(tidb_executor_statement_total{component="tidb"}[5m])) by (cluster_id) ) > 0.1 for: 1m labels: tier: next-gen severity: critical component: tidb visibility: external ruleID: "103" canEdit: "all" status: disabled monitorTarget: "Query Error Rate" operator: "has been above {thresholdPercentage}%" description: "Request Error Rate Over {thresholdPercentage}%" message: | The Query Error Rate of cluster TiDBClusterName has been above {thresholdPercentage}% for {triggerInterval} minutes. Please check the recent errors and overall statement execution status for this cluster. annotations: description: "Request Error Rate Over {thresholdPercentage}%" message: | The Query Error Rate of cluster TiDBClusterName has been above {thresholdPercentage}% for {triggerInterval} minutes. Please check the recent errors and overall statement execution status for this cluster.