groups: # Be cautious to change the group name unless you know what you are doing - name: instance-customer rules: - alert: EssentialHighResourceUnitUsage expr: |- (sum(rate(resource_manager_resource_unit_read_request_unit_sum{}[1m])) by(name) + sum(rate(resource_manager_resource_unit_write_request_unit_sum{}[1m])) by(name) + sum(rate(resource_manager_resource_unit_sql_layer_request_unit_sum{}[1m])) by(name)) / max by(name) (resource_manager_resource_unit_service_limit) > 0.8 for: 5m labels: tier: serverless severity: warning component: tidb visibility: external ruleID: "200" canEdit: "all" status: active monitorTarget: "Request Unit/s" operator: "is greater than {thresholdPercentage}% of Max RCU" description: "The Request Unit/s exceeded {thresholdPercentage}% of Max RCU" message: | The request unit/s of instance TiDBClusterName has exceeded {thresholdPercentage}% of Maximum RCU for more than {triggerInterval} minutes. 1. Check your traffic pattern in the RU metrics to determine if it's a gradual increase or a sharp spike. 2. If it's a gradual increase, verify if Query Duration has increased. If so, your current max RCU configuration is likely insufficient. 3. Scale up your capacity by manually increasing the max RCU in the TiDB Cloud console to meet demand. 4. For long-term optimization, you can identify and optimize queries consuming the most RUs using the slow query feature. If the problem cannot be fixed by your side, you can contact TiDB Cloud Support for further assistance. annotations: description: "The Request Unit/s exceeded {thresholdPercentage}% of Max RCU" message: | The request unit/s of instance TiDBClusterName has exceeded {thresholdPercentage}% of Maximum RCU for more than {triggerInterval} minutes. 1. Check your traffic pattern in the RU metrics to determine if it's a gradual increase or a sharp spike. 2. If it's a gradual increase, verify if Query Duration has increased. If so, your current max RCU configuration is likely insufficient. 3. Scale up your capacity by manually increasing the max RCU in the TiDB Cloud console to meet demand. 4. For long-term optimization, you can identify and optimize queries consuming the most RUs using the slow query feature. If the problem cannot be fixed by your side, you can contact TiDB Cloud Support for further assistance. - alert: EssentialQPSSuddenDrop expr: |- 1 - sum by (cluster_id) (rate(tidb_executor_statement_total[2m])) / min by (cluster_id) ( avg_over_time(sum by (cluster_id) (rate(tidb_executor_statement_total[2m]))[1h:]) or avg_over_time(sum by (component, cluster_id) (rate(tidb_executor_statement_total[2m]))[2h:]) ) > 0.2 for: 5m labels: tier: serverless severity: critical component: tidb visibility: external ruleID: "205" canEdit: "all" status: disabled monitorTarget: "QPS of past 2 minutes" operator: "has dropped by more than {thresholdPercentage}%" description: "QPS suddenly dropped by {thresholdPercentage}%" message: | The QPS to instance TiDBClusterName has dropped by more than {thresholdPercentage}% for {triggerInterval} minutes. Suggested actions: 1. Check if it's caused by increasing query latency. 2. Check if your application is healthy. If this drop is intentional, please ignore this alert. If not, and you cannot identify the root cause, please contact Support immediately. annotations: description: "QPS suddenly dropped by {thresholdPercentage}%" message: | The QPS to instance TiDBClusterName has dropped by more than {thresholdPercentage}% for {triggerInterval} minutes. Suggested actions: 1. Check if it's caused by increasing query latency. 2. Check if your application is healthy. If this drop is intentional, please ignore this alert. If not, and you cannot identify the root cause, please contact Support immediately. - alert: EssentialTiDBQueryP95LatencyIncreased expr: |- histogram_quantile(0.95, sum(rate(tidb_server_handle_query_duration_seconds_bucket{component="tidb", sql_type!="internal"}[1m])) by (vmrange, cluster_id)) * 1000 > 200 for: 60m labels: tier: serverless severity: critical component: tidb visibility: external ruleID: "210" canEdit: "all" status: disabled monitorTarget: "Query P95 Latency within the past 1 minute" operator: "has exceeded {threshold} ms" description: "Query P95 latency exceeded {threshold} ms" message: | Query P95 Latency of instance TiDBClusterName within the past 1 minute has exceeded {threshold}ms for {triggerInterval} minutes. Recommended Actions: 1. Investigate Slow Queries: Go to Slow Query page. Filter by the recent time range to find newly appeared or slower-running queries. 2. Review Recent Changes: Was there a recent application deployment, schema change, or data import job that could explain the change in traffic pattern? If you cannot identify the root cause, please contact Support immediately. annotations: description: "Query P95 latency exceeded {threshold} ms" message: | Query P95 Latency of instance TiDBClusterName within the past 1 minute has exceeded {threshold}ms for {triggerInterval} minutes. Recommended Actions: 1. Investigate Slow Queries: Go to Slow Query page. Filter by the recent time range to find newly appeared or slower-running queries. 2. Review Recent Changes: Was there a recent application deployment, schema change, or data import job that could explain the change in traffic pattern? If you cannot identify the root cause, please contact Support immediately. - alert: EssentialTiDBQueryP99LatencyIncreased expr: |- histogram_quantile(0.99, sum(rate(tidb_server_handle_query_duration_seconds_bucket{component="tidb", sql_type!="internal"}[1m])) by (vmrange, cluster_id)) * 1000 > 200 for: 60m labels: tier: serverless severity: critical component: tidb visibility: external ruleID: "215" canEdit: "all" status: disabled monitorTarget: "Query P99 Latency within the past 1 minute" operator: "has exceeded {threshold} ms" description: "Query P99 latency exceeded {threshold} ms" message: | Query P99 Latency of instance TiDBClusterName within the past 1 minute has exceeded {threshold}ms for {triggerInterval} minutes. Recommended Actions: 1. Investigate Slow Queries: Go to Slow Query page. Filter by the recent time range to find newly appeared or slower-running queries. 2. Review Recent Changes: Was there a recent application deployment, schema change, or data import job that could explain the change in traffic pattern? If you cannot identify the root cause, please contact Support immediately. annotations: description: "Query P99 latency exceeded {threshold} ms" message: | Query P99 Latency of instance TiDBClusterName within the past 1 minute has exceeded {threshold}ms for {triggerInterval} minutes. Recommended Actions: 1. Investigate Slow Queries: Go to Slow Query page. Filter by the recent time range to find newly appeared or slower-running queries. 2. Review Recent Changes: Was there a recent application deployment, schema change, or data import job that could explain the change in traffic pattern? If you cannot identify the root cause, please contact Support immediately. - alert: EssentialRequestErrorRateIncreased expr: |- ( sum(increase(tidb_server_execute_error_total[5m])) by (cluster_id) / sum(increase(tidb_executor_statement_total[5m])) by (cluster_id) ) > 0.1 for: 5m labels: tier: serverless severity: critical component: tidb visibility: external ruleID: "220" canEdit: "all" status: disabled monitorTarget: "Query Error Rate" operator: "has been above {thresholdPercentage}%" description: "Request Error Rate exceeded {thresholdPercentage}%" message: | The Query Error Rate of instance TiDBClusterName has been above {thresholdPercentage}% for {triggerInterval} minutes. Please check the recent errors and overall statement execution status for this instance. annotations: description: "Request Error Rate exceeded {thresholdPercentage}%" message: | The Query Error Rate of instance TiDBClusterName has been above {thresholdPercentage}% for {triggerInterval} minutes. Please check the recent errors and overall statement execution status for this instance.