groups:
# Be cautious to change the group name unless you know what you are doing
- name: resources
  rules:
  - alert: TotalTiDBMemoryUsageTooHigh
    annotations:
      message: |
        Total TiDB node memory utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiDB to reduce the memory usage percentage of the current workload. To monitor node memory utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiDB node memory utilization across cluster exceeded {thresholdPercentage}% for {triggerInterval} minutes
    expr: |-
      1 - avg by (tidb_cluster_id) (
        (
          node_memory_MemFree_bytes{component="tidb"} +
          node_memory_Buffers_bytes{component="tidb"} +
          node_memory_Cached_bytes{component="tidb"}
        )
        /
        node_memory_MemTotal_bytes{component="tidb"}
      ) > 0.7
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tidb
      ruleID: "10"
      status: active
      canEdit: all
      monitorTarget: "Total TiDB node memory utilization "
      operator: " is greater than {thresholdPercentage}%"
      message: |
        Total TiDB node memory utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiDB to reduce the memory usage percentage of the current workload. To monitor node memory utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiDB node memory utilization across cluster exceeded {thresholdPercentage}% for {triggerInterval} minutes
  - alert: TotalTiKVMemoryUsageTooHigh
    annotations:
      message: |
        Total TiKV node memory utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiKV to reduce the memory usage percentage of the current workload. To monitor node memory utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiKV node memory utilization across cluster exceeded {thresholdPercentage}% for {triggerInterval} minutes
    expr: |-
      1 - avg by (tidb_cluster_id) (
        (
          node_memory_MemFree_bytes{component="tikv"} +
          node_memory_Buffers_bytes{component="tikv"} +
          node_memory_Cached_bytes{component="tikv"}
        )
        /
        node_memory_MemTotal_bytes{component="tikv"}
      ) > 0.7
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tikv
      ruleID: "15"
      status: active
      canEdit: all
      monitorTarget: "Total TiKV node memory utilization"
      operator: " is greater than {thresholdPercentage} %"
      message: |
        Total TiKV node memory utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiKV to reduce the memory usage percentage of the current workload. To monitor node memory utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiKV node memory utilization across cluster exceeded {thresholdPercentage}% for {triggerInterval} minutes
  - alert: TotalTiFlashMemoryUsageTooHigh
    annotations:
      message: |
        Total TiFlash node memory utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiKV to reduce the memory usage percentage of the current workload. To monitor node memory utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiFlash node memory utilization across cluster exceeded {thresholdPercentage}% for {triggerInterval} minutes
    expr: |-
      1 - avg by (tidb_cluster_id) (
        (
          node_memory_MemFree_bytes{component="tiflash"} +
          node_memory_Buffers_bytes{component="tiflash"} +
          node_memory_Cached_bytes{component="tiflash"}
        )
        /
        node_memory_MemTotal_bytes{component="tiflash"}
      ) > 0.7
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tiflash
      ruleID: "20"
      status: active
      canEdit: all
      monitorTarget: "Total TiFlash node memory utilization "
      operator: " is greater than {thresholdPercentage} %"
      message: |
        Total TiFlash node memory utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiKV to reduce the memory usage percentage of the current workload. To monitor node memory utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiFlash node memory utilization across cluster exceeded {thresholdPercentage}% for {triggerInterval} minutes
  - alert: TotalTiDBCPUUsageTooHigh
    annotations:
      message: |
        Total TiDB node CPU utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiDB to reduce the CPU usage percentage of the current workload. To monitor node CPU utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiDB node CPU utilization exceeded {thresholdPercentage}% for {triggerInterval} minutes
    expr: avg(sum by (cpu, instance, tidb_cluster_id) (irate(node_cpu_seconds_total{component="tidb", mode!~"idle"}[2m]))) by (tidb_cluster_id) > 0.8
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tidb
      ruleID: "30"
      status : active
      canEdit : all
      monitorTarget: "Total TiDB node CPU utilization "
      operator: " is greater than {thresholdPercentage}%"
      message: |
        Total TiDB node CPU utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiDB to reduce the CPU usage percentage of the current workload. To monitor node CPU utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiDB node CPU utilization exceeded {thresholdPercentage}% for {triggerInterval} minutes
  - alert: TotalTiKVCPUUsageTooHigh
    annotations:
      message: |
        Total TiKV node CPU utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiKV to reduce the CPU usage percentage of the current workload. To monitor node CPU utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiKV node CPU utilization exceeded {thresholdPercentage}% for {triggerInterval} minutes
    expr: avg(sum by (cpu, instance, tidb_cluster_id) (irate(node_cpu_seconds_total{component="tikv", mode!~"idle"}[2m]))) by (tidb_cluster_id) > 0.8
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tikv
      ruleID: "35"
      status : active
      canEdit : all
      monitorTarget: "Total TiKV node CPU utilization"
      operator: " is greater than {thresholdPercentage} %"
      message: |
        Total TiKV node CPU utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiKV to reduce the CPU usage percentage of the current workload. To monitor node CPU utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiKV node CPU utilization exceeded {thresholdPercentage}% for {triggerInterval} minutes
  - alert: TotalTiFlashCPUUsageTooHigh
    annotations:
      message: |
        Total TiFlash node CPU utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiFlash to reduce the CPU usage percentage of the current workload. To monitor node CPU utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiFlash node CPU utilization exceeded {thresholdPercentage}% for {triggerInterval} minutes
    expr: avg(sum by (cpu, instance, tidb_cluster_id) (irate(node_cpu_seconds_total{component="tiflash", mode!~"idle"}[2m]))) by (tidb_cluster_id) > 0.8
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tiflash
      ruleID: "40"
      status : active
      canEdit : all
      monitorTarget: "Total TiFlash node CPU utilization "
      operator: " is greater than {thresholdPercentage} %"
      message: |
        Total TiFlash node CPU utilization of cluster TiDBClusterName in project TiDBProjectName has exceeded exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider increasing the node number or node size for TiFlash to reduce the CPU usage percentage of the current workload. To monitor node CPU utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Total TiFlash node CPU utilization exceeded {thresholdPercentage}% for {triggerInterval} minutes
  - alert: TiKVStorageUsageTooHigh
    annotations:
      message: |
        Total TiKV storage utilization of cluster TiDBClusterName in project TiDBProjectName exceeds {thresholdPercentage}%. Consider increasing the node number or node storage size for TiKV to increase your storage capacity. To monitor storage utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: TiKV storage utilization exceeds {thresholdPercentage}%
    expr: (sum by (tidb_cluster_id) (tikv_store_size_bytes{type="capacity"}) - sum by(tidb_cluster_id) (tikv_store_size_bytes{type="available"})) / sum by (tidb_cluster_id) (tikv_store_size_bytes{type="capacity"}) > 0.8
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tikv
      ruleID: "45"
      status : active
      canEdit : all
      monitorTarget: "Total TiKV storage utilization "
      operator: " is greater than {thresholdPercentage} %"
      message: |
        Total TiKV storage utilization of cluster TiDBClusterName in project TiDBProjectName exceeds {thresholdPercentage}%. Consider increasing the node number or node storage size for TiKV to increase your storage capacity. To monitor storage utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: TiKV storage utilization exceeds {thresholdPercentage}%
  - alert: TiFlashStorageUsageTooHigh
    annotations:
      message: |
        Total TiFlash storage utilization of cluster TiDBClusterName in project TiDBProjectName exceeds {thresholdPercentage}%. Consider increasing the node number or node storage size for TiFlash to increase your storage capacity. To monitor storage utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: TiFlash storage utilization exceeds {thresholdPercentage}%
    expr: (sum by (tidb_cluster_id) (tiflash_system_current_metric_StoreSizeCapacity) - sum by(tidb_cluster_id) (tiflash_system_current_metric_StoreSizeAvailable)) / sum by (tidb_cluster_id) (tiflash_system_current_metric_StoreSizeCapacity) > 0.8
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tiflash
      ruleID: "50"
      status : active
      canEdit : all
      monitorTarget: "Total TiFlash storage utilization "
      operator: " is greater than {thresholdPercentage} %"
      message: |
        Total TiFlash storage utilization of cluster TiDBClusterName in project TiDBProjectName exceeds {thresholdPercentage}%. Consider increasing the node number or node storage size for TiFlash to increase your storage capacity. To monitor storage utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: TiFlash storage utilization exceeds {thresholdPercentage}%
  - alert: MaxTiDBMemoryUsageTooHigh
    annotations:
      message: |
        The max memory utilization across TiDB nodes of cluster TiDBClusterName in project TiDBProjectName exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider checking if there is any  <a href="https://docs.pingcap.com/tidbcloud/tidb-cloud-sql-tuning-overview#hotspot-issues">Hotspot</a>  in the cluster or increasing the node number or node size for TiDB to reduce the memory usage percentage of the current workload. To monitor node memory utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Max memory utilization across TiDB nodes exceeded {thresholdPercentage}% for {triggerInterval} minutes
    expr: |-
      1 - min by (tidb_cluster_id) (
        avg by (instance, tidb_cluster_id) (
          (
            node_memory_MemFree_bytes{component="tidb"} +
            node_memory_Buffers_bytes{component="tidb"} +
            node_memory_Cached_bytes{component="tidb"}
          )
          /
          node_memory_MemTotal_bytes{component="tidb"}
        )
      ) > 0.7
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tidb
      ruleID: "60"
      status : disabled
      canEdit : all
      monitorTarget: "TiDB node Max memory utilization "
      operator: " is greater than {thresholdPercentage}%"
      message: |
        The max memory utilization across TiDB nodes of cluster TiDBClusterName in project TiDBProjectName exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider checking if there is any  <a href="https://docs.pingcap.com/tidbcloud/tidb-cloud-sql-tuning-overview#hotspot-issues">Hotspot</a>  in the cluster or increasing the node number or node size for TiDB to reduce the memory usage percentage of the current workload. To monitor node memory utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Max memory utilization across TiDB nodes exceeded {thresholdPercentage}% for {triggerInterval} minutes
  - alert: MaxTiDBCPUUsageTooHigh
    annotations:
      message: |
        The max CPU utilization across TiDB nodes of cluster TiDBClusterName in project TiDBProjectName exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider checking if there is any  <a href="https://docs.pingcap.com/tidbcloud/tidb-cloud-sql-tuning-overview#hotspot-issues">Hotspot</a>  in the cluster or increasing the node number or node size for TiDB to reduce the CPU usage percentage of the current workload. To monitor node CPU utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Max CPU utilization across TiDB nodes exceeded {thresholdPercentage}% for {triggerInterval} minutes
    expr: max(avg(sum by (cpu, instance, tidb_cluster_id) (irate(node_cpu_seconds_total{component="tidb", mode!~"idle"}[2m]))) by (instance, tidb_cluster_id))by (tidb_cluster_id) > 0.8
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tidb
      ruleID: "62"
      status : disabled
      canEdit : all
      monitorTarget: "TiDB node Max CPU utilization "
      operator: " is greater than {thresholdPercentage}%"
      message: |
        The max CPU utilization across TiDB nodes of cluster TiDBClusterName in project TiDBProjectName exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider checking if there is any  <a href="https://docs.pingcap.com/tidbcloud/tidb-cloud-sql-tuning-overview#hotspot-issues">Hotspot</a>  in the cluster or increasing the node number or node size for TiDB to reduce the CPU usage percentage of the current workload. To monitor node CPU utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Max CPU utilization across TiDB nodes exceeded {thresholdPercentage}% for {triggerInterval} minutes
  - alert: MaxTiKVMemoryUsageTooHigh
    annotations:
      message: |
        The max memory utilization across TiKV nodes of cluster TiDBClusterName in project TiDBProjectName exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider checking if there is any  <a href="https://docs.pingcap.com/tidbcloud/tidb-cloud-sql-tuning-overview#hotspot-issues">Hotspot</a>  in the cluster or increasing the node number or node size for TiKV to reduce the memory usage percentage of the current workload. To monitor node memory utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Max memory utilization across TiKV nodes exceeded {thresholdPercentage}% for {triggerInterval} minutes
    expr: |-
      1 - min by (tidb_cluster_id) (
        avg by (instance, tidb_cluster_id) (
          (
            node_memory_MemFree_bytes{component="tikv"} +
            node_memory_Buffers_bytes{component="tikv"} +
            node_memory_Cached_bytes{component="tikv"}
          )
          /
          node_memory_MemTotal_bytes{component="tikv"}
        )
      ) > 0.7
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tikv
      ruleID: "61"
      status : disabled
      canEdit : all
      monitorTarget: "TiKV node Max memory utilization "
      operator: " is greater than {thresholdPercentage}%"
      message: |
        The max memory utilization across TiKV nodes of cluster TiDBClusterName in project TiDBProjectName exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider checking if there is any  <a href="https://docs.pingcap.com/tidbcloud/tidb-cloud-sql-tuning-overview#hotspot-issues">Hotspot</a>  in the cluster or increasing the node number or node size for TiKV to reduce the memory usage percentage of the current workload. To monitor node memory utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Max memory utilization across TiKV nodes exceeded {thresholdPercentage}% for {triggerInterval} minutes
  - alert: MaxTiKVCPUUsageTooHigh
    annotations:
      message: |
        The max CPU utilization across TiKV nodes of cluster TiDBClusterName in project TiDBProjectName exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider checking if there is any  <a href="https://docs.pingcap.com/tidbcloud/tidb-cloud-sql-tuning-overview#hotspot-issues">Hotspot</a>  in the cluster or increasing the node number or node size for TiKV to reduce the CPU usage percentage of the current workload. To monitor node CPU utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Max CPU utilization across TiKV nodes exceeded {thresholdPercentage}% for {triggerInterval} minutes
    expr: max(avg(sum by (cpu, instance, tidb_cluster_id) (irate(node_cpu_seconds_total{component="tikv", mode!~"idle"}[2m]))) by (instance, tidb_cluster_id))by (tidb_cluster_id) > 0.8
    for: 10m
    labels:
      visibility: external
      severity: major
      component: tikv
      ruleID: "65"
      status : disabled
      canEdit : all
      monitorTarget: "TiKV node Max CPU utilization "
      operator: " is greater than {thresholdPercentage}%"
      message: |
       The max CPU utilization across TiKV nodes of cluster TiDBClusterName in project TiDBProjectName exceeded {thresholdPercentage}% for {triggerInterval} minutes. Consider checking if there is any  <a href="https://docs.pingcap.com/tidbcloud/tidb-cloud-sql-tuning-overview#hotspot-issues">Hotspot</a>  in the cluster or increasing the node number or node size for TiKV to reduce the CPU usage percentage of the current workload. To monitor node CPU utilization, see <a href="https://TiDBCloudDomainName/console/clusters/{{ $labels.tidb_cluster_id }}/metrics">Metrics page</a> topic.
      description: Max CPU utilization across TiKV nodes exceeded {thresholdPercentage}% for {triggerInterval} minutes