# TiDB METRICS_SCHEMA - 其他指标 > TiDB 8.5.4 METRICS_SCHEMA > > 参考: https://docs.pingcap.com/zh/tidb/stable/metrics-schema/ ## 说明 **其他指标**是特定场景或高级调试时使用的指标，包括： - 各类详细计数和耗时统计 - 内部实现细节指标 - 特殊功能指标（备份、TiFlash 等） - Node 级别系统指标（如需要可通过 DataPlane 获取） **使用场景**： - 高级调试 - 特定功能分析 - 开发人员排查 ## 统计 | 组件 | 数量 | |------|------| | TiDB | 148 | | TiKV | 300 | | PD | 57 | | TiFlash | 3 | | Node | 52 | | Etcd | 4 | | Other | 8 | | **总计** | **572** | --- ## DataPlane Infrastructure Metrics > These metrics are **not** part of TiDB built-in `METRICS_SCHEMA`. > They are DataPlane / Kubernetes / cloud infrastructure derived queries documented here for dashboard reuse. ### AWS / Kubernetes #### `tikv_exceeded_iops` **说明**: TiKV pod IOPS exceeded flag derived from `aws_ebs_csi_exceeded_iops_seconds_total`. - **Labels**: `pod,volume_id` - **Quantile**: `-` **PromQL**: ```promql avg by (pod,volume_id) ( rate(aws_ebs_csi_exceeded_iops_seconds_total{instance=~"$instance"}[$__rate_interval]) * on(instance) group_left(pod) label_replace( kube_pod_info{namespace=~"$namespace", pod=~".*tikv.*"}, "instance", "$1", "node", "(.*)" ) ) > 0 ``` **注意**: Use this as the **primary** TiKV IOPS limit-breach signal. This query is expected to return data when the dashboard panel has data. If it unexpectedly returns empty, first verify: - `namespace` is expanded to the concrete cluster namespace such as `tidb${cluster_id}` - `instance` keeps the same effective filter as the dashboard - `label_replace(..., "instance", "$1", "node", "(.*)")` preserves the literal `"$1"` and is not mangled by shell / string interpolation If it is still empty through Clinic Data Proxy, use `ClinicAPI.query_exceeded_iops_by_pod(..., component="tikv")`, which internally does `query_series(kube_pod_info)` plus client-side join by `instance=node`. To match the current dashboard more closely, prefer `rate_window="2m"` when using the helper. Only after that should you fall back to `tikv_node_disk_iops`. #### `tiflash_exceeded_iops` **说明**: TiFlash pod IOPS exceeded flag derived from `aws_ebs_csi_exceeded_iops_seconds_total`. - **Labels**: `pod,volume_id` - **Quantile**: `-` **PromQL**: ```promql avg by (pod,volume_id) ( rate(aws_ebs_csi_exceeded_iops_seconds_total{instance=~"$instance"}[$__rate_interval]) * on(instance) group_left(pod) label_replace( kube_pod_info{namespace=~"$namespace", pod=~".*tiflash.*"}, "instance", "$1", "node", "(.*)" ) ) > 0 ``` **注意**: Use this as the **primary** TiFlash IOPS limit-breach signal. This query is expected to return data when the dashboard panel has data. If it unexpectedly returns empty, first verify: - `namespace` is expanded to the concrete cluster namespace such as `tidb${cluster_id}` - `instance` keeps the same effective filter as the dashboard - `label_replace(..., "instance", "$1", "node", "(.*)")` preserves the literal `"$1"` and is not mangled by shell / string interpolation If it is still empty through Clinic Data Proxy, use `ClinicAPI.query_exceeded_iops_by_pod(..., component="tiflash")`, which internally does `query_series(kube_pod_info)` plus client-side join by `instance=node`. To match the current dashboard more closely, prefer `rate_window="2m"` when using the helper. Only after that should you fall back to `tiflash_node_disk_iops`. #### `tikv_node_disk_iops` **说明**: Pod-level storage disk IOPS for TiKV, derived by mapping node disk counters to TiKV pods via `kube_pod_info`. - **Labels**: `pod,device` - **Quantile**: `-` **PromQL**: ```promql sum by (pod,device) ( ( rate(node_disk_reads_completed_total{instance=~"$instance"}[$__rate_interval]) + rate(node_disk_writes_completed_total{instance=~"$instance"}[$__rate_interval]) ) * on(instance) group_left(pod) label_replace( kube_pod_info{namespace=~"$namespace", pod=~".*tikv.*"}, "instance", "$1", "node", "(.*)" ) ) ``` **注意**: If multiple TiKV pods share one node, the same node/device IOPS series is duplicated to each matched pod rather than split by PVC. Use this as a **fallback** / corroborating signal when `tikv_exceeded_iops` returns no data. #### `tiflash_node_disk_iops` **说明**: Pod-level storage disk IOPS for TiFlash, derived by mapping node disk counters to TiFlash pods via `kube_pod_info`. - **Labels**: `pod,device` - **Quantile**: `-` **PromQL**: ```promql sum by (pod,device) ( ( rate(node_disk_reads_completed_total{instance=~"$instance"}[$__rate_interval]) + rate(node_disk_writes_completed_total{instance=~"$instance"}[$__rate_interval]) ) * on(instance) group_left(pod) label_replace( kube_pod_info{namespace=~"$namespace", pod=~".*tiflash.*"}, "instance", "$1", "node", "(.*)" ) ) ``` **注意**: If multiple TiFlash pods share one node, the same node/device IOPS series is duplicated to each matched pod rather than split by PVC. Use this as a **fallback** / corroborating signal when `tiflash_exceeded_iops` returns no data. --- ## Etcd ### Etcd #### `etcd_disk_wal_fsync_rate` **说明**: The rate of writing WAL into the persistent storage - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql delta(etcd_disk_wal_fsync_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `etcd_wal_fsync_duration` **说明**: The quantile time consumed of writing WAL into the persistent storage - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `etcd_wal_fsync_total_count` **说明**: The total count of writing WAL into the persistent storage - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(etcd_disk_wal_fsync_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `etcd_wal_fsync_total_time` **说明**: The total time of writing WAL into the persistent storage - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(etcd_disk_wal_fsync_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ## Node ### CPU #### `node_cpu_usage` **说明**: - - **Labels**: `instance,mode` - **Quantile**: `-` **PromQL**: ```promql sum(rate(node_cpu_seconds_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (mode,instance) * 100 / count(node_cpu_seconds_total{$LABEL_CONDITIONS}) by (mode,instance) or sum(irate(node_cpu_seconds_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (mode,instance) * 100 / count(node_cpu_seconds_total{$LABEL_CONDITIONS}) by (mode,instance) ``` #### `node_virtual_cpus` **说明**: node virtual cpu count - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql count(node_cpu_seconds_total{mode="user"}) by (instance) ``` ### 其他 #### `node_file_descriptor_allocated` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_filefd_allocated{$LABEL_CONDITIONS} ``` #### `node_process_open_fd_count` **说明**: Process opened file descriptors count - **Labels**: `instance,job` - **Quantile**: `-` **PromQL**: ```promql process_open_fds{$LABEL_CONDITIONS} ``` #### `node_processes_blocked` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_procs_blocked{$LABEL_CONDITIONS} ``` #### `node_processes_running` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_procs_running{$LABEL_CONDITIONS} ``` #### `node_uptime` **说明**: node uptime, units are seconds - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_time_seconds{$LABEL_CONDITIONS} - node_boot_time_seconds{$LABEL_CONDITIONS} ``` ### 内存 #### `node_memory_active` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_Active_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_available` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_MemAvailable_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_buffers` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_Buffers_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_cached` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_Cached_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_dirty` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_Dirty_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_free` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_MemFree_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_inactive` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_Inactive_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_mapped` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_Mapped_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_shared` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_Shmem_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_swap_used` **说明**: bytes used of node swap memory - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_SwapTotal_bytes{$LABEL_CONDITIONS} - node_memory_SwapFree_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_usage` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql 100* (1-(node_memory_MemAvailable_bytes{$LABEL_CONDITIONS}/node_memory_MemTotal_bytes{$LABEL_CONDITIONS})) ``` #### `node_memory_writeback` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_Writeback_bytes{$LABEL_CONDITIONS} ``` #### `node_memory_writeback_tmp` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_WritebackTmp_bytes{$LABEL_CONDITIONS} ``` #### `node_total_memory` **说明**: total memory in node - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_memory_MemTotal_bytes{$LABEL_CONDITIONS} ``` ### 内核 #### `node_kernel_context_switches` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql rate(node_context_switches_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) or irate(node_context_switches_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_kernel_forks` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql rate(node_forks_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) or irate(node_forks_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_kernel_interrupts` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql rate(node_intr_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) or irate(node_intr_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` ### 磁盘 #### `node_disk_available_size` **说明**: - - **Labels**: `instance,device,fstype,mountpoint` - **Quantile**: `-` **PromQL**: ```promql node_filesystem_avail_bytes{$LABEL_CONDITIONS} ``` #### `node_disk_io_util` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql rate(node_disk_io_time_seconds_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) or irate(node_disk_io_time_seconds_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_disk_read_latency` **说明**: node disk read latency - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql (rate(node_disk_read_time_seconds_total{$LABEL_CONDITIONS}[$RANGE_DURATION])/ rate(node_disk_reads_completed_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `node_disk_size` **说明**: - - **Labels**: `instance,device,fstype,mountpoint` - **Quantile**: `-` **PromQL**: ```promql node_filesystem_size_bytes{$LABEL_CONDITIONS} ``` #### `node_disk_state` **说明**: - - **Labels**: `instance,device,fstype,mountpoint` - **Quantile**: `-` **PromQL**: ```promql node_filesystem_readonly{$LABEL_CONDITIONS} ``` #### `node_disk_throughput` **说明**: Units is byte - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql irate(node_disk_read_bytes_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) + irate(node_disk_written_bytes_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_disk_usage` **说明**: Filesystem used space. If is > 80% then is Critical. - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql ((node_filesystem_size_bytes{$LABEL_CONDITIONS} - node_filesystem_avail_bytes{$LABEL_CONDITIONS}) / node_filesystem_size_bytes{$LABEL_CONDITIONS}) * 100 ``` #### `node_disk_write_latency` **说明**: node disk write latency - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql (rate(node_disk_write_time_seconds_total{$LABEL_CONDITIONS}[$RANGE_DURATION])/ rate(node_disk_writes_completed_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` ### 系统负载 #### `node_load1` **说明**: 1 minute load averages in node - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_load1{$LABEL_CONDITIONS} ``` #### `node_load15` **说明**: 15 minutes load averages in node - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_load15{$LABEL_CONDITIONS} ``` #### `node_load5` **说明**: 5 minutes load averages in node - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_load5{$LABEL_CONDITIONS} ``` ### 网络 #### `node_network_in_drops` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql rate(node_network_receive_drop_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_network_in_errors` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql rate(node_network_receive_errs_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_network_in_errors_total_count` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql sum(increase(node_network_receive_errs_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by(instance, device) ``` #### `node_network_in_packets` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql rate(node_network_receive_packets_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) or irate(node_network_receive_packets_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_network_in_traffic` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql rate(node_network_receive_bytes_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) or irate(node_network_receive_bytes_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_network_interface_speed` **说明**: node_network_transmit_queue_length = transmit_queue_length value of /sys/class/net/. - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql node_network_transmit_queue_length{$LABEL_CONDITIONS} ``` #### `node_network_out_drops` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql rate(node_network_transmit_drop_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_network_out_errors` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql rate(node_network_transmit_errs_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_network_out_errors_total_count` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql sum(increase(node_network_transmit_errs_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, device) ``` #### `node_network_out_packets` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql rate(node_network_transmit_packets_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) or irate(node_network_transmit_packets_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_network_out_traffic` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql rate(node_network_transmit_bytes_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) or irate(node_network_transmit_bytes_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `node_network_utilization_in_hourly` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql sum(increase(node_network_receive_bytes_total{$LABEL_CONDITIONS}[1h])) ``` #### `node_network_utilization_out_hourly` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql sum(increase(node_network_transmit_bytes_total{$LABEL_CONDITIONS}[1h])) ``` #### `node_tcp_connections` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_netstat_Tcp_CurrEstab{$LABEL_CONDITIONS} ``` #### `node_tcp_in_use` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql node_sockstat_TCP_inuse{$LABEL_CONDITIONS} ``` #### `node_tcp_segments_retransmitted` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql rate(node_netstat_Tcp_RetransSegs{$LABEL_CONDITIONS}[$RANGE_DURATION]) or irate(node_netstat_Tcp_RetransSegs{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` ## Other ### 其他 #### `abnormal_stores` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(pd_cluster_status{ type=~"store_disconnected_count|store_unhealth_count|store_low_space_count|store_down_count|store_offline_count|store_tombstone_count"}) ``` #### `normal_stores` **说明**: The count of healthy stores - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(pd_cluster_status{type="store_up_count"}) by (instance) ``` #### `process_cpu_usage` **说明**: - - **Labels**: `instance,job` - **Quantile**: `-` **PromQL**: ```promql rate(process_cpu_seconds_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `resource_manager_resource_unit` **说明**: The Total RU consumption per second - **Labels**: `-` - **Quantile**: `-` **PromQL**: ```promql sum(rate(resource_manager_resource_unit_read_request_unit_sum{type=~"|tp"}[$RANGE_DURATION])) + sum(rate(resource_manager_resource_unit_write_request_unit_sum{type=~"|tp"}[$RANGE_DURATION])) ``` #### `store_available_ratio` **说明**: It is equal to Store available capacity size over Store capacity size for each TiKV instance - **Labels**: `address,store` - **Quantile**: `-` **PromQL**: ```promql sum(pd_scheduler_store_status{type="store_available"}) by (address, store) / sum(pd_scheduler_store_status{type="store_capacity"}) by (address, store) ``` #### `store_size_amplification` **说明**: The size amplification, which is equal to Store Region size over Store used capacity size, of each TiKV instance - **Labels**: `address,store` - **Quantile**: `-` **PromQL**: ```promql sum(pd_scheduler_store_status{type="region_size"}) by (address, store) / sum(pd_scheduler_store_status{type="store_used"}) by (address, store) * 2^20 ``` #### `up` **说明**: whether the instance is up. 1 is up, 0 is down(off-line) - **Labels**: `instance,job` - **Quantile**: `-` **PromQL**: ```promql up{$LABEL_CONDITIONS} ``` #### `uptime` **说明**: TiDB uptime since last restart(second) - **Labels**: `instance,job` - **Quantile**: `-` **PromQL**: ```promql (time() - process_start_time_seconds{$LABEL_CONDITIONS}) ``` ## PD ### Checker #### `pd_checker_event_count` **说明**: The replica/region checker's status - **Labels**: `instance,name,type` - **Quantile**: `-` **PromQL**: ```promql sum(delta(pd_checker_event_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (name,instance,type) ``` ### Label #### `pd_label_distribution` **说明**: - - **Labels**: `name` - **Quantile**: `-` **PromQL**: ```promql pd_cluster_placement_status{$LABEL_CONDITIONS} ``` ### Operator #### `pd_operator_finish_total_count` **说明**: The total count of the operator is finished - **Labels**: `type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_schedule_finish_operators_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type) ``` #### `pd_operator_finish_total_time` **说明**: The total time consumed when the operator is finished - **Labels**: `type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_schedule_finish_operators_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type) ``` #### `pd_operator_step_finish_duration` **说明**: The quantile time consumed when the operator step is finished - **Labels**: `type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(pd_schedule_finish_operator_steps_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,type)) ``` #### `pd_operator_step_finish_total_count` **说明**: The total count of the operator step is finished - **Labels**: `type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_schedule_finish_operator_steps_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type) ``` #### `pd_operator_step_finish_total_time` **说明**: The total time consumed when the operator step is finished - **Labels**: `type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_schedule_finish_operator_steps_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type) ``` ### Region 管理 #### `pd_region_health` **说明**: It records the unusual Regions' count which may include pending peers, down peers, extra peers, offline peers, missing peers or learner peers - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(pd_regions_status{$LABEL_CONDITIONS}) by (instance, type) ``` #### `pd_region_heartbeat_duration` **说明**: The quantile of heartbeat latency of each TiKV instance in - **Labels**: `address,store` - **Quantile**: `0.99` **PromQL**: ```promql round(histogram_quantile($QUANTILE, sum(rate(pd_scheduler_region_heartbeat_latency_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,address, store)), 1000) ``` #### `pd_region_heartbeat_total_count` **说明**: The total count of heartbeat latency of each TiKV instance in - **Labels**: `address,store` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_scheduler_region_heartbeat_latency_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (address,store) ``` #### `pd_region_heartbeat_total_time` **说明**: The total time of heartbeat latency of each TiKV instance in - **Labels**: `address,store` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_scheduler_region_heartbeat_latency_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (address,store) ``` #### `pd_region_label_isolation_level` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql pd_regions_label_level{$LABEL_CONDITIONS} ``` #### `pd_region_syncer_status` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql pd_region_syncer_status{$LABEL_CONDITIONS} ``` ### TSO 服务 #### `pd_start_tso_wait_duration` **说明**: The quantile duration of the waiting time for getting the start timestamp oracle - **Labels**: `instance` - **Quantile**: `0.999` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_pdclient_ts_future_wait_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `pd_start_tso_wait_total_count` **说明**: The total count of the waiting for getting the start timestamp oracle - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_pdclient_ts_future_wait_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `pd_start_tso_wait_total_time` **说明**: The total time of duration of the waiting time for getting the start timestamp oracle - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_pdclient_ts_future_wait_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `pd_tso_rpc_duration` **说明**: The quantile duration of a client sending TSO request until received the response. - **Labels**: `instance` - **Quantile**: `0.999` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(pd_client_request_handle_requests_duration_seconds_bucket{type="tso"}[$RANGE_DURATION])) by (le,instance)) ``` #### `pd_tso_rpc_total_count` **说明**: The total count of a client sending TSO request until received the response. - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_client_request_handle_requests_duration_seconds_count{type="tso"}[$RANGE_DURATION])) by (instance) ``` #### `pd_tso_rpc_total_time` **说明**: The total time of a client sending TSO request until received the response. - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_client_request_handle_requests_duration_seconds_sum{type="tso"}[$RANGE_DURATION])) by (instance) ``` #### `pd_tso_wait_total_count` **说明**: The total count of a client starting to wait for the TS until received the TS result. - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_client_cmd_handle_cmds_duration_seconds_count{type="wait"}[$RANGE_DURATION])) by (instance) ``` #### `pd_tso_wait_total_time` **说明**: The total time of a client starting to wait for the TS until received the TS result. - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_client_cmd_handle_cmds_duration_seconds_sum{type="wait"}[$RANGE_DURATION])) by (instance) ``` ### gRPC #### `pd_grpc_completed_commands_rate` **说明**: The rate of completing each kind of gRPC commands - **Labels**: `instance,grpc_method` - **Quantile**: `-` **PromQL**: ```promql sum(rate(grpc_server_handling_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (grpc_method,instance) ``` #### `pd_grpc_completed_commands_total_count` **说明**: The total count of completing each kind of gRPC commands - **Labels**: `instance,grpc_method` - **Quantile**: `-` **PromQL**: ```promql sum(increase(grpc_server_handling_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,grpc_method) ``` #### `pd_grpc_completed_commands_total_time` **说明**: The total time of completing each kind of gRPC commands - **Labels**: `instance,grpc_method` - **Quantile**: `-` **PromQL**: ```promql sum(increase(grpc_server_handling_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,grpc_method) ``` ### 其他 #### `pd_cmd_fail_ops` **说明**: pd client command fail count - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(pd_client_cmd_handle_failed_cmds_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `pd_cmd_fail_total_count` **说明**: The total count of pd client command fail - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_client_cmd_handle_failed_cmds_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `pd_handle_transactions_duration` **说明**: The quantile time consumed of handling etcd transactions - **Labels**: `instance,result` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(pd_txn_handle_txns_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, instance, result)) ``` #### `pd_handle_transactions_rate` **说明**: The rate of handling etcd transactions - **Labels**: `instance,result` - **Quantile**: `-` **PromQL**: ```promql sum(rate(pd_txn_handle_txns_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, result) ``` #### `pd_handle_transactions_total_count` **说明**: The total count of handling etcd transactions - **Labels**: `instance,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_txn_handle_txns_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,result) ``` #### `pd_handle_transactions_total_time` **说明**: The total time of handling etcd transactions - **Labels**: `instance,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_txn_handle_txns_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,result) ``` #### `pd_peer_round_trip_duration` **说明**: The quantile latency of the network in .99 - **Labels**: `instance,To` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(etcd_network_peer_round_trip_time_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance,To)) ``` #### `pd_peer_round_trip_total_count` **说明**: The total count of the network in .99 - **Labels**: `instance,To` - **Quantile**: `-` **PromQL**: ```promql sum(increase(etcd_network_peer_round_trip_time_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,To) ``` #### `pd_peer_round_trip_total_time` **说明**: The total time of latency of the network in .99 - **Labels**: `instance,To` - **Quantile**: `-` **PromQL**: ```promql sum(increase(etcd_network_peer_round_trip_time_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,To) ``` #### `pd_request_rpc_duration` **说明**: The quantile of pd client handle request duration(second) - **Labels**: `instance,type` - **Quantile**: `0.999` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(pd_client_request_handle_requests_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,type,instance)) ``` #### `pd_request_rpc_duration_avg` **说明**: - - **Labels**: `type` - **Quantile**: `-` **PromQL**: ```promql avg(rate(pd_client_request_handle_requests_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type) / avg(rate(pd_client_request_handle_requests_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type) ``` #### `pd_request_rpc_ops` **说明**: pd client handle request operation per second - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(pd_client_request_handle_requests_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `pd_request_rpc_total_count` **说明**: The total count of pd client handle request duration(second) - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_client_request_handle_requests_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `pd_request_rpc_total_time` **说明**: The total time of pd client handle request duration(second) - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_client_request_handle_requests_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `pd_role` **说明**: It indicates whether the current PD is the leader or a follower. - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql delta(pd_tso_events{type="save"}[$RANGE_DURATION]) > bool 0 ``` #### `pd_server_etcd_state` **说明**: The current term of Raft - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql pd_server_etcd_state{$LABEL_CONDITIONS} ``` ### 客户端 #### `pd_client_cmd_duration` **说明**: The quantile of pd client command durations - **Labels**: `instance,type` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(pd_client_cmd_handle_cmds_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, type,instance)) ``` #### `pd_client_cmd_ops` **说明**: pd client command ops - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(pd_client_cmd_handle_cmds_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `pd_client_cmd_total_count` **说明**: The total count of pd client command durations - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_client_cmd_handle_cmds_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `pd_client_cmd_total_time` **说明**: The total time of pd client command durations - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_client_cmd_handle_cmds_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` ### 调度 #### `pd_balance_scheduler_status` **说明**: The inner status of balance leader scheduler - **Labels**: `instance,name,type` - **Quantile**: `-` **PromQL**: ```promql sum(delta(pd_scheduler_event_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,name) ``` #### `pd_schedule_filter` **说明**: - - **Labels**: `instance,scope,store,type` - **Quantile**: `-` **PromQL**: ```promql sum(delta(pd_schedule_filter{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (store, type, scope, instance) ``` #### `pd_schedule_operator` **说明**: The number of different operators - **Labels**: `instance,type,event` - **Quantile**: `-` **PromQL**: ```promql sum(delta(pd_schedule_operators_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,event,instance) ``` #### `pd_schedule_operator_total_num` **说明**: The total number of different operators - **Labels**: `instance,type,event` - **Quantile**: `-` **PromQL**: ```promql sum(increase(pd_schedule_operators_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,event,instance) ``` #### `pd_schedule_store_limit` **说明**: - - **Labels**: `instance,store,type` - **Quantile**: `-` **PromQL**: ```promql pd_schedule_store_limit{$LABEL_CONDITIONS} ``` #### `pd_scheduler_balance_direction` **说明**: - - **Labels**: `instance,source,target,type` - **Quantile**: `-` **PromQL**: ```promql sum(delta(pd_scheduler_balance_direction{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,source,target,instance) ``` #### `pd_scheduler_config` **说明**: - - **Labels**: `type` - **Quantile**: `-` **PromQL**: ```promql pd_config_status{$LABEL_CONDITIONS} ``` #### `pd_scheduler_op_influence` **说明**: - - **Labels**: `instance,scheduler,store,type` - **Quantile**: `-` **PromQL**: ```promql pd_scheduler_op_influence{$LABEL_CONDITIONS} ``` #### `pd_scheduler_region_heartbeat` **说明**: - - **Labels**: `instance,address,status,store,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(pd_scheduler_region_heartbeat{$LABEL_CONDITIONS}[$RANGE_DURATION])*60) by (address,instance, store, status,type) ``` #### `pd_scheduler_status` **说明**: - - **Labels**: `instance,kind,type` - **Quantile**: `-` **PromQL**: ```promql pd_scheduler_status{$LABEL_CONDITIONS} ``` #### `pd_scheduler_store_status` **说明**: - - **Labels**: `instance,address,store,type` - **Quantile**: `-` **PromQL**: ```promql pd_scheduler_store_status{$LABEL_CONDITIONS} ``` #### `pd_scheduler_tolerant_resource` **说明**: - - **Labels**: `instance,scheduler,source,target` - **Quantile**: `-` **PromQL**: ```promql pd_scheduler_tolerant_resource{$LABEL_CONDITIONS} ``` ### 集群状态 #### `pd_cluster_metadata` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql pd_cluster_metadata{$LABEL_CONDITIONS} ``` ## TiDB ### Coprocessor #### `tidb_cop_duration` **说明**: The quantile of kv storage coprocessor processing durations - **Labels**: `instance` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_tikvclient_request_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tidb_cop_total_count` **说明**: The total count of kv storage coprocessor processing durations - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_cop_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_cop_total_time` **说明**: The total time of kv storage coprocessor processing durations - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_cop_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### DDL 操作 #### `tidb_ddl_add_index_speed` **说明**: TiDB add index speed - **Labels**: `-` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_ddl_add_index_total[$RANGE_DURATION])) by (type) ``` #### `tidb_ddl_batch_add_index_duration` **说明**: The quantile of TiDB batch add index durations by histogram buckets - **Labels**: `instance,type` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_ddl_batch_add_idx_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, type, instance)) ``` #### `tidb_ddl_batch_add_index_total_count` **说明**: The total count of TiDB batch add index durations by histogram buckets - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_batch_add_idx_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_ddl_batch_add_index_total_time` **说明**: The total time of TiDB batch add index durations by histogram buckets - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_batch_add_idx_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_ddl_deploy_syncer_duration` **说明**: The quantile of TiDB ddl schema syncer statistics, including init, start, watch, clear function call time cost - **Labels**: `instance,type,result` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_ddl_deploy_syncer_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, type, result,instance)) ``` #### `tidb_ddl_deploy_syncer_total_count` **说明**: The total count of TiDB ddl schema syncer statistics, including init, start, watch, clear function call - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_deploy_syncer_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,result) ``` #### `tidb_ddl_deploy_syncer_total_time` **说明**: The total time of TiDB ddl schema syncer statistics, including init, start, watch, clear function call time cost - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_deploy_syncer_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,result) ``` #### `tidb_ddl_meta_opm` **说明**: TiDB different ddl worker numbers - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql increase(tidb_ddl_worker_operation_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `tidb_ddl_total_count` **说明**: The total count of TiDB DDL duration statistics - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_handle_job_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_ddl_total_time` **说明**: The total time of TiDB DDL duration statistics - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_handle_job_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_ddl_update_self_version_duration` **说明**: The quantile of TiDB schema syncer version update time duration - **Labels**: `instance,result` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_ddl_update_self_ver_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, result,instance)) ``` #### `tidb_ddl_update_self_version_total_count` **说明**: The total count of TiDB schema syncer version update - **Labels**: `instance,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_update_self_ver_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,result) ``` #### `tidb_ddl_update_self_version_total_time` **说明**: The total time of TiDB schema syncer version update time duration - **Labels**: `instance,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_update_self_ver_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,result) ``` #### `tidb_ddl_worker_duration` **说明**: The quantile of TiDB ddl worker duration - **Labels**: `instance,type,result,action` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(increase(tidb_ddl_worker_operation_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, type, action, result,instance)) ``` #### `tidb_ddl_worker_total_count` **说明**: The total count of TiDB ddl worker duration - **Labels**: `instance,type,result,action` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_worker_operation_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,result,action) ``` #### `tidb_ddl_worker_total_time` **说明**: The total time of TiDB ddl worker duration - **Labels**: `instance,type,result,action` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_worker_operation_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,result,action) ``` ### GC #### `tidb_gc_delete_range_fail_opm` **说明**: kv storage unsafe destroy range failed counts - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_gc_unsafe_destroy_range_failures{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tidb_gc_delete_range_task_status` **说明**: kv storage delete range task execution status by type - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(tidb_tikvclient_range_task_stats{$LABEL_CONDITIONS}) by (type, result,instance) ``` #### `tidb_gc_fail_opm` **说明**: kv storage garbage collection failing counts - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_gc_failure{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tidb_gc_push_task_duration` **说明**: The quantile of kv storage range worker processing one task duration - **Labels**: `instance,type` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_tikvclient_range_task_push_duration_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,type,instance)) ``` #### `tidb_gc_push_task_total_count` **说明**: The total count of kv storage range worker processing one task duration - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_range_task_push_duration_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_gc_push_task_total_time` **说明**: The total time of kv storage range worker processing one task duration - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_range_task_push_duration_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_gc_too_many_locks_opm` **说明**: kv storage region garbage collection clean too many locks count - **Labels**: `-` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_gc_region_too_many_locks[$RANGE_DURATION])) ``` #### `tidb_gc_total_count` **说明**: The total count of kv storage garbage collection - **Labels**: `instance,stage` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_gc_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,stage) ``` #### `tidb_gc_total_time` **说明**: The total time of kv storage garbage collection time durations - **Labels**: `instance,stage` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_gc_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,stage) ``` #### `tidb_gc_worker_action_opm` **说明**: kv storage garbage collection counts by type - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_gc_worker_actions_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` ### KV 请求 #### `tidb_kv_backoff_total_count` **说明**: The total count of kv backoff - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_backoff_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_kv_backoff_total_time` **说明**: The total time of kv backoff time durations(second) - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_backoff_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_kv_region_error_total_count` **说明**: The total count of kv region error - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_region_err_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tidb_kv_request_total_count` **说明**: The total count of kv requests durations by store - **Labels**: `instance,type,store` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_request_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,store) ``` #### `tidb_kv_request_total_time` **说明**: The total time of kv requests durations by store - **Labels**: `instance,type,store` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_request_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,store) ``` #### `tidb_kv_snapshot_ops` **说明**: using snapshots total - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_tikvclient_snapshot_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_kv_write_num` **说明**: The quantile of kv write count per transaction execution - **Labels**: `instance` - **Quantile**: `1` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_tikvclient_txn_write_kv_num_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, instance)) ``` #### `tidb_kv_write_num_total_count` **说明**: The total count of kv write in transaction execution - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_txn_write_kv_num_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_kv_write_size` **说明**: The quantile of kv write size per transaction execution - **Labels**: `instance` - **Quantile**: `1` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_tikvclient_txn_write_size_bytes_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, instance)) ``` #### `tidb_kv_write_size_total_count` **说明**: The total count of kv write size per transaction execution - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_txn_write_size_bytes_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_kv_write_total_num` **说明**: The total num of kv write in transaction execution - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_txn_write_kv_num_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_kv_write_total_size` **说明**: The total kv write size in transaction execution - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_txn_write_size_bytes_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### Owner 管理 #### `tidb_owner_handle_syncer_duration` **说明**: The quantile of TiDB ddl owner time operations on etcd duration statistics - **Labels**: `instance,type,result` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_ddl_owner_handle_syncer_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, type, result,instance)) ``` #### `tidb_owner_handle_syncer_total_count` **说明**: The total count of TiDB ddl owner operations on etcd - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_owner_handle_syncer_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,result) ``` #### `tidb_owner_handle_syncer_total_time` **说明**: The total time of TiDB ddl owner time operations on etcd duration statistics - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_ddl_owner_handle_syncer_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,result) ``` #### `tidb_owner_watcher_ops` **说明**: TiDB owner watcher counts - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_owner_watch_owner_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type, result, instance) ``` ### Region #### `tidb_region_cache_ops` **说明**: TiDB region cache operations count - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_tikvclient_region_cache_operations_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,result,instance) ``` ### SQL 编译 #### `tidb_compile_total_count` **说明**: The total count of building the query plan(second) - **Labels**: `instance,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_compile_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,sql_type) ``` #### `tidb_compile_total_time` **说明**: The total time of cost of building the query plan(second) - **Labels**: `instance,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_compile_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,sql_type) ``` #### `tidb_parse_total_count` **说明**: The total count of parsing SQL to AST(second) - **Labels**: `instance,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_parse_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,sql_type) ``` #### `tidb_parse_total_time` **说明**: The total time cost of parsing SQL to AST(second) - **Labels**: `instance,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_parse_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,sql_type) ``` ### Schema 加载 #### `tidb_load_schema_total_count` **说明**: The total count of TiDB loading schema by instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_domain_load_schema_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_load_schema_total_time` **说明**: The total time of TiDB loading schema time durations by instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_domain_load_schema_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_schema_lease_error_opm` **说明**: TiDB schema lease error counts - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_schema_lease_error_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_schema_lease_error_total_count` **说明**: The total count of TiDB schema lease error - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_schema_lease_error_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### 事务 #### `tidb_kv_txn_ops` **说明**: TiDB total kv transaction counts - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_tikvclient_txn_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_transaction_local_latch_wait_duration` **说明**: The quantile of TiDB transaction latch wait time on key value storage(second) - **Labels**: `instance` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_tikvclient_local_latch_wait_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tidb_transaction_local_latch_wait_total_count` **说明**: The total count of TiDB transaction latch wait on key value storage(second) - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_local_latch_wait_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_transaction_local_latch_wait_total_time` **说明**: The total time of TiDB transaction latch wait time on key value storage(second) - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_local_latch_wait_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_transaction_retry_error_total_count` **说明**: The total count of transaction retry - **Labels**: `instance,type,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_retry_error_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,sql_type,instance) ``` #### `tidb_transaction_retry_num_total_count` **说明**: The total count of TiDB transaction retry num - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_retry_num_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_transaction_retry_total_num` **说明**: The total num of TiDB transaction retry num - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_retry_num_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_transaction_statement_num` **说明**: The quantile of TiDB statements numbers within one transaction. Internal means TiDB inner transaction - **Labels**: `instance,sql_type` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_session_transaction_statement_num_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance,sql_type)) ``` #### `tidb_transaction_statement_num_total_count` **说明**: The total count of TiDB statements numbers within one transaction. Internal means TiDB inner transaction - **Labels**: `instance,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_transaction_statement_num_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,sql_type) ``` #### `tidb_transaction_statement_total_num` **说明**: The total num of TiDB statements numbers within one transaction. Internal means TiDB inner transaction - **Labels**: `instance,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_transaction_statement_num_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,sql_type) ``` #### `tidb_transaction_total_count` **说明**: The total count of transaction execution durations, including retry(second) - **Labels**: `instance,type,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_transaction_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,sql_type) ``` #### `tidb_transaction_total_time` **说明**: The total time of transaction execution durations, including retry(second) - **Labels**: `instance,type,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_transaction_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,sql_type) ``` #### `tidb_txn_cmd_duration` **说明**: The quantile of TiDB transaction command durations(second) - **Labels**: `instance,type` - **Quantile**: `0.9` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_tikvclient_txn_cmd_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,type,instance)) ``` #### `tidb_txn_cmd_total_count` **说明**: The total count of TiDB transaction command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_txn_cmd_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_txn_cmd_total_time` **说明**: The total time of TiDB transaction command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_txn_cmd_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_txn_region_num` **说明**: The quantile of regions transaction operates on count - **Labels**: `instance` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_tikvclient_txn_regions_num_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, instance)) ``` #### `tidb_txn_region_num_total_count` **说明**: The total count of regions transaction operates on count - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_txn_regions_num_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_txn_region_total_num` **说明**: The total num of regions transaction operates on count - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_txn_regions_num_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### 其他 #### `tidb_auto_id_request_duration` **说明**: The quantile of TiDB auto id requests durations - **Labels**: `instance,type` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_autoid_operation_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, type,instance)) ``` #### `tidb_auto_id_request_total_count` **说明**: The total count of TiDB auto id requests durations - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_autoid_operation_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_auto_id_request_total_time` **说明**: The total time of TiDB auto id requests durations - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_autoid_operation_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_event_opm` **说明**: TiDB Server critical events total, including start/close/shutdown/hang etc - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql increase(tidb_server_event_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `tidb_execute_duration` **说明**: The quantile time cost of executing the SQL which does not include the time to get the results of the query(second) - **Labels**: `instance,sql_type` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_session_execute_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, sql_type, instance)) ``` #### `tidb_execute_total_count` **说明**: The total count of of TiDB executing the SQL - **Labels**: `instance,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_execute_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,sql_type) ``` #### `tidb_execute_total_time` **说明**: The total time cost of executing the SQL which does not include the time to get the results of the query(second) - **Labels**: `instance,sql_type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_session_execute_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,sql_type) ``` #### `tidb_get_token_duration` **说明**: The quantile of Duration (us) for getting token, it should be small until concurrency limit is reached(microsecond) - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_server_get_token_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tidb_get_token_total_count` **说明**: The total count of Duration (us) for getting token, it should be small until concurrency limit is reached - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_get_token_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_get_token_total_time` **说明**: The total time of Duration (us) for getting token, it should be small until concurrency limit is reached(microsecond) - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_get_token_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_keep_alive_opm` **说明**: TiDB instance monitor average keep alive times - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_monitor_keep_alive_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_load_privilege_ops` **说明**: TiDB load privilege counts - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_domain_load_privilege_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_load_safepoint_ops` **说明**: The OPS of load safe point loading - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_tikvclient_load_safepoint_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_load_safepoint_total_num` **说明**: The total count of safe point loading - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_load_safepoint_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_meta_operation_duration` **说明**: The quantile of TiDB meta operation durations including get/set schema and ddl jobs - **Labels**: `instance,type,result` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_meta_operation_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, type,result,instance)) ``` #### `tidb_meta_operation_total_count` **说明**: The total count of TiDB meta operation durations including get/set schema and ddl jobs - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_meta_operation_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,result) ``` #### `tidb_meta_operation_total_time` **说明**: The total time of TiDB meta operation durations including get/set schema and ddl jobs - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_meta_operation_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,result) ``` #### `tidb_ops_internal` **说明**: TiDB internal SQL is used by TiDB itself. - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_session_restricted_sql_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_time_jump_back_ops` **说明**: TiDB monitor time jump back count - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_monitor_time_jump_back_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### 内存 #### `tidb_process_mem_usage` **说明**: process rss memory usage - **Labels**: `instance,job` - **Quantile**: `-` **PromQL**: ```promql process_resident_memory_bytes{$LABEL_CONDITIONS} ``` ### 分布式查询 #### `tidb_distsql_copr_cache` **说明**: The total count of TiDB distsql coprocessor cache - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_distsql_copr_cache{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tidb_distsql_execution_total_count` **说明**: The total count of distsql execution(second) - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_distsql_handle_query_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_distsql_execution_total_time` **说明**: The total time of distsql execution(second) - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_distsql_handle_query_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tidb_distsql_partial_num` **说明**: The quantile of distsql partial numbers per query - **Labels**: `instance` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_distsql_partial_num_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tidb_distsql_partial_num_total_count` **说明**: The total count of distsql partial numbers per query - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_distsql_partial_num_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_distsql_partial_scan_key_num` **说明**: The quantile numebr of distsql partial scan key numbers - **Labels**: `instance` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_distsql_scan_keys_partial_num_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tidb_distsql_partial_scan_key_num_total_count` **说明**: The total count of distsql partial scan key numbers - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_distsql_scan_keys_partial_num_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_distsql_partial_scan_key_total_num` **说明**: The total num of distsql partial scan key numbers - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_distsql_scan_keys_partial_num_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_distsql_partial_total_num` **说明**: The total num of distsql partial numbers per query - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_distsql_partial_num_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_distsql_scan_key_num_total_count` **说明**: The total count of distsql scan numbers - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_distsql_scan_keys_num_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_distsql_scan_key_total_num` **说明**: The total num of distsql scan numbers - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_distsql_scan_keys_num_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### 慢查询分析 #### `tidb_slow_query_cop_process_total_count` **说明**: The total count of TiDB slow query cop process - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_slow_query_cop_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_slow_query_cop_process_total_time` **说明**: The total time of TiDB slow query statistics with slow query total cop process time(second) - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_slow_query_cop_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_slow_query_cop_wait_total_count` **说明**: The total count of TiDB slow query cop wait - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_slow_query_wait_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_slow_query_cop_wait_total_time` **说明**: The total time of TiDB slow query statistics with slow query total cop wait time(second) - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_slow_query_wait_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### 执行器 #### `tidb_expensive_executors_ops` **说明**: TiDB executors using more cpu and memory resources - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_executor_expensive_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` ### 批处理 #### `tidb_batch_client_pending_req_count` **说明**: kv storage batch requests in queue - **Labels**: `instance,store` - **Quantile**: `-` **PromQL**: ```promql sum(tidb_tikvclient_pending_batch_requests{$LABEL_CONDITIONS}) by (store,instance) ``` #### `tidb_batch_client_unavailable_duration` **说明**: The quantile of kv storage batch processing unvailable durations - **Labels**: `instance` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_tikvclient_batch_client_unavailable_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, instance)) ``` #### `tidb_batch_client_unavailable_total_count` **说明**: The total count of kv storage batch processing unvailable durations - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_batch_client_unavailable_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_batch_client_unavailable_total_time` **说明**: The total time of kv storage batch processing unvailable durations - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_batch_client_unavailable_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_batch_client_wait_conn_duration` **说明**: The quantile of batch client wait new connection establish durations - **Labels**: `instance` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_tikvclient_batch_client_wait_connection_establish_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, instance)) ``` #### `tidb_batch_client_wait_conn_total_count` **说明**: The total count of batch client wait new connection establish - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_batch_client_wait_connection_establish_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_batch_client_wait_conn_total_time` **说明**: The total time of batch client wait new connection establish - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_batch_client_wait_connection_establish_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_batch_client_wait_duration` **说明**: The quantile of kv storage batch processing durations, the unit is nanosecond - **Labels**: `instance` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_tikvclient_batch_wait_duration_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, instance)) ``` #### `tidb_batch_client_wait_total_count` **说明**: The total count of kv storage batch processing durations - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_batch_wait_duration_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_batch_client_wait_total_time` **说明**: The total time of kv storage batch processing durations, the unit is nanosecond - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_batch_wait_duration_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### 服务器 #### `tidb_server_maxprocs` **说明**: The Total CPU quota of each TiDB instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql tidb_server_maxprocs{$LABEL_CONDITIONS} ``` ### 查询性能 #### `tidb_auto_id_qps` **说明**: TiDB auto id requests per second including single table/global auto id processing and single table auto id rebase processing - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_autoid_operation_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tidb_distsql_partial_qps` **说明**: the numebr of distsql partial scan numbers - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_distsql_scan_keys_partial_num_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tidb_qps_ideal` **说明**: - - **Labels**: `-` - **Quantile**: `-` **PromQL**: ```promql sum(tidb_server_connections) * sum(rate(tidb_server_handle_query_duration_seconds_count[$RANGE_DURATION])) / sum(rate(tidb_server_handle_query_duration_seconds_sum[$RANGE_DURATION])) ``` #### `tidb_query_using_plan_cache_ops` **说明**: TiDB plan cache hit ops - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_server_plan_cache_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` ### 统计信息 #### `tidb_statistics_auto_analyze_ops` **说明**: TiDB auto analyze query per second - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_statistics_auto_analyze_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tidb_statistics_auto_analyze_total_count` **说明**: The total count of TiDB auto analyze - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_statistics_auto_analyze_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_statistics_auto_analyze_total_time` **说明**: The total time of TiDB auto analyze time durations within 95 percent histogram buckets - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_statistics_auto_analyze_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_statistics_pseudo_estimation_ops` **说明**: TiDB optimizer using pseudo estimation counts - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_statistics_pseudo_estimation_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tidb_statistics_pseudo_estimation_total_count` **说明**: The total count of TiDB optimizer using pseudo estimation - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_statistics_pseudo_estimation_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tidb_statistics_stats_inaccuracy_rate` **说明**: The quantile of TiDB statistics inaccurate rate - **Labels**: `instance` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_statistics_stats_inaccuracy_rate_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tidb_statistics_stats_inaccuracy_rate_total_count` **说明**: The total count of TiDB statistics inaccurate rate - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_statistics_stats_inaccuracy_rate_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_statistics_stats_inaccuracy_total_rate` **说明**: The total time of TiDB statistics inaccurate rate - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_statistics_stats_inaccuracy_rate_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_statistics_update_stats_ops` **说明**: TiDB updating statistics using feed back counts - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_statistics_update_stats_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tidb_statistics_update_stats_total_count` **说明**: The total count of TiDB updating statistics using feed back - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_statistics_update_stats_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` ### 连接管理 #### `tidb_connection_idle_total_count` **说明**: The total count of TiDB connection idle - **Labels**: `instance,in_txn` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_conn_idle_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (in_txn,instance) ``` #### `tidb_connection_idle_total_time` **说明**: The total time of TiDB connection idle - **Labels**: `instance,in_txn` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_conn_idle_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (in_txn,instance) ``` #### `tidb_handshake_error_opm` **说明**: The OPM of TiDB processing handshake error - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_handshake_error_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_handshake_error_total_count` **说明**: The total count of TiDB processing handshake error - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_handshake_error_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_new_etcd_session_duration` **说明**: The quantile of TiDB new session durations for new etcd sessions - **Labels**: `instance,type,result` - **Quantile**: `0.95` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tidb_owner_new_session_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,type,result, instance)) ``` #### `tidb_new_etcd_session_total_count` **说明**: The total count of TiDB new session durations for new etcd sessions - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_owner_new_session_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,result) ``` #### `tidb_new_etcd_session_total_time` **说明**: The total time of TiDB new session durations for new etcd sessions - **Labels**: `instance,type,result` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_owner_new_session_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,result) ``` ### 锁 #### `tidb_lock_resolver_ops` **说明**: lock resolve times - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_tikvclient_lock_resolver_actions_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tidb_lock_resolver_total_num` **说明**: The total number of lock resolve - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_tikvclient_lock_resolver_actions_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` ### 错误监控 #### `tidb_binlog_error_count` **说明**: TiDB write binlog error, skip binlog count - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql tidb_server_critical_error_total{$LABEL_CONDITIONS} ``` #### `tidb_binlog_error_total_count` **说明**: The total count of TiDB write binlog error and skip binlog - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_critical_error_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tidb_load_safepoint_fail_ops` **说明**: safe point update ops - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_tikvclient_load_safepoint_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tidb_lock_cleanup_fail_ops` **说明**: lock cleanup failed ops - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tidb_tikvclient_lock_cleanup_task_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tidb_panic_count` **说明**: TiDB instance panic count - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql increase(tidb_server_panic_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `tidb_panic_count_total_count` **说明**: The total count of TiDB instance panic - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tidb_server_panic_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ## TiFlash ### 其他 #### `tiflash_cpu_quota` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql tiflash_system_current_metric_LogicalCPUCores{$LABEL_CONDITIONS} ``` #### `tiflash_process_cpu_usage` **说明**: - - **Labels**: `instance,job` - **Quantile**: `-` **PromQL**: ```promql rate(tiflash_proxy_process_cpu_seconds_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `tiflash_resource_manager_resource_unit` **说明**: - - **Labels**: `-` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tiflash_compute_request_unit[$RANGE_DURATION])) ``` ## TiKV ### Apply #### `tikv_admin_apply` **说明**: The number of the processed apply command - **Labels**: `instance,type,status` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_admin_cmd_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,status,instance) ``` #### `tikv_apply_avg_wait_duration` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_apply_wait_time_duration_secs_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_raftstore_apply_wait_time_duration_secs_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` ### Coprocessor #### `tikv_cop_dag_executors_ops` **说明**: The number of DAG executors per seconds - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_coprocessor_executor_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_cop_dag_requests_ops` **说明**: - - **Labels**: `instance,vec_type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_coprocessor_dag_request_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (vec_type,instance) ``` #### `tikv_cop_handle_duration` **说明**: The quantile of time consumed when handling coprocessor requests - **Labels**: `instance,req` - **Quantile**: `1` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_coprocessor_request_handle_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,req,instance)) ``` #### `tikv_cop_handle_total_count` **说明**: The total count of tikv coprocessor handling coprocessor requests - **Labels**: `instance,req` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_request_handle_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,req) ``` #### `tikv_cop_handle_total_time` **说明**: The total time of time consumed when handling coprocessor requests - **Labels**: `instance,req` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_request_handle_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,req) ``` #### `tikv_cop_kv_cursor_operations` **说明**: - - **Labels**: `instance,req` - **Quantile**: `1` **PromQL**: ```promql histogram_quantile($QUANTILE, avg(rate(tikv_coprocessor_scan_keys_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,req,instance)) ``` #### `tikv_cop_kv_cursor_operations_total_count` **说明**: - - **Labels**: `instance,req` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_scan_keys_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,req) ``` #### `tikv_cop_request_duration` **说明**: The quantile of time consumed to handle coprocessor read requests - **Labels**: `instance,req` - **Quantile**: `1` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_coprocessor_request_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,req,instance)) ``` #### `tikv_cop_request_durations` **说明**: The time consumed to handle coprocessor read requests - **Labels**: `instance,req` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_coprocessor_request_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance,req) ``` #### `tikv_cop_request_total_count` **说明**: The total count of tikv handle coprocessor read requests - **Labels**: `instance,req` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_request_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,req) ``` #### `tikv_cop_request_total_time` **说明**: The total time of time consumed to handle coprocessor read requests - **Labels**: `instance,req` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_request_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,req) ``` #### `tikv_cop_requests_ops` **说明**: - - **Labels**: `instance,req` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_coprocessor_request_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (req,instance) ``` #### `tikv_cop_scan_details` **说明**: - - **Labels**: `instance,tag,req,cf` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_coprocessor_scan_details{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (tag,req,cf,instance) ``` #### `tikv_cop_scan_details_total` **说明**: - - **Labels**: `instance,tag,req,cf` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_scan_details{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (tag,req,cf,instance) ``` #### `tikv_cop_scan_keys_num` **说明**: - - **Labels**: `instance,req` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_coprocessor_scan_keys_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (req,instance) ``` #### `tikv_cop_scan_keys_total_num` **说明**: - - **Labels**: `instance,req` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_scan_keys_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,req) ``` #### `tikv_cop_total_response_size_per_seconds` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_coprocessor_response_bytes{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_cop_total_response_total_size` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_response_bytes{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_cop_total_rocksdb_perf_statistics` **说明**: - - **Labels**: `instance,req,metric` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_coprocessor_rocksdb_perf{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (req,metric,instance) ``` #### `tikv_cop_wait_duration` **说明**: The quantile of time consumed when coprocessor requests are wait for being handled - **Labels**: `instance,req,type` - **Quantile**: `1` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_coprocessor_request_wait_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,req,type,instance)) ``` #### `tikv_cop_wait_total_count` **说明**: The total count of coprocessor requests that wait for being handled - **Labels**: `instance,req,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_request_wait_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,req,type) ``` #### `tikv_cop_wait_total_time` **说明**: The total time of time consumed when coprocessor requests are wait for being handled - **Labels**: `instance,req,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_request_wait_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,req,type) ``` #### `tikv_coprocessor_is_busy` **说明**: The ops of Coprocessor Full events that make the TiKV instance unavailable temporarily - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_coprocessor_request_error{type='full'}[$RANGE_DURATION])) by (instance,db,type) ``` #### `tikv_coprocessor_is_busy_total_count` **说明**: The total count of Coprocessor Full events that make the TiKV instance unavailable temporarily - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_request_error{type='full'}[$RANGE_DURATION])) by (instance,db,type) ``` #### `tikv_coprocessor_request_error_total_count` **说明**: The total number of different coprocessor errors on each TiKV instance - **Labels**: `instance,reason` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_coprocessor_request_error{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, reason) ``` ### GC #### `tikv_auto_gc_progress` **说明**: Progress of TiKV's GC - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(tikv_gcworker_autogc_processed_regions{type="scan"}) by (instance,type) / sum(tikv_raftstore_region_count{type="region"}) by (instance) ``` #### `tikv_auto_gc_safepoint` **说明**: SafePoint used for TiKV's Auto GC - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql max(tikv_gcworker_autogc_safe_point{$LABEL_CONDITIONS}) by (instance) / (2^18) ``` #### `tikv_auto_gc_working` **说明**: - - **Labels**: `instance,state` - **Quantile**: `-` **PromQL**: ```promql sum(max_over_time(tikv_gcworker_autogc_status{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,state) ``` #### `tikv_gc_fail_tasks` **说明**: The count of GC tasks processed fail by gc_worker - **Labels**: `instance,task` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_gcworker_gc_task_fail_vec{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (task,instance) ``` #### `tikv_gc_keys` **说明**: The count of keys in write CF affected during GC - **Labels**: `instance,tag,cf` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_gcworker_gc_keys{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (tag,cf,instance) ``` #### `tikv_gc_keys_total_num` **说明**: The total number of keys in write CF affected during GC - **Labels**: `instance,tag,cf` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_gcworker_gc_keys{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (tag,cf,instance) ``` #### `tikv_gc_skipped_tasks` **说明**: The count of GC skipped tasks processed by gc_worker - **Labels**: `instance,task` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_storage_gc_skipped_counter{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (task,instance) ``` #### `tikv_gc_speed` **说明**: The GC keys per second - **Labels**: `-` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_storage_mvcc_gc_delete_versions_sum[$RANGE_DURATION])) ``` #### `tikv_gc_tasks_avg_duration` **说明**: The time consumed when executing GC tasks - **Labels**: `instance,task` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_gcworker_gc_task_duration_vec_sum{}[$RANGE_DURATION])) by (task,instance) / sum(rate(tikv_gcworker_gc_task_duration_vec_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (task,instance) ``` #### `tikv_gc_tasks_duration` **说明**: The quantile of time consumed when executing GC tasks - **Labels**: `instance,task` - **Quantile**: `1` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_gcworker_gc_task_duration_vec_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,task,instance)) ``` #### `tikv_gc_tasks_ops` **说明**: The count of GC total tasks processed by gc_worker per second - **Labels**: `instance,task` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_gcworker_gc_tasks_vec{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (task,instance) ``` #### `tikv_gc_tasks_total_count` **说明**: The total count of executing GC tasks - **Labels**: `instance,task` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_gcworker_gc_task_duration_vec_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,task) ``` #### `tikv_gc_tasks_total_time` **说明**: The total time of time consumed when executing GC tasks - **Labels**: `instance,task` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_gcworker_gc_task_duration_vec_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,task) ``` #### `tikv_gc_too_busy` **说明**: The count of GC worker too busy - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_gc_worker_too_busy{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` ### IO #### `tikv_backup_avg_duration` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_backup_request_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_backup_request_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_backup_duration` **说明**: - - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_backup_request_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_backup_range_avg_duration` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_backup_range_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) / sum(rate(tikv_backup_range_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_backup_range_duration` **说明**: - - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_backup_range_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,type,instance)) ``` #### `tikv_check_split_duration` **说明**: The quantile of time consumed when running split check in .9999 - **Labels**: `instance` - **Quantile**: `0.9999` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_raftstore_check_split_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, instance)) ``` #### `tikv_compression_ratio` **说明**: The compression ratio of each level - **Labels**: `instance,level,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_compression_ratio{$LABEL_CONDITIONS}) by (level,instance,db) ``` #### `tikv_io_utilization` **说明**: The I/O utilization per TiKV instance - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql rate(node_disk_io_time_seconds_total{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `tikv_mvcc_delete_versions` **说明**: The number of versions deleted by GC for each key - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_storage_mvcc_gc_delete_versions_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_mvcc_versions` **说明**: The number of versions for each key - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_storage_mvcc_versions_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_propose_avg_wait_duration` **说明**: The average wait time of each proposal - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_request_wait_time_duration_secs_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_raftstore_request_wait_time_duration_secs_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_read_amplication` **说明**: The read amplification per TiKV instance - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_read_amp_flow_bytes{type="read_amp_total_read_bytes"}[$RANGE_DURATION])) by (instance,db) / sum(rate(tikv_engine_read_amp_flow_bytes{type="read_amp_estimate_useful_bytes"}[$RANGE_DURATION])) by (instance,db) ``` #### `tikv_request_batch_ratio` **说明**: The quantile ratio of request batch output to input per TiKV instance - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_server_request_batch_ratio_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,type,instance)) ``` #### `tikv_request_batch_ratio_total_count` **说明**: The total count of request batch output to input per TiKV instance - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_request_batch_ratio_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_request_batch_total_ratio` **说明**: The total ratio of request batch output to input per TiKV instance - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_request_batch_ratio_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_stall_conditions_changed_of_each_cf` **说明**: Stall conditions changed of each column family - **Labels**: `instance,cf,type,db` - **Quantile**: `-` **PromQL**: ```promql tikv_engine_stall_conditions_changed{$LABEL_CONDITIONS} ``` #### `tikv_wal_sync_duration` **说明**: The time consumed when executing WAL sync operation, the unit is microsecond - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_wal_file_sync_micro_seconds{$LABEL_CONDITIONS}) by (db,type,instance) ``` #### `tikv_wal_sync_max_duration` **说明**: The max time consumed when executing WAL sync operation, the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_wal_file_sync_micro_seconds{$LABEL_CONDITIONS}) by (db,type,instance) ``` #### `tikv_write_stall_avg_duration` **说明**: The time which is caused by write stall - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_write_stall{$LABEL_CONDITIONS}) by (type,instance,db) ``` #### `tikv_write_stall_max_duration` **说明**: The time which is caused by write stall - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_write_stall{$LABEL_CONDITIONS}) by (type,instance,db) ``` ### Leader/Learner #### `tikv_active_written_leaders` **说明**: The number of leaders being written on each TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_region_written_keys_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_leader_missing` **说明**: The count of missing leaders per TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(tikv_raftstore_leader_missing{$LABEL_CONDITIONS}) by (instance) ``` ### PD 通信 #### `tikv_pd_heartbeat` **说明**: The total number of the gRPC message failures - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(delta(tikv_pd_heartbeat_message_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_pd_heartbeats` **说明**: The total number of PD heartbeat messages - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_pd_heartbeat_message_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type ,instance) ``` #### `tikv_pd_request_avg_duration` **说明**: The time consumed by requests that TiKV sends to PD - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_pd_request_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type ,instance) / sum(rate(tikv_pd_request_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_pd_request_duration` **说明**: - - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_pd_request_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance,type)) ``` #### `tikv_pd_request_ops` **说明**: The OPS of requests that TiKV sends to PD - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_pd_request_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_pd_request_total_count` **说明**: The count of requests that TiKV sends to PD - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_pd_request_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_pd_request_total_time` **说明**: The count of requests that TiKV sends to PD - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_pd_request_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_pd_validate_peers` **说明**: The total number of peers validated by the PD worker - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_pd_validate_peer_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type ,instance) ``` ### Raft 共识 #### `tikv_config_raftstore` **说明**: TiKV rocksdb config value - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql tikv_config_raftstore{$LABEL_CONDITIONS} ``` #### `tikv_raft_dropped_messages` **说明**: The number of dropped Raft messages per type - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_raft_dropped_message_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_raft_dropped_messages_total` **说明**: The total number of dropped Raft messages per type - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_raft_dropped_message_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_raft_log_speed` **说明**: The rate at which peers propose logs - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql avg(rate(tikv_raftstore_propose_log_size_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raft_message_avg_batch_size` **说明**: - - **Labels**: `-` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_server_raft_message_batch_size_sum[$RANGE_DURATION])) / sum(rate(tikv_server_raft_message_batch_size_count[$RANGE_DURATION])) ``` #### `tikv_raft_message_batch_size` **说明**: - - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_server_raft_message_batch_size_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_raft_message_batch_size_total_count` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_raft_message_batch_size_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raft_message_batch_total_size` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_raft_message_batch_size_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raft_proposals` **说明**: The number of proposals per type in raft - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_proposal_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_raft_proposals_per_ready` **说明**: The quantile proposal count of all Regions in a mio tick - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_raftstore_apply_proposal_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le, instance)) ``` #### `tikv_raft_proposals_per_ready_total_count` **说明**: The total count of proposal count of all Regions in a mio tick - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_apply_proposal_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raft_proposals_per_total_ready` **说明**: The total proposal count of all Regions in a mio tick - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_apply_proposal_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raft_proposals_total_num` **说明**: The total number of proposals per type in raft - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_proposal_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_raft_sent_messages` **说明**: The number of Raft messages sent by each TiKV instance - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_raft_sent_message_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_raft_sent_messages_total_num` **说明**: The total number of Raft messages sent by each TiKV instance - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_raft_sent_message_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_raft_store_events_duration` **说明**: The quantile time consumed by raftstore events (P99).99 - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_raftstore_event_duration_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,type,instance)) ``` #### `tikv_raft_store_events_total_count` **说明**: The total count of raftstore events (P99).99 - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_event_duration_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_raft_store_events_total_time` **说明**: The total time of raftstore events (P99).99 - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_event_duration_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_raftstore_append_log_avg_duration` **说明**: The avg time consumed when Raft appends log - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_append_log_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_raftstore_append_log_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_raftstore_append_log_duration` **说明**: The quantile time consumed when Raft appends log - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_raftstore_append_log_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_raftstore_append_log_total_count` **说明**: The total count of Raft appends log - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_append_log_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raftstore_append_log_total_time` **说明**: The total time of Raft appends log - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_append_log_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raftstore_apply_log_avg_duration` **说明**: The average time consumed when Raft applies log - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_apply_log_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_raftstore_apply_log_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_raftstore_apply_log_total_count` **说明**: The total count of Raft applies log - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_apply_log_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raftstore_apply_log_total_time` **说明**: The total time of Raft applies log - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_apply_log_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raftstore_apply_wait_duration` **说明**: - - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_raftstore_apply_wait_time_duration_secs_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_raftstore_apply_wait_total_count` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_apply_wait_time_duration_secs_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raftstore_apply_wait_total_time` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_apply_wait_time_duration_secs_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raftstore_commit_log_avg_duration` **说明**: The time consumed when Raft commits log - **Labels**: `-` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_commit_log_duration_seconds_sum[$RANGE_DURATION])) / sum(rate(tikv_raftstore_commit_log_duration_seconds_count[$RANGE_DURATION])) ``` #### `tikv_raftstore_commit_log_total_count` **说明**: The total count of Raft commits log - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_commit_log_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raftstore_commit_log_total_time` **说明**: The total time of Raft commits log - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_commit_log_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raftstore_process_duration` **说明**: The quantile time consumed for peer processes in Raft - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_raftstore_raft_process_duration_secs_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance,type)) ``` #### `tikv_raftstore_process_handled` **说明**: The count of different process type of Raft - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_raft_process_duration_secs_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_raftstore_process_total_count` **说明**: The total count of peer processes in Raft - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_raft_process_duration_secs_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_raftstore_process_total_time` **说明**: The total time of peer processes in Raft - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_raft_process_duration_secs_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_raftstore_propose_wait_duration` **说明**: The quantile wait time of each proposal - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_raftstore_request_wait_time_duration_secs_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_raftstore_propose_wait_total_count` **说明**: The total count of each proposal - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_request_wait_time_duration_secs_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_raftstore_propose_wait_total_time` **说明**: The total time of wait time of each proposal - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_request_wait_time_duration_secs_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### Region 管理 #### `tikv_approximate_avg_region_size` **说明**: The avg approximate Region size - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_region_size_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_raftstore_region_size_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_approximate_region_size` **说明**: The quantile of approximate Region size, the default value is P99 - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_raftstore_region_size_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_approximate_region_size_histogram` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_region_size_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_approximate_region_size_total_count` **说明**: The total count of approximate Region size, the default value is P99 - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_region_size_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_approximate_region_total_size` **说明**: The total size of approximate Region size - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_region_size_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_region_average_written_bytes` **说明**: The average rate of writing bytes to Regions per TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_region_written_bytes_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) / sum(rate(tikv_region_written_bytes_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_region_average_written_keys` **说明**: The average rate of written keys to Regions per TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_region_written_keys_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) / sum(rate(tikv_region_written_keys_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_region_change` **说明**: The count of region change per TiKV instance - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(delta(tikv_raftstore_region_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_region_written_bytes` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_region_written_bytes_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_region_written_keys` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_region_written_keys_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` ### gRPC 通信 #### `tikv_average_grpc_messge_duration` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_grpc_msg_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) / sum(rate(tikv_grpc_msg_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_grpc_avg_req_batch_size` **说明**: - - **Labels**: `-` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_server_grpc_req_batch_size_sum[$RANGE_DURATION])) / sum(rate(tikv_server_grpc_req_batch_size_count[$RANGE_DURATION])) ``` #### `tikv_grpc_avg_resp_batch_size` **说明**: - - **Labels**: `-` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_server_grpc_resp_batch_size_sum[$RANGE_DURATION])) / sum(rate(tikv_server_grpc_resp_batch_size_count[$RANGE_DURATION])) ``` #### `tikv_grpc_error_total_count` **说明**: The total count of the gRPC message failures - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_grpc_msg_fail_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_grpc_errors` **说明**: The OPS of the gRPC message failures - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_grpc_msg_fail_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_grpc_message_duration` **说明**: The quantile execution time of gRPC message - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_grpc_msg_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,type,instance)) ``` #### `tikv_grpc_message_total_count` **说明**: The total count of tikv execution gRPC message - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_grpc_msg_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_grpc_message_total_time` **说明**: The total time of execution time of gRPC message - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_grpc_msg_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_grpc_req_batch_size` **说明**: - - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_server_grpc_req_batch_size_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_grpc_req_batch_size_total_count` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_grpc_req_batch_size_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_grpc_req_batch_total_size` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_grpc_req_batch_size_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_grpc_resp_batch_size` **说明**: - - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_server_grpc_resp_batch_size_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_grpc_resp_batch_size_total_count` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_grpc_resp_batch_size_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_grpc_resp_batch_total_size` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_grpc_resp_batch_size_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### 其他 #### `tikv_allocator_stats` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql tikv_allocator_stats{$LABEL_CONDITIONS} ``` #### `tikv_channel_full` **说明**: The ops of channel full errors on each TiKV instance, it will make the TiKV instance unavailable temporarily - **Labels**: `instance,db,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_channel_full_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,db) ``` #### `tikv_channel_full_total_count` **说明**: The total number of channel full errors on each TiKV instance, it will make the TiKV instance unavailable temporarily - **Labels**: `instance,db,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_channel_full_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,db) ``` #### `tikv_check_split` **说明**: The number of raftstore split checks - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_check_split_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_check_split_total_count` **说明**: The total count of running split check - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_check_split_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_check_split_total_time` **说明**: The total time of time consumed when running split check in .9999 - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_check_split_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_client_task_progress` **说明**: The progress of tikv client task - **Labels**: `result,type` - **Quantile**: `-` **PromQL**: ```promql max(tidb_tikvclient_range_task_stats{$LABEL_CONDITIONS}) by (result,type) ``` #### `tikv_corrrput_keys_flow` **说明**: The flow of corrupt operations on keys - **Labels**: `instance,db,cf` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_compaction_num_corrupt_keys{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,cf,instance) ``` #### `tikv_cpu_quota` **说明**: The Total CPU quota of each TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql tikv_server_cpu_cores_quota{$LABEL_CONDITIONS} ``` #### `tikv_critical_error` **说明**: The OPS of the TiKV critical error - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_critical_error_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, type) ``` #### `tikv_critical_error_total_count` **说明**: The total number of the TiKV critical error - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_critical_error_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, type) ``` #### `tikv_disk_read_bytes` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql sum(irate(node_disk_read_bytes_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,device) ``` #### `tikv_disk_write_bytes` **说明**: - - **Labels**: `instance,device` - **Quantile**: `-` **PromQL**: ```promql sum(irate(node_disk_written_bytes_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,device) ``` #### `tikv_flow_mbps` **说明**: The total bytes of read and write in each TiKV instance - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_flow_bytes{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type,db) ``` #### `tikv_flush_messages` **说明**: The number of Raft messages flushed by each TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_server_raft_message_flush_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_flush_messages_total_num` **说明**: The total number of Raft messages flushed by each TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_raft_message_flush_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_local_reader_execute_requests` **说明**: The number of total requests from the local read thread - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_local_read_executed_requests{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_local_reader_reject_requests` **说明**: The number of rejections from the local read thread - **Labels**: `instance,reason` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_local_read_reject_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, reason) ``` #### `tikv_number_files_at_each_level` **说明**: The number of SST files for different column families in each level - **Labels**: `instance,cf,level,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_num_files_at_level{$LABEL_CONDITIONS}) by (cf, level,db,instance) ``` #### `tikv_per_read_avg_bytes` **说明**: The avg bytes per read - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_bytes_per_read{$LABEL_CONDITIONS}) by (type,db,instance) ``` #### `tikv_per_read_max_bytes` **说明**: The max bytes per read - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_bytes_per_read{$LABEL_CONDITIONS}) by (type,db,instance) ``` #### `tikv_per_write_avg_bytes` **说明**: The avg bytes per write - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_bytes_per_write{$LABEL_CONDITIONS}) by (type,db,instance) ``` #### `tikv_per_write_max_bytes` **说明**: The max bytes per write - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_bytes_per_write{$LABEL_CONDITIONS}) by (type,db,instance) ``` #### `tikv_ready_handled` **说明**: The count of ready handled of Raft - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_raftstore_raft_ready_handled_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_receive_messages` **说明**: The number of Raft messages received by each TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_server_raft_message_recv_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_receive_messages_total_num` **说明**: The total number of Raft messages received by each TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_raft_message_recv_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_request_batch_avg` **说明**: The ratio of request batch output to input per TiKV instance - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_server_request_batch_ratio_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) / sum(rate(tikv_server_request_batch_ratio_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_request_batch_size` **说明**: The quantile size of requests into request batch per TiKV instance - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_server_request_batch_size_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,type,instance)) ``` #### `tikv_request_batch_size_avg` **说明**: The avg size of requests into request batch per TiKV instance - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_server_request_batch_size_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) / sum(rate(tikv_server_request_batch_size_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_request_batch_size_total_count` **说明**: The total count of request batch per TiKV instance - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_request_batch_size_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_request_batch_total_size` **说明**: The total size of requests into request batch per TiKV instance - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_request_batch_size_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_total_keys` **说明**: The count of keys in each column family - **Labels**: `instance,db,cf` - **Quantile**: `-` **PromQL**: ```promql sum(tikv_engine_estimate_num_keys{$LABEL_CONDITIONS}) by (db,cf,instance) ``` #### `tikv_worker_handled_tasks` **说明**: The number of tasks handled by worker - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_worker_handled_task_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (name,instance) ``` #### `tikv_worker_handled_tasks_total_num` **说明**: Total number of tasks handled by worker - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_worker_handled_task_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (name,instance) ``` #### `tikv_worker_pending_tasks` **说明**: Current pending and running tasks of worker - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_worker_pending_task_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (name,instance) ``` #### `tikv_worker_pending_tasks_total_num` **说明**: Total pending and running tasks of worker - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_worker_pending_task_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (name,instance) ``` #### `tikv_write_stall_reason` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_engine_write_stall_reason{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,type,instance) ``` ### 内存 #### `tikv_memory` **说明**: The memory usage per TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql avg(process_resident_memory_bytes{$LABEL_CONDITIONS}) by (instance) ``` #### `tikv_memtable_hit` **说明**: The hit rate of memtable - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_memtable_efficiency{type="memtable_hit"}[$RANGE_DURATION])) by (instance,db) / (sum(rate(tikv_engine_memtable_efficiency{}[$RANGE_DURATION])) by (instance,db) + sum(rate(tikv_engine_memtable_efficiency{}[$RANGE_DURATION])) by (instance,db)) ``` #### `tikv_memtable_size` **说明**: The memtable size of each column family - **Labels**: `instance,cf,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_memory_bytes{$LABEL_CONDITIONS}) by (type,instance,db,cf) ``` ### 备份 #### `tikv_backup_errors` **说明**: - - **Labels**: `instance,error` - **Quantile**: `-` **PromQL**: ```promql rate(tikv_backup_error_counter{$LABEL_CONDITIONS}[$RANGE_DURATION]) ``` #### `tikv_backup_errors_total_count` **说明**: - - **Labels**: `instance,error` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_backup_error_counter{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,error) ``` #### `tikv_backup_flow` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_backup_range_size_bytes_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_backup_range_size` **说明**: - - **Labels**: `instance,cf` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_backup_range_size_bytes_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,cf,instance)) ``` #### `tikv_backup_range_size_total_count` **说明**: - - **Labels**: `instance,cf` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_backup_range_size_bytes_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,cf) ``` #### `tikv_backup_range_total_count` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_backup_range_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_backup_range_total_size` **说明**: - - **Labels**: `instance,cf` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_backup_range_size_bytes_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,cf) ``` #### `tikv_backup_range_total_time` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_backup_range_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_backup_total_count` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_backup_request_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_backup_total_time` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_backup_request_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### 存储层 #### `tikv_storage_async_request_avg_duration` **说明**: The time consumed by processing asynchronous snapshot requests - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_storage_engine_async_request_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_storage_engine_async_request_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_storage_async_request_total_count` **说明**: The total count of processing asynchronous snapshot requests - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_storage_engine_async_request_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_storage_async_request_total_time` **说明**: The total time of time consumed by processing asynchronous snapshot requests - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_storage_engine_async_request_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_storage_async_requests` **说明**: The number of different raftstore errors on each TiKV instance - **Labels**: `instance,status,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_storage_engine_async_request_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, status, type) ``` #### `tikv_storage_async_requests_total_count` **说明**: The total number of different raftstore errors on each TiKV instance - **Labels**: `instance,status,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_storage_engine_async_request_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, status, type) ``` #### `tikv_storage_command_ops` **说明**: The total count of different kinds of commands received per seconds - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_storage_command_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` ### 存储引擎 #### `tikv_compaction_max_duration` **说明**: The time consumed when executing the compaction and flush operations - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_compaction_time{$LABEL_CONDITIONS}) by (type,instance,db) ``` #### `tikv_compaction_operations` **说明**: The count of compaction and flush operations - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_event_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance,db) ``` #### `tikv_compaction_reason` **说明**: - - **Labels**: `instance,cf,reason,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_compaction_reason{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,cf,reason,instance) ``` #### `tikv_config_rocksdb` **说明**: TiKV rocksdb config value - **Labels**: `instance,cf,name` - **Quantile**: `-` **PromQL**: ```promql tikv_config_rocksdb{$LABEL_CONDITIONS} ``` #### `tikv_engine_avg_get_duration` **说明**: The average time consumed when executing get operations, the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_get_micro_seconds{$LABEL_CONDITIONS}) by (db,type,instance) ``` #### `tikv_engine_avg_seek_duration` **说明**: The time consumed when executing seek operation, the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_seek_micro_seconds{$LABEL_CONDITIONS}) by (db,type,instance) ``` #### `tikv_engine_blob_bytes_flow` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_blob_flow_bytes{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance,db) ``` #### `tikv_engine_blob_file_count` **说明**: - - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_titandb_num_obsolete_blob_file{$LABEL_CONDITIONS}) by (instance,db) ``` #### `tikv_engine_blob_file_read_duration` **说明**: the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_blob_file_read_micros_seconds{$LABEL_CONDITIONS}) by (type,instance,db) ``` #### `tikv_engine_blob_file_size` **说明**: - - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_titandb_obsolete_blob_file_size{$LABEL_CONDITIONS}) by (instance,db) ``` #### `tikv_engine_blob_file_sync_duration` **说明**: the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_blob_file_sync_micros_seconds{$LABEL_CONDITIONS}) by (instance,type,db) ``` #### `tikv_engine_blob_file_sync_operations` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_blob_file_synced{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_engine_blob_file_write_duration` **说明**: the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_blob_file_write_micros_seconds{$LABEL_CONDITIONS}) by (type,instance,db) ``` #### `tikv_engine_blob_gc_bytes_flow` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_blob_gc_flow_bytes{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance,db) ``` #### `tikv_engine_blob_gc_duration` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_blob_gc_micros_seconds{$LABEL_CONDITIONS}) by (db,instance,type) ``` #### `tikv_engine_blob_gc_file` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_blob_gc_file_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance,db) ``` #### `tikv_engine_blob_gc_keys_flow` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_blob_gc_flow_bytes{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance,db) ``` #### `tikv_engine_blob_get_duration` **说明**: the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_blob_get_micros_seconds{$LABEL_CONDITIONS}) by (type,db,instance) ``` #### `tikv_engine_blob_key_avg_size` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_blob_key_size{$LABEL_CONDITIONS}) by (db,instance,type) ``` #### `tikv_engine_blob_key_max_size` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_blob_key_size{$LABEL_CONDITIONS}) by (db,instance,type) ``` #### `tikv_engine_blob_seek_duration` **说明**: the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_blob_seek_micros_seconds{$LABEL_CONDITIONS}) by (db,type,instance) ``` #### `tikv_engine_blob_seek_operations` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_blob_locate{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,type,instance) ``` #### `tikv_engine_blob_value_avg_size` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_blob_value_size{$LABEL_CONDITIONS}) by (db,instance,type) ``` #### `tikv_engine_blob_value_max_size` **说明**: - - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_blob_value_size{$LABEL_CONDITIONS}) by (db,instance,type) ``` #### `tikv_engine_compaction_flow_bytes` **说明**: The flow rate of compaction operations per type - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_compaction_flow_bytes{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,type,instance) ``` #### `tikv_engine_get_block_cache_operations` **说明**: The count of get memtable operations - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_cache_efficiency{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,type,instance) ``` #### `tikv_engine_get_cpu_cache_operations` **说明**: The count of get l0,l1,l2 operations - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_get_served{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,type,instance) ``` #### `tikv_engine_get_memtable_operations` **说明**: The count of get memtable operations - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_memtable_efficiency{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,type,instance) ``` #### `tikv_engine_live_blob_size` **说明**: - - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_titandb_live_blob_size{$LABEL_CONDITIONS}) by (instance,db) ``` #### `tikv_engine_max_get_duration` **说明**: The max time consumed when executing get operations, the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_get_micro_seconds{$LABEL_CONDITIONS}) by (db,type,instance) ``` #### `tikv_engine_max_seek_duration` **说明**: The time consumed when executing seek operation, the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_seek_micro_seconds{$LABEL_CONDITIONS}) by (db,type,instance) ``` #### `tikv_engine_seek_operations` **说明**: The count of seek operations - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_locate{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,type,instance) ``` #### `tikv_engine_size` **说明**: The storage size per TiKV instance - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(tikv_engine_size_bytes{$LABEL_CONDITIONS}) by (instance, type, db) ``` #### `tikv_engine_wal_sync_operations` **说明**: The count of WAL sync operations - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_wal_file_synced{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,instance) ``` #### `tikv_engine_write_max_duration` **说明**: The time consumed when executing write operation, the unit is microsecond - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_write_micro_seconds{$LABEL_CONDITIONS}) by (db,type,instance) ``` #### `tikv_engine_write_operations` **说明**: The count of write operations - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_write_served{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (db,type,instance) ``` #### `tikv_engine_write_stall` **说明**: Indicates occurrences of Write Stall events that make the TiKV instance unavailable temporarily - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_write_stall{type="write_stall_percentile99"}) by (instance, db) ``` #### `tikv_ingest_sst_avg_duration` **说明**: The average time consumed when ingesting SST files - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_snapshot_ingest_sst_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_snapshot_ingest_sst_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_ingest_sst_duration` **说明**: The quantile of time consumed when ingesting SST files - **Labels**: `instance,db` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_snapshot_ingest_sst_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance,db)) ``` #### `tikv_ingest_sst_total_count` **说明**: The total count of ingesting SST files - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_snapshot_ingest_sst_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,db) ``` #### `tikv_ingest_sst_total_time` **说明**: The total time of time consumed when ingesting SST files - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_snapshot_ingest_sst_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,db) ``` #### `tikv_sst_read_duration` **说明**: The time consumed when reading SST files - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql avg(tikv_engine_sst_read_micros{$LABEL_CONDITIONS}) by (type,instance,db) ``` #### `tikv_sst_read_max_duration` **说明**: The max time consumed when reading SST files - **Labels**: `instance,type,db` - **Quantile**: `-` **PromQL**: ```promql max(tikv_engine_sst_read_micros{$LABEL_CONDITIONS}) by (type,instance,db) ``` ### 快照 #### `tikv_handle_snapshot_duration` **说明**: The quantile of time consumed when handling snapshots - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_raftstore_snapshot_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance,type)) ``` #### `tikv_handle_snapshot_total_count` **说明**: The total count of tikv handling snapshots - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_snapshot_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_handle_snapshot_total_time` **说明**: The total time of time consumed when handling snapshots - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_raftstore_snapshot_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_number_of_snapshots` **说明**: The number of snapshot of each TiKV instance - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql tikv_engine_num_snapshots{$LABEL_CONDITIONS} ``` #### `tikv_oldest_snapshots_duration` **说明**: The time that the oldest unreleased snapshot survivals - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql tikv_engine_oldest_snapshot_duration{$LABEL_CONDITIONS} ``` #### `tikv_send_snapshot_duration` **说明**: The quantile of time consumed when sending snapshots - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_server_send_snapshot_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_send_snapshot_total_count` **说明**: The total count of sending snapshots - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_send_snapshot_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_send_snapshot_total_time` **说明**: The total time of time consumed when sending snapshots - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_send_snapshot_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_snapshot_kv_count` **说明**: The quantile of number of KV within a snapshot - **Labels**: `instance` - **Quantile**: `0.9999` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_snapshot_kv_count_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_snapshot_kv_count_total_count` **说明**: The total count of number of KV within a snapshot - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_snapshot_kv_count_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_snapshot_kv_total_count` **说明**: The total number of KV within a snapshot - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_snapshot_kv_count_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_snapshot_size_total_count` **说明**: The total count of snapshot size - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_snapshot_size_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_snapshot_state_count` **说明**: The number of snapshots in different states - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(tikv_raftstore_snapshot_traffic_total{$LABEL_CONDITIONS}) by (type,instance) ``` #### `tikv_snapshot_state_total_count` **说明**: The total number of snapshots in different states - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(delta(tikv_raftstore_snapshot_traffic_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_snapshot_total_size` **说明**: The total size of snapshot size - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_snapshot_size_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` ### 服务器 #### `tikv_server_report_failures` **说明**: The total number of reported failure messages - **Labels**: `instance,store_id,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_server_report_failure_msg_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance,store_id) ``` #### `tikv_server_report_failures_total_count` **说明**: The total number of reported failure messages - **Labels**: `instance,store_id,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_server_report_failure_msg_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance,store_id) ``` ### 线程池 #### `tikv_futurepool_handled_tasks` **说明**: The number of tasks handled by future_pool - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_futurepool_handled_task_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (name,instance) ``` #### `tikv_futurepool_handled_tasks_total_num` **说明**: Total number of tasks handled by future_pool - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_futurepool_handled_task_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (name,instance) ``` #### `tikv_futurepool_pending_tasks` **说明**: Current pending and running tasks of future_pool - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_futurepool_pending_task_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (name,instance) ``` #### `tikv_futurepool_pending_tasks_total_num` **说明**: Total pending and running tasks of future_pool - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_futurepool_pending_task_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (name,instance) ``` #### `tikv_thread_nonvoluntary_context_switches` **说明**: - - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_thread_nonvoluntary_context_switches{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, name) ``` #### `tikv_thread_voluntary_context_switches` **说明**: - - **Labels**: `instance,name` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_thread_voluntary_context_switches{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, name) ``` #### `tikv_threads_io` **说明**: - - **Labels**: `instance,io,name` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_threads_io_bytes_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (name,io,instance) ``` #### `tikv_threads_state` **说明**: - - **Labels**: `instance,state` - **Quantile**: `-` **PromQL**: ```promql sum(tikv_threads_state{$LABEL_CONDITIONS}) by (instance,state) ``` ### 调度器 #### `tikv_scheduler_command_avg_duration` **说明**: The average time consumed when executing command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_scheduler_command_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_scheduler_command_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_scheduler_command_total_time` **说明**: The total time of time consumed when executing command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_command_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_scheduler_is_busy` **说明**: Indicates occurrences of Scheduler Busy events that make the TiKV instance unavailable temporarily - **Labels**: `instance,db,type,stage` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_scheduler_too_busy_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,db,type,stage) ``` #### `tikv_scheduler_is_busy_total_count` **说明**: The total count of Scheduler Busy events that make the TiKV instance unavailable temporarily - **Labels**: `instance,db,type,stage` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_too_busy_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,db,type,stage) ``` #### `tikv_scheduler_keys_read` **说明**: The quantile count of keys read by command - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_scheduler_kv_command_key_read_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance,type)) ``` #### `tikv_scheduler_keys_read_avg` **说明**: The average count of keys read by command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_scheduler_kv_command_key_read_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_scheduler_kv_command_key_read_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_scheduler_keys_read_total_count` **说明**: The total count of keys read by a command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_kv_command_key_read_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_scheduler_keys_total_read` **说明**: The total count of keys read by command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_kv_command_key_read_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_scheduler_keys_total_written` **说明**: The total count of keys written by a command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_kv_command_key_write_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_scheduler_keys_written` **说明**: The quantile of count of keys written by a command - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_scheduler_kv_command_key_write_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance,type)) ``` #### `tikv_scheduler_keys_written_avg` **说明**: The average count of keys written by a command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_scheduler_kv_command_key_write_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_scheduler_kv_command_key_write_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_scheduler_keys_written_total_count` **说明**: The total count of keys written by a command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_kv_command_key_write_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_scheduler_latch_wait_avg_duration` **说明**: The average time which is caused by latch wait in command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_scheduler_latch_wait_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_scheduler_latch_wait_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_scheduler_latch_wait_total_count` **说明**: The total count which is caused by latch wait in command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_latch_wait_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_scheduler_latch_wait_total_time` **说明**: The total time which is caused by latch wait in command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_latch_wait_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_scheduler_pending_commands` **说明**: The count of pending commands per TiKV instance - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(tikv_scheduler_contex_total{$LABEL_CONDITIONS}) by (instance) ``` #### `tikv_scheduler_priority_commands` **说明**: The count of different priority commands - **Labels**: `instance,priority` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_scheduler_commands_pri_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (priority,instance) ``` #### `tikv_scheduler_processing_read_duration` **说明**: The quantile time of scheduler processing read in command - **Labels**: `instance,type` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_scheduler_processing_read_duration_seconds_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance,type)) ``` #### `tikv_scheduler_processing_read_total_count` **说明**: The total count of scheduler processing read in command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_processing_read_duration_seconds_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_scheduler_processing_read_total_time` **说明**: The total time of scheduler processing read in command - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_processing_read_duration_seconds_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance,type) ``` #### `tikv_scheduler_scan_details` **说明**: The keys scan details of each CF when executing command - **Labels**: `instance,tag,req,cf` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_scheduler_kv_scan_details{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (tag,instance,req,cf) ``` #### `tikv_scheduler_scan_details_total_num` **说明**: - - **Labels**: `instance,tag,req,cf` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_kv_scan_details{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (tag,instance,req,cf) ``` #### `tikv_scheduler_stage` **说明**: The number of scheduler state on each TiKV instance - **Labels**: `instance,stage,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_scheduler_stage_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, stage,type) ``` #### `tikv_scheduler_stage_total_num` **说明**: The total number of scheduler state on each TiKV instance - **Labels**: `instance,stage,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_scheduler_stage_total{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance, stage,type) ``` #### `tikv_scheduler_writing_bytes` **说明**: The total writing bytes of commands on each stage - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(tikv_scheduler_writing_bytes{$LABEL_CONDITIONS}) by (instance) ``` ### 锁管理 #### `tikv_block_all_cache_hit` **说明**: The hit rate of all block cache - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_cache_efficiency{type="block_cache_hit"}[$RANGE_DURATION])) by (db,instance) / (sum(rate(tikv_engine_cache_efficiency{type="block_cache_hit"}[$RANGE_DURATION])) by (db,instance) + sum(rate(tikv_engine_cache_efficiency{type="block_cache_miss"}[$RANGE_DURATION])) by (db,instance)) ``` #### `tikv_block_bloom_prefix_cache_hit` **说明**: The hit rate of data block cache - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_bloom_efficiency{type="bloom_prefix_useful"}[$RANGE_DURATION])) by (db,instance) / sum(rate(tikv_engine_bloom_efficiency{type="bloom_prefix_checked"}[$RANGE_DURATION])) by (db,instance) ``` #### `tikv_block_cache_size` **说明**: The block cache size. Broken down by column family if shared block cache is disabled. - **Labels**: `instance,cf,db` - **Quantile**: `-` **PromQL**: ```promql topk(20, avg(tikv_engine_block_cache_size_bytes{$LABEL_CONDITIONS}) by(cf, instance, db)) ``` #### `tikv_block_data_cache_hit` **说明**: The hit rate of data block cache - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_cache_efficiency{type="block_cache_data_hit"}[$RANGE_DURATION])) by (db,instance) / (sum(rate(tikv_engine_cache_efficiency{type="block_cache_data_hit"}[$RANGE_DURATION])) by (db,instance) + sum(rate(tikv_engine_cache_efficiency{type="block_cache_data_miss"}[$RANGE_DURATION])) by (db,instance)) ``` #### `tikv_block_filter_cache_hit` **说明**: The hit rate of data block cache - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_cache_efficiency{type="block_cache_filter_hit"}[$RANGE_DURATION])) by (db,instance) / (sum(rate(tikv_engine_cache_efficiency{type="block_cache_filter_hit"}[$RANGE_DURATION])) by (db,instance) + sum(rate(tikv_engine_cache_efficiency{type="block_cache_filter_miss"}[$RANGE_DURATION])) by (db,instance)) ``` #### `tikv_block_index_cache_hit` **说明**: The hit rate of data block cache - **Labels**: `instance,db` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_engine_cache_efficiency{type="block_cache_index_hit"}[$RANGE_DURATION])) by (db,instance) / (sum(rate(tikv_engine_cache_efficiency{type="block_cache_index_hit"}[$RANGE_DURATION])) by (db,instance) + sum(rate(tikv_engine_cache_efficiency{type="block_cache_index_miss"}[$RANGE_DURATION])) by (db,instance)) ``` #### `tikv_lock_manager_deadlock_detect_avg_duration` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_lock_manager_detect_duration_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_lock_manager_detect_duration_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_lock_manager_deadlock_detect_duration` **说明**: - - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_lock_manager_detect_duration_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_lock_manager_deadlock_detect_total_count` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_lock_manager_detect_duration_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_lock_manager_deadlock_detect_total_time` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_lock_manager_detect_duration_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_lock_manager_deadlock_detector_leader` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(max_over_time(tikv_lock_manager_detector_leader_heartbeat{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_lock_manager_detect_error` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_lock_manager_error_counter{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_lock_manager_detect_error_total_count` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_lock_manager_error_counter{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_lock_manager_handled_tasks` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_lock_manager_task_counter{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_lock_manager_wait_table` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(max_over_time(tikv_lock_manager_wait_table_status{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (type,instance) ``` #### `tikv_lock_manager_waiter_lifetime_avg_duration` **说明**: - - **Labels**: `instance,type` - **Quantile**: `-` **PromQL**: ```promql sum(rate(tikv_lock_manager_waiter_lifetime_duration_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) / sum(rate(tikv_lock_manager_waiter_lifetime_duration_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) ``` #### `tikv_lock_manager_waiter_lifetime_duration` **说明**: - - **Labels**: `instance` - **Quantile**: `0.99` **PromQL**: ```promql histogram_quantile($QUANTILE, sum(rate(tikv_lock_manager_waiter_lifetime_duration_bucket{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (le,instance)) ``` #### `tikv_lock_manager_waiter_lifetime_total_count` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_lock_manager_waiter_lifetime_duration_count{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ``` #### `tikv_lock_manager_waiter_lifetime_total_time` **说明**: - - **Labels**: `instance` - **Quantile**: `-` **PromQL**: ```promql sum(increase(tikv_lock_manager_waiter_lifetime_duration_sum{$LABEL_CONDITIONS}[$RANGE_DURATION])) by (instance) ```