# PromQL Reference

This file records the PromQL expressions used by automatic TiFlash health
inspection. Execute these expressions through Clinic Data Proxy, O11Y metrics,
or a user-provided Prometheus endpoint. Do not fetch Grafana JSON at runtime.

## Selector Template

Default TiFlash summary selector, when these labels exist on the target metric:

```promql
k8s_cluster="<k8s_cluster>", tidb_cluster="<tidb_cluster>", instance=~"<instance_regex>"
```

Build selectors from `query_series` results for the same metric family being
queried. Cloud TiFlash metrics may use `tidb_cluster`, `tidb_cluster_id`,
`cluster_id`, or only `instance`. Keep only labels that exist on the returned
series and are needed to isolate the target cluster or TiFlash instances.

Common alternatives:

```promql
tidb_cluster_id="<cluster_id>", instance=~"<instance_regex>"
cluster_id="tidb<cluster_id>", instance=~"<instance_regex>"
instance=~"<instance_regex>"
```

Do not add `job` selectors to TiFlash-prefixed metrics by default. Metric
families such as `tiflash_*` and `tiflash_proxy_*` are already TiFlash-scoped. If
a query returns no data, remove only unsupported selector labels after
confirming with `query_series`; do not change the metric family or aggregation.

For ratio queries that combine multiple metric families, build a separate
selector for each metric family and join only on stable labels such as
`instance`. A selector discovered from one metric family can contain labels that
do not exist on the other side of the ratio.

Before executing through Clinic Data Proxy or direct Prometheus, replace any
Grafana variables with concrete values:

| Grafana variable | Replacement |
|---|---|
| `$__rate_interval` | `1m` unless a rule states otherwise |
| `$namespace` | concrete namespace such as `tidb<cluster_id>` |
| `$instance` | concrete instance regex discovered from series |
| `$k8s_cluster` | concrete Kubernetes cluster label |
| `$tidb_cluster` | concrete TiDB cluster label |

## Query Health

### Request QPS

Reference: `Coprocessor / Request QPS`

```promql
sum(rate(tiflash_coprocessor_request_count{<selector>}[1m])) by (type)
```

### Request Duration

Reference: `Coprocessor / Request Duration`

```promql
histogram_quantile(0.99, sum(rate(tiflash_coprocessor_request_duration_seconds_bucket{<selector>}[1m])) by (le, type))
```

Also inspect `0.999` when looking for tail latency.

### Error QPS

Reference: `Coprocessor / Error QPS`

```promql
sum(rate(tiflash_coprocessor_request_error{<selector>}[1m])) by (reason)
```

### Handling Requests

Reference: `Threads / Coprocessor Handling Request`

```promql
sum(tiflash_coprocessor_handling_request_count{<selector>}) by (type)
```

### Response Bytes

Reference: `Coprocessor / Response Bytes`

```promql
sum(rate(tiflash_coprocessor_response_bytes{<selector>}[1m])) by (type)
```

For MPP tunnel response bytes, focus on `type="mpp_tunnel"` and
`type="mpp_tunnel_local"`.

## MPP Task Health

### MPP Query Count

Reference: `Threads / MPP Query count`

```promql
max(tiflash_mpp_task_manager{<selector>}) by (instance, type)
```

### Longest Live MPP Task

Reference: `Threads / Time of the Longest Live MPP Task`

```promql
tiflash_mpp_task_monitor{<selector>}
```

For M1 evaluation, ignore samples whose value is older than two days:

```promql
tiflash_mpp_task_monitor{<selector>} <= 172800
```

### Exchange Queueing Data

Reference: `Threads / Data size in send and receive queue`

```promql
tiflash_exchange_queueing_data_bytes{<selector>}
```

### Exchange Data Bytes

Reference: `Coprocessor / Exchange Data Bytes`

```promql
sum(rate(tiflash_exchange_data_bytes{<selector>}[1m])) by (type)
```

### MPP Establish Connection Duration

Reference: `Coprocessor / Request Duration`, filtered by type.

```promql
histogram_quantile(0.99, sum(rate(tiflash_coprocessor_request_duration_seconds_bucket{<selector>, type="mpp_establish_conn"}[1m])) by (le, type))
```

### Run MPP Task Duration

Reference: `Coprocessor / Request Duration`, filtered by type.

```promql
histogram_quantile(0.99, sum(rate(tiflash_coprocessor_request_duration_seconds_bucket{<selector>, type="run_mpp_task"}[1m])) by (le, type))
```

## Resource Health

### CPU Usage

Reference: `Server / CPU Usage`

```promql
sum by (instance) (
  rate(tiflash_proxy_process_cpu_seconds_total{<selector>}[1m])
)
```

CPU quota:

```promql
sum(tiflash_system_current_metric_LogicalCPUCores{<selector>}) by (instance)
```

CPU usage ratio is `process_cpu_rate / logical_cpu_cores`.

Complete ratio query:

```promql
sum by (instance) (
  rate(tiflash_proxy_process_cpu_seconds_total{<process_selector>}[1m])
)
/
on(instance)
sum by (instance) (
  tiflash_system_current_metric_LogicalCPUCores{<cpu_capacity_selector>}
)
```

### Memory Usage

Reference: `Server / Memory`

```promql
max by (instance) (
  tiflash_proxy_process_resident_memory_bytes{<selector>}
)
```

Memory capacity:

```promql
sum(tiflash_system_current_metric_MemoryCapacity{<selector>}) by (instance)
```

Memory usage ratio is `rss / memory_capacity`.

Complete ratio query:

```promql
max by (instance) (
  tiflash_proxy_process_resident_memory_bytes{<memory_selector>}
)
/
on(instance)
sum by (instance) (
  tiflash_system_current_metric_MemoryCapacity{<memory_capacity_selector>}
)
```

### Store Size

Reference: `Storage / Store size`

```promql
sum(tiflash_system_current_metric_StoreSizeUsed{<selector>}) by (instance)
```

Remote store size:

```promql
sum(tiflash_system_current_metric_StoreSizeUsedRemote{<selector>}) by (instance)
```

### Available Size

Reference: `Storage / Available size`

```promql
sum(tiflash_system_current_metric_StoreSizeAvailable{<selector>}) by (instance)
```

### Capacity Size

Reference: `Storage / Capacity size`

```promql
sum(tiflash_system_current_metric_StoreSizeCapacity{<selector>}) by (instance)
```

## Storage IO

### TiFlash IO Throughput

Reference: `Server / IO Throughput`

```promql
sum by (instance) (irate(tiflash_proxy_threads_io_bytes_total{<io_selector>}[1m]))
```

### Fsync Duration

Reference: `Storage / Fsync OPS/Duration`

```promql
histogram_quantile(1.00, sum(round(1000000000 * rate(tiflash_system_seconds_bucket{<selector>, type=~"fsync"}[1m]))) by (le, instance) / 1000000000)
```

### Write Stall Duration

Reference: `Storage Write Stall / Write Stall Duration`

```promql
histogram_quantile(1.00, sum(round(1000000000 * rate(tiflash_storage_write_stall_duration_seconds_bucket{<selector>}[1m]))) by (le, type, instance) / 1000000000)
```

P99 auxiliary:

```promql
histogram_quantile(0.99, sum(rate(tiflash_storage_write_stall_duration_seconds_bucket{<selector>}[1m])) by (le, type, instance))
```

### IO Limiter Pending

Reference: `Storage / I/O Limiter Current Pending Count`

```promql
avg(tiflash_system_current_metric_IOLimiterPendingBgWriteReq{<selector>}) by (instance)
```

```promql
avg(tiflash_system_current_metric_IOLimiterPendingFgWriteReq{<selector>}) by (instance)
```

```promql
avg(tiflash_system_current_metric_IOLimiterPendingBgReadReq{<selector>}) by (instance)
```

```promql
avg(tiflash_system_current_metric_IOLimiterPendingFgReadReq{<selector>}) by (instance)
```

## Network IO

### Node Network Drops And Errors

```promql
rate(node_network_receive_drop_total{instance=~"<node_instance_regex>"}[1m])
```

```promql
rate(node_network_transmit_drop_total{instance=~"<node_instance_regex>"}[1m])
```

```promql
rate(node_network_receive_errs_total{instance=~"<node_instance_regex>"}[1m])
```

```promql
rate(node_network_transmit_errs_total{instance=~"<node_instance_regex>"}[1m])
```

### Node Network Traffic

```promql
rate(node_network_receive_bytes_total{instance=~"<node_instance_regex>"}[1m])
```

```promql
rate(node_network_transmit_bytes_total{instance=~"<node_instance_regex>"}[1m])
```

### TiFlash Proxy Server Report Failures

Reference: `Server report failures`

```promql
sum(rate(tiflash_proxy_tikv_server_report_failure_msg_total{<selector>}[1m])) by (type, instance, store_id)
```

Focus on `type="unreachable"`, but do not conclude network failure without
checking CPU, gRPC threads, memory, and host stalls.

### gRPC Async Client Threads

Reference: `Threads CPU / GRPC Async Client`

```promql
sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"GRPCComp.*"}[1m]))
```

Thread count:

```promql
count by (instance) (tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"GRPCComp.*"})
```

## Raft And Replica Health

### Raft Wait Index Duration

Reference: `Raft / Raft Wait Index Duration`

```promql
histogram_quantile(0.99, sum(rate(tiflash_raft_wait_index_duration_seconds_bucket{<selector>}[1m])) by (le))
```

### Raft Wait Index Timeout

Reference: `Raft / Raft Wait Index Duration`

```promql
sum(increase(tiflash_system_profile_event_RaftWaitIndexTimeout{<selector>}[1m])) by (instance)
```

### Raft Throughput

Reference: `Raft / Raft throughput`

```promql
sum(rate(tiflash_raft_throughput_bytes{<selector>}[1m])) by (type)
```

## Pipeline Model

### Task Pool Size

Reference: `Pipeline Model / Task Thread Pool Size`

```promql
max(tiflash_pipeline_scheduler{<selector>, type=~".*_task_thread_pool_size"}) by (instance, type)
```

### Task Count

Reference: `Pipeline Model / Task Count`

```promql
max(tiflash_pipeline_scheduler{<selector>, type=~".*_tasks_count"}) by (instance, type)
```

### Task Status Change OPS

Reference: `Pipeline Model / Task Status Change OPS`

```promql
sum(rate(tiflash_pipeline_task_change_to_status{<selector>}[1m])) by (type)
```

### Task Duration

Reference: `Pipeline Model / Task Duration`

```promql
histogram_quantile(0.99, sum(rate(tiflash_pipeline_task_duration_seconds_bucket{<selector>}[1m])) by (le, type))
```

Also compute averages for:

```promql
sum(rate(tiflash_pipeline_task_duration_seconds_sum{<selector>, type="cpu_execute"}[1m])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{<selector>, type="cpu_execute"}[1m]))
```

```promql
sum(rate(tiflash_pipeline_task_duration_seconds_sum{<selector>, type="cpu_queue"}[1m])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{<selector>, type="cpu_queue"}[1m]))
```

```promql
sum(rate(tiflash_pipeline_task_duration_seconds_sum{<selector>, type="io_execute"}[1m])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{<selector>, type="io_execute"}[1m]))
```

```promql
sum(rate(tiflash_pipeline_task_duration_seconds_sum{<selector>, type="io_queue"}[1m])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{<selector>, type="io_queue"}[1m]))
```

```promql
sum(rate(tiflash_pipeline_task_duration_seconds_sum{<selector>, type="await"}[1m])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{<selector>, type="await"}[1m]))
```

### Task Max Execute Time Per Round

Reference: `Pipeline Model / Task Max Execute Time Per Round`

```promql
histogram_quantile(1.00, sum(round(1000000000 * rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket{<selector>}[1m]))) by (le, type) / 1000000000)
```

### CPU Task Thread Pool

Reference: `Pipeline Model / Threads CPU of CPU Task Thread Pool`

```promql
sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"cpu_pool"}[1m]))
```

Thread count:

```promql
count by (instance) (tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"cpu_pool"})
```

Saturation ratio:

```promql
sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"cpu_pool"}[1m]))
/
on(instance)
count by (instance) (tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"cpu_pool"})
```

### IO Task Thread Pool

Reference: `Pipeline Model / Threads CPU of IO Task Thread Pool`

```promql
sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"io_pool"}[1m]))
```

Thread count:

```promql
count by (instance) (tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"io_pool"})
```

Saturation ratio:

```promql
sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"io_pool"}[1m]))
/
on(instance)
count by (instance) (tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"io_pool"})
```

### Wait Reactor

Reference: `Pipeline Model / Threads CPU of Wait Reactor`

```promql
sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{<selector>, name=~"WaitReactor"}[1m]))
```