# Inspection Rules ## Rule Conventions - Default incident window: `30m`. - Default trend window: `24h`. - Default persistence for warnings: `>= 5m`. - Treat isolated one-sample spikes as weak evidence unless the metric is a hard error counter. - Emit per-instance findings first, then aggregate to cluster-level verdict. - Thresholds marked as empirical are health-check heuristics, not official TiFlash alerting thresholds. Use these calculation conventions unless a rule overrides them: - `sustained >= Nm`: at least `N` one-minute samples, or at least 80% of samples in an `N` minute interval, satisfy the condition. - `grows for >= 5m`: the last sample is at least 20% higher than the first usable sample in the window and the fitted trend is positive. For tiny values, require an absolute increase of at least 1 MiB for byte gauges. - `near zero`: max value is `<= 0.01` per second for counters, or all usable samples are zero for gauges. - For histogram duration rules, if the bucket metric family exists after `query_series()` but `histogram_quantile()` returns no usable samples because all bucket rates are zero in the inspected window, mark the rule `Healthy` as `no observed events`, not `Unknown`. - Do not report a generic `rule cannot be evaluated` effect. Classify unknown rule effects as one of: - `metric missing`: the metric family has no series in the inspected window. - `selector unresolved`: TiFlash instance or cluster labels could not be narrowed safely. - `metric samples`: series exist, but the rule query returned no usable samples after selector discovery. - `query failed`: the data source returned an error or timed out. - `series discovery failed`: label discovery failed before the rule could be evaluated. - `does not drain`: current value remains above 80% of the window maximum after request QPS drops by at least 30%. - `elevated against baseline`: incident-window p99 is greater than `max(trend_p95 * 2, trend_p95 + 1s)` for duration metrics, or incident-window max is greater than `max(trend_p95 * 2, trend_p95 + 1)` for count gauges. - `baseline`: use the same metric and labels over the 24h trend window. Compare incident p99 with trend-window p95. If the 24h baseline is unavailable, mark the baseline-dependent check `Unknown` instead of inventing a threshold. For every rule, evaluate and report the same contract: | Field | Meaning | |---|---| | Rule | Stable rule id and title, such as `R1 CPU usage ratio is high` | | Query | PromQL or helper used, with concrete selectors after label discovery | | Aggregation | How series are grouped, usually by `instance`, `type`, `pod`, or `reason` | | Condition | Numeric threshold, baseline comparison, trend, or correlation condition | | Duration | Required persistence window, or `instant` for hard counters | | Severity | `Healthy`, `Warning`, `Critical`, or `Unknown` | | Dependencies | Other signals needed before escalating or attributing bottleneck | For every evaluated rule, extract these result fields when available: | Field | Meaning | |---|---| | `status` | Rule verdict | | `max` | Maximum observed value in the incident window | | `avg` | Average observed value in the incident window | | `p99` | P99 or histogram quantile when applicable | | `duration_above_threshold` | Total sustained time satisfying the rule condition | | `affected_labels` | Instances, pods, volumes, types, reasons, or store ids involved | | `sample_count` | Number of usable non-NaN samples | | `query` | Final executed query or helper name | | `evidence_type` | `hard signal`, `empirical heuristic`, `baseline`, `trend`, or `correlation` | ## Data Coverage ### D1 Unknown: minimal core TiFlash metrics missing Minimal core metric families: - `tiflash_coprocessor_request_duration_seconds_bucket` - `tiflash_coprocessor_request_count` - `tiflash_proxy_process_cpu_seconds_total` - `tiflash_system_current_metric_LogicalCPUCores` - `tiflash_proxy_process_resident_memory_bytes` - `tiflash_system_current_metric_MemoryCapacity` Workload-specific metric families: - `tiflash_coprocessor_request_error` - `tiflash_mpp_task_manager` - `tiflash_mpp_task_monitor` - `tiflash_exchange_queueing_data_bytes` Overall status is `Unknown` when minimal core metrics are absent after label discovery. Missing workload-specific metrics make the affected workload domain `Unknown`, but do not force the whole report to `Unknown` when the inspected window has no matching workload. Domain-specific coverage: | Domain | Metric families | |---|---| | Storage IO | `tiflash_system_seconds_bucket`, `tiflash_storage_write_stall_duration_seconds_bucket`, `tiflash_system_current_metric_IOLimiterPending*` | | Network IO | `node_network_*`, `tiflash_proxy_tikv_server_report_failure_msg_total`, `tiflash_proxy_thread_cpu_seconds_total` | | Raft | `tiflash_raft_wait_index_duration_seconds_bucket`, `tiflash_system_profile_event_RaftWaitIndexTimeout`, `tiflash_raft_throughput_bytes` | | Pipeline Model | `tiflash_pipeline_scheduler`, `tiflash_pipeline_task_change_to_status`, `tiflash_pipeline_task_duration_seconds_bucket` | If a domain-specific metric family is absent after label discovery, mark that domain `Unknown`. Do not make the whole report `Unknown` unless minimal core metrics are missing or the unknown domain is necessary to judge an active Critical/Warning candidate. ## Query Health ### Q1 Warning: TiFlash request error QPS is nonzero ```promql sum(rate(tiflash_coprocessor_request_error{}[1m])) by (reason) ``` Warning when any reason is `> 0` for `>= 5m`, but classify the reason before using Q1 for impact or bottleneck attribution. Common retry or data-movement reasons such as `epoch_not_match`, `region_not_found`, and `meet_lock` are weak warning signals by themselves. They should be reported, but they do not prove TiFlash service impact unless Q2, Q3, M3, N2, A2, or another user-facing signal is abnormal in the same window. When only these weak reasons are present, keep Q1 out of Top Risks and do not raise the overall cluster status above `Healthy`; include the series in the evidence section as supporting context. Critical when `reason` matches an internal/server failure class, the error QPS is `> 0` for `>= 5m`, and Q2 is also `Warning` or `Critical` in the same time window. ### Q2 Warning: request duration is elevated ```promql histogram_quantile(0.99, sum(rate(tiflash_coprocessor_request_duration_seconds_bucket{}[1m])) by (le, type)) ``` Warning when p99 for important request types is elevated for `>= 5m`. Use this order: - If the 24h baseline is available, warn when incident p99 is greater than `max(trend_p95 * 2, trend_p95 + 1s)`. - If the baseline is unavailable, warn when `mpp_establish_conn` p99 is `> 3s`, `run_mpp_task` p99 is `> 30s`, or `cop` p99 is `> 10s`. These are empirical fallback thresholds. Use p999 for tail confirmation. ### Q3 Warning: MPP establish connection is slow ```promql histogram_quantile(0.99, sum(rate(tiflash_coprocessor_request_duration_seconds_bucket{, type="mpp_establish_conn"}[1m])) by (le, type)) ``` Warning when p99 `> 3s` for `>= 5m`. This is an empirical threshold. If `tiflash_coprocessor_request_duration_seconds_bucket` exists but the `type="mpp_establish_conn"` bucket rates are all zero or the quantile query has no usable samples, treat Q3 as `Healthy` with `no observed mpp_establish_conn events`, not `Unknown`. If combined with exchange queue growth or network failures, raise network/MPP transport suspicion. ### Q4 Warning: active handling requests do not drain ```promql sum(tiflash_coprocessor_handling_request_count{}) by (type) ``` Warning when `run_mpp_task`, `mpp_establish_conn`, or `cop` handling count is elevated against the 24h baseline for `>= 5m`. If no 24h baseline is available, warn only when the handling count is `> 0`, request QPS drops by at least 30%, and the handling count does not drain for `>= 5m`. ## MPP Task Health ### M1 Warning: longest live MPP task is suspicious ```promql tiflash_mpp_task_monitor{} ``` Warning when the longest live MPP task is `> 10m` and persists. Ignore `tiflash_mpp_task_monitor` samples whose value is `> 2d` (`172800s`); these are treated as stale long-lived task artifacts and must not trigger M1. When all samples in the window are older than `2d`, report M1 as `Healthy` with the ignored stale sample count. Critical only when combined with at least two of: - `tiflash_exchange_queueing_data_bytes` grows. - `tiflash_pipeline_task_change_to_status` is near zero. - `run_mpp_task` handling count does not drain. - `mpp_establish_conn` or `run_mpp_task` duration is elevated. - Storage, network, or Raft hard error signals fire. Do not call a long-running MPP task stuck from lifetime alone. ### M2 Warning: MPP query count stays high ```promql max(tiflash_mpp_task_manager{}) by (instance, type) ``` Warning when active query count is elevated against the 24h baseline for `>= 5m` while request QPS is flat or falling. If no 24h baseline is available, warn only when active query count is `> 0`, request QPS drops by at least 30%, and active query count remains above 80% of the incident-window maximum for `>= 5m`. ### M3 Warning: exchange queueing data grows ```promql tiflash_exchange_queueing_data_bytes{} ``` Warning when send/receive queue bytes are `> 128 MiB` for `>= 5m`. Critical when queue bytes are `> 128 MiB` while `tiflash_exchange_data_bytes` or `tiflash_coprocessor_response_bytes` is near zero for `>= 5m`. ## Resource Health ### R1 Warning: CPU usage ratio is high CPU usage ratio. Do not add `job` selectors to TiFlash-prefixed metrics by default; the `tiflash_proxy_process_cpu_seconds_total` metric family is already TiFlash-scoped. Discover `process_selector` and `cpu_capacity_selector` independently from their own metric families. ```promql sum by (instance) ( rate(tiflash_proxy_process_cpu_seconds_total{}[1m]) ) / on(instance) sum by (instance) ( tiflash_system_current_metric_LogicalCPUCores{} ) ``` Warning when max per-instance ratio `> 85%` for `>= 3m`. If Pipeline `cpu_queue` is also high, this supports compute bottleneck. ### R2 Warning: memory usage ratio is high Memory usage ratio. Do not add `job` selectors to TiFlash-prefixed metrics by default; the `tiflash_proxy_process_resident_memory_bytes` metric family is already TiFlash-scoped. Discover `memory_selector` and `memory_capacity_selector` independently from their own metric families. ```promql max by (instance) ( tiflash_proxy_process_resident_memory_bytes{} ) / on(instance) sum by (instance) ( tiflash_system_current_metric_MemoryCapacity{} ) ``` Warning when max per-instance ratio `> 80%` for `>= 3m`. Critical when memory pressure coincides with process restarts, OOM evidence, or failed requests. ### R3 Warning: store capacity risk ```promql sum(tiflash_system_current_metric_StoreSizeUsed{}) by (instance) / on(instance) sum(tiflash_system_current_metric_StoreSizeCapacity{}) by (instance) ``` If `StoreSizeUsed` is absent but `StoreSizeAvailable` and `StoreSizeCapacity` are present, derive used ratio as `1 - available / capacity` and record the derived query in the evidence. Warning when used ratio approaches the configured low-space threshold, or when used ratio is `> 85%` and the recent growth trend suggests the threshold will be reached within five days. If neither configured threshold nor growth trend is available, warn at `> 90%` used ratio for `>= 5m`. ## Storage IO ### S2 Warning: fsync duration is elevated ```promql histogram_quantile(1.00, sum(round(1000000000 * rate(tiflash_system_seconds_bucket{, type=~"fsync"}[1m]))) by (le, instance) / 1000000000) ``` Warning when max fsync duration is elevated against the 24h baseline for `>= 5m`, and Q2, S3, or P2 is also abnormal in the same time window. If no 24h baseline is available, warn when max fsync duration is `> 1s` for `>= 5m` and Q2, S3, or P2 is also abnormal. This is an empirical fallback threshold. ### S3 Warning: write stall duration is high ```promql histogram_quantile(1.00, sum(round(1000000000 * rate(tiflash_storage_write_stall_duration_seconds_bucket{}[1m]))) by (le, type, instance) / 1000000000) ``` Warning when max write stall duration `> 10s` for `>= 5m`. This is an empirical threshold. Critical when combined with S2 or Q2 in the same time window. ### S4 Warning: IO limiter pending requests persist Check all pending read/write gauges: - `tiflash_system_current_metric_IOLimiterPendingBgWriteReq` - `tiflash_system_current_metric_IOLimiterPendingFgWriteReq` - `tiflash_system_current_metric_IOLimiterPendingBgReadReq` - `tiflash_system_current_metric_IOLimiterPendingFgReadReq` Warning when foreground pending count stays `> 0` for `>= 5m`. Foreground pending is stronger evidence than background pending. ## Network IO ### N1 Warning: node network drops or errors are nonzero ```promql rate(node_network_receive_drop_total{instance=~""}[1m]) rate(node_network_transmit_drop_total{instance=~""}[1m]) rate(node_network_receive_errs_total{instance=~""}[1m]) rate(node_network_transmit_errs_total{instance=~""}[1m]) ``` Warning when any drop/error rate is `> 0` for `>= 5m`. Critical when network errors coincide with MPP queueing, server report failures, or Raft timeouts. ### N2 Warning: TiFlash Proxy reports server failures ```promql sum(rate(tiflash_proxy_tikv_server_report_failure_msg_total{}[1m])) by (type, instance, store_id) ``` Warning when any type is `> 0` for `>= 3m`. Critical when `type="unreachable"` is `> 0` for `>= 3m` and coincides with MPP/Raft impact. Do not conclude network failure from `unreachable` alone; check CPU, gRPC thread saturation, memory pressure, and storage stalls as false positives. ### N3 Warning: MPP exchange queue indicates transport or downstream wait ```promql tiflash_exchange_queueing_data_bytes{} ``` Warning when queueing bytes grow for `>= 5m` and Pipeline `await` p99 is `> 1s` for `>= 5m`. Classify as `Network/Exchange` only when storage and CPU signals do not explain the queueing. ## Raft And Replica Health ### A1 Warning: Raft wait index duration is high ```promql histogram_quantile(0.99, sum(rate(tiflash_raft_wait_index_duration_seconds_bucket{}[1m])) by (le)) ``` Warning when p99 `> 30s` for `>= 5m`. This is an empirical threshold. ### A2 Critical: Raft wait index timeout occurs ```promql sum(increase(tiflash_system_profile_event_RaftWaitIndexTimeout{}[1m])) by (instance) ``` Critical when timeout count increases by `> 0` in any one-minute interval. If combined with server report failures or node network errors, classify as network/Raft transport risk. If combined with storage stalls, classify as mixed. ## Pipeline Model ### P1 Warning: CPU queue duration is high ```promql histogram_quantile(0.99, sum(rate(tiflash_pipeline_task_duration_seconds_bucket{, type="cpu_queue"}[1m])) by (le, type)) ``` Warning when `cpu_queue` p99 is `> 1s` for `>= 5m` and CPU pool saturation is also present. This is an empirical threshold. This supports compute bottleneck. ### P2 Warning: IO queue duration is high ```promql histogram_quantile(0.99, sum(rate(tiflash_pipeline_task_duration_seconds_bucket{, type="io_queue"}[1m])) by (le, type)) ``` Warning when `io_queue` p99 is `> 1s` for `>= 5m` with IO limiter pending, fsync, write stall, storage subtask duration, or S3 request duration. This is an empirical threshold. This supports storage bottleneck. ### P3 Warning: await duration is high ```promql histogram_quantile(0.99, sum(rate(tiflash_pipeline_task_duration_seconds_bucket{, type="await"}[1m])) by (le, type)) ``` Warning when `await` p99 is `> 1s` for `>= 5m` with exchange queueing, MPP tunnel bytes, or downstream wait evidence. This is an empirical threshold. This supports exchange/downstream wait, not pure compute or storage by itself. ### P4 Critical: live MPP task has no pipeline progress ```promql sum(rate(tiflash_pipeline_task_change_to_status{}[1m])) by (type) ``` Critical only when: - The longest live MPP task grows for `>= 5m`. - Active MPP tasks remain. - Task status change OPS is near zero. - Exchange or response throughput is near zero, or resource/transport evidence shows no progress. Use Warning when exactly three of these conditions are true. Keep the rule `Healthy` when fewer than three conditions are true. ### P5 Warning: task pool saturation CPU pool: ```promql sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{, name=~"cpu_pool"}[1m])) ``` IO pool: ```promql sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{, name=~"io_pool"}[1m])) ``` Compute the pool saturation ratio as `pool_cpu_rate / pool_thread_count`. Warning when the ratio is `> 0.85` for `>= 3m` and the matching queue duration is `> 1s` for `>= 5m`. Match CPU pool saturation with `cpu_queue`; match IO pool saturation with `io_queue`. If the thread count query is missing, mark pool saturation `Unknown` instead of inferring saturation from CPU rate alone. ## Bottleneck Attribution Rules ### B1 Compute-like bottleneck Classify as `Compute` with high confidence when: - R1 is `Warning` or worse. - CPU pool is saturated. - P1 or CPU-side P5 is `Warning` or worse. - Storage IO and network/exchange hard signals are normal. ### B2 Storage-like bottleneck Classify as `Storage` with high confidence when: - IO limiter pending, fsync, write stall, or storage subtask duration is abnormal. - P2 or IO-side P5 is `Warning` or worse. - CPU is not the dominant saturated resource. ### B3 Network/exchange-like bottleneck Classify as `Network/Exchange` with high confidence when: - Node network drops/errors or TiFlash Proxy server report failures exist. - `mpp_establish_conn` is slow or exchange queueing grows. - P3 is `Warning` or worse. - CPU and storage indicators do not explain the delay. ### B4 Mixed bottleneck Classify as `Mixed` when two or more layers have sustained abnormal signals and the timing does not isolate a dominant cause. ### B5 Unknown bottleneck Classify as `Unknown` when required metrics are missing or the signal is only a single weak indicator.