# Serverless Daily Inspection Rules ## Scope and Method This inspection adapts the Dedicated TiKV daily-inspection structure to a shared Serverless TiKV pool. The required identity is `(control_plane_info, k8s_cluster_info)`. The full Grafana `control_plane_info` value (`//`) routes the parent APIs, while PromQL uses its derived `/` label plus `k8s_cluster_info`, `namespace="tidb-serverless"`, and `component="tikv"`. For node-exporter disk metrics, do **not** apply the namespace or component labels directly. Instead, map each node-exporter `instance` to a TiKV storage pod with `label_replace(kube_pod_info, "instance", "$1", "node", "(.*)")`. The `kube_pod_info` side carries the derived control-plane label, `k8s_cluster_info`, `namespace="tidb-serverless"`, and the strict TiKV storage pod regex `^tikv(?:-[a-z0-9]+)*-[0-9]+$`. Parent Loki routes by HTTP `control_plane_info`; its documented selector is `cluster=""`, `namespace="tidb-serverless"`, and `container="tikv-log"`. `control_plane_info` and `k8s_cluster_info` are internal, undocumented labels. They are treated as mandatory routing and isolation labels, not as user-facing topology metadata. ## Sections Generate these sections in order: 1. Summary 2. Key Findings 3. Error Signals 4. gRPC QPS and Latency 5. CPU and Memory 6. Critical Thread Pool CPU 7. Disk Usage and Region Count 8. Disk Load 9. Compaction 10. ERROR/WARN Logs 11. Follow-up Investigation API/query errors and malformed non-empty Prometheus responses are `Unknown`. Every successful Prometheus response with an empty result is known zero/healthy, including scalar, labeled-vector, range, B3, and thread-group paths. D8 remains structurally unavailable and therefore `Unknown` in its own report row. Structurally unavailable placeholder rules are visible evidence only: they never drive the verdict, the Summary unknown-rule list, or follow-up actions. Only unexpected `Unknown` results make a pool `Needs Review`. For a multi-day report, preserve the same section order and rule semantics, but render every chapter as an item × date table. Each date cell is the result of a separate day-bounded inspection window; do not calculate a range average or combine statuses across dates. The `ERROR/WARN Logs` chapter additionally has one Top 10 `Content × Count` table per date. Those tables are independently grouped, sorted by count descending then content ascending, and capped at ten groups; never merge log groups between dates. ## Metric Source of Truth - Dashboard: `Starter-Essential-TiKV-Details-for-Clinic` - Dashboard UID: `_eCx0ZDvz` - Source inspected: `/Users/pingyu/workspace/cloud-storage-engine/metrics/grafana/tikv_details_serverless.json` - Generated-source cross-check: `tikv_details_serverless.dashboard.py` and `tikv_details.dashboard.py` The Serverless wrapper provides the title, UID, and Serverless PromQL policy. The generated base dashboard supplies the panel purposes and metric families. Every PromQL selector adds the derived two-segment `control_plane_info` label, `k8s_cluster_info`, `namespace="tidb-serverless"`, and `component="tikv"`. The node-exporter side of D4-D7 is the exception: it has only the conservative non-root NVMe matcher `device=~"^nvme[1-9][0-9]*n[1-9][0-9]*$"`; the required pool and TiKV scope is applied to its joined `kube_pod_info` side. T3, D5-D7, and F2 are validated parent-only metrics and therefore are intentionally absent from the external dashboard source. D8 remains unavailable because Serverless parent metrics have no `aws_ebs_*` exceeded-IOPS series. Parent Loki uses the full Grafana `control_plane_info` as an HTTP routing parameter and the documented stream labels `cluster`, `namespace`, and `container`. Status meanings: - `exact`: the Serverless metric preserves the Dedicated rule's unit and purpose - `replacement`: a related Serverless metric preserves the operational intent - `partial`: only part of the original signal is available - `unavailable`: no valid pool-wide alternative; report `Unknown` - `log`: a LogQL rule rather than a Prometheus metric ## Rule Table | Rule ID | Area | Metric or Logic | Threshold | Severity | Status | Source | |---|---|---|---|---|---|---| | E1 | Critical Error | `increase(tikv_critical_error_total[window])`; an empty successful response is zero | `> 0` | Error | exact | Critical error panel | | E2 | Server is busy | `increase(tikv_channel_full_total[window])`; an empty successful response is zero | `> 0` | Warning | exact | Server is busy panel | | E3 | Server is busy | `increase(tikv_scheduler_too_busy_total[window])`; an empty successful response is zero | `> 0` | Warning | exact | Server is busy panel | | E4 | Server is busy | `increase(tikv_coprocessor_request_error{type="full"}[window])`; an empty successful response is zero | `> 0` | Warning | exact | Server is busy panel | | E5 | Server report failures | `increase(tikv_server_report_failure_msg_total[window])` | `> 0` | Warning | exact | Server report failures panel | | G1 | gRPC message rate | pool-wide `sum by(type)(rate(tikv_grpc_msg_duration_seconds_count[1m]))`; report the full-window maximum and average 1-minute rate for read types `coprocessor`, `kv_get`, and `kv_batch_get`, and write types `kv_prewrite` and `kv_commit` | four observation rows: read max, read avg, write max, and write avg, each in messages/s by request type | Observation | exact | gRPC message count panel | | G2 | gRPC message failed | maximum rolling `sum(increase(tikv_grpc_msg_fail_total[30m])) by (type)` across all failure types | `>= 100` in a rolling 30m window | Warning | exact | gRPC message failed panel | | G3 | Write-path gRPC P99 | `histogram_quantile(0.99, rate(tikv_grpc_msg_duration_seconds_bucket[5m]))` for `kv_prewrite` / `kv_commit` | `>= 1000ms` for 3 contiguous 5m samples | Warning | exact | gRPC message duration panel | | G4 | Average gRPC duration | `increase(tikv_grpc_msg_duration_seconds_sum[window]) / increase(..._count[window])` by type | observation | Observation | exact | Average gRPC message duration panel | | G5 | Read-path gRPC P99 | gRPC duration P99 for `coprocessor`, `kv_get`, `kv_batch_get`, and `kv_batch_get_command` | observation | Observation | exact | gRPC message duration panel | | B1 | TiKV CPU | `process_cpu_seconds_total / tikv_server_cpu_cores_quota` by instance | `>= 85%` for 6 contiguous 5m samples | Warning | exact | CPU panel | | B2 | TiKV memory | `process_resident_memory_bytes / tikv_server_memory_quota_bytes` by instance | warning `>= 85%`; Error `>= 90%`, each for 6 contiguous 5m samples | Warning/Error | replacement | Memory panel; quota replaces Dedicated topology memory | | B3 | Memory-leak watch | current-window versus prior-7-day memory average by matching instance, with Region/QPS/CPU growth exclusions | observe at `>= 3` percentage points; warn at `>= 5` percentage points unless the same instance also has Region growth `>= 100` and `>= 10%`, gRPC workload QPS growth `>= 100` and `>= 20%`, or CPU growth `>= 5` percentage points | Warning | replacement | Serverless-scoped memory, Region, gRPC QPS, and CPU queries | | B4 | Restart | `changes(process_start_time_seconds[window])` | `> 0` | Warning | exact | Uptime panel | | T1 | Thread pool CPU | per ordered dashboard group, pool-scoped `sum by (instance) (rate(tikv_thread_cpu_seconds_total{name=~""}[5m]))`; retain per-instance query-range maximum and arithmetic average | observation: largest per-instance `max` and mean per-instance `avg` CPU cores; no threshold | Observation | replacement | Thread CPU panels with Serverless thread-name regexes | | T2 | Unified Read Pool running tasks | `tikv_unified_read_pool_running_tasks` | observation | Observation | exact | Unified Read Pool Running Tasks panel | | T3 | Unified Read Pool time used | `sum by(instance)(increase(multilevel_level_elapsed{name="unified-read-pool",level="total"}[window])) / 1000000` | observation | Observation | exact | Dedicated elapsed-time query shape | | C1 | Store usage | `tikv_store_size_bytes{type="all_used"} / tikv_store_size_bytes{type="capacity"}` by instance | `>= 80%` after a `>= 10` pp rise from the minimum prior value within 30m | Warning | replacement | Store size and Capacity size panels | | C2 | Region / Leader balance | `sum by(instance,type)(tikv_raftstore_region_count{type=~"region|leader"})` | report separate per-instance/type evidence | Observation | exact | Region and Leader panels | | D1 | Raft Engine P99 | maximum window P99 across append, commit, and apply log duration histograms; stage labels are preserved before reduction | observation | Observation | exact | Write Pipeline Duration / Raft IO panels | | D2 | Raft Engine WAL write throttle | `sum(increase(raft_engine_wal_write_throttle_duration_seconds_sum[window]))` | `> 0` seconds | Warning | exact | `components/rfengine/src/writer.rs`: histogram observes only when `throttle_sleep > 0` | | D3 | Pending Raft WAL compaction | window peak of `raft_engine_pending_compaction_wals{container="tikv"}` | `>= 2` | Warning | exact | Raft Engine pending-WAL gauge; count formula is validated in `components/rfengine/src/engine.rs` | | D4 | Disk utilization | maximum window `rate(node_disk_io_time_seconds_total[5m])`, joined to scoped TiKV storage pods | observation only | Observation | exact | IO utilization panel | | D5 | Disk queue | maximum window `rate(node_disk_io_time_weighted_seconds_total[5m])`, joined to scoped TiKV storage pods | observation only; never creates an automated warning | Observation | exact | Validated parent node-exporter metric | | D6 | Disk read latency | maximum window `rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])`, joined to scoped TiKV storage pods | `> 20ms` | Warning | exact | Validated parent node-exporter metric | | D7 | Disk write latency | maximum window `rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])`, joined to scoped TiKV storage pods | `> 20ms` | Warning | exact | Validated parent node-exporter metric | | D8 | Disk IOPS ceiling | `aws_ebs_*` exceeded-IOPS purpose | unavailable; no `aws_ebs_*` series exists in Serverless parent metrics | Unknown | unavailable | No volume-attributed exceeded-IOPS series | | F1 | Flow control throttle duration | P99 from `tikv_scheduler_throttle_duration_seconds_bucket` | observation | Observation | exact | Flow Control Throttle duration panel | | F2 | Flow control discard ratio | `max by(instance)(max_over_time((tikv_scheduler_discard_ratio)[window:5m]))`; an empty successful response is zero | peak ratio `> 0` | Warning | exact | Dedicated rule; validated parent-only metric | | P1 | TiKV slow score | maximum window `store_slow_score` | score `>= 100` is high risk | Warning | replacement | Slow Node Statistics / Store Slow Score panel | | R1 | Raft log lag | maximum window 5m rate of `tikv_raftstore_log_lag_sum` | observation only; Dedicated P99 `> 5000` threshold does not transfer | Observation | partial | Raft Log / Raft log lag panel | | K1 | Pending compaction shards | `kv_engine_pending_compaction_shards` | observation only; Dedicated byte thresholds do not apply to shard counts | Observation | replacement | Kv Engine Shards panel | | K2 | Compaction failures | `sum by(container,result)(increase(kv_engine_compaction_result_counter{container=~"tikv|tikv-worker",result=~"failure|local_compaction_disabled"}[window]))` | `> 0` | Warning | exact | `components/kvengine/src/metrics.rs` declaration and `components/kvengine/src/compaction.rs` updates | | K3 | Compaction result evidence | `sum by(container,result)(increase(kv_engine_compaction_result_counter{container=~"tikv|tikv-worker"}[window]))` | report labeled count | Observation | exact | Source-defined counter; evidence is retained by container and result | | L1 | Error / warning logs | parent Loki selector `{cluster="",namespace="tidb-serverless",container="tikv-log"}` with noise exclusions; trim leading datetimes, group normalized lines, show top 10 by count | no unexpected actionable `ERROR`, `FATAL`, or `WARN` lines | Warning | log | Clinic Serverless parent logs API | `kv_engine_compaction_result_counter` is declared in `components/kvengine/src/metrics.rs` and updated by the compaction result paths in `components/kvengine/src/compaction.rs`. The source-defined result values are `success`, `local_compaction_disabled`, `not_retryable`, `schema_out_of_date`, `failure`, and `empty`. K3 preserves every observed result value; K2 warns only on the requested `failure` and `local_compaction_disabled` values. In multi-day item × date tables, the K3 cell lists the date's window total plus every container/result count for that date; a date with no observed compaction results shows `0.00`. ## Thread Groups T1 uses this ordered, dashboard-derived catalog. Exact duplicate dashboard aliases are omitted; the two `KvEngine worker CPU` spellings are combined into one selector because they have the same panel meaning. `Import CPU` retains one `sst_.*` selector despite the duplicate `CPU Usage` alias. `Backup CPU` and `Backup CPU Utilization` remain separate because they are distinct panel meanings. Every group query is pool-scoped through the required `metric(...)` labels and has this shape, grouping only by `instance` and never by `name`: ```promql sum by (instance) (rate(tikv_thread_cpu_seconds_total{...,name=~""}[5m])) ``` For every query-range window, retain the maximum and arithmetic average of each instance's returned sample values in **CPU cores**. If duplicate series for an instance occur, combine their values by timestamp before calculating the statistics, matching `sum by (instance)` semantics. A successful empty result is `No observed instances; max: 0.00 cores, avg: 0.00 cores`. A query error, malformed response, or missing `instance` label makes only that group row `Unknown: `; known group observations are retained, while the overall T1 result is `Unknown` for partial telemetry. T1 has no threshold or automated warning. Report cells retain one group row and render only `max: cores, avg: cores`: `max` is the largest per-instance maximum, while `avg` is the arithmetic mean of the per-instance averages. Never render instance names or individual instance values. A successful empty group is known zero internally and omitted. Omit another known group row only when its `max` is below `0.10` CPU cores; `0.10` remains visible. In multi-day tables, omit a row only when every displayed window is known and below that line. If an empty window shares a row with another window at or above `0.10`, retain the row for date-column alignment and render the empty cell as `No observed instances; max: 0.00 cores, avg: 0.00 cores`. Never omit a group with missing, malformed, or failed telemetry. | Ordered group | Required Serverless `name` regex | |---|---| | Raft store CPU | `(raftstore|rs|rfstore)_.*` | | Async apply CPU | `apply_[0-9]+|apply_follower_[0-9]+` | | Raft async writer CPU | `raft_io.*` | | Scheduler pool CPU | `sched_tokio_pool.*|sched_yatp_pool.*|sched_background.*` | | gRPC poll CPU | `grpc.*` | | KvEngine worker CPU | `txn-chunk-worker.*|txn_chunk_worker.*` | | RfEngine worker CPU | `sub-compact-wkr.*|compact-wal-worker.*` | | Unified read pool CPU | `unified_read_po.*` | | Background worker CPU | `background.*` | | GC worker CPU | `gc_worker.*` | | Import CPU | `sst_.*` | | Backup CPU | `(backup-worker|bkwkr|backup_endpoint).*` | | CDC worker CPU | `(cdcwkr.*|tso|cdc_.*)` | | TSO worker CPU | `tso_worker` | | IME CPU | `ime.*` | | IA CPU | `s3fifo.*` | | DFS worker CPU | `(builtin-dfs.*|memory-fs.*|s3-client.*)` | | Cloud worker CPU | `worker-server.*` | | Lock Manager Thread CPU | `(waiter_manager.*|deadlock_detect.*)` | | Resolved TS Worker CPU | `resolved_ts.*` | | Advance ts Worker CPU | `advance_ts.*` | | Scan lock Worker CPU | `inc_scan.*` | | Backup CPU Utilization | `(b.*k.*w.*k.*|backup_io)` | ## Log Scope Pass the full Grafana `control_plane_info` to the parent logs API as its routing parameter. Use `{cluster="",namespace="tidb-serverless",container="tikv-log"}` in LogQL; do not assume control-plane, k8s, or component labels exist in Loki. The proxy replaces/injects `cluster` consistently, so the explicit matcher validates the supplied k8s identity. Identify actionable `ERROR`, `FATAL`, and `WARN` patterns. Exclude routine Raft/CDC noise including `KV:Raft:StepPeerNotFound`, `no valid key found for split.`, `not_leader {`, `stale command`, `region_not_found {`, `epoch_not_match {`, `cdc register for a not found region`, `oneshot canceled`, `scan canceled`, and `cdc initialize fail`.