--- name: tidbcloud-serverless-daily-inspection description: Run pool-wide TiKV daily inspections for TiDB Cloud Serverless shared pools and generate English, screenshot-free daily report sections. Use when inspecting Starter or Essential shared TiKV pools identified by control-plane and Kubernetes-cluster routing labels. --- # TiDB Cloud Serverless Daily Inspection ## Overview Run a fixed TiKV-focused daily inspection for shared Serverless TiKV pools. The inspection identity is the pair `(control_plane_info, k8s_cluster_info)`, not a logical TiDB cluster, tenant, keyspace, or organization. Every result is **pool-wide** and must be described as such. The fixed scope is: - `namespace="tidb-serverless"` - `component="tikv"` - log `container="tikv-log"` `control_plane_info` and `k8s_cluster_info` are internal, undocumented labels. They are nevertheless required routing and isolation identities for this skill. Do not infer either label from a cluster ID or from a tenant. Obtain both values from the same Serverless dashboard variable/metric-label series. The `control_plane_info` CLI input is the full Grafana routing value `//` (for example, `us-west-2-f04/aws-us-west-2/devtier-infra`). When the caller gives only the pool name (for example, `us-west-2-f04`), resolve the full routing value with the `tidbcloud-serverless-pool-routing` skill before running the script; `k8s_cluster_info` is not derivable from the pool name and still comes from the dashboard series. The script passes it unchanged to parent APIs, derives `/` for PromQL's `control_plane_info` matcher, and uses `k8s_cluster_info` unchanged to scope metrics and validate the Loki-injected `cluster` identity. The script checks the pair with pool-scoped `tikv_engine_size_bytes` before inspection. For node-exporter disk checks, the node metric itself is never filtered by `namespace` or `component`. The script maps its `instance` to `kube_pod_info` with `label_replace(..., "instance", "$1", "node", "(.*)")`; that pod mapping is constrained by the derived `control_plane_info`, `k8s_cluster_info`, `namespace="tidb-serverless"`, and the strict TiKV storage pod regex `^tikv(?:-[a-z0-9]+)*-[0-9]+$`. Disk checks accept only non-root NVMe devices matching `^nvme[1-9][0-9]*n[1-9][0-9]*$`. ## Read First Read these references before modifying queries or interpreting a report: - [inspection rules](references/inspection-rules.md) - [Feishu template hints](references/feishu-template-hints.md) Use `scripts/serverless_daily_inspection.py`. Do not substitute Dedicated topology, node dashboards, `tidb` namespaces, or keyspace filters. ## Prerequisites Use Python 3.10 or newer in an activated virtual environment. From the loaded skill directory, install the helper dependency before running an inspection: ```bash python3 -m pip install -r requirements.txt ``` ## Inputs and Windows The tool accepts a repeatable two-value pool identity: ```bash --pool CONTROL_PLANE K8S_CLUSTER ``` - `CONTROL_PLANE` is the full Grafana routing value `//`. When the caller provides only a pool name such as `us-west-2-f04`, resolve it first with the `tidbcloud-serverless-pool-routing` skill (for example, `us-west-2-f04` → `us-west-2-f04/aws-us-west-2/devtier-infra`). - `inspect` requires exactly one `--pool`. - `daily-report` accepts one to 20 `--pool` flags. - Inputs are validated and deduplicated as identity pairs. - `--report-date` defaults to today in `Asia/Shanghai`. - A past report date uses its full natural day. Today uses midnight through now. - `--relative-window` accepts positive hour/day values up to 24h and may use `--end` as its anchor. `--start` cannot be combined with `--relative-window`. - An `inspect` window must be no longer than 24 elapsed hours. - A `daily-report` may span up to 31 local dates. The script slices natural days and explicit `--start`/`--end` ranges at local-day and 24-elapsed-hour boundaries, so a DST fall-back day is inspected in two windows and no parent API request exceeds 24 hours. - A multi-window `daily-report` produces one document body. Each shared-pool section renders every chapter as an item × date/window table; split windows on the same local date include their time boundaries and are not averaged or merged. - JSON output includes `report_range` with the canonical `start`, `end`, `timezone_name`, inclusive `start_date` and `end_date`, and a title-ready `label`. Use `report_range.label` for document titles; `report_date` remains the last included local date for backward compatibility. ## Data Sources Use Clinic's parent endpoints, with no logical cluster ID or organization ID: - `/data-proxy/serverless-parent-metrics/api/v1/{query,query_range,series}` - `/data-proxy/serverless-parent-logs/api/v1/query_range` `CLINIC_API_KEY` must be an internal Clinic access key. Store it only in the environment or an uncommitted local env file; never commit or print it. Pass the full Grafana `control_plane_info` value as the endpoint routing parameter. Each PromQL selector binds the derived two-segment `control_plane_info` label plus `k8s_cluster_info`, `namespace`, and `component`. Node-exporter D4-D7 selectors instead use only the NVMe device matcher and join to a `kube_pod_info` selector carrying the derived routing labels, namespace, and strict TiKV storage-pod regex. Parent Loki instead uses documented stream labels: `cluster=""`, `namespace="tidb-serverless"`, and `container="tikv-log"`. The proxy replaces/injects the cluster matcher from the routing identity; supplying it validates the requested k8s identity. Do not invent `control_plane_info`, `k8s_cluster_info`, or `component` as Loki labels. This is intentionally a shared-parent identity contract: do not use logical cluster IDs, canonical organization IDs, Dashboard `orgID`, or `clinic_get_cluster_by_id` routing for this workflow. API/query errors and malformed non-empty Prometheus responses are `Unknown`. Every successful Prometheus response with an empty result is known zero/healthy, including scalar, labeled-vector, range, B3, and thread-group paths. D8 remains unavailable/`Unknown` because it is structurally unsupported rather than a successful empty query result. ## Workflow 1. Resolve the inspection window and normalized pool list. 2. Run the bundled script for each shared pool. 3. Keep this section order: `Summary`, `Key Findings`, `Error Signals`, `gRPC QPS and Latency`, `CPU and Memory`, `Critical Thread Pool CPU`, `Disk Usage and Region Count`, `Disk Load`, `Compaction`, `ERROR/WARN Logs`, `Follow-up Investigation`. 4. Use the report markdown directly in the daily Feishu document. For a multi-day report, use one document for the requested date range; every pool section contains item × date chapter tables. Do not add screenshots or repeat the document title as an H1. 5. Always create a NEW Feishu document for each report run. NEVER overwrite, update, or append to an existing document, even when a document with the same title or date range already exists. The title MUST include the pool (cluster) name: `TiDBCloud Serverless Daily Inspection - - YYYY-MM-DD` for a single day or `TiDBCloud Serverless Daily Inspection - - YYYY-MM-DD to YYYY-MM-DD` for a multi-day range (for example, `TiDBCloud Serverless Daily Inspection - us-west-2-f04 - 2026-07-24 to 2026-07-26`). When a report covers multiple pools, include every pool name in the title in `--pool` order, separated by `+`. 6. Create the document in `my_library`. Do not search for or reuse existing documents; each run produces its own standalone report document. 7. Use colored Feishu tags for verdicts, actions, and severities; make the document readable by all PingCAP members. If no permission API exists, state the manual permission follow-up explicitly. ## Commands Run these commands with the loaded `tidbcloud-serverless-daily-inspection` skill directory as the shell working directory. Bundled helper paths are relative to that skill root. Inspect one shared pool: ```bash python3 scripts/serverless_daily_inspection.py inspect \ --pool 'us-east-1-f04/aws-us-east-1/devtier-infra' 'eks-prod-a' \ --report-date 2026-07-23 \ --env-file ~/.env ``` Create a report for several pools: ```bash python3 scripts/serverless_daily_inspection.py daily-report \ --pool 'us-east-1-f04/aws-us-east-1/devtier-infra' 'eks-prod-a' \ --pool 'eu-central-1-f02/aws-eu-central-1/devtier-infra' 'eks-prod-b' \ --relative-window 6h \ --env-file ~/.env ``` Create one multi-day report with chapter-level item × date tables: ```bash python3 scripts/serverless_daily_inspection.py daily-report \ --pool 'us-east-1-f04/aws-us-east-1/devtier-infra' 'eks-prod-a' \ --start '2026-07-20T00:00:00+08:00' \ --end '2026-07-23T00:00:00+08:00' \ --env-file ~/.env ``` ## Interpretation Rules - Reuse a Dedicated threshold only when the Serverless metric has the same unit and purpose. The script does not transfer byte thresholds to shard or WAL counts. - G1 is a pool-wide message-rate observation. Calculate `sum by(type)(rate(tikv_grpc_msg_duration_seconds_count[1m]))`, then report the inspection-window maximum and average separately for read request types `coprocessor`, `kv_get`, and `kv_batch_get`, and write request types `kv_prewrite` and `kv_commit`. Feishu output uses four G1 rows and the unit messages/s; never substitute the full-window counter increase. - B3 compares the current-window average memory percentage with the prior seven-day average by matching instance. It observes rises of at least 3 percentage points and warns at 5 points unless matching Region growth (at least 100 and 10%), gRPC workload QPS growth (at least 100 and 20%), or CPU growth (at least 5 percentage points) explains the rise. - Duration rules use parent `query_range`: G2 is the maximum rolling 30-minute failure count; G3 requires P99 `>=1s` for 15 minutes; B1/B2 require their 85/90% thresholds for 30 minutes; C1 requires `>=80%` plus a 10 percentage- point rise within 30 minutes. - T1 is a dashboard-group CPU-rate observation, not a threshold check. For each ordered group in [inspection rules](references/inspection-rules.md#thread-groups), query the pool-scoped `sum by (instance) (rate(tikv_thread_cpu_seconds_total{...}[5m]))` with that group's `name=~` matcher. The query intentionally has no `name` grouping. For each inspection window and TiKV instance, report the maximum and arithmetic average query-range sample values in CPU cores. Keep that per-instance evidence internally, but render one group-level pair: the largest instance `max` and the arithmetic mean of the instance `avg` values. A successful empty group is `No observed instances; max: 0.00 cores, avg: 0.00 cores`; a failed, malformed, or instance-less group is independently `Unknown` without discarding other groups' evidence. Single-day output uses one T1 row per group in an `Item × Observation` Markdown table; multi-day output uses the same stable rows across date/window columns. Report rendering omits a known group when its group-level maximum is below `0.10` CPU cores; `0.10` is included. A successful empty group is known zero internally and is omitted. In multi-day output, omit a row only when every displayed window is known and below that line. If an empty window shares a row with another window at or above `0.10`, retain the aligned row and render the empty cell as `No observed instances; max: 0.00 cores, avg: 0.00 cores`. Always retain a row with missing, malformed, or failed telemetry. Do not list instance names or individual instance values in report output, infer a per-instance thread-count threshold, or generate a T1 warning. - `ERROR`, `FATAL`, and actionable `WARN` logs are reviewed after excluding known noise, including `cdc initialize fail`. Trim leading datetimes, group otherwise equivalent normalized lines, and show the top 10 groups by count. In a multi-day report, the `ERROR/WARN Logs` chapter renders a separate `Content × Count` top-10 table for each date; log groups are never merged across dates. - Structurally unavailable placeholder rules (currently D8) remain visibly `Unknown` in their own report rows, but they never drive the verdict, the Summary unknown-rule list, or follow-up actions; they are placeholders to be fixed later. Only unexpected `Unknown` results (API/query errors or malformed non-empty responses) make a pool `Needs Review`, not `Healthy`. - D5 is a non-root NVMe weighted-queue observation only. D6 and D7 warn only when their mapped disk latency is strictly greater than 20 ms. D8 remains `Unknown`: no `aws_ebs_*` exceeded-IOPS series exists in Serverless parent metrics, and node IOPS must not be used as a ceiling approximation. - F2 uses the Dedicated flow-control discard-ratio rule and warns when the window peak is positive. It is a validated parent-only metric because it is absent from the Serverless dashboard. A successful response with no series is healthy zero under the global missing-data policy. - T3 uses the Dedicated-shape pool-wide elapsed-time query `sum by(instance)(increase(multilevel_level_elapsed{name="unified-read-pool",level="total"}[window])) / 1000000`. D2 sums `raft_engine_wal_write_throttle_duration_seconds_sum` increases and warns above zero; the histogram is observed only after `throttle_sleep > 0` in `components/rfengine/src/writer.rs`. D3 selects only `container="tikv"` and warns at two pending Raft Engine WAL compactions. - Compaction K1-K3 retain the required pool scope. K1 observes only `kv_engine_pending_compaction_shards`. K2 and K3 select `container=~"tikv|tikv-worker"`; K2 warns on failure or `local_compaction_disabled` increases, and K3 reports all result counts by container and result. The `cloud-engine` source declares `kv_engine_compaction_result_counter` in `components/kvengine/src/metrics.rs`, updates it in `components/kvengine/src/compaction.rs`, and validates `success`, `local_compaction_disabled`, `not_retryable`, `schema_out_of_date`, `failure`, and `empty` result semantics. K3 observes every result value; K2 warns only for the requested `failure` and `local_compaction_disabled` values. In a multi-day item × date table, the K3 cell lists the date's window total plus every container/result count for that date; a date with no observed compaction results shows `0.00`. - The output is English and screenshot-free. It reports shared-pool capacity and behavior only; it must not attribute results to individual tenants.