--- name: tidbcloud-dedicated-daily-inspection description: | Run daily inspection for TiDB Cloud Dedicated clusters, generate one English report section per cluster, and archive results into one date-based daily report document. Use when the user wants a reusable Dedicated inspection skill, needs a daily health report, wants to inspect one or more Dedicated cluster IDs, or wants to convert an internal inspection SOP into a public skill. Prefer clinic-api for metadata and metrics, use the bundled script to produce markdown evidence, then create or update the date-based Feishu report file. --- # TiDB Cloud Dedicated Daily Inspection ## Overview Run a fixed TiKV-focused daily inspection for TiDB Cloud Dedicated clusters. The report follows the internal Feishu template structure, but all generated content should be in English and screenshot-free. For TiKV metrics, the monitoring target object must be `tikv-details`. The Feishu template shape is also persisted locally in `references/feishu-template-hints.md` so routine iterations do not need to reload the same source template. Reviewer-driven scope additions already folded into this skill: - the gRPC section must cover both write-path and read-path request types - the CPU and Memory section must compare current-window TiKV average memory usage against the previous 7-day baseline to detect leak-like growth - the memory-leak watch must suppress warning tags when the same TiKV nodes also show strong Region-count, workload-QPS, or CPU growth versus the same 7-day baseline, because that makes the memory increase workload-driven - the thread-pool section must include Unified Read Pool running-task pressure - the storage / write-path section must include flow-control, slow-score, and raft-log-lag add-on observations when those metrics are available The skill has three outputs: - one English markdown report section per cluster - one date-based daily report document that is readable by all PingCAP members - one reusable public skill with explicit thresholds and caveats ## Load References Read: - `references/inspection-rules.md` - `references/feishu-template-hints.md` Use the bundled script: - `scripts/dedicated_daily_inspection.py` Do not draft ad hoc PromQL before reading `references/inspection-rules.md`. ## Inputs Collect: - `cluster_ids` - preferred input - optional `source_doc` - for example `https://pingcap.feishu.cn/wiki/NEEuwYglZiGieakLhskc5Bt3nFh` - `report_date` - default: today in `Asia/Shanghai` - optional `relative_window` - for example `2h`, `6h`, or `1d` - optional `start` / `end` - use only when the user wants a custom window ## Data Sources Use sources in this order: 1. `clinic-api` - exact cluster lookup - cluster topology - Prometheus metrics via Data Proxy - TiKV metric review scoped to `tikv-details` - node disk review scoped to `dedicated-dataplane-node-info` - Loki logs via Data Proxy 2. `mcp__lark_mcp__` - fetch inspection source docs - create or update the final Feishu report ## Window Rules Window precedence: - if `start` and `end` are both provided, use that absolute window - else if `relative_window` is provided, inspect the latest duration ending at `end` when `end` is provided, otherwise ending at `now` - else use the date-based daily window below Date-based default: - for a past `report_date`, inspect the full natural day `00:00:00` to `23:59:59` - for today's `report_date`, inspect `00:00:00` to `now` - default timezone: `Asia/Shanghai` Always print the absolute inspection window in the final report. ## Cluster Discovery Preferred behavior: 1. Use explicit `cluster_ids` when available. 2. If the user gives a source Feishu doc, fetch it first. 3. If the doc exposes a Clinic cluster-list URL with `query=`, treat that value as a Clinic search input and discover candidate clusters with `clinic_search_clusters`. 4. Filter discovered candidates to: - `clusterStatus == active` - `clusterDeployTypeV2 == dedicated` Important caveat: - `fetch_doc` does not expand embedded Feishu Sheets into rows. - If the source doc relies on a Sheet for the exact cluster shortlist, the Clinic tenant query can over-include clusters. - In that case, use the discovered active Dedicated list only for sampling or ask the user for exact `cluster_ids`. ## Workflow Run steps in this order: 1. Resolve the cluster list. 2. Resolve the report window. 3. Use the bundled script to generate one section per cluster or one combined daily report. 4. Keep the section order fixed: - `Summary` - `Key Findings` - `Error Signals` - `gRPC QPS and Latency` - `CPU and Memory` - `Critical Thread Pool CPU` - `Disk Usage and Region Count` - `Disk Load` - `ERROR/WARN Logs` - `Follow-up Investigation` 5. Search for the daily Feishu doc by title: - `TiDBCloud Dedicated Daily Inspection - YYYY-MM-DD` 6. If `search_doc` is unavailable or returns validation errors, fall back to `list_docs my_library=true` and filter by exact title. 7. If the doc exists, append or replace the matching cluster section. 8. If the doc does not exist, create it in `my_library`. 9. Ensure the final Feishu doc is readable by all PingCAP members so readers do not need per-user re-authorization. ## Bundled Script Single cluster: ```bash python3 diagnosis/tidbcloud-dedicated-daily-inspection/scripts/dedicated_daily_inspection.py inspect \ --cluster-id 10400917620589115003 \ --report-date 2026-06-02 \ --env-file ~/.env ``` Latest 2 hours: ```bash python3 diagnosis/tidbcloud-dedicated-daily-inspection/scripts/dedicated_daily_inspection.py inspect \ --cluster-id 10400917620589115003 \ --relative-window 2h \ --env-file ~/.env ``` Date-based daily report: ```bash python3 diagnosis/tidbcloud-dedicated-daily-inspection/scripts/dedicated_daily_inspection.py daily-report \ --cluster-id 10400917620589115003 \ --report-date 2026-06-02 \ --env-file ~/.env ``` Latest 6 hours: ```bash python3 diagnosis/tidbcloud-dedicated-daily-inspection/scripts/dedicated_daily_inspection.py daily-report \ --cluster-id 10400917620589115003 \ --relative-window 6h \ --env-file ~/.env ``` Multiple clusters in one daily report: ```bash python3 diagnosis/tidbcloud-dedicated-daily-inspection/scripts/dedicated_daily_inspection.py daily-report \ --cluster-id 10400917620589115003 \ --cluster-id 10400917620589115004 \ --report-date 2026-06-02 \ --env-file ~/.env ``` Discovery: ```bash python3 diagnosis/tidbcloud-dedicated-daily-inspection/scripts/dedicated_daily_inspection.py discover \ --query 1372813089209061633 \ --env-file ~/.env ``` ## Report Rules The script returns English markdown. Reuse it directly when creating the Feishu doc. Daily report structure: ```text - Inspection Date: YYYY-MM-DD - Inspection Window: ... - Data Sources: Clinic API / Clinic Data Proxy / Loki ## () - TiKV Monitoring Target: tikv-details - TiKV Dashboard: tikv-details deep link - Node Monitoring Target: dedicated-dataplane-node-info - Node Dashboard: dedicated-dataplane-node-info deep link - Log Filter Target: tikv - Log Explore Link: tikv explore deep link - Summary - Verdict: Healthy / Warning / Error / Needs Review - Health Summary: healthy / keep watching / check now / needs review - Recommended Action: routine watch / keep watching / check now / review missing signals - Watch Items: top warning-focus metrics ### Key Findings ### Error Signals ### gRPC QPS and Latency ### CPU and Memory ### Critical Thread Pool CPU ### Disk Usage and Region Count ### Disk Load ### ERROR/WARN Logs ### Follow-up Investigation ``` Section-specific content requirements: - `CPU and Memory` - include current-window average TiKV memory usage versus the previous 7-day baseline - add a `WARNING` finding for potential memory-leak phenomenon when a TiKV node's current-window average exceeds its previous-7-day average by `>= 5` percentage points, unless the same node also shows strong Region-count, workload-QPS, or CPU growth that explains the memory increase - `gRPC QPS and Latency` - include write-path `kv_prewrite` / `kv_commit` - include read-path `coprocessor` / `kv_get` / `kv_batch_get` - `Critical Thread Pool CPU` - include per-instance thread-pool CPU interpretation - include Unified Read Pool running tasks and total time-used observation - `Disk Load` - include flow-control throttle / discard observations when available - include slow-score and raft-log-lag observations when available One date corresponds to one document. Put all cluster sections for that date into the same report. ## Feishu Output Preferred title: ```text TiDBCloud Dedicated Daily Inspection - YYYY-MM-DD ``` Creation flow: 1. Try `search_doc` with the exact title text. 2. If `search_doc` is unavailable, use `list_docs my_library=true` and filter by exact title locally. 3. If an exact title match exists, use `update_doc` append mode or `overwrite` when regenerating the full report. 4. Otherwise use `create_doc` with `wiki_space: "my_library"`. 5. After create or update, set the document visibility to readable by all PingCAP members. ## Notes - Keep the public skill and the generated report in English. - Separate source-defined thresholds from heuristic thresholds; `references/inspection-rules.md` already does this. - Use `clinic-api` first. Only fall back when the required signal is unavailable there. - Keep wording aligned to `tikv-details` rather than generic TiKV dashboard wording. - The report header should include direct links for `tikv-details`, `dedicated-dataplane-node-info`, and the `tikv` Loki query. - The automated `ERROR/WARN Logs` section should suppress generic CDC initializer noise, including `cdc initialize fail`, so the daily report stays focused on actionable TiKV errors. - If a metric is missing, mark the related check as `Unknown` instead of silently passing it. - Prefer sustained-window judgments over single-point spikes for `gRPC failed messages`, `gRPC message duration P99`, `TiKV CPU`, `TiKV memory`, `Store usage`, and `Compaction pending bytes`. - Interpret `Critical Thread Pool CPU` per TiKV instance. Do not use cluster-wide summed thread-pool cores to judge whether one node exceeded its per-node CPU limit. - Use Feishu-supported colored text tags for verdicts, actions, and finding severities so status is easy to scan in the final document. - Do not repeat the document title as an H1 inside the Feishu body. Put the title in the document metadata only. - Screenshots are currently out of scope for this skill. Keep the workflow metrics-first and text-report-first. - If the current tool session does not provide a Feishu permission-setting API, say so explicitly and treat the final share-setting step as a manual follow-up instead of silently leaving the doc private.