--- name: clinic-cli description: | Primary Clinic data access skill for Investibook. Use this builtin skill for PingCAP Clinic cluster metadata, metrics, tier-aware slow query / TopSQL / Statements, TiCDC changefeed metadata, Data Proxy SQL, Loki logs, Activity Hub events, continuous profiling, and diagnostic files. --- # Clinic CLI `clinic-cli` is the enabled Clinic data access skill. The legacy `clinic-api` skill is not enabled by default during normal Investibook sessions. Use this skill when the user needs Clinic cluster metadata, metrics, slow query or TopSQL data, Loki logs, Activity Hub events, continuous profiling data, or diagnostic files. Use it for read-only TiCDC changefeed metadata as well. ## Entry Point ```bash python3 .opencode/skills/builtin/clinic-cli/scripts/clinic_cli.py ... ``` Before the first use of any CLI subcommand in a diagnosis, run its `--help` and use that command's declared arguments. Do not infer argument names from another subcommand (for example, use `--start`/`--end`, not `--from`/`--to`; Top SQL uses `instances` and `summary`, not `list`; Statements does not accept `--limit`). ```bash python3 .opencode/skills/builtin/clinic-cli/scripts/clinic_cli.py --help python3 .opencode/skills/builtin/clinic-cli/scripts/clinic_cli.py --help ``` ## Capabilities | Area | Sub-skill | CLI subcommand | |------|-----------|----------------| | Cluster list / detail / topology | `cluster-metadata/SUBSKILL.md` | `search clusters`, `cluster-metadata`, `topology-snapshot` | | Persisted Slow Query / TopSQL (supported non-BYOC tiers) | `data-proxy/SUBSKILL.md` | `sql` | | Tier-aware slow queries (BYOC O11y) | `data-proxy/SUBSKILL.md` | `slow-query ...` | | Tier-aware TopSQL (BYOC O11y) | `data-proxy/SUBSKILL.md` | `top-sql ...` | | Live Statements and plans (BYOC O11y) | `data-proxy/SUBSKILL.md` | `statements ...` | | TiCDC changefeed metadata | `ticdc/SUBSKILL.md` | `ticdc list`, `ticdc detail` | | Prometheus metrics | `metrics/SUBSKILL.md` | `metrics` | | Loki logs | `loki/SUBSKILL.md` | `logs` | | Common ad-hoc queries | this file | `ad-hoc ...` | | Activity Hub events | `activity-hub/SUBSKILL.md` | `activity-events` | | Continuous profiling metadata | `continuous-profiling/SUBSKILL.md` | `continuous-profiling` | | Continuous profiling downloads | `continuous-profiling/SUBSKILL.md` | `continuous-profiling-download` | | Diagnostic file lists | `diagnostic-files/SUBSKILL.md` | `diagnostic-files` | | Diagnostic file downloads | `diagnostic-files/SUBSKILL.md` | `diagnostic-file-download` | Read `SUBSKILLS_INDEX.md` first, then open the matching `SUBSKILL.md` for domain-specific query guidance, metric names, SQL examples, log query patterns, or diagnostic-analysis notes. ### BYOC routing boundary The CLI resolves the cluster tier from exact cluster metadata. For a logical or physical BYOC cluster: - use `metrics` and `logs` normally; the canonical org/cluster routes enforce BYOC scope; - use `slow-query`, `top-sql`, and `statements` for customer-side O11y data; - do not use `sql` as a fallback for Slow Query, TopSQL, Statements, LogStats, Plan Replayer, or OOM records; - data-plane metrics use the physical namespace `tidb{physical_cluster_id}`; - profiling and diagnostic-file capability boundaries are defined in their respective subskills; follow those boundaries instead of retrying through a different tier's API. `statements list`, `statements plans`, and `statements plan-detail` are merged into `queries.json.statements` with an `operation` field. They also appear in the workspace report with `source: "statement"`. Full-window `statement_summary` is persisted as its own query evidence source. Discovery operations (`available-fields`, `statement-types`) are returned without persistence. For topology, runtime image tag, component node count, CPU, memory, or storage limits such as IOPS / throughput, use `cluster-metadata --include-components`. That path uses Clinic topology data from `/resource-pool/components` for dedicated, resource-pool, premium, and BYOC clusters when available; do not infer those facts from image-tag strings or stale detail payload fields. For historical topology or CPU core count at a requested timestamp, use `topology-snapshot --at ` instead of current `cluster-metadata`. It combines point-in-time metrics with nearby Activity Hub topology-change events and explicitly marks current topology as comparison-only. When the user gives an ISO8601 or timezone-bearing time such as `2026-06-30 12:00 UTC+8`, pass it to `--at` as an ISO8601 string such as `2026-06-30T12:00:00+08:00`. Do not manually convert it to Unix epoch before calling the CLI unless the user already provided epoch seconds. In the returned evidence, `metric_query_type=instant` means the component counts come from Prometheus instant queries evaluated at the requested timestamp. Prometheus may use a recent sample within its lookback window, so use Activity Hub as change evidence near scale boundaries. Do not describe `metric_window` as the node-count query range; it is report metadata only. If current topology differs from the instant metric evidence, do not infer a scale operation unless Activity Hub or another event source shows one. If `historical_metrics_complete=false` or a requested component has `historical_metric_available=false`, do not use current `cluster-metadata` as a fallback to answer historical node/core counts. Current topology may be shown only as current reference, and "no Activity Hub event" alone is not enough to claim historical topology equality. For cluster discovery, use `search clusters`. When the user gives a tenant id, pass it as `--tenant-id`; this is an exact filter on user-facing `tenantID`, not the internal Clinic `orgID`. ## Ad-hoc Fast Path When the user asks a targeted follow-up after diagnosis, prefer the stable `ad-hoc` recipes before composing raw SQL, PromQL, or LogQL by hand. These commands encode the known label joins, date partition formats, exact instance matching, and aggregation shape used by common ad-hoc evaluation cases. Use the nutshell `clinic-api` references for detailed upstream contracts when they exist in the workspace: ```text .opencode/.all-skills/nutshell-skills/platform/clinic-api/SKILL.md .opencode/.all-skills/nutshell-skills/platform/clinic-api/SUBSKILLS_INDEX.md .opencode/.all-skills/nutshell-skills/platform/clinic-api//SUBSKILL.md ``` Consult those references as needed, but execute through this builtin `clinic-cli` runtime adapter whenever possible so results are persisted to Investibook evidence files. Do not execute nutshell shell helpers directly unless the builtin CLI cannot express the request. For ad-hoc Activity Hub questions, use `activity-events` rather than direct REST/curl calls. This is especially important for Premium clusters: the builtin command queries the requested child cluster target first and automatically falls back to the parent/resource-pool target when Clinic rejects the child target. Direct API calls can miss those events or report a false failure. For matching ad-hoc requests, the recipe output is the terminal evidence. Do not launch `evidence-agent`, `diagnosis-analyst`, or `collect_evidence.py`, and do not keep exploring raw metrics/logs after a matching recipe returns the fields the user asked for. If the recipe reports `evidence_sufficient=false`, answer that the requested evidence is insufficient instead of escalating into a full diagnosis workflow. Use: ```bash python3 .opencode/skills/builtin/clinic-cli/scripts/clinic_cli.py ad-hoc --help python3 .opencode/skills/builtin/clinic-cli/scripts/clinic_cli.py ad-hoc --help ``` Recipe routing: | User intent | Recipe | | --- | --- | | Determine whether PD stores are TiKV or TiFlash | `ad-hoc store-roles` | | Get top slow-query digests in an incident window | `ad-hoc slow-query-top` | | Check TiDB TTL row processing counters in a bounded window | `ad-hoc ttl-rows` | | Build TopSQL links for the hottest TiKV read-IOPS nodes | `ad-hoc topsql-links` | | Rank TiDB P99 latency nodes with KV/TSO/GC breakdowns | `ad-hoc tidb-p99-breakdown` | | Aggregate focused error / timeout / unavailable log patterns | `ad-hoc logs` | | Rank top TiDB/TiKV/PD CPU overload nodes and count ERROR/WARN logs per node | `ad-hoc cpu-overload-logs` | | Find hottest component instances and related log patterns | `ad-hoc cpu-logs` | Notes: - `ad-hoc slow-query-top` builds the correct `slow_query_logs` date partitions and uses epoch-second `time` predicates for the bounded window. - `ad-hoc ttl-rows` uses the TiDB server counters `tidb_server_ttl_insert_rows` and `tidb_server_ttl_processed_expired_rows`. Do not substitute guessed `tidb_ttl_*` metric names for TTL row-processing questions. - `ad-hoc topsql-links` ranks TiKV nodes by read IOPS using `node_disk_reads_completed_total` joined to `kube_pod_info`, then returns ready-to-open Clinic TopSQL URLs for the selected TiKV pods. - `ad-hoc tidb-p99-breakdown` gathers TiDB query P99 plus plan-build, distributed execution, KV request, TSO wait, and Go GC STW pause breakdowns; if a required signal is missing, treat the returned `evidence_sufficient` flag as authoritative instead of fabricating a ranking. - `ad-hoc cpu-overload-logs` computes component CPU overload from `process_cpu_seconds_total` divided by each component's quota/maxprocs metric, then counts ERROR/WARN logs for the selected nodes. Prefer it over manually composing many per-node PromQL and Loki queries. - `ad-hoc cpu-logs` writes both metric and log evidence. Read the returned `persisted_evidence.metrics` and `persisted_evidence.logs` paths. ## Environment The CLI uses the same Clinic auth/env behavior: - `CLINIC_API_KEY` - `CLINIC_ENV` - `CLINIC_ENDPOINT` - `ENV_FILE` Optional override: - `CLINIC_API_SKILL_ROOT`: alternate skill directory containing `scripts/clinic_api.py` - The backend provides the workspace context. Do not set or override evidence directory environment variables in CLI commands. The CLI output includes `persisted_evidence` paths for the updated raw file and report, and omits the raw `payload` from stdout. ## File-First Workflow For data commands (`cluster-metadata`, `metrics`, `sql`, `slow-query`, `top-sql`, `statements`, `logs`, `ad-hoc`, `activity-events`, `continuous-profiling`, `diagnostic-files`, and download variants), treat the workspace files as the source of record: 1. Run the CLI command. 2. If the output contains `persisted_evidence`, read the referenced file to confirm the data was written. 3. Analyze from the persisted file or the concise command summary, not from any hidden output protocol. The CLI does not require the agent or backend to parse a `workspace_evidence` field or raw `payload` from stdout. Large stdout may be truncated by the agent runtime; the file referenced by `persisted_evidence` remains the durable copy. For metrics, `workspace_evidence_report.json` intentionally excludes raw `chart_data`. That is not data loss. Raw datapoints are stored in the `metrics.json` file referenced by `persisted_evidence.evidence_file_path` when `--inline-chart-data` is used, or can be replayed later from `chart_data_ref`. Do not call `ClinicAPI.query_range()` or other lower-level Python APIs just because the compact report lacks `chart_data`; read `metrics.json` instead. When you need chart datapoints for a known metric, look up the row by `metric_group_id` in `metrics.json.metrics[]` and read that row's `chart_data.series`. If no matching row has `chart_data`, use its `chart_data_ref` as the durable replay reference. Example: ```bash python3 - <<'PY' import json from pathlib import Path metric_group_id = "tidb_qps" data = json.loads(Path("evidences/metrics.json").read_text()) rows = [ row for row in data.get("metrics", []) if isinstance(row, dict) and row.get("metric_group_id") == metric_group_id ] for row in rows: chart = row.get("chart_data") if isinstance(row.get("chart_data"), dict) else {} series = chart.get("series") if isinstance(chart.get("series"), list) else [] print({ "name": row.get("name"), "instance": row.get("instance"), "summary": row.get("summary"), "series_count": len(series), "chart_data_ref": row.get("chart_data_ref"), }) PY ``` If the CLI reports a missing workspace context, surface the environment problem. Do not retry with a stdout-only workflow.