---
name: clinic-metrics
description: |
  PromQL metrics guidance, common TiDB/TiKV signals, and schema references for clinic-cli metrics.
---

# Metrics

Use the parent CLI command:

```text
python3 .opencode/skills/builtin/clinic-cli/scripts/clinic_cli.py metrics --help
```

Typical use:

```text
python3 .opencode/skills/builtin/clinic-cli/scripts/clinic_cli.py metrics \
  --cluster-id <cluster_id> \
  --name tidb_qps \
  --query "sum(rate(tidb_server_query_total[1m]))" \
  --start <start_rfc3339_or_unix> \
  --end <end_rfc3339_or_unix> \
  --step 60s \
  --metric-group-id tidb_qps \
  --unit qps
```

Command output includes overall `summary`, per-series `series_summaries`, and a `chart_data_ref`.
Use `--inline-chart-data` only when raw datapoints need to be persisted for UI
chart rendering or local inspection.

Important evidence boundary:

- `workspace_evidence_report.json` is a compact report and intentionally omits
  `chart_data`.
- Raw series from `--inline-chart-data` are stored in the persisted
  `metrics.json`, not in the report.
- If raw datapoints are needed after a successful command, read the
  `persisted_evidence.evidence_file_path` / `<workspace>/evidences/metrics.json`.
- To find chart data, filter `metrics.json.metrics[]` by `metric_group_id`, then
  read the matching row's `chart_data.series`. If `chart_data` is absent, use
  that row's `chart_data_ref`.
- Do not import `clinic_api.py` or call `ClinicAPI.query_range()` just because
  the compact report lacks `chart_data`.

## Query rules

- For common follow-up questions, prefer parent `ad-hoc` recipes before
  hand-writing PromQL: `ad-hoc store-roles` and `ad-hoc cpu-logs`.
- Use range queries for incident windows and trends.
- Choose `step` based on the time range; `60s` is a practical default for hour-level diagnosis.
- Cluster-store labels are generally handled by the backend. Do **not** add a
  `k8s_cluster`, `tidb_cluster_id`, or other guessed cluster matcher to TiDB
  QPS/SQL metrics: the canonical API already applies the requested cluster
  scope. For BYOC, query tenant-scoped TiDB metrics with the logical ID; query
  TiKV/PD/TiFlash host metrics with the `parent_id` physical BYOC ID. Exact
  Kubernetes/node joins use the physical namespace `tidb{physical_cluster_id}`.
- When diagnosing high TiDB CPU, also check Go GC metrics such as `go_gc_duration_seconds`, GC pause series, and count/sum/bucket variants.

## Common metric groups

| Group | PromQL starting point |
| --- | --- |
| TiDB QPS | `sum(rate(tidb_server_query_total[1m]))` |
| TiDB QPS by type | `sum(rate(tidb_server_query_total[1m])) by (type)` |
| TiDB QPS by instance | `sum(rate(tidb_server_query_total[1m])) by (instance)` |
| TiDB CPU | `sum(rate(process_cpu_seconds_total{component="tidb"}[1m])) by (instance)` |
| TiKV CPU | `sum(rate(tikv_thread_cpu_seconds_total[1m])) by (instance)` |
| TiKV gRPC p99 | `histogram_quantile(0.99, sum(rate(tikv_grpc_msg_duration_seconds_bucket[1m])) by (le,instance))` |
| TiDB GC duration | `go_gc_duration_seconds` and bucket/sum/count variants |
| KV request | `sum(rate(tidb_tikvclient_request_seconds_count[1m])) by (type,instance)` |
| Region count | `tikv_raftstore_region_count` |

## Ad-hoc metric recipes

Use these instead of reconstructing joins or regexes manually:

```text
clinic_cli.py ad-hoc store-roles --cluster-id <id> --start <start> --end <end>
clinic_cli.py ad-hoc cpu-logs --cluster-id <id> --start <start> --end <end>
```

For IOPS rankings, compaction-level details, or other metric-specific follow-up
questions, use the metric definitions in this skill plus a narrow
`clinic_cli.py metrics` command. Do not use a nearby ad-hoc recipe when the
requested metric shape differs.

More metric definitions are in:

- `metrics_1_core.md`
- `metrics_2_common.md`
- `metrics_3_other.md`

## Metric-Name Discovery (components not in the docs above)

When the component you need (for example TiCDC/changefeed, TiFlash, PD internals) has no recipe in the metric docs above, do not guess `--name`/`--query` values one command at a time — each miss costs a full round trip. Follow the `diagnosis-overview` skill's `references/metric-discovery.md` (`.opencode/skills/builtin/diagnosis-overview/references/metric-discovery.md`, fallback `.opencode/.all-skills/...`): one lookup in the PSC panel knowledge base (`/opt/agentkit/data/psc_knowledge_base_<tier>.json`) lists the ready-to-use metric names and PromQL, then run a handful of `metrics` commands.

- Compare incident-window metrics with a nearby baseline when possible.
- QPS changes alone do not explain latency. Correlate with CPU, KV request duration, lock/latch metrics, storage write metrics, and slow query / TopSQL.
- If a query returns no data, validate the time range, metric availability, and whether the cluster is shared, Premium, BYOC, or Dedicated.