---
name: clinic-metrics
description: |
  Clinic Data Proxy Prometheus metrics (PromQL), series discovery, METRICS_SCHEMA reference files.
---

# Metrics query API (PromQL)

Bearer `CLINIC_API_KEY` on the canonical org-scoped routes under `${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/metrics/...`. Here `orgID` is the TiDB Cloud/DBAAS organization ID. METRICS_SCHEMA reference files live in this same `metrics/` directory.

Query Prometheus metrics for TiDB clusters using PromQL.

**BYOC routing rule**: cluster-store metrics retain server-side `tidb_cluster_id` scoping. Exact BYOC data-plane queries are the deliberate exception: Data Proxy does **not** inject `tidb_cluster_id`, because Kubernetes and node metrics such as `kube_pod_info` and `node_*` do not carry that label. Scope those queries with the physical Kubernetes namespace (`tidb{physicalParentID}`) and the real labels shown by Grafana/series discovery. Do not manually add `tidb_cluster_id` to BYOC data-plane PromQL. Premium and every other tier keep their established injection behavior.


## Python helper (preferred)

Use `clinic_api.py` for all metrics queries — it resolves the canonical `orgID` from cluster metadata, then handles auth, endpoint selection, and JSON parsing automatically. The Python and shell helpers intentionally remain cluster-only at their call sites; callers do not need to look up `orgID` themselves.

When the caller already has the required TiDB Cloud `orgId` and `clusterId`, use
the batch helper instead. It takes the exact typed pair without metadata search,
runs independent queries concurrently within a six-call total budget including
recovery, compacts the time series, and
handles `cross_source_metrics_rejected` itself. Do not ask an agent to design a
retry after that provider error.

```bash
python3 scripts/clinic_api.py query_range_batch plan.json result.json
```

An orchestrator may provide a trusted context file separately:

```bash
python3 scripts/clinic_api.py query_range_batch context.json plan.json result.json
```

In that form, `context.json` owns `scenarioId`, exact `timeWindow`, `orgId`, and
`clusterId`; values copied into the agent-written plan are ignored. Historical
absolute `start`/`end` are derived from that trusted window.

`plan.json` uses this bounded contract:

```json
{
  "schemaVersion": "clinic-metrics-batch-plan-v1",
  "scenarioId": "memory-oom",
  "timeWindow": "historical:absolute;start=...;end=...;timezone=UTC;source=replay_cutoff_lookback",
  "target": {"orgId": "DBAAS_ORG_ID", "clusterId": "CLUSTER_ID"},
  "start": 1700000000,
  "end": 1700003600,
  "step": "60",
  "queries": [
    {
      "queryId": "mechanism",
      "source": "mixed",
      "description": "TiKV process RSS and allocator memory in the incident window",
      "query": "DATA_PLANE_QUERY or CLUSTER_STORE_QUERY",
      "fallbackQueries": [
        {"source": "data_plane", "query": "DATA_PLANE_QUERY"},
        {"source": "cluster_store", "query": "CLUSTER_STORE_QUERY"}
      ]
    }
  ]
}
```

Prefer separate source-homogeneous entries in the initial plan. When a
load-bearing expression must combine sources, declare both fallback queries
before execution. If Clinic returns `cross_source_metrics_rejected`, the helper
launches those fallbacks concurrently and returns one recovered query result.
For a simple top-level PromQL `or`, it can split the unchanged arms
automatically. It never rewrites arithmetic or aggregation semantics.

The output is `clinic-metrics-batch-v1`. It contains one bounded summary per
query (`status`, neutral `statement`, duration, series/sample counts, min/max/
average, capped per-series statistics, and recovery metadata). It does not
contain raw time-series arrays and is suitable for a downstream deterministic
evidence materializer.

### Built-in bounded recipes

For these scenarios, do not search the large metrics reference files or invent
PromQL. Use the built-in same-name recipe:

The fixed recipes are not a substitute for BYOC namespace scoping. When a BYOC investigation requires data-plane metrics and the recipe does not include the physical namespace, use an explicit bounded plan whose data-plane PromQL includes `namespace="tidb{physicalParentID}"`.

| Recipe | Fixed coverage |
|---|---|
| `memory-oom` | TiKV working-set/limit alignment, allocator composition, engine and block-cache memory |
| `capacity-resource-pressure` | TiKV working-set/limit alignment, thread CPU, Region and leader distribution |

The plan is only:

```json
{"schemaVersion":"clinic-metrics-batch-plan-v1","recipe":"memory-oom"}
```

Recipes use the trusted context file for target and time bounds, run their
fixed queries concurrently, and remain inside the same six-call total budget.

```python
import sys; sys.path.insert(0, ".opencode/skills/nutshell-skills/platform/clinic-api/scripts")
from clinic_api import ClinicAPI
api = ClinicAPI()

# Range query
r = api.query_range("CLUSTER_ID", 'sum(rate(tikv_thread_cpu_seconds_total{component="tikv"}[1m])) by (instance)', START, END, step="60")
for series in r.get("data", {}).get("result", []):
    inst = series["metric"].get("instance", "")
    vals = series["values"]  # [[timestamp, value], ...]
    print(f"{inst}: max={max(float(v[1]) for v in vals):.2f}")

# Instant query
r = api.query_instant("CLUSTER_ID", "tikv_raftstore_region_count", time=END)

# Series discovery
r = api.query_series("CLUSTER_ID", "tikv_grpc_msg_duration_seconds_bucket", START, END)

# Client-side join helper for infra metrics that need instance -> pod mapping.
# For logical BYOC, use the physical parent namespace explicitly.
metadata = api.get_cluster_by_id("CLUSTER_ID")
physical_id = (
    metadata.get("diagnosticContext", {}).get("parentId")
    or metadata.get("parentId")
    or "CLUSTER_ID"
)
r = api.query_exceeded_iops_by_pod(
    "CLUSTER_ID", "tikv", START, END, step="60", rate_window="2m",
    namespace=f"tidb{physical_id}",
)
for series in r.get("data", {}).get("result", []):
    pod = series["metric"].get("pod", "")
    volume_id = series["metric"].get("volume_id", "")
    source_pod = series["metric"].get("source_pod", "")  # ebs-csi-node-*
    vals = [float(v[1]) for v in series["values"]]
    print(f"{pod} {volume_id} max={max(vals):.3f} source={source_pod}")
```

## Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/metrics/query` | GET/POST | Instant query |
| `/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/metrics/query-range` | GET/POST | Range query |
| `/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/metrics/series` | GET/POST | Query series |

## BYOC logical and physical targets

The three canonical metrics endpoints accept a logical BYOC ID or its physical `byoc-host` ID. In both cases Data Proxy resolves the authorized customer-side O11y target and routing headers; callers must not send provider, region, deploy type, O11y URL, or placement options.

Use exact metadata to derive the physical namespace for data-plane queries:

```bash
METADATA=$(clinic_get_cluster_by_id "$LOGICAL_CLUSTER_ID")
PHYSICAL_CLUSTER_ID=$(printf '%s' "$METADATA" | jq -r '.diagnosticContext.parentId // .parentId // empty')

clinic_metrics_query_range "$LOGICAL_CLUSTER_ID" \
  "kube_pod_info{namespace=\"tidb${PHYSICAL_CLUSTER_ID}\"}" \
  "$START" "$END" 60
```

This matches the Clinic/Grafana Nodes Info panel shape. If a BYOC data-plane query is empty, inspect the panel PromQL and use `metrics/series` to confirm `namespace`, `pod`, `node`, `instance`, or `k8s_cluster_info`; do not add a synthetic `tidb_cluster_id` matcher. For cluster metrics such as `tidb_server_query_total`, keep using the logical or physical ID in the canonical path and let Data Proxy apply cluster-store scoping.

Route support does not guarantee that a given cluster/window has samples. If the corresponding Grafana panel is also empty, an empty Data Proxy result is not by itself a defect.

### Discovery workflow (recommended when a query returns no data)

Use the canonical `metrics/series` route to discover which labels exist for a metric on the target cluster:

```bash
CLUSTER_ID="..."
DBAAS_ORG_ID="..." # TiDB Cloud/DBAAS organization ID
END=$(date +%s)
START=$((END - 600))

curl -s -G "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/metrics/series" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  --data-urlencode 'match[]=tidb_server_query_total{cluster_id="'"${CLUSTER_ID}"'"}' \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" | jq '.data[0]'
```

If `series` shows the metric exists but your range query is empty, check time range and aggregation. If `series` itself returns empty, the metric is not available for that cluster.

## Instant Query

Evaluates a PromQL expression at a single point in time.

```bash
# GET method
curl -X GET "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/metrics/query?query=sum(tidb_server_query_total)" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}"

# POST method
curl -X POST "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/metrics/query" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -d "query=sum(tidb_server_query_total)"

# With timestamp
curl -X GET "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/metrics/query?query=sum(tidb_server_query_total)&time=2026-03-06T22:00:00Z" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}"
```

| Parameter | Required | Description |
|-----------|----------|-------------|
| `query` | Yes | PromQL expression |
| `time` | No | Evaluation timestamp (RFC3339 or Unix), default: now |
| `timeout` | No | Evaluation timeout |

## Range Query

Evaluates a PromQL expression over a time range.

```bash
# GET method
curl -X GET "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/metrics/query-range?query=sum(tidb_server_query_total)&start=2026-03-06T21:00:00Z&end=2026-03-06T22:00:00Z&step=15s" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}"

# POST method
curl -X POST "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/metrics/query-range" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -d "query=sum(tidb_server_query_total)" \
  -d "start=2026-03-06T21:00:00Z" \
  -d "end=2026-03-06T22:00:00Z" \
  -d "step=15s"
```

| Parameter | Required | Description |
|-----------|----------|-------------|
| `query` | Yes | PromQL expression |
| `start` | Yes | Start timestamp (RFC3339 or Unix) |
| `end` | Yes | End timestamp (RFC3339 or Unix) |
| `step` | Yes | Query resolution step (e.g., `15s`, `1m`, `5m`) |
| `timeout` | No | Evaluation timeout |

## Metrics Query Scenarios

### Scenario 1: Query TiDB CPU Usage

```bash
# CPU usage rate for all TiDB instances
curl -X GET "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/metrics/query-range" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -G \
  --data-urlencode "query=sum(rate(process_cpu_seconds_total{job=\"tidb\"}[1m]))" \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" \
  --data-urlencode "step=1m"
```

**Use cases**:
- Analyze database resource usage
- Troubleshoot performance bottlenecks

### Scenario 2: Query Database QPS

```bash
# Total QPS
curl -X GET "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/metrics/query-range" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -G \
  --data-urlencode "query=sum(rate(tidb_server_query_total[1m]))" \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" \
  --data-urlencode "step=1m"
```

**Use cases**:
- Monitor database load changes
- Analyze business traffic trends

## Common PromQL Queries

### TiDB Metrics

```promql
# Total QPS
sum(rate(tidb_server_query_total[1m]))

# QPS by type (Select, Insert, Update, Delete)
sum(rate(tidb_server_query_total[1m])) by (type)

# QPS by instance
sum(rate(tidb_server_query_total[1m])) by (instance)

# Current connections
tidb_server_connections

# Connections by instance
sum(tidb_server_connections) by (instance)

# P99 query duration
histogram_quantile(0.99, sum(rate(tidb_server_handle_query_duration_seconds_bucket[1m])) by (le))

# P95 query duration
histogram_quantile(0.95, sum(rate(tidb_server_handle_query_duration_seconds_bucket[1m])) by (le))

# Average query duration
sum(rate(tidb_server_handle_query_duration_seconds_sum[1m])) / sum(rate(tidb_server_handle_query_duration_seconds_count[1m]))

# CPU usage rate
rate(process_cpu_seconds_total{job="tidb"}[1m])

# Total CPU across all TiDB instances
sum(rate(process_cpu_seconds_total{job="tidb"}[1m]))

# Memory usage
process_resident_memory_bytes{job="tidb"}

# Statement execution count
sum(rate(tidb_executor_statement_total[1m])) by (type)

# TiKV client request duration
histogram_quantile(0.99, sum(rate(tidb_tikvclient_request_seconds_bucket[1m])) by (le))
```

### TiKV Metrics

```promql
# Storage size
sum(tikv_store_size_bytes) by (instance)

# Engine size
sum(tikv_engine_size_bytes) by (instance, type)

# gRPC message duration P99
histogram_quantile(0.99, sum(rate(tikv_grpc_msg_duration_seconds_bucket[1m])) by (le, type))

# Region count
sum(tikv_raftstore_region_count) by (instance)

# Leader count
sum(tikv_raftstore_leader_count) by (instance)

# Scheduler command duration
histogram_quantile(0.99, sum(rate(tikv_scheduler_command_duration_seconds_bucket[1m])) by (le))

# Coprocessor request duration
histogram_quantile(0.99, sum(rate(tikv_coprocessor_request_duration_seconds_bucket[1m])) by (le))

# Raftstore apply log duration
histogram_quantile(0.99, sum(rate(tikv_raftstore_apply_log_duration_seconds_bucket[1m])) by (le))

# Thread CPU usage
sum(rate(tikv_thread_cpu_seconds_total[1m])) by (name)
```

### PD Metrics

```promql
# Cluster status
pd_cluster_status

# Region status
pd_regions_status

# Scheduler region heartbeat
pd_scheduler_region_heartbeat

# Store status
pd_scheduler_store_status
```

### TiFlash Metrics

```promql
# Query queue size
tiflash_system_current_metric_QueryQueueSize

# Coprocessor request duration
histogram_quantile(0.99, sum(rate(tiflash_coprocessor_request_duration_seconds_bucket[1m])) by (le))

# CPU usage
rate(tiflash_proxy_process_cpu_seconds_total[1m])

# Logical CPU cores
tiflash_system_current_metric_LogicalCPUCores
```

## Metrics Reference Documentation

Complete TiDB METRICS_SCHEMA reference (637 metrics from TiDB 8.5.4) with full PromQL expressions:

| File | Description | Count |
|------|-------------|-------|
| [metrics_1_core.md](./metrics_1_core.md) | **Core Metrics** - Daily monitoring, first to check during troubleshooting | 30 |
| [metrics_2_common.md](./metrics_2_common.md) | **Common Metrics** - Deep analysis, performance tuning | 39 |
| [metrics_3_other.md](./metrics_3_other.md) | **Other Metrics** - Special scenarios, advanced debugging | 572 |

Each file contains metrics organized by component (TiDB/TiKV/PD/Process/Go Runtime) with:
- Full PromQL expression
- Labels and Quantile info
- Functional category (Query, Transaction, Storage, etc.)

### Core Metrics Quick Reference

**Query Performance**:
- `tidb_query_duration` - Query latency distribution (P90/P95/P99)
- `tidb_qps` / `tidb_server_query_total` - Queries per second
- `tidb_slow_query_duration` - Slow query latency

**Connections & Transactions**:
- `tidb_connection_count` / `tidb_server_connections` - Active connections
- `tidb_transaction_duration` - Transaction latency
- `tidb_transaction_retry_num` - Transaction retry count (conflict indicator)

**TiKV Storage**:
- `tikv_grpc_msg_duration` - TiKV gRPC latency
- `tikv_storage_async_request_duration` - Async request latency
- `tikv_coprocessor_request_duration` - Coprocessor latency
- `tikv_scheduler_command_duration` - Scheduler latency
- `tikv_scheduler_latch_wait_duration` - Latch wait time

**Raft Consensus**:
- `tikv_raftstore_commit_log_duration` - Raft commit latency
- `tikv_raftstore_apply_log_duration` - Raft apply latency

**PD**:
- `pd_tso_wait_duration` - TSO wait time (affects transaction start)
- `pd_cluster_status` - Cluster status

**Process Resources**:
- `process_cpu_seconds_total` - TiDB/TiKV/PD CPU usage
- `process_resident_memory_bytes` - TiDB/TiKV/PD memory usage
- `go_memstats_heap_alloc_bytes` - Go heap memory
- `tikv_engine_size_bytes` - TiKV storage engine size

### Additional Metrics (Not in Reference Files)

The reference files document TiDB's built-in METRICS_SCHEMA. Additional metrics available via DataPlane include:

**Kubernetes Metrics** (for TiDB Cloud):
- `kube_pod_*` - Pod status and resources
- `kube_node_*` - Node status
- `container_*` - Container resource usage

**AWS/GCP Infrastructure Metrics**:
- CloudWatch/Stackdriver metrics via Prometheus adapter
- EBS/Persistent Disk volume metrics
- Network interface metrics

These infrastructure metrics are cluster/environment specific. For reusable metric definitions, see [metrics_3_other.md](./metrics_3_other.md) under `DataPlane Infrastructure Metrics`, including:

- `tikv_exceeded_iops`
- `tiflash_exceeded_iops`
- `tikv_node_disk_iops`
- `tiflash_node_disk_iops`

**Recommended query order for storage IOPS troubleshooting**:
- Prefer `tikv_exceeded_iops` / `tiflash_exceeded_iops` first when checking whether the storage volume is actually hitting configured IOPS limits.
- If the exceeded metric returns no data, treat that as a **query-construction problem to verify first**, not as a valid “no exceed” result. Check concrete `namespace` / `instance` expansion and make sure `label_replace(..., "$1", ...)` is preserved literally.
- If the PromQL still returns empty through Clinic Data Proxy, use `clinic_api.py` helper `query_exceeded_iops_by_pod(...)` / `query_node_disk_iops_by_pod(...)`, which internally does `query_series(kube_pod_info)` and then joins client-side by `instance=node`.
- Only after that verification still fails should you fall back to `tikv_node_disk_iops` / `tiflash_node_disk_iops` to identify hot pods and estimate IOPS pressure at pod level.
- Treat `*_node_disk_iops` as a fallback / corroborating signal rather than a direct limit-breach signal.
- For parity with the current dashboard, prefer `rate_window="2m"` for exceeded IOPS helpers unless you explicitly want a different window.

## Response Format

```json
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {"__name__": "tidb_server_query_total", "instance": "tidb-0:10080"},
        "value": [1709766000, "12345.67"]
      }
    ]
  }
}
```

## Example: TiDB QPS (last hour)

```bash
source ~/.claude/skills/clinic-api/scripts/clinic-shell.sh
get_clinic_auth

CLUSTER_ID="10040083716234509777"
DBAAS_ORG_ID="1234567890" # TiDB Cloud/DBAAS organization ID
END=$(date -u +%Y-%m-%dT%H:%M:%SZ)
START=$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)   # Linux; macOS: START=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)

curl -s -X GET "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/metrics/query-range" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -G \
  --data-urlencode "query=sum(rate(tidb_server_query_total[1m]))" \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" \
  --data-urlencode "step=1m" | jq
```

## Compatibility-only routes

Use the canonical routes and helpers above for new calls. The old cluster metrics entry points remain available only for compatibility:

- `/data-proxy/metrics/{clusterID}/api/v1/query`
- `/data-proxy/metrics/{clusterID}/api/v1/query_range`
- `/data-proxy/metrics/{clusterID}/api/v1/series`

The shell equivalents are explicitly named `clinic_legacy_metrics_query`, `clinic_legacy_metrics_query_range`, and `clinic_legacy_metrics_series`.

Starter/Essential parent metrics identified by `control_plane_info` do not have an org/cluster canonical alias. Continue to use `/data-proxy/serverless-parent-metrics/api/v1/...` only for that parent/control-plane case; do not invent an org-scoped path for it.