---
name: o11y-metrics-api
description: |
  Query TiDB Cloud O11Y (Observability) metrics via Prometheus-compatible API.
  Use when fetching TiDB cluster metrics, K8s node metrics, or any O11Y data.
---

# O11Y Metrics API

Query TiDB Cloud observability metrics using the internal Prometheus-compatible API.

## Clinic Integration (IMPORTANT)

**When only given a cluster ID**, use `clinic-api` skill to get cluster metadata first:

```bash
# Step 1: Get cluster metadata from Clinic
source ~/.claude/skills/clinic-api/scripts/clinic-shell.sh
get_clinic_auth
CLUSTER_INFO=$(clinic_get_cluster_by_id "2018131130879836160")

# Extract O11Y-required fields
VENDOR=$(echo "$CLUSTER_INFO" | jq -r '.clusterProviderName')  # aws, gcp, alicloud
REGION=$(echo "$CLUSTER_INFO" | jq -r '.clusterRegionName')    # us-west-2, ap-southeast-1
DEPLOY_TYPE=$(echo "$CLUSTER_INFO" | jq -r '.deployType')      # dedicated, resource_pool, premium

# For Premium clusters, get the parent Resource Pool ID
if [ "$DEPLOY_TYPE" = "premium" ]; then
  SHARED_POOL_ID=$(echo "$CLUSTER_INFO" | jq -r '.parentID')
else
  SHARED_POOL_ID="$CLUSTER_ID"
fi

# Step 2: Now query O11Y with vendor/region/sharedPoolId
source ~/.claude/skills/o11y-metrics-api/o11y-token.sh
AUTH0_TOKEN=$(get_o11y_token)
# ... continue with O11Y query flow
```

### Clinic Response Fields for O11Y

| Clinic Field | O11Y Parameter | Example |
|--------------|----------------|---------|
| `clusterProviderName` | `vendor` | `aws`, `gcp`, `alicloud` |
| `clusterRegionName` | `region` | `us-west-2`, `ap-southeast-1` |
| `clusterID` or `parentID` | `applicationID` | `2018131130879836160` |
| `tenantID` | `tenantID` | `1372813089209271775` |

### Cluster Type → O11Y Query Target

| Clinic deployType | O11Y applicationID | Notes |
|-------------------|-------------------|-------|
| `dedicated` | Use `clusterID` directly | Has its own O11Y registration |
| `resource_pool` | Use `clusterID` directly | SharedPool with TiKV/TiDB metrics |
| `premium` | Use `parentID` (Resource Pool) | Premium runs on parent's infra |
| `starter` | Not applicable | No O11Y metrics available |

## Architecture

```
┌─────────────────┐     ┌──────────────────────┐     ┌─────────────────────┐
│  Auth0 Token    │ --> │  Global Server (GS)  │ --> │  Data Server (DS)   │
│  (24h validity) │     │  (cluster metadata)  │     │  (metrics query)    │
└─────────────────┘     └──────────────────────┘     └─────────────────────┘
```

## Token Management (IMPORTANT)

**DO NOT fetch a new token for every request.** Token is valid for 24 hours.

### Token Helper Script

Use `.claude/skills/o11y-metrics-api/o11y-token.sh` to manage tokens:

```bash
# Source the helper and get token (fetches new only if expired)
source .claude/skills/o11y-metrics-api/o11y-token.sh
AUTH0_TOKEN=$(get_o11y_token)
```

The script:
1. Checks if cached token exists in `.env`
2. If valid (not expiring within 5 min), returns cached token
3. If expired/missing, fetches new token and saves to `.env`

### Manual Token Fetch (if needed)

```bash
# Only run this if you don't have a valid token
curl -s 'https://tidb-soc2.us.auth0.com/oauth/token' \
  -H 'content-type: application/x-www-form-urlencoded' \
  -d 'grant_type=client_credentials' \
  -d "client_id=${O11Y_CLIENT_ID}" \
  -d "client_secret=${O11Y_CLIENT_SECRET}" \
  -d 'audience=https://tidb-soc2.us.auth0.com/api/v2/'

# Response: {"access_token": "...", "expires_in": 86400, ...}
```

### .env Token Storage

```bash
# Credentials (permanent)
O11Y_CLIENT_ID=Sd4aekPCa5nMUSQBIvf8eGSL895WtV4o
O11Y_CLIENT_SECRET=<secret>

# Cached token (managed by script)
O11Y_AUTH0_TOKEN=eyJhbGciOiJSUzI1NiIs...
O11Y_AUTH0_TOKEN_EXPIRES=1770427199
```

## Two O11Y Cluster Types

Each TiDB SharedPool has **two** separate O11Y registrations with different metrics:

| Type | applicationID Pattern | Contains | Use For |
|------|----------------------|----------|---------|
| **Application** | `{sharedPoolId}` (e.g., `2013516099403907072`) | TiKV/TiDB metrics | P99 latency, request rate, queue depth |
| **K8s Infrastructure** | `dp-{dataplane-uuid}` (e.g., `dp-df6d0f93-...`) | kube-state-metrics | HPA replicas, CPU utilization, throttling |

Both share the same `tenantID`, `vendor`, and `region`. They use the same DS endpoint
but different O11Y UUIDs.

**IMPORTANT**: For NextGen clusters (`prod-tidbcloud-nextgen`), K8s infra uses
`dp-{dataplane-uuid}` as applicationID. The legacy `k8s-infra` + `projectID` pattern
only works for legacy `tidbcloud` bizType.

### Metric Family -> Source Mapping

| Metric family | Correct source |
|--------------|----------------|
| `tidb_*`, `tikv_*`, `pd_*` | Application UUID |
| `node_*` | K8s infra UUID |
| `kube_*` | K8s infra UUID |
| `aws_ebs_*`, `aws_ebs_csi_*` | K8s infra UUID |

**Guardrail**:
- If `node_disk_*`, `kube_*`, `aws_ebs_*`, or `aws_ebs_csi_*` returns `success` with an empty result on the application UUID, do **not** conclude the metric is absent.
- Resolve the infra UUID first, then re-run the query.
- For Dedicated / legacy `tidbcloud`, prefer `applicationID=k8s-infra` + `projectID`.
- For NextGen, use the tenant-scoped `dp-*` discovery flow below, with `x-org-id=<tenantID>`, `x-biz-type=<cluster bizType>`, and `x-region-id=<vendor>-<region>`.
- For Premium AWS disk analysis, if `aws_ebs_*` is still absent on the correct infra target, fall back to `kube_persistentvolume* + kube_volumeattachment* + node_disk_*` and map `PV -> volume_id -> /dev/xvd* -> nvme*`.

## Query Flow

**Required inputs**: SharedPool ID, vendor, region.

### Step 1: Resolve Application O11Y UUID

**GS endpoint is ALWAYS `www.gs.us-west-2.aws.observability.tidbcloud.com`** regardless
of target cluster location. Pass `vendor` and `region` as query parameters.

```bash
source .claude/skills/o11y-metrics-api/o11y-token.sh
AUTH0_TOKEN=$(get_o11y_token)

SHARED_POOL_ID="2013516099403907072"
VENDOR="alicloud"
REGION="ap-southeast-1"

# Query app cluster metadata (GS endpoint is always us-west-2.aws)
APP_INFO=$(curl -s "http://www.gs.us-west-2.aws.observability.tidbcloud.com/api/v1/clusters/" \
  -H "Authorization: Bearer ${AUTH0_TOKEN}" \
  -G \
  --data-urlencode "vendor=${VENDOR}" \
  --data-urlencode "region=${REGION}" \
  --data-urlencode "applicationID=${SHARED_POOL_ID}")

APP_UUID=$(echo "$APP_INFO" | jq -r '.data.clusters[0].id')
TENANT_ID=$(echo "$APP_INFO" | jq -r '.data.clusters[0].tenantID')
DS_ENDPOINT="https://www.ds.${REGION}.${VENDOR}.observability.tidbcloud.com"
```

**Response:**
```json
{
  "code": 0,
  "data": {
    "clusters": [{
      "applicationID": "2013516099403907072",
      "bizType": "prod-tidbcloud-nextgen",
      "id": "019bda6f-3beb-7eb8-bfb8-9178f018329e",
      "internalReadUri": "https://www.ds.ap-southeast-1.alicloud.observability.tidbcloud.com/internal/v2/metrics/019bda6f-3beb-7eb8-bfb8-9178f018329e/api/v1/",
      "tenantID": "1372813089209289635",
      "vendor": "alicloud",
      "region": "ap-southeast-1"
    }]
  }
}
```

Note: `bizType` is optional in the query -- GS auto-matches if omitted.

### Step 2: Resolve K8s Infrastructure O11Y UUID (Auto-Discovery)

The K8s infra O11Y UUID is registered under `applicationID=dp-{dataplane-uuid}`.
Since the dataplane UUID is not in the app response, use this auto-discovery algorithm:

```bash
NAMESPACE="tidb${SHARED_POOL_ID}"

# 2a. List all dp- entries for same tenant + vendor + region
DP_ENTRIES=$(curl -s "http://www.gs.us-west-2.aws.observability.tidbcloud.com/api/v1/clusters/" \
  -H "Authorization: Bearer ${AUTH0_TOKEN}" \
  -G \
  --data-urlencode "vendor=${VENDOR}" \
  --data-urlencode "region=${REGION}" \
  --data-urlencode "tenantID=${TENANT_ID}" \
  | jq '[.data.clusters[] | select(.applicationID | startswith("dp-"))]')

DP_COUNT=$(echo "$DP_ENTRIES" | jq 'length')

# 2b. If single dp- entry, use it directly
if [ "$DP_COUNT" -eq 1 ]; then
  K8S_UUID=$(echo "$DP_ENTRIES" | jq -r '.[0].id')
else
  # 2c. If multiple, probe each for our namespace (~0.5s per probe)
  for i in $(seq 0 $((DP_COUNT - 1))); do
    UUID=$(echo "$DP_ENTRIES" | jq -r ".[$i].id")
    COUNT=$(curl -s "${DS_ENDPOINT}/internal/v2/metrics/${UUID}/api/v1/query" \
      -G --data-urlencode "query=kube_horizontalpodautoscaler_status_current_replicas{namespace=\"${NAMESPACE}\"}" \
      | jq '.data.result | length')
    if [ "$COUNT" -gt 0 ]; then
      K8S_UUID="$UUID"
      break
    fi
  done
fi
```

**How it works:**
1. Query GS for all entries matching the tenant, filter to `dp-*` applicationIDs
2. If only one `dp-` entry exists, it must be the right one
3. If multiple exist, probe each with an instant query for `kube_*{namespace="tidb{id}"}`
   -- the first hit (~0.5s per probe) identifies the correct K8s infra UUID

### Step 3: Query Metrics from Data Server

**DS endpoint varies by vendor/region** -- construct from Step 1.

```bash
# Application metrics (TiKV/TiDB) -- use APP_UUID
curl -s "${DS_ENDPOINT}/internal/v2/metrics/${APP_UUID}/api/v1/query" \
  -G --data-urlencode "query=up"

# K8s infrastructure metrics (HPA, CPU) -- use K8S_UUID
curl -s "${DS_ENDPOINT}/internal/v2/metrics/${K8S_UUID}/api/v1/query" \
  -G --data-urlencode "query=kube_horizontalpodautoscaler_status_current_replicas{namespace=\"${NAMESPACE}\"}"

# Node / disk / EBS metrics -- also use K8S_UUID, never APP_UUID
curl -s "${DS_ENDPOINT}/internal/v2/metrics/${K8S_UUID}/api/v1/query" \
  -G --data-urlencode "query=count(node_disk_io_time_seconds_total)"

curl -s "${DS_ENDPOINT}/internal/v2/metrics/${K8S_UUID}/api/v1/query" \
  -G --data-urlencode "query=topk(5, rate(aws_ebs_csi_read_bytes_total[2m]))"

# Range query example
curl -s "${DS_ENDPOINT}/internal/v2/metrics/${APP_UUID}/api/v1/query_range" \
  -G \
  --data-urlencode "query=sum(rate(tikv_coprocessor_request_duration_seconds_count[5m]))" \
  --data-urlencode "start=1769875200" \
  --data-urlencode "end=1769961599" \
  --data-urlencode "step=300s"
```

## URL Patterns

| Service | URL Pattern | Notes |
|---------|-------------|-------|
| Auth0 | `https://tidb-soc2.us.auth0.com/oauth/token` | Token endpoint |
| Global Server | `http://www.gs.us-west-2.aws.observability.tidbcloud.com/api/v1/clusters/` | **ALWAYS this endpoint** |
| Data Server | `https://www.ds.{region}.{vendor}.observability.tidbcloud.com/internal/v2/metrics/{uuid}/api/v1/` | Varies by cluster |

## Global Server Query Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| vendor | Yes | Cloud provider: `aws`, `gcp`, `alicloud` |
| region | Yes | Cloud region: `us-west-2`, `ap-southeast-1`, etc. |
| bizType | No | Business type filter (auto-matches if omitted; see below) |
| applicationID | No | SharedPool ID, `dp-{dataplane-uuid}`, or other identifier |
| tenantID | No | Tenant ID filter (useful for listing all dp- entries) |
| projectID | No | Project ID (legacy `k8s-infra` only, not needed for NextGen) |

### BizType Values

| bizType | Description |
|---------|-------------|
| `tidbcloud` | Legacy dedicated clusters |
| `prod-tidbcloud-nextgen` | Production NextGen clusters |
| `staging-tidbcloud-nextgen` | Staging NextGen clusters |
| `dev-tidbcloud-nextgen` | Dev NextGen clusters |
| `prod-devtier` | Production serverless |
| `autonomous` | Autonomous clusters |
| `byoc` | Bring Your Own Cloud |

## Data Server Endpoints

| Endpoint | Method | Parameters |
|----------|--------|------------|
| `/query` | GET | `query` (PromQL), `time` (optional) |
| `/query_range` | GET | `query`, `start`, `end`, `step` |

## Common Metrics

### TiDB/TiKV Application Metrics

```promql
# Coprocessor P99 latency
histogram_quantile(0.99, sum(rate(tikv_coprocessor_request_duration_seconds_bucket[5m])) by (le))

# Coprocessor request rate
sum(rate(tikv_coprocessor_request_duration_seconds_count[5m]))

# Remote coprocessor worker request rate
sum(rate(tikv_worker_remote_cop_request_duration_seconds_count[5m]))

# Remote coprocessor P99 process duration
histogram_quantile(0.99, sum(rate(tikv_worker_remote_cop_process_duration_seconds_bucket[5m])) by (le))

# TiDB query rate
sum(rate(tidb_server_query_total[5m]))
```

### K8s Infrastructure Metrics

Query using the **K8S_UUID** resolved via auto-discovery (Step 2 above).
Use `namespace="tidb{SHARED_POOL_ID}"` to scope to the correct SharedPool.

```promql
# HPA current replicas
kube_horizontalpodautoscaler_status_current_replicas{namespace="tidb2013516099403907072", horizontalpodautoscaler=~".*coprocessor.*"}

# HPA desired replicas
kube_horizontalpodautoscaler_status_desired_replicas{namespace="tidb2013516099403907072", horizontalpodautoscaler=~".*coprocessor.*"}

# HPA observed CPU utilization (MUST include metric_target_type label)
kube_horizontalpodautoscaler_status_target_metric{namespace="tidb2013516099403907072", horizontalpodautoscaler=~".*coprocessor.*", metric_name="cpu", metric_target_type="utilization"}

# CPU throttling
rate(container_cpu_cfs_throttled_periods_total{namespace="tidb2013516099403907072", pod=~".*coprocessor.*"}[5m]) / rate(container_cpu_cfs_periods_total{namespace="tidb2013516099403907072", pod=~".*coprocessor.*"}[5m])

# Node info
kube_node_info
```

**Legacy note**: The `applicationID=k8s-infra` + `projectID` pattern only works
for legacy `tidbcloud` bizType. For NextGen (`prod-tidbcloud-nextgen`), always use
the `dp-` auto-discovery method.

## Response Format

Standard Prometheus response:

```json
{
  "status": "success",
  "isPartial": false,
  "data": {
    "resultType": "matrix",
    "result": [{
      "metric": {"__name__": "tikv_coprocessor_request_duration_seconds_count"},
      "values": [
        [1769875200, "901.49"],
        [1769875500, "877.27"]
      ]
    }]
  }
}
```

## Complete Example

```bash
#!/bin/bash
cd /path/to/project

# 1. Get token (cached, only fetches if expired)
source .claude/skills/o11y-metrics-api/o11y-token.sh
AUTH0_TOKEN=$(get_o11y_token)

# 2. Define inputs (only these 3 are required)
SHARED_POOL_ID="2013516099403907072"
VENDOR="alicloud"
REGION="ap-southeast-1"
NAMESPACE="tidb${SHARED_POOL_ID}"

# 3. Resolve Application O11Y UUID from GS
APP_INFO=$(curl -s "http://www.gs.us-west-2.aws.observability.tidbcloud.com/api/v1/clusters/" \
  -H "Authorization: Bearer ${AUTH0_TOKEN}" \
  -G \
  --data-urlencode "vendor=${VENDOR}" \
  --data-urlencode "region=${REGION}" \
  --data-urlencode "applicationID=${SHARED_POOL_ID}")

APP_UUID=$(echo "$APP_INFO" | jq -r '.data.clusters[0].id')
TENANT_ID=$(echo "$APP_INFO" | jq -r '.data.clusters[0].tenantID')
DS_ENDPOINT="https://www.ds.${REGION}.${VENDOR}.observability.tidbcloud.com"

# 4. Auto-discover K8s Infrastructure O11Y UUID
DP_ENTRIES=$(curl -s "http://www.gs.us-west-2.aws.observability.tidbcloud.com/api/v1/clusters/" \
  -H "Authorization: Bearer ${AUTH0_TOKEN}" \
  -G \
  --data-urlencode "vendor=${VENDOR}" \
  --data-urlencode "region=${REGION}" \
  --data-urlencode "tenantID=${TENANT_ID}" \
  | jq '[.data.clusters[] | select(.applicationID | startswith("dp-"))]')

DP_COUNT=$(echo "$DP_ENTRIES" | jq 'length')
K8S_UUID=""

if [ "$DP_COUNT" -eq 1 ]; then
  K8S_UUID=$(echo "$DP_ENTRIES" | jq -r '.[0].id')
elif [ "$DP_COUNT" -gt 1 ]; then
  for i in $(seq 0 $((DP_COUNT - 1))); do
    UUID=$(echo "$DP_ENTRIES" | jq -r ".[$i].id")
    COUNT=$(curl -s "${DS_ENDPOINT}/internal/v2/metrics/${UUID}/api/v1/query" \
      -G --data-urlencode "query=kube_horizontalpodautoscaler_status_current_replicas{namespace=\"${NAMESPACE}\"}" \
      | jq '.data.result | length')
    if [ "$COUNT" -gt 0 ]; then
      K8S_UUID="$UUID"
      break
    fi
  done
fi

echo "APP_UUID=${APP_UUID}"   # For TiKV/TiDB application metrics
echo "K8S_UUID=${K8S_UUID}"   # For K8s infrastructure metrics

# 5. Query application metrics (TiKV P99 latency)
START_TIME=1769875200  # 2026-02-01 00:00:00 CST
END_TIME=1769961599    # 2026-02-01 23:59:59 CST

curl -s "${DS_ENDPOINT}/internal/v2/metrics/${APP_UUID}/api/v1/query_range" \
  -G \
  --data-urlencode "query=histogram_quantile(0.99, sum(rate(tikv_worker_remote_cop_process_duration_seconds_bucket[5m])) by (le))" \
  --data-urlencode "start=${START_TIME}" \
  --data-urlencode "end=${END_TIME}" \
  --data-urlencode "step=300s" | jq '.data.result[0].values[:5]'

# 6. Query K8s infrastructure metrics (HPA replicas)
curl -s "${DS_ENDPOINT}/internal/v2/metrics/${K8S_UUID}/api/v1/query_range" \
  -G \
  --data-urlencode "query=kube_horizontalpodautoscaler_status_current_replicas{namespace=\"${NAMESPACE}\", horizontalpodautoscaler=~\".*coprocessor.*\"}" \
  --data-urlencode "start=${START_TIME}" \
  --data-urlencode "end=${END_TIME}" \
  --data-urlencode "step=300s" | jq '.data.result[0].values[:5]'
```

## Troubleshooting

| Issue | Cause | Solution |
|-------|-------|----------|
| `{"clusters":[]}` | Wrong vendor/region/bizType | Check cluster's actual location and bizType |
| 405 Blocked (Alicloud WAF) | IP not whitelisted | Use whitelisted IP or internal network |
| 401 Unauthorized | Token expired | Run `get_o11y_token` to refresh |
| 404 Not Found | Wrong O11Y UUID or DS endpoint | Verify UUID from GS response |
| Empty metrics | Cluster not registered in O11Y | Check if O11Y is configured for this cluster |

## Codebase Integration

Existing Go implementation: `pkg/autoscale/input/metric_fetcher.go`

```go
// Get O11Y cluster UUID
uuid, err := fetcher.QueryO11yClusterUUID(req, &o11yConfig)

// Query metrics with PromQL
metrics, err := fetcher.QueryO11yForMetrics(req, promQL, uuid)
```

## Related Files

- `.claude/skills/o11y-metrics-api/o11y-token.sh` - Token management helper
- `.env` - Credentials and cached token
- `pkg/autoscale/input/metric_fetcher.go` - Go implementation