---
name: clinic-continuous-profiling
description: |
  Canonical physical-BYOC profiling inventory and the existing NGM
  CPU/heap/mutex/goroutine pprof download and analysis workflow.
---

# Continuous profiling (ConProf / Go pprof)

Clinic has two separate Continuous Profiling contracts:

- canonical Data Proxy exposes read-only group inventory for a physical BYOC `byoc-host` cluster;
- the established NGM APIs expose profile token/download flows for supported Dedicated/Premium workflows.

Do not mix their target IDs, headers, or capabilities.

## Canonical BYOC read-only inventory

BYOC profiles remain in the customer's O11y environment. The canonical routes proxy only JSON inventory and accept only the physical `byoc-host` cluster ID:

```text
GET /data-proxy/api/v1/orgs/{orgID}/clusters/{physicalClusterID}/continuous-profiling/group-profiles?begin_time=...&end_time=...
GET /data-proxy/api/v1/orgs/{orgID}/clusters/{physicalClusterID}/continuous-profiling/group-profiles/{ts}
```

A logical BYOC ID is deliberately unsupported for these routes. Resolve its physical parent through exact metadata:

```bash
source scripts/clinic-shell.sh
get_clinic_auth

METADATA=$(clinic_get_cluster_by_id "$LOGICAL_CLUSTER_ID")
PHYSICAL_CLUSTER_ID=$(printf '%s' "$METADATA" | jq -r '.diagnosticContext.parentId // .parentId // empty')

clinic_byoc_continuous_profiling_group_profiles \
  "$PHYSICAL_CLUSTER_ID" "$BEGIN" "$END"
clinic_byoc_continuous_profiling_group_detail \
  "$PHYSICAL_CLUSTER_ID" "$TS"
```

`begin_time`, `end_time`, and detail `ts` are positive Unix seconds. O11y floors list bounds to minutes, and the minute-rounded list span must not exceed three hours. A `null` list is normalized to `[]`; detail may return JSON `null` when the requested group does not exist.

This canonical BYOC contract does **not** expose action tokens, profile bytes, download, view, or config. Do not infer canonical routes for them and do not copy profiles into Clinic storage. Route support also does not guarantee that every physical cluster has a finished profile group in every window.

## Existing NGM pprof contract

Clinic exposes **Next-Gen Monitoring (NGM)** HTTP APIs under `${CLINIC_ENDPOINT}/ngm/api/v1/continuous_profiling/` for profiling data collected from supported non-BYOC workflows. Profiles are **Go `pprof` protobuf** (or compatible) binaries suitable for `go tool pprof` and flame graph tools.

**Typical availability**: Dedicated, and Premium with special header rules

## Authentication and headers

Use the **same** internal API key as other Clinic APIs, plus **cluster placement headers** (mirrors Clinic UI).

| Header | Source | Notes |
|--------|--------|--------|
| `Authorization` | `Bearer ${CLINIC_API_KEY}` | Required for programmatic access |
| `x-cluster-id` | User-facing cluster ID | Same as `clusterID` in List/Get cluster |
| `x-org-id` | List cluster **`tenantID`** | NGM expects the **tenant** id (same as user-facing tenant ID). This is **not** the internal `orgID` used in `GET /orgs/{orgID}/clusters/...` |
| `x-project-id` | List or Get cluster `projectID` / `projectId` | If missing on list row, call Get Cluster and read project id |
| `x-deploy-type` | **`clusterDeployType`** (List or Get cluster) | NGM uses this field, **not** `clusterDeployTypeV2`. Lowercase, e.g. `dedicated`, `premium` (`ascii_downcase`; spaces → `_` if needed) |
| `x-provider` | e.g. `aws`, `gcp`, `alicloud` | From cluster metadata: Get Cluster often exposes **`provider`**; list row may use `cloudProvider` / `vendor` |
| `x-region` | e.g. `us-east-1` | From cluster `region` |

**Agent workflow**: Run `clinic_search_cluster_by_id` → jq-select exact `clusterID` → read **`tenantID`** (for `x-org-id`), `projectID`, **`clusterDeployType`**, **`clusterProviderName`** / **`clusterRegionName`** (or Get Cluster fallbacks) → attach headers on every NGM request below. Keep internal **`orgID`** only for Clinic REST paths such as Get Cluster, not for NGM `x-org-id`.

Canonical physical-BYOC helpers are `clinic_byoc_continuous_profiling_group_profiles` and `clinic_byoc_continuous_profiling_group_detail` in `byoc-diagnostics/scripts/clinic-byoc-diagnostics.sh`. The existing NGM helpers remain `clinic_continuous_profiling_group_profiles`, `clinic_continuous_profiling_detail`, `clinic_continuous_profiling_action_token`, `clinic_continuous_profiling_download`, and `clinic_continuous_profiling_fetch` in `continuous-profiling/scripts/clinic-continuous-profiling.sh`. Both modules are sourced by `scripts/clinic-shell.sh`. Do not call the NGM download helpers as though they were canonical BYOC operations.

**Default download directory** (when `clinic_continuous_profiling_download` omits the third argument, or `clinic_continuous_profiling_fetch` omits the sixth): same as diagnostic files — **`CLINIC_DIAGNOSTIC_DOWNLOAD_DIR`** if set when the command runs, else **`$PWD`**. Generated filenames: `pprof-download-<unix>.bin` (`download` only) or `pprof-<ts>-<component>-<address>-<profile_type>.pb` (`fetch`, with `/` and `:` in component/address sanitized to `_`). Override by passing an explicit path. Stderr: **`clinic_continuous_profiling_download: wrote <path>`**.

Dedicated clusters continue to use the standard headers above and the existing helper functions.

## Premium cluster pitfalls

Premium continuous profiling is easy to break if you reuse Dedicated assumptions. Follow these rules exactly:

| Pitfall | Correct rule | Why |
|--------|---------------|-----|
| 1. `x-deploy-type` uses API-returned `nextgen-host` | **Always send `x-deploy-type: premium`** | NGM routes Premium profiling by deploy type, not by the UI/metadata host flavor |
| 2. `x-cluster-id` uses the Premium child cluster ID | **Use the Resource Pool ID (`parent_id`)** | Premium profiling data is keyed by the parent resource pool, not the child cluster record |
| 3. `x-project-id` is required like Dedicated | **Omit `x-project-id` entirely** | Premium NGM accepts the request without it; sending it is unnecessary and can mislead troubleshooting |
| 4. Helper functions should work the same | **Do not use `clinic_continuous_profiling_*` helpers for Premium** | Current helpers are built around the standard cluster-id / project-id path and are not safe for Premium |

### Premium-specific header mapping

When the target cluster is Premium:

- `x-deploy-type: premium`
- `x-cluster-id: <resource_pool_id>` where `resource_pool_id = parent_id`
- `x-org-id: <tenantID>`
- omit `x-project-id`
- keep `x-provider` / `x-region` from cluster metadata as usual

Do not copy `nextgen-host` into `x-deploy-type`, and do not pass the Premium child `clusterID` into `x-cluster-id`.

If the request unexpectedly returns empty data or auth-like errors for a Premium cluster, check these four pitfalls first before investigating anything else.

## 1. List profile groups in a time window

```
GET /ngm/api/v1/continuous_profiling/group_profiles?begin_time={unix}&end_time={unix}
```

| Field | Description |
|-------|-------------|
| `ts` | Unix seconds — snapshot start key for detail/download |
| `profile_duration_secs` | Sample window length (often 10) |
| `state` | e.g. `finished` |
| `component_num` | Counts of pd/tidb/tikv/tiflash/ticdc with profiles in that group |

```bash
# Example: last hour (Unix seconds)
END=$(date +%s)
BEGIN=$((END - 3600))

curl -s -G "${CLINIC_ENDPOINT}/ngm/api/v1/continuous_profiling/group_profiles" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -H "x-cluster-id: ${CLUSTER_ID}" \
  -H "x-org-id: ${TENANT_ID}" \
  -H "x-project-id: ${PROJECT_ID}" \
  -H "x-deploy-type: ${DEPLOY_TYPE}" \
  -H "x-provider: ${CLOUD_PROVIDER}" \
  -H "x-region: ${REGION}" \
  --data-urlencode "begin_time=${BEGIN}" \
  --data-urlencode "end_time=${END}" | jq
```

Pick a row with `state == "finished"` and note `ts` (or the newest `ts` overlapping the incident).

### Premium example: use raw curl, not helpers

```bash
source scripts/clinic-shell.sh
get_clinic_auth

PREMIUM_CLUSTER_ID="10324983984131567830"
CLUSTER_JSON="$(clinic_get_cluster_by_id "${PREMIUM_CLUSTER_ID}")"
TENANT_ID="$(printf '%s' "${CLUSTER_JSON}" | jq -r '.orgId')"
PARENT_ID="$(printf '%s' "${CLUSTER_JSON}" | jq -r '.diagnosticContext.parentId')"
PROVIDER="$(printf '%s' "${CLUSTER_JSON}" | jq -r '.provider')"
REGION="$(printf '%s' "${CLUSTER_JSON}" | jq -r '.region')"

END=$(date +%s)
BEGIN=$((END - 3600))

curl -s -G "${CLINIC_ENDPOINT}/ngm/api/v1/continuous_profiling/group_profiles" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -H "x-cluster-id: ${PARENT_ID}" \
  -H "x-org-id: ${TENANT_ID}" \
  -H "x-deploy-type: premium" \
  -H "x-provider: ${PROVIDER}" \
  -H "x-region: ${REGION}" \
  --data-urlencode "begin_time=${BEGIN}" \
  --data-urlencode "end_time=${END}" | jq
```

## 2. List per-target profile types for one snapshot

```
GET /ngm/api/v1/continuous_profiling/group_profile/detail?ts={ts}
```

Response includes `target_profiles[]`: each entry has `profile_type` (`cpu`, `heap`, `mutex`, `goroutine`), `state`, `error`, and `target.component` / `target.address`.

- Filter by `component` (`pd`, `tidb`, `tikv`, …) and `profile_type`.
- Skip entries where `state != "finished"` or `error` is non-empty; surface failures to the user.

## 3. Obtain a short-lived download token

```
GET /ngm/api/v1/continuous_profiling/action_token?q={encoded_query}
```

`q` is a **single URL-encoded** string containing **ampersand-separated** key/value pairs (encode `&` as `%26` when building the query component of the URL, or use curl `--data-urlencode` on `q`).

**Plaintext form of `q`** (before encoding):

```
ts={unix}&profile_type={cpu|heap|mutex|goroutine}&component={pd|tidb|tikv|...}&address={pod_or_host_id}&data_format=protobuf
```

- `address` must match `target.address` from detail exactly.
- Response body is **raw JWT** (text), not JSON—use it as `token` in step 4.

```bash
# Build q and request token (curl encodes q)
Q="ts=${TS}&profile_type=cpu&component=tidb&address=${ADDRESS}&data_format=protobuf"
curl -s -G "${CLINIC_ENDPOINT}/ngm/api/v1/continuous_profiling/action_token" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -H "x-cluster-id: ${CLUSTER_ID}" \
  -H "x-org-id: ${TENANT_ID}" \
  -H "x-project-id: ${PROJECT_ID}" \
  -H "x-deploy-type: ${DEPLOY_TYPE}" \
  -H "x-provider: ${CLOUD_PROVIDER}" \
  -H "x-region: ${REGION}" \
  --data-urlencode "q=${Q}"
```

Premium uses the same endpoint with the Premium header mapping above.

## 4. Download the profile blob

```
GET /ngm/api/v1/continuous_profiling/download?token={jwt}
```

Same NGM **x-*** headers as above. Response is often a **ZIP** archive containing a `*.proto` profile and a small `README.md` — unzip and run `go tool pprof` on the inner `*.proto` (raw single-file protobuf may also occur; if `pprof` errors, `file` the download).

```bash
TOKEN="$( ... step 3, strip newlines ... )"
curl -s -L "${CLINIC_ENDPOINT}/ngm/api/v1/continuous_profiling/download?token=${TOKEN}" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -o "pprof-${TS}-${COMPONENT}-${PROFILE_TYPE}.pb"
```

If the file is gzip, `go tool pprof` often accepts it directly.

## Agent playbook: download → analyze → report

1. **Align time**: Convert incident window to Unix seconds; call `group_profiles`; choose `ts` with `finished` and non-zero relevant `component_num`.
2. **Scope targets**: Call `group_profile/detail` for that `ts`. Prefer the TiDB/PD/TiKV instance matching metrics or slow-query `instance` when known; otherwise sample one hot node per component.
3. **Download**: For each `(profile_type, component, address)` triplet of interest, get `action_token` then `download`.
4. **Analyze locally** (requires Go toolchain: `go tool pprof`):

| Profile | Commands (examples) | What to look for |
|---------|---------------------|------------------|
| **cpu** | Interactive: run `go tool pprof ./x.pb` then `top 40 -cum`. Scripted: see **After download** below. Optional `go tool pprof -http=:0 ./x.pb` | Hot stacks in `executor`, `coprocessor`, parser, DDL, client-go; high `runtime`/`GC` → allocation or GC pressure; PD: `etcd`/`grpc` loops |
| **heap** | Interactive: `sample_index=inuse_space` then `top`. Scripted: pipe `sample_index=inuse_space` + `top` + `quit` | Large `inuse_space` in session/cache structures, chunk memory, or single packages → memory leak or oversized working set |
| **mutex** | Same pattern; `top` on mutex profile | Contention at `sync.(*Mutex).Lock`, channel ops, or TiDB/TiKV client paths → lock scaling or serialized critical sections |
| **goroutine** | Same pattern; `top` on goroutine profile | Huge counts on one stack (wait, sleep, lock wait) → goroutine leaks, stuck DDL/drain, or thundering herd |

5. **Synthesize**: Tie findings to symptoms (latency, OOM risk, CPU saturation). Recommend follow-ups: metrics (`metrics/SUBSKILL.md`), slow query (`data-proxy/SUBSKILL.md`), Loki (`loki/SUBSKILL.md`), or TiKV memory / perf skills when the profile points to storage.

**Note**: Token `exp` is short; download immediately. Do not paste JWTs or raw profiles into public channels.

## After download: scripted / non-interactive analysis

The skill does **not** parse profile bytes by itself. **Automation** means: shell (or CI) runs **`go tool pprof`** with **no TTY**, then you or an agent read the **text output**.

**Prerequisites**: [Go](https://go.dev/dl/) installed so `go tool pprof` exists; for ZIP downloads, `unzip`.

### 1) Resolve the profile file

```bash
# If download is a ZIP (typical from Clinic):
unzip -l "${OUT_ZIP}"
PROTO_IN_ZIP=$(unzip -Z1 "${OUT_ZIP}" | grep '\.proto$' | head -1)
unzip -p "${OUT_ZIP}" "${PROTO_IN_ZIP}" > /tmp/conprof.pb
PROFILE=/tmp/conprof.pb

# If download is already a single .proto / .pb / gzip, point PROFILE at that path.
```

### 2) Non-interactive `pprof` commands (pipe)

`go tool pprof` is interactive by default; feed it **one-line commands** on stdin, ending with **`quit`**:

```bash
# CPU (cumulative top 30)
printf 'top 30 -cum\nquit\n' | go tool pprof "${PROFILE}"

# Heap — inuse space (adjust sample index if pprof prompts for options)
printf 'sample_index=inuse_space\ntop 30\nquit\n' | go tool pprof "${PROFILE}"

# Mutex
printf 'top 30\nquit\n' | go tool pprof "${PROFILE}"

# Goroutine
printf 'top 30\nquit\n' | go tool pprof "${PROFILE}"
```

Redirect to a file for logs or agent ingestion: `... | tee conprof-cpu-top.txt`.

### 3) Optional: web UI (manual, not headless)

```bash
go tool pprof -http=:0 "${PROFILE}"
```

### 4) Agent / CI pattern

1. `clinic_continuous_profiling_fetch` → `${OUT_ZIP}`
2. Shell: unzip → pick `*.proto` → `printf 'top 30 -cum\nquit\n' | go tool pprof` → attach **text summary** to the incident (not raw JWT or full profile in public channels).

## Example

```bash
source ~/.claude/skills/clinic-api/scripts/clinic-shell.sh
get_clinic_auth

CLUSTER_ID="10324983984131567830"
END=$(date +%s)
BEGIN=$((END - 3600))

clinic_continuous_profiling_group_profiles "${CLUSTER_ID}" "${BEGIN}" "${END}" | jq

TS=1775121240
clinic_continuous_profiling_detail "${CLUSTER_ID}" "${TS}" | jq '.target_profiles[] | select(.target.component=="tidb" and .profile_type=="cpu")'

ADDR="db-tidb-0"
OUT="/tmp/tidb-cpu-${TS}.zip"
clinic_continuous_profiling_fetch "${CLUSTER_ID}" "${TS}" "cpu" "tidb" "${ADDR}" "${OUT}"
unzip -l "${OUT}"
PROTO=$(unzip -Z1 "${OUT}" | grep '\.proto$' | head -1)
unzip -p "${OUT}" "${PROTO}" > "/tmp/tidb-cpu-${TS}.pb"
printf 'top 25 -cum\nquit\n' | go tool pprof "/tmp/tidb-cpu-${TS}.pb"
```