---
name: clinic-diagnostic-files
description: |
  Download NGM diagnostic artifacts from Clinic: plan_replayer ZIPs and OOM records (Dedicated tier).
  List via plan_replayer/list and oom_record/list; download via oom_record/files (default dir: CLINIC_DIAGNOSTIC_DOWNLOAD_DIR or $PWD at download time).
  Offline: scan plan_replayer zip contents; oom_record goroutine*/running_sql* text and heap* zip for go tool pprof.
---

# Diagnostic files (plan_replayer, oom_record)

Clinic exposes **Next-Gen Monitoring (NGM)** HTTP APIs under `https://clinic.pingcap.com/ngm/api/v1/` for **diagnostic file** bundles collected from the cluster (TiDB plan replayer dumps, OOM-related captures, etc.).

**Supported in this sub-skill (documented / tested path):** **Dedicated** clusters only — `plan_replayer` and `oom_record` **list + download**. Other deploy types may return empty lists or errors; do not assume parity with Premium or shared tiers until documented.

## Authentication and headers

Same pattern as **`continuous-profiling`**: internal API key plus **cluster placement** headers.

| Header | Source | Notes |
|--------|--------|-------|
| `Authorization` | `Bearer ${CLINIC_API_KEY}` | Required |
| `x-cluster-id` | Cluster `clusterID` | User-facing cluster id |
| `x-org-id` | List/Get cluster **`tenantID`** | NGM uses **tenant** id, not internal REST `orgID` |
| `x-project-id` | `projectID` / `projectId` | From list row or Get Cluster |
| `x-deploy-type` | **`clusterDeployType`** | Lowercase, e.g. `dedicated` — **not** `clusterDeployTypeV2` |
| `x-provider` | e.g. `aws`, `gcp`, `alicloud` | From cluster metadata |
| `x-region` | e.g. `us-east-1` | From cluster `region` |

**Agent workflow**: `clinic_search_cluster_by_id` → exact `clusterID` → **`tenantID`** → `projectID` → **`clusterDeployType`** → provider / region → use on every request below.

Shell helpers: `clinic_diagnostic_list_plan_replayer`, `clinic_diagnostic_list_oom_record`, `clinic_diagnostic_download_file` in `diagnostic-files/scripts/clinic-diagnostic-files.sh` (sourced by `scripts/clinic-shell.sh`).

## 1. List plan_replayer files in a time window

```
GET /ngm/api/v1/plan_replayer/list?start={unix_sec}&end={unix_sec}
```

Response shape typically includes `records[]` with `instance` and `files[]`; each file has `filename`, `key`, optional `download_url` / `download_path`, `size`, `created_at_unix_sec`.

```bash
END=$(date +%s)
START=$((END - 86400))

curl -s -G "https://clinic.pingcap.com/ngm/api/v1/plan_replayer/list" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -H "x-cluster-id: ${CLUSTER_ID}" \
  -H "x-org-id: ${TENANT_ID}" \
  -H "x-project-id: ${PROJECT_ID}" \
  -H "x-deploy-type: ${DEPLOY_TYPE}" \
  -H "x-provider: ${CLOUD_PROVIDER}" \
  -H "x-region: ${REGION}" \
  --data-urlencode "start=${START}" \
  --data-urlencode "end=${END}" | jq
```

## 2. List oom_record files in a time window

```
GET /ngm/api/v1/oom_record/list?start={unix_sec}&end={unix_sec}
```

Same header set as §1. Response structure mirrors plan_replayer-style `records` / `files` (exact fields depend on backend).

## 3. Download a file by storage key

Both **plan_replayer** and **oom_record** artifacts are fetched through the **same** download route (relative paths in list responses often point here):

```
GET /ngm/api/v1/oom_record/files?provider={cloud}&region={region}&key={object_key}
```

| Query param | Value |
|-------------|--------|
| `provider` | Same as `x-provider` (e.g. `aws`) |
| `region` | Same as `x-region` (e.g. `us-east-1`) |
| `key` | **`key`** from the list response `files[]` entry (URL-encoded when passed as a query value) |

Use **the same** NGM headers as list calls (`x-cluster-id`, `x-org-id`, `x-project-id`, `x-deploy-type`, `x-provider`, `x-region`).

```bash
# key from JSON: files[].key — pass raw string to curl --data-urlencode
curl -s -L -G "https://clinic.pingcap.com/ngm/api/v1/oom_record/files" \
  -H "Authorization: Bearer ${CLINIC_API_KEY}" \
  -H "x-cluster-id: ${CLUSTER_ID}" \
  -H "x-org-id: ${TENANT_ID}" \
  -H "x-project-id: ${PROJECT_ID}" \
  -H "x-deploy-type: ${DEPLOY_TYPE}" \
  -H "x-provider: ${CLOUD_PROVIDER}" \
  -H "x-region: ${REGION}" \
  --data-urlencode "provider=${CLOUD_PROVIDER}" \
  --data-urlencode "region=${REGION}" \
  --data-urlencode "key=${KEY}" \
  -o "$(pwd)/$(basename "${KEY}")"
```

For raw `curl`, **`cd`** to the desired directory first (or set **`CLINIC_DIAGNOSTIC_DOWNLOAD_DIR`** when using the helper). The **`clinic_diagnostic_download_file`** helper defaults to **`$PWD`** unless that variable is set (see below).

**Note:** List payloads may expose `download_url` as a **relative** path (e.g. `/ngm/api/v1/oom_record/files?...`). Prepend `https://clinic.pingcap.com` and still send `Authorization` + **x-*** headers, or call `clinic_diagnostic_download_file` which resolves context from `cluster_id` and `key`.

## Shell helpers (bash / zsh)

Set **`CLINIC_DIAGNOSTIC_DOWNLOAD_DIR`** when you want downloads outside the current **`$PWD`** (recommended for agents):

```bash
export CLINIC_DIAGNOSTIC_DOWNLOAD_DIR="/path/to/out"
source /path/to/clinic-api/scripts/clinic-shell.sh
get_clinic_auth
```

```bash
# Unix seconds window
clinic_diagnostic_list_plan_replayer "$CLUSTER_ID" "$START" "$END" | jq
clinic_diagnostic_list_oom_record "$CLUSTER_ID" "$START" "$END" | jq

# Download using files[].key — third argument optional
clinic_diagnostic_download_file "$CLUSTER_ID" "$KEY"
```

**Default output path** when the third argument is omitted:

| Priority | Directory |
|----------|-----------|
| 1 | **`CLINIC_DIAGNOSTIC_DOWNLOAD_DIR`** — if set when **`clinic_diagnostic_download_file`** runs |
| 2 | **`$PWD`** at download time, it almost is a tmp dir |

Filename is always **`basename(key)`** (e.g. `replayer_….zip`, `running_sql…`). The helper prints **`clinic_diagnostic_download_file: wrote <path>`** to stderr.

```bash
# Explicit path (optional)
clinic_diagnostic_download_file "$CLUSTER_ID" "$KEY" "/tmp/artifact.zip"
```

If `clusterDeployType` is not `dedicated`, helpers **stderr-warn** but still issue the request (for debugging); prefer Dedicated for supported use cases.

## Offline analysis (no TiDB import)

Many workflows **cannot** run `PLAN REPLAYER LOAD` on a live TiDB. You can still **inspect downloaded artifacts locally**: enumerate files, read text bundles, and run **`go tool pprof`** on heap profiles. Summarize findings (suspicious stacks, hot SQL, allocation hotspots) without pasting customer SQL verbatim into public channels.

### plan_replayer (single `.zip`)

- One **ZIP** per token (e.g. `replayer_….zip`). Unzip to a directory named like `replayer_…_<unix>`.
- **Scan for problems** (non-exhaustive):
  - **`errors.txt`** (if present): export/loader warnings.
  - Any **EXPLAIN** / plan / optimizer text: compare operator rows, index use, estimates vs hints in the text.
  - Schema/stats metadata files: confirm tables/indexes referenced by the incident SQL exist in the bundle.
- This is **static** review; full plan parity still requires TiDB + `PLAN REPLAYER LOAD` when allowed ([PLAN REPLAYER](https://docs.pingcap.com/tidb/stable/sql-plan-replayer)).

```bash
unzip -l replayer_….zip
unzip -q replayer_….zip -d ./pr-unpack && find ./pr-unpack -type f | head -50
# optional: rg -n 'error|ERROR|fail' ./pr-unpack
```

### oom_record (often **multiple** files per capture)

Downloads are **not** always one monolithic story: a capture may include several artifacts with predictable **name prefixes** (exact suffixes vary by version/instance):

| Pattern | Type | What it is | How to analyze |
|---------|------|--------------|----------------|
| `goroutine*` | **Plain text** | Snapshot of **all goroutine stacks** (runtime stack dump) | Search for **blocked** / **chan** / **mutex** / component names (`tidb`, `tikv`, `pd`); look for deep stacks, lock contention, runaway goroutine counts |
| `running_sql*` | **Plain text** | **Currently running SQL** (or session/query lines at capture time) | Identify long runners, missing indexes hinted by predicates, unexpected full scans if the text includes plan snippets |
| `heap*` | **ZIP** (outer) | Inner payload is typically a **Go heap profile** (e.g. `*.pb.gz` / `*.prof` / similar) | Unzip the `heap*` file, then run **`go tool pprof`** on the inner profile |

**Heap (`heap*`) — `go tool pprof`**

```bash
unzip -l heap….zip
unzip -q heap….zip -d ./heap-unpack
file ./heap-unpack/*
# replace PATH with the actual profile file:
go tool pprof -http=:0 ./heap-unpack/<profile-file>
# non-interactive top entries:
go tool pprof -top ./heap-unpack/<profile-file>
```

Use **the same Go major version** as the producing binary when possible (TiDB/TiKV/Pd use Go toolchain); if `go tool pprof` errors on format, note the profile extension and try `go version` / another toolchain per PingCAP release notes.

**Quick inventory**

```bash
find . -maxdepth 3 \( -name 'goroutine*' -o -name 'running_sql*' -o -name 'heap*' \) -print
```

### Agent checklist (import-free)

1. Classify artifact: **plan_replayer** zip vs **oom_record** bundle(s).
2. Text: sample line counts / `head` / targeted `rg` for errors and hot keywords; **do not** dump full customer SQL into logs.
3. Heap: unzip `heap*` → `go tool pprof -top` or `-http` for allocation-heavy functions.
4. Goroutines: scan stacks for deadlocks, stuck IO, or known component symbols tied to the incident.

## Privacy / agent notes

- Do not paste full API keys or customer-identifying bundle contents into chat; summarize file names, sizes, and time ranges only.
- Artifacts are often **ZIP**; inspect locally with `unzip -l` / `file` before sharing paths.
- **oom_record** may contain **SQL text and stacks** — treat as sensitive data.