---
name: clinic-byoc-diagnostics
description: |
  Canonical Data Proxy contracts for BYOC logs, metrics, Slow Query, Top SQL,
  SQL Statements, and read-only Continuous Profiling inventory.
---

# BYOC diagnostics

BYOC diagnostic data remains in the customer's observability environment. Clinic Data Proxy resolves the authorized customer-side O11y target and proxies fixed APIs; callers provide only the canonical TiDB Cloud org and cluster scope.

For BYOC, never create or query S3/Athena/BigQuery/Synapse tables, copy diagnostic payloads into Clinic storage, or add caller-selected placement headers. The warehouse SQL contracts documented in `../data-proxy/SUBSKILL.md` remain valid for their supported non-BYOC tiers, but they are not the BYOC path for Slow Query, Top SQL, or live SQL Statements.

## Resolve the target

```bash
source scripts/clinic-shell.sh
get_clinic_auth

LOGICAL_CLUSTER_ID="..."
METADATA=$(clinic_get_cluster_by_id "$LOGICAL_CLUSTER_ID")
DBAAS_ORG_ID=$(printf '%s' "$METADATA" | jq -r '.orgId')
PHYSICAL_CLUSTER_ID=$(printf '%s' "$METADATA" | jq -r '.diagnosticContext.parentId // .parentId // empty')
```

`orgId` is the TiDB Cloud/DBAAS org ID used in canonical paths. For a logical BYOC cluster, `diagnosticContext.parentId` identifies its physical `byoc-host` cluster when that distinction is needed. Do not substitute the legacy Dashboard database field named `orgID`.

Most BYOC diagnostic routes accept either the logical BYOC cluster ID or its physical `byoc-host` ID. Continuous Profiling is the exception: its canonical inventory routes accept only the physical ID.

Shell helpers live in `byoc-diagnostics/scripts/clinic-byoc-diagnostics.sh` and are loaded by `source scripts/clinic-shell.sh`.

## Support matrix

| Diagnostic data | Logical BYOC ID | Physical `byoc-host` ID | Data path |
|-----------------|-----------------|-------------------------|-----------|
| Metadata/components | Yes, when exposed by the control plane | Yes, when exposed by the control plane | Clinic metadata APIs |
| Logs | Yes | Yes | Customer-side O11y Loki |
| Cluster metrics | Yes | Yes | Customer-side O11y Prometheus |
| Data-plane/infra metrics | Yes; filter with the physical namespace | Yes; filter with the physical namespace | Customer-side O11y Prometheus |
| Slow Query | Yes | Yes | Customer-side O11y diagnosis API |
| Top SQL | Yes | Yes | Customer-side O11y diagnosis API |
| Live SQL Statements | Yes | Yes | Customer-side O11y diagnosis API |
| Continuous Profiling inventory | No; resolve the physical parent first | Yes | Customer-side O11y diagnosis API |
| Continuous Profiling token/download/view/config | No canonical support | No canonical support | Use no inferred endpoint |
| Aggregated LogStats | No canonical BYOC support | No canonical BYOC support | Raw Logs are supported; warehouse LogStats remains non-BYOC |
| Plan Replayer / OOM diagnostic files | No canonical BYOC support | No canonical BYOC support | Existing NGM file workflow is Dedicated-only |
| Top RU | Not part of the BYOC contract | Not part of the BYOC contract | See `../data-proxy/SUBSKILL.md` for supported non-BYOC tiers |

“Supported” means the route and routing contract are implemented. It does not guarantee that a particular cluster has collected data for the requested window; an empty valid response can be correct.

## Common request rules

- Use `GET /data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/...` and a Bearer `CLINIC_API_KEY`.
- Both IDs are validated against authoritative metadata. Do not pass provider, region, O11y URL, data source, project, deploy type, or routing headers.
- Do not send a request body on these canonical GET routes.
- Slow Query, Top SQL, Statements, and Continuous Profiling cap the encoded query at 16 KiB. Canonical logs/metrics cap GET query strings or form bodies at 64 KiB.
- Time parameters are Unix seconds except Loki business-log timestamps, which follow Loki's nanosecond timestamp contract.
- Query only the incident window needed. Slow Query and Statements allow at most 259500 seconds; Top SQL allows at most 15 days; Continuous Profiling allows at most a minute-rounded 3 hours.

## Python helpers

`scripts/clinic_api.py` exposes the same canonical routes without caller-selected placement:

| Area | `ClinicAPI` methods |
|------|---------------------|
| Logs | `logs_query_range`, `logs_labels`, `logs_label_values`, `logs_series` |
| Metrics | `query_instant`, `query_range`, `query_series`, `byoc_data_plane_namespace` |
| Slow Query | `byoc_slow_query_available_fields`, `byoc_slow_query_list`, `byoc_slow_query_detail` |
| Top SQL | `byoc_top_sql_instances`, `byoc_top_sql_summary` |
| Statements | `byoc_statements_available_fields`, `byoc_statements_statement_types`, `byoc_statements_list`, `byoc_statements_plans`, `byoc_statements_plan_detail` |
| Continuous Profiling | `byoc_continuous_profiling_group_profiles`, `byoc_continuous_profiling_group_detail` |

```python
from clinic_api import ClinicAPI

api = ClinicAPI()
rows = api.byoc_slow_query_list(
    "LOGICAL_BYOC_CLUSTER_ID", START, END, limit=20, order_by="query_time"
)
namespace = api.byoc_data_plane_namespace("LOGICAL_BYOC_CLUSTER_ID")
pods = api.query_range(
    "LOGICAL_BYOC_CLUSTER_ID",
    f'kube_pod_info{{namespace="{namespace}"}}',
    START,
    END,
)
```

## Logs

The existing canonical Loki helpers work for both BYOC IDs:

```bash
START_NS="$(( $(date +%s) - 3600 ))000000000"
END_NS="$(date +%s)000000000"

clinic_logs_query_range "$LOGICAL_CLUSTER_ID" \
  '{container="tidb"} |~ `(?i)panic|fatal`' \
  "$START_NS" "$END_NS" 500 backward
clinic_logs_labels "$LOGICAL_CLUSTER_ID" "$START_NS" "$END_NS"
clinic_logs_label_values "$LOGICAL_CLUSTER_ID" container "$START_NS" "$END_NS"
clinic_logs_series "$LOGICAL_CLUSTER_ID" '{container="tidb"}' "$START_NS" "$END_NS"
```

Routes:

- `GET .../logs/query-range`
- `GET .../logs/labels`
- `GET .../logs/labels/{name}/values`
- `GET .../logs/series`

For BYOC, Data Proxy enforces `cluster_env` and the physical namespace (`tidb{parentID}` for a logical ID, `tidb{clusterID}` for a physical ID). A range query must contain exactly one LogQL stream selector. Do not try to override those labels or send placement query parameters. See `../loki/SUBSKILL.md` for parameters and safe LogQL quoting.

## Metrics

The existing metrics helpers also accept both BYOC IDs:

```bash
# Cluster-store metric: Data Proxy keeps BYOC cluster scoping server-side.
clinic_metrics_query_range "$LOGICAL_CLUSTER_ID" \
  'sum(rate(tidb_server_query_total[1m])) by (type)' \
  "$START" "$END" 60

# Data-plane metric: use the physical Kubernetes namespace explicitly.
clinic_metrics_query_range "$LOGICAL_CLUSTER_ID" \
  "kube_pod_info{namespace=\"tidb${PHYSICAL_CLUSTER_ID}\"}" \
  "$START" "$END" 60
```

Routes are `GET/POST .../metrics/query`, `.../metrics/query-range`, and `.../metrics/series`.

Exact BYOC data-plane queries do **not** receive an injected `tidb_cluster_id` matcher. Kubernetes and node metrics such as `kube_pod_info` and `node_*` do not carry that label; adding it makes the same query visible in Grafana return empty. Scope data-plane queries with the physical namespace and the metric's real labels. BYOC cluster-store metrics retain their established cluster scoping, and this exception does not change Premium or any other tier. See `../metrics/SUBSKILL.md`.

## Slow Query

Routes and helpers:

| Operation | Canonical route | Shell helper |
|-----------|-----------------|--------------|
| Available fields | `GET .../slow-queries/available-fields` | `clinic_byoc_slow_query_available_fields` |
| List | `GET .../slow-queries` | `clinic_byoc_slow_query_list` |
| Detail | `GET .../slow-queries/detail` | `clinic_byoc_slow_query_detail` |

List requires `begin_time` and `end_time`. Optional parameters are `fields`, repeated `db`, `limit` (1-1000), `text`, `orderBy` (`timestamp`, `query_time`, `memory_max`, `total_keys`, or `process_keys`), `desc`, repeated `plans`, `digest`, and `show_internal`.

```bash
clinic_byoc_slow_query_available_fields "$LOGICAL_CLUSTER_ID"
clinic_byoc_slow_query_list "$LOGICAL_CLUSTER_ID" "$START" "$END" \
  20 query_time true 'query,timestamp,query_time,memory_max,digest,connection_id'
```

The shell helper's final optional `db_csv` and `plans_csv` arguments expand to repeated `db` and `plans` query parameters. They are a shell convenience, not CSV on the HTTP wire. If a value itself contains a comma, use the Python list helper (`urlencode(..., doseq=True)`) instead of the shell CSV form.

Detail requires `digest`, a finite positive `timestamp`, and unsigned decimal `connect_id`; `show_internal` is optional. List rows expose the composite-key field as `connection_id`, while the detail request parameter is named `connect_id`:

```bash
clinic_byoc_slow_query_detail "$LOGICAL_CLUSTER_ID" \
  "$DIGEST" "$TIMESTAMP" "$CONNECTION_ID" false
```

When `fields` is supplied, the proxy projects the response to those fields plus the detail key fields. Use `available-fields` instead of assuming a field exists on every cluster/version.

## Top SQL

| Operation | Canonical route | Shell helper |
|-----------|-----------------|--------------|
| Instances | `GET .../top-sql/instances` | `clinic_byoc_top_sql_instances` |
| Summary | `GET .../top-sql/summary` | `clinic_byoc_top_sql_summary` |

Instances requires decimal Unix-second `start` and `end`. Use one returned instance in the summary request, which additionally requires `window`, `top` (1-100), and `instance_type` (`tidb`, `tikv`, or `pd`). The range is at most 15 days and the window must be a positive whole-second Go duration that does not produce more than 4096 buckets.

```bash
clinic_byoc_top_sql_instances "$LOGICAL_CLUSTER_ID" "$START" "$END"
clinic_byoc_top_sql_summary "$LOGICAL_CLUSTER_ID" \
  "$START" "$END" 1m 20 "$INSTANCE" tidb
```

Do not send `data_source=deltalake` or query the warehouse `topsql` table for BYOC. The response is the validated O11y `{status,data}` envelope; `data` can legitimately be empty.

## Live SQL Statements

These routes expose customer-side live Statement Summary data. They are separate from the persisted, warehouse-backed Statements SQL contract used by supported Premium variants.

| Operation | Canonical route | Shell helper |
|-----------|-----------------|--------------|
| Available fields | `GET .../statements/available-fields` | `clinic_byoc_statements_available_fields` |
| Statement types | `GET .../statements/statement-types` | `clinic_byoc_statements_statement_types` |
| List | `GET .../statements` | `clinic_byoc_statements_list` |
| Plans | `GET .../statements/plans` | `clinic_byoc_statements_plans` |
| Plan detail | `GET .../statements/plan-detail` | `clinic_byoc_statements_plan_detail` |

List requires `begin_time` and `end_time`. Optional parameters are repeated `schemas`, repeated `stmt_types`, `text`, and comma-separated `fields`. Discover valid fields and statement types before filtering.

Plans requires `begin_time`, `end_time`, and a 64-character hexadecimal SQL `digest`; `schema_name` is optional. Plan detail accepts the same parameters plus optional repeated `plans` values, each a 64-character hexadecimal plan digest or `N/A`.

```bash
clinic_byoc_statements_available_fields "$LOGICAL_CLUSTER_ID"
clinic_byoc_statements_statement_types "$LOGICAL_CLUSTER_ID"
clinic_byoc_statements_list "$LOGICAL_CLUSTER_ID" "$START" "$END"
clinic_byoc_statements_plans "$LOGICAL_CLUSTER_ID" \
  "$START" "$END" "$SQL_DIGEST" "$SCHEMA_NAME"
clinic_byoc_statements_plan_detail "$LOGICAL_CLUSTER_ID" \
  "$START" "$END" "$SQL_DIGEST" "$PLAN_DIGEST" "$SCHEMA_NAME"
```

For shell calls, `schemas_csv`, `stmt_types_csv`, and `plans_csv` expand to repeated query parameters; CSV is not sent on the HTTP wire. Pass an empty placeholder when a later optional positional argument is needed without an earlier one. If a value itself contains a comma, use the Python list helper (`doseq=True`) instead.

List and plans normalize an upstream JSON `null` collection to `[]`. Plan detail returns an object; no matching data is `{}`. When an encoded binary plan is present, `binary_plan_text` is best-effort enrichment and may be absent if decoding fails.

## Continuous Profiling inventory

Canonical BYOC Continuous Profiling is a read-only inventory and accepts only the physical `byoc-host` ID.

| Operation | Canonical route | Shell helper |
|-----------|-----------------|--------------|
| Group list | `GET .../continuous-profiling/group-profiles` | `clinic_byoc_continuous_profiling_group_profiles` |
| Group detail | `GET .../continuous-profiling/group-profiles/{ts}` | `clinic_byoc_continuous_profiling_group_detail` |

```bash
clinic_byoc_continuous_profiling_group_profiles \
  "$PHYSICAL_CLUSTER_ID" "$START" "$END"
clinic_byoc_continuous_profiling_group_detail \
  "$PHYSICAL_CLUSTER_ID" "$PROFILE_TS"
```

The list requires positive Unix-second `begin_time` and `end_time`; after minute flooring, the span must not exceed three hours. An upstream `null` list becomes `[]`. Detail uses a positive Unix-second path `ts` and may return JSON `null` when no group exists.

List entries include `ts`, `profile_duration_secs`, `state`, and per-component counts in `component_num`. Detail adds `target_profiles[]`, whose entries identify `profile_type`, `state`, `error`, and `target.component` / `target.address`. These fields describe inventory only; they are not profile bytes.

These canonical routes do not expose action tokens, raw profile bytes, download, view, or config. Do not infer those routes from the historical NGM API. The NGM Dedicated/Premium workflow remains separately documented in `../continuous-profiling/SUBSKILL.md`.

## Interpreting empty and error responses

- Empty `[]`, `{status,data:[]}`, `{}`, or JSON `null` has the operation-specific meanings described above; it is not proof of a proxy failure.
- Compare the same window and grouping in Clinic UI/Grafana when deciding whether empty metrics or inventory are unexpected.
- `400` commonly means an unsupported tier/operation or invalid strict parameters. It does not authorize falling back to S3/Athena for BYOC.
- `404` can mean the cluster or requested data is unavailable. `429`, `503`, and `504` are transient target/rate/timeout signals; retry with a narrower window rather than changing placement.
- Never paste access keys, customer SQL/log bodies, O11y URLs, routing headers, or raw profile bytes into reports.