--- name: clinic-loki description: | Clinic Data Proxy Loki log queries (LogQL), label discovery, and canonical clinic_logs_* aliases while retaining the stable canonical clinic_loki_* helpers. Default triage: panic/fatal → [error] → [warn] → broad query. --- # Loki log query API Query raw TiDB cluster logs and Clinic control-plane system logs using Loki query language. Preferred business-log routes are scoped by TiDB Cloud/DBAAS `orgID` and `clusterID`; preferred system-log routes are scoped by the datasource UID returned by the canonical catalog. ## Clinic Data Proxy: `|~` — use **backticks**, not double quotes On the Clinic Data Proxy log routes, including **`${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/logs/...`**, a regex line filter wrapped in **double quotes** cannot use **`\[` / `\]`** escapes — the server returns: ```text parse error at line 1, col …: invalid char escape ``` **Wrong (do not use on Clinic):** ```logql {container="tidb"} |~ "(?i)\[error\]" {container="tidb"} |~ "(?i)\[warn(ing)?\]" ``` **Right:** delimit the regex with **backticks** (`` ` ``). Then bracket escapes work as usual: ```logql {container="tidb"} |~ `(?i)\[error\]` {container="tidb"} |~ `(?i)\[warn(ing)?\]` {container="tidb"} |~ `(?i)panic|fatal` ``` **Shell / curl:** pass the whole LogQL in **single quotes** so bash leaves inner **backticks** literal (no command substitution). Example: `'... |~ `(?i)\[error\]`'` — see the **Query Loki Logs** bash block below. **Optional:** **`|=`** literals (`|="[ERROR]"`, etc.) still work if you prefer substring match without regex. **`limit`**: the Data Proxy may enforce a **per-query maximum** (e.g. **1000**). Requests with `limit` above the cap return an error like `max entries limit per query exceeded`. Use **`limit` ≤ that cap** unless documentation says otherwise. ## Investigation query order (default) **Unless the user specifies a different filter**, use this **staged** order to limit noise and token size: | Step | When | LogQL pattern (example: TiDB) | Notes | |------|------|--------------------------------|-------| | **1** | Always try first | `{container="tidb"} \|~ \`(?i)panic\|fatal\`` | **Backtick** regex; repeat for `tikv`, `pd`, `tiflash` as needed. | | **2** | After step 1 (or if step 1 empty and you need signals) | `{container="tidb"} \|~ \`(?i)\[error\]\`` | **Error** level tags (`(?i)` covers `[ERROR]` / `[error]`). Use **backticks**, not ` \|~ "(?i)\[error\]"`. Run **before** warn. | | **3** | After step 2, if you still need warnings | `{container="tidb"} \|~ \`(?i)\[warn(ing)?\]\`` | **Warn** level: `[WARN]`, `[warning]`, etc. Prefer this over ` \|~ "(?i)\[warn(ing)?\]"` (double quotes break on Clinic). | | **4** | Last resort / full context | `{container="tidb"}` or `{}` | Whole stream for that component, or all labels (heavy); use **allowed** `limit` only. | **Agent workflow**: 1. Run **step 1** per relevant `container`. 2. Run **step 2** (**error** regex, backticks) on the **same** time window — **before** warn. 3. If you still need warning-level lines, run **step 3** (**warn(ing)?** regex, backticks). 4. Only then run **step 4**; prefer `{container="..."}` over `{}` when the component is known. **Copy-paste LogQL — step 2 (error)**: ```logql {container="tidb"} |~ `(?i)\[error\]` ``` **Copy-paste LogQL — step 3 (warn)**: ```logql {container="tidb"} |~ `(?i)\[warn(ing)?\]` ``` Adjust `container` and time range; keep **`limit` within the proxy cap** (often **1000**). ## Large result handling When the target time range produces too many warn/error lines to return directly, do **not** dump raw logs first. Summarize them. **Default behavior for large warn/error result sets**: 1. Group repeated lines into **patterns**. 2. Return only a compact summary table with: - `pattern` - `count` - `first_seen` - `last_seen` - `sample_line` 3. Prefer grouping by the stable error/warn message shape, not by the full raw line with timestamps, IDs, or addresses. 4. Call out the top patterns first, especially `panic`, `fatal`, `oom`, RPC timeout, region/store unavailable, Raft, GC, and allocator pressure messages. If the grouped summary is still too large to present cleanly, **stop and ask the user how to reduce scope** before continuing. Offer options such as: - truncate to top `N` patterns - narrow the time window - split by component or instance - focus on only `error/fatal` or only `warn` Do not silently truncate a very large grouped result unless the user explicitly accepts truncation or has already requested a brief summary. ## Endpoints | Endpoint | Method | Description | |----------|--------|-------------| | `/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/logs/query-range` | GET | Business-log range query | | `/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/logs/labels` | GET | List business-log labels | | `/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/logs/labels/{name}/values` | GET | Get business-log values for a label | | `/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/logs/series` | GET | Discover business-log series; requires `match[]` | | `/data-proxy/api/v1/system-log-datasources` | GET | Catalog system-log datasource UIDs and the default UID | | `/data-proxy/api/v1/system-log-datasources/{datasourceUID}/logs/query-range` | GET | System-log range query | | `/data-proxy/api/v1/system-log-datasources/{datasourceUID}/logs/labels` | GET | List system-log labels | | `/data-proxy/api/v1/system-log-datasources/{datasourceUID}/logs/labels/{name}/values` | GET | Get system-log values for a label | | `/data-proxy/api/v1/system-log-datasources/{datasourceUID}/logs/series` | GET | Discover system-log series; requires `match[]` | The canonical-name aliases are `clinic_logs_query_range`, `clinic_logs_labels`, `clinic_logs_label_values`, and `clinic_logs_series`. The stable `clinic_loki_query_range`, `clinic_loki_labels`, `clinic_loki_label_values`, and `clinic_loki_series` helpers are retained and still use the same canonical routes. Both families remain cluster-only at the call site and first resolve canonical `orgID` from metadata. Raw `curl` callers must supply the TiDB Cloud/DBAAS `DBAAS_ORG_ID` themselves. ## BYOC business logs The same canonical business-log routes and `clinic_loki_*` helpers accept a logical BYOC ID or its physical `byoc-host` ID. For BYOC only, Data Proxy sends the request to the authorized customer-side O11y Loki target. It does not copy logs into Clinic storage. Data Proxy enforces both routing labels server-side: - logical BYOC ID: `namespace="tidb{physicalParentID}"` - physical `byoc-host` ID: `namespace="tidb{clusterID}"` - both: the configured `cluster_env` Callers can use ordinary component selectors such as `{container="tidb"}`; do not pass provider/region/deploy headers or placement query parameters, and do not try to override `namespace` or `cluster_env`. A BYOC range expression must contain exactly one LogQL stream selector. Expressions with multiple selector branches are rejected because scoping only one branch would be unsafe. Labels accept at most one selector; series may repeat `match[]`, with each selector scoped independently. ```bash METADATA=$(clinic_get_cluster_by_id "$LOGICAL_CLUSTER_ID") PHYSICAL_CLUSTER_ID=$(printf '%s' "$METADATA" | jq -r '.diagnosticContext.parentId // .parentId // empty') # Both target forms are supported. Start with the logical cluster in normal workflows. clinic_logs_query_range "$LOGICAL_CLUSTER_ID" \ '{container="tidb"} |~ `(?i)\[error\]`' "$START_NS" "$END_NS" 1000 backward clinic_logs_series "$PHYSICAL_CLUSTER_ID" \ '{container="tikv"}' "$START_NS" "$END_NS" ``` An empty valid result can mean the cluster collected no matching lines in that window. Compare the same selector/window in Clinic UI before treating it as a proxy failure. ## System Logs Datasource Selection System logs are not scoped by business `clusterID`. Select a **datasource UID** from the canonical catalog and put it in the request path. Do not send a `datasource` query parameter on canonical requests. When the user does not know which UID to use: 1. Run `clinic_system_logs_datasources_table` to list configured choices in a readable table. 2. Prefer the catalog's `data.default_datasource`; the helpers do this automatically. 3. If the provider and region are known, run `clinic_system_logs_datasource_for_region ` and use the returned UID. 4. Override with `CLINIC_SYSTEM_LOGS_DATASOURCE` or pass the UID explicitly (`clinic_system_logs_query_range` uses the 6th argument; label helpers document their arguments below). The environment variable keeps its historical name, but its value is a UID. The catalog is served by Clinic Data Proxy from its own `system_endpoints` configuration. It is a migrated snapshot of the Grafana Loki datasource set, but the system logs API does **not** call Grafana or expose a Grafana datasource API. The UID is routing metadata and is not forwarded to Loki. ```bash source ~/.claude/skills/platform/clinic-api/scripts/clinic-shell.sh get_clinic_auth # Discover datasources. clinic_system_logs_datasources clinic_system_logs_datasources_table # Use the catalog default, or resolve a UID for a known provider/region. clinic_system_logs_default_datasource DATASOURCE_UID=$(clinic_system_logs_datasource_for_region aws us-west-2) # Optional: change the default for the current shell. export CLINIC_SYSTEM_LOGS_DATASOURCE="${DATASOURCE_UID}" ``` ### Query System Logs ```bash START=$(date -d '6 hours ago' +%s)000000000 END=$(date +%s)000000000 clinic_system_logs_query_range \ '{cluster_env="prod",cluster="prod-clinic",namespace="clinic-prod-cloud",container=~"clinic",stream=~"stderr|stdout"} |~ `test`' \ "${START}" \ "${END}" \ 1000 # Override datasource only when needed: clinic_system_logs_query_range \ '{cluster_env="prod",namespace="clinic-prod-cloud"} |~ `test`' \ "${START}" \ "${END}" \ 1000 \ backward \ "${DATASOURCE_UID}" ``` ### System Logs Label Discovery ```bash clinic_system_logs_labels "${DATASOURCE_UID}" "${START}" "${END}" clinic_system_logs_label_values container "${START}" "${END}" "${DATASOURCE_UID}" clinic_system_logs_series '{namespace="clinic-prod-cloud"}' "${START}" "${END}" "${DATASOURCE_UID}" ``` ## Label Discovery ### List All Labels Get all available labels for the cluster. ```bash curl -G "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/logs/labels" \ -H "Authorization: Bearer ${CLINIC_API_KEY}" \ --data-urlencode 'start=1773125534000000000' \ --data-urlencode 'end=1773132734000000000' ``` | Parameter | Required | Description | |-----------|----------|-------------| | `start` | No | Start timestamp (nanoseconds since Unix epoch) | | `end` | No | End timestamp (nanoseconds since Unix epoch) | Response: ```json { "status": "success", "data": ["container", "instance", "namespace", "stream"] } ``` ### Get Label Values Get all values for a specific label. ```bash curl -G "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/logs/labels/${LABEL_NAME}/values" \ -H "Authorization: Bearer ${CLINIC_API_KEY}" \ --data-urlencode 'start=1773125534000000000' \ --data-urlencode 'end=1773132734000000000' ``` | Parameter | Required | Description | |-----------|----------|-------------| | `start` | No | Start timestamp (nanoseconds since Unix epoch) | | `end` | No | End timestamp (nanoseconds since Unix epoch) | Response: ```json { "status": "success", "data": ["tidb", "tikv", "pd", "tiflash"] } ``` ## Query Range Queries logs over a specified time range. ```bash curl -G "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/logs/query-range" \ -H "Authorization: Bearer ${CLINIC_API_KEY}" \ --data-urlencode 'query={container="tidb"}' \ --data-urlencode 'start=1773125534000000000' \ --data-urlencode 'end=1773132734000000000' \ --data-urlencode 'limit=1000' ``` | Parameter | Required | Description | |-----------|----------|-------------| | `query` | Yes | LogQL query expression (e.g., `{container="tidb"}`, `{container="slowlog"}`) | | `start` | Yes | Start timestamp (nanoseconds since Unix epoch) | | `end` | Yes | End timestamp (nanoseconds since Unix epoch) | | `limit` | No | Max log lines per query; **Clinic Data Proxy often caps at 1000** (higher values may be rejected) | | `direction` | No | Sort direction: `forward` or `backward` (default: `backward`) | | `step` | No | Query resolution step for aggregations (e.g., `1m`, `5m`) | ## Series Discovery ```bash curl -G "${CLINIC_ENDPOINT}/data-proxy/api/v1/orgs/${DBAAS_ORG_ID}/clusters/${CLUSTER_ID}/logs/series" \ -H "Authorization: Bearer ${CLINIC_API_KEY}" \ --data-urlencode 'match[]={container="tidb"}' \ --data-urlencode 'start=1773125534000000000' \ --data-urlencode 'end=1773132734000000000' ``` ## Response Format ### Query Range Response ```json { "status": "success", "data": { "resultType": "streams", "result": [ { "stream": { "container": "tidb", "instance": "tidb-0" }, "values": [ ["1773125534000000000", "2026-01-08T10:05:34.000Z [INFO] server started"], ["1773125535000000000", "2026-01-08T10:05:35.000Z [INFO] connection accepted"] ] } ] } } ``` Each entry in `values` is a tuple: `[timestamp_ns, log_line]` ## Example: Query Loki Logs (default triage) ```bash source ~/.claude/skills/platform/clinic-api/scripts/clinic-shell.sh # (loads `loki/scripts/clinic-loki.sh` among others) get_clinic_auth CLUSTER_ID="10139314816168987482" START=$(date -v-2H +%s)000000000 # macOS: -v-2H; Linux: -d '2 hours ago' END=$(date +%s)000000000 # Step 1: panic / fatal — regex delimited with backticks; entire LogQL in single quotes (backticks literal in bash) clinic_logs_query_range "${CLUSTER_ID}" '{container="tidb"} |~ `(?i)panic|fatal`' "${START}" "${END}" 500 # Step 2: error — use |~ `(?i)\[error\]` not |~ "(?i)\[error\]" (double quotes → invalid char escape on Clinic) clinic_logs_query_range "${CLUSTER_ID}" '{container="tidb"} |~ `(?i)\[error\]`' "${START}" "${END}" 1000 # Step 3: warn — |~ `(?i)\[warn(ing)?\]` clinic_logs_query_range "${CLUSTER_ID}" '{container="tidb"} |~ `(?i)\[warn(ing)?\]`' "${START}" "${END}" 1000 # Step 4: all lines for container (heavy); keep limit ≤ proxy cap # clinic_logs_query_range "${CLUSTER_ID}" '{container="tidb"}' "${START}" "${END}" 1000 ``` The helper resolves `orgID` through canonical cluster metadata before issuing each business-log request. ## Compatibility-only routes Use the canonical routes and helpers above for new calls. The following paths remain only for compatibility: - Business logs: `/data-proxy/loki/{clusterID}/api/v1/query_range`, `/labels`, `/label/{name}/values`, and `/series`. - System logs: `/data-proxy/system-logs/api/v1/datasources`, `/query_range`, `/labels`, `/label/{name}/values`, and `/series`; these historical routes use a `datasource` query parameter. Explicit legacy-route shell helpers use the `clinic_legacy_loki_*` and `clinic_legacy_system_logs_*` prefixes. Stable `clinic_loki_query_range`, `clinic_loki_labels`, `clinic_loki_label_values`, and `clinic_loki_series` do not use legacy routes; the new `clinic_logs_*` aliases call them. Loki instant query has no canonical alias. `clinic_loki_query` therefore delegates to `clinic_legacy_loki_query`, which calls `/data-proxy/loki/{clusterID}/api/v1/query`. Starter/Essential parent logs identified by `control_plane_info` also have no org/cluster canonical alias. Continue to use `/data-proxy/serverless-parent-logs/api/v1/...` only for that parent/control-plane case; do not invent an org-scoped path for it. ---