---
name: clinic-api
description: Use the Agent9 managed Auto Oncall Clinic Tool for bounded, read-only TiDB Cloud metadata, component, and Prometheus evidence. Use only for an ST or TCID workspace whose diagnostic manifest selects Clinic and whose runner-owned scenario context contains an exact orgId and resolved cluster target set.
---

# Clinic evidence through Agent9

Use the separately installed `evidence-query-planning` Skill for shared metric
scope, source-cardinality, and selector-provenance requirements. This Provider
Skill owns supported operations, actual label mappings, and transport only in
that planning boundary; its existing product-capability guidance still applies.

Use one bounded `managed_auto_oncall_clinic` batch call per acquisition round as
the only Clinic entrypoint. It runs in a
Tool Sandbox separate from the Agent Sandbox. Agent9 injects the Clinic secret
only into that Tool Sandbox; never inspect credentials or contact Clinic with
shell, curl, Python, browser, or another Tool.

The provider request script remains inside the managed Tool package and reads
`CLINIC_API_KEY` from its injected environment. The Skill package contains no
credential value and does not execute that provider script in the Agent Sandbox.

## Entry gate

1. Treat the runner-owned immutable context supplied in the current Evidence
   Agent Turn as binding. When its projected file representation is needed,
   read only `context/clinic-metrics-context.json`; parent-run `input/` files
   are outside this role's workspace and are not an additional gate.
2. Continue only when the current Turn selects Clinic, the issue key is
   `ST-*` or `TCID-*`, and the runner-owned scenario context contains an exact
   `orgId` plus one to three resolved cluster IDs.
3. Apply the same bounded plan to each supplied target and pass every value
   unchanged. Never discover, add, remove, or substitute an organization,
   cluster, tenant, account, or similarly named resource.
4. The runner owns direct-artifact and Provider admission before this Turn. Do
   not search a parent workspace to repeat those checks.

## Managed operations

In round 1, reuse current-run, runner-supplied Clinic metadata only when it
belongs to the exact supplied organization and cluster and identifies a
recognized service tier. Do not repeat `cluster_detail` merely to rediscover
that tier. For a target without that metadata, first call `cluster_detail`
once in a metadata-only batch; an absent or unrecognized tier is not proof of
Dedicated capability. When the supplied metadata or that result says the tier is `Starter`,
`Essential`, or `shared`, apply this product boundary before planning: Clinic
does not expose physical component topology, TopSQL, persisted Statements,
dedicated-cluster configuration, or physical-node `node_*` Prometheus metrics
for that target.
Do not request those surfaces. Slow Query remains available through NGM, with
`orderBy` limited to `query_time`, `memory_max`, or `total_keys`; use relevant
supported logical-cluster metrics, NGM Slow Query, logs, and activities. An
expected unavailable surface is not an evidence failure and must not cause a
retry, refinement, customer request, or human handoff by itself.

- `cluster_detail`: bounded non-commercial cluster metadata.
- `components`: sanitized component counts only.
- `configuration`: 1 to 16 exact non-secret PD, TiKV, TiDB, or TiDB-variable
  keys for one resolved cluster. Put every requested key in one `configKeys`
  array inside one `queries` entry; do not split keys into separate Tool calls.
  Each selector is an object with exactly `component` and `key`, for example
  `{"component":"pd","key":"schedule.max-snapshot-count"}`. Never pass
  strings or include the component name in `key`. The Tool validates canonical
  org ownership, queries Clinic Dashboard first with both `x-org-id` and
  `x-cluster-id`, and
  uses DevOps CloudConfig only for rejected or missing keys. It never omits the
  org header, returns a full configuration payload, or mutates configuration.
  When CloudConfig supplies a missing value, an explicit current override takes
  precedence over the selected cluster version's default. Use this operation
  once for a current-value request; do not enumerate Prometheus metric families
  or query activity history to rediscover configuration. Never call, plan, or
  authenticate to CloudConfig directly: Clinic-to-CloudConfig fallback is one
  deterministic execution inside this helper, and its fallback provenance does
  not require another Agent call.
- `metrics_query`: one Agent-authored PromQL expression at one bounded time.
- `metrics_query_range`: one Agent-authored PromQL expression over one bounded
  range. It may use a runner-authorized multi-day trend up to 30 days. Start
  with the narrowest incident and baseline window that answers the question,
  normally no more than 24 hours, and expand beyond that only when the evidence
  objective explicitly needs a multi-day trend. Keep a useful step and bounded
  aggregation so a longer trend does not become an unbounded series scan.
- `topsql`: canonical date-partitioned TopSQL aggregates over one bounded
  absolute range through the Data Proxy SQL API. For a node-specific question,
  pass `instance` plus `component=tidb|tikv`; this ranks SQL inside that node
  and is the default attribution path. `instance` accepts either the exact
  TopSQL `host:port` or the exact Clinic display ordinal `TiKV-N` / `TiDB-N`;
  the helper converts the latter to a component-bound DNS-prefix selector and
  returns the actual matched instance in row facts. Omit both only when the question is
  which SQL is globally largest. `digest` alone localizes one SQL across
  instances; `instance+component+digest` localizes its plans on that node.
  Metric `instance` labels and `entityRef` preserve original instance values
  and can be reused as selectors. Optional `planDigest` further narrows that selected SQL. Both Digests accept
  an exact value or a 6-to-63-character prefix, and `planDigest` requires
  `digest`. `rankBy` is one of `cpu_ms`, `exec_count`, `avg_latency_ms`,
  `read_keys`, `write_keys`, logical read/write bytes, or network in/out bytes;
  it defaults to CPU. CPU is always included. Use optional `measures` for no
  more than four load-bearing non-CPU aggregates in total, counting `rankBy`.
  `measures` uses those same logical names; never put underlying Data Proxy
  columns such as `topsql_stmt_exec_count` or `topsql_write_keys` in the plan;
  do not request all exported columns by default because each Parquet column
  adds scan cost. A no-Digest query is a fixed SQL-level scope Top20 plus an
  independent supplied-SQL candidate/plan lookup and a separate CPU-only scope
  total, so the candidate does not consume ranking slots and the total does not
  require an analytic window over ranked rows. Compact output carries scope
  total CPU, per-row share, average cores, plan count, explicitly requested
  aggregates when exported, and Top20 membership or a conservative cutoff
  share bound instead of tail rows. Null component-specific measures remain
  unavailable rather than becoming zero. Managed Tool broad lists retain full
  API-returned Digests and metrics, not normalized SQL or detailed plans.
  `includeSqlText` defaults to false for TopSQL/Slow Query discovery and true
  for Digest-selected detail. Set it explicitly to true only for single-query
  target matching or a load-bearing candidate detail question. Overall
  cluster-query-latency and workload-hotspot lists stay text-free; do not fetch every
  candidate's SQL. `sqlTextOmitted` is deliberate evidence selection, not
  redaction, unsupported capability or failed acquisition. `cpu_ms` is
  integrated CPU time across cores; never
  multiply it by core count. Use `cpu_ms / (window_seconds * 1000)` for average
  CPU cores. Keep the initial batch on the narrowest incident-aligned
  sub-window and never exceed 24 hours; do not default to the full canonical
  range when it is wider. Only a later adaptive refinement may use comparable
  `start`/`end` phases inside that range; express bounds as epoch seconds or
  timezone-explicit RFC3339 values. For an increase-at-onset question, the baseline phase must precede
  that onset; an after-onset low point is not a valid baseline. Never
  substitute a guessed PromQL metric.
- `sql_query`: one Agent-authored read-only query over Clinic's exported
  observability datasets, such as TopSQL, Statements, or LogStats, with every
  UTC date partition intersecting the exact absolute range, up to 24 hours.
  In a Managed Tool call, put the SQL text in the `sql` field, never `query` or
  `statement`, and never send both `sql` and `query`. The Tool accepts `query`
  only as a deterministic compatibility alias when `sql` is absent.
  Data Proxy owns its supported logical tables and SQL dialect; do not add a
  `clinic_data_proxy.` prefix merely to satisfy a client-side allowlist. The
  exact Statements table is the exception and is always
  `clinic_data_proxy.statements`; the Tool deterministically qualifies an exact
  unqualified `statements` table and does not rewrite unknown table names.
- `slow_query`: up to 24 hours of tier-aware bounded Slow Query rows. Default
  discovery omits detailed plans. A supplied full Digest or reusable six-character
  prefix selecting one SQL returns up to three execution-paired decoded plans
  with compact operator facts. Rows retain diagnostic fields,
  including instance, full plan Digest, `cop_proc_max` / `cop_proc_addr`, and
  `cop_wait_max` / `cop_wait_addr`, and API-returned query text in both discovery
  and selected-query rows. Keep complete returned Digests for exact follow-up queries.
- `logs_query_range`: one bounded Loki query of at most 24 hours with optional
  safe marker counts and an explicit `detailLevel=summary|events`. Before every
  log plan, read
  [log identity and detail contract](references/log-identity-and-detail-contract.md).
  It defines decimal connection/processlist/SQL-ID handling, exact and prefix
  Digest shapes, literal-first LogQL, and when the Provider Executor may return
  bounded typed event facts and up to twenty credential-redacted log lines.
- `activities`: bounded ActivityHub change-correlation facts over at most 30
  days. Start incident-centered and expand only when change history remains
  load-bearing.
- `ticdc_changefeeds_list`: bounded, sanitized TiCDC changefeed and TiCDC
  cluster metadata for the exact runner-supplied Cloud org and source cluster.
  Use the ticket's business-facing changefeed display ID to select one returned
  entry. The Tool does not expose sink endpoints, credentials, filter rules, or
  the kernel/internal changefeed ID. Pair this operation with one bounded
  `logs_query_range` query only when a runtime fact such as `newarch=true` is
  independently load-bearing; `newarch` is an allowed safe marker.
- `ticdc_changefeed_architecture_evidence`: preferred typed operation for a
  `ticdc-deployment-topology` question with an exact business-facing changefeed
  display ID. It exact-matches the bounded changefeed metadata, preserves the
  independent version/deployment/control-plane fields, and, only when the
  explicit architecture field is empty, deterministically checks the safe
  `newarch` startup marker in a creation-centered window inside the runner-owned
  canonical range. Do not replace it with an Agent-authored Loki selector.

For any selected `ticdc-*` diagnostic scenario, read
[TiCDC diagnostic fields](references/ticdc-diagnostic-fields.md) before writing
the plan. It is the small exhaustive architecture-aware metric and metadata
reference for this provider. Do not mix its classic and new-architecture metric
families or infer architecture from a metric name that merely happens to
return data.

## Database-access boundary

Clinic is not a connection to the customer's TiDB cluster. `sql_query` sends
SQL syntax to Clinic Data Proxy, whose Provider-owned logical-table and dialect
validation determines what observability data is available. The Managed Tool
does not maintain a second table-name allowlist and does not infer from a table
name that the query reached a customer database.

Do not assume that Data Proxy can retrieve customer database schema. When the
ticket or a reply-safe attachment supplies DDL, keep that artifact as the schema
input and use Clinic runtime evidence to test the diagnostic mechanism. Data
Proxy itself returns the safe logical-table, field, dialect, or partition error
when a read-only SQL statement is unsupported; preserve and correct that
per-query result instead of pre-classifying table names in this Skill.

PromQL is not restricted to a fixed metric allowlist. Start with the direct
mechanism named by the selected diagnosis branch, then its closest counterpart,
and only the strongest alternative. Keep the time range narrow and stay inside
the runtime-selected historical window.

For `capacity-resource-pressure`, use range metrics as the primary evidence:
compare resource usage with its real quota or effective limit, preserve the
same-entity time trend, and check aligned queue, throttle, backlog, or demand
signals when they are load-bearing. Use Activities when a bounded
configuration, scaling, maintenance, or deployment change could materially
explain the capacity trend. Do not query TopSQL, Slow Query, or
Statements merely to explain a capacity trend. Those detail surfaces belong to
a separately selected workload-hotspot, cluster-query-latency, write-path, or other
workload-attribution scenario.

For a TiKV CPU hotspot or CPU-capacity question, start with TiKV process CPU
per instance from `rate(process_cpu_seconds_total[2m])`, aggregated by the
metric family's proven `instance` label. Compare it with the actual TiKV quota
from `tikv_server_cpu_cores_quota` by that family's proven instance label. Add
only TiKV-scope matchers already established by the current Clinic metric
contract or compact result; do not invent a component, job, pod, namespace, or
cluster label.

Once the affected TiKV instance is established, query TopSQL with its exact
host:port or Clinic `TiKV-N` display ordinal and `component=tikv` before
attributing the node's CPU to a SQL. Do
not replace this with a global TopSQL rank. Global discovery is appropriate
only when the load-bearing question is which SQL is largest across the
cluster; if that discovery produces a candidate for a node-specific claim,
localize the candidate back to the affected instance.

"Native grouping" means grouping by labels exposed by the metric family being
queried. It does not mean switching to Kubernetes pod or container metrics.
Do not replace TiKV process CPU and quota with
`container_cpu_usage_seconds_total`, `kube_pod_container_resource_*`, or
`container_cpu_cfs_*` merely because the objective mentions CPU or capacity.
Select those Kubernetes families only when the supplied evidence question
explicitly requires verification of a Kubernetes quota, throttle, or cgroup
capacity boundary and their pod/namespace/cluster label scope is independently
known from the same family or the current provider contract. Never derive a
pod selector from a TiKV instance value or the reverse.

When a later evidence objective asks whether one target SQL explains a CPU
increase, compare equal-duration or otherwise normalized phases on the same
TiKV instance. The baseline must precede the reported increase and the hot
phase must follow it; if the canonical range does not contain both, return that
exact limitation instead of substituting two after-onset phases. Acquire
same-instance process CPU and the node-scoped TopSQL distribution for both
valid phases. Preserve the supplied SQL candidate, the leading node-local SQL,
and each phase's scope CPU total; do not reduce the comparison to target-Digest
rows alone. Keep the plan factual: the Scenario Diagnostic Agent owns
interpreting target, leading-SQL, and scope TopSQL average-core changes against
the process-CPU change.
Do not preemptively add opening, middle, or closing phase TopSQL queries to the
initial batch. Do not request rows below the complete global Top20 merely to
obtain an exact tail rank; the deterministic cutoff-share bound already answers
dominance.

### TiKV store and flow-control localization

For a write-path-contention question, the Scenario Diagnostic Agent's evidence
request is a semantic draft. The Evidence Agent owns translating that draft to
the exact Clinic operations, metric families, labels, grouping, and bounded
same-round repair. Do not require the Scenario Diagnostic Agent to spell
PromQL or provider field names.

Use this provider-owned relationship when the question asks whether one
specific store experienced flow control. Follow it in whichever direction the
current handoff makes observable:

1. On an already known exact TiDB `instance` and a focused 5-15 minute window,
   use `tidb_tikvclient_request_seconds_bucket` for the load-bearing
   `Commit`/`Prewrite` request percentile grouped by its native `store` and
   `type` labels. The returned numeric `store` is the TiKV store ID; do not
   query this high-cardinality family cluster-wide merely to discover stores.
2. Map that exact numeric store with `pd_scheduler_store_status`, preserving
   the returned `store` and `address` labels. Use `type="store_capacity"` for
   the bounded identity lookup. In this metric family, `instance` and `pod`
   identify the PD scheduler scraper and must not be treated as TiKV identity.
   For a numeric store plus a TiKV DNS address ending in `:20160`, the Provider
   Executor strips the port and DNS suffix and emits `tikvStoreMappings` with
   the exact `store` and first-label `tikvInstance`. This mapping is a
   provider-owned topology join, not permission to guess that unrelated metric
   families share an instance vocabulary.
   The exact `max by (store, address)` form is supported even though Prometheus
   removes `__name__` and the filtered `type` label from its result series. The
   Provider Executor binds normalization to that exact selector and grouping;
   an unrelated aggregate with address-shaped rows does not qualify.
3. Query `tikv_scheduler_throttle_duration_seconds_bucket` for that exact
   mapped TiKV instance and aligned window, and compute the requested bounded
   percentile (normally P99) by `instance`. A material aligned throttle
   duration supports store-local flow control; compare its magnitude with the
   end-to-end write latency rather than treating any non-zero sample as causal.
4. `tikv_scheduler_throttle_action_total`,
   `tikv_txn_command_throttle_time_total`, and their non-transaction
   counterparts are optional corroboration only when the current cluster
   exposes those series. Their absence does not negate a non-empty throttle
   duration histogram.

When no exact TiDB instance or numeric store is known, but store-local flow
control is a load-bearing alternative, start with a bounded top-k
`tikv_scheduler_throttle_duration_seconds_bucket` percentile grouped by its
native TiKV `instance`. If one exact instance has material aligned throttle
duration, the next authorized round queries a bounded
`pd_scheduler_store_status{type="store_capacity"}` mapping without filtering
`address` by the short instance name. Correlate that prior native instance only
with the Provider Executor's exact `tikvStoreMappings` output to obtain numeric
`store`; do not use the metric's PD-scraper `instance` or `pod` labels.
Once that store is present in the compact handoff, the next focused round uses
`store="..."` on `tidb_tikvclient_request_seconds_bucket` to test aligned
`Commit`/`Prewrite` latency without a cluster-wide source scan. This reverse
direction is the same identity chain; it does not permit guessing that other
metric families share instance labels.

Do not replace the chain with a generic top-k battery of scheduler, storage,
Raft, Region, and leader metrics. Those may bound alternatives, but they do not
prove which numeric store was throttled or whether Commit/Prewrite reached it.
The chain may consume both adaptive refinements after the initial batch because
each completed batch supplies the exact selector for the next one. Stop when a
required exact mapping is absent instead of substituting an unrelated SQL
lookup.

When the first batch does not yet know the exact store or TiKV instance, obtain
the lowest-cardinality aligned identity facts first. If the Scenario Diagnostic
Agent then asks the load-bearing store-local question, continue the same
Evidence Agent thread across the runner-authorized changed rounds and use only
exact labels returned in the compact handoff. Stop with the precise provider gap when
the TiDB instance-to-store or store-to-address link cannot be established; do
not infer it from names or dashboard legends.

Treat the Managed Tool as an ordinary read-only Provider program. Its batch
result contains one independent entry per `queryId`; a top-level completed call
does not hide an entry whose `status=error`. Every failed entry includes its
stable `queryId`, operation, safe
error code, and, when available, `errorClass`, `field`, `expected`, HTTP status,
bounded upstream detail, and repair instruction. For every local validation
rejection and every HTTP 4xx, inspect those fields and either correct that same
diagnostic objective in the current Turn or stop when no bounded caller-owned
correction exists. Send corrections in another batch containing only changed
failed entries. Reuse the same `queryId`, change at least one rejected argument,
and make at most four correction attempts for the objective. Never repeat the
same returned `requestDigest`, widen the authorized target or
window, or treat a rejected query as evidence that the metric or incident is
absent. HTTP 422 is not a special case; the same rule applies to the full 4xx
class. A configuration selector-validation rejection permits at most one
changed correction; any remaining configuration error stops because the
helper, not the Evidence Agent, owns Provider fallback.

Authentication, shared Provider-session bootstrap, readiness, authorization,
and runtime availability are Provider-wide prerequisites rather than query
errors. If the Tool reports `failureScope=provider_session`,
`repair.action=stop_provider_calls`, or an equivalent failure before a Provider
request, preserve that first failure and stop all remaining Provider calls in
the current Turn. Do not correct arguments, repeat authentication, or start
another objective against the unavailable shared session.

The one initial call must include the exact runner-supplied issue key, `orgId`,
and a `queries` array with at most twelve entries. Each entry contains one
supplied `clusterId`, one stable semantic `queryId`, operation, and
operation-specific query/time arguments. Use a distinct `queryId` for each
initial objective-target pair and reuse it only for a changed corrected attempt
of that same objective in the current Turn. The Tool executes independent
entries with a concurrency limit of six and preserves their input order in the
result.
The Agent-visible schema is batch-only. The Tool script retains a non-schema
single-operation compatibility path for existing deterministic callers, but
the Evidence Agent must not use or infer it. The Tool script validates the
ticket shape, supported operation, and
operation-specific maximum time range: 30 days for `metrics_query_range` and
Activities, and 24 hours for TopSQL, Slow Query, exported SQL, and logs. These
limits do not apply to the time-independent `configuration` operation. They are
ceilings, not default query sizes; begin with the narrowest
load-bearing sub-window and expand metrics or Activities beyond 24 hours only
for an explicit multi-day trend or change-history objective. Clinic
verifies whether the organization owns the requested cluster. Do not use
installation policy as a source for any current-run value.

For every metric plan, set `emptyResultExpectation` to
`expected_non_empty`, `sparse_allowed`, or `unknown`. Use
`expected_non_empty` only for a common baseline that this skill establishes for
the exact target/component and window. Use `sparse_allowed` only for a
legitimately sparse event/condition metric whose exact selector/label contract
is established here. Otherwise use `unknown`.

A successful expected-baseline empty gets one same-round, same-family, minimal
selector/label validation. If it remains empty, preserve an unexplained gap and
never use it as counter-evidence. A sparse empty is verified only under the
trusted contract above. HTTP rejection or a nonexistent metric family remains
failed, never empty.

## Output boundary

Use only the compact credential-protected Tool result. SQL, PromQL, full
Digests, addresses, and resource specifications are internal diagnostic data,
not credentials. Never request credentials or bypass bounded Provider
projections to open full payloads. Customer-facing cost, specification and
commercial disclosure restrictions are enforced at the audience boundary,
not by deleting internal diagnostic facts. Treat empty, rejected, or failed evidence as `unverified`; never
describe a failed query as a healthy cluster.