# Clinic TopSQL Instance Scope

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "clinic-topsql-instance-scope-2026-08-18",
  "problems": [
    {
      "problemId": "clinic-topsql-global-ranking-does-not-localize-node-cpu-driver",
      "summary": "A node-specific TiKV CPU investigation used cluster-wide TopSQL discovery because the typed Clinic helper did not expose component and instance selectors, so the dominant SQL was aggregated across 44 TiKV instances and could not be attributed to TiKV-13.",
      "status": "optimized_verified",
      "applicability": "Clinic workload-hotspot diagnosis where an affected TiDB or TiKV instance is known and TopSQL must rank or quantify SQL inside that exact node scope; global discovery remains available only for cluster-wide workload ranking.",
      "issueKeys": ["TCID-12986"],
      "sourceRefs": ["https://tidb.atlassian.net/browse/TCID-12986#problem:clinic-topsql-instance-scope"],
      "signalTextIncludesAny": ["TopSQL instance filter", "global SQL does not prove TiKV node attribution", "dominant digest across instances", "node-local SQL CPU share"],
      "observedThrough": "2026-08-19",
      "effectiveAt": "2026-08-19T00:57:50+08:00",
      "verifiedAt": "2026-08-19T01:22:49+08:00",
      "attemptedAction": "Expose typed component, exact host:port or Clinic TiKV-N/TiDB-N display-ordinal instance, SQL/plan Digest, bounded rank, and on-demand public measure selectors in both Clinic Provider Executor backends; make instance-scoped TopSQL the default for node attribution; obtain the scope CPU total through a separate CPU-only query and return row shares, average cores, plan count, and only requested execution/duration, key, logical-IO, or network aggregates without exposing raw SQL or plans. Install the repository-owned Clinic skill from the runtime image bundle so the deployed Evidence Agent receives the same contract.",
      "evidenceUrl": "https://tidb.atlassian.net/browse/TCID-12986"
    }
  ]
}
-->

- Owner: Clinic Provider Executor for typed selectors and neutral aggregates;
  Scenario Diagnostic Agent for choosing node scope versus global discovery
- Source: TCID-12986 human-versus-AI TiKV-13 CPU attribution review
- Implementation status: effective in staging
- Effectiveness status: optimized_verified

## Bounded problem

The ticket asked what drove CPU on TiKV-13. The helper could rank SQL globally
or localize an already known Digest by instance, but it could not rank SQL
inside one exact instance. The Scenario therefore found a globally dominant
Digest without proving whether it was concentrated on TiKV-13.

## Implemented change

- Add closed `component`, exact `instance`, optional `planDigest`, and bounded
  `rankBy` fields to the typed TopSQL operation in both Provider Executor
  backends.
- Accept the exact Clinic display ordinal (`TiKV-N` / `TiDB-N`) when a ticket or
  metric does not carry TopSQL's full `host:port`; bind it to the same component
  and DNS ordinal, while preserving the actual matched instance in row facts.
- Publish public measure names (`exec_count`, `avg_latency_ms`, key, logical-IO,
  and network aggregates) to the Evidence Agent. Normalize the corresponding
  exported column aliases at the Provider Executor boundary instead of rejecting
  an otherwise usable plan.
- Infer four deterministic scopes from selectors: instance TopN, Digest by
  instance, Digest/plan on one instance, and global TopN only when no instance
  or Digest is supplied.
- Return CPU and scope-relative facts by default. Expose other neutral
  aggregates through a closed, at-most-four on-demand measure list while
  keeping normalized SQL and plans internal to deterministic matching or safe
  plan compaction.
- Make the workload-hotspot Evidence Agent prefer the known affected instance;
  use global discovery only when the load-bearing question is which workload
  is globally largest.

## Verification evidence

Before implementation, a direct canonical Data Proxy query over the frozen
TCID-12986 window proved that exact `component='tikv'` plus the TiKV-13
`instance` is supported. In that scope `a862ec` ranked first at 7,828,242
CPU-ms (29.16% of the node's captured TopSQL CPU), while `bc6171` ranked second
at 2,612,601 CPU-ms (9.73%). The former had one plan and the latter four.

After the implementation was changed to separate ranking from the CPU-only
scope total, the local production-backed helper completed the same exact
instance/window query in 22.7 seconds. It returned scope total 26,842,803
CPU-ms and reproduced the same first two Digests, shares, average cores (8.6980
and 2.9029), and plan counts without exposing full Digests or SQL text. This is
pre-deployment capability evidence, not yet runtime replay verification.

The first post-deployment read replay on 2026-08-19 reached the new typed
TopSQL plan but failed before provider execution with
`clinic_topsql_instance_invalid` for the Clinic display value `TiKV-13` and
`clinic_topsql_measures_invalid` because the Agent used exported column names
from the plan contract. Runs `4bd41e0e-7954-4b6b-87a0-074b567892f5` and
`4c7f20af-3888-473e-afb7-018a1287e170` therefore remain negative runtime
evidence, not verification. The same replay also proved that the mounted
external Clinic skill was stale relative to the repository-owned skill.

After the follow-up repair, a production-backed pre-deployment execution using
the exact failed plan shape (`instance=TiKV-13` plus the four exported measure
aliases) completed successfully. The helper matched
`db-tikv-13...:20160`, returned the same 26,842,803 CPU-ms scope total, and
again ranked `a862ec` first at 29.16% and `bc6171` second at 9.73%. Unsupported
TiKV execution-duration measures remained null while `read_keys` was preserved.
This proves the repaired Provider Executor contract but is still not a deployed
replay.

The follow-up staging image
`00add7fc3ad53efc10d99e6fa9e687b1fae3197c` installed the required bundled
Clinic skill and completed read replay
`8f624dbd-0ba4-4e17-82b8-90b86d6362fe`. Its first workload TopSQL query used
`component=tikv` and `instance=TiKV-13`; all exact-instance TopSQL operations
completed without instance or measure validation errors. The result preserved
the actual `db-tikv-13...:20160` instance, measured a 26,842,803 CPU-ms node
scope, attributed at most 9.73% to `bc6171`, and identified `a862ec` at 29.16%
in that same node scope. The Scenario therefore treated the supplied SQL as a
partial, non-dominant contributor instead of inferring node attribution from a
global ranking.

Authorized write replay `b975f4a1-22a3-4cb6-8069-87779f626a02` then completed
with Clinic evidence acquisition `complete`, provider capability verified, and
no failure codes. It added phase-aligned node-local checks for the larger SQL
and kept the ticket-level distinction between the confirmed supplied-query
latency mechanism and incomplete attribution of the whole TiKV-13 CPU increase.
Publication reused run DB id `150094` and updated the existing Jira internal
comment `23357171` in place. Jira readback confirmed the exact same comment id,
`public=false`, and the expected paired customer/internal body; no duplicate or
customer-visible comment was created.

The same source revision was also delivered to the passive Agent9 staging
backend through app-delivery PR `77631`. Live readback showed image
`00add7fc3ad53efc10d99e6fa9e687b1fae3197c`, one Ready pod with zero restarts,
the bundled Clinic skill startup marker, and HTTP 200 health while automation,
polling, and Jira writes remained disabled.

## Unresolved gaps

The first pre-deployment live helper trial showed that one query scanning every
exported measure plus an analytic scope-total window exceeded 120 seconds. The
verified implementation therefore keeps a separate CPU-only total query and
scans other measures only when explicitly requested. Component-specific TopSQL
measures may still be null when that component does not export them and must
remain unavailable rather than be rewritten as zero. The full-window node share
does not by itself measure each SQL's contribution to a phase-to-phase process
CPU delta; that remains a Scenario-level correlation question, not a helper
contract gap.

## Next validation action

Monitor later Clinic node-hotspot replays for regressions in exact-instance
selection, bounded measure use, and the distinction between full-window SQL CPU
share and phase-to-phase process CPU attribution.
