# Separate target-SQL diagnosis from overall latency and workload lists

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "single-query-scenario-2026-09-08",
  "problems": [
    {
      "problemId": "target-sql-and-overall-latency-scope",
      "summary": "Target SQL diagnosis shared the overall query-latency Scenario and automatically opened a hotspot branch; broad SQL lists consumed context with full SQL text.",
      "status": "optimized_unverified",
      "applicability": "Target-specific SQL diagnosis and overall latency/workload discovery using the Agent9 Clinic or Wetech Managed Tools.",
      "issueKeys": ["TCID-13132"],
      "sourceRefs": ["https://tidb.atlassian.net/browse/TCID-13132#problem:target-sql-and-overall-latency-scope"],
      "signalTextIncludesAny": ["target-sql-and-overall-latency-scope"],
      "observedThrough": "2026-09-08",
      "effectiveAt": "2026-09-08T12:11:38.555Z",
      "verifiedAt": null,
      "attemptedAction": "Merged PR 1411 with Clinic v33 and Wetech v65, then PR 1412 (ed03eb41) to preserve Router-owned cluster scope; verified deployed SHAs and completed three isolated case reads plus the corrected aggregate rerun. Routing and list projection passed. Target identity remains unmet on TCID-13132; TCID-13103 has a causal-attribution gap; TCID-13139's hotspot refinement timed out. No Jira publication requested; not a verified root-cause repair.",
      "evidenceUrl": "https://tidb.atlassian.net/browse/TCID-13132"
    }
  ]
}
-->

Owner: AOS Scenario Router, Scenario Diagnostic guidance and selected Managed
Provider result projection. Source: operator-requested Scenario split.
Base: `ab80172a03e63e485742a2838e08f72163c64fa7` (fresh origin/main).

## Bounded capability

- `single-query`: the selected SQL/request's identity, execution latency,
  plan-choice explanation and bounded optimization advice. Multiple executions,
  plans or engine comparisons remain one target-specific question.
- `cluster-query-latency`: overall/multi-query latency distribution and representative
  phase/wait evidence; no automatic single-SQL tuning audit.
- `workload-hotspot`: resource contribution. It is not automatically added for
  one slow SQL without an independent resource question/signal.
- Managed Clinic and Wetech broad TopSQL/Slow Query lists default to
  `includeSqlText=false`; full Digests, instance/time and numeric facts remain.
  Digest detail defaults to true; explicit true is available for bounded
  target-SQL matching. This is selection, not redaction or failed acquisition.
- Single-query Clinic discovery and a dependent Digest detail may complete in
  the same Evidence Agent Turn within the existing combined query budget.
  Evidence plans acquisition; Scenario Diagnostic judges identity and causality.

## Verification and remaining work

Local verification at `2026-09-08T11:34:31Z`:

- `npm test`: 1,208 passed, 2 skipped, 0 failed; isolated runner script tests
  additionally passed 6/6. These are contract/fixture tests, not real LLM or
  Provider execution.
- `npm run deploy:agent9-tools -- --validate-only` and `git diff --check`
  passed. Regression coverage includes route separation, exact role Skill
  ownership, broad-list text omission, explicit text inclusion, preserved
  full Digests and distinct 4.5/20.8-second execution records.
- Prepared Clinic v33 and Wetech v65 immutable artifacts; downloaded the new
  bundle's exact S3 version and verified SHA-256
  `e03069506ed3afca6ffee842728fef4b0d6c778c0b1bcbc9cc9416845376d7a4`.
  Version ID: `jSk3nRTdXod9LabQBQLbEydEpUCrWu5Z`.
  At this initial verification point only source configuration referenced the
  bundle. The subsequent operator-authorized rollout is recorded below.

## Merged rollout and read verification

PR #1411 merged as `07a9c8c7208ccc26457f9f6cd5230a9f4b015453`.
The live Ready AOS Pod reported that exact build SHA and the new Scenario IDs.
Agent9 loaded the bundle above without changing its image; Evidence Agent
installations were read back as Clinic v33, Wetech v65 and Manus v7.
Skills from the same source were published and role installations verified.
Staging image-source PRs `tidbcloud/app-delivery#79521` and `#79522` merged.
After the source merged but the Deployment remained on the old image, the one
active Deployment's image was converged to that exact merged source value and
its rollout was verified Ready. No poller/worker policy was changed.

Release workflow `34222842841` initially stopped at
`managed_tool_installation_in_use`; the observed running Turn subsequently
succeeded. Rerunning the workflow then hit `idempotency_key_reused` in the
already-completed Skill step. No release implementation was changed: the
existing Managed Tool release script completed the remaining installations,
and the existing app-delivery workflows completed the image-source updates.
The two workflow failures are release observations, not replay failures.

Before rollout, three local real-model workload-hotspot trials used the
unchanged route/handoff pair from TCID-13132 run
`5f9cb4a4-ec38-45be-87a0-218614c36001`. All returned `suspected`, attribution
`correlated`, and no further refinement; none treated the absent target match
or resource corroboration as proven causality. This verifies only the offline
Scenario Diagnostic boundary, not new routing or Provider execution.

Selected cases, all created since 2026-08-25:

- TCID-13132 (2026-09-03): same SQL's TiFlash/TiKV execution comparison;
  expected `single-query` with target identity and distinct execution facts.
- TCID-13103 (2026-08-30): high TiDB CPU attribution; expected
  `workload-hotspot` without a routine full-SQL/plan inventory.
- TCID-13139 (2026-09-07): a batch of slow queries and 60 timeouts; expected
  `cluster-query-latency` with aligned aggregate and representative phase facts.

These are full isolated `mode=read` replays with no Jira publication, DB/Base
write or evaluation-state update. Results and frozen publication artifacts are
stored under the private operator artifact directory
`scenario-split-20260908.hLw7GW`.

TCID-13132 completed in 684,570 ms as run
`a13d6ab2-6690-4952-9b51-8276acf5e210`, fingerprint
`68ba75fd2ca29f80e570c92bc50231ed1dd8c2ecb9b9c57ce869fb33038750ee`.
It selected `single-query` primary plus `tiflash-io-apply`, with no automatic
hotspot branch. Both Scenario executions and response generation completed;
the 1 metadata preflight plus 12 TiFlash queries did not trip the historical
count check. There were 16 Provider attempts overall, no execution failure
codes, and 10 empty TiFlash metric results. Both diagnoses were `inconclusive`
because the target SQL was not precisely matched and aligned engine/resource
evidence remained insufficient. Judge was `different_but_valid`, not proof of
root-cause recovery. The focused identity criterion remains unmet.
The single-query handoff was 48,261 bytes.

TCID-13103 completed in 1,012,551 ms as run
`7b0f4454-a70c-41d7-adcb-13fe8f2070df`, fingerprint
`d3c04d5a10d767f68c7a57315a92fb4706d518debc59236eb9fd73e87802daf5`.
It selected `workload-hotspot` primary plus `capacity-resource-pressure`,
without single-query tuning. Both executions completed without Provider
failure codes. Three first-round lists and the peak-local refinement TopSQL
all had `sqlTextOmitted=true`, preserving full Digests and numeric metrics.
The initial lists contained 5/5 TopSQL entries and 18 Slow Query records;
their compact reports were 4,065/3,946/21,722 bytes. The initial hotspot
handoff was 48,812 bytes. The one authorized refinement compared same-window
TopSQL CPU and process CPU without requesting every SQL's text.

Judge was `human_better`: the final answer treated 84.5% SQL CPU accounting
as the cause of the CPU increase and called the incident cluster-wide while
explicitly lacking a comparable pre-peak baseline. That accounting shows
where CPU was consumed, not why the anomalous increment occurred. The human
comparison instead described unchanged request rates and node-local TSO
wait/connection buildup. Thus list projection and routing passed, but this
replay exposed a separate diagnosis/refinement gap; do not mark the case's
root cause repaired or correct it in Final Response formatting.

TCID-13139's initial run `46a6f151-3346-44bb-8ec0-9b7c7f418a71` exposed a
deterministic normalization regression introduced by the Scenario split:
the raw Router selected `cluster-query-latency` for the batch incident without
a selected SQL, but normalization replaced it with `single-query` solely
because the classifier category was `sql_performance_tuning`. The original
run is preserved and is not a passing cluster-latency verification.
The bounded correction removes that classifier-driven override, leaving the
scope decision with Scenario Router. It changes no Provider package, diagnosis
SOP, model, budget, or publication path. Its full suite passed 1,209 tests,
with 2 skipped and 6 additional isolated-runner tests passed. After release,
only this aggregate case needs rerunning: the first two cases already had
explicit Router selections that the removed override did not change.
PR #1412 merged the correction as
`ed03eb4102ed69b4946cb1b65fc17b71876fc581`; release workflow `34228369167`
succeeded, with image-source PRs `app-delivery#79524` and `#79525` merged.
The initial misrouted run completed in 717,932 ms, with fingerprint
`1f4a5374fe5698e4a3bf908b60755a2d808817be82315c9ae730fce3b7a9754b`
and Judge `not_compared`; execution completion does not cure its wrong scope.
Post-correction rollout initially failed verification: a manual image convergence to the
merged ed03eb41 source was followed by Argo CD reconciling the Deployment back
to 07a9c8c7. The Ready Pod SHA readback caught this; no new replay was submitted.
Managed-field evidence names `argocd-controller` as the image owner at
`2026-09-08T12:55:33Z`, and the last-applied image also remained 07a9c8c7.
The app-delivery source was independently read back as ed03eb41. The staging
Argo API returned HTTP 403 from this host, and the target workload cluster does
not host the Application CRD. The operator was asked for its control-plane
context or an Application refresh; do not repeatedly fight self-healing with
manual image updates. The controller subsequently converged; at
`2026-09-08T13:03:19.178Z` the Ready Pod reported the exact ed03eb41 SHA.
The post-correction isolated replay was then started in its own new artifact
directory `TCID-13139-route-fix`. Never substitute the earlier misrouted,
completed process for this verification.

The corrected run selected `cluster-query-latency` primary and
`workload-hotspot` secondary. All six first-round Slow Query lists omitted
SQL text. Their numeric execution records still occupied substantial space:
the cluster-latency handoff was 118,351 bytes and the hotspot handoff was
23,793 bytes. Removing SQL text does not deduplicate execution records across
windows and sort orders or guarantee a small total handoff.

After one refinement, cluster-query-latency returned `confirmed` for a bounded
Digest cohort's coprocessor response/wait mechanism, while retaining the
unproven link to the reported 60 application timeouts and the unproven deeper
configuration trigger. The hotspot second-round diagnostic Turn timed out
after 264,479 ms against 264,476 ms remaining; its trace recorded no terminal
event or final response. This was after Provider acquisition, with replay Pod
restart count zero, not an observed replay Pod rotation or Provider failure.
Router reconciliation and Final Response subsequently completed. The
diagnostic timeout is a remaining finding, not permission to raise global
budgets or silently rerun the case.

The corrected run `e4322416-536b-44ea-ba46-f7e9870b89e8` completed export at
`2026-09-08T13:24:16.929Z` after 1,214,518 ms, with fingerprint
`ae79cc24227c415ce946848e2e04a6bcd40d086b55f3d74a7a1f34d50eb148d7`.
Provider acquisition and response generation were complete with no final
Provider failure codes. Of 55 ordered attempts, seven round-two log queries
initially returned `invalid_query`; each was corrected in the same round and
its second attempt returned complete or empty. These are corrected
intermediate errors, not unresolved Provider failures. The scenario summary
explicitly retained hotspot
`outcome=timeout`, `status=bounded_failure`, `completion=partial`.
The final ticket decision was `human_required`. Judge was `not_compared` with
`eligibility=no_human_reply`: the only human reply acknowledged investigation
and contained no substantive diagnosis. This is not a quality pass from Judge.
The final answer distinguished the observed slow-query/cancellation cohort
from unproven one-to-one client timeout attribution and did not assert the
configuration change as the root cause. All 16 compact Slow Query/TopSQL
reports across both rounds had `sqlTextOmitted=true`; targeted Digest records
could still supply execution/plan details for the precise refinement.
The full run, frozen read-publication artifact, summary and logs were exported
before Job cleanup. No result was published to Jira, DB or Base.

Historical
TCID-13132 artifacts use the former query-latency scope; do not relabel or edit
them to manufacture a single-query pass. A fresh isolated post-deployment
mode=read replay must verify Router selection, focused SQL lookup, preserved
execution/plan evidence, complete diagnosis and reduced broad-list payloads.
The reads above verify route separation and SQL-text omission, but do not
meet all of those diagnosis criteria; the record remains optimized_unverified.

The earlier SQL keyword-filter implementation and paused target-focus worktrees
are separate unmerged work. They are not included or claimed deployed by this
change. Package integration must preserve both changes when merging them.
