# Demand-source diagnosis and refinement priority

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "demand-source-diagnostic-priority-2026-09-07",
  "problems": [
    {
      "problemId": "demand-source-evidence-priority",
      "summary": "CPU source diagnosis prioritizes unmatched lock samples or recurring high-water classification over aligned demand changes.",
      "status": "optimized_unverified",
      "applicability": "CPU or workload increases with supplied SQL candidates and incident-aligned demand evidence.",
      "issueKeys": [
        "APID-29464"
      ],
      "sourceRefs": [
        "https://tidb.atlassian.net/browse/APID-29464#problem:demand-source-evidence-priority"
      ],
      "signalTextIncludesAny": [
        "CPU",
        "lock_keys_time",
        "prior_7_daily_same_clock"
      ],
      "observedThrough": "2026-09-07",
      "effectiveAt": "2026-09-07T03:40:59.387Z",
      "verifiedAt": null,
      "attemptedAction": "Change Scenario Diagnostic target relevance and demand-source refinement priorities while retaining matched wait mechanisms and explicit historical capacity questions.",
      "evidenceUrl": "https://tidb.atlassian.net/browse/APID-29464?focusedCommentId=23812455"
    }
  ]
}
-->

Owner: Scenario Diagnostic Agent initial evidence question and local decision
in `src/diagnostic-scenarios.ts`. Base: fetched `origin/main`
`03d6d2d1e8effc22ae501edf71b714d0285affb0` on 2026-09-07. Final Response and Provider behavior are unchanged.

The fresh APID-29464 read run `85e690ca-deb2-4cca-aaf0-9b516ca6af52`
(answer fingerprint `6faeefc28d20d3c92a72950cfd9817096728e16a0f9851356ee8a9c133b8ca6a`)
ran image `c5f5ce2a`. Query-latency explicitly lacked a supplied-SQL match but
ranked 22–33 second lock/retry samples as the leading mechanism. Workload-hotspot
recognized peer-wide CPU/QPS growth but requested seven daily reference windows.
The final answer led with shared demand growth; Judge returned
`different_but_valid`. This remains distinct from the original answer's human
`wrong` feedback and does not negate it. Browser navigation failures and two
Agent9 `terminated` streams are separate execution limitations.

The one blocking criterion is that a CPU-source question prioritizes evidence
applicable to the reported workload and scope: unmatched waits cannot dominate
target diagnosis, and immediate demand-change discrimination cannot be replaced
by historical absolute-level classification merely because CPU was high.
Preserve directly matched waits, confirmed scope-aligned CPU attribution,
node-local transients, and explicit historically unusual/capacity questions.

The change distinguishes execution volume, workload mix, and CPU per execution.
A missing pre-increase segment remains a window limitation; this change does not
widen the Router's canonical window or invent baseline values. It does not prove
that all queries grew equally, that plans were unchanged, or identify the
application/caller/task which generated the demand.

Status: deployed, with one completed post-change read replay; overall source
investigation remains incomplete (`optimized_unverified`). The active staging
runtime was verified Ready and healthy on image
`47b394c201a33a0ae388e729d972640daf9650a3` at 03:40 UTC on 2026-09-07.
Release run `34080134982` succeeded; app-delivery PRs #79355 and #79356 merged.
Post-change read run: `9b197053-63e0-4acf-b0d4-96a480309018`.
The local regression below tests local diagnosis/refinement only. The completed
live replay is recorded separately below; neither Judge nor local passes establish
whole-ticket resolution.

## Local regression result

Final source-case trials use `gpt-5.6-sol` with high reasoning, three fresh
executions per paired first-round handoff. Query-latency passed 3/3: unmatched
locks were excluded from both the leading and strongest-alternative mechanisms,
and refinement searched the named targets instead of chasing the unrelated
lock samples. Workload-hotspot passed 3/3: shared demand remained the leading
bounded source category, with no seven-day refinement or unmatched lock
alternative. All six preserved suspected strength and did not invent a
pre-increase baseline or unchanged plans.

This passes only the local priority criterion. All three workload trials still
requested endpoint-local timing/discrimination rather than establishing the
application source or a complete before/incident decomposition. The frozen
range has no authorized pre-increase segment. A live replay must separately
verify the initial question, target matching, available comparison scope,
Provider transport, and Router synthesis before this problem can be verified.

TypeScript build and all 1,110 tests passed on the final source and updated
contract assertions (including the 81 scenario contract tests). The first revision
also passed the full local suite. Its first CI attempt failed the existing 25 ms
stream-cleanup test (cleanup count 0 versus 1); the same commit passed on one
unchanged rerun. Final-diff CI is required before merge. Final-source CI also exposed one stale
text assertion for the narrowed alternative-candidate rule; the assertion was
updated to require target relevance, and the full local suite passed again.

Retained local evidence: `~/.cache/auto-oncall-services/regressions/apid-29464-demand-source-20260907/`
contains the frozen route/handoffs, SHA-256 manifest, intermediate and final
validated decisions, and a compact report. Review and CI:
https://github.com/tidbcloud/aisupport/pull/1357.

## Deployed read replay: partial improvement

The single post-deployment read replay completed with HTTP 200 in 768.117129
seconds (12 minutes 48 seconds). Run:
`9b197053-63e0-4acf-b0d4-96a480309018`; answer fingerprint:
`9a187fe45fc060fbdad0d55419c39e00e04e8cc884d4f6750f332f0e54000163`.
The live image was verified directly before the request, with `/healthz` and
`/readyz` both `ok`. Deployment evidence:
https://github.com/tidbcloud/aisupport/actions/runs/34080134982;
https://github.com/tidbcloud/app-delivery/pull/79355;
https://github.com/tidbcloud/app-delivery/pull/79356.
Status-record base: fetched `origin/main`
`1152cadfd0f880197eb506beaedc77e354685202`.

Historical cutoff remained 2026-09-05 00:29:40 +08:00, matching the pre-change
replay. Later human plan conclusions and feedback were excluded from diagnostic
input. Judge returned `different_but_valid`, with no listed AI gaps, while the
independent source-case assessment remains partial improvement. The historical
human `wrong` still belongs to the original answer, not this fingerprint.

Both Scenario Diagnostic decisions rejected unmatched lock/retry samples as a
target cause or strongest alternative. Neither requested a seven-day window or
another refinement. The final reply led with broad execution pressure and did
not attribute CPU to Other or one supplied SQL. This verifies the narrow negative
priority checks in live integration. It does not verify Other knowledge uptake:
the returned TopSQL compact top 10 omitted Other, and the Router selected only
the OP environment-boundary topic.

The query-latency batch acquired six peers' QPS peaks of 48.8k–53.7k/s, broad CPU
maxima of 30.66–30.89 cores, and the reported endpoint's 9.7%–16.65% QPS share.
These are incident-window summaries, not a labeled before/incident comparison;
peers' separate maxima must not be summed into a simultaneous cluster CPS peak.
The final answer and handoff still concentrate on obtaining target plans and
execution phases. They do not complete demand-growth decomposition or identify
the application/caller/task behind the requests.

Remaining gaps have separate owners from this diagnostic-priority change:

- Router window: still only 23:44–23:50 +08:00, without an authorized pre-increase
  baseline for execution-volume / workload-mix / CPU-per-execution comparison.
- Evidence acquisition: the batch chased two unmatched broad Slow Query samples
  using 12-character digest prefixes; both requests returned `invalid digest`.
  The five supplied targets were not resolved in the materialized handoff.
- Managed Wetech v41: successful TopSQL still exposed anonymized identities and
  redacted SQL/plans, with 21 rows truncated to 10. Other calls hit Dashboard
  signin `ERR_ABORTED`, Grafana launcher/chat-button timeout, and session-not-ready
  failures. These are not evidence of invalid username/password credentials.

All model stages completed; response generation was `complete`, acquisition was
`degraded`, and the decision was `human_required`. No Agent stream termination or
scenario timeout occurred. Jira, database, persisted evaluation, and Feishu were
all `not_requested`. No publication or duplicate replay was performed. After the
HTTP result was saved, a concurrent rollout replaced the original Pod, so the
additional server `read-publication.json` was not recovered; do not assume this
read run remains publishable from the server artifact.

Retained evidence:
`~/.cache/auto-oncall-services/replays/apid-29464-demand-source-47b394c2-20260907/`
contains the complete HTTP response, final-answer-stage route/trace/handoffs and
compact Provider facts, baseline response, checksums, and Chinese result report.
The next validation must resolve the identity/query and comparison-window gaps,
then assess demand-source investigation. Keep this problem
`optimized_unverified`; it is not a whole-ticket success or an authorized Jira
refresh.
