# TCID-13103 historical hotspot baseline - Owner: workload-hotspot Scenario Diagnostic Agent refinement decision, runner-owned comparison-window expansion, Evidence Agent planning, and Provider Executor validation - Source: TCID-13103 post-change Judge gap after Jira internal comment `23760613` - Implementation status: third bounded repair merged as `49ecd1ecac7ccafdb43cb38baae1270c34e08c25` and deployed to staging - Effective status: `optimized_verified` ## Bounded problem The prior optimization correctly preserved a short node-local excursion inside the incident window. Its p95, p99, maximum, and peer separation answer whether the entity diverged during that event. They do not answer whether the absolute CPU level was unusual for that entity or merely revisited a recurring high-water mark. Without that second comparison, words such as "abnormal hotspot" can overstate capacity or headroom evidence. ## Previous ineffective action For workload-hotspot cases where the absolute classification can change the diagnosis, capacity wording, or mitigation, the Scenario Router authorizes the incident plus seven preceding calendar days as the immutable evidence envelope. The initial Scenario question remains incident-local. Only after the event-local shape and mechanism are known may the Scenario Diagnostic Agent use one adaptive refinement for three to seven prior, equal-duration windows at the same local clock time. The intended Evidence Agent behavior was to keep every reference window separate and plan the same metric semantics, unit, aggregation, entity and peer scope, and sampling step. Topology, node specification, resource quota, and relevant configuration must match or be explicitly normalized with an observed denominator. A missing or incomparable baseline stays unknown. A recurring high-water mark does not erase the event-local excursion, while an incident-local p99 or maximum does not by itself prove new absolute abnormality, capacity exhaustion, or a scale-out need. ## Verification The TypeScript build passed. The focused diagnostic-scenario and optimization-record suites passed 83/83. The full local suite passed 1,032/1,032. The new contract case covers Router authorization, conditional Scenario refinement, neutral Evidence Agent comparison planning, bounded unknown behavior for incomparable baselines, and the absence of historical-diagnosis instructions from the Final Response Agent. Source PR [#1197](https://github.com/tidbcloud/aisupport/pull/1197) was merged as `1cbde420c37c78d2fc32503e2386bfcbb49c1854`. Unified release [run 33492252135](https://github.com/tidbcloud/aisupport/actions/runs/33492252135) completed successfully, app-delivery PRs [#78856](https://github.com/tidbcloud/app-delivery/pull/78856) and [#78857](https://github.com/tidbcloud/app-delivery/pull/78857) merged, and the staging analysis Deployment became Ready with zero restarts on that exact image at `2026-09-01T09:33:50Z`. Root build metadata, `/healthz`, and `/readyz` all confirmed the deployed SHA and readiness. One bounded `mode=read` replay of TCID-13103 created run `dc65cd14-1e85-4f4b-ad97-1b8cd118c559` at `2026-09-01T09:35:44Z` and produced its final output at `2026-09-01T09:43:54Z`. The request client reached its five-minute response-header timeout, but the server-side run continued and completed; no second replay was started and no Jira write was requested. The replay did not satisfy the bounded problem criterion. The Scenario Router correctly authorized `2026-08-23T12:21:29Z` through `2026-08-30T13:21:29Z`, covering the incident hour plus seven prior days. The Evidence Agent then used that entire seven-day-and-one-hour envelope as one query window for all eight Provider operations. It did not first query the incident hour and did not materialize three to seven separate same-clock, equal-duration reference windows. Consequently the returned per-instance p95, p99, maximum, and time-series summaries mixed the incident with the historical period and could not establish whether the incident hour was above the recurring historical range. No adaptive refinement separated the windows. The Router therefore returned `human_required` and explicitly said both the incident-aligned investigation and the historical baseline were still unfinished, rather than producing the intended event-local versus historical classification. ## Second bounded action The failed replay showed that historical need and acquisition mechanics had the wrong owner. The Router now keeps only the immutable incident window. Once incident-local evidence has been materialized, the workload-hotspot Scenario Diagnostic Agent may set the typed `comparisonWindowPolicy=prior_7_daily_same_clock` only when historical abnormality still changes the diagnosis, capacity/headroom wording, or mitigation. The runner deterministically derives `day-1` through `day-7` as equal-duration same-clock windows. The Evidence Agent receives those explicit windows and must plan one identical range-metric operation per day. Clinic, Wetech, Manus, and the Agent9 Managed Tool materializer reject a missing day, duplicated day, non-metric operation, out-of-window operation, or one combined multi-day range. One batch remains allowed as a transport container, but not as one logical historical query. ## Unresolved gaps The second action was merged in source PR [#1209](https://github.com/tidbcloud/aisupport/pull/1209) and deployed by [release run 33508497686](https://github.com/tidbcloud/aisupport/actions/runs/33508497686). App-delivery PRs [#78882](https://github.com/tidbcloud/app-delivery/pull/78882) and [#78883](https://github.com/tidbcloud/app-delivery/pull/78883) merged, and the staging analysis Deployment became Ready with zero restarts on the exact image. Root build metadata, `/healthz`, and `/readyz` all confirmed the deployed SHA. The one authorized post-deployment `mode=read` replay returned HTTP 200 after 596.094 seconds as run `b4c3d60f-4eaa-4b66-ab7d-4053c7c4ca60`. Its initial Provider round stayed inside the incident-local window `1788092489..1788096089`. The Scenario Diagnostic Agent then requested one load-bearing refinement for rise/peak/recovery alignment, but explicitly set `comparisonWindowPolicy` to null. Round 2 therefore executed 12 incident-local metric sub-window operations rather than the seven prior daily same-clock operations. The runner and Provider history validators were never reached. Router reconciliation again stated that no comparable historical baseline was available, so the replay did not satisfy the bounded source-case criterion. The Judge verdict was `different_but_valid`; publication reported Jira, database, evaluation, and Feishu as `not_requested`. No Jira refresh was performed. The action also does not establish a fixed CPU threshold or universal capacity rule; comparable topology, specification, quota, configuration, and metric semantics remain required before classifying historical abnormality. ## Third bounded action and local simulation The replay showed a refinement-order conflict inside the workload-hotspot Scenario Diagnostic prompt. One rule required a historical comparison when absolute abnormality remained load-bearing, while a later rule prioritized incident-local connection, backlog, TSO, or PD-client alignment for a short-lived node hotspot. The Agent selected the latter. Although the schema allows another adaptive refinement, the second round consumed nearly the remaining Scenario budget, so historical classification was never requested. The revised owner rule makes the Agent perform a terminal-answer check before choosing the current refinement and treats later rounds as unavailable for planning purposes. If the customer asks why the reported CPU is high, or if the eventual brief would still depend on whether the level was abnormal, capacity-constrained, short of headroom, or mitigation-worthy, the Agent must request `prior_7_daily_same_clock` first. It may prioritize deeper incident-local mechanism alignment only after that absolute-level distinction is no longer load-bearing. A repository-local simulator now builds the production Scenario Diagnostic finalization prompt from a frozen Router packet and runner-owned Evidence Agent handoff, invokes the real model with the production decision schema, and validates the result with the production parser. It does not query a Provider, write Jira, database, or Feishu state, or require a deployment. Against the frozen TCID-13103 route and round-one handoff from replay `b4c3d60f-4eaa-4b66-ab7d-4053c7c4ca60`, three independent trials all selected `comparisonWindowPolicy=prior_7_daily_same_clock` as the next refinement. The local result validated only the Scenario Diagnostic choice. Live integration still required one bounded read replay to prove that the runner expanded and executed the seven separate historical windows. ## Post-deployment verification Source PR [#1211](https://github.com/tidbcloud/aisupport/pull/1211) passed CI and merged as `49ecd1ecac7ccafdb43cb38baae1270c34e08c25`. Unified staging [release run 33516929971](https://github.com/tidbcloud/aisupport/actions/runs/33516929971) completed successfully, including the image, same-source Agent9 Skills, Managed Tool package, and both application bumps. App-delivery PRs [#78891](https://github.com/tidbcloud/app-delivery/pull/78891) and [#78892](https://github.com/tidbcloud/app-delivery/pull/78892) merged. The live analysis Deployment reached generation 101 with the exact target image, one Ready replica, and zero restarts. Its build metadata reported the same SHA and `/`, `/healthz`, and `/readyz` returned HTTP 200. The one post-deployment TCID-13103 `mode=read` replay returned HTTP 200 in 614.529 seconds as run `8a6f33ce-6466-4293-8e04-ec3d888802b4`. Round 1 kept the incident window at `1788092489..1788096089`. The workload-hotspot Scenario Diagnostic Agent then requested `comparisonWindowPolicy=prior_7_daily_same_clock`, explicitly asking whether the incident p95, p99, or maximum exceeded comparable recurring upper tails. The runner expanded that typed policy to exactly seven separate equal-duration same-clock windows, `day-1` through `day-7`, from `1788006089..1788009689` back to `1787487689..1787491289`. The Evidence Agent received those immutable windows and round 2 executed exactly seven independent `metrics_query_range` attempts named `tidb-cpu-peer-historical-day-1` through `tidb-cpu-peer-historical-day-7`. It did not re-query the incident, omit a day, or combine the days into one range. This passes the bounded refinement-order and runner-expansion criterion. All seven historical Provider results were empty. The final answer therefore kept historical abnormality, capacity pressure, and a production change unsupported instead of presenting the incident-local peak as historically exceptional. Judge returned `different_but_valid`. Jira, database, evaluation, and Feishu publication were all `not_requested`; no Jira refresh was performed. ## Unresolved follow-up The empty results mean this replay did not obtain the historical values needed to match the later human investigation's absolute CPU and capacity conclusion. That is a separate Provider-query or historical-data-availability signal, not a failure of the now-verified Scenario Diagnostic refinement choice. A later optimization may investigate why all seven otherwise valid daily metric operations returned empty, but it must not reopen this problem by moving historical diagnosis into the Router, Evidence Agent, Provider Executor, or Final Response Agent. Any Jira refresh remains unauthorized until a separate verified result supplies the missing historical classification. ## 2026-09-02 status refresh An authorized `mode=write` replay completed as run `006cd376-5bed-4639-bc67-0106398dc1f6` and persisted DB run `150187`. Provider acquisition was complete with no failure code, the `workload-hotspot` Scenario Diagnostic completed after its historical comparison refinement, and Router reconciliation succeeded. The resulting answer preserved the short node-local excursion while keeping historical abnormality, a unique workload, and capacity action unproven. The Final Response Agent itself fell back with `diagnostic_final_response_failed`; the fallback used the Router-owned canonical answer and did not reopen this bounded Scenario Diagnostic criterion. The replay updated existing Jira internal comment `23760613` in place. Direct readback confirmed `public=false` and both the customer-draft and internal sections. The persisted `analyze_write` evaluation was `different_but_valid`, and the Feishu Base row read back with `AI Acceptable Value=1` at `2026-09-02T19:18:18.729+08:00`. No customer-visible comment was created.