# TCID-13103 temporal peak alignment - Owner: Managed Clinic Provider Executor and workload-hotspot Scenario Diagnostic Agent - Source: TCID-13103 feedback 129308 and single-ticket task `003f5b7267e9cc41` - Implementation status: deployed; same-owner refinement read replay passed; stable Jira internal comment refreshed and read back - Effective status: `optimized_verified` ## Bounded problem The affected answer compared one-hour means across TiDB nodes and treated their similarity as evidence against a short single-node hotspot. It then used a cluster-wide TopSQL CPU share to support a node-specific conclusion. Neither comparison answers whether one node diverged briefly inside the range window. ## Intended change The Provider Executor keeps temporal p50, p95, p99, maximum, and the first maximum timestamp for each returned metric series while retaining the stable entity reference. The workload-hotspot Scenario Diagnostic Agent must compare the upper tail with that entity's central tendency and peers, align other same-entity signals around the peak, and preserve an explicit limitation when alignment is unavailable. Full-window averages and global TopSQL remain valid for their own questions but cannot negate the transient node question. The first deployed replay exposed one narrower continuation at the same owner: after finding the peak, the Scenario Diagnostic Agent spent its next question on an unavailable TopSQL instance-identity chain. For a transient TiDB CPU peak without a supplied SQL target, it must instead prioritize peer-comparable same-entity connection/session accumulation, query-processing demand/backlog, and TSO/PD-client wait around `maxAt`. Missing TopSQL identity limits only SQL attribution and must not block that immediate-mechanism comparison. The same-owner continuation is implemented in [#1181](https://github.com/tidbcloud/aisupport/pull/1181). ## Verification - Source PR [#1176](https://github.com/tidbcloud/aisupport/pull/1176) merged as `f2bebd0681158587c2bb2d78e720e7bd234a03f7`. Focused tests passed 173/173 and the full suite passed 1,023/1,023. - Managed Clinic v22 was published with package digest `6bc1bbf3d8ed9930f9538fe630ce5cd8a9558635dd73182caa716861b4dc53d5`; release PR [#1177](https://github.com/tidbcloud/aisupport/pull/1177) merged as `095b5bd14ab0abcf9134f635089709084cb98824`. - The staging Deployment rolled out that exact image with one Ready Pod, zero restarts, and HTTP 200 from `/`, `/healthz`, and `/readyz`. The root endpoint reported the same build SHA. - The single authorized TCID-13103 historical `mode=read` replay completed as run `87baaebb-d934-45ab-8269-e22a8fab928c` with HTTP 200 in 733.821492 seconds. Its canonical cutoff remained `2026-08-30T21:21:29+0800`, and its Agent9 operation recorded Managed Clinic v22 with the exact package digest above. - The peak entity's compact CPU facts preserved mean 6.18 cores, p50 5.94, p95 7.15, p99 7.18, and max 7.24 at the first maximum timestamp. The replay answer consequently identified one instance's short rise to 7.24 cores / 90.49% instead of using the one-hour mean to exclude a node-local peak. It treated cluster-wide TopSQL only as insufficient evidence for the node-specific attribution and kept the exact instance mapping and deeper trigger unresolved. - Same-owner refinement PR [#1181](https://github.com/tidbcloud/aisupport/pull/1181) merged as `d110f01a51d11b9b72b8bed152bfb9e92872aec9`. The focused scenario suite passed 82/82, the local full suite passed 1,028/1,028, and the GitHub CI full suite passed on its bounded rerun. The initial CI attempt exposed an unchanged streamed-iterator timing assertion; that exact test passed 21 consecutive runs on untouched `origin/main` and 21 on the repair branch, so the unrelated runner behavior was not added to this optimization. - Unified staging release [run 33464340063](https://github.com/tidbcloud/aisupport/actions/runs/33464340063) deployed main SHA `e92b4e34a5ef39bf0dc95c2744f17de2253fd6f3`, which contains the repair merge. App-delivery PRs [#78771](https://github.com/tidbcloud/app-delivery/pull/78771) and [#78772](https://github.com/tidbcloud/app-delivery/pull/78772) merged. The live Deployment reported that exact image, one Ready replica, zero restarts, and HTTP 200 from `/`, `/healthz`, and `/readyz`; the root response reported the same build SHA. - The one post-refinement TCID-13103 historical `mode=read` replay completed as run `65e00e36-62dc-47fd-8e9c-652de589b5a1` with HTTP 200 in 687.232308 seconds. Its cutoff remained `2026-08-30T21:21:29+0800`, and its Agent9 operations recorded Managed Clinic v23 with package digest `6ddddb1171235b49b986045d9ce9252d03a6dc53e8a6f544f41e55406ae2a89b`. - The first evidence round executed nine bounded range-metric queries for CPU, connections/session availability, query rate and handling/token wait, and TSO/PD-client wait. The Scenario Diagnostic Agent then asked which signal became peer-distinct no later than the CPU rise. The second round executed twelve 30-second range and point-change comparisons around the peak for the same stable entity and its peers. Every Provider operation was `metrics_query_range`; there was no TopSQL call and no guessed `TiDB-0..N` identity fan-out. - The replay answer retained the 7.24-core short node-local excursion, but stated that connection, query-handling, and PD-client peaks followed CPU by about 4-9 minutes, so the data shows one correlated episode rather than a proven cause. It also kept SQL/workload attribution unresolved and did not use cluster-wide SQL share to answer the node-specific question. The replay evaluator returned `different_but_valid` rather than the prior wrong-answer signal. - With explicit user authorization, the already verified draft from run `65e00e36-62dc-47fd-8e9c-652de589b5a1` was recovered directly without another replay. Before the write, the persisted idempotency mapping resolved `TCID-13103` to existing Jira comment `23760613`, and Jira readback confirmed that exact comment was `public=false`. - One Jira v3 `PUT` updated comment `23760613` in place. Post-write readback of the same comment remained `public=false` and matched the expected plain-text body digest `edd19d455507ead194a96135e9d11348ffd12774bdc148a973fb52d021a822a1`. The issue's comment count remained 12 before and after the write, proving no duplicate comment was created. Historical DB run, evaluation, and Base data were not rewritten by this direct recovery. ## Unresolved gaps The mechanism metrics were acquired and aligned, but their strongest peaks followed the CPU maximum; active-session, connection-accept-rate, and one direct TSO-wait series remained empty. The deeper workload or external trigger is therefore still unknown, and the answer correctly stops at correlated evidence. The evaluator also identified a separate remaining gap: calling the event an "instant local hotspot" can imply that the CPU level itself was abnormal when the historical replay cutoff did not contain a wider baseline or specification comparison. Later human evidence outside that cutoff treated the CPU level as near its existing high-water mark and emphasized capacity margin and connection distribution. That baseline/capacity distinction is not needed to pass this record's temporal-alignment criterion and remains a separate optimization candidate rather than permission to broaden this repair. ## Next validation action Monitor later workload-hotspot cases for regression of this exact temporal-alignment signal. Treat the separate baseline/capacity wording gap as its own candidate rather than reopening this verified problem. ## 2026-09-02 status refresh Authorized `mode=write` replay `006cd376-5bed-4639-bc67-0106398dc1f6` completed Provider acquisition without a failure code, preserved the short-lived node-local excursion, and did not use a full-window mean or global TopSQL share to rule it out. Router reconciliation succeeded and the persisted evaluation was `different_but_valid`. The Final Response Agent fell back with `diagnostic_final_response_failed`, but the Router-owned canonical answer retained the verified temporal-alignment boundary. DB run `150187` was persisted, existing Jira internal comment `23760613` was updated in place and read back as `public=false`, and its body retained both the customer-draft and internal sections. The Feishu Base row read back with `AI Acceptable Value=1` at `2026-09-02T19:18:18.729+08:00`.