# Diagnostic context efficiency: post-deployment verification

Status: three read-only replays completed; no ticket-wide repair or optimization-status
promotion. This is a read-only replay report, not a new behavior change.

## Release and execution boundary

- Implementation PR: #1416; deployed AOS build
  `02bdc964eca3b58d0f3835acd5224737eb9e3d0d`.
- Ready image digest:
  `sha256:08020e5147ce6924a05d8625c945d8a4628d0601c67b7990d7d4f22d440646e0`.
- Clinic Skill v22 and shared evidence-query-planning v2 were released; no
  Managed Tool package change was part of this release.
- Actual runs use isolated Kubernetes Jobs in the authorized staging context,
  production `mode=read`, and explicit `noDiagnosticTimeout=true`. The one-hour
  Job and collector limits, Provider operation limits, and cancellation remain.
  Default auto-oncall timeout behavior is unchanged.
- No Jira, database, evaluation-publication, or Base write is authorized.
  Export full runs, frozen read publication, summary, and logs before Job cleanup.
- The first two replays overlap. Times below are observed wall time, not a
  controlled performance benchmark or proof that a single code change sped up
  the model.

## Runs and comparison identity

| Ticket | New run | Baseline run | New / old wall time; delta | Evaluation |
| --- | --- | --- | --- | --- |
| TCID-13103 | `8da37d69-2676-4b96-9ea0-5a55c5419dbb` | `7b0f4454-a70c-41d7-adcb-13fe8f2070df` | 19m13s / 16m53s; +2m21s | `human_better` |
| TCID-13139 | `8e3d2c69-7ab4-4d86-b4db-934ce4b20310` | `e4322416-536b-44ea-ba46-f7e9870b89e8` | 15m59s / 20m15s; -4m15s | `not_compared` |
| TCID-13132 | `9e70c8f4-9a43-4462-a77b-876f7e421828` | `a13d6ab2-6690-4952-9b51-8276acf5e210` | 20m30.1s / 11m24.6s; +9m05.5s | `different_but_valid` |

Answer fingerprints:

- TCID-13103: `7f771246571775ab6a2194b77f03aa08ac90cc09e05ffff563413f52d8592aa1`.
- TCID-13139: `70d31b26489058b2c0af3f9d45f78177174e03c5238f852db6e0a1a96fc9311f`.
- TCID-13132: `8f84933a4d1248af2e6b306328c6c60df425dcdb4ab606a3b03b2d2969b984ab`.

TCID-13103 ran from 2026-09-09 00:22:23 to 00:41:37 (+08). Its ticket JSON,
visible comments, replay target/cutoff, original image, and customer-evidence
file are byte-identical to the baseline. The cutoff remains the first support
reply at 2026-08-30 21:21:29 (+08). The new Judge used substantive human reference
comment `23755233`, body digest
`7d0b35a1f0bd3b127cecd28dc5da9de63553e5d0a0a42c16dd05c221df9dc043`;
the old Judge also included another comment. Judge labels alone are therefore
not a matched quantitative quality comparison.

TCID-13139 ran from 2026-09-09 00:24:28 to 00:40:27 (+08). Its baseline is the
route-fixed run above, not the earlier incorrectly routed intermediate replay.
Ticket JSON, visible comments, replay target/cutoff, and customer evidence are
byte-identical to that baseline.
The current human comparison contained only an acknowledgement, hence
`not_compared`; this does not mean equivalent to a substantive human diagnosis.

TCID-13132 ran from 2026-09-09 00:43:13 to 01:03:44 (+08). Its ticket JSON,
visible comments, customer evidence, and replay target are byte-identical to
the listed baseline (fingerprint
`68ba75fd2ca29f80e570c92bc50231ed1dd8c2ecb9b9c57ce869fb33038750ee`).
Both use the 2026-09-03 10:57:09 (+08) cutoff before target comment `23794717`.
Both Judge verdicts are `different_but_valid`; the new evaluation has
`previousAnswerVersion=null`, so it is not a Judge comparison with the old AI.

## TCID-13103: incident alignment improved, causal diagnosis remains bounded

The Router retained the affected TiDB node, its separation from peers, and the
approximately 21:02 (+08) CPU rise. Evidence compared 20:49–21:02 with
21:02–21:15, then refined the onset/plateau within 20:54–21:15. Neither the
diagnosis nor final ticket answer replaced this event with the old 20:32
cluster-global peak. The workload and capacity scenarios both finished, including
the workload refinement.

The additional measurements supported node-local connection/session accumulation
as a suspected mechanism: connection-stock growth peaked shortly before the CPU
rise, while PD-client wait maxima were later. The answer preserved the missing
direct accounting between session work and incremental CPU. It did not treat
empty node TopSQL as evidence that no local workload concentration existed.

Remaining limits and observed risks:

- Two initial node TopSQL requests failed instance-format validation, then were
  corrected in the same Evidence turn. The accepted `TiDB-2` requests returned
  zero rows in both 13-minute phases, classified as `unclassified_empty`.
  Global results were nonempty. The response does not establish why the node
  results were empty; do not silently equate this with verified no activity.
- A derivative of connection stock measures net accumulation, not new-connection
  arrival rate. Disconnect rate and connection lifetime were not acquired.
  The internal request to inspect "new connections" is terminologically
  ambiguous, not proof that the answer blamed a customer traffic surge.
- The capacity decision and Router/Final wording strengthen temporal alignment
  into CPU pressure "causing" backlog and latency. The corresponding human
  evidence says the earliest changes cannot be temporally separated at its
  observation granularity. This causal-strength risk remains.
- The customer answer copies the detailed monitoring-semantics checklist;
  that is a presentation issue, not evidence that the underlying metric was
  validated.

Router received the original image, but no Scenario Diagnostic turn requested
an image in this replay. Same-run on-demand image delivery is supported by unit
and local-model tests, not exercised end to end by this particular Agent9 run.
The original finalization-only local trial's UTC error remains retained; the
subsequent full-context local trials passed only the bounded image/time criterion.

## TCID-13139: both scenarios finish, with remaining acquisition limits

Both `cluster-query-latency` and `workload-hotspot` ran through round-2 diagnosis.
The cluster scenario completed; workload acquisition remained partial with
`clinic_http_500`, even though the Scenario Diagnostic turn itself succeeded.
The final answer identified an incident-local sparse tail-latency execution
batch and distinguished observed TiKV Cop waiting from an unproven configuration
trigger. Exact mapping to all client timeouts remained open.

The actual Evidence Agent emitted four purpose-specific Slow Query summary
groups across the two scenarios and their two rounds. The cluster scenario's
round-2 summary retained the complete Digest and plan identity, slow execution
timing and key counts, unavailable waiting fields, and the same-plan faster
controls; it did not replace them with a generic "slow SQL" conclusion.
Final cumulative handoffs were 20,194 bytes for workload and 22,366 bytes for
cluster latency, versus 129,938 and 126,705 bytes in the old run. These are
different query/result sets, so this is observed runtime size reduction, not a
paired same-input compression ratio or proof that all omitted facts were safe.

The per-task coverage risk is concrete: Evidence retained another candidate's
`request_count=110–154`, but the cluster Diagnostic R1 called the control not
genuinely small-task because its whole execution exceeded one million keys,
then refined only the dominant one-task Digest. Whole-execution keys do not
establish the size of each Cop task. The old same-plan detail for that other
candidate included a large index scan plus 98 lookup tasks with only 15 maximum
processed keys per lookup task, a relevant candidate for the customer's
small-task hypothesis. Its old slow execution cannot be substituted for the
new faster samples, but the candidate's follow-up coverage became weaker.
This was not a summary loss of `request_count`; the selection happened in Diag.

The new run also did not reproduce the old 60-execution/Cop-cancellation-log
corroboration. It correctly retains the unmatched-client-timeout gap rather than
claiming every timeout was explained. The current HTTP 500 came from an added
configuration lookup that the old run did not perform; it is not evidence that
the same previously successful operation regressed. These limits prevent
promoting `completion=complete` or a non-compared Judge result into a root-cause
pass.

## TCID-13132: live image rereading works, target-SQL criterion not met

Both `single-query` and `tiflash-io-apply` completed their diagnostic turns with
`inconclusive` status. All 33 Provider attempts have no final failure code, but
15 metric results are empty. A successful execution does not establish TiFlash
metric availability or a matched target statement.

The TiFlash Diagnostic R2 requested the cutoff-visible original execution-plan
image, then successfully continued in the same session after the runner supplied
that one image. The continuation took 151.452 seconds and received 735 text
bytes plus one image. This exercises the live Agent9 on-demand image branch,
unlike TCID-13103 where Diag did not request an image. The final answer correctly
reads about 14.2 seconds and 2,957 returned rows, notes that visible operator
times are inclusive, and preserves the missing lower scan/storage section.

The requested SQL was still not matched in Slow Query results. A same-table
count query was correctly rejected as a substitute. Neither the same-target,
same-row-count/different-duration comparison nor the TiFlash CPU rise/recovery
curve was obtained. The different-but-valid Judge result does not satisfy those
specific case criteria. The final answer keeps TiFlash resource, replica, and
MPP scheduling mechanisms unresolved instead of treating empty metrics as health.
An earlier historical run `5f9cb4a4-ec38-45be-87a0-218614c36001` did recover a
target-shaped execution through one-minute Slow Query slices. The current run
did not recover it. This remains a discovery-coverage difference, not proof
that records no longer exist, and that older run is not the strict baseline
used for the timing comparison above.

Summary use is optional and did not shrink every scenario: the single-query
final handoff was 8,112 bytes, but the TiFlash cumulative handoff was 198,450
bytes (baseline: 48,261 and 28,427 bytes respectively), including a 191,384-byte
R2 handoff retaining all 12 query observations.
That round's Diagnostic text input was 213,820 bytes. No claim is made that the
large TiFlash handoff was eliminated merely because TCID-13139 used summaries.

## Runtime cost and the parts not yet verified

TCID-13103's critical scenario took 876.4 seconds. Router selection took 51.7
seconds, ticket synthesis 83.8 seconds, and Final Response 58.3 seconds. Its
Evidence R1/R2 took 261.7/204.6 seconds, whereas their actual Agent9 Managed Tool
operations took 15.1/5.2 seconds. Capacity Evidence took 222.0 seconds with a
5.8-second Tool operation. The `llm.complete` operation encloses child tools;
do not add those durations together. Most elapsed time was outside Provider
execution, including model planning, interpretation, and context/tool overhead.

TCID-13139's four Managed Tool operations took 13.814/10.101 seconds for cluster
R1/R2 and 10.001/7.621 seconds for workload R1/R2: 41.537 seconds total inside
720.478 seconds of accumulated Evidence-turn time. Because the scenarios ran
in parallel, neither sum is the ticket's wall time. This independently points
to substantial non-Provider overhead. The window also narrowed from 60 to 10
minutes and query selection changed; the wall-time decrease is not a strict
benchmark. No `diagnostic_would_timeout` event occurred in this run either, so
its completion cannot be attributed to the observe flag.

TCID-13132's Evidence turns took 196.041/109.777 seconds for single-query R1/R2
and 239.654/137.693 seconds for TiFlash R1/R2. The corresponding Managed Tool
durations were 13.920/11.025 and 10.422/10.534 seconds: 45.901 seconds inside
683.165 accumulated Evidence seconds, again not a wall-time decomposition.
Its additional live-image continuation adds diagnostic work that the baseline
did not perform. There were no `diagnostic_would_timeout` events in this run.

In TCID-13103, the Evidence thread was reused, but R2 still reread Clinic and the shared Skill.
Scenario Diagnostic also reread `op-diagnosis`. The actual R2 Evidence message
explicitly instructed it not to reload; the production Diagnostic prompt likewise
instructs reuse. The available Agent9 model audit redacts the higher-priority
prompt, so the cause of the repeated reads is not established. Do not claim
that Skill-reload overhead was eliminated, or invent a cache/controller fix.
TCID-13132's two Evidence R2 turns did not reread Skills, but its Diagnostic
turns still repeatedly read `op-diagnosis`, including the image continuation.
Evidence reuse on that ticket must not be generalized to every Agent or case.

No `diagnostic_would_timeout` event occurred in TCID-13103. Its completion cannot
be attributed to bypassing an old deadline. Its cumulative workload handoff was
60,213 bytes versus 55,779 in the old replay; no Slow Query was acquired, so this
ticket does not test Agent-generated Slow Query summary compression.

## Private evidence and remaining work

All paths below are operator-local internal artifacts, not public CI uploads:

- TCID-13103: `/home/nizheming/.cache/auto-oncall-services/replays/diagnostic-context-13103-20260909-0017/`.
  Includes `operation-timing-audit.json`, exact run `output/result.json`, frozen
  `output/read-publication.json`, `trace/context-trace.json`, and Provider facts.
- TCID-13139: `/home/nizheming/.cache/auto-oncall-services/replays/tcid-13139-context-efficiency-20260909/`.
- TCID-13132: `/home/nizheming/.cache/auto-oncall-services/replays/tcid-13132-02bdc964-20260909.wKaOKN/TCID-13132/`.
  Live image trace: `trace/codex-diagnostic-scenario-tiflash-io-apply-clinic-diagnostic-round-2-turn-images-1.json`.
- Original local trials, including the failed timezone result:
  `/home/nizheming/.cache/auto-oncall-services/replays/diagnostic-context-local-20260908.FpnNGA/`.
- Baselines: `/home/nizheming/.cache/auto-oncall-services/replays/scenario-split-20260908.hLw7GW/`.

All three runs, frozen publications, summaries, and logs were exported before
their isolated Jobs were removed. None was published to Jira/DB/Base. Target-SQL
identity/contrast, stronger causal evidence, per-task coverage, and repeated
Skill reads remain distinct unresolved findings. No new repair, deployment,
replay retry, publication, or `optimized_verified` update follows automatically
from this report. The 25-minute first response / later final design remains
TODO only.