# Clinic Evidence Batch Orchestration

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "clinic-evidence-batch-orchestration-2026-08-05",
  "problems": [
    {
      "problemId": "clinic-evidence-agent-reloads-diagnosis-skills-and-serializes-small-queries",
      "summary": "Clinic scenario parents and evidence children wrapped fast provider calls and evidence JSON maintenance in repeated model turns, including manual cross-source recovery and handoff patch loops.",
      "status": "optimized_verified",
      "issueKeys": ["ST-2865", "TCID-12973"],
      "sourceRefs": [
        "https://tidb.atlassian.net/browse/ST-2865#problem:clinic-evidence-serial-agent-latency",
        "https://tidb.atlassian.net/browse/TCID-12973#problem:clinic-evidence-serial-agent-latency"
      ],
      "signalTextIncludesAny": [
        "clinic evidence child model latency dominates provider latency",
        "parent loads clinic-api or diagnosis skills before child spawn",
        "one model turn per metric query",
        "cross_source_metrics_rejected stops the evidence child",
        "child repeatedly patches compact-report or child-evidence-handoff JSON"
      ],
      "observedThrough": "2026-08-05",
      "effectiveAt": "2026-08-05T10:42:00Z",
      "verifiedAt": "2026-08-05T10:53:48Z",
      "attemptedAction": "For Clinic scenarios, keep provider skills out of the parent, have the evidence child invoke one exact-target Clinic query_range_batch helper, move concurrency and cross_source_metrics_rejected recovery into that helper, and have the auto-oncall runner deterministically materialize compact reports and the strict handoff from the generic batch result.",
      "evidenceUrl": "https://tidb.atlassian.net/browse/ST-2865"
    }
  ]
}
-->

- Optimization ID: `clinic-evidence-batch-orchestration-2026-08-05`
- Owner: auto-oncall-services
- Source references: branch-local replays of `ST-2865` and `TCID-12973`
- Implementation status: deployed to staging
- Effective status: `optimized_verified`

## Bounded problem

The provider calls were not the dominant delay. In the measured replays,
Clinic/provider execution consumed seconds to tens of seconds, while the
scenario parent and child consumed minutes in repeated instruction discovery,
planning, small command calls, handoff rewrites, and model inference. The
largest child accumulated dozens of tool calls and millions of cached input
tokens around twelve provider calls whose total execution time was about 22
seconds.

## Implementation

- The Clinic scenario parent must spawn its evidence child immediately and may
  not load `clinic-api`, Clinic subskills, `op-diagnosis`, or another diagnosis
  or provider skill. It reasons only from inline route context and the validated
  child handoff.
- The Clinic evidence child may load exactly the `clinic-api` package and only
  the package subskill instructions required by its predeclared queries. It may
  not load general diagnosis, query-tuning, Wetech, Manus, runner, schema,
  validator, or test instructions.
- The child writes one bounded plan and invokes Clinic's `query_range_batch`
  helper once. The helper uses the exact `orgId`/`clusterId`, runs independent
  queries concurrently, and compacts raw time series under a six-provider-call
  total budget.
- `cross_source_metrics_rejected` recovery belongs to the Clinic helper. It
  executes predeclared source-homogeneous fallbacks or unchanged top-level
  PromQL `or` arms concurrently; the agent does not design a retry after the
  error.
- Clinic emits generic `clinic-metrics-batch-v1`. The auto-oncall runner writes
  compact reports and `auto-oncall-diagnostic-child-evidence-v2` atomically
  from that result. The child no longer writes or patches evidence JSON.
- Optional query-coverage IDs are not a checklist. The child records only
  selected or attempted queries, which avoids running low-value probes merely
  to fill the schema.

## Local verification

- Clinic helper unit tests cover concurrent execution, exact-target use,
  deterministic cross-source recovery, compact summaries, and the six-call
  budget. Auto-oncall tests cover deterministic compact-report/handoff
  materialization for complete, empty, failed, and recovered queries.
- Prompt tests assert the strict parent/child skill boundary, exact Cloud target,
  search prohibition, one helper invocation, and prohibition on agent-owned
  provider retry or evidence JSON patching.
- The existing Wetech and supplied-evidence paths retain their provider-specific
  instruction behavior.
- An isolated staging-pod `mode=read` replay of `ST-2865` on 2026-08-05 used
  run `3ed6a9fd-e511-486d-acaa-720b1e071f1a`. It selected both
  `memory-oom` and `capacity-resource-pressure`; both scenarios and top-level
  evidence acquisition completed without failure codes. The exact trusted
  replay window, org ID, and cluster ID were preserved in both batches and in
  both runner-generated handoffs.
- The two deterministic Clinic batches completed in 0.94 seconds and 3.94
  seconds. Their evidence children completed in about 37 and 31 seconds,
  respectively, without reading `metrics_1_core.md`, `metrics_2_common.md`, or
  `metrics_3_other.md`; each child wrote one built-in recipe plan and invoked
  the helper once. The parent scenario durations were 116 and 96 seconds, and
  the complete two-scenario replay took about 168 seconds end to end.
- Before this change, the comparable two-scenario replay took about 516 seconds
  end to end and the memory evidence child took about 225 seconds. This
  staging-equivalent run therefore reduced total latency by about 67% and the
  measured memory evidence-child latency by about 84%.
- Staging image `4edcfd9abeadd4c2a47ee25c3d8fb9537d630f13` and the
  Clinic skill from `tidbcloud/nutshell-skills` PR `#189` were deployed
  together. The live Clinic helper hash was
  `af5d0cb151117e9f5c1b088d87b855264c7dd0372b48f6bb43c8e27c145331cf`.
- Deployed `mode=write` replay `ST-2865` run
  `06876c15-de4f-46a5-9a7a-6e48d1e766e5` completed both Clinic scenarios,
  all six queries, both handoffs, response generation, and evidence acquisition.
  Its batches took 0.77 and 3.60 seconds. DB run `120016`, Feishu Base, and
  existing private Jira comment `23432370` were updated successfully.
- Deployed `mode=write` replay `TCID-12973` run
  `60f1bb4f-3f2b-4382-ba4e-3f1e2c41ea0f` completed the custom Clinic batch,
  five evidence artifacts, terminal parent decision, response generation, and
  evidence acquisition. DB run `120017`, Feishu Base, and existing private
  Jira comment `23327373` were updated successfully.

## Unresolved gaps

- The helper's automatic `cross_source_metrics_rejected` recovery is covered by
  deterministic tests; the ST-2865 recipes were already source-homogeneous, so
  that provider rejection did not occur in the live query sample.
- APID/OP Wetech evidence remains a separate provider path and is not an
  applicability case for this Clinic optimization.

## Next validation action

Monitor later Clinic regressions for repeated agent-owned provider calls,
`cross_source_metrics_rejected`, or missing runner-generated evidence. A new
matching signal after the verification timestamp is a regression rather than
evidence that this verified problem should be selected again unchanged.
