# Observable evidence discovery

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "observable-evidence-discovery-2026-08-11",
  "problems": [
    {
      "problemId": "missing-ideal-identifier-blocks-provider-discovery",
      "summary": "Evidence acquisition treats a missing Digest, heap profile, or OOM artifact as a prerequisite instead of first using available metrics, TopSQL, Slow Query, and Statements to discover the affected workload.",
      "status": "optimized_unverified",
      "issueKeys": ["APID-29296"],
      "sourceRefs": [
        "https://tidb.atlassian.net/browse/APID-29296#problem:missing-ideal-identifier-blocks-provider-discovery"
      ],
      "signalTextIncludesAny": [
        "missing digest blocks topsql discovery",
        "asks for oom evidence before querying available workload signals",
        "does not split instance metrics before slow query discovery"
      ],
      "observedThrough": "2026-08-11",
      "effectiveAt": "2026-08-11T05:47:29Z",
      "verifiedAt": null,
      "attemptedAction": "Give every Evidence Agent the same observable-first planning rule: use supported metrics to lock component, instance, and event window, then use available TopSQL, Slow Query, or Statements to discover workload candidates before requesting an unavailable ideal identifier or artifact.",
      "evidenceUrl": null
    }
  ]
}
-->

- Optimization ID: `observable-evidence-discovery-2026-08-11`
- Owner: auto-oncall-services diagnostic orchestration
- Source reference: `APID-29296` answer-version feedback
- Implementation status: deployed to staging
- Effective status: `optimized_unverified`

## Bounded problem

The affected case reported transient duration and memory signals, but the
diagnosis asked for OOM, backlog, disk, network, and customer-supplied identity
instead of first splitting the visible event by component, instance, and time.
The human investigation found one load-bearing `ANALYZE` candidate by comparing
TiDB instances and sorting Slow Query records by memory, and separately
identified a later write-side event.

An exact Digest or heap artifact can be required for final causal confidence
without being a prerequisite for provider discovery. Missing ideal evidence
must not stop supported metrics, TopSQL, Slow Query, or Statements acquisition.

## Candidate change

- Put one provider-neutral observable-first rule in the common Evidence Agent
  planning prompt used by Clinic, Wetech, and Manus.
- Treat the customer-proposed component and cause as hypotheses; first lock the
  observed component, instance, and bounded event window.
- Use provider-supported workload surfaces to discover candidates before
  requesting a Digest or other ideal artifact.
- Preserve exact provider failure, retention, unsupported-operation, empty, or
  top-N limitations rather than claiming no workload or blindly retrying.

## Verification evidence

Implemented in the shared Evidence Agent contract for Clinic, Wetech, and
Manus. On 2026-08-11, the `APID-29296`-shaped provider regression passed, and
the complete TypeScript suite passed with 723 tests. This is local contract
evidence only: availability to auto-oncall and a case-specific post-change
replay are still required before promotion beyond `candidate`.

An isolated, automation-disabled staging Pod then ran `APID-29296` in
`mode=read` as run `5fec418f-1e22-4a87-9e07-20d12c066c13`. The Evidence Agent
planned instance-labelled duration, TiKV memory and resource metrics plus a
bounded Slow Query discovery query, showing that the common observable-first
instruction was applied. Provider execution did not occur because the frozen
ticket had no exact cluster target and the Scenario Router returned
`anchor.entity=null`. The attached dashboard visibly named the cluster, but
that image-derived target was not admitted into the canonical provider target.
The final draft therefore remained unimproved and requested customer context.
No Jira, DB, or Feishu write occurred.

Staging replay `34eff525-726b-4f5f-b0b0-c9c6325a6d6b` on image
`6fcd789dfe896594b405cfdb0b8df7fc50a47fc3` then executed the observable-first
plan against `wetech:middle-east-prod:SaaS`. Both scenarios completed their
instance-labelled Prometheus queries. The first Dashboard workload query failed,
and the later Slow Query, Statements, and TopSQL operations failed fast with
`wetech_dashboard_session_unavailable`, so workload attribution and the final
answer remained incomplete.

## Unresolved gaps

- TopSQL does not guarantee complete workload identity and may be retention or
  top-N limited; the final causal boundary must remain unchanged.
- The image-derived target and Prometheus provider execution now work in the
  shared staging runtime, but Dashboard workload discovery remains unavailable.
- No Jira refresh has been performed.

## Next validation action

Restore the Dashboard workload path, rerun `APID-29296`, and verify that Slow
Query or TopSQL narrows the workload before refreshing the existing Jira
internal comment.