# Evidence Acquisition Objective And Recovery

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "evidence-acquisition-recovery-2026-08-11",
  "problems": [
    {
      "problemId": "evidence-command-success-without-acquisition-completion",
      "summary": "The Evidence Agent could stop after runnable metric queries without a workload discovery result, while Wetech Slow Query request-shape and transient HTTP failures were materialized without bounded in-batch recovery.",
      "status": "optimized_verified",
      "issueKeys": ["APID-29296"],
      "sourceRefs": [
        "https://tidb.atlassian.net/browse/APID-29296#problem:evidence-acquisition-incomplete"
      ],
      "signalTextIncludesAny": [
        "slow_query_failed common.bad_request",
        "fields must be a non-empty string",
        "successful query count treated as evidence completion",
        "TopSQL and Slow Query were not used for workload discovery"
      ],
      "observedThrough": "2026-08-11",
      "effectiveAt": "2026-08-11T09:28:05Z",
      "verifiedAt": "2026-08-11T10:01:42Z",
      "attemptedAction": "Give the Evidence Agent an observable acquisition completion target, normalize fields arrays at the Wetech Provider Executor boundary, retry one transient Dashboard 5xx, and repair one Slow Query common.bad_request with a changed minimal projection inside the same batch.",
      "evidenceUrl": "https://github.com/tidbcloud/aisupport/tree/main/auto-oncall-services/docs/opts/2026-08-11-evidence-acquisition-recovery.md"
    }
  ]
}
-->

- Optimization ID: `evidence-acquisition-recovery-2026-08-11`
- Owner: auto-oncall-services Evidence Agent and Wetech Provider Executor
- Source reference: `APID-29296` staging replay `08e4e184-6135-46de-bbad-b2d4a23d259f`
- Implementation status: deployed to staging in image `184f890885ff352d16731ea9d5e660ab557c044a`
- Effective status: `optimized_verified`

## Bounded problem

The Evidence Agent had general observable-first guidance but no explicit
completion condition distinguishing “a query ran” from “the evidence objective
was acquired.” In the observed replay, Slow Query returned
`common.bad_request`, later plans supplied `fields` as arrays, and no usable
Slow Query or TopSQL workload evidence reached diagnosis.

This record owns acquisition-objective clarity and deterministic recovery at
the Wetech query boundary. It does not move diagnosis into the Evidence Agent,
authorize provider fallback, broaden the target/window, or consume a Scenario
Diagnostic Agent refinement round for transport work.

## Candidate change

- Require an aligned impact/mechanism fact and, when applicable, a stable
  workload candidate from TopSQL, Slow Query, or Statements; otherwise record
  the exact supported-surface limitation.
- Tell the Evidence Agent to include already-known alternative evidence
  surfaces in the current batch instead of waiting for a customer Digest.
- Declare `fields` as a comma-separated string and deterministically normalize
  a non-empty string array before provider validation.
- Retry one Dashboard HTTP 500/502/503/504 response in the helper.
- On Slow Query `common.bad_request`, issue one changed minimal projection
  without optional sorting, then materialize the final result or exact error.

## Verification evidence

Focused TypeScript tests cover the observable completion prompt and plan field
contract. Wetech helper tests cover string-array normalization, one transient
5xx retry, and the changed Slow Query request after `common.bad_request`.

The staging replay for `APID-29296` completed as run
`33d54cec-e4fb-4faf-9cd1-2735c11d78f9` (DB run `120076`) on
2026-08-11. Slow Query requests used the normalized comma-separated `fields`
shape and returned 100 records per queried segment without
`common.bad_request`. After the first TopSQL query was blocked by an empty
runtime-p99 prerequisite, the Evidence Agent refined the plan and acquired 101
TopSQL records plus four 100-record Slow Query segments. Later refinement
confirmed 29-42 second outliers and one 294.252-second successful request, and
materialized the exact remaining limitation: the available identity and phase
projections could not bind the extreme request to a stable workload or measured
execution subphase. Evidence acquisition completed with provider capability
verified and no failure codes. Existing Jira comment `23399342` was updated in
place, and a Jira Service Management readback confirmed `public=false`.

## Unresolved gaps

- The evidence path is repaired, but workload attribution remains unresolved:
  TopSQL aggregate groups could not be linked to the 294.252-second request.
- The available slow-query phase fields did not account for the elapsed time,
  so the operator-versus-connection delay mechanism remains unknown.
- The unscoped runtime-p99 series conflicts with the dashboard report and still
  requires metric-definition, scope, aggregation, and display-chain review.

## Next validation action

Treat workload identity and execution-phase attribution as a separate diagnosis
gap. On recurrence, capture a timestamp-aligned sanitized statement identity
and operator/connection phase breakdown instead of repeating the repaired
request-shape work.
