# Clinic Metrics Source Sample Budget

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "clinic-metrics-source-sample-budget-2026-08-13",
  "problems": [
    {
      "problemId": "clinic-metrics-victoriametrics-sample-limit-hidden-as-retryable-502",
      "summary": "Clinic translated VictoriaMetrics HTTP 422 maximum-sample rejections into HTTP 502, so Auto Oncall retried unchanged high-source-sample range queries three times and still lost the load-bearing metric evidence.",
      "status": "optimized_verified",
      "issueKeys": ["TCID-12986"],
      "sourceRefs": ["https://tidb.atlassian.net/browse/TCID-12986#problem:clinic-metrics-source-sample-budget"],
      "signalTextIncludesAny": ["metrics upstream request failed", "clinic_http_502", "maximum samples", "VictoriaMetrics 422"],
      "observedThrough": "2026-08-13",
      "effectiveAt": "2026-08-13T13:10:15Z",
      "verifiedAt": "2026-08-13T13:18:39Z",
      "attemptedAction": "Prevent whole-cluster tidb_tikvclient_request_seconds queries. Require a focused sub-window plus an exact known TiDB instance or TiKV store selector before provider execution, otherwise make the same Evidence Agent choose a lower-cardinality counterpart.",
      "evidenceUrl": "https://github.com/tidbcloud/aisupport/tree/main/auto-oncall-services/docs/opts/2026-08-13-clinic-metrics-source-sample-budget.md"
    }
  ]
}
-->

- Owner: auto-oncall-services Evidence Agent and Clinic Provider Executor
- Source: TCID-12986 replay `3c5a266b-e0f4-48a5-992f-b46b6e20867a`
- Implementation status: deployed to staging in image `6dbef26ce03dbf813f6801d1b7fc74128dda1975`
- Effectiveness status: optimized_verified for this bounded sample-budget problem

## Bounded problem

The mean and p99 `tidb_tikvclient_request_seconds` range queries covered the
whole one-hour cluster window. This metric describes TiDB-to-TiKV requests, so
its series count grows with both component node sets. VictoriaMetrics rejected
the resulting source scan, Clinic surfaced it as HTTP 502, and the Provider
Executor retried each unchanged request three times. Final aggregation and
`topk` bound returned series but do not avoid scanning the underlying node-pair
series. Increasing `step` can also hide short spikes without addressing that
fan-out.

## Candidate change

- State the source-sample versus output-cardinality distinction in the Clinic
  Evidence Agent contract.
- For the observed TiDB-to-TiKV request bucket/sum/count family, prohibit a
  whole-cluster query. Require a sub-window of at most 15 minutes and an exact
  known `instance` or `store` equality selector on every referenced family.
- Preserve useful time resolution. A larger `step`, `type` alone, `store!="0"`,
  a broad regex, and output aggregation do not satisfy the source-scope bound.
- Reject a riskier plan before the Provider call and return the exact plan error
  to the same Evidence Agent thread. Without a known TiDB or TiKV endpoint, the
  Agent must choose a lower-cardinality counterpart.

## Verification evidence

The initial step-based implementation was superseded after review because a
coarse step loses useful detail. On 2026-08-13, the replacement source-scope
implementation passed 175/175 focused tests and 804/804 full-suite tests. The
tests prove that whole-cluster, endpoint-unbound, and wider-than-15-minute
queries fail before a Clinic call; short exact-instance/store queries retain a
60-second step and execute with their requested sub-window; ordinary metrics
remain unchanged; and the same Evidence Agent receives the exact repair.

PR [#890](https://github.com/tidbcloud/aisupport/pull/890) was deployed through
[staging run 31703004214](https://github.com/tidbcloud/aisupport/actions/runs/31703004214)
and [app-delivery #77254](https://github.com/tidbcloud/app-delivery/pull/77254).
The live Pod started at `2026-08-13T13:10:15Z` with the merge-SHA image, Ready,
and zero restarts.

TCID-12986 historical replay `5fd16fb1-d604-4182-a53b-419fbe075daa` completed at
`2026-08-13T13:18:39Z`. It made no HTTP 502 or VictoriaMetrics 422 attempt and
did not repeat the whole-cluster mean or p99
`tidb_tikvclient_request_seconds` queries. The query-latency first round instead
used two 15-minute lower-cardinality coprocessor, backoff, and gRPC sum/count
windows; all 13 HTTP attempts in that round returned 200. The bounded
TiKV-request-type report also completed. This verifies the selected
sample-budget problem, not the whole ticket.

## Unresolved gaps

Clinic currently hides the VictoriaMetrics 422 as a generic 502. Auto Oncall
therefore applies the preflight only to the concretely observed expensive
families and does not infer that every Clinic 502 is a sample-budget error.

The same replay remained `evidenceAcquisition=degraded`: two Statements queries
returned `clinic_datasource_not_configured` through HTTP 400, and two TopSQL
queries each exhausted three HTTP 500 attempts. The final answer was
`human_required`. These are separate provider problems and do not invalidate
the verified absence of the original maximum-sample queries.

## Next validation action

After the Clinic Statements and TopSQL failures are deployed or otherwise
resolved, replay TCID-12986 again. Only if the whole replay is no longer
degraded, continue with two additional historical Clinic cases.