# Manus metrics timeout recovery

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "manus-metrics-timeout-recovery-2026-07-22",
  "problems": [
    {
      "problemId": "manus-metrics-timeout-discards-provider-evidence",
      "summary": "A wide Manus metric request can time out and cause replay to treat the whole read-only provider as unavailable instead of preserving successful dimensions and retrying one bounded query.",
      "status": "optimized_verified",
      "issueKeys": ["APID-29174"],
      "sourceRefs": [],
      "signalTextIncludesAny": ["manus metrics timeout", "grafana http 504", "read-only metrics backend unavailable", "tikv_scheduler_pending_commands"],
      "observedThrough": "2026-07-21",
      "effectiveAt": "2026-07-22T03:46:05Z",
      "verifiedAt": "2026-07-22T03:57:00Z",
      "attemptedAction": "Serialize Manus metric reads, use narrow instance-scoped windows, retry one timeout or HTTP 504 over the centered half-window, preserve successful evidence, and use tikv_scheduler_contex_total for pending commands.",
      "evidenceUrl": "https://tidb.atlassian.net/rest/api/3/issue/7895053/comment/23147378"
    }
  ]
}
-->

- Optimization ID: `manus-metrics-timeout-recovery-2026-07-22`
- Owner: auto-oncall-services
- Source type: Focus #375 replay diagnosis
- Source reference: `APID-29174` baseline and current-main replay evidence
- Bounded problem: one wide or high-cardinality metric query can return a
  timeout/HTTP 504, after which the analysis may discard the entire Manus
  provider and miss already available scheduler evidence.

## Status

- Implementation: merged in `tidbcloud/aisupport#445`.
- Effective/deployment: staging image
  `d6bdef16a5f6be29ed5211e6ea75b21041fe13ff`, rolled out through
  `tidbcloud/app-delivery#75165` at 2026-07-22 03:46 UTC.
- Problem status: `optimized_verified` for provider-level timeout recovery.

## Baseline evidence

- The historical replay retained only `TimeoutError` / HTTP 504, not the exact
  PromQL, so query concurrency cannot be proven as the cause.
- Current `vida` health, datasource, Dashboard, and credentials are available.
- Equivalent 20-minute, one-metric, instance-filtered reads complete and expose
  scheduler command backlog, latch/command latency, and flow control evidence.
- The supported pending scheduler command metric uses the historical exported
  spelling `tikv_scheduler_contex_total`; the previously attempted
  `tikv_scheduler_pending_commands` name is absent.

## Post-change verification

- Python unit tests passed 19/19, the skill package passed `quick_validate.py`,
  and the repository typecheck and full Node test suite passed locally and in
  PR CI.
- The live Deployment generation 186 and Recent Optimizer CronJob both rendered
  image `d6bdef16...`; the Deployment reached 1/1 Ready. The pod contained the
  retry guard and corrected metric guidance.
- In-pod `health --target vida` authenticated to Dashboard and resolved the
  exact Grafana datasource `vida`.
- The APID-29174 20-minute incident window query, filtered to the reported host,
  returned one `tikv_scheduler_contex_total` series with 41 points and a peak of
  2,882 commands on its first attempt.
- Historical reply replay run `24558b72-eda6-44e2-8817-c0478b57679b` used the
  cutoff at comment `23147640` and returned an evidence-backed scheduler queue
  explanation instead of classifying the Manus provider as unavailable.
- Write replay DB run `60080` (`5ba9c694-79af-4e25-b211-64ae2f0c2a2d`)
  updated the existing Jira internal comment `23147378` via the stable key
  `auto-live:v1:APID-29174:initial`; API readback confirmed it remains internal
  (`public=false`) and contains the 2,882-command evidence.

## Unresolved gaps and next action

- This optimization verifies only bounded metrics recovery and preservation of
  provider evidence. The replay still did not prove the initiating L0 flow
  control, write-hotspot, SQL, or storage mechanism, so those remain separate
  unverified problem signals and are not promoted by this record.
- If a future case reproduces a timeout, retain the successful dimensions and
  inspect the `recovery` metadata to confirm that only the timed-out metric was
  narrowed and retried.