# Clinic 5xx Recovery and Reporting

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "clinic-5xx-recovery-reporting-2026-08-11",
  "problems": [
    {
      "problemId": "clinic-5xx-was-not-retried-or-reportable",
      "summary": "Clinic HTTP 5xx responses ended a bounded query after one attempt and retained only the status code, so transient gateway failures could not recover and persistent provider failures lacked the details needed for owner communication.",
      "status": "optimized_unverified",
      "issueKeys": ["ST-2863"],
      "sourceRefs": ["https://tidb.atlassian.net/browse/ST-2863#problem:clinic-http-5xx-unretried-unobservable"],
      "signalTextIncludesAny": ["clinic_http_500", "clinic_http_502", "query execution failed", "provider 5xx"],
      "observedThrough": "2026-08-11",
      "effectiveAt": "2026-08-11T05:47:29Z",
      "verifiedAt": null,
      "attemptedAction": "Retry read-only Clinic HTTP 500/502/503/504 queries up to three attempts inside the same batch, retain bounded recovery or exhausted-error metadata, propagate provider system events into execution state, and list them deterministically in the weekly report.",
      "evidenceUrl": "https://github.com/tidbcloud/aisupport/tree/main/auto-oncall-services/docs/opts/2026-08-11-clinic-5xx-recovery-reporting.md"
    }
  ]
}
-->

- Optimization ID: `clinic-5xx-recovery-reporting-2026-08-11`
- Owner: auto-oncall-services
- Source: ST-2863 temporary-pod replays and the 2026-08-11 direct TopSQL reproduction
- Implementation status: deployed to staging, verification pending

## Bounded problem

Clinic 5xx transport/provider failures were attempted once and collapsed to
`clinic_http_<status>`. A transient 502 therefore could not recover, while a
repeated 500 did not retain the safe upstream error code, message, request IDs,
or attempt history needed for escalation.

## Attempted change

- Retry read-only HTTP 500, 502, 503, and 504 responses twice after the initial
  call, within the same provider batch.
- Record successful retry recovery separately from exhausted retry failure.
- Persist only bounded, redacted upstream error fields and request IDs.
- Carry these events into `execution_json` and a deterministic weekly report
  section.

## Current evidence

The same ST-2863 TopSQL request was issued three times on 2026-08-11. All three
returned HTTP 500 with `InternalError: query execution failed`; request IDs were
captured for provider-owner follow-up. Unit tests cover one 502 recovery and one
three-attempt 500 exhaustion. Temporary-pod replay
`868e6f7e-04a2-41ac-ac14-8230566099d8` then verified real three-attempt
exhaustion for both 500 and 502 operations, exact bounded upstream errors and
request IDs, and degraded execution status instead of falsely complete evidence.

## Unresolved gaps

The change is available in staging image `6fcd789dfe896594b405cfdb0b8df7fc50a47fc3`.
Deployed ST-2863 persistence and the next weekly report artifact remain
unverified.

## Next validation action

Rerun ST-2863 through the persisted staging path, then verify the next weekly
report artifact.