# Transient Fault Domain Evidence Gate

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "transient-fault-domain-evidence-gate-2026-07-28",
  "problems": [
    {
      "problemId": "transient-infra-fault-domain-discrimination",
      "summary": "Transient backoff, disconnect, node-hang, leader-change, and Region-movement incidents can be routed as direct answers or synthesized from one favored hypothesis without independently checking host/process, component-network, and scheduler/leadership evidence.",
      "status": "optimized_verified",
      "issueKeys": ["APID-29220", "APID-29230"],
      "sourceRefs": [
        "auto-oncall-report:staging:daily:2026-07-22:case:APID-29220#problem:ai_gap:477d7bce7aee11a3",
        "auto-oncall-report:staging:daily:2026-07-23:case:APID-29230#problem:ai_gap:ad2b5d8d496909cc"
      ],
      "signalTextIncludesAny": [
        "主机存活",
        "组件间网络可达性",
        "调度活动",
        "故障域判别",
        "第三方网络结论",
        "leader 迁移"
      ],
      "observedThrough": "2026-07-28",
      "effectiveAt": "2026-07-28T04:34:51Z",
      "verifiedAt": "2026-07-28T04:54:46Z",
      "attemptedAction": "Apply canonical auto-coverage normalization to default replay workspaces, then enforce a deterministic publication guard that requires complete entity/time-aligned host/process, component-network, and scheduler/leadership rows plus a compact Jira-draft status summary before a transient incident result is accepted.",
      "evidenceUrl": "https://github.com/tidbcloud/aisupport/issues/500"
    }
  ]
}
-->

- Optimization ID: `transient-fault-domain-evidence-gate-2026-07-28`
- Owner: auto-oncall-services
- Source: Recent Focus #500
- Implementation status: deployed to staging on image
  `8d7b058d6d3732cc680133e6a896faa2b9c84ab9`
- Effective status: `optimized_verified`

## Bounded problem

An explicit TiDB/TiKV runtime alert can be mislabeled as a provider-free
direct-answer scenario. Even when provider diagnosis runs, one plausible
network or scheduling hypothesis can suppress the other transient fault
domains. This produces a confident root-cause statement without a consistent
host/process, component-network, and scheduler/leadership comparison.

## Implementation

- Correct a non-operational, non-backup classification when the visible ticket
  has explicit TiDB runtime-incident and component signals.
- Require a three-row fault-domain matrix on one locked entity/window. Every
  row must be `supported`, `contradicted`, or `unverified`, with direct
  evidence, counter-evidence, and one bounded next check.
- Reuse the incident anchor and compact summaries, with a six-command matrix
  cap and a two-command per-domain cap after readiness; prohibit broad-plus-
  exact duplicate discovery after the entity/window lock.
- Apply canonical auto-coverage normalization in default replay workspaces,
  closing the path that left runtime incidents in provider-free direct-answer
  mode when no external classifier was injected.
- Add a deterministic final publication guard. A transient result is accepted
  only when its internal evidence record has all three complete matrix rows
  and its Jira draft carries the three compact statuses; a correction may mark
  missing evidence `unverified` but may not invent it.
- Validate the corrected result and fail closed when any domain, aligned
  window, evidence boundary, counter-evidence, or bounded next check is
  omitted.
- Preserve customer or third-party conclusions as hypotheses unless the
  current replay independently observes aligned evidence.

## Verification evidence

- Repository CI passed all 475 tests for the final guard-boundary change.
- The rendered staging Deployment and Recent Focus CronJob both used image
  `8d7b058d6d3732cc680133e6a896faa2b9c84ab9`. The running pod reported
  publication guard `60000`, diagnostic multiturn enabled, staging automation
  active, polling enabled, and customer reply disabled.
- `APID-29230` post-deployment write replay
  `401adb70-1f86-4007-b84c-185492fe8661` updated stable Jira internal comment
  `23276484`. Its aligned matrix reported `host_process=unverified`,
  `component_network=supported`, and `scheduler_leadership=unverified`; it
  preserved the observed QPS impact while leaving the deeper cause
  unconfirmed.
- `APID-29220` final write replay
  `c1969803-ef8c-4d3b-af0a-b5cd72f4405d` updated stable Jira internal comment
  `23262615`. It kept the AWS network conclusion as an unverified hypothesis,
  did not promote Region leader movement to a trigger, and did not recommend
  applying the proposed production parameters without human review. Runtime
  trace confirmed the legacy-path publication guard used `timeoutMs=60000`
  and completed in 45,171 ms.
- Both Jira comments were read back by comment ID after the write.
- Final fresh baseline on current main auto-closed Recent Focus #500 with
  `findingStatus=criteria_met`: both target cases passed, no case failed, and
  no optimization PR was created.

## Unresolved gaps

- The available historical evidence still does not prove the deeper trigger
  for either incident; the optimization intentionally records those domains as
  `unverified` instead of inventing a root cause.
- A later problem signal remains eligible if newer case evidence reproduces an
  incorrect fault-domain promotion.

## Next validation action

Keep this problem suppressed only for matching stale signals. If a newer
answer again promotes host/process, component-network, or
scheduler/leadership evidence across domains without an aligned direct signal,
select that new problem ref as a regression and require a fresh replay.
