# Historical Incident Root-Cause Continuation

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "historical-incident-root-cause-2026-07-23",
  "problems": [
    {
      "problemId": "historical-incident-recovery-stops-root-cause-diagnosis",
      "summary": "Auto-oncall can treat a recovered alert metric as a sufficient preliminary conclusion even when the customer is asking why the historical alert fired.",
      "status": "optimized_verified",
      "issueKeys": ["APID-29215"],
      "sourceRefs": [
        "https://tidb.atlassian.net/browse/APID-29215#problem:ai_gap:recovery-treated-as-root-cause"
      ],
      "signalTextIncludesAny": [
        "指标已回落",
        "当前已经恢复",
        "没有继续超过告警阈值",
        "未继续定位历史告警根因"
      ],
      "observedThrough": "2026-07-23",
      "effectiveAt": "2026-07-23T08:22:04Z",
      "verifiedAt": "2026-07-23T08:32:46Z",
      "attemptedAction": "Require root-cause-oriented incident analysis to distinguish current recovery from the historical trigger and continue with bounded metrics and existing ticket evidence instead of treating recovery as the root cause; automated component-log search remains disabled.",
      "evidenceUrl": "https://tidb.atlassian.net/browse/APID-29215?focusedCommentId=23257106"
    }
  ]
}
-->

- Optimization ID: `historical-incident-root-cause-2026-07-23`
- Owner: auto-oncall-services and external `wetech-op-grafana` skill
- Source reference: `APID-29215`
- Implementation status: effective in staging
- Effective status: `optimized_verified` for the bounded recovery-short-circuit
  problem; the deeper incident cause remains unresolved

## Bounded problem

The initial analysis correctly detected the historical large-Region alert and
its later metric recovery, but stopped before querying the component logs that
could explain why the Region remained large. Recovery answers the current
state question only; it does not answer a customer's historical root-cause
question.

The bounded optimization requires an incident RCA to keep the historical
trigger, current state, and causal mechanism separate. When the customer asks
why an alert fired, a recovered metric is not a stop condition. The Agent
should continue with bounded, time-aligned metrics and existing ticket
evidence, then report only the mechanism supported by that evidence. Automated
component-log search is not part of this path because the available Dashboard
task-group implementation can cause unsafe PD memory growth and no configured
Wetech cluster exposes a usable Loki datasource.

## Verification evidence

- Auto-oncall build and full local suite passed on 2026-07-23: 412 tests,
  50 suites, 0 failures.
- At the original verification point, the external `wetech-op-grafana` Loki
  transport passed 22 unit tests and the skill structural validator, but no
  Wetech cluster exposed a usable Loki datasource.
- A live read-only datasource check covered both configured `java-back`
  Grafana entries. Neither exposed a Loki datasource, and the bounded Loki
  command failed closed before issuing a log query.
- Staging pod `auto-oncall-services-667d78978-v6f8z` became effective at
  2026-07-23T08:22:04Z on image `c5daf000fece745405f059dc6e6bf1d638723e5c`,
  a descendant of implementation commit
  `fc776d1f47af340c3d7777f276820d6a577e9966`. The live pod contained both the
  historical-RCA contract and the matching `browser-query-logs` transport.
- The `APID-29215` historical-cutoff `mode:write` replay completed as run
  `c4587e27-a591-4ec8-8a05-8e96feb2aadf` / DB run `60105`. It re-established
  the historical alert window, checked time-aligned write-throughput metrics,
  attempted one bounded Loki query, and kept the deeper cause unresolved when
  no Loki datasource was available. Recovery was explicitly treated as current
  state rather than root cause.
- Jira internal comment `23257106` was updated in place under
  `auto-live:v1:APID-29215:initial`. The comment remained `public=false`; the
  issue stayed at 9 total comments and 6 public comments.
- The write replay persisted evaluation `ai_better` and refreshed the ticket
  Base projection at 2026-07-23T08:32:46Z.
- A later safety review found that all 25 configured Wetech clusters lacked a
  Loki datasource mapping and the only real `browser-query-logs` attempt
  failed before querying logs. Dashboard task-group log search was also judged
  unsafe because it can cause PD memory growth.
- At `2026-07-23T10:54:19Z`, staging deployment generation `208` installed the
  safety revision in pod `auto-oncall-services-5fd66bc499-h7f6r`. The live
  skill exposes neither `browser-query-logs` nor `dashboard-query --kind logs`;
  metrics, TopSQL, Statements, and Slow Query remain available.

## Unresolved gaps

- Component-log search is intentionally disabled. The current `java-back`
  Grafana inventory does not expose a Loki datasource, and the Dashboard
  task-group implementation is not safe for automated use.
- The replay therefore verified the improved diagnostic behavior, not the
  alert's deeper causal mechanism. No specific Region mapping or Region-state
  lookup is required by this optimization.

## Next validation action

Keep component-log search disabled. For similar historical alerts, continue
past current recovery using narrow time-aligned metrics and existing ticket
evidence, and label the result as suspected unless the complete causal chain is
proved. Revisit logs only after an explicitly approved provider-side design
isolates PD from query memory risk.