# Incident Entity-Time Alignment

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "incident-entity-time-alignment-2026-07-27",
  "problems": [
    {
      "problemId": "incident-entity-selection-uses-wide-proxy-maximum",
      "summary": "A mechanism diagnosis can select its entity from a wider-window CPU or resource maximum instead of the direct result at the incident sample, then incorrectly rule out the actual mechanism on the wrong entity.",
      "status": "optimized_verified",
      "issueKeys": ["APID-29231"],
      "sourceRefs": [
        "https://tidb.atlassian.net/browse/APID-29231#problem:diagnostic_regression:wide-proxy-max-selects-wrong-incident-entity"
      ],
      "signalTextIncludesAny": [
        "wide cpu maximum selects diagnostic entity",
        "direct result entity and incident sample are not aligned",
        "mechanism ruled out on the wrong instance",
        "mechanism objective runs unnecessary topsql"
      ],
      "observedThrough": "2026-07-27",
      "effectiveAt": "2026-07-27T09:52:53.000Z",
      "verifiedAt": "2026-07-27T11:35:44.882Z",
      "attemptedAction": "Lock a canonical direct-result entity and narrow incident interval before mechanism exploration; when the entity is unknown, select it from the direct result at the event peak or change sample rather than a resource proxy. Keep mechanism-only diagnosis off workload surfaces unless one load-bearing workload gap can change the conclusion, require the aligned scheduler command and latch chain plus one strongest storage alternative, and require a structured current-read-only follow-up when an unconfirmed root cause still has an outcome-changing provider gap.",
      "evidenceUrl": "https://tidb.atlassian.net/browse/APID-29231?focusedCommentId=23279460"
    }
  ]
}
-->

- Optimization ID: `incident-entity-time-alignment-2026-07-27`
- Owner: auto-oncall-services
- Source reference: `APID-29231`
- Implementation status: deployed to staging
- Effective status: `optimized_verified`

## Bounded problem

A direct-result incident may have a narrow event time but no explicitly bound
entity. Selecting the diagnostic entity from a wider-window CPU maximum,
dashboard legend, or workload rank can choose an unrelated instance. Every
later scheduler, latch, storage, and workload query may then be internally
consistent while answering the wrong causal path.

This record owns incident entity/time selection and the immediate mechanism
path. It does not claim to identify every workload trigger or replace the
separate workload-attribution optimization.

## Regression evidence

Read-only historical replay `fb31868a-9e2f-4500-8f86-43a364bf499b` completed
in 358.997 seconds. It used the broad 02:25–03:34 CPU maximum to select TiKV
`10.107.42.119:20180`, then queried scheduler command latency, latch wait, and
TopSQL only on that store. Those checks returned roughly 20–22 ms scheduler
latency, about 36 microseconds of Commit/Prewrite latch wait, and zero TopSQL
rows, so the result incorrectly stated that latch or lock contention could not
explain the incident.

Earlier aligned evidence locates the direct Commit/Prewrite result and
second-scale scheduler latch/lock mechanism on a different TiKV store in the
narrow event interval. The regression therefore came from entity selection,
not from a missing metric or a valid causal-magnitude rejection. The run also
left the root cause unconfirmed while setting
`diagnosticFollowup.required=false`.

The replay was analysis-only. It did not write Jira, database, Feishu, provider
configuration, or production state.

## Candidate change

- Lock a canonical direct-result tuple: result, entity, component, narrow
  interval, timezone, and query step.
- Reuse a reliable artifact-provided entity only when it is bound to the direct
  result at the incident sample.
- When the entity is absent, locate the direct-result peak or change point
  first and select the entity at that sample or within one query step.
- Prohibit wider-window CPU/resource maxima, dashboard legends, and TopSQL rank
  from selecting or replacing the entity before direct-result alignment.
- Keep TopSQL, Statements, and Slow Query off mechanism-only paths unless
  workload attribution is required or one explicit load-bearing workload gap
  can change the conclusion.
- For write-path mechanism diagnosis, require the aligned client/TiKV result,
  scheduler command and Commit/Prewrite latch chain, one strongest storage
  alternative, and magnitude comparison. CPU remains pressure and cumulative
  `ANALYZE` rank remains non-causal.
- If the final root cause remains unconfirmed and an available read-only check
  can change the leading outcome, require a structured diagnostic follow-up.

## Verification status

Local contract validation passed on 2026-07-27:

- `op-diagnosis` and `tidb-latency-diagnosis` both passed `quick_validate.py`.
- Focused contract tests passed: 58/58.
- The full repository test suite passed: 448/448.
- TypeScript typecheck and `git diff --check` passed.

The staging runtime started the deployed image at
`2026-07-27T09:52:53.000Z`. Deployed read replay
`6b72bd78-74c6-42a0-b3da-b8f01050f0a1` completed with `ai_better`, selected
the correct store `10.107.3.2:20180`, found second-scale Commit, Prewrite,
scheduler, and latch evidence, kept Raft and CPU at their supported causal
strength, and did not query TopSQL for this mechanism-only objective.

Deployed write replay `ed2b8958-bcde-4e01-8937-8d0cc8d0f076` updated the
existing Jira internal comment `23279460` in place. Jira readback verified the
exact comment ID and updated body.

## Unresolved gaps

- Workload identity remains outside this mechanism-only problem and must not be
  inferred from the entity-alignment result.

## Next validation action

Monitor later matching signals as regressions. Keep workload attribution and
any remaining lower-level trigger as separate problem signals rather than
promoting the entire ticket.