# Write-Path Scheduler Latch Evidence

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "write-path-scheduler-latch-evidence-2026-07-24",
  "problems": [
    {
      "problemId": "write-latency-skips-scheduler-latch-mechanism",
      "summary": "After locating aligned Commit or Prewrite latency on one TiKV, auto-oncall can stop at gRPC latency and CPU or move to workload ranking without querying the direct scheduler latch-wait mechanism.",
      "status": "candidate",
      "issueKeys": ["APID-29231"],
      "sourceRefs": [
        "https://tidb.atlassian.net/browse/APID-29231#problem:ao_feedback:scheduler-latch-mechanism-omitted"
      ],
      "signalTextIncludesAny": [
        "未查询 scheduler latch wait",
        "停在 commit prewrite grpc 和 cpu",
        "缺少写侧 latch contention 机制证据"
      ],
      "observedThrough": "2026-07-24",
      "effectiveAt": null,
      "verifiedAt": null,
      "attemptedAction": "When Commit or Prewrite client latency and its TiKV gRPC counterpart align on one store, require one bounded scheduler latch-wait histogram query for those command types on that exact store and incident window before generic CPU or workload ranking.",
      "evidenceUrl": "https://tidb.atlassian.net/browse/APID-29231"
    }
  ]
}
-->

- Optimization ID: `write-path-scheduler-latch-evidence-2026-07-24`
- Owner: auto-oncall-services
- Source reference: `APID-29231`
- Implementation status: implemented on the causal-alignment branch; not
  deployed
- Effective status: `candidate`

## Bounded problem

The aligned historical replay found the correct TiKV and Commit/Prewrite
interval, but then used process CPU and TopSQL context without querying the
write-side scheduler latch mechanism. That left the direct cause of the
Prewrite tail latency untested.

This change adds one mechanism-triggered check. When client Commit or Prewrite
latency and the TiKV gRPC counterpart align on one store, diagnosis queries the
scheduler latch-wait histogram for only those command types, on that store and
incident window, before generic resource or workload evidence. It does not add
a scheduler metric inventory or a generic storage sweep.

## Implementation evidence

- `skills/tidb-latency-diagnosis/references/storage-latency.md` names the
  aligned Commit/Prewrite trigger and the bounded
  `tikv_scheduler_latch_wait_duration_seconds_bucket` check.
- `skills/op-diagnosis/SKILL.md` keeps that mechanism ahead of CPU and workload
  ranking and prohibits substituting indirect proxies.
- The OP contract also closes the unsafe legacy-log ambiguity: when a selected
  transport says component logs are disabled, no legacy log command or
  Dashboard log task-group may be invoked.
- Contract tests cover the trigger, exact entity/window scope, direct metric,
  anti-inventory boundary, and log-disable boundary.

## Read-only provider evidence

At `2026-07-24T04:04:19Z`, one Wetech Prometheus query was run against cluster
`SaaS02`, datasource `wQrYqYMvk`, store `10.107.3.2:20180`, and the historical
five-minute interval `1784834700` through `1784835000` with a 15-second step:

```promql
histogram_quantile(
  0.99,
  sum(
    rate(
      tikv_scheduler_latch_wait_duration_seconds_bucket{
        instance="10.107.3.2:20180",
        type=~"(?i)commit|prewrite"
      }[1m]
    )
  ) by (le, instance, type)
)
```

The query returned two series. Prewrite p99 rose above `1s`, peaking at
`1.113858591s` at epoch `1784834820`, while Commit p99 peaked at
`0.000079376s` at the same point. The Prewrite latch spike overlaps the
previously observed client and TiKV gRPC Commit/Prewrite incident samples.
Evidence is stored outside the repository at
`/tmp/apid-29231-scheduler-latch/metrics.json` and
`/tmp/apid-29231-scheduler-latch/workspace_evidence_report.json`.

This provider check confirms that the metric is available and material for the
case. It does not make the branch change effective in auto-oncall and therefore
does not promote the problem beyond `candidate`.

## Branch-local historical replay

One and only one branch replay was run for `APID-29231` from
`2026-07-24T04:10:00Z` through `2026-07-24T04:15:25Z`:

- Run ID: `cbf5d129-1abe-4a70-b599-b0f7efa68b10`
- Source: job `90007`, run DB id `90003`
- Cutoff: `2026-07-23T19:46:48.509Z`
  (`first_run_created_at`)
- Workspace:
  `/tmp/auto-oncall-causal-alignment-validation.JkhwiT/runs/cbf5d129-1abe-4a70-b599-b0f7efa68b10`
- Core output: `output/replay-context.md`
- Execution trace: `trace/context-trace.json`
- Latch evidence: `evidences/latch_wait/metrics.json`

The isolated `CODEX_HOME` contained exact copies of the changed contracts:

- `op-diagnosis` SHA-256:
  `1041912e285d3abefc0e8bb122b1195852df636c560d9a5382ceb2e43c2f0e63`
- storage-latency reference SHA-256:
  `4a410fc06921ec05b25fd2457db388f22d7d53ad55cc8b92fa8a1f569e6157c1`

The replay first found the client duration interval, narrowed TiKV gRPC
Commit/Prewrite to `10.107.3.2:20180`, and then issued the bounded scheduler
latch-wait query for only that instance and those command types. Prewrite
latch-wait p99 reached `1.05s` at 03:26:45 and `1.11s` at 03:27:00 UTC+8;
Commit latch wait stayed below `0.1ms`. The output used this as the direct
mechanism, rejected cumulative `ANALYZE TABLE` ranking as causal proof, and
left the deeper workload or hot-key trigger unverified.

No component-log or Dashboard log task-group command was invoked. No Jira,
database, deployment, or provider configuration write was enabled.

## Unresolved gaps

- The change is not deployed to auto-oncall.
- The branch replay proves only branch behavior; it is not evidence that the
  change is effective in staging or production.
- The existing Jira internal comment has not been refreshed or verified.

## Next validation action

Deploy the skill change through the normal rollout path, then repeat the
case-specific `APID-29231` replay before promoting the optimization status.
If the deployed replay passes, update the existing stable Jira internal comment
in place and verify it by comment ID or idempotency key.