# TiKV Hotspot Mitigation Boundary

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "tikv-hotspot-mitigation-boundary-2026-08-13",
  "problems": [
    {
      "problemId": "unexplained-cluster-tail-stops-before-tikv-backend-discrimination",
      "summary": "The workload-hotspot branch can stop after correlated SQL, CPU, or aggregate TiKV request evidence without proving the workload mechanism or capacity exhaustion and without testing a store-local apply, snapshot, RocksDB, storage, scheduler, or leader-distribution alternative.",
      "status": "optimized_verified",
      "issueKeys": ["APID-29253"],
      "sourceRefs": [
        "https://tidb.atlassian.net/browse/APID-29253?focusedCommentId=23310306#problem:diagnostic_gap:unexplained-cluster-tail-stops-before-tikv-backend-discrimination"
      ],
      "signalTextIncludesAny": [
        "aggregate resolve lock treated as the mechanism",
        "tikv request latency not localized to a store",
        "cluster tail attributed to correlated sql without mechanism proof",
        "aggregate request latency used to reject store-local backend",
        "hot store alternative not tested",
        "tikv hotspot mitigation missing"
      ],
      "observedThrough": "2026-08-13",
      "effectiveAt": "2026-08-13T14:53:00Z",
      "verifiedAt": "2026-08-13T15:59:00Z",
      "attemptedAction": "For an unexplained cluster-wide tail, spend the one adaptive refinement on a direct store-local backend discriminator when SQL causality and capacity exhaustion remain unproven; also localize material aggregate Commit, Prewrite, or ResolveLock latency before assigning a mechanism. Admit only readback-first, reversible, human-owned TiKV hotspot mitigation without historical store ids or parameter values.",
      "evidenceUrl": "https://tidb.atlassian.net/browse/APID-29253?focusedCommentId=23310306"
    }
  ]
}
-->

- Optimization ID: `tikv-hotspot-mitigation-boundary-2026-08-13`
- Owner: workload-hotspot Scenario Diagnostic Agent
- Source reference: APID-29253 comment `23310306`
- Implementation status: deployed to staging and verified by write replay
- Effective status: `optimized_verified`

## Bounded problem

The frozen-input read-only replay
`bd01ac41-ccc8-4521-925d-1de3bb735da5` successfully acquired Wetech evidence
with only comments before `23310306` visible to analysis. It confirmed a
cluster P999 incident, but neither mapped the correlated SELECT class to a
stable statement mechanism nor proved CPU capacity exhaustion. It rejected
"broad TiKV" from lower aggregate request-type latency without testing a
store-local apply, snapshot, RocksDB/storage, scheduler, or leader-distribution
boundary. The workload-hotspot branch did not use its adaptive refinement.

A second current-main read-only replay
`d713536f-6ac6-48b2-a453-f4f5b52e82a4` exposed the other form of the same stop
gap: ResolveLock P99 reached about 814 ms and Prewrite P99 about 636 ms, but the
query aggregated away store identity and the branch still returned
`evidenceRefinement.required=false`.

The resulting causal brief correctly kept the TiKV path as an alternative
instead of repeating the old lock-contention claim, but it could neither bound
the TiKV hotspot mechanism nor support the requested emergency playbook. This
is a diagnostic stop-condition gap, not a Provider failure and not a missing
Scenario.

## Candidate change

- For a cluster-wide tail with neither SQL causality nor capacity exhaustion
  proved, require one direct store-local backend discriminator before stopping.
- Treat a newly observed, material aggregate TiKV request-latency alternative
  as load-bearing when store identity and one direct backend discriminator are
  still absent.
- Use the existing single adaptive refinement to localize that same result and
  compare one aligned apply/snapshot, RocksDB/storage, scheduler, or
  leader-distribution boundary.
- Allow only conditional, reversible, human-owned mitigation after direct
  per-store evidence: read back current scheduling state and leader
  distribution, change one boundary, observe the same signals, and restore the
  recorded values when appropriate.
- Never copy the historical store number or scheduling values into a new
  recommendation.

## Verification evidence

- `node --test dist/test/op-diagnosis-skill.test.js` must preserve the narrow
  Scenario contract after compilation.
- The current baseline used `mode=read`; it did not write Jira, DB evaluation,
  Feishu, or production state.
- Isolated Pod run `64fe8da3-bb15-4738-8206-c4dd50c3450d` first proved that
  the workload-hotspot branch requested and executed the store-local adaptive
  refinement instead of promoting ResolveLock by name.
- Staging write replay `6a78a834-012a-476d-ae9c-da09c255df31` repeated that
  behavior. It localized simultaneous storage-write, prewrite scheduler, and
  snapshot-apply tail latency on one TiKV store, kept background maintenance as
  the material alternative, and did not identify one SQL as root cause.
- The same run persisted as DB run `150034`, updated the mapped internal Jira
  comment `23310306`, and readback confirmed `public=false` with separate
  customer and internal sections.

## Unresolved gaps

The automatic replay target is later public escalation comment `23310364`, not
the exact creation boundary of original bot comment `23310306`. The Support
status update between them was excluded, while several bridged coordination
comments were visible. Therefore this verifies the diagnostic mechanism and
publication boundary, but not the original answer's recovery-state claim.

## Next validation action

Monitor later hotspot cases for store-local discrimination. A separate replay
cutoff improvement would be required to evaluate the original `23310306`
recovery-state claim at its exact creation time.