# Write Path Contention

## Question

Which scheduler, lock, Raft, storage, or service boundary turned the observed
Commit/Prewrite demand into the user-visible result and, only when the
objective requires it, which workload and transaction shape triggered that
demand?

## Enter When

Use for aligned Commit or Prewrite latency/rate, scheduler wait, lock/backoff,
write conflict, deadlock, or write-path ServerIsBusy evidence.

Keep this scenario primary while TopSQL, Statements, or Slow Query are used to
identify the originating write workload. Add `query-execution` only after a
concrete DML digest or plan creates a separate plan/operator tuning question.

When the ticket already supplies old/new DDL, use that DDL directly as artifact
evidence. Do not ask an observability Provider to query customer table, index,
partition, or statistics metadata. Treat `AUTO_RANDOM`, partitioning, primary
keys, and global indexes as workload-distribution or write-amplification
triggers to evaluate; none of them is the immediate latency mechanism without
an aligned scheduler, latch, Raft, storage, flow-control, or service boundary.

## Objective Exit Gates

Set `workload_identity` and `transaction_shape` to `required` when the
diagnostic objective asks which workload, SQL, batch, or transaction caused the
symptom, or asks for SQL/transaction remediation.

`workload_identity` passes only when a concrete write DML, exact Digest,
normalized SQL identity, or equivalently stable workload key is aligned with
the incident window. Generic Commit/Prewrite rows, cumulative TopSQL rank, and
high write QPS do not pass this gate.

`transaction_shape` passes only when the evidence establishes enough of the
relevant boundary to evaluate the requested attribution or remediation:
autocommit versus explicit transaction, statement grouping or batch size,
connection/transaction mapping, concurrency, write keys/size, and material
Prewrite/Commit/retry/lock stage times. Record unsupported elements precisely;
do not infer them from Commit/Prewrite latency.

When a required gate cannot pass after the available read-only provider
surfaces are exhausted, mark it `bounded_failure`. Do not claim resolved
workload attribution or concrete SQL/transaction remediation, but continue to
a final bounded mechanism finding.

## Incident Entity And Window Gate

Before any scheduler, CPU, or workload query, lock the direct-result tuple:

1. Reuse an entity only when a reliable artifact or metric binds it to the
   Commit/Prewrite result in the narrow incident interval.
2. Preserve the artifact's statistic. If it shows Commit/Prewrite P99 or
   another histogram percentile, confirm that same percentile and operations;
   do not replace it with mean duration, request rate, cumulative rank, or a
   different transaction operation.
3. If the interval is known but the store is not, first confirm the bounded
   client Commit/Prewrite percentile with only those operation types. Then use
   the matching TiKV gRPC Commit/Prewrite percentile grouped by instance to
   locate the same peak/change sample and select the store at that sample or
   within one query step.
4. If a provider rejects a percentile query for excessive samples or memory,
   reduce the interval and label cardinality while keeping Commit/Prewrite and
   the percentile unchanged. Do not issue an all-type inventory or downgrade
   to a mean as the diagnostic result.
5. Do not select a store from a dashboard legend, a wider-window CPU maximum,
   a resource rank, or a cumulative workload rank.

For Clinic, ask for the relationship semantically rather than writing provider
syntax: identify the numeric store carried by the aligned TiDB client
Commit/Prewrite result, map that store to its TiKV address/instance through the
provider-owned topology surface, and determine whether that exact TiKV instance
has aligned scheduler throttle duration large enough to explain the reported
latency. The Evidence Agent owns concrete metric families, labels, PromQL, and
same-round synonym correction. A flow-control counter absent on one version is
not counter-evidence when the provider's throttle-duration histogram exists.

Keep the client result, TiKV gRPC counterpart, scheduler command, latch wait,
and strongest storage alternative on the locked store and interval. Replace
the tuple only when new direct-result evidence disproves it.

## Mechanism-Only Path

When `mechanism` is required but workload identity and transaction shape are
not:

1. align the client Commit/Prewrite percentile with its matching TiKV gRPC
   percentile, filtered to those operations and the narrow incident interval;
2. query the matching scheduler command percentile and the Commit/Prewrite
   scheduler latch-wait percentile on the locked store and interval; a
   sum/count mean is supporting context only and cannot replace either
   percentile;
3. check one strongest storage percentile alternative on that same tuple, such
   as the most relevant Raft append/apply or engine/storage duration;
4. compare every stage magnitude with the aligned end-to-end result;
5. keep CPU as pressure context only and keep cumulative `ANALYZE` or TopSQL
   rank out of the direct causal chain.

Do not run TopSQL, Statements, or Slow Query on this path merely because those
surfaces are available. Use a workload surface only when workload attribution
becomes required or one explicit load-bearing current-read-only gap can change
the leading mechanism, severity, safety decision, or recommendation.

## Separate Trigger From Mechanism

Maintain two independent chains:

```text
Workload trigger:
write DML/digest -> frequency and transaction grouping ->
write keys/size/retries -> Commit or Prewrite demand

Write-path mechanism:
Commit or Prewrite demand -> scheduler/lock/Raft/storage/service boundary ->
observed latency, CPU, error, or hotspot
```

Do not call a workload causal merely because it is top-ranked in the same
window. Do not call a scheduler, Raft, or storage wait the workload trigger.
Join the chains only through aligned time/entity evidence and an observed
Digest, connection, transaction, request, or dependency mapping.

## Workload And Transaction Drilldown

Treat generic TiKV `COMMIT`, Prewrite, two-phase-commit, and storage-command
rows as wrappers, not originating SQL. Continue adaptively:

### Hard order when workload gates are required

When `workload_identity` or `transaction_shape` is required, mechanism
exploration cannot consume the workload-attribution budget:

1. Run transport readiness once at most. Do not count it as a diagnostic
   mechanism check and do not repeat it after the provider is ready.
2. Classify every supplied workload-identity surface before provider mechanism
   expansion. A UI TopSQL or statement aggregate for `COMMIT` is
   `aggregate_commit_summary` and can only be `correlation_only`. A
   `prev_stmt` value has unresolved identity unless the provider API proves
   that it belongs to the same Slow Query `COMMIT` record and preserves a
   stable record, session, transaction, or request linkage. That mapping makes
   the DML a high-risk candidate; it does not prove that the DML caused high
   load or Commit latency. Never copy SQL or `prev_stmt` text into the scenario
   packet.
3. Run at most one alignment check before the selected single-instance TopSQL
   query, and only when no supplied artifact already binds the direct result
   to the incident entity and sample. The alignment check may locate the
   affected instance; it must not expand into scheduler, latch, Raft, storage,
   CPU, or resource inventory.
4. A broad scheduler query followed by the same scheduler query on the
   affected store is a prohibited duplicate. When a customer screenshot or
   artifact binds a TiKV host/instance to the direct result at the incident
   sample, use that exact mapping for TiKV TopSQL and perform only the minimum
   metric validation needed for the mechanism. A legend or wider-window
   maximum does not bind the instance.
5. Immediately after the optional alignment check, MUST execute the selected
   single-instance TopSQL query. Until that workload gate is attempted, do not
   query an optional TiDB commit counterpart, scheduler or latch metrics,
   calculate an additional aligned ratio, or descend into Raft, RocksDB,
   storage, CPU, or resource checks.
6. If TopSQL returns generic `COMMIT` or Prewrite, MUST spend the one allowed
   post-TopSQL refinement on a narrow write-specific Slow Query lookup on the
   selected request path to identify originating DML. If TopSQL also shows SQL
   on a related business table, use the provider's equivalent of
   `--text <business table>` and order by `write_keys`. Include the presence
   and provenance of `prev_stmt`, write keys/size, stage timing, retry, lock,
   and backoff fields. `prev_stmt` can identify originating DML only when the
   API result proves it is on that same Slow Query `COMMIT` record and exposes
   a stable record, session, transaction, or request linkage. An adjacent row,
   same-window row, UI association, or unkeyed summary remains correlation.
   Even valid identity proof remains causally unverified unless
   single-statement/autocommit shape or a complete
   transaction/write/request-to-latency mechanism mapping is observed.
   Do not add the Digest of a lower-ranked co-window DML without direct
   transaction linkage. Only after TopSQL and that lookup return an exact
   bounded failure may the final draft request DML or transaction details from
   the customer. This is an exclusive post-TiKV-TopSQL dominant-wrapper
   branch: skip TiDB TopSQL and Statements and go directly from the generic
   wrapper result to the table-text Slow Query.

Then use this bounded provider sequence:

1. Start summary-first with the exact incident interval, one affected or
   highest-impact instance selected only among entities aligned to the locked
   direct-result sample for the TopSQL component, Top 10 Digests, and any
   customer-supplied Digest. Pass the exact instance to the provider; never use
   the first instance returned by Dashboard discovery or a wider-window
   resource maximum as "top". Persist raw output through the run-scoped provider
   output path, but keep only compact aggregates, capability/truncation
   metadata, and the decision rows in working context. Do not read the complete
   raw artifact. At most one wide TopSQL discovery query is allowed for this
   scenario. For Wetech, spell the duration argument exactly as
   `--window 10s`; `--window 10` is invalid and prohibited.
2. Use TiKV TopSQL on that one affected store and interval to locate the generic
   wrapper, its exact timing, and competing workload. Retain Digests and
   instance labels when exposed.
   Compare each candidate's execution rate and CPU with the wrapper. If generic
   `COMMIT` or Prewrite materially dominates both dimensions and there is no
   direct `prev_stmt`, connection, transaction, request, or Digest mapping,
   lower-ranked DML in the same window remains a co-window candidate only. Do
   not label it originating DML or filter the one Slow Query lookup by its
   Digest. Exact-Digest priority is allowed only when direct linkage already
   exists or the wrapper does not materially dominate.
3. Use the TiDB TopSQL or Statements branch only when direct request,
   transaction, connection, or Digest mapping already selects the TiDB request
   path or a concrete originating DML, and generic `COMMIT` or Prewrite does not
   materially dominate the relevant TopSQL result. In that branch, use
   TiDB-side metrics or the direct mapping to select exactly one TiDB instance.
   A TiKV store alone does not identify that TiDB instance. Query only that TiDB
   request path over the same interval. If one compact Statements query needs a
   type filter, combine `INSERT`, `UPDATE`, `DELETE`, and `REPLACE` in that one
   bounded query when supported; never issue four separate broad Statements
   queries. Match exact query/plan Digests across surfaces when available.
   When the generic wrapper materially dominates, this branch is forbidden;
   skip directly to step 5 without TiDB TopSQL or Statements.
4. After TopSQL returns a concrete originating DML Digest, stop all broad
   Statements discovery. "Concrete" here requires the direct linkage rule
   above, not mere appearance in the same TopSQL window. The initial turn has exactly one highest-value
   post-TopSQL drilldown for that Digest. Prefer one write-specific Slow Query
   or transaction-field lookup on the same TiDB instance that can jointly
   expose `prev_stmt`, transaction size, and stage timing. Use an exact
   Statements lookup only when it provides a required field unavailable from
   the higher-value transaction surface. Return at most 10 compact decision
   rows and keep full detail in the provider artifact. Do not sort
   only by total `query_time`; inspect every supported discriminator needed to
   find the transaction, including `write_keys`, `write_size`,
   `write_sql_response_total`, Prewrite/Commit stage time, `txn_retry`,
   `exec_retry_time`, `lock_keys_time`, `backoff_time`, and backoff type.
5. If TopSQL returns only a generic `COMMIT`/Prewrite wrapper, do not prioritize
   an exact wrapper lookup. MUST use the one post-TopSQL drilldown for a
   write-specific Slow Query lookup constrained by the selected TiDB instance
   or observed request path, incident window, and visible workload/table clue.
   When a business-table clue is present, use the provider's equivalent of
   `--text <business table>` and order by `write_keys`, without a low-confidence
   DML Digest filter. Include `prev_stmt`, `write_keys`, `write_size`, stage
   timing, retry, lock, and backoff fields. Mark a DML identity proof only when
   `prev_stmt` is returned on the same Slow Query `COMMIT` record and the API
   preserves a stable record, session, transaction, or request linkage. The
   resulting DML is a high-risk candidate, while causal attribution remains
   correlation-only without single-statement/autocommit or a complete
   transaction/write/request-to-latency mechanism mapping. An adjacent or co-window
   `prev_stmt` is not a transaction mapping. If that one lookup
   cannot produce a concrete DML candidate, mark workload attribution
   `bounded_failure`; do not fall back to TiDB TopSQL, Statements, or four
   broad DML-type searches.
6. If the one exact result is empty or top-N limited, do not enumerate every
   instance, switch sort orders repeatedly, retry a wide query, or consume
   another initial-turn refinement. Absence from one page is not negative
   evidence. Preserve the exact candidate and missing field as a possible
   causal gap.
   Keep every refinement on the instance already selected for that provider
   surface. If metrics cannot select one TiDB instance and no direct mapping is
   available, return a bounded failure instead of enumerating instances.
7. Map the exact Digest to connection/transaction context and compare batch or
   statement grouping, write keys/size, retries, and stage times with another
   write workload or a nearby normal execution.

For request-type metrics, discover the active label values once in the exact
incident window before querying latency. Then issue one exact-type query; do
not try a wide by-type latency query, repeat it in a narrower window, and only
then discover the active types.

Read provider field/capability metadata before interpreting missing values.
Unsupported fields are unavailable, not zero. A short high-frequency write can
be absent from Slow Query while still appearing in Statements or TopSQL.

Stop workload enumeration once one exact candidate has aligned Slow Query or
equivalent transaction evidence containing same-record identity provenance,
write-key/size, retry/lock/backoff, and Commit/Prewrite stage fields needed for
the objective. The runtime deadline and the causal-gap `stopCondition` are hard
bounds; if the remaining gate cannot be resolved inside them, return one exact
`bounded_failure`.

When the customer supplied two incident windows, the initial turn must query
only the canonical first window. It must never execute the second window, even
after the first window produces a concrete Digest or passes the required
objective gates. If the second window can close one remaining load-bearing
transaction mapping or otherwise change the leading cause, severity, safety
decision, or recommendation, reserve it for one high-information causal gap,
set that gap `resolvableBy=current_read_only_tools`, name the exact Digest or
table-text transaction signature and stop condition, and end the initial turn.
Recurrence alone is not load-bearing when the first window already establishes
the same leading cause, transaction shape, safety decision, and recommendation;
in that case set `diagnosticFollowup.required=false` and finalize. Only the
backend gap round may query a selected second-window gap. If the first-window
table-text Slow Query already identified the transaction through a same-record
`prev_stmt` with stable record, session, transaction, or request linkage, reuse
the same `--text <business table>`, `write_keys` order, and transaction-field projection
directly in the second window. Do not repeat instance selection or TopSQL. Do
not turn the gap into another broad TopSQL or Statements discovery pass.

For an aligned Commit or Prewrite spike, check the exact instance's scheduler
command latency and Commit/Prewrite latch wait, then check one strongest
storage alternative on the same tuple. If scheduler command latency explains
the end-to-end result while latch wait and the storage alternative are
materially smaller, record scheduler command as the immediate boundary and the
smaller stage as a partial contributor or insufficient explanation. Do not
turn this one alternative into a Raft, RocksDB, and disk inventory.

## Mechanism Drilldown

1. Align the client result and TiKV gRPC Commit/Prewrite result on exact
   instances and samples.
2. Follow the strongest direct boundary: scheduler latch, transaction
   lock/backoff/retry, Raft append/apply, RocksDB/storage/flow control, or gRPC
   service saturation. For scheduler contention, prefer
   `tikv_scheduler_latch_wait_duration_seconds_bucket` for Commit and Prewrite
   on the affected TiKV and interval.
3. Compare Commit with Prewrite, the same mechanism on a peer/normal sample,
   and one strongest materially different storage mechanism.

Do not substitute process CPU, pending-command count, window maxima, or
cumulative workload rank for the direct mechanism. Before calling a wait the
primary mechanism, compare its magnitude with the aligned end-to-end result. A
microsecond- or low-millisecond wait cannot by itself explain a
tens-of-milliseconds result.

## Strongest Alternative And Stop

Keep one strongest alternative alive for each chain:

- workload trigger: another write DML, transaction grouping, hotspot, or
  workload-volume shift;
- mechanism: lock/retry, scheduler, Raft/apply, storage/flow control, or
  service saturation.

Stop when the mechanism and strongest alternative are bounded and every
required objective gate is either `passed` or `bounded_failure`. Empty,
aggregate-only, unsupported, or top-N-limited output is an evidence gap, not
proof that a candidate is absent.