# Storage Latency

Use this branch when latency is concentrated on one TiKV or TiFlash entity, or
when Raft/RocksDB/EBS evidence suggests storage delay.

## Workflow

1. Confirm the customer-visible latency or QPS change and its exact time window.
2. Locate the affected entity using TiDB-to-storage request duration or
   component gRPC duration grouped by instance/store. When the event time is
   known but the store is not, select the store at the direct-result peak or
   change sample, not from a wider-window CPU maximum or dashboard legend.
3. When Commit or Prewrite latency and its TiKV gRPC counterpart spike on the
   same store, query the scheduler latch-wait histogram for only Commit and
   Prewrite on that store and incident window before CPU or workload ranking.
   Use
   `tikv_scheduler_latch_wait_duration_seconds_bucket` when that series is
   exposed, compute an appropriate tail quantile grouped by `instance` and
   `type`, and preserve the aligned points. This is one direct mechanism check,
   not permission to inventory scheduler metrics.
4. For a write-path mechanism objective, retain scheduler command latency and
   the Commit/Prewrite latch-wait series, then check one strongest storage
   alternative on the same store and interval. Use the most relevant Raft
   append/apply or engine/storage duration rather than inventorying all of
   them.
5. Follow the write path into Raft Engine/RocksDB duration and node/cloud-volume
   latency or limit-exceeded signals only when the strongest storage
   alternative remains material or conflicting.
6. Check the strongest alternatives on the same entity and timeline:
   workload growth, read/write hotspot, CPU/thread saturation, compaction,
   scheduling, or network delay.

## Interpretation

- Append/apply or RocksDB/Raft Engine latency rising while work decreases or
  stays stable supports a storage mechanism.
- Commit latency without matching append/apply latency can indicate network or
  quorum delay instead of local disk.
- An aligned Prewrite or Commit latch-wait spike supports a write-side
  scheduler contention mechanism. Compare the two command types explicitly;
  do not substitute process CPU, pending-command count, or cumulative TopSQL
  rank for the latch-wait series.
- Compare scheduler command, latch wait, and the strongest storage alternative
  with the aligned end-to-end latency. Keep a materially smaller storage stage
  as a partial contributor, and keep CPU as pressure rather than the direct
  cause.
- A long-tail spike can disappear in an average or one fixed quantile. Query
  the available histogram at more than one appropriate quantile and compare
  with the raw heatmap/maximum only when the transport supports it.
- A slow entity observed only in a top-N result is not proof that other entities
  are healthy.
- Node disk averages can miss cloud-volume long-tail latency. Treat cloud
  volume metrics as a separate evidence class when available.

Do not recommend hardware, scheduler, or TiKV configuration changes without
current version verification and human review.