# Raftstore Write-Pipeline Known Cases

Source note: adapted from the Feishu `Raftstore handbook` <https://pingcap.feishu.cn/wiki/HaZTwLVC0iDxngk2hX3cbR33nqn>.

## Large Numbers of Raft Admin Commands Cause Raftstore Stall

- Phenomenon:
  - Overall write latency rises in raftstore stages.
  - The effect is often visible in `tikv-details -> storage -> storage async write duration`.
  - Breakdown panels show pressure in `propose wait`, `append log`, `commit log`, or `apply`.
- Trigger / Preconditions:
  - PD issues a large burst of admin commands such as scatter, region scheduling, or configuration change operations.
- Root cause:
  - Admin-command processing includes raftstore IO work and can raise end-to-end raftstore latency when issued too aggressively.
- Workaround:
  - Reduce the PD-side issuance rate for admin commands and avoid large scheduling bursts.
- Related oncall:
  - v7.5.6 TiKV raftstore stalled during massive scatter operations in add index `TCID-11575` / `TCOC-3605`

## Split or Merge Operations Block Business Writes Because of Embedded IO

- Phenomenon:
  - Split or merge operators increase `propose wait`, `commit log`, and `apply log` latency when underlying IO is slow.
- Trigger / Preconditions:
  - Table split, truncate table, or large empty-region merge activity overlaps with slow storage.
- Root cause:
  - Raftstore split and merge operators contain embedded IO work that directly amplifies latency when the storage path is slow.
- Workaround:
  - Throttle split and scatter rates.
  - Upgrade to a release line that includes the relevant fixes if available.
- Related oncall:
  - v8.5.4 OLTP latency observed during index creation `TCOC-4704`
  - v8.5.2 table split caused latency spike `ONCALL-7924`

## Long-Tail Latency on Large Single TiKV Nodes from Metadata Processing

- Phenomenon:
  - Even without a clear CPU or IO bottleneck, a large single TiKV node with many regions can show significant long-tail latency.
  - Store and peer message handling can stall on mutex or metadata processing.
- Trigger / Preconditions:
  - One TiKV holds very large data volume or very high region count.
- Root cause:
  - Region metadata processing becomes expensive and can block store-message and peer-message handling.
- Workaround:
  - No immediate universal workaround is documented.
  - Newer release lines reduce the cost of some message paths and may mitigate the issue.
- Related oncall:
  - v7.5.6 slow query analysis `ST-1907` / `TCOC-3754`
  - v7.5.6 single node with 4.7 TB had high `propose wait` latency `TCOC-3728`
  - v7.5.2 single node with 7.2 TB saw frequent leader drop `TICKET-6164`

## False Slow-Store Judgment Under High Write Speed

- Phenomenon:
  - High write throughput can make one TiKV look like a slow store even when the real problem is write pressure rather than storage failure.
  - If `evict-slow-store-scheduler` is enabled, excessive leader eviction can amplify the performance impact.
- Trigger / Preconditions:
  - Write throughput is very high in bytes or key count.
- Root cause:
  - The slow-store signal can be distorted by high write speed and not only by actual disk stalls.
- Workaround:
  - Increase the slow-store inspect interval.
  - Disable `evict-slow-store-scheduler` if false positives dominate the incident.
- Related oncall:
  - v7.5.6 a TiKV instance dropped leaders frequently under heavy write load `APID-11596`