# Write Path Diagnosis Read this file when `Prewrite`, `Commit`, `Scheduler`, `async write`, or raftstore-related metrics are slow. ## 1. Write Path Decomposition Use these formulas to keep causality straight. ```plaintext gRPC message duration = gRPC handle time + Scheduler -> {Command} -> Command Duration ``` ```plaintext Scheduler -> {Command} -> Command Duration = write throttle duration + latch wait + scheduler worker wait + async get snapshot + processing read duration + Resource Control priority wait + quota limiter wait + async write duration ``` For `acquire_pessimistic_lock`, add: - lock wait - get-memory-lock or async-write related wait ## 2. Scheduler Layer ### Main metrics map Check: - `Scheduler -> {Command} -> Schedule command duration` - `Scheduler -> {Command} -> Scheduler latch wait duration` - `Scheduler -> Txn Scheduler Pool Wait Duration` - `Scheduler Worker Pool -> Wait duration` - `Storage -> Storage async snapshot duration` - `Scheduler -> {Command} -> Scheduler command process duration` - `Flow Control -> Txn command throttled duration` - `Flow Control -> Throttle duration` - `Resource Control -> Priority Task Wait Duration` - `Storage -> Storage async write duration` If `ServerIsBusy`, `Throttle duration`, or `Scheduler discard ratio` is prominent: - also read `references/flow-control.md` Important note: - `processing_read_duration` is badly named - it is the in-memory preparation stage before async write, including conflict checks and write-record preparation ### Scheduler diagnosis order 1. Compare `gRPC duration` with `scheduler command duration` - gRPC slow, scheduler normal: go back to gRPC 2. Compare `scheduler command duration` with `Storage async write duration` - async write slow in the same trend and magnitude: go to raftstore, do not blame scheduler first 3. Check whether flow control or Resource Control explains part of the wait 4. Only then inspect scheduler internals ### Typical scheduler cases #### Latch wait high Meaning: - a key-hash slot cannot proceed yet Remember: - latch wait is not the same as transaction lock wait - latch is key-hash-slot level and only protects concurrent writes in scheduler Typical reasons: - scheduler is slow, so latch is released late - hot-key or hash-slot conflicts Useful signals: - `Scheduler command duration` tails look similar to latch wait tails - pessimistic lock wait also rises when true lock conflict exists #### Scheduler worker wait high Main panel: - `Scheduler -> Txn Scheduler Pool Wait Duration` Typical reasons: - too many tasks arrive at once - worker CPU is saturated - conflict checking causes heavy synchronous read / scan work - snapshot is slow Check: - `Thread CPU -> Scheduler worker CPU` - `Scheduler -> Txn Scheduler Pool Wait Duration` - `Scheduler -> Txn Scheduler Pool Exec Duration` - `Scheduler -> {Command} -> Scheduler scan details` - `Scheduler -> {Command} -> Scheduler command process duration` - `Scheduler -> {Command} -> Scheduler command block read duration` Mitigation direction: - if CPU is the bottleneck, increase `scheduler-worker-pool-size` - if block read grows, look for cache miss, extra scans, or disk-read slowdown #### Pessimistic lock wait high Main panels: - `Pessimistic Locking -> Lock Manager -> Waiter lifetime duration` - `Pessimistic Locking -> Lock Waiting Queue` - `Scheduler -> AcquirePessimisticLock -> Scheduler command duration` Two main explanations: - lock hold time is long - waiter queue is long To separate them: - inspect TiKV-side `Prewrite` / `Commit` latency, because slow commit extends lock hold time - inspect TiDB-side `Transaction -> KV Transaction Duration` to determine whether a big transaction is holding locks for a long time #### Async write high Main panel: - `Storage -> Storage async write duration` If its rise matches scheduler command duration, scheduler is downstream-blocked and raftstore is the next investigation target. ## 3. Raftstore: Diagnose In Reverse Order Do not follow write flow strictly top-down. Diagnose raftstore in reverse order so downstream bottlenecks are not mistaken for queue-wait symptoms. Recommended order: 1. `Raft IO -> Process Ready duration` 2. `Raft IO -> Store write loop duration` 3. `Raft IO -> Commit log duration` 4. `Raft IO -> Apply log duration` 5. `Raft IO -> Store write wait duration` 6. `Raft IO -> Propose wait duration` ## 4. Raftstore Stage Guide ### Process Ready duration Meaning: - one raftstore poller round cost Interpretation depends on async IO: - async IO enabled: - `raftstore.store-io-pool-size > 0` - `Thread CPU -> Store writer CPU` is present - then focus on raftstore CPU and admin IO - async IO disabled: - process-ready also includes more direct IO cost - inspect append / write duration together Typical reasons: - `Thread CPU -> Raft store CPU by thread` is high - admin apply work is heavy, such as `add learner` - some raftstore message handling path is slow ### Store write loop duration Meaning: - from store-writer task pickup to raft log write completion If this is slow, the most likely problem is the write-to-engine stage itself, especially raft-engine persistence. If the spike is short-lived and only one store is the outlier, also read `references/io-jitter.md`. ### Commit log duration Meaning: - end-to-end from leader propose to majority commit It includes: - leader local persistence - replication to followers - follower persistence - follower response back to leader - leader commit-index advance Useful approximation: ```plaintext commit log duration ~ leader propose wait + leader process ready + leader store write wait + leader store write loop + follower propose wait + follower process ready + follower store write wait + follower store write loop + network / raft message transfer + leader round-trip processing ``` Diagnosis order: 1. see whether propose / process-ready / store-write-wait / store-write-loop already explain the rise 2. if not, investigate replication transport Replication transport possibilities: - gRPC raft-client bottleneck - network latency or retransmit - raft inflight-message pressure on hot regions - memory-allocation stalls Useful checks: - `TiDB -> KV Request -> RPC Layer Latency` - blackbox ping latency - `Node exporter -> TCP drop` - `Node exporter -> TCP retransmit` - `Node exporter -> Numa page -> allocstall` If `Commit log duration` rises much more than `Append log duration` and `unreachable` failures appear: - also read `references/network-issues.md` Version / config clue: - older versions with small `grpc_raft_conn_num` can bottleneck raft message sending ### Apply log duration Meaning: - from apply-worker handling start to RocksDB write completion This is usually where RocksDB write cost shows up most directly. Main checks: - `Raft IO -> Apply log duration` - `Thread CPU -> Async apply CPU` - `Raft Propose -> Perf Context duration` - `RocksDB KV -> Write duration / Write Operation` - `RocksDB KV -> Write WAL duration / WAL sync duration` Typical reasons: - apply CPU saturation - disk jitter - compaction pressure, especially periodic full compaction - manifest rewrite - especially when TiKV data size exceeds `3T` - RocksDB log shows `Creating manifest` - SST ingest blocking foreground write - region-schedule related `add learner` - Lightning / backup / import - fast DDL ingest ### Propose wait Meaning: - time from command arrival at raftstore to actual processing start Typical reasons: - raftstore CPU high - mitigation: increase `raftstore.store-pool-size` - async IO disabled and raftstore still performs IO inline - signal: `Store write loop duration` absent - mitigation: set `raftstore.store-io-pool-size >= 1` - raftstore stuck on admin IO - raftstore message handling path is slow - hot region starvation / reschedule effects - too many regions per store, making default batching too small Useful checks: - `Raft Process -> Max duration of raft store events` - `Raft Admin -> apply operations` - `RocksDB -> WAL sync operations` - `Cluster -> Region and Server -> Hibernate Peers` - for `v8.5`, store-FSM poll-round and count-per-poll tails Typical mitigations: - increase `raftstore.store-pool-size` - enable async IO - pre-split or scatter hotspot regions - increase `store-max-batch-size` when many regions are active ### Store write wait Meaning: - wait from raftstore queueing a write task to a store-writer thread starting it Typical case: - store-writer CPU is busy Rule of thumb: - because store writers perform a lot of IO, CPU above about `50%` can already be meaningful Mitigation: - increase `raftstore.store-io-pool-size` ### Apply wait Meaning: - wait from apply task creation to apply-poller pickup Typical reasons: - apply CPU high - apply duration high - region-level hotspot - too many active regions - too much write data volume causing queue buildup Useful checks: - `Raft Process -> Apply FSM` - apply FSM scheduler wait distribution - apply FSM poll duration - apply FSM poll round - apply FSM poll count per round - `RocksDB KV -> Write Flow` Typical mitigations: - increase `raftstore.apply-pool-size` - increase `raftstore.apply-yield-write-size` - scatter hotspots ## 5. Root-Cause Rules - If wait metrics are high only because a downstream processing metric already rose first, blame the downstream metric. - If `async write duration` is the main contributor, keep descending until you reach raftstore or RocksDB evidence. - If `Commit log duration` is high but local waits are flat, do not forget network and raft-message transport. - If lock wait is high, distinguish lock-hold time from waiter-queue size before deciding the issue is contention.