# IO Jitter Diagnosis Read this file when a short-lived TiKV slowdown looks store-local and disk tail latency is plausible. This guide is derived from the internal KB page: - `[Troubleshooting Guide] How to quickly identify IO jitter in TiKV` ## When To Read Read this file when the active branch is `io-jitter` and the problem looks like a short-lived, store-local long-tail event that may come from raft-engine, RocksDB-KV, or node / cloud-disk latency. ## Goal The goal is to determine: - whether workload growth or hotspot movement already explains the slowdown - which store is the real outlier - whether the first strong evidence points to raft-engine, RocksDB-KV, or node / cloud-disk latency - whether the correct wording is `suspected IO jitter` or a stronger node / disk conclusion ## Key Questions Keep these questions aligned with the main skill: - is workload volume, request mix, and hotspot distribution stable enough for local IO jitter to be plausible - which store is actually slow - is the slow path read, write, or both - which tail quantile exposes the problem - what node-level or vendor-level evidence is still missing ## Guardrails - if many stores slow together, investigate workload-wide or broader infra causes before naming IO jitter - do not treat average-only disk metrics as decisive proof for short long-tail events - if only store-local tails are proven, write `suspected IO jitter` - only upgrade to node / cloud-disk wording when higher-level evidence aligns in the same window ## 1. What IO-Jitter Diagnosis Must Answer Every IO-jitter analysis must answer these questions in order: 1. Is workload volume, request mix, and hotspot distribution stable enough that local IO jitter is plausible? 2. Which TiKV store is actually slow? 3. Is the slow path write, read, or both? 4. Does the first strong evidence point to raft-engine IO, RocksDB-KV IO, or node / cloud-disk latency? 5. Which tail quantile exposes the issue? 6. What node-level or vendor-level evidence is still missing? Important wording rule: - if only store-local tail spikes are proven, write `suspected IO jitter` - only upgrade to `node / cloud-disk IO jitter` when node or cloud-disk panels align in the same window ## 2. When To Suspect IO Jitter IO jitter is more plausible when these conditions hold together: - business volume is roughly stable - no obvious hotspot move or load redistribution explains the spike - the incident is short-lived instead of sustained for a long period - one TiKV or a small subset is much slower than the fleet - cloud disks are involved, especially AWS EBS Common symptom patterns: - many slow SQL appear suddenly, but overall QPS does not jump much - `p99` looks only mildly elevated, while `p999`, `p99999`, or `max` shows obvious bursts - `Commit`, `Prewrite`, `Cop`, or `BatchGet` tails spike only on one store Guardrail: - if many stores slow together, suspect workload growth, hotspot spread, network, or a broader infra event before naming IO jitter ## 3. Rule Out Business Growth And Hotspot First Check: - `TiDB -> KV Request OPS` - `TiDB -> KV Request duration by store/type` - request counts or QPS by store / instance - hotspot or request-distribution panels when available - coprocessor or write request totals in the same window Interpretation: - workload stable, only one store slow: - local IO jitter becomes plausible - one store gets much more traffic or leader load at the same time: - do not jump to disk jitter yet - request mix changes sharply, such as new scan-heavy or write-heavy traffic: - this may be a business-shape issue rather than storage jitter ## 4. Find The Slow TiKV First Start with these panels: - `TiDB -> KV Request -> KV Request Duration 99 by store` - `TiKV -> gRPC -> 99% gRPC message duration` by instance - `TiKV -> Raft IO -> 99% Append log duration per server` - `TiKV -> Raft IO -> 99% Apply log duration per server` - `TiKV -> Errors -> Server is busy` - `TiKV -> PD -> Store Slow Score` Interpretation: - one store is the outlier across several panels: - that store is the primary IO-jitter candidate - `store-write-channelfull-*` appears: - StoreWrite is writing slowly, especially meaningful when async IO is enabled - `Store Slow Score` rises: - useful as slowdown evidence, but not by itself proof of PD `evict-slow-store` Version note: - in `v7.6+`, slow-store detection can participate in automatic eviction when the scheduler is enabled - in `v7.5` and earlier, TiKV can report slow-score but PD does not automatically evict slow stores by default ## 5. Quantile Escalation Rules Do not stop at `p99` if the incident is short and store-local. Use this order: 1. `0.99` 2. `0.999` 3. `0.99999` 4. `1` Rules: - compare the same quantile across healthy and slow stores - prefer per-store or per-instance views over fleet averages - if only `max` spikes but `p99` stays flat, describe it as `extreme tail latency` or `bursty store-local jitter`, not sustained saturation Important limitation: - node `Disk wait latency` is usually an average metric, so it can miss long-tail IO jitter completely ## 6. Write-Side IO Jitter Use this branch when `Prewrite`, `Commit`, or raft-related metrics spike on one store. Key checks: - `TiDB -> KV Request duration by store/type` - `TiKV -> gRPC -> 99% gRPC message duration` - `TiKV -> Raft IO -> 99% Append log duration per server` - `TiKV -> Raft IO -> 99% Apply log duration per server` - `TiKV -> Errors -> Server is busy` - `TiKV -> PD -> Store Slow Score` How to separate the layer: ### A. Raft-engine / raft-persistence side More likely when: - `append log` or related raft write metrics spike first on one store - `commit log duration` rises with that same store - `store-write-channelfull-*` appears under async IO Interpretation: - the local raft persistence path is a strong suspect - if async IO is disabled, inline IO can also amplify `propose wait` ### B. RocksDB-KV write side More likely when: - `apply log duration` is the clearer outlier - RocksDB write, WAL sync, or write-operation duration also rises - the incident is localized to one store Interpretation: - foreground apply or RocksDB persistence is the stronger suspect than raft transport Guardrails: - `Store Slow Score` is supporting evidence, not the root cause - if onset order is unclear, do not claim raft-engine or RocksDB-KV as final root cause yet ## 7. Read-Side IO Jitter Use this branch when `BatchGet` or `Cop` slows and the evidence points to storage-read cost on one store. Check: - `TiDB -> KV Request duration by store/type` - `Coprocessor Detail -> Handle duration` - `RocksDB KV -> Get duration` - `RocksDB KV -> Seek duration` - `RocksDB KV -> SST read duration` - `Node exporter -> disk latency` or equivalent node disk panels Interpretation: - handle duration high together with RocksDB read metrics high on one store: - local storage-read jitter is plausible - read latency high but handle duration is not: - go back to gRPC, snapshot, or response-send path before blaming disk If the issue involves coprocessor disk-read amplification, use: ```promql sum(rate(tikv_coprocessor_rocksdb_perf{k8s_cluster="$k8s_cluster", tidb_cluster_id="$tidb_cluster", instance=~"$instance", metric="block_read_count"}[2m])) by (req) ``` Interpretation: - `block_read_count` rises sharply with the slowdown: - first consider scan-volume or query-shape change - workload is stable but RocksDB read durations spike on one store: - IO jitter remains plausible ## 8. Node And Cloud-Disk Confirmation If store-level metrics strongly suggest IO jitter, continue upward to node or cloud-disk evidence. Check: - `Node exporter -> Disk wait latency` - `EBS Volume Performance` - equivalent cloud-disk throughput / IOPS / latency panels - vendor-side evidence if available Interpretation: - `Disk wait latency` is average-oriented: - useful for sustained issues - weak for short long-tail jitter - `EBS Volume Performance` is more useful for long-tail spikes or `volume IO exceeded` style events - if node or cloud-disk panels align on the same store and time window, confidence rises a lot - if node panels stay flat, the right wording is often still `suspected store-local IO jitter` rather than a proven disk fault ## 9. Slow Log And Log Fallback When the incident is too short or affects only a few requests, logs can be more sensitive than coarse metrics. Useful sources: - TiDB slow log - TiKV slow log - clinic download logs when available Use logs to answer: - whether slow requests concentrate on one store - whether the slow stage is read-side or write-side - whether the slow requests are actually larger scans, larger writes, or a workload-shape change ## 10. Mitigation Directions Match the mitigation to the proven layer: - suspected raft-engine or raft-persistence jitter: - reduce disk pressure - increase disk headroom - verify async IO is enabled with `raftstore.store-io-pool-size >= 1` - suspected RocksDB-KV jitter: - inspect compaction, WAL sync, manifest rewrite, or ingest pressure - reduce write amplification or compaction debt - node / cloud-disk corroborated: - check volume limits - evaluate higher-performance disk - involve the cloud vendor when platform evidence is strong Version / config note: - async IO is enabled by default since `v8.1` - older versions or clusters with `store-io-pool-size == 0` are more vulnerable to inline IO stalls ## 11. Minimal IO-Jitter Checklist Always check these before writing the conclusion: - workload volume and request mix in the same window - hotspot or store-load redistribution - which store is the outlier - which quantile exposes the tail - whether the issue is read-side, write-side, or both - whether node or cloud-disk evidence exists - what evidence is still missing ## 12. Output Template State the conclusion in this order: 1. whether workload growth or hotspot change was ruled out 2. which store is the slow outlier 3. whether the issue is on the read path, write path, or both 4. which layer is the best current suspect: - raft-engine - RocksDB-KV - node / cloud-disk 5. whether the conclusion is `suspected IO jitter` or supported by node / cloud-disk evidence Do not compress everything into `disk slow`.