# Read Path Diagnosis Read this file when `BatchGet`, `Cop`, unified read pool, or coprocessor metrics are slow. ## 1. Read Path Overview For slow read incidents, keep three questions separate: 1. Is the delay before the task starts running? 2. Is snapshot acquisition slow? 3. Is the actual scan / operator execution slow? Do not blame coprocessor immediately just because gRPC coprocessor duration is high. When the incident is store-local, prefer proving the earliest slow stage on the affected instance before comparing fleet-wide averages. ## 2. Coprocessor Handling Flow High-level flow: - gRPC receives the request - overload check may reject if read pool is already too busy - request is parsed and checked - task enters unified read pool - task may queue there - worker gets snapshot, possibly through local read or read index - handler executes scan and operators - response returns through gRPC This means slow read symptoms can come from: - gRPC - unified read pool scheduling - snapshot - actual handler execution - large response sending If coprocessor errors or `ServerIsBusy` reasons are prominent: - read `references/flow-control.md` ## 3. Main Diagnosis Order ### Step 1: Check unified read pool pressure Look at: - `Thread CPU -> Unified read pool thread` - `Unified Read Pool -> Time used level` - `Unified Read Pool -> Running task` - `Unified Read Pool -> Wait duration` - `Coprocessor Overview -> Cop wait` Interpretation: - thread CPU above about `80%` means the pool is probably saturated - `Time used level` close to thread count means the pool has little slack - high running task count means queueing pressure exists - if `cop wait` rises together with `unified read pool schedule wait`, the dominant delay is before handler execution - if running tasks surge while `handle` and `task exec` stay flat, prefer `queueing / saturation` over `coprocessor compute slow` - if ticket or comment text already says `unified read pool` is full, `time used by level` stayed near thread count, and `handle duration` stayed flat, the runtime diagnosis may already converge to `unified_read_pool_queueing` before replay-stable metrics are collected If this layer is saturated, fix hotspot or concurrency pressure before diving deeper. Important wording rule: - if the pool is the first proven bottleneck, prefer the canonical labels `unified_read_pool_queueing` or `localized_read_pool_saturation` - prose can still explain them as `unified read pool queueing` or `localized read-pool saturation` - do not jump directly to `CPU saturation` unless whole-instance CPU and quota evidence support that stronger claim ### Step 2: Compare gRPC coprocessor duration with internal coprocessor duration Compare: - gRPC coprocessor duration - `Coprocessor Overview -> Request Duration` - `Coprocessor Overview -> Cop wait` - `Unified Read Pool -> Wait duration` Interpretation: - both high: continue into coprocessor investigation - only gRPC-side metric high: go back to gRPC diagnosis - if read latency rises together with `RPC Layer Latency`, also read `references/network-issues.md` - if gRPC coprocessor duration is high but `cop wait` and read-pool wait are already high on the same instance, treat gRPC as a visible consequence first and only return to gRPC as root cause if internal queueing cannot explain the onset order ### Step 3: Split request duration into handle versus non-handle Inspect: - `Coprocessor Detail -> Handle duration` - `Unified Read Pool -> Task Execute Duration` - `Unified Read Pool -> Duration of One Time Slice` Interpretation: - handle duration high: real scan / operator work is slow - request duration high but handle duration not high: - large response size may be delaying reply - big cop tasks may be suspended and resumed - some time may be outside handler execution - wait high plus handle / task-exec flat: - the first proven bottleneck is queueing before execution - handle / task-exec high together: - continue into snapshot, RocksDB, or scan-volume analysis Also check: - `Coprocessor Overview -> Total Response Size` ### Step 4: Distinguish localized read-pool saturation from instance-wide CPU saturation Compare: - `Thread CPU -> Unified read pool thread` - TiKV process CPU - CPU quota - `Unified Read Pool -> Running task` - `Unified Read Pool -> Time used level` Interpretation: - read-pool CPU high, running tasks high, `time used level` near thread count, but process CPU still well below quota: - write `localized_read_pool_saturation` - process CPU broadly reaches quota and multiple major pools are busy: - stronger `instance-wide CPU bottleneck` wording is justified Guardrail: - do not upgrade a busy read pool into whole-host CPU saturation unless quota pressure is also proven ## 4. Snapshot-Related Read Slowness If coprocessor or read requests show high snapshot time: - go to `references/cross-cutting.md` - inspect local-read versus read-index split before blaming storage scan Main clue: - if handle duration is not high but snapshot-related metrics are, the root cause is usually before scan execution ## 5. Handle Duration High When handle duration is high, the two most common explanations are: - disk / RocksDB read path is slow - scan volume is too large ### Disk / RocksDB slow Check: - `RocksDB KV -> Get duration` - `RocksDB KV -> Seek duration` - `RocksDB KV -> SST read duration` - `Node exporter -> disk latency` - cloud-disk performance panels such as EBS throughput / IOPS / latency Interpretation: - these durations rising together usually means block-cache misses are falling through to slow storage - if the spike is short-lived and only one store is the outlier, also read `references/io-jitter.md` ### Scan volume too large Check workload volume: - `gRPC -> Request count by type (coprocessor)` - `Coprocessor Overview -> Total Request` Check scanned key volume: - `Coprocessor Overview -> KV cursor operations` - `Coprocessor Detail -> Total Ops Details` Important note: - `seek` work is especially expensive in CPU and latency terms ### MVCC-heavy scans Check for excessive MVCC traversal: - `Coprocessor Overview -> Total RocksDB Perf Statistics` - `Coprocessor Detail -> Total Ops Details` Typical signals: - a lot of `delete_skipped` - `next` much higher than `processed_keys` Interpretation: - the request is not just reading target keys; it is walking through many obsolete or skipped MVCC versions ## 6. Typical Read-Side Conclusions - unified read pool busy: - bottleneck is scheduling / CPU capacity or hotspot pressure - unified read pool queueing with flat handle / exec: - bottleneck is before execution, not coprocessor compute itself - prefer the canonical label `unified_read_pool_queueing` - high gRPC coprocessor duration with matching read-pool queueing: - treat gRPC duration as fallout unless separate gRPC evidence contradicts that - snapshot high: - bottleneck is before handler execution; follow snapshot analysis - handle duration high with RocksDB read metrics high: - bottleneck is storage-read path - handle duration high with scan / cursor metrics high: - bottleneck is query shape or data-volume explosion - request duration high but handle duration flat: - suspect gRPC send path, suspend behavior, or oversized responses ## 7. Guardrails - Do not confuse `gRPC coprocessor duration` with proof that coprocessor compute is slow. - Do not stop at `Request Duration` until handle duration, snapshot, and response-size angles are separated. - If only a few stores or threads are slow, prefer per-instance evidence over cluster averages. - Do not write `CPU saturation` when the evidence only proves `unified read pool` saturation on one store.