# Cross-Cutting Checks Read this file when you need to confirm that slowness is really on TiKV, or when gRPC, flow control, snapshot, or Resource Control may be involved. ## 1. Confirm TiKV-Side Slowness Start with: - `TiDB -> KV Request OPS` - `TiDB -> KV Request duration by store/type` - `TiDB -> KV Request -> RPC Layer Latency` Use this split: ```plaintext KV Request duration = TiDB gRPC send/receive + network transmit + TiKV request processing + TiKV gRPC send/receive ``` `RPC Layer Latency` is the TiDB-side gRPC plus network portion: ```plaintext RPC Layer Latency = TiDB gRPC send/receive + network transmit ``` Interpretation: - `KV Request duration` up, `RPC Layer Latency` flat: likely TiKV-side - both up together: suspect network or TiDB-side gRPC first If `KV Request duration` and `RPC Layer Latency` rise together: - read `references/network-issues.md` From the first pass, always extract: - slow request type - slow stores versus healthy stores - QPS / workload-shape changes - whether tails are sparse or broad If the issue is a short-lived tail on one or a few stores while workload stays flat: - raise the quantile instead of trusting `p99` alone - read `references/io-jitter.md` If `ServerIsBusy`, `deadline exceeded`, or backoff-by-type becomes prominent: - read `references/flow-control.md` For fast triage: - read path: `BatchGet`, `Cop` - write path: `Prewrite`, `Commit` ## 2. Basic Health Check Check: - `TiKV -> Cluster -> CPU` - `TiKV -> Cluster -> Memory` - `TiKV -> Cluster -> Available size` - `TiKV -> Thread CPU -> Busy Threads (>80%)` - `TiKV -> Error -> Server report failures` - `TiKV -> Error -> Server is busy` - `PD -> Abnormal stores -> Slow stores` - `PD -> Abnormal stores -> Lowspace stores` - `PD -> Abnormal stores -> Others stores` Interpretation reminders: - `Server report failures` often points to network or internal communication problems - `Server is busy` means a flow-control or overload symptom exists, not necessarily the root cause - if only a subset of stores is slow, follow that subset all the way through node, thread, disk, and network checks - if the incident is short-lived and store-local, do not rely on average-only node disk panels - if `unreachable` appears, separate true network failure from gRPC-thread exhaustion or stuck threads ## 3. gRPC Layer ### What gRPC does gRPC threads: - receive requests and deserialize - dispatch requests to worker pools - serialize, optionally compress, and send responses They are the entry and exit layer, not the main data-processing layer. ### What `tikv_grpc_msg_duration_seconds` means Included: - downstream handling time in non-gRPC threads - waiting time before the response is actually sent Not included: - time before a message gets scheduled onto a gRPC thread - serialization / deserialization cost outside the measured window - time after the metric is recorded but before bytes really leave the node ### How to interpret high gRPC duration High gRPC duration usually means one of two things: 1. downstream work is slow 2. the response is ready but gRPC cannot send it promptly Diagnosis order: 1. compare with downstream read / write stages first 2. if downstream stages are normal, inspect gRPC CPU and per-thread imbalance Important checks: - `TiKV -> Thread CPU -> GRPC` - per-thread imbalance in gRPC threads - overall TiKV CPU saturation even when gRPC average CPU does not look dramatic Typical gRPC cases: - gRPC CPU exhausted or badly imbalanced - mitigation: increase `grpc-concurrency` or overall TiKV CPU - network jitter - signals: `RPC Layer Latency` rises with gRPC symptoms - huge SQL payloads causing serialization / deserialization spikes - typical symptom: bursty gRPC stalls, raft side tails, and oversized SQL text - gRPC compression overhead - typical symptom: large response sizes plus gRPC stalls without an obvious average CPU spike ## 4. Flow Control ### Write flow control Write flow control exists to protect lower layers when they cannot keep up. Main knobs: - `storage.scheduler-pending-write-threshold` - `storage.flow-control.soft-pending-compaction-bytes-limit` - `storage.flow-control.hard-pending-compaction-bytes-limit` - `storage.flow-control.l0-files-threshold` - `storage.flow-control.memtables-threshold` Observed behaviors: - hard reject with `ServerIsBusy` - probabilistic reject under pending compaction pressure - throttle wait instead of reject Key rule: - flow control is evidence that lower layers are slower than incoming write production - do not stop at flow control; continue downward to find what is actually slow - for detailed per-reason diagnosis, read `references/flow-control.md` ### Read flow control Read flow control is simpler and usually returns `ServerIsBusy`, after which clients retry with backoff. Common related settings: - `tidb_load_based_replica_read_threshold` - `max-tasks-per-worker` - `end-point-memory-quota` ## 5. Snapshot Path Read both read-path and write-path investigations often depend on snapshot behavior. Main panel: - `Storage -> Storage async snapshot duration` Normally, `p99` should be only a few milliseconds. Tens of milliseconds or worse is already suspicious. Split snapshot into: - pure local read: - `Storage -> Storage async snapshot duration (pure local read)` - read index: - `Storage -> Read index propose wait duration` - `Storage -> Read index confirm duration` Interpretation: - pure local slow: suspect thread-pool pressure or RocksDB snapshot / mutex effects - read index slow but pure local clean: suspect raftstore processing or network / gRPC path ### Snapshot bottleneck checklist Check thread pools involved: - `Thread CPU -> Raftstore` - `Thread CPU -> Unified Read Pool` - `Thread CPU -> Scheduler Worker Pool` - `Thread CPU -> GRPC` Typical cases: - thread CPU bottleneck in one of the involved pools - raftstore slow, especially high `Raft IO -> Propose wait` - metric bias from [tikv/issues/17579](https://github.com/tikv/tikv/issues/17579) - affected until: - `v6.5.12` - `v7.1.6` - `v7.5.4` - symptom: async snapshot and scheduler pool wait move together - huge SST count causing RocksDB global lock wait - signal: single-store SST count above `100k`, pure local snapshot `p9999` around `100ms` - fixed in `v8.5.4` via [tikv/issues/18743](https://github.com/tikv/tikv/issues/18743) - large compaction jobs coinciding with snapshot stalls - gRPC / network stalls when read index is slow but local snapshot and raftstore are not ## 6. Resource Control Resource Control is enabled by default in `v7.5` and `v8.5`. Inside TiKV it mainly appears as priority-based scheduling, not RU accounting at the point where the request has already entered TiKV. Main panel: - `Resource Control -> Priority Task Control -> Priority Task Wait Duration` Interpretation: - if this wait rises with latency, low- or medium-priority tasks may be intentionally delayed Important version note: - known issue: [tikv/issues/18939](https://github.com/tikv/tikv/issues/18939) - affects `v7.5` and `v8.5` - disabled in `v8.5.4` - workaround: set involved resource groups to high priority so they bypass this limiter