# Flow Control Diagnosis Read this file when `ServerIsBusy`, `deadline exceeded`, throttling, or read/write admission rejection is visible. This guide is derived from the internal note: - `Diagnosis of the "ServerIsBusy" Error` ## When To Read Read this file when the active branch is `flow-control` and the visible symptom is rejection, throttling, or admission control rather than a directly proven lower-layer bottleneck. ## Goal The goal is to determine: - which exact protection mechanism fired first - whether it is on the read path, write path, or both - which metric or log proves the threshold crossing - which lower-layer bottleneck triggered the protection ## Key Questions Keep these questions aligned with the main skill: - which exact `ServerIsBusy`, throttle, or rejection mechanism fired first - is it on the write path, the read path, or both - which metric or log proves the threshold crossing or effective crossing - what lower-layer bottleneck triggered the protection - is threshold tuning only mitigation, or is there also a proven root bottleneck ## Guardrails - do not stop at `ServerIsBusy` - the real RCA is the lower-layer bottleneck that caused the protection to fire - use logs first for the `reason` field when available - do not clear a path only because a sampled gauge looks normal ## 1. What Flow-Control Diagnosis Must Answer Every flow-control analysis must answer these questions in order: 1. Which exact `ServerIsBusy`, throttle, or rejection mechanism fired first? 2. Is it on the write path, the read path, or both? 3. Which metric or log proves the threshold crossing or effective threshold crossing? 4. What lower-layer bottleneck triggered the protection? 5. Is threshold tuning only a mitigation, or is there also a proven root bottleneck? Important wording rule: - do not stop at `ServerIsBusy` - the real RCA is the bottleneck that caused the protection to fire ## 2. First Identify The Reason Prefer logs first, because they often include the concrete `reason`. Useful sources: - TiDB logs - TiKV logs - `TiDB -> KV Errors -> TiClient Region Error OPS` - `TiDB -> KV Errors -> KV Backoff OPS` - `TiDB -> KV Errors -> KV Backoff Duration` Important practice: - for `KV Backoff Duration`, group by `type` - that makes it clear whether `server_is_busy`, `deadline_exceeded`, or another backoff dominates Guardrail: - metrics show categories and rate - logs often show the exact `reason` ## 3. Write Flow Control ### A. `storage.scheduler-pending-write-threshold` Work mechanism: - this is an admission-control limit at the scheduler entry - when in-flight write bytes exceed the threshold, new writes are rejected instead of queued Main checks: - `TiKV -> Errors -> Server is Busy -> scheduler-{instance}` - `TiKV -> Scheduler -> Scheduler writing bytes` - `TiKV -> Raft Engine -> Bytes / Written` Interpretation: - `scheduler-{instance}` plus high scheduler writing bytes: - scheduler pending-write threshold is plausible - if `scheduler-{instance}` appears but `Scheduler writing bytes` does not look high: - do not clear this path yet - the gauge is sampled and can miss a short burst - raise `Bytes / Written` quantile from `0.9999` to `1` if needed ### B. `storage.flow-control.soft/hard-pending-compaction-bytes-limit` Work mechanism: - as pending compaction bytes rise from the soft to hard limit, reject probability rises from `0%` to `100%` Main checks: - `TiKV -> Flow Control -> Scheduler discard ratio` - `TiKV -> Flow Control -> Compaction pending bytes` - `TiKV -> Thread CPU -> Rocksdb` - compaction flow and GC signals Interpretation: - discard ratio above `0`: - this protection path is active - compaction pending bytes high on one CF or instance: - compaction cannot keep up Then continue downward: - check whether compaction CPU is saturated - check whether a large data drop or delete burst changed the workload ### C. `deadline exceeded` On The Write Path Work mechanism: - write requests have a deadline - if throttling or slow write handling pushes the request past that deadline, TiKV returns `ServerIsBusy` with reason `deadline is exceeded` Main checks: - TiDB or TiKV logs for `deadline is exceeded` - `TiKV -> gRPC -> gRPC message Duration` - `TiKV -> Flow Control -> Throttle duration` - write-path latency stages if throttling is not the explanation Interpretation: - if throttle duration rises with slow writes: - find the throttle cause next - if slow writes are not explained by throttle: - continue into write-path diagnosis; flow control is only the symptom ### D. `storage.flow-control.l0-files-threshold` Work mechanism: - TiKV throttles write speed when L0 SST files exceed the threshold Main checks: - `TiKV -> Flow Control -> Throttle duration` - `TiKV -> Flow Control -> Scheduler throttled CF` - `TiKV -> Flow Control -> Flow controller factors` Then find the L0 source: - foreground write / memtable flush - PD scheduling - SST ingest from Lightning, add index, or related ingest activity ### E. `storage.flow-control.memtables-threshold` Work mechanism: - same pattern as the L0 threshold, but triggered by too many immutable memtables Main checks: - `TiKV -> Flow Control -> Throttle duration` - `TiKV -> Flow Control -> Scheduler throttled CF` - `TiKV Trouble-Shooting -> Write Stall -> Immutable mem-table number` ### F. `raftstore is busy` Work mechanism: - the request reaches the raft router, but the channel to raftstore is already full - TiKV returns `ServerIsBusy` immediately Main checks: - `TiKV -> Server Is Busy -> channelfull-{instance}-{type}` - raftstore CPU and downstream raftstore stages Interpretation: - this does not prove CPU by itself - if raftstore CPU is not exhausted, continue through raftstore stages to find the real bottleneck ### G. `gc worker is busy` Work mechanism: - when GC worker pending tasks exceed the internal limit, new GC tasks are rejected Main checks: - logs with `gc worker is busy` - logs showing `failed to schedule gc task` Then split: - too many GC tasks at once - or GC worker handling is too slow ## 4. Read Flow Control ### A. `tidb_load_based_replica_read_threshold` Work mechanism: - when estimated read-pool wait exceeds the threshold, TiDB routes reads to followers Main checks: - logs showing `estimated wait time exceeds threshold` - `TiKV -> Error -> Coprocessor error` - `TiKV -> Unified Read Pool -> Wait duration` - `TiKV -> Unified Read Pool -> Running tasks` Interpretation: - high wait duration and long queue: - read-pool pressure is real ### B. `readpool.unified.max-tasks-per-worker` Work mechanism: - when total running tasks exceed `max_tasks_per_worker * pool_size`, new tasks are rejected Main checks: - `TiKV -> Error -> Coprocessor error` with `max_pending_tasks_exceeded` - `TiKV -> Unified Read Pool -> Running tasks` ### C. `server.end-point-memory-quota` Work mechanism: - coprocessor requests are rejected if the endpoint memory quota is exhausted before execution Main checks: - `TiKV Coprocessor Overview -> Memory Quota` - `TiKV -> Error -> Coprocessor error` with `memory_quota_exceeded` ### D. `deadline exceeded` On The Read Path Work mechanism: - coprocessor requests are aborted when handling exceeds the deadline budget Main checks: - `tikv_coprocessor_request_error{reason="deadline_exceeded"}` - `TiKV -> Unified Read Pool -> Wait duration` - `TiKV -> Unified Read Pool -> Task Execute Duration` Interpretation: - high wait plus high execute duration: - the read pool is overloaded or individual tasks are too slow ## 5. Gauge And Sampling Caveat Some factor metrics are gauges and can miss short spikes. Important examples: - `tikv_scheduler_writing_bytes` - other flow-control factor metrics such as L0 or memtable-related gauges Rules: - do not clear a flow-control path only because the gauge looks normal - use supporting metrics such as reject reason, throttle duration, discard ratio, queue length, and higher-quantile burst metrics ## 6. Diagnostic Order Use this order: 1. identify the exact `reason` 2. decide whether it is write-side or read-side 3. prove the threshold crossing or effective crossing 4. find the lower-layer bottleneck 5. only then discuss threshold tuning Do not reverse steps `3` and `4`. ## 7. Mitigation Direction Match mitigation to the proven mechanism: - scheduler pending-write threshold: - reduce burst write pressure - remove write bottlenecks - scatter hotspots if present - compaction pending bytes: - improve compaction throughput - reduce short-term delete or drop bursts - L0 or memtable threshold: - reduce foreground write pressure - improve compaction - reduce PD scheduling or SST ingest pressure when they are the source - raftstore busy: - investigate raftstore CPU or downstream stalls - read-pool rejection: - optimize slow queries - scale read-pool capacity - memory quota: - raise quota only if memory headroom exists - otherwise reduce query memory footprint Guardrail: - threshold tuning without bottleneck analysis is only symptom mitigation ## 8. Minimal Flow-Control Checklist Always check these before writing the conclusion: - TiDB `KV Backoff` grouped by `type` - whether logs expose the `reason` - whether it is write-side or read-side - which threshold or factor is implicated - which lower-layer bottleneck triggered it - whether gauges may have missed a short spike - whether the proposed mitigation only masks the symptom ## 9. Output Template State the conclusion in this order: 1. which exact `ServerIsBusy` or throttle mechanism fired 2. whether it is write-side or read-side 3. which metric or log proves the threshold crossing 4. which bottleneck triggered the protection 5. whether the proposed mitigation addresses the root cause or only the symptom Do not compress everything into `flow control`.