# Leader Drop Diagnosis Read this file when the primary symptom is TiKV `leader drop`, not generic read or write latency. This guide is derived from the internal KB page: - `[Troubleshooting Guide] Common reasons for TiKV leader drops` ## When To Read Read this file when the active branch is `leader-drop` and the main task is to explain the direct leader-movement mechanism before discussing deeper write-path or infrastructure causes. ## Goal The goal is to determine: - whether leader movement was PD-driven or follower-side re-election - which signal appeared first in the onset window - which evidence supports the direct mechanism versus only an upstream candidate cause ## Key Questions Keep these questions aligned with the main skill: - was the movement PD-driven or follower-side re-election - if PD-driven, which scheduler or operator did it - if not PD-driven, which onset signal appeared first - what evidence rules out `evict-slow-store` - which later metrics are only secondary amplification ## Guardrails - do not write `commit log duration high caused leader drop` - do not write `slow store caused leader drop` if the real meaning is only `the node became slow` - if `evict-slow-store` is not proven, do not use PD scheduling language - compare onset on the narrowest available alignment instead of relying on smoothed averages ## 1. What Leader-Drop Diagnosis Must Answer Every leader-drop analysis must answer these questions in order: 1. Was the leader movement PD-driven, or was it follower-side re-election? 2. If PD-driven, which scheduler or operator did it? 3. If not PD-driven, which signal appeared first in the onset window? 4. Is the issue isolated to one store, a subset of stores, or cluster-wide? 5. What evidence supports the direct mechanism of leader drop? 6. What evidence supports only an upstream candidate root cause? Important wording rule: - do not write `commit log duration high caused leader drop` - do not write `slow store caused leader drop` if what you really mean is `the node became slow` - if `evict-slow-store` is not proven, do not use PD scheduling language ## 2. First Split: PD-Driven Or Follower Re-Election Use this split before reading deep write-path metrics. ### A. PD-Driven Leader Movement Common cases: - `evict-leader` during upgrade or rolling restart - `evict-slow-store` after slow-score reaches 100 and PD schedules leader eviction ### B. Follower-Side Re-Election Common cases: - gRPC thread busy or stuck, so heartbeats are delayed - raftstore slowdown, so heartbeats or raft processing are delayed - network jitter between TiKV nodes - host-level stall such as allocstall or swap If you do not first decide between `A` and `B`, the rest of the analysis becomes ambiguous. ## 3. Onset Alignment Rules For leader-drop incidents, timing discipline matters. Rules: - anchor on the first visible `Leader drop` spike - compare nearby metrics on the same 15-second alignment when possible - do not rely on 1-minute smoothing to decide which signal started first - if a metric rises after leader drop already starts, treat it as secondary amplification or fallout Always align at least: - `TiKV -> Leader drop` - `TiKV -> Leader missing` - `TiDB -> KV Request -> RPC Layer Latency` - `TiKV -> Thread CPU -> gRPC` - `TiKV -> Thread CPU -> Raftstore` - key raftstore write-path tails if write slowdown is suspected ## 4. PD-Driven Branch ### 4.1 Evict-Leader During Upgrade Or Rolling Update Look for: - PD audit logs creating and deleting `evict-leader-scheduler` - operator / scheduler records that point to restart or rolling update Typical conclusion: - this is by design - do not keep searching for TiKV internal bottlenecks unless the timing does not match ### 4.2 Evict-Slow-Store This path needs strict evidence. Do not infer it from `leader drop` plus one slow TiKV. Required evidence: - `PD -> Slow store score` reaches `100` - PD scheduler or operator evidence shows `evict-slow-store` or downstream `evict leader` Useful checks: - `PD -> Abnormal stores -> Slow stores` - PD operator view - PD scheduler logs or API output Guardrail: - if the slow-score or PD scheduling evidence is missing, do not write `evict-slow-store` - instead, write that the node showed slowdown-like symptoms but PD-driven eviction is not proven ### 4.3 Slow-Store False Positives Use this only when `evict-slow-store` really happened. Typical pattern: - very heavy write rate or raft-log append backlog - slow-store inspection times out - PD triages the store as slow and evicts leaders even though the main issue is metric bias or transient backlog Evidence: - clear high write volume to that TiKV - slow-store path active - append backlog style signals ## 5. Follower Re-Election Branch Use this branch when PD-driven movement is not proven. The direct mechanism to state is: - follower did not receive valid heartbeat / lease renewal in time - or leader-side raft / heartbeat handling was delayed enough that followers re-elected Your job is then to identify which upstream bottleneck delayed those heartbeats. ## 6. Internal Causes ### 6.1 gRPC Thread Busy Mechanism: - raft heartbeats are sent and received through gRPC threads - if gRPC threads are too busy, heartbeat transmission or reception is delayed - followers may start a new election Main evidence: - `TiKV -> Thread CPU -> gRPC thread pool` - per-thread imbalance in gRPC CPU - `TiKV -> Error -> Server report failures -> full` - `TiDB -> KV Request -> RPC Layer Latency` Interpretation rule: - gRPC must be abnormal before or at onset - if gRPC CPU rises only after leader drop begins, it is not the primary trigger Typical mitigation direction: - increase `server.grpc-concurrency` - increase `server.grpc-raft-conn-num` - reduce oversized request payloads if large SQL or message size is suspected ### 6.2 gRPC Stuck By Large SQL Or Payload Mechanism: - large SQL or large messages keep gRPC threads busy in serialization or deserialization - heartbeats are delayed - followers re-elect Main evidence: - gRPC thread stress - `server report failures -> full` - TiDB RPC layer latency spike - large SQL or abnormal message-size clues from logs / SQL context / profiling Do not claim this branch without a payload-size clue. ### 6.3 Raftstore Disk I/O Jitter This branch is especially important on versions before async IO is enabled by default. Mechanism: - if raftstore-side IO or related write-path work stalls, heartbeat and raft processing can be delayed - followers then re-elect Main evidence: - onset-aligned rise in raftstore write-path tails - especially `propose wait`, `append log`, `process ready`, or other raftstore stages depending on version and config - write-path slowdown begins before or at leader-drop onset Version / config clue: - if `raftstore.store-io-pool-size == 0` or async IO is effectively disabled, inline IO is more plausible Guardrail: - do not blame disk from averages alone - if only `max` tails spike while `p99` stays flat, write `bursty node-local IO jitter` or `extreme tail latency`, not `sustained disk saturation` ### 6.4 High Workload Pressure Mechanism: - even without explicit IO hang, heavy load can slow raftstore enough that heartbeats or processing are delayed Evidence: - workload-shape or hotspot change in the same window - broad thread-pool pressure - raftstore or apply path stress Do not use this branch if workload volume is flat and only one node shows a local tail spike. ### 6.5 Internal Slow-Store-Like Stall Sometimes TiKV store messages themselves are delayed for several seconds, for example `LatencyInspect` or other internal store messages. Evidence: - TiKV logs showing store message handling taking seconds - local store-message backlog or internal stall evidence This is not the same as PD `evict-slow-store`. ## 7. Hardware / Node Causes ### 7.1 Network Jitter Mechanism: - leader heartbeats cannot reach followers in time - follower lease expires and re-election starts Main evidence: - network latency or jitter in the same onset window - retransmit or drop signals - broad RPC-layer latency increase without clear TiKV internal bottleneck - `Commit log duration` rises more than `Append log duration` Useful checks: - blackbox ping - `Node exporter -> TCP retransmit` - `Node exporter -> TCP drop` For a dedicated network workflow: - read `references/network-issues.md` ### 7.2 Memory Allocstall Or Swap Mechanism: - kernel stalls the process during memory reclaim - TiKV pauses in arbitrary stages - heartbeats and processing are delayed Main evidence: - `Node exporter -> Vmstat -> allocstall` - swap activity - leader drop coincides with host-level memory stall This branch can explain both gRPC and raftstore symptoms if the whole process is intermittently stalled. ## 8. Evidence Priority Use this order: 1. direct mechanism evidence - PD scheduler / operator evidence - or follower re-election timing evidence 2. onset-aligned first abnormal signal 3. supporting path metrics 4. host or config evidence 5. comments or prior ticket conclusions as reference only Do not promote comments into the main conclusion when the metrics disagree. ## 9. Minimal Leader-Drop Checklist Always check these before writing the RCA: - `TiKV -> Leader drop` - `TiKV -> Leader missing` - `TiDB -> KV Request -> RPC Layer Latency` - `PD -> Abnormal stores -> Slow stores` - PD scheduler / operator evidence if `evict-leader` or `evict-slow-store` is suspected - `TiKV -> Thread CPU -> gRPC` - `TiKV -> Thread CPU -> Raftstore` - `TiKV -> Error -> Server report failures` - `TiKV -> Error -> Server is busy` - relevant write-path or read-path tails on the affected store only - workload-shape comparison in the same window - host-level network or memory stall signals if TiKV internals do not fully explain onset ## 10. Output Template Use this structure: 1. Direct mechanism - `PD-driven evict-leader` - `PD-driven evict-slow-store` - `follower-side re-election after heartbeat delay` 2. Earliest abnormal signal 3. Supporting evidence 4. Ruled-out branches 5. Candidate upstream root cause 6. Missing evidence 7. Mitigation / next step Example wording: ```text The direct mechanism of leader drop was follower-side re-election after heartbeat delay, not PD-driven evict-slow-store. The earliest abnormal signal was on at