# TiKV Leader Drop Troubleshooting Author: Gao Zhenjiao --- ## Objectives - Recognize Leader drop issue symptoms - Understand the causes of Leader drop issues - Diagnose and solve Leader drop issues ## Introduction Leader drop happens when a region leader migrates from one TiKV node to another node. There are various underlying causes that could lead to leader drop, some are expected operations, while some are problematic requiring further investigation. For example, PD scheduler transfers leaders or peers among nodes for load balancing purposes, which is normal; TiKV write performance bottleneck could trigger a large number of leader drop, which is an indication of a problem. When there is a problem, the leader drop error is usually accompanied by a large number of other errors. Grafana TiKV-Details --> Errors panel shows the overall number of errors. Below is the explanation of each errors from [TiDB documentation](https://www.google.com/url?q=https://docs.pingcap.com/tidb/stable/grafana-tikv-dashboard&sa=D&source=editors&ust=1654304429266597&usg=AOvVaw1Dtgsb4v_xf3XB6aowjvbP) ![explanation](/hc/article_attachments/6587781540503/image9.png) ## Symptoms ### Leader drop symptoms (I)[\[a\]](#cmnt1)[\[b\]](#cmnt2) When there is a large number of Leader transferring in the cluster, we can find the corresponding metric in both TiDB and TiKV's Grafana monitoring. - TiDB --> KV Errors --> TiClient Region Error OPS: the number of Region related error messages returned by TiKV ![Generated](/hc/article_attachments/6587802115095/image14.png)[\[c\]](#cmnt3) - TiKV-Details --> Errors --> Leader drop: The count of dropped leaders per TiKV instance ![Generated](/hc/article_attachments/6587781645847/image13.png) - or TiKV-Trouble-Shooting --> Leader Drop --> Leader ### Leader drop symptoms (II) When the problem happens, the duration of TiDB could increase, impacting business. In addition, we can usually see a lot of errors in the TiKV-Details \-> Error panel. - TiDB-> Query Summary -> Duration. Observe duration spikes ![Generated](/hc/article_attachments/6587773004951/image16.png) - TiKV-Details --> Errors panel shows a lot of errors ![Generated](/hc/article_attachments/6587802128791/image15.png) ![Generated](/hc/article_attachments/6587781742743/image18.png) Leader transfers, i.e. leaders migrated from one TiKV node to another, could be the result of normal PD scheduling, which is expected. The leader drop problem that needs investigation is usually accompanied by a large number of TiKV and TiDB errors, which causes the increase of duration at TiDB side,  affecting the overall cluster performance. ## Causes ### Raft Group Election Timeout A TiKV instance has multiple Regions on it. The Raftstore module drives the Raft state machine to process Region messages. These messages include processing read or write requests on Regions, persisting or replicating Raft logs, and processing Raft heartbeats. The workflow of Raftstore is shown below: ![Generated](/hc/article_attachments/6587781745175/image17.png) From the above diagram, you can see that requests from the TiDB servers, after passing through the gRPC and storage modules, become read and write messages of KV (key-value), and are sent to the corresponding Regions. These messages are not immediately processed but are temporarily stored. Raftstore polls to check whether each Region has messages to process. If a Region has messages to process, Raftstore drives the Raft state machine of this Region to process these messages and perform subsequent operations according to the state changes of these messages. For example, when write requests come in, the Raft state machine stores logs into disk and sends logs to other Region replicas; when the heartbeat interval is reached, the Raft state machine sends heartbeat information to other Region replicas. We can see that Raftstore mainly processes two types of messages, messages generated by write requests and heartbeat messages between Regions. Currently there is no separate thread in Raftstore to handle Region heartbeat messages. Heartbeat messages are sent to other Region peers via gRPC.  Messages generated by write requests will be written into local disks and then sent to its peers. Therefore the heartbeat messages depend on two factors: the write performance and the network latency. When disk performance or network latency degrades, or TiKV restarts abnormally, the Raft state machine may not send out heartbeat messages in a timely fashion, when reaching the Raft timeout, triggering the Leader election. To help better understand how heartbeat messaging mechanism works, the Raft Group model can be conceptually simplified as a batch processing system. The thread pool of the Raftstore processes messages from different regions in a batch, including both write requests and heartbeat messages. The Raftstore can only start processing the next batch after completing the current batch of messages, including disk writes. Therefore, disk or network performance degradation could lead to heartbeat messages not being sent in time and triggering election timeouts. ![election timeouts](/hc/article_attachments/6587785045271/image2.png) For details, refer to [Best Practices for TiKV Performance Tuning with Massive Regions | PingCAP Docs](https://www.google.com/url?q=https://docs.pingcap.com/tidb/stable/massive-regions-best-practices&sa=D&source=editors&ust=1654304429274470&usg=AOvVaw3WoGS7EWi1HthAe-mFafIB) ### PD Scheduling In PD scheduling, balance-leader and hot-balance-region-schedulers are the two schedulers that involve Region Leader transfers. The purpose of balance-leader is to spread the pressure of handling client requests and to make the leader distribution even across Storage nodes. In scheduling, the sum of all region sizes corresponding to leaders in a store is used as the score. From v4.0 / v3.0.15 onwards it also provides a score based on the count of leaders as the basis for scheduling. 1. Scoring is based on the amount of resources available to each Store 2. Constantly select Leaders from higher scoring Stores to migrate to lower scoring Stores For hot regions scheduling, the server uses the hot-region-scheduler. Since TiDB v3.0, the process is performed as follows: 1. Count hot regions by determining read/write traffic that exceeds a certain threshold for a certain period based on the information reported by stores. 2. Redistribute these regions in a similar way to load balancing. For hot-write regions, the hot-region-scheduler attempts to redistribute both region peers and leaders; for hot-read regions, hot-region-scheduler only redistributes region leaders. ![hot-write regions](/hc/article_attachments/6587785083287/image5.png) [PD Scheduling Policy Best Practices](https://www.google.com/url?q=https://docs.pingcap.com/tidb/stable/pd-scheduling-best-practices/&sa=D&source=editors&ust=1654304429276204&usg=AOvVaw0sklmiJ64irdz4X1iNcpgT) provides a detailed description of PD Scheduling ## Troubleshooting and Resolution ### Identify the source of lead drops When the TiKV-Details Grafana monitor observes a large number of Leader drops, check the following metrics to determine the source of the leader drop - TiKV-Details --> Errors --> Leader drop ![Generated](/hc/article_attachments/6587802314391/image21.png) - TiKV-Details --> Raft message --> Messages ![Generated](/hc/article_attachments/6587811055383/image19.png) Use this metric to confirm where the Leader drop is triggered: Raftgroup election or PD scheduling. - transfer_leader is generated by PD scheduling - The prevote (on by default) message is generated by the election timeout There are log entries generated when leader transferring happens. For examples: - pd.log entry example showing leader transfer from balance-leader \[2022/01/06 17:20:01.148 -05:00\] \[INFO\] \[operator_controller.go:438\] \["add operator"\] \[region-id=8033\] \[operator="\\"balance-leader {transfer leader: store 4 to 1} (kind:leader, region:8033(84,5), createAt:2022-01-06 17:20:01.148814866 -0500 EST m=+26321.806460722, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, steps:\[transfer leader from store 4 to store 1\])\\""\] \[additional-info="{\\"sourceScore\\":\\"6.00\\",\\"targetScore\\":\\"5.00\\"}"\] - tikv.log entry showing leader transfer scheduled by pd \[2022/02/10 16:23:00.417 -05:00\] \[INFO\] \[[pd.rs:1110](https://www.google.com/url?q=http://pd.rs:1110&sa=D&source=editors&ust=1654304429278304&usg=AOvVaw3Ity9l3kieYDpbJUbCmrc9)\] \["try to transfer leader"\] \[to_peer="id: 6124 store_id: 4"\] \[from_peer="id: 6122 store_id: 1"\] \[region_id=6121\] - tikv.log entry showing leader transfer due to election timeout \[2022/02/10 16:23:52.626 -05:00\] \[INFO\] \[raft.rs:2304\] \["\[term 43\] received MsgTimeoutNow from 6124 and starts an election to get leadership."\] \[from=6124\] \[term=43\] \[raft_id=6122\] \[region_id=6121\] \[2022/02/10 16:23:52.626 -05:00\] \[INFO\] \[raft.rs:1517\] \["starting a new election"\] \[term=43\] \[raft_id=6122\] \[region_id=6121\] The next two sections provide the detailed steps for these two cases. ### Raftgroup election timeout For Raftgroup election timeout issue, follow the steps listed in the below diagram ![Raftgroup election timeout](/hc/article_attachments/6587801806487/image3.png) #### Raft store CPU utilization - TiKV-Details --> Thread CPU --> Raft store CPU ![Generated](/hc/article_attachments/6587797707543/image20.png) Check the Raft store CPU utilization, in general Raftstore CPU utilization is recommended to be kept less than 60%. That is, when the Raftstore thread count configuration parameter store-pool-size is set to the default value of 2, the value on the Raft store CPU panel should be kept within 120%. When the utilization rate is too high, consider adjusting the store-pool-size (default value of 2, i.e., 200% when full) parameter to avoid the possibility that the heartbeat of the Region cannot be issued in time due to insufficient thread pool. Note the decision to adjust store-pool-size should take various factors into consideration, otherwise it could impact performance adversely. Refer to [Tune TiKV Thread Pool Performance | PingCAP Docs](https://www.google.com/url?q=https://docs.pingcap.com/tidb/stable/tune-tikv-thread-performance/&sa=D&source=editors&ust=1654304429280286&usg=AOvVaw2-fc6g7Kl-qIyHfgT7RXlV) for details. #### Massive number of regions If the cluster has a large number of regions,  Raftstore needs to take time to process the heartbeat messages, which can cause delay. There are two features that, when enabled, can reduce the number of heartbeat messages. - Hibernate Region - Hibernate Region feature minimizes the number of heartbeat messages for idle Regions. When the Hibernate Region feature is enabled, Raftstore stops sending tick messages to the idle Regions, thus avoiding generating heartbeat messages. Hibernate Region is enabled by default in TiKV master. For details, refer to [Configure Hibernate Region](https://www.google.com/url?q=https://docs.pingcap.com/tidb/stable/tikv-configuration-file&sa=D&source=editors&ust=1654304429281126&usg=AOvVaw3KCa8QCpypk1Jyz8t9ILSW). - Region Merge - Enabling Region Merge also reduces the number of Regions. In contrast to Region Split, Region Merge is the process of merging small adjacent Regions through scheduling. After deleting data in a cluster or executing Drop Table / Truncate Table statements, small or even empty Regions can be merged to reduce the consumption of resources. See details in "[PD Scheduling Diagnosis and Handling FAQ](https://www.google.com/url?q=https://pingcap.feishu.cn/docs/doccnHtYhamWLA1Bkvp5Qti77sd&sa=D&source=editors&ust=1654304429281587&usg=AOvVaw0Sl8uG8KdV2SzmCDTdrKk2)". These two cases are also explained in detail in [Best Practices for TiKV Performance Tuning with Massive Regions](https://www.google.com/url?q=https://docs.pingcap.com/tidb/stable/massive-regions-best-practices&sa=D&source=editors&ust=1654304429282004&usg=AOvVaw0Ym1zty6UMVNmXZ7b2aekf) If Raft store CPU resources are sufficient and Raftstore heartbeat burden is reduced by Hibernate Region and Region Merge, and there is still a large number of Leader drops generated by Raftgroup election, the TiKV cluster may have slow write scenarios such as disk stalling, causing the Raft store stuck, or the network between stores has long latency, eventually triggers the election timeout. #### Disk latency/bottleneck As explained in the previous section[\[d\]](#cmnt4), when there are disk stalls or performance bottlenecks, the Raft state machine could reach timeout, causing Leader drops. Check the Disk Latency panel for IO latency. - Overview -> System Info -> IO Util -> Disk Latency ![Generated](/hc/article_attachments/6587811158679/image23.png) For IO troubleshooting, refer to [Troubleshoot High Disk I/O Usage in TiDB | PingCAP Docs](https://www.google.com/url?q=https://docs.pingcap.com/tidb/dev/troubleshoot-high-disk-io/&sa=D&source=editors&ust=1654304429283274&usg=AOvVaw0R4mQ1pfZI_R248JHi9GDy) #### TiKV cluster writes slow From the time the client initiates a write request to the time it finally returns a successful write status, the whole write process is relatively long, and the Raftstore is mainly involved in the TiKV side. When the Raftstore has slow write performance problems, then to a certain extent, the messages will pile up, triggering Leader drop. ![triggering Leader drop](/hc/article_attachments/6587810840599/image10.png) A typical cause is when RocksDB has a write stall, causing the TiKV node to drop Leaders. Note from 5.2.0, TiKV introduced a new flow control mechanism to replace the previous RocksDB write stall mechanism to improve the stability of TiKV flow control. ![Generated](/hc/article_attachments/6587802383255/image24.png) - TiKV Details -> Errors -> Server is busy ![Generated](/hc/article_attachments/6587797890583/image25.png) For RocksDB flow control, refer to [Troubleshooting Guide: Write Stalls in TiKV](https://www.google.com/url?q=https://docs.google.com/document/d/1JK4xX62p1WUOl-PwLEfuBxs3bwRtAvq6PfPAeHtYcjY/edit%23heading%3Dh.96hr3tvoi8ax&sa=D&source=editors&ust=1654304429284639&usg=AOvVaw1bp62Ioqb4xCpH0JBFIIys). ![Write Stalls in TiKV](/hc/article_attachments/6587802333463/image22.png) The following are some references for TiKV write performance issue - [Common ways to deal with slow TiDB database writes](https://www.google.com/url?q=https://learning.pingcap.com/kng/course/package/video/d5c90cfe66f249f3871c2769b184b079_b8c2b5b11e4a4d169688cc4ff69ee1a3.html?uniqueid%3D1642698849836%26uniqueid%3D637783244501272565&sa=D&source=editors&ust=1654304429285293&usg=AOvVaw2h5lupvFAqdI4RuJh8tGyq) - [TiKV writing flow and metrics](https://www.google.com/url?q=https://docs.google.com/presentation/d/1zaAtaj8EG5fxBojcpg2MQ8tWE614be00tFjl2mIFYrc/edit%23slide%3Did.g446c4deb4d_0_341&sa=D&source=editors&ust=1654304429285709&usg=AOvVaw0dH5Qf5KsOVwRvUIE_b_ez) - [TiKV performance tuning](https://www.google.com/url?q=https://edu.pingcap.com/learner/courseinfo/id:156&sa=D&source=editors&ust=1654304429286074&usg=AOvVaw1lt7FD80nWGxCiPXQ7ojeP) #### Network Latency Below diagnostics can help determine if there are network problems among stores: - TiKV-Details --> Errors --> Server report failures: The number of unreachable indicates communications issues among stores. Note that Unreachable could also be the result of other issues in the TiKV node, such as hangs, OOM, etc. ![Unreachable](/hc/article_attachments/6587810627991/image1.png) - Blackbox_exporter --> Ping Latency: With this monitoring item, you can determine whether the network communication between each Store is normal. ![Ping Latency](/hc/article_attachments/6587810720407/image6.png) - Overview -> System Info -> TCP Retrans: TCP Retrans counts the number of TCP packets that need to be retransmitted ![TCP Retrans](/hc/article_attachments/6587797500951/image12.png) - The gRPC communication exception messages in TiKV logs record the detailed error information returned by the network layer. \[2021/09/03 16:44:22.488 +08:00\] \[ERROR\] \[draft_client.rs:407\] \["connection aborted"\] \[addr=172.16.6.13:22194\] \[receiver_err="Some( RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: \\"Socket closed\\", details: \[\] }))"\] \[sink_error="Some(RpcFinished(Some(RpcStatus { code : 14-UNAVAILABLE, message: \\"Socket closed\\", details: \[\] })))"\] \[store_id=63\] \[2021/09/03 16:44:22.488 +08:00\] \[ERROR\] \[draft_client.rs:707\] \["connection abort"\] \[addr=172.16.6.13:22194\] \[store_id=63\] Contact the network administrator if an underlying network issue is identified. #### TiKV unexpected restart - TiKV-Details --> Cluster --> Uptime: indicate how long a TiKV has been up and running. Unexpected TiKV reboots are usually caused by - OOM - Panic: tikv.log has clear Panic information ![Generated](/hc/article_attachments/6587781592855/image11.png) For TiKV OOM issue, refer to [TiKV Database OOM Problem Diagnosis and Handling (in Chinese)](https://www.google.com/url?q=https://learning.pingcap.com/kng/course/package/video/d5c90cfe66f249f3871c2769b184b079_194e00deae9c4ad8a79a2cee4900f998.html?uniqueid%3D1643993388004%26uniqueid%3D637796189884368241&sa=D&source=editors&ust=1654304429288978&usg=AOvVaw2X1DVvu5usiRVWZr3RPryD). Tikv.log has detailed panic information. There are various causes. Check known issues and submit a new one when needed. ### Leader Scheduling Leader scheduling by PD is normally generated by the balance-leader and hot-region-scheduler schedulers, and usually does not result in problems, except in the case of evict-leader scheduling. PD could also schedule leaders indirectly, typically resulting from Raftgroup election timeout. The leader transfer from PD scheduling will also be reflected in the metric of ‘Leader drop’. In such cases, we need to investigate the cause of Raftgroup election timeout as documented in the previous section. - PD --> Operator --> Schedule operator create ![Generated](/hc/article_attachments/6587772885527/image8.png) - PD --> Statistics - balance --> Store leader score / size / count ![Generated](/hc/article_attachments/6587797335703/image4.png) - TiKV-Details --> Errors --> Leader drop ![Generated](/hc/article_attachments/6587810765591/image7.png) ## Summary Leader drop is a general issue with various root causes and symptoms. This article provides some detailed insights of problem diagnostics, it can be used as a reference guide for step by step troubleshooting to identify the problematic areas and approach corresponding resolutions. [\[a\]](#cmnt_ref1)It will be also helpful to call out the symptoms from KvClient perspective? (e.g. what's the error of KvClient when the region's leader is dropped) [\[b\]](#cmnt_ref2)Is this available in some grafana panel? [\[c\]](#cmnt_ref3)I believe this is only informative. If the tidb client attempt to write to a tikv that is no longer master it will retry.. correct? [\[d\]](#cmnt_ref4)id remove this