# Uneven Distribution Of Leader/Region: Solution (A)


![left](./student-guide/diagram/25-Leader-Region-solution-01-01.png)

------------------------------------------------------------
  - First, observe the ratings of different stores

    - If the ratings of different stores are close, the PD believes that it is already in equilibrium at this point, the possible reasons are:
      	- The presence of hot spots causes an imbalance in the load. For further analysis based on the information related to hotspot scheduling, please refer to the section on hotspot scheduling below.
     	 - There are a large number of empty regions or small regions, resulting in a very large difference in the number of leaders in different stores, resulting in overburdening raftstores. If you need to turn on Region Merge and speed up merging as much as possible, please refer to the section on Region Merge below.
      	- There are differences in the software and hardware environment of different stores, such as disk space capacity. Then you can adjust the leader-weight and region-weight as appropriate to control the leader/region distribution.
      	- Other unknown reasons. You can also adjust the weight to the bottom of the pocket by adjusting the leader-weight and region-weight to the distribution that the user feels reasonable.
      

    - If the ratings of different stores vary greatly, it is necessary to check the Operator-related metrics further, paying special attention to the generation and execution of the Operator. The relevant monitoring is as follows:
        - Grafana PD —> Operator —> Schedule Operator Create：The creation of the Operator
        - Grafana PD —> Operator —> Operator finish duration：Operator execution time consuming
        - Grafana PD —> Operator —> Operator Step duration：When it takes time to execute different Operator Steps


One situation is when the generated schedule is normal, but the schedule is slow
        	- Scheduling speed is limited by limit configuration
          	- The limit configured by default for PD is conservative, and the leader-schedule-limit or region-schedule-limit can be increased as appropriate without significantly impacting normal business. In addition,  `max-pending-peer-count` And `max-snapshot-count`  restrictions can also be relaxed.
         	 - The Store Limit configuration is too conservative and limits the speed of operator consumption
        	- There is competition for other scheduling tasks running in the system at the same time, causing the balance speed to fail
          	- In this case, if the balance schedule has a higher priority, you can stop other schedules first or limit the speed of other schedules. For example, if you do an offline node operation when the region is not balanced, the offline schedule and region balance will preempt the `region-schedule-limit`  quota.  `replica-schedule-limit` Tune down to limit the speed of offline scheduling, or simply set it `disable-replace-offline-replica = true`  to temporarily turn off the offline process.
       	 - Scheduling is running too slowly
          	- You can check the time taken by the Operator Step to make a judgment. Normally steps that do not involve sending and receiving snapshots (such as TransferLeader, RemovePeer, PromoteLearner, etc.) should take milliseconds to complete, and the number of steps involved in snapshots (such as AddLearner, AddPeer, etc.) Ten seconds. If the time taken is obviously too high, it may be caused by excessive pressure on TiKV or bottlenecks in the network, etc., which need to be analyzed on a case-by-case basis.

- Another case is that the corresponding balance schedule could not be generated
        	- scheduler is not enabled
		For example, the corresponding Scheduler is deleted, or the limit is set to 0.
        	- Unable to schedule due to other constraints
		For example, if there is an evict-leader-scheduler in the system, it is not possible to migrate the Leader to the corresponding Store at this time. Another example is setting the Label property, which also causes some stores to not accept Leader. What you see at this point is that the number of leaders on a TiKV is 0. Take evict-leader as an example:
         		 - Grafana PD —> Scheduler —> Scheduler is running: Check if there is an evict-leader schedule in the cluster
          		- Use pd-ctl config show all to confirm that the TiKV of evict-leader exists
          		- Remove evict-leader scheduling on TiKV using pd-ctl
       	 - Imbalance due to limitations of cluster topology
		For example, in a cluster of 3 replicas and 3 data centers, due to the limitation of replica isolation, 3 replicas in each region are distributed in different data centers. If the number of stores in these 3 data centers is not the same, the final scheduling will converge to a state where each data center is balanced, but globally unbalanced.