# [Know Issue] PD 异常切换 leader 耗时超过 1 分钟，日志报错 “coordinator is stopping” 和 “scheduler has been stopped”

## Key Words

PD, leader,切换,耗时, 1min

## Description

问题出现在 leader 退出的清理流程中，通过日志看到 `balance-region-scheduler` 的推出比较慢，关键日志如下：

```golang
12021/05/17 09:22:05.090 +08:00] (INFO] [cluster.go:566] ["leader changed"] [region-id=9728976] [from=15078417] [to=23]
[2021/05/17 09:22:05.090 +08:00] (INFO] [operator_controller.go:537] ["operator finish"] [region-id=9728976] (takes=2.430045ms] [operator=")"transfer-hot-write-leader (transfer leader: store 15078417 to 23}
[2021/05/17 09:22:05.497 +08:00] [INFO] [server.go:1175] ["lease expired, leader step down"]
[2021/05/17 09:22:05.497 +08:00] [INFO] [cluster.go:311] ["metrics are reset"]
[2021/05/17 09:22:05.497 +08:00] [INF0] [cluster.go:332] ["coordinator is stopping"]
[2021/05/17 09:22:05.497 708:00] [INFO] [coordinator.go:654] ["scheduler has been stopped"] [scheduler-name=label-scheduler] [error=" context canceled" ]
[2021/05/17 09:22:05.497 +08:00] (INFO] [server.go:108] ["region syncer has been stopped"]
[2021/05/17 09:22:05.497 +08:00] [INFO] [coordinator.go:654] ["scheduler has been stopped"] [scheduler-name=balance-hot-region-scheduler] [error="context canceled" ]
[2021/05/17 09:22:05.497 +08:00] [INFO] [coordinator.go:205] ["drive push operator has been stopped" ]
[2021/05/17 09:22:05.497 +08:00] [INF0] [cluster.go:313] ["background jobs has been stopped" ]
(2021/05/17 09:22:05.522 +08:00] [INF0] [coordinator.go:103] ["patrol regions has been stopped" ]
[2021/05/17 09:22:06.861 +08:00] [INF0] [coordinator.go:654] ["scheduler has been stopped"] [scheduler-name=balance-leader-scheduler] [error="context canceled" ]
[2021/05/17 09:23:57.648 +08:00] [INFO] [operator_controller.go:424] ["add operator"] [region-id=8432158] [operator=")"balance-region (mv peer: store [15084471] to (15084441]}
[2021/05/17 09:23:57.648 +08:00] [INFO] [operator_controller.go:620] ["send schedule command"] [region-id=8432158] [step="add learner peer 15321810 on store 15084441"][source=create]
[2021/05/17 09:23:57.648 +08:00] (INF0] [coordinator.go:654] ["scheduler has been stopped"] [scheduler-name=balance-region-scheduler] ferror-"context canceled"
[2021/05/17 09: 23:3/.648 +08:00] TINFO] Icluster.go.326]["tuvruinator nas veen stopped" ]
[2021/05/17 09:23:57.653 +08:00] [INFO] [server.go:1112] ["start to campaign leader"] [campaign-leader-name=pd-10.4.78.224-2379]
[2021/05/17 09:23:57.674 +08:00] [INFO] [server.go:1131] ["campaign leader ok"] [campaign-leader-name=pd-10.4.78.224-2379]
[2021/05/17 09:23:57.675 +08:00] (INFO] (tso.go:158] ["sync and save timestamp"] (last=2021/05/17 09:22:06.397 +08:00] [save=2021/05/17 09:24:00.674 +08:00] [next=2021/05/17 09:23:57.674 +08:00]
[2021/05/17 09:23:57.676 +08:00] [INF0] [server.go:1220] ["server enable region storage]
[2021/05/17 09:23:57.836 +08:00] [INF0] [cluster.go: 280] ["load stores"] [count=181] [cost=159.123576ms]
```

## Root Cause

目前 store 数量多的情况下（100+ TiKV 实例），balance region 调度器可能没法立即退出，导致切换 leader 的时间太长。

GitHub issue:<https://github.com/tikv/pd/issues/3697>

## Diagnostic Steps

1. 通过 PD-ctl 或者 PD 监控确认旧 leader 节点，并查询旧 Leader 日志；
2. 找到关键词 ”lease expired, leader step down“，继续向下定位关键日志；
3. “coordinator is stopping“ 和 ”scheduler has been stopped“ 说明旧 Leader 在进行 evcit 过程中，卡在调度进行中没有立即退出；

## Resolution

该问题已经在一下版本修复，最终方案升级到相应的 Release 版本。

`>=` v4.0.14

`>=` v5.0.2 

## Workaround

尝试重启 PD 实例看问题是否解决，最终重启 PD 所在的云主机。

## Affect Version(optional)

`<` v4.0.13

`<` v5.0.2