# [KB] Kill pd leader 后 QPS 掉零时间过长

## Key Words

kill PD, PD, QPS

## Description

1. kill PD leader 的故障场景模拟, sysbench 模拟业务负载，kill PD leader 之后, QPS 暂时掉零 0, 持续时间 18s 左右。PD 日志每个 6s 刷 [pd] tso stream is not ready 的 info 日志：

```golang
[2022/06/21 15:22:39.372 +08:00] [ERROR] [client.go:792] ["[pd] create tso stream error"] [dc-location=global] [error="[PD:client:ErrClientCreateTSOStream]create TSO stream failed, retry timeout"]
[2022/06/21 15:22:39.372 +08:00] [INFO] [client.go:788] ["[pd] tso stream is not ready"] [dc=global]
[2022/06/21 15:22:39.372 +08:00] [INFO] [base_client.go:275] ["[pd] cannot update member from this address"] [address=http://127.0.0.1:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\" target:127.0.0.1:2379 status:TRANSIENT_FAILURE: error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\" target:127.0.0.1:2379 status:TRANSIENT_FAILURE"]
[2022/06/21 15:22:39.372 +08:00] [INFO] [base_client.go:275] ["[pd] cannot update member from this address"] [address=http://127.0.0.1:2382] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2382: connect: connection refused\" target:127.0.0.1:2382 status:TRANSIENT_FAILURE: error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2382: connect: connection refused\" target:127.0.0.1:2382 status:TRANSIENT_FAILURE"]
[2022/06/21 15:22:39.372 +08:00] [INFO] [base_client.go:275] ["[pd] cannot update member from this address"] [address=http://127.0.0.1:2384] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2384: connect: connection refused\" target:127.0.0.1:2384 status:TRANSIENT_FAILURE: error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2384: connect: connection refused\" target:127.0.0.1:2384 status:TRANSIENT_FAILURE"]
[2022/06/21 15:22:39.372 +08:00] [ERROR] [base_client.go:144] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://127.0.0.1:2379 http://127.0.0.1:2382 http://127.0.0.1:2384] error"]
[2022/06/21 15:22:39.579 +08:00] [INFO] [client_batch.go:581] ["batchRecvLoop re-create streaming fail"] [target=127.0.0.1:20160] [forwardedHost=] [error="context deadline exceeded"]
[2022/06/21 15:22:40.481 +08:00] [INFO] [client_batch.go:581] ["batchRecvLoop re-create streaming fail"] [target=127.0.0.1:20160] [forwardedHost=] [error="context deadline exceeded"]
[2022/06/21 15:22:40.559 +08:00] [INFO] [domain.go:578] ["server topology syncer restarted"]
[2022/06/21 15:22:41.278 +08:00] [INFO] [client_batch.go:573] ["batchRecvLoop re-create streaming success"] [target=127.0.0.1:20160] [forwardedHost=]
[2022/06/21 15:22:41.412 +08:00] [INFO] [client_batch.go:581] ["batchRecvLoop re-create streaming fail"] [target=127.0.0.1:20160] [forwardedHost=] [error="context deadline exceeded"]
[2022/06/21 15:22:43.279 +08:00] [INFO] [client_batch.go:573] ["batchRecvLoop re-create streaming success"] [target=127.0.0.1:20160] [forwardedHost=]
[2022/06/21 15:22:44.375 +08:00] [ERROR] [client.go:907] ["[pd] update connection contexts failed"] [dc=global] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""]
[2022/06/21 15:22:45.279 +08:00] [INFO] [client_batch.go:573] ["batchRecvLoop re-create streaming success"] [target=127.0.0.1:20160] [forwardedHost=]
[2022/06/21 15:22:45.376 +08:00] [INFO] [client.go:788] ["[pd] tso stream is not ready"] [dc=global]
[2022/06/21 15:22:46.587 +08:00] [INFO] [client_batch.go:581] ["batchRecvLoop re-create streaming fail"] [target=127.0.0.1:20160] [forwardedHost=] [error="context deadline exceeded"]
[2022/06/21 15:22:48.583 +08:00] [INFO] [client_batch.go:573] ["batchRecvLoop re-create streaming success"] [target=127.0.0.1:20160] [forwardedHost=]
[2022/06/21 15:22:50.379 +08:00] [ERROR] [client.go:907] ["[pd] update connection contexts failed"] [dc=global] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""]
[2022/06/21 15:22:51.380 +08:00] [INFO] [client.go:788] ["[pd] tso stream is not ready"] [dc=global]
[2022/06/21 15:22:51.637 +08:00] [INFO] [data_window.go:249] ["Error exists when getting the SQL Metric."]
[2022/06/21 15:22:56.383 +08:00] [ERROR] [client.go:907] ["[pd] update connection contexts failed"] [dc=global] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""]
[2022/06/21 15:22:57.384 +08:00] [INFO] [client.go:788] ["[pd] tso stream is not ready"] [dc=global]
[2022/06/21 15:23:02.387 +08:00] [ERROR] [client.go:907] ["[pd] update connection contexts failed"] [dc=global] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""]
[2022/06/21 15:23:02.387 +08:00] [ERROR] [client.go:792] ["[pd] create tso stream error"] [dc-location=global] [error="[PD:client:ErrClientCreateTSOStream]create TSO stream failed, retry timeout"]
[2022/06/21 15:23:02.387 +08:00] [INFO] [client.go:788] ["[pd] tso stream is not ready"] [dc=global]
[2022/06/21 15:23:02.388 +08:00] [INFO] [base_client.go:275] ["[pd] cannot update member from this address"] [address=http://127.0.0.1:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\" target:127.0.0.1:2379 status:TRANSIENT_FAILURE: error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\" target:127.0.0.1:2379 status:TRANSIENT_FAILURE"]
[2022/06/21 15:23:02.388 +08:00] [ERROR] [pd.go:236] ["updateTS error"] [txnScope=global] [error="[PD:client:ErrClientCreateTSOStream]create TSO stream failed, retry timeout"]
[2022/06/21 15:23:02.391 +08:00] [INFO] [base_client.go:350] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2384] [old-leader=http://127.0.0.1:2379]
[2022/06/21 15:23:07.391 +08:00] [ERROR] [client.go:907] ["[pd] update connection contexts failed"] [dc=global] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""]
[2022/06/21 15:23:08.391 +08:00] [INFO] [client.go:788] ["[pd] tso stream is not ready"] [dc=global]
```

## Root Cause

当 kill PD leader 后 pd cluster 触发重新选举，但是 [PD client](https://github.com/tikv/pd/blob/f04b89e82b72446833316687ebe08f1bea3639ce/client/client.go##L784-L822) 重试逻辑存在每次 loop 时间过长问题，具体参考 [bug](https://github.com/tikv/pd/issues/5207) 描述。

## Diagnostic Steps

1. 新部署的 TiDB 集群，使用 sysbench 进行负载模拟；
2. kill PD leader；
3. sysbench QPS 掉零时长大约 18s 左右，重复测试时间稳定；

## Resolution

升级到未受影响版本，[fix pr](https://github.com/tikv/pd/pull/5551)。

## Workaround

N/A

## Affected Versions

v5.0.0~v5.0.6

v5.1.0~v5.1.4

v5.2.0~v5.2.4

v5.3.0~v5.3.3

v5.4.0~v5.4.3

v6.0.0

v6.1.0~v6.1.1

v6.2.0

v6.3.0