# [KB]关于 TiCDC 同步卡住 checkpoint 不推进，日志告警  The resolvedTs is fallen back in kvclient  的问题.md

## Description

TiCDC 查看任务同步卡住，checkpoint 不推进，并且期间 TiKV 多次 OOM

```shell
tiup ctl:v5.0.x cdc changefeed query --pd=http://xxx.xxx.xxx.xxx:2379 --changefeed-id=xxx { "state": "normal", "tso": xxx, "checkpoint": "xxxx-xx-xx xx:xx:xx.xxx", "error": null }
```

## Root Cause

Issue : [https://github.com/pingcap/ticdc/issues/1626](https://github.com/pingcap/ticdc/issues/1626) BUG 导致 TiKV 中存在 region 的 resolved ts 停止推进。

## Diagnostic Steps

1. 查看 TiCDC 日志有包含以下关键字告警，并且有打印具体 region id 和 resolvets 的时间，可以使用 pd-ctl tso 确认卡住的大概时间 ： "The resolvedTs is fallen back in kvclient" "region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock"

2. 根据告警的 region id 查看所有 TiKV 日志,查看最后一次 register 和 deregister 的日志信息 grep * | grep register

    ```shell
    grep 2256981 * | grep register
    [2021/05/15 23:48:22.801 +08:00] [INFO] [endpoint.rs:484] ["cdc register region"] [downstream_id=DownstreamID(2020)] [req_id=87945] [conn_id=ConnID(98)] [region_id=2256981]
    [2021/05/16 05:28:48.574 +08:00] [INFO] [endpoint.rs:351] ["cdc deregister"] [deregister="Deregister { deregister: "region", region_id: 2256981, observe_id: ObserveID(1887), err: Request(message: "peer is not leader for region 2256981, leader may None" not_leader { region_id: 2256981 }) }"]
    ```

3. 查看这个region id 最后一次 schedlue resolver 的日志 grep * | grep 'schedule resolver ready'

    ```shell
    grep 2256981 * | grep 'schedule resolver ready'
    [2021/05/15 23:48:21.777 +08:00] [INFO] [endpoint.rs:1228] ["cdc resolver initialized and schedule resolver ready"] [takes=176ms] [observe_id=ObserveID(1887)] [lock_count=0] [resolved_ts=0] [downstream_id=DownstreamID(1887)] [conn_id=ConnID(36)] [region_id=2256981]
    ```

4. 如果步骤 2 最后一次 register 时间 > 步骤 3 schedule resolver 的时间，说明 TiKV 卡了

## Resolution

预计 v5.0.2 修复,修复 PR [https://github.com/tikv/tikv/pull/10028](https://github.com/tikv/tikv/pull/10028)

## Workaround

先尝试 TiCDC 任务暂停重启

```
cdc cli changefeed pause --pd=http://xx.xx.xx.xx:2379 --changefeed-id= cdc cli changefeed resume --pd=http://xx.xx.xx.xx:2379 --changefeed-id=
```

如果 TiCDC 无法解决，需要尝试重启 region 所在的 TiKV 节点 pd-ctl region 可以查看 region 所在 tikv 节点 tiup cluster reload -N

## Affect Version(optional)

v4.0.0、v4.0.1、v4.0.2、v4.0.3、v4.0.4、v4.0.5、v4.0.6、v4.0.7、v4.0.8、v4.0.9、v4.0.10、v4.0.11、v4.0.12、v5.0.0、v5.0.1