---
name: tikv-cdc
description: |
  Reference handbook for TiKV and TiCDC interaction issues.
  Use when investigating CDC-triggered TiKV panic, changefeed lag,
  invalid PD endpoint reuse, store-address reuse, or CDC synchronization
  failures that depend on TiKV state.
---

# TiKV CDC Handbook

Known TiKV and TiCDC interaction patterns for on-call diagnosis.

Source note: adapted from the Feishu `Storage oncall handbook` <https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb>.

## When To Use This Skill

Use this skill when:

- CDC processing triggers TiKV panic
- changefeed lag increases after scale-in or scale-out
- TiCDC keeps using stale PD or store endpoints
- CDC behavior depends on TiKV in-memory lock or store metadata state

## First Checks

- Confirm whether the first symptom is TiKV panic, changefeed lag, or endpoint churn.
- Check whether scale-in, scale-out, PD topology change, or TiKV address reuse happened recently.
- Record affected `changefeed`, `store_id`, `addr`, and exact TiCDC/TiKV versions.
- Distinguish whether the root cause is CDC metadata drift or TiKV runtime state.

## Known Cases

### 🧩 TiKV panic caused by missing rollback-lock handling in CDC check

- References:
  - [tikv#19025](https://github.com/tikv/tikv/issues/19025)
  - [tikv#18498](https://github.com/tikv/tikv/issues/18498)
- Phenomenon: CDC does not properly handle locks left on non-primary keys after rollback. TiKV may still keep a rolled-back in-memory lock `t1`, while RocksDB contains a newer lock `t2`. CDC sees `t2` but reads `t1`, causing assertion failure and panic.

```text
assertion `left == right` failed
 left: TimeStamp(458355287523328275)
 right: TimeStamp(458355287523328316)
```

- Workaround: restart the TiKV node so in-memory locks are cleared.
- Affected version: `v8.5+`
- Related oncall:
  - v8.5.2 CDC assertion failure `GTOC-8041`
  - v8.5.3 TiKV assert panic `ONCALL-7972`
  - v8.5.3 TiKV panic at CDC module `TICKET-6722` / `ONCALL-8005`

### CDC keeps adding a scaled-in PD back as an etcd endpoint ([tiflow#12368](https://github.com/pingcap/tiflow/issues/12368))

- Phenomenon: after scaling in a PD node, TiCDC keeps re-adding the removed PD address into its etcd endpoint set every 30 seconds.
- Example logs:

```text
[2025/10/16 14:00:30.537 +08:00] [INFO] [client.go:447] ["update endpoints"] [numChange=3->4] [lastEndpoints="[http://127.0.0.1:2379,http://127.0.0.1:2384,http://127.0.0.1:63065]"] [endpoints="[http://127.0.0.1:2382,http://127.0.0.1:2379,http://127.0.0.1:63065,http://127.0.0.1:2384]"]
```

- Related oncall:
  - v7.5.6 CDC sync stops after restart due to invalid PD endpoint used `APID-11856` / `GTOC-7966`

### CDC changefeed lag keeps increasing due to TiKV address reuse ([tiflow#12162](https://github.com/pingcap/tiflow/issues/12162))

- Detailed reference: [`references/tikv-address-reuse-changefeed-lag.md`](references/tikv-address-reuse-changefeed-lag.md)
- Phenomenon: when scaling out a new TiKV using the address of a previously scaled-in TiKV, PD assigns a new `store_id`, but CDC may continue using the old `store_id` and fail to fetch store metadata.
- Example log:

```text
[2025/05/01 15:59:07.296 +08:00] [INFO] [shared_stream.go:93] ["event feed check store version fails"] [namespace=default] [changefeed=xxxx] [streamID=15] [storeID=6] [addr=10.107.4.96:20160] [error="[CDC:ErrGetAllStoresFailed]get stores from pd failed: type:UNKNOWN message:"invalid store ID 6, not found" "]
```

- Workaround: restart the CDC node.
- Related oncall:
  - v7.5.6 CDC sync stops after restart due to invalid PD endpoint used `APID-11856` / `GTOC-7966`

## Related Skills

- For TiKV panic analysis, read `skills/diagnosis/tikv-panic/SKILL.md`.
- For store or region movement during scale operations, read `skills/diagnosis/tikv-raftstore/SKILL.md`.