---
name: tikv-raftstore
description: |
  Reference handbook for TiKV raftstore and region-lifecycle issues.
  Use when investigating split, add learner, snapshot, region relocation,
  peer cleanup, large-region scheduling, ingest SST during movement, or
  raft-log and peer-state related incidents. Includes diagnostic patterns,
  workarounds, and version notes.
---

# TiKV Raftstore Handbook

Known region-lifecycle and raftstore incident patterns for TiKV on-call diagnosis.

Source note: adapted from the Feishu `Storage oncall handbook` <https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb> and `Raftstore handbook` <https://pingcap.feishu.cn/wiki/HaZTwLVC0iDxngk2hX3cbR33nqn>.

## When To Use This Skill

Use this skill when:

- the incident involves region split, merge, peer transfer, add learner, or snapshot
- recovery is slow because peers cannot catch up or snapshots are blocked
- a panic is clearly tied to raftstore state transitions
- region relocation or large-region scheduling is the trigger

## First Checks

- Identify the affected `region_id`, `peer_id`, and `store_id`.
- Confirm whether the issue happens during split, add learner, snapshot apply, or peer cleanup.
- Check whether hotspot scheduling or load-based split is acting on the same region.
- Verify whether the region is oversized or moving between stores.

## Topics

### Split and hotspot interaction

- `Hotspots in a subrange of one region cause TiKV CPU spikes`
- `TiKV may panic when split and hot-region scheduling race`

### Large-region side effects

- `Hot row updates cause oversized regions`
- `Large-region scheduling causes level-0 SST accumulation and flow-control throttling`

### Snapshot and learner issues

- `Ingest SST blocked by write pressure slows down peer recovery`
- `Multiple TiKV nodes offline causes add-learner timeout loops`
- `Offline node delete-snapshot backlog blocks region-worker`
- `TiKV may repeatedly restart after a crash when apply-snapshot state is left inconsistent`
- `Some peers stay apply-busy during TiKV startup and delay leader scheduling`

### Region movement and cleanup

- `Ingest SST during data movement causes cluster latency spikes`
- `Destroying a region and GC raft log in the same batch may panic TiKV`
- `clean_stale_range` causes large compaction

## Common Commands

```bash
tiup ctl:<version> pd -u http://<pd>:2379 region <region_id>
tikv-ctl --host <tikv_addr> region-properties -r <region_id>
tikv-ctl --pd <pd_addr> raft region -r <region_id>
```

## Detailed Cases

### ✅ Hotspots in a subrange of one region cause TiKV CPU spikes ([tikv#18932](https://github.com/tikv/tikv/issues/18932))

- Description: with load-base-split enabled, TiKV tries to find a split key when a region becomes hot. After it finds a candidate split key, it still checks whether the access pattern would be balanced after the split. This is controlled by `split.split-balance-score`.
  - The smaller the value, the stricter the balance requirement.
  - The default is `0.25`.
  - If a region has hotspots only in one contiguous subrange, the threshold may be too strict, so TiKV cannot find a valid split key and the region remains unsplit.
- Diagnostic hint: in Grafana `tikv-details -> raft admin -> load base split event`, if `no_balance_key` stays high while TiKV CPU also stays high, this strongly suggests a subrange hotspot inside one region.
- Workaround: increase `split.split-balance-score` appropriately:

```sql
SET CONFIG tikv `split.split-balance-score` = '0.55';
SHOW CONFIG WHERE name LIKE "split.split-balance-score";
```

- Related oncall:
  - v8.5 how to mitigate read imbalance across TiKV `GTOC-7880` / `NAID-11609`

### ✅ Hot row updates cause oversized regions

- Description: when GC malfunctions, for example GC is stuck or GC lifetime is too large, old MVCC versions of a frequently updated key cannot be cleaned up in time. Because all MVCC versions of one key must stay in the same region, the region containing that hot key can grow to gigabytes and become unsplittable.
- Impact:
  - region size imbalance
  - scheduling becomes difficult
  - hotspots become more concentrated in CPU, memory, and I/O
- Diagnostic steps:
  - check the metric `tikv-details -> server -> approximate region size`
  - search TiKV logs for `invalid split key`
  - inspect region properties:

```bash
tikv-ctl --host <addr> region-properties -r <region_id>
```

- Workaround:
  - if GC is the cause, fix the GC issue or reduce GC lifetime
  - if delayed RocksDB compaction is the cause, try manual compaction:

```bash
tikv-ctl --host <addr> region-properties -r <region_id>
tikv-ctl --pd IP:port --bottommost force -c write --region <region-id>
tikv-ctl --pd IP:port --bottommost force -c default --region <region-id>
tikv-ctl --host <addr> region-properties -r <region_id>
```

- Related oncall:
  - v8.1.1 troubleshooting error `failed to handle split req` `NAID-11323` / `GTOC-7609`
  - v8.5.1 200 GB region could not split `TICKET-6121`

### ✅ Large-region scheduling causes level-0 SST accumulation and flow-control throttling

- Description: when a region becomes large, it covers a wider key range and includes more SST files. During scheduling operations such as peer migration, balance-region, or merge, if business writes overlap with those SST ranges, SST ingestion is more likely to land in level 0.
- Result:
  - more SST files are ingested into L0
  - L0 files accumulate quickly and trigger flow control
  - write throughput drops and latency increases, sometimes with `8022` errors
- Key metrics:
  - `tikv-details -> rocksdb-kv -> ingestion picked level`
  - `tikv-details -> flow-control`
- Workaround: increase `l0-files-threshold`. For versions earlier than `v8.0`, changing this may still require restarting TiKV.
- Related oncall:
  - v7.1.5 TiKV region size adjusted to 960 MB and frequent `8022` errors appeared `ONCALL-7598` / `TICKET-5547`
  - v7.1.5 cluster suddenly stalled after a traffic burst without obvious resource saturation `TICKET-5535`

### Ingest SST blocked by write pressure slows down peer recovery ([tikv#17078](https://github.com/tikv/tikv/issues/17078))

- Description: when a TiKV node stays offline for more than 30 minutes, PD starts replica recovery. If write pressure is high and L0 files accumulate, SST ingest may be paused, so snapshot apply stays pending for a long time and peer recovery slows down.
- Workaround: avoid taking TiKV offline during peak traffic.
- Related oncall:
  - v7.1.0 two TiKV nodes were taken offline one after another, causing one replica to stay down and another to remain stuck in snapshot apply `GTOC-6932`

### Multiple TiKV nodes offline causes add-learner timeout loops

- Detailed reference: [`references/add-learner-timeout-loop.md`](references/add-learner-timeout-loop.md)
- Description: PD may observe repeated `add learner` timeouts and fall into a `replace-rule-offline(leader)-peer` loop.
- Root cause:
  - PD issues `add learner`, times out after about 20 minutes, then later observes the learner promoted and removes it again
  - contributing factors include region-worker CPU saturation, snap-generator saturation, and frequent L0 overflow blocking snapshot apply
- Workaround:
  - throttle scheduling on PD when resources are saturated
  - evict leaders from the snapshot-heavy node to reduce snapshot generation pressure
  - if necessary, force restart TiKV
- Related oncall:
  - v6.5.7 taking 10 TiKV nodes offline at the same time got stuck `TICKET-4746` / `TICKET-4681` / `ONCALL-7328`

### ✅ Offline node delete-snapshot backlog blocks region-worker ([tikv#12587](https://github.com/tikv/tikv/issues/12587))

- Description: during scale-in, delete-snapshot tasks can pile up on the offline node, block generate-snapshot, and stall replica migration.
- Workaround: upgrade to a fixed version, or evict leaders from the offline node to reduce snapshot work.
- Fixed version: `v8.4.0+`
- Related oncall:
  - v6.5.6 TiKV scale-in caused slow region migration `ONCALL-7429` / `TICKET-4975`

### ✅ Slow SST ingest logging holds RocksDB mutex and causes latency spikes ([tikv#18549](https://github.com/tikv/tikv/issues/18549))

- Phenomenon: when SST ingest is slow, TiKV logs `sst ingest is too slow`. While collecting SST range metadata for the log, TiKV may hold the RocksDB KV mutex and cause read/write latency spikes.
- Sample log:

```text
[2025/06/16 04:56:33.853 +00:00] [WARN] [sst_service.rs:484] ["sst ingest is too slow"] [sst_ranges="[([122, 116, 128, 0, 0, 0, 0, 0, 0, 255, ... 215, 255, 245])]"] [thread_id=13]
```

- Workaround: upgrade to a fixed version.
- Fixed version: `v8.5.3` / `v8.1.3` / `v7.5.7` / `v7.1.7` / `v6.5.13`
- Related oncall:
  - v8.1.2 QPS dropped during index creation and returned error code `8027` `TCOC-4434`

### Ingest SST during data movement causes cluster latency spikes ([tikv#5911](https://github.com/tikv/tikv/issues/5911) [tikv#18096](https://github.com/tikv/tikv/pull/18096))

- Detailed reference: [`references/ingest-sst-during-data-movement.md`](references/ingest-sst-during-data-movement.md)
- Phenomenon: `apply wait` becomes high while `apply` itself stays low and the apply pool remains mostly idle. In practice, `apply wait` latency often correlates strongly with snapshot apply.
- Root cause: scale-in, scale-out, and merge operations move data. Deleting old regions during that process may use SST ingest, which increases `apply wait` and write latency, often because ingest interacts with RocksDB global mutex contention.
- Workaround:
  - follow the SST-ingest mitigation SOP and consider:
    - [tikv#18461](https://github.com/tikv/tikv/pull/18461): avoid SST ingestion during destroy peer
    - [tikv#17408](https://github.com/tikv/tikv/pull/17408): avoid SST ingestion during region merge
    - [tikv#18096](https://github.com/tikv/tikv/pull/18096): do not block foreground writes while applying raft snapshots by ingesting SST
    - [tikv#18522](https://github.com/tikv/tikv/pull/18522): do not block foreground writes during add index by ingesting SST
  - scale out in smaller steps; avoid `1:3` or larger expansion ratios
  - during peak traffic, throttle new nodes with `store limit add-peer`
- Related oncall:
  - v7.5.6 ingest SST caused cluster latency spikes `GTOC-7709` / `APID-11430`
  - v6.5.4 latency `999` doubled while scaling from 3 TiKV to 9 TiKV `GTOC-6948` / `ST-612`

### `clean_stale_range` causes large compaction ([tikv#19113](https://github.com/tikv/tikv/issues/19113) [tikv#18107](https://github.com/tikv/tikv/issues/18107) [tikv#18040](https://github.com/tikv/tikv/pull/18040))

- Description: `clean_stale_range` may merge ranges from multiple regions and ingest them together, producing an SST file that covers a much larger key range than intended. This can later trigger very large compaction tasks.
- Example logs:

```text
[db/external_sst_file_ingestion_job.cc:491] [AddFile] External SST file /var/lib/tikv/snap/del_range_xxx.sst.tmp was ingested in L5 with path /var/lib/tikv/db/123456.sst (global_seqno=xxxxxx)
[INFO] EVENT_LOG_v1 {"time_micros": xxxxxx, "event": "ingest_finished", "files_ingested": ["file", "/var/lib/tikv/db/123456.sst", "level", 5], "lsm_state": [x,x,xx,xxx,xxx]}
[2025/11/11 19:58:46.298 +00:00][4][INFO] EVENT_LOG_v1 {"time_micros": xxxxx, "job": xxxxx, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L5": [123456], "files_L6": [xxx,xxxx...(thousands of files)], "score": -1, "input_data_size": 213074130014}
```

- Workaround: enlarge disk capacity so the large compaction can finish, or scale out the cluster.
- Related oncall:
  - v7.5.1 available TiKV storage dropped from 1 TB to 200 GB in one hour `GTOC-8034`

### Some peers stay apply-busy during TiKV startup and delay leader scheduling

- Detailed reference: [`references/startup-apply-busy-and-leader-scheduling.md`](references/startup-apply-busy-and-leader-scheduling.md)
- Phenomenon: after TiKV restarts, some peers can remain in an apply-busy state for minutes, `ServerIsBusy` may appear, and PD does not move leaders onto that TiKV even after raft-log lag returns to zero.
- Workaround:
  - increase `raftstore.min-pending-apply-region-count` when the startup path is dominated by this behavior
- Fixed version:
  - `v8.5.2` / `v7.5.7`
- Related oncall:
  - v8.5.X upgrade process had slow TiKV region leader callbacks `TICKET-8183`
  - v8.5.3 how to make TiKV scale up faster `TCOC-4544`
  - v8.1 upgrade process showed uneven TiKV restart time `TCOC-3137`

### ✅ TiKV may panic when split and hot-region scheduling race ([tikv#17469](https://github.com/tikv/tikv/issues/17469))

- Phenomenon: a hot region is being transferred with `add learner`, while `load-base-split` concurrently splits the same region. The newly split peer may be created twice on the target store and later panic when snapshot handling starts.
- Diagnostic hints:

```text
[2024/08/31 10:14:07.945 +08:00] [Info] [apply.rs:1611] ["execute admin command"] [command="cmd_type: BatchSplit splits { requests { split_key: >>SPLIT_KEY<< new_region_id: 25435689 new_peer_ids: 25435690 new_peer_ids: 25435691 new_peer_ids: 25435692 new_peer_ids: 25435693 } right_derive: true share_source_region_size: true }"] [index=14749] [term=7] [peer_id=25435688] [region_id=25433193]
[2024/08/31 10:14:07.946 +08:00] [Info] [peer.rs:4093] ["insert new region"] [store_id=4] [is_uninitialized_peer_exist=true] ...
[2024/08/31 10:14:07.947 +08:00] [Info] [apply.rs:1611] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 25435693 store_id: 4 role: Learner } } }"] [index=7] [term=6] [peer_id=25435693] [region_id=25435689]
```

- Workaround:
  - TiKV restarts automatically after the panic, so immediate manual intervention is usually unnecessary
  - if it happens repeatedly and upgrade is not possible, tune `load-base-split` and hot-region scheduling to reduce concurrency on the same region
  - upgrade to `v6.5.11`, `v8.1.2`, `v7.5.4`, `v7.1.6`, `v6.1.8`, or `v5.4.4`
- Related oncall:
  - v6.5.6 TiKV panic while receiving snapshot `ONCALL-7373` `TICKET-4905`
  - v6.5.6 TiKV panic with `no entry found for key` `TICKET-5190` `ONCALL-7482`

### ✅ Destroying a region and GC raft log in the same batch may panic TiKV ([tikv#17840](https://github.com/tikv/tikv/issues/17840))

- Phenomenon: TiKV destroys a region and compacts raft logs in the same batch. When the GC raft-log path runs after the region has already been cleaned up, TiKV may hit `Option::unwrap()` and panic.
- Diagnostic hints:
  - before the panic, the region is usually migrated away from the store
  - the region has already been fully cleaned up on the local TiKV
  - the panic happens around:

```rust
if let Some(e) = self.cache.entry(idx) {
    Ok(e.get_term())
} else {
    Ok(self.raft_engine.get_entry(self.region_id, idx).unwrap().unwrap().get_term())
}
```

- Fixed version:
  - `v8.5.0` / `v8.1.2`
  - `v7.5.5` / `v7.1.7`
  - `v6.5.12`
- Related oncall:
  - v6.5.4 TiKV panic `GTOC-7253`

### ✅ TiKV may repeatedly restart after a crash when apply-snapshot state is left inconsistent ([tikv#15292](https://github.com/tikv/tikv/issues/15292))

- Phenomenon: after an abnormal restart, usually with prior IO stall or similar instability, a region is in `PeerState::Apply` and starts to apply a snapshot, but the required snapshot file has already been deleted by a processed GC snapshot message. TiKV then exits with `applying snapshot failed`.
- Workaround:
- Force the affected regions to `tombstone` before starting TiKV again.

```bash
tikv-ctl --data-dir /path/to/tikv tombstone -p 127.0.0.1:2379 -r <region_id>,<region_id> --force
pd-ctl operator add remove-peer <region-id> <store-id>
```

- Upgrade to the expected fixed version: `v8.1.0` / `v7.5.3`.
- Related oncall:
  - v6.5.4 TiKV panic with `applying snapshot failed` `GTOC-6773`

## Related Skills

- For process crash-first issues, read `skills/diagnosis/tikv-panic/SKILL.md`.
- For unsafe recovery and tombstone workflows, read `skills/diagnosis/tikv-recovery/SKILL.md`.
- For storage-engine pressure and compaction issues, read `skills/diagnosis/tikv-storage/SKILL.md`.