---
name: tikv-storage
description: |
  Reference handbook for TiKV storage-engine and disk-related incidents.
  Use when investigating RocksDB compaction pressure, Titan issues, disk
  exhaustion, SST corruption, pending compaction bytes growth, manifest
  rewrite cost, or storage permission and IO anomalies.
---

# TiKV Storage Handbook

Known storage-engine, RocksDB, Titan, and disk-behavior patterns for TiKV on-call diagnosis.

Source note: adapted from the Feishu `Storage oncall handbook` <https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb>.

## When To Use This Skill

Use this skill when:

- RocksDB level files or pending compaction bytes grow abnormally
- SST ingest, SST corruption, or manifest rewrite is involved
- disk capacity, disk encryption, or WAL/SST handling is the main concern
- Titan-specific GC or region-size estimation behavior is suspected

## First Checks

- Check disk usage, pending compaction bytes, and L0/L1 file growth.
- Confirm whether recent events include BR, Lightning, add index, scale-out, or large ingest.
- Determine whether the issue is logical throttling or physical disk failure.
- Check whether Titan is enabled.

## RocksDB

### Host and disk behavior known cases

- Detailed reference: [`references/host-and-disk-behavior-known-cases.md`](references/host-and-disk-behavior-known-cases.md)
- Use this reference when the symptom looks like a host-level or disk-level anomaly rather than a pure TiKV internal bottleneck.

### TiKV node clock skew causes cluster QPS drop to zero or compaction stall

- References:
  - [tikv#17995](https://github.com/tikv/tikv/issues/17995)
  - [tikv/rocksdb#401](https://github.com/tikv/rocksdb/issues/401)
  - [tikv#18102](https://github.com/tikv/tikv/issues/18102)
  - [rust-rocksdb#815](https://github.com/tikv/rust-rocksdb/issues/815)
- Phenomenon: when the system clock of a TiKV node is adjusted backward or forward abruptly, TiKV QPS may drop to zero and compaction may stall.
- Fixed version:
  - `v8.5.2`
  - `v8.1.3`
  - `v7.5.6`
  - `v7.1.7`
  - `v6.5.12`
- Related oncall:
  - v6.5.7 L0 continued increasing due to compaction stuck `TICKET-8240`

### Large compaction caused by non-overlapping SST ingestion ([tikv#18866](https://github.com/tikv/tikv/pull/18866))

- Phenomenon: `compaction-guard-min-output-file-size` can merge adjacent small-region data into one SST file, which later prevents new SST files from being ingested into the bottommost level and may produce large compaction tasks.
- Workaround:
  - refer to the flow-control mitigation around non-overlapping SST ingestion
  - the long-term fix is the force-partition and targeted-compaction API set introduced around [tikv#18866](https://github.com/tikv/tikv/pull/18866)
- Related oncall:
  - v8.5.2 disk usage continued to grow until the node ran out of space `TICKET-6471` / `ONCALL-7939`

### Ingesting large regions causes RocksDB throttling ([tikv#18340](https://github.com/tikv/tikv/issues/18340))

- Phenomenon: large SST ingest can overlap heavily and fall into L0, causing flow control and compaction pressure.
- Example log:

```text
build_sst_cf_file_list build: 96 files in cf default...
```

- Workaround: increase flow-control thresholds or improve the client-side ingest pattern.
- Related oncall:
  - v7.5.5 Lightning import paused split and formed large regions, causing latency jitter `GTOC-7486`

### Pending compaction bytes exceeds soft or hard threshold and returns `ServerIsBusy`

- Phenomenon: when compaction pending bytes exceeds `soft-pending-compaction-bytes-limit`, some writes start returning `ServerIsBusy`. When it exceeds the hard threshold, all writes may be rejected.
- Workaround:
  - enlarge the soft limit appropriately
  - increase compaction concurrency
  - if flow control is enabled, check `storage.flow-control.soft-pending-compaction-bytes-limit`

```sql
SHOW CONFIG WHERE name LIKE '%storage.flow-control.enable%';
```

- Related oncall:
  - v7.1.3 insert latency increased and one TiKV returned `ServerIsBusy` `TCOC-2948`

### TTL compaction saturates compaction CPU and causes L0 buildup ([tikv#15355](https://github.com/tikv/tikv/issues/15355))

- Phenomenon: RocksDB level-0 files suddenly accumulate because compaction CPU is saturated by TTL compaction.
- Workaround:
  - upgrade to `v6.5.4` / `v7.1.4` / `v7.4.0`
  - on `v6.5`, reduce compaction speed and observe; `rate-bytes-per-sec` supports online change
- Related oncall:
  - v6.5.3 lock CF level-0 buildup `ONCALL-7652` / `TICKET-5581`

### Large L2 file count triggers expensive L0-L2 compaction

- Phenomenon: when L2 contains more than one thousand files, each L0->L2 compaction may need to overlap against the entire L2 set.
- Example:

```text
[2026/01/10 02:16:51.269 +08:00][5][INFO] [db/compaction/compaction_job.cc:2082] [write] [JOB 216763] Compacting 25@0 + 1054@2 files to L2, score 62.50
```

- Workaround: speed up compaction.
- Related oncall:
  - v8.5.2 disk usage was very high while store data remained balanced `TICKET-8137`

### RocksDB compaction task gets stuck and L0 files continue to accumulate ([rocksdb#407](https://github.com/facebook/rocksdb/issues/407))

- Phenomenon: L0 file count keeps increasing but no effective compaction progress appears in RocksDB logs.
- Workaround: restart TiKV to recover.
- KB: `[Known issue] RocksDB compaction task gets stuck and level-0 files continue to accumulate`
- Related oncall:
  - v8.5.1 cluster hit `9004 Resolve lock timeout` `ONCALL-7703` / `TICKET-5782`
  - v8.1.1 cluster suddenly saw `9004` anomalies `GTOC-7457` / `NAID-11186`

### RocksDB WAL `.log` files require encryption-at-rest

- Phenomenon: plaintext sensitive data may appear in `db/*.log`, which are RocksDB WAL files and cannot be deleted directly.
- Workaround: enable encryption at rest. For existing clusters, only newly written data can be encrypted.
- Reference: <https://docs.pingcap.com/zh/tidb/stable/encryption-at-rest#tikv>
- Related oncall:
  - v6.5.4 plaintext sensitive fields appeared in `db/*.log` `TICKET-4650`

### Large numbers of SST files require increasing `max-manifest-file-size`

- Phenomenon: when SST file count becomes very large, manifest rewrite overhead increases, memory rises, region-worker CPU rises, and `apply wait` becomes high.
- Workaround:
  - enlarge `max-manifest-file-size`
  - use region merge to reduce SST file count over time
- SOP: `Mitigating TiDB Latency Spike Due to RocksDB SST Ingestion and Manifest Rewrite Issues`
- Related oncall:
  - v7.5.6 single TiKV with 7 TB data and 500k SST files caused high apply wait `GTOC-7709` / `APID-11430`

### Increased tail latency with many SST files under high snapshot/write concurrency ([tikv#18743](https://github.com/tikv/tikv/issues/18743))

- Phenomenon: when RocksDB data volume is large, operations such as apply snapshot and ordinary writes hold RocksDB global mutex longer due to many SST files, increasing `apply wait`.
- Workaround:
  - upgrade to `v8.5.4`
  - keep single-TiKV storage within about `4 TB`
- Related oncall:
  - v8.5.0 CDC downstream slowed down due to TiKV with 5 TB in one TiDB `GTOC-8217` / `GTOC-8209`

### RocksDB SST file corruption causes panic

- Phenomenon: hardware disk faults may corrupt SST files and later cause TiKV panic while reading them.
- Diagnostic command:

```bash
tikv-ctl --data-dir </path/to/tikv> bad-ssts --pd <endpoint>
```

- Related oncall:
  - single-replica DR cluster TiKV reported `block checksum mismatch` `TICKET-5264` / `ONCALL-7510`

### Corruption: SST file is ahead of WALs in CF lock caused by disk error

- Phenomenon: TiKV logs `Corruption: SST file is ahead of WALs in CF lock`.
- Root cause: disk error.
- Workaround:

```bash
tikv-ctl ldb repair --db=${rocksdb_data}
tikv-ctl --data-dir /tidb-data/tikv-20160 bad-regions
pd-ctl region ${region_id}
tikv-ctl --data-dir /path/to/tikv tombstone -r <region_id>,<region_id> --force
tikv-ctl --db /path/to/tikv-data/db recreate-region --pd <pd endpoint> -r <region_id>
```

- Related oncall:
  - v8.5.3 Data recovery `TICKET-8108`

### RocksDB `max-bytes-for-level-base` too large causes high compaction CPU

- Phenomenon: if `max-bytes-for-level-base` is set too large, L1 file count grows and each L0 compaction overlaps too many L1 files.
- Workaround:
  - restore `max-bytes-for-level-base` to default
  - or use `rate-bytes-per-sec` to limit compaction CPU
- Related oncall:
  - v8.1.1 periodic high compaction CPU `APID-11014`

### RocksDB `rate-bytes-per-sec` too small causes compaction to fall behind

- Phenomenon: compaction cannot keep up with write pressure, eventually triggering compaction pending bytes limits and throttling.
- Workaround: enlarge `rate-bytes-per-sec`.
- Related oncall:
  - v7.5.6 TiDB QPS dropped to 0 `TCOC-3626`

## Titan

### TiKV apply-log latency regression or abnormal memory usage caused by THP

- Detailed reference: [`references/thp-latency-and-memory-regression.md`](references/thp-latency-and-memory-regression.md)
- Phenomenon: when transparent huge pages are enabled on TiKV hosts, TiKV can show abnormal memory behavior and elevated append-log latency.
- Workaround: disable THP manually and make the setting persistent across reboots.
- Related oncall:
  - v8.5.1 TiKV memory usage was abnormal on some nodes `TICKET-8007`
  - v5.1.4 PD follower restart caused duration increase `TICKET-3477` / `ONCALL-6686`

### Titan GC causes read and write slowdown

- Detailed reference: [`references/titan-behavior-known-cases.md`](references/titan-behavior-known-cases.md)
- Phenomenon: frequent Titan GC consumes IO resources and increases foreground read/write latency.
- Workaround:
  - lower `max-background-gc=1`
  - increase `discardable-ratio` if disk space allows
- Related oncall:
  - v6.5.4 IO usage spike with Titan GC `GTOC-6955`

### Inaccurate region size estimation in Titan

- Detailed reference: [`references/titan-behavior-known-cases.md`](references/titan-behavior-known-cases.md)
- References:
  - [tikv#18600](https://github.com/tikv/tikv/issues/18600)
  - [tikv#18628](https://github.com/tikv/tikv/issues/18628)
  - [tikv#15996](https://github.com/tikv/tikv/issues/15996)
- Impact:
  - large regions may not split in time
  - uneven space utilization across TiKV instances
- Workaround:
  - lower region-size threshold carefully if mitigating large regions
  - upgrade to fixed version `v8.5.4` for uneven distribution improvement
- Related oncall:
  - v8.5.3 store sizes severely imbalanced `TICKET-6721` / `ONCALL-8008`
  - v8.1.1 uneven space utilization across TiKV instances `NAID-10959` / `GTOC-7266`

### Titan enabled causes severe data skew across TiKV instances ([tikv#15996](https://github.com/tikv/tikv/issues/15996))

- Detailed reference: [`references/titan-behavior-known-cases.md`](references/titan-behavior-known-cases.md)
- Phenomenon: when wide-table data dominates and Titan is enabled, Titan and RocksDB compaction rhythms differ and can produce severe skew.
- Affected version: all supported versions with Titan enabled
- Workaround: N/A

## Related Skills

- For flow control and workload-driven throttling, read `skills/diagnosis/tikv-performance/SKILL.md`.
- For region movement and snapshot-related SST behavior, read `skills/diagnosis/tikv-raftstore/SKILL.md`.
- For SST corruption that surfaces as process panic, read `skills/diagnosis/tikv-panic/SKILL.md`.