# TiKV & Storage Troubleshooting -- Document Summaries

> Generated: 2026-02-15
> Source: 6 Feishu wiki documents from TiKV/Storage oncall knowledge base
> Total content processed: ~350KB across all documents

---

## Document 1: Storage Oncall Handbook
**Source**: https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb
**Type**: Oncall Handbook (comprehensive)
**Reading Status**: Full Read (128KB, processed via explore agent)

### Key Points
- Massive handbook covering 22+ topic areas: unsafe-recover, gRPC, disk, flow control, load-base split, large regions, add-index, region relocation, coprocessor, TiKV panics, RocksDB, restart/upgrade, stale read, kernel, Titan, CDC, and more
- Bilingual (EN/CN) with each issue described in both languages
- Status-coded: checkmark=fixed, wrench=in-progress, pause=won't-fix/no-PR, X=unconfirmed
- Heavy focus on RocksDB internals (compaction, SST ingestion, L0 accumulation, write stall)
- Contains critical tikv-ctl and pd-ctl recovery procedures for data loss scenarios

### Diagnostic Procedures
- **Unsafe-recover post-recovery GC failure**: Shut down leader TiKV -> tikv-ctl tombstone bad replica -> delete PD region cache -> restart -> check for holes
- **Orphan peer cleanup**: pd-ctl region scan to find affected regions -> search historical logs -> tombstone or unsafe-recover
- **Large region diagnosis**: Check tikv-details->server->approximate region size -> grep "invalid split key" -> tikv-ctl region-properties -> manual compaction
- **SST corruption diagnosis**: `tikv-ctl bad-ssts --pd <endpoint>` -> `tikv-ctl ldb repair` -> tombstone -> recreate-region
- **IOPS exhaustion**: Check Coprocessor Overview -> Total KV Cursor Operations -> block-cache hit rate -> search slow logs for high block_read_count
- **Stuck compaction**: grep RocksDB logs for `Compacting .* [0-9]{4,}@` -> compare with normal logs -> restart TiKV if stuck

### Metrics / Thresholds
| Metric/Config | Default/Threshold | Context |
|---|---|---|
| `high-space-ratio` | 0.7 | PD ignores remaining space above this |
| `low-space-ratio` | 0.8 | PD avoids scheduling to this node |
| `reserve-space` | max(5GB, capacity*0.05) | TiKV self-protection |
| AlmostFull | available <= reserve-space | Rejects normal writes |
| AlreadyFull | available <= reserve-space/2 | Rejects ALL writes |
| `l0-files-threshold` | 20 | Flow control triggers rate limiting |
| `soft-pending-compaction-bytes-limit` | 192GB | Partial write rejection (ServerIsBusy) |
| `hard-pending-compaction-bytes-limit` | 1024GB | Total write rejection |
| `split.split-balance-score` | 0.25 | Increase to 0.55 for subrange hotspots |
| `max-background-jobs` | 9 | Increase to 12 under compaction pressure |
| `max-sub-compactions` | 3 | Increase to 6 (restart needed) |
| `compaction-guard-min-output-file-size` | 8MB | Min SST output size |
| `max-manifest-file-size` | 128MB | Increase to 1GB when SST count >100K |
| `raftstore.min-pending-apply-region-count` | 10 | Increase to 50-100 for faster post-restart |
| Coprocessor response panic | >4GB | Triggers protobuf panic (fixed v5.2.2+) |
| TiKV recommended storage limit | 4TB/node | Higher amounts increase apply wait latency |

### Tools / Commands
```bash
# tikv-ctl: check region properties
tikv-ctl --host <addr> region-properties -r <region_id>

# tikv-ctl: manual compaction on a region
tikv-ctl --pd IP:port --bottommost force -c write --region {region-id}
tikv-ctl --pd IP:port --bottommost force -c default --region {region-id}

# tikv-ctl: tombstone a region (TiKV must be stopped)
tikv-ctl --data-dir /path/to/tikv tombstone -p 127.0.0.1:2379 -r <region_id>

# tikv-ctl: check corrupt SST files
tikv-ctl --data-dir /path/to/tikv bad-ssts --pd <endpoint>

# tikv-ctl: repair RocksDB
tikv-ctl ldb repair --db=${rocksdb_data}

# tikv-ctl: recreate a region
tikv-ctl --db /path/to/tikv-data/db recreate-region --pd <pd_endpoint> -r <region_id>

# pd-ctl: delete region cache
curl -X DELETE http://{pd}:2379/pd/api/v1/admin/cache/region/<region_id>

# pd-ctl: online unsafe-recover
pd-ctl -u <pd_addr> unsafe remove-failed-stores <store_id>

# TiKV CPU profile
curl -o tikv_cpu.proto -H 'Content-Type: application/protobuf' \
  'http://127.0.0.1:20182/debug/pprof/profile?seconds=30'
go tool pprof --http=0.0.0.0:1234 tikv_cpu.proto
```

```sql
-- Disable historical statistics (fix write surge)
SET GLOBAL tidb_enable_historical_stats = "OFF";

-- Increase L0 files threshold (online)
SET CONFIG tikv `storage.flow-control.l0-files-threshold` = 30;

-- Increase compaction parallelism (online)
SET CONFIG tikv `rocksdb.max-background-jobs` = 12;

-- Adjust split balance score
SET CONFIG tikv `split.split-balance-score` = '0.55';

-- Slow log search for high block reads (Loki)
-- {cluster_env="prod", namespace="tidbxx", container=~"slowlog.*"} |~ block_read_count: [0-9]{5,}
```

### Raw Content Excerpt
> **RocksDB Flow Control & Disk Space**: When there is a high volume of writes in a short period, or when compaction cannot keep up with the write speed, level-0 files may accumulate to 20. TiKV will then trigger rate limiting, causing latency spikes. Workaround: Increase l0-files-threshold (online config). Adjust compaction configs: max-background-jobs 9->12 (online), max-sub-compactions 3->6 (restart needed).
>
> **TiKV disk space exhaustion**: TiKV and PD jointly manage disk space. PD scheduling: at high-space-ratio (0.7), PD ignores remaining space; at low-space-ratio (0.8), PD avoids scheduling data to the node. TiKV self-protection: reserves reserve-space (max(5G, capacity*0.05)). Returns "AlmostFull" (rejects writes) when available <= reserve-space. Returns "AlreadyFull" when available <= reserve-space/2.
>
> **SST ingestion causes large compaction (tikv#18866)**: CompactionGuard + compaction-guard-min-output-file-size (8MB) can cause multiple small adjacent regions to merge into one SST file. When new data is later ingested for the middle range, it cannot go to L6 due to overlap, falls to L0, causing massive compaction. Long-term fix: AddForcePartitionRange API.
>
> **Raft peers stuck in busy apply state post startup (tikv#18233)**: When TiKV starts, some peers stay in busy apply state for ~2 minutes even when raft log lag is 0. Server reports "Server is busy" (applystore_busy). Workaround: increase raftstore.min-pending-apply-region-count from 10 to 50-100. Fixed: v8.5.2/v7.5.7.

---

## Document 2: Transaction Oncall Handbook
**Source**: https://pingcap.feishu.cn/wiki/FF5Jwc1EniDGZPk31jJcvSXDnPe
**Type**: Oncall Handbook (comprehensive)
**Reading Status**: Full Read (76KB, processed via explore agent)

### Key Points
- Covers 11 major topic areas: unconfirmed issues, pessimistic transactions, async-commit, pipelined DML, follower read, stale read, system variables, client-go, TiDB-side transaction issues, hotspots, BR/PITR anomalies
- Multiple correctness-related issues marked with puzzle-piece emoji (data integrity risks)
- Two unconfirmed correctness issues: (1) async commit suspected data loss under network instability, (2) secondaries key stuck with min_commit_ts > pk.commit_ts
- Critical interaction between fair_locking (default ON since v7.0) + async-commit + 1PC causing PessimisticLockNotFound
- GC stuck by corrupt transaction (tikv#11187) is the most frequently recurring issue (6+ oncall cases)

### Diagnostic Procedures
- **Hot write keys causing TiKV CPU hotspot**: Check batch-get/pessimistic_lock/pessimistic_rollback QPS -> query information_schema.data_lock_waits -> pd-ctl evict-leader on hot region leaders
- **GC stuck by corrupt transaction**: Check gc_worker logs for frozen resolved-ts -> look for `unexpected resolve err: commit_ts_expired` -> follow KB
- **TiKV panic from start_ts=0**: Check logs for `assertion failed: self.0 > 0` -> verify fast_reorg=ON + dist_task=OFF -> drop problematic indexes -> set tidb_ddl_enable_fast_reorg=OFF
- **Async commit resolve lock loop**: Look for large `query_check_secondary_locks` count -> check for repeated lock timeout on same region -> manually delete locks or upgrade
- **Replica data corruption after restart (tikv#16706)**: Leader reports "default not found" but follower read works -> transfer leader to healthy replica -> remove corrupted peer

### Metrics / Thresholds
| Metric/Parameter | Value | Context |
|---|---|---|
| `pessimistic-txn.wait-for-lock-timeout` | Default 1s | TiKV waits before returning KeyIsLocked |
| `pessimistic-txn.wake-up-delay-duration` | Tunable | Increase to reduce thundering herd |
| Pessimistic lock TTL/expiry | ~20 seconds | Wait time for residual locks after kill |
| Election timeout (follower read) | 10s default | Follower read latency spike during leader change |
| `innodb_lock_wait_timeout` | Not enforced during write conflict retry (bug) | Use max_execution_time as workaround |
| `tidb_store_limit` | Default unlimited | Reduce to prevent TiKV CPU saturation |
| Scheduler pending commands | Continuously increasing | Symptom of pipelined DML deadlock |
| Raftstore CPU drops to ~0 | -- | Symptom of pipelined DML deadlock |

### Tools / Commands
```sql
-- Disable fair locking to avoid PessimisticLockNotFound
SET GLOBAL tidb_pessimistic_txn_fair_locking = 0;

-- Disable unchanged key locking (risk: inconsistency on upgrade from <v7)
SET GLOBAL tidb_lock_unchanged_keys = OFF;

-- Disable plan cache (fix SELECT FOR UPDATE lock semantic loss)
SET GLOBAL tidb_enable_prepared_plan_cache = OFF;

-- Disable DDL fast reorg (prevent start_ts=0 panic)
SET GLOBAL tidb_ddl_enable_fast_reorg = OFF;

-- Query lock waits to find hot keys
SELECT * FROM information_schema.data_lock_waits;

-- BR auto_random rebase after restore
SELECT (id & (1 << (64-1-5)) - 1) AS inc_id FROM BB ORDER BY inc_id DESC LIMIT 1;
SHOW TABLE BB next_row_id;
ALTER TABLE BB auto_random_base = max(inc_id) + 1;
```

```bash
# PD-CTL: transfer leader for corruption recovery
pd-ctl operator add transfer-leader <region-id> <target-store-id>
pd-ctl operator add remove-peer <region-id> <store-id>
pd-ctl operator add evict-leader-scheduler <store-id>
```

### Raw Content Excerpt
> **PessimisticLockNotFound caused by calculated commit_ts conflict with for_update_ts under fair lock mode (tidb#41792, tidb#60679)**: When tidb_pessimistic_txn_fair_locking (enabled by default since v7.0.0) is enabled, in scenarios when commit_ts is calculated (e.g., async-commit or 1PC), lock conflicts may lead to PessimisticLockNotFound errors during transaction commit. The transaction attempts to lock k1. When awakened by fair_locking, it uses the current k1's last_commit_ts(=10) as for_update_ts. TiDB fetches a new tso2 from PD as for_update_ts, since the previous last_commit_ts(=10) is calculated, the new for_update_ts ts2 may be the same. When retrying, if k1 no longer needs locking but k2 does, TiDB calls exitAggressiveLockingIfInapplicable and asynchronously rolls back k1's lock, leaving a for_update_ts=10 rollback record. Subsequent attempts to lock k1 still use for_update_ts=10 and get rollbacked. On commit, TiKV meets the rollback with for_update_ts=10 and reports PessimisticLockNotFound. Workaround: disable async-commit & 1PC, OR disable fair_locking. Affected: >v7.0.
>
> **GC stalls due to transaction write corruption during network instability (tikv#11187)**: TiKV GC stalls with resolved-ts frozen. Error: `commit_ts_expired:<start_ts:... attempted_commit_ts:... min_commit_ts:...>`. Most frequently recurring issue with 6+ oncall cases across v4.0-v6.5.
>
> **Hot insert conflicts**: Each pessimistic lock retry waits ~1s (wait-for-lock-timeout), multiplied by retry count. Example: 7 RPCs * ~1s each = 7s+ latency.

---

## Document 3: Raftstore Handbook
**Source**: https://pingcap.feishu.cn/wiki/HaZTwLVC0iDxngk2hX3cbR33nqn
**Type**: Oncall Handbook (raftstore-focused)
**Reading Status**: Full Read

### Key Points
- Focused entirely on raftstore layer: write latency, region/store status, raft log lag, raft engine disk
- Write latency breakdown: append log -> apply wait -> apply -> propose wait -> commit (visible in tikv-details->duration->write pipeline duration)
- Critical issue: TiKV node recovery after downtime causes cluster-wide latency due to raft log read/write lock contention (tikv#18617, fixed v7.5.7/v8.5.3)
- Large clusters with many regions suffer from compaction-triggered meta update storms (tikv#18532, fixed v8.5.4)
- Memory allocstall is a major hidden cause of TiKV latency tails -- check node-exporter for free memory <10GB
- Slow store false-positive under high write speed (tikv#18775) -- adjust inspect-interval or disable evict-slow-store-scheduler

### Diagnostic Procedures
- **Overall write latency analysis**: tikv-details->storage->storage async write duration (overall) -> tikv-details->duration->write pipeline duration (per-stage breakdown: append log, apply wait, apply, propose wait, commit)
- **Large cluster meta update storm**: Check Raft Process->Max duration of raftstore events for StoreMsg -> search logs for `store message` keyword with high takes value -> expand cluster or increase region size
- **Memory allocstall diagnosis**: Check node-exporter for free memory <10GB + allocstall events -> temporary: `echo 1 > /proc/sys/vm/drop_caches` -> long-term: configure vm.min_free_kbytes
- **Slow store false-positive**: Check if write speed is high (bytes or keys) -> increase inspect-interval or disable evict-slow-store-scheduler
- **Raft log lag after node recovery**: Check if recently restarted TiKV node is catching up logs -> reduce raftstore.raft-max-size-per-msg or upgrade to v7.5.7+
- **Down peer diagnosis**: Check tikv-details->raft-message->raft drop messages for region-overlap -> find stale region via "leader missing longer than abnormal_leader_missing_duration" log

### Metrics / Thresholds
| Metric/Config | Value | Context |
|---|---|---|
| `memory-usage-limit` high watermark | limit * 0.9 (e.g., 12GB * 0.9 = 10.8GB) | TiKV rejects raft log when exceeded |
| Free memory threshold | <10GB | Triggers allocstall risk |
| `vm.min_free_kbytes` recommended | min(num_numa_nodes * 1G, 16G) | e.g., 2097152 for dual-NUMA |
| PD disconnect store timeout | 20 seconds | No store heartbeat -> disconnect |
| `max-peer-down-duration` | 10 minutes | Peer marked as down |
| TiFlash `abnormal_leader_missing_duration` | 2 hours (default) | Reduce to 10m to fix stale peer alerts |
| `store-max-batch-size` | Configurable | Increase to reduce tail latency (tradeoff: avg latency increases) |
| `grpc-raft-conn-num` | 1 (default) | Increase when raft client reports channel full |
| `raft-max-inflight-msgs` | Configurable | Increase to reduce raft log send throttling |
| `raft_entry_max_size` | Configurable | If too large + cmd_batch=true, causes write jitter |

### Tools / Commands
```bash
# Emergency: evict leaders from jittery TiKV
pd-ctl operator add evict-leader-scheduler <store-id>

# Drop page cache before traffic peak (temporary allocstall fix)
echo 1 > /proc/sys/vm/drop_caches

# Configure min_free_kbytes (long-term allocstall fix)
vi /etc/sysctl.conf
# add: vm.min_free_kbytes = 2097152
sysctl -p
cat /proc/sys/vm/min_free_kbytes

# TiFlash: reduce stale peer detection time (requires restart)
# In tiflash-proxy config: set max_leader_missing_duration = "10m"
```

### Raw Content Excerpt
> **TiKV node recovery causes cluster performance degradation (tikv#18617, tikv#18605)**: When a TiKV goes down, business writings continue normally. After the node is restarted, it needs to catch up on Raft logs from other TiKV nodes. During this process, when reading old raft logs for a specific region from other TiKV nodes, it shares the same lock with normal writes for this region. When raft log lag is significant, this contention blocks normal write operations for the affected region, causing overall (hot-write region) latency degradation. Workaround: Upgrade to v7.5.7/v8.5.3 (removes the lock). Or reduce raftstore.raft-max-size-per-msg (may reduce throughput for wide tables).
>
> **Memory allocstall causing TiKV read/write latency tail**: Symptoms: noticeable latency tail, increased raftstore write latency. In node-exporter: free memory <10GB, observable allocstall events. Temporary: `echo 1 > /proc/sys/vm/drop_caches`. Long-term: configure vm.min_free_kbytes = min(num_numa_nodes * 1G, 16G). Caution: if system memory is tight, setting min_free_kbytes too high causes forced memory reclamation -> gRPC threads blocked -> TiKV unreachable.
>
> **Large clusters region meta update storm (tikv#18532)**: When cluster has too many regions, updating store.meta after RocksDB compaction becomes expensive (locking + iterating all regions). StoreMsg::CompactedEvent triggers calc_ranges_declined_bytes. Key indicator: Raft Process->Max duration of raftstore events shows StoreMsg latency spike. Log: `[store N] handle 1 store messages ("StoreUnreachable", 1)] [takes=1330]`. Workaround: expand cluster or increase region size. Fixed: v8.5.4.

---

## Document 4: Understanding TiKV Memory Management and Common Issues
**Source**: https://pingcap.feishu.cn/wiki/AtKgweGsxiTJh5kjkUacfbbanSe
**Type**: Deep-dive Technical Guide + Oncall Reference
**Reading Status**: Full Read

### Key Points
- Core challenge: TiKV cannot prevent OOM -- no single config controls total memory; memory-usage-limit excludes block-cache and is only an estimate
- Memory breakdown formula: block-cache (45% of system) + region metadata (500MB per 10K regions) + raftstore memtrace + resolved-ts locks + coprocessor buffers + ~1.3GB baseline
- Critical memory leak bugs: (1) crossbeam-skiplist in scheduler/lock_key (tikv#19284), (2) HashMap for pessimistic locks never shrinks (tikv#18799, fixed v8.5+), (3) LRUCache in raft-router (tikv#15431, fixed v7.1.2/v6.5.6)
- Resolved-ts is a major OOM contributor -- caches uncommitted transaction locks; disable if not using stale read/FLASHBACK/EBS recovery
- On TiDB Cloud: 4c16g TiKV actually limited to ~13GB; 8c32g limited to ~28GB

### Diagnostic Procedures
- **Memory usage analysis**: tikv-details->cluster->memory (overall) -> v8.5+ tikv-details->memory (per-module breakdown)
- **Module-by-module check**:
  1. RocksDB block cache: tikv-details->rocksdb kv->block-cache (stable, equals config)
  2. RocksDB memtable: tikv-details->rocksdb kv->memtable size (small, config-bounded)
  3. Region metadata: estimate from region count (500MB per 10K regions), check tikv-details->cluster->region
  4. Resolved-ts: tikv-details->resolved ts panel (often inaccurate, calculated as total minus others)
  5. Raftstore: tikv-details->server->memory-trace (estimated)
  6. Coprocessor backlog: tikv-details->unified read pool->running tasks + coprocessor-overview->total kv cursor operations
- **Heap profiling (pre-v7.5)**:
  ```bash
  curl -k --cacert /var/lib/tikv-tls/ca.crt --cert /var/lib/tikv-tls/tls.crt \
    --key /var/lib/tikv-tls/tls.key \
    https://127.0.0.1:20180/debug/pprof/heap_activate?interval=600
  ```
- **Heap profiling (v7.5+)**: Available directly from TiDB Dashboard

### Metrics / Thresholds
| Metric/Config | Value | Context |
|---|---|---|
| `memory-usage-limit` | Configurable (NOT a hard limit) | Excludes block-cache; estimated only; mainly for raft memory |
| `storage.block-cache.capacity` | 45% of system memory (default) | Online configurable on v8.5 |
| Region metadata memory | ~500MB per 10K regions | Estimate for capacity planning |
| TiKV baseline memory (no load) | ~1.3GB | Approximate |
| `evict-cache-on-memory-ratio` | 0.1 recommended for ServerIsBusy | Default OFF in v7.0 but setting works |
| `end-point-memory-quota` | Since v8.2.0 | Coprocessor memory limit; rejects with ServerIsBusy |
| Raft entry cache eviction | Triggers at memory-limit * 90% | Sends PeerTick::EntryCacheEvict continuously until below watermark |
| Raft-router LRU leak | ~2GB for 400K regions | Experience value |
| Cloud 4c16g actual limit | ~13GB | TiDB Cloud specific |
| Cloud 8c32g actual limit | ~28GB | TiDB Cloud specific |

### Tools / Commands
```bash
# Heap profile (pre-v7.5, with TLS)
curl -k --cacert /var/lib/tikv-tls/ca.crt --cert /var/lib/tikv-tls/tls.crt \
  --key /var/lib/tikv-tls/tls.key \
  https://127.0.0.1:20180/debug/pprof/heap_activate?interval=600

# Heap profile (pre-v7.5, no TLS)
curl -k http://127.0.0.1:20180/debug/pprof/heap_activate?interval=600

# List captured profiles
curl -k https://127.0.0.1:20180/debug/pprof/heap_list

# Deactivate heap profiling (export first! -- cleans all files)
curl -k https://127.0.0.1:20180/debug/pprof/heap_deactivate

# View heap file
jeprof --svg /xxx/tikv-20160/bin/tikv-server 000001.heap > a.svg

# Install jeprof
yum install epel-release && yum install jemalloc-devel -y
jeprof --version
```

```sql
-- Disable resolved-ts (requires rolling restart, NOT online configurable)
-- Cannot use: SET CONFIG tikv `resolved-ts.enable`='false';  -- WILL FAIL
-- Must modify tikv config file and rolling restart

-- Force cleanup locks (emergency for resolved-ts OOM)
SELECT count(1) FROM <table>;  -- triggers lock cleanup

-- Limit coprocessor memory (v8.2+)
SET CONFIG tikv `end-point-memory-quota` = '<value>';

-- Reduce max tasks to prevent coprocessor OOM
SET CONFIG tikv `readpool.unified.max-tasks-per-worker` = <lower_value>;
```

### Raw Content Excerpt
> **Memory-usage-limit is NOT a hard limit** and cannot completely limit process memory usage. It currently mainly handles raft-related memory, so do not depend on this config. When memory reaches the watermark (90%), tikv#18462 checks whether active regions' entry cache exceeds a threshold; if so, evicts half. This check sends PeerTick::EntryCacheEvict continuously until memory drops below the high watermark, incurring additional CPU consumption.
>
> **Resolved-ts causes TiKV OOM**: High memory usage mainly from system residual locks. Accompanies: large transactions (locks committed only after completion), GC safepoint stuck (locks before safepoint cleared during GC), OOM recurrence (memory immediately climbs after restart). Workaround: disable resolved-ts (rolling restart required). Cannot disable if using: stale read, FLASHBACK TO TIMESTAMP, EBS Recovery. Safe to disable -- does NOT affect: CDC, PITR, follower read. Emergency: force lock cleanup via SELECT count(1).
>
> **Pessimistic lock HashMap memory leak (tikv#18799)**: TiKV stores pessimistic locks in hashmap per region. No proper release mechanism -- memory only grows. Long-term operation causes severe leak. Heap profile shows `try_write_in_memory_pessimistic_locks`. Workaround: evict-leader (destroys hashmap). Fixed: v8.5+.

---

## Document 5: [Top Issue] Hotspot Scheduling vs TiKV CPU Utilization
**Source**: https://pingcap.feishu.cn/wiki/QFpWw2UE1ibmNnkIkr6cDwWFnFc
**Type**: Top Issue Analysis / Design Gap Document
**Reading Status**: Full Read

### Key Points
- **Core problem**: PD hotspot scheduling does NOT consider TiKV CPU utilization as an independent scheduling dimension
- Persistent TiKV CPU hotspots occur even without hot regions -- caused by many regions on one TiKV each accessed at low frequency
- Two existing mechanisms fail in this scenario: (1) load-base-split only triggers when single region exceeds threshold for 10s, (2) hot-region-scheduler balances by hotspot region count, not CPU
- Typical trigger: TiSpark full-scan workloads, multi-table access concentrated on single TiKV
- Hibernate regions frequently awakened on hotspot TiKV nodes (visible in tikv-details->server->hibernate peers)
- Table-level leader/region imbalance: PD considers cluster-wide balance but not per-table balance
- Tracked as FRM-2877 with 6+ related oncall cases (v7.5.5, v7.5.6, v8.5.2)

### Diagnostic Procedures
1. **Confirm TiKV CPU hotspot**: tikv-details->cluster->CPU -- some instances show sustained high CPU
2. **Rule out hot regions**: pd->statistics-hot-write/hot-read->Hot region's leader/peer distribution -- NO obvious hotspot
3. **Check load-base-split**: tikv-details->raft admin->TopN QPS exceeds threshold -- no hotspot regions identified
4. **Check hibernate regions**: tikv-details->server->hibernate peers -- more peers frequently awakened on hotspot TiKV
5. **Check raftstore wake-up traffic**: tikv->raft messages->messages -- hotspot wake-up causes raftstore CPU spike for heartbeats
6. **Check traffic by store**: tikv-details->cluster->MBPS/QPS and pd->statistics hot-write/read->Store write/read rate -- sometimes hotspot visible, sometimes not

### Metrics / Thresholds
| Metric | Location | What to Look For |
|---|---|---|
| TiKV CPU per instance | tikv-details->cluster->CPU | Sustained imbalance between instances |
| Hot region distribution | pd->statistics-hot-write/hot-read | Should show NO obvious hotspot |
| Hibernate peers | tikv-details->server->hibernate peers | Frequent wake-ups on one node |
| Raft messages | tikv->raft messages->messages | Spike from region wake-ups |
| Store traffic | tikv-details->cluster->MBPS/QPS | Intermittent or subtle imbalance |

### Tools / Commands
```bash
# Scatter table regions (manual hotspot mitigation)
curl -X POST http://{TiDBIP}:10080/tables/{db}/{table}/scatter

# Stop scattering
curl http://{TiDBIP}:10080/tables/{db}/{table}/stop-scatter

# Evict leaders from hot TiKV (temporary relief)
pd-ctl operator add evict-leader-scheduler <store-id>
```

### Raw Content Excerpt
> **Root Cause analysis**: Currently TiDB hotspot scheduling does not consider TiKV CPU utilization as an independent scheduling dimension. TiDB mainly relies on two hotspot mitigation mechanisms: (1) Load-base-split (only for read) -- when a region's read traffic or CPU usage continuously exceeds a threshold for 10 seconds, TiKV will split the region at an appropriate key boundary. (2) Hot region scheduler -- PD balances hot regions across TiKV nodes based on the number of hotspot regions on each node.
>
> When one or two TiKV instances handle a large number of regional requests, with each region being accessed at a low frequency, the following scenarios are difficult to handle: Access to multiple tables concentrated on a single TiKV; no single region becomes hot in a large TiKV node (example: TiSpark full-scan workloads, each region scanned only 1 time); SQL-level optimization challenges -- in large clusters, problematic SQL is hard to identify via TopSQL.
>
> Workarounds: (1) Identify SQL and apply scatter (effective in some cases); (2) scatter-range-schedule may help if SQL identified (not GA, practical limitations); (3) Use evict-leader to migrate load (effective but with performance impact). Table-Level imbalance: PD considers cluster-wide leader distribution but not per-table balance. Workaround: `curl -X POST http://{TiDBIP}:10080/tables/{db}/{table}/scatter`.

---

## Document 6: Troubleshooting: TiKV OOM
**Source**: https://pingcap.feishu.cn/wiki/BertwM4rsid3kjkgI6HcSe6gndg
**Type**: Troubleshooting Runbook (OOM-focused)
**Reading Status**: Full Read

### Key Points
- Structured OOM diagnosis checklist with specific metrics panels to check
- Memory breakdown formula: region memory (500MB/10K) + block-cache (45% default) + memtable + raftstore memtrace + resolved-ts lock heap + CDC memory + ~1.3GB baseline + unified read pool tasks
- Top OOM causes by category: (1) coprocessor task backlog, (2) resolved-ts with stuck GC, (3) pessimistic lock accumulation, (4) too many regions for available memory, (5) scale-in overload, (6) swap-induced gRPC deadlock, (7) memory leaks (resolved-ts hashmap, raft-router LRU)
- Key insight: memory-usage-limit is NOT a hard limit and mainly handles raft memory -- cannot prevent OOM
- NUMA node memory exhaustion is a hidden cause (128GB total but single NUMA node full at 80GB)

### Diagnostic Procedures
**OOM Metrics Checklist (check these panels in order)**:
1. `tikv-details -> cluster -> memory` (overall trend)
2. `tikv-details -> cluster -> region` (region count)
3. `tikv-details -> rocksDB-KV -> block cache` (cache size)
4. `tikv-details -> rocksDB-KV -> memtable` (memtable size)
5. `tikv-details -> server -> memory trace` (raftstore memory)
6. `tikv-details -> memory -> allocator stats` (fragmentation)
7. `tikv-details -> resolved-ts -> lock heap size` (resolved-ts locks)
8. `TiCDC -> TiKV -> CDC memory` (CDC old-value + sink)
9. `tikv-details -> Task` (pending tasks)
10. `tikv-details -> Unified Read Pool -> running task` (backlogged reads)
11. `GC -> TiKV Auto GC SafePoint` (stuck GC)
12. `TiDB -> Transaction Regions Num` (transaction size)

**OOM Category Decision Tree**:
- Coprocessor running tasks high -> reduce `max-tasks-per-worker` or set `end-point-memory-quota`
- Resolved-ts lock heap growing -> disable resolved-ts (rolling restart) or force lock cleanup
- Pessimistic lock queue long + deadlock detector CPU 100% -> reduce `pessimistic-txn.max-retry-count`
- Region count high + memory tight -> scale up/out or reduce block-cache
- During scale-in -> avoid scale-in when memory tight (evict-leader adds read pressure + region metadata pressure at 500MB/10K)
- Swap enabled -> disable swap immediately

### Metrics / Thresholds
| Metric/Config | Value | Context |
|---|---|---|
| Region metadata memory | 500MB / 10K regions | Capacity planning |
| Block cache default | 45% of system memory | storage.block-cache.capacity |
| TiKV baseline (no load) | ~1.3GB | Minimum memory |
| Coprocessor running tasks | Watch for backlog (8000+ = danger) | 5GB in 4 minutes observed |
| `max-tasks-per-worker` | Reduce to prevent OOM | Side effect: Server is Busy errors |
| `pessimistic-txn.max-retry-count` | Reduce for lock accumulation | Affects v7.1/v7.5/v8.1 (not v6.5) |
| memory-usage-limit | NOT a hard limit | Only controls raft-related memory |
| Scale-in memory overhead | 500MB per 10K regions migrated | Region metadata for replenished replicas |

### Tools / Commands
```sql
-- Check memory-related configs
SHOW CONFIG WHERE name LIKE '%memory%';

-- Reduce coprocessor task limit
SET CONFIG tikv `readpool.unified.max-tasks-per-worker` = <value>;

-- Reduce pessimistic retry count
SET CONFIG tikv `pessimistic-txn.max-retry-count` = <value>;

-- Disable resolved-ts (rolling restart required)
-- Edit tikv config: resolved-ts.enable = false
-- Then rolling restart all TiKV nodes
```

```bash
# Disable Linux swap
swapoff -a

# Check NUMA node memory distribution
numactl --hardware
numastat -p <tikv_pid>
```

### Raw Content Excerpt
> **Coprocessor task backlog OOM (EMID-10023, v7.1.1)**: Large tablescan requests hit hotspot, read pool maxed out, unified thread pool tasks backlogged (8000+). Backlogged tasks cache data in memory. Result: 5GB memory increase in 4 minutes -> OOM. Workaround: limit max-tasks-per-worker to cap read thread pool tasks. Side effect: Server is Busy errors. No unified config to control total TiKV process memory.
>
> **Pessimistic lock accumulation OOM (GTOC-6879, v7.5.1)**: Airbnb incident -- external system failure caused massive data writes. One TiKV node's deadlock_detector thread hit 100% CPU, preventing other nodes from sending pessimistic lock RPCs. Locks accumulated in memory across nodes -> cascading OOM. Workaround: reduce pessimistic-txn.max-retry-count. Bug affects v7.1/v7.5/v8.1, not v6.5.
>
> **Resolved-ts OOM (TCOC-2175, v6.5.3)**: User ran batch jobs with large transactions + resolved-ts enabled. GC stuck -> residual locks cached in resolved-ts couldn't be released -> 12 hours of continuous OOM. GC lock cleanup took 4 hours due to too many locks + low system specs. Workaround: disable resolved-ts if not using stale read/FLASHBACK/EBS. Memory-usage-limit does NOT help.
>
> **Scale-in OOM (TCOC-2185, v7.1.3)**: 3-TiKV cluster rolling restart. During evict-leader phase, remaining 2 TiKVs absorbed additional read/write pressure + region metadata for replenished replicas (500MB/10K regions). Memory already near limit -> OOM during restart. Avoid scale-in when TiKV memory is tight.
>
> **Swap-induced gRPC deadlock (TICKET-4728, v6.5.6)**: Multiple TiKV nodes on same physical machine. Swap enabled -> memory pressure -> gRPC threads hung (keepalive watchdog timeout). TiKV enters down state. Workaround: disable swap, set reasonable block-cache size, restart.

---

## Cross-Document Index: Most Critical Issues

### By Frequency (most oncall cases)
1. **GC stuck by corrupt transaction** (tikv#11187) -- 6+ cases, Transaction handbook
2. **Disk jitter causing raftstore latency** -- 15+ cases, Raftstore handbook
3. **TiKV CPU hotspot without hot regions** (FRM-2877) -- 6+ cases, Hotspot doc
4. **Memory allocstall latency tails** -- 4+ cases, Raftstore handbook
5. **Coprocessor task backlog OOM** -- 5+ cases, Memory/OOM docs
6. **Resolved-ts OOM** -- 5+ cases, Memory/OOM docs

### By Severity (data integrity risk)
1. **Async commit suspected correctness issue** -- UNCONFIRMED, Transaction handbook
2. **Secondaries key stuck (min_commit_ts > pk.commit_ts)** -- UNCONFIRMED, Transaction handbook
3. **Data loss: default value not found** (tikv#16706) -- Transaction handbook
4. **Raft applied memory statistics error** (tikv#15371) -- Memory handbook

### By Impact (cluster-wide)
1. **ServerIsBusy from memory limit rejection** (tikv#15800, tikv#17392) -- Raftstore handbook
2. **TiKV node recovery causing cluster latency** (tikv#18617) -- Raftstore handbook
3. **Large cluster meta update storm** (tikv#18532) -- Raftstore handbook
4. **Pipelined DML deadlock** (tikv#18384) -- Transaction handbook
5. **Slow store false-positive under high writes** (tikv#18775) -- Raftstore handbook