---
name: tikv-memory
description: |
  Reference skill for TiKV memory analysis and OOM diagnosis.
  Use when investigating TiKV memory growth, OOM, memory hotspots,
  heap profiles, resolved-ts memory pressure, coprocessor backlog,
  pessimistic-lock leaks, or swap-related memory anomalies.
---

# TiKV Memory

Known memory-management behaviors, OOM patterns, and troubleshooting guidance for TiKV.

Source note: adapted from the Feishu `Understanding TiKV Memory Management and common issues` <https://pingcap.feishu.cn/wiki/AtKgweGsxiTJh5kjkUacfbbanSe>.

## When To Use This Skill

Use this skill when:

- TiKV memory keeps growing or reaches OOM
- the user wants to know which modules are consuming memory
- heap profile analysis is required
- memory pressure is suspected to come from resolved-ts, coprocessor backlog, pessimistic locks, or raftstore
- swap, region count, or scale-in activity may be amplifying memory pressure

## First Checks

- Confirm whether the issue is gradual growth, sudden spike, or repeated OOM.
- Check whether the impact is limited to one TiKV or affects multiple stores.
- Record exact versions of TiKV, PD, TiDB, and whether CDC, follower read, or PITR is enabled.
- Check whether recent operations include scale-in, large deletes, large transactions, DDL, backup, or hotspot traffic.
- Decide whether the first priority is preventing the next OOM or finding the root cause.

## Routing Guide

- `Key Configuration Items`: memory-related limits, quotas, and cache controls
- `Key Metrics`: where to estimate major memory consumers
- `Heap Profile Collection`: how to capture heap profiles before and after `v7.5`
- `Known Issues`: memory leak, backlog, swap, resolved-ts, raftstore, and hotspot patterns
- detailed references are linked directly under the related issues when available

## Memory Management Overview

TiKV memory issues are usually hard for two reasons:

- TiKV has many modules, but not every module exposes precise and unified memory accounting.
- Older versions could not strictly limit the overall TiKV memory footprint, so avoiding OOM relied heavily on workload shape and operator actions.

In practice, the two biggest pain points are:

1. OOM prevention is incomplete, because TiKV cannot always rebalance memory usage dynamically across modules.
2. Root-cause tracing is difficult, because a unified memory heat map is still incomplete in many versions.

## Key Configuration Items

- `memory-usage-limit`
  - Needs restart on `v8.5`
  - Default: about `75%` of total system memory in most cases
  - Limits TiKV instance memory only
  - Does not include block cache
  - Uses estimation and cannot cap total memory reliably
- `grpc-memory-pool-quota`
  - Online config on `v8.5`
  - Default: no limit
  - Limits gRPC memory usage
  - Restricting it too aggressively can introduce stalls
- `end-point-memory-quota`
  - Online config on `v8.5`
  - Available since `v8.2.0`
  - Default: the larger of `12.5%` of total system memory and `500MiB`
  - Limits memory used by coprocessor requests
  - Requests may return `server is busy` after the quota is exceeded
- `evict-cache-on-memory-ratio`
  - Restart required
  - Available since `v7.5.0`
  - Default: `0.1`
  - `0` disables this feature
  - Controls when TiKV evicts raft entry cache under memory pressure
- `raftengine.memory-limit`
  - Needs restart on `v8.5`
  - Default: about `15%` of total machine memory
  - Caps memory usage of raft engine related data
- `storage.block-cache.capacity`
  - Online config on `v8.5`
  - Default: `30%` of total memory on `partitioned-raft-kv`, `45%` on `raft-kv`
  - Controls shared block cache size
- `in-memory-engine.capacity`
  - Available since `v8.5`
  - Default: `min(10% of total system memory, 5GiB)`
  - Controls the maximum memory size of the TiKV MVCC in-memory engine
- `pessimistic-txn.in-memory-instance-size-limit`
  - Available since `v8.4.0`
  - Default: `100MiB`
  - Limits memory used by in-memory pessimistic locks for one TiKV instance
  - When exceeded, TiKV persists pessimistic locks instead of keeping them in memory
- `import.memory-use-ratio`
  - Available since `v6.5.0`
  - Default: `0.3`
  - Range: `[0.0, 0.5]`
  - Controls how much of total TiKV memory PITR can use for in-memory restore
  - `0.0` falls back to downloading log files to a local directory instead of restoring in memory
- `cdc.sink-memory-quota`
  - Default: `512MiB`
  - Upper memory limit for TiCDC data change events
- `cdc.old-value-cache-memory-quota`
  - Default: `512MiB`
  - Upper memory limit for TiCDC old-value cache

## Key Metrics

### Overall memory

- `tikv-details -> cluster -> memory`
- Since `v8.5`, per-module memory allocation is visible under `tikv-details -> memory`

### Main memory consumers

- RocksDB KV block cache
  - `tikv-details -> rocksdb kv -> block-cache`
  - Usually stable and close to configured size
- RocksDB KV memtable
  - `tikv-details -> rocksdb kv -> memtable size`
  - Normally smaller and more workload-dependent
- Region metadata
  - No exact metric
  - Estimate from region count
  - A rough reference is about `500 MB / 10K regions`
- Resolved-ts memory
  - Metric exists but is approximate
  - Often inferred from total memory minus other major components
  - Can become large when many locks remain uncommitted
- Raftstore memtrace
  - `tikv-details -> server -> memory-trace`
  - Estimated value for raftstore internals such as messages and routing
- Coprocessor backlog
  - No direct memory metric
  - Watch:
    - `tikv-details -> unifiedread pool -> running tasks`
    - `tikv-details -> coprocessor-overview -> total kv cursor operations`
- Baseline memory
  - TiKV may use around `1.3 GB` even without production load

## Heap Profile Collection

### `v7.5` and later

- Prefer collecting memory profile directly from TiDB Dashboard.

### Before `v7.5`

- Heap profile often needs to be collected manually inside the pod or host.
- The captured profile is incremental, so collection usually needs to run for a while before it becomes useful.

Example commands with TLS:

```bash
curl -k --cacert /var/lib/tikv-tls/ca.crt --cert /var/lib/tikv-tls/tls.crt --key /var/lib/tikv-tls/tls.key https://127.0.0.1:20180/debug/pprof/heap_activate?interval=600
curl -k --cacert /var/lib/tikv-tls/ca.crt --cert /var/lib/tikv-tls/tls.crt --key /var/lib/tikv-tls/tls.key https://127.0.0.1:20180/debug/pprof/heap_list
curl -k --cacert /var/lib/tikv-tls/ca.crt --cert /var/lib/tikv-tls/tls.crt --key /var/lib/tikv-tls/tls.key https://127.0.0.1:20180/debug/pprof/heap_deactivate
jeprof --svg /xxx/tikv-20160/bin/tikv-server 000001.heap > a.svg
```

Example commands without TLS:

```bash
curl -k http://127.0.0.1:20180/debug/pprof/heap_activate?interval=600
curl -k http://127.0.0.1:20180/debug/pprof/heap_list
curl -k http://127.0.0.1:20180/debug/pprof/heap_deactivate
```

## Known Issues

### ✅ Memory hotspot in scheduler `lock_key` leads to OOM ([tikv#19284](https://github.com/tikv/tikv/issues/19284))

- Detailed reference: [`references/tikv#19284.md`](references/tikv#19284.md)
- Description: TiKV's use of `crossbeam-skiplist` for `lock_key` can cause continuous memory leakage.
- Phenomenon:
  - gradual memory growth until OOM
  - the hotspot can move with leadership
  - heap profile shows unexpected allocation around `lock_keys`
- Workaround:
  - avoid `async commit` and `1pc`
  - avoid concurrent range reads while `async-commit` or `1PC` transactions are still prewriting
- Related oncall:
  - v8.5.4 gogame TiKV memory leak `TCOC-4632`
  - v7.5.6 TiKV OOM `TCID-11612` / `TCOC-3623`

### Memory leak in in-memory pessimistic locks ([tikv#18799](https://github.com/tikv/tikv/issues/18799))

- Description: pessimistic locks are stored in memory by `HashMap`, but old entries are not released effectively, so memory only grows.
- Reproduction hint: a large volume of writes on different keys can keep expanding this memory usage.
- Workaround: evict leaders from the affected TiKV so the related lock map is destroyed.
- Fixed version: `v8.5+`
- Related oncall:
  - v7.5.6 potential memory leak in TiKV in production `GTOC-8029`
  - v7.5.6 8 GB of TiKV memory occupied by pessimistic locks `ONCALL-7929` / `TICKET-6405`

### Single-region write hotspot and large-region side effects

- Phenomenon:
  - oversized regions remain visible in `tikv-details -> approximate region size`
  - TiKV logs keep showing repeated split attempts for the same region. grep key words `BatchSplit splits` in TiKV's log
  - memory pressure rises together with hotspot concentration
- Workaround: fix the hotspot from schema design or workload distribution.
- Related oncall:
  - v7.5.3 TiKV memory hotspot investigation `TCID-11349` / `TCID-11307` / `TCOC-3214`
  - v6.5.3 TiKV OOM `TICKET-3663`

### OOM caused by coprocessor task backlog

- Phenomenon: when coprocessor tasks pile up, TiKV buffers more data in memory and may OOM. Memory growth often matches the backlog growth.
- Workaround:
  - set `end-point-memory-quota` on `v8.2+`
  - reduce `max-tasks-per-worker`
  - note that a stricter limit may cause `server is busy`
- Related oncall:
  - v7.5.6 TiKV hotspot led to OOM `TCOC-4453`
  - v6.5.4 backup-time readpool backlog caused OOM `GTOC-7002` / `NAID-10595`
  - v8.1.2 TiKV OOM `GTOC-7252`

### Too many regions per TiKV causes system-memory OOM

- Phenomenon: memory grows slowly until OOM on small-memory nodes, especially in cloud environments. Region metadata itself becomes expensive when region count is high.
- Workaround:
  - scale up or scale out
  - if necessary, reduce block cache or increase region size, but both may hurt performance
- Related oncall:
  - v6.5.3 frequent CPU 100 percent and repeated OOM `TCOC-2175`
  - v6.5.3 120K regions kept one TiKV near 25 GB memory on a 32 GB node `TCOC-2199`
  - v8.1.1 excessive region count caused alternating TiKV and PD OOM `GTOC-7417`

### Scaling in TiKV can trigger OOM on low-memory clusters

- Phenomenon:
  - evict-leader shifts read and write pressure to the remaining stores
  - replica replenishment increases metadata pressure on the remaining stores
- Workaround: avoid scaling in TiKV when memory headroom is already tight.
- Related oncall:
  - v7.1.3 a 3-TiKV cluster OOMed during maintenance when only 2 TiKV instances remained `TCOC-2185`

### Using swap can cause gRPC disconnection and TiKV down state

- Detailed reference: [`references/resolved-ts-and-swap-known-cases.md`](references/resolved-ts-and-swap-known-cases.md)
- Phenomenon: after memory reaches a certain level with swap enabled, TiKV logs many gRPC disconnections, gRPC becomes stuck, and TiKV may enter down state. Node exporter usually shows heavy page swap in/out before failure.
- Root cause: suspected swap-induced internal gRPC hang.
- Workaround:
  - disable Linux swap
  - reduce block cache to free memory
  - restart the affected TiKV instance
- Related oncall:
  - v6.5.6 three TiKV nodes stayed down for a long time and recovered after restart `TICKET-4728` / `ONCALL-7316`

### Enabling swap causes performance jitter

- Detailed reference: [`references/resolved-ts-and-swap-known-cases.md`](references/resolved-ts-and-swap-known-cases.md)
- Recommendation: follow the official guidance and disable swap.
- Related oncall:
  - v6.5.8 performance testing TPS could not increase `ONCALL-7714`
  - v7.5.4 swap increased suspend duration inside unified read pool `TICKET-5948` / `ONCALL-7757`
  - v6.5.6 TiKV latency increased without obvious resource bottlenecks `TICKET-5962`

### Resolved-ts causes TiKV OOM

- Detailed reference: [`references/resolved-ts-and-swap-known-cases.md`](references/resolved-ts-and-swap-known-cases.md)
- Phenomenon:
  - usually associated with large transactions or stuck GC safepoint
  - memory rises again quickly after OOM and can keep OOMing
  - large deletes can amplify the problem; see [tikv#15458](https://github.com/tikv/tikv/issues/15458)
- Workaround:
  - disable `resolved-ts` with rolling restart when the dependent features are not needed
  - when it cannot be disabled, clean residual locks as quickly as possible
  - avoid setting GC safepoint too large
  - do not expect `memory-usage-limit` to solve this case directly
- Note:
  - `resolved-ts` is required by some features such as stale read, `FLASHBACK TO TIMESTAMP`, and some recovery flows
  - it does not need to stay enabled only for CDC, PITR, or follower read in every case
- Related oncall:
  - TiKV v6.5.5 OOM `ONCALL-7027`
  - v6.5.3 TiKV frequently OOMed before GC got stuck `TCOC-2175`
  - v5.1.4 TiKV had OOM risk `ONCALL-6891`
  - massive deletion caused TiKV memory OOM `TCOC-1299`

### Resolved-ts worker CPU saturation causes instant memory spike ([tikv#18359](https://github.com/tikv/tikv/issues/18359))

- Detailed reference: [`references/resolved-ts-and-swap-known-cases.md`](references/resolved-ts-and-swap-known-cases.md)
- Phenomenon: after resolved-ts worker CPU reaches full utilization, pending tasks surge and memory can jump suddenly into OOM.
- Key metrics:
  - `tikv-details -> resolved ts`
  - resolved-ts CPU fully utilized
  - pending command size rising
  - worker pending tasks heavily backlogged
- Workaround:
  - disable `resolved-ts` if the feature is not required
- Related oncall:
  - v8.5.1 TiKV OOM due to full resolved-ts worker `TICKET-6935` / `ONCALL-8062`
  - v6.5.2 TiKV OOM analysis `TICKET-5877` / `ONCALL-7736`

### Cloud memory alarm format may not match TiKV memory panels

- Phenomenon: cloud node-level memory alarms may trigger even when TiKV dashboards look normal, because the alarm expression differs from TiKV panel semantics.
- Related oncall:
  - v7.5.3 `NODE_memory_used_more_than_80%` `GTOC-7400` / `ST-1426`

### Pessimistic-lock high-conflict scenarios can leak memory ([tikv#17394](https://github.com/tikv/tikv/issues/17394))

- Phenomenon: under extreme single-key contention, if deadlock detection leader is remote and RPC delivery stalls, messages can accumulate in channels and consume too much memory.
- Diagnostic hints:
  - `TiKV-details -> pessimistic locking -> lock manager thread CPU` is near 100 percent
  - `TiKV-details -> pessimistic locking -> lock waiting queue` is high
- Workaround:
  - decrease `pessimistic-txn.max-retry-count`
- Affected versions: `v7.1`, `v7.5`, `v8.1`
- Related oncall:
  - v7.5.1 TiKV OOM `GTOC-6879`
  - v7.5.2 wetech TiKV OOM `GTOC-7067`

### Incorrect raft apply memory statistics block writes ([tikv#15371](https://github.com/tikv/tikv/issues/15371))

- Phenomenon: incorrect raftstore apply-memory statistics keep growing until TiKV believes memory usage exceeds the high watermark, then Raft rejects local messages and writes may get stuck with `server is busy`.
- Diagnostic hint: `tikv-details -> server -> memory trace` shows abnormal `raftstore-applys`.
- Workaround: restart TiKV.
- Fixed version: `v7.1.2` / `v6.5.6` / `v6.1.7`
- Related oncall:
  - v6.5.2 cluster stuck `TICKET-6115` / `ONCALL-7835`
  - v7.1.1 log replication rejection caused very long commit time `GTOC-7668`

### Boxing raft messages reduces redundant memory clones ([tikv#17326](https://github.com/tikv/tikv/issues/17326))

- Phenomenon: using `Box` for raft-message transfer reduces unnecessary clones in raftstore batch-system paths, especially in large clusters with many regions.
- Fixed version: `v8.1.3` / `v7.5.4` / `v7.1.6` / `v6.5.13` / `v6.1.8`
- Related oncall:
  - TiKV 7.1.2 saw memory spike and OOM during create-table phase for 80K table imports `tikv#16229`

### LRU cache in raft-router fails to release memory ([tikv#15431](https://github.com/tikv/tikv/issues/15431) [tikv#15430](https://github.com/tikv/tikv/issues/15430))

- Phenomenon:
  - old raft-router implementations used LRU cache per gRPC connection
  - mailbox clone memory grows during startup
  - deleted peers are not removed quickly enough from cache
  - memory grows and does not fall after workload drops
- Additional effect: after memory reaches the high watermark, repeated entry-cache eviction checks can add CPU overhead; see [tikv#18462](https://github.com/tikv/tikv/issues/18462)
- Workaround:
  - upgrade to `v7.1.2` / `v6.5.6`
  - or restart TiKV periodically as a temporary mitigation
- Related oncall:
  - v6.5.3 root-cause analysis of TiKV restart `TICKET-6060`
  - v6.5.5 high TiKV memory usage `TICKET-5981` / `ONCALL-7788`
  - v6.1.7 TiKV memory kept growing slowly `TICKET-5761`

## Related Skills

- For storage-engine, compaction, and SST-related memory side effects, read `skills/diagnosis/tikv-storage/SKILL.md`.
- For region count, large-region, and hotspot scheduling effects, read `skills/diagnosis/tikv-raftstore/SKILL.md`.
- For performance and throttling topics beyond memory, read `skills/diagnosis/tikv-performance/SKILL.md`.