---
name: tikv-performance
description: |
  Reference handbook for TiKV gRPC, disk, flow-control, and large-region
  diagnosis.
  Use when investigating TiDB -> TiKV RPC long-tail latency, disk IOPS
  saturation, scheduler busy errors, flow-control throttling, load-based
  split imbalance, or large-region side effects.
---

# TiKV Performance Handbook

Known gRPC, disk, throttling, and large-region patterns for TiKV on-call diagnosis.

## When To Use This Skill

Use this skill when:

- duration, latency, or throughput changes without obvious crash symptoms
- `ServerIsBusy` appears during normal traffic
- internal SQL, large scans, or write bursts saturate TiKV resources
- gRPC-related long-tail latency appears between TiDB and TiKV
- load-based split does not mitigate TiKV hotspots as expected
- large regions amplify scheduling or throttling side effects

## First Checks

- Identify whether the main bottleneck is CPU, disk IOPS, memory, or network.
- Check whether recent operations include `ANALYZE`, DDL, large scan, or large transaction.
- Confirm whether throttling is from scheduler, flow control, or storage engine.
- Record whether the issue is store-local or cluster-wide.

## Routing Guide

- `gRPC`: TiDB -> TiKV RPC long-tail latency, compression overhead, or connection bottlenecks
- `Disk`: IOPS saturation, full scan pressure, or disk-space exhaustion
- `Flow Control`: write bursts, level-0 accumulation, or scheduler throttling
- `Load-Based Split`: subrange hotspots that fail to split cleanly
- `Large Regions`: oversized-region split failures or scheduling side effects
- detailed references are linked directly under the related cases when available

Source note: adapted from [internal Feishu notes](https://pingcap.feishu.cn/wiki/LBOEw7aRbi5q2QkoWndcCd61nue).

## gRPC

gRPC-related bottlenecks and behavior changes between TiDB and TiKV.

### ⏸️ gRPC thread hotspot causes long-tail latency ([tikv#11009](https://github.com/tikv/tikv/issues/11009))

- Detailed reference: [`references/tikv#11009.md`](references/tikv#11009.md)
- Description: TiKV stream RPC connections are statically bound to gRPC worker threads after establishment. Under heavy traffic, this can create thread-level load imbalance and long-tail latency.
- Critical streams include:
  - client read/write requests through [`BatchCommands`](https://github.com/pingcap/kvproto/blob/release-8.5/proto/tikvpb.proto#L146-L201)
  - raft communication through [`Raft` and `BatchRaft`](https://github.com/pingcap/kvproto/blob/release-8.5/proto/tikvpb.proto#L78-L79)
- Symptoms:
  - uneven utilization in `TiKV-details -> CPU -> gRPC pool CPU` when grouped by thread name
  - long-tail latency in the `TiDB -> TiKV RPC layer`
  - long-tail latency in `replica raft log duration`
- Workaround:
  - enlarge `server.grpc-concurrency`
  - enlarge `server.grpc-raft-conn-num`
  - both changes require restarting TiKV
- Related references:
  - internal tracking: [FRM-2527](https://internal.pingcap.net/jira/browse/FRM-2527)
  - client-go enhancement: [client-go#1098](https://github.com/tikv/client-go/pull/1098) (not GA)
- Related oncall:
  - v7.5.6 gRPC CPU hotspot during write surge `TCOC-3877`
  - v7.5.2 `LockNotFound` errors `TICKET-6135`
  - v7.5.3 TiKV RPC layer latency spike `TCID-11431` / `TCID-11441` / `TCOC-3397`
  - v6.5.10 replica raft log duration spike and raft log gap increase
  - v6.5.10 `TxnLockNotFound` burst `TICKET-5912` / `ONCALL-7746`

### ✅ `grpc-compression-type` does not affect TiKV responses ([tikv#17176](https://github.com/tikv/tikv/issues/17176))

- Detailed reference: [`references/tikv#17176.md`](references/tikv#17176.md)
- Phenomenon: TiKV does not apply `grpc-compression-type` to client response paths in the affected versions, so changing the TiKV-side setting alone has no effect for those responses.
- Workaround: N/A
- Affected versions:
  - `<= v8.1.0`
  - `<= v7.5.2`
  - `<= v6.5.10`
- Related oncall:
  - v6.5.2 `grpc-compression-type` had no effect in some cases `ONCALL-7215` / `TICKET-4410`

### ⏸️ gRPC compression bottleneck from wide tables

- Symptom: when gRPC compression is enabled, processing wide tables with small `process-key` but large `process_keys_size` can consume excessive gRPC resources.

```text
tidb: tikv-client.grpc-compression-type = "gzip"
tikv: server.grpc-compression-type = "gzip"
```

- Impact:
  - long-tail latency in the TiDB -> TiKV RPC layer
  - long-tail latency in replica raft log duration
- Workaround:
  - scatter regions for the wide table to distribute load and avoid merge-back
  - disable gRPC compression, which requires restarting TiKV
  - enlarge `grpc_raft_conn_num`
  - adjust `grpc_concurrency` if needed
- SOP: `[GTOC-7734] SOP - gRPC connection bottleneck causes excessive long-tail latency in the cluster`
- Related oncall:
  - v7.5.6 Performance issue analyze `APID-11605`
  - v7.5.6 slow log analyze `APID-11539`
  - v7.5.6 slow log analyze `APID-11505` / `GTOC-7782`
  - v7.5.6 high retry errors(backoff) `APID-11454` / `GTOC-7734`

## Disk

### ⏸️ TiKV's IOPS limit was reached due to internal SQL (`ANALYZE`) or large SQL ([tikv#8616](https://github.com/tikv/tikv/issues/8616))

- Detailed reference: [`references/tikv#8616.md`](references/tikv#8616.md)
- Description: internal SQL such as `ANALYZE` can trigger TiKV disk-read hotspots, causing frontend latency spikes once IOPS reaches the device limit. TiKV currently lacks direct configuration to cap internal SQL IOPS usage. Internal SQL tasks like `ANALYZE` can easily hit IOPS limits when scanning a large amount of cold data and degrade cluster-wide performance.
- Key metrics:
  - `TiKV-details -> Coprocessor overview -> total kv cursor operators`: shows RocksDB read operations for `ANALYZE`-like SQL, but there is no direct metric for IOPS usage. Note that TTL task consumption cannot be measured here.
  - `TiKV-details -> rocksdb kv -> block-cache operations`: indirectly reflects SST read frequency
  - `TiKV-details -> rocksdb kv -> block cache hit`: low hit rates can indicate cold-data access, but low concurrency of internal SQL can make this signal less obvious
- Workaround: there is no direct IOPS throttling configuration for internal SQL or general SQL. Only indirect traffic-control options are available, with varying effectiveness.
  - TiKV: set [background-read-bandwidth](https://docs.pingcap.com/zh/tidb/stable/tikv-configuration-file/#background-read-bandwidth-%E4%BB%8E-v620-%E7%89%88%E6%9C%AC%E5%BC%80%E5%A7%8B%E5%BC%95%E5%85%A5)
  - TiDB: reduce concurrency
    - [tidb_sysproc_scan_concurrency](https://docs.pingcap.com/tidb/stable/system-variables/#tidb_sysproc_scan_concurrency-new-in-v650)
    - [tidb_auto_build_stats_concurrency](https://docs.pingcap.com/tidb/stable/system-variables/#tidb_auto_build_stats_concurrency-new-in-v650)
    - [tidb_build_sampling_stats_concurrency](https://docs.pingcap.com/tidb/stable/system-variables/#tidb_build_sampling_stats_concurrency-new-in-v750)

  ```sql
  SET GLOBAL tidb_sysproc_scan_concurrency = X;
  SET GLOBAL tidb_auto_build_stats_concurrency = Y;
  SET GLOBAL tidb_build_sampling_stats_concurrency = Z;
  ```

  - Use resource control.
- Related oncall:
  - v7.5.6 Triage Root cause of EBS IOPS Exceeded `TCOC-4356`
  - v7.5.3 IOPS exceeded `ST-2120` / `TCOC-4070`
  - v7.5.6 iops exhaustion due to BIG SQL `TCOC-3993`
  - v7.5.6 iops exhaustion due to analyzing SQL `TCOC-3992`

### ⏸️ TiKV's IOPS limit was reached due to expensive SQL

- Description: similar to the previous case, but triggered by foreground expensive SQL instead of internal jobs such as `ANALYZE`.
- Related oncall:
  - v7.5.6 Expensive SQL causes iops saturation `TCOC-3905`
  - v6.5.5 Duration spike caused by unknown IOPS increase `GTOC-7847`

### ⏸️ Large table full scan triggers EBS throttling

- Phenomenon: a table full scan may read a large amount of cold data, causing heavy disk I/O and EBS throttling.
- Workaround: reduce TiDB scan concurrency by lowering `tidb_distsql_scan_concurrency`.

```sql
SET tidb_distsql_scan_concurrency = 10;
SET SESSION tidb_distsql_scan_concurrency = 10;
SET @@global.tidb_distsql_scan_concurrency = 10;
SET GLOBAL tidb_distsql_scan_concurrency = 10;
```

- Related oncall:
  - v7.5.5 cluster commit latency increased and p99 store duration spiked `TCID-11512`

### ✅ TiKV disk space exhaustion

- Description: TiKV and PD jointly manage TiKV disk space through scheduling thresholds and TiKV self-protection.
  - PD scheduling control:
    - at `high-space-ratio` (`0.7` by default), PD ignores remaining space and balances by used space
    - at `low-space-ratio` (`0.8` by default), PD avoids scheduling more data to the TiKV node
  - TiKV self-protection:
    - [reserve-space](https://docs.pingcap.com/zh/tidb/stable/tikv-configuration-file/#reserve-space) is `max(5G, capacity * 0.05)`
    - returns `AlmostFull` when available space is below reserve-space
    - returns `AlreadyFull` when available space is below half of reserve-space
- Workaround: scale out or scale up TiKV nodes.
- Related oncall:
  - v7.1.1 tiflash learner node addition timeout during cluster scaling `GTOC-6788`
  - v5.4.1 write stall due to filled TiKV node `ONCALL-6911`

### ✅ TiDB goroutine panic prevents purging of `mysql.stats_history` ([tidb#59702](https://github.com/pingcap/tidb/issues/59702))

- Symptom: `information_schema.tikv_region_status` shows excessive storage consumption by `mysql.stats_history`, eventually causing TiKV disk exhaustion.
- Workaround: N/A. Clients generally cannot drop the `mysql.stats_history` system table directly.
- Related oncall:
  - low RocksDB compression ratio in staging environment `TCID-11284` / `TCOC-3125`

## Related Skills

- `skills/diagnosis/transaction-oncall-handbook/SKILL.md`: transaction-focused diagnosis such as large transactions, lock conflicts, or commit-path anomalies
- `skills/diagnosis/tidb-perf-diagnosis/SKILL.md`: broader TiDB and cluster-level performance diagnosis beyond the TiKV-focused bottlenecks in this skill

## Flow Control

### ⏸️ TiDB historical stats causes write surge and throttling ([tidb#52826](https://github.com/pingcap/tidb/issues/52826))

- Detailed reference: [`references/tidb#52826.md`](references/tidb#52826.md)
- Phenomenon: before `v8.2.0`, TiDB enabled historical statistics by default. Recording them can generate a burst of writes and trigger TiKV throttling such as:
  - TiKV report `server is busy`
  - A noticeable spike in bytes/written metrics from the raft-engine
- Workaround: disable historical statistics.

```sql
SET GLOBAL tidb_enable_historical_stats = "OFF";
SHOW GLOBAL VARIABLES LIKE 'tidb_enable_historical_stats';
```

- Related oncall:
  - v7.5.6 `server is busy` triaged without large foreground transactions `TICKET-6732`

### Raftstore write-pipeline known cases

- Detailed reference: [`references/raftstore-write-pipeline-known-cases.md`](references/raftstore-write-pipeline-known-cases.md)
- Use this reference when write latency rises in raftstore stages such as `propose wait`, `append log`, `commit log`, or `apply`, especially during admin-command bursts or metadata-heavy clusters.

### ⏸️ High write workload causes level-0 file accumulation and rate limiting

- Phenomenon: when write traffic is very high, or compaction cannot keep up, level-0 files may accumulate until TiKV starts rate limiting and latency spikes appear.
- Workaround:
  - increase `l0-files-threshold`(20 by default) using online config

    ```sql
    SET CONFIG tikv `storage.flow-control.l0-files-threshold` = 30;
    SHOW CONFIG WHERE name LIKE "%l0_files_threshold%";
    ```

  - adjust compaction-related configurations
    - increase `rocksdb.max-background-jobs` using online config:

      ```sql
      SET CONFIG tikv `rocksdb.max-background-jobs` = 12;
      ```

    - increase `max-sub-compactions` from `3` to `6` if a restart is acceptable
- Related oncall:
  - v6.5.6 TiKV reported `server is busy` `TICKET-6343`
  - v7.5.6 latency spikes `TCID-11493`
  - v7.5.5 latency spike in testing TiDB cluster `GTOC-7588` / `NAID-11353`

### ⏸️ Single large transaction causes scheduler to report `ServerIsBusy`

- Phenomenon: a single large transaction can spike scheduler writing bytes so quickly that the spike may not be obvious in normal metrics, but it can still trigger scheduler throttling and report `ServerIsBusy`.
- Diagnostic hint: check `raft-engine -> bytes/written` to see whether there was a sudden large write burst.
- Workaround:
  - increase `scheduler-pending-write-threshold` if a restart is acceptable
- Related oncall:
  - TiKV reported scheduler `server is busy` `APID-11236` / `GTOC-7614`

## Load-Based Split

### ✅ Hotspots in a subrange of one region can cause TiKV CPU spikes ([tikv#18932](https://github.com/tikv/tikv/issues/18932))

- Detailed reference: [`references/tikv#18932.md`](references/tikv#18932.md)
- Description: with load-based split enabled, TiKV tries to find a split key when a region becomes hot. After finding a candidate split key, TiKV checks whether the post-split traffic would be balanced. This final decision is controlled by `split.split-balance-score`.
- Notes:
  - the smaller the value, the stricter the balance requirement
  - the default is `0.25`
  - if hotspots are concentrated only in one subrange of a region, the threshold can be too strict and prevent TiKV from finding a valid split key
- Diagnostic hints:
  - check `tikv-details -> raft admin -> load base split event`
  - if `no_balance_key` stays high while the related TiKV CPU also stays high, it usually indicates subrange hotspots inside one region
- Workaround: increase `split.split-balance-score` appropriately

```sql
SET CONFIG tikv `split.split-balance-score` = '0.55';
SHOW CONFIG WHERE name LIKE 'split.split-balance-score';
```

- Related oncall:
  - v8.5 how to mitigate read unbalance across TiKV `GTOC-7880` / `NAID-11609`

## Large Regions

### ⏸️ Hot-row updates can create oversized regions

- Detailed reference: [`references/hot-row-updates-can-create-oversized-regions.md`](references/hot-row-updates-can-create-oversized-regions.md)
- Description: when GC malfunctions, for example because GC is stuck or GC lifetime is too large, old MVCC versions of a frequently updated hot key cannot be cleaned up in time. Because all MVCC versions of one key must stay in the same region, that region can grow to GB scale and become hard to split.
- Impact:
  - region size imbalance
  - scheduling becomes difficult
  - hotspot concentration increases for memory, CPU, and I/O
- Diagnostic steps:
  - check `tikv-details -> server -> approximate region size`
  - search TiKV logs for `invalid split key`
  - inspect region properties

```bash
tikv-ctl --host ADDR region-properties -r REGION_ID
```

- Workaround:
  - if GC is the root cause:
    - resolve the GC malfunction
    - reduce GC lifetime so old versions can be cleaned earlier
  - if delayed RocksDB compaction is the main cause, try manual compaction:

```bash
# Check the region's properties first
tikv-ctl --host ADDR region-properties -r REGION_ID

# Trigger compaction
tikv-ctl --pd IP:PORT --bottommost force -c write --region REGION_ID
tikv-ctl --pd IP:PORT --bottommost force -c default --region REGION_ID

# Check the region's properties again
tikv-ctl --host ADDR region-properties -r REGION_ID
```

- Related oncall:
  - v8.1.1 troubleshooting error `failed to handle split req` `NAID-11323` / `GTOC-7609`
  - v8.5.1 a 200 GB region could not split `TICKET-6121`

### ✅ Large-region scheduling can cause level-0 SST accumulation and flow-control throttling

- Detailed reference: [`references/large-region-scheduling-causes-l0-accumulation-and-throttling.md`](references/large-region-scheduling-causes-l0-accumulation-and-throttling.md)
- Description: when a region becomes large, it covers a wider key range and contains more SST files. During scheduling operations such as peer migration, balance region, or merge, business writes are more likely to overlap with these SST key ranges. This increases the chance that SST files are ingested into level 0.
- Impact:
  - level-0 files accumulate quickly
  - `l0-files-threshold` is triggered
  - write throughput drops and latency increases, sometimes with `8022` errors
- Key metrics:
  - `tikv-details -> rocksdb kv -> ingestion picked level`
  - `tikv-details -> flow-control`
- Workaround:
  - increase `l0-files-threshold`
  - note that versions earlier than `v8.0` still require a restart for this change
- Related oncall:
  - v7.1.5 TiKV frequently reported `8022` after region size was adjusted to 960 MB `ONCALL-7598` / `TICKET-5547`
  - v7.1.5 the cluster suddenly became stuck after burst traffic without obvious resource bottlenecks `TICKET-5535`