# Hot-row updates can create oversized regions

Source note: adapted from the internal Feishu notes referenced by the `tikv-performance` skill: <https://pingcap.feishu.cn/wiki/LBOEw7aRbi5q2QkoWndcCd61nue>.

## Issue

- Summary: when GC is stuck, GC lifetime is too large, or compaction is delayed, old MVCC versions of a frequently updated hot key cannot be cleaned up in time. Because all versions of one key must stay in the same region, the region can grow to GB scale and become difficult to split.

## Phenomenon

- A single region keeps growing and can become extremely large.
- Region size becomes imbalanced across stores.
- Scheduling becomes harder and hotspots become more concentrated on memory, CPU, and I/O.
- TiKV may log `failed to handle split req` or `invalid split key`.

## Root Cause

This issue is usually caused by one of two patterns:

- GC malfunction: old MVCC versions are retained for too long because GC is stuck or GC lifetime is too large.
- Delayed compaction: stale versions have already become reclaimable, but RocksDB compaction is too slow to clean them promptly.

Because all MVCC versions of one key must remain in the same region, hot-row update patterns are especially likely to inflate one region dramatically.

## Diagnostic Steps

1. Check `tikv-details -> server -> approximate region size`.
1. Search TiKV logs for `invalid split key` or related split failures.
1. Inspect region properties:

```bash
tikv-ctl --host ADDR region-properties -r REGION_ID
```

1. Determine whether the main blocker is GC retention or delayed compaction.

## Workaround

If GC is the root cause:

- Resolve the GC malfunction.
- Reduce GC lifetime so old versions can be cleaned earlier.

If delayed RocksDB compaction is the main cause, try manual compaction:

```bash
# Check the region's properties first
tikv-ctl --host ADDR region-properties -r REGION_ID

# Trigger compaction
tikv-ctl --pd IP:PORT --bottommost force -c write --region REGION_ID
tikv-ctl --pd IP:PORT --bottommost force -c default --region REGION_ID

# Check the region's properties again
tikv-ctl --host ADDR region-properties -r REGION_ID
```

## Related Oncall

- v8.1.1 troubleshooting error `failed to handle split req` `NAID-11323` / `GTOC-7609`
- v8.5.1 a 200 GB region could not split `TICKET-6121`