# TiKV Apply-Log Latency Regression or Abnormal Memory Usage Caused by THP

Source note: adapted from the Feishu `Storage oncall handbook` <https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb>.

- Phenomenon:
  - TiKV append-log latency rises noticeably.
  - Some nodes can show abnormal memory usage or memory-allocation stalls.
  - The impact is often visible without a corresponding TiKV logical bottleneck.
- Trigger / Preconditions:
  - Transparent Huge Pages (THP) are enabled on the TiKV host.
  - The workload causes frequent small memory allocations inside TiKV.
- Key logs:
  - No mandatory fatal log signature is required. Diagnosis usually depends on host configuration plus latency and memory symptoms.
- Key metrics:
  - Append-log or raft-log latency increases.
  - Host memory behavior can look unusual on a subset of nodes.
- Root cause:
  - THP uses 2 MB huge pages.
  - TiKV allocates many small buffers, especially around raft log buffers, coprocessor temporary buffers, and raftstore internal buffers.
  - Huge-page allocation and splitting can introduce expensive stalls and increase latency in allocation-sensitive paths.
- Workaround:
  - Disable THP manually on the host and ensure the setting persists after reboot.
  - Note that `tiup cluster check --apply` may disable THP temporarily but does not guarantee persistence after reboot.
- Fixed version:
  - This is primarily an environment and operating-system configuration issue rather than a TiKV code fix.
- Related oncall:
  - v8.5.1 TiKV memory usage was abnormal on some nodes `TICKET-8007`
  - v5.1.4 PD follower restart caused duration increase `TICKET-3477` / `ONCALL-6686`