# Resolved-TS and Swap Known Cases

Source note: adapted from the Feishu `Understanding TiKV Memory Management and common issues` <https://pingcap.feishu.cn/wiki/AtKgweGsxiTJh5kjkUacfbbanSe>.

## Using Swap Can Cause gRPC Disconnection and TiKV Down State

- Phenomenon:
  - After memory usage rises with swap enabled, TiKV logs many gRPC disconnections.
  - gRPC can stall and TiKV may enter a down state.
  - Node-exporter usually shows heavy page swap in and swap out around the incident.
- Root cause:
  - Swap is suspected to trigger an internal gRPC hang path under memory pressure.
- Workaround:
  - Disable Linux swap.
  - Reduce block cache if memory headroom is too small.
  - Restart the affected TiKV instance after stabilizing memory configuration.
- Related oncall:
  - v6.5.6 three TiKV nodes stayed down for a long time and recovered after restart `TICKET-4728` / `ONCALL-7316`

## Enabling Swap Causes Performance Jitter

- Phenomenon:
  - TiKV latency rises without an obvious CPU or disk bottleneck.
  - Unified read pool suspend duration can increase.
- Workaround:
  - Follow official guidance and disable swap.
- Related oncall:
  - v6.5.8 performance testing TPS could not increase `ONCALL-7714`
  - v7.5.4 swap increased suspend duration inside unified read pool `TICKET-5948` / `ONCALL-7757`
  - v6.5.6 TiKV latency increased without obvious resource bottlenecks `TICKET-5962`

## Resolved-TS Causes TiKV OOM

- Phenomenon:
  - Usually appears together with large transactions or a stuck GC safepoint.
  - Memory rises again quickly after OOM and repeated OOM can follow.
  - Large deletes can amplify the problem.
- Trigger / Preconditions:
  - Resolved-ts remains enabled while many residual locks accumulate.
- Root cause:
  - Resolved-ts retains lock-related state and can consume significant memory when old locks are not cleaned quickly.
- Workaround:
  - Disable `resolved-ts` with rolling restart when dependent features are not needed.
  - If it cannot be disabled, prioritize residual-lock cleanup and avoid an excessively large GC safepoint window.
  - Do not expect `memory-usage-limit` alone to solve this case.
- Related oncall:
  - TiKV v6.5.5 OOM `ONCALL-7027`
  - v6.5.3 TiKV frequently OOMed before GC got stuck `TCOC-2175`
  - v5.1.4 TiKV had OOM risk `ONCALL-6891`
  - massive deletion caused TiKV memory OOM `TCOC-1299`

## Resolved-TS Worker CPU Saturation Causes Instant Memory Spike

- Phenomenon:
  - After resolved-ts worker CPU becomes fully utilized, pending tasks surge and TiKV memory can jump suddenly into OOM.
- Key metrics:
  - `tikv-details -> resolved ts`
  - resolved-ts CPU fully utilized
  - pending command size rising
  - worker pending tasks heavily backlogged
- Workaround:
  - Disable `resolved-ts` if the feature is not required.
- Related oncall:
  - v8.5.1 TiKV OOM due to full resolved-ts worker `TICKET-6935` / `ONCALL-8062`
  - v6.5.2 TiKV OOM analysis `TICKET-5877` / `ONCALL-7736`