# Ingest SST During Data Movement Causes Cluster Latency Spikes

Source note: adapted from the Feishu `Storage oncall handbook` <https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb>.

- Phenomenon:
  - `apply wait` increases significantly while `apply` itself remains relatively low.
  - The apply pool can look mostly idle even though cluster write latency rises.
  - The latency trend often correlates strongly with snapshot apply or region movement windows.
- Trigger / Preconditions:
  - Scale-out, scale-in, region merge, or other data-movement operations are active.
  - The cluster is also serving foreground traffic, especially write-heavy workloads.
- Key logs:
  - No single mandatory log signature exists, but SST-ingest and compaction-related event logs often appear around the same time as the latency spike.
- Key metrics:
  - `apply wait` rises and tracks snapshot activity.
  - Ingestion-level and flow-control panels may show pressure from SST movement.
  - L0 file accumulation can increase if ingest overlaps heavily with existing SST ranges.
- Root cause:
  - Data movement deletes or rewrites old region state through SST ingest paths.
  - In some versions or workloads, SST ingest interacts poorly with RocksDB global mutexes.
  - The result is elevated apply wait and visible foreground write latency, even without an obvious CPU or thread-pool bottleneck.
- Workaround:
  - Scale out in smaller steps and avoid aggressive expansion ratios such as `1:3`.
  - During peak traffic, throttle new nodes with a small `store limit add-peer`.
  - Follow the SST-ingest mitigation SOP where available.
  - Consider fixes or cherry-picks associated with:
    - `tikv#18461`: avoid SST ingestion during destroy peer
    - `tikv#17408`: avoid SST ingestion during region merge
    - `tikv#18096`: do not block foreground writes while applying raft snapshot by ingesting SST
    - `tikv#18522`: do not block foreground writes during add index by ingesting SST
- Fixed version:
  - Version-dependent. The exact fix path depends on which ingest path dominates in the incident.
- Related oncall:
  - v7.5.6 ingest SST caused cluster latency spikes `GTOC-7709` / `APID-11430`
  - v7.5.6 scale-in caused obvious latency increase `ST-1824` / `TCOC-3589`
  - v7.1.0 scale-in doubled `apply wait` `TCOC-2212`
  - v6.5.4 latency `999` doubled while scaling from 3 TiKV to 9 TiKV `GTOC-6948` / `ST-612`