# Multiple TiKV Nodes Offline Cause Add-Learner Timeout Loops

Source note: adapted from the Feishu `Storage oncall handbook` <https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb>.

- Phenomenon:
  - PD reports repeated `add learner` timeouts.
  - The operator can loop in `replace-rule-offline(leader)-peer` and scale-in gets stuck.
- Trigger / Preconditions:
  - Multiple TiKV nodes are taken offline at the same time.
  - Replica recovery overlaps with high snapshot pressure and heavy write load.
- Key logs:
  - PD side symptoms are repeated `add learner` timeout records and repeated recreation of replacement operators.
- Key metrics:
  - `apply-delay` stays high.
  - Region-worker CPU is saturated on the snapshot-heavy node.
  - `snap-generator` thread pool remains busy.
  - RocksDB level-0 files frequently exceed the ingest-safe range during recovery.
- Root cause:
  - PD issues `add learner` and times out after about 20 minutes.
  - TiKV may still finish the learner add after roughly 30 minutes and then promote it.
  - PD later observes the promoted learner as an orphan peer and removes it.
  - The rule checker then retries the same unhealthy-peer repair path and re-enters the loop.
  - The timeout mismatch is amplified by region-worker CPU saturation, snapshot generation pressure, and level-0 buildup that blocks or delays snapshot apply.
- Workaround:
  - Throttle PD scheduling when the recovery path is resource-bound.
  - Evict leaders from the node generating too many snapshots to reduce snapshot generation pressure.
  - Increase `snap-generator-pool-size` if snapshot generation is the bottleneck.
  - If `evict-leader` cannot proceed because other peers are already offline, force restart the TiKV node.
  - Operationally, avoid taking many TiKV nodes offline at once.
- Fixed version:
  - No single fixed version is documented in the source material. Treat this as an operational failure mode and mitigate through scheduling control and rollout discipline.
- Related oncall:
  - v6.5.7 taking 10 TiKV nodes offline at the same time got stuck `TICKET-4746` / `TICKET-4681` / `ONCALL-7328`