# Restart and Upgrade Known Cases

Source note: adapted from the Feishu `Storage oncall handbook` <https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb>.

## TiKV Rolling Restart Increases Cluster Duration

- Phenomenon:
  - Cluster duration rises during rolling restart.
  - The latency increase can be noticeable even without a hard failure.
- Trigger / Preconditions:
  - A rolling restart is in progress.
- Root cause:
  - `evict-leader` during restart reduces block-cache hit rate and increases IO traffic.
- Workaround:
  - Perform rolling restart during off-peak hours.
- Fixed version:
  - No fixed version is documented in the source material.
- Related oncall:
  - v7.1.1 duration increased noticeably during upgrade `TCOC-3000`

## TiKV Restart Stuck by Compaction

- Phenomenon:
  - TiKV takes unusually long to shut down or restart because compaction is still running.
- Trigger / Preconditions:
  - TiKV restart overlaps with ongoing compaction, especially manual compaction.
- Root cause:
  - TiKV waits for compaction to finish before shutdown.
  - Some fixes only block new or queued manual compaction and cannot cancel a compaction that is already running.
- Workaround:
  - If restart is blocked, try to kill the TiKV process.
- Related oncall:
  - v6.5.6 TiKV took 7 minutes to shut down `GTOC-6780`
  - v8.5.1 TiKV process did not terminate because of running manual compaction `NAID-11276` / `GTOC-7524`

## TiKV Cross-Version Upgrade Failure Due to `enable_multi_thread_write` Compatibility

- Phenomenon:
  - TiKV fails to start during a multi-version upgrade and reports:

```text
[FATAL] [lib.rs:479] ["failed to load_latest_options \"Invalid argument: Could not find option: : enable_multi_thread_write\""]
```

- Trigger / Preconditions:
  - TiKV is upgraded across multiple major release lines, for example `v5.4.1 -> v8.5.1`.
- Root cause:
  - Newer versions cannot recognize some legacy RocksDB option state written by older versions.
- Workaround:
  - Follow a staged upgrade path such as `v5.4.1 -> v6.5.7 -> v8.5.1`.
  - For emergency recovery, roll back only TiKV to `v6.5.7`, start the cluster, then upgrade TiKV forward again while keeping other components on the newer line.
  - Do not attempt to downgrade PD from `v8.5.1` to `v6.5.7`.
- Related oncall:
  - v5.4.1 -> v8.5.1 upgrade failure `TICKET-6319` / `ONCALL-7898`
  - v5.4.1 -> v8.5.1 upgrade failure `TICKET-5651` / `ONCALL-7646`