--- name: tikv-panic description: | Reference handbook for TiKV panic, crash, and restart-loop incidents. Use when investigating SIGSEGV, abnormal restart, clock rollback, apply snapshot failure, missing SST files, PITR pause/resume panic, coprocessor response overflow, or other fatal TiKV process exits. Includes symptoms, log signatures, workarounds, and version notes. --- # TiKV Panic Handbook Known panic, crash, and restart-loop patterns for TiKV on-call diagnosis. Source note: adapted from the Feishu `Storage oncall handbook` . ## When To Use This Skill Use this skill when: - TiKV exits with `FATAL`, panic backtrace, or segmentation fault - one or more TiKV nodes enter restart loops - the first visible symptom is process crash instead of latency or throughput degradation - oncall needs quick log signatures, workarounds, and fixed-version guidance ## First Checks - Identify whether the crash is single-store or cluster-wide. - Capture the first fatal log around the incident start time. - Check whether recent operations exist: restart, upgrade, PITR, disk expansion, ingest SST, or time-sync change. - Confirm exact versions of TiKV, PD, and TiDB. ## Issue Index ### SIGSEGV - `TiKV crashes with SIGSEGV` ### Storage-related panic - `Missing SST files during ingest may panic TiKV` - `SST file permission changed causes TiKV panic` ### Restart-related panic - `TiKV abnormal restart damages raft state` - `Clock rollback causes TiKV panic` ### Task-triggered panic - `TiKV resume or pause PITR task causes panic` - `Coprocessor response too large causes TiKV panic` ## Common Commands ```bash tiup cluster display tiup ctl: pd -u http://:2379 store rg -n "FATAL|panic|SIGSEGV" /path/to/tikv.log ``` ## Detailed Cases ### TiKV crashes with SIGSEGV - Detailed reference: [`references/sigsegv-on-arm-rocksdb.md`](references/sigsegv-on-arm-rocksdb.md) - Core dumps are useful for this class of crash. See `How to: Enable Core Dumps on TiKV (TiUP-deployed)`. - Root cause: an upstream RocksDB issue in `MemTable::GetBloomFilter()` can trigger a data race on ARM weak-memory platforms; see [rocksdb#19096](https://github.com/facebook/rocksdb/issues/19096). - Trigger condition: ARM weak-memory environments. The issue is rare and hard to reproduce. - Related oncall: - v8.5.3 Segment fault in RocksDB write operation `ONCALL-8053` - Manus v8.5.1 TiKV segfault `GTOC-7804` `APID-11531` - Blot v8.5.1 57/63 TiKV instances crashed in two minutes `EMID-10050` `GTOC-7770` - Airbnb v8.5.1 TiKV segmentation violation (`SIGSEGV`) crash `GTOC-7785` `NAID-11514` ### Missing SST files during ingest may panic TiKV - Phenomenon: operator mistakes or hardware faults remove an SST file that TiKV is about to ingest, causing a fatal IO error and process panic. - Example log: ```text [2025/06/23 11:35:28.116 +07:00] [FATAL] [lib.rs:509] ["[region 4408007] 4408008 ingest uuid: A0A7C56D3998465A913DFA0A256B0B7F ... IO error: No such file or directory: while stat a file for size: /var/lib/tikv/import/a0a7c56d-3998-465a-913d-fa0a256b0b7f_4408007_47491_207554_write.sst: No such file or directory"] ``` - Workaround: - Force the affected regions to `tombstone` before restarting TiKV. If the TiKV operator cannot stop the instance, attach the disk to another environment with PD access and run `tombstone` there. ```bash tikv-ctl --data-dir /path/to/tikv tombstone -p 127.0.0.1:2379 -r , --force pd-ctl operator add remove-peer ``` - If too many regions are affected, follow the full TiKV decommission workflow instead of repairing them one by one. - Related oncall: - v6.5.7 TiKV panic after disk-capacity expansion `APID-11405` `GTOC-7703` ### TiKV abnormal restart damages raft state - Phenomenon: after an abnormal restart, hardware or OS issues can make `sync` return success even though data was not fully persisted. TiKV then fails to start with logs like `to_commit 777777 is out of range [last_index 777776], raft_id:2222, region_id:1111`. - Workaround: - Tombstone the affected peer before startup: ```bash tikv-ctl --data-dir /path/to/tikv tombstone -p -r ``` - If TiKV cannot start at all, the PD parameter is not required: ```bash tikv-ctl --data-dir /path/to/tikv tombstone -r ``` - Related oncall: - v6.5.2 TiKV could not start after leader eviction and direct host restart `ONCALL-7734` `TICKET-5879` ### Restart and upgrade known cases - Detailed reference: [`references/restart-and-upgrade-known-cases.md`](references/restart-and-upgrade-known-cases.md) - Use this reference when TiKV cannot restart cleanly, startup takes unexpectedly long, or a multi-version upgrade fails during TiKV startup. ### TiKV resume or pause PITR task causes panic ([tikv#17020](https://github.com/tikv/tikv/issues/17020)) - Phenomenon: when resuming a PITR task, if the resume path hits a PD connection error, TiKV can panic with: ```text [FATAL] [lib.rs:509] ["there is no reactor running, must be called from the context of a Tokio 1.x runtime"] [location=/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/backup-stream/src/endpoint.rs:690] [thread_name=backup-stream-0] ``` - Workaround: - avoid frequent pause/resume operations on PITR tasks - upgrade to expected fixed versions: `v6.5.10`, `v7.1.6`, `v7.5.2`, `v8.1.0` - Related oncall: - TiKV v6.5.6 abnormal panic restart `ONCALL-7122` ### Coprocessor response too large causes TiKV panic ([tikv#9012](https://github.com/tikv/tikv/issues/9012)) - Phenomenon: due to [rust-protobuf#530](https://github.com/stepancheg/rust-protobuf/issues/530), if a coprocessor response exceeds 4 GB, TiKV may panic. Large regions or chunk-mode amplification with many `NULL` columns make this easier to trigger. - Workaround: - upgrade to `v5.2.2` or `v6.0+`, where an error is returned instead of a panic - disable chunk mode on the TiDB side via `tidb_enable_chunk_rpc` - locate and fix large-region root causes - Related oncall: - v5.1.4 TiKV panic `TICKET-4573` - v5.1.4 TiKV panic caused by coprocessor response too big `ONCALL-6924` ### SST file permission changes cause TiKV panic - Phenomenon: if SST file permissions are changed unexpectedly, TiKV may panic with `Permission denied` when reading the file. - Workaround: restore the file permissions, delete `panic_mark_file`, and then start TiKV. - Related oncall: - v6.5.5 antivirus software caused TiKV to fail reading SST files and panic `TICKET-4471` `ONCALL-7217` ### Clock rollback causes TiKV panic ([tikv#14518](https://github.com/tikv/tikv/issues/14518)) - Phenomenon: TiKV panics after system time jumps backward and logs messages like: ```text [2024/01/06 06:57:14.850 +00:00] [FATAL] [lib.rs:465] ["elapsed=37162644864; when=37162644862"] [location=/rust/git/checkouts/tokio-8e927faba632ed16/e8ac149/tokio-timer/src/wheel/mod.rs:225] [thread_name=steady-timer] ``` - Workaround: - usually all TiKV instances on the same host may panic together - search for nearby `ntp` or `clock wrong` logs to confirm the time anomaly - Fixed version: `v8.5.2` / `v8.1.3` / `v7.5.8` / `v7.1.7` / `v6.5.13` - Related oncall: - v8.5.1 TiKV panic `TICKET-6929` - v7.1.5 clock jump caused TiKV crash and restart `GTOC-7416` / `APID-11074` - v5.4.3 suspected OS time rollback caused TiKV panic `APID-10332` - v5.4.0 TiKV logged time rollback before panic `TICKET-1093` - v6.5.3 TiKV panicked due to time rollback `TICKET-4770` / `TICKET-4769` ### Clock re-synchronization notes - If only TiKV clocks are out of sync, synchronize them directly. The worst case is TiKV panic and restart; choosing not to restart may also be acceptable. - If PD clocks are out of sync, no manual action is usually needed. TSO physical-part allocation may stall for about 3 seconds. - Related oncall: - v6.5.5 cluster NTP clock drift `TICKET-5852` ## Related Skills - For split, snapshot, add learner, region movement, and peer lifecycle issues, read `skills/diagnosis/tikv-raftstore/SKILL.md`. - For unsafe recovery and tombstone workflows, read `skills/diagnosis/tikv-recovery/SKILL.md`. - For compaction and storage-engine pressure, read `skills/diagnosis/tikv-storage/SKILL.md`.