# [KB]-TiKV-实例-OOM-之后实例无法启动

## Key Words

TiKV, failed start, applying snapshot failed

## Description

TiKV 集群实例触发 OOM 重启后，在重启过程中报错 applying snapshot failed 而启动失败：

```golang
[FATAL] [lib.rs:465] ["[region 157802790] 175749763 applying snapshot failed"] [backtrace="stack backtrace:\n   0: tikv_util::set_panic_hook::{{closure}}\n   1: std::panicking::rust_panic_with_hook\n             at /rustc/bc39d4d9c514e5fdb40a5782e6ca08924f979c35//library/std/src/panicking.rs:595\n   2: std::panicking::begin_panic_handler::{{closure}}\n             at /rustc/bc39d4d9c514e5fdb40a5782e6ca08924f979c35//library/std/src/panicking.rs:497\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n             at /rustc/bc39d4d9c514e5fdb40a5782e6ca08924f979c35//library/std/src/sys_common/backtrace.rs:141\n   4: rust_begin_unwind\n             at /rustc/bc39d4d9c514e5fdb40a5782e6ca08924f979c35//library/std/src/panicking.rs:493\n   5: std::panicking::begin_panic_fmt\n             at /rustc/bc39d4d9c514e5fdb40a5782e6ca08924f979c35//library/std/src/panicking.rs:435\n   6: raftstore::store::peer_storage::PeerStorage<EK,ER>::check_applying_snap\n   7: raftstore::store::peer::Peer<EK,ER>::handle_raft_ready_append\n   8: <raftstore::store::fsm::store::RaftPoller<EK,ER,T> as batch_system::batch::PollHandler<raftstore::store::fsm::peer::PeerFsm<EK,ER>,raftstore::store::fsm::store::StoreFsm<EK>>>::handle_normal\n   9: batch_system::batch::Poller<N,C,Handler>::poll\n  10: std::sys_common::backtrace::__rust_begin_short_backtrace\n  11: core::ops::function::FnOnce::call_once{{vtable.shim}}\n  12: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rustc/bc39d4d9c514e5fdb40a5782e6ca08924f979c35/library/alloc/src/boxed.rs:1484\n      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rustc/bc39d4d9c514e5fdb40a5782e6ca08924f979c35/library/alloc/src/boxed.rs:1484\n      std::sys::unix::thread::Thread::new::thread_start\n             at /rustc/bc39d4d9c514e5fdb40a5782e6ca08924f979c35//library/std/src/sys/unix/thread.rs:71\n  13: start_thread\n  14: __clone\n"] [location=/data2/qizhi/zto-hotfix/tics/contrib/tiflash-proxy/components/raftstore/src/store/peer_storage.rs:1275] [thread_name=raftstore-2]
```

## Root Cause

TiKV 重启后部分 region peer 状态为 PeerState::Applying 开始 apply snpahost，但是发现对应 snapshot 已经被清理了（重启前收到了一个 gc snapshot 的消息 ）这个时候就会触发 panic。

## Diagnostic Steps

N/A

## Resolution

升级到未受影响版本

## Workaround

通过 tikv-ctl --data-dir /path/to/tikv tombstone -p 127.0.0.1:2379 -r <region_id>,<region_id> --force 手动将有问题的 region ID 执行 --force 设置为 tombstone，如果是被 merge 了，那之后另外一个 region 找不到这个 peer 可能还会 panic，到时候再执行手工 tombstone 操作。

## Affected Versions

v5.0.0~v5.0.6

v5.1.0~v5.1.4

v5.2.0~v5.2.3

v5.3.0

v5.4.0