# [Critical bug] TiKV 运行超过两年可能 panic

  这个问题出现在 [ONCALL-4441](https://internal.pingcap.net/jira/browse/ONCALL-4441)中，影响所有历史版本。

## Issue

TiKV 持续运行超过 795 天可能会出现 panic 的问题。

```golang
[2022/02/08 20:32:34.418 +08:00] [ERROR] [util.rs:419] ["request failed, retry"] [err_code=KV:PD:Unknown] [err="Other(\"[components/pd_client/src/tso.rs:85]: TimestampRequest channel is closed\")"]
[2022/02/08 20:32:34.418 +08:00] [ERROR] [util.rs:419] ["request failed, retry"] [err_code=KV:PD:Unknown] [err="Other(\"[components/pd_client/src/tso.rs:85]: TimestampRequest channel is closed\")"]
[2022/02/08 20:32:34.418 +08:00] [ERROR] [util.rs:419] ["request failed, retry"] [err_code=KV:PD:Unknown] [err="Other(\"[components/pd_client/src/tso.rs:85]: TimestampRequest channel is closed\")"]
[2022/02/08 20:32:34.418 +08:00] [ERROR] [util.rs:419] ["request failed, retry"] [err_code=KV:PD:Unknown] [err="Other(\"[components/pd_client/src/tso.rs:85]: TimestampRequest channel is closed\")"]
[2022/02/08 20:32:34.421 +08:00] [FATAL] [lib.rs:465] ["index out of bounds: the len is 6 but the index is 6"] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}\n             at /home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:464:18\n   1: std::panicking::rust_panic_with_hook\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:626:17\n   2: std::panicking::begin_panic_handler::{{closure}}\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:519:13\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:141:18\n   4: rust_begin_unwind\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:515:5\n   5: core::panicking::panic_fmt\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/panicking.rs:92:14\n   6: core::panicking::panic_bounds_check\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/panicking.rs:69:5\n   7: <usize as core::slice::index::SliceIndex<[T]>>::index_mut\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/slice/index.rs:190:14\n      core::slice::index::<impl core::ops::index::IndexMut<I> for [T]>::index_mut\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/slice/index.rs:26:9\n      <alloc::vec::Vec<T,A> as core::ops::index::IndexMut<I>>::index_mut\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/alloc/src/vec/mod.rs:2445:9\n      tokio_timer::wheel::Wheel<T>::insert\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/wheel/mod.rs:114:9\n      tokio_timer::timer::Timer<T,N>::add_entry\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/timer/mod.rs:324:15\n   8: tokio_timer::timer::Timer<T,N>::process_queue\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/timer/mod.rs:301:21\n   9: <tokio_timer::timer::Timer<T,N> as tokio_executor::park::Park>::park\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/timer/mod.rs:361:9\n      tokio_timer::timer::Timer<T,N>::turn\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/timer/mod.rs:256:21\n  10: tikv_util::timer::start_global_timer::{{closure}}\n             at /home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tikv/components/tikv_util/src/timer.rs:98:17\n  11: std::sys_common::backtrace::__rust_begin_short_backtrace\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:125:18\n  12: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/thread/mod.rs:476:17\n  13: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panic.rs:347:9\n  14: std::panicking::try::do_call\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:401:40\n      std::panicking::try\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:365:19\n      std::panic::catch_unwind\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panic.rs:434:14\n      std::thread::Builder::spawn_unchecked::{{closure}}\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/thread/mod.rs:475:30\n      core::ops::function::FnOnce::call_once{{vtable.shim}}\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/ops/function.rs:227:5\n  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/alloc/src/boxed.rs:1572:9\n      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/alloc/src/boxed.rs:1572:9\n      std::sys::unix::thread::Thread::new::thread_start\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys/unix/thread.rs:91:17\n  16: start_thread\n  17: __GI___clone\n"] [location=/rust/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/wheel/mod.rs:114] [thread_name=timer]
```

## Root Cause

TiKV 使用了 deprecated 状态的 tokio-timer，该库存在进程运行时间超过 2^36 ms 可能出现 index out of bounds 的问题。

发生问题的代码:

```golang
fn level_for(elapsed: u64, when: u64) -> usize {
    const SLOT_MASK: u64 = (1 << 6) - 1;

    // Mask in the trailing bits ignored by the level calculation in order to cap
    // the possible leading zeros
    let masked = elapsed ^ when | SLOT_MASK;

    let leading_zeros = masked.leading_zeros() as usize;
    let significant = 63 - leading_zeros;
    significant / 6
}
```

elapsed 为进程启动后经过的时间，when 为期待被唤醒的时间，期望计算所得的 level 在 0 - 5 之间，超过则会导致 index 越界，如 elapsed 为 2 ^ 36 - 1，when 为 2 ^ 36，讲计算得到 level = 6，发生 index out of bounds 导致 panic。

## Diagnostic Steps

检查 TiKV 进程是否已经启动超过两年，可以参考监控中的 uptime。

## Resolution

fix version: 5.0.7, 5.1.5, 5.2.4, 5.3.1, 5.4.1, 6.0.0.

## Workaround

该 bug 只影响运行时间超过 795 天的 TiKV 进程，以两年的周期重启 TiKV 能够预防。出现 bug panic 的 TiKV 重启后恢复正常。