---
name: tikv-oncall-handbook
description: |
  Entry skill for TiKV on-call diagnosis.
  Use when triaging TiKV incidents and deciding whether the problem belongs to
  panic/crash, raftstore and region lifecycle, unsafe recovery, storage and
  RocksDB/Titan, performance and flow-control, status-server profiling, or CDC.
  Includes routing guidance, first-response checks, and links to specialized
  TiKV diagnosis skills.
---

# TiKV Oncall Handbook

Entry guide for TiKV on-call diagnosis. Use this skill first to classify the incident,
then continue with the specialized handbook for that symptom family.

Source note: adapted from the Feishu `Storage oncall handbook` <https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb>.

## Skill Family

- `tikv-panic`: crash, restart loop, fatal startup failure, upgrade failure
- `tikv-raftstore`: split, add learner, snapshot apply, peer cleanup, startup apply-busy
- `tikv-recovery`: majority loss, unsafe recovery, tombstone, orphan peers
- `tikv-performance`: gRPC bottleneck, flow control, disk saturation, hot regions
- `tikv-storage`: RocksDB, Titan, compaction, host or disk behavior, SST corruption
- `tikv-status-server`: status port, pprof, profile collection, FD leak, slow backtrace collection
- `tikv-cdc`: CDC-triggered TiKV panic, stale endpoint reuse, TiKV address reuse
- `transaction-oncall-handbook`: stale read, follower read, transaction correctness, client-go cache behavior

## Routing Guide

| If you see... | Read this skill |
| --- | --- |
| panic, crash, restart loop, fatal log | `skills/diagnosis/tikv-panic/SKILL.md` |
| split, add learner, snapshot, region move, peer cleanup, startup apply-busy | `skills/diagnosis/tikv-raftstore/SKILL.md` |
| majority lost, unsafe recovery, tombstone, orphan peer | `skills/diagnosis/tikv-recovery/SKILL.md` |
| flow control, gRPC bottleneck, IOPS saturation, latency spike, network jitter | `skills/diagnosis/tikv-performance/SKILL.md` |
| RocksDB, Titan, SST corruption, compaction pressure, disk-engine behavior, host-level disk anomalies | `skills/diagnosis/tikv-storage/SKILL.md` |
| status port, pprof, profile collection, FD leak | `skills/diagnosis/tikv-status-server/SKILL.md` |
| CDC-triggered panic, changefeed lag, stale endpoint or store reuse | `skills/diagnosis/tikv-cdc/SKILL.md` |
| transaction-specific TiDB/TiKV behaviors | `skills/diagnosis/transaction-oncall-handbook/SKILL.md` |

## First Response Checklist

- Confirm whether the issue is primarily correctness, availability, or performance.
- Identify whether the blast radius is single-store, single-region, or cluster-wide.
- Capture the first fatal or abnormal log around the incident start time.
- Check whether recent operations exist: scale-in/out, upgrade, restart, disk expansion, recovery, DDL, PITR, CDC, or ingest SST.
- Record exact versions of PD, TiKV, TiDB, TiCDC, and whether TiFlash is involved.

## Common Commands

```bash
tiup cluster display <cluster-name>
tiup ctl:<version> pd -u http://<pd>:2379 store
tiup ctl:<version> pd -u http://<pd>:2379 region <region_id>
tikv-ctl --host <tikv_addr> region-properties -r <region_id>
rg -n "FATAL|panic|SIGSEGV|ServerIsBusy" /path/to/tikv.log
```

## Specialized Skills

- `skills/diagnosis/tikv-panic/SKILL.md`
- `skills/diagnosis/tikv-raftstore/SKILL.md`
- `skills/diagnosis/tikv-recovery/SKILL.md`
- `skills/diagnosis/tikv-performance/SKILL.md`
- `skills/diagnosis/tikv-storage/SKILL.md`
- `skills/diagnosis/tikv-status-server/SKILL.md`
- `skills/diagnosis/tikv-cdc/SKILL.md`
- `skills/diagnosis/transaction-oncall-handbook/SKILL.md`

## Triage Hints

- If the process exits or loops on restart, start with `tikv-panic`.
- If the issue appears only during region split, scale-in, add learner, snapshot apply, or startup apply-busy after restart, start with `tikv-raftstore`.
- If the cluster already lost replicas or requires manual data repair, start with `tikv-recovery`.
- If the incident is mostly latency, throttling, or saturation, start with `tikv-performance`.
- If the symptom centers on RocksDB, Titan, SST files, compaction, disk growth, or host-level disk anomalies, start with `tikv-storage`.
- If the problem appears while collecting profile, pprof, or status-port data, start with `tikv-status-server`.
- If the symptom starts from CDC panic, changefeed lag, or endpoint reuse after topology changes, start with `tikv-cdc`.
- If the symptom involves stale read, follower read, schema-load side effects, or client-go store-cache behavior, start with `transaction-oncall-handbook`.