--- name: tikv-scale description: | Diagnose TiKV scale-out and scale-in behavior in TiDB clusters. Use when investigating TiKV scale-out, scale-in, store online or offline transitions, slow data migration, slow operator creation, add-learner timeout, snapshot migration bottlenecks, offline or down store recovery, patrol-region or rule-checker scheduling problems, store state transitions, or when explaining TiKV node scaling behavior with PD and TiKV logs and metrics. --- # TiKV Scale Handbook Source note: adapted from the Feishu page about TiKV scale-in and scale-out behavior and the repository root PDF with the same topic. TiKV scaling diagnosis can usually be split into two stages: 1. How fast `PD` creates scheduling operators 2. How fast `TiKV` executes replica migration Identify which stage is the bottleneck first. Do not start by tuning both PD and TiKV at the same time. ## When To Use This Skill Use this skill when: - the user wants to explain TiKV scale-out or scale-in behavior - a new TiKV node stays empty and does not receive data - scale-in is slow and `replace-rule-offline-peer` or `replace-rule-offline-leader-peer` is created slowly - `add learner` times out, loops, or gets stuck in snapshot generate, send, receive, or apply - PD metrics show low `operator create` throughput or much slower `operator finish` - you need to decide whether the bottleneck is PD operator generation or TiKV data migration - you need to explain `offline`, `down`, or `tombstone` state changes and the related logs - you need to explain why data migration starts from the transition among `Up`, `Disconnect`, `Down`, `Offline`, and `Tombstone` ## First Pass Collect these facts first: - operation type: `scale-out`, `scale-in`, automatic healing for abnormal `down`, or ordinary rebalancing - affected store state: `up`, `offline`, `disconnect`, `down`, or `tombstone` - `PD -> operators -> schedule operator create` - `PD -> operators -> schedule operator finish` - `PD -> scheduler/filter` and `PD -> operator/scheduler reach limit` - `tikv-details -> task -> worker pending tasks -> snapshot-worker` - `tikv-details -> snapshot -> transport speed / state count` - whether `balance-region`, `replace-rule-offline-peer`, or `replace-rule-offline-leader-peer` time out - whether a `schedule=deny` label rule exists or PD becomes slow in a large cluster Quick interpretation: - low or unstable `create`: start with `PD` - high `create` but lagging `finish`: start with `TiKV` - when scale-in only needs to move a few replicas, `create` jitter is not always a bug and may only reflect `PatrolRegion` scan granularity ## Workflow ### 1. Split by scenario - scale-out: focus on `balance-region-scheduler` - scale-in: focus on `PatrolRegion -> rule-checker -> replace-rule-offline-*` - abnormal store loss: distinguish `offline` from `down` ### 2. Split by stage - `PD` creates operators - `TiKV` executes `add learner -> joint consensus -> remove peer` ### 3. Find the blocking step - no operator: inspect scheduler, rule-checker, and `schedule deny` - operator created but stuck: compare with PD logs to identify the blocked step - slow `add learner`: determine whether snapshot generation, sending, receiving, or apply is slow ## Quick Routing - scale-out scheduling generation and common filters: see [scale-out.md](references/scale-out.md) - scale-in state machine, PatrolRegion, rule-checker, and offline logs: see [scale-in.md](references/scale-in.md) ## Common Commands ```bash tiup ctl: pd store tiup ctl: pd region tiup ctl: pd config show tiup ctl: pd schedule show tiup ctl: pd schedule remove balance-region-scheduler tiup ctl: pd schedule add balance-region-scheduler grep -n "add operator\\|operator finish\\|schedule deny\\|store has been offline\\|store has been Tombstone" pd.log rg -n "requesting snapshot|sent snapshot|saving snapshot file|begin to apply snapshot|apply new data" /path/to/tikv.log ``` ## Fast Heuristics - a new TiKV node stays empty without clear inbound traffic: check `balance-region-scheduler`, `store limit add-peer`, and `max-snapshot-count` - `offline` never finishes: first inspect `replace-rule-offline-*` create rate, then inspect `add learner` - `add learner` times out after 20 minutes and later succeeds anyway: suspect an orphan learner loop - `apply wait duration` is high but apply CPU is not: suspect `ingest sst` and the RocksDB `global mutex` - in large clusters, PD region-related requests become slow during store online or offline operations: check `pd#7248`-like issues and `max-store-preparing-time` - `Disconnect` does not trigger migration immediately; `Down` triggers replica recovery; `Offline` is an explicit safe decommission; `Tombstone` is irreversible ## Key Knobs Only change these after you have identified a clear bottleneck: - `region-schedule-limit` - `replica-schedule-limit` - `store limit remove-peer / add-peer` - `max-snapshot-count` - `max-pending-peer-count` - `raftstore.snap-generator-pool-size` - `server.snap-io-max-bytes-per-sec` - `server.concurrent-send-snap-limit` - `server.concurrent-recv-snap-limit` - `max-store-preparing-time` ## Notes - `balance-leader-scheduler` only changes leader metadata and is usually not the main scaling bottleneck - after scale-in finishes, the store moves from `offline` to `tombstone`; `tombstone` is irreversible - this skill is an operational entry for first-response diagnosis and for explaining scaling principles