# Scale-In Source note: adapted from the Feishu page about TiKV scale-in and scale-out behavior . ## Store States Before analyzing scale-in, distinguish the store state first: - `Up`: healthy - `Offline`: manually decommissioned - `Disconnect`: PD has not received heartbeats for about 20 seconds and does not immediately migrate data - `Down`: PD has not received heartbeats for about 30 minutes and starts replica recovery - `Tombstone`: all data has been moved away, the store has been safely removed, and it is irreversible To distinguish manual versus passive removal, inspect: - `PD -> abnormal stores -> Offline stores` - `PD -> abnormal stores -> Down stores` - `PD -> region health -> Offline-peer-region-count` - `PD -> region health -> Down-peer-region-count` Additional notes: - `Disconnect -> Up` may recover by itself and does not imply migration - `Down -> Up` may also recover; without a manual decommission it will not become `Offline` - a store that stays `Down` for a long time and has already been drained still needs manual decommission before it becomes `Tombstone` ## Core Model Scale-in is still replica migration. The operators are usually: - `replace-rule-offline-peer` - `replace-rule-offline-leader-peer` If the peer being removed is the leader, there is one extra step: 1. `add learner` 2. `transfer leader` 3. `use joint consensus` 4. `leave joint state` 5. `remove peer` ## PD-Side Generation Path During scale-in, PD relies on `PatrolRegion` to scan regions periodically and uses `rule-checker` to create operators. Main checkers: - `jointStateChecker` - `splitChecker` - `rule-checker` - `merge-checker` The most important checker for scale-in is `rule-checker`, which handles: - orphan peer - missing peer - unhealthy peer - replacing offline or down peers ## Key Knobs And Metrics ### Factors affecting operator creation speed - `patrol-region-interval` - `replica-schedule-limit` - `store limit remove-peer/add-peer` - `max-snapshot-count` - `max-pending-peer-count` Key metrics: - `PD -> schedule -> patrol region time` - `PD -> schedule -> Rule checker` - `PD -> operator -> scheduler reach limit` - `PD -> scheduler -> filter target/resource` - `PD -> operator -> scheduler operator create` ## Common Scale-In Problems ### `replace-rule-offline-*` creation rate is unstable Phenomenon: - when one TiKV is being decommissioned in a large cluster, operator creation can frequently drop to 0 Root cause: - `PatrolRegion` only scans part of the regions at a time - the cluster has many regions overall, but only a small subset belong to the store being decommissioned Workaround: - reduce `patrol-region-interval` - if the store must be removed quickly, scripted `pd-ctl` operator creation is often faster - if business impact is acceptable, avoid aggressive tuning ### `replace-rule-offline-*` stays at 0 for a long time Phenomenon: - the store is already being decommissioned, but almost no operator is created Check first: - whether PD logs contain `schedule deny` Root cause: - some tool has added `schedule=deny` to a specific key range Workaround: - remove the label rule and restore scheduling ## How To Read PD Logs During Scale-In The normal key log sequence for manual decommission is: 1. `DeleteStore` API is called 2. `store has been offline` 3. PD automatically raises the store's `remove-peer` limit to a very high value 4. `PatrolRegion` starts creating `replace-rule-offline-*` 5. regions finish execution one by one 6. `store has been Tombstone` 7. `store limit removed` If this chain breaks: - if step 4 is missing, inspect `PatrolRegion`, `rule-checker`, and `schedule deny` - if step 4 exists but finish is slow, switch to the `add learner` path and inspect TiKV If the input is PD logs, these keywords are the most recognizable: - `DeleteStore` - `store has been offline` - `store limit changed` - `replace-rule-offline-peer` - `replace-rule-offline-leader-peer` - `store has been Tombstone` - `store limit removed` ## Offline vs Down - `offline`: manual decommission, PD migrates data away safely and gradually - `down`: passive failure, PD starts replica recovery after the threshold is reached The TiKV-side migration path is similar for both, but: - `offline` is more of an operations control-plane scenario - `down` is more about failure healing and availability recovery If the leader peer is on the offline store, the operator has one extra `transfer leader` step: 1. `add learner` 2. `transfer leader` 3. `use joint consensus` 4. `leave joint state` 5. `remove peer` ## Large-Cluster PD Stall Symptoms: - during store online or offline operations, all region-related PD requests slow down - including region heartbeat and TiDB `get region` Known cause: - PD estimates remaining online or offline progress through `GetRegionSizeByRange`, which contends on the large region metadata lock Workaround: - upgrade to a fixed version: `v6.5.6 / v7.1.3 / v7.5.0` - reduce `max-store-preparing-time` to a value like `10s` so PD stops progress estimation sooner ## Scale-In Knob Summary | Config | Side | Use | | --- | --- | --- | | `patrol-region-interval` | PD | Controls the scan cadence of `PatrolRegion` | | `replica-schedule-limit` | PD | Limits concurrent replica operators created by `rule-checker` | | `store limit add-peer/remove-peer` | PD | Controls migration speed for the store being decommissioned | | `max-snapshot-count` | PD | Limits concurrent snapshot send and receive per store | | `max-pending-peer-count` | PD | Limits pending peers per store | | `max-store-preparing-time` | PD | Limits how long PD spends estimating online or offline progress to avoid large-cluster slowdown | | `raftstore.snap-generator-pool-size` | TiKV | Increases snapshot generation concurrency | | `server.snap-io-max-bytes-per-sec` | TiKV | Increases the bandwidth cap for snapshot-related disk IO | | `server.concurrent-send-snap-limit` | TiKV | Controls snapshot send concurrency | | `server.concurrent-recv-snap-limit` | TiKV | Controls snapshot receive concurrency | ## Quick Summary Recommended scale-in troubleshooting order: 1. Confirm that the store state matches expectations. 2. Check whether `replace-rule-offline-*` has been created. 3. If creation is slow, inspect `PatrolRegion`, `rule-checker`, and `schedule deny`. 4. If creation is normal but finish is slow, switch to the `add learner` and snapshot migration path.