# Scale-Out

Source note: adapted from the Feishu page about TiKV scale-in and scale-out behavior <https://pingcap.feishu.cn/wiki/VaYBwGJPaileuhk3VbRcGXpfnqd>.

## Core Model

During scale-out, PD uses `balance-region-scheduler` to create `balance-region` operators and move replicas from old stores to new stores.

Typical steps for one region:

1. `add learner` on the new store
2. `use joint consensus` to promote the new peer to voter and demote the old peer to learner
3. `leave joint state`
4. `remove peer` on the old store

If you need to locate the blocking point from PD logs, compare whether these steps continue to move forward:

- `add learner peer ... on store ...`
- `use joint consensus`
- `leave joint state`
- `remove peer on store ...`
- `operator finish`

## PD-Side Generation Path

`balance-region-scheduler` attempts scheduling every `10ms ~ 5s/10s`. When creating a `balance-region` operator, the key checks include:

1. `region-schedule-limit`
2. choosing the source store
3. choosing the peer to move
4. choosing the target store

Important filters:

- source:
  - `store limit remove-peer`
  - `max-snapshot-count`
- target:
  - placement rule
  - `store limit add-peer`
  - `max-snapshot-count`
  - `max-pending-peer-count`

## What To Check First

### No `balance-region` operator is created

Check in this order:

1. whether `balance-region-scheduler` is enabled
2. whether `region-schedule-limit` is saturated
3. whether the source store is limited by `remove-peer`
4. whether the target store is filtered by `add-peer`, `max-snapshot-count`, or `max-pending-peer-count`

Related metrics:

- `PD -> scheduler -> scheduler is running`
- `PD -> operator -> schedule reach limit`
- `PD -> scheduler -> filter source`
- `PD -> scheduler -> filter target`
- `PD -> operators -> schedule operator create -> balance region`

If the goal is to identify the exact blocking point from PD logs, look for these log sequences first:

- `add operator` with `balance-region {mv peer: store [...] to [...]}`
- `send schedule command` steps:
  - `add learner peer ...`
  - `use joint consensus ...`
  - `leave joint state ...`
  - `remove peer on store ...`
- `region ConfVer changed`
- `operator finish`

Heuristics:

- only `add operator` appears and there are no later steps: usually the region leader heartbeat is not advancing the operator
- stuck at `add learner` for a long time: switch to the TiKV snapshot path first
- joint consensus finishes quickly but the operator never finishes: inspect whether `remove peer` keeps retrying

## How To Decide PD vs TiKV

Compare these two metrics:

- `PD -> operators -> schedule operator create`
- `PD -> operators -> schedule operator finish`

Interpretation:

- low `create`: PD creates operators slowly
- high `create` but low `finish`: TiKV executes slowly

## TiKV-Side Snapshot Pipeline

`add learner` is essentially snapshot migration from the leader TiKV to the learner TiKV, followed by apply.

Critical path:

1. `pd-worker` receives the PD heartbeat
2. `raftstore` broadcasts conf change
3. `apply` updates metadata and triggers snapshot generation
4. `region-worker` forwards `Task::Gen`
5. `snap-generator` builds the snapshot
6. `snap-handler` or `snap-sender` sends the snapshot
7. the learner side receives the snapshot
8. the learner-side `region-worker` applies the snapshot to RocksDB

## Common Scale-Out Problems

### `add learner` is stuck

Common causes:

- thread-pool CPU bottleneck
- snapshot-related IO or network bottleneck
- RocksDB `apply snapshot` is stuck

Useful log keywords:

- `requesting snapshot`
- `scan snapshot`
- `start sending snapshot`
- `sent snapshot`
- `saving snapshot file`
- `begin to apply snapshot`
- `apply new data`

Important resource dimensions:

- leader side: CPU and disk bandwidth for `snap-generator`
- network: whether snapshot transport speed is clearly too low
- learner side: whether `apply snapshot` is slowed by RocksDB ingest or L0 pressure

### `add learner` enters a loop

Typical sequence:

1. PD creates `replace-rule-offline-*`
2. TiKV executes `add learner`
3. PD does not see success within 20 minutes, so the operator times out and is cancelled
4. 30 minutes later TiKV finally adds the learner
5. on the next rule-checker pass, PD treats this learner as an orphan peer and deletes it
6. the same cycle repeats

If you observe this loop, do not focus only on PD. The root cause is usually still that `add learner` is too slow.

### Newly added scale-out nodes become hotspot bottlenecks

During scale-out, old nodes concurrently stream regions into new nodes. If old nodes greatly outnumber new nodes, the new nodes are the first to saturate on write pressure.

Check first:

- `store limit add-peer` on the target store
- `max-snapshot-count`
- `max-pending-peer-count`
- `tikv-details -> snapshot -> snapshot state count`
- `tikv-details -> task -> worker pending tasks -> snapshot-worker`

## Thread Pools And Knobs

### `region-worker`

- the worker thread count is fixed at `1`
- it is responsible for snapshot `generate`, `apply`, and `destroy`
- check `tikv-details -> task -> worker pending tasks -> snapshot-worker`

Common mitigations:

- generate pressure is high: move hot leaders away, or use `evict-leader` if needed
- apply pressure is high: lower `store limit add-peer` on the PD side
- destroy pressure is high: lower `store limit remove-peer` on the PD side

### `snap-generator`

- config: `raftstore.snap-generator-pool-size`
- config: `server.snap-io-max-bytes-per-sec`

Useful for:

- insufficient CPU for snapshot generation
- insufficient disk bandwidth for snapshot generation

### `snap-handler` / `snap-sender`

- watch:
  - `server.concurrent-send-snap-limit`
  - `server.concurrent-recv-snap-limit`

Metrics:

- `tikv-details -> snapshot -> snapshot transport speed`
- `tikv-details -> snapshot -> snapshot state count`

## Important Online Config Examples

```sql
SET CONFIG "<tikv_ip:port>" `server.snap-io-max-bytes-per-sec` = '200MiB';
SET CONFIG "<tikv_ip:port>" `raftstore.snap-generator-pool-size` = 6;
SET CONFIG "<tikv_ip:port>" `server.concurrent-send-snap-limit` = 1;
SET CONFIG "<tikv_ip:port>" `server.concurrent-recv-snap-limit` = 48;
SHOW CONFIG WHERE type = 'tikv' AND name LIKE '%snap%';
```

## Special Case: `apply wait duration` increases

Phenomenon:

- `apply wait duration` is high
- apply CPU is not high
- enlarging the apply pool does not help

Suspect first:

- `ingest sst` triggering the RocksDB `global mutex`

Workaround:

- reduce migration speed during peak traffic
- if needed, lower `store limit` aggressively to contain impact first

## Scale-Out Knob Summary

| Config | Side | Use |
| --- | --- | --- |
| `region-schedule-limit` | PD | Limits concurrent region operators such as `balance-region` |
| `store limit add-peer/remove-peer` | PD | Controls migration speed for a specific store |
| `max-snapshot-count` | PD | Limits concurrent snapshot send and receive per store |
| `max-pending-peer-count` | PD | Limits pending peers per store |
| `raftstore.snap-generator-pool-size` | TiKV | Increases snapshot generation concurrency |
| `server.snap-io-max-bytes-per-sec` | TiKV | Increases the bandwidth cap for snapshot-related disk IO |
| `server.concurrent-send-snap-limit` | TiKV | Controls snapshot send concurrency |
| `server.concurrent-recv-snap-limit` | TiKV | Controls snapshot receive concurrency |