# Deterministic batched peer startup plan for large-region stores ## Problem During store startup, rfstore currently loads all local peers, registers them, and then sends `PeerMsg::Start` to every peer immediately. On stores with a very large number of regions, this creates a large startup burst: - all peers are activated at once; - peer startup messages and real incoming Raft messages share the same peer queue; - periodic peer ticks begin for the full peer set as soon as startup starts. Under high region counts, this can delay heartbeats, append messages, and early leader election traffic long enough that many regions appear unable to rejoin their Raft groups promptly. ## Goals - Reduce startup message and tick pressure on stores with very large numbers of regions. - Make peer activation order deterministic across stores so the same region is activated in the same startup batch on all nodes. - Preserve correctness by allowing peers to be registered and able to receive Raft messages before their periodic ticks are activated. - Keep the first version simple, localized, and operationally predictable. ## Non-goals - Dynamic or adaptive batch sizing in the first version. - Cross-store coordination through PD or a separate control plane. - Changing Raft recovery semantics, snapshot rules, or peer creation flow. - Reworking unrelated readiness or apply-catch-up logic beyond the minimum needed for batched startup. ## Design overview Introduce deterministic batched peer activation at startup for large-region stores. The key rule is: - register all loaded peers first; - only activate periodic peer ticking and election behavior in batches. This is important. Delaying peer creation or registration would force startup traffic for not-yet-registered peers through the existing pending-message path, which is intentionally small and not suitable for large startup fan-in. Batch activation is enabled only when the number of local peers loaded at startup is greater than a threshold: - `startup_batch_region_threshold = 30000` by default; - `startup_peer_per_batch = 10000` by default; - derived `startup_batch_count = ceil(local_peer_count / startup_peer_per_batch)`. Here `local peer count` means the number of persisted local peers loaded on the current store during startup, not the global region count of the whole cluster. When enabled, a peer belongs to startup batch: - `batch_id = stable_hash(region_id) % startup_batch_count` Batches are activated in increasing batch order: - `0, 1, 2, ..., startup_batch_count - 1` Within a store, the batch function depends only on `region_id` and the derived `startup_batch_count`, so activation order is deterministic for that store. The derived batch count depends on the local peer count, so strict cross-store batch alignment is not guaranteed. ## Why this is safe The proposal does not delay peer loading or peer registration. The peer remains present in local metadata and in the in-memory peer map before its startup batch is activated. This means: - incoming `RaftMessage`s can still be routed to the peer; - the peer can still step incoming append, heartbeat, vote, and snapshot traffic; - only locally generated periodic ticking and self-driven election startup are delayed. This makes the change a load-shedding and sequencing change, not a membership or storage correctness change. ## Activation model Add startup activation state for loaded peers. Suggested peer lifecycle during startup: 1. `Registered`: peer is loaded from disk and inserted into the store peer map. 2. `Inactive`: peer is registered but excluded from startup tick generation. 3. `Active`: peer has entered its startup batch, consumed `PeerMsg::Start`, and its ticker is initialized. 4. `Connected`: peer has observed enough Raft communication to be counted as having joined its group for batch progression purposes. ### Important behavior Inactive peers must be skipped by periodic tick generation. Only delaying `PeerMsg::Start` is not enough, because the Raft worker independently injects `PeerMsg::Tick` for all peers during normal processing. Without an explicit inactive-peer filter, the store still pays per-peer tick cost across the full peer set. ## Startup policy At startup: 1. Load all peers from disk as today. 2. Register all peers in the store context as today. 3. Compute `local_peer_count`. 4. Derive `startup_batch_count = ceil(local_peer_count / startup_peer_per_batch)`. 5. If `local_peer_count <= startup_batch_region_threshold` or `startup_batch_count <= 1`: - keep the existing startup behavior. 6. Otherwise: - mark all peers as startup-inactive; - activate only peers in the first non-empty `stable_hash(region_id) % startup_batch_count` batch; - activate later batches sequentially. ### Startup flow ```ascii RaftBatchSystem::spawn | v load_peers from disk | v StartupBatchState::new(region_ids) | +-- batching disabled ----------------------------------------------+ | | | StoreMsg::Start | | | | | v | | start_all_loaded_peers | | | | | v | | send PeerMsg::Start to every loaded peer | | | | | v | | mark_startup_activation_completed | | | +-- batching enabled -----------------------------------------------+ | v mark every loaded peer as INACTIVE | v StoreMsg::Start | v activate_next_startup_batch | v choose next non-empty batch by stable_hash(region_id) % startup_batch_count | v for each peer in the batch: INACTIVE -> ACTIVE send PeerMsg::Start | v peer consumes PeerMsg::Start | v initialize peer ticker | v send StoreMsg::PeerStartupActivated(region_id) | v are all peers in current batch activated? | +-- no --> wait for remaining peers to consume PeerMsg::Start | +-- yes --> arm current batch wait timer | v is this the final batch? | +-- yes --> mark_startup_activation_completed | +-- no --> wait for connectivity progress | v peer handles raft traffic / ready | v peer observes leader or becomes leader | v ACTIVE -> CONNECTED | v send StoreMsg::PeerStartupConnected(region_id) | v are all/over threshold peers in current batch connected? | +-- yes --> clear_current_batch | | | v | activate_next_startup_batch | +-- no --> did wait timeout expire? | +-- no --> keep waiting | +-- yes --> record timeout clear_current_batch | v activate_next_startup_batch ``` ## Batch completion criteria The design should avoid strict all-or-nothing barriers. A single slow peer, a snapshot wait, or a temporarily unhealthy region must not block startup forever. A peer is considered `Connected` when any of the following happens after its batch is activated: - the peer becomes leader; - the peer learns a valid leader through Raft state transition; - the peer observes leader commit progress from incoming Raft traffic. This keeps the signal tied to actual group communication rather than full apply catch-up. A batch completes when either: - all peers in the batch reach `Connected`; or - the connected peer ratio reaches `startup_batch_connected_threshold_percent`; or - a batch wait timeout expires. Recommended first-version timeout: - `startup_batch_wait_timeout = 2 * raft election timeout` This keeps the implementation simple and prevents a permanently stalled batch from blocking the rest of startup forever. The timeout should start only after all peers in the current batch have consumed `PeerMsg::Start` and initialized their tickers, so queueing delay for startup messages does not consume the connectivity wait budget. After a batch completes, the next batch is activated immediately. After the final batch is activated, all peers continue under normal steady-state behavior, regardless of whether some peers only progressed by timeout rather than by positive `Connected` observation. ## Configuration rfstore configuration items: - `startup_peer_per_batch: usize` - default: `10000` - `startup_batch_region_threshold: u64` - default: `30000` - `startup_batch_connected_threshold_percent: u64` - default: `90` Derived value (per store): - `startup_batch_count = ceil(local_peer_count / startup_peer_per_batch)` Behavior: - if derived `startup_batch_count <= 1`, batching is disabled; - if `local_peer_count <= startup_batch_region_threshold`, batching is disabled; - otherwise batching is enabled and `stable_hash(region_id) % startup_batch_count` selects the batch; - once a batch is fully activated, it can advance when all peers connect, when the connected peer ratio reaches `startup_batch_connected_threshold_percent`, or when the wait timeout expires. The first version can keep the batch timeout internal rather than user-facing if we want to minimize config surface area. ## Readiness impact Current readiness mainly tracks: - connection to PD; - whether Raft peers have caught up applying logs. That is not sufficient for batched peer startup, because a store could satisfy apply-based readiness while some peers have not even been activated yet. The proposal should add a startup-specific readiness condition, for example: - `startup_peer_activation_completed` `/ready` should become true only after: - connected to PD; - Raft peers caught up under existing rules; - all startup batches have been activated. This keeps readiness aligned with externally visible startup behavior. ## Observability Add startup logs and metrics so operators can understand startup progress on large stores. Suggested metrics: - total loaded peers at startup; - whether batched startup is enabled; - current startup batch id; - total peers in current batch; - connected peers in current batch; - batch activation duration; - number of batches completed by timeout. Suggested logs: - startup batching enabled/disabled and the reason; - batch activation start; - batch activation completion; - batch timeout completion with remaining unconnected peer count. ## Affected modules - `components/cloud_server/src/tikv_server.rs` - startup wiring and local peer count handoff. - `components/rfstore/src/store/store_fsm.rs` - startup sequencing and store-level startup controller state. - `components/rfstore/src/store/peer_worker.rs` - tick generation must skip inactive peers. - `components/rfstore/src/store/peer_fsm.rs` - peer activation entry point and connection-progress observation hooks. - `components/rfstore/src/store/peer.rs` - role-change and leader-progress signals used to mark a peer connected. - `components/tikv_util/src/lib.rs` - server readiness state if a new readiness flag is added. ## Alternatives considered ### Keep the current all-at-once startup Rejected. This is the source of the startup burst and does not scale to very large region counts. ### Delay peer registration and only create peers batch by batch Rejected. This would push startup traffic into the pending-message path for not-yet-created peers and makes startup much more fragile. ### Use a hash-based batch function instead of `region_id % N` Chosen for the first version. A stable hash spreads skewed or sequential region IDs more evenly while remaining deterministic across stores. ### Adaptive batch sizing Future work. The first version should keep fixed defaults and predictable behavior. ## Risks and mitigations ### Risk: one slow batch delays startup too much Mitigation: - use a bounded batch timeout; - continue to the next batch after timeout. ### Risk: inactive peers still consume periodic tick budget Mitigation: - explicitly exclude inactive peers from tick generation in the Raft worker. ### Risk: readiness becomes misleading Mitigation: - add a startup-completion readiness gate. ### Risk: very large stores still have too-large batches Mitigation: - keep `startup_peer_per_batch` configurable; - use conservative defaults first and tune later based on production data. ## Testing ### Unit tests 1. Threshold behavior: - batching disabled when `local_peer_count <= startup_batch_region_threshold`; - batching enabled when `local_peer_count > startup_batch_region_threshold` and derived `startup_batch_count > 1`. 2. Batch assignment: - peers are assigned by `stable_hash(region_id) % startup_batch_count`. 3. Tick suppression: - inactive peers do not receive periodic startup ticks. 4. Batch progression: - next batch activates on success; - next batch activates on timeout. ### Integration tests 1. Large store restart: - build a test store with more than the threshold number of peers; - verify startup activates batches in deterministic order. 3. Slow-peer case: - inject a peer that cannot quickly establish communication; - verify startup progresses after timeout instead of deadlocking. 4. Readiness: - verify `/ready` does not become healthy before startup batching completes. ## Rollout - Default behavior for stores with `<= startup_batch_region_threshold` peers remains unchanged. - Default behavior for large stores becomes deterministic batched activation. - The configuration should be documented and surfaced in startup logs so operators can confirm when the feature is active. ## Implementation steps 1. Add rfstore config fields: - `startup_peer_per_batch` - `startup_batch_region_threshold` - `startup_batch_connected_threshold_percent` 2. Add store-level startup controller state to track: - whether batching is enabled; - current batch id; - inactive regions; - per-batch connection progress; - batch start time. 3. Keep peer loading and registration unchanged. 4. Replace all-at-once startup activation for large stores with deterministic batch activation. 5. Update Raft worker tick generation to skip inactive peers. 6. Add peer connection-progress signals based on role changes and leader-commit observations. 7. Advance batches on success or timeout. 8. Add a startup completion readiness condition. 9. Add metrics, logs, and tests.