# Cold Region On-Demand Raft Activity ## Purpose Large rfstore deployments can contain many regions without tenant traffic. Running the normal Raft tick for every such region creates background heartbeat and election work that is not useful to the workload. Cold-region mode keeps those peers routable while suspending their periodic Raft activity. The feature is designed for TiDB Cloud deployments with replication factor at least three. It assumes one region covers a whole tenant or keyspace and that an active keyspace periodically advances its shard `max_ts`, for example through `gc_safe_point` writes. The first request after leadership loss may need a retry. No stale read is introduced. ## Peer States `ColdPeerState` is runtime-only: - `Regular` runs the normal peer and Raft ticks. - `TempRegular` is still a cold-data candidate, but retains regular scheduling while a vote lease drains or Raft, apply, merge, snapshot, encryption, leader transfer, or read-index work needs progress. - `Cold` is removed from regular peer ticks and receives only the segmented cold tick. Followers in `TempRegular` do not call `RawNode::tick()`. Leaders call it only while active progress needs heartbeat, replication, or check-quorum work. Otherwise `election_elapsed` advances only to the base election timeout, without generating an automatic campaign. `PreCandidate` and `Candidate` peers are always `Regular` and use normal Raft ticks until the election resolves. A write proposal immediately moves a cold-managed peer to `Regular`. Other state changes are re-evaluated by the split-check path or by a cold-specific message. ## Eligibility A peer can enter `Cold` only when: - cold-region policy is not `disabled`, and its current Raft role is allowed; - the peer is initialized, not inactive in startup batching, and not pending removal; - shard initialization and persistence are complete; - the shard is empty, or covers a whole keyspace and its `max_ts` is older than `cold_region.max_ts_threshold`; - apply and snapshot state is caught up; - no tick-driven progress obligation remains; and - `election_elapsed` has reached the base election timeout. An inactive lagging replica does not keep a leader out of `Cold`. A recently active lagging replica or an in-flight snapshot does, until both replicated and committed progress catch up. Runtime eligibility checks reuse the split-check tick, which already loads the shard and its size. A role change is re-evaluated immediately, so a follower that becomes leader under `enable-follower` returns to `Regular`. When a leader's Region epoch changes, it retains one runtime-only heartbeat retry. A newly split child on the parent leader's Store gets the same marker. The marker is cleared by the next scheduled PD heartbeat or consumed before entering `Cold`, so a lost immediate metadata report does not wait for a cold sweep. Ordinary followers are not marked, and entering `Cold` clears any leftover marker without reporting unless the peer is leader. ## Startup `cloud_server` records a monotonic timestamp before engine recovery and delays Raft peer creation until any pre-restart leader lease has expired. When `PeerMsg::Start` is handled: - an eligible peer without a current vote lease saturates `election_elapsed` and may enter `Cold` immediately; - recent append or heartbeat traffic keeps the peer in `TempRegular`; and - an ineligible peer remains `Regular`. An initialized cold peer satisfies startup-batch connectivity without waiting for a leader connection. Later recovery is driven by its cold tick or by a foreground request. ## Cold Tick The raft worker keeps cold peers in 4096 Region-hashed segments. The configured `cold_region.region_heartbeat_interval` is the nominal time for one sweep. Once a segment interval has elapsed, a worker loop processes exactly the next segment. Delayed loops neither catch up multiple segments nor skip segments, so they extend the current sweep without creating a burst. When a cold peer is ticked, it first rechecks eligibility. ### Leader A cold leader: - sends a normal region heartbeat to PD on each cold tick; and - sends one standard Raft heartbeat round on the configured `cold_region.leader_heartbeat_interval` cadence, defaulting to 30 minutes. The Raft heartbeat cadence is distributed by hashing each Region into a sweep phase. The phase advances only after every segment has been processed, so a delayed sweep stretches the cadence without permanently missing a phase. These Raft heartbeats let valid followers observe the leader before the default stale-peer validation deadline, without resuming normal Raft ticks. ### Follower A cold follower runs the normal stale-peer state check on its cold tick. `LeaderMissing` is expected while a cold leader is quiet, so that intermediate state does not emit the regular warning or metric. `ToValidate` still schedules PD peer validation. The follower also starts a leader-discovery probe when: - it has no cached leader; or - PD has confirmed that the cached leader's Store is down. The probe sends ReadIndex messages to at most two voter peers and waits one randomized election timeout. A matching response restores the leader hint and cancels the task. If the recovery condition still holds at timeout, a promotable voter starts a campaign and becomes `Regular`; a learner validates its membership with PD instead. Normal Raft election retries continue until the campaign resolves. The probe itself has no background retry loop. Expired probe timeouts are consumed only while a cold segment is processed, with work capped by that segment's valid peer count and a minimum of one for sparse sets. ## Foreground Reads A cold follower with a cached leader handles an explicit ReadIndex through the normal follower read-index path. An ordinary Get or Snap with a cached leader returns `NotLeader` with the cached hint and starts one shared leader probe. A matching response cancels the probe; timeout may start one campaign. If the peer has no cached leader, a valid foreground read starts a campaign immediately when the local peer is promotable and returns `NotLeader(region, None)`. Concurrent requests do not restart a campaign already in pre-candidate or candidate state. Pending follower read-index work has a request-local Raft election timeout. When it expires, unresolved reads complete with retryable `StaleCommand` and a promotable voter starts a campaign and becomes `Regular`. The raft worker stores these deadlines in a deduplicated map and min-heap and caps timeout dispatch per receive loop. All request-driven recovery happens after normal store, peer, initialization, snapshot, term, and epoch validation. ## Down-Store Recovery PD Store heartbeat age is the authority for proactive cold-region recovery. Raft client connection events do not change this state. After a successful local Store heartbeat, the PD worker periodically fetches all Stores: - the refresh interval is `min(1 minute, cold_region.store_down_grace_period)`; - at most one refresh is in flight; - heartbeat age is calculated by comparing remote and local Store heartbeat values from the same PD snapshot; and - a Store is confirmed down only when two consecutive snapshots contain the same stale remote heartbeat token beyond the grace period. A Store-level `ColdStoreLiveness` shared by `GlobalContext` and the PD worker publishes the completed observation to peers without coupling it to routing. A fresh heartbeat clears the down state immediately. A failed PD request leaves the last completed state unchanged. When the affected follower's segment is reached, it probes before campaigning. The timeout rechecks the current PD-derived state, so Store recovery cancels stale work. The new leader reports itself through its next cold region heartbeat. Cold leader region heartbeats report only peers whose Stores are confirmed down by this mechanism. Quiet, healthy cold followers are not reported as down. A failure limited to the Raft service or Store-to-Store network does not trigger proactive election while the Store continues heartbeating to PD. Foreground reads still recover on demand, and stale-peer validation remains available. ## Scheduling and Memory Bounds - Cold peers remain in the main peer map for routing and metadata lookup. - Cold peers are absent from the regular tick set and present in one cold segment. - The regular tick set, cold ticker, and idle worker exist under every policy. `Regular` and `TempRegular` peers may run on the idle worker; `Cold` peers run outside the idle worker and are scheduled by the main worker. - If an idle peer becomes `Cold`, the idle worker hands that peer back to the main worker through the normal single-peer wake-up path. - Startup or bulk transitions into `Cold` do not enqueue an immediate tick. - Probe tasks and read-index timeouts are deduplicated per Region. - Deadline heaps are rebuilt when stale entries become disproportionate to live tasks. ## Configuration and Rollout - `cold_region.policy`: `disabled` permits no peer to enter `Cold`; `enable-follower` permits only followers; `enabled` permits both leaders and followers. - `cold_region.max_ts_threshold`: minimum shard `max_ts` age; default 24 hours. - `cold_region.store_down_grace_period`: minimum PD Store heartbeat age for down-Store recovery; default 10 minutes. - `cold_region.region_heartbeat_interval`: nominal cold sweep and region heartbeat interval; default 5 minutes. - `cold_region.leader_heartbeat_interval`: nominal Raft heartbeat interval for cold leaders; default 30 minutes. `split_region_check_tick_interval` must be nonzero because it drives runtime re-evaluation into `Cold`. `region_heartbeat_interval` must not be shorter than `raft_base_tick_interval`. When leaders may enter `Cold`, the leader heartbeat interval rounded up to a whole sweep must remain shorter than `max_leader_missing_duration`. Online activation must follow this order: 1. Change every Store from `disabled` to `enable-follower`. This does not scan or wake existing peers in bulk. Idle peers observe the policy through their normal ticks, while leaders remain `Regular` and keep followers active. 2. After every Store is on `enable-follower`, change the policy online to `enabled`. Skipping directly from `disabled` to `enabled` and changing from `enable-follower` to `disabled` online are rejected. `enabled` can be reduced online to `enable-follower` before a binary rollback. The rollback procedure must leave more than ten minutes between successful policy propagation and binary replacement; this is an operational settling period rather than an in-process drain barrier. Changing the other cold-region fields online is rejected. The same policy values can also be set at startup. The policy controls Cold eligibility only; the existing `peer_idle_duration` and idle-worker behavior remain available under every policy. ## Metrics - `rfstore_cold_regions_count`: current local peers in `Cold`. - `rfstore_cold_region_state_transitions_total{state}`: transitions into `regular`, `temp_regular`, or `cold`. - `rfstore_cold_leader_probe_pending_count`: live one-shot leader probes. - `rfstore_cold_campaign_events_total{reason}`: campaign attempts caused by `read_index`, `leader_unknown`, or `store_down`.