# cloud_server Maintainer Guide This document is for maintainers of the `components/cloud_server` crate. It focuses on design intent, invariants, and failure behavior. It intentionally skips routine API details that are easy to read in code. When you need exact struct fields, read the source under `components/cloud_server/src`. This guide assumes you already understand distributed storage systems and Raft at a high level. It explains how cloud_server wires the cloud engine stack together. The goal is to make it safe to change, debug, and extend this component. ## Quick index - Scope and mental model - Component boundaries and responsibility split - High level architecture diagram - Project map and reading order - Threading model and workers - Startup lifecycle (prepare -> setup -> run) - Identity, StoreIdent, and bootstrap - Engine initialization and DFS integration - Flow control and resource control - RaftKv engine wrapper - Custom raft log encoding and txn semantics - Async write and snapshot flows - Replica read lock check and read index handling - Raft transport overview - RaftClient queueing, batching, and backoff - Address resolver and store cache - gRPC server lifecycle - gRPC service patterns (unary, streaming, batch) - Request batching heuristics - Import SST service flow - Backup and diagnostics services - Status server architecture - Status endpoints (by family, not exhaustive) - Status server security gating - Recovery, blacklist, and panic region handling - Observability and metrics - Failure modes and invariants - Testing and harnesses - Debugging playbook - Glossary - References ## Scope and mental model cloud_server is the server runtime for the Cloud Storage Engine stack. It is the glue that wires the storage engine, Raft store, and RPC services. It owns the startup sequence, gRPC server, status server, and transport. It does not own Raft state machines or storage engines themselves. Think of cloud_server as the process coordinator. It configures and starts rfstore (Raft layer) and kvengine (data layer). It exposes RPC endpoints used by TiDB, client libraries, and internal workers. It also exposes administrative HTTP endpoints via the status server. What this guide covers. - How cloud_server initializes, owns, and shuts down core components. - How it wires Raft transport, address resolution, and gRPC services. - How async write, snapshot, and custom raft log flows work together. - Why specific ordering and gating choices exist in startup and shutdown. What this guide intentionally skips. - Exhaustive RPC method lists and request/response structs. - Routine config fields whose meaning is obvious from code. - Implementation details inside kvengine, rfengine, and rfstore. If you need deeper details. - rfstore behavior: `components/rfstore/README.md`. - kvengine behavior: `components/kvengine/README.md`. - rfengine behavior: `components/rfengine/README.md`. - txn file feature: `doc/features/txn_file.md`. ## Component boundaries and responsibility split cloud_server sits between external callers and the storage stack. Its responsibilities are process and network facing. Its dependencies implement the storage semantics. Boundary summary. - rfstore owns Raft groups and apply logic. - kvengine owns user data and shard metadata. - rfengine owns Raft log persistence. - cloud_server owns lifecycle, network, and service wiring. Why this separation matters. - It keeps Raft and storage correctness logic in dedicated crates. - It keeps server orchestration changes localized and testable. - It allows alternative server front ends without rewriting rfstore. ## High level architecture cloud_server is the component that binds together IO, Raft, and RPC. It also maintains a control plane (status server) separate from data plane. Simplified diagram. ```ascii +-------------------+ +-------------------+ | RPC clients | | PD | | (TiDB, tools, | | metadata + ts | | tikv-worker) | +---------+---------+ +---------+---------+ ^ | | v | +---------+----------------------------+---------+ | cloud_server | | - TikvServer lifecycle | | - gRPC server + services | | - Status server (HTTP) | | - Raft transport (RaftClient) | +---------+----------------------------+---------+ | | v v +---------+---------+ +---------+---------+ | rfstore | | kvengine | | Raft groups | | data + shards | +---------+---------+ +---------+---------+ | | v v rfengine DFS / local ``` Control plane vs data plane. - Data plane: gRPC Tikv service, Raft transport, storage RPCs. - Control plane: status server, config management, recovery endpoints. ## Project map and reading order Start here if you are new to cloud_server. This order mirrors the startup path and request flow. Core lifecycle and wiring. - `components/cloud_server/src/tikv_server.rs`. - `components/cloud_server/src/setup.rs`. - `components/cloud_server/src/node.rs`. RPC and server side surface. - `components/cloud_server/src/server.rs`. - `components/cloud_server/src/service/kv.rs`. - `components/cloud_server/src/service/batch.rs`. - `components/cloud_server/src/service/sst_service.rs`. - `components/cloud_server/src/service/diagnostics/mod.rs`. Raft transport and address resolution. - `components/cloud_server/src/raft_client.rs`. - `components/cloud_server/src/transport.rs`. - `components/cloud_server/src/resolve.rs`. Engine wrapper and request encoding. - `components/cloud_server/src/raftkv.rs`. Status server. - `components/cloud_server/src/status_server/mod.rs`. - `components/cloud_server/src/status_server/metrics.rs`. Heap profile utilities (shared component used by cloud_server and cloud_worker). - `components/memory_profile/src/lib.rs`. Entry points. - `cmd/tikv-server/src/main.rs` uses `cloud_server::TikvServer`. Related docs worth keeping open. - `components/rfstore/README.md` for Raft store internals. - `components/kvengine/README.md` for shard and file layout. - `doc/features/txn_file.md` for txn file write flow details. ## Threading model and workers cloud_server uses multiple thread pools and worker types. These are configured to isolate workloads and avoid starvation. Thread pools used by cloud_server. - gRPC worker threads (grpcio env, sized by cpu_cores quota and grpc_concurrency_factor). - Read pools (unified read pool, tokio or yatp based). - Scheduler worker pool, which defaults to a dedicated pool and only reuses the unified read pool when `storage.use-separated-scheduler-pool` is false. - Debug thread pool (used by diagnostics). - Status server runtime thread pool (HTTP server). - Import SST runtime (per importer). - Background worker (general purpose). Additional runtime loops. - stats_pool for gRPC thread load and memory sampling. - global timer tasks used by load statistics and sampling intervals. Why stats_pool exists. - The gRPC server itself should avoid expensive periodic work. - Load statistics and memory sampling are isolated from request paths. Worker loops and schedulers. - PD worker (for PD tasks from rfstore). - Address resolver worker (resolve store address via PD). - Resource metering workers (recorder, reporter, single target). Read pool selection. - Unified read pool can run on tokio or yatp based on config. - The choice is made during init_servers and affects storage reads. - `storage.enable-get-in-read-pool` controls whether transactional point-get storage work executes as one task in the unified read pool. It can be changed online. Keyspace fairness takes precedence and always sends that task through the keyspace fair queue; gRPC request decoding and response encoding remain on the gRPC side. Design rationale. - Keep admin and debug work off the data plane. - Avoid blocking gRPC threads with heavy IO. - Preserve predictable latency under high load. Key invariants. - gRPC services are registered before server start. - Workers that depend on engines are started after engines open. - Background workers are stopped before engines are closed. ## Startup lifecycle Startup in cloud_server is deliberate and sequential. Many components depend on earlier initialization steps. The startup is split across `prepare`, `setup`, and `run`. ### Phase 1: prepare `TikvServer::prepare` does early environment and connection setup. It runs before full config validation and before engines are opened. Key reasons for this ordering. - Logging must be configured before anything else can fail. - PD connection is needed to derive cluster id and feature gate. - DFS selection determines engine options and encryption setup. Key steps. - Initialize logging (`setup::initial_logger`). - Print build version and resource quota. - Check critical environment variables and set panic hooks. - Build `SecurityManager` for TLS and certificate validation. - Build gRPC environment with thread count from CPU quota. - Connect to PD and fetch the cluster id. - Choose DFS implementation based on config. DFS selection logic (simplified). - If S3 bucket and endpoint are empty, use builtin DFS. - If endpoint is `memory`, use in-memory DFS for tests. - Else use S3 DFS with configured endpoint and credentials. Important design note. - DFS choice happens before engines open because it affects engine paths. - The choice also controls IA defaults and remote compactor behavior. ### Phase 2: setup `TikvServer::setup` constructs the server object and opens engines. It does config validation and initializes resource controllers. Key steps. - Validate config and persist it (`init_config`). - Probe the effective advertised store and status endpoints before opening engines. Hostname resolution waits up to the shared network-inspection threshold per endpoint. A timeout is advisory and does not cancel the underlying system resolver thread. - Initialize flow control and store limiter. - Build IO rate limiter and set the IO budget. - Create master key for encryption from security config. - For either active cold-region policy, record the lease guard timestamp before engine recovery and wait out any remainder before creating `RaftBatchSystem`. - Open rfengine and kvengine (see Engine initialization). - Create the rfstore `RaftBatchSystem` and router. - Create background workers (worker pool, resolver worker). - Initialize concurrency manager using PD TSO. - Initialize resource controller. - Build the startup-fixed legacy admission runtime. Quota and overload objects are constructed only when read-pool keyspace fairness is disabled. - Lock in storage-size metric label schemas via `init_store_size_metric` and `init_keyspace_size_metric` before any emit. Design rationale. - Config validation happens before engine open to prevent wrong path usage. - Flow control and rate limiting must be ready before storage starts. - Concurrency manager must use a recent PD TSO for lock correctness. ### Phase 3: run `TikvServer::run` wires everything and starts serving. It should be called only after `setup` has completed. Key steps in run order. - Register memory usage high water thresholds. - Check for conflicting addresses in lock dir. - Acquire file locks and check for panic mark files. - Initialize Yatp metrics (one time global). - Build `RaftKv` wrapper and local reader. - Build servers, storage, and Raft node (`init_servers`). - Register gRPC services (`register_services`). - Start metrics flushing background task. - Bind and start the gRPC server. - Start status server if enabled. - Start memory profile auto-dump observer. - Start safe point watchers and metrics push (if configured). - Start the resource controller. Design rationale. - gRPC services must be registered before the server is built and started. - Status server is intentionally separate from gRPC server for isolation. - The run order ensures the Raft node is started before exposing RPCs. ### Shutdown `TikvServer::stop` and `TikvServer::force_stop` drive shutdown. They ensure that services stop before engines are closed. Key shutdown steps. - Stop gRPC server and health service. - Stop rfstore node and region info accessor. - Stop lock manager, background workers, and status server. - Close rfengine writer and stop its worker. - Stop the legacy overload protector worker when it exists. - Stop resource controller. - Release lock files (best effort). Important invariant. - Never close engines before stopping components that may access them. ## Config controller and online config cloud_server uses ConfigController to manage runtime configuration changes. It is created during init_config and passed to status server and modules. Registration highlights. - Resource metering config manager updates recorder and reporter behavior. - Quota and overload config managers are registered only when read-pool keyspace fairness was disabled at startup. - Resource control config manager updates resource controller policy. - Backup endpoint config manager allows runtime tuning of backup settings. Why this matters. - Cloud workloads require safe online tuning without restarts. - The status server exposes config updates and validation. ## Identity, StoreIdent, and bootstrap cloud_server owns store identity and bootstrap decisions. This logic lives in `node.rs` and is invoked during `init_servers`. ### Store identity Each store is identified by a store id assigned by PD. On disk, the store id and cluster id are stored in `StoreIdent`. When starting, cloud_server checks for an existing StoreIdent. Key behaviors. - If the cluster id in StoreIdent mismatches PD, startup fails. - If StoreIdent is missing, a new store id is allocated. - The store id is also set on kvengine and rfengine. ### Cluster bootstrap If the cluster is not bootstrapped, the first store may bootstrap it. The sequence is explicitly guarded and retried. Bootstrap flow (simplified). 1. Check if a prepared bootstrap region exists on disk. 2. Ask PD if the cluster is bootstrapped. 3. If not bootstrapped, allocate region id and peer id. 4. Prepare bootstrap state in engines. 5. Try PD bootstrap. 6. Clear prepared state on success or if PD has bootstrapped already. Why this design. - Prevents multiple stores bootstrapping simultaneously. - Allows idempotent restart if the process dies mid bootstrap. - The prepared bootstrap state survives restarts. ### Store start Once identity is established, the Raft store is started. `Node::start` spawns rfstore workers via `RaftBatchSystem`. It also registers the store to PD and loads existing store list. Important invariants. - `Node::start` must run after store id and cluster id are set. - The router and transports must exist before rfstore starts. ## Server wiring details `init_servers` is where the core runtime wiring happens. It builds storage, coprocessor, lock manager, and the gRPC server. Key wiring steps. - Create LockManager and register deadlock observer hooks. - Build unified read pool (tokio or yatp). - Create debug thread pool shared by diagnostics service. - Initialize resource metering recorder, reporter, and sink workers. - Register ConfigController modules for resource metering and the startup-allowed legacy admission mechanisms. - Create Storage backed by RaftKv and ConcurrencyManager. - Register ReplicaReadLockChecker in CoprocessorHost. - Create coprocessor Endpoint and configure remote execution settings. - Create Server with gRPC environment, transport, and read pool. - Initialize SstImporter and set compression types for CFs. - Start Node with transports, store meta, and importer. - Start CDC after Node. Before requesting a snapshot, each incremental scan records a deadline equal to two thirds of the startup value of `raftstore.local-file-gc-timeout`. The scan checks it before its initial snapshot access and before each delta-scan batch; an expired scan truncates pending events and returns `Congested` instead of reading another batch. Resolver lock scans are covered by the initial check and are expected to be short. Compaction refreshes the mtime of retired local files, so the deadline leaves one third of the configured lifetime for an in-flight batch. CDC does not track online changes to the GC timeout; rfstore preserves the same safety bound by rejecting online decreases below the process startup value. Why this structure. - LockManager and concurrency manager must be in place before storage. - The coprocessor endpoint needs the read pool and resource control. - Install the coprocessor resource controller before building its remote context, because load-aware remote dispatch captures the controller's total CPU monitor at construction. - Importer and backup services depend on storage and router. Resource metering detail. - Recorder and reporter workers are started and stored for shutdown. - A single target worker sends reports to a configured receiver. - ConfigController updates metering behavior online. Why this matters. - Metering is used for billing and workload attribution in cloud deployments. - It must be isolated from latency critical paths. ## Engine initialization and DFS integration cloud_server opens and configures both rfengine and kvengine. This is where cloud-specific storage behavior is bound in. ### rfengine rfengine is opened by `init_raft_engine`. It is given the raftdb path, data dir, and optional DFS handle. Why DFS matters for rfengine. - rfengine can place or access raft WAL chunks on DFS. - The DFS handle must be available to avoid corrupting WAL layout. ### kvengine kvengine is opened by `init_kv_engine`. It needs DFS, multiple local directories, and config options. Key initialization steps. - Compute block cache capacity from total memory or config. - Ensure all local directories exist (main + extra dirs). - Apply RocksDB-derived settings (memtable, block size, compression). - Enable flow control, IA, and columnar settings. - Configure remote compactor addresses and concurrency limits. - Choose txn file worker pool size based on memory. - Build the PD id allocator and meta change listener. - Pass recovery handler and blacklist to kvengine. Important constraints. - `max_mem_table_size` is capped to kvengine maximum. - Flow control thresholds are derived from config and must be consistent. - Some options are intentionally dynamic (IA capacity is always dynamic). ### DFS selection DFS selection is done in `prepare`. DFS controls where data files are stored and how reads are performed. Paths and semantics. - builtin DFS uses PD-backed metadata for local operations. - S3 DFS uses an object store for file persistence. - In-memory DFS is only for tests and does not persist data. ### IO rate limiter and store limiter Two layers of IO control are initialized. - `IoRateLimiter` enforces global bytes per second limit. - `StoreLimiter` enforces storage flow control (write throttling). Why two layers. - The IO rate limiter is a global budget shared across operations. - The store limiter is a dynamic throttle based on memory usage feedback. IO metrics flusher. - IO stats are collected from either the OS collector or the rate limiter. - A background task periodically flushes IO metrics to Prometheus. ### Kubernetes disk capacity check On Kubernetes, startup checks disk capacity for the data dir. If capacity is too low, startup fails fast. Why this exists. - Prevents booting a pod before volumes are properly mounted. - Avoids silent data loss from running on ephemeral filesystem. ### Panic regions and blacklist escalation cloud_server uses panic region files to build a blacklist. This is a recovery safety net after repeated panics. Key inputs. - Panic region files are discovered in the data directory. - Region to peer mapping is loaded from rfengine. - Snapshot metadata gives keyspace and table id. Escalation logic. - Count panics per table and keyspace. - If counts exceed thresholds, add to blacklist. - Merge with blacklist config files and recovery blacklist. - Apply whitelist overrides (whitelist wins). Design rationale. - Prevents repeatedly crashing regions from being reopened immediately. - Escalation to table and keyspace avoids many individual region entries. - The logic is data driven and resilient to partial data loss. ## Flow control and resource control Several subsystems regulate workload and resource usage. These are initialized in `setup` and registered in config controller. ### Flow control Flow control uses a `StoreLimiter` and `FlowController`. It ties memory usage to write throttling. Key design points. - The max speed is derived from hard limit and PD heartbeat interval. - The min speed is clamped to `CLOUD_MIN_THROTTLE_SPEED`. - Flow control is enabled via storage config. ### Resource controller `ResourceController` coordinates resource control subscriptions. It registers subscribers for read and transfer leader tasks. The legacy Starter read limiter and the newer RC mechanisms share the resource-controller umbrella but have separate runtime switches: | Deployment posture | Legacy scan limiter | CPU-token throttle | Read-pool fairness | | --- | --- | --- | --- | | Starter | `resource-control.read-enabled = true` | `resource-control.cpu-throttle.enabled = false` | `readpool.unified.keyspace-fairness.enabled = false` | | RC-enabled tier | `resource-control.read-enabled = false` | rollout-controlled | rollout-controlled | `enable-qps-bytes-limiter` controls adaptive QPS/bytes calculation only after the legacy `read-enabled` path is active. These are runtime configuration boundaries, not product-tier Cargo features. Keep the postures mutually exclusive in deployment configuration so the legacy per-scan waits cannot stack with RC CPU admission or fair scheduling. Config validation warns when `resource-control.read-enabled = true` and `resource-control.cpu-throttle.enabled = true` are both set, but preserves the pre-fairness behavior of accepting the configuration. `LegacyAdmissionRuntime` is the hard boundary for the older global quota and transaction overload protector. With keyspace fairness disabled, it restores the historical behavior: Storage receives the configured quota limiter, Coprocessor receives its separate default limiter, and an overload worker plus both online config managers are available. The quota defaults to unlimited and overload protection defaults to disabled. With keyspace fairness enabled, the runtime keeps all three request-path objects absent, starts no overload worker, and registers neither legacy config manager. Non-default quota or enabled overload settings are ignored with a warning. This construction-time `None` boundary is deliberate: an online config update cannot reactivate observation, delay, metrics, task-stat reporting, or rejection in the fair path. Remote cloud workers likewise enter the restored coprocessor code with no legacy quota object. The legacy quota is instance-wide rather than per-keyspace, and overload state is keyed by transaction start timestamp. Neither mechanism creates persistent per-keyspace state or metric labels on the million-keyspace default path. Where it matters. - The unified read pool uses read limiter subscriptions. - Transfer leader operations are throttled to avoid cluster churn. - CPU throttling separates burst capacity (`burst-window`) from refill cadence (`refill-interval`). Both settings are restart-only. Fair-queue dispatch, legacy request admission, and runtime token allocation retry at the exact absolute deadline of the bucket that blocked them. - With keyspace fairness disabled, `resource-control.max-read-cpu-ratio` remains online-adjustable for the legacy read limiter, including the historical CPU-throttle posture; it does not resize an existing global token bucket. With fairness enabled, it is the startup bucket anchor and changes require restart. Use the fairness-specific ratio for online fair-path adjustment. - `resource-control.max-read-wait-time` preserves the historical `50ms` startup limit. An online update changes the legacy scan limiter afterward; the setting has no effect while keyspace fairness is enabled. - The `100ms` burst-window default preserves the effective bucket capacity from before this separation. Raising it to `300ms` or `500ms` would allow three or five times as many cached tokens and therefore requires explicit burst/cap validation instead of being treated as a refill-cadence optimization. - `enable-fair-allocation` is a legacy-throttle burst-share guard. It is not the read-pool keyspace fairness switch. ### Coprocessor concurrency compatibility `server.end-point-max-concurrency` retains the pre-fairness concurrency bound for the regular YATP read pool. Unary coprocessor work may run for up to 5ms without a permit; work that remains active after that must hold one of the configured permits. Streaming work acquires a permit before execution. The default permit count is the larger of the available CPU count and four. When keyspace fairness is enabled, the endpoint does not install this process-global semaphore. The fair YATP path instead uses `readpool.unified.keyspace-fairness.max-tasks-per-keyspace`, so a global waiter cannot sit in front of keyspace selection. Tokio read pools keep their existing behavior and do not install the semaphore. ### Concurrency manager and lock tracking ConcurrencyManager is initialized using PD TSO. It is used by RaftKv, replica read lock checking, and backup flows. Key usage points. - Update max ts on read index paths to enforce monotonicity. - Track backup_ts to coordinate snapshot and backup safety. - Provide lock checks for replica reads via ReadIndexObserver. ## RaftKv engine wrapper `RaftKv` is the engine wrapper exposed to the Storage layer. It bridges storage requests to rfstore via Raft commands. ### Purpose - Convert Storage write and read calls into Raft commands. - Provide async write and snapshot APIs for Storage. - Track write progress and provide proposed and committed events. ### Async write flow `async_write` is the central write path for KV operations. It builds a `RaftCmdRequest` and sends it to rfstore. It returns a stream of `WriteEvent` values. Key stages. 1. Validate request (non empty writes, failpoints). 2. Convert `WriteData` into a custom raft log request. 3. Build request header and set flags (one_pc, stale read, etc). 4. Send to rfstore with a callback that completes the WriteRes stream. 5. Update metrics based on result and duration. Why a stream. - Some callers want proposed and committed events before apply. - The stream model lets them subscribe to those events efficiently. Safety handling. - If the on_applied callback is dropped, the write becomes undetermined. - The callback is wrapped with `must_call` to enforce completion. ### Async snapshot flow `async_snapshot` issues a Raft `Snap` command. It returns a future that resolves to a `RegionSnapshot`. Key behaviors. - The request may include key ranges and a read index start_ts. - Stale reads set a special flag and include start_ts in flag_data. - The response may contain either a snapshot or an error response. Why it is async. - Snapshot acquisition can be slow and should not block threads. - It also allows cancellation and integrates with async scheduling. ### Custom raft log encoding `modifies_to_requests` encodes logical txn operations into a custom log. This is a key cloud engine optimization. Design goals. - Shrink Raft log size by storing only essential metadata. - Avoid repeated value payloads when they can be fetched on apply. - Preserve transactional semantics and rollback safety. Key cases (conceptual). - Prewrite merges default CF values into lock values when possible. - Commit stores key + commit_ts and applies the value on apply. - OnePc handles put, del, and lock in a single log entry. - Rollback and CheckTxnStatus encode rollback semantics compactly. - ResolveLock may contain Commit and Rollback sub-entries in one log. This mixed shape is triggered by kv_commit batches that also need stale pessimistic-lock cleanup, and intentionally reuses the existing persisted format to avoid introducing a new raft-log format during rolling upgrades. - Shared-lock holder updates carry one batch-level provenance bit in `WriteData.extra.shared_lock_lifecycle`. Ordinary requests keep the old fast path unchanged. For request types whose compact encoding cannot represent shared-lock holder updates, a request containing shared-lock lifecycle work is emitted as `SharedLockModifies` instead of trying to split the batch per key. Pessimistic-lock acquire can use the same bit to avoid in-memory lock writes while still using the normal pessimistic-lock custom log. Why this is complex. - The same set of Modify operations can represent different txn stages. - Some cases depend on whether one_pc is set or not. - Correctness depends on matching the exact txn command type. Common pitfalls. - Prewrite must merge default CF values into lock values correctly. - OnePc must detect deleted locks to preserve semantics. - Rollback logic must avoid conflicting rollback and commit records. - Shared-lock commits, rollbacks, and resolve-lock batches cannot assume the final lock action is delete-only. If other holders remain, the custom log must preserve the updated shared-lock bytes so rfstore apply can write them back. - `SharedLockModifies` encodes write puts before lock updates. Some write-put replay paths consult the current lock bytes, so lock updates must not overwrite or delete those bytes before the write record is applied. - Mixed batches are intentional in the current design. A single logical request can contain an ordinary primary plus shared secondary keys. Once the batch is marked as shared-lock lifecycle, the whole batch goes through `SharedLockModifies`, and rfstore apply must preserve the existing normal commit behavior for any ordinary `Put` or `Delete` entries replayed there. ### Txn file integration When `WriteData` contains `txn_file`, `modifies_to_requests` stores a ref. The ref is carried in the custom request to rfstore. This integrates with the txn file feature described in `doc/features/txn_file.md`. Design note. - The custom request carries a `TxnFileRef`, not the file content. - The actual file data is stored in DFS and loaded by kvengine. - cloud_server also starts a dedicated `txn-file-lock-resolve-worker` for maintenance-tick-triggered active resolution of old txn-file locks. - That worker owns a small multi-thread tokio runtime plus a single shared `tikv_client::TransactionClient` cache built lazily in `api_v2_no_prefix` mode from PD endpoints plus `SecurityConfig`. - A shared `TxnFileLockResolveLimiter` gates how many started resolve tasks may run concurrently. - A separate admission semaphore, sized to the resolver worker's pending capacity, bounds how many spawned attempts may live in the runtime at once. If admission is full, the task is dropped before start, the region returns to `Ready`, and a distinct load-shedding metric is recorded. - rfstore now keeps only per-region `Inflight` scheduler state. cloud_server receives tasks only after rfstore has already done the authoritative per-region schedule decision. - The resolver creates an attempt guard before `runtime.spawn` and keeps that guard alive for the entire async attempt. - If startup, admission overflow, client initialization, limiter acquisition, or runtime shutdown prevents the future from completing normally, dropping the guard returns the region to `Ready` immediately. - There is no post-attempt cooldown; retries are paced by later maintenance ticks plus the existing worker/admission/concurrency bounds. - It resolves locks through the normal online `resolve_locks` path, so region routing comes from each lock key instead of the original region id. ### Replica read lock check `ReplicaReadLockChecker` is registered as a read index observer. It checks for lock conflicts during read index processing. Key behavior. - It runs only on leaders and only for MsgReadIndex. - It parses the read index context and checks key ranges. - It updates ReadIndexContext with lock info if conflicts exist. Why this matters. - Replica reads can bypass leader lease checks. - This provides a safety guard to ensure a lock conflict is visible. ### Write path sequence (simplified) This is a high level view of a write request through cloud_server. It omits details inside rfstore and kvengine. Sequence. 1. gRPC KvService receives a write request. 2. Storage builds WriteData and calls RaftKv::async_write. 3. RaftKv encodes WriteData into a custom raft log request. 4. rfstore proposes the Raft command and replicates it. 5. rfengine persists the log entry. 6. rfstore apply workers apply to kvengine. 7. RaftKv callback completes the WriteEvent stream. Why the sequence matters. - The custom log encoding must align with apply logic in rfstore. - The callback is the only signal that the write is durable and applied. ### Read path sequence (simplified) This is a high level view of a read request. It focuses on linearizable reads and replica reads. Sequence. 1. gRPC KvService receives a read request. 2. Storage chooses a read path and calls RaftKv::async_snapshot. 3. rfstore processes the read index request if needed. 4. ReplicaReadLockChecker injects lock info when leader handles read index. 5. RaftKv returns a RegionSnapshot or error. Why the sequence matters. - Read index contexts carry lock info that affects correctness. - Replica reads must be checked against lock conflicts explicitly. ## Raft transport and address resolution cloud_server provides rfstore with a transport implementation. This transport is implemented by `RaftClient` and `ServerTransport`. ### ServerTransport `ServerTransport` is a thin adapter. It forwards `send`, `need_flush`, and `flush` to `RaftClient`. rfstore owns a boxed Transport and calls these methods. Two transports are created. - `trans` for normal traffic. - `trans_idle` for idle or lower priority traffic. The separation helps rfstore differentiate workloads. It also provides independent RaftClient instances for scheduling. ### Address resolver `PdStoreAddrResolver` resolves store ids to addresses using PD. It runs in a dedicated worker and caches results by TTL. Key behaviors. - Uses PD get_store and handles tombstone stores explicitly. - Caches address for `STORE_ADDRESS_REFRESH_SECONDS`. - Returns error for empty address (tests may set it empty). Why caching exists. - Raft message transport is hot path and must avoid PD hammering. - Store address changes are infrequent compared to Raft traffic. Resolver execution model. - Each resolve call enqueues a Task onto a worker Scheduler. - The callback is executed on the worker thread and returns a Result. ### RaftClient architecture RaftClient manages gRPC streams per (store_id, conn_id). It has a connection pool, a local LRU cache, and a queue per stream. Key data structures. - Queue holds outbound RaftMessage instances. - ConnectionPool tracks active connections and tombstone stores. - CachedQueue tracks dirtiness and full state for flushing. - BatchMessageBuffer batches raft messages into BatchRaftMessage. Why this architecture. - Raft messages are frequent and must be batched. - gRPC streams are long lived and need reconnection logic. - A small cache avoids locking the connection pool on every send. Connection sharding. - The target connection id is derived from region id and grpc_raft_conn_num. - This spreads regions across multiple streams to reduce head of line blocking. Dynamic config tracking. - BatchMessageBuffer uses a VersionTrack tracker to refresh limits. - This allows adjusting batching behavior without restart. ### Connection lifecycle Connection lifecycle is managed by `start` in `raft_client.rs`. It is a loop that retries on failure with backoff. Lifecycle sequence (simplified). 1. Resolve store address (PD resolver). 2. Connect gRPC channel to the address. 3. Start batch_raft stream. 4. If unsupported, fallback to legacy raft stream. 5. Wait until stream ends, then retry. Failure handling. - Resolve failure clears pending messages and increments metrics. - Tombstone errors remove the connection and mark store as tombstoned. - Connection timeouts broadcast unreachable to rfstore. - Cold-region store-down recovery does not consume connection reachability; it uses PD Store heartbeat metadata instead. - Disconnection triggers backoff and stream rebuild. Why fallback exists. - Mixed version clusters may not support batch_raft. - Fallback keeps compatibility with older nodes. Connection config notes. - ChannelBuilder applies keepalive, compression, and window sizes. - Reconnect backoff is bounded by raft_client_* config. - A random channel arg forces new connections on rebuild. ### Queue and flush semantics `RaftClient::send` enqueues messages into a queue. It does not send immediately to gRPC. The caller must call `flush` to notify queues. Why explicit flush. - Reduces wakeups and allows higher batching efficiency. - Lets rfstore send many messages and then flush once. Queue states. - Established: accepts messages. - Paused: rejects messages with DiscardReason::Paused. - Disconnected: rejects messages and triggers reconnection. When queue becomes full. - `send` returns DiscardReason::Full. - The queue is notified so the sender drains it. - Metrics are incremented for observability. ### Store allowlist `RaftClient` can pause traffic to stores not in allowlist. This is used to limit traffic in specific scenarios. Behavior summary. - If allowlist is empty, all stores are allowed. - If allowlist is non empty, others are paused. - Paused stores return DiscardReason::Paused on send. ### Raft message send sequence (simplified) This illustrates the interaction between rfstore and RaftClient. Sequence. 1. rfstore calls Transport::send for each RaftMessage. 2. RaftClient enqueues into Queue and marks it dirty. 3. rfstore calls Transport::flush after batching. 4. RaftClient notifies queues to wake the AsyncRaftSender. 5. AsyncRaftSender batches and flushes to gRPC stream. Why this matters. - Flush boundaries define batching efficiency. - Queue capacity and pause state determine backpressure behavior. ## gRPC server lifecycle `Server` encapsulates gRPC server creation and lifecycle. It owns the gRPC environment, server builder, and load stats. ### Build vs start `build_and_bind` creates the gRPC server and binds to a port. `start` starts serving and initializes load tracking. Why two phases. - Services must be registered before the server is built. - Tests may use port 0 and need the resolved address. Registration behavior. - register_service returns the service if called after start. - This prevents late registration from silently being ignored. ### gRPC configuration Several settings are tuned from config. Examples include. - Stream window size and message size limits. - Memory quota for gRPC internal buffers. - Keepalive settings and compression level. Design rationale. - Cloud workloads can have large messages (SST, backup). - Keepalive tuning avoids idle connection drops. ### Load and memory metrics The server tracks load on gRPC threads. It periodically records thread load and memory usage. Why this exists. - Load spikes must be observed early for stability. - Memory usage is sampled to support alerting and auto-tuning. ### Health service A gRPC health service is registered and updated. It reports Serving only after the server is fully started. This avoids early routing to an unready node. ## gRPC service patterns The `service` module implements the TiKV-compatible gRPC services. It relies on Storage, coprocessor endpoint, and raft router. ### Proxy forwarding KvService uses Proxy to optionally forward RPCs. Forwarding is invoked at the start of each RPC. It enables flexible routing without changing service code. Design note. - Forwarding is a cross cutting concern handled before request work. - It keeps service logic focused on request execution. ### Unary request pattern Most KV RPCs use a standard pattern. - Convert request into a future (for example, future_get). - Await and send response on sink. - Record metrics and error counters. Why this matters. - Ensures consistent latency and failure metrics. - Minimizes per method boilerplate using macros. Thread load tracking. - KvService receives a ThreadLoadPool used to record gRPC thread load. - The load signal is used to report system pressure and guide batching. ### Streaming request pattern Streaming RPCs (coprocessor_stream, batch_commands, raft) use streams. They use async tasks and channels to decouple request handling. Design rationale. - Avoid blocking gRPC threads on heavy processing. - Provide backpressure via stream consumption. ### Batch commands flow batch_commands is a duplex stream for heterogeneous requests. It accepts a stream of requests and returns a stream of responses. Key behaviors. - Request handlers are spawned on the tokio runtime. - Responses are multiplexed by request id. - A batcher may coalesce get requests into batch_get. Why this exists. - Reduces RPC overhead for high throughput workloads. - Enables mixed request types in a single stream. ### Request batching heuristics ReqBatcher coalesces get requests under certain conditions. It is intentionally conservative to avoid latency spikes. Heuristics used. - Only batch normal priority GETs. - Batch size is capped by MAX_BATCH_GET_REQUEST_COUNT. - Batching depends on pool size and queue size. - Batches are always partitioned by keyspace. While load-based split is enabled, they are also partitioned by Region so one shared read-pool future has one Region CPU owner. The resulting resource tag carries every point key in that Region batch and charges the future CPU once. Design rationale. - Batch GET only helps when enough requests are present. - Over batching can harm tail latency and fairness. - Keeping cross-Region batching when load-based split is disabled preserves the existing snapshot batching behavior. ### Coprocessor requests Coprocessor requests are handled by the Endpoint. The endpoint can be configured for remote execution. Why remote coprocessor. - Cloud deployments may offload heavy scans to remote workers. - The endpoint supports remote URLs and minimum thresholds. ### Raft and batch_raft RPCs KvService handles both raft and batch_raft streams. These are used by RaftClient and other nodes for replication. Key points. - batch_raft accepts BatchRaftMessage and splits them. - Both streams forward messages to rfstore router. - Observability counters track message counts and failures. ### Service registration order Service registration happens in register_services. The order matters because some services depend on others being ready. Key registrations. - ImportSST service and its runtime handle. - Deadlock service from LockManager. - Backup service and backup endpoint worker. - Diagnostics service using the debug thread pool. LockManager start. - LockManager is started after gRPC service registration. - It uses PD, resolver, and security manager for lock role changes. Why the ordering. - Deadlock service depends on LockManager availability. - LockManager callbacks may rely on gRPC server readiness. ## Import SST service ImportSstService handles SST ingestion into rfstore. It uses a dedicated tokio runtime for importer tasks. Key stages. - Create a snapshot for the target region. - Acquire a per SST lock to avoid duplicate ingest. - Download or receive SST data and write to importer path. - Send raft ingest command to rfstore. - Release lock and report result. Design reasons. - Snapshot ensures the region is at a consistent term and epoch. - Locks prevent two concurrent ingests for the same SST. - Dedicated runtime isolates IO heavy ingest from gRPC threads. Additional notes. - Import threads use IO type tagging to isolate IO metrics. - A periodic tick shrinks importer state to control memory usage. ## Backup and diagnostics services cloud_server registers several additional gRPC services. These are started during register_services. ### Backup service Backup is handled by the backup crate. It uses a background worker and a dedicated endpoint. Key points. - The endpoint uses RaftKv and region info accessor. - Config is registered for online updates. - The service is registered before server start. ### Diagnostics service Diagnostics service exposes log search and server info. It uses the debug thread pool and tokio runtime handle. Why it exists. - Helps operators inspect logs without SSH access. - Provides hardware and load info for debugging. ## Status server architecture The status server is an HTTP server for admin endpoints. It is separate from the gRPC server by design. ### Runtime and threading The status server creates its own tokio runtime. It runs on a dedicated thread pool with a configurable size. Requests are handled by hyper using this runtime. Design rationale. - Separates admin workload from user RPC workload. - Allows blocking or heavy operations without gRPC impact. ### TLS and security The status server accepts either plain or TLS connections. It uses the PD security manager to build the acceptor. Certificate gating. - Most endpoints require a valid peer cert by CN allowlist. - Some GET endpoints are allowed without cert. Allowlist behavior. - If cert_allowed_cn is empty, the check is skipped. - If present, common name must match allowed patterns. Endpoints allowed without cert. - /metrics - /status - /config (GET) - /debug/pprof/profile Why gating exists. - Status endpoints can mutate config or expose key ranges. - Certificate checks prevent unintended access. ### Request dispatch Requests are matched by method and path. The handler uses a shared StatusContext with key dependencies. Shared context includes. - ConfigController for online config updates. - RaftRouter to send administrative raft messages. - kvengine and rfengine handles. - ConcurrencyManager for backup and safe point logic. - Store info cache for PD stats. Shutdown behavior. - Status server runs in a dedicated thread that listens for close_rx. - Shutdown requests stop serving without killing the runtime abruptly. ### Endpoint families (high level) The list below is not exhaustive. It groups endpoints by behavior and purpose. Observability and debugging. - Metrics export (/metrics). - PProf endpoints (/debug/pprof/*). - Log level and config changes (/config). - Failpoints (when enabled). Cluster and region operations. - Region sync and region info (/region/*). - Unsafe recovery and split and merge support. - Major compaction and flush control. Storage engine operations. - `/rfengine/backup` captures a regular lightweight-backup WAL boundary; the DFS worker must already be enabled and healthy. - `/rfengine/track_wal_progress` reports async WAL progress. - `GET /rfengine/dfs_worker` reports runtime `enabled`, `running`, and `healthy`. Runtime enable and disable use `POST /config` with the `rfengine.lightweight-backup` field; there is no DFS-worker-specific POST. - Restore shard (ChangeSet based). - Online snapshot Store prepare and full-CF Region capture. - Metadata-only typed Region file-reference dumps. - DFS file read and write helpers. - Columnar build and cleanup. Backup and WAL tracking notes. - rfengine backup validates cluster id from StoreIdent. - Optional backup_ts waits are coordinated via ConcurrencyManager. Recovery and blacklist. - Recovery mode management (/recovery/*). - Whitelist and blacklist updates and queries. Design note. - Many endpoints are intentionally rate limited or guarded. - Expensive operations run on blocking threads or via worker tasks. ### Concurrency control in status server Some endpoints limit concurrency explicitly. The DFS file read path combines a configurable in-flight request limit with kvengine's non-blocking cache-read memory admission. Online `capture` and `file_refs` dumps share a separate per-Store semaphore sized to half of the Store's CPU quota, rounded down with a minimum concurrency of one, and a 512 MiB response limit. Capture also uses a 1 GiB weighted memory budget, charging twice the captured memtable skiplist KV bytes in 16 MiB units. The memory permit remains attached to the response body until it is consumed or dropped. These endpoints do not consume the legacy snapshot-cache concurrency budget. Their CPU-intensive snapshot and response construction runs on the Tokio blocking pool instead of occupying the status server's async workers. Online snapshot control endpoints: - `POST /kvengine/online-snapshot/prepare` advances the Store backup-ts boundary and returns success only after old tracked transactions drain. Branch timestamps must exclude the zero and maximum sentinels. - `GET /kvengine/online-snapshot/capture/` requires that Store prepare and returns V3 full-CF memtable data without entering the legacy snapshot cache path. Capture validates the local Region version; the native-BR caller checks each source Region version again at that Region's final file-reference observation. - `GET /kvengine/online-snapshot/file-refs/` returns typed references from one stable `SnapAccess` without entering the snapshot dump path. - Online `POST /restore-shard` requests carry `expected_conf_ver`; the status server preserves it when routing to the Region leader, while legacy restore requests omit it and retain their existing version-only behavior. Successful prepare requests use the HTTP status as their complete acknowledgement and return an empty body. Capture responses likewise do not echo request identity in headers; consumers validate the structured payload and the V3 memtable format instead. Why. - DFS file reads can be large and block the runtime. - Request count bounds per-request overhead; byte admission bounds full source buffers using spare process headroom without consuming required-load quota. - Larger cache reservations preserve the complete future required-load quota in the process headroom check; rejected reads return HTTP 503 for fallback. - DFS reads are executed via spawn_blocking to avoid reactor stalls. - Request and byte guards are acquired before allocating the source buffer and stay with the response body until completion, cancellation, or drop. ### Store info cache Store stats are cached with a TTL. The cache is per store id and protected by an async mutex. Behavior summary. - Fetch fresh stats if cache is older than TTL. - If PD fails, return stale cache until max TTL. - After max TTL, propagate PD error to the caller. Why this design. - Avoids PD hot loops under many status queries. - Still allows stale information for best effort operations. ### Status request flow (simplified) This is the typical control plane request flow. Sequence. 1. Hyper accepts connection with or without TLS. 2. The service_fn builds a handler with StatusContext. 3. Certificate gating checks cert_allowed_cn when required. 4. Path and method dispatch select a handler. 5. Handler runs, often via spawn_blocking for heavy IO. 6. Response is returned and latency is recorded in histogram. ## Recovery, blacklist, and panic region handling cloud_server uses multiple safeguards for unstable regions. These safeguards are tied to the startup sequence. ### Panic region files Panic region files are created on repeated failures. Startup collects them and uses the counts to derive blacklists. Why. - Repeated panics often indicate corrupt data or inconsistent state. - Blacklisting prevents the region from crashing the node repeatedly. ### Blacklist sources Blacklists come from multiple sources. - Static blacklist file (black_list_path). - Recovery mode blacklist files. - Auto blacklist derived from panic regions. - Additional keyspace files in data dir. Whitelist override. - Whitelist file can remove keyspaces from blacklist. - This allows targeted recovery and exception handling. ### Recovery mode Recovery mode is toggled via config and status server. It enables recovery oriented behavior in kvengine. Design note. - Recovery mode is intentionally explicit and opt in. - Use it cautiously and document in runbooks. ## Observability and metrics Observability is built into each layer. cloud_server emits metrics and logs for critical paths. Key metric groups. - Shared `tikv_server_is_busy_total{type}` counts busy responses constructed for clients when service and lower storage layers construct them. - gRPC request latency and failure counts. - Raft message batching and flush counters. - Async write and snapshot counters. - Status server request histograms. - Memory usage and load statistics. - Shared memory profile auto-dump counter with `component` + `action` labels. Where metrics come from. - server.rs records gRPC thread load and memory usage. - raftkv.rs records async write and snapshot metrics. - raft_client.rs records flush and error counters. - status_server/metrics.rs records HTTP endpoint latency and snapshot-cache validation, lock-check, build, and payload metrics for `/kvengine/snapshot`. Startup info metrics. - SERVER_INFO_GAUGE_VEC records build version and startup timestamp. Logging practices. - Startup logs record key configuration and addresses. - Errors include store id, region id, and addresses where possible. - Slow status requests are logged with path and elapsed time. ## Failure modes and invariants This section lists failure modes that influence design choices. Understanding them is key to safe changes. ### Startup invariants - Config must be validated before engine open. - build_and_bind must run before Server::start. - Services must be registered before the gRPC server is built. - Store id must be set before rfstore is started. ### Async write failure modes - Dropped on_applied callback yields undetermined error. - Empty write requests are rejected early. - Early error injection is supported via failpoints. ### Transport failure modes - Address resolve failures clear pending messages. - Tombstone store resolution removes connection and pauses traffic. - Queue full returns DiscardReason::Full and increments metrics. - Paused stores return DiscardReason::Paused and do not reconnect. ### Status server failure modes - Missing TLS or invalid cert returns 403 for sensitive endpoints. - DFS file reads can fail with too many concurrent requests. - PD stats failures return stale data until max TTL expires. ### Storage safety invariants - Panic mark file indicates prior panic and blocks startup. - LOCK file ensures only one process uses a data directory. - Cluster id mismatch prevents accidental cross-cluster join. ## Testing and harnesses Several crates and tests exercise cloud_server behavior. Use them when making changes to the server or transport layers. Local integration. - components/test_cloud_server provides a test harness. - It wraps cloud_server and offers cluster builders. Integration suites. - tests/cloud_engine exercises the storage engine. - tests/cloud_engine_failpoints cover failpoint scenarios. - tests/random uses test_cloud_server for randomized testing. Why these matter. - Many bugs only appear under multi node or failpoint scenarios. - The harness simulates PD, store bootstrap, and raft traffic. ## Debugging playbook This is a lightweight guide to common issues. Keep it updated as new failure modes appear. ### Startup failures If startup fails early. - Check logs for config validation errors. - Verify cluster id and StoreIdent match PD. - Check data directory LOCK file and panic mark. - Confirm disk capacity if running on Kubernetes. ### RPC issues If client requests fail. - Check gRPC server health status. - Inspect gRPC request metrics and failure counters. - Verify proxy forwarding rules if configured. ### Raft transport issues If raft traffic stalls. - Check resolve failures and unreachable logs. - Inspect RAFT_MESSAGE_FLUSH_COUNTER for high full counts. - Verify store allowlist settings. - Check if any store is marked tombstone. ### Status server issues If status endpoints fail. - Verify TLS configuration and cert CN allowlist. - Check for 403 responses due to cert gating. - Review status server logs for slow requests. ### Backup and restore issues For backup. - Confirm cluster id match in backup config. - Ensure concurrency manager backup ts is advanced. - Temporary backup orchestration is client-side. It changes the normal DFS worker through `/config`, polls health, invokes the ordinary cloud-worker instant backup, and restores the initial disabled state. For restore. - Validate ChangeSet input and shard range coverage. - Check for unsafe recovery restrictions. ## Glossary - cloud_server: this crate, process coordinator for the Cloud Storage Engine server. - rfstore: Raft store implementation for cloud engine. - rfengine: Raft log and state persistence engine. - kvengine: data engine storing shards and tables. - DFS: distributed file system, builtin or S3. - RaftKv: engine wrapper converting storage ops to Raft commands. - Status server: HTTP server exposing admin and debug endpoints. ## References - `components/cloud_server/src/tikv_server.rs` for lifecycle. - `components/cloud_server/src/raftkv.rs` for custom raft logs. - `components/cloud_server/src/raft_client.rs` for transport logic. - `components/cloud_server/src/service/kv.rs` for gRPC service patterns. - `components/cloud_server/src/status_server/mod.rs` for HTTP admin logic. - `components/rfstore/README.md` for Raft store internals. - `doc/features/txn_file.md` for txn file feature flow.