# Remote Coprocessor Functionality This document elaborates the key code path associated with the **Remote Coprocessor** feature, presented in the sequence of processing procedure. There are two different implementations of the remote coprocessor. The main call path of the first way is `tidb` -> `tikv-worker` -> `tikv` -> `tikv-worker` -> `tidb`. The second way's call path is `tidb` -> `tikv` -> `tikv-worker` -> `tikv` -> `tidb`. The second way is easier to operate in the production environment, so we use the second way to enable the remote coprocessor. ## First way ### tidb-server side The code is housed in [tidb-server](https://github.com/tidbcloud/tidb-cse/pull/395): - The tidb-server configures the remote-coprocessor address in the config file. - It utilizes a session variable to regulate the usage of the remote coprocessor. - When enabled, coprocessor requests are directed to the remote-coprocessor address, bypassing the tikv-server. ### tikv-worker side The tikv-worker, running a `RemoteCopServer`, handles the remote-coprocessor requests. Upon receipt of the `Coprocessor` request, the tikv-worker: - Constructs a `DelegateRequest` and dispatches it to the tikv-server. ### tikv-server side Upon receipt of the `DelegateRequest`, the tikv-server: - Executes `check_memory_locks`. - Acquires the `SnapAccess` and forms a `CloudStore`. - Checks locks and assembles the memory table data via `new_memtable_iterator`. - Responds to the tikv-worker with the collected memory table data and snapshot change set. ### tikv-worker side Upon receipt of the `DelegateResponse`, the tikv-worker: - Constructs the SnapAccess using the received memory table data and snapshot change set. - Reads the SST files in the Moka cache. If a cache miss occurs, it reads from S3. - Handle the request via `parse_request_and_handle_remote_cop`. ## Second way ### tikv-server side The tikv-server can be configured to offload resource-intensive coprocessor requests to remote workers. Before the offloading, the tikv-server check the range blocks of the coprocessor request. If the size exceeds the threshold, tikv-server will offload the request to the remote worker. This can significantly improve service stability in the case of unexpected large queries exhaust the cluster resource. example config: ``` [kvengine] remote-coprocessor-addr = "http://127.0.0.1:19000/coprocessor" remote-coprocessor-min-blocks-size = 32000000 remote-coprocessor-topn-max-pushdown-rows = 50000 ``` - The `remote-coprocessor-addr` parameter specifies the address of the remote coprocessor worker to which heavy coprocessor requests will be offloaded. - The `remote-coprocessor-min-blocks-size` parameter defines the blocks size threshold for offloading a coprocessor request to the remote worker. - The `remote-coprocessor-topn-max-pushdown-rows` parameter limits which non-partition TopN operators are pushed down to the remote worker. Its default is 50,000, and zero disables the policy. For a larger TopN, TiKV sends `TopN.limit - 1` as the paging size so the worker returns resumable source batches instead of building and draining the entire TopN heap in one batch. The tikv-server encodes the offload request with related memtable and snapshot data, and then dispatches the encoded request to worker using http/https. ### Offload decision reasons `tikv_coprocessor_remote_decision_total{result,reason}` records one primary, low-cardinality reason for each request that reaches the load-aware routing decision. Requests routed remotely use these reasons: - `blocks_size`: the estimated blocks size reaches the configured minimum. - `num_ranges`: the range count reaches the configured shortcut threshold. - `long_query_blocks_size`: query age lowers the minimum blocks-size threshold. - `lazy_pattern_blocks_size` / `lazy_pattern_duration`: historical executions of a lazy remote pattern exceed the size or duration threshold. Size takes precedence when both match. - `load_aware_force_blocks_size` / `load_aware_force_num_ranges`: a load-aware force threshold bypasses the CPU preference. - `cpu_busy`: the load-aware state allows remote execution after a busy signal. - `no_cpu_signal`: no usable CPU signal is available, so routing fails open to remote execution. Local decisions use `cpu_idle`. The metric records a single primary reason per decision, so its per-reason QPS sums to the total decision QPS without double counting. ### tikv-worker side The tikv-worker using `/coprocessor` endpoint to handle the offload request. - Decode the request to get the memtable data and snapshot data. - Construct the SnapAccess using the received memory table data and snapshot change set. - Reads the SST files in the cachefs. - Handle the request via `parse_request_and_handle_remote_cop`. ### Coprocessor cache data version For the full protocol and CSE implementation details, see [Coprocessor Cache In Cloud Storage Engine](coprocessor_cache.md). TiDB's coprocessor cache follows the same validation shape as HTTP cache: - TiDB sends a cache-enabled coprocessor request. - TiKV returns a normal response with `cache_last_version`. - TiDB stores the request, response, and `cache_last_version`. - On the next identical request, TiDB sends `cache_if_match_version`. - TiKV compares that version with `SnapshotExt::get_data_version()`. - If the version is unchanged for a unary request, TiKV returns an empty response with `is_cache_hit` set, and TiDB fills the result from its local cache. For rfstore, `SnapshotExt::get_data_version()` is backed by the kvengine snapshot's captured `write_sequence`, which represents the applied Raft index of the data snapshot. Local coprocessor execution uses the store snapshot directly, so the version is naturally preserved. Remote coprocessor execution is different because TiKV serializes a `SnapAccess` into ChangeSet bytes plus memtable data, and the worker reconstructs a new snapshot before executing the DAG. The ChangeSet does not transport the captured `write_sequence`: `Snapshot.data_sequence` represents persisted shard data and must not be reused as an in-memory apply sequence. Instead, the production TiKV offload path keeps the source `SnapAccess` in `RemoteDagDispatcher`. The worker reports `can_be_cached` after checking its reconstructed snapshot. The origin separately remembers newer WRITE_CF versions filtered from serialized memtables and locks found before dispatch, then vetoes worker cache admission if either exists. It also overwrites `cache_last_version` with the retained snapshot's `write_sequence`. That is the same captured applied index used by local cache validation, without extending or overloading the worker snapshot protocol. Remote workers do not perform the cache-hit fast path. In the TiKV offload route, the TiKV endpoint checks unary `CachedRequestHandler` eligibility after snapshot acquisition and before constructing any remote handler, so a cache-valid unary request never reaches `tikv-worker`. Streaming requests do not use `CachedRequestHandler`; they follow normal local or remote execution and can only return ordinary response chunks. The rebuilt offload request only preserves `Request.is_cache_enabled`; this lets a normal remote DAG response run newer-timestamp checks. The origin dispatcher combines worker and origin cache-admission evidence and adds `cache_last_version` before responding to TiDB. Normal responses are admitted to TiDB's cache only when TiKV sets `can_be_cached`. CloudStore therefore opts into newer-timestamp checking for DAG execution. Scans and incremental gets keep reporting uncacheable results when they observe a write or lock newer than the request timestamp, because TiDB's coprocessor cache key does not include the timestamp. If no newer data is observed, the response can carry `can_be_cached = true` together with the snapshot `cache_last_version`. ### Large remote TopN paging For a worker-first rolling upgrade, an upgraded tikv-worker can temporarily apply the same TopN paging policy to requests from older TiKV servers: ``` cop-topn-max-pushdown-rows = 50000 ``` The worker setting defaults to zero. With zero, the worker does not decode the nested DAG or change its paging size. After all TiKV servers have been upgraded and `remote-coprocessor-topn-max-pushdown-rows` is active there, reset the worker setting to zero; finite paging from the upgraded TiKV servers is passed through without a second DAG decode. Both settings are row-count heuristics, not hard response-memory limits. Choose a lower threshold if the workload contains wide rows, and continue to monitor remote response sizes and process memory during rollout. ### cop-limiter The cop-limit is configured on the tikv-worker to throttle the coprocessor requests when the memory usage exceeds the threshold to avoid OOM. example config: ``` [cop-limiter] high-mem-ratio = 0.8 wait-min-duration = "100ms" wait-max-duration = "3s" sample-ttl = "60s" ``` When the memory usage above `high-mem-ratio`, the cop-limiter will throttle the coprocessor requests. Each throttled request will wait until the memory usage drops below the `high-mem-ratio`. To avoid thundering herd, a single wait sleeps for a random duration between `wait-min-duration` and `wait-max-duration`. Each request's response size is sampled and collected in a sliding window to calculate each keyspace's throughput. The `sample-ttl` defined the sliding window duration.