# Coprocessor Cache In Cloud Storage Engine

This note explains how TiDB coprocessor cache works when the TiKV side is
Cloud Storage Engine (CSE). It focuses on the TiKV/CSE implementation, including
remote coprocessor execution, snapshot data-version ownership, cache
admission, and common debugging checks.

## Goal

The coprocessor cache lets TiDB reuse the result of an identical coprocessor
request without making TiKV execute the DAG again. The protocol is similar to
HTTP cache validation:

- TiDB sends a coprocessor request to TiKV.
- TiKV executes the request and returns a normal response with the current data
  version.
- TiDB stores the request, response, and data version in its local cache.
- On the next identical request, TiDB sends the cached data version back to
  TiKV.
- TiKV compares TiDB's version with the latest snapshot data version.
- If the version matches for a unary coprocessor request, TiKV returns a small
  cache-hit response and TiDB fills the real response from its local cache.
- If the version does not match, TiKV executes the request normally and returns
  a fresh response with the latest data version.

This avoids repeated scan and aggregation work for stable regions while keeping
TiKV as the authority that validates whether cached data is still current.

## Wire Fields

The protocol uses fields in `kvproto::coprocessor::{Request, Response}`.

Request-side fields:

- `Request.is_cache_enabled`: TiDB sets this when the request participates in
  coprocessor cache validation.
- `Request.cache_if_match_version`: TiDB's cached data version for the same
  request. This is meaningful only when `is_cache_enabled` is true.

Response-side fields:

- `Response.can_be_cached`: TiKV sets this on a normal response when the result
  is safe for TiDB to admit into its local cache.
- `Response.cache_last_version`: TiKV's data version for the snapshot used by
  this response.
- `Response.is_cache_hit`: TiKV sets this on unary responses when it did not
  execute the request because TiDB's cached response is still valid.

The cache-hit response intentionally does not carry the full DAG result. TiDB is
expected to read the cached result locally when `is_cache_hit` is true.
Streaming coprocessor requests do not use this empty-response cache-hit shape;
they always execute normally and stream normal response chunks.

## Data Version Source

The TiKV-side validation point is `tikv_kv::SnapshotExt::get_data_version()`.
Engines that do not support a data version return `None`, and those snapshots
cannot produce a cache hit.

For CSE's rfstore path, the data version is the region snapshot's applied Raft
index:

- `components/tikv_kv/src/rfstore_impls.rs` implements
  `SnapshotExt::get_data_version()` by returning
  `RegionSnapshot::get_applied_index()`.
- This value changes when the region applies new data, so it is a conservative
  invalidation token for cached coprocessor results.

CSE remote coprocessor execution needs extra care because the worker executes on
a reconstructed snapshot. TiKV serializes a `kvengine::SnapAccess` into
ChangeSet bytes plus memtable data, but that message does not carry the captured
`write_sequence` used as the coprocessor cache token. In particular,
`Snapshot.data_sequence` describes persisted shard data and must not be
overloaded with the in-memory `write_sequence`.

The production offload path therefore keeps cache-version ownership at the
origin TiKV:

- `RemoteDagDispatcher` retains the exact `SnapAccess` used to serialize the
  remote request.
- The worker executes the DAG and checks its reconstructed snapshot for hidden
  newer MVCC state.
- The origin also records newer WRITE_CF versions filtered from serialized
  memtables and locks found by its pre-dispatch scan. Either signal vetoes
  `Response.can_be_cached`, because the worker cannot observe omitted data.
- After decoding a successful worker response,
  `RemoteDagDispatcher::handle_request` sets `Response.cache_last_version` from
  `SnapAccess::get_write_sequence()` when cache admission is enabled.

This is the same captured applied index that the origin `RegionSnapshot`
exposes through `SnapshotExt::get_data_version()`. The version cannot drift
between request serialization and response handling, and no cache-specific
sequence field needs to be added to the worker snapshot protocol.

## Local TiKV Path

For a normal TiDB to TiKV coprocessor request, the cache flow is:

1. `src/coprocessor/endpoint.rs` parses `Request.is_cache_enabled` and
   `Request.cache_if_match_version` into `ReqContext.cache_match_version`.
2. TiKV retrieves the storage snapshot for the request.
3. After the snapshot is available, TiKV compares
   `ReqContext.cache_match_version` with `snapshot.ext().get_data_version()`.
4. For unary requests, if they match, TiKV uses `CachedRequestHandler` instead
   of building the DAG executor tree.
5. `src/coprocessor/cache.rs` returns `Response.is_cache_hit = true` and, when
   available, the current `cache_last_version`.
6. If they do not match, TiKV builds the normal DAG handler.
7. Normal DAG responses set `Response.is_cache_hit = false`,
   `Response.can_be_cached`, and `Response.cache_last_version` in
   `src/coprocessor/dag/mod.rs`.

The key property is that the unary cache-hit decision happens after snapshot
acquisition. This avoids comparing TiDB's cached version with a moving live
store value that might not match the actual read snapshot.

Streaming endpoint paths intentionally do not use `CachedRequestHandler`.
`CachedRequestHandler` returns an empty metadata-only response, while streaming
requests expect normal streamed chunks. A streaming request with a matching
`cache_if_match_version` therefore still follows the normal local or remote
execution policy. It may produce a normal response with `can_be_cached` and
`cache_last_version`, but it will not return `Response.is_cache_hit = true`.

## Remote Coprocessor Paths

CSE supports two remote coprocessor shapes, both documented in
`doc/features/remote_coprocessor.md`.

### Direct Remote Coprocessor

The direct path is:

```text
TiDB -> tikv-worker -> TiKV -> tikv-worker -> TiDB
```

The worker receives the TiDB coprocessor request and delegates snapshot
construction to TiKV. TiKV returns the ChangeSet and memtable data. The worker
reconstructs a snapshot and calls `parse_request_and_handle_remote_cop`.

The direct remote-coprocessor route does not use the generic
`CachedRequestHandler` fast path. It executes the request normally on the worker
snapshot. This legacy route also has no origin-side `RemoteDagDispatcher` to
stamp the captured applied index, so it is not part of the supported CSE
coprocessor-cache metadata path.

### TiKV Offload To Remote Worker

The production-oriented offload path is:

```text
TiDB -> TiKV -> tikv-worker -> TiKV -> TiDB
```

TiKV first receives the TiDB request, parses it, and decides whether to offload
the request based on remote-coprocessor policy. The offload dispatcher rebuilds
a new `kvproto::coprocessor::Request` for the worker in
`src/coprocessor/remote_dispatcher.rs`.

For unary requests, the local TiKV endpoint checks `CachedRequestHandler`
eligibility before it asks the offload dispatcher to create a remote handler. A
cache-valid unary request should therefore never be sent to `tikv-worker`; it
returns a local cache-hit response immediately after snapshot validation.
Streaming requests skip this fast path and continue through normal local or
remote execution.

The rebuilt offload request still preserves cache enablement, but not the match
version. Preserving `Request.is_cache_enabled` lets the normal worker-side DAG
execution run newer-timestamp checks. The origin separately retains evidence
for newer memtable versions filtered during serialization and locks found
before dispatch. When the response returns,
`RemoteDagDispatcher::handle_request` vetoes `can_be_cached` if that origin
evidence exists and stamps `cache_last_version` from the retained origin
snapshot. The worker does not compare `cache_if_match_version`, return
`CachedRequestHandler`, or need the origin snapshot's `write_sequence`.

DB9 DAG requests still do not participate in this generic DAG cache metadata
path.

## Cache Admission And MVCC Correctness

TiKV should not ask TiDB to cache every normal response. A response is admitted
only when `Response.can_be_cached` is true.

The central correctness issue is that TiDB's coprocessor cache key does not
include the request timestamp. A response read at timestamp `t1` can be reused
for a later timestamp `t2` only if the scanned data range had no MVCC write or
lock newer than `t1`. Otherwise the same request at `t2` might legitimately see
different data even if the region apply index did not change after the first
response was cached.

CSE handles this through the query executor storage layer:

- `src/coprocessor/endpoint.rs` reads `is_cache_enabled` from the request and
  passes it as `local_is_cache_enabled` into `DagHandlerBuilder`.
- `src/coprocessor/dag/mod.rs` passes `local_is_cache_enabled` into
  `TikvStorage::new(...)`.
- `src/coprocessor/dag/storage_impl.rs` initializes
  `NewerTsCheckState::NotMetYet` when checking is enabled.
- Scans and incremental point gets update that state when they observe newer
  writes or locks.
- Incremental point gets use the metadata-only `SnapAccess::has_newer` probe.
  It detects hidden newer MVCC versions without resolving a `BlobRef`, so cache
  admission does not add object-store IO or blob-read failures for values that
  are invisible at the request timestamp.
- The executor chain exposes this as `BatchExecutor::can_be_cached()`.
- `handle_qe_response` copies the final value to `Response.can_be_cached`.

The range scan check in `CloudStore` uses a delta write iterator so newer delete
markers are also detected. This is important because a normal read iterator can
hide a newer delete from the visible result while the newer delete still makes
the old response unsafe to reuse at a later timestamp.

## End-To-End Timeline

Initial request:

```text
TiDB cache lookup misses
TiDB -> TiKV: is_cache_enabled may be true, no useful match version
TiKV gets snapshot data_version = 11616
TiKV executes DAG
TiKV -> TiDB: is_cache_hit = false,
              can_be_cached = true,
              cache_last_version = 11616,
              data = full result
TiDB stores request -> response at version 11616
```

Repeated request with unchanged data:

```text
TiDB cache lookup finds request at version 11616
TiDB -> TiKV: is_cache_enabled = true,
              cache_if_match_version = 11616
TiKV gets snapshot data_version = 11616
TiKV -> TiDB: is_cache_hit = true,
              cache_last_version = 11616,
              data = empty
TiDB fills the result from its local cache
```

Repeated request after region data changed:

```text
TiDB cache lookup finds request at version 11616
TiDB -> TiKV: is_cache_enabled = true,
              cache_if_match_version = 11616
TiKV gets snapshot data_version = 11640
TiKV executes DAG
TiKV -> TiDB: is_cache_hit = false,
              can_be_cached = true or false,
              cache_last_version = 11640,
              data = fresh result
TiDB replaces or skips the local cache entry based on can_be_cached
```

## Observability

The most direct user-visible signal is TiDB's `EXPLAIN ANALYZE` output:

```sql
explain analyze select count(1) from sbtest1 where id > 1000;
```

Expected shape for a stable repeated request:

- First run: `copr_cache_hit_ratio: 0.00`, normal `proc_keys`.
- Second run: `copr_cache_hit_ratio: 1.00`, usually `proc_keys: 0`, and much
  lower `fetch_resp_duration`.

When debugging a miss, inspect these layers in order:

1. TiDB must send `Request.is_cache_enabled = true` with the previous
   `cache_if_match_version`.
2. TiKV must preserve that value in `ReqContext.cache_match_version`.
3. If the request is offloaded, `build_remote_dag_request` must preserve cache
   enablement for normal response admission.
4. `RemoteDagDispatcher::handle_request` must overwrite the worker response's
   `cache_last_version` with the retained origin snapshot's `write_sequence`.
5. For unary requests, the local endpoint must choose `CachedRequestHandler`
   before constructing a remote handler when the versions match. Streaming
   requests must not choose `CachedRequestHandler`.
6. Normal responses must set both `can_be_cached = true` and
   `cache_last_version`.
7. `CloudStore` must mark responses uncacheable when newer MVCC state is seen.

## Tests

The focused tests for this behavior are:

- `test_build_remote_dag_request_preserves_cache_enablement` in
  `src/coprocessor/remote_dispatcher.rs`
- `test_set_remote_cache_metadata_uses_origin_evidence` in
  `src/coprocessor/remote_dispatcher.rs`
- `test_build_mem_data_reports_filtered_newer_write` in
  `components/kvengine/src/read.rs`
- `test_cache_hit_skips_remote_coprocessor_dispatch` in
  `src/coprocessor/endpoint.rs`
- `test_cloud_store_enables_newer_ts_cacheability_check` in
  `src/storage/txn/cloud_store.rs`
- `test_marshal_omits_snapshot_versions` in
  `components/kvengine/src/read.rs`

Useful targeted commands:

```bash
cargo test -p tikv test_build_remote_dag_request_preserves_cache_enablement
cargo test -p tikv test_set_remote_cache_metadata_uses_origin_evidence
cargo test -p kvengine test_build_mem_data_reports_filtered_newer_write
cargo test -p tikv test_cache_hit_skips_remote_coprocessor_dispatch
cargo test -p tikv test_cloud_store_enables_newer_ts_cacheability_check
cargo test -p kvengine test_marshal_omits_snapshot_versions
```

For a live local NextGen stack, the practical validation is two identical
`EXPLAIN ANALYZE` executions through `tidb-normal`. The first execution should
populate TiDB's cache; the second should report `copr_cache_hit_ratio: 1.00`
when no relevant data changed between the two runs.

## Current Boundaries

- The cache validation token is region-snapshot level. Writes in the same
  region can invalidate a cached response even when they do not affect the
  specific SQL result. This is conservative and correct.
- Engines that return `None` from `SnapshotExt::get_data_version()` cannot
  produce a cache hit.
- DB9 DAG requests are not handled by the generic worker cache-hit fast path.
- Streaming coprocessor requests do not return cache-hit responses. They may
  still return normal cacheable responses for TiDB to store.
- Cache admission depends on MVCC newer-version tracking. Disabling or
  bypassing that tracking can create timestamp-related stale-read risk, because
  TiDB's cache key omits the request timestamp.