# Native BR PD Meta Watch Cache

## Problem

`tikv-worker` runs native BR lightweight backup periodically. Each backup calls
`backup_pd_keyspace_meta` to copy PD metadata into
`ClusterBackupMeta.keyspace_meta`.

The current implementation already reuses the last backup meta and uses
`with_min_mod_revision(old_revision)`, but every backup still issues range
reads over the raw PD/etcd metadata prefixes. With many keyspaces and a short
backup interval, these repeated range scans can put avoidable pressure on PD.

Current path:

- `components/cloud_worker/src/native_br.rs` creates `BackupWorker`.
- `components/native_br/src/backup_worker.rs` injects periodic backup requests.
- `components/native_br/src/backup.rs::backup_cluster_with_ts` calls
  `backup_pd_keyspace_meta`.
- `backup_pd_keyspace_meta` scans all prefixes in `PD_KEY_SPACE_META_PATH` and
  paginates with `MAX_BATCH_GET_CNT`.

## Goals

- Avoid repeated large PD/etcd range reads during periodic backup.
- Keep `ClusterBackupMeta` and restore behavior unchanged.
- Continue backing up raw PD metadata instead of parsed PD structures.
- Fall back to the existing scan path whenever the cache cannot prove it is
  complete.
- Keep the change local to native BR and tikv-worker wiring.

## Non-goals

- Do not introduce delta backup meta files.
- Do not change full restore or keyspace restore semantics.
- Do not make one-shot CLI backup depend on a background cache.

## Metadata Scope

The cache must cover the same raw metadata prefixes currently scanned by
`PD_KEY_SPACE_META_PATH`:

- `/pd/$cluster_id/keyspaces/`
- `/pd/$cluster_id/region_label/keyspaces/`
- `/pd/$cluster_id/rules/`
- `/pd/$cluster_id/rule_group/`
- `/pd/$cluster_id/tso/keyspace_groups/membership/`
- `resource_group/`
- `/tidb/remote/worker/`
- `/pd/$cluster_id/config`
- `/pd/$cluster_id/meta_service_groups/`

The backup meta still stores a full raw KV map. This proposal only changes how
the map is collected.
The live watch is intentionally broader than this metadata scope: it watches
all etcd keys on one stream and filters events back down to these prefixes
before mutating the cached map.

## Design

Add a long-lived PD metadata watch cache owned by native BR backup worker. The
cache maintains an in-memory raw KV snapshot for the metadata prefixes above.

Backup-time behavior:

1. Ask the cache for a complete snapshot.
2. The cache requests watch progress and waits until the stream catches up.
3. If progress succeeds, clone the filtered local KV map into
   `ClusterBackupMeta`.
4. If the cache is stale, initializing, or times out, run the existing scan
   path.

```text
tikv-worker startup
    |
    v
NativeBrManager -> BackupWorker
    |
    v
PdMetaWatchCache bootstrap
    |
    +-- seed from latest backup meta when possible
    |       |
    |       +-- watch all keys from meta_revision + 1
    |
    +-- otherwise full snapshot at a fixed revision
            |
            +-- watch all keys from snapshot_revision + 1

periodic backup
    |
    +-- cache snapshot succeeds -> write keyspace_meta from cache
    |
    +-- cache snapshot fails    -> existing PD/etcd scan fallback
```

## Main Interfaces

Add a small provider abstraction so the backup path does not depend directly on
watch implementation details:

```rust
pub struct PdMetaSnapshot {
    pub revision: i64,
    pub kvs: HashMap<Vec<u8>, Vec<u8>>,
}

pub trait PdMetaProvider: Send + Sync {
    fn snapshot(&self, timeout: Duration) -> Result<PdMetaSnapshot>;
}
```

`backup_pd_keyspace_meta` should accept an optional provider:

- If `skip_keyspace_meta` is enabled, preserve the current test behavior.
- If the provider returns a snapshot, set `keyspace_meta` and `meta_revision`
  from it.
- If the provider returns stale/unavailable/timeout, log and use the existing
  range scan.

`BackupRunner` should own the provider because it owns periodic backup state
and `last_backup_meta`. `NativeBrManager::new` creates the provider only when a
new config flag is enabled.

## Bootstrap

The cache supports two bootstrap modes.

### Seed From Latest Backup

If the latest backup meta exists and contains `keyspace_meta` plus a non-zero
`meta_revision`, initialize the cache from it and start watching from
`meta_revision + 1`.

If that revision has already been compacted, discard the seed and take a full
snapshot.

### Full Snapshot

When no usable seed exists:

1. Read one metadata prefix at the latest revision and record the response
   header revision as `snapshot_revision`.
2. Read all metadata prefixes at `snapshot_revision` with
   `GetOptions::with_revision(snapshot_revision)`.
3. Populate the cache with the returned KV pairs.
4. Start the all-key watch from `snapshot_revision + 1`.

Using one fixed revision keeps the initial cache internally consistent across
all prefixes.

## Watch Handling

Use one all-key etcd watch stream and keep the same prefix boundaries as the
existing scan path when applying events:

- Resolve `PD_KEY_SPACE_META_PATH` for the current cluster ID.
- Create a single watch over all keys from the bootstrap revision.
- Keep the defensive key-prefix check before applying an event.
- Ignore events whose keys are outside the metadata prefixes.

For each watch response:

- In-scope put event: insert or update `kvs[key] = value`.
- In-scope delete event: remove `key`.
- Out-of-scope event: ignore the key/value payload.
- Compacted watch: mark cache stale, incrementally scan live keys modified
  since the local snapshot revision, and restart the watch from the scan
  revision.
- Canceled watch: mark cache stale and reconnect from the local snapshot
  revision when possible.
- Stream error or EOF: reconnect from `applied_revision + 1`.
- Successful event response: advance `applied_revision` to the maximum
  delivered event modification revision.
- Empty progress response: advance `applied_revision` to the response header
  revision.

The single all-key watch intentionally avoids coordinating progress across
multiple watch IDs. A progress response on the only watch stream proves that all
future events on the stream have a higher revision than the response header, so
a cache snapshot acknowledged by that response is complete for every filtered
metadata prefix at that revision.

## Correctness

The cache must satisfy these invariants:

- A returned snapshot contains a complete raw KV image for every metadata
  prefix.
- `snapshot.revision` represents a revision at or after all returned events.
- Events outside the metadata prefixes may advance the watch revision, but must
  never be stored in the backup metadata map.
- If watch progress cannot be proven, backup must fall back to range scan.
- Delete events observed by the watch must be reflected in the cache.
- A compacted watch gap follows the same append-only metadata assumption as the
  existing scan path, because an etcd range scan with `min_mod_revision` cannot
  report keys that were deleted during the gap.
- Full snapshot resync must replace the local map atomically from backup
  callers' perspective.

The backup timestamp and PD metadata revision are different timelines. This is
already true in the current implementation; this design only changes how the PD
metadata image is obtained.

## Failure Handling

Cache states:

- `Initializing`
- `Ready`
- `Stale`
- `Resyncing`
- `Stopped`

Fallback to the existing scan path when:

- the cache is not `Ready`;
- progress request times out;
- watch revision was compacted and resync is not complete;
- bootstrap or reconnect is failing;
- the cache provider is disabled.

If fallback scan also fails, return the original backup error.

## Configuration

Add rollout knobs under `[native-br]`:

```toml
[native-br]
backup-pd-meta-watch-cache-enabled = false
backup-pd-meta-watch-cache-progress-timeout = "2s"
```

Initial default should be disabled. After canary validation, consider enabling
it automatically when periodic backup is enabled.

## Metrics

Add low-cardinality metrics:

- `tikv_worker_native_br_pd_meta_cache_state{state}`
- `tikv_worker_native_br_pd_meta_cache_kvs`
- `tikv_worker_native_br_pd_meta_cache_revision`
- `tikv_worker_native_br_pd_meta_watch_events_total{type}`
- `tikv_worker_native_br_pd_meta_watch_resync_total{reason}`
- `tikv_worker_native_br_pd_meta_backup_snapshot_total{source}`
- `tikv_worker_native_br_pd_meta_fallback_total{reason}`

Do not label by keyspace ID.

Important logs:

- bootstrap source, key count, and revision;
- watch compacted or canceled;
- resync start and finish;
- backup fallback reason.

## Compatibility

Backup meta format is unchanged. Restore code remains unchanged because it
already restores raw `ClusterBackupMeta.keyspace_meta` entries to PD/etcd.

One-shot backup paths without a provider keep using the existing scan path.

## Rollout

1. Add provider abstraction and keep default behavior unchanged.
2. Implement watch cache behind a disabled config flag.
3. Add metrics and fallback logs.
4. Enable in a test environment with many keyspaces.
5. Validate PD/etcd range-read reduction and successful restore.
6. Exercise restart, watch compaction, stream error, and PD leader change.
7. Enable in one production canary.

## Test Plan

Unit tests:

- full snapshot pagination builds a complete map;
- put and delete events update the cache correctly;
- out-of-scope watch events are ignored and not stored in the cache;
- compacted watch marks cache stale and triggers incremental scan resync;
- full snapshot resync replaces the map atomically;
- latest-backup seed starts from `meta_revision + 1`;
- compacted seed falls back to full snapshot.

Backup-path tests with a fake provider:

- provider snapshot populates `keyspace_meta` and `meta_revision`;
- provider failure falls back to existing scan;
- `skip_keyspace_meta` still skips metadata backup in test builds.

Integration/manual tests:

- many-keyspace periodic backup with cache disabled vs enabled;
- tikv-worker restart with latest-backup seed;
- forced watch compaction and resync;
- unrelated etcd key churn while the cache is enabled;
- restore from backups produced by both paths.

## Alternatives

### Increase Backup Interval

This lowers PD pressure but weakens recovery point objectives and does not fix
the scaling issue.

### Store Delta Metadata Files

This would reduce backup meta object size too, but requires a new metadata
format, restore compatibility logic, and archive updates. It should be a
separate project.

## Estimated Work

Recommended implementation size: 5 to 8 person-days.

- Provider abstraction and backup path wiring: 1 day.
- Watch cache bootstrap, event application, and progress handling: 2 to 3 days.
- Resync, fallback, shutdown, and configuration: 1 to 2 days.
- Metrics, logging, and docs: 0.5 to 1 day.
- Tests and manual validation: 1 to 2 days.

The main risks are catch-up correctness, compacted revision fallback, and
avoiding partial snapshots during resync.