# Backup Virtualization Design

## Scope

This document describes the backup virtualization changes on branch `codex/backup-virtualization`.
Merge commits are intentionally skipped; the design is based on the branch's non-merge commits and the current working tree.

## Background

Before this change, the scheduled backup worker materialized one `user_backups` row per scheduled backup. The worker periodically:

1. Found due backup schedules.
2. Queried cluster-proxy for the first physical backup after the scheduled time.
3. Inserted a `user_backups` row containing `BackupID`, `BackupName`, `BackupTime`, and `ScheduleTime`.
4. Advanced `backup_schedules.next_schedule_time`.

That model creates a persistent row for every retained scheduled backup. It also requires a recurring worker to scan due schedules and update many rows even though serverless physical backups already exist independently.

This branch adds virtual scheduled backups derived from the current schedule plus archived schedule versions. The virtualized implementation is now the single public API path; existing persisted `user_backups` are retained as historical input while they expire naturally.

## Goals

- Serve the virtual API model from schedules, histories, and physical slots for every organization.
- Retire scheduled `user_backups` materialization while retaining persisted rows for compatibility and cleanup.
- Generate scheduled backup list entries from current schedules, archived versions, and cron definitions.
- Keep restore and detail APIs compatible with existing callers by resolving virtual backup IDs to physical cluster-proxy backups lazily.
- Preserve delete semantics for virtual backups with tombstones.
- Avoid duplicate rows during migration while old persisted `user_backups` still exist.
- Avoid showing virtual backups before the cluster is actually ready.
- Avoid listing a scheduled backup as successful before its Kubernetes-level physical backup has materialized.
- Keep the old cleanup path temporarily while historical rows and backup jobs drain.

## Data Model

### Scale Assumptions

The production fleet contains millions of clusters. The virtual path avoids a recurring per-schedule materialization scan and instead uses Kubernetes-level slot synchronization, bounded batch reads of unique slot keys, history/tombstone storage, and periodic cleanup. Online virtual reads remain cluster- or tenant-scoped; large cluster-ID and slot-key sets are queried in bounded batches rather than one query per backup candidate.

### Virtual Backup ID

Virtual backup IDs are deterministic and encode:

- `cluster_id`
- `schedule_time`

The format is:

```text
<backup-id-prefix>v1-<base64url(cluster_id:unix_schedule_time)>
```

This ID identifies the scheduled restore point, not a physical cluster-proxy backup. The physical backup is resolved only when detail or restore needs it.

### Domain Object Boundary

`VirtualBackup` is an in-memory scheduled restore-point value. It is generated only after its Kubernetes-level physical slot has been found; the resolved physical backup metadata is attached in memory and the value is never persisted.

List paths convert persisted `UserBackup` and `VirtualBackup` values independently to the existing API `Backup` view before sorting and pagination. Detail and restore paths resolve a `VirtualBackup` against cluster-proxy first, then construct a complete `UserBackup` containing the physical backup fields. This keeps `UserBackup` and `RestoreJob.BackupInfo` persisted-backup-shaped and does not change the external API schema.

An internal `backupResolver` is the only component that classifies a backup ID as persisted or virtual. It centralizes virtual-ID parsing, tombstone/readiness/history/retention validation, and optional physical resolution. Callers choose one of two levels:

- restore-point resolution for delete, which validates the virtual backup without contacting cluster-proxy
- physical resolution for detail and restore, which additionally resolves the physical backup and returns a complete `UserBackup`

This avoids duplicating virtual/legacy branches in API entry points while preserving the lower-cost and more reliable tombstone delete path.

### Backup Schedule History

`backup_schedule_histories` is an archive, not a mirror of `backup_schedules`. It records only a schedule version that was replaced or deleted:

- cluster and tenant identity
- region, cloud provider, kubernetes
- cron and retention
- source token and snapshot fields
- `valid_from`, `valid_to`, and `deleted_at`

For an active cluster, listing synthesizes the current version from `backup_schedules` and combines it with archived rows. This keeps the common “never modified” case at zero history rows.

### Backup Tombstone

`backup_tombstones` records deleted restore points keyed by `(cluster_id, schedule_time)`. The virtual backup ID is retained for diagnostics, but suppression uses the restore-point key so a legacy persisted row and a virtual row cannot represent the same deleted backup under different IDs. Each tombstone also stores the restore point's actual expiry, derived from the resolved physical backup time.

## Migration

Migration `202607081200` creates:

- `backup_schedule_histories`
- `backup_tombstones`
- `backup_virtualization_metadata`, containing one immutable rollout cutover timestamp

The migration is schema-only and does not scan or backfill `backup_schedules`. Existing deleted clusters continue to use their persisted `user_backups` until those rows expire. This avoids reconstructing inaccurate historical cron versions from only the final deleted schedule configuration.

The cutover timestamp is written once when the migration runs. Current schedules are synthesized with `valid_from = max(schedule.created_at, cutover_at, latest_history.valid_to)`, so their current cron is never projected backward into the pre-virtualization period. Before cutover, persisted `user_backups` remain authoritative; after cutover, new virtual backups come from current schedules and newly archived history versions.

Existing non-expired `user_backups` are not deleted by the migration. They continue to be listed and used as persisted backups until they expire.

## Listing Backups

`ListBackups` now merges two sources:

1. Non-expired persisted `user_backups`.
2. Virtual backups generated from the current schedule and archived histories.

Virtual backups are generated by:

1. Loading active and archived histories that have not been physically expired by cleanup.
2. Computing the regular schedule window:

   ```text
   start = max(history.valid_from, virtualization.cutover_at, now - retention)
   end   = min(now, history.valid_to, history.deleted_at)
   ```

3. Enumerating cron occurrences in `[start, end]` and loading their physical slots.
4. Loading slots whose physical `backup_time` is still inside the largest relevant retention window. Slots whose `slot_time` predates the regular schedule window are checked against the history cron and validity interval. This reverse lookup recovers arbitrarily delayed physical backups without imposing a schedule-to-backup delay limit.
5. Calculating expiry exclusively as `physical_backup_time + retention` and keeping only unexpired occurrences.
6. Removing tombstoned restore points and deduplicating repeated virtual IDs across histories.

For active-cluster list and detail paths, `history.valid_from` is the local readiness boundary. Current histories derive it from the schedule creation time and the virtualization cutover, so these DB-only reads do not add a synchronous management-service dependency.

### Physical Backup Slot Index

The backup list must continue to show the real Kubernetes-level physical
backup time and restore the exact physical backup selected by a user. Looking
up cluster-proxy during every list request would make list traffic proportional
to user reads, while recreating one `user_backups` row per cluster would
reintroduce the old materialization cost.

`physical_backup_slots` stores the physical backup chosen for each half-hour
UTC slot. The worker enumerates Kubernetes directly from
`WorkerConfig.RegionalConfig[].K8sConfig[]`; it does not read or register
backup schedules. Backup policies only accept `HH:00` and `HH:30`, so there
are at most 48 slots per Kubernetes per day. The worker runs every minute and
uses the latest indexed slot as a continuous watermark. Each run processes a
bounded batch of at most 48 consecutive slots per Kubernetes. A Kubernetes
with no slot starts at the first half-hour slot on or after the global cutover.
The worker pages physical backups from the first unresolved slot, maps each
slot to the first physical backup at or after that slot time, and batch-upserts
the resolved continuous prefix. It stops at the first unresolved slot, so a
missing backup or failed query cannot create a watermark gap. Once the backup
becomes available, the next run can catch up by a full batch rather than one
slot per minute.

Virtual list, detail, and restore paths read the same slot row. A virtual
restore point is not listed as succeeded until its slot row exists. This makes
availability data-driven rather than relying on a fixed visibility delay, and
ensures the list's `backup_time`, selected `backup_id`, and restore target are
the same physical backup.

### Materialization Delay

Physical backups belong to the Kubernetes-level backup stream, not to an individual serverless cluster. The virtual listing derives restore points directly from schedule history and therefore uses the physical slot index as its physical-existence gate.

`ListBackups` and `ListRecycledInstanceBackups` use the physical slot index as
their materialization gate. A virtual schedule is listed as `succeeded` only
after its slot has been synchronized. The first synchronization attempt occurs
five minutes after the slot, which absorbs scheduler and propagation delay
without adding a cluster-proxy request to every list call.

This is deliberately fail-closed for recent scheduled backups: the list does not present a restore point before the corresponding physical slot has been indexed.

Existing persisted rows and new virtual rows can describe the same scheduled backup. Once the virtual restore point is ready, the virtual row wins and the matching persisted auto backup remains stored until normal expiry and cleanup. Manual backups and persisted backups without a ready virtual counterpart remain visible. Both `ListBackups` and `ListRecycledInstanceBackups` suppress duplicates and apply tombstones to both sources by `(cluster_id, schedule_time)`.

## Getting And Restoring A Virtual Backup

`GetBackup` and restore paths parse the virtual backup ID to recover `cluster_id` and `schedule_time`, regenerate the canonical ID, and require an exact string match.

They then validate:

- cluster ID matches the request when provided
- no tombstone exists for `(cluster_id, schedule_time)`
- a schedule history covers `schedule_time`
- the backup has not expired

Only after those checks does the service resolve the virtual backup to a physical cluster-proxy backup.

Detail and restore read the same physical slot row as the list. If it has not been synchronized yet, these paths return `RecordNotFound` with the message: "This backup is not available yet. Please try again later."

Physical lookup selects the first physical backup at or after the scheduled
time:

```text
choose the earliest physical backup whose time is >= schedule_time
```

Cluster-proxy `ListBackups` is called with:

```text
last_backup_time = schedule_time - 1 second
max_count = 100
```

Returned items are sorted by time. The earliest item at or after the schedule
time is used to fill:

- `BackupID`
- `BackupName`
- `BackupTime`

There is no upper-bound generation interval: the same selection rule is used
for slot indexing, backup detail, and restore, so all paths resolve a virtual
restore point to the same physical backup.

## Recycled Instances

Recycled instance list and backup list also use virtual backups.

During migration, recycled instance discovery merges clusters backed by non-expired persisted `user_backups` with clusters backed by virtual schedule histories. This keeps a recycled instance visible as long as either source still has an available backup; matching cluster IDs are returned only once.

For recycled instance discovery, deleted cluster metadata is loaded for identity, filtering, and display fields. Backup availability uses the same schedule-history and physical-slot boundaries as active-cluster listing; `creation_finished_at` is not an additional backup cutoff.

Availability checks use a newest-first cursor per schedule history. Each round batches the current candidate from every unresolved cluster into slot queries; clusters that miss advance to their previous candidate, while clusters that find an unexpired physical backup stop immediately. This avoids both full-window materialization and serial cluster-by-cluster queries. Recycled instances with the same deletion time use `cluster_id DESC` as a deterministic pagination tie-breaker.

Before cursor processing, recycled discovery performs the same physical-backup-time reverse lookup used by backup listing. A cluster with an unexpired delayed slot is immediately considered available even when its schedule time is older than its retention window.

Region predicates are pushed into the history query. Tombstones are loaded in batches of at most 500 cluster IDs, and deleted-cluster metadata is requested in bounded single-page batches to avoid N+1 queries, oversized `IN` clauses, and redundant pagination RPCs.

## Delete Semantics

Deleting a virtual backup creates a tombstone:

- tenant ID
- cluster ID
- virtual backup ID
- schedule time
- deleted time
- actual expire time

Deleting either representation writes the same restore-point tombstone. For a persisted scheduled backup, tombstone creation and row deletion run in one transaction; any stale persisted row for that restore point remains hidden from list/get/restore paths.

Delete writes and persisted reads apply tombstones in the single virtualized implementation, so deleted restore points cannot be resurrected by old persisted data.

Every recycled-instance get/list/delete path also requires cluster metadata to contain a deletion timestamp; `GetClusterIncludeDeleted` returning an active cluster is treated as not found.

## Worker Changes

Scheduled backup materialization through the legacy cron has been retired. No worker creates scheduled `user_backups` rows or `backup_jobs` records, and `backup_schedules.next_schedule_time` is no longer periodically advanced. The column remains populated by schedule create/update paths until a future schema cleanup.

The retained schedule-check worker also performs a bounded repair pass for existing schedules whose restore snapshot fields are incomplete. It reads at most 500 rows per run, fills source token, service plan, encryption, and zone-redundancy fields from cluster and billing metadata, and saves only the schedule metadata. It does not create backups, advance schedule time, or query cluster-proxy backups. Missing clusters and permanently empty source tokens are logged and excluded from the failed retry count; they require separate orphan/data cleanup. The pass is intentionally bounded and can be removed after production data has been verified complete.

Other worker responsibilities remain:

- cleaning expired backup-related records
- syncing restore progress
- executing restore jobs
- checking and repairing missing backup schedules for newly created clusters

The virtual path runs a Kubernetes-level slot synchronizer every minute and a bounded cleanup worker every 12 hours for expired slots, tombstones, and closed histories. The existing backup cleanup worker remains active while historical persisted rows and backup jobs drain; its soft-deleted schedule cleanup remains needed by the virtual path.

When a Kubernetes has no indexed slot yet, the synchronizer starts at the first half-hour slot on or after `virtualization_cutover` and advances from there in batches of at most 48 consecutive slots. This backfills restore points created between migration and the first worker run without introducing a schedule-to-physical-backup delay limit; a 30-day gap needs about 30 successful worker rounds instead of 1,440.

Legacy schedule cleanup also keeps a soft-deleted schedule while one of its history intervals can still produce an unexpired virtual backup, because restore-to-new still reads schedule-owned cluster parameters.

Tombstone cleanup uses the tombstone's actual physical expiry and removes it only when no persisted `user_backups` row for the same `(cluster_id, schedule_time)` remains unexpired.

`CheckBackupSchedule` is intentionally retained because virtual backup listing depends on the current schedule. If cluster creation events are missed, this check still backfills missing schedules.

## Event Handling

Schedule create/update/delete events maintain the archive only when it becomes necessary:

- creating a schedule writes no history row
- changing cron or retention archives the previous current version, then updates the current schedule
- deleting a schedule archives the final current version and marks every archived version for that cluster deleted

Updates that do not change cron or retention, including service-plan-only changes, do not create redundant history versions. A closed history is removed only after the slot watermark has advanced past its effective end and every physical slot in the interval is expired according to `backup_time + history.retention`. This places no upper bound on physical backup delay.

This lets virtual backup listing derive the correct schedule configuration for historical restore points.

## Compatibility

Persisted `user_backups` remain valid and are merged by the single virtualized API implementation. A ready virtual row hides the matching persisted auto backup, while older persisted rows remain available until their normal expiry and cleanup.
- Manual backups and persisted backups without a ready virtual counterpart remain listed until expiration.
- Restore continues to use their stored physical backup metadata.

Virtual backup IDs are deterministic and can be constructed by callers, so detail, restore, and delete paths perform the same existence checks as listing paths instead of trusting that a virtual ID came from a prior list response.

## Tests

The branch adds coverage for:

- virtual backup ID round trip
- cron schedule enumeration
- history coverage checks
- list merging of persisted and virtual backups
- tombstone suppression
- persisted-over-virtual deduplication
- physical slot indexing and readiness gating for recent virtual schedules
- local history/cutover cutoff for active list and direct get, plus deleted-cluster readiness cutoff for recycled instances
- virtual backup resolution through cluster-proxy
- restore request construction from a virtual backup
- virtual backup delete tombstone creation
- bounded repair of incomplete schedule snapshot fields for deleted-cluster restore