# Spike: recovery API (PITR) - findings Verified 2026-07-22 against a live TiDB Cloud Essential cluster. Two documented endpoints did not behave as specified (see below); the rest of the record resolves every gap against a live Essential cluster: PITR restore, snapshot backup + restore-from-backup, and delete were all verified end to end. Billable steps created short-lived clusters (each deleted + verified in the same session, well under the $10 guardrail); everything else was read-only or rejected (free) requests. The sections below are kept in chronological order (initial blockers first, then the resolutions) so the reasoning is auditable. ## Non-billable probes (against the existing Starter demo cluster 10612738041866767921) Host `https://cloud.tidbapi.com/v1beta2`, same Digest key auth as the serverless API. Confirmed reachable and authenticating. | Probe | Result | Notes | | ------------------------------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /tidbs/{id}/backups` | 200 | Shape: `{"backups":[],"totalSize":0,"nextPageToken":""}`. Starter cluster has no manual snapshots. Endpoint confirmed. | | `POST /tidbs:restore` (empty body) | 400 `"Tidb is required."` | Endpoint EXISTS; body needs a `tidb` field. Confirmed. | | `GET /tidbs/{id}/backupSetting` | 404 `"Get cluster failed."` | On a Starter cluster. Likely Essential-only (Starter has no PITR window), which would be a valid tier signal, but UNCONFIRMED until tested on Essential. | | `GET /tidbs/{id}:getRestoreStatus` | 404 `"invalid path"` | Does NOT exist as described. The `:getRestoreStatus` custom verb is not a valid path. | | `information_schema.DDL_JOBS` via read path | ok, 3 rows | Readable on the read-only ephemeral user. Columns: JOB_ID, DB_NAME, TABLE_NAME, JOB_TYPE, STATE, CREATE_TIME. `tidb_recent_ddl` feasible. | ## The foundational ambiguity: what is `tidbId`? The same id (`10612738041866767921`, our serverless Starter cluster) behaves inconsistently across the v1beta2 endpoints: | Endpoint | Result for this id | | ------------------------------- | --------------------------- | | `GET /tidbs/{id}/backups` | 200 (empty list) | | `GET /tidbs/{id}` | 404 `"Cluster not found."` | | `GET /tidbs/{id}/backupSetting` | 404 `"Get cluster failed."` | So `backups` accepts the id but `get-cluster` and `backupSetting` say the cluster does not exist. That means the `tidbId` these endpoints key on is probably NOT the serverless cluster id we use elsewhere, OR these endpoints do not apply to a serverless Starter cluster. Until we know the correct identifier (and how to obtain it from a serverless cluster id), none of get-recovery-info, restore, or status can be wired correctly. This is the primary blocker. ## Hard-gate discrepancies (STOP and report) 1. **Restore status endpoint is wrong.** `GET /tidbs/{id}:getRestoreStatus` returns 404 "invalid path". Since a PITR restore creates a NEW cluster, restore status is most likely the new cluster's own state (a normal get-cluster on the restored id, state Creating/Restoring -> Active/Failed), or a differently-named endpoint. Need the correct mechanism before wiring `tidb_get_restore_status`. 2. **backupSetting unconfirmed.** 404 "Get cluster failed" on Starter. Consistent with Starter having no PITR window (a tier signal), but whether the endpoint and its earliest-restorable timestamp field work on Essential is unverified. Needs an Essential test or corrected path. ## What is NOT yet done (blocked on the above) - (c) trigger a PITR restore and poll to terminal state: blocked on the correct status mechanism. - (d) restore-from-backupId request shape. - (f) Starter degrade behavior: partially seen (backupSetting 404); the full picture depends on confirming Essential behavior. - (g) out-of-window rejection: needs the Essential window first. - Essential billing granularity check (hourly vs daily minimum) vs the $10 guardrail: not started; will do before any billable create. ## Essential-cluster probes (id 10939402612054875070, read-only, zero cost) An Essential cluster resolved the `tidbId` ambiguity and confirmed the two gaps are real, not tier artifacts. - `GET /tidbs/{id}` -> 200. `tidbId` equals the cluster id. Returns `state` (ACTIVE), `servicePlan` (`Essential_V2`), `createTime`, `endpoints`. This is the tier-detection source AND the restore- status source (poll the restored cluster's state here). The earlier Starter 404s were because the old serverless Starter cluster is not registered on this v1beta2 host. - `GET /tidbs/{id}/backupSetting` -> 200, but the shape is the SNAPSHOT schedule, not a PITR window: `{tidbId, backupPolicy:"TIER1", mode:"STANDARD_BUNDLE", retentionDays:7, dailyBackupTimeUtc:"00:00"}`. There is NO earliest-restorable-timestamp / PITR window field, and `retentionDays` is 7, not the documented 30. So the plan's "backupSetting returns the restorable window" is wrong. - `GET /tidbs/{id}/backups` -> 200, empty (fresh cluster, no snapshot taken yet). - `GET /tidbs/{id}:getRestoreStatus` -> 404 "invalid path" on Essential too. The documented status endpoint does not exist. Use the restored cluster's `state` via `GET /tidbs/{newId}` instead. ## Resolved / adjusted design - `tidbId` == cluster id. Confirmed. - Tier detection: read `servicePlan` from `GET /tidbs/{id}` (Essential_V2 vs Starter). - Restore status: poll `GET /tidbs/{newId}` `state`, not the non-existent `:getRestoreStatus`. ## Step 0 completed: one real PITR restore (2026-07-22) A single billable PITR restore was triggered, polled, verified, and deleted, all within the same session and under the $10 guardrail. Findings: ### Restore request (confirmed working) ``` POST https://cloud.tidbapi.com/v1beta2/tidbs:restore { "tidb": { "displayName": "recovery-spike-DELETE-ME", "regionId": "aws-us-west-2", "servicePlan": "Essential_V2", "minRcu": "2000", "maxRcu": "2000" }, "restoreMode": "PITR", "sourceTidbId": "", "pointInTime": "" } ``` 200 returns the FULL new-cluster object immediately: `tidbId` (the new id), `state: "CREATING"`, `displayName`, `regionId`, `servicePlan`, `minRcu`/`maxRcu`, `labels`, `createTime`. The response shape is identical to a normal create-cluster. `restoreMode` also accepts `SNAPSHOT` (backupId) per the docs; only PITR was exercised here. ### The PITR window: latest-restorable LAGS wall-clock by ~13-15 min on a fresh cluster Source cluster created 18:49 UTC. Probed at ~19:18 UTC (cluster ~29 min old). Results: | pointInTime | offset from now | result | | ----------- | --------------- | -------------------------------------------------- | | 19:16:05 | now-2m | 400 "Point in time is not in the pitr time range." | | 19:13:06 | now-5m | 400 rejected | | 19:09:16 | now-8m | 400 rejected | | 19:03:06 | now-15m | **200 ACCEPTED, cluster created** | So the restorable ceiling is NOT `now`; it trails real time (here by >13 min, <=15 min). On a fresh cluster the PITR log/checkpoint takes time to catch up. Design consequence: NEVER pass `now` or a near-now timestamp. Out-of-window requests are rejected server-side with 400 "Point in time is not in the pitr time range." and create NOTHING (free to probe). The tool must NOT try to compute the exact ceiling client-side (there is no endpoint that returns it); it relies on this server-side 400 as the gate, surfaces the exact message, and should suggest stepping the timestamp further back. ### Restore creates a NEW cluster; source untouched New cluster state progression observed: `CREATING` -> `RESTORING` -> (would reach `ACTIVE`). Source cluster stayed `ACTIVE` throughout. Confirms: PITR restore is always out-of-place, never in-place. ### Restore status = poll GET /tidbs/{newId} `state` Confirmed again: `:getRestoreStatus` does not exist. Status is the new cluster's own `state` (CREATING/RESTORING -> ACTIVE, or FAILED). `tidb_get_restore_status` wraps GET /tidbs/{newId}. ### Project: restore defaults to "default virtual project" UNLESS labels are passed (RESOLVED) First restore (no labels): source is in `chainscope` (1372813089454786113) but the restored cluster landed in `1372813089454636502` = "default virtual project". The restore body has no top-level project field, so it defaults. Second restore, passing `tidb.labels = {"tidb.cloud/project": "1372813089454786113"}`: the new cluster landed in `chainscope`. Confirmed. So the restore `tidb` object honors the same `tidb.cloud/project` label mechanism as create-cluster. DESIGN DECISION: `tidb_restore_cluster` reads the SOURCE cluster's project label and passes it as the new cluster's label by default, so a recovery lands in the same project as the source (never silently in "default virtual project"). The operator can override with an explicit projectId. The preflight surfaces the target project so it is never a surprise. When the server is `--project` confined, the restore is refused if the source is outside that project, and the new cluster is forced into the confined project. ### Delete works mid-RESTORING `DELETE /tidbs/{newId}` returned 200 with `state: "DELETED"` while the cluster was still RESTORING (no need to wait for ACTIVE). A subsequent GET returns 403 "does not have permission to access the resource" (the deleted cluster is no longer accessible), and the cluster list shows 0 non-deleted recovery-spike clusters. Deletion is immediate and verifiable. ### Cost Negligible and within guardrail. The one created cluster lived ~3 minutes (created 19:18:07, deleted 19:21) as an idle Essential (usage-based RCU) cluster that never took traffic. Deleted and deletion-verified in the same session. All other probes were GETs or rejected (free) POSTs. ## Step 0b: snapshot backup + restore-from-backup (2026-07-22, verified live) Verified `tidb_restore_backup`'s path end to end on the Essential test cluster (10939402612054875070), staying under the $10 ceiling; all created resources deleted and verified. ### Create a manual backup ``` POST /tidbs/{id}/backups body: {"name": ""} -> 200 {"backupId": "111039"} ``` The body needs `name` (a minimal body 400s "Name is required."; `displayName`/`description` alone do not satisfy it). The create response returns the id as **`backupId`**. ### API key inconsistency: `id` (list) vs `backupId` (create) The LIST endpoint (`GET /tidbs/{id}/backups`) returns each backup's id as **`id`**, but create-backup returns it as **`backupId`**. Same value, different key. Code must read both (helper `backupIdOf`). List object fields: `id`, `tidbId`, `displayName`, `description`, `state`, `type` (SNAPSHOT), `triggerType` (AUTO|MANUAL), `sizeBytes`, `createTime`, `expirationTime`, `backupTs`, `regionId`. ### Backup states: RUNNING -> SUCCEEDED (not COMPLETED) The terminal success state is **`SUCCEEDED`**, not `COMPLETED`/`SUCCESS`. A manual snapshot of this ~1.6 MB cluster took **~6 minutes** (created 19:56:30, SUCCEEDED by ~20:02). Only a SUCCEEDED backup can be restored. Note: the cluster's scheduled AUTO backup showed `state: FAILED, sizeBytes: 0`, so Essential snapshots can fail; the tool should not assume a backup exists or succeeded. ### Restore from backup (verified request shape) ``` POST /tidbs:restore { "tidb": { ...same new-cluster spec as PITR, incl. labels for project... }, "restoreMode": "SNAPSHOT", "backupId": "111039", "sourceTidbId": "" } ``` 200, returns the new cluster (state CREATING), landed in the project set via the `tidb` labels (chainscope, confirmed). Same host and response shape as the PITR restore. ### Delete a backup ``` DELETE /tidbs/{id}/backups/{backupId} -> 200 {} ``` Confirmed: the manual backup was gone from the list afterward. Cleanup complete (restored cluster DELETED, manual backup deleted), leaving only the pre-existing FAILED auto-backup. ## Step 0c: create-backup tool spike + re-verify (2026-07-22) Repeated create-backup + restore-from-backup once more to back the new `tidb_create_backup` tool: - Create-backup shape re-confirmed: `POST /tidbs/{id}/backups {"name":...}` -> `{"backupId":"111090"}`. - Second manual snapshot reached SUCCEEDED in ~4 min (~6 min the first time); size ~1.6 MB. - More AUTO backups showed FAILED on this cluster (111012, 111045). Essential snapshots here are genuinely flaky, which is exactly why the tools never assume a backup exists or succeeded. - Restore from 111090 -> new cluster CREATING in chainscope (project label honored again). - **Delete-cluster returned a transient 500 once**, then 200 on the immediate retry. Lesson: cluster delete is not always first-try; retry on 5xx and re-verify with a GET (403 == gone). The recovery cleanup path should tolerate a transient 5xx, not conclude the cluster survived. - Everything created (restored cluster + backup 111090) deleted and verified gone (GET 403, backup absent from list, 0 stray clusters). Cost trivial, within the $10 ceiling. Design note: `tidb_delete_backup` is deliberately NOT exposed as a tool even though the endpoint is verified. An agent with both delete-backup and delete-cluster is the ransomware sequence; backup deletion stays a human console action.