# Design & Plan: CRRF (Cross Region Replication Failover) for BRService Generated by Claude Opus 4.6 with Copilot. ## Overview CRRF is a feature based on snapshot backup + log backup (PiTR) + AWS S3 Cross Region Replication. It replicates backup data (snapshot + log) from an upstream region to a downstream region via CRR, then uses PiTR restore in the downstream region to create a disaster-recovery cluster. This design adds 4 new RPCs to BRService: `EnableCRRF`, `DisableCRRF`, `GetCRRF`, `UpdateCRRF`, with full-stack implementation across cluster-server and cluster-controller, plus a new `crrf-daemon` Helm chart. ### Key Design Decisions - **1:1 relationship**: One CRRF per log backup - **crrf_id**: Caller-provided (not server-generated) - **Annotation-driven**: cluster-server writes annotations on Backup CR; cluster-controller reconciles based on annotation presence/absence - **Helm-based deployment**: crrf-daemon deployed via Helm install/uninstall in dataplane - **BR image reuse**: crrf-daemon uses the same BR image as the log backup by default, overridable via parameters --- ## 1. State Machine ``` EnableCRRF called │ ▼ ┌──────────────────────────────────┐ │ PENDING │ ← crrf-spec annotation set, │ (helm install in progress) │ crrf-status = {"phase":"Pending"} └──────────┬───────────────────────┘ │ helm install success + │ deployment ready ▼ ┌──────────────────────────────────┐ │ RUNNING │ ← crrf-status = {"phase":"Running"} │ (crrf-daemon active) │ └──────────┬───────────────────────┘ │ DisableCRRF called │ (crrf-spec annotation removed) ▼ ┌──────────────────────────────────┐ │ DESTROYING │ ← crrf-status = {"phase":"Destroying"} │ (helm uninstall in progress) │ └──────────┬───────────────────────┘ │ helm uninstall success ▼ ┌──────────────────────────────────┐ │ DISABLED │ ← crrf-status annotation removed │ (no crrf-daemon) │ └──────────────────────────────────┘ Error handling: PENDING ──(helm install failed)──▶ FAILED ← crrf-status = {"phase":"Failed","message":"..."} RUNNING ──(daemon unhealthy)────▶ FAILED (via health check, future enhancement) ``` ### State Transitions Table | Current State | Trigger | Next State | Action | |---|---|---|---| | (none) | EnableCRRF | PENDING | Set crrf-spec anno, set crrf-status=Pending | | PENDING | Helm install success + deployment ready | RUNNING | Update crrf-status=Running | | PENDING | Helm install failed | FAILED | Update crrf-status=Failed | | RUNNING | DisableCRRF | DESTROYING | Remove crrf-spec anno, set crrf-status=Destroying | | DESTROYING | Helm uninstall success | (none) | Remove crrf-status anno | | FAILED | EnableCRRF (retry) | PENDING | Update crrf-spec anno, reset crrf-status=Pending | | FAILED | DisableCRRF | DESTROYING | Remove crrf-spec anno, set crrf-status=Destroying | | RUNNING | UpdateCRRF (image change) | PENDING | Update crrf-spec anno (new image), helm upgrade, reset crrf-status=Pending | --- ## 2. Sequence Diagrams ### EnableCRRF Flow ``` Caller cluster-server K8s API cluster-controller Dataplane K8s │ │ │ │ │ │ EnableCRRF(req) │ │ │ │ │─────────────────────▶│ │ │ │ │ │ Get Cluster CR │ │ │ │ │──────────────────────▶│ │ │ │ │◀──────────────────────│ │ │ │ │ Get Backup CR │ │ │ │ │──────────────────────▶│ │ │ │ │◀──────────────────────│ │ │ │ │ │ │ │ │ │ Validate: │ │ │ │ │ - mode == log │ │ │ │ │ - cluster_id match │ │ │ │ │ - no existing crrf │ │ │ │ │ or same crrf_id │ │ │ │ │ │ │ │ │ │ Set annotation: │ │ │ │ │ crrf-spec={json} │ │ │ │ │──────────────────────▶│ │ │ │ │◀──────────────────────│ │ │ │ Response(OK) │ │ │ │ │◀─────────────────────│ │ │ │ │ │ │ Reconcile event │ │ │ │ │───────────────────▶│ │ │ │ │ │ Read crrf-spec anno │ │ │ │ │ │ │ │ │ │ Set crrf-status= │ │ │ │ │ Pending │ │ │ │◀───────────────────│ │ │ │ │ │ │ │ │ │ │ helm.Apply(crrf-daemon) │ │ │ │──────────────────────▶│ │ │ │ │◀──────────────────────│ │ │ │ │ │ │ │ │ │ Check Deployment ready│ │ │ │ │──────────────────────▶│ │ │ │ │◀──────────────────────│ │ │ │ │ │ │ │ │ │ Update crrf-status= │ │ │ │ │ Running │ │ │ │◀───────────────────│ │ ``` ### DisableCRRF Flow ``` Caller cluster-server K8s API cluster-controller Dataplane K8s │ │ │ │ │ │ DisableCRRF(req) │ │ │ │ │─────────────────────▶│ │ │ │ │ │ Get Cluster + Backup │ │ │ │ │──────────────────────▶│ │ │ │ │◀──────────────────────│ │ │ │ │ │ │ │ │ │ Validate: │ │ │ │ │ - crrf exists │ │ │ │ │ - crrf_id match │ │ │ │ │ │ │ │ │ │ Remove crrf-spec │ │ │ │ │ annotation │ │ │ │ │──────────────────────▶│ │ │ │ │◀──────────────────────│ │ │ │ Response(OK) │ │ │ │ │◀─────────────────────│ │ │ │ │ │ │ Reconcile event │ │ │ │ │───────────────────▶│ │ │ │ │ │ No crrf-spec but │ │ │ │ │ crrf-status exists │ │ │ │ │ │ │ │ │ │ Set crrf-status= │ │ │ │ │ Destroying │ │ │ │◀───────────────────│ │ │ │ │ │ │ │ │ │ │ helm.Uninstall(crrf) │ │ │ │ │──────────────────────▶│ │ │ │ │◀──────────────────────│ │ │ │ │ │ │ │ │ │ Remove crrf-status │ │ │ │ │ annotation │ │ │ │◀───────────────────│ │ ``` ### GetCRRF Flow ``` Caller cluster-server K8s API │ │ │ │ GetCRRF(req) │ │ │─────────────────────▶│ │ │ │ Get Cluster + Backup │ │ │──────────────────────▶│ │ │◀──────────────────────│ │ │ │ │ │ Read crrf-spec and │ │ │ crrf-status annos │ │ │ │ │ Response(CRRFInfo) │ │ │◀─────────────────────│ │ ``` --- ## 3. API Definition (Proto) ### New RPCs in BRService ```protobuf // In api/tidbcloud/cluster-server/v1/services.proto, add to BRService: rpc EnableCRRF(EnableCRRFRequest) returns (CRRFInfo) { option (google.api.http) = { post: "/clusters/{cluster_id}/logbackups/{log_backup_name}:enableCRRF" body: "*" }; } rpc DisableCRRF(DisableCRRFRequest) returns (CRRFInfo) { option (google.api.http) = { post: "/clusters/{cluster_id}/logbackups/{log_backup_name}:disableCRRF" body: "*" }; } rpc GetCRRF(GetCRRFRequest) returns (CRRFInfo) { option (google.api.http) = { get: "/clusters/{cluster_id}/logbackups/{log_backup_name}/crrf" }; } rpc UpdateCRRF(UpdateCRRFRequest) returns (CRRFInfo) { option (google.api.http) = { post: "/clusters/{cluster_id}/logbackups/{log_backup_name}:updateCRRF" body: "*" }; } ``` ### New Messages ```protobuf // In api/tidbcloud/cluster-server/v1/resources.proto: message EnableCRRFRequest { string cluster_id = 1 [(buf.validate.field).required = true]; string log_backup_name = 2 [(buf.validate.field).required = true]; string crrf_id = 3 [(buf.validate.field).required = true]; // Optional: sub-directory for state storage, defaults to "crr-checkpoint/" string state_storage_sub_dir = 4; // Optional: downstream S3 storage URI. If empty, --downstream-storage is not set. string downstream_storage = 5; // Optional: override the BR image for crrf-daemon. Defaults to log backup's image. string image = 6; } message DisableCRRFRequest { string cluster_id = 1 [(buf.validate.field).required = true]; string log_backup_name = 2 [(buf.validate.field).required = true]; string crrf_id = 3 [(buf.validate.field).required = true]; } message GetCRRFRequest { string cluster_id = 1 [(buf.validate.field).required = true]; string log_backup_name = 2 [(buf.validate.field).required = true]; // Note: no crrf_id field — Get is looked up by backup name only } message UpdateCRRFRequest { string cluster_id = 1 [(buf.validate.field).required = true]; string log_backup_name = 2 [(buf.validate.field).required = true]; string crrf_id = 3 [(buf.validate.field).required = true]; // Optional: set a new BR image; triggers helm upgrade + status reset to Pending optional string image = 4; } message CRRFInfo { string crrf_id = 1; string cluster_id = 2; string log_backup_name = 3; CRRFPhase phase = 4; string message = 5; string helm_release_name = 6; // upstream_storage is the source S3 URI (from the log backup spec.storage) string upstream_storage = 7; // downstream_storage is the optional destination S3 URI string downstream_storage = 8; // image is the BR image used by the crrf-daemon string image = 9; enum CRRFPhase { CRRF_PHASE_UNSPECIFIED = 0; PENDING = 1; RUNNING = 2; DESTROYING = 3; FAILED = 4; DISABLED = 5; } } ``` --- ## 4. Annotation Schema ### `backup.tidbcloud.com/crrf-spec` Set by cluster-server when EnableCRRF is called. Removed by cluster-server when DisableCRRF is called. ```json { "crrf_id": "crrf-123", "state_storage_sub_dir": "crr-checkpoint/", "downstream_storage": "s3://bucket-region2/dir1/dir2", "image": "pingcap/br:v8.1.0" } ``` ### `backup.tidbcloud.com/crrf-status` Managed by cluster-controller. Tracks the lifecycle of the crrf-daemon. ```json { "crrf_id": "crrf-123", "phase": "Running", "message": "", "helm_release_name": "crrf-daemon-", "upstream_storage": "s3://source-bucket/prefix", "downstream_storage": "s3://dest-bucket/prefix", "image": "pingcap/br:v8.1.0", "last_transition_time": "2026-04-08T10:00:00Z" } ``` The `upstream_storage`, `downstream_storage`, and `image` fields are written when the status first transitions to Pending and updated again when it transitions to Running. They allow `GetCRRF` and `UpdateCRRF` to return the currently active configuration without reading the spec. --- ## 5. Annotation Constants Add to `infra-api/api/v1/backup_types.go`: ```go const ( BackupAnnotationKeyCRRFSpec = BackupAnnotationKeyPrefix + "crrf-spec" BackupAnnotationKeyCRRFStatus = BackupAnnotationKeyPrefix + "crrf-status" ) ``` --- ## 6. cluster-server Implementation ### File: `cluster-server/server/br.go` Add 3 new methods to `brServer`: #### EnableCRRF 1. Validate cluster exists via `s.clusterReader.Get()` 2. Get Backup CR by `log_backup_name` 3. Validate: - Backup mode is `log` (`backup.Spec.Mode == BackupModeLog`) - Backup belongs to `cluster_id` (`backup.Spec.ClusterName` matches cluster.Name) - No existing crrf-spec annotation, OR existing crrf-spec has same crrf_id (idempotent) 4. Build CRRFSpec JSON: `{crrf_id, state_storage_sub_dir (default "crr-checkpoint/"), downstream_storage, image (default from backup.Spec.Configuration)}` 5. Set annotation `backup.tidbcloud.com/crrf-spec` on Backup CR 6. `s.kubeClient.Update(ctx, backup)` 7. Return `CRRFInfo{crrf_id, phase: PENDING}` #### DisableCRRF 1. Validate cluster exists 2. Get Backup CR by `log_backup_name` 3. Validate: - crrf-spec annotation exists - crrf_id in annotation matches request crrf_id 4. Remove annotation `backup.tidbcloud.com/crrf-spec` 5. `s.kubeClient.Update(ctx, backup)` 6. Return `CRRFInfo{crrf_id, phase: DESTROYING}` #### GetCRRF 1. Validate cluster exists 2. Get Backup CR by `log_backup_name` 3. Read `crrf-spec` and `crrf-status` annotations 4. If neither annotation exists → return NotFound 5. Return `CRRFStatus2PB(clusterID, logBackupName, backup)` (status takes precedence over spec) #### UpdateCRRF 1. Validate cluster exists 2. Get Backup CR by `log_backup_name` 3. Validate crrf-spec annotation exists and `crrf_id` matches 4. Apply mutators to the spec (currently only `image` can be updated) 5. Marshal updated spec back to annotation and call `s.kubeClient.Update()` 6. Return `CRRFStatus2PB()` — response reflects the current (pre-reconcile) status plus updated spec fields ### File: `cluster-server/server/cvt/br.go` Add conversion functions: - `CRRFStatus2PB(spec, status) -> *CRRFInfo` — converts annotation JSONs to proto response --- ## 7. cluster-controller Implementation ### Reconcile Logic Change **File:** `cluster-controller/controllers/backup/task/backup.go` In the existing backup task's `PostCreate()` or as a new task, add CRRF logic: When `backup.Spec.Mode == BackupModeLog`: - If annotation `crrf-spec` is set → ensure crrf-daemon is deployed (CRRF task Create) - If annotation `crrf-spec` is NOT set but `crrf-status` IS set → destroy crrf-daemon (CRRF task Delete) ### New Task: CRRF **File:** `cluster-controller/controllers/backup/task/crrf.go` ```go type crrfTask struct { *baseTask config *CRRFDaemonConfig // implements helm.Package } ``` #### Create(ctx) 1. Parse `crrf-spec` annotation from `t.backup` 2. Parse `crrf-status` annotation (if exists) 3. If crrf-status phase is already Running → check deployment health, return Done or requeue 4. Set crrf-status = `{phase: "Pending", crrf_id: spec.crrf_id}` 5. Build `CRRFDaemonConfig` with Helm values: - image: from crrf-spec (or default from backup config) - upstream_storage: from `backup.Spec.Storage` - downstream_storage: from crrf-spec (optional) - task_name: `t.backup.Name` - state_storage_sub_dir: from crrf-spec - pd_addr: `-pd..svc:2379` - TLS cert paths: `/tls/ca.crt`, `/tls/tls.key`, `/tls/tls.crt` 6. `helm.Apply(ctx, getter, t.config)` to install/upgrade crrf-daemon 7. Check deployment readiness in dataplane 8. If ready → set crrf-status = `{phase: "Running"}` 9. If not ready → requeue after 15s #### Delete(ctx) 1. Parse `crrf-status` annotation 2. If crrf-status is empty → return Done 3. Set crrf-status = `{phase: "Destroying"}` 4. `helm.Uninstall(ctx, getter, t.config, releaseName)` 5. Remove crrf-status annotation 6. Return Done ### Task Registration **File:** `cluster-controller/controllers/backup/controller.go` Add CRRF task to the task list conditionally: ```go tasks := []deploy.Task{ task.NewGuardian(c), task.NewNodePool(c), task.NewBackup(c), } // Add CRRF task for log backups that need CRRF management if backup.Spec.Mode != nil && *backup.Spec.Mode == cloudv1.BackupModeLog { if hasCRRFSpec(backup) || hasCRRFStatus(backup) { tasks = append(tasks, task.NewCRRF(c)) } } ``` ### CRRFDaemonConfig (Helm Package) **File:** `cluster-controller/controllers/backup/task/crrf_config.go` Implements `helm.Package` interface: - `GetName()` → `"crrf-daemon-"` - `GetNamespace()` → cluster's PrefixedId (same namespace as log backup) - `ToHelmChart()` → loads from config chart path (e.g., `helm-charts/crrf-daemon/`) - `ToHelmValues()` → returns values map with all runtime parameters - `GetCRDs()` → nil (no CRDs) --- ## 8. crrf-daemon Helm Chart ### Directory: `helm-charts/crrf-daemon/` ### Chart.yaml ```yaml apiVersion: v2 name: crrf-daemon description: Cross Region Replication Failover daemon for TiDB Cloud log backup type: application version: 0.1.0 appVersion: "1.0.0" ``` ### values.yaml (Parameters exposed) ```yaml replicaCount: 1 image: repository: pingcap/br tag: "latest" pullPolicy: IfNotPresent # Required: upstream S3 storage URI (from log backup) upstreamStorage: "" # Optional: downstream S3 storage URI downstreamStorage: "" # Required: log backup task name taskName: "" # Optional: state storage sub-directory stateStorageSubDir: "crr-checkpoint/" # Required: PD address pdAddr: "" # Status listen address statusAddr: ":10081" # TLS settings tls: enabled: true secretName: "db-cluster-client-secret" mountPath: "/tls" caFile: "ca.crt" certFile: "tls.crt" keyFile: "tls.key" # K8s Service service: type: ClusterIP port: 10081 # VMServiceScrape for metrics collection metrics: enabled: true interval: 60s scrapeTimeout: 60s # Resources resources: limits: memory: "512Mi" requests: cpu: "100m" memory: "128Mi" nameOverride: "" fullnameOverride: "" podAnnotations: {} podSecurityContext: runAsGroup: 1000 runAsUser: 1000 securityContext: {} nodeSelector: {} tolerations: [] affinity: {} ``` ### templates/deployment.yaml - Single container running `./br operator crr-checkpoint` with args from values - Volume mount for client TLS certs from `db-cluster-client-secret` - Liveness/readiness probe on status HTTP port - Labels: `app: crrf-daemon`, `app.kubernetes.io/component: crrf` ### templates/service.yaml - ClusterIP service exposing port 10081 (metrics + status) - Label: `app: crrf-daemon` (selector for VMServiceScrape) ### templates/vmservicescrape.yaml - VictoriaMetrics VMServiceScrape resource - Scrapes metrics port at configured interval - Selector matches service labels ### templates/_helpers.tpl - Standard Helm helper templates (fullname, labels, selectorLabels) ### Command Construction (in deployment.yaml) ```yaml command: - ./br - operator - crr-checkpoint args: - "--upstream-storage" - "{{ .Values.upstreamStorage }}" {{- if .Values.downstreamStorage }} - "--downstream-storage" - "{{ .Values.downstreamStorage }}" {{- end }} - "--task-name" - "{{ .Values.taskName }}" - "--state-storage-sub-dir" - "{{ .Values.stateStorageSubDir }}" - "-u" - "{{ .Values.pdAddr }}" - "--status-addr" - "{{ .Values.statusAddr }}" {{- if .Values.tls.enabled }} - "--ca" - "{{ .Values.tls.mountPath }}/{{ .Values.tls.caFile }}" - "--key" - "{{ .Values.tls.mountPath }}/{{ .Values.tls.keyFile }}" - "--cert" - "{{ .Values.tls.mountPath }}/{{ .Values.tls.certFile }}" {{- end }} ``` --- ## 9. Implementation Steps ### Phase 1: API & Proto Definition 1. Add `EnableCRRF`, `DisableCRRF`, `GetCRRF` RPCs to BRService in `api/tidbcloud/cluster-server/v1/services.proto` 2. Add request/response messages to `api/tidbcloud/cluster-server/v1/resources.proto` 3. Run `make proto` in `api/` to generate Go code (requires buf + BSR access) ### Phase 2: Annotation Constants & Types 4. Add `BackupAnnotationKeyCRRFSpec` and `BackupAnnotationKeyCRRFStatus` to `infra-api/api/v1/backup_types.go` 5. Define `CRRFSpec` and `CRRFStatus` Go structs in a new file `infra-api/api/v1/crrf_types.go` (or in backup_types.go) ### Phase 3: Helm Chart (*parallel with Phase 2*) 6. Create `helm-charts/crrf-daemon/` directory with Chart.yaml, values.yaml 7. Create templates: `_helpers.tpl`, `deployment.yaml`, `service.yaml`, `vmservicescrape.yaml` ### Phase 4: cluster-server (*depends on Phase 1 + 2*) 8. Implement `EnableCRRF`, `DisableCRRF`, `GetCRRF` in `cluster-server/server/br.go` 9. Add conversion function in `cluster-server/server/cvt/br.go` 10. Register new RPCs in server initialization (if not auto-registered) ### Phase 5: cluster-controller (*depends on Phase 2 + 3*) 11. Add `CRRFDaemonConfig` implementing `helm.Package` in `cluster-controller/controllers/backup/task/crrf_config.go` 12. Implement CRRF task (`Create`, `Delete`, `PostCreate`) in `cluster-controller/controllers/backup/task/crrf.go` 13. Add CRRF task to backup controller's task list in `cluster-controller/controllers/backup/controller.go` 14. Add CRRF chart path to `config.BRConfig` struct ### Phase 6: Configuration 15. Add crrf-daemon chart path to cluster-controller config (similar to `ClusterAgent.ChartPath`) ### Phase 7: Testing 16. Unit tests for cluster-server EnableCRRF/DisableCRRF/GetCRRF handlers 17. Unit tests for CRRF task Create/Delete logic 18. Unit tests for CRRFDaemonConfig helm values generation 19. Integration test for the full Enable→Running→Disable→Destroyed flow --- ## 10. Files to Modify/Create ### New Files | File | Purpose | |---|---| | `helm-charts/crrf-daemon/Chart.yaml` | Helm chart metadata | | `helm-charts/crrf-daemon/values.yaml` | Default values for crrf-daemon | | `helm-charts/crrf-daemon/templates/_helpers.tpl` | Template helpers | | `helm-charts/crrf-daemon/templates/deployment.yaml` | Deployment spec | | `helm-charts/crrf-daemon/templates/service.yaml` | Service for metrics port | | `helm-charts/crrf-daemon/templates/vmservicescrape.yaml` | VictoriaMetrics scrape config | | `cluster-controller/controllers/backup/task/crrf.go` | CRRF task implementation | | `cluster-controller/controllers/backup/task/crrf_config.go` | Helm package config for CRRF | ### Modified Files | File | Change | |---|---| | `api/tidbcloud/cluster-server/v1/services.proto` | Add 3 RPCs to BRService | | `api/tidbcloud/cluster-server/v1/resources.proto` | Add request/response messages | | `infra-api/api/v1/backup_types.go` | Add CRRF annotation constants + CRRFSpec/CRRFStatus types | | `cluster-server/server/br.go` | Implement EnableCRRF/DisableCRRF/GetCRRF handlers | | `cluster-server/server/cvt/br.go` | Add CRRFStatus2PB conversion | | `cluster-controller/controllers/backup/controller.go` | Add CRRF task to task list | | `cluster-controller config` | Add crrf-daemon chart path | --- ## 11. Verification 1. **Proto lint**: `cd api && make lint-proto` — verify proto definitions are valid 2. **cluster-server build**: `cd cluster-server && make build` — verify compilation 3. **cluster-server unit tests**: `cd cluster-server && make unit` — verify handler tests pass 4. **cluster-controller build**: `cd cluster-controller && make build` — verify compilation 5. **cluster-controller unit tests**: `cd cluster-controller && make unit` — verify CRRF task tests pass 6. **Helm template lint**: `helm lint helm-charts/crrf-daemon/` — verify chart is valid 7. **Helm template render**: `helm template test helm-charts/crrf-daemon/ --set upstreamStorage=s3://test,taskName=test,pdAddr=pd:2379` — verify rendered manifests 8. **Manual E2E**: Deploy in dev environment, call EnableCRRF → verify daemon pods running → DisableCRRF → verify cleanup --- ## 12. Scope ### In Scope - Proto API definition for EnableCRRF/DisableCRRF/GetCRRF - cluster-server handler implementation (annotation management) - cluster-controller CRRF task (helm install/uninstall) - crrf-daemon Helm chart (deployment + service + VMServiceScrape) - Unit tests for new code ### Out of Scope - crrf-daemon application logic (BR's `operator crr-checkpoint` command itself) - Health monitoring / auto-recovery for crrf-daemon - RBAC permissions for crrf-daemon ServiceAccount (if needed, add later) - Multiple CRRFs per log backup (1:N not needed now) - CRR S3 bucket replication setup (separate infrastructure concern)