# Auto-CRRF Feature Design

**Date:** 2026-05-20  
**Status:** Approved  
**Author:** Brainstorming session

---

## Problem

When both PiTR and dual-region backup are simultaneously active on a Dedicated cluster, CRRF should be automatically enabled. Conversely, when either is disabled, CRRF should be automatically disabled. Today this requires manual operator intervention.

---

## Solution Overview

Insert new auto-CRRF trigger actions as the **last step** of four existing DAGs:

| Trigger DAG | Action |
|-------------|--------|
| `DagEnablePitBackup` | Auto-enable CRRF if dual-region is already active |
| `DagDisablePitBackup` | Auto-disable CRRF if active |
| `DagEnableDualRegionBackupAWS` | Auto-enable CRRF if PiTR is already active |
| `DagDisableDualRegionBackupAWS` | Auto-disable CRRF if active |

Each action uses the **Run/RunAfter** pattern to wait for the downstream CRRF DAG to complete.

Additionally, the CRRF enable DAG is simplified by removing the `enable_dual_region` and `enable_pitr` steps, since both are now guaranteed prerequisites.

---

## Design Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Trigger placement | **Second-to-last step** (before the "set status to normal" final step) | Backup setting stays locked (`editing_pitr` / `editing_region`) while CRRF DAG runs; prevents concurrent user changes |
| When conditions not met | Skip silently (log warning, DAG succeeds) | Avoids breaking existing flows when cluster is GCP, wrong version, etc. |
| Wait for CRRF DAG? | Yes — Run/RunAfter pattern for both enable and disable | Ensures consistency before parent DAG is marked complete |
| CRRF is already active during enable trigger? | Skip silently | Idempotent |
| Existing PiTR-disable guard | Remove | Auto-disable in `DagDisablePitBackup` makes it redundant |
| Action package location | `app/workflow/action/crrf/auto/` | Consistent with CRRF action convention |
| `EnableCRRF` status check | Relax to also allow `editing_pitr` and `editing_region` | Auto-CRRF runs while backup setting is still in editing state; the old "must be normal" check would block it |

---

## 1. New Action Package

**Location:** `app/workflow/action/crrf/auto/`

### 1.1 `enable.go`

Two action structs sharing a `tryAutoEnableCRRF()` helper:

**`AutoCRRFEnableFromPitAction`** (for `DagEnablePitBackup`)
- Embeds `backup/base.BaseAction` to use `QueryPitBackupSwitch` → ClusterID → BackupSetting

**`AutoCRRFEnableFromDualRegionAction`** (for `DagEnableDualRegionBackupAWS`)
- Receives `backup_setting_id` as task param; queries BackupSetting by ID

**Shared `Run()` logic (`tryAutoEnableCRRF`):**
1. Check `InfraProviderType` is AWS/EKS — skip if not
2. Load cluster via `CoreGrpcClient.GetAnyClusterByID()`
3. Check feature flag: `cfg.CRRFOverrides.GetMatched(...).AutoCRRF` must be `true` — skip if not
4. Check version `>= CRRFEnableVersionConstraint` (config default: `>=v8.5.6`) — skip if below
5. Check the *other* feature is enabled:
   - For PiTR action: `backupSetting.Bucket != nil && backupSetting.Bucket.DualRegionEnable`
   - For dual-region action: `backupSetting.PiTR != nil && backupSetting.PiTR.Enable`
6. Query `CRRFModel.QueryCRRF()`; skip if status is already `enabling|enabled`
7. Extract secondary region: call `backupSetting.Bucket.SecondaryRegions()`, take `[0]`
8. Call `CRRFModel.EnableCRRF(ctx, &EnableCRRFReq{ClusterID, TenantID, ProjectID, SecondaryRegion, SkipEnablePiTRAndDualRegion: true})`
9. Call `markAutoCRRFEnableTriggered(eCtx)` to signal `poll()` that enable was triggered
10. Return `nil` → Wrapper proceeds to `RunAfter`

**Skip path:** return `nil` from `run()` — the action succeeds without triggering `poll()` (which checks `isAutoCRRFEnableTriggered()` before polling).

**`RunAfter()` logic:**
1. If `isAutoCRRFEnableTriggered()` is false (enable was skipped) → return `nil` immediately
2. Call `CRRFModel.QueryCRRF(ctx, clusterID)`
3. If `status == "enabled"` → return `nil` (stops retry loop)
4. If `status == "disabled"` with non-empty `ErrorMessage` → return `retry.Unrecoverable(...)` (terminal failure)
5. If `status == "disabled"` with empty `ErrorMessage` → return `nil` (treat as no-op success — CRRF was reset cleanly)
6. Otherwise return transient error → Wrapper retries with exponential backoff

### 1.2 `disable.go`

Two action structs sharing a `tryAutoDisableCRRF()` helper:

**`AutoCRRFDisableFromPitAction`** + **`AutoCRRFDisableFromDualRegionAction`** (same param source pattern as enable).

**Shared `Run()` logic (`tryAutoDisableCRRF`):**
1. Call `CRRFModel.QueryCRRF(ctx, clusterID)`
2. If CRRF absent or `status == "disabled"` → log info, return `nil` (skip silently)
3. If `status == "disabling"` → log info, return `nil` (already in progress; `poll()` will wait for completion)
4. Call `CRRFModel.DisableCRRF(ctx, &DisableCRRFReq{ClusterID, TenantID, ProjectID})`
   - Note: `DisableCRRF` only accepts `status == "enabled"`; if CRRF is in `enabling` or `failed` state when the disable trigger fires, `DisableCRRF` returns an error and the task fails — which is intentional behavior for this edge case
5. Return `nil` → Wrapper proceeds to `RunAfter`

**`RunAfter()` logic:**
1. Call `CRRFModel.QueryCRRF(ctx, clusterID)`
2. If `status == "disabled"` → return `nil` (stops retry loop)
3. If `status == "failed"` → return `retry.Unrecoverable(...)` (terminal failure)
4. Otherwise return transient error → retry

### 1.3 Retry Strategy

Both files use `invokeRunAfterWithSlowExponentialRetry` (copied from `enable_pitr.go`):
- 15 s initial delay, exponential backoff up to 5 min cap, unlimited attempts, stops on `action.StopRetry`

---

## 2. Configuration Changes

**File:** `internal/config/config.go` — add to `BackupConfig`:

```go
CRRFEnableVersionConstraintStr string              `mapstructure:"crrfEnableVersionConstraint" default:">=v8.5.6"`
CRRFEnableVersionConstraint    *semver.Constraints // parsed from CRRFEnableVersionConstraintStr at startup
CRRFOverrides                  CRRFOverrides       `mapstructure:"crrfOverrides"`
```

Default value: `">=v8.5.6"` (same parsing pattern as `LogBackupEnableMinVerConstraint`).

**`CRRFOverride` struct** (feature-flag mechanism per cluster/tenant/project/region):

```go
type CRRFOverride struct {
    FeatureGate
    // AutoCRRF enables automatic CRRF enable/disable triggered by PiTR and dual-region backup changes.
    AutoCRRF bool `mapstructure:"autoCRRF" default:"false"`
}
```

Auto-CRRF is gated behind `AutoCRRF: true` in a matching `CRRFOverride` entry. This allows controlled rollout per tenant/cluster before enabling globally.

---

## 3. CRRF Enable Model Preconditions

**File:** `app/models/crrf/impl/crrf.go` — in `EnableCRRF()`, make two changes:

### 3.1 Relax backup setting status check (existing check, lines 65–68)

The current check requires `normal` status. Relax it to also accept `editing_pitr` and `editing_region`, since auto-CRRF now fires while the parent DAG has the backup setting locked in an editing state:

```go
// Replace:
if backupSetting.Status != entity.BackupSettingStatusNormal {
    return bizerr_wrapper.NewInvalidInputError(
        "backup setting is in %s state, must be normal to enable CRRF", backupSetting.Status)
}

// With:
allowedStatuses := map[entity.BackupSettingStatus]bool{
    entity.BackupSettingStatusNormal:        true,
    entity.BackupSettingStatusEditingRegion: true,
    entity.BackupSettingStatusEditingPitr:   true,
}
if !allowedStatuses[backupSetting.Status] {
    return bizerr_wrapper.NewInvalidInputError(
        "backup setting is in %s state, must be normal/editing_pitr/editing_region to enable CRRF", backupSetting.Status)
}
```

### 3.2 Add PiTR + dual-region preconditions (new checks, after step 1)

After the relaxed status check and the existing AWS check, add:

```go
if backupSetting.PiTR == nil || !backupSetting.PiTR.Enable {
    return bizerr_wrapper.NewInvalidInputError("PiTR must be enabled before CRRF can be enabled")
}
if backupSetting.Bucket == nil || !backupSetting.Bucket.DualRegionEnable {
    return bizerr_wrapper.NewInvalidInputError("dual-region backup must be enabled before CRRF can be enabled")
}
```

This enforces that all callers (auto and manual) respect the invariant.

### 3.3 `EnableCRRFReq` extension

`domain.EnableCRRFReq` gains a new field:

```go
// SkipEnablePiTRAndDualRegion instructs the enable_crrf DAG to skip the
// enable_dual_region and enable_pitr tasks (they are already active).
SkipEnablePiTRAndDualRegion bool
```

The auto-CRRF path always sets this to `true`. Manual API calls leave it `false` (default).

> **Note:** Both `PiTR.Enable` and `DualRegionEnable` are written to the DB at the *start* of their respective enable/disable requests (before the DAG is launched), so they reflect the intended final state when the auto-CRRF step reads them mid-DAG.

---

## 4. CRRF Enable DAG — Skip Flag

**File:** `app/workflow/dag/crrf/enable.go`

The `enable_dual_region` and `enable_pitr` tasks are **kept** in the DAG. A new boolean DAG var `KeySkipEnablePiTRDualRegion` is added. When the auto-CRRF path calls `EnableCRRF`, it sets `SkipEnablePiTRAndDualRegion: true` in `EnableCRRFReq`, which propagates to the DAG as `KeySkipEnablePiTRDualRegion=true`. Both tasks check this flag and exit early (no-op) when it is set, since PiTR and dual-region are already active.

`KeySecondaryRegion` is **kept** in DAG vars — it is still required by the `enable_dual_region` task for the manual (non-skip) path.

**Updated DAG vars:**
```go
workflow.KeyCRRFSettingID:            {},
workflow.KeyBackupSettingId:          {},
workflow.ClusterID:                   {},
workflow.KeySecondaryRegion:          {},
workflow.KeySkipEnablePiTRDualRegion: {},  // new: "true" → skip enable_dual_region + enable_pitr
```

**DAG sequence** (unchanged step IDs and structure):
```
enable_dual_region  (no-op if KeySkipEnablePiTRDualRegion=true)
  → enable_pitr     (no-op if KeySkipEnablePiTRDualRegion=true)
    → modify_replication_rtc_enable  (synchronous — direct AWS S3 API + transactional DB update)
      → enable_crrf_daemon
        → toggle_periodic_backup_enable
          → mark_enabled
```

> Note: `enable_dual_region.go` and `enable_pitr.go` action files are kept and still registered in the DAG; the skip flag makes them no-ops when invoked from the auto-CRRF path.

---

## 5. Four Trigger DAGs — New Second-to-Last Step

The auto-CRRF action is inserted **before** the "set status to normal" final step in each DAG.
This keeps the backup setting locked in its editing state while the CRRF DAG runs, preventing concurrent user operations.
The final step's `DependOn` is updated to depend on the new auto-CRRF step.

**Workflow constants to add** (`app/workflow/constants.go`):
```go
ActionAutoCRRFEnableFromPit          = "auto_crrf_enable_from_pit"
ActionAutoCRRFDisableFromPit         = "auto_crrf_disable_from_pit"
ActionAutoCRRFEnableFromDualRegion   = "auto_crrf_enable_from_dual_region"
ActionAutoCRRFDisableFromDualRegion  = "auto_crrf_disable_from_dual_region"
```

### `DagEnablePitBackup` (`dag/backup/pit/enbale.go`)

Current final chain: `ActionUpdatePitBackupSwitchStatusFromEnablingToEnabled → ActionUpdatePitBackupSettingStatusToNormal`

New chain:
```
ActionUpdatePitBackupSwitchStatusFromEnablingToEnabled
  → ActionAutoCRRFEnableFromPit           (new, DependOn: [ActionUpdatePitBackupSwitchStatusFromEnablingToEnabled])
    → ActionUpdatePitBackupSettingStatusToNormal  (DependOn updated: [ActionAutoCRRFEnableFromPit])
```

Add to `actions` slice:
```go
auto_crrf.NewAutoCRRFEnableFromPitAction(
    base, container.Global().CRRFModel, container.Global().CoreGrpcClient,
    container.Global().BackupSettingRepo, internal_config.Global().BackupConfig,
)
```

Add new `Task` and update existing final task:
```go
// New task (insert before ActionUpdatePitBackupSettingStatusToNormal):
{
    ID:          workflow.ActionAutoCRRFEnableFromPit,
    ActionName:  workflow.ActionAutoCRRFEnableFromPit,
    DependOn:    []string{workflow.ActionUpdatePitBackupSwitchStatusFromEnablingToEnabled},
    TimeoutSecs: 60 * 60, // 1 hour
},
// Update existing final task DependOn:
{
    ID:          workflow.ActionUpdatePitBackupSettingStatusToNormal,
    ActionName:  workflow.ActionUpdatePitBackupSettingStatusToNormal,
    DependOn:    []string{workflow.ActionAutoCRRFEnableFromPit},  // was: ActionUpdatePitBackupSwitchStatusFromEnablingToEnabled
    TimeoutSecs: 5 * 60,
},
```

### `DagDisablePitBackup` (`dag/backup/pit/disable.go`)

Current final chain: `ActionUpdatePitBackupSwitchStatusToDisabled → ActionUpdatePitBackupSettingStatusToNormal`

New chain:
```
ActionUpdatePitBackupSwitchStatusToDisabled
  → ActionAutoCRRFDisableFromPit          (new, DependOn: [ActionUpdatePitBackupSwitchStatusToDisabled])
    → ActionUpdatePitBackupSettingStatusToNormal  (DependOn updated: [ActionAutoCRRFDisableFromPit])
```

Same pattern as enable, using `AutoCRRFDisableFromPitAction`, `TimeoutSecs: 40 * 60`.

### `DagEnableDualRegionBackupAWS` (`dag/backup_setting/aws/enable_dual_region.go`)

Current final chain: `ActionCreateBucketReplication → ActionEnableDualRegionBucketFinalStep`

New chain:
```
ActionCreateBucketReplication
  → ActionAutoCRRFEnableFromDualRegion    (new, DependOn: [ActionCreateBucketReplication])
    → ActionEnableDualRegionBucketFinalStep       (DependOn updated: [ActionAutoCRRFEnableFromDualRegion])
```

Add to `actions` slice:
```go
auto_crrf.NewAutoCRRFEnableFromDualRegionAction(
    container.Global().CRRFModel, container.Global().CoreGrpcClient,
    container.Global().BackupSettingRepo, internal_config.Global().BackupConfig,
)
```

Add new `Task` and update existing final task:
```go
// New task:
{
    ID:         workflow.ActionAutoCRRFEnableFromDualRegion,
    ActionName: workflow.ActionAutoCRRFEnableFromDualRegion,
    Params: map[string]interface{}{
        workflow.KeyBackupSettingId: fmt.Sprintf("{{%v}}", workflow.KeyBackupSettingId),
    },
    DependOn:    []string{workflow.ActionCreateBucketReplication},
    TimeoutSecs: 60 * 60,
},
// Update existing final task DependOn:
{
    ID:         workflow.ActionEnableDualRegionBucketFinalStep,
    ActionName: workflow.ActionEnableDualRegionBucketFinalStep,
    Params: map[string]interface{}{
        workflow.KeyBackupSettingId: fmt.Sprintf("{{%v}}", workflow.KeyBackupSettingId),
    },
    DependOn:    []string{workflow.ActionAutoCRRFEnableFromDualRegion},  // was: ActionCreateBucketReplication
    TimeoutSecs: 30,
},
```

### `DagDisableDualRegionBackupAWS` (`dag/backup_setting/aws/disable_dual_region.go`)

Current final chain: `ActionAddNodeRoleToBucketPolicy → ActionDisableDualRegionBucketFinalStep`

New chain:
```
ActionAddNodeRoleToBucketPolicy
  → ActionAutoCRRFDisableFromDualRegion   (new, DependOn: [ActionAddNodeRoleToBucketPolicy])
    → ActionDisableDualRegionBucketFinalStep      (DependOn updated: [ActionAutoCRRFDisableFromDualRegion])
```

Same pattern, using `AutoCRRFDisableFromDualRegionAction`, `TimeoutSecs: 40 * 60`.

---

## 6. Guard Removal

**File:** `app/models/backup/impl/backup_setting.go`

In `preCheckModifyPitBackupSetting()`, remove the `else` sub-block that checks CRRF activity before a PiTR disable (lines 842–853):

```go
// REMOVE THIS BLOCK:
crrfSetting, crrfErr := m.crrfSettingRepo.GetByClusterID(ctx, req.ClusterID)
if crrfErr != nil && crrfErr.ErrCode() != bizerr.ErrCodeRecordNotFound {
    ...
}
if crrfSetting != nil && crrfSetting.IsActive() {
    return bizerr_wrapper.NewInvalidInputError("cannot disable PiTR while CRRF is active; disable CRRF first")
}
```

The dry-run disable path that follows (lines 855–860) is **kept**.

---

## 7. Error Handling Summary

| Scenario | Behavior |
|----------|----------|
| Cluster is GCP | Skip silently (warning log) |
| Cluster version < v8.5.6 | Skip silently (warning log) |
| Other feature (PiTR/dual-region) not yet enabled | Skip silently (warning log) |
| CRRF already active on enable trigger | Skip silently (warning log) |
| CRRF absent/disabled on disable trigger | Skip silently (warning log) |
| EnableCRRF/DisableCRRF returns transient error | Run() retried by Wrapper |
| CRRF DAG stuck or terminal error | RunAfter() returns error; parent DAG task fails |
| Manual EnableCRRF without PiTR or dual-region | Fails with `InvalidInputError` (new precondition) |

---

## 8. Files Changed

| File | Change Type |
|------|-------------|
| `app/workflow/action/crrf/auto/enable.go` | **New** |
| `app/workflow/action/crrf/auto/disable.go` | **New** |
| `internal/config/config.go` | **Modified** — add `CRRFEnableMinVer` + constraint |
| `app/models/crrf/impl/crrf.go` | **Modified** — relax status check (`editing_pitr`/`editing_region` allowed); add PiTR + dual-region preconditions |
| `app/workflow/dag/crrf/enable.go` | **Modified** — remove 2 tasks + `KeySecondaryRegion` var |
| `app/workflow/dag/backup/pit/enbale.go` | **Modified** — append auto-enable action + task |
| `app/workflow/dag/backup/pit/disable.go` | **Modified** — append auto-disable action + task |
| `app/workflow/dag/backup_setting/aws/enable_dual_region.go` | **Modified** — append auto-enable action + task |
| `app/workflow/dag/backup_setting/aws/disable_dual_region.go` | **Modified** — append auto-disable action + task |
| `app/workflow/constants.go` | **Modified** — add 4 action name constants |
| `app/models/backup/impl/backup_setting.go` | **Modified** — remove CRRF-active guard in `preCheckModifyPitBackupSetting` |