# Design: Refactor RTC Enable/Disable into Backup Setting Layer

**Date:** 2026-05-18  
**Branch:** feat/rtc-replication-billing  
**PR:** [#3394](https://github.com/tidbcloud/dataflow-service/pull/3394)

---

## Problem

RTC (Replication Time Control) enable/disable logic currently lives as standalone CRRF workflow actions (`enable_rtc.go`, `disable_rtc.go`). This violates the intended architecture:

- RTC is a property of the S3 bucket replication configuration, which belongs in the `backup_setting` model layer
- RTC state should be tracked on `BackupSetting.Bucket` alongside `DualRegionEnable`
- The CRRF layer should be thin and delegate to `ModifyRegionSetting`

---

## Goals

1. Track per-bucket AWS RTC state in `AWSBucketConfiguration`
2. Extend `RegionSettingModificationReqV2` to carry `EnableRTC` intent
3. Make `awsEnableDualRegionHook` and `awsDisableDualRegionHook` manage RTC state in `BucketSetting`
4. Move the actual AWS `ModifyReplicationRTC` call into the `DagEnableDualRegionBackupAWS` workflow (as a new `ActionEnableReplicationRTC` step) — **not** inside a DB transaction
5. Support re-entrant RTC toggle (when dual-region is already active) via a lightweight `DagModifyReplicationRTC`
6. Delete standalone `enable_rtc.go` and `disable_rtc.go` CRRF actions
7. Metering continues to work without changes to `getRTCEnabled()`

---

## Constraints

- `EnableRTC` is AWS-only
- `EnableRTC=true` requires `Enable=true` OR dual-region already active (re-entrant)
- Disabling dual-region auto-destroys RTC (AWS deletes the replication rule) — no explicit AWS disable call needed
- AWS API calls must NOT run inside DB transactions (hooks run inside transactions)

---

## Non-Goals

- GCP / Azure RTC support (not applicable)
- Changes to how billing records anchor their start time
- Changing `getRTCEnabled()` fallback logic in the metering code

---

## Design

### 1. Entity Layer

**File:** `app/repo/entity/entity.go`

Add `RTCEnabled *bool` to `AWSBucketConfiguration`. Pointer type so `nil` represents unknown/legacy buckets.

```go
type AWSBucketConfiguration struct {
    BucketPolicy            AWSBucketPolicy            `json:"bucket_policy,omitempty"`
    VersioningEnable        bool                       `json:"versioning_enable,omitempty"`
    GCRuleEnable            bool                       `json:"gc_rule_enable,omitempty"`
    BucketReplicationPolicy AWSBucketReplicationPolicy `json:"bucket_replication_policy,omitempty"`
    RTCEnabled              *bool                      `json:"rtc_enabled,omitempty"` // ← NEW
}
```

`AWSBucketReplicationPolicy` (`RuleIDs []string`) is unchanged — RTC state lives on the configuration, not the policy.

`BucketSetting.RTCEnabled bool` already exists (line 1133) and is used for billing snapshots — no change.

---

### 2. Domain Layer

**File:** `app/models/domain/backupv2.go`

Add `EnableRTC bool` to `RegionSettingModificationReqV2`:

```go
type RegionSettingModificationReqV2 struct {
    ClusterID uint64
    Enable    bool
    EnableRTC bool      // only effective for AWS + Enable=true or re-entrant case
    Regions   []string
    DryRun    bool
}
```

---

### 3. Model Layer — Hooks

**File:** `app/models/backup/impl/region_modification.go`

#### `awsEnableDualRegionHook`

**`paramCheck`** — add RTC validation:
```go
if req.EnableRTC && !req.Enable && !setting.Bucket.DualRegionEnable {
    return bizerr: "EnableRTC requires Enable=true or dual-region already active"
}
```

**`updateEntity`** — record RTC intent in `BucketSetting`:
```go
bucketSetting := &entity.BucketSetting{
    DualRegionEnable: true,
    RTCEnabled:       req.EnableRTC, // ← NEW
    Buckets:          ...,
}
```

> **Edge case:** There is no standalone "disable RTC while keeping dual-region" operation. `EnableRTC=false` means "don't enable RTC"; it does NOT disable RTC if already active (no AWS API exists for this). Callers must never pass `EnableRTC=false` on a cluster that already has RTC active — that would incorrectly set `RTCEnabled=false` in DB while AWS still has RTC on. To disable RTC, callers must disable dual-region (`Enable=false`), which triggers `awsDisableDualRegionHook` and auto-removes RTC at the AWS level.

**`buildDagVars`** — branch based on current state:

```go
// Re-entrant: dual-region already active, only RTC is changing
if req.oldBackupSetting.Bucket != nil && req.oldBackupSetting.Bucket.DualRegionEnable {
    return workflow.DagModifyReplicationRTC, map[string]string{
        workflow.KeyBackupSettingId: fmt.Sprintf("%d", newBackupSetting.ID),
        workflow.KeyEnableRTC:       strconv.FormatBool(req.EnableRTC),
    }, nil
}

// Normal: full dual-region setup
return workflow.DagEnableDualRegionBackupAWS, map[string]string{
    workflow.KeyBackupSettingId: ...,
    workflow.KeyRegion:          ...,
    workflow.KeySecondaryRegion: ...,
    workflow.KeyNetworkNodeRole: ...,
    workflow.KeyEnableRTC:       strconv.FormatBool(req.EnableRTC), // ← NEW
}, nil
```

#### `awsDisableDualRegionHook`

**`updateEntity`** — explicitly clear `RTCEnabled`:
```go
bucketSetting := &entity.BucketSetting{
    DualRegionEnable: false,
    RTCEnabled:       false, // ← NEW: AWS replication rule deletion auto-destroys RTC
    Buckets: []entity.BackupSettingBucket{{Regions: []string{req.Regions[0]}, Master: true}},
}
```

No AWS API call for RTC disable — the `DagDisableDualRegionBackupAWS` deletes the replication rule, which implicitly removes RTC.

---

### 4. Workflow Constants

**File:** `app/workflow/constants.go`

**Add:**
```go
KeyEnableRTC               = "enable_rtc"
ActionEnableReplicationRTC = "enable_replication_rtc"
DagModifyReplicationRTC    = "dag_modify_replication_rtc"
```

**Remove:**
```go
ActionCRRFEnableRTC   // was line 155
ActionCRRFDisableRTC  // was line 161
```

---

### 5. New Action: `ActionEnableReplicationRTC`

**File:** `app/workflow/action/backup_setting/enable_dual_region/enable_replication_rtc.go`

**Params:** `KeyBackupSettingId`, `KeyEnableRTC`

**`Run()`** logic:
1. Parse `KeyEnableRTC` — if `false`, return nil (no-op; supports `EnableRTC=false` in DAG var)
2. Load `BackupSetting` by ID
3. Find master `Bucket` entity
4. Call `awsBucketManager.ModifyReplicationRTC(ctx, {BucketID: masterBucketID, EnableRTC: true})`
5. Update `Bucket.Configuration.RTCEnabled = ptr(true)` — sets `AWSBucketConfiguration.RTCEnabled`
6. Call `Bucket.SetRTCEnabled(true)` — preserves legacy `Extension["rtc_enabled"]` fallback for metering
7. Save bucket entity

**Idempotency:** Check `AWSBucketConfiguration.RTCEnabled != nil && *AWSBucketConfiguration.RTCEnabled` before calling AWS to skip redundant calls.

---

### 6. `DagEnableDualRegionBackupAWS` (updated)

**File:** `app/workflow/dag/backup_setting/aws/enable_dual_region.go`

Add `KeyEnableRTC` to DAG vars. Insert `ActionEnableReplicationRTC` between `CreateBucketReplication` and `EnableDualRegionFinalStep`:

```
CreateAndCheckBucketReady (master + secondary)
  ↓
AddNodeRoleToBucketPolicy (master + secondary)
  ↓
CreateBucketReplication
  ↓
ActionEnableReplicationRTC   ← NEW (DependOn: CreateBucketReplication)
  ↓
EnableDualRegionFinalStep    (DependOn: ActionEnableReplicationRTC)
```

Task definition:
```go
{
    ID:          workflow.ActionEnableReplicationRTC,
    ActionName:  workflow.ActionEnableReplicationRTC,
    Params: map[string]interface{}{
        workflow.KeyBackupSettingId: fmt.Sprintf("{{%v}}", workflow.KeyBackupSettingId),
        workflow.KeyEnableRTC:       fmt.Sprintf("{{%v}}", workflow.KeyEnableRTC),
    },
    DependOn:    []string{workflow.ActionCreateBucketReplication},
    TimeoutSecs: 5 * 60,
},
```

`EnableDualRegionFinalStep` task's `DependOn` changes from `[ActionCreateBucketReplication]` to `[ActionEnableReplicationRTC]`.

---

### 7. New DAG: `DagModifyReplicationRTC`

**File:** `app/workflow/dag/backup_setting/aws/modify_replication_rtc.go`

Lightweight single-task DAG for the re-entrant RTC-only toggle:

```go
func GetModifyReplicationRTCDag() (*entity.Dag, []run.Action) {
    c := container.Global()
    actions := []run.Action{
        backup_setting_enable.NewEnableReplicationRTCAction(c.BackupSettingRepo, c.BucketRepo, c.AWSBucketManagerModel),
    }
    dag := &entity.Dag{
        BaseInfo: entity.BaseInfo{ID: workflow.DagModifyReplicationRTC},
        Name:     workflow.DagModifyReplicationRTC,
        Vars:     entity.DagVars{
            workflow.KeyBackupSettingId: {},
            workflow.KeyEnableRTC:       {},
        },
        Status: entity.DagStatusNormal,
        Tasks: []entity.Task{{
            ID:          workflow.ActionEnableReplicationRTC,
            ActionName:  workflow.ActionEnableReplicationRTC,
            Params: map[string]interface{}{
                workflow.KeyBackupSettingId: fmt.Sprintf("{{%v}}", workflow.KeyBackupSettingId),
                workflow.KeyEnableRTC:       fmt.Sprintf("{{%v}}", workflow.KeyEnableRTC),
            },
            TimeoutSecs: 5 * 60,
        }},
    }
    return dag, actions
}
```

This DAG must be registered in the workflow startup alongside other backup_setting DAGs.

---

### 8. CRRF Enable Action (updated)

**File:** `app/workflow/action/crrf/enable/enable_dual_region.go`

**`Run()`** changes:

```go
// Old: skip if DualRegionEnable == true
// New: skip only if dual-region AND RTC are both already in desired state
if backupSetting.Bucket != nil && backupSetting.Bucket.DualRegionEnable && backupSetting.Bucket.RTCEnabled {
    logger.Info("dual-region + RTC already configured, skipping trigger")
    return nil
}

// Pass EnableRTC: true (hardcoded for CRRF — CRRF always enables RTC)
_, bizErr := a.backupSettingModel.ModifyRegionSetting(ctx, &domain.RegionSettingModificationReqV2{
    ClusterID: ...,
    Enable:    true,
    EnableRTC: true,   // ← NEW
    Regions:   ...,
})
```

**`RunAfter()`** — unchanged. It polls until `DualRegionEnable=true` + status Normal, then caches bucket IDs into `CRRFSetting`. By that point, `ActionEnableReplicationRTC` inside `DagEnableDualRegionBackupAWS` has already succeeded.

---

### 9. CRRF DAG Cleanup

**File:** `app/workflow/dag/crrf/enable.go`

Remove `ActionCRRFEnableRTC` task. Update `ActionCRRFEnableDaemon.DependOn`:

```
Before: enable_dual_region → enable_pitr → enable_rtc → enable_daemon → ...
After:  enable_dual_region → enable_pitr → enable_daemon → ...
```

Remove `crrf_enable.NewEnableRTCAction(...)` from the actions slice.

**File:** `app/workflow/dag/crrf/disable.go`

Remove `ActionCRRFDisableRTC` task. Update `ActionCRRFMarkDisabled.DependOn`:

```
Before: ... → disable_daemon → disable_rtc → mark_disabled
After:  ... → disable_daemon → mark_disabled
```

Remove `crrf_disable.NewDisableRTCAction(...)` from the actions slice.

**Delete:**
- `app/workflow/action/crrf/enable/enable_rtc.go`
- `app/workflow/action/crrf/disable/disable_rtc.go`

---

### 10. Metering — No Change Required

**File:** `app/models/backup/impl/backup_billing_record_generator.go`

`getRTCEnabled()` currently:
1. Reads `BucketSettingSnapshot.RTCEnabled` (set by hook's `updateEntity` into `BucketSetting.RTCEnabled`)
2. Falls back to `bucket.GetRTCEnabled()` (from `Bucket.Extension["rtc_enabled"]`)

After the refactor:
- Path 1 is now **correctly populated** because `awsEnableDualRegionHook.updateEntity` sets `BucketSetting.RTCEnabled = req.EnableRTC`
- Path 2 remains valid because `ActionEnableReplicationRTC` still calls `bucket.SetRTCEnabled(true)` — preserving the legacy fallback

**No change to `getRTCEnabled()` or `generateMeteringInfoForReplicationRTC`.**

---

## Change Inventory

| File | Type | Change |
|------|------|--------|
| `app/repo/entity/entity.go` | Modify | Add `RTCEnabled *bool` to `AWSBucketConfiguration` |
| `app/models/domain/backupv2.go` | Modify | Add `EnableRTC bool` to `RegionSettingModificationReqV2` |
| `app/models/backup/impl/region_modification.go` | Modify | Extend `paramCheck`, `updateEntity`, `buildDagVars` on both hooks |
| `app/workflow/constants.go` | Modify | Add 3 constants; remove `ActionCRRFEnableRTC`, `ActionCRRFDisableRTC` |
| `app/workflow/action/crrf/enable/enable_dual_region.go` | Modify | Fix early-return guard; hardcode `EnableRTC: true` in `ModifyRegionSetting` |
| `app/workflow/action/backup_setting/enable_dual_region/enable_replication_rtc.go` | **Create** | New `ActionEnableReplicationRTC` action |
| `app/workflow/dag/backup_setting/aws/enable_dual_region.go` | Modify | Add `KeyEnableRTC` var; add `ActionEnableReplicationRTC` task |
| `app/workflow/dag/backup_setting/aws/modify_replication_rtc.go` | **Create** | New `DagModifyReplicationRTC` |
| `app/workflow/dag/crrf/enable.go` | Modify | Remove `ActionCRRFEnableRTC` task; fix dependency chain |
| `app/workflow/dag/crrf/disable.go` | Modify | Remove `ActionCRRFDisableRTC` task; fix dependency chain |
| `app/workflow/action/crrf/enable/enable_rtc.go` | **Delete** | — |
| `app/workflow/action/crrf/disable/disable_rtc.go` | **Delete** | — |

---

## Risk Assessment

- **Low risk:** Entity/domain changes are additive (new optional fields)
- **Medium risk:** `buildDagVars` branching logic — the re-entrant path must correctly detect `oldBackupSetting.Bucket.DualRegionEnable == true`
- **Medium risk:** DAG task insertion in `DagEnableDualRegionBackupAWS` — `EnableDualRegionFinalStep` dependency chain must be updated correctly
- **Low risk:** Metering — no changes; both fallback paths continue to work