# Auto-CRRF E2E Test Cases Design

## Overview

Add per-track E2E test cases for the Auto-CRRF feature under `repos/dataflow-service/test/e2e/cases/dedicated/crrf/`. Tracks 1 and 2 from the [Auto-CRRF Feature Matrix Test Plan](https://pingcap.feishu.cn/wiki/FwuLw5jDYiDjwokb9Z1cfXBOnpb) get their own test case file.

## Scope

| Track | File | Test Case Name |
|-------|------|----------------|
| 1 — DR-trigger path | `crrf_auto_dr_trigger.go` | `AWSCRRFAutoDRTrigger` |
| 2 — PiTR-trigger path | `crrf_auto_pitr_trigger.go` | `AWSCRRFAutoPiTRTrigger` |

Both use framework-created AWS clusters.

## Track 1 — DR-Trigger Path (`crrf_auto_dr_trigger.go`)

**Preconditions:** Fresh AWS cluster, PiTR=OFF, DR=OFF, CRRF not enabled.

**Sequence:**

```
 1. Capture original backup setting
 2. Enable PiTR
 3. WaitForBackupSettingReady
 4. Trigger snapshot backup (activates log backup)
 5. Enable dual-region backup (with secondary region S1)
 6. WaitForBackupSettingReady
 7. Expect: CRRF enabled
 8. Disable dual-region backup
 9. WaitForBackupSettingReady
10. Expect: CRRF disabled
11. Re-enable dual-region backup (same secondary region S1)
12. WaitForBackupSettingReady
13. Expect: CRRF enabled
14. Disable dual-region backup
15. WaitForBackupSettingReady
16. Expect: CRRF disabled
17. Re-enable dual-region backup (different secondary region S2)
18. WaitForBackupSettingReady
19. Trigger snapshot backup
20. Expect: CRRF enabled (with S2)
```

**Key points:**
- Steps 5-10 and 11-16 test DR enable/disable cycling with the same secondary region
- Steps 17-20 test switching to a different secondary region
- `WaitForBackupSettingReady` (poll until `status == Normal`) is called after every PiTR/DR change, following the pattern in `crrf_enable_disable_smoke.go`
- `DisableDualRegion` calls `ModifyBackupSetting` with `Regions: [{Name: primary}]`
- Secondary regions S1 and S2 resolved via `GetBackupOptions`; S2 picked by excluding S1
- If only 1 secondary region available, steps 17-20 are skipped with a log message

**Cleanup:** Restore original backup setting (single region, PiTR off, CRRF disabled if still enabled).

## Track 2 — PiTR-Trigger Path (`crrf_auto_pitr_trigger.go`)

**Preconditions:** Fresh AWS cluster, PiTR=OFF, DR=OFF, CRRF not enabled.

**Sequence:**

```
1. Capture original backup setting
2. Enable dual-region backup
3. WaitForBackupSettingReady
4. Enable PiTR
5. WaitForBackupSettingReady
6. Trigger snapshot backup
7. Expect: CRRF enabled
8. WaitForBackupSettingReady
9. Disable PiTR
10. WaitForBackupSettingReady
11. Expect: CRRF disabled
```

**Key points:**
- Enables DR first, then PiTR — the PiTR enable DAG's `AutoCRRFEnableFromPit` fires because both conditions (PiTR=ON, DR=ON) are met after the snapshot
- `WaitForBackupSettingReady` after every state change ensures the system is settled before the next operation

**Cleanup:** Restore original backup setting.

## New Helper: `DisableDualRegion`

Add to `BackupValidator` (`cases/dedicated/common/backup_validator.go`):

```go
func (v *BackupValidator) DisableDualRegion(tidbCluster *TiDBDedicatedEntity) error {
    req := &consoleapi.ModifyBackupSettingRequest{
        Regions: []*consoleapi.RegionInfoRequest{
            {Name: tidbCluster.Region},
        },
    }
    return v.backupW.ModifyBackupSetting(tidbCluster, req)
}
```

Downgrades from dual-region to single-region by passing only the primary region.

## New Helper: `GetDifferentSecondaryRegion`

Add to `BackupValidator`:

```go
func (v *BackupValidator) GetDifferentSecondaryRegion(tidbCluster *TiDBDedicatedEntity, excludeRegion string) (string, error) {
    params := console_br_params.NewGetBackupOptionsParamsWithContext(log.WithLogger(context.Background(), v.lgr)).
        WithOrgID(tidbCluster.OrgID).
        WithProjectID(tidbCluster.ProjectID).
        WithClusterID(tidbCluster.ClusterID)

    resp, err := consoleapi.BrBackupServiceClient().GetBackupOptions(params, nil)
    if err != nil {
        return "", err
    }

    if resp.Payload == nil || resp.Payload.DualRegionOptions == nil {
        return "", fmt.Errorf("no dual_region_options for cluster %s", tidbCluster.ClusterID)
    }

    for _, opt := range resp.Payload.DualRegionOptions.RegionOptions {
        if opt != nil && opt.Region != nil && opt.Region.Name != "" && opt.Region.Name != excludeRegion {
            return opt.Region.Name, nil
        }
    }
    return "", fmt.Errorf("no secondary region other than %s available for cluster %s", excludeRegion, tidbCluster.ClusterID)
}
```

Used by Track 1 step 17 to get a second secondary region S2 ≠ S1. If only 1 secondary region is available, the step is skipped.

## Case Registration

In `cases_registry_dedicated.go`, add:

```go
// Auto-CRRF trigger-path cases
awsCRRFAutoDRTriggerCase := crrf.NewCRRFAutoDRTriggerCase(AWS)
awsCRRFAutoPiTRTriggerCase := crrf.NewCRRFAutoPiTRTriggerCase(AWS)
addCases(awsCRRFAutoDRTriggerCase, awsCRRFAutoPiTRTriggerCase)
```

Add both to `awsWeeklyRegression` group.

## Verification Strategy

All verification uses the **backup setting API's `crrf_status` field**:

| Expected | Check | Timing |
|----------|-------|--------|
| CRRF triggers | `ExpectCRRFStatus(cluster, "enabled")` | Poll up to 15 min (30 × 30s) |
| CRRF disables | `ExpectCRRFStatus(cluster, "disabled")` | Poll up to 10 min (30 × 20s) |

After each PiTR/DR change, `WaitForBackupSettingReady` (poll until `status == Normal`) ensures the system is settled before the next operation or verification.

No metadb or kubectl access required.

## Cleanup Pattern

Each test case captures original backup settings at the start and restores them in a `defer` block with retry logic, following the pattern in `crrf_enable_disable_smoke.go`:

1. Capture original state (PiTR, regions) at test start
2. `defer` cleanup restores original state with `retry.Attempts(3)`
3. If CRRF still enabled at cleanup, disable via `CRRFValidator.DisableIfEnabled`
4. Restore regions to original single-region config
5. Restore PiTR to original state
6. `WaitForBackupSettingReady` to converge to `Normal`

## Files Changed

| File | Action |
|------|--------|
| `cases/dedicated/crrf/crrf_auto_dr_trigger.go` | **New** — Track 1 |
| `cases/dedicated/crrf/crrf_auto_pitr_trigger.go` | **New** — Track 2 |
| `cases/dedicated/common/backup_validator.go` | **Edit** — Add `DisableDualRegion`, `GetDifferentSecondaryRegion` |
| `internal/case_driver/cases_registry_dedicated.go` | **Edit** — Register 2 new cases |