# Rollback Plan

> Recovery procedures when baseline publication or downstream branch validation fails.

## Purpose

Define clear recovery procedures to:
1. Minimize business disruption
2. Restore the baseline or test branch to a known-good state
3. Preserve audit trail of what happened
4. Enable safe retry after issues are resolved

This document is intentionally written for a pre-production TiDB Cloud branch workflow. The primary recovery mechanism is branch reset / branch recreate from a validated baseline, not emergency production cutover rollback.

---

## Rollback Triggers

### Immediate Rollback Triggers

| Trigger | Severity | Action |
|---------|----------|--------|
| Data corruption detected | CRITICAL | Immediate branch reset / baseline rebuild |
| Security/permission breach | CRITICAL | Immediate branch reset / access-model review |
| Bulk data loss | CRITICAL | Immediate branch reset / baseline rebuild |
| System instability | CRITICAL | Immediate branch reset / investigation |

### Conditional Rollback Triggers

| Trigger | Severity | Action |
|---------|----------|--------|
| Validation failure > 5% records | HIGH | Rebuild baseline recommended |
| Critical flow broken | HIGH | Reset branch and fix baseline inputs |
| Lookup relationship failure > 1% | MEDIUM | Investigate first |
| Performance degradation | MEDIUM | Monitor before rollback |

---

## Pre-Baseline Safeguards

Before publishing or refreshing a baseline, create:

```
1. Baseline branch checkpoint
   - Confirm which TiDB Cloud branch is the current canonical baseline
   - Record the exact branch name / version / timestamp

2. Optional database snapshot
   - TiDB backup via mysqldump or br tool if a persistent backup is desired
   - Store in S3: s3://nexuscrm-migration/backups/pre-baseline_YYYYMMDD/

3. S3 configuration backup
   - Field mapping config
   - Migration state files
   - Loader ledger / manifest files

4. Application state
   - Flow definitions (from pack)
   - Target pack version / install evidence
   - Custom object schemas
```

### Backup Command Template

```bash
# TiDB full backup
tiup br backup db --pd <pd-address> --backup-dir s3://nexuscrm-migration/backups/pre-migration_$(date +%Y%m%d)/

# Or via mysqldump for smaller databases
mysqldump -h <host> -P <port> -u root <database> > pre-migration-$(date +%Y%m%d).sql
```

---

## Recovery Procedure

### Phase 1: Immediate Actions (0-5 minutes)

```
1. STOP ALL MIGRATION PROCESSES
   - Kill running import scripts
   - Disable EC2 scheduled tasks
   - Stop any batch jobs

2. NOTIFY STAKEHOLDERS
   - Send alert: Migration rollback initiated
   - Include: What happened, expected duration, impact

3. PRESERVE CURRENT STATE
   - Capture failing branch name / loader run ID / pack version
   - Snapshot current database state if deeper debugging is needed
   - Document loader ledger position
```

### Phase 2: Restore Baseline or Test Branch (5-30 minutes)

```
1. IDENTIFY RECOVERY TARGET
   - Canonical validated baseline branch
   - Or the most recent known-good persistent backup

2. RESTORE / RECREATE
   - Option A: Reset or recreate the failing TiDB test branch from the canonical baseline
   - Option B: Rebuild the canonical baseline from frozen pack + S3 export + controlled loader
   - Option C: Restore from persistent backup only if branch-based recovery is insufficient

3. VERIFY RESTORE
   - Check record counts match expected
   - Verify key relationships intact
   - Confirm system operational
   - Confirm branch is ready for re-validation
```

### Phase 3: Cleanup (30-60 minutes)

```
1. CLEAN UP FAILED BRANCH OR FAILED BASELINE ATTEMPT
   - Archive or discard the dirty test branch
   - Verify the canonical baseline remains untouched

2. RESET MIGRATION STATE
   - Clear migration lock files
   - Reset S3 state files
   - Clear or archive loader run state

3. VERIFY CLEAN STATE
   - Branching source is healthy
   - No stale branch is still being treated as canonical
```

---

## Rollback Decision Matrix

| Scenario | Rollback Recommended | Alternative |
|----------|---------------------|-------------|
| Dirty test branch only | Reset / recreate branch | - |
| Baseline build failed before publish | Rebuild baseline | - |
| Published baseline found invalid | Replace canonical baseline after fix | Persistent backup restore if needed |
| Security breach | Reset branch / rebuild baseline | - |
| 10-50% validation failure | Evaluate | Fix loader / pack / schema then rebuild |
| <10% validation failure | No | Fix specific records if the branch is disposable |
| Single object failure | No | Re-load that object or rebuild branch |

---

## Loader Ledger Based Recovery

Instead of treating recovery as only a full database restore, use the controlled loader ledger:

```
1. Loader produces batch ledger
   - Each batch: start_time, records, status, errors
   - Stored: s3://nexuscrm-migration/logs/migration_YYYYMMDD.log

2. Rebuild or replay from ledger
   - Identify the failed object / batch
   - Decide whether to rebuild the whole baseline or discard the test branch
   - Re-run from the frozen source manifest if appropriate

3. Verify cleanup
   - Ensure no dirty branch is still in use
   - Verify relationship integrity on the recovered branch
```

### Transaction Log Format

```json
{
  "timestamp": "2026-05-16T10:30:00Z",
  "batch_id": "account_001",
  "object": "account",
  "records": 3307,
  "status": "COMMITTED",
  "external_id_range": ["001xx000003XXXXAA", "001xx000003YYYYBB"],
  "errors": []
}
```

### Ledger-Based Rebuild Script

```python
#!/usr/bin/env python3
"""Inspect loader batches and choose rebuild / replay actions."""

def inspect_loader_run(log_path: str):
    """Inspect committed loader batches from log file"""

    with open(log_path) as f:
        transactions = json.load(f)

    # Sort in reverse order (newest first)
    transactions.sort(key=lambda x: x["timestamp"], reverse=True)

    for txn in transactions:
        if txn["status"] != "COMMITTED":
            continue

        print(f"Committed batch: {txn['object']} / {txn['batch_id']}")

    print("Use this ledger to decide branch reset, baseline rebuild, or selective replay.")
```

---

## Rollback Verification Checklist

```
1. Database State
   - [ ] Recovered branch matches expected baseline state
   - [ ] Record counts match expected baseline or checkpoint
   - [ ] No orphaned records
   - [ ] Foreign key integrity verified

2. Application State
   - [ ] System operational
   - [ ] Login works
   - [ ] Basic CRUD works
   - [ ] No error logs spike

3. Data Integrity
   - [ ] Source data unchanged (source is read-only during migration)
   - [ ] Lookup relationships valid
   - [ ] Access model valid for the intended validation scope

4. Audit Trail
   - [ ] Rollback event logged
   - [ ] Incident report filed
   - [ ] Stakeholders notified

5. Next Steps
   - [ ] Root cause identified
   - [ ] Fix implemented
   - [ ] Baseline rebuild or branch recreate scheduled
```

---

## Rollback Notification Template

```
Subject: [CRITICAL] Baseline / Branch Recovery Initiated

Incident: Recovery initiated due to [REASON]

Impact:
- Migration paused
- System under recovery
- Estimated resolution: [TIME]

Current Status:
- [ ] Migration processes stopped
- [ ] Baseline / branch recovery target identified
- [ ] Reset / rebuild in progress
- [ ] Verification pending

Next Update: [TIME]

Contact: [NAME]
```

---

## Recovery Timeline

```
Hour 0: Issue Detected
  - Alert fires
  - On-call engineer notified
  - Initial assessment

Hour 0-1: Decision Made
  - Root cause identified
  - Rollback vs fix decision
  - Stakeholders notified

Hour 1-2: Recovery Target Verification
  - Canonical baseline or backup identified
  - Recovery integrity verified
  - Reset / rebuild command prepared

Hour 2-4: Reset / Rebuild Execution
  - Branch reset or baseline rebuild initiated
  - Monitoring progress
  - Handling any issues

Hour 4-5: Verification
  - Record counts verified
  - System checks pass
  - Basic functionality confirmed

Hour 5-6: Cleanup
  - Loader state cleared
  - S3 artifacts cleaned
  - Audit trail complete

Hour 6+: Post-Mortem
  - Root cause analysis
  - Fix implementation
  - Rebuild / re-validation re-scheduled
```

---

## Related Documents

- SALESFORCE_DATA_MIGRATION.md — Migration overview
- SCHEMA_COMPATIBILITY.md — Pre-migration check
- BUSINESS_VALIDATION_CHECKLIST.md — Acceptance criteria
- FIELD_MAPPING.md — Field correspondence