# Premium Import Into migration runbook (probe-first)

## Scope

- Migration from legacy `dataflow-service` Import Into path to `dataflow-service-ng`
- `CLUSTER_IDS` can narrow scope; if unset, scripts run whole-table mode

## Toolkit layout

```text
tools/migration/import_into/
├── migrate_data.sh
├── verify_parity.sh
├── lib.sh
└── sql/
    └── verify/
        ├── import_into_tasks_count.sql.tmpl
        ├── import_into_precheck_tasks_count.sql.tmpl
        ├── import_into_tasks_status_distribution.sql.tmpl
        ├── import_into_tasks_inflight.sql.tmpl
        └── probe_import_task.sql.tmpl
```

## Inputs

- `OLD_DB_HOST`, `OLD_DB_PORT`, `OLD_DB_USER`, `OLD_DB_PASSWORD`, `OLD_DB_NAME`
- `NEW_DB_HOST`, `NEW_DB_PORT`, `NEW_DB_USER`, `NEW_DB_PASSWORD`, `NEW_DB_NAME`
- `CLUSTER_IDS` (optional; comma-separated cluster IDs in migration scope)
- `SNAPSHOT_DIR` (artifact output directory for migration/verify runs)
- `DRY_RUN` (optional for migration script; when true only dump data)
- `PROBE_IMPORT_ID` (optional for parity script; required for probe check)

If `CLUSTER_IDS` is unset, scripts run in whole-table mode (`import_into_tasks` + `import_into_precheck_tasks`).

## Step 1. Create probe task before freeze

1. Create one import-into task via legacy API.
2. Record probe metadata:
   - probe import ID
   - tenant/project/cluster
   - expected state progression

## Step 2. Freeze legacy write + worker

1. Set dataflow-service:
   - `import.importIntoWriteEnabled=false`
   - `import.importIntoWorkerEnabled=false`
2. Confirm:
   - legacy `Create/Cancel/CreateCheckImportSourceTask` return permission error
   - legacy `List/Get` still work

## Step 3. Migrate metadb rows

```bash
OLD_DB_HOST="..." OLD_DB_PORT="3306" OLD_DB_USER="..." OLD_DB_PASSWORD="..." OLD_DB_NAME="..." \
NEW_DB_HOST="..." NEW_DB_PORT="3306" NEW_DB_USER="..." NEW_DB_PASSWORD="..." NEW_DB_NAME="..." \
CLUSTER_IDS="1001,1002" SNAPSHOT_DIR="./snapshot" \
./tools/migration/import_into/migrate_data.sh
```

Keep the generated dump under the run directory for audit and rollback analysis.

Migration behavior notes:
1. Target tables are not cleaned before import.
2. Dump is generated with `INSERT IGNORE` + column list (`--complete-insert`), so rerun is allowed and column-order drift between old/new tables does not break replay.
3. Import step removes `NO_BACKSLASH_ESCAPES` from session `sql_mode` before applying dump to avoid JSON-escape import failures.
4. For dry run, set `DRY_RUN=true`; script exits after dump generation.

Whole-table migration example:

```bash
OLD_DB_HOST="..." OLD_DB_PORT="3306" OLD_DB_USER="..." OLD_DB_PASSWORD="..." OLD_DB_NAME="..." \
NEW_DB_HOST="..." NEW_DB_PORT="3306" NEW_DB_USER="..." NEW_DB_PASSWORD="..." NEW_DB_NAME="..." \
SNAPSHOT_DIR="./snapshot" \
./tools/migration/import_into/migrate_data.sh
```

Dry-run example:

```bash
OLD_DB_HOST="..." OLD_DB_PORT="3306" OLD_DB_USER="..." OLD_DB_PASSWORD="..." OLD_DB_NAME="..." \
NEW_DB_HOST="..." NEW_DB_PORT="3306" NEW_DB_USER="..." NEW_DB_PASSWORD="..." NEW_DB_NAME="..." \
CLUSTER_IDS="1001,1002" SNAPSHOT_DIR="./snapshot" DRY_RUN="true" \
./tools/migration/import_into/migrate_data.sh
```

## Step 4. Verify parity before gateway cutover

```bash
OLD_DB_HOST="..." OLD_DB_PORT="3306" OLD_DB_USER="..." OLD_DB_PASSWORD="..." OLD_DB_NAME="..." \
NEW_DB_HOST="..." NEW_DB_PORT="3306" NEW_DB_USER="..." NEW_DB_PASSWORD="..." NEW_DB_NAME="..." \
CLUSTER_IDS="1001,1002" SNAPSHOT_DIR="./snapshot" PROBE_IMPORT_ID="12345" \
./tools/migration/import_into/verify_parity.sh
```

If parity fails, stop cutover and investigate diffs under the verification run directory.

Whole-table parity example:

```bash
OLD_DB_HOST="..." OLD_DB_PORT="3306" OLD_DB_USER="..." OLD_DB_PASSWORD="..." OLD_DB_NAME="..." \
NEW_DB_HOST="..." NEW_DB_PORT="3306" NEW_DB_USER="..." NEW_DB_PASSWORD="..." NEW_DB_NAME="..." \
SNAPSHOT_DIR="./snapshot" \
./tools/migration/import_into/verify_parity.sh
```

## Step 5. Switch API gateway to dataflow-service-ng

1. Keep NG gates disabled initially:
   - `importInto.writeEnabled = []`
   - `importInto.workerEnabled = []`
2. Switch routing to NG.
3. Confirm NG `List/Get` parity against baseline.

## Step 6. Enable NG write + worker for probe scope only

1. Set NG scoped gates for probe tenant/project/cluster:
   - `importInto.writeEnabled=[{tenantID:[...],projectID:[...],clusterID:[...]}]`
   - `importInto.workerEnabled=[{tenantID:[...],projectID:[...],clusterID:[...]}]`
2. Validate:
   - probe task state continues correctly
   - create/cancel works for probe scope

## Step 7. Expand NG gates to full Premium scope

1. Add all Premium tenant/project/cluster scopes to `writeEnabled` and `workerEnabled`.
2. Keep legacy service frozen.

## Step 8. Post-cutover observation

1. Monitor import-into task failures and inflight tasks.
2. Keep snapshot artifacts and parity outputs for incident review.