# Migrate Premium DM from Legacy to dataflow-service-ng

## TL;DR

Move Premium DM (shared-pool MySQL→TiDB migration) from `internal/` (legacy, co-located with Dedicated DM) to `dataflow-service-ng/` as a first-class citizen. **PR1 prioritizes minimal refactoring** — keep code structure as-is, only change imports/package paths. Pattern modernization (Interface+Impl models, functional-option repos, domain types) deferred to follow-up PRs.

---

## Migration Approach

### PR1 — Move Code (This PR)
- **Goal**: Move all Premium DM code to ng with minimal diff
- **Principle**: Keep existing code structure, patterns, and naming intact
- **Only change**: Package paths, import paths, DB access adaptation (`db.GetDB()` → injected `*gorm.DB`)
- **Agent proxy**: Move `NGRegionalClient` and related types to `pkg/regional-api/agent_ng.go` (HTTP-based, same behavior)

### Follow-up PRs (Deferred Refactoring)
- [ ] Refactor DAO to functional-option repo pattern with `db.Provider` read/write split
- [ ] Extract model interface + impl pattern (decouple from service layer)
- [ ] Convert to domain types in model interfaces (vs raw entity types)
- [ ] Add QuerySet generation for Premium DM entities
- [ ] Adopt `bizerr.WrapGormError` consistently
- [ ] Split service layer into thin controller + model (currently monolithic)

---

## Current State

### Premium DM in legacy `internal/`
| Layer | Path | Description |
|-------|------|-------------|
| Service (gRPC) | `internal/openapi/service/premium_dm/` | `PremiumDMService` implements `v1beta2.PremiumMigrationServiceServer` |
| Model/Worker | `internal/model/dm/premium_dm/` | `PremiumDMTaskWorker` (5s), `PremiumDMPrecheckWorker` (5s), `PremiumDMProxy`, `PremiumDMTaskDAO` |
| Entity | `internal/db/entity/premium_dm.go` | `PremiumDMTask`, `PremiumDMSource`, `PremiumDMPrecheck`, `PremiumDMPrecheckItem`, `PremiumDMVpcEndpoint` |
| Config | `internal/config/config.go` (`PremiumDMConfig`) | `ImportThreads`, `ExportThreads`, `ReplThreads`, `ReplBatch` |
| Metrics | `internal/metrics/premium_dm/` | Prometheus collectors |
| DI | `internal/container/container.go` | `PremiumDMAPI`, `PremiumDMWorker`, `PremiumDMPrecheckWorker` |
| Worker Reg | `internal/worker/global_worker.go` | gocron 5s interval |
| Proto | `idl/idl/dataflow/premium_dm/v1beta2/premium_dm.proto` | gRPC service + messages |
| Migrations | `migrate/record_202601041800_dm_premium.go`, `migrate/record_202602031001_dm_premium_precheck_schema.go` | Schema creation |
| Agent Proxy | `internal/handler/agent/regional-ng.go` | `NGRegionalClient.DMOpenAPIProxy()`, `CreateDMCluster()`, `ScaleDMCluster()`, `GetDMClusterByID()` |
| Bootstrap | `internal/cmd/bootstrap/dataflow_http.go`, `dataflow_worker.go` | gRPC+HTTP registration, metrics |

### Key Dependencies on Legacy Infrastructure
1. **`agent.NGRegionalClient`** — HTTP proxy to DM-Master via agent-ng
2. **`tiflow/dm/openapi`** — DM source/task payload types (already in dataflow-service-ng go.mod)
3. **`tidb-management-service`** — `ListClusters` for cluster validation (already `MgmtSvcGrpcClient` in ng)
4. **`encryptor-wrapper`** — Password encryption (already in ng `pkg/encryptor-wrapper/`)
5. **`dataflow-agent-ng/idl/pbgen/proto/ng_agent`** — Agent protobuf types. **Not importable from ng** (separate go.mod)

---

## Steps

### Phase 1: Proto & Generated Code ✅
1. Copy Premium DM proto to `dataflow-service-ng/idl/idl/dataflow/premium_dm/v1beta2/premium_dm.proto`
2. Update `go_package` to ng module path
3. Add `--path ./dataflow/premium_dm` to Makefile `generate-idl`
4. Run `make generate-idl` in `dataflow-service-ng/`

### Phase 2: Entity & Migration
5. Move `internal/db/entity/premium_dm.go` → `dataflow-service-ng/app/repo/entity/premium_dm.go` (change imports only)
6. Move `internal/db/entity/premium_dm_ext.go` → `dataflow-service-ng/app/repo/entity/premium_dm_ext.go`
7. Register entities in `entity.ORMModels`
8. Create idempotent migration record

### Phase 3: Agent Proxy (pkg/regional-api)
9. Move NGRegionalClient + DM types to `dataflow-service-ng/pkg/regional-api/agent_ng.go`
10. Move DM-specific methods (DMOpenAPIProxy, CreateDMCluster, etc.) to `pkg/regional-api/dm_cluster.go`
11. Define local agentpb equivalent types as Go structs (HTTP JSON, not gRPC proto)
12. Adapt constructors to use injected MgmtSvcGrpcClient + config (no global container access)

### Phase 4: DAO / Data Access
13. Move `internal/model/dm/premium_dm/dao.go` → `dataflow-service-ng/app/repo/premium_dm/dao.go`
14. Minimal change: inject `*gorm.DB` via constructor instead of `db.GetDB()`

### Phase 5: Model / Worker / Precheck
15. Move `internal/model/dm/premium_dm/proxy.go` → `dataflow-service-ng/app/models/premium_dm/proxy.go`
16. Move `internal/model/dm/premium_dm/worker.go` → `dataflow-service-ng/app/models/premium_dm/worker.go`
17. Move `internal/model/dm/premium_dm/precheck_*.go` → `dataflow-service-ng/app/models/premium_dm/`
18. Adapt imports (entity, agent, config, container)

### Phase 6: Service / Controller
19. Move `internal/openapi/service/premium_dm/premium_dm.go` → `dataflow-service-ng/app/controllers/premium_dm/premium_dm.go`
20. Move `internal/openapi/service/premium_dm/premium_dm_vo.go` → `dataflow-service-ng/app/controllers/premium_dm/premium_dm_vo.go`
21. Update proto import to ng-generated types

### Phase 7: Config & Metrics
22. Add `PremiumDMConfig` to ng config
23. Move `internal/metrics/premium_dm/` → `dataflow-service-ng/app/metrics/premium_dm/`

### Phase 8: DI Wiring & Bootstrap
24. Wire into container (`SetUpContainers`)
25. Register gRPC+HTTP routes in bootstrap
26. Register gocron worker

### Phase 9: Verification
27. `make build-dataflow` — must compile
28. `make app-unit-test-local` — all tests pass

---

## Decisions

- **Agent proto types**: Define locally as Go structs in `pkg/regional-api/` (HTTP JSON, not gRPC)
- **Database migration**: Idempotent `AutoMigrate` (tables already exist from legacy)
- **No workflow (fastflow)**: Keep gocron worker pattern (poll-based state machine)
- **Scope**: Premium DM only. Dedicated DM stays in `internal/`

## Risk Assessment

### 🔴 High Risk
1. **Dual-worker race condition**: Never run both legacy and ng workers simultaneously
2. **Agent communication compatibility**: ng must produce identical HTTP requests

### 🟡 Medium Risk
3. **Proto package path change**: Separate binary, internal types only
4. **Config deployment**: Add defaults in config parsing

### 🟢 Low Risk
5. **Migration record ordering**: `AutoMigrate` is idempotent
6. **Metrics gap**: Brief gap during cutover, acceptable

## PR Strategy
- **PR1 (this)**: Move all Premium DM code to ng — proto + entity + DAO + agent proxy + model/worker + service + config + metrics + wiring. Minimal refactoring.
- **PR2**: Refactor to ng patterns — Interface+Impl model, functional-option repos, domain types, QuerySet
- **PR3**: Legacy cleanup — remove Premium DM from `internal/`
- **PR4**: Tests + mocks