# Workspace Architecture & Implementation Summary

- `tx-migration`: a migration orchestration API for exporting/importing data between TiDB/MySQL-compatible clusters.

## High-Level Repository Structure

- `tx-migration/`
  - `cmd/tidb-x-migration`: service entrypoint.
  - `internal/api`: HTTP handlers and routing.
  - `internal/task`: async task lifecycle, retries, worker model.
  - `internal/storage`: task metadata backends (`memory` + MySQL-protocol compatible storage).
  - `internal/export`: Dumpling wrapper.
  - `internal/importer`: TiDB-Lightning wrapper.
  - `internal/config`: env-based runtime configuration.
  - `deployments/`: Kubernetes manifests.
  - `build.Dockerfile`: runtime image setup (tiup/dumpling/lightning).

## `tx-migration` Architecture

### 1) API Layer

- Exposes:
  - `POST /api/migration`: create migration task.
  - `GET /api/migration/:taskId`: query task status.
  - `GET /health`: health check.
- Input validation enforces MySQL-style connection URI format (`mysql://...`).

### 2) Task Orchestration Layer

- `Manager` owns task lifecycle:
  - Create task metadata.
  - Worker-based async execution.
  - Export + import pipeline.
  - Retry logic with backoff + jitter.
  - Status transitions (`PENDING/RUNNING/RETRYING/SUCCESS/FAILED`).
- Idempotent create behavior:
  - duplicate task ID returns existing task instead of hard failure.

### 3) Storage Layer

- Unified storage interface:
  - create/get/update task
  - claim next pending task
  - requeue stale tasks
- Implementations:
  - `MemoryStorage` (local dev/testing)
  - MySQL-protocol compatible SQL storage (works with TiDB/MySQL)

### 4) Data Movement Layer

- Export via `tiup dumpling`.
- Import via `tiup tidb-lightning`.
- Runtime-generated Lightning config with tunable concurrency and batch settings.

## Scalability & Reliability Enhancements Implemented

- Migrated from in-memory-only task state to shared SQL metadata backend.
- Added worker claim model for horizontal scaling (multi-replica safe queueing semantics).
- Added stale-task requeue on startup to recover abandoned `RUNNING/RETRYING` tasks.
- Added panic recovery around worker execution to avoid process-level task loss.
- Added configurable timeouts for Dumpling/Lightning phases.
- Added retry jitter to reduce synchronized retry storms.
- Added deployment-level support for multi-replica operation and env-driven tuning.

## Configuration Model (Env-Driven)

Key settings include:

- Storage/backend:
  - `STORAGE_TYPE` (`mysql` / `tidb` / `memory`)
  - `MYSQL_DSN` (preferred), fallback to legacy `TIDB_DSN`
- Scheduling/retry:
  - `MAX_RETRIES`
  - `MAX_CONCURRENT_TASKS`
  - `TASK_POLL_INTERVAL_MS`
  - `STALE_TASK_TTL_SECONDS`
- Performance tuning:
  - `LIGHTNING_*` concurrency/batch knobs
  - `DUMPLING_TIMEOUT_SECONDS`
  - `LIGHTNING_TIMEOUT_SECONDS`
  - `ENABLE_POST_IMPORT_VERIFICATION`

## Testing Status

Unit tests were added and passed for core non-external logic:

- `internal/config`: defaults + DSN fallback/precedence.
- `internal/storage` (memory): create/get/claim/requeue behavior.
- `internal/task`: idempotent create fallback, helper logic.
- `internal/api`: connection format validator.

Current tests are focused on deterministic logic and avoid external DB/tool dependencies.

## Current Trade-offs / Follow-up Opportunities

- SQL storage integration tests can be expanded with `sqlmock`.
- HTTP handler tests can be expanded to full `httptest` endpoint tests.
- Task cancellation and lease-heartbeat renewal can further improve long-running task resilience.
- Docker image can optionally add `ca-certificates` for stricter TLS robustness in cloud DB environments.