# TiDB Cloud Tool Source Code Analysis - Repo Inventory Overview > This document summarizes all code repositories (repos) involved in each tool (Tier x Tool), including the module name (go module name) and primary responsibility of each one. --- ## 1. Quick Repo Reference Table | # | Repo Name | GitHub Repository | Go Module Name | Responsibility Summary | |---|-----------|-------------------|----------------|------------------------| | 1 | dataflow-service | [tidbcloud/dataflow-service](https://github.com/tidbcloud/dataflow-service) | github.com/tidbcloud/dataflow-service | Dedicated control plane (gRPC API + Worker + Regional API Client) | | 2 | dataflow-service-ng | same as above ↑ | github.com/tidbcloud/dataflow-service-ng | Premium control plane (gRPC API + DAG Workflow + Regional API Client)
**Note**: It lives in the same GitHub repository as dataflow-service (subdirectory). | | 3 | serverless-service | [tidbcloud/serverless-service](https://github.com/tidbcloud/serverless-service) | github.com/tidbcloud/serverless-service | Essential control plane (gRPC API + Worker + ClusterProxy Client) | | 4 | infra-provider | [tidbcloud/infra-provider](https://github.com/tidbcloud/infra-provider) | github.com/tidbcloud/infra-provider | Shared Kubernetes controller (Import/Backup/TiCDC/DM CR Reconciler) | | 5 | cluster-service-ng | [tidbcloud/cluster-service-ng](https://github.com/tidbcloud/cluster-service-ng) | github.com/tidbcloud/cluster-service-ng | Regional control plane (DAG Workflow + K8s resource rendering + BR Workflow Actions) | | 6 | tidb-management-service | [tidbcloud/tidb-management-service](https://github.com/tidbcloud/tidb-management-service) | github.com/tidbcloud/tidb-management-service | Global control plane (Restore orchestration + Gateway proxy) | | 7 | aws-shared-provider | [tidbcloud/aws-shared-provider](https://github.com/tidbcloud/aws-shared-provider) | github.com/tidbcloud/aws-shared-provider | Essential Import controller (Lightning Job + Secret + PVC creation) | | 8 | infra-api | [tidbcloud/infra-api](https://github.com/tidbcloud/infra-api) | github.com/tidbcloud/infra-api | CRD definitions (Import/Backup/Restore CRDs + Kubernetes API types) | | 9 | infra-provider/api | [tidbcloud/infra-provider](https://github.com/tidbcloud/infra-provider) | github.com/tidbcloud/eks-provider/api | CRD definitions (DMJob CRD + TiCDC Service API definitions)
**Note**: The Go module name is `eks-provider/api`, but the actual code is under `infra-provider/api/`. | | 10 | tcidm | [tidbcloud/tcidm](https://github.com/tidbcloud/tcidm) | tcidm | Regional HTTP-to-CR translation layer (HTTP Handler -> K8s Client -> Kubernetes) | | 11 | tidb-operator | [pingcap/tidb-operator](https://github.com/pingcap/tidb-operator) | github.com/pingcap/tidb-operator/v2 | TiBR/Backup/Restore CRD definitions + TiBR/Backup/Restore Controller
**Note**: **v1 (release-1.x)** is used by Dedicated.
**Note**: **v2 (main)** is used by Premium. | | 12 | tidb | [pingcap/tidb](https://github.com/pingcap/tidb) | github.com/pingcap/tidb | TiDB kernel (IMPORT INTO SQL executor, Lightning, Dumpling, BR backup/restore) | | 13 | tiflow | [pingcap/tiflow](https://github.com/pingcap/tiflow) | github.com/pingcap/tiflow | TiCDC/TiDB DM kernel (Changefeed Pipeline + DM Master/Worker + OpenAPI) | | 14 | **runbooks** | [tidbcloud/runbooks](https://github.com/tidbcloud/runbooks) | - | **Alert rule definitions** (Prometheus Rules + Grafana Dashboards + Runbook docs) | | 15 | **app-delivery** | [tidbcloud/app-delivery](https://github.com/tidbcloud/app-delivery) | - | **GitOps deployment repository** (ArgoCD App definitions + version files + bump workflow) | --- ## 2. Tier 1 - Dedicated (4 tools) ### 2.1 Dedicated Import (Data Import) Involves **5 repositories**: | # | Repo | Module Name | Covered Components | |---|------|-------------|--------------------| | 1 | **dataflow-service** | github.com/tidbcloud/dataflow-service | Core control plane: API entry -> Model -> Worker state machine -> Regional API Client -> dataflow-agent | | 2 | **tcidm** | tcidm | Regional HTTP layer: HTTP Handler -> K8s Client -> Import CR (`cloudv1.Import`) | | 3 | **infra-provider** | github.com/tidbcloud/infra-provider | Controller: `import-controller` (`cluster-controller/controllers/imports/`) -> Lightning Job/Pod creation | | 4 | **infra-provider/infra-api** | github.com/tidbcloud/infra-api | CRD definition: `Import` CRD (`cloud.pingcap.com/v1`) | | 5 | **tidb** | github.com/pingcap/tidb | Lightning: `pkg/lightning/` (TiDB Lightning execution engine) | **Simplified call chain**: ```text dataflow-service → tcidm (HTTP /api/v1/tcidm/import/cr) → infra-provider Import Controller (cluster-controller/controllers/imports/) → Import CR (infra-provider/infra-api/api/v1/import_types.go) → Lightning Pod (tidb/pkg/lightning/) ``` --- ### 2.2 Dedicated Changefeed (Data Change Capture) Involves **4 repositories** (confirmed at source level): | # | Repo | Module Name | Covered Components (source confirmed) | |---|------|-------------|---------------------------------------| | 1 | **dataflow-service** | github.com/tidbcloud/dataflow-service | Core control plane: BizChangefeed Model + DAG Workflow + Scheduler + Regional API Client | | 2 | **infra-provider/ticdc-service** | github.com/tidbcloud/ticdc-service | TiCDC Controller + CRD: `infra-provider/ticdc-service/controllers/ticdc/` + `infra-provider/ticdc-service/api/v1/` | | 3 | **infra-provider/api** | github.com/tidbcloud/eks-provider/api | TiCDC Service API definitions: `api/tidbcloud/ticdc-service/v1/` | | 4 | **tiflow** | github.com/pingcap/tiflow | TiCDC kernel (Changefeed Pipeline + DDL/DML replication + OpenAPI) | > **Note**: TiCDC Controller location: `infra-provider/ticdc-service/controllers/ticdc/controller.go` > **Note**: TiCDC CRD location: `infra-provider/ticdc-service/api/v1/ticdc_types.go` **Call chain (source confirmed)**: ```text dataflow-service (app/models/biz_changefeed/) → infra-provider/ticdc-service (gRPC Regional API) → TiCDCReconciler (infra-provider/ticdc-service/controllers/ticdc/) → TiCDC CR (infra-provider/ticdc-service/api/v1/ticdc_types.go) → TiCDC Pods (tiflow kernel) ``` **Key source locations**: | Component | File Path | Description | |-----------|-----------|-------------| | BizChangefeed | `dataflow-service/app/models/biz_changefeed/impl/biz_changefeed.go` | Business Changefeed model | | TiCDC CRD | `infra-provider/ticdc-service/api/v1/ticdc_types.go` | TiCDC resource definition | | TiCDC Controller | `infra-provider/ticdc-service/controllers/ticdc/controller.go` | K8s Controller | | TiCDC Service API | `infra-provider/api/tidbcloud/ticdc-service/v1/` | gRPC API definitions | --- ### 2.3 Dedicated DM (Data Migration) Involves **5 repositories** (confirmed at source level): | # | Repo | Module Name | Covered Components (source confirmed) | |---|------|-------------|---------------------------------------| | 1 | **dataflow-service** | github.com/tidbcloud/dataflow-service | Core control plane: DM Model + Worker state machine + Regional API Client | | 2 | **tcidm** | tcidm | Regional HTTP layer: `regional-tcidm-service` (HTTP -> DMJob CR) | | 3 | **infra-provider/api** | github.com/tidbcloud/eks-provider/api | CRD definition: `DMJob` CRD (`eks.infra.pingcap.com/v1alpha1`) | | 4 | **infra-provider/cluster-controller** | github.com/tidbcloud/eks-provider | DM Controller: `controllers/dm/controller.go` | | 5 | **tiflow** | github.com/pingcap/tiflow | DM kernel: DM-Master + DM-Worker + DM OpenAPI | > **Note**: `eks-provider/api` and `eks-provider` are Go module names. The actual code is under the `infra-provider/` directory. > **Note**: DMJob CRD location: `infra-provider/api/v1alpha1/dmjob_types.go` > **Note**: DM Controller location: `infra-provider/cluster-controller/controllers/dm/controller.go` **Call chain (source confirmed)**: ```text dataflow-service (internal/model/dm/) → tcidm (HTTP /api/v1/tcidm/dmjob/cr) → infra-provider DM Controller (cluster-controller/controllers/dm/) → DMJob CR (infra-provider/api/v1alpha1/dmjob_types.go) → DMCluster → DM-Master/Worker Pods (tiflow kernel) ``` **Key source locations**: | Component | File Path | Description | |-----------|-----------|-------------| | DM Model | `dataflow-service/internal/model/dm/model.go` | Core DM model | | DM State Machine | `dataflow-service/internal/model/dm/state_*.go` | State machine implementation | | DMJob CRD | `infra-provider/api/v1alpha1/dmjob_types.go` | DMJob resource definition (Go module: `eks-provider`) | | DM Controller | `infra-provider/cluster-controller/controllers/dm/controller.go` | K8s Controller | --- ### 2.4 Dedicated Backup & Restore Involves **5 repositories**: | # | Repo | Module Name | Covered Components | |---|------|-------------|--------------------| | 1 | **dataflow-service** | github.com/tidbcloud/dataflow-service | Core control plane: Backup Model + Restore Model + DAG Workflow + Regional API Client | | 2 | **infra-provider/infra-api** | github.com/tidbcloud/infra-api | CRD definitions: `Backup` / `Restore` CR (`cloud.pingcap.com/v1`) + BackupReconciler/RestoreReconciler | | 3 | **infra-provider/cluster-controller** | github.com/tidbcloud/eks-provider | BR Controller: `controllers/br/` | | 4 | **tidb-operator** | github.com/pingcap/tidb-operator/v2 | Backup/Restore Controller + CRD (`pingcap.com/v1alpha1`) | | 5 | **tidb** | github.com/pingcap/tidb | BR kernel: `pkg/br/` | **Simplified call chain**: ```text dataflow-service → infra-api BackupFacade/RestoreFacade → infra-provider BackupReconciler / RestoreReconciler → infra-api (CRD definitions: cloud.pingcap.com/v1) → tidb-operator Backup CR Controller → tidb-operator (CRD definitions: pingcap.com/v1alpha1) → BR Pod (tidb kernel br command) ``` --- ## 3. Tier 2 - Premium (5 tools) ### 3.1 Premium Changefeed (Data Change Capture) Involves **3 repositories** (confirmed at source level): | # | Repo | Module Name | Covered Components (source confirmed) | |---|------|-------------|---------------------------------------| | 1 | **dataflow-service-ng** | github.com/tidbcloud/dataflow-service-ng | Core control plane: BizChangefeedModel + DAG Workflow + Shared Pool Scheduler | | 2 | **cluster-service-ng** | github.com/tidbcloud/cluster-service-ng | Regional control plane: `api/tidbcloud/ticdc-service/v1` (TiCDC API)
**Source confirmation**: `dataflow-service-ng` directly calls `cluster-service-ng` via `import ticdc_v1 "github.com/tidbcloud/cluster-service-ng/api/tidbcloud/ticdc-service/v1"` | | 3 | **tiflow** | github.com/pingcap/tiflow | TiCDC kernel (Changefeed Pipeline + Shared Pool TiCDC cluster) | > **Note**: Regional logic is in `cluster-service-ng`: the TiCDC Controller lives in `cluster-service-ng` (`ticdc-service`), not `infra-provider`. **Call chain (source confirmed)**: ```text dataflow-service-ng (app/models/ticdc/impl/) → cluster-service-ng (api/tidbcloud/ticdc-service/v1) → TiCDCReconciler (K8s Controller) → TiCDC Cluster (tiflow kernel, Shared Pool mode) ``` **Key source locations**: | Component | File Path | Description | |-----------|-----------|-------------| | BizChangefeedModel | `dataflow-service-ng/app/models/biz_changefeed/impl/biz_changefeed.go` | Business Changefeed model | | TiCDC Regional | `dataflow-service-ng/app/models/ticdc/impl/changefeed_regional.go` | Regional API call | | TiCDC Worker | `dataflow-service-ng/app/models/ticdc/impl/changefeed_worker_regional.go` | Worker state machine | | TiCDC Service API | `cluster-service-ng/api/tidbcloud/ticdc-service/v1` | TiCDC API definitions | --- ### 3.2 Premium ImportInto (Data Import) Involves **4 repositories** (confirmed at source level): | # | Repo | Module Name | Covered Components (source confirmed) | |---|------|-------------|---------------------------------------| | 1 | **dataflow-service** | github.com/tidbcloud/dataflow-service | Global control plane: ImportIntoModel + ImportInto DAG
**Source confirmation**: The Global code is in the main `dataflow-service` directory (`internal/model/import/import_into.go`), **not** in `dataflow-service-ng`. | | 2 | **cluster-service-ng** | github.com/tidbcloud/cluster-service-ng | Regional control plane: Regional API routing | | 3 | **dataflow-agent-ng** | github.com/tidbcloud/dataflow-service/dataflow-agent-ng | Data plane agent: Lightning Job management
**Source confirmation**: `import agentpb "github.com/tidbcloud/dataflow-service/dataflow-agent-ng/idl/pbgen/proto/ng_agent"` | | 4 | **tidb** | github.com/pingcap/tidb | Lightning + ImportInto executor: `pkg/executor/import_into.go` + `pkg/lightning/` | **Call chain (source confirmed)**: ```text dataflow-service (internal/model/import/import_into.go — Global layer) → agent.NewNGRegionalClientByClusterID() (HTTP REST) → cluster-service-ng (Regional API routing) → dataflow-agent-ng (pkg/agent/ — Lightning Job management) → Lightning Pod → tidb kernel (pkg/lightning/ + pkg/executor/import_into.go) ``` **Key source locations**: | Component | File Path | Description | |-----------|-----------|-------------| | ImportIntoModel | `dataflow-service/internal/model/import/import_into.go` | Core ImportInto logic | | NGRegionalClient | `dataflow-service/internal/handler/agent/regional-agent.go` | Regional Agent client | | Lightning API | `dataflow-service/internal/model/import/agent.go` | Lightning Job API | | ImportInto Exec | `tidb/pkg/executor/import_into.go` | TiDB ImportInto executor | | Lightning | `tidb/pkg/lightning/` | Lightning import engine | --- ### 3.3 Premium Export (Data Export) Involves **3 repositories** (confirmed at source level, **does not use `dataflow-service`**): | # | Repo | Module Name | Covered Components (source confirmed) | |---|------|-------------|---------------------------------------| | 1 | **tidb-management-service** | github.com/tidbcloud/tidb-management-service | Global control plane: `nextgen-global-service` (CreateExport DAG + Export TiDB Group management) | | 2 | **cluster-service-ng** | github.com/tidbcloud/cluster-service-ng | Regional control plane: Export DAG Actions + Dumpling Job + TiDB Group | | 3 | **tidb** | github.com/pingcap/tidb | Dumpling: `dumpling/` (data export tool) | **Call chain (source confirmed)**: ```text tidb-management-service (nextgen-global-service) → cluster-service-ng (Export DAG Actions) → Dumpling Job (tidb/dumpling/ -> exports to S3/GCS/Azure) ``` **Key source locations**: | Component | File Path | Description | |-----------|-----------|-------------| | Export DAG | `tidb-management-service/domain/service/workflow/dag/` | Export orchestration | | Export Actions | `cluster-service-ng/domain/workflow/action/export/` | Export Actions | | Dumpling | `tidb/pkg/util/mydump/` | Dumpling export tool | --- ### 3.4 Premium Backup & Restore Involves **4 repositories** (confirmed at source level): | # | Repo | Module Name | Covered Components (source confirmed) | |---|------|-------------|---------------------------------------| | 1 | **tidb-management-service** | github.com/tidbcloud/tidb-management-service | Global control plane: `nextgen_restore.go` -> Restore DAG orchestration | | 2 | **cluster-service-ng** | github.com/tidbcloud/cluster-service-ng | Regional control plane: `domain/workflow/action/br/` - BR Workflow Actions | | 3 | **tidb-operator (v2)** | github.com/pingcap/tidb-operator/v2 | TiBR CRD definitions (`api/br/v1alpha1/tibr_types.go`) + TiBR Controller
**Source confirmation**: `cluster-service-ng` imports it via `import v1alpha1br "github.com/pingcap/tidb-operator/api/v2/br/v1alpha1"` | | 4 | **tidb** | github.com/pingcap/tidb | BR kernel: `br/` (invoked inside the TiBR Pod) | **Call chain (source confirmed)**: ```text tidb-management-service (nextgen_restore.go) → cluster-service-ng (domain/workflow/action/br/) → tidb-operator v2 (api/br/v1alpha1) → TiBR Controller → TiBR Pod → tidb kernel (pkg/br/) ``` --- ### 3.5 Premium DM (Data Migration) Involves **5 repositories** (confirmed at source level): | # | Repo | Module Name | Covered Components (source confirmed) | |---|------|-------------|---------------------------------------| | 1 | **dataflow-service-ng** | github.com/tidbcloud/dataflow-service-ng | Core control plane: DM Worker state machine + DM Proxy + NGRegionalClient | | 2 | **tidb-management-service** | github.com/tidbcloud/tidb-management-service | Gateway routing: `dataplane/proxy/sharedPools/{id}/dataflow-agent` | | 3 | **cluster-service-ng** | github.com/tidbcloud/cluster-service-ng | Regional control plane: dataplane-manager (`/dataplane/proxy/` routing) | | 4 | **dataflow-agent-ng** | github.com/tidbcloud/dataflow-service/dataflow-agent-ng | Data plane agent: DM Cluster management | | 5 | **tiflow** | github.com/pingcap/tiflow | DM kernel: DM-Master + DM-Worker + DM OpenAPI | **Call chain (source confirmed)**: ```text dataflow-service-ng (app/models/premium_dm/worker.go) → NGRegionalClient (pkg/agent/dm.go) → tidb-management-service (dataplane/proxy/) → cluster-service-ng (dataplane-manager) → dataflow-agent-ng (DM Cluster management) → DMCluster CR → DM Pods → tiflow ``` --- ## 4. Tier 3 - Essential (5 tools) ### 4.1 Essential Import (Data Import) Involves **4 repositories**: | # | Repo | Module Name | Covered Components | |---|------|-------------|--------------------| | 1 | **serverless-service** | github.com/tidbcloud/serverless-service | Core control plane: Import API + Domain Service + ImportSyncer + ClusterProxy Client | | 2 | **aws-shared-provider** | github.com/tidbcloud/aws-shared-provider | **Import Controller**: `ImportReconciler` -> Lightning Job/Pod/Service creation | | 3 | **infra-api** | github.com/tidbcloud/infra-api | CRD definition: `Import` CRD (`cloud.pingcap.com/v1`)
**Note**: The actual code is at `infra-provider/infra-api/api/v1/import_types.go` (resolved via `replace`). | | 4 | **tidb** | github.com/pingcap/tidb | Lightning: `pkg/lightning/` (Server Mode HTTP API) | --- ### 4.2 Essential Changefeed (Data Change Capture) Involves **3 repositories**: | # | Repo | Module Name | Covered Components | |---|------|-------------|--------------------| | 1 | **serverless-service** | github.com/tidbcloud/serverless-service | Core control plane: CDC API + CDCService + CDCWorker + ClusterProxy Client | | 2 | **infra-api** | github.com/tidbcloud/infra-api | CRD definition: `TiCDC` CRD (`cloud.pingcap.com/v1`)
**Note**: The CRD type name is `TiCDC`, not `ServerlessCDC` or `ServerlessChangefeed`.
**Note**: The actual code is at `infra-provider/infra-api/api/v1/ticdc_types.go` (resolved via `replace`). | | 3 | **tiflow** | github.com/pingcap/tiflow | TiCDC kernel (serverless TiCDC cluster managed by keyspace) | **Simplified call chain**: ```text serverless-service → ClusterProxy (HTTP -> Serverless K8s) → TiCDC Controller (infra-provider/infra-api/api/v1/ticdc_types.go — TiCDC CR) → TiCDC Pod (tiflow kernel, managed by keyspace) ``` --- ### 4.3 Essential DM (Data Migration) Involves **3 repositories**: | # | Repo | Module Name | Covered Components | |---|------|-------------|--------------------| | 1 | **serverless-service** | github.com/tidbcloud/serverless-service | Core control plane: DM API + DMService + DMTaskWorker + gRPC Client | | 2 | **tidb-management-service/serverless-global-service** | github.com/tidbcloud/tidb-management-service/serverless-global-service | Regional gRPC API: `ServerlessDMRegionalService` (Precheck/CreateVpcEndpoint, etc.) | | 3 | **tiflow** | github.com/pingcap/tiflow | DM kernel (shared DM cluster mode) | > **Source confirmation**: Essential DM **does not use** infra-api or eks-provider CRDs. Instead, control plane communication happens through the **serverless-global-service gRPC API** (`ServerlessDMRegionalService`). **Simplified call chain**: ```text serverless-service → ServerlessDMRegionalService (gRPC -> serverless-global-service) → DM Task execution → DM-Master/Worker Pods (tiflow kernel, shared cluster mode) ``` --- ### 4.4 Essential Export (Data Export) Involves **4 repositories**: | # | Repo | Module Name | Covered Components | |---|------|-------------|--------------------| | 1 | **serverless-service** | github.com/tidbcloud/serverless-service | Core control plane: Export API + Domain Service + BackupWorker + ClusterProxy Client | | 2 | **aws-shared-provider** | github.com/tidbcloud/aws-shared-provider | **Export Controller**: `ExportReconciler` -> Dumpling Job/Pod creation | | 3 | **infra-api** | github.com/tidbcloud/infra-api | CRD definition: `Export` CRD (`cloud.pingcap.com/v1`)
**Note**: The actual code is at `infra-provider/infra-api/api/v1/export_types.go` (resolved via `replace`). | | 4 | **tidb** | github.com/pingcap/tidb | Dumpling: `dumpling/` (data export tool) | **Simplified call chain**: ```text serverless-service → ClusterProxy (HTTP -> Serverless K8s) → aws-shared-provider ExportReconciler (pkg/controllers/export/) → Export CR (infra-provider/infra-api/api/v1/export_types.go) → Dumpling Pod (tidb/dumpling/ -> exports to S3/GCS) ``` --- ### 4.5 Essential BR (Backup & Restore) Involves **2 repositories**: | # | Repo | Module Name | Covered Components | |---|------|-------------|--------------------| | 1 | **serverless-service** | github.com/tidbcloud/serverless-service | Core control plane: Backup/Restore API + BackupWorker + ClusterProxy Client | | 2 | **tidb** | github.com/pingcap/tidb | BR: `br/` (BR backup/restore library invoked via ClusterProxy HTTP calls) | > **Source confirmation**: Essential BR **does not use** infra-api CRDs. Instead, it directly invokes BR inside the Serverless cluster through ClusterProxy HTTP. **Simplified call chain**: ```text serverless-service → pkgv2/ClusterProxy (HTTP) → Cluster Proxy (deployed inside the Serverless cluster) → TiDB BR (tidb/br/ library) → TiKV (RocksDB Snapshot) → S3 Storage ``` --- ## 5. Repo x Tool Matrix | Repo | Ded.Import | Ded.Change-feed | Ded.DM | Ded.BR | Prm.ImportInto | Prm.Change-feed | Prm.Export | Prm.BR | Prm.DM | Ess.Import | Ess.Change-feed | Ess.DM | Ess.Export | Ess.BR | |------|:----------:|:---------------:|:------:|:------:|:--------------:|:---------------:|:----------:|:------:|:------:|:----------:|:----------------:|:------:|:----------:|:------:| | **dataflow-service** | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | | **dataflow-service-ng** | | | | | | ✅ | | | ✅ | | | | | | **dataflow-agent-ng** | | | | | ✅ | | | | ✅ | | | | | | **serverless-service** | | | | | | | | | | ✅ | ✅ | ✅ | ✅ | ✅ | | **infra-provider/cluster-controller** | ✅ | | ✅ | ✅ | | | | | | | | | | | **infra-provider/ticdc-service** | | ✅ | | | | | | | | | | | | | **infra-provider/api** | | ✅ | ✅ | | | | | | | | | | | | **infra-provider/infra-api** | ✅ | | | ✅ | | | | | | ✅ | ✅ | ❌ | ✅ | ❌ | | **cluster-service-ng** | | | | | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | **tidb-management-service** | | | | | | | ✅ | ✅ | ✅ | | | ✅ | | | | **aws-shared-provider** | | | | | | | | | | ✅ | | | ✅ | | | **tcidm** | ✅ | | ✅ | | | | | | | | | | | | **tidb-operator** | | | | ✅(v1) | | | | ✅(v2) | | | | | | | **tidb** | ✅ | | | | ✅ | | ✅ | ✅ | | ✅ | | | ✅ | ✅ | | **tiflow** | | ✅ | ✅ | | | ✅ | | | ✅ | | ✅ | ✅ | | | | **runbooks** | 📊 | 📊 | 📊 | 📊 | 📊 | 📊 | 📊 | 📊 | 📊 | 📊 | 📊 | 📊 | 📊 | 📊 | | **app-delivery** | 🚀 | 🚀 | 🚀 | 🚀 | 🚀 | 🚀 | 🚀 | 🚀 | 🚀 | 🚀 | 🚀 | 🚀 | 🚀 | 🚀 | > `🚀` means the tool is deployed through the `app-delivery` repository using GitOps (applies to all tools). > `📊` means the alert rules for the tool are defined in the `runbooks` repository (applies to all tools). --- ## 6. Repo Responsibilities Quick Reference ### dataflow-service - **Path**: `dataflow-service` - **Go Module**: `github.com/tidbcloud/dataflow-service` - **Responsibility**: Dedicated control plane (Import/DM/Backup Restore) + Premium ImportInto Global layer - **Key code**: `internal/model/{import,dm,backup}/` ### dataflow-service-ng - **Path**: `dataflow-service/dataflow-service-ng` - **Go Module**: `github.com/tidbcloud/dataflow-service-ng` - **Responsibility**: Premium control plane (Changefeed/DM Shared Pool) - **Key code**: `app/models/biz_changefeed/`, `app/models/premium_dm/` ### dataflow-agent-ng - **Path**: `dataflow-service/dataflow-agent-ng` - **Go Module**: `github.com/tidbcloud/dataflow-service/dataflow-agent-ng` - **Responsibility**: Premium data plane agent (ImportInto Lightning Job + DM Cluster management) - **Key code**: `agent/grpc/services.go`, `idl/pbgen/proto/ng_agent/` ### serverless-service - **Path**: `serverless-service` - **Go Module**: `github.com/tidbcloud/serverless-service` - **Responsibility**: Essential control plane (Import/CDC/DM/Export/BR) - **Key code**: `internal/domain/service/{import.go,cdc.go,dm.go,export.go,backup_restore.go}` ### aws-shared-provider - **Path**: `aws-shared-provider` - **Go Module**: `github.com/tidbcloud/aws-shared-provider` - **Responsibility**: Essential Import Controller + Essential Export Controller - **Key code**: `pkg/controllers/imports/`, `pkg/controllers/export/` ### infra-provider (monorepo) - **Path**: `infra-provider` | Subdirectory | Go Module | Responsibility | |--------------|-----------|----------------| | `cluster-controller/` | `github.com/tidbcloud/eks-provider` | Dedicated Import/DM/BR Controller | | `api/` | `github.com/tidbcloud/eks-provider/api` | DMJob CRD + TiCDC Service API definitions | | `ticdc-service/` | `github.com/tidbcloud/ticdc-service` | Dedicated TiCDC Controller + CRD | | `infra-api/` | `github.com/tidbcloud/infra-api` | Essential/Dedicated BR Controller + CRD | | `infra-api/api/` | `github.com/tidbcloud/infra-api/api` | Import/Export/Backup/Restore/TiCDC CRD type definitions | ### cluster-service-ng - **Path**: `cluster-service-ng` - **Go Module**: `github.com/tidbcloud/cluster-service-ng` - **Responsibility**: Regional control plane (Export DAG Actions + BR Workflow Actions + DM Cluster + TiCDC API) - **Key code**: `domain/workflow/action/{export,br}/`, `domain/workflow/action/apply_dm_cluster.go`, `api/tidbcloud/ticdc-service/v1/` ### tidb-management-service (monorepo) - **Path**: `tidb-management-service` | Subdirectory | Go Module | Responsibility | |--------------|-----------|----------------| | `nextgen-global-service/` | `github.com/tidbcloud/tidb-management-service/nextgen-global-service` | Premium Export/BR Global control plane | | `serverless-global-service/` | `github.com/tidbcloud/tidb-management-service/serverless-global-service` | Essential DM Global control plane (`ServerlessDMRegionalService`) | | `dedicated-global-service/` | `github.com/tidbcloud/tidb-management-service/dedicated-global-service` | Dedicated Global control plane | | `tidb-mgmt-service/` | `github.com/tidbcloud/tidb-management-service/tidb-mgmt-service` | Management service gateway | ### tcidm - **Path**: `/tcidm` - **Go Module**: `tcidm` - **Responsibility**: Regional HTTP-to-CR translation layer (Dedicated Import -> Import CR, Dedicated DM -> DMJob CR) - **Key code**: `api/api.go`, `internal/service/regional_import_service/`, `internal/service/dmjob.go` ### tidb-operator - **Path**: `tidb-operator` - **Go Module**: `github.com/pingcap/tidb-operator/v2` - **Responsibility**: TiBR/Backup/Restore Controller + CRD - **Version**: v1 (`release-1.x`) is used for Dedicated, v2 (`main`) is used for Premium ### runbooks - **Path**: `runbooks` - **Responsibility**: Alert rule definitions (Prometheus Rules + Grafana Dashboards + Runbook docs) ### app-delivery - **Path**: `app-delivery` - **Responsibility**: GitOps deployment repository (ArgoCD App definitions + version files + bump workflow) --- ## 7. Important `replace` Mappings > The following `replace` directives affect module resolution. The import paths seen in source code may differ from the actual code locations. | Source `go.mod` | `replace` Mapping | Meaning | |-----------------|-------------------|---------| | `serverless-service/go.mod` | `github.com/tidbcloud/infra-api => github.com/tidbcloud/infra-provider/infra-api` | When `serverless-service` imports `infra-api`, it is actually resolved to `infra-provider/infra-api/`. | | `serverless-service/go.mod` | `github.com/tidbcloud/infra-api/api => github.com/tidbcloud/infra-provider/infra-api/api` | The CRD types are actually located in `infra-provider/infra-api/api/`. | | `dataflow-service/go.mod` | `github.com/tidbcloud/infra-api => github.com/tidbcloud/infra-provider/infra-api` | When `dataflow-service` imports `infra-api`, it is actually resolved to `infra-provider/infra-api/`. | | `dataflow-service/go.mod` | `github.com/tidbcloud/infra-api/api => github.com/tidbcloud/infra-provider/infra-api/api` | The CRD types are actually located in `infra-provider/infra-api/api/`. | | `dataflow-service/dataflow-service-ng/go.mod` | `github.com/tidbcloud/infra-api => github.com/tidbcloud/infra-provider/infra-api` | Premium dataflow is also resolved to `infra-provider/infra-api/`. |