# controlplane9: Stakeholder Guide **Date:** 2026-04-13 | **Author:** @yifan | **Status:** DRAFT > This document is for team leads across TiDB Cloud console development. It explains what's changing in the dev environment, how it affects your team, and what we need from you. --- ## Why This Matters to Your Team The entire dev environment is being decommissioned — global CP (dev-base EKS), regional CP (dev seed clusters), data plane (dev dataplane EKS), and all related cloud resources. It causes test flakiness from shared state, wastes infrastructure cost, and blocks agent-driven development. Every team that uses any part of the dev environment is affected. **controlplane9 (cp9)** replaces it with local ephemeral environments. Each developer runs `cp9 up` on their Mac and gets an isolated control plane with only the services they need. When done, `cp9 destroy` cleans up everything. --- ## What Stays the Same - **Staging** remains for PM/QA final validation and performance testing — unchanged - **Production** — unchanged - **Your repos** — no structural changes required. Your existing Makefiles, Dockerfiles, proto definitions, and swagger specs continue to work - **CI/CD pipelines** — ArgoCD staging/prod deployments unchanged - **Code review process** — unchanged ## What Changes | Before (shared dev env) | After (cp9) | | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------- | | Deploy to shared EKS (global CP, regional CP, dataplane), wait for other teams | `cp9 up` — isolated env on your Mac; CP in minutes, real dataplane via prod createcluster on first use | | All 42+ services running, most irrelevant | Only feature-path services run; rest auto-stubbed or skipped | | Shared database state, test pollution | Per-session real dataplane + staging data mirror; torn down on `cp9 destroy` | | Manual E2E after all teams deploy | Test locally end-to-end in your own environment anytime | | Dev env runs 24/7, significant infrastructure cost | Ephemeral — exists only while you develop, near-zero idle cost | --- ## Three-Layer Scenario Taxonomy Every TiDB Cloud feature touches one or more layers. Understanding which layer your team operates in determines your cp9 migration path. ``` Open API call (input) | v +-------------------------------------------------------+ | L1: Microservice & Controller Code | | Business logic, cross-service RPC, cross-VPC/region. | | Includes self-defined CRD controllers | | (appbox, approute, etc.) — these are business logic. | | All code owned by us. Delivery: FAST. | +-------------------------------------------------------+ | | system edge: crossplane CRD, pulumi, terraform, AWS SDK v +-------------------------------------------------------+ | L2: Cloud Infrastructure | | Crossplane CRDs, pulumi stacks, terraform configs, | | AWS SDK direct calls. Cloud provider dependency. | | Delivery: MEDIUM (slow, fragile, rate-limited). | +-------------------------------------------------------+ | | system edge: tidb-operator CRD, k8s manifests v +-------------------------------------------------------+ | L3: TiDB Kernel | | TiDB/TiKV/PD/TiFlash/TiCDC component changes. | | Stable codebase, slowest iteration, highest risk. | | Delivery: SLOW. | +-------------------------------------------------------+ ``` | Layer | cp9 Approach | |---|---| | L1: Microservice & controller code | Feature-path services run REAL in local k3d. Off-path deps run as proto/swagger stubs. Self-defined CRD controllers (appbox, approute) are always REAL. | | L2: Cloud infrastructure | **Real cloud, no stubs.** Crossplane runs REAL in local k3d with per-cloud dev credentials. AWS resources land in a dev Control Plane VPC A; non-AWS clouds (GCP / Azure / Alicloud) are reached via NAT egress from VPC A. All resources tagged `cp9-owner` / `cp9-session` and reaped on destroy. | | L3: TiDB kernel | **Real cloud, no stubs.** The production `createcluster` path stands up a per-session dataplane (EKS / GKE / AKS / ACK + tidb-operator + real TiDB/TiKV/PD/TiFlash/TiCDC) in the target cloud. No shared pool; nothing shared across developers. | **`cp9 destroy` is a two-stage guarantee.** Stage 1 runs the production teardown path. Stage 2 is an authoritative tag-scoped force-purge that sweeps any leftover `cp9-owner`/`cp9-session` resources across all AWS types and regions. `cp9 destroy` returns 0 only when the post-purge scan finds zero tagged resources. --- ## Your Team's Migration Phase ### Phase 1: Global Control **Teams:** Global control, Billing, Account, Auth **Capability:** L1 (done). Services: Everything in `staging-ms` + `tidb-management-service` namespaces. O1 is feature-complete. Verification ongoing with CMEK M1+M2 historical redo and Connection Setting V2. **What we need from you:** Validate `cp9 up` boots your services. Report gaps. ### Phase 2: Core Product Lines + Core Ecosystem + Kernel **Product lines:** Starter/Essential + Premium Regional @yuqing, Dedicated @cwen, Premium @tennix **Ecosystem:** BR/Import/Migrate @jianyuan, CDC/Dataflow @wangbing **Kernel:** @wangcong (same capability set, not a separate phase) **Capabilities needed:** CR-mediated topology for self-defined CRDs (L1 extension) + tunnel from local k3d to AWS dev Control Plane VPC A (NAT egress reaches GCP / Azure / Alicloud public APIs) + crossplane installed REAL in local k3d with per-cloud dev creds + cluster-lifecycle services configured to target dev so the prod `createcluster` path creates real per-session dataplane in the target cloud + force-purge reaper for the destroy guarantee. One capability set serves everyone who needs real cloud — kernel uses the same path because its feature-path services happen to include TiDB kernel components. **What we need from you:** Validate cp9 covers your full vertical (global + regional + dataplane code paths). Develop and test one real feature end-to-end using cp9. ### Phase 3: Peripheral + New Region **Teams:** Frontend @bosn, Clinic @zheming, OPS portal @charles, O11Y @xinglu, New Region @alexquan **Capability needed:** L1 sufficient for peripheral (API gateway exposure). New Region uses the tunnel + crossplane + reaper subset when it touches cloud resources. **What we need from you:** Confirm your services can connect to a local cp9 environment. --- ## Stakeholder Scenario Matrix | Team | Services (cluster) | Layers | Phase | Status | |---|---|---|---|---| | Global control | account-*, authserver-*, tidb-mgmt-*, serverless-svc-* (staging-base) | L1 | Phase 1 | O1 ready, verification in progress | | Starter/Essential + Premium Regional @yuqing | serverless-*, cluster-ng-* (staging-base, staging-us-east-1-f01) | L1+L2+L3 | Phase 2 | Aligned | | Dedicated @cwen | cluster-service (staging-seed), infra/dedicated-api-server (staging-base) | L1+L2 | Phase 2 | Aligned | | BR, Import, Migrate @jianyuan | dataflow-*, tcidm (staging-base, staging-seed) | L1+L2+L3 | Phase 2 | Aligned | | Premium @tennix | cluster-ng-service, dataplane-manager (staging-seed-ng) | L1+L2+L3 | Phase 2 | Aligned | | CDC Dataflow @wangbing | dataflow-*, ticdc-service (staging-base, staging-seed) | L1 | Phase 2 | Aligned | | Kernel @wangcong | tidb-operator, tidb components (tidb-admin) | L3 | Phase 2 | Aligned — uses the same real-cloud path as product lines | | O11Y @xinglu | o11y-*, monitoring (various) | L1 | Phase 3 | Isolated env, minimal changes | | Clinic @zheming | clinic (staging-base) | L1 | Phase 3 | Pending discussion | | OPS portal @charles | ops portal (staging-base) | L1 | Phase 3 | Pending discussion | | Frontend @bosn | Frontend app -> API gateway | L1 | Phase 3 | Pending discussion | | New Region @alexquan | Regional CP deployment + registration | L2 | Phase 3 | Pending discussion | **Cloud coverage per product line:** Dedicated @cwen — AWS / GCP / Azure. Premium @tennix — AWS / Azure / Alicloud. Starter/Essential @yuqing — AWS. One local k3d + one AWS Control Plane VPC A serves all four clouds (see [20260417-design-cp9-expanded-scope.md](20260417-design-cp9-expanded-scope.md) — Network reachability). --- ## Discussion Questionnaire We will schedule a 30-minute 1:1 with each team. Please think about these questions beforehand: 1. **Services owned:** Which deployments do you maintain in which staging cluster(s)? 2. **Dev workflow:** Walk through your last feature development cycle end-to-end — from picking up the task to merging the PR. 3. **Wait points:** Where do you wait? What's manual that could be automated? 4. **Cross-team dependencies:** Which other teams' services do you call? How do you coordinate? 5. **Cloud infra needs:** Do you need real AWS/GCP resources during development (crossplane, terraform)? 6. **Testing strategy:** How do you test today? Unit / integration / E2E? Where do E2E tests run? 7. **Adoption willingness:** What would make you adopt a new dev environment tool? What would block you? 8. **Pain points:** Top 3 things that slow you down in your current dev workflow. --- ## Open Questions — By Team These are specific questions we'd like to discuss in the 1:1 meetings: 1. **Dedicated @cwen:** Does your dev workflow require real crossplane resource provisioning, or is CRD status simulation sufficient? How many crossplane CRDs are actively used in feature development? Answer: For most features, CRD status simulation is sufficient. Crossplane CRDs are rarely directly manipulated in feature development, while self-defined CRDS (e.g. appbox, approute) are more commonly used. 2. **Premium @tennix:** Same as dedicated. Additionally, how does autoscaler-engine testing work today? Does it need real dataplane metrics or can they be simulated? Answer: For CRD part, same as dedicated. For autoscaler-engine, before PR submitted local development testing, metrics can be simulated. After deployment to staging, real metrics are needed for validation. 3. **Starter/Essential @yuqing:** The staging-us-east-1-f01 cluster has 250+ tenant namespaces and 45 nodes. What does a typical feature dev cycle look like? Do you need actual TiDB clusters running or just the control path? Answer: Doing edge-case testing currently. - **Dev workflow:** RU v2 example: kernel develops and provides image first. Then use pulumi to swap the dev image. Typically modify provider (controller in f01) first, then serverless service; if OpenAPI changes are needed, also modify mgmt. Also have manual dev image replacement for testing. - **Wait points:** Different developers interfere with each other in the shared dev environment. Cloud resources cause conflicts — e.g., pulumi creates resource A, someone else modifies pulumi, same pulumi stack executed by different people overwrites each other. - **Cross-team dependencies:** PM, kernel, mgmt, dataflow. - **Feature dev involving cloud resources:** Use pulumi (not crossplane — all static) for testing on dev. - **Mock data needed:** e.g., global metadb. - **Other comments:** Kernel end-to-end testing is an important verification step. Before broader rollout, need to have dataplane + kernel integration testing capability ready. 4. **BR, Import, Migrate @jianyuan:** dataflow-*, tcidm spans both global and regional clusters. How does your dev workflow cover L1+L2+L3? Answer: - Development covers all tiers, vertical across three-layer architecture: from mgmt, global service, regional controller, dataplane stub, to operating cloud resources via crossplane. - Testing requires real clusters and prepared kernel data. - Premium and Dedicated are similar — Premium's cloud interaction is encapsulated in dataplane manager; Dedicated has more direct crossplane operations. Serverless is a very different two-layer architecture with complex dependencies in -f01 EKS. - Example: Premium import-into. Global (dataflow service) connects directly to dataplane for pre-check, then connects to cluster service to initiate operation. Development requires waiting for kernel to provide pre-check interface first. - Pain points: In addition to known pain points, image replacement testing conflicts. 5. **CDC dataflow @wangbing:** dataflow-service and ticdc-service appear in both global and regional clusters. What's the testing boundary? Do you test globally, regionally, or both? Answer: Both. 6. **Kernel @wangcong:** Your workflow is "replace TiDB cluster image and test." Can this be done in k3d, or does it require a full cloud dataplane? Answer: It requires a full cloud dataplane. ### Architecture decisions (resolved) - **L2/L3 strategy:** No stubs below L1. Crossplane runs REAL in local k3d with per-cloud dev credentials. Cluster-lifecycle services run REAL and call their production `createcluster` path, which stands up a real per-session dataplane (EKS / GKE / AKS / ACK + TiDB cluster) in the target cloud. Kernel work uses the same path, so there is no separate kernel phase. - **Multi-cloud networking:** One local k3d, one AWS dev Control Plane VPC A, NAT egress to GCP / Azure / Alicloud public APIs. Inter-service auth is identity-based (IAM trust, Auth0 M2M, k8s public endpoints), so cross-cloud VPC peering is not required. - **Dev-environment isolation:** No shared pool, no namespace-prefix scheme. Each cp9 session owns its own real cloud resources end-to-end. - **Destroy guarantee:** `cp9 destroy` is two-stage — run the production teardown, then an authoritative tag-scoped force-purge across all clouds. Exits non-zero if any `cp9-owner` / `cp9-session` resource survives. - **Customer-side resources:** Out of scope. When a feature needs customer-analog S3 / RDS / KMS, the developer provisions them manually. - **Team 1:1 answers from @cwen, @tennix, @yuqing, @jianyuan, @wangcong** (above) about stubbing fidelity predate this direction and should be re-read in that light — the design question those answers addressed ("is CRD stubbing sufficient?") is no longer posed because cp9 does not stub L2/L3 at all. --- ## Adoption Cost | Factor | What to expect | | --------------------- | ------------------------------------------------------------------------------------------------------- | | **Integration cost** | Low — your repos already have Makefiles, Dockerfiles, proto/swagger defs. No structural changes needed. | | **Learning cost** | Low — `cp9 up` and `cp9 destroy` replace kubectl/argocd commands. Self-describing CLI. | | **Migration cost** | Medium — need to validate all your services boot locally and stubs are sufficient. | | **Ops complexity** | Lower than shared dev env — per-developer isolation eliminates shared state issues. | | **Default usability** | Good — `cp9 up` works out of box with minimal config. Topology engine auto-classifies services. | --- ## Get Involved - **Repo:** [tidbcloud/controlplane9](https://github.com/tidbcloud/controlplane9) - **Detailed design:** `design.md` in this wiki directory - **Contact:** @yifan