# Design: Expanded Scope — Entire TiDB Cloud Console Dev Environment Codename: **controlplane9** | Repo: [tidbcloud/controlplane9](https://github.com/tidbcloud/controlplane9) Supersedes: v2 design ([20260331-design-cp9-v2-global-cp-only.md](20260331-design-cp9-v2-global-cp-only.md), global CP only) and prior revision of this document (2026-04-15, stub-heavy L2/L3) | Status: APPROVED | Date: 2026-04-17 Related docs, kept in sync with this revision: [20260413-design-cp9-stakeholder-guide.md](20260413-design-cp9-stakeholder-guide.md), [20260417-design-cp9-landing-page.md](20260417-design-cp9-landing-page.md). > Historical command-surface note (2026-06-01): this design predates the command cleanup train. Its command examples may mention removed ideas such as `sync`, `onboard`, `config`, `persona`, `crd`, `migrate`, `test`, `cost`, `reaper`, `stub-gen`, `overlay`, `verify`, `feature-cycle`, `topology`, or `up --persona` / `up --discover` / `up --profile` / `up --remote`. Current runnable surfaces are `setup`, `doctor`, `up`, `status`, `logs`, `deploy`, `destroy`, direct `pause`, and `db dump-init`. --- ## Problem Statement TiDB Cloud's dev environment spans AWS, GCP, Azure, and Alicloud — 6 cluster types, 73 services, 11+ teams, 112+ idle AWS nodes plus a smaller non-AWS footprint running 24/7. Shared state causes flakiness, collisions block cross-team work, and most compute is idle most of the time. The v2 design ([20260331-design-cp9-v2-global-cp-only.md](20260331-design-cp9-v2-global-cp-only.md)) proved per-developer ephemeral environments work for the global CP (O1, ~42 services). cp9 extends that model to the full dev environment. **Goal.** Decommission the entire dev environment. Replace with per-developer ephemeral environments driven by the real production code path. **Strategy.** L1 services run locally in k3d, with proto/swagger stubs for off-path deps. L2 and L3 are not stubbed — cp9 runs real cloud via the production code path. `cp9 destroy` is a two-stage guarantee: local cleanup (k3d + Zero), then tag-scoped force-purge across clouds (authoritative). Rationale: stubs cannot exercise the paths that carry the actual risk for features crossing the customer trust boundary (private link, VPC peering, IAM trust, KMS). (Tech decision rationale captured 2026-04-16; working notes not committed here.) **First wedge.** Phase 2 ships Premium autoscaler end-to-end in cp9 before expanding to other product lines — a bounded proof that the real-cloud path carries a full L1 + L2 + L3 feature. See [Rollout](#rollout). --- ## Three-Layer Scenario Taxonomy Every TiDB Cloud feature touches one or more layers. cp9's fidelity strategy differs per layer. ``` Open API call (input) | v +-------------------------------------------------------+ | L1: Microservice & Controller Code | | Business logic, cross-service RPC, cross-VPC/region. | | Includes self-defined CRD controllers | | (appbox, approute, etc.) — business logic. | | All code owned by us. Delivery: FAST. | +-------------------------------------------------------+ | system edge: crossplane CRD, pulumi, terraform, cloud SDK v +-------------------------------------------------------+ | L2: Cloud Infrastructure | | Crossplane CRDs, pulumi stacks, terraform configs, | | cloud SDK calls (AWS / GCP / Azure / Alicloud). | | Delivery: MEDIUM (slow, fragile, rate-limited). | +-------------------------------------------------------+ | system edge: tidb-operator CRD, k8s manifests v +-------------------------------------------------------+ | L3: TiDB Kernel | | TiDB / TiKV / PD / TiFlash / TiCDC component changes. | | Stable codebase, slowest iteration, highest risk. | | Delivery: SLOW. | +-------------------------------------------------------+ ``` Self-defined CRDs (appbox, approute) and their controllers are **L1** — they encode business logic. Only crossplane / pulumi / terraform / cloud-SDK interactions are L2. --- ## Architecture ``` +=============================== DEVELOPER MAC ===============================+ | cp9 CLI + Agent Plugin | | local k3d — single cluster, namespace-isolated: | | ns: staging-ms (Global CP) | | ns: cluster-service (Dedicated Regional — AWS / GCP / Azure) | | ns: cluster-ng-service (Premium Regional — AWS / Azure / Alicloud) | | ns: tidb-admin (Dataplane controllers) | | ns: crossplane-system (REAL providers, per-cloud dev creds) | +======================================|=====================================+ | tunnel (VPN / SSM / private link) v +=========================== AWS CONTROL PLANE VPC A =========================+ | CP-CLOUD: network / IAM created by prod code path via crossplane + | | cloud SDK. VPC endpoints, SGs, IAM roles + trust, KMS grants, | | Route53, ... tagged cp9-owner / cp9-session. | | | | NAT gateway — stable EIP = single cp9 dev egress address | +===========|=============================|===================================+ | | | prod createcluster path | public internet from NAT EIP v v +=========================+ +==================================================+ | PER-SESSION DATAPLANE | | GCP / Azure / Alicloud public APIs | | (in target cloud) | | (per-cloud dev creds on the calling pod) | | | | Auth0 tenant | | EKS / GKE / AKS / ACK | | Dataplane k8s API servers (public endpoint) | | + tidb-operator | | Dataplane business APIs | | + real TiDB cluster | +==================================================+ +=========================+ Out of scope: customer VPC. S3 / RDS / KMS provisioned manually per feature. ``` **Single k3d cluster, namespace isolation** for the control plane (unchanged from O1). Services must use in-cluster k8s service discovery, not hardcoded cluster endpoint URLs. CP↔CP network-policy and cross-VPC auth bugs still verify on staging. **No stubs below L1, no cp9-owned pool layer.** The dataplane exists because the real production code path creates it, and goes away because the two-stage destroy guarantee destroys it. Kernel work uses the same path; kernel's feature-path services just happen to include TiDB kernel components. ### Service Classification | Classification | What | Where | | ----------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------- | | **REAL (always)** | Cluster-lifecycle services — cp9 provisions the real dataplane by running their production code path | local k3d | | **REAL (feature-path)** | Feature-path L1 services + self-defined CRD controllers | local k3d, built from source | | **STUB (API)** | L1 deps not on the feature path and not cluster-lifecycle | local k3d, proto/swagger-generated | | **CP-CLOUD** | Real network / IAM created by CP services via crossplane + cloud SDK | AWS Control Plane VPC A; tagged + reaped | | **DP-REAL** | Real dataplane (EKS/GKE/AKS/ACK + tidb-operator + TiDB/TiKV/PD/TiFlash/TiCDC) from the prod path | target cloud, per-session | | **SKIP** | Unreachable from the feature path | not deployed | Stubs exist only for L1 deps off the feature path. Self-defined CRD controllers are never stubbed. L2 and L3 have no stubs at all. ### Network reachability Regional control planes span four clouds (AWS: serverless / dedicated / premium; GCP: dedicated; Azure: dedicated / premium; Alicloud: premium). cp9 collapses this to a **single network topology**: one local k3d, one AWS dev VPC A, NAT egress to every cloud's public APIs. Why a single AWS VPC covers all four clouds — every inter-service auth boundary in prod is identity-based, not network-based: | Boundary | Mechanism | Needs VPC membership? | | -------------------------------- | ----------------------------------------- | --------------------- | | CP → dataplane business API | Auth0 M2M token | No | | CP → dataplane k8s API | kubeconfig + public endpoint | No | | AWS regional CP → dataplane | IAM trust policy (STS AssumeRole) | No | | GCP regional CP → dataplane | SA trust / Workload Identity Federation | No | | Azure regional CP → dataplane | Managed Identity / service principal | No | | Alicloud regional CP → dataplane | RAM trust | No | Cross-cloud reachability reduces to public internet + the right credential on the pod. The NAT EIP is the single cp9 egress identity that non-AWS dev accounts allowlist. **Known gaps** (accepted scope exclusions): - **Private-network features.** PrivateLink / PSC / Private Endpoint from the CP side — CP traffic is public internet. Dataplane-side private-link creation still happens as part of the feature's real code path. - **IP-bound IAM conditions.** Dev policies must allowlist the NAT EIP where `aws:SourceIp` or equivalents are used. - **GCP VPC-SC / equivalent perimeters.** Dev project exempts the NAT EIP or runs without VPC-SC. - **WIF bound to specific cluster principals.** Dev bindings accept a cp9 AWS principal as a valid federated subject. - **Egress geography.** A GCP europe-west4 call from AWS us-west-2 NAT adds transatlantic latency; correctness unaffected. **Operational prerequisites.** - Allocate a stable EIP for VPC A's NAT gateway; document as the cp9 dev egress address. - Register the EIP in each non-AWS dev account's network + IAM allowlists. - Create dev-equivalent trust policies in each non-AWS cloud that accept the cp9 AWS principal. Owned per product line — Dedicated @cwen (GCP / Azure), Premium @tennix (Azure / Alicloud). --- ## End-to-End Flow 1. **`cp9 up`.** Local k3d boots. Crossplane installs with per-cloud dev credentials. Tunnel to VPC A comes up. 2. **Control plane provisioning.** Crossplane providers and CP services on k3d make real cloud calls. AWS resources land in VPC A; non-AWS cloud resources are created via public APIs using per-cloud creds. All are tagged `cp9-owner` / `cp9-session`. 3. **Dataplane provisioning.** The feature hits a cluster-lifecycle service (tidb-mgmt-service, cluster-service, cluster-ng-service, dataplane-manager — varies by product line and cloud). That service runs its production provisioning path, which creates the per-session dataplane (EKS/GKE/AKS/ACK + tidb-operator + the requested TiDB cluster) in the target cloud. 4. **Feature execution.** The feature runs end-to-end. Real network, IAM, private link, and peering paths execute as in prod. 5. **`cp9 destroy`.** Runs the two-stage destroy guarantee. --- ## Destroy Guarantee Cloud resources created on behalf of a session cost real money and hold real IAM privileges — leaks are unacceptable. `cp9 destroy` is a **two-stage, idempotent** operation. Both stages always run: ``` Stage 1 — Local cleanup (soft): Delete the k3d cluster (k3d cluster delete ), which kills every pod and all local state in one shot: CP services, crossplane, metadb, CRDs, secrets. Delete the session's TiDB Cloud Zero instance via Zero API. Remove the .cp9/ session directory. cp9 does NOT call cluster-lifecycle services' delete APIs — pods die with the cluster; cloud resources carry cp9-owner / cp9-session tags and Stage 2 finds them. Stage 1 failures are logged but NOT fatal — Stage 2 still runs. Stage 2 — Force purge (hard, authoritative): Tag-scoped sweep of every cloud resource carrying this session's cp9-owner / cp9-session tag (or cloud-equivalent label). Covers: - Every resource type the prod path can create, across all clouds (EKS/GKE/AKS/ACK, EC2/GCE/VM, ELB, VPC + subnets + routes + gw, SG, IAM roles/policies, KMS, Route53, S3/GCS/Blob, SQS, ...) - Crossplane-created resources in VPC A Deletes in dependency-safe order with retries. Exits non-zero if any tagged resource survives. Exit contract: cp9 destroy returns 0 ONLY when a post-purge tag scan finds zero cp9-owner / cp9-session resources across all clouds and regions. ``` Stage 2 builds on the existing O1 AWS reaper and extends it to the full per-cloud resource-type matrix the prod `createcluster` path can create. **Load-bearing invariant:** every resource created on behalf of a session — including indirectly by crossplane providers and by the prod path — must carry `cp9-owner` / `cp9-session` tags (or cloud-equivalent labels). Without this, Stage 2 can't find what to purge. Force-purge is now an internal stage of `cp9 destroy`, not a standalone user command. Recovery from sessions whose `cp9 destroy` was never invoked should re-run `cp9 destroy` against the saved session state; separate admin sweeps must stay behind explicit operator tooling, not the product CLI surface. --- ## Capabilities | Capability | Layer | Description | | ---------- | ----- | ---------------------------------------------------------------------------------------------------------- | | **C1** | L2/L3 | Tunnel from local k3d to AWS Control Plane VPC A; VPC A NAT egress carries traffic to all non-AWS cloud APIs | | **C2** | L2/L3 | Crossplane + per-cloud dev creds installed REAL in local k3d (no stub CRD controller) | | **C3** | L2/L3 | Cluster-lifecycle services run unchanged in cp9; per-service audit enforces externally-configurable + tag-complete preconditions (no runtime cp9 code; audit-only) | | **C4** | L2/L3 | Force-purge reaper — tag-scoped authoritative cleanup across clouds; `cp9 destroy` stage 2 | | **C5** | L1 | Per-persona topology templates declaring each feature's service set + dependency edges; self-defined CRD controllers always REAL | **Primary risks:** - **C3 per-service audit.** Each cluster-lifecycle service must (a) accept dev account / region / VPC / infra-stack target purely via env/ConfigMap and (b) apply `cp9-owner` + `cp9-session` tags to every resource it creates (directly via cloud SDK and indirectly via crossplane CRs). Failing either precondition is a blocking Phase-2 prerequisite that requires a code-change PR against the owning repo. Audit scope per service, per cloud. - **C4 tag discipline.** The reaper is only as good as the tags. Any resource created without `cp9-owner`/`cp9-session` (or cloud-equivalent labels) is invisible to stage 2. Full-surface audit required — see Open Questions. - **C5 template drift.** A stale template means cp9 boots the wrong service set. Mitigation: templates live in the controlplane9 repo alongside capability code; per-team owners review template PRs in their vertical; cp9 fails loudly on missing services rather than silently producing a broken environment. ### C5 detail — Topology templates `cp9 up --persona ` loads a template that declares the feature's service set: L1 services that run REAL (including self-defined CRD controllers), L1 deps to stub, and required CP-CLOUD / DP-REAL capabilities. ``` # topologies/premium-autoscaler.yaml (example) persona: premium-autoscaler real: - autoscaler-engine - cluster-ng-service - dataplane-manager - appbox-controller - approute-controller stub: - account-svc # proto - billing-svc # proto requires: - cp-cloud # VPC A + crossplane-created IAM/network - dp-real # per-session real dataplane via prod createcluster ``` Templates live in `controlplane9/topologies/` and ship with the CLI — one template per active persona, reviewed by team leads in each vertical. Auto-discovery (static analysis of CRD ownership and client usage) is a future enhancement, not a Phase 1 or Phase 2 requirement. --- ## Rollout Rollout follows vertical product domains. Teams adopt cp9 as soon as their required capabilities are ready. **Phase 1 — Global Control** (L1; needs C5 + O1 work; in progress) Teams: Global control, Billing, Account, Auth. **Phase 2 — Core Products + Ecosystem + Kernel** (needs C1 + C2 + C3 + C4 + C5) One capability set serves everyone who needs real cloud. Kernel is not a separate phase — its feature-path services happen to include TiDB kernel components. **First target: Premium autoscaler end-to-end.** Before expanding across product lines, ship one bounded E2E demo — the narrowest wedge that proves the real-cloud path carries a full L1 + L2 + L3 feature on cp9. AWS-only; Azure / Alicloud coverage for Premium follows. See [Premium Autoscale landing page](https://pingcap.feishu.cn/wiki/DOhowJv0Zib9eIkdz4mc9XEznXg). Exit criteria: - `cp9 up --persona premium-autoscaler` stands up local k3d (global CP + premium regional CP + autoscaler-engine), AWS VPC A CP-CLOUD resources, and a per-session AWS premium dataplane (EKS + tidb-operator + TiDB/TiKV/PD + warm pool). - Developer drives a workload generator; CPU sustains >80% on the target component; autoscaler-engine emits a scale-out and replicas increase in the dataplane. - Scale-in triggers after cooldown (CPU < 30%, ≥ 5 min). - `cp9 destroy` exits 0 with zero surviving tagged resources. **Expand from there:** - **Product lines:** Starter/Essential @yuqing (L1+L2+L3), Dedicated @cwen (L1+L2), Premium @tennix (L1+L2+L3; non-AWS clouds after the wedge) - **Ecosystem:** BR/Import/Changefeed @jianyuan (L1+L2+L3) - **Kernel:** @wangcong compute, @jinpeng storege (L3) **Phase 3 — Peripheral + New Region** - **Peripheral** (L1; needs C5): Frontend @bosn, Clinic @zheming, OPS portal @charles, O11Y @xinglu - **New Region** (L2; needs C1 + C2 + C4): @alexquan **Phase 4 — Decommission.** Once all domains confirm cp9 replaces their dev workflow, freeze and remove the shared dev clusters. --- ## Decommission & Cost **Decommission trigger.** Each product domain team lead signs off that cp9 replaces their dev workflow. Staging remains for PM/QA validation and performance testing. ``` Phase 1 adopted -> freeze dev-base staging-ms workloads. Phase 2 adopted -> freeze seed clusters, dev-us-east-1-f01, seed-ng clusters, dataplane clusters. Phase 3 adopted -> all remaining dev workloads frozen. All confirmed -> decommission: dev-base-us-west-2 23 nodes dev seed clusters 6 nodes dev-us-east-1-f01 45 nodes dev dataplane clusters 38 nodes + static IAM, ECR, Route53, VPC, KMS, S3, SQS + equivalent non-AWS dev regional CPs + dataplanes (GCP / Azure / Alicloud; count per product line — Open Question) --- AWS: 112+ nodes. Non-AWS: TBD. ``` **Not decommissioned.** Dev AWS account itself, IAM scaffolding for the `cp9-developer` role, base VPC/Route53 records the real cloud path relies on — cheap relative to the idle-node footprint. **Cost shape after cp9:** - **Eliminated:** 24/7 idle compute/storage across all dev clouds. - **Per-session:** real EKS/GKE/AKS/ACK + TiDB provisioned by prod createcluster while the developer works; torn down on destroy. Order-of-magnitude smaller than the 24/7 baseline because it exists only while work is happening. - **Ongoing:** cheap VPC A network/IAM (created on demand, reaped); NAT egress to non-AWS clouds — negligible at control-plane API volumes. - **Local k3d:** developer's Mac — $0. **Seed data mirrors from staging** (databases, etcd CRs, config) so per-session environments start in a realistic state rather than synthesized from scratch. Final savings number depends on concurrency + session duration — see Open Questions. --- ## Open Questions **Networking & Tunneling** 1. **Tunnel from Mac → VPC A.** VPN (Tailscale / OpenVPN), AWS SSM port-forward, or private-link? Latency and auth impact on crossplane provider calls and cloud SDK calls from k3d-hosted CP services. 2. **NAT EIP allowlisting in non-AWS dev accounts.** Who owns landing the cp9 dev egress IP in GCP / Azure / Alicloud dev network + IAM allowlists? Infra-team coordination with per-cloud dev account owners. **Blocking for multi-cloud Phase 2 adoption.** 3. **Dev-equivalent trust policies per non-AWS cloud.** Dev versions must accept a cp9 AWS principal as a federated subject. Policy authoring per product line — @cwen for GCP/Azure dedicated, @tennix for Azure/Alicloud premium. 4. **VPC-SC and equivalent perimeter exceptions.** If the dev GCP project enforces VPC Service Controls (or Azure/Alicloud equivalents), the AWS NAT EIP is an outside source and traffic blocks. Need perimeter exception or disable in dev. **Tagging & Destroy Guarantee** 1. **Tag-coverage audit for C4.** Every creation path must carry `cp9-owner`/`cp9-session` tags (or cloud-equivalent labels) — crossplane providers, cloud-SDK call sites in cluster-lifecycle services, indirect resources (EKS-managed ELBs, EBS, GKE/AKS/ACK equivalents that tag asymmetrically). Any untagged creation path is a silent leak vector. **Blocking for Phase 2.** **Rollout Prerequisites** 1. **Wiring cluster-lifecycle services to dev.** The prod code path picks target environment (account, region, infra stack) from config. Pure config, or does it require code changes in cluster-service / cluster-ng-service / dataplane-manager / per-cloud variants? Spike per service. 2. **Cold-start latency budget.** Prod `createcluster` routinely takes 15–30 min (EKS). GKE / AKS / ACK vary. Keep O1's <10 min `cp9 up` SLA, or accept longer first boot for features needing the dataplane and fast iteration afterward? 3. **Cloud-resource provisioning time via crossplane (VPC A).** VPC endpoints, IAM trust, KMS grants take minutes. Does this fit alongside the dataplane cold-start budget, or can they run in parallel? 4. **Starter/Essential shared compute model.** staging-us-east-1-f01 is a different architecture (shared serverless platform, 250+ tenant namespaces, 45 nodes). Does one-dataplane-per-session apply, or does Starter/Essential need a modified model? **Owner:** @yuqing. 5. **Premium autoscaler warm pool.** The scaling executor relies on a warm pool for seconds-scale scale-out. Does the prod `createcluster` path provision the warm pool as part of a per-session dataplane, or does cp9 need to prime it separately before the wedge demo? **Owner:** @tennix. **Validation** 1. **Topology template catalog — ownership & lifecycle.** Templates live in `controlplane9/topologies/`. Per-team ownership for template correctness in their vertical. Do templates get reviewed per capability PR, or on a fixed cadence? Who approves a new persona? 2. **Adoption validation criteria.** Premium autoscaler wedge (see Rollout) is the first concrete validation. For subsequent teams, what does "team lead signs off" concretely mean? Proposal: team develops and tests one real feature end-to-end using cp9, documented with comparison to previous workflow. --- ## Out of Scope - **O1 Verification Plan** — separate document ([o1-verification-plan.md](o1-verification-plan.md)). - **Detailed implementation design** — lives in [20260425-design-cp9-detail.md](20260425-design-cp9-detail.md) (full implementation contract) and per-capability follow-ups as needed. - **Customer-analog resource setup** — developer provisions customer-side S3 / RDS / KMS manually when a feature requires them. - **O3 Multi-Agent Orchestration** — mentioned only. - **O4 Remote Environment & DR** — mentioned only. - **Agent plugin detailed spec** — ships with O2. - **Per-team onboarding guides** — written as each domain adopts.