# PRD: cp9 Cloud Dev Integration Substrate Status: APPROVED Date: 2026-06-15 Owner: cp9 ## Problem Statement Premium frontline developers need the current always-running dev environment to go fully offline. Its replacement must be temporary, session isolated, and capable of serving the same product integration role that the old-dev environment served. Local laptop cp9 is not sufficient for old-dev replacement acceptance. It runs the control plane on a developer laptop, which blocks or weakens several integration classes: - Product API traffic needs a public backend API endpoint. A local laptop does not provide the public api-gateway entrypoint shape used for old-dev style integration. - Product paths that provision cloud resources need enough cloud-hosted runtime fidelity to exercise behavior beyond local process wiring. - Laptop CPU, memory, network, and sleep behavior make the environment unstable as a shared replacement for old-dev integration. The new cp9 direction is a Cloud-hosted substrate for session-scoped product integration. The developer's laptop remains the controller and editor, but the product runtime runs in a temporary cloud session. Engineering decision on 2026-06-16: for the remote EC2+k3d substrate, the local `cp9` CLI is only a management/orchestration client. It may resolve configuration, create/select the remote EC2 session, package source inputs, transfer them over SSH, fetch logs/status, and destroy resources. It must not use the developer's local Docker Desktop or local k3d as part of the workload build/import/runtime path. Docker pulls, service image builds, k3d image imports, the k3d runtime, and workload preflight checks run on the remote EC2 host. The guiding model is Compressed staging: combine the staging Global CP and Regional CP network shape into one session VPC, compress compute from multiple EKS clusters into one EC2-hosted k3d runtime, and preserve staging product contracts where they affect the api-gateway public entrypoint, service identity, API routing, and private-network behavior. The implementation methodology is staging-first: mirror, mimic, or copy real staging deployment architecture and setup first; introduce a new cp9-specific solution only when the staging shape cannot work in a temporary session. Scope correction on 2026-06-15: the implementation issue train from this PRD focuses on the Cloud-hosted substrate itself. Front-end branch integration and PrivateLink network fixture work are not part of this issue train. Local laptop k3d mode must remain supported for lightweight development, but it is not the old-dev replacement acceptance path. This PRD covers TiDB Cloud product functions needed by cp9 personas except chat2query, dataservice, front-end branch integration, PrivateLink network fixture work, and BYOC. ## Solution cp9 will provide a Cloud-hosted substrate built around a session-scoped remote VPC and EC2 host. The remote host runs the cp9 k3d control plane and live-service graph. The local machine drives source edits, commands, logs, and status, but product traffic enters through cloud endpoints. The remote session is not a generic cloud VM. It is a Compressed staging environment: - one session VPC instead of separate staging Global CP and Regional CP VPCs; - one EC2-hosted k3d runtime instead of multiple EKS clusters; - staging-faithful api-gateway public entrypoint, API routing, service identity, service graph, and VPC endpoint behavior where those behaviors affect product integration. Remote EC2 sizing has two independent knobs. The EC2 instance type controls CPU and memory. The remote root EBS volume size controls disk headroom for Docker, k3d, image cache, and build contexts. Root EBS disk size must be configured before host creation and validated by remote preflight; it is not an EC2 memory setting. The refactor must cover the major cp9 product surfaces one by one: Premium, Starter/Essential v1, and Dedicated. Each product surface should get its own evidence, graph, gap list, implementation slice, and acceptance proof instead of assuming one persona proves all product families. Every persona must treat `api-gateway` as a first-class live service. The public entrypoint for public-facing UI APIs and OpenAPI traffic is the staging-like api-gateway public entrypoint: public load balancer, Kong ingress/proxy, `api-gateway` routes/plugins, then persona service. It is not a generic cp9 ingress, direct service endpoint, local port-forward, or simplified service-level load balancer that bypasses the staging gateway shape. The current issue train is three persona end-to-end proofs: Premium, Starter/Essential v1, and Dedicated. Each issue must prove one persona through the api-gateway public entrypoint, create a real cluster, reach the returned endpoint, run host `mysql` `select 1`, and then recycle resources. All cloud-hosted substrate proofs for this train must run in AWS account `804837308162` with profile `dev-offline-afterworld`. That account is approved for destructive cp9 acceptance-proof operations because it is empty and dedicated to this work; every run must still close with `cp9 destroy` and cost-saving cleanup evidence. BYOC sandboxing is out of scope for cp9 in this requirement. ## Architecture Decision cp9 will treat the remote VPC/EC2 k3d runtime as the primary substrate for old-dev replacement acceptance. The engineering method is staging-first. For each product surface, cp9 should mirror, mimic, or copy real staging deployment architecture and setup first. Only introduce a new cp9-specific solution after proving that the staging shape does not work for a temporary session. The remote environment is Compressed staging: - merge today's staging Global CP and Regional CP network shape into one session-scoped VPC; - compress compute from multiple EKS clusters into one EC2-hosted k3d runtime; - keep the api-gateway public entrypoint, gateway behavior, API routing, VPC endpoint, service identity, and product behavior as close to staging as practical. For `remote-ec2-k3d`, the workload boundary is the EC2 host. Local checks are limited to controller prerequisites such as config, AWS auth, SSH key material, and SSH reachability. Remote preflight checks the root filesystem headroom, Docker daemon, base-image pull, and trivial Docker build on EC2 before workload deployment. Paid proof runs should not start while source, live, or stub service build/import paths still depend on local Docker Desktop. This refactor must cover product surfaces one by one: Premium, Starter/Essential v1, and Dedicated. Each product surface needs its own staging evidence, compressed-staging graph, implementation slice, and acceptance proof. A passing proof for one surface does not prove another surface. The expected implementation order is: 1. Premium. 2. Starter/Essential v1. 3. Dedicated. Each persona implementation slice must pass the same four gates: 1. Staging evidence: collect fresh AWS CLI or kubectl evidence for the persona's real staging service graph, network setup, gateway route setup, API entrypoints, and cleanup-relevant resource classes. 2. Compressed-staging graph: document what the slice copies from staging, what it compresses into the session VPC or EC2-hosted k3d runtime, and every intentional divergence. 3. Implementation: build only after the staging-derived graph exists. 4. Acceptance report: prove the api-gateway public entrypoint, real cluster create, returned endpoint connectivity, host `mysql` `select 1`, dynamic data-plane behavior where relevant, and destroy cleanup. No persona implementation task should start from an abstract design alone. ## Acceptance Seams The highest acceptance seam is outside-in product traffic, not internal service readiness: - api-gateway public entrypoint: an OpenAPI or product client calls the public load balancer, then traffic flows through Kong ingress/proxy and `api-gateway` routes/plugins into the persona service graph. - New-account substrate proof: every proof runs in AWS account `804837308162` with profile `dev-offline-afterworld`, and all proof resources are recycled. - Persona end-to-end proof: each of Premium, Starter/Essential v1, and Dedicated has its own independent proof that creates a real cluster through the api-gateway public entrypoint, reaches the returned endpoint, runs host `mysql` `select 1`, and destroys resources. - Dynamic data-plane proof: Premium and Dedicated acceptance uses the product's real create and destroy path when the scenario depends on data-plane lifecycle, returned endpoints, or customer-visible database behavior. - Shared regional proof: Starter/Essential v1 acceptance proves the shared Regional CP/DP surface through its staging-derived graph. - Cleanup proof: `cp9 destroy` removes local state, host resources, VPC children, load balancer children, gateway resources, cluster resources, and any other session-tagged cloud resources. Process health, direct service calls, local port-forwards, and generic ingress health are useful diagnostics but do not satisfy old-dev replacement acceptance. ## Staging Evidence Evidence collected on 2026-06-15 with AWS CLI and kubectl against the staging AWS account. Kubernetes evidence uses the `dbaas-dev` AWS profile: - `staging-base-us-west-2` is active in `staging-base-us-west-2-vpc` and exposes both public and private EKS API endpoints across six subnets. It is the shared Global CP for Premium, Starter/Essential v1, and Dedicated. - `staging-seed-us-west-2-ng-0711` is active in its own VPC and is the Premium regional CP. kubectl access via the `dbaas-dev` AWS profile confirmed regional workloads including `cluster-ng-service`, `dataplane-manager`, Kong, autoscaler, and monitoring operators. - `staging-us-east-1-f01` is active in its own VPC and is the Starter/Essential v1 regional CP/DP. kubectl access via the `dbaas-dev` AWS profile confirmed serverless regional workloads including `tidb-admin` manager/gateway/PrivateLink/public endpoint deployments, `tidb-serverless`, `tidb-worker`, metering, and regional observability operators. - `staging-seed-us-west-2` is active in `staging-seed-us-west-2-vpc` and is the Dedicated regional CP. kubectl access via the `dbaas-dev` AWS profile confirmed regional workloads including `cluster-service`, `regional-server`, Crossplane AWS providers, `tcidm`, and `ticdc-service`. - Dedicated and Premium data planes are created and destroyed dynamically in response to customer requests; they are not fixed regional CP clusters. - The staging base VPC has an active internet-facing application load balancer named `k8s-kongingressalb-d0d9d6d02e`, tagged with Kubernetes ingress stack `kong-ingress-alb` and cluster `staging-base-us-west-2`. - That gateway load balancer has HTTP 80 and HTTPS 443 listeners. - kubectl access to `staging-base-us-west-2` via the `dbaas-dev` AWS profile confirmed the `kong/kong-ingress` ingress uses `internet-facing`, `kong-ingress-alb`, and the same live load balancer DNS name; the `api-gateway-kong-proxy` service is a NodePort selected by `app.kubernetes.io/instance=api-gateway`. - kubectl access to `staging-base-us-west-2` via the `dbaas-dev` AWS profile confirmed live gateway and product-entrypoint workloads including `kong/api-gateway-kong`, `staging-ms/serverless-svc-*`, `tidb-management-service/tidb-mgmt-portal`, and `tidb-management-service/serverless-global-regional-proxy`. - kubectl access to `staging-base-us-west-2` via the `dbaas-dev` AWS profile confirmed gateway plugin shape including CORS, cookie auth/filter, bearer token auth/fallback, o11y proxy, Prometheus, and trace plugins. - The staging base VPC has available interface VPC endpoints, confirming that VPC endpoint behavior is part of the real staging network shape. ## User Stories 1. As a Premium frontline developer, I want a temporary cp9 session that can replace old-dev integration, so that old always-running dev can be shut down. 2. As a Premium frontline developer, I want each cp9 session isolated from other developers, so that my tests do not collide with other sessions. 3. As a Premium frontline developer, I want my local laptop to remain the command and source-editing surface, so that I can keep my existing development loop. 4. As a Premium frontline developer, I want the control-plane runtime to execute on cloud EC2 capacity, so that laptop resources do not limit the environment. 5. As a Premium frontline developer, I want product API traffic to call a public cp9 backend endpoint, so that old-dev replacement traffic enters through a product-like gateway path. 6. As a product developer, I want the public backend endpoint to use the api-gateway public entrypoint, so that routing, auth, CORS, cookies, and gateway plugins match the product entrypoint. 7. As a product developer, I want cp9 to expose the endpoint I should configure for proof traffic, so that I do not need to discover AWS load balancers manually. 8. As a cp9 persona owner, I want `api-gateway` to be live in every persona, so that public API behavior is not bypassed by direct service calls. 9. As a cp9 persona owner, I want every public-facing UI API and OpenAPI request to enter through `api-gateway`, so that public product behavior is tested at the product seam. 10. As a cp9 persona owner, I want serverless OpenAPI traffic to route through the same gateway entrypoint used by product traffic, so that Starter/Essential v1 behavior is tested at the product seam. 11. As a cp9 persona owner, I want the tidb-management-portal-hosted serverless OpenAPI surface included in coverage, so that management-portal product APIs are not accidentally skipped. 12. As a cp9 persona owner, I want the serverless-service-hosted serverless API surface included in coverage, so that regional/serverless product APIs are not accidentally skipped. 13. As a cp9 maintainer, I want all three persona proofs to run in AWS account `804837308162`, so that the old-dev replacement work is isolated from the legacy dev account. 14. As a cp9 maintainer, I want proof runs to use profile `dev-offline-afterworld`, so that agents use the approved administrator path for the empty proof account. 15. As a cp9 maintainer, I want every proof to recycle resources with `cp9 destroy`, so that the empty account does not accumulate cost. 16. As a persona owner, I want each proof to create a real cluster and execute `select 1`, so that route-only or create-only evidence is not mistaken for old-dev replacement acceptance. 17. As a cp9 maintainer, I want the local laptop mode preserved, so that small local development tasks are not broken by the cloud-hosted substrate work. 18. As a platform engineer, I want `cp9 status` to report the remote host, public gateway endpoint, gateway health, and persona proof state, so that session health is inspectable without manual AWS spelunking. 19. As a platform engineer, I want `cp9 logs` and `cp9 deploy` to work against the remote runtime, so that moving compute to EC2 does not break the local developer loop. 20. As a platform engineer, I want `cp9 destroy` to clean the remote EC2 host, VPC children, load balancer children, gateway resources, and cluster resources, so that temporary sessions do not leak cloud resources. 21. As a cp9 maintainer, I want local-only k3d to remain useful for lightweight development where it still fits, so that small tasks are not forced through the heavier cloud path. 22. As a cp9 maintainer, I want old-dev replacement acceptance to use the cloud path, so that the acceptance proof matches the requirement. 23. As a reviewer, I want each persona acceptance report to include public gateway calls, real cluster id, returned endpoint, host `mysql` `select 1`, and destroy proof, so that the replacement can be judged without trusting implementation internals. 24. As a cp9 maintainer, I want each implementation slice to copy staging architecture first, so that cp9 does not invent unnecessary substitutes. 25. As a cp9 maintainer, I want a clear reason recorded whenever cp9 diverges from staging, so that exceptions stay intentional and reviewable. 26. As a Premium persona owner, I want Premium covered independently, so that Premium old-dev replacement does not depend on Starter or Dedicated proof. 27. As a Starter/Essential v1 persona owner, I want Starter covered independently, so that the shared Regional CP/DP product surface is proven through its own staging-derived graph. 28. As a Dedicated persona owner, I want Dedicated covered independently, so that its old Dedicated control-plane and data-plane shape is not inferred from Premium or Starter. ## Implementation Decisions - Real staging evidence must be collected with AWS CLI or kubectl before turning a requirement into a concrete implementation slice. - Implementation must copy real staging deployment architecture and setup first. New cp9-specific solutions are allowed only after the staging shape is proven unworkable for a temporary session. - The primary old-dev replacement substrate is a Cloud-hosted cp9 session running k3d and the persona service graph. - The remote VPC/EC2 substrate should be designed as a Compressed staging analogue, not as an unrelated test environment. - The refactor must proceed product surface by product surface: Premium, Starter/Essential v1, and Dedicated unless a human reorders the train. Each surface needs separate staging evidence and acceptance. - Each product-surface implementation slice must pass the staging evidence, compressed-staging graph, implementation, and acceptance-report gates before it is treated as complete. - The local laptop remains the command, source-editing, and control surface. Build/deploy/log/status operations target the remote session. - `api-gateway` is mandatory live-service infrastructure for every persona that participates in the cloud dev integration substrate. - The api-gateway public entrypoint is the public product entrypoint. cp9 must not invent a separate generic public ingress or raw service-level load balancer for public-facing UI APIs or OpenAPI traffic. - Direct service calls and local port-forwards are debug shortcuts only. They do not satisfy old-dev replacement acceptance for public-facing behavior. - The gateway endpoint must support product API proof traffic: HTTPS, expected auth/cookie behavior, CORS where the product path requires it, and routing to persona services. - Each persona proof must create a real cluster, reach the returned endpoint, and run host `mysql` `select 1`. - All three persona proofs must run in AWS account `804837308162` with profile `dev-offline-afterworld`. - Old local laptop mode must remain supported and unbroken, but old-dev replacement acceptance must use the cloud-hosted substrate. - Premium and Dedicated acceptance must use the product's real create and destroy path whenever a scenario depends on data-plane lifecycle, returned endpoints, or customer-visible database behavior. - Starter/Essential v1 is different: its Regional CP/DP is shared, so acceptance proves the shared regional CP/DP surface through its staging-derived graph instead of inventing a customer data plane. - All cp9-created cloud resources must use the cp9 namespace prefix and carry owner/session tags. - BYOC sandboxing is not part of this cp9 scope. - chat2query and dataservice are excluded from this requirement. ## Testing Decisions - The highest acceptance seam is product traffic through the api-gateway public entrypoint into a cloud-hosted cp9 persona. - Each product surface acceptance starts from staging evidence for that surface: live service graph, network setup, gateway route setup, API entrypoints, and cleanup-relevant resource classes. - Acceptance reports should explicitly cite the staging evidence and compressed-staging graph used by the implementation slice. - Tests should assert externally visible behavior: reachable gateway endpoint, correct routing, auth/CORS behavior, API responses, real cluster endpoint SQL proof, and cleanup. They should not assert internal implementation details of the host abstraction. - Prior art for this testing style is cp9's existing full-persona proof pattern: public product API call, returned endpoint, database connectivity where applicable, `cp9 status` evidence, and `cp9 destroy` closeout. - Persona graph tests should prove that every cloud-dev persona includes live `api-gateway`. - Gateway integration tests should call the public endpoint instead of local port-forwards. - A persona acceptance proof is incomplete if public-facing UI API or OpenAPI traffic cannot enter through that persona's api-gateway public entrypoint. - Premium, Starter/Essential v1, and Dedicated proofs should be independent acceptance reports. A passing proof for one surface must not be used as a substitute for another surface. - Each persona acceptance report must include the AWS account, profile, region, cp9 SHA, source SHAs, public gateway endpoint, real cluster id, returned SQL endpoint, `select 1` result, and `cp9 destroy` closeout. - Premium and Dedicated acceptance reports must include dynamic data-plane proof when the scenario touches data-plane lifecycle, returned endpoints, or customer-visible database behavior. - Destroy tests should verify there are no remaining session-tagged EC2, VPC, load balancer, gateway, cluster, security-group, DNS, or IAM resources. ## Out of Scope - BYOC customer-account sandboxing. - chat2query. - dataservice. - Front-end branch integration / Netlify proof. - PrivateLink network fixture work. - Replacing staging or shared production-like environments. - Shared multi-developer cp9 sessions. - Making real Databend, lake, or observability service dependencies mandatory in this issue train. - Treating local laptop k3d as the old-dev replacement acceptance path. - Using a novel cp9-only architecture before checking whether the staging deployment shape can be mirrored. ## Further Notes - The current `api-gateway` staging deployment shape uses Kong plus an internet-facing ALB ingress with HTTPS listeners, WAF, CORS/cookie origin configuration, and o11y proxy plugin wiring. cp9 should preserve this product entrypoint shape where practical. - This issue train should publish exactly three persona implementation issues: Premium, Starter/Essential v1, and Dedicated. Each issue must be independently provable end to end through the api-gateway public entrypoint, real cluster create, returned endpoint, host `mysql` `select 1`, and destroy cleanup.