# Retro: cp9 Premium Reserved Dogfood to Public SQL Date: 2026-05-20 Status: retrospective for merged dogfood backport Source discussion: internal cp9 dogfood thread; durable references are issue `tidbcloud/controlplane9#192` and PRs #246/#247. Primary task: cp9 task #29 / GitHub issue `tidbcloud/controlplane9#192` Backport PR: `tidbcloud/controlplane9#246` Merge commit: `4acda28c063cf8e869fcfe9e049af9cc6428f704` Follow-up PR: `tidbcloud/controlplane9#249` ## Executive Summary The dogfood goal was deliberately end-to-end: 1. Create a Premium Reserved cluster through public OpenAPI. 2. Build a real cp9-owned AWS/EKS dataplane. 3. Reach AppBox/DataPlane Active. 4. Let OpenAPI report the cluster as `AVAILABLE`. 5. Enable the public endpoint with `UpdatePublicConnectionSetting`. 6. Fetch the public endpoint. 7. Run SQL `select 1` through that endpoint. The live lab reached the finish line. Public SQL went through: ```text AWS NLB -> TiProxy -> TiDB -> select 1 ``` The successful live lab used a public AWS NLB endpoint shaped like: ```text .elb.us-west-2.amazonaws.com:4000 ``` `select 1` returned `OK: 1` five times with the expected OpenAPI password `` after a temporary live auth bypass was reverted. PR #246 packaged the source/test backport for the validated cp9 local runtime gaps. It did not include the temporary auth bypass, did not mutate staging, and did not perform batch cleanup of old cp9 AWS resources. PR #249 added the manual get-started guide and then tightened the local seed data from staging RDS. That follow-up matters because some dataplane container differences in the first live snapshots were not merely Kubernetes differences; the authoritative image/template facts are stored in staging RDS seed tables. The follow-up mirrored those seed families into cp9, materialized the staging image repositories through the local dev ECR registry for remote EKS pulls, and proved the documented manual `curl` + `mysql` path again. This retro is meant to be read before new cp9 persona work, new Premium Reserved work, or cp9 refactors that touch local dogfood, Tilt generation, source-built services, dataplane-manager CRs, cluster-ng overlays, or OpenAPI-driven finish paths. ## Main Lesson The most important process correction was that the dogfood goal outranks cleanup bookkeeping and intermediate evidence packets. A run that has already paid the cost to create EKS, AppBox, nodegroups, and shared-pool resources is a live lab. Use it. The productive loop became: ```text observe live pod/log/event/state -> patch the live EKS/k3d object only when the wrong object is clear -> validate the exact failed pod/task progresses -> backport the source/render/config path that should have produced that state -> use Tilt hot reload for source-built local services when possible -> restart from OpenAPI create only when the lab is gone or state is unrecoverable ``` The unproductive loop was: ```text run from zero -> stop at an intermediate artifact checkpoint -> cleanup all resources -> repeat ``` That loop spent most of its time in cloud create/delete and repeatedly lost the most valuable debugging asset: a ready remote EKS dataplane. ## 2026-05-21 Follow-Up: RDS Seed Alignment and Manual Proof After the first retro was merged, the dataplane image comparison exposed an important source-of-truth gap: image and template facts for the staging dataplane are not derived from pod snapshots alone. They are seeded in RDS. The follow-up in PR #249 used staging RDS as read-only input and did not commit or print credentials. The regional seed update mirrored these staging config-center families from `config_center_regional_staging_us_west_2_ng_0711` and checked the paired regional database `staging_us_west_2_ng_0711` for the same local init path: - `component_images` - `default_versions` - `default_templates` - `templates` - `template_values` The global staging databases `nextgen_global_service_staging` and `tidb_management_service_staging` were checked as well. They did not contain real Premium Reserved global seed rows that could replace cp9's local synthetic Premium Reserved seed, so that local seed remains cp9-owned. The source lesson is precise: mirror staging image/version/template facts, then materialize those image repositories into the dev ECR registry/repo prefix that the cp9-created remote EKS cluster can pull. In this update, staging repos such as `gcr.io/pingcap-public/dbaas/{tidb,tikv,tiproxy}` and the TiBR / cluster-agent worker image families are rendered through the local dev ECR shape. The same follow-up corrected the local TiKV template to `aux-worker-count=3` for the 3 CPU pod quota. The follow-up was validated with the written get-started flow, not a hidden Go test: 1. Start cp9 and port-forward the OpenAPI entrypoint. 2. `curl` the OpenAPI create request. 3. Poll OpenAPI until the cluster is customer-ready (`ACTIVE` / `AVAILABLE`). 4. `curl` `UpdatePublicConnectionSetting`. 5. `curl` the public endpoint response. 6. Run `mysql ... -e 'select 1'` against the returned public endpoint. The fresh manual run `cp9-david-rds-102422` accepted OpenAPI create for cluster `10909280891066734231`, reached AppBox/DataPlane and OpenAPI `ACTIVE`, returned 200 from `UpdatePublicConnectionSetting`, returned an AWS NLB public endpoint, and `select 1` returned `1`. ## Timeline ### 2026-05-13 to 2026-05-15: Initial Dogfood Framing The thread started with the Premium Reserved local dogfood path as the main cp9 train. The desired result was not just local generation or CRD application; it was to move through the real Premium Reserved runtime until the public SQL endpoint worked. Early operating rules were established: - Integration Gate / Integration Tests could be waived for this dogfood train. - Non-waived Build/Lint/Test checks still had to pass before merge. - Staging was a reference, not a place to patch. - Work should stay in `controlplane9` unless a human explicitly authorized a different repository. The first practical blockers were local authentication and environment setup. Repeated SSO/device-code loops wasted time. The better rule became: only start a new login flow when a human is ready to complete it, then verify credential export succeeds without printing secrets. ### 2026-05-17 to 2026-05-18: Source-Built Dataplane Manager The first major runtime mismatch was architecture/image related. A pinned dataplane-manager image path did not work for the local arm64 cp9 environment. The direction was corrected: - Do not change external product source for cp9 dogfood. - Treat dataplane-manager as a cp9 `source` service. - Build it from `cluster-service-ng/internal/dataplane-manager`. This produced the first durable pattern for cp9: ```text if a required staging component has no usable local image, model it as a cp9 source-built service and generate the local image through cp9 ``` That work led to earlier PRs in the train: - #235: source service for Premium Reserved dataplane-manager. - #237/#239 era: harness and generated source-build status/report fixes. The thread also established that staging EKS clusters should be used as read-only component references. The requested first artifact was a component matrix: staging object, cp9 rendered/live state, gap, and cp9-owned port or refactor. ### 2026-05-19: S3, Internet, AppBox, and the Cleanup Trap Several runs proved that S3 gateway endpoint children and route-table associations eventually become Ready, but the harness was stopping too early or classifying from stale parent status. Extending the outer S3 wait showed: - S3 endpoint became Ready/Synced. - All route-table associations became Ready/Synced. - Internet rechecked and became Active. - DataPlane re-reconciled and moved to AppBox. That closed the S3 lane as not a product S3 parent propagation bug. Then the work fell into a cleanup trap. Post-run guardrails found EKS/VPC/NAT resources and the team repeatedly waited for full cleanup before doing the next run. This was corrected twice: - First, dogfood goal was set as the highest priority. If test code blocks the dogfood path, change the code. - Later, per-run cleanup was explicitly deprioritized. Old cp9 resources should warn/report unless they cause a hard quota/name/API conflict. Batch cleanup can happen later on request. The AppBox/EKS lane showed that IAM and EKS timing was mostly normal: - Node role and policies can become ready. - EKS can become Active. - All nodegroups can become Active. - Early `NoSuchEntity` role policy attachment errors were timing, not a product patch by themselves. The next real AppBox blocker was missing O11y API support: ```text no matches for kind "KubernetesAgent" in version "agent.observability.pingcap.com/v2" ``` The cp9 local dataplane-manager CRD package needed the O11y `KubernetesAgent` CRD. ### 2026-05-20 Morning: AppBox Active and OpenAPI Availability Adding the `KubernetesAgent` CRD let AppBox and DataPlane become Active. The next blocker was OpenAPI still reporting the customer cluster as `CREATING`. The first attempt to call `UpdatePublicConnectionSetting` immediately after DataPlane Active returned: ```text 400 Cluster is not available. ``` The harness was corrected to wait for OpenAPI cluster state `AVAILABLE` before public endpoint enablement. Further investigation showed OpenAPI was not stale yet. The shared pool itself was still creating. Specific causes included: - cluster-ng source-built images lacked `aws-iam-authenticator`. - Long waits expired mounted AWS credentials. - Bucket CRs were created but not reconciled because local controller predicates required `cp9-owner` / `cp9-session` labels. - Premium Reserved local config rendered impossible quotas such as `pd_scheduling minNodeCount=2 maxNodeCount=1`. Backports from this phase: - Add `aws-iam-authenticator` and mounted exported AWS credentials to cluster-ng server/executor local source-built images. - Refresh mounted AWS credentials during long waits. - Merge DataPlane cloud tags into Bucket CR labels. - Render `pd_scheduling: 2` for Premium Reserved server/executor config. ### 2026-05-20 Midday: Live EKS Lab Becomes the Main Tool The next critical correction came when a run had remote EKS resources available and a blocker inside the dataplane. The instruction was to stop restarting from zero if `kubectl` could validate and fix the issue live. This changed the loop. The team started using the remote EKS lab to patch one affected object, validate the exact pod or resource moved forward, then backport the source path. Concrete live-lab validations: - TiKV worker image: - Before: normal TiKV image tried to execute `/tikv-worker`. - Failure: `stat /tikv-worker: no such file or directory`. - Live/source fix: use the staging ECR TiKV worker image family. - TiKV worker concurrency: - Before: coprocessor worker failed validation. - Failure: `global concurrency factor must be greater than keyspace concurrency_factor`. - Rendered state: `global-concurrency-factor=8` without keyspace factor. - Live validation: adding `keyspace-concurrency-factor=4` let the worker pass config validation. - Source fix: seed/render both values. - Readiness feature gates: - Before: pods were Running but NotReady. - PD `/pd/api/v2/readyz` returned 404. - Scheduling `/health` returned 404. - TiKV `/ready` returned 404. - TiFlash `/tiflash/readyz` returned 404. - Live validation: remove `UsePDReadyAPIV2`, `UseSchedulingReadyAPI`, `UseTiKVReadyAPI`, and `UseTiFlashReadyAPI`. - Source fix: local cp9 source-built template drops those gates. - Additional image/materialization gaps: - `tibr` needed the worker image, not normal TiKV image. - `cluster-agent` init needed the worker image for `/cse-ctl`. - `cluster-agent` main image needed staging `cluster-agent-ng` because the old image did not support rendered TLS flags. - Audit sidecar needed staging ECR `audit-ng`, not `docker.io/tidbcloud/audit-ng:latest`. - SYSTEM keyspace: - Before: worker TiDB crashed on `Load keyspace SYSTEM failed: keyspace does not exist`. - PD only had `DEFAULT`. - Live validation: create `SYSTEM` through PD v2 and restart worker TiDB. - Result: TiDBGroup Ready and remote `Cluster/db AVAILABLE=True`. - Source fix: bootstrap `SYSTEM` before worker TiDB starts. This was the highest-value phase of the dogfood. It proved that live EKS inspection is not optional; it is the fastest way to separate local cp9 rendering bugs from normal cloud latency. ### 2026-05-20 Afternoon: Latest Heads and Tilt Hot Reload Another process correction was to pull latest heads for related repositories instead of patching around old source: - TiDB management: `c780e303` on `main`, including the fastflow module/schema path for the `cmd`/`has_cmd` issue. - cluster-service-ng: `f99c5b7` on `main`. - config-center: `3add065a` on `master`. The fastflow `cmd` set while `has_cmd` was NULL issue was therefore handled by updating the TiDB management dependency path, not by carrying a local cp9 workaround. Tilt hot reload was validated and should remain the default for local source-built cp9 services: - Latest cluster-service-ng changed `CreateTiDBClusterMeta` to require `sharedPool`. - `cluster-ng-server` failed to build under Tilt. - The cp9 overlay was patched. - `tilt trigger cluster-ng-server` rebuilt and redeployed the service in place. - The rebuilt pod was Ready and logs no longer showed the signature compile error. Latest-head contract fixes then followed: - Premium Reserved OpenAPI create now requires `reserved_capacity_template_id`. - cp9 seeded template id `1` and sent `reservedCapacityTemplateId=1`, omitting old direct `maxRcu` / `minRcu` fields. - Latest topology validation accepts only `PD`, `TIKV`, `TIKV_WORKER`, and `TIDB_WORKER`. - cp9 seed/template data stopped emitting rejected names such as `PD_TSO`, `PD_SCHEDULING`, `COPROCESSOR_WORKER`, `TIDB`, and `TIPROXY` in that shared-pool seed. ### 2026-05-20 Evening: Public Endpoint and SQL After runtime `Cluster/db` became available, the final path exposed several remaining local cp9 mismatches: - `s3:PutLifecycleConfiguration` was missing from the dataplane-manager role. Live-adding it let BucketLifecycleConfiguration become Synced/Ready and sharedPool/OpenAPI move to `AVAILABLE`. - Public endpoint enablement created a regional `LoadBalancer` CR, but dataplane-manager did not reconcile it because it missed `cp9-owner` / `cp9-session` labels. This was the same predicate-label class as Bucket CRs. Live-adding the labels made dataplane-manager create the remote Service and fill status. - Local Route53 config was empty, but the DNS path tried to form a Route53 record with an empty domain, creating a trailing-dot name. The local path was changed to skip Route53 when Route53 config is incomplete and use the AWS NLB hostname directly. - SQL reached TiProxy, but TiProxy returned `No available TiDB instances`. Injecting the missing TiDB topology entry let TiProxy discover a healthy `keyspace=SYSTEM` backend. - The final SQL blocker was password initialization. OpenAPI had accepted a root password, but the live TiDB bootstrap path had not set it. A temporary live auth bypass was used only to set the expected password, then reverted. After revert, public `select 1` succeeded five times. The backport kept the real source paths and did not include bypass behavior. ## Detailed cp9 vs Staging Compare This compare was redone read-only while addressing review feedback. Staging was not mutated. Compared environments: - cp9 local control plane: the k3d/k3s control plane from the successful run was already gone at review time, so live pod inspection was not possible. The local control-plane rows below come from the cp9 rendered/Tilt source-service model and PR #246 backport inventory. - cp9 dataplane EKS: `arn:aws:eks:us-west-2:385595570414:cluster/tidbcloud-c96cd7019b69dc6946eb3307af71bfea-427b7503` - staging control-plane EKS clusters: `staging-base-us-west-2` and `staging-seed-us-west-2-ng-0711` - real staging dataplane EKS: `arn:aws:eks:us-west-2:790636686964:cluster/tidbcloud-1183114ddc50bfb3f8c06916876c26c2-dacf0c7d` ### Physical Deployment Architecture Diagrams The cp9 dogfood deployment is not a small in-cluster mock. It is a local source-built control plane that provisions and drives a real AWS/EKS dataplane. ```text +------------------------ developer machine: local cp9 control plane ------------------------+ | macOS arm64 + Docker + k3d/k3s | | | | Tilt hot-reload resources | | - cluster-ng-server - cluster-ng-executor | | - dataplane-manager - config-center-regional | | - nextgen-global-service - tidb-mgmt-service / tidb-mgmt-portal | | | | Local platform pieces | | - local Kong / Gateway API / HTTPRoute | | - Crossplane providers and cp9-generated CRDs | | - exported AWS SSO credentials + aws-iam-authenticator mounted into source services | +-----------------------------------------------+-------------------------------------------+ | | AWS APIs, OpenAPI create, Crossplane CRs v +------------------------ AWS account 385595570414: cp9 dataplane --------------------------+ | EKS: cp9-created Premium Reserved dataplane | | - public/private EKS API; public access restricted to the local /32 | | - VPC with 9 subnets, 7 route tables, 3 NAT gateways, S3 gateway endpoint, 5 SGs | | - 2 NLBs; final SQL path used the internet-facing public NLB | | | | Nodes and namespaces observed | | - 10 nodes: 3 amd64 t3.medium, 7 arm64 c8g.xlarge | | - kube-system, cert-manager, monitoring, tidb-admin, tidb | | | | TiDB runtime at finish | | - Cluster/db AVAILABLE=True | | - PD=1, TiKV=1, TiDB=2, TiProxy=1 | +-----------------------------------------------+-------------------------------------------+ | | UpdatePublicConnectionSetting v public AWS NLB :4000 -> TiProxy -> TiDB -> select 1 ``` The real staging deployment splits the control plane across persistent EKS clusters and then drives a separate staging dataplane EKS cluster. ```text +---------------------- staging-base-us-west-2: global control-plane EKS -------------------+ | 23 amd64 nodes | | namespaces include infra, staging-ms, staging, kong, tidb-management-service, monitoring | | Kong plus public/internal ALB/NLB front doors | | broad staging control-plane services and shared platform dependencies | +-----------------------------------------------+-------------------------------------------+ | | global APIs and regional orchestration v +------------------ staging-seed-us-west-2-ng-0711: regional/seed EKS ----------------------+ | 4 amd64 m6i.large nodes | | cluster-ng-service: cluster-ng-server + executor | | dataplane-manager: dataplane-controller + dataplane-server | | crossplane-system: provider-aws family/ec2/eks/iam/kms/route53/s3 | | kong, monitoring, logging, regional helper services | +-----------------------------------------------+-------------------------------------------+ | | AWS APIs and dataplane provisioning v +---------------------- AWS account 790636686964: real staging dataplane -------------------+ | EKS: tidbcloud-1183114ddc50bfb3f8c06916876c26c2-dacf0c7d | | - VPC with 9 subnets, 7 route tables, 3 NAT gateways, S3 gateway + interface endpoints | | - 5 SGs, 2 NLBs, production-style service tags | | | | Nodes and namespaces observed | | - 22 nodes: 3 amd64 t3.medium, 6 arm64 c8g.2xlarge, 9 arm64 c8g.xlarge, 4 arm64 m8g.large | | - kube-system, monitoring, tidb-admin, one TiDB namespace, cert-manager | | | | TiDB runtime | | - Cluster/db AVAILABLE=True | | - PD=3, TiKV=3, TiDB=2, TiFlash=2, TiProxy=2 | | - TSO=2, Scheduling=2, default and coprocessor TiKV workers ready | +-----------------------------------------------+-------------------------------------------+ | | private/internal LB and Route53/DNS shape v staging SQL path -> TiProxy -> TiDB ``` Side-by-side, the important physical difference is where code changes land and how much of staging cp9 intentionally compresses. ```text cp9 dogfood path real staging path ----------------------------------------------- ----------------------------------------------- local Mac/k3d/k3s control plane persistent EKS global + regional control planes Tilt rebuilds source services in place released images deployed in staging clusters local seed/config overlays staging data/config sources cp9 predicate labels and cp9 resource tags production-style tags and lifecycle ownership smaller Premium Reserved dataplane shape broader production-like SKU matrix internet-facing public NLB with /32 source range private/internal LB + Route53/DNS service path no per-run cleanup by default persistent environments and managed lifecycle Operational rule learned: when the cp9 dataplane EKS is alive, use it as a lab with kubectl/Tilt/live validation first; restart from OpenAPI create only when the lab cannot answer the question or one-shot generated artifacts must be regenerated. ``` ### Control-Plane Physical Deployment | Component class | cp9 local k3s/Tilt | staging-base-us-west-2 | staging-seed-us-west-2-ng-0711 | Difference | | --- | --- | --- | --- | --- | | Host shape | k3d/k3s on a local arm64 Mac; pods run in local Docker/k3d nodes | EKS, 23 amd64 nodes across legacy and AL2023 groups | EKS, 4 amd64 `m6i.large` nodes | cp9 is a local single-machine dev plane; staging is real multi-node EKS. | | API gateway | local Kong/gateway resources generated by cp9 | `kong` namespace with Kong 2.8 pods and public/internal ALB/NLB front doors | `kong` namespace with Kong 3.6 pod and internet-facing NLB | cp9 must generate enough Kong route shape for OpenAPI and dataplane proxy tests; staging has full shared gateway deployment. | | cluster-ng server | source-built `cluster-ng-server` Tilt resource | not hosted in base; base is global/control platform | `cluster-ng-service/cluster-ng-server` image `tidbcloud/cluster-service-ng:release-v0.23-...` | cp9 builds from source and must keep overlays compatible with latest cluster-service-ng APIs. | | cluster-ng executor | source-built `cluster-ng-executor` Tilt resource | not hosted in base | `cluster-ng-service/executor` image `tidbcloud/ng-executor:release-v0.23-...` | cp9 must install `aws-iam-authenticator`, mount refreshed AWS credentials, and carry local config overrides. | | dataplane-manager | source-built `dataplane-manager` plus local CRDs/providers | not hosted in base | `dataplane-manager/dataplane-controller` and `dataplane-server`, image `tidbcloud/dataplane-manager:074081...` | cp9 builds source locally but must mirror staging CRDs, IAM calls, and predicate-label behavior. | | Crossplane providers | local upbound providers generated by cp9 | not the dogfood regional seed | provider-aws family/ec2/eks/iam/kms/route53/s3 v1.21.1 | cp9 must keep the provider set broad enough for EKS/VPC/IAM/S3/LB resources. | | config-center / TiDB management | source-built/stubbed cp9 services: `config-center-regional`, `nextgen-global-service`, `tidb-mgmt-service`, `tidb-mgmt-portal` | full staging management services in `staging-ms`, `infra`, `tidb-management-service` | regional seed services plus helper/mysql | cp9 intentionally carries a smaller local subset, but its seeded contracts must match latest staging APIs. | Staging-base is the broad global/control-plane platform. It had 766 pods at the snapshot time, spread mainly across `infra`, `staging-ms`, `kube-system`, `staging`, `monitoring`, `metering`, and `kong`. Staging-seed is the closer regional control-plane reference for this dogfood: 61 pods, including `cluster-ng-service`, `dataplane-manager`, `crossplane-system`, `kong`, monitoring, and logging. ### Dataplane Physical Deployment | Area | cp9 dataplane EKS | real staging dataplane EKS | Difference | | --- | --- | --- | --- | | Nodes observed | 10 nodes: 3 amd64 `t3.medium`, 7 arm64 `c8g.xlarge` | 22 nodes: 3 amd64 `t3.medium`, 6 arm64 `c8g.2xlarge`, 9 arm64 `c8g.xlarge`, 4 arm64 `m8g.large` | cp9 is smaller and only scaled the instance classes needed by the dogfood; staging has broader arm64 capacity. | | EKS nodegroups | 7 active nodegroups: one base `t3.medium` group plus six c8g xlarge/2xlarge groups across AZs | 52 active nodegroups covering c8g, i8g/i8ge, m8g/m8gd, and r8g classes across AZs | cp9 renders a reduced Premium Reserved quota set; staging exposes the full production-like SKU matrix. | | Namespaces | `kube-system`, `monitoring`, `tidb-admin`, one TiDB namespace, `cert-manager` | same classes plus one TiDB namespace | cp9 has the same essential dataplane layout but fewer observability/TiDB replicas. | | TiDB cluster CR | `Cluster/db AVAILABLE=True`, PD=1, TiKV=1, TiDB=2, TiProxy=1 | `Cluster/db AVAILABLE=True`, PD=3, TiKV=3, TiDB=2, TiFlash=2, TiProxy=2 | cp9 proves the path with minimum viable counts; staging has HA counts and TiFlash. | | PD/TSO/Scheduling | cp9 has PD Ready; no separate TSO/Scheduling groups in the final cp9 shape | staging has PD 3/3, TSO 2/2, Scheduling 2/2 | cp9 local seed reduced topology; earlier local readiness gates still had to be compatible with staging-style components. | | TiKV workers | cp9 has default TiKVWorkerGroup 1/1 | staging has default and coprocessor TiKVWorkerGroups 1/1 each | cp9 backport covers the worker image and concurrency bug even though final cp9 scale is smaller. | | TiProxy | cp9 TiProxyGroup 1/1 Ready | staging TiProxyGroup 2/2 Ready | cp9 has enough to validate backend discovery and public SQL; staging has redundant TiProxy. | | O11y | cp9 has monitoring namespace and O11y CRD support; O11y app set is smaller | staging has O11y/vector/vmagent pods in the TiDB namespace | cp9 needed `KubernetesAgent` CRD to pass AppBox install even with smaller O11y footprint. | Important dataplane container differences from the live snapshots: | Container purpose | cp9 image class | staging image class | What cp9 had to learn | | --- | --- | --- | --- | | cluster-agent main | ECR `tidbcloud/cluster-agent-ng:main-29d2...` | ECR `tidbcloud/cluster-agent-ng:main-7ea...` | Use the `cluster-agent-ng` family, not the old cluster-agent image. | | cluster-agent `copy-cse-ctl` init | ECR TiKV worker image `release-nextgen-202603` | ECR TiKV image `v8.5.4-nextgen.202510.23` | The init image must contain `/cse-ctl`; normal public TiKV did not. | | PD ctl init | ECR/public PD v8.1.2 in cp9 | ECR PD `release-nextgen-20251011-...` in staging | cp9 uses older minimal runtime but must match command availability. | | TiDB | cp9 public/ECR v8.1.2 family | staging ECR `tidb:v8.5.4-nextgen...` | Version differs, but keyspace/password/bootstrap behavior must be structurally compatible. | | TiKV | cp9 public v8.1.2 for normal TiKV; ECR worker image for worker/tibr | staging ECR `tikv:v8.5.4-nextgen...` for normal and worker roles | cp9 must never run `/tikv-worker` from the normal public TiKV image. | | TiProxy | cp9 live lab still showed public `gcr.io/.../tiproxy:v1.3.1`; PR #246 seed was corrected to staging ECR `tidbcloud/tiproxy:v1.4.0-nextgen.202510.6` | staging ECR `tidbcloud/tiproxy:v1.4.0-nextgen.202510.6` | The source seed should follow staging even if a live patch proved an older public image could run. | | audit | ECR `tidbcloud/audit-ng:aecee...` | same ECR class | Public `audit-ng:latest` was invalid for cp9. | | tibr | ECR TiKV worker image | ECR TiKV nextgen image | tibr belongs to the same image/command family as TiKV worker. | | datacleaner | ECR `aws-shared-provider-datacleaner:6c829...` | staging uses ECR service images | cp9 should use staging ECR class for support sidecars. | The 2026-05-21 RDS seed follow-up refined this table's conclusion. Pod snapshots are useful evidence, but the durable source of truth for these image families is the regional config-center seed data in RDS. Future cp9 seed work should first compare `component_images`, `default_versions`, `default_templates`, `templates`, and `template_values`, then assert that the local cp9 materialization layer rewrites staging repositories into the dev ECR shape that remote EKS can pull. ### AWS Resource Shape | Resource class | cp9 dataplane | staging-base control plane | staging-seed control plane | real staging dataplane | Notes | | --- | ---: | ---: | ---: | ---: | --- | | VPCs for compared EKS | 1 | 1 | 1 | 1 | Each compared EKS is in one VPC. | | Subnets in VPC | 9 | 6 | 6 | 9 | Dataplane VPCs use 9 subnets; staging control-plane VPCs use 6. | | Route tables | 7 | 7 | 5 | 7 | cp9 dataplane and real staging dataplane have the same count. | | NAT gateways | 3 | 3 | 1 | 3 | cp9 dataplane matches real staging dataplane. | | VPC endpoints | 1 | 5 | 2 | 2 | cp9 has S3 Gateway only; real staging dataplane has S3 Gateway plus one Interface endpoint. | | Security groups | 5 | 44 | 7 | 5 | cp9 dataplane SG count matches real staging dataplane; staging-base has many platform services. | | Network interfaces | 25 | 162 | 19 | 40 | Counts reflect active pods/LBs/node scale at snapshot time. | | ELBv2 load balancers | 2 | 24 | 1 | 2 | cp9 dataplane matches real staging dataplane count but endpoint policy differs. | | EKS nodegroups | 7 | 6 | 1 | 52 | cp9 renders only the local dogfood SKU subset; staging dataplane exposes a broad matrix. | Endpoint and network differences: - cp9 dataplane EKS had public and private API access enabled with a single `/32` public CIDR. This matched the local dogfood requirement for host access. - staging-base and staging-seed control planes had public EKS access open more broadly at the snapshot time; those are long-lived control-plane clusters. - real staging dataplane had public and private API access enabled with several public CIDRs. - cp9 dataplane had two NLBs: one internal cluster gateway and one internet-facing public SQL NLB. Real staging dataplane also had two NLBs, but the SQL/customer path was internal/private for staging. - cp9 local Route53 config is intentionally empty. The local code must skip Route53 and use the NLB hostname. Real staging has the private DNS/LB path. ### What This Compare Changed in cp9 The compare directly supported these source decisions: - Include the `KubernetesAgent` CRD because staging seed has it and AppBox O11y install expects it. - Use staging ECR image families for cluster-agent, TiKV worker/tibr, datacleaner, audit, and TiProxy. - Keep TiProxy seed aligned to staging ECR `tidbcloud/tiproxy:v1.4.0-nextgen.202510.6`. - Keep cp9's local SQL endpoint path different from staging on purpose: cp9 uses internet-facing NLB plus `/32`; staging uses private/internal LB and DNS. - Keep Bucket/LoadBalancer label inheritance because cp9 local predicate filters are stricter than staging's long-lived control plane. - Keep old-resource cleanup separate from dogfood because cp9's short-lived AWS resources have a different lifecycle from staging's persistent resources. ## Source Backport Inventory PR #246 modified these cp9 areas: - `internal/cmd/premium_integration_test.go` - `internal/lifecycle/context_materialize_step.go` - `internal/lifecycle/context_materialize_step_test.go` - `internal/lifecycle/tiltfile_generate_step.go` - `internal/lifecycle/tiltfile_generate_step_test.go` - `internal/persona/shipped_test.go` - `personas/app-delivery/base/dataplane-manager-crds/agent.observability.pingcap.com_kubernetesagents.yaml` - `personas/app-delivery/base/dataplane-manager-crds/kustomization.yaml` - `personas/app-delivery/envs/premium/cluster-ng-executor/config.yaml` - `personas/app-delivery/envs/premium/cluster-ng-server/config.yaml` - `personas/app-delivery/envs/premium/cluster-ng-server/nextgen-global-service.sql` - `personas/app-delivery/envs/premium/dataplane-manager/tidb-cluster-policy-template.json` Functional areas covered: - Local source-built runtime/toolchain: - cluster-ng server/executor get `aws-iam-authenticator`. - cluster-ng server/executor mount exported AWS credential material. - old-resource guardrails warn instead of blocking iteration. - cluster-ng local overlays: - staging-aligned images for cluster-agent, datacleaner, audit-ng, TiKV worker family, tibr, and TiProxy. - readiness feature gates removed for the runtime shape that returns 404 on newer probe endpoints. - `SYSTEM` keyspace bootstrap before worker TiDB starts. - local shared-pool DAG/order and image materialization paths. - Premium Reserved quotas for `pd_scheduling` and `tiproxy` render high enough to avoid `maxNodeCount < minNodeCount`. - config-center / nextgen local seed: - Premium Reserved reserved-capacity template seed for latest-head OpenAPI. - topology components accepted by latest validation. - TiKV worker family image seed. - staging-aligned TiProxy image/version. - TiKV worker concurrency config. - dataplane-manager local overlays: - `KubernetesAgent` CRD included. - Bucket and LoadBalancer CR labels inherit dataplane cloud tags so local predicate filters see cp9-created CRs. - empty local Route53 config skips Route53 and uses the AWS NLB hostname. - TiDB cluster policy includes `s3:PutLifecycleConfiguration`. - Harness: - drives to OpenAPI `AVAILABLE`; - calls `UpdatePublicConnectionSetting`; - fetches the public endpoint; - runs SQL `select 1`; - uses latest-head `reservedCapacityTemplateId` contract; - preserves assertions for the source/render paths that regressed during the dogfood. PR #249 then added the manual get-started guide and the RDS-backed seed corrections: - Get-started validation is documented as bare `curl` plus `mysql`, not as a `go test` wrapper. - Regional config-center staging seed families are mirrored from RDS: `component_images`, `default_versions`, `default_templates`, `templates`, and `template_values`. - Global staging RDS was checked, but cp9 kept its local synthetic Premium Reserved global seed because staging had no real Premium Reserved global seed rows to mirror. - Staging image repositories are materialized through the dev ECR registry/repo prefix so the cp9-created EKS dataplane can pull them. - The local TiKV template uses `aux-worker-count=3` for the 3 CPU pod quota. ## Failure Catalog | Area | Symptom | Cause | Fix / rule | | --- | --- | --- | --- | | SSO | Repeated device-code sessions expired | Login was started when no human was ready | Ask once, wait for human, verify credential export only | | arm64 source service | Dataplane-manager image could not run locally | Required image was not compatible with local cp9 | Model as `services.source` and build from source | | S3/Internet | Parent Internet looked stale while children were Ready | Harness waited/classified at wrong time | Extend wait and classify from saved state | | cleanup | Runs spent time deleting old AWS sets | Guardrail bookkeeping became the main path | Warn/report old resources; batch cleanup later | | AppBox O11y | Missing `KubernetesAgent` kind | CRD absent from cp9 dataplane-manager CRDs | Include O11y CRD locally | | cluster-ng auth | Remote EKS calls failed | Source-built cluster-ng images lacked `aws-iam-authenticator` | Add authenticator and mounted credentials | | credential lifetime | Long wait hit `ExpiredToken` | Mounted exported credentials expired | Refresh credential material during waits | | Bucket CR | Buckets stayed Pending | Predicate labels missing | Merge dataplane cloud tags into Bucket labels | | Premium quota | `maxNodeCount=1 < minNodeCount=2` | Local quota seed too low | Render `pd_scheduling=2`, `tiproxy=2` | | TiKV worker image | `/tikv-worker` missing | Normal TiKV image used for worker | Use staging worker image family | | TiKV worker config | Concurrency validation failed | Missing keyspace concurrency factor | Render keyspace factor with global factor | | readiness | PD/Scheduling/TiKV/TiFlash NotReady | Unsupported readiness gates produced 404 probes | Remove four local gates | | tibr | `/tikv-worker` missing | Same image mapping class as worker | Use worker image | | audit | ImagePullBackOff | Public/latest image did not pull | Use staging ECR audit image | | cluster-agent | init/main failures | Wrong init image and old main image | Use worker init image and `cluster-agent-ng` | | worker TiDB | `SYSTEM` keyspace missing | Bootstrap did not create keyspace | Create `SYSTEM` before worker TiDB starts | | fastflow | retry command ignored | `cmd` set while `has_cmd` NULL | Pull TiDB management latest module fix | | OpenAPI create | missing reserved capacity template | Latest contract requires template id | Seed and send `reservedCapacityTemplateId=1` | | topology seed | validation rejected components | Seed used old component names | Emit accepted latest names only | | RDS seed drift | dataplane images/templates differed across runs | cp9 inferred too much from snapshots instead of staging RDS seed rows | Mirror config-center image/version/template rows from RDS, then materialize to dev ECR | | TiKV template | worker started with `aux-worker-count=4` on 3 CPU quota | local template did not match pod quota | Render `aux-worker-count=3` | | LoadBalancer CR | no remote Service/status | Predicate labels missing | Merge dataplane cloud tags into LoadBalancer labels | | DNS | trailing-dot Route53 name | Empty local Route53 config | Skip Route53 and use NLB hostname | | S3 lifecycle | Bucket lifecycle not Ready | IAM policy missing `s3:PutLifecycleConfiguration` | Add permission to local policy template | | TiProxy backend | `No available TiDB instances` | Missing TiDB topology entry | Backport keyspace-aware topology registration | | password | SQL auth failed | OpenAPI password not initialized in TiDB | Backport real password-init/change-password path | ## What Worked - Human corrections were applied quickly when the loop became unproductive. - Live `kubectl` validation converted several unknowns into exact source fixes. - Tilt hot reload preserved a live lab through source-built service fixes. - Staging read-only compare corrected image/version assumptions. - Focused assertions were added for each local render/seed contract that had failed during runtime. - The final PR kept temporary live hacks out of source. ## What Did Not Work - Early runs stopped at intermediate evidence stops instead of driving to the dogfood finish line. - Cleanup and guardrail counts consumed too much time and repeatedly destroyed useful live labs. - Several harness assertions encoded old artifact path names or old source contracts, causing false stops. - Waiting for full cloud create/delete on every iteration made feedback loops slow and expensive. - Some reports used jargon that obscured the actual first failing object. ## Future cp9 Feature / Refactor Checklist Use this checklist before adding a persona, changing Premium Reserved, or refactoring cp9 local delivery code. ### Define the Finish Line - State the user-visible result before the first run. - For this class of work, the finish line is not "CRD applied" or "DataPlane exists"; it is public endpoint plus SQL. - Keep the harness driving toward that finish line. ### Keep Staging Read-Only but Useful - Compare against real staging objects for images, CRDs, feature gates, services, endpoints, topology, IAM, and DNS shape. - Do not mutate staging. - If staging and cp9 differ intentionally, document the reason in cp9. ### Prefer Live-Lab Debugging - Once EKS/AppBox/DataPlane are ready enough for `kubectl`, use the live lab. - Patch one live object only after naming the wrong object. - Validate the exact pod/resource moves forward. - Then backport the source path. - Restart only when the lab is gone, one-shot generated artifacts must be regenerated, or state cannot be made consistent. ### Use Tilt for Source-Built Services - Source-built local services should be Tilt-backed. - Expected loop: patch source/render path, `tilt trigger` affected resource, verify rebuilt image and pod logs, continue the same lab. - If a cp9-owned source-built service cannot hot rebuild/redeploy through Tilt, treat that as a harness gap. ### Keep Guardrails on the Critical Path Only When They Prevent Real Conflict - Old-resource checks should warn/report by default. - Fail only for hard conflicts: name collision, quota exhaustion, credential failure, or a resource that will directly corrupt the active run. - Do batch cleanup separately when requested. ### Check These Local Rendering Classes - All controller-created CRs that local predicates watch must inherit `cp9-owner` / `cp9-session` labels or the equivalent dataplane cloud tags. - All local images must match the command they execute. Do not use a normal TiKV image for `/tikv-worker` or `/cse-ctl`. - Treat staging RDS config-center seed rows as the durable source for component images, versions, and templates; do not rely only on pod snapshots. - When staging seed rows reference external image repositories, assert the cp9 materialization layer rewrites them into the dev ECR shape remote EKS can pull. - Readiness feature gates must match APIs served by the selected runtime image. - Premium Reserved quota seeds must not render `maxNodeCount < minNodeCount`. - Template CPU/resource knobs must match the small local cp9 quota shape, such as `aux-worker-count=3` for a 3 CPU TiKV pod. - Keyspace-aware components need `SYSTEM` keyspace and keyspace labels before worker TiDB/TiProxy are expected to be ready. - Local empty DNS config must skip DNS creation and use the load balancer hostname. - IAM policy templates must include the AWS calls made by rendered controllers, including bucket lifecycle configuration. ### Keep Latest Contracts Current - Pull latest related repo heads before a long dogfood loop. - Prefer upstream fixes for upstream-owned issues, such as fastflow `cmd`/`has_cmd`. - Add assertions for latest OpenAPI request fields and seed component names. - Keep the public finish-path guide runnable as manual OpenAPI `curl` commands plus `mysql select 1`; tests may automate it, but the operator guide should expose the actual calls. ## Review and Merge Rules Used Here - Build/Lint/Test were non-waived and had to be green. - Integration Gate / Integration Tests were waived/skipped for this dogfood train. - The final PR was merge-ready only after non-waived CI passed. - The final PR body included: - live dogfood finish; - validation commands; - staging compare; - caveats around the temporary live auth bypass; - no cleanup side path. ## Caveats and Follow-Ups - The temporary live SQL/auth bypass was a diagnostic technique only. It was reverted before final SQL and was not backported. - Old cp9 AWS resources were intentionally not batch-cleaned as part of PR #246. Cleanup should be requested and run separately. - The broader cp9 tag gap remains issue #194. - The read-only staging compare artifacts were generated under `artifacts/staging-compare-20260520/` during PR #246 preparation. - The detailed compare counts are point-in-time read-only snapshots. They should be refreshed before making capacity or security decisions from this document. - RDS credentials used for the 2026-05-21 seed comparison are intentionally not recorded here. Only table families and resulting cp9 seed behavior are documented. - The 2026-05-21 manual validation lab was left alive for review/inspection; it was not part of the earlier batch cleanup snapshot. - Future docs should link back to this retro when they introduce new cp9 persona delivery, new local source-built services, or Premium Reserved runtime changes.