Premium remote-ec2-k3d — where the daily-loop time goes

Headline answers

• Which steps dominate the loop? Two cloud-wait phases: session_ready 7m17s (66%) — remote image builds + 72 Tilt resources becoming ready — and host_ready 2m11s (20%) — EC2 boot + SSH + remote host setup. Then the OpenAPI create→cluster-ACTIVE wait (real dataplane EKS provisioning) is the next huge slice (≥15–20 min when healthy). cp9-internal work (terraform plan/apply, owner/iam) is only ~12%.
• Where to add telemetry? cp9 already prints a per-stage "Timing breakdown", but the two dominant stages are black boxes. Add sub-stage timing inside session_ready (per-image build, per-Tilt-resource ready) and host_ready (EC2 RunInstances→SSH-ready→remote bootstrap), plus the OpenAPI train (create→ACTIVE, per readiness gate).
• Tilt start→fully-ready duration? Already ≈ tilt_ready (6.5s, launch) + session_ready (7m17s, wait-for-all). cp9 records tilt StartedAt + ready/total counts but no explicit "fully-ready-at" timestamp or per-resource times — those come from tilt get uiresources -o json buildHistory (used below).

cp9 up — per-stage timing (total 11m02s)

cp9's own native "Timing breakdown" on success.

Before this run, also: git clone main 2.7s, make build 19.5s, setup 4.9s, doctor 4.3s, up dry-run 0.03s.

Inside session_ready (the 7m17s) — top Tilt build durations

From tilt get uiresources buildHistory. Builds run in parallel, so wall-clock (437s) < sum; the longest builds gate the critical path.

116s  dedicated-infra-api-crds      ← old-crossplane CRDs imported by #601
 74s  dedicated-account-svc-image   ← source-built image
 70s  kms-local-image               ← source-built stub
 70s  dataplane-manager
 67s  billing-svc-image             ← source-built stub
 31s  regional-server
 28s  dataflow
 27s  cluster-controller            ← dedicated runtime
 26s  dedicated-global-service      ← dedicated runtime
 22s  nextgen-crossplane-ready
 17s  upbound-aws-provider

Finding: a large share of session_ready is (a) source-built local images (*-image: kms/billing/dedicated-account ~67–74s each) and (b) the Dedicated runtime imported by #601 (dedicated-infra-api-crds 116s + dedicated-account/global-service/cluster-controller/dedicated-kong). The dual Dedicated+Premium runtime materially inflates Premium up time.

Train B (Premium lifecycle smoke) — blocked at create→ACTIVE

OpenAPI CreateTidb returned 200 (cluster id 10752…) in ~1s; the cluster then sat CREATING for 32+ min and never reached ACTIVE (run stopped). So the SQL proof (ResetRootPassword → public endpoint → host mysql select 1 → delete) could not run. Two dataplane EKS clusters were created (ACTIVE in AWS) but the cluster never became usable.

Root cause (confirmed): dual Crossplane providers coexist after #601 imported the Dedicated runtime — eks.aws.upbound.io (Premium/new) and eks.aws.crossplane.io (Dedicated/old). The dataplane's dataplane-metadata-… cluster CR landed on the old CRD (READY=False), but dataplane-manager GetDataPlane looks it up under the new upbound CRD → "not found" → pre-create-tidbcluster "failed to load shared pool" → create DAG deadlocks → config-center never registers components → cluster-ng-server loops "tidb cluster version not found" → CREATING forever. (Not a DB seed gap — config-center defaults are all present.)

Two bugs found on main HEAD (both from the #601 Dedicated import)

• IAM policy size (FIXED + merged): the Premium workload managed policy hit 6902 chars > AWS's hard 6144 limit → substrate_apply failed. Split into <6144 policies. PR #615 merged.
• Crossplane CRD split (diagnosed, fix needs an architecture call): metadata cluster on the old CRD vs dataplane-manager querying the new one. Fix is in the dataplane-manager image / which runtime owns the metadata cluster.

Recommendations

1. Instrument the two black boxes. Emit sub-timings inside host_ready (RunInstances → SSH-ready → remote bootstrap/docker/k3d) and session_ready (per source-image build, per-Tilt-resource ready, "all-ready-at" timestamp). Add OpenAPI train timing (create→ACTIVE, per readiness gate).
2. Attack the dominant costs. session_ready is gated by source-image builds (cache them / prebuild) and the Dedicated-runtime footprint (#601). host_ready (EC2 cold boot 2m11s) is fixed per-run overhead — a warm/pooled host would remove it.
3. The create→ACTIVE dataplane wait is the real ceiling for the daily loop — real EKS provisioning per cluster. For agent dogfood, a pre-warmed shared pool would avoid per-create EKS spin-up.