# cp9 iteration speed-up — analysis + multi-angle brainstorm

Goal: accelerate the **agent auto-loop** (cp9's main purpose). Two targets:
- **(A) tilt `session_ready`** — with 5 services recompiling per iteration → **~3 min** (today ~7m10s, mostly image *pulls* not compiles).
- **(B) `create → ACTIVE`** → **~15 min** (today ~33–35 min).

## The one idea behind both

Both bottlenecks are **cold-provision costs paid on every iteration**, and in both the product **already has the architecture for warm reuse — only the warm half is unbuilt**:
- (A) services are deployed as raw `kustomize` `local_resource`s, so Tilt can't hot-reload them. Move source-built services onto Tilt-native image objects → **`live_update`** (sync the recompiled binary into the running pod) instead of rebuild→import→reschedule.
- (B) a create already runs two DAG phases — `create-sharedpool` (provisions the dataplane EKS, ~18 min) then `create-tidbcluster` (lands a logical `TiDBGroup` on the pool, designed via `spec.SharedPoolId`). **Keep pools warm** → a create runs only phase 2.

For an agent loop the principle is: **keep the env and the dataplanes warm; only hot-swap what changed.**

---

## (A) tilt `session_ready` — mechanism + levers

### How it works today (file refs)
- premium-full uses a **hand-authored** Tiltfile (`personas/premium-full/example.Tiltfile`), pointed at directly (`internal/lifecycle/steps.go:52-54`); the `tiltfile_generate` codegen path is NOT used (≈0s stage).
- Every service is a `local_resource` + shell script + `kustomize` image-swap. **No `docker_build`/`custom_build`/`live_update`/`sync` anywhere** (verified repo-wide).
- Build vs pull is one static list: `source_build_services` (`example.Tiltfile:19-22`, **currently empty → everything PULLS staging ECR images**). `_product_image` (`:533-536`) routes a listed service to `_local_image_resource` (build) else to a staging pin. The `fast-start`/`staging_image=…`/`always_local_stub=…` summary is just evidence JSON (`:675-695`).
- remote-ec2-k3d build path (`build/remote/build-remote-image.sh` → runs `build/build-local-image.sh` **on the EC2 host**): syncs source once per git-stamp (reused across services), serves the Go **module-download** cache over an SSH reverse tunnel, then on the host does `go build` (single-stage, `CGO_ENABLED=0`) → `docker build` → `k3d image import`.
- **The three structural slow-points:**
  1. **No build caching**: no `GOCACHE` set/persisted, no Docker BuildKit `--cache-from` (plain `docker build`; starter even forces `--no-cache`). Fresh host = cold compiles every time.
  2. **k3d import is globally serialized** — a filesystem mutex (`build-local-image.sh:252-268`) means 5 services import one-at-a-time even though builds run in parallel (`max_parallel_updates`, `:167`).
  3. **`session_ready` blocks on the whole dependency chain** (`tilt_ready_step.go:3187-3224`): metadb seed → dedicated-global-service rollout → cluster-ng → gateway → portal probe. The 7 min ≈ parallel pulls (serialized import) + this sequential rollout/seed chain.

### Levers (multi-angle), prioritized
**T1 — structural, decisive**
1. **`live_update` for the ~5 hot services.** Restructure source-built services onto Tilt-native `custom_build`/`docker_build` + `k8s_yaml` so Tilt owns the image→deploy edge, then `live_update` syncs the freshly cross-compiled binary into the container and restarts the process — skipping image build, the serialized k3d import, and pod reschedule. The binary is already built standalone (`build-local-image.sh:81`), so syncing it is natural. → multi-minute rebuild becomes **seconds**.
2. **"dev-loop / fast-redeploy" mode that decouples iteration from the full proof chain.** Per iteration you only need the 5 changed deployments rebuilt + their `rollout status` — not a re-run of metadb seed + gateway/portal proof. A mode that triggers only changed resources and waits only on them bounds the per-iteration cost to build+reload+per-service rollout. (The 7-min number is largely one-time cold start; the *per-iteration* cost is what matters for agents.)

**T2 — make the unavoidable compiles cheap**
3. **Persistent + shared `GOCACHE` on the EC2 host** (under `remote_state_dir`), warmed once → incremental single-package recompiles are seconds.
4. **Docker BuildKit layer cache / `--cache-from` (+ a stable base image, drop `--no-cache`)** → the docker step becomes a trivial binary-layer swap.
5. **A long-lived remote builder** (warm container on the EC2 holding GOCACHE + module cache + buildkit cache) instead of cold per-build environments; optionally a Go remote build cache (sccache/bazel-style) shared across sessions.

**T3 — remove serialization / scope**
6. **De-serialize k3d image distribution**: push to a small in-cluster registry once and let k3d *pull* in parallel, instead of the globally-locked `k3d image import`.
7. **Fine-grained change detection**: `<service>-image` deps are the whole repo (`:428`,`:441`) and the stamp is the whole sync-root git state — narrow to each service's package subtree so one change doesn't invalidate all.
8. **Skip-the-image hot path for dev**: mount the new binary via a shared volume and restart the process (a poor-man's `live_update`) for services not yet on Tilt-native build.

**Recommendation for ~3 min:** (1)+(2) are decisive — `live_update` removes build/import/reschedule from the inner loop and the fast-redeploy mode stops re-paying the proof chain; (3)+(4) make the compiles incremental. 5 changed services then ≈ 5 incremental Go compiles (parallel, cached) + binary syncs ≈ well under 3 min, often seconds.

---

## (B) `create → ACTIVE` — mechanism + levers

### How it works today (file refs)
- Ingress (sub-second): tidb-mgmt-portal → tidb-mgmt-service → nextgen-global-service → gRPC `CreateTiDBCluster` on cluster-ng-server.
- **Two sequential DAGs** on cluster-ng-executor: **`create-sharedpool`** (provisions the pool = the dataplane EKS + infra) then **`create-tidbcluster`** (creates a logical `TiDBGroup` *on* the pool; `TiDBGroup.spec.SharedPoolId` links them — `…_tidbgroups.yaml:23-24,54-55`). Per-component node quotas drive `apply-pd/tikv/tidb/tiproxy`.
- dataplane-manager provisions the dataplane EKS via the **Upbound** Crossplane provider: network (~5-6 min) → **EKS control plane (~10 min, AWS floor)** → **node groups joining (~7 min, via cluster-autoscaler — reactive, not Karpenter)** → a long **sequential Helm addon stack** (autoscaler, LB controller, cert-manager+PCA, coredns/cni/csi, node-local-dns, metrics, …).
- TiDB rollout (~11 min): tidb-operator → PD→TiKV→TiDB statefulsets, `imagePullPolicy: Always` from ECR on cold nodes, then **S3 Bucket CR reconciliation on the critical path** (`pool→AVAILABLE` waits on `BucketLifecycleConfiguration` Ready — retro) + public LB.
- **AGENTS.md is explicit: "DPs are create/delete dynamically regarding API call. Not provisioned in CD stage."** → the dataplane EKS is built **fresh every create; nothing is pre-warmed.** Pool *idle-GC* is half-wired (`gcConfig` on the dedicated regional-server; `cleanSharedPoolCron` exists but `enableCleanSharedPool: false`); pre-*warm* is an explicit, unstaffed open follow-up (`docs/20260417-…:323`, `docs/20260428-…:657`).

### Where the ~33–35 min goes
network ~5-6 · **EKS control plane ~10 · nodegroups ~7** · addons (overlap) · **TiDB rollout ~11** (image pulls + S3 CR + LB). ~90% is AWS EKS + node + pod provisioning; cp9 orchestration is seconds.

### Levers (multi-angle), prioritized
**T1 — decisive**
1. **Warm / pre-provisioned shared pool.** Keep N idle Active dataplane pools; on create run **only `create-tidbcluster`** (land a `TiDBGroup` + scale a node group) → skip the entire ~18 min EKS/network/addon build. Architecturally supported today (the pool model, `resource_pools`, `shared_pool_id`, `TiDBGroup.SharedPoolId`); the missing piece is a **pre-warm/keep-N-ready controller** + routing create onto a ready pool. **→ ~18 min saved → ~15-17 min (hits the goal on its own).** Cost: idle EKS control plane + small admin nodegroup running continuously.

**T2 — attack the residual ~11-15 min**
2. **Warm worker nodes.** Even with a warm control plane, scaling from zero re-pays ~7 min. Use **EC2 ASG warm pools** (already in the Crossplane CRDs: `warmPoolConfiguration/warmPoolSize`) or **Karpenter** (faster than the current cluster-autoscaler), or keep a small standby capacity sized for the default cluster. → ~5-7 min.
3. **Pre-pull TiDB images onto warm nodes** (AMI bake or a pre-pull DaemonSet; flip `Always`→`IfNotPresent`) + **enable `internetConfig.enableECRInterfaceEndpoints`** (currently false) to keep ECR pulls in-VPC. → ~2-4 min off the rollout.

**T3 — cheap config/perms trims**
4. **Smaller default dev topology** (single-AZ PD/TiKV/TiDB, drop the forced `pd_scheduling:2`/`tiproxy:2` where the product version allows) → fewer pods/nodes to Ready. ~2-4 min.
5. **Overlap the addon Helm stack with PD/TiKV bring-up** and prune dogfood-only addons (goldpinger, node-pinger, …). ~1-3 min.
6. **Faster S3 bucket-CR reconcile** (correct `cp9-owner/cp9-session` labels + `s3:PutLifecycleConfiguration` perms so it doesn't stall the AVAILABLE tail). ~1-2 min.

**Radical angles (for the agent-loop profile specifically)**
7. **One long-lived shared dataplane hosting many logical clusters.** For agent loops you rarely need a *fresh* EKS per test cluster — a single warm dataplane can host every test cluster as a `TiDBGroup`; create = deploy the group (~minutes), delete = remove the group. This is lever 1 taken to its limit and is the highest-leverage shift for iteration speed.
8. **Snapshot/clone a pre-provisioned cluster** instead of create-from-scratch.
9. **Lighter dataplane substrate for non-fidelity loops** (e.g. nodes via Fargate to skip node-join wait, or a local/kind dataplane for pure control-path iteration), reserving real-EKS provisioning for fidelity runs.

**Recommendation for ~15 min:** Lever 1 (warm pool) alone reaches ~15-17 min and is the design's intended-but-unbuilt path; add 2+3 (warm nodes + pre-pulled images) to get toward ~8-10 min; 4-6 are cheap trims. Lever 7 is the strategic end-state for agent loops.

---

## Cross-cutting: measure first
Add per-sub-step telemetry before tuning (so each lever is verifiable):
- cluster-ng: per-DAG-task (`create-sharedpool`/`create-tidbcluster`, `apply-*`) start/finish timestamps.
- dataplane-manager: `DataPlane` phase transitions (network-ready / eks-ready / nodegroups-ready / appbox-ready).
- Crossplane/upbound managed CRs: scrape `.status.conditions` Ready transition times (VPC/EKS/Nodegroup/Bucket) — free AWS-side attribution.
- tilt: per-`uiresource` Ready timestamps (which staging pulls / rollouts dominate the 7 min).

## Suggested phasing
1. **Telemetry** (both paths) — validate the splits.
2. **(A) `live_update` + GOCACHE/buildkit cache + fast-redeploy mode** → the 3-min loop.
3. **(B) warm-pool keeper** (the ~18-min win) → ~15 min, then **warm nodes + pre-pulled images** → ~8-10 min.
4. Cheap trims (topology, addon overlap, S3 perms, ECR endpoint) opportunistically.

Both #2 and #3 are "build the warm-reuse half the product already half-models," and both pay off every single agent iteration.