# PRD: Accelerate cp9 single-iteration speed for agent auto-loops

Status note, 2026-07-06: the warm-pool prewarm command/status surface was
withdrawn in the command-surface cleanup. `cp9 prewarm` and `cp9 prewarm
status` are no longer available; warm-pool material in this PRD is historical
design context, not current implementation guidance.

## Problem Statement

I run automated loops against cp9: bring up an environment, create a Premium TiDB cluster,
exercise the control path, tear it down, change something, repeat. Today every iteration
pays large **cold-provision** costs:

- `cp9 up` takes ~11 min (dominated by `session_ready` ~7m10s = image pulls + all services
  becoming Ready, plus `host_ready` ~2m6s).
- When I change service code, Tilt re-builds cold (no compile cache) and re-readies through
  the whole dependency chain.
- A Premium cluster takes **~33–35 min** to reach ACTIVE — dominated by **~18 min** of real
  EKS dataplane provisioning (control plane ~10 min + node groups ~7 min) and **~11 min** of
  TiDB component rollout.

All the cp9 orchestration itself (OpenAPI, the create DAG, IAM, terraform) is sub-second;
the wall-clock is real AWS provisioning. These costs are the main thing blocking fast
iteration, which is cp9's entire purpose (fast agent auto-loops). Three things make this
hard to fix naively: we are under a **tight cost budget** (can't add always-on
infrastructure), cp9 must stay **production-fidelity** (the dataplane EKS is the customer's
data plane and must remain a real EKS), and the fix must live **inside the cp9 repo** —
changing other teams' upstream service repos adds hidden coordination cost and risk.

## Solution

An iteration should cost minutes, not tens of minutes, while keeping a real,
production-fidelity dataplane and adding no always-on infrastructure.

- **Tilt re-iteration:** when ~5 services change, rebuild + reload them in **~3 min** — by
  syncing the freshly-compiled binary into the running pod (`live_update`) instead of
  rebuild→k3d-import→reschedule, backed by persistent build caches, and a "fast-redeploy"
  mode that re-rolls only the changed services and waits only on their readiness.
- **`create → ACTIVE`:** cp9 **pre-warms the environment's single real shared dataplane
  pool** during `up`; every subsequent create lands a logical cluster on that warm pool and
  **skips the ~18 min EKS provision** → ~15 min, trending to ~5–8 min once images are
  pre-pulled and a smaller dev topology is used.
- **Sharing:** an already-built, warm environment can be handed directly to a trusted
  teammate or agent (full trust → no access-control hardening), so the setup cost is
  amortized across whoever reuses it.
- **Prerequisite — credentials:** the environment's injected AWS session expires in ~1 h
  with no supported refresh, which breaks any long-lived or shared environment. Provide a
  `cp9 refresh-creds` command now, and move the in-cluster controllers to a non-expiring
  credential source (host instance-profile / EKS IRSA) so there is no 1 h cliff.

Everything lands inside the cp9 repo (CLI/lifecycle orchestration, persona config,
Tiltfile/build scripts, substrate); no upstream business-repo changes; the dataplane stays a
real EKS; and no extra always-on EKS beyond the environment's own single dataplane.

## User Stories

1. As an agent running a cp9 loop, I want a single changed service to rebuild and reload in seconds, so that I am not blocked minutes per code edit.
2. As an agent, I want ~5 changed services to rebuild and reload in ~3 minutes total, so that a typical multi-service change does not stall the loop.
3. As an agent, I want only the services I changed to rebuild and re-roll, so that I do not re-pay the full env-ready proof chain (DB seed, gateway, portal) every iteration.
4. As an agent, I want incremental Go compiles to be cached across builds, so that re-compiling one changed package is seconds, not a cold full build.
5. As an agent, I want the container image step to reuse layer cache, so that only the changed binary layer is rebuilt.
6. As an agent, I want building several changed services to happen in parallel without a serialized image-import bottleneck, so that N changes do not cost N×(import time).
7. As an agent, I want a Premium cluster to reach ACTIVE in ~15 minutes or less, so that the create step does not dominate my loop.
8. As an agent, I want my create to land on an already-provisioned warm dataplane, so that I skip the ~18 minutes of EKS provisioning.
9. As an agent, I want the warm dataplane to be a real EKS, so that what I test matches production fidelity.
10. As an agent, I want creating and deleting many clusters in one environment to reuse the same warm pool, so that the provisioning cost is paid once per environment, not once per cluster.
11. As an agent, I want TiDB component images pre-pulled on the warm pool's nodes, so that the first PD/TiKV/TiDB pods do not pay a cold ECR pull.
12. As an agent, I want a smaller default cluster topology in dev, so that fewer pods/nodes need to become Ready.
13. As a teammate/agent, I want to directly reuse a colleague's already-built warm environment, so that I do not rebuild from scratch.
14. As a teammate sharing an environment, I want to hand over access without security ceremony, because we are fully trusted, so that sharing is frictionless.
15. As an agent on a shared environment, I want my clusters isolated as my own logical clusters on the shared pool, so that my work does not collide with others'.
16. As a cp9 user, I want a command to refresh the environment's AWS credentials, so that a long-running environment does not fail with ExpiredToken.
17. As a cp9 maintainer, I want the in-cluster controllers to use a non-expiring credential source, so that long-lived and shared environments never hit the 1 h credential cliff.
18. As a cost owner, I want no extra always-on EKS beyond the environment's own single dataplane, so that the speed-up does not blow the budget.
19. As a cp9 maintainer, I want every change to live in the cp9 repo, so that we do not take on cross-team coordination dependencies.
20. As an agent or maintainer, I want per-step timing telemetry on both the build path and the create path, so that I can verify where the time goes and confirm each optimization's effect.
21. As an agent, I want the warm-pool pre-warm to overlap with `cp9 up` where possible, so that the first create is already fast.
22. As an agent iterating on the control path, I want the shared-environment loop to cover cluster lifecycle (create→ACTIVE→delete), so that I can validate control-path behavior fast.
23. As an agent iterating on service code, I want to understand that code changes are global to a shared environment, so that I use a private environment for code-change loops.

## Implementation Decisions

**Tilt fast re-iteration (path A)** — decisions in ADR 0011:
- Restructure the source-built services (the currently-empty `source_build_services` set in the premium-full Tiltfile) onto Tilt-native image objects so Tilt owns the image→deploy edge, enabling `live_update`: incrementally cross-compile the binary on the host (persistent GOCACHE), sync it into the running pod, and restart the process. Only cp9-built binaries are synced — no service-source changes.
- Introduce a persistent, shared Go build cache (`GOCACHE`) and Docker BuildKit layer cache on the build host; stop forcing `--no-cache` (for the full-rebuild path).
- Replace the globally-serialized `k3d image import` with parallel distribution (push once to an in-cluster registry that k3d pulls from, or per-image scoping) for the full-rebuild path.
- Make change detection per-service (narrow the image-resource deps and the source stamp to each service's package subtree) so one change does not invalidate all.
- _Dropped: a separate "fast-redeploy / skip proof chain" mode — the ~7 min `session_ready` proof chain is a one-time `cp9 up` cost, not re-run on iteration, so skipping it is automatic (ADR 0011)._

**Warm dataplane reuse (path B)** — decisions settled in ADR 0009:
- Add a cp9 lifecycle step that pre-provisions the environment's single shared dataplane pool by calling the **`create-sharedpool` gRPC OPS API on tidb-mgmt-service** (pool-only, ~18 min — no keeper cluster). **Async pre-warm:** `cp9 up` does not block on the pool reaching ACTIVE — it provisions in the background — and cp9 **exposes a readiness poll/status** for "is the warm pool ready". Subsequent creates reuse the pool via the existing shared-pool mechanism (a `TiDBGroup` lands on it by `SharedPoolId`). cp9 is only a gRPC client + config — no cluster-ng / dataplane-manager source changes.
- **Minimal pool topology** (smallest/fewest TiDB components + EC2) to shorten pool creation, via cluster-ng meta-DB config (cp9-owned seed) + `create-sharedpool` API params.
- **Nodes scale-to-zero** while the pool is idle (warm = control plane + network + appbox only, no idle worker nodes); the first create on an empty pool pays the ~7 min node-join once per env.
- **No keeper:** rely on pool persistence (`enableCleanSharedPool: false`); verified no idle-GC reaps the empty premium pool (dataplane-manager has no idle/GC config; #624 excludes premium from the dedicated regional-server's 24 h GC). The pool lives for the env's lifetime.
- **Teardown:** tag the pool's resources with `cp9-session`/`cp9-owner` via **Crossplane provider-level default tags** (cp9 providerConfig) so `cp9 destroy`'s tag-scoped reaper cleans it — no dataplane-manager source change.
- Reduce the residual component-rollout time with cp9-side moves: a cp9-deployed pre-pull DaemonSet for TiDB images on the warm pool's nodes; a smaller default dev topology via cp9's `cluster-ng-server` config (`nodeCountQuota`); node/AMI/addon tuning in cp9 persona configs (e.g. faster node bring-up, pruning dogfood-only addons, ECR VPC endpoint).

**Shared environment (path C)** — withdrawn:
- The historical `cp9 share` bundle from ADR 0012 was retired. `cp9 status` now reports the current session's read-only access facts; credential transfer and access mutation stay outside cp9.

**Env credentials — root fix (prerequisite for long-lived/shared envs)** — decision in ADR 0010:
- Move the in-cluster controllers off the injected `aws-creds` session and onto the **EC2 host instance-profile via IMDS** (auto-rotating, no 1 h cliff, no session injection). The cp9 substrate gives the host an instance-profile with the controllers' permissions (or able to assume the per-controller sub-roles); the controllers read creds from the host IMDS. IRSA is rejected — the cp9 control plane is k3d on EC2, not a real EKS (no OIDC). Go straight to this root fix; no interim `cp9 refresh-creds` command. Lives in cp9 substrate + controller config. (The human-driven `cp9` CLI's own SSO is separate and unaffected.)

**Telemetry (enabler)**
- Emit per-step timing: cluster-ng create-DAG task start/finish; `DataPlane` phase transitions (network/eks/nodegroups/appbox-ready); Crossplane/upbound managed-resource `.status.conditions` Ready transitions; per-Tilt-resource Ready timestamps.

## Testing Decisions

- A good test asserts **external behavior** — end-to-end wall-clock and end-state — not internal implementation. The highest, single seam is the **existing premium integration-test harness** (`TestIntegration_*` / the premium lifecycle path that already drives create→poll-ACTIVE→delete via the real OpenAPI). Prefer extending it over adding new seams.
- **Path B:** at that seam, assert `create → ACTIVE` wall-clock is under the target on a pre-warmed environment, and assert the dataplane is a real EKS (the managed-resource family is upbound, the EKS cluster is real). Prior art: the existing premium lifecycle smoke / integration test.
- **Credentials:** test that the env's controllers keep making successful AWS calls past the old ~1 h session boundary without intervention (the instance-profile / IMDS root fix — ADR 0010).
- **Path A:** assert that a change to a service triggers only that service's rebuild+roll and completes under the target, and that unchanged services are not rebuilt; measured via the build path's evidence/timing output rather than internal cache state.
- **Path C:** test that a second party can, with shared access, create and reach ACTIVE for their own isolated cluster on the shared warm pool.
- Telemetry is validated by the timing assertions above (the per-step numbers must add up to the measured end-to-end).

## Out of Scope

- A fleet-level managed environment pool / scheduler (explicitly out — keep sharing simple and direct).
- Mock or lightweight (k3d/kind) dataplane substitution — the dataplane must stay a real EKS for fidelity.
- Access-control / multi-tenant isolation hardening for shared environments — full trust is assumed.
- Eliminating AWS-imposed floors (EKS control-plane create ~10 min; EKS control-plane cost while warm) — these are amortized by warm reuse, not removed.
- Changes to upstream business-line repos (cluster-service-ng, tidb-management-service, dataplane-manager, etc.).
- Optimizing the one-time `cp9 up` beyond what directly helps the iteration loop.

## Further Notes

- 2026-07-08 scoped acceleration: cp9's Terraform runner accepts
  `CP9_TERRAFORM_PARALLELISM=<n>` and passes `-parallelism=<n>` to
  `plan`/`apply`/`destroy`. This changes Terraform's maximum concurrent
  operations for graph-independent resources; unset keeps Terraform's default
  of 10. The dev-full Tiltfile keeps causal ordering in `resource_deps` for
  namespaces, session context, images, Crossplane readiness, credentials,
  providers, provider configs, and service-specific runtime config, but does
  not treat transient service crashloops as a reason to serialize otherwise
  independent Kubernetes workloads. If a workload can self-converge after its
  config/image/readiness inputs exist, short startup chaos is acceptable. Local
  resources without shared output paths run with `allow_parallel=True` once
  their explicit dependencies are ready.
- The dominant fact: both bottlenecks are cold-provision-every-iteration, and the product
  already half-models the warm-reuse fix (Tilt-native builds → `live_update`; the
  `TiDBGroup`-on-`SharedPool` model → warm pools). Only the warm half is unbuilt.
- The **source SSO** the env credentials derive from also expires; the credential root fix
  (non-expiring controller source) removes that dependency for the controllers, though
  human-driven `cp9` CLI calls still need a valid SSO.
- Suggested phasing: P0 = telemetry + the credential root fix (ADR 0010, unblocks
  long-lived/shared envs); P1 = path A (ADR 0011); P2 = path B (ADR 0009 — warm pool, then
  minimal topology + pre-pull); P3 = path C (ADR 0012, shared env).
- Measured baselines and full mechanism breakdown (with file references) are in
  `cp9-premium-train-b-timing-detailed.md` and `cp9-iteration-speedup-design.md`.