# Premium remote-ec2-k3d — Detailed end-to-end timing analysis

Env: cp9 Premium full persona, `remote-ec2-k3d`, region us-west-2, m7i.xlarge host.
Measured on a clean run after the fix in PR #624 (which is what finally lets the Train B
cluster reach ACTIVE — before it, the cluster deadlocked at the most expensive step).

There are **two cost phases**: (A) bringing the cp9 environment up, and (B) the Train B
cluster lifecycle (create → ACTIVE). The time sink in both is **real AWS resource
provisioning**, not cp9 orchestration.

---

## Phase A — `cp9 up` (environment provisioning): **10m55s**

Native per-stage breakdown (cp9 emits this):

| stage | duration | what it is |
|---|---|---|
| preflight / config_load | ~0s | local checks |
| owner_identity_ready | 4.0s | resolve AWS identity |
| iam_ready | 4.8s | IAM readiness |
| substrate_init | 0.05s | terraform init |
| substrate_plan | 35.1s | terraform plan |
| substrate_apply | 48.2s | terraform apply (substrate) |
| **host_ready** | **2m6s** | EC2 launch + remote bootstrap (docker/k3d on the host) |
| service_plan / metadb_seed / context_materialize / tiltfile_generate | ~0s | local prep |
| tilt_ready | 6.6s | Tilt API up |
| **session_ready** | **7m10s** | **all 72 Tilt resources to Ready (image builds + service rollout)** |

**Dominant:** `session_ready` (7m10s, 65% of `up`) = image builds + every cp9 service
becoming Ready, then `host_ready` (2m6s) = EC2 + remote runtime bootstrap. Everything else
is seconds.

---

## Phase B — Train B cluster lifecycle: `create → ACTIVE` ≈ **33–35 min**

The OpenAPI calls themselves are sub-second; all the time is downstream provisioning.
Milestone timeline (precise deltas from the live monitor; create T0 is ~2–3 min before the
monitor started, so create→EKS-CR is approximate, the rest are exact):

| step | duration | what is happening |
|---|---|---|
| `CreateTidb` API call | **~2s** | request accepted, returns tidbId |
| create → dataplane network + EKS CR submitted | **~5–6 min** | VPC + **9 subnets** + IGW + 3 NAT + route tables (all `*.ec2.aws.upbound.io`), then the `Cluster.eks.aws.upbound.io` CR is created |
| EKS CR → **dataplane Active** | **~17–18 min** | **EKS control-plane provisioning (~10 min, AWS-side) + node groups joining (~7 min) + appbox/management setup** |
| dataplane Active → **cluster ACTIVE** | **~11 min** | TiDB operator + PD/TiKV/TiDB component rollout on the dataplane + endpoint (LB) creation |
| reset-password / enable-public API | sub-second | mutations are async; underlying provision is separate |
| **create → ACTIVE total** | **~33–35 min** | |

Live anchors from the monitor:
- EKS CR present by 08:49:57 (upbound EKS = 1)
- dataplane phase `Active` at 09:07:24
- cluster state `ACTIVE` at 09:18:20

(The dedicated regional-server stayed at 0 reconciles the whole time — the #624 fix —
so there was no duplicate crossplane.io stack and no deadlock.)

---

## Where the time goes (找到耗时步骤)

1. **EKS control-plane + node-group provisioning inside the dataplane: ~17–18 min** — the
   single biggest item in the whole loop. This is pure AWS EKS latency (control plane create
   + workers joining), not cp9.
2. **TiDB component rollout (operator + PD/TiKV/TiDB + endpoints): ~11 min.**
3. **cp9 `session_ready` (image builds + service Ready): ~7 min** — only paid once per env.
4. **EC2 host bootstrap (`host_ready`): ~2 min** — once per env.
5. Dataplane network (VPC/subnets/IGW/NAT): ~5–6 min.

Everything cp9-orchestration-side (OpenAPI calls, the create DAG steps, IAM, terraform plan)
is seconds. **~90% of a cold cluster's time is AWS EKS + node + TiDB-pod provisioning.**

---

## Recommendations

### (a) Where to add telemetry
- **cp9 `up`** already has native per-stage timing (table above) — good; no change needed
  beyond optionally splitting `session_ready` per-Tilt-resource (see (c)).
- **Train B create has no per-sub-step timing exposed** — OpenAPI only surfaces coarse
  `CREATING → ACTIVE`. Add timestamps at:
  - the **cluster-ng create DAG** (`create-sharedpool`, `create-tidbcluster` tasks) — emit
    per-task start/finish so the ~11-min component rollout is attributable.
  - the **dataplane-manager**: emit `DataPlane` phase-transition timestamps with sub-phases
    (network-ready, eks-ready, nodegroups-ready, appbox-ready) so the ~18-min dataplane
    provision is attributable.
  - the **Crossplane/upbound managed resources**: record `Ready` transition timestamps for
    VPC / EKS Cluster / Nodegroup CRs (they're already in `.status.conditions`); scraping
    these gives the exact AWS-side split for free.

### (b) Which processes dominate
EKS control-plane + node-group provisioning (~18 min), then TiDB component rollout (~11 min).
These are the two to optimize (e.g. warm node pools / pre-provisioned shared pools) — the
cp9 control path is not the bottleneck.

### (c) How to record Tilt start → fully-ready
cp9 already does it: `tilt_ready` (API up, 6.6s) + `session_ready` (all resources Ready,
7m10s). `session_ready` **is** the Tilt-start→fully-ready duration. For per-resource detail
(which image builds dominate the 7 min), query the Tilt API
(`tilt get uiresources -o json`) and read each resource's Ready condition timestamp.

---

## Caveat — what is NOT in this report
The **public-endpoint → live `SELECT 1`** tail was not captured: on cp9-dev the private
`VPC_PEERING` endpoint isn't reachable from the laptop or the k3d pod (separate VPC, no
peering), and enabling public access never produced a reachable `PUBLIC` host in the
attempts made (possible cp9-dev limitation in wiring the public NLB + customer DNS).
A dedicated isolated env would be needed to time that tail; per decision, we stopped here.