# cp9 Host & Tenancy Refactor — Design Evaluation

Codename: **controlplane9** | Issue: [tidbcloud/controlplane9#79](https://github.com/tidbcloud/controlplane9/issues/79)

Status: Proposed | Date: 2026-04-30 | Author: @cdx-55

Derives from: [05_detail-design.md](20260425-design-cp9-detail.md), [07_wedge-premium-reserved.md](20260428-design-cp9-wedge-premium-reserved.md) | Inputs: AWS pricing pages (retrieved 2026-04-30); current cp9 branch after merging `origin/main@238b3c8`. Code-coupling audit remains against `tidbcloud/controlplane9@06b913b` because the newer main change is documentation-only for the host-refactor surface.

> All retrieval dates are 2026-04-30. Pricing is region-specific. EKS, NAT, and gp3 numbers below come from AWS pricing pages; EC2 instance rates are planning inputs from public EC2 price tables and must be re-checked in AWS Pricing Calculator or the AWS Price List API before an implementation commit.

> Historical command-surface note (2026-06-01): this design predates the command cleanup train. Its command examples may mention removed ideas such as `sync`, `onboard`, `config`, `persona`, `crd`, `migrate`, `test`, `cost`, `reaper`, `stub-gen`, `overlay`, `verify`, `feature-cycle`, `topology`, or `up --persona` / `up --discover` / `up --profile` / `up --remote`. Current runnable surfaces are `setup`, `doctor`, `up`, `status`, `logs`, `deploy`, `destroy`, direct `pause`, and `db dump-init`.

> Remote-workload decision (2026-06-16): for `host.mode: remote-ec2-k3d`, the local `cp9` CLI is a management/orchestration client. It may resolve config, create or select the remote EC2 session, package inputs, transfer them over SSH, fetch status/logs, and clean up. It must not use the developer's local Docker Desktop or local k3d as part of the workload build/import/runtime path. Service image pulls, Docker builds, k3d image imports, k3d runtime, and workload preflight checks run on the remote EC2 host.

> Sizing terminology (2026-06-16): EC2 instance type controls CPU and memory. Remote root EBS volume size is a separate disk-headroom setting for Docker, k3d, image cache, and build contexts. Do not describe root EBS disk size as "EC2 memory."

---

## TL;DR

Three options were on the table. After the 2026-04-30 updates to 05/07 and PR review feedback, the recommendation is narrower: keep cp9 **all-in on k3d** for both local and remote modes, and do not pursue shared EKS/vcluster for this issue.

| | Alt 0 (current) | Alt 1 (per-dev EC2) | Alt 2 (shared EKS + vcluster) | **Alt 3 (recommended: tiered k3d)** |
|---|---|---|---|---|
| **Incremental host cost / dev / mo (5 dev team)** | $0 | $38 active / $144 always-on | $57 shared host | mode-dependent |
| **Cold start (host)** | ~30 s k3d | ~3 min (EC2 boot + k3d) | ~30 s (vcluster boot) | mode-dependent |
| **Iteration loop (`cp9 deploy svc`)** | ~30 s local | ~60-120 s (rsync + remote build) | ~60-90 s (image push + redeploy) | mode-dependent |
| **Debug ergonomics** | full-local (best) | ssh + port-forward | RBAC-scoped (worst) | mode-dependent |
| **Dev control** | total | total (own EC2) | shared-cluster constraints | total |
| **Net new code surface** | none | ~700-1,000 LOC EC2 host adapter + sync + state | ~600-900 LOC vcluster lifecycle + auth + state | ~800-1,200 LOC |
| **Wedge-thesis impact** | none | none | none if customer dataplane stays real; **breaking** if also virtualizes dataplane | none |
| **Failure-mode blast radius** | one dev | one dev | **whole team** if host EKS is unhealthy | one dev (modes are isolated) |

**Recommended path: Alt 3 — tiered k3d offering.**

- Local k3d is still the **default** because the productivity properties (sub-ms CLI, full-local debug, total dev control) dominate when the developer's laptop can handle it.
- Per-dev EC2 + k3d (Alt 1) is the **opt-in escape hatch** for constrained laptops, big-cluster work, or "I need a beefy box today." Same architecture, just a different host adapter.
- Shared-EKS-vcluster (Alt 2) is **rejected for this issue**. It adds platform ownership, shared-cluster blast radius, RBAC/secret risks, and another Kubernetes tenancy model while the stated reviewer preference is all-in k3d.

The cp9 architecture changes additively: a `host.mode` field in `cp9.yaml` (`local-k3d` / `remote-ec2-k3d`) plus a host-adapter layer behind the existing `cluster_create`, kubeconfig selection, CRD apply, deploy/log, and destroy flows.

**Next experiment** (acceptance criteria below): first make the current local k3d path explicit behind `host.mode: local-k3d`, then build EC2 as the opt-in remote adapter behind `host.mode: remote-ec2-k3d`; measure real-world cold-start, iteration-loop, and debug-ergonomic deltas vs local. Do not spend implementation time on Alt 2 unless a future issue reopens shared-cluster tenancy.

---

## Problem framing — separate the host axis from customer dataplane side effects

The user's three alternatives implicitly mix two dimensions. The 2026-04-30 update to 05/07 clarifies an important boundary: cp9 does not have a separate dataplane persona capability. Customer dataplane provisioning is a normal production CP service side effect of OpenAPI create/update paths.

### Two independent dimensions

```
                           cp9 control-plane host:
                           local-k3d     EC2+k3d         shared-EKS-vcluster
Customer dataplane side effect:
  real EKS via OpenAPI     Alt 0         Alt 1            Alt 2 (a)
  create/update paths      (current)
  shared/virtualized dp    n/a           n/a              Alt 2 (b)
```

- The **cp9 control-plane host** runs the 4 REAL services (`tidb-mgmt-portal`, `tidb-mgmt-service`, `cluster-ng-service`, `dataplane-manager`), the 9 sub-apps (crossplane, cert-manager, kong, gateway-api-crds, ...), the 3-4 STUB services, and the Zero metadb. This is the **k3d cluster the design currently puts on a developer's Mac.** Cost: 0 local; per-dev EC2 cost; or amortized.
- The **customer dataplane** is the EKS that production service code provisions for the actual TiDB cluster (tidb-operator + TiDB pods) after the E2E test calls normal OpenAPI create/update APIs. It is not a cp9 bootstrap concern, but it is still the dominant cloud cost side effect of a realistic cp9 session.

**The wedge thesis** ([07 § wedge thesis](20260428-design-cp9-wedge-premium-reserved.md#the-wedge-thesis)) is anchored on normal OpenAPI create/update paths provisioning a **real customer dataplane**. Sharing or virtualizing that dataplane (Alt 2 (b)) breaks that thesis. Sharing only the cp9 control-plane host (Alt 2 (a)) does not break it, but it is still rejected here on productivity, ownership, and reviewer-preference grounds.

### Why the dataplane is the dominant cost lever

Pricing snapshot (us-west-2 where available, retrieved 2026-04-30):

| Component | Cost |
|---|---|
| EKS control plane (standard support, first 14 mo) | $0.10 / hr ≈ $73 / mo |
| EKS control plane (extended support, after 14 mo) | $0.60 / hr ≈ $438 / mo |
| EKS control plane (avg over 26-mo lifecycle) | $0.33 / hr ≈ $241 / mo |
| 3 × m6i.large EKS nodes (24×7) | 3 × $70.08 = $210 / mo |
| EBS gp3 (root + cluster), ~150 GB | ~$12 / mo |
| NAT gateway (per AZ, 24×7) | $32.85 / mo + $0.045 / GB processed |
| 1 × m6i.xlarge EC2 (24×7) | $140 / mo |
| 1 × m6i.large EC2 (24×7) | $70 / mo |

Sources: [AWS EC2 on-demand pricing](https://aws.amazon.com/ec2/pricing/on-demand/) plus public EC2 price tables for the specific m6i rates, [EKS pricing](https://aws.amazon.com/eks/pricing/), [VPC NAT pricing](https://aws.amazon.com/vpc/pricing/), [EBS pricing](https://aws.amazon.com/ebs/pricing/).

A **customer dataplane EKS** running 8 hours/day for 22 working days is **not** $73/mo flat — that assumes 24×7. The session is destroyed when work is done. Realistic active-only cost:

- EKS control plane: 8 × 22 = 176 hr × $0.10 = $17.60/mo
- 3 × m6i.large (sized for tidb-operator + a small TiDB cluster): 3 × 176 × $0.0960 ≈ $51/mo
- EBS gp3 for TiDB data, ~50 GB: $4/mo (charged 24×7 even when EKS is down)
- NAT gateway (always-on, per AZ): $32.85/mo

Per-session active-only ≈ **$70-105/mo per developer**, dominated by NAT. NAT can be amortized across developers if shared (it usually is — VPC A's existing NAT EIP per the design's tunnel contract).

So the **customer dataplane EKS side effect is ~$70-100 per active-developer per month** for a daily user. That dominates anything we do with the control-plane host.

**Implication.** If we want a step-change in cost, the lever is "share or virtualize the customer dataplane" (Alt 2 (b)) — but that breaks the current wedge thesis. The lever for "save dev's laptop" is host swap (Alt 1), which is small money.

---

## Current code checkpoint

Surveyed `tidbcloud/controlplane9@06b913b` on 2026-04-30; merged `origin/main@238b3c8` before this revision. The 2026-04-30 main update changes 05/07 docs, not the code paths below.

- `cp9 up` has a real lifecycle engine, but it is hard-wired to local k3d in `internal/cmd/cluster_cmd.go` and `internal/cluster/k3d_step.go`.
- `cp9 deploy` and `cp9 logs` are registered Day-2 commands; `verify` is no longer a product CLI surface, and feature tests now run as direct `go test` invocations against the active cp9 session via `pkg/cp9test`. Host-mode work must make the remaining registered commands real enough to route through a selected kubeconfig.
- `cp9 destroy` currently preserves the local k3d cluster and removes state + AWS resources. The design docs say "delete k3d"; implementation says "preserve k3d" because recreation is expensive. The host refactor must resolve this drift before adding EC2 destroy semantics.
- `config.Config` has `cluster.k3d` but no host-mode abstraction; `LocalConfig` only contains source, service, and secret overrides.
- `internal/crd.Applier` and the staging dumper are already kubeconfig/context based. They are the clearest proof that the right abstraction boundary is "session kubeconfig", not "local vs remote".
- No `internal/tunnel` package exists on main at this commit; tunnel behavior is still a design capability, not a merged implementation package.

## Existing code coupling — what we'd actually touch

Host-coupled surface from the current repo:

| Subsystem | LOC (host-coupled) | Files | Notes |
|---|---|---|---|
| k3d lifecycle | ~590 | `internal/cluster/k3d.go` (189), `k3d_step.go` (66), `destroy.go` (109), `state.go` (~40 of 226), `engine.go` call sites | Shells out to `k3d`; lifecycle state names assume a local cluster. `K3dClient` helps tests, but it is not yet a host abstraction. |
| Command wiring | ~130 | `internal/cmd/cluster_cmd.go`, `commands.go` | `up` injects `NewK3dClusterStep`; `destroy` does not delete k3d; deploy/log are the remaining Day-2 surfaces. |
| Config schema | ~80 | `internal/config/config.go` + tests | `cluster.k3d` is first-class. Need `host.mode`, `host.ec2`, and backwards-compatible defaults. |
| Doctor (preflight) | ~300 | `internal/doctor/preflight.go`, `health.go`, tests | Checks docker / kubectl / k3d / aws on the local Mac. Needs mode-conditional checks for EC2 launch/ssh. |
| CRD install | ~150 | `internal/crd/` | Targets a kubeconfig; agnostic to which cluster — drop-in for local k3d and EC2+k3d. |
| AWS cleanup / reaper | ~1,300 | `internal/aws/` | Already tag-scoped for SQS/S3/DynamoDB/SNS/Lambda. EC2 host mode needs EC2 instances, volumes, security groups, and possibly EIPs in the type table. |
| Persona / topology | ~600 (PR #70 merged) | `internal/persona/` | Host-agnostic. No change. |

**Total host-coupled surface to refactor:** ~1,200-1,600 existing LOC across 5 packages, plus ~700-1,000 net-new LOC for EC2+k3d mode. This is a real feature, not a small config toggle.

**What stays unchanged in every option:**

- C5 persona / topology loader (PR #70 merged).
- C4 reaper logic (the type table and tag scoping).
- C3 audit gates (every service's externally-configurable + tag-complete + topology-agnostic preconditions still apply).
- Wedge tests T1 / T2 / T3 (table-driven create + update + billing pipeline).

**What changes per option:**

| Subsystem | Alt 0 → Alt 1 (per-dev EC2+k3d) | Alt 0 → Alt 2 (shared-EKS-vcluster, rejected) | Alt 0 → Alt 3 (tiered k3d) |
|---|---|---|---|
| `cluster_create` lifecycle state | new `host_provision` sub-state to launch EC2; `cluster_create` runs `k3d` over ssh | would run `vcluster create`; do not implement for this issue | local and remote k3d, gated by `host.mode` |
| `cp9 deploy svc` | rsync source → remote build → `kubectl apply` over ssh | would require registry push or in-cluster registry | local k3d import or remote k3d build |
| `cp9 logs --trace-id` | `kubectl logs` over ssh-tunneled API | would query vcluster API | kubeconfig-routed local/remote k3d |
| `cp9 destroy` | terminate EC2 and delete/preserve remote k3d according to TTL policy; stage 2 reaper unchanged but extended to EC2 resources | would delete vcluster and tenant namespace | local k3d cleanup or EC2 termination |
| Kubeconfig handling | new `kubeconfig` field per-mode; ssh fetch from EC2 vs local file | would fetch via `vcluster connect` | local file vs EC2-fetched kubeconfig |
| Tunnel / VPC reachability | tunnel runs on EC2 if retained; if EC2 is in VPC A, some tunnel work may disappear | not pursued | mode-conditional |
| `cp9 doctor` | adds EC2 reachability + ssh-key + IAM-can-launch checks | not pursued | mode-conditional local/EC2 checks |

The **abstraction line** should be the kubeconfig: every downstream operation should route through `.cp9/<name>/kubeconfig.yaml` and context metadata. The current CRD path already works that way; deploy/log/test still need to be implemented against that contract.

**Code-shape recommendation.** Introduce `internal/host/` package with two implementations of an `Host` interface:

```go
type Host interface {
    Provision(ctx, sessionName) (HostHandle, error)
    Kubeconfig(ctx, h HostHandle) ([]byte, error)
    KubectlContext(ctx, h HostHandle) string
    DeployImage(ctx, h HostHandle, svc string, imageTar []byte) error
    StreamLogs(ctx, h HostHandle, svc, traceID string) (io.Reader, error)
    Destroy(ctx, h HostHandle) error
}

// Implementations:
//   internal/host/local/k3d/       — current k3d on Mac
//   internal/host/ec2/k3d/         — new (Alt 1)
```

The lifecycle state machine becomes mode-aware but not mode-specific; all mode-specific code stays in `internal/host/{local,ec2}`. This is the **minimum surface** that supports local k3d and EC2+k3d without rewriting persona, CRD, DB, and AWS cleanup packages.

---

## Cost model — broken out per cdx-55's review criteria

All numbers retrieved 2026-04-30; us-west-2 when available; `*estimate*` flags my reasoning.

### Cost components

| Component | Source | Per unit | Notes |
|---|---|---|---|
| EKS control plane (standard) | [AWS EKS pricing](https://aws.amazon.com/eks/pricing/) | $0.10 / hr | First 14 mo of a k8s minor version. |
| EKS control plane (extended) | same | $0.60 / hr | After 14 mo. Realistic only if cluster lives long. |
| m6i.large (EC2) | [AWS EC2 on-demand](https://aws.amazon.com/ec2/pricing/on-demand/) + [DevZero m6i.large table](https://www.devzero.io/instances/aws/m6i.large) | $0.096 / hr ~= $70 / mo @ 24x7 | 2 vCPU + 8 GiB. Fits k3d + 4 REAL + 4 stubs *barely*. |
| m6i.xlarge (EC2) | [AWS EC2 on-demand](https://aws.amazon.com/ec2/pricing/on-demand/) + [DevZero m6i.xlarge table](https://www.devzero.io/instances/aws/m6i.xlarge) | $0.192 / hr ~= $140 / mo @ 24x7 | 4 vCPU + 16 GiB. Comfortable for k3d + 9 sub-apps + 4 services. |
| m6i.2xlarge (EC2) | [AWS EC2 on-demand](https://aws.amazon.com/ec2/pricing/on-demand/) + [DevZero m6i.2xlarge table](https://www.devzero.io/instances/aws/m6i.2xlarge) | $0.384 / hr ~= $280 / mo @ 24x7 | 8 vCPU + 32 GiB. Headroom for spike loads. |
| EBS gp3 | [AWS EBS pricing](https://aws.amazon.com/ebs/pricing/) | $0.08 / GB-mo | 50 GB root + cluster ≈ $4 / mo. |
| NAT gateway | [AWS VPC pricing](https://aws.amazon.com/vpc/pricing/) | $0.045 / hr ($32.85 / mo) + $0.045 / GB | Already required by current design (VPC A egress); shared across all devs. |
| Data transfer (intra-AZ) | AWS standard | $0 | If EC2 + EKS in same AZ. |
| Data transfer (cross-AZ) | AWS standard | $0.01 / GB / direction | Avoid by single-AZ. |
| Shared-EKS/vcluster overhead | vCluster docs | *estimate:* 0.25-1 vCPU + 512 MiB-2 GiB RAM per tenant control plane | Included only for rejection analysis; do not implement in this issue. |

### Per-developer per-month cost (active-only: 8 hr/day × 22 days = 176 hr/mo)

Active-only is the realistic shape — sessions are destroyed at end-of-day, so 24×7 over-counts.

| Line item | Alt 0 (local) | Alt 1 (per-dev EC2) | Alt 2 (a) (shared host EKS) | Alt 3 default = local |
|---|---|---|---|---|
| Local PC cost | $0 (out-of-scope) | $0 | $0 | $0 |
| Per-dev EC2 (m6i.xlarge × 176 hr) | — | 176 × $0.192 = **$33.79** | — | mode-dependent |
| EBS gp3 50 GB (24×7) | — | $4 | — | mode-dependent |
| Shared-EKS host control plane share (24×7 / N devs, N = 5) | — | — | $73 / 5 = **$14.60** | — |
| Shared-EKS host nodes share (3 × m6i.large × 24×7 / 5) | — | — | $210 / 5 = **$42** | — |
| Shared-EKS NAT share (24×7 / 5) | $32.85 / 5 = **$6.57** | $32.85 / 5 = **$6.57** | $32.85 / 5 = **$6.57** | $6.57 |
| Customer dataplane EKS side effect (active 176 hr) | $17.60 + $51 nodes + $4 EBS = **$72.60** | same = **$72.60** | same = **$72.60** | $72.60 |
| **Total per-developer per-month** | **$79** | **$117** | **$136** | $79 (default) / $117 (opt-in EC2) |

Sensitivity:

- 10-dev team scales Alt 2's flat cost down: $73 + $210 = $283 / 10 + NAT share + customer dataplane side effect = $28 + $7 + $72 = **$107/mo/dev**. This is still rejected for this issue.
- Customer dataplane EKS is **the dominant cost in every option**. Sharing it (Alt 2 (b)) would save $50-70/dev/mo but breaks the wedge thesis.
- 24×7 vs active-only: if sessions are left running overnight, Alt 0 stays $0 local but the customer dataplane EKS jumps from $73 to $241 (avg lifecycle), and Alt 1's EC2 jumps from $34 to $140. **Tear-down discipline matters more than host choice.**

### Cost narrative

- **Alt 0 is the cheapest** by ~$40-60/dev/mo over Alt 1 / Alt 2, *if you're willing to ignore local-PC strain*. The user's framing ("不计入公司的 dev 环境开发成本") explicitly accepts this.
- **Alt 1 costs $34-140/dev/mo extra** for an EC2 — small money in absolute terms; whether it's worth it depends on the productivity delta below.
- **Alt 2 costs about the same as Alt 1** at small team size (5 devs); becomes cheaper than Alt 1 at 10+ devs. **Never beats Alt 0 in raw cost.**
- **Reducing customer dataplane lifetime** (e.g. teardown discipline + shared NAT) is a higher-leverage cost lever than any host-axis choice.

---

## Productivity probe — Cara's day in the life

Walk-through per option for the canonical Day-in-the-Life from [07 § Day-in-the-life](20260428-design-cp9-wedge-premium-reserved.md#day-in-the-life--cara-ships-a-reserved-sizing-calculator-fix). Each step measured against goal 2 (network delay, debug-friendly, quick code-deploy-verify, strong dev control).

### Step 1 — `cp9 setup` + `cp9 doctor`

| | Alt 0 | Alt 1 | Alt 2 |
|---|---|---|---|
| Time-to-first-prompt | ~5 s | ~5 s | ~5 s |
| Failure-mode | local: docker not running | remote: ec2 not reachable / sso expired | remote: shared-EKS down / RBAC denied |
| Recovery | local restart | ssh debug + EC2 console | requires platform team |

Alt 0 / 1 are roughly equivalent. Alt 2 has the highest "i'm blocked, ask someone" surface.

### Step 2 — `cp9 up` (cold start)

| | Alt 0 | Alt 1 | Alt 2 |
|---|---|---|---|
| Host bring-up | k3d up: ~15-30 s | EC2 launch + ssh: ~2-3 min | vcluster create: ~10-30 s |
| 9-sub-app install (crossplane, kong, …) | ~5-10 min | ~5-10 min | ~3-5 min (host EKS may already have cached images) |
| Customer dataplane EKS provision, triggered by OpenAPI create/update tests | ~15-30 min (unchanged) | ~15-30 min (unchanged) | ~15-30 min (unchanged) |
| **Cold-start total** | **~20-40 min** | **~22-43 min** | **~18-35 min** |
| Re-entry (warm) | ~30 s (k3d cache) | ~30 s (EC2 already up + k3d cache) | ~5 s (vcluster persists) |

vcluster's only real win is **warm re-entry** (5 s vs 30 s). For initial cold-start, the customer dataplane EKS side effect dominates regardless.

### Step 3 — Iterate (`cp9 deploy <svc>`)

| | Alt 0 | Alt 1 | Alt 2 |
|---|---|---|---|
| Source available | local | rsync to EC2 (~2-10 s for incremental) | local |
| Build | local docker build (~30-60 s) | remote docker build on EC2 (~30-60 s) | local docker build + push to ECR (~30-90 s incl. push) |
| Image transfer | k3d image import (in-VM, ~2 s) | already on EC2 | ECR pull on host EKS (~5-15 s) |
| Pod restart | ~5-10 s | ~5-10 s | ~5-10 s |
| **Total iteration** | **~40-75 s** | **~40-90 s** | **~50-115 s** |
| Network delay per CLI cmd | <5 ms | ~30-100 ms ssh round-trip | ~10-30 ms kubectl-over-VPN |

Alt 0 is the fastest iteration loop **by 10-40 s per cycle**. Over a day of dozens of `cp9 deploy` invocations, this compounds: ~5-20 min/day. Alt 2 is slowest because of the registry round-trip.

### Step 4-5 — E2E (normal OpenAPI create/update)

These exercise the real customer dataplane side effect, which is identical in Alt 0 and Alt 1. In 07's current flow this is driven through:

```bash
go test ./test/e2e -run TestPremiumReserved_CreateTopology/Create56
go test ./test/e2e -run TestPremiumReserved_UpdateTopology/Update56To104
```

Productivity is equivalent between local k3d and EC2+k3d for the dataplane provision itself; host choice mainly affects the code-deploy-debug loop around those tests.

### Step 6 — `cp9 destroy`

| | Alt 0 | Alt 1 | Alt 2 |
|---|---|---|---|
| Stage 1 (host cleanup) | current code preserves k3d; optional k3d delete ~10 s | EC2 terminate (~30-60 s) and optionally remote k3d delete | vcluster delete (~5 s) |
| Stage 2 (reaper) | unchanged (customer dataplane EKS + tagged AWS resources) | unchanged | unchanged |
| **Total destroy** | **~3-8 min** | **~4-9 min** | **~3-8 min** |

Equivalent.

### Debug ergonomics

| | Alt 0 | Alt 1 | Alt 2 |
|---|---|---|---|
| Attach IDE / debugger to a pod | local docker exec + IDE (best) | ssh + port-forward + IDE remote (works, more friction) | kubectl port-forward + RBAC permits (works, more layers) |
| Network capture (tcpdump) | local docker exec (good) | ssh into EC2 + docker exec (workable) | RBAC may deny (depends) |
| Inspect raw cluster state | `kubectl --context <local>` (instant) | `kubectl --kubeconfig <ec2-fetched>` (works) | `kubectl --context <vcluster>` (works) |
| Read host kernel logs / dmesg | not relevant (Mac) | ssh + sudo dmesg (works) | RBAC deny (host kernel is shared infra) |
| Cluster admin (kubectl edit ds, etc.) | full | full | restricted (vcluster owns its API; host EKS admin is platform team) |

**Alt 0 wins on debug ergonomics.** Alt 1 is close (you own the EC2 — `sudo` works). Alt 2 has ergonomic constraints at the host-EKS layer (you don't own the host).

### Productivity verdict

For the **median dev-day** (lots of small iterations, occasional create/update E2E), **Alt 0 wins clearly** on iteration loop and debug ergonomics. Alt 1 is acceptable as an opt-in escape hatch. Alt 2 has the worst per-iteration loop *but the best multi-session re-entry latency*; that tradeoff is not enough to justify the shared-cluster ownership burden for this issue.

---

## Security / isolation tradeoffs

Per cdx-55's review criteria.

### IAM blast radius

| | Alt 0 | Alt 1 | Alt 2 |
|---|---|---|---|
| AWS identity used for tunnel | per-dev SSO email → `cp9-developer` role | per-dev SSO email → `cp9-developer` role (assumed by EC2 instance profile or via SSO token forwarding) | shared-EKS service-account (IRSA) → role assumed by vcluster pod |
| Boundary on what cp9 can create | `cp9-developer` permission boundary (existing) | same | **same role**, but multi-developer activity goes through one identity by default |
| Secret material location | dev's Mac (`.aws/credentials` or SSO cache) | EC2 (`.aws/credentials` or SSO cache forwarded) | shared-EKS secret manager / IRSA |
| Compromise scenario | one dev's Mac → one dev's blast radius | one dev's EC2 → one dev's blast radius | shared-EKS pod escape → **whole team's blast radius** |
| Owner-scoped reaper | works (each dev's `cp9-owner` distinct) | works | **works only if `cp9-owner` is propagated per-vcluster** (audit gate) |

**Alt 2 has the largest IAM blast radius** if not carefully scoped. The mitigation is per-vcluster IRSA + per-developer impersonation — workable, but it is extra platform-team operational cost for an option this issue rejects.

### vcluster tenant isolation

From [vCluster multi-tenancy guide](https://www.vcluster.com/guides/tenancy-models-with-vcluster) (retrieved 2026-04-30):

- **Shared nodes** (default, OSS): tenant pods are scheduled as host pods in a host namespace. Each vcluster has its own API server but shares **kernel, network, and storage**.
- **Dedicated nodes**: tenant pods on labeled nodes; better workload isolation; same control plane.
- **Private nodes** (Pro): tenant gets a separate physical node pool; full hardware isolation.

vCluster's own docs treat shared-node mode as soft isolation: the virtual API server is isolated, but stronger workload isolation needs dedicated/private nodes or extra runtime/network controls. For cp9 this is avoidable complexity because local k3d and EC2+k3d preserve single-developer isolation.

### Secret handling

| | Alt 0 | Alt 1 | Alt 2 |
|---|---|---|---|
| Dev secret store | local file / SSO cache | EC2-local file / SSO cache | shared-EKS Secret resources, RBAC-scoped per-vcluster |
| Cross-developer secret leak risk | none (per-Mac) | none (per-EC2) | **non-zero** if vcluster-API misconfig leaks secrets to wrong tenant |
| C5 persona's `secrets_inject` state | unchanged | unchanged | needs per-vcluster Secret synthesis; vcluster syncs Secrets from host by default — **flag for audit** |

### Shared-cluster failure modes (Alt 2 only)

- **Host EKS upgrade.** Forces every vcluster session to reset; a 10-dev team can't all be in-session during a host upgrade. → Schedule windowed upgrades; surface `cp9 status` warning. **Operational cost.**
- **Host node-pool exhaustion.** One dev's runaway pod (OOM, fork bomb) starves other tenants. → ResourceQuotas per vcluster + host-level LimitRanges. **Mitigation needed before adoption.**
- **vcluster control-plane crash.** The vcluster pod itself dies → tenant's API server is gone; tenant can't `cp9 deploy`. Recovery time: ~30-60 s. Compared to Alt 0 / Alt 1 where only one dev is affected, Alt 2 affects one dev per vcluster — but a host-EKS-wide problem affects all.
- **CRD conflicts.** All vclusters share the host EKS's CRDs by default. If one team needs `crossplane@v1.16` and another needs `v1.17`, they conflict. → vcluster supports per-vcluster CRD shadowing, but it's a configuration gate.

**Mitigation discipline.** Alt 2 introduces a **Platform Team responsibility** that Alt 0 / Alt 1 don't have. Whoever owns the shared-EKS host carries operational pages.

---

## Per-option deep dives

### Considered, not recommended — Lima `vz` + k3s

Lima `vz` + k3s was evaluated because it keeps a Kubernetes API while reducing Docker Desktop dependence. It should **not** be the recommendation for this issue.

Reasons:

- The repo and docs already assume k3d as the local substrate.
- k3d is also the cleaner remote-EC2 story: same cluster lifecycle, same image import/build assumptions, same debugging vocabulary.
- Adding Lima introduces a second local Kubernetes distribution before cp9 has proven the simpler host abstraction.
- Reviewer preference is explicit: all-in k3d for both local and EC2 escape hatch.

The "better local default" should therefore mean **better k3d defaults and diagnostics**, not a substrate swap: resource sizing checks, Docker runtime guidance, image/cache cleanup hints, and clear fallback to EC2+k3d when a laptop cannot carry the environment.

### Rejected alternative — Docker Swarm as the local host

Docker Swarm was proposed as a way to lower local laptop load. It should not be the cp9 control-plane substrate.

Reason: the Premium Reserved wedge depends on Kubernetes-native behavior, not just containers. The cp9 control plane needs CRD install, `kubectl`-driven overlays, Crossplane/cert-manager/Kong/Gateway API CRDs, tidb-operator interaction in the customer dataplane, and a kubeconfig-shaped developer interface. Swarm would save one lightweight Kubernetes API server, but it would force cp9 to either fake Kubernetes APIs or maintain a second deploy path for every controller-shaped dependency. That undermines the "same CP services, same APIs, same code path" invariant from 05.

The cheaper local optimization is to keep Kubernetes and improve the k3d path: Docker runtime guidance, resource checks, and a remote EC2+k3d escape hatch.

### Alt 0 — Local PC + k3d + tunnel *(current)*

**What stays.** Everything in the current 07. C1 tunnel; local k3d; customer dataplane EKS created by normal OpenAPI create/update paths through `cluster-ng-service` and `dataplane-manager`.

**What's good.**
- Sub-millisecond CLI latency.
- Full-local debug — `docker exec`, IDE attach, breakpoints all work natively.
- Total developer control. No platform-team dependency.
- $0 marginal cost (local PC out-of-scope per user).
- Works today; no migration risk.

**What's bad.**
- Local resource use: ~5-6 GB RAM + ~50 % CPU under load. Low-spec Macs (8 GB unified) struggle. Mid-spec Macs (16 GB) OK with discipline. High-spec (32 GB+) comfortable.
- Docker Desktop is the heaviest single line item (~1.5-2 GB just for the macOS VM); can be replaced with OrbStack / Colima as an opt-in laptop optimization.
- Tunnel adds latency to every AWS API call from the local k3d. Typically <50 ms; not iteration-loop-bottleneck.
- Network-quality-dependent: tunnel reliability varies on hotel/conference WiFi.

**When this option is right.** Daily dev work on a capable Mac with stable network. The default for most cp9 sessions today.

### Alt 1 — Remote EC2 + ssh + k3d-on-EC2

**What changes.**

- New `internal/host/ec2/` adapter (~600 LOC *estimate*, modeled on `internal/cluster/k3d.go`).
- New `cp9.yaml` / `cp9.local.yaml` field `host.mode: remote-ec2-k3d` + `host.ec2.{instance_type, ami, key_pair, root_volume_size_gib, ...}`.
- New lifecycle sub-state `host_provision` before `cluster_create`.
- `cp9 deploy <svc>` / Tilt custom builds: local `cp9` packages and transfers source/build inputs over SSH; the EC2 host runs Docker pulls/builds and `k3d image import`; `kubectl apply` targets the EC2-hosted k3d API via ssh-tunneled kubeconfig.
- `cp9 logs --trace-id`: same ssh tunnel.
- `cp9 destroy`: terminate EC2 (tagged `cp9-owner`/`cp9-session` so reaper can finish if stage 1 crashes) and remove remote session state.
- Tunnel (C1): runs on the EC2, not the Mac. Remote preflight validates EC2 root filesystem headroom, Docker daemon health, base-image pull, and trivial Docker build. Local preflight is limited to controller prerequisites such as config, AWS auth, SSH key material, and SSH reachability.

**What stays.** Persona, capability contracts, reaper logic, customer dataplane side effect, and the 07 Premium Reserved create/update E2E tests.

**What's good.**
- Local PC cost stays $0 (still out-of-scope per user). EC2 cost is ~$34/mo per active developer (m6i.xlarge × 176 hr).
- Beefier compute available — 16 GiB VM is way more than most laptops can spare.
- Network: EC2 in VPC A → AWS API calls are intra-AZ, ~5-10 ms vs. tunnel's ~50 ms.
- Same architecture as Alt 0 — `kubectl` doesn't know the difference.
- Single-developer blast radius preserved.
- Works offline-from-VPN ergonomically because the EC2 owns the VPC presence; the dev just needs ssh.

**What's bad.**
- ssh round-trip: 30-100 ms per CLI command. Compounds over a day.
- Source rsync friction on every `cp9 deploy`. Solvable with watch-and-sync but adds another moving part.
- EC2 tear-down discipline matters: forgetting to `cp9 destroy` leaves an EC2 running at $0.192/hr × 24 = $4.61/day = ~$140/mo. **Add idle-timeout auto-shutdown.**
- Debug ergonomics: ssh + port-forward works, but IDE remote-extension setup is one more thing per developer.
- Build cache lives on the EC2; new EC2 per session = no warm cache. Mitigation: persistent EBS snapshot reused across sessions for build-cache + image-cache.

**When this option is right.** Constrained-laptop developers; dev on a phone/iPad via SSH; unreliable-network developers; "I want a beefy box" days. Should be **opt-in**, not default.

### Alt 2 — Shared longrunning EKS + vcluster per session

**Rejected for this issue.** Even the safer variant, where only the cp9 control-plane host is shared and the customer dataplane remains real, adds a platform-owned host cluster, vcluster lifecycle, tenant RBAC, quota, secret-sync, CRD-version coordination, and whole-team failure modes. The per-developer cost only becomes attractive at larger team sizes, and the productivity tradeoff is worse for the daily edit/deploy/debug loop.

The more aggressive variant, where the customer dataplane is also shared or virtualized, is not compatible with the updated 07 wedge thesis because 07 validates normal OpenAPI create/update paths provisioning a real customer dataplane. That variant should stay out of scope unless the wedge thesis is intentionally rewritten.

### Alt 3 — Tiered k3d (recommended)

A `host.mode` field in `cp9.yaml` + `cp9.local.yaml` selects the host adapter:

```yaml
host:
  mode: local-k3d            # repo-level default

# cp9.local.yaml — per-developer override
host:
  mode: remote-ec2-k3d      # opt-in escape hatch
```

**Behavior.**

- `host.mode: local-k3d` → current Alt 0 behavior; default for daily dev.
- `host.mode: remote-ec2-k3d` → Alt 1. Provisions an EC2 and runs the same k3d-based cp9 cluster there; opt-in per developer/session.

**Architectural shape.** Both modes implement the `Host` interface above. The lifecycle state machine is mode-agnostic; mode-specific behavior is isolated in `internal/host/{local,ec2}/`.

**What's good.**
- Each developer picks the right k3d host for their context.
- Migration risk minimized: `local-k3d` stays the default while `remote-ec2-k3d` is additive.
- The team avoids the complexity of a second local Kubernetes distribution and a shared-cluster tenancy model.
- Failure-mode blast radius capped at one developer in local and remote-EC2 modes.

**What's bad.**
- Two host-adapter paths to maintain (`local-k3d`, `remote-ec2-k3d`).
- `cp9 doctor`, `cp9 logs`, and `cp9 destroy` need local/remote routing.
- More documentation surface.

**When this option is right.** When the team wants the current local productivity model plus a beefier single-developer host for sessions that exceed laptop capacity.

---

## Recommendation

**Adopt Alt 3 — tiered k3d host adapter — but ship in stages.**

### Stage 1 (now → 5/19 wedge): make local k3d explicit and healthier

- Add `host.mode: local-k3d` as the explicit default while preserving existing `cluster.k3d` compatibility.
- Extract the current local k3d logic into `internal/host/local/k3d` without changing behavior.
- Update `cp9 doctor` to check Docker/k3d memory, CPU, disk, image-cache size, and common Docker runtime pitfalls.
- Align destroy semantics with 05: decide whether Stage 1 deletes k3d or preserves it for warm re-entry, document it, and make code/docs consistent.
- Acceptance: existing local `cp9 up` behavior remains green; no persona/topology change; 07's create/update E2E command paths remain unchanged.

### Stage 2 (5/19 → 5/26): build the EC2+k3d host adapter (Alt 1)

- Extend `internal/host/` with `ec2/k3d` implementation.
- New `cp9.yaml host.mode:` field supports `remote-ec2-k3d`; default remains `local-k3d`.
- New lifecycle sub-state `host_provision`.
- Add `cp9 destroy` EC2 termination and remote state cleanup.
- Reaper extension: add EC2 instance, EBS volume, security group, and optional EIP cleanup with `cp9-owner`/`cp9-session` propagation.
- Idle-timeout auto-shutdown for the EC2 (cron + script, killed by cp9 on next session).

**Acceptance criteria for Stage 2:**

- `cp9 up` with `host.mode: remote-ec2-k3d` provisions an EC2, brings up k3d + 9 sub-apps, deploys the persona's REAL services, and reports `persona_ready` within **35 minutes cold / 5 minutes warm-restart** (slightly more than local).
- No `remote-ec2-k3d` service path depends on local Docker Desktop or local k3d. Generated source, live, and stub build scripts must contain only local input packaging/SSH orchestration on the controller side; Docker build/pull/load/import and `k3d image import` happen on the EC2 host. Tests must fail if local `docker build`, `docker save`, or local `k3d image import` re-enters the remote host mode.
- `host.ec2.root_volume_size_gib` is applied before EC2 launch as root EBS sizing, and remote preflight fails before workload deployment if disk/Docker/base-image checks are not healthy.
- Cara's day-in-the-life works through 07's normal create/update tests:
  - `go test ./test/e2e -run TestPremiumReserved_CreateTopology/Create56`
  - `go test ./test/e2e -run TestPremiumReserved_UpdateTopology/Update56To104`
- `cp9 destroy` exits 0 and the reaper post-scan confirms zero tagged resources, including the EC2.
- Iteration-loop measured: `cp9 deploy cluster-ng-service` after a code change ≤ 90 s end-to-end.
- ssh round-trip latency to EC2 from a typical dev location: median ≤ 80 ms.
- One developer per shift volunteers as the Alt 1 guinea pig for two weeks; produces a 1-page experience report.

### Not doing in this issue

- No Lima/k3s adapter.
- No shared EKS/vcluster adapter.
- No virtualized customer dataplane.

### Rollback path

Alt 0 is **always available**. Host mode is a `cp9.local.yaml` per-developer setting; reverting to `host.mode: local-k3d` is one YAML change. No data loss -- `cp9 destroy` cleans state regardless of mode.

If Alt 1 is shipped and proves a net negative (operational cost > productivity gain), `host.mode: remote-ec2-k3d` can be deprecated and the package archived without affecting Alt 0 users.

### Explicit "keep current" criterion

If after Stage 2 the volunteer-dev report shows median iteration loop > 120 s **and** the laptop-relief argument doesn't materialize, freeze Alt 1 development and keep `local-k3d` as the only daily-dev path. Alt 2 should remain forgotten unless a new, explicit shared-cluster mandate appears.

---

## Open questions

1. **Tunnel relevance in Alt 1.** If the EC2 is in VPC A directly, the tunnel may only be needed for specific AWS-resource-from-pod paths. Define the exact C1 contract for remote mode before implementation.
2. **Source-sync strategy for Alt 1.** rsync-on-deploy (lazy), watch-and-sync daemon, or pull-from-git on the EC2. Default recommendation: rsync-on-deploy + a `cp9 sync` escape hatch.
3. **EC2 size.** m6i.large is only a tight-cost option; m6i.xlarge is the safer default for k3d plus the 9 sub-apps and 4 REAL services. Recommend xlarge default; large as cost-saver opt-in after measurement.
4. **Destroy semantics.** 05 now says Stage 1 deletes k3d; current code preserves k3d. Resolve this before introducing remote EC2 termination semantics.

---

## Self-consistency checks

- ✅ Alt 0 keeps the wedge thesis (normal create/update provisions a real customer dataplane) intact.
- ✅ Alt 1 keeps the wedge thesis intact — only the cp9 control plane host moves.
- ✅ Alt 2 is rejected for this issue per review feedback, even though a control-plane-only version could preserve the wedge thesis.
- ⚠️ Virtualizing or sharing the customer dataplane **breaks** the current 07 wedge thesis.
- ✅ The C5 persona / topology loader (PR #70 merged) is host-agnostic; no rework needed.
- ✅ Local k3d and EC2+k3d preserve the same Kubernetes distribution and debugging model.
- ✅ Docker Swarm rejection keeps CRD/controller/Crossplane coverage intact.
- ✅ The C1 tunnel capability is mode-conditional — Alt 1 may reduce its scope; the current main branch has no merged tunnel package, so this should not block host abstraction work.
- ✅ The C4 reaper logic is host-agnostic; tag scoping works for local and EC2 modes, with EC2 resources added to the type table.
- ✅ Cost numbers cited with retrieval date (2026-04-30); estimates flagged.
- ✅ Per-option code-change matrix explicit.
- ✅ Security/isolation tradeoffs surfaced per cdx-55's review criteria.
- ✅ Recommendation is actionable: Stage 1 / 2 with acceptance criteria + rollback.

---

## Sources

All retrieved 2026-04-30.

- [AWS EC2 on-demand pricing](https://aws.amazon.com/ec2/pricing/on-demand/) — base reference.
- [DevZero m6i.large](https://www.devzero.io/instances/aws/m6i.large) — public EC2 table used for the specific us-west-2 `$0.096/hr` planning rate.
- [DevZero m6i.xlarge](https://www.devzero.io/instances/aws/m6i.xlarge) — public EC2 table used for the specific us-west-2 `$0.192/hr` planning rate.
- [DevZero m6i.2xlarge](https://www.devzero.io/instances/aws/m6i.2xlarge) — public EC2 table used for the specific us-west-2 `$0.384/hr` planning rate.
- [AWS EKS pricing](https://aws.amazon.com/eks/pricing/) — $0.10/hr standard support; $0.60/hr extended.
- [AWS VPC pricing](https://aws.amazon.com/vpc/pricing/) — NAT gateway $0.045/hr + $0.045/GB.
- [vCluster multi-tenancy guide](https://www.vcluster.com/guides/tenancy-models-with-vcluster) — shared-nodes / dedicated-nodes / private-nodes models.
- [vCluster resource-limit docs](https://www.vcluster.com/docs/vcluster/next/learn-how-to/control-plane/container/configure-memory-limits) — set per-vcluster CPU/memory limits; actual overhead must be measured.
- [VS Code Remote SSH docs](https://code.visualstudio.com/docs/remote/ssh).

Internal references:

- [07_wedge-premium-reserved.md](20260428-design-cp9-wedge-premium-reserved.md)
- [05_detail-design.md](20260425-design-cp9-detail.md)
- [tidbcloud/controlplane9 issue #79](https://github.com/tidbcloud/controlplane9/issues/79)
- [PR #70 (C5 persona loader, merged)](https://github.com/tidbcloud/controlplane9/pull/70)