# Detail Design: controlplane9

Codename: **controlplane9** | Repo: [tidbcloud/controlplane9](https://github.com/tidbcloud/controlplane9)

Supersedes: [20260331-design-cp9-v2-global-cp-only.md](20260331-design-cp9-v2-global-cp-only.md) (v2 global-CP-only design — deletable now) | Derives from: [20260417-design-cp9-expanded-scope.md](20260417-design-cp9-expanded-scope.md) | Status: APPROVED | Date: 2026-04-25

Related: [20260417-design-cp9-expanded-scope.md](20260417-design-cp9-expanded-scope.md) (architecture source of truth), [20260413-design-cp9-stakeholder-guide.md](20260413-design-cp9-stakeholder-guide.md), [20260417-design-cp9-landing-page.md](20260417-design-cp9-landing-page.md), current wedge [20260428-design-cp9-wedge-premium-reserved.md](20260428-design-cp9-wedge-premium-reserved.md), historical wedge [20260426-design-cp9-wedge-premium-autoscaler.md](20260426-design-cp9-wedge-premium-autoscaler.md).

Revision: 2026-04-25 — CEO review pass applied (CI identity path; owner-scoped reaper default; metadb single-provider fail-loud; `cp9 up` flock + orphan-state + activity-based TTL; load-bearing invariant tests; soak gate; risk-table additions). Eng review pass applied (Distribution subsection: `go install @<release-tag>`; M-series only; reaper type-table location; cross-repo compatibility baseline; C4 multi-region concurrency bound; crossplane bump cadence; 5 state-machine invariant tests).

Revision: 2026-04-27 — 06 eng review surfacing pass: `cp9 logs` extends to per-session dataplane EKS; CP layer collapse documented in Migration; C3 audit gains third precondition (topology-agnostic).

Revision: 2026-04-30 — removed the separate dataplane persona capability; customer dataplane provisioning is a normal CP service side effect of OpenAPI create/update paths, not a cp9 bootstrap concern. Premium Reserved replaces Premium Autoscaler as the current wedge.

Revision: 2026-05-09 — issue #106 env-creation refactor supersedes the split `topologies/<persona>.yaml` + repo `cp9.yaml` + local `cp9.local.yaml` config model and the special `crossplane_install` / staging `crd_install` lifecycle states. See [20260506-design-cp9-env-creation-refactor.md](20260506-design-cp9-env-creation-refactor.md) for the proposed unified `cp9.yaml`, `source` / `live` / `stub` service taxonomy, persona extension contract (`cp9.<persona>.example.yaml` + persona Terraform `.tfvars` + persona app-delivery env), Terraform substrate contract, and Tilt-centered lifecycle v3. For the 07 narrow wedge, network/tunnel setup is optional and local IAM setup is required.

> Historical command-surface note (2026-06-01): this design preserves architecture history and predates the final command cleanup train. Its command examples may mention removed ideas such as `sync`, `onboard`, `config`, `persona`, `crd`, `migrate`, `test`, `cost`, `reaper`, `stub-gen`, `overlay`, `verify`, `feature-cycle`, `topology`, or `up --persona` / `up --discover` / `up --profile` / `up --remote`. Current runnable surfaces are `setup`, `doctor`, `up`, `status`, `logs`, `deploy`, `destroy`, direct `pause`, and `db dump-init`.

---

## Purpose

Implementation contract for controlplane9, at **architectural depth**: flows, contracts, invariants, error modes, operational patterns. Does not prescribe package paths, Go type signatures, or code bodies — the implementing agent makes those calls per repo conventions ([CLAUDE.md](https://github.com/tidbcloud/controlplane9/blob/main/CLAUDE.md)). Content carries forward from 01 as needed, restated in terms of the new architecture; no `see 01` references.

---

## Core mental model — cp9 is an environment, not a translator

cp9 runs the **same CP services, same APIs, same code path** as production — on the selected k3d host, with dev cloud credentials. When the developer calls the OpenAPI Create TiDB Cluster path (which flows into downstream `CreateCluster` / `CreateTiDBCluster` calls), the production code path executes unchanged: allocate metadb rows + CRs, apply crossplane/provider resources via service-owned controllers or cloud SDKs, and bring up a real TiDB cluster in the dev account. Runtime is fully separated from prod and staging.

- cp9 does **not** intercept services, overlay them with "dev redirections," or poll for hooks. Services run with their normal config surface; cp9 populates that surface correctly and stands up the surrounding environment. In the issue #106 model, that environment boundary is selected k3d host + local IAM proof + Terraform support infra outputs + cp9-owned Argo envs + generated Tilt inputs + reaper.
- Every cloud resource the CP code path creates is tagged. Destroy Stage 1 is local/session cleanup (selected host + metadb/session artifacts). Destroy Stage 2 is a tag-scoped cloud reaper that authoritatively verifies nothing survived. cp9 does not call services' delete APIs.
- Any service that cannot run in cp9 is a **service-side integration gap**, not a cp9 gap. The per-service audit catches those.

---

## Design principles

1. **Real over simulated below L1.** Crossplane, cluster-lifecycle services, and dataplane provisioning run the real production code path unchanged. Stubs exist only for L1 deps off the feature path.
2. **cp9 is an environment.** It owns the selected k3d host, IAM/credential proof, Terraform invocation, session state, generated context/Tilt inputs, the reaper, and tags. app-delivery owns live app envs; auto-deploy owns Terraform source; Tilt owns the runtime dev loop. cp9 does **not** own behavior inside services.
3. **One universal interface.** The CLI is the only surface. Every command returns parseable JSON via `{sys.output.format=json}` and self-describes via `--help`.
4. **Per-session isolation by construction.** Every cloud resource carries `cp9-owner` + `cp9-session` tags (or cloud-equivalent labels). No shared pool.
5. **Force-purge over best-effort teardown.** Stage 1 is local cleanup; Stage 2 is the tag-scoped cloud reaper and is authoritative.
6. **Declarative persona extension, not discovered topology.** Which services are `source`, `live`, or `stub` is declared in `cp9.<persona>.example.yaml` and backed by the persona-specific Terraform `.tfvars` plus app-delivery env, not inferred.
7. **Cross-repo only.** Single-repo concerns (build, lint, proto gen) live in each repo's Makefile; cp9 orchestrates across repos and clouds.
8. **Every outbound call is verbose-wrappable.** Cloud SDK calls, crossplane CR applies, RPCs, `exec.Command` all go through `internal/verbose`.
9. **Namespace-prefix isolation.** Every external resource name begins with the session `name`; additional safety net beyond tags.

### Agent-native CLI properties

The CLI is the delivery interface for both humans and agents. These properties are invariants:

| Property | Architectural requirement |
|---|---|
| Structured + self-describing | Commands on a ticat tree; `--help`, `cp9 cmds.tree`, structured JSON output. An agent reading the help surface can drive cp9 without this doc. |
| Lightweight + universal | Single static Go binary; 2026-04 runtime deps: Docker, kubectl, k3d, chosen tunnel binary. Issue #106 adds Tilt for the dev loop and makes the tunnel binary optional for 07 M1. **M-series Mac only** (success-criteria fleet); the binary itself builds for whatever host runs `go install`, but service builds are arm64-only by design. |
| Deterministic + re-entrant | Same inputs → same transitions. Idempotent destroy and reaper. State in `.cp9/<name>/state.json`. |
| Cross-repo focused | No per-repo Makefile duplication. cp9 owns cross-repo orchestration only. |
| Plugin-packaged | Ships with an agent plugin (skills/agents/hooks) — see [Agent-driven delivery](#agent-driven-delivery-o2). |

---

## Architecture

See [03](20260417-design-cp9-expanded-scope.md#architecture) for Mac → k3d → AWS VPC A → multi-cloud. Inside cp9:

```
+------------------------- cp9 CLI (ticat) --------------------------+
|                                                                   |
|  cp9 up | destroy | status | logs | reaper | tunnel | persona     |
|     |         |                                                   |
|     v         v                                                   |
|  Lifecycle state machine  |  Destroy orchestrator                 |
|         |                        (two-stage guarantee)            |
|         v                                                         |
|  +-------------------------------------------------------------+  |
|  |                   Capability surfaces                       |  |
|  |  C1 Tunnel + AWS creds    C2 Crossplane + per-cloud creds   |  |
|  |  C3 CL-service audit      C4 Tag-scoped reaper              |  |
|  |  C5 Persona / topology template loader                      |  |
|  +-------------------------------------------------------------+  |
|                                                                   |
|  Supporting mechanisms:                                           |
|    k3d lifecycle | CRD install (staging) | metadb (Zero or       |
|    fallback) | Kustomize overlays | secrets | classify/deploy    |
|    | L1 stub generation | IAM + AWS runtime | tag enforcement    |
|    | namespace-prefix | verbose wrap | error categories          |
+-------------------------------------------------------------------+
```

C1–C5 are independently testable — runtime sequence is imposed by the `cp9 up` [lifecycle](#cp9-up-lifecycle), not by capability coupling. They share only the [Tag invariant](#tag-invariant) and the cp9 config contract. Supporting mechanisms are cross-cutting ([Supporting mechanisms](#supporting-mechanisms)).

> 2026-05-10 consistency note: the diagram above is the approved 2026-04 baseline. Under issue #106, C4/destroy/reaper, owner/session identity, tag invariants, and selected-host ownership remain authoritative. Env creation moves to the v3 flow: local IAM + Terraform substrate + cp9-owned Argo env + generated Tiltfile. Tunnel/network is optional for 07 M1, Crossplane/provider are `live` services, and CRDs install through the Argo-style manifest contract.

---

## Session and ownership identity

Two strings identify every session. They are the tag values the reaper scans on, the namespace prefix, and the simulated-tenant identifier in CP metadb.

### Session name

- **Field**: `name` in the active session config. In the issue #106 model this is unified `cp9.yaml` copied from `cp9.<persona>.example.yaml`; `cp9.local.yaml` was the 2026-04 baseline. **Required.** Developer sets it before `cp9 up`; no default.
- **Format**: DNS-compatible (lowercase alphanumeric + hyphens), 3–32 chars. Unique per developer. Enforced at config-load.
- **One value, many uses**: k3d cluster name, namespace prefix, `cp9-session` tag, simulated-tenant identifier inside CP metadb.
- **Lifetime**: one `cp9 up` → `cp9 destroy`. Re-entering `cp9 up` reuses the existing `name`. Changing mid-session is rejected (developer must `cp9 destroy` first).
- **Conflict (orphan k3d)**: on `cp9 up`, if no state file exists but a k3d cluster of that name exists, cp9 errors and asks for a new `name`.
- **Conflict (orphan state)**: if the state file exists but the k3d cluster is absent (Docker restart, manual `k3d cluster delete`), `cp9 up` resumes at `cluster_create` — state survives Docker, cluster does not.
- **Concurrency**: `cp9 up` takes a `flock` on `.cp9/<name>/lock` at entry. A second concurrent invocation fails fast with the lock-holder's PID and a stale-lock recovery hint. Prevents racing writes to `state.json`.
- **Activity timestamp**: every `cp9 <any-subcommand>` on the session updates `.cp9/<name>/last-active`. TTL enforcement (reaper + `cp9.yaml clouds.*.ttl`) computes age from `last-active`, not session creation, so idle-but-recent sessions are safe.

### Owner identity

- **Value (human developer path, default)**: developer's AWS SSO email, captured at `cp9 up` from the active SSO session via STS identity resolution (session tag on the `cp9-developer` AssumeRole call).
- **Value (CI / automation path)**: when SSO is absent, cp9 reads `CP9_OWNER` env var and validates it against `cp9.yaml ci.owner_allowlist: { <CI-runner-IAM-principal-ARN>: [<permitted-owner-values>] }`. The caller's resolved IAM principal (STS caller identity) must map to the asserted owner in the allowlist. Forging a teammate's owner requires compromising both the CI runner's IAM and the allowlist.
- **Output**: stored in session state; emitted as `cp9-owner` tag on every created cloud resource.
- **Normalization**: lowercased; `@` and `.` kept in AWS tags. Other clouds see [Tag invariant](#tag-invariant) for value mapping.
- **Fail-loud**: cp9 refuses to run if neither path resolves (config category, remediation: `aws sso login` OR set `CP9_OWNER` + add allowlist entry). Never inferred from OS username or a plain `cp9.yaml` field.
- **Reaper scoping**: Stage 2 of `cp9 destroy` scopes to the invoker's own resolved `cp9-owner` (intersect with any session filter). Cross-owner sweeps are an admin/operator concern outside the product CLI and require credentials carrying the boundary permission. Full contract in [C4](#c4--force-purge-reaper-tag-scoped-multi-cloud).

---

## Capability contracts

Each C below: **Purpose / Contract / Flow / Invariants / Error modes / Open questions**. Error-mode arrow `transient → permanent` means: retried within the state's budget (`cp9.yaml retry`); on exhaustion the category escalates to the RHS.

### C1 — Local IAM and optional tunnel wiring

> 2026-05-08 update: for the 07 narrow wedge and issue #106, the load-bearing C1 requirement is local IAM/owner identity soundness. Tunnel/network remains an optional capability for future paths that prove private network reachability is required.

**Purpose.** k3d pods have a provable developer-owned AWS identity and, when a persona requires private reachability, can also reach AWS Control Plane VPC A through a validated tunnel.

**Contract.**

- In: unified `cp9.yaml` AWS profile/role/account allowlist; optional VPC A coordinates (account, region, mechanism, `expected_egress_eip`) only when network is enabled.
- Out (cluster): every pod can use the session AWS identity; optional tunnel paths additionally validate VPC reachability and observed egress.
- Out (cloud): none directly (validates only).
- Out (state): IAM readiness, `cp9-owner` captured from SSO, and optional tunnel health/mechanism/observed EIP when network is enabled.

**Flow.**

```
1. Preflight: AssumeRole cp9-developer; capture SSO email as cp9-owner;
   validate account allowlist and session tags.
2. Validate the selected k3d host can use the session AWS identity.
3. If network/tunnel is enabled for the persona: validate tunnel binary
   (Tailscale / SSM plugin / …), bring tunnel up, and smoke test from a k3d
   pod: DNS resolves; STS GetCallerIdentity; observed egress IP equals
   cp9.yaml tunnel.vpc_a.expected_egress_eip.
4. Record IAM readiness, owner, and optional tunnel state in session state.
```

**Invariants.** IAM is verified before any service can create cloud resources. If tunnel/network is enabled, it is fully up and smoke-tested, or reported down — no "partially up." Observed EIP must match registered (`permanent` error if not). No capability after C1 touches clouds without reading IAM readiness and any required network state.

**Error modes.**

| Shape | Category | Handling |
|---|---|---|
| AssumeRole denied / SSO expired | config | Surface role ARN + `aws sso login` hint |
| `CP9_OWNER` set but not in allowlist for caller's principal | config | Surface caller principal ARN + allowlist file path |
| Tunnel binary missing when network is enabled | config | Exit non-zero with install link |
| Smoke test fails when network is enabled (tunnel down OR k3d-to-host network issue) | transient → permanent | Retry within C1 budget, then fail; message disambiguates tunnel vs. host-network |
| Observed EIP ≠ registered when network is enabled | permanent | Abort; operational fix |

**Open questions.** Tunnel mechanism default (Tailscale / SSM / VPN Client / private-link) — decide after a per-mechanism latency spike. **Only one mechanism ships initially**; others are future adapters under the same field. EIP + NAT gateway are pre-provisioned by infra; cp9 validates, does not create.

### C2 — Crossplane REAL + per-cloud dev credentials

> 2026-05-07 update: issue #106 changes the runtime model. Crossplane core and its providers are no longer installed by a special cp9 capability state. They are `live` app-delivery services rendered/applied with the rest of the service graph; Terraform supplies only supporting infra and credential inputs. The operational intent of this section remains, but the lifecycle owner moves from `crossplane_install` to app-delivery manifest rendering plus Terraform outputs.

**Purpose.** Crossplane runs as a real controller in k3d with real per-cloud credentials; CP code's crossplane CRs land as real cloud resources in the correct dev accounts.

**Contract.**

- In: `cp9.yaml clouds:` (subset of `aws | gcp | azure | alicloud`); per-cloud credential reference (profile / SA key / WIF / MI / RAM).
- Out (cluster): crossplane in `ns: crossplane-system`; one provider per in-scope cloud, healthy; default-tag hook configured where provider supports it.
- Out (state): provider health per cloud.

**Flow.**

```
1. Install crossplane (helm or manifest; pinned version).
2. Per in-scope cloud: install provider; create/refresh ProviderConfig from
   credential; configure default-tag injection (cp9-owner + cp9-session);
   wait for provider.healthy.
3. Sanity CR per provider: a cheap read-only CR (AWS STS caller identity;
   GCP project.get; Azure RG read; Alicloud account.get). Fail here, not
   at usage time.
4. Record per-cloud provider health.
```

**Invariants.** Every in-scope provider healthy before anything downstream runs. Where a provider can't inject default tags, **the code path applying the CR tags** (C3 audit item). cp9 refuses to run if the credential resolves outside `cp9.yaml clouds.*.account_allowlist` — the sole runtime account-allowlist enforcement point.

**Error modes.**

| Shape | Category | Handling |
|---|---|---|
| Provider CRD install fails | permanent | Abort with k8s version-compat note |
| ProviderConfig secret rejected | config | Surface credential path + remediation |
| Sanity CR never healthy | transient → permanent | Retry C2 budget; fail with CR events |
| Credential outside allowlist | permanent | Abort immediately |

**Open questions.** GCP: WIF from AWS STS vs. static SA key? Default proposed WIF, static-key fallback behind flag. Non-AWS default-tag injection: native / webhook / CR-caller per provider — audit per provider version. Pin provider versions to app-delivery to avoid dev/prod drift.

### C3 — Cluster-lifecycle service preconditions

**Purpose.** Cluster-lifecycle services (tidb-mgmt-service, cluster-service, cluster-ng-service, dataplane-manager, per-cloud variants) run in cp9 **unchanged from prod**. C3 is not a translator and has **no runtime flow** — it is an audit artifact per service before onboarding.

**Contract.**

- In: persona's list of REAL cluster-lifecycle services; cp9's normal config/secret surface.
- Out (cluster): services deploy via the normal `deploy` state; no cp9-specific overlay.
- Out (cloud): none directly — cloud resources appear as side effects of the developer calling normal OpenAPI create/update paths. Every such resource carries the tags because services are audit-verified for tag coverage.
- Out (state): none.

**Preconditions (per-service audit).** A service is cp9-compatible only if both hold. Audit is the capability's deliverable, owned per product line. **Blocking Phase 2 prerequisite** per team; remediation scope is unknown until the audit runs.

| Precondition | Audit question | Remediation |
|---|---|---|
| Externally configurable | Are dev account / region / VPC / infra-stack purely env/ConfigMap — or hardcoded to prod? | Patch the service to externalize. |
| Tag-complete | Are `cp9-owner` + `cp9-session` applied to every resource the service creates (directly via cloud SDK + indirectly via crossplane CRs)? | Patch creation call-sites; this is usually the largest remediation. |
| Topology-agnostic | Does the service rely on regional-vs-global VPC properties (private network reach, regional IAM trust, regional service discovery)? cp9 collapses all CP layers into one k3d cluster. | Externalize the assumption (config / overlay) or make the service work over the C1 tunnel. |

**Invariants.** Services run unchanged — no cp9-specific code, no dev-redirection overlay, no cp9 teardown hook. The simulated tenant is internal to the service's metadb and irrelevant to C4; only cloud-resource tags matter for cleanup.

**Error mode.** Service fails audit → `permanent` (until patched). Block the persona from Phase-2 adoption; remediation documented in the audit artifact.

**Open questions.** Each service's existing default-tag config surface (many have "default tags applied to all created resources" for prod-side tenant tagging) — the audit records the exact field per service. Which services currently hardcode any target coordinate — audit surfaces these as remediation PRs.

### C4 — Force-purge reaper (tag-scoped, multi-cloud)

**Purpose.** Authoritative Stage 2 of `cp9 destroy`: scan every in-scope cloud for resources tagged `cp9-owner` + `cp9-session`, delete in dependency-safe order, exit 0 only when post-scan is empty.

**Contract.**

- In: the active or explicitly selected session state. **Owner-scoping is the default** — the destroy reaper uses the invoker's resolved `cp9-owner` and the session's persisted `cp9-session` scope. Cross-owner or fleet-wide sweeps are not a user CLI surface; they require explicit admin/operator tooling with boundary permissions. Scope comes from the resolved `cp9.yaml` cloud configuration and persisted AWS session state.
- Out: deletion of all discovered tagged resources; final scan result (pass: zero; fail: surviving list with reason).
- State: `destroyed` on success; surviving-list on failure so a re-run picks up where it stopped.

**Flow.**

```
1. Discovery per cloud per region (concurrency: parallel across regions
   within a cloud, bounded to 4 in flight; sequential across clouds to
   keep blast radius readable in logs):
     AWS: Resource Groups Tagging API GetResources.
     GCP: Asset Inventory + labels filter.
     Azure: Resource Graph + tags filter.
     Alicloud: Tagged-Resources API per region.
2. Per-cloud dependency graph over discovered resources using type rules;
   unknown types default to "delete last," retried up to K post-purge scans
   before being warned.
3. Delete in dependency-safe order with retries on throttling.
4. Post-purge scan: empty (tagged + unknowns after K retries) → exit 0;
   non-empty → exit non-zero with surviving list.
5. Idempotent: re-run picks up from the surviving list of the prior run.
```

**Invariants.**

- **Authoritative.** C4 alone returns "destroyed" status. Stage 1 success is never sufficient.
- **Tag-discovery-or-bust.** Untagged creation paths are invisible to C4. This is why the tag invariant is mandatory across all capabilities.
- **Credential-scoped.** Same per-cloud credentials as everywhere else; `cp9-developer` permission boundary + analogs keep scope in dev accounts. No runtime allowlist beyond this.
- **Owner-scoping by default.** Default scope intersects discovered tags with the invoker's own `cp9-owner`. Cross-owner sweeps require `--all-owners` + a caller credential carrying admin boundary permission. Prevents cross-developer collision on accidentally-equal `name` values.
- **Never deletes untagged resources** "just in case."
- **Empty-scope discrimination.** `cp9 destroy` reaching Stage 2 with state claiming resources but an empty scan → exit non-zero (tagging gap). Operator-only sweeps without session state may legitimately find no matches, but that path is not a product CLI command.

**Error modes.**

| Shape | Category | Handling |
|---|---|---|
| Tagging API throttled or pagination token error | transient | Exponential backoff within C4 budget |
| External protection prevents delete | resource | Record, exit non-zero, surface remediation |
| Unknown type has dependents | permanent | Retry K times; warn + abort |
| Cloud-equivalent tag schema differs | config | Normalize during discovery per [Tag invariant](#tag-invariant) |

**Phase-2 prerequisite.** Per-cloud dependency table lives as a Go data file in the cp9 repo (one file per cloud — e.g., `internal/reaper/types/aws.go`), versioned with the binary. Seeded from prod `createcluster` output; new resource types ship in cp9 release tags. AWS first — without it, the Phase-2 wedge has no authoritative cleanup. Unit tests assert every entry has a defined parent type or is marked leaf.

### C5 — Persona / topology template loader

> 2026-05-09 update: issue #106 supersedes this standalone loader. The current persona contract is three persona-keyed artifacts: `cp9.<persona>.example.yaml`, persona Terraform `.tfvars`, and persona app-delivery env. The unified `cp9.yaml` service graph replaces runtime `--persona` loading and `topologies/<persona>.yaml`.

**Historical 2026-04 purpose.** Drive `cp9 up --persona <name>` from a declarative template listing REAL / STUB services and required capabilities. Replaces 01's static-analysis `cp9 topology`.

**Contract.**

- In: `topologies/<persona>.yaml` loaded by name.
- Out: per-run plan — service-list with refs/branches, capability set, cloud scope.

**Flow.** Load template by name (fail clearly if not found — list available); merge `cp9.local.yaml` overrides; validate (every REAL has a source; every STUB has a reachable proto/swagger; every required capability's prerequisites met); hand the plan to the lifecycle.

**Invariants.** Fail loud on missing services (never silent boot with a hole). Templates are source-of-truth; auto-discovery is out of scope. CODEOWNERS per `topologies/` subdirectory — product-line teams own their templates.

**Error modes.**

| Shape | Category | Handling |
|---|---|---|
| Persona not found | config | List available with fuzzy-match |
| References unknown capability | config | Reject with named offender |
| Service has no source | config | Surface missing `cp9.local.yaml sources:` entry |
| Stub has no API definition | config | Permissive stub + `cp9 doctor` warning |
| Partial validation | permanent | Refuse; never partial-boot |

**Open questions.** Override grammar (`force-real`, `force-stub`, extra env) — every override surfaced in `cp9 doctor`. Nightly CI check: no persona references a deleted service.

---

## Cross-capability invariants

### Tag invariant

Every cloud resource created on behalf of a session — by cp9, crossplane, cluster-lifecycle services during feature execution, any cloud-SDK call in any REAL service — **must** carry:

- `cp9-owner` = developer's AWS SSO email
- `cp9-session` = session `name`

**Cloud-equivalent labels.**

| Cloud | Mechanism | Value normalization |
|---|---|---|
| AWS | Resource tags via Resource Groups Tagging API | None; `@` and `.` allowed |
| GCP | Labels per resource | Lowercase-alphanumeric-hyphen. `alice@pingcap.com` → `alice-pingcap-com`. `cp9-session` already DNS-compat. |
| Azure | Resource tags (case-insensitive keys) | Lowercased |
| Alicloud | Tag service per product | Not all products tag; exceptions documented per-product in C4 type-table |

**Audit procedure** (the implementing agent executes for each Phase-2 team):

1. Per cluster-lifecycle service per target cloud, enumerate creation call-sites (crossplane CRs + direct cloud-SDK calls).
2. Confirm each call-site either sets tags explicitly or uses a provider default-tag hook. Otherwise file remediation against the owning service repo.
3. Confirm each resource type is discoverable by the cloud's tag-search API. Document asymmetric-tagging exceptions in the per-cloud C4 type-table.
4. Ship audit as a per-service checklist — **blocking acceptance artifact** for Phase-2.

**CI enforcement.** Each Phase-2 repo gets a CI gate rejecting PRs that introduce untagged new cloud-resource call-sites. Implementation varies per repo; the commitment is that tag coverage doesn't silently regress.

### Multi-cloud adapter contract

C2 and C4 need per-cloud behavior; C3 does not (per-cloud concerns live inside the cluster-lifecycle services). Each cloud adapter provides:

| Capability | Adapter supplies |
|---|---|
| C2 | Provider name/version (pinned); credential shape; sanity-CR type; default-tag hook semantics |
| C4 | Tag-search API + pagination; resource-type dependency table; account-identity validator; cloud-equivalent-label mapping |

Adapters are independently testable; AWS ships first (Phase-2 wedge is AWS-only per [03 Rollout](20260417-design-cp9-expanded-scope.md#rollout)).

---

## `cp9 up` lifecycle

State machine; re-entrant. State in `.cp9/<name>/state.json`.

> 2026-05-08 update: the table below is the approved 2026-04 lifecycle baseline. Issue #106 replaces it with lifecycle v3: one schema-validated `cp9.yaml`, local IAM validation, Terraform substrate states, cp9-managed local k3d, context materialization, generated Tiltfile, and Tilt-owned build/apply/readiness. In that model `crossplane_install`, staging `crd_install`, standalone `metadb_plan`, and cp9-owned deploy/readiness states are removed or folded into substrate/context/Tilt handoff.

| State | Kind | Owner / what it does |
|---|---|---|
| `preflight` | once | Acquire `flock` on `.cp9/<name>/lock`; load configs; validate `name`; capture `cp9-owner` from SSO (or CI path per [Owner identity](#owner-identity)); check Docker / kubectl / k3d / tunnel binary; conflict-detect on stale k3d cluster; conflict-detect on orphan state (resume at `cluster_create`); low-RAM warning |
| `persona_load` | once | C5 loads template, merges `cp9.local.yaml` overrides, validates, produces per-run plan |
| `cluster_create` | once | `k3d cluster create <name>` (arm64-native on Apple Silicon; loadbalancer port exposed) |
| `tunnel_up` | once | C1 — see C1 flow |
| `crossplane_install` | once | C2 install (crossplane in `ns: crossplane-system`) |
| `cred_wire` | once | C2 per-cloud provider + credential + sanity CR |
| `cp_cloud_ready` | gate | C1 + C2 smoke tests; blocks downstream |
| `persona_sources_ready` | once | Per REAL service: ensure source at specified `ref` (checkout or fetch-and-checkout) |
| `crd_install` | once | Dump CRDs from staging by API group, cache (24h TTL), apply to k3d |
| `secrets_inject` | once | Opaque Secret per service from `.env.dev` / inline values |
| `metadb_plan` | once | Provision or locate Zero instance `<name>-dev` (provider from `cp9.yaml metadb.provider`; fail-loud if unhealthy — no cascade) |
| `build_real` | parallel | Per REAL service: `docker build --platform=linux/arm64` + `k3d image import` |
| `gen_stubs` | parallel | Per STUB: generate proto/swagger server, cache by content hash, import |
| `config_gen` | once | Kustomize overlays from staging ArgoCD configs, patched for this session (DB DSN, k3d DNS, default tags, overrides) |
| `metadb_seed` | once | Dump schema (+ selected data) from staging metadb into per-service databases in Zero |
| `deploy` | parallel | Apply overlays per service to k3d — REAL and STUB alike; cluster-lifecycle services have no special handling |
| `readiness` | parallel | REAL: `readiness` probe (default: real business API, expect configured status); STUB: `/healthz` |
| `persona_ready` | gate | Every REAL ready; every STUB healthy; every required capability green |

**State-machine semantics.** `SUCCESS → next`; `TRANSIENT_FAILURE → retry` within `cp9.yaml retry.<state>` budget; `PERMANENT_FAILURE → abort` (re-run resumes at the failed state); `PARTIAL → record per-item progress`, re-run resumes remaining items; `SIGINT/SIGTERM → interrupted`, next run retries.

**`cp9.yaml retry:`.** Architectural shape: `retry.<state>` (per-state int, default 3), `retry.total` (session cap, default 5), `retry.on_budget_exceeded` (`abort` default, or `draft-pr` to open a draft PR with state + correlated logs).

**Metadb vs. customer dataplane — independent axes.** In the 2026-04 baseline, Zero is the metadb replacement for AWS RDS: every REAL CP service with state gets its own database on one shared Zero instance for this session. In the issue #106 model, Terraform may create metadb support infra and cp9 still owns app-level `metadb_seed`. Either way, metadb setup operates only on CP state. The customer dataplane is not provisioned by `cp9 up` and is not a persona capability. It appears only when the developer drives the production OpenAPI create/update paths: cluster-lifecycle services render and apply CRs through Crossplane/provider paths and `dataplane-manager` as they do in prod.

---

## `cp9 destroy` — two-stage guarantee

**Stage 1 — Local/session cleanup.** Delete the selected k3d host (kills every pod + host-local state in one shot); clean metadb/session artifacts using the active provider/substrate state; remove `.cp9/<name>/`. cp9 does **not** call services' delete APIs — pods die with the cluster; cloud resources carry tags for Stage 2 to find. Stage 1 failures are logged, not fatal.

**Stage 2 — Force-purge (authoritative).** C4 runs against every cloud in scope, scoped by `cp9-owner` + `cp9-session`. Dependency-safe deletion. Re-scans. Exits 0 only on empty.

**Exit contract.** Return 0 iff post-Stage-2 scan finds zero tagged resources across every cloud and every in-scope region.

**Partial-failure example.**

```
cp9 destroy
  stage 1: selected k3d host + metadb/session artifacts deleted
  stage 2: deleted 47; 3 survived
  POST-SCAN FAIL:
    aws us-west-2: ebs vol-abc123 (attached to terminated instance)
    aws us-west-2: ebs vol-def456 (attached to terminated instance)
    gcp my-dev-project: compute/disks/pd-ghi789
  exit 2
  hint: cp9 destroy --session <name> to re-run; common "storage outlives
        cluster" pattern — safe to re-run after the cluster de-attaches.
```

**`--force`.** Re-runs Stage 1 even if session state marks it done — recovers from interrupted first pass that left stray local state. Stage 2 is always re-runnable.

**Idempotency.** Every re-run picks up where the last stopped.

**Force-purge ownership.** C4 now runs through `cp9 destroy`; standalone `cp9 reaper` is not a user command. Recovery from sessions whose destroy was never invoked should re-run `cp9 destroy` with the saved session state. Cross-owner or TTL sweeps, if needed, belong in explicit admin/operator tooling with boundary permissions.

---

## CLI surface

ticat conventions: `key=value` args, `:` chaining, `{env.key=value}` env, JSON via `{sys.output.format=json}`.

> 2026-05-08 update: issue #106 keeps the CLI/JSON/ticat principles but changes the primary input: `cp9 up` reads the unified `cp9.yaml`; persona selection happens by copying a committed `cp9.<persona>.example.yaml`, not by passing `persona=<name>`.

**First-run order**: `aws sso login --profile cp9` → `cp9 setup` → `cp9 doctor` → `cp9 up`.

**Core.**

| Command | Purpose | Key args |
|---|---|---|
| `cp9 up` | Stand up environment for the unified `cp9.yaml` service graph (re-entrant) | `cp9.yaml` copied from `cp9.<persona>.example.yaml`; `target.cloud/region`; `host.mode` |
| `cp9 destroy` | Two-stage teardown | `session=<name>` (default: current); `force=true` |
| `cp9 status` | Per-capability health + `persona_ready` gate | `--json` |
| `cp9 doctor` | Pre-flight + capability checks + tag-audit hint | — |
| `cp9 logs` | Cross-service + cross-cluster log aggregation (local k3d AND per-session dataplane EKS, via the kubeconfig wired by `cluster-ng-service` during the normal create path) | `trace-id=<id>`; `service=<svc>`; `since=5m`; `level=error` |

**Capabilities.**

| Command | Purpose |
|---|---|
| `cp9 tunnel up / down / status` | C1 independent of `cp9 up` |
| `cp9 persona list / show / validate` | Historical C5; issue #106 follow-up may replace this with checks for `cp9.<persona>.example.yaml` + persona Terraform `.tfvars` + persona app-delivery env |

**Cross-repo.**

| Command | Purpose |
|---|---|
| `cp9 deploy <svc>` | Single-service rebuild + redeploy |
| `cp9 sync` | Refresh template index + stub cache (no topology.json; that file no longer exists) |
| `cp9 migrate` | Staging → metadb schema + selected seed; `--reset` drop+reapply; `--seed` data dump |
| `cp9 setup` | Validate `cp9-developer` + each in-scope cloud credential |
| `cp9 stub-gen <svc>` | Regenerate one service's stub |
| `cp9 onboard <repo>` | Scaffold AGENTS.md + `cp9.yaml` stub. TOPOLOGY.md is not generated. |

**Removed from 01** (implementing agent deletes commands + packages): `cp9 topology` (all subcommands), `cp9 up --discover`.

**Global flags**: `--json` / `{sys.output.format=json}`; `--help`; `{cp9.verbose=true}` or `CP9_VERBOSE=1` (MANDATORY per CLAUDE.md for outbound-call capture).

---

## Configuration

> 2026-05-09 consistency note: the examples below are historical 2026-04 config shapes. Issue #106's active model is one gitignored, schema-validated `cp9.yaml` copied from a committed `cp9.<persona>.example.yaml`. That example points at the matching persona Terraform `.tfvars` in auto-deploy and persona Argo/kustomize env in app-delivery.

### cp9.yaml (repo-level, checked in)

```yaml
apiVersion: cp9/v1            # required; reject unknown versions
name: cp9-${USER}             # default session name if cp9.local.yaml omits
cluster:
  k3d:
    agents: 2                 # default 2
    memory: "8G"              # default 8G; warn below RAM required

clouds:                       # AWS required; others optional per persona needs
  aws:
    profile: cp9              # ~/.aws/config profile → cp9-developer role
    regions: [us-west-2]
    account_allowlist: [...]  # dev account IDs; enforced by C2
    ttl: 72h                  # session TTL; reaper-enforced
  gcp: { ... }                # project_allowlist
  azure: { ... }              # subscription_allowlist
  alicloud: { ... }           # account_allowlist

tunnel:
  mechanism: tailscale        # tailscale | ssm | vpn-client | private-link
  vpc_a:
    account: "<dev acct id>"
    region: us-west-2
    expected_egress_eip: "..."

reaper:
  clouds: [aws, gcp, azure, alicloud]
  default_regions: [...]

personas:
  default: ""                 # optional; avoids passing persona= on cp9 up

metadb:                       # metadb provider (replaces RDS for CP state)
  provider: tidb-cloud-zero   # zero | starter | local-mysql. Single provider per
                              # session; `cp9 up` fails loud if unhealthy (no cascade).

crds:
  staging_access:
    kubeconfig: ~/.kube/staging
    context: staging-cluster
  mappings:
    tidb-operator:
      api_groups: ["pingcap.com"]
    infra-api-controller:
      api_groups: ["infra.tidbcloud.com"]
      exclude: true
  seed:
    default: crds-only        # crds-only | config-crs | full

readiness:
  tidb-mgmt-service:
    type: api                 # api | healthz
    method: GET
    path: /api/v1beta1/projects
    expect_status: 200

retry:
  build_real: 3
  gen_stubs: 3
  deploy: 2
  readiness: 3
  total: 5
  on_budget_exceeded: abort   # abort | draft-pr
```

### cp9.local.yaml (gitignored, per-developer)

> 2026-05-07 update: issue #106 merges this file, repo `cp9.yaml`, and `topologies/<persona>.yaml` into one user/session `cp9.yaml` validated by JSON Schema. Keep the example below only as historical 2026-04 context.

```yaml
name: cp9-alice-wedge1        # REQUIRED; no default. 3–32 chars DNS-compatible.
                              # = k3d cluster name = namespace prefix =
                              # cp9-session tag = simulated-tenant in metadb

persona: premium-autoscaler   # optional if set in cp9.yaml personas.default

sources:                      # REAL services
  tidb-mgmt-service:
    path: ../tidb-management-service
    ref: feature/cmek-m2
  cluster-ng-service:
    path: ../cluster-service-ng
    ref: main

overrides:
  force_real: []              # deploy REAL even if persona template marks STUB
  force_stub: []              # deploy STUB even if persona marks REAL
  extra_env:
    tidb-mgmt-service:
      LOG_LEVEL: debug

secrets:
  tidb-mgmt-service:
    env_file: ../tidb-management-service/.env.dev
  billing-svc:
    values:                   # inline overrides env_file
      STRIPE_KEY: sk_test_xxx

credentials:                  # optional per-cloud overrides
  gcp:
    service_account_key: ~/.cp9/gcp-alice.json
```

Env-file format: `KEY=VALUE`, `"VALUE"`, `'VALUE'`, `export KEY=VALUE`; blank lines and `#` comments ignored.

### Topology template

> 2026-05-07 update: issue #106 removes this standalone topology template. Persona examples become `cp9.<persona>.example.yaml`; runtime input is one `cp9.yaml` with `source`, `live`, and `stub` service sets.

```yaml
persona: premium-autoscaler
owners: ["@tennix"]
real:
  - autoscaler-engine
  - cluster-ng-service
  - dataplane-manager
  - appbox-controller
  - approute-controller
stub:
  - account-svc
  - billing-svc
requires:
  - cp-cloud                  # C1+C2
  - metadb                    # Zero / fallback
  - crds-staging
clouds:
  - aws
```

Capabilities not in `requires` are not activated. Do not list a separate dataplane capability: cp9 starts the CP environment, not the customer dataplane. The customer dataplane is produced by the CP service code after an OpenAPI create/update call, so the relevant persona choice is which cluster-lifecycle services live in `real:`.

---

## Distribution

cp9 ships as a Go module with semver-ish release tags on the `tidbcloud/controlplane9` repo. Developers install per release tag:

```bash
go install github.com/tidbcloud/controlplane9/cmd/cp9@v1.2.3   # any tag
go install github.com/tidbcloud/controlplane9/cmd/cp9@latest  # latest tag
```

`go install` builds for the host architecture (M-series → arm64), so the binary is platform-correct by construction. Tag cadence is per merged feature increment; release notes live on the GitHub release. There is no auto-update — fleet upgrades are a coordinated `go install @<new-tag>` (typically broadcast in the cp9 channel when a tag carries a behavior change).

---

## Get started

```yaml
# cp9.yaml (checked in)
apiVersion: cp9/v1
clouds:
  aws:
    profile: cp9
    regions: [us-west-2]
    account_allowlist: ["<dev account id>"]
tunnel:
  mechanism: tailscale
  vpc_a:
    account: "<dev account id>"
    region: us-west-2
    expected_egress_eip: "<stable EIP>"
metadb:
  provider: tidb-cloud-zero
```

```yaml
# cp9.local.yaml (gitignored)
name: cp9-alice-wedge1
persona: premium-autoscaler
sources:
  cluster-ng-service:
    path: ../cluster-service-ng
    ref: feature/autoscaler
  autoscaler-engine:
    path: ../autoscaler-engine
    ref: main
```

```bash
aws sso login --profile cp9
cp9 setup                 # validate prerequisites
cp9 doctor                # pre-flight
cp9 up                    # stand up (first run slow: real cloud provisioning)
cp9 status
cp9 logs --trace-id <id>
cp9 destroy               # two-stage
```

First `cp9 up` is slower (cold cloud resource creation); subsequent iterations are fast — state is re-entrant.

---

## Supporting mechanisms

### Service classification and deploy

> 2026-05-08 update: issue #106 replaces `REAL` / `STUB` / `SKIP` with `source` / `live` / `stub`. `source` services are local source-built and watched by Tilt; `live` services are real app-delivery services without local source edits; `stub` services remain generated or recording replacements. The 2026-04 flow below is historical.

Per the loaded persona template, each service is **REAL** (built from source, runs as a real pod; always REAL for cluster-lifecycle services and self-defined CRD controllers), **STUB** (proto/swagger-derived server returning schema-valid zero-value responses; real pod with real DNS so REAL callers dial it normally), or **SKIP** (not deployed).

```
REAL:   git checkout ref → docker build --platform=linux/arm64
        → k3d image import → kustomize apply
STUB:   gen stub server (cached by content hash) → minimal container
        → k3d image import → kustomize apply (service name matches REAL
        so DNS works unchanged)
SKIP:   nothing
```

All builds are native arm64 on Apple Silicon — no cross-compile.

### L1 stub generation

Server-side stubs (not client-side mocks). Stubs run as real k3s pods with real DNS; REAL callers dial them like production.

```
.proto  → protoc + grpc-go      → gRPC stub server   ─┐
swagger → kin-openapi           → HTTP stub server   ─┤
(none)  → permissive 200 OK {}                       ─┤── docker build
          (cp9 doctor warns)                          │    + k3d import
cache hit (SHA256 of protos/swagger) → skip rebuild ──┘
```

Defaults: schema-valid, zero-value responses. Improvement paths: manual edits in `.cp9/<name>/stubs/<svc>/` (persist via git merge across `cp9 sync`); agent-assisted generation from staging logs + source.

Contract drift: `cp9 sync` regenerates on proto/swagger changes (cache miss); `cp9 doctor` flags schema drift vs. callers.

### CRD installation from staging

> 2026-05-07 update: this mechanism is invalid for the issue #106 design. CRDs are defined in app-delivery service directories and install with those service manifests. cp9 should render/apply those service manifests instead of dumping CRDs from staging.

CRD schemas are pulled from the staging cluster rather than discovered from source — staging is source-of-truth, analogous to the metadb schema pattern.

```
For each controller service active in the persona and in crds.mappings:
  kubectl get crd from staging (filtered by API group)
  cache in .cp9/<name>/crd-dumps/<svc>/ (24h TTL)
  kubectl apply -f to local k3d
  optionally seed CRs from staging (--seed=config-crs|full)
```

### Secrets injection

> 2026-05-08 update: issue #106 keeps local secret inputs but routes the active contract through unified `cp9.yaml`, local secret stores/env files, Terraform outputs, and app-delivery envs. `cp9.local.yaml` references below are historical.

- **AWS secrets**: injected via local AWS credentials; the service reads Secrets Manager (dev-scoped copies in the dev account — same code path as prod).
- **K8s secrets**: one Opaque Secret per service from `cp9.local.yaml secrets.<svc>` (env_file or inline values).
- **TLS**: k3s self-signed for internal service traffic.
- **Missing**: `cp9 doctor` warns.

Third-party secrets (Stripe, etc.) must be in developer's `.env.dev` or inline — cp9 does not manage them.

### Metadb seeding from staging

> 2026-05-08 update: issue #106 moves metadb support infra creation to Terraform substrate outputs, while cp9 still owns app-level `metadb_seed` before Tilt handoff. The old standalone Zero/fallback provider chain is historical for the current 07 design.

Zero (or fallback) serves the per-developer RDS replacement. Instead of replicating each service's migration tool locally, cp9 mirrors schema + selected data from the staging metadb.

```
cp9 migrate
  per REAL service's database (per-database atomicity — each service's
    schema dump is all-or-nothing; PARTIAL per-state resumes at the
    next service):
    connect to staging metadb (read-only; DSN from staging K8s secret or
      cp9.yaml metadb.staging_access)
    mysqldump --no-data --routines --triggers
    dump selected seed data (default: schema-only; --seed: config/lookup
      tables; --seed=full: full data mirror)
    apply to Zero: mysql -h <zero-host> <service_db> < schema.sql
  cache dumps in .cp9/<name>/db-dumps/<svc>/ (24h TTL)

cp9 migrate --reset: DROP + recreate + reapply
```

Why this shape: staging schema is correct by construction (no per-service migration-tool reproduction); seeded config/enum tables prevent empty-table crashes; typical schema-only dump is < 1 MB. Requires staging-metadb read access (port-forward or direct); feature-branch additive schema changes stay backward-compatible. Staging data is treated as safe for developer Macs — no PII sanitization layer in cp9.

### Config generation (Kustomize overlays)

> 2026-05-10 update: issue #106 replaces heuristic staging overlay generation with the cp9-owned persona env contract: shared Argo-style base -> `envs/<persona>/<service>` -> user overrides -> cp9 generated patch layer -> generated Tilt input. Staging remains evidence/source material, not the runtime pipeline.

No Go templates. Overlays patched from real staging ArgoCD configs:

```
Input: app-delivery repo Application manifests (primary) or
       kubectl get application from staging (fallback).
Patch per service:
  DB DSN → Zero connection string
  AWS    → cp9 profile + dev region + dev VPC A
  gRPC / HTTP endpoints → k3d DNS names
  STUB deps → stub pod DNS (service name unchanged)
  default tags → cp9-owner + cp9-session injected into env/ConfigMap
  overrides → cp9.local.yaml overrides.extra_env
Output: overlay applied at deploy; regenerated every cp9 up.
```

Cluster-lifecycle services receive their dev account/region/VPC target through this same overlay mechanism, via the env/ConfigMap surface the services already expose in prod (per C3 audit).

### IAM and AWS resource runtime

Services provision their own AWS resources at runtime (SQS queues, S3 buckets, DynamoDB tables, …) using the developer's local CLI identity — same credential chain as production. cp9 injects creds, tracks via tags, cleans via C4.

```
cp9 up → inject AWS creds into pods (mount ~/.aws/*; AWS_PROFILE=cp9)
      → services create resources at runtime via AWS SDK (tagged
        cp9-owner + cp9-session)
cp9 status  → session health + available resource/cost reporting
cp9 destroy → Stage 2 reaper (tag-scoped, authoritative)
```

**IAM setup** (one-time org work, not cp9 code):

- Shared `cp9-developer` role in the dev AWS account; ABAC via session tag `cp9-owner=<email>`.
- Permission boundary restricts role to the dev account.
- Resource-level conditions: developers cannot touch each other's sessions (`cp9-owner` scoping).
- Rare cross-account needs use a separate staging-read-only role with explicit opt-in.
- `cp9 setup` validates AssumeRole; does NOT create IAM resources.

Per-cloud analogs: GCP WIF / static SA key; Azure Managed Identity / service principal; Alicloud RAM. Only the AWS dev account hosts the tunnel-side identity (VPC A); other clouds are reached via NAT egress + the appropriate per-cloud credential on the pod.

**Safety**: CloudTrail for forensics; AWS Budget alarms; `cp9 status` for visible session/resource reporting.

---

## Test strategy

| Level | Scope | What's tested |
|---|---|---|
| Unit | per-capability, pure Go | Config parsing; template validation; tag invariant; state transitions; reaper graph; per-cloud adapter logic with mock clients; verbose-wrapping of outbound calls |
| Integration | capability + cloud against dev account | C1 tunnel smoke; C2 provider install + sanity CR; C4 discovery+purge on a pre-seeded fixture; C5 template loading; Stage 1 local cleanup |
| E2E | full persona | `cp9 up` with unified `cp9.yaml` against real dev clouds through `cp9 destroy` exit-0 |

**Premium Reserved E2E is the current wedge exit criterion** (see [07 Reserved wedge](20260428-design-cp9-wedge-premium-reserved.md#phase-2-exit-criteria--reserved-specific)). It runs after C1–C5 ship and becomes the ongoing regression. Premium Autoscaler remains historical context in [06](20260426-design-cp9-wedge-premium-autoscaler.md).

**Integration-test hygiene.** Every test targets dev accounts in `cp9.yaml clouds.*.account_allowlist` and asserts account identity before destructive ops. Every created test resource is tagged — teardown goes through C4 (dogfoods the reaper on every CI run).

**Verbose invariant in tests.** Unit tests assert that every outbound-call surface goes through `internal/verbose`; a regression that bypasses is a unit-test failure.

**Load-bearing invariant tests.** Three integration tests each capability review must cover — explicitly called out because the invariants are load-bearing:

| Invariant | Test shape |
|---|---|
| Tag invariant | Drive a cluster-lifecycle service through a real creation path in the dev account; assert every resource returned by the cloud tag-search API carries both `cp9-owner` and `cp9-session` values matching the session. Include at least one async-creation path (queue worker, scheduler) so deferred creation is exercised. |
| Cross-owner isolation | Two sessions with the same `name` but different owners; each `cp9 destroy` cleans only the invoker's own resources; the teammate's session survives. Asserts owner-scoping default on C4. |
| Destroy exit contract | Inject a deliberately-untagged AWS resource into the session's logical scope; `cp9 destroy` must exit non-zero with that resource in the surviving list (never exit 0 with state expecting non-empty). |

**State-machine invariant tests.** Five additional integration tests covering the lifecycle invariants the design relies on:

| Invariant | Test shape |
|---|---|
| CI-identity allowlist | With SSO unavailable + `CP9_OWNER` set + caller principal NOT in allowlist for that owner: `cp9 up` exits non-zero (config category) and surfaces the caller principal ARN. With principal in allowlist: `cp9 up` proceeds and tags carry the asserted owner. |
| `flock` concurrency | Start `cp9 up` in foreground; from a second terminal start `cp9 up` for the same `name`. Second invocation exits non-zero within seconds with the lock-holder's PID. State file is unchanged. |
| Idempotent reaper resume | Simulate a Stage-2 partial purge (e.g., kill the process between deletions). Re-run `cp9 destroy`. Surviving list from the first run is consumed; second run completes and exits 0 only on empty post-scan. |
| `cp9 destroy --force` | After a Stage-1 partial first pass that leaves stray local state, `cp9 destroy` (no `--force`) refuses to re-run Stage 1; `--force` re-runs it cleanly and proceeds. |
| Orphan-state resume | Provision a session, then delete the k3d cluster manually. `cp9 up` for the same `name` resumes at `cluster_create` (not `preflight`-error). State file is preserved across the resume. |

**CI.** `run-integration` label workflow extends with per-cloud adapter integration tests behind an opt-in matrix entry. Default CI is pure-Go unit + lint.

---

## Agent-driven delivery (O2)

### Feature-dev agent pipeline

```
Feature spec (PRD / design doc)
  │
  v
1. Agent reads spec + AGENTS.md + persona template
  │
  │  ─── GATE 1: Detail Design Doc (refs a persona; proposes a new one
  │             if none fits). HUMAN REVIEW; agent STOPS.
  │
  v
2. Agent edits code (proto / handler / config).
3. Per-repo Makefile validation: make dev / make fmt / unit tests.
   On fail → agent reads error, fixes, retries.
  │
  │  ─── GATE 2: Detail Test Plan + Go test code (in proto/swagger repo).
  │             HUMAN REVIEW; agent STOPS.
  │
  v
4. cp9 deploy <svc> → direct `go test` with `pkg/cp9test` session helpers
   On fail → cp9 logs --trace-id <id> → fix → goto 3.
  │
  v
5. All tests pass → agent opens PR → human review → staging → prod.
```

**Two human gates only** — design doc, test plan. Single-repo ops (build / lint / fmt) use per-repo Makefiles; cp9 is not invoked. When a new feature needs services no existing persona covers, the agent proposes a new persona template as part of the design gate.

### Tracing and log correlation

Services already propagate trace IDs in HTTP response headers and gRPC metadata; tilog tags every log line. No Jaeger, no OTel collector — structured logs + grep.

```
E2E test response → X-Trace-Id: abc123
cp9 logs --trace-id abc123
  → for each pod: kubectl logs <pod> | grep "abc123"
  → aggregate, sort by timestamp, render cross-service flow:
    [10:00:00.001] tidb-mgmt-service  | POST /v1beta1/clusters → handler
    [10:00:00.012] tidb-mgmt-service  | CreateCluster RPC
    [10:00:00.015] cluster-ng-service | CreateCluster called
    [10:00:00.020] cluster-ng-service | crossplane CR applied
    [10:00:00.025] tidb-mgmt-service  | Response: 200, cluster_id=xyz
```

Other queries: `--all` / `--service <svc>` / `--since 5m` / `--level error`. `--json` for agents.

### AGENTS.md authoring

`cp9 onboard <repo>` scaffolds AGENTS.md + `cp9.yaml` stub per repo. Content: build/test/fmt/lint; how to run the service via a persona; key dependencies; proto/swagger source-of-truth paths; owning team. TOPOLOGY.md is not generated — there's no `topology.json` source anymore; teams describe features through persona templates.

### E2E test harness

Standard Go test code, driven by `go test`. Tests live with the proto/swagger in the service's repo. Agent authors via Gate 2; test code ships in the feature PR.

Illustrative shape — the implementing agent matches the project's actual helper surface:

```go
func TestCluster_CreateAndWaitReady(t *testing.T) {
  c := cp9test.NewClient(t) // connects to the active cp9 session
  resp := c.POST(t, "/v1beta1/projects/{pid}/clusters", map[string]any{...})
  cp9test.RequireStatus(t, resp, 200)
  traceID := resp.Header.Get("X-Trace-Id")
  cp9test.PollUntil(t, c, fmt.Sprintf(".../clusters/%s", clusterID),
    cp9test.JSONPathEquals("state", "ACTIVE"), 15*time.Minute)
  t.Logf("trace_id=%s", traceID)
}
```

Feature E2E tests run through direct `go test` invocations using `pkg/cp9test` session helpers for API endpoint, state, and credentials. Failures carry trace IDs; `cp9 logs --trace-id <id>` correlates.

### Agent plugin

Ships compatible with Claude Code + VS Code Copilot:

```
cp9-plugin/
  .claude-plugin/plugin.json
  plugin.json                    # VS Code Copilot format
  skills/
    up/ deploy/ logs/ destroy/ status/
    persona-config/ dogfood/ shared/
  agents/
    feature-dev.agent.md         # two gates; single-feature delivery
  hooks/
    hooks.json                   # PostToolUse Write/Edit → repo-owned compatibility checks
                                 # PostToolUse cp9 deploy → readiness check
  settings.json                  # default agent: feature-dev
```

Install: Claude Code `claude --plugin-dir ./cp9-plugin`; VS Code `"chat.pluginLocations": { "./cp9-plugin": true }`.

---

## Migration from 01

The existing controlplane9 repo implements 01's design. Architectural migration (no file paths):

> 2026-05-09 consistency note: the migration bullets below describe the 2026-04 C1-C5 plan. Issue #82 already removed the old topology flow. Issue #106 is the next env-creation refactor: no runtime `persona=` gate, no split `cp9.local.yaml`, no standalone topology templates, no special Crossplane/CRD lifecycle states, and no mandatory tunnel for 07 M1. The current cutover unit is the unified `cp9.yaml` + persona Terraform `.tfvars` + persona app-delivery env contract.

**Delete.** `cp9 topology` (all subcommands + scanners + linker + topology.json generation + dynamic-discovery + refine + profiles); CRD status-simulation; TiDB cluster API stubs (PD/TiDB/TiKV/TiFlash stub servers); `cp9.yaml topology.*` + `services.skip`; any `cp9 up` states existing only for topology.

**Refactor.** Classifier → template-plan computation driven by personas. AWS module → multi-cloud adapter set (AWS adapter stays in place; GCP/Azure/Alicloud siblings added). Reaper → tag-scoped, multi-cloud, credential-scoped; tighter exit semantics (zero post-scan only). Destroy flow → explicit two-stage. State machine → new `tunnel_up` / `crossplane_install` / `cred_wire` / `persona_load` states; no `cl_target_install` (folds into normal `config_gen`+`deploy`). `name` field → repo-level default/hint in `cp9.yaml`; **required** in `cp9.local.yaml`. Metadb module → same mechanics, re-labeled from "database" to "metadb" (prose + `cp9.yaml` key) to distinguish from customer dataplane. Fallback-cascade language removed — single provider per session, fail-loud.

**Add.** C1 tunnel + smoke + SSO capture. C2 crossplane install + per-cloud creds + default-tag hook. C3 audit artifacts (no runtime code). C4 multi-cloud reaper. C5 template loader + `cp9 persona` command tree.

**Keep unchanged.** k3d lifecycle; CRD install from staging; metadb seed from staging; Kustomize overlays; secrets injection; readiness checks; retry budgets; namespace-prefix isolation; verbose-wrap; error categories; trace-ID log correlation; E2E harness; plugin structure; ticat registration; AWS runtime resource creation + tagging pattern.

**Architectural deviation: CP layer collapse.** In production, Global CP and per-product Regional CPs deploy to separate clusters / VPCs. cp9 collapses all of them into one local k3d cluster (a session). Services must work in the collapsed topology. Per-service C3 audit checks a third precondition: **topology-agnostic** — does the service rely on regional-vs-global VPC properties (private network reach, regional IAM trust, regional service discovery)? If yes, externalize the assumption or surface as an audit gap. Premium Reserved carries the current audit surface in [07](20260428-design-cp9-wedge-premium-reserved.md#architecture-deviation-cp-layer-collapse).

**Cutover — additive then subtractive.**

1. Issue #82 landed the initial Premium Reserved implementation train and removed the old topology/discovery code.
2. Issue #106 defines the next flow: unified `cp9.yaml`, persona-keyed extension artifacts, Terraform substrate, selected host, context materialization, and Tilt handoff.
3. Premium Reserved wedge ships E2E on the issue #106 flow — see [20260428-design-cp9-wedge-premium-reserved.md](20260428-design-cp9-wedge-premium-reserved.md).
4. **Soak gate.** Wedge runs green for ≥ 4 weeks across the Phase-2 teams before subtractive removal of any remaining 2026-04 compatibility surfaces. Rollback during soak is "use the last accepted `cp9.premium-reserved.example.yaml` / issue #82 flow."

---

## Dependencies

- **Infra team**: pre-provisioned `cp9-developer` IAM role; stable EIP + NAT gateway in VPC A; dev-equivalent trust policies per non-AWS dev account accepting a cp9 AWS principal.
- **Platform team**: ECR read access for source references; staging kubeconfig for CRD + metadb dumps.
- **Service teams**: proto/swagger up-to-date; externally configurable per C3 audit; tag-complete per C3 audit.
- **R&D leadership**: mandate to decommission dev-base + per-cloud dev regional CPs + dev dataplanes.
- **TiDB Cloud Zero team**: API stability + rate-limit headroom for per-developer instances; programmatic create/delete at session granularity.
- **App-delivery repo**: source-of-truth for staging ArgoCD configs (Kustomize input) + pinned crossplane provider versions.
- **Product-line leads**: sign-off per [03 Decommission](20260417-design-cp9-expanded-scope.md#decommission--cost).

---

## Maintenance and ownership

**Owner**: controlplane9 team (1–2 people initially; 3–4 as adoption spreads).

| Task | Frequency | Effort |
|---|---|---|
| Persona template review | Per PR | Minutes; per-team owners |
| Stub regen on API change | `cp9 sync` triggered | Automated |
| New service cp9-readiness audit | Per onboarding team | Days per service per cloud |
| New persona authoring | As feature cross-sections emerge | Hours |
| Per-cloud reaper type-table | Per prod createcluster change | Hours; CI tag-coverage gate amortizes |
| TTL sweep | Operator-owned sweep using cp9 tag/session contracts | Automated |
| CLI / capability bug fixes | As reported | Variable |
| Crossplane provider version bumps | Quarterly + on-demand for security; CI sync job in cp9 reads `app-delivery`'s pinned versions and opens a PR | Hours; gated on adapter integration tests |

Largest growth surface + largest maintenance risk: per-cloud adapters. Shared integration-test scaffolding contains the matrix.

---

## Risks

| Risk | Mitigation |
|---|---|
| k3d too RAM-heavy on 16GB Mac | Persona scoping keeps count small (~5 REAL + 5–10 STUB, ~4GB total); `cp9 doctor` warns; 32GB recommended for 8+ REAL |
| Persona templates drift from reality | CODEOWNERS + nightly CI check "no persona references deleted service"; fail-loud on missing services |
| Stubs diverge from real behavior | Manual + agent-assisted improvement; `cp9 sync` regens on API change; `cp9 doctor` flags schema drift |
| Tunnel mechanism fragility | Smoke tests before anything downstream; one mechanism initially |
| Tag-coverage gaps (silent leak) | Per-service audit as Phase-2 prerequisite; CI gate on new call-sites per repo |
| Adapter drift across clouds | Shared integration-test scaffolding; same pass/fail contract; provider versions pinned to app-delivery |
| Crossplane dev-vs-prod drift | Pin provider versions to app-delivery's |
| Per-session cost drift | TTL cleanup; per-developer budget alarms; status/resource reporting |
| Audit scope unknown until run | Run audit before wedge implementation; surface remediation in 06 |
| Per-cloud C4 type-table scale | AWS first (Phase-2 prereq); others phased; unknown-type retry + CI gate |
| Cross-team resistance | Wedge first as proof; staging retained for validation; leadership mandate |
| AWS resource leaks on reaper failure | Stage 2 tag-scope + TTL + Budget alarms + status/resource visibility |
| VPC A / NAT EIP single egress identity | Multi-AZ NAT; infra runbook on EIP change; risk isolated to dev fleet only (no prod impact) |
| Staging metadb uptime coupling (seed blocks on outage) | `--seed=false` fallback; cache-hit path on cached dump; risk acknowledged as dev-only blocker, not design goal |
| Malicious / compromised service in persona exfiltrates via mounted creds | Persona templates have CODEOWNERS review; permission boundary scopes blast radius to dev account; residual risk accepted |
| No per-developer spend cap (enforcement deferred) | Visibility-only: status/resource reporting + AWS Budget alarms + TTL sweep. Accepted risk per /plan-ceo-review 2026-04-25; revisit if fleet spend exceeds threshold |
| Gates slow agents | Only two gates (design + test plan); async PR-style review |
| ticat maturity (single maintainer) | Cobra documented fallback; command surface is framework-independent |

---

## Operational prerequisites

Infra deliverables (not cp9 code). See [03](20260417-design-cp9-expanded-scope.md#network-reachability) for the full list. Detail-design additions:

- `cp9.yaml tunnel.vpc_a.expected_egress_eip` = stable EIP on VPC A's NAT gateway. C1 validates observed egress; mismatch aborts.
- Non-AWS dev accounts' network + IAM allowlists accept that EIP. AWS-only wedge is not gated on this; multi-cloud personas are.

---

## Success criteria

**O1 — local ephemeral dev environment (persona-scoped).**

- `cp9 up` with a unified `cp9.yaml` copied from `cp9.<persona>.example.yaml` creates a working environment on M-series Mac within the cold-start budget (exact budget: Open Question; prod createcluster is 15–30 min).
- Persona extension artifacts (`cp9.<persona>.example.yaml`, persona Terraform `.tfvars`, persona app-delivery env) drive `source` / `live` / `stub`; no static-analysis topology needed; `cp9 doctor` or future persona audit tooling flags drift.
- A feature runs E2E through the real RPC chain (`source`/`live` → `source`/`live` → `stub` where the persona allows).
- `cp9 destroy` returns 0 only when post-Stage-2 tag scan is empty across every cloud in scope.
- Any product-line team can onboard via the cp9-readiness audit; no per-team code in cp9 itself.
- Staging-retained workflow (PM QA + perf testing) unchanged.

**O2 — agent-driven E2E delivery + plugin.**

- Agent delivers a Premium Reserved-equivalent feature from spec to passing E2E inside a cp9 session in < 60 min of agent time (excluding cold-start wait).
- Two human gates enforced: design doc + test plan; no mid-code gates.
- `cp9 logs --trace-id` shows cross-service timeline for any failing test.
- Cross-repo breaking-change detection through repo-owned CI/tooling plus the persona's service set.
- Plugin installs in one command (Claude Code + VS Code).
- Agent never invokes cp9 for per-repo Makefile operations.

**Milestone**: [03 Decommission](20260417-design-cp9-expanded-scope.md#decommission--cost). This doc's success criteria are prerequisites.

---

## Open questions

On top of [03 Open Questions](20260417-design-cp9-expanded-scope.md#open-questions):

- **Tunnel mechanism spike** first implementation task; blocks C1 closure.
- **Tag audit** is a deliverable (not an assumption) per Phase-2 team.
- **Cp9-readiness audit (C3)** is the concrete artifact for 03 Q1; runs early to surface cross-repo remediation scope.
- **Cold-start latency (03 Q2) + crossplane-provisioning (03 Q3)** may need parallelization in `cp9 up`; confirm both paths are safely concurrent.
- **Starter/Essential shared-compute model (03 Q4)** deferred behind the wedge.
- **Warm pool (03 Q5)** resolves in 06.
- **Template catalog lifecycle** per-PR CODEOWNERS review; no fixed cadence.

---

## Out of scope and follow-ups

**Out of scope**: package-level code layout + Go interfaces; wedge-specific details (covered by [07 Reserved](20260428-design-cp9-wedge-premium-reserved.md)); customer-analog resources (developer provisions manually per feature); O3 Multi-Agent Orchestration + O4 Remote Environment & DR.

**Follow-ups, ordered.**

1. **Wedge detail design** — [20260428-design-cp9-wedge-premium-reserved.md](20260428-design-cp9-wedge-premium-reserved.md): current C1–C5 cross-section + Premium Reserved persona + E2E harness hitting Phase-2 exit criteria.
2. **Per-service cp9-readiness audit** — one checklist per cluster-lifecycle service per cloud; blocking per product-line team.
3. **Per-cloud C4 reaper type-table** — AWS first (Phase-2 prereq); others phased.
4. **Migration PR train** — additive-then-subtractive cutover; old flow survives until wedge proves new flow.

---

## Self-consistency checks

Verify against [03](20260417-design-cp9-expanded-scope.md) before approval:

- Issue #106 is the active env-creation design for 07: unified `cp9.yaml`, persona extension artifacts (`cp9.<persona>.example.yaml`, persona Terraform `.tfvars`, persona app-delivery env), Terraform substrate, local IAM, selected host, app-delivery/Tilt handoff.
- Every C1–C5 in 03 has a Capability contract here; C3 framing: services run unchanged, audit-only, no runtime flow.
- Every 03 Open Question has a detail-level resolution path here.
- Every 03 Rollout phase points at capabilities defined here.
- No "CRD stub" / "TiDB cluster API stub" references remain.
- No `cp9 topology` static-analysis references remain beyond the explicit DELETED line.
- Tag invariant is load-bearing in C2, C3, C4 — not only in C4.
- `cp9 destroy` exit contract matches 03 exactly. Stage 1 = local cleanup; Stage 2 = tag-scoped cloud reaper, authoritative. (03 and 04 synced 2026-04-25.)
- Metadb vs. customer-dataplane distinction explicit and maintained: metadb = Zero-replaces-RDS for CP's own state; customer dataplane = customer-facing TiDB plus supporting cloud resources created by CP code on OpenAPI create/update paths. Independent axes; no separate dataplane persona flag.
- Session identity = `name` in unified `cp9.yaml` after issue #106 (`cp9.local.yaml` was the 2026-04 baseline); owner identity = AWS SSO email (human path) or allowlisted `CP9_OWNER` (CI path, principal→email mapping). Reaper defaults to invoker-owner-scoped; `--all-owners` is admin-only; tag + credential scoping otherwise.
- Metadb: issue #106 creates/supports metadb infra through Terraform outputs and keeps app-level `metadb_seed` in cp9 before Tilt handoff. The old `cp9.yaml metadb.provider` cascade is not part of the 07 v3 contract.
- `cp9 up` holds a `flock` on `.cp9/<name>/lock`; orphan-state handling is symmetric (state exists → resume at `cluster_create`; k3d exists without state → error); TTL computed from `last-active`, not creation.
- Every topic from 01 that remained valid is either restated here or explicitly superseded by issue #106. CRD install, Kustomize overlay generation, Crossplane installation, and deploy/readiness loops must be read through the 106 app-delivery/Tilt model for 07.