# cp9 Env Creation Refactor

Codename: **controlplane9** | Issue: [tidbcloud/controlplane9#106](https://github.com/tidbcloud/controlplane9/issues/106)

Status: Proposed | Date: 2026-05-06 | Revision: 2026-05-08 | Author: @cdx-55

Derives from: [05_detail-design](20260425-design-cp9-detail.md), [07_wedge-premium-reserved](20260428-design-cp9-wedge-premium-reserved.md), [08_host-tenancy-refactor](20260430-design-cp9-host-tenancy-refactor.md)

Inputs reviewed:

- `tidbcloud/infra/deploy/eks` Pulumi staging base (`Pulumi.staging-base-us-west-2.yaml`, `index.ts`)
- `tidbcloud/auto-deploy/terraform/aws/premium`
- `tidbcloud/auto-deploy/terraform/modules/app`
- `tidbcloud/auto-deploy/terraform/aws/aws_modules/*`
- `tidbcloud/app-delivery` Application/ApplicationSet app registries and service env trees for:
  - `dedicated-cluster-ng-service/cluster-ng-server`
  - `nextgen-regional-control-plane/crossplane`
  - `nextgen-regional-control-plane/upbound-aws-provider`
  - `nextgen-regional-control-plane/dataplane-manager`
- Current `controlplane9` config, persona, lifecycle, host, kustomize, destroy, and reaper implementation after issue #82
- Official Tilt documentation for `Tiltfile` resources, `docker_build`, `custom_build`, `k8s_yaml`, `k8s_resource`, `local_resource`, `resource_deps`, `allow_k8s_contexts`, `tilt up`, `tilt ci`, `tilt logs`, `tilt trigger`, and `tilt down`

---

## TL;DR

The current PR #107 should become a v3 design for issue #106, not a small add-on to the 05/07 lifecycle.

Key decisions:

- Keep the core 05/07 invariant: cp9 control plane runs in one selected k3d host (`local-k3d` or `remote-ec2-k3d`).
- The cp9 Terraform substrate must never create EKS. It creates AWS support infra for the selected k3d session only.
- The customer/dataplane EKS remains a production service-code side effect of normal OpenAPI create/update flows. `cp9 up` does not create it.
- Merge the old topology YAML, `cp9.yaml`, and `cp9.local.yaml` into one JSON-schema-validated `cp9.yaml`.
- Replace overloaded service categories with:
  - `source`: services under active feature development; cp9 resolves local source context and Tilt builds/redeploys them in the dev loop.
  - `live`: real critical-path services deployed from app-delivery manifests/images without local code changes.
  - `stub`: generated or recording stubs.
- Treat Crossplane and the AWS provider as `live` services, not special lifecycle capabilities.
- Remove the special `crd_install` lifecycle. CRDs are installed by service manifests from app-delivery, just as they are in the ApplicationSet path.
- Replace `metadb_plan` with Terraform-created metadb support infra outputs. App data/schema seeding remains a cp9/Tilt session concern.
- After cp9 proves Terraform substrate, local IAM, selected host, and session context, delegate the change-deploy-verify loop to Tilt. cp9 composes the `Tiltfile` and inputs; Tilt owns image build/watch, manifest apply, readiness, logs, port-forward, and manual rebuild triggers.
- Network and tunnel are optional capabilities. For the 07 narrowest wedge, do not require a tunnel or dedicated network lifecycle. The local path must instead make AWS/IAM setup sound.
- Put the cp9 Terraform substrate under `tidbcloud/auto-deploy/tree/main/terraform`.
- For the M1 wedge, keep cp9-compatible Argo-style app envs in
  `controlplane9/personas/app-delivery/envs/<persona>/`; real app-delivery
  relocation is a follow-up only after the cp9-owned shape proves useful.

Terraform remains intentional source-of-truth coupling across teams, with
visible ownership and credit. The app env starts cp9-owned for M1. Neither path
directly executes staging Terraform, GitHub workflows, or ArgoCD registration.

---

## Superseded 05/07 Assumptions

This design supersedes these pieces of 05/07 once accepted:

| Old assumption | New decision |
|---|---|
| `topologies/<persona>.yaml` is the persona source of truth. | One schema-validated `cp9.yaml` is the source of truth. Persona examples are checked in as `cp9.<persona>.example.yaml`. |
| `cp9.yaml` is repo config and `cp9.local.yaml` is local/session config. | They merge into one gitignored user/session config, with example files committed. |
| `persona:` is a runtime field. | Remove it. Persona is encoded by the example file the user starts from, and the expanded service graph in `cp9.yaml` is the runtime contract. |
| `real:` means source-built services. | Use `source:` for source-built services. Use `live:` for real services pulled from app-delivery images/manifests. |
| `owners:` in persona YAML maps product owners. | Remove it. cp9 team owns persona definitions; cross-team ownership is expressed through app-delivery and auto-deploy PRs. |
| `requires:` activates separate capability flags. | Remove it or make it derived. Required capabilities come from the service graph and Terraform substrate contract. |
| `crossplane_install` is a lifecycle state. | Crossplane is a `live` app-delivery service. |
| `crd_install` dumps CRDs from staging. | Invalid. CRDs live in app-delivery service directories and install with those service manifests. |
| `metadb_plan` is a standalone cp9 provider chain. | Replace with Terraform substrate output for cloud-backed metadb support infra. Keep only app-level seed/migration in cp9/Tilt. |
| `config_gen` heuristically patches staging manifests. | Replace with `context_materialize` + `tiltfile_generate`: cp9 resolves inputs and Tilt owns the runtime deploy loop. |

---

## Non-Goals

cp9 must not:

- create control-plane EKS,
- create customer/dataplane EKS during `cp9 up`,
- ask Terraform to create or manage local k3d,
- ask Tilt to create or destroy the Terraform substrate, selected host, or tag-scoped cloud resources,
- invoke existing `terraform/aws/premium` as its daily runtime root module,
- invoke existing app-delivery GitHub workflows for per-developer sessions,
- register per-developer sessions into global ArgoCD,
- create app-delivery PRs during a session,
- mutate app-delivery or auto-deploy during `cp9 up`,
- depend on shared staging/prod Terraform state as normal per-session input.

Staging IaC and app-delivery manifests are evidence and source material. They are not the runtime pipeline.

---

## Ownership Model

The design deliberately spans three repos because repo placement also sets communication, review, and credit boundaries.

| Repo | Owner | Owns | Does not own |
|---|---|---|---|
| `tidbcloud/controlplane9` | cp9 team | CLI, config schema, lifecycle state, Terraform invocation, selected host, Tiltfile/context generation, destroy/reaper guarantees | Terraform source-of-truth modules; app-delivery source-of-truth envs; Tilt's runtime control loop |
| `tidbcloud/auto-deploy` | deployment/IaC team | cp9 Terraform substrate root module and reviewed source-material cherry-picks | cp9 CLI state machine; per-session orchestration |
| `tidbcloud/app-delivery` | app-delivery team | cp9-compatible app env overlays and service render contracts | cp9 CLI state machine; Terraform orchestration |

This makes collaboration visible:

- auto-deploy owners get review and credit for making support infra reusable by cp9,
- app-delivery owners get review and credit for making app manifests reusable by cp9,
- cp9 owners get review and credit for turning those contracts into the developer workflow.

The risk is not coupling. The risk is accidental coupling to staging/prod onboarding side effects. The contracts below keep the coupling explicit.

---

## Unified `cp9.yaml`

### Decision

Merge current `topologies/*.yaml`, repo-level `cp9.yaml`, and local `cp9.local.yaml` into one `cp9.yaml`.

Committed examples:

```text
cp9.premium-reserved.example.yaml
cp9.<future-persona>.example.yaml
```

Each example file is the persona template. After a user copies one to `cp9.yaml`, cp9 should not need a separate `persona:` field: the explicit `source` / `live` / `stub` graph is the runtime source of truth.

User/session file:

```text
cp9.yaml        # gitignored in normal service repos
```

Validation:

- JSON Schema validation first.
- Go semantic validation second.
- CLI validation against app-delivery Application/ApplicationSet names third.

### Example

```yaml
apiVersion: cp9.tidbcloud.com/v1alpha2
kind: Session

name: cp9-cara-rs1

host:
  mode: local-k3d # local-k3d | remote-ec2-k3d
  ttl: 8h

target:
  cloud: aws
  region: us-west-2
  aws:
    account_allowlist:
      - "123456789012"
    iam:
      profile: cp9
      role_arn: arn:aws:iam::123456789012:role/cp9-developer
      session_tags:
        cp9-owner: required
        cp9-session: required

terraform:
  repo: ../auto-deploy
  ref: main
  path: terraform/aws/cp9-substrate
  var_file: terraform/aws/cp9-substrate/premium.tfvars
  state: local

appDelivery:
  # cp9-owned Argo-style inputs. Registries and bases are shared; each
  # persona owns one dedicated env under envs/<persona>/.
  repo: ./personas/app-delivery
  ref: main
  env: premium-reserved

tilt:
  mode: up # up | ci | generate-only
  port: 10350

services:
  source:
    tidb-mgmt-service:
      repo: ../tidb-management-service/tidb-mgmt-service
      ref: main # override with a feature branch when testing in-flight service changes
      build:
        main: cmd/tidb-mgmt-service/main.go
      app:
        registry: registry/tidb-mgmt-service.yaml
        env: premium-reserved
      overrides:
        - ./cp9-overrides/tidb-mgmt-service.yaml

    cluster-ng-server:
      repo: ../cluster-service-ng
      ref: main # override with a feature branch when testing in-flight service changes
      build:
        main: cmd/cluster_ng_server/main.go
      app:
        registry: registry/cluster-ng-server.yaml
        env: premium-reserved

  live:
    crossplane:
      app:
        name: nextgen-crossplane
        registry: registry/crossplane.yaml
        env: premium-reserved

    upbound-aws-provider:
      app:
        registry: registry/upbound-aws-provider.yaml
        env: premium-reserved

    dataplane-manager:
      app:
        registry: registry/dataplane-manager.yaml
        env: premium-reserved

  stub:
    account-svc:
      swagger: ../account-svc/openapi.yaml
    metrics-service:
      proto: ../metrics-service/api/metrics.proto
    metering:
      kind: metering-recorder
      output: .cp9/${name}/stubs/metering/payloads.jsonl
    billing-svc:
      kind: billing-recorder
      output: .cp9/${name}/stubs/billing-svc/invoices.jsonl
```

### Naming Rationale

I recommend `source`, `live`, and `stub`.

| Name | Meaning | Why this name |
|---|---|---|
| `source` | Feature-changing service that cp9 builds from local source and redeploys in the day-2 loop. | It matches the user action: edit source, build, deploy, test. |
| `live` | Real dependency service pulled from app-delivery env/image/manifests, with no local source edits for this feature. | It avoids overloading the old `REAL` term and is shorter than `runtime` or `dependency`. |
| `stub` | Generated or recording service used when the real service is off the feature path. | Existing concept stays clear. |

Rejected names:

- `real`: too overloaded. Existing cp9 code used REAL to mean source-built.
- `runtime`: accurate but longer and less direct.
- `dep`: too vague. A `source` service is also a dependency.

### App Name Enforcement

For `source` and `live`, the service key must match one app-delivery Application/ApplicationSet name or an explicit alias declared in the app-delivery cp9 contract.

Examples from app-delivery today:

| Config key | Evidence path | App kind/name |
|---|---|---|
| `cluster-ng-server` | `registry/cluster-ng-server.yaml` | `ApplicationSet/cluster-ng-server` |
| `dataplane-manager` | `registry/dataplane-manager.yaml` | `ApplicationSet/dataplane-manager` |
| `crossplane` | `registry/crossplane.yaml` | `ApplicationSet/nextgen-crossplane` |
| `upbound-aws-provider` | `registry/upbound-aws-provider.yaml` | `ApplicationSet/upbound-aws-provider` |

The `crossplane` case must be explicit in schema-visible config. The user-facing service key may remain `crossplane`, but cp9 must bind it to the real app-delivery object with a field such as `app.name: nextgen-crossplane`. cp9 must not infer this as a hidden fallback.

### Target Cloud Semantics

`target.cloud` and `target.region` select the cloud-specific runtime configuration cp9 must generate for the real services. They do not mean cp9 creates the customer dataplane.

For M1:

```yaml
target:
  cloud: aws
  region: us-west-2
```

This drives:

- which cp9-owned Argo envs are valid for services such as `dataplane-manager`, `cluster-ng-server`, and Crossplane provider config,
- which Terraform substrate variables are written,
- which AWS account/role/profile and session-tag checks `iam_ready` performs,
- which service runtime configs are patched into the generated Tilt inputs.

The customer/dataplane EKS still appears only when the real services execute the normal production OpenAPI create/update path. Different clouds require different service runtime config; cp9's job is to select, validate, patch, and deploy those service configs for the chosen target cloud.

### Local IAM Contract

For the 07 narrowest wedge, local AWS/IAM soundness replaces the old mandatory tunnel/network gate.

`iam_ready` must prove:

- the configured AWS profile or SSO export resolves credentials for an allowlisted account,
- the caller can assume the cp9 developer role or is already running under it,
- the resolved owner is stable and lowercased,
- session tags or equivalent request tags include `cp9-owner` and `cp9-session`,
- Terraform receives only that owner/session/account/region scope,
- Crossplane/provider credentials rendered into the selected k3d host use the same scope,
- the tag-scoped reaper can scan the same owner/session scope before `cp9 up` is considered safe.

This does not require a VPC tunnel. If a future service path requires private network reachability, cp9 can add an optional `network_ready` capability for that path, but the M1 local happy path must stay `local-k3d` + local IAM + Terraform-created support infra + Tilt.

---

## App-Delivery Contract

### Source Placement

Use a cp9-owned, Argo-style shape. It borrows Application/ApplicationSet,
kustomize, and Helm concepts from app-delivery, but it is intentionally not a
mirror of the app-delivery repository:

```text
controlplane9/personas/app-delivery/
  registry/
    tidb-mgmt-service.yaml
    cluster-ng-server.yaml
    crossplane.yaml
  base/
    tidb-mgmt-service/
    cluster-ng-server/
    upbound-aws-provider/
    dataplane-manager/
  charts/
    crossplane/
  envs/
    premium-reserved/
      tidb-mgmt-service/
      cluster-ng-server/
      crossplane/
      upbound-aws-provider/
      dataplane-manager/
```

`registry/`, `base/`, and `charts/` are shared inputs across personas. One
persona owns one env directory, such as `envs/premium/`. A future
persona should add a sibling env directory and reuse the shared inputs where
possible.

This is preferred over placing cp9 envs under `staging/.../envs/cp9` or
mirroring app-delivery owner directories because cp9 is not a staging
environment and the narrow wedge should stay reviewable by persona key.

The cp9 env may reuse ideas from production/staging bases and variants, but it
is a distinct cp9-owned persona contract. The env name should use the persona
key, such as `premium-reserved`, so the cp9 example file, Terraform `.tfvars`
file, and Argo-style env are tied together by one reviewable name.

### Runtime Contract

cp9 reads app-delivery manifests but does not run ArgoCD. cp9 also no longer owns the steady-state apply/rebuild loop after session bootstrap. It prepares Tilt inputs and a generated `Tiltfile`; Tilt owns the runtime loop.

For each `source` or `live` service:

1. Load the Application/ApplicationSet from `services.*.<name>.app.registry`.
2. Validate the requested `env` exists in the cp9 app contract.
3. Resolve the render input path:
   - kustomize env path for normal services,
   - helm chart + values for Helm-backed apps such as Crossplane,
  - multi-source rendering only when the cp9-owned Argo env contract declares it.
  For remote Helm charts, cp9 keeps the Argo source shape and uses a local `charts/<chart>` mirror from the checked-out app root for Tilt rendering.
4. Materialize deterministic layers into Tilt inputs:

```text
cp9-owned Argo base
  -> cp9-owned persona env
  -> user override manifests from cp9.yaml
  -> cp9 generated patch layer
  -> Tilt k8s_yaml / custom deploy input
```

The cp9 generated patch layer owns only per-session values:

- namespace,
- image refs for `source` services,
- Terraform outputs,
- metadb endpoint/credentials references,
- in-cluster service endpoints,
- `cp9-owner` / `cp9-session` labels/env/tags,
- stub endpoints and recorder mounts.

### Context Artifacts

Every handoff input and output must be inspectable:

```text
.cp9/<name>/tilt/
  Tiltfile
  inputs/
    app-delivery/<service>/
    patches/<service>/
    stubs/<service>/
  reports/
    context-report.json
    tilt-status.json
```

`context-report.json` must include:

- app-delivery repo/ref/path,
- Application/ApplicationSet name,
- selected env,
- Tilt resource name,
- deploy input type (`kustomize`, `helm`, rendered YAML, or explicitly declared multi-source),
- user overrides applied,
- Terraform outputs consumed,
- IAM/account/owner/session scope consumed,
- fallback status.

Fallback is allowed only during migration and must be loud in `cp9 doctor`.

---

## Tilt Handoff Contract

### Boundary

Tilt is the control loop for local development after cp9 prepares the session. It is not the environment owner.

| Surface | Owner |
|---|---|
| `cp9.yaml` validation, owner/session identity, IAM/account allowlist | cp9 |
| Terraform substrate init/plan/apply/destroy | cp9 invoking auto-deploy-owned Terraform |
| selected host create/delete and selected kubeconfig/context | cp9 host adapter factory |
| optional remote EC2/tunnel/network setup | cp9 or Terraform substrate, only when enabled |
| source image build/watch, deploy/apply, readiness, logs, port-forward, manual rebuild triggers | Tilt |
| final cleanup proof and tag-scoped cloud reaper | cp9 |

The important handoff line is:

```text
cp9 bootstrap + context materialization
  -> generated Tiltfile
  -> tilt up / tilt ci owns change-deploy-verify
```

### Generated Tiltfile

The generated `Tiltfile` must be deterministic and reviewable. It should use official Tilt primitives directly:

- `allow_k8s_contexts(<selected cp9 context>)` as a safety guard.
- `docker_build` or `custom_build` for `source` and generated `stub` images.
- `k8s_yaml`, `kustomize`, or `helm` for cp9-owned Argo envs.
- `k8s_resource` for resource naming, `port_forward`, and dependency wiring.
- `resource_deps` for first-start ordering where the service graph requires it.
- `local_resource` only for local commands that are truly part of the dev loop, not for Terraform substrate creation.
- `trigger_mode` and `tilt trigger` for compatibility with `cp9 deploy <svc>`.

Resource mapping:

| cp9 service kind | Tilt resource behavior |
|---|---|
| `source` | Build from local source, watch source files, deploy to selected k3d, expose logs/readiness/port-forward. |
| `live` | Apply app-delivery cp9 manifest using declared image refs or app-delivery-defined image source. No local source watch. |
| `stub` | Build/apply generated or recording stub resources. Recording file mounts stay under `.cp9/<name>/stubs`. |

Crossplane, upbound AWS provider, and dataplane-manager are normal `live` resources in this model.

### CLI Behavior

| Command | v3 behavior |
|---|---|
| `cp9 up` | Run cp9 bootstrap, generate `.cp9/<name>/tilt/Tiltfile`, then start `tilt up -f .cp9/<name>/tilt/Tiltfile` by default. `--no-tilt` or `tilt.mode: generate-only` prints the command instead. |
| `cp9 up --ci` | Run bootstrap, then `tilt ci -f ...`; this is the integration-gate path because `tilt ci` exits only after resources are healthy or failed. |
| `cp9 deploy <svc>` | Compatibility wrapper around `tilt trigger <resource>` when a Tilt session is running. The primary day-2 loop is Tilt's watch/rebuild/apply. |
| `cp9 logs <svc>` | Prefer `tilt logs <resource>` for host-cluster resources. Selected-host kubectl diagnostics remain a fallback/debug path. |
| `cp9 status` | Show cp9 bootstrap state plus Tilt session/resource status. |
| Feature `go test` + `pkg/cp9test` | Feature repos own E2E execution; cp9 exposes reusable helpers and session facts. |
| `cp9 destroy` | Best-effort `tilt down -f ...`, then cp9 authoritative host cleanup, Terraform destroy, and tag-scoped reaper. |

### Why Tilt Is the Right Boundary

Tilt already models the pieces cp9 was starting to reimplement: file watching, image rebuilds, Kubernetes apply, readiness, logs, port-forwards, resource dependencies, manual triggers, and CI mode. Keeping those in cp9 would duplicate an existing dev-loop engine.

The same argument does not apply to substrate or cleanup. Tilt does not prove owner/session IAM scope, Terraform output correctness, k3d ownership, or empty final cloud scans. cp9 must keep those responsibilities.

### Host Adapter Factory

cp9 should implement selected-host management behind a host adapter factory, not hard-code local k3d into the lifecycle engine.

Adapter contract:

- input: validated `host.mode`, session name, owner, TTL, target cloud/region, and Terraform outputs when relevant,
- output: selected kubeconfig, context, namespace, image import/build strategy, and cleanup handle,
- required operations: `Ensure`, `Load`, `RunKubectl`, image availability, diagnostics, and `Destroy`.

M1 implements only `local-k3d`. Later adapters can add `kind`, `remote-ec2-k3d`, or even remote managed Kubernetes hosts such as EKS/GKE without changing the service graph or Tiltfile generator boundary.

---

## Terraform Substrate Contract

### Source Placement

The cp9 substrate belongs in `tidbcloud/auto-deploy/tree/main/terraform`.

Proposed shape:

```text
auto-deploy/
  terraform/
    aws/
      cp9-substrate/
        main.tf
        variables.tf
        outputs.tf
        versions.tf
        premium.tfvars
        README.md
```

The substrate root module is independent from `terraform/aws/premium`.
Persona-specific HCL var files select and parameterize the subset of supporting infra used by that persona. For 07 M1, `premium.tfvars` is manually derived from staging source material by the persona owner and should start with local IAM plus any proved metadb support requirements. cp9 copies this file into `.cp9/<name>/terraform/` and mutates that session-local copy with runtime owner/session/account/tag values.

### State

State is local by default, per user decision.

cp9 materializes a per-session Terraform working directory:

```text
.cp9/<name>/terraform/
  source/
  premium.tfvars
  terraform.tfstate
  outputs.json
  logs/
```

cp9 does not run Terraform inside the auto-deploy checkout.

Remote state may be added later only with a per-session key:

```text
cp9/<owner>/<session>/terraform.tfstate
```

### What Terraform May Create

Terraform may create AWS support infra for one cp9 session:

- IAM roles/policies consumed by Crossplane/provider configs, app manifests, or local developer role assumption,
- metadb support infra when the M1 path needs a cloud-backed metadb endpoint,
- Secrets Manager/SSM secret references for app manifests,
- VPC/subnet/security-group support only when remote host or cloud reachability requires it,
- remote EC2 host, instance profile, user data, and security group for `remote-ec2-k3d`,
- Tailscale or equivalent bootstrap material only when an optional remote/network mode is enabled.

For the 07 narrowest wedge, the first Terraform graph should start with IAM and metadb support infra. It should not add VPC/tunnel resources unless the acceptance path proves they are required.

Terraform must tag every resource:

- `cp9-owner`
- `cp9-session`
- `cp9-managed = terraform-substrate`
- `cp9-ttl` or equivalent expiry timestamp

### What Terraform Must Not Create

Terraform must not create:

- control-plane EKS,
- customer/dataplane EKS,
- EKS node groups,
- permanent ArgoCD cluster registration,
- global region registration,
- app-delivery workflow dispatch resources,
- app-delivery PR/automerge state,
- shared staging/prod dependencies as normal runtime state.

### Existing auto-deploy Source-Material Decision

After reading current auto-deploy:

| Existing code | Decision for cp9 | Reason |
|---|---|---|
| `terraform/aws/premium` root | Source material only; do not import. | It creates EKS and reads shared env state. |
| `terraform/modules/app` | Source material only; do not import. | It dispatches GitHub workflows through `gh`, waits for Argo-created resources, and models region onboarding side effects. |
| `terraform/modules/argocd_cluster` | Source material only; do not import. | It registers clusters into ArgoCD, explicitly out of scope. |
| `terraform/modules/global_registration` | Source material only; do not import. | It registers real regions globally, explicitly out of scope. |
| `terraform/modules/kubeconfig` | Source material only; do not import. | It writes EKS kubeconfig; cp9 selected-host kubeconfig is local/remote k3d. |
| `terraform/aws/aws_modules/rds-db` | Source material only; cherry-pick concepts with modification. | It applies DB resources through a kubeconfig and assumes the premium EKS path. |
| IAM helper patterns in `terraform/aws/aws_modules/*` | Source material only; cherry-pick with modification. | Some are service-specific and may carry shared-state, EKS, or static-tag assumptions. |

The cp9 substrate should start as a new root module. Do not directly import existing auto-deploy root modules or helper modules into the cp9 runtime. Copy/cherry-pick only the minimal logic that survives cp9-specific review: no EKS, no Argo registration, no GitHub workflow dispatch, no shared staging/prod state, and every resource session-tagged.

---

## Runtime Ownership Decisions

The latest review resolves the main ownership questions.

| Surface | Decision | Reason |
|---|---|---|
| local k3d | cp9-owned, not Terraform-owned | It is a local developer-machine operation; Terraform would make the default debug path worse. |
| remote EC2 host | optional follow-up; may be Terraform-owned | EC2, instance profile, security group, user data, and TTL fit Terraform state when remote mode is enabled. |
| network/tunnel | optional capability, not required for 07 M1 | The narrow wedge should avoid a mandatory tunnel and prove local IAM first. |
| IAM support | Terraform substrate + cp9 validation | Terraform creates scoped support infra; cp9 verifies caller identity, account allowlist, role/profile, and owner/session tags. |
| cloud-backed metadb support infra | Terraform output | It has lifecycle/cost/tag semantics and should be visible in substrate state. |
| metadb schema/data seed | cp9 or Tilt local resource, depending on implementation | It is app/session initialization, not infra provisioning. |
| source/live/stub build/deploy/readiness/logs | Tilt | Tilt is the right control loop for changes after cp9 has prepared context. |

This is no longer a hybrid-host open question for M1. The M1 default is:

```text
local IAM ready
  -> Terraform support infra ready
  -> cp9-managed local k3d ready
  -> Tilt handoff
```

Remote EC2 and tunnel/network paths stay design-compatible but should not be required by the Premium Reserved acceptance path.

---

## Lifecycle v3

### Current Invalid States

These current states must be removed or renamed:

| Current state | Action |
|---|---|
| `crossplane_install` | Remove. Crossplane becomes a `live` service rendered/applied from app-delivery. |
| `crd_install` | Remove. CRDs are applied with their owning app-delivery service manifests. |
| `metadb_plan` | Replace with Terraform-created metadb support infra outputs plus explicit cp9-owned `metadb_seed`. |
| `config_gen` | Replace with `context_materialize` and `tiltfile_generate`. |
| `build_real` / `build_source` | Remove from cp9 lifecycle. Tilt builds/watches source images after handoff. |
| `gen_stubs` | Remove from cp9 lifecycle as a deploy-loop step. cp9 may generate stub source/inputs; Tilt builds/applies them. |
| `deploy` / `app_apply` | Remove from cp9 lifecycle. Tilt applies Kubernetes resources. |
| `readiness` | Remove from cp9 lifecycle. Tilt owns resource readiness and `tilt ci` is the CI gate. |

### Proposed State Machine

```text
preflight
-> config_load
-> owner_identity_ready
-> iam_ready
-> substrate_init
-> substrate_plan
-> substrate_apply
-> host_ready
-> service_plan
-> metadb_seed
-> context_materialize
-> tiltfile_generate
-> tilt_ready
-> session_ready
```

Optional extension states:

```text
network_ready       # only when a service path or host mode explicitly needs private reachability
remote_host_ready   # only when host.mode=remote-ec2-k3d
```

State meaning:

| State | Meaning |
|---|---|
| `preflight` | Lock `.cp9/<name>`, check required binaries, validate repo paths exist. |
| `config_load` | Load one `cp9.yaml`, validate JSON Schema, then Go semantics. |
| `owner_identity_ready` | Resolve `cp9-owner` from SSO or allowlisted CI path. |
| `iam_ready` | Validate AWS profile/role/account allowlist/session tags and prove the reaper can scan the owner/session scope. |
| `substrate_init` | Materialize Terraform source into `.cp9/<name>/terraform/source` and run init. |
| `substrate_plan` | Copy the persona-owned `.tfvars` into `.cp9/<name>/terraform/`, inject runtime values/tags into that copy, run plan, persist plan logs. |
| `substrate_apply` | Apply Terraform, persist `outputs.json`. |
| `host_ready` | Verify/create selected host. For M1 local path, create/verify local k3d. For optional remote mode, resolve remote host from Terraform outputs. |
| `service_plan` | Resolve `source` / `live` / `stub` graph; validate app-delivery names and envs. |
| `metadb_seed` | cp9-owned schema/data seed and migration preparation for the session metadb. It consumes Terraform metadb outputs and service seed requirements, then persists a seed report. |
| `context_materialize` | Resolve cp9-owned Argo-style envs, source repo paths, user overrides, Terraform outputs, IAM scope, secrets, stubs, recorder mounts, and metadb DSNs/seed report into `.cp9/<name>/tilt/inputs`. |
| `tiltfile_generate` | Generate `.cp9/<name>/tilt/Tiltfile` plus `context-report.json`. |
| `tilt_ready` | Launch or validate Tilt: `tilt up` for interactive mode, `tilt ci` for CI/integration mode, or print the handoff command in generate-only mode. |
| `session_ready` | Persist final summary and unblock day-2 commands. |

### State Artifacts

```text
.cp9/<name>/
  state.json
  cp9.resolved.yaml
  host.json
  kubeconfig.yaml
  terraform/
    premium.tfvars
    terraform.tfstate
    outputs.json
  services/
    plan.json
  metadb/
    seed-report.json
  tilt/
    Tiltfile
    inputs/
    reports/
      context-report.json
      tilt-status.json
  stubs/
  destroy/
```

`cp9.resolved.yaml` is important: it lets reviewers see exactly what schema defaults, env resolution, and app aliases produced.

---

## Service Graph for Premium Reserved M1

The smallest support graph must include Crossplane because dataplane-manager and cluster-ng-service rely on Crossplane provider APIs in the production path.

Initial Premium Reserved M1 graph:

```yaml
services:
  source:
    tidb-mgmt-portal: {}
    tidb-mgmt-service: {}
    cluster-ng-server: {}

  live:
    crossplane: {}
    upbound-aws-provider: {}
    dataplane-manager: {}

  stub:
    account-svc: {}
    metrics-service: {}
    metering: {}
    billing-svc: {}
```

For Nicole CPU / Premium Reserved M1, `dataplane-manager` is `live`. If a later feature changes dataplane-manager code, that later feature can move it to `source`. This is exactly why `source` is separate from `live`: feature scope decides the edit loop, not whether the service is real.

---

## Persona Extension Contract

Problem solved: cp9 cannot infer the right narrow wedge by scanning staging code alone. Real staging Pulumi, Terraform, and ArgoCD/app-delivery code contain hidden context: which resources are actually needed, which configs are staging-only, which assumptions must be preserved, and which values are safe to replace for cp9. If that decision stays implicit, implementation either blocks on uncertainty or accidentally recreates staging.

Decision: before implementing a persona, the persona owner must do the human discovery work and extend three cp9 integration points. The contract is not a separate runtime persona registry; the contract is the set of persona-keyed artifacts that cp9 already needs to run.

Contract artifacts:

| Artifact | Owning repo | Purpose |
|---|---|---|
| `cp9.<persona>.example.yaml` | `controlplane9` | Developer-facing session template: host mode, target cloud/region, service graph, local source paths, Tilt behavior, Terraform root/config location, and app-delivery env location. |
| persona-specific HCL `.tfvars` | `auto-deploy/terraform` | Inputs for the cp9 Terraform substrate. It declares which supporting infra modules are enabled for the persona and their variables. cp9 mutates only the session-local copy with runtime owner/session/account/tags. It must not create EKS. |
| persona-specific Argo env / kustomize overlay | `app-delivery` | Runtime app configuration for `live` and `source` services under cp9. It is the Argo/app-delivery-side extension point that cp9 consumes when generating the final manifests. |

The persona name is the business key across all three artifacts. For Premium Reserved, `premium-reserved` must appear consistently in:

```text
controlplane9/
  cp9.premium-reserved.example.yaml

auto-deploy/
  terraform/aws/cp9-substrate/
    premium.tfvars

app-delivery/
  <cp9-argo-env-root>/
    premium-reserved/
```

Exact directory names should follow each owning repo's convention, but cp9 should not maintain a hidden mapping table. `cp9.premium-reserved.example.yaml` points to the `premium-reserved` Terraform var file and the `premium-reserved` Argo env. That common name is the review and debug handle.

For the narrowest 07 wedge, there is no need to build a generic persona authoring platform first. The persona owner manually traverses staging Pulumi, existing Terraform, and app-delivery/ArgoCD code, then hand-writes the three artifacts:

1. Extend cp9 Terraform in `auto-deploy/terraform` with the minimal supporting infra needed by the persona.
2. Provide the persona HCL `.tfvars` file for that substrate. For 07, this should focus on local IAM and other supporting infra that lets the single k3d host behave like the required control-plane environment. It must not create EKS.
3. Extend the cp9 Argo/kustomize env in `app-delivery` with the service runtime config for the persona.
4. Update `cp9.<persona>.example.yaml` to point at those two persona-specific artifacts and declare the `source` / `live` / `stub` graph.
5. Record the classification table below in this design or issue until cp9 has first-class authoring/audit commands.

Future tooling can make this easier with schema checks for `.tfvars` and Argo env references plus an audit command that verifies the three artifacts share the same persona key and cover every required input. That is follow-up leverage, not a blocker for 07 M1.

Outcome:

- Saba/team can feed hidden staging context into cp9 through concrete extension artifacts, not an abstract pack.
- New personas can be injected by adding three persona-keyed artifacts in the correct owning repos.
- Implementation can proceed after the owner classifies the inputs that otherwise block the service graph, Terraform substrate, and Argo env decisions.

## Input Classification Buckets

Each persona-specific input must map to exactly one bucket.

Owner buckets:

| Bucket | Meaning | Typical usage |
|---|---|---|
| cp9-owned Argo env | Static manifest/env content owned by the persona in controlplane9. | Namespace, deployment layout, RBAC, baseline config maps, service accounts, Helm values. |
| cp9 generated patch | Per-session values generated by cp9. | Image refs, `cp9-owner`, `cp9-session`, selected namespace, service DNS rewrites, recorder mounts. |
| Terraform output | Cloud support resource created by cp9 substrate. | IAM role ARN, metadb endpoint, secret ARN. |
| user override | Developer-provided manifest or scalar override. | Enable debug logs, shrink resources, point to a branch-specific endpoint. |
| secret input | Sensitive value supplied by SSO/CI/local secret store. | API token, DB password, Tailscale auth material if approved later. |
| stub/mock | Local replacement for an off-path service. | account-svc, metrics-service, metering recorder, billing recorder. |
| not needed for M1 | Staging input excluded from the first acceptance path. | O11y integration, optional public DNS, full production alerting, tunnel-only config. |

Initial M1 rows to classify:

| Input | Expected M1 bucket |
|---|---|
| local AWS profile / role | secret input + `iam_ready` validation |
| `cp9-owner` / `cp9-session` tag contract | cp9 generated patch |
| metadb endpoint / credentials | Terraform output + secret input |
| metadb schema/data seed | cp9-owned `metadb_seed` |
| Crossplane core chart/values | cp9-owned Argo env |
| Upbound AWS provider config | cp9-owned Argo env + Terraform output |
| IAM role ARN/name | Terraform output |
| dataplane account list | cp9-owned Argo env or user override, depending on feature path |
| image registry / image refs | cp9-owned Argo env for `live`; cp9 generated patch for `source`/`stub` |
| resource tags | cp9 generated patch |
| DNS/hosted zone | not needed for M1 unless acceptance proves otherwise |
| PCA/cert inputs | not needed for M1 unless acceptance proves otherwise |
| public access CIDRs | not needed for M1 unless acceptance proves otherwise |
| metering bucket/roles | not needed while metering is a recorder stub |
| node-pool templates | cp9-owned Argo env or user override if dataplane-manager requires them for AWS M1 |

---

## Destroy Semantics

Destroy must work when any earlier phase failed.

Order:

1. Load persisted `cp9.resolved.yaml`, selected host, Terraform state, Terraform outputs, and owner/session scope.
2. Run `tilt down -f .cp9/<name>/tilt/Tiltfile` best-effort if a Tiltfile exists.
3. Delete selected host:
   - local: delete k3d cluster,
   - remote: destroy/terminate via Terraform state, then verify no selected host remains.
4. Run Terraform destroy for cp9 substrate state.
5. Run tag-scoped cloud reaper for the owner/session.
6. Exit 0 only if the final post-scan is empty.

Terraform destroy failures must not prevent the reaper from running. Return both errors if both happen.

## Cross-Doc Consistency Checks

Before accepting this design, verify the three active docs agree:

- [05](20260425-design-cp9-detail.md) remains authoritative for owner/session identity, tag invariant, selected-host ownership, destroy/reaper exit contract, and the customer-dataplane-as-service-side-effect invariant.
- [07](20260428-design-cp9-wedge-premium-reserved.md) is the concrete wedge: `local-k3d` is the M1 host, network/tunnel is out of M1, Premium Reserved uses the `source` / `live` / `stub` graph, and OpenAPI create/update remains the validation path.
- This document supersedes 05/07 only for env creation: unified `cp9.yaml`, persona extension artifacts, Terraform substrate, app-delivery env materialization, and Tilt handoff.
- The same persona key appears in all durable extension artifacts: `cp9.premium-reserved.example.yaml`, `terraform/aws/cp9-substrate/premium.tfvars`, and the app-delivery `premium-reserved` env.
- No current, non-historical section should describe `crossplane_install`, staging `crd_install`, standalone `metadb_plan`, or mandatory tunnel/network as required for the 07 narrow wedge.

## Remaining Implementation Risks

1. **Tilt remote mode.** The local path is straightforward. Remote EC2-k3d needs a careful `custom_build` / image-transfer strategy and should follow after local `tilt ci` is stable.
2. **App-delivery render type.** Some services are kustomize, some may be Helm or multi-source. The first implementation uses Argo CD `v1alpha1` Go API types from the v2.14 module and a narrow Crossplane Helm + `$values` multi-source resolver; moving to the v3 module should be a mechanical pin change once v3 is consumable without local `gitops-engine` replaces.
3. **IAM proof.** The local IAM gate must be strict enough that customer-dataplane resources created by service code are reaper-scannable by owner/session before M1 acceptance.
4. **Metadb seed complexity.** Terraform creates endpoint/credentials/support infra, but cp9 must own schema/data seed and migration preparation explicitly before Tilt starts services.
5. **Tilt status mapping.** cp9 should summarize Tilt status without duplicating Tilt's per-resource runtime state.