# Design: Agent Developer Experience for cp9

Issue: [tidbcloud/controlplane9#2](https://github.com/tidbcloud/controlplane9/issues/2)
Status: Proposed
Date: 2026-05-14
Author: @Bob

## Summary

cp9 should stay CLI-first.

The restarted O2 work should not begin with a plugin, a new command family, or a generic feature-cycle abstraction. It should make the existing cp9 CLI easier for agents and engineers to use, then add small cp9 skills that teach the daily workflow.

Delivery order:

1. Improve the persona example config experience.
2. Keep the existing CLI surfaces clear for agents: `setup`, `doctor`, `up`, `status`, `logs`, and `destroy`.
3. Add cp9 skills as the agent catch-up layer.
4. Prove the flow on Premium Reserved.
5. Package plugins only after CLI and skills prove repeatable.

First product proof:

```text
public OpenAPI create
  -> real cp9-owned Premium dataplane
  -> returned public SQL endpoint
  -> mysql select 1
```

First agent-DX proof:

```text
fresh agent starts from a GitHub issue
  -> chooses the right cp9.<persona>.example.yaml
  -> tunes a small cp9.yaml for the ad hoc case
  -> runs setup / doctor / up / status
  -> knows current runtime state and next safe action
```

## Decisions

| Decision | Rationale |
|---|---|
| CLI is the source of truth | Agents need stable commands and structured output, not hidden shell recipes from chat. |
| Example configs are the entrypoint | Real cp9 users start by choosing and tuning a `cp9.<persona>.example.yaml`. |
| Existing commands stay central | `status` and `doctor` already cover runtime state and validation. Adding another command family would be over-designed. |
| Feature repos own tests | Test setup, case code, engines, and reports vary by language and feature. cp9 should provide environment facts, not own every test workflow. |
| `cp9 destroy` is the cleanup command | The tag-scoped reaper is an implementation stage behind destroy, not a separate daily user workflow. |
| Skills teach orchestration | Skills should explain when and how to use cp9 commands. They must not duplicate cp9 business logic. |
| Premium Reserved is the first proving path | It is the active cp9-owned environment wedge and exercises real control-plane behavior. |
| Staging is template input only | cp9 must own the runtime environment. Staging should not be a runtime dependency or success path. |
| Plugin packaging is deferred | Packaging before CLI and skills stabilize would hide the real DX gap. |

Supersedes:

- The agent-delivery and plugin-first portions of the original O2 issue #2.

Derives from:

- [20260425-design-cp9-detail](20260425-design-cp9-detail.md)
- [20260506-design-cp9-env-creation-refactor](20260506-design-cp9-env-creation-refactor.md)
- [20260509-premium-reserved-v1alpha2-readiness-audit](20260509-premium-reserved-v1alpha2-readiness-audit.md)
- [#185 P7 Premium dogfood meta](https://github.com/tidbcloud/controlplane9/issues/185)
- [#192 current real-create boundary](https://github.com/tidbcloud/controlplane9/issues/192)

## Reader Path

The design optimizes for two readers.

| Reader | Needs | Should not do |
|---|---|---|
| Fresh agent or engineer | Pick a persona config, tune it safely, run the next existing cp9 command | Reconstruct history from chat threads, old PR comments, or long design archaeology |
| Reviewer or project lead | See proof, runtime state, remaining boundary, and next cut | Rebuild the run timeline from terminal output |

Target first-five-minute outcome:

```text
git clone / checkout repo
choose cp9.premium-full.example.yaml
copy or adapt it as cp9.yaml
run cp9 setup / cp9 doctor
know whether cp9 up is safe to run
```

After those steps, a fresh agent must be able to answer:

- Which persona example is closest to this issue?
- Which fields need ad hoc tuning?
- Is staging only template input?
- What resources may cp9 create?
- Which owner/session tags are required?
- What does `cp9 doctor` require before `cp9 up`?
- Which human gate, if any, blocks the next step?

Full environment readiness can take longer than five minutes. Picking the right example config and finding blockers should not.

## Problem

cp9 work still depends on human memory.

Agents can read the repo and still miss:

- which persona example to start from,
- which fields are safe to tune for an ad hoc run,
- which services are source, live, or stubbed,
- which prerequisite is missing,
- which runtime resource is failing now,
- which command proves the current state,
- whether cleanup has a single user-facing closeout path.

The current Premium Reserved dogfood train uses good execution discipline: small PR-backed cuts, each proving one boundary. The repo does not yet encode enough of that discipline in example configs, CLI output, and skills.

The gap:

```text
cp9 must make persona example selection and tuning obvious.
cp9 status and doctor must carry the runtime facts agents need.
cp9 skills must teach agents the daily workflow around existing commands.
```

## Non-Goals

- Do not make cp9 own feature test setup, test case code, test engines, or test reports.
- Do not replace repo Makefile workflows. cp9 does not own single-repo `make build`, `make lint`, or `make test`.
- Do not use staging as a runtime dependency. Staging remains source/template input.
- Do not solve all of issue #192 in this design.
- Do not ship a plugin before CLI and skill contracts stabilize.
- Do not let agents merge, deploy staging, or deploy production without normal review.
- Do not require agents to read this whole document before every task.
- Do not optimize for a hosted playground, IDE extension, or plugin marketplace in this lane.

## Current State

### Original O2

The original O2 goal remains useful:

- agent delivers a feature from spec to passing E2E in less than 60 minutes,
- human gates are limited to spec approval, design review, test strategy review, and code review,
- plugin ships as part of O2.

The delivery order changed. cp9 needs example-config DX and skills before plugin packaging can help.

### Existing User Flow

Typical cp9 user flow:

1. cp9 owner provides persona examples, such as `cp9.premium-full.example.yaml`.
2. User picks the closest example for the daily development persona.
3. User tunes a small set of fields for the ad hoc case: session name, owner, source paths, image refs, account allowlist, TTL, or service overrides.
4. User runs `cp9 setup`, `cp9 doctor`, and `cp9 up`.
5. User enters the normal development loop: edit feature repo, run feature-owned tests, inspect `cp9 status`/`cp9 logs`, then destroy.

The missing product work is not another command. The missing work is helping users choose and tune the right example config, then making the existing commands explain the current state.

### Existing CLI

Source: `internal/cmd/commands.go`, command implementations under `internal/cmd/`, and command registration tests.

Code facts from the current v1alpha2 path:

- `setup` validates AWS identity, owner allowlist, IAM role assumptions, and cleanup permissions. It does not execute Terraform today.
- `up` runs the side-effecting lifecycle: config load, owner/IAM checks, Terraform source materialization, Terraform preview, Terraform apply, host readiness, service graph load, metadb seed, runtime materialization, Tiltfile generation, and Tilt readiness.
- `destroy` already runs the closeout sequence: Tilt down, host cleanup, Terraform destroy, and tag-scoped cloud reaping.
- `deploy` triggers a generated Tilt resource for a running session.
- The removed `test` command used to shell out to feature-specific Go tests with cp9 environment variables. Feature repos now run direct `go test` and consume `pkg/cp9test`.

Current dogfood metadb handling:

- The v1alpha2 `metadb_seed` lifecycle step does not use the removed standalone migration command. It reads the Terraform metadb output contract and persists `.cp9/<session>/metadb/seed-report.json` with admin and per-service DSNs.
- The runtime input materialization step consumes that seed report and passes metadb DSNs into the generated service input bundle.
- Persona app-delivery inputs own schema and data needed by the local dogfood path. Premium Reserved uses one shared `premium-metadb` MySQL instance and separate service databases inside it; its committed init SQL files are mounted through one ConfigMap under `/docker-entrypoint-initdb.d`.
- The desired RDS shape is one provisioned database instance per cp9 session, with separate service databases inside that instance. This is already the contract of `internal/db.Provision`: create one provider instance, then `CREATE DATABASE` per service and publish per-service DSNs. The product surface should explain and validate that contract instead of implying one RDS instance per service.
- The old standalone migration command was a legacy staging-DSN dump/apply helper. It was never part of the current v1alpha2 `up` path and has been removed from the CLI surface.
- `cp9 db dump-init` is the seed-authoring surface for producing persona-owned init SQL from a real MySQL/RDS source when users need to edit persona seed records. The generated SQL should be reviewed and committed under the persona; runtime `cp9 up` consumes those committed SQL files instead of running ad hoc config-center seed curls.

Daily user surface:

| Command | Origin usage | Gap from current dogfood use | Action |
|---|---|---|---|
| `cp9 setup` | Validate identity, account, IAM, owner/session tags, and cleanup permission before cloud work. | Users need a no-mutation preview before expensive work. Code today does not run Terraform, so any Terraform preview must be explicit in output or routed through `up dry-run`. | Improve with `dry-run` output for resolved account, owner, session, IAM role, tags, Terraform target, and mutation risk. |
| `cp9 doctor` | Preflight host, config, IAM, service inputs, schema, and readiness checks. | This is the main validation surface for example config problems; blockers must be specific enough for agents. | Improve. |
| `cp9 up` | Stand up the cp9-owned control-plane environment from config. | It currently performs Terraform apply, selected-host work, generated Tiltfile handoff, and Tilt readiness. Users need a safe preview path. | Improve with `dry-run`: stop before mutation, report Terraform preview, generated Tiltfile path, selected services, and what a real run would create. |
| `cp9 status` | Show environment status, service health, artifact paths, and cost availability. | Reviewers need session context and cost status in the same runtime report. | Keep as the reporting surface; cost visibility belongs here, not in a separate command. |
| `cp9 logs` | Fetch service logs with service, trace, time, and level filters. | Useful for failure inspection; live follow is not a supported command arg. | Keep and improve bounded output. |
| `cp9 deploy` | Trigger one generated Tilt resource after `cp9 up`. | It mutates a running session and can be confused with a broader sync/regeneration operation. | Improve with `dry-run`: show target resource, namespace, image, and Tilt trigger command without executing. |
| `cp9 db dump-init` | Dump schema and selected non-deleted rows from a real MySQL/RDS source into committed persona init SQL. | Persona owners often need to edit seed records from staging/RDS facts without generated Tilt curl jobs. | Keep as the seed-authoring surface; require reviewed SQL committed under the persona and keep runtime loading inside `cp9 up`. |
| `cp9 destroy` | Tear down local state, Terraform resources, selected host, and tagged cloud resources. | Closeout should remain one user-facing command with explicit survivor/owner/session evidence. | Keep as the cleanup surface; do not reintroduce standalone `reaper` / `reap` commands. |

Commands removed from the current delivery surface:

| Command | Code fact | Action |
|---|---|---|
| `cp9 reaper` | Removed standalone tag-scoped cleanup surface. | Keep removed. `destroy` owns Stage-2 reap behavior and survivor output. |
| `cp9 cost` | Removed standalone spend-reporting surface. | Keep removed. Cost availability/reporting belongs in `status` output. |
| `cp9 test` | Removed feature-test wrapper. | Keep removed. Feature repos and skills own direct `go test` validation instructions with `pkg/cp9test`. |
| `cp9 config validate` | Removed; it was a low-level parse/validation helper. | Keep removed from product surface; `doctor` is the validation command users should know. |
| `cp9 persona list/show/validate` | Removed; these were template helpers around `topologies/`. | Keep removed from product surface; users choose committed example configs, and skills explain that choice. |
| `cp9 db provision/destroy` | Removed owner/debug helper that provisioned one database instance and created separate service databases inside it. | Keep removed from the product surface. `up` owns the session metadb lifecycle, and `doctor/status` should explain the resolved one-instance/per-service-DB contract. |
| `cp9 crd` | Removed direct CRD/secret installer. | Keep removed from the product surface; lifecycle service startup logic in persona/app-delivery inputs covers it. |
| `cp9 migrate` | Removed legacy command that dumped schema/seed from staging DSNs and applied it to local DBs. Current v1alpha2 dogfood does not invoke it; `up` handles the metadb DSN contract, and persona app-delivery inputs carry local schema/seed material. | Keep removed from the product surface. Seed authoring belongs in `cp9 db dump-init` and reviewed persona SQL, while runtime metadb readiness belongs inside `up`, `doctor`, and `status`. |
| `cp9 sync` | Removed; it was a registered stub and never implemented. | Keep removed. It overlaps with deploy/regeneration language and should be covered by skills if agents need catch-up. |
| `cp9 onboard` | Removed; it was a registered stub and never implemented. | Keep removed. Skills cover repo/agent catch-up first. |
| `cp9 stub-gen` | Removed partial stub artifact generator. | Keep removed from the product surface. Stub generation is lifecycle-owned through `up` / `up dry-run`, with `doctor` reporting source drift. |
| `cp9 overlay` | Removed partial Kustomize overlay artifact generator. | Keep removed from the product surface. Overlay generation is lifecycle-owned through `up` / `up dry-run`. |

Generation surface:

- `overlay` and `stub-gen` were partial artifact generators and are no longer user commands.
- The comprehensive Tiltfile generator exists in code as the lifecycle's `tiltfile_generate` step and is part of `up`.
- The user/debug surface is `cp9 up dry-run`, because it follows the real `up` lifecycle and avoids creating another daily command.

`cp9 verify` and `cp9 feature-cycle` are not current commands. Tests assert they stay unregistered.

### Premium Reserved Boundary

As of PR #210:

- customer-facing OpenAPI create returns 200,
- nextgen returns `state=CREATING`,
- real cluster-ng returns `CreateTiDBCluster{phase=CREATING, version=v8.1.2}`,
- customer-facing OpenAPI get returns 200 and `state=CREATING`,
- the old localhost executor path is absent,
- the current boundary is async dataplane/provider readiness and status propagation after accepted create.

That is the first proving path because it uses real cp9-owned environment work.

## Target Workflow

```text
GitHub issue / feature spec
  -> choose cp9.<persona>.example.yaml
  -> tune ad hoc cp9.yaml
  -> cp9 setup
  -> cp9 doctor
  -> human gate if config/resource risk is unclear
  -> cp9 up
  -> cp9 status
  -> code change in feature repo
  -> feature-owned validation
  -> cp9 logs / status / doctor if failing
  -> PR with feature-owned proof plus cp9 state snippets
  -> review
  -> cp9 destroy
```

Human gates:

| Gate | Trigger | Human reviews | Agent behavior |
|---|---|---|---|
| Design | New feature path, new architecture, unclear scope | Workflow, non-goals, acceptance, risks | Stop before implementation |
| Resource risk | Expensive or risky cloud path, unclear owner/session tags, unknown staging dependency | Config, resource scope, owner, TTL | Stop before `cp9 up` |
| PR | Diff is ready | Code, feature tests, cp9 state, cleanup state | Wait for review |

Small changes inside an approved issue contract should not need a new gate.

## Feature Deliveries

### D1. Persona Example Config Experience

cp9 owners should ship example configs for daily development personas:

```text
cp9.premium-full.example.yaml
cp9.<typical-persona>.example.yaml
```

Each example should make these choices obvious:

- persona name,
- host mode,
- owner/session/TTL fields,
- AWS account allowlist and profile fields,
- source service paths,
- live service image or app-delivery source,
- stub choices,
- staging input paths, clearly marked as template/source only,
- comments or docs for fields users commonly tune.

The user-facing problem:

```text
Which example do I start from, and which fields do I change for this issue?
```

The first delivery should answer that problem before adding new commands.

### D2. Existing CLI Surfaces for Agents

The design should improve the user-facing surfaces and remove direct delivery pressure from helper commands.

| Surface | Agent-facing contract |
|---|---|
| `cp9 setup` | States missing local/cloud prerequisites before any expensive action; dry-run output names account, owner, session, IAM, tags, Terraform target, and mutation risk |
| `cp9 doctor` | States config, persona, AWS/IAM, schema, drift, and readiness blockers |
| `cp9 up` | States selected persona, session, owner/TTL, created resource scope, generated Tiltfile path, and dry-run result |
| `cp9 status` | States current runtime/session/resource state, artifact paths, and cost section |
| `cp9 logs` | Produces bounded log artifacts for a trace, service, or recent window |
| `cp9 deploy` | Triggers one generated Tilt resource; dry-run output names the resource and command without executing |
| `cp9 destroy` | Performs local cleanup plus tagged cloud cleanup; `reap` behavior is part of destroy and reports survivors |

When output is structured, it should use existing ticat/model output behavior. JSON belongs on these existing commands, not on a new command family.

Commands removed from the delivery surface are still code facts, but they should not shape the agent workflow. Feature tests, repo onboarding, persona choice, low-level config checks, database helpers, CRD helpers, staging DB dump migration, partial overlay generation, and partial stub generation either belong inside the lifecycle or inside skills.

### D3. Feature-Owned Tests

Feature developers own tests.

Examples:

- `make test`,
- Go test packages,
- a repo-specific E2E harness,
- an external test engine,
- custom report generation.

cp9 should provide environment facts that tests need:

- API endpoint,
- kubeconfig and selected Kubernetes target,
- namespace,
- state directory,
- session name,
- persona,
- relevant status/log output.

Skills may tell agents how to find and run the feature repo's test command. cp9 should not standardize all test setup, case code, or reports.

### D4. cp9 Skills Catch-Up Layer

Add a small `skills/` tree modeled after lark-cli:

```text
skills/
  get-started/
    SKILL.md
    references/
      safety-and-ownership.md
      cli-output.md
      dev-full-aws-runtime.md
  premium/       # SKILL.md + retained API operation map
  dedicated/     # SKILL.md + retained API operation map
  starter/       # SKILL.md + retained API operation map
  essential-v2/  # SKILL.md
  byoc/          # SKILL.md
```

Config selection/tuning is intentionally outside this tree: the comments in
repository-root `cp9.dev-full.remote-ec2-k3d.example.yaml` are the single
config reference.

Each skill should answer:

- when to use the skill,
- which existing cp9 commands to inspect first,
- when to use the root example config reference,
- which safety rules are mandatory,
- which human gates stop the agent,
- how to run feature-owned validation,
- what final state must be reported before stopping.

Shared safety rules:

- Staging is source/template input only.
- Any cloud-resource path requires `cp9-owner` and `cp9-session`.
- Cleanup closeout goes through `cp9 destroy`.
- Unclear goal, resource risk, or success criteria means stop and ask.
- Implementation work ends in a PR, not an attachment-only checkpoint.

### D5. First Proving Feature

Premium Reserved is the first accepted proving path:

```text
public OpenAPI create
  -> real cp9-owned Premium dataplane
  -> returned public SQL endpoint
  -> mysql select 1
```

The design should serve this dogfood path first. Do not broaden to all personas before the Premium Reserved path repeats.

## Gap Table

| Area | Current state | Target state | First delivery |
|---|---|---|---|
| Persona selection | User reads docs/thread to infer starting config | User can choose a persona example directly | Example config guide and skill |
| Config tuning | Example exists, tunable fields are implicit | Common ad hoc fields are documented and safe to change | Annotated `cp9.premium-full.example.yaml` or companion guide |
| Preflight | `cp9 setup` and `cp9 doctor` exist | Output gives actionable blockers before `cp9 up` | Improve wording/structured fields where needed |
| Dry-run | Side-effecting work is tied to direct command execution | `setup`, `up`, and `deploy` have no-mutation previews | Add dry-run behavior to those surfaces |
| Runtime state | `cp9 status` exists | Output names current session/resource state, artifacts, and cost | Clarify status output and merge cost section |
| Logs | `cp9 logs` exists | Bounded trace/service log artifacts with paths | Stabilize logs output where needed |
| Tests | Feature output plus notes | Feature repo owns test command and report | Skills point to feature-owned validation |
| Cleanup | `cp9 destroy` includes cleanup stages and the standalone reaper surface is removed | One user-facing closeout command with survivor/owner/session evidence | Make destroy output clear on survivors/owner |
| Generation | Partial overlay/stub commands are removed; full Tiltfile generation is inside `up` | Full generated Tilt handoff is available through `up dry-run` or the same lifecycle code | Keep partial generators out of the user command surface |
| Agent catch-up | AGENTS.md plus long docs | Small repo skills with references | `skills/shared`, `skills/get-started`, `skills/persona-premium-full-aws` |

## Metrics

Track these per PR-backed dogfood cut:

| Metric | Definition |
|---|---|
| Time to choose example config | Fresh agent start to correct `cp9.<persona>.example.yaml` choice |
| Time to safe config | Example config choice to tuned `cp9.yaml` ready for `doctor` |
| Time to first runtime state | `cp9 up` start to useful `cp9 status` output |
| Time to first failing boundary | Run start to precise blocker from `status`, `doctor`, or logs |
| Dogfood proof | Public endpoint returned and `select 1` executed |
| Cleanup closeout | `cp9 destroy` success or survivor/owner state recorded |
| Human interventions | Count and category of required human messages |

Keep the original less-than-60-minute O2 goal as the long-term metric. For real-cloud dogfood, optimize correct config choice, useful runtime state, and `select 1` proof first.

## Rollout

| Phase | Deliverable |
|---|---|
| 0 | Accept this design and link issue #2 to the PR |
| 1 | Improve Premium Reserved example config selection and tuning docs |
| 2 | Add safe dry-run surfaces for `setup`, `up`, and `deploy` |
| 3 | Improve `doctor`, `status`, `logs`, and `destroy` output only where the dogfood path shows gaps |
| 4 | Ship repo skills with small references and unprefixed skill names |
| 5 | Use the skills on the Premium Reserved dogfood path |
| 6 | Package plugins after CLI and skills prove useful |

## Implementation Tasks

Cut these into small issues before code work.

| Task | Priority | Human estimate | Agent estimate | Scope | Verification |
|---|---|---:|---:|---|---|
| T1 | P1 | 2h | 30m | Document how to choose and tune `cp9.premium-full.example.yaml` | Fresh agent can explain which fields to change before `cp9 doctor` |
| T2 | P1 | 3h | 1h | Add no-mutation output to `cp9 setup` and `cp9 up` | Dry-run shows account, owner, session, tags, Terraform target, generated Tiltfile path, and skipped mutations |
| T3 | P1 | 2h | 45m | Improve `cp9 doctor` output for persona/config blockers found by dogfood | Missing config/AWS/persona prerequisites produce actionable messages |
| T4 | P1 | 2h | 45m | Improve `cp9 status` output and merge cost data | Status shows session, persona, resource readiness, artifact paths, and cost section |
| T5 | P2 | 2h | 45m | Add `cp9 deploy dry-run` for generated Tilt resources | Dry-run shows target resource, namespace, image, and trigger command without executing |
| T6 | P2 | 2h | 45m | Fold standalone reaper delivery into `cp9 destroy` closeout | Destroy exposes reap behavior and reports success or survivor owner/session data through one command |
| T7 | P2 | 3h | 1h | Add initial `skills/` tree | Fresh agent can run the agreed user flow without reading this full design |
| T8 | P2 | 2h | 45m | Record DX metrics in issue/PR notes manually | Notes include config-choice time, first runtime state, dogfood proof, cleanup, and human intervention count |

## Acceptance Criteria

- The design PR adds this document and links it from issue #2.
- The document states that CLI is core and skills are the agent catch-up layer.
- Historical O2 plugin-first assumptions are deferred or superseded.
- The document includes a code-backed command inventory with keep, improve, or remove actions.
- The design leaves test execution/reporting to feature repos.
- The design keeps cleanup as one user-facing `cp9 destroy` flow.
- The design adds safe dry-run direction for side-effecting surfaces and moves cost into status.
- Premium Reserved is the first proving path.
- Follow-up issues can be cut from the gap table without hidden assumptions.
- No cp9 code changes are required in this design PR.