# PRD: cp9 Black-Box Proof Workflow Status: PROPOSED Date: 2026-06-19 Owner: cp9 Related context: - GitHub tracker: #518 - Issue train: #519, #520, #521, #522, #523, #524 - [`20260615-prd-cp9-cloud-dev-integration-substrate.md`](./20260615-prd-cp9-cloud-dev-integration-substrate.md) - [`20260616-adr-session-tiltfile-runtime-graph.md`](./20260616-adr-session-tiltfile-runtime-graph.md) ## Problem Statement cp9 can now run the Premium full persona through meaningful local and remote proof paths, but it still behaves like a set of expert-operated primitives rather than a black-box product proof workflow. During the Premium full local and remote reruns, successful delivery depended on manually stitching together several independent evidence streams: - `setup`, `doctor`, `up`, Tilt, OpenAPI trains, and cleanup logs did not produce one canonical proof artifact. - Local persona fidelity gaps surfaced as downstream product errors such as `403`, `404`, and `500` rather than as named missing contract checks. - Remote mode failures were often presented as generic image, Tilt, kubeconfig, or service errors even when the actionable cause was an execution-boundary problem such as source sync, module proxy freshness, remote image build, or remote k3d import. - Cleanup evidence required expert interpretation because Resource Groups Tagging may report stale deleted resources while direct AWS service APIs show no live survivors. - `cp9 status` reports lifecycle and Tilt details, but it does not separate lifecycle readiness, Tilt readiness, proof readiness, product route readiness, and cleanup truth. - The shipped skills guide agents and humans through the workflow, but the workflow still relies on remembered operating conventions instead of a single command/product contract. The result is that cp9 can prove serious product behavior, but a non-expert user cannot confidently answer: 1. Is this session safe to start? 2. What exactly did this proof prove? 3. Which phase failed, and what class of action fixes it? 4. Are all cloud resources gone? 5. Which local persona dependencies are real, stubbed, or acceptance-critical? 6. In remote mode, which machine built, imported, cached, and ran the workload? ## Goals Turn cp9 into a black-box proof workflow where the CLI and shipped skills agree on the same evidence model. The first product line is Premium full because it has the current deepest proof history. The contracts should be generic enough for Starter/Essential v1 and Dedicated to adopt after Premium proves the shape. The highest-leverage outcomes are: 1. Persona-aware readiness through `cp9 doctor`. 2. A first-class proof runner and proof manifest. 3. Cleanup truth based on direct service APIs, not stale tag discovery. 4. Layered status that reports lifecycle, Tilt, proof, product, and cleanup readiness separately. 5. Local persona fidelity contracts that expose missing seed/schema/stub/live dependencies before product traffic reaches opaque failures. 6. Remote host-mode abstractions that make source sync, module proxy, image build/import, kubeconfig, and tunnel failures actionable. ## Non-Goals - Replacing Session Tiltfiles as the persona runtime graph owner. - Rewriting every OpenAPI curl train. - Making final Premium product acceptance easier than the current requirement: public OpenAPI or gateway create, real cluster, returned SQL endpoint, host `mysql select 1`, and cleanup. - Treating staging as the runtime success path. Staging remains source evidence and template input only. - Replacing `cp9 destroy` with ad hoc manual cleanup commands. ## Product Terms | Term | Meaning | | --- | --- | | Doctor readiness | No-mutation preflight verdict for local tools, config, credentials, account, persona contract, and selected host mode. | | Lifecycle ready | cp9 substrate/session state exists and `cp9 up` reached the lifecycle handoff point. | | Tilt ready | Tilt is running against the selected session and all required Tilt resources are ready/up-to-date for the chosen proof scope. | | Product route ready | The persona's public or local product entrypoint, gateway, backend route, plugin/auth shape, and target service path are reachable enough for the selected proof. | | Proof ready | The selected proof train has run to a terminal pass/fail state and emitted a proof manifest. | | Direct-zero cleanup | Direct service API verification shows no live current-session survivors for the resource classes the session can own. | | Stale tag warning | Resource Groups Tagging reported mappings that direct service APIs say are already gone; this is diagnostic, not cleanup truth. | | Failure class | A stable, action-oriented category such as `source-profile-missing`, `module-proxy-stale`, `image-import-host-mismatch`, or `iam-delete-conflict`. | ## Solution ### 1. Persona-Aware Doctor `cp9 doctor` becomes the authoritative no-mutation readiness surface for a selected persona and host mode. For Premium full, doctor must check: - owner, session name, TTL, account allowlist, region, and tag inputs; - local binaries and versions required by the selected path; - AWS profile/source profile/role assumptions and GitHub workflow permission where cleanup handoff is configured; - source repository paths, required refs, and private-module access; - local MySQL and persona seed prerequisites that are acceptance-critical; - local-k3d host prerequisites when `host.mode=local-k3d`; - remote SSH, remote Docker, remote k3d, remote kubeconfig/tunnel, remote image build/import, source sync, and module proxy/cache facts when `host.mode=remote-ec2-k3d`; - persona fidelity contract checks for real local services, stubs, required schemas, seed data, gateway plugins/routes, and proof dependencies. Doctor output must use stable categories and remediation-oriented messages. Human-readable output can be concise, but JSON output must preserve enough structure for skills and proof manifests to consume. ### 2. Historical First-Class Proof Runner And Manifest Status: superseded. The first-class proof runner was removed; `cp9 status` continues to project historical proof-summary artifacts when present. The original proposal added a proof command family, starting with Premium full: ```bash cp9 --trains D,N --host local-k3d --config cp9.yaml cp9 --trains D,N --host remote-ec2-k3d --config cp9.yaml ``` The first implementation may wrap existing `setup`, `doctor`, `up`, Premium OpenAPI trains, and `destroy` rather than rewriting them. Every proof run emits: - `proof-summary.json` as the canonical machine-readable manifest; - `proof-summary.md` as a human-readable report; - partial summary artifacts even when the proof fails. The manifest records at least: - cp9 commit, relevant source repo commits, config digest, owner/session, account, region, host mode, and timestamp; - stage states for setup, doctor, up, Tilt, product route checks, train execution, and cleanup; - Tilt endpoint, namespace, resource readiness totals, and not-ready resource names; - train names, request counts, failed request details, created resource ids, endpoint/SQL evidence, and artifact paths; - cleanup workflow URL/result, direct-zero status, stale-tag warnings, failure phase, fallback result, and final survivor verdict. ### 3. Cleanup Verifier Productization Cleanup closeout must treat direct service APIs as survivor truth. Resource Groups Tagging remains useful for discovery and broad scans, but stale tag mappings must not override direct API evidence that the resource no longer exists. If direct verification reaches zero live survivors, stale tag entries are warnings. `cp9 destroy` closeout should include: - local state/Tilt/host/Terraform destroy status; - delegated cleanup workflow URL, run id, conclusion, and failed phase if any; - fallback result when workflow dispatch or workflow cleanup fails; - direct-zero summary path and resource-class counts; - stale-tag warning counts and examples; - final cleanup verdict. Eventual dependency races such as IAM `DeleteConflict` and VPC `DependencyViolation` should share retry/settle behavior instead of producing one-off proof failures that require manual cleanup scripts. ### 4. Layered Status `cp9 status` must separate readiness layers instead of presenting one flat status. Required top-level layers: - lifecycle; - selected host mode; - Tilt; - product route; - proof; - cleanup. Premium status should include the public OpenAPI route, Kong/backend route, source/restored TiDB state when known, SQL endpoint readiness when known, last proof summary path, and last failure class. Status should read persisted session/proof artifacts by default. Expensive live refresh should be explicit, for example through a future `--refresh` flag. ### 5. Local Persona Fidelity Contract Each persona must document and test its fidelity boundary. For Premium full, the contract must distinguish: - real local services; - source-built services; - live cloud-backed services; - deterministic stubs; - acceptance-critical schema and seed data; - diagnostic-only helpers. The first Premium contract should cover the dependencies that repeatedly caused opaque runtime failures: - Account seed/schema/RBAC/resource-relations/API-key shape; - Billing plan endpoint and seed shape; - KMS local stub and nextgen wiring; - Kong plugins/routes/backend references; - metadata seed/schema required by Train D and Train N; - source/restored TiDB lifecycle evidence required by the proof train. Contract tests and doctor checks must fail before product traffic reaches a generic `403`, `404`, or `500` whenever the missing dependency is predictable. ### 6. Remote Host-Mode Abstraction Remote mode must make the execution boundary explicit and stable. For `remote-ec2-k3d`, the remote host owns: - source sync destination; - module proxy/cache stamp and invalidation; - Docker pulls and source image builds; - k3d image imports; - k3d runtime; - kubeconfig and API tunnel facts; - remote proof artifacts that are copied back to the local controller. The local machine remains the controller/editor. It must not silently use local Docker Desktop or local k3d as part of the remote workload build/import/runtime path. Remote failures should map to stable classes, including: - `source-sync-failed`; - `module-proxy-stale`; - `remote-image-build-failed`; - `remote-image-import-failed`; - `remote-k3d-unreachable`; - `remote-kubeconfig-unreachable`; - `remote-tunnel-unreachable`. ## Skill Requirements The shipped skills are part of the product surface. They must consume the same CLI contracts instead of duplicating tribal knowledge. Required skill updates: - `skills/shared/SKILL.md`: define direct-zero cleanup, layered status, and proof manifest as the default evidence path. - `skills/get-started/SKILL.md`: route ambiguous proof requests toward the proof runner when a persona and train are selected. - `skills/persona-premium-full-aws/SKILL.md`: require persona-aware doctor before `up`, use proof manifest for reporting, and use direct-zero closeout. - `skills/premium-openapi/SKILL.md`: when request records are produced by the historical proof runner, prefer the proof manifest and train artifacts as the debug entrypoint. - Starter/Dedicated skills should adopt the same evidence vocabulary after the Premium implementation proves it. ## Issue Train The implementation should use narrow PRs. Each issue is independently reviewable, but later issues may depend on artifacts from earlier ones. ### Issue #519: Persona-aware doctor contract for Premium full Deliver `cp9 doctor` categories and Premium full checks for local and remote host modes. Acceptance: - Doctor JSON exposes stable check categories, status, message, remediation, and host mode. - Premium full local and remote fixture tests cover pass and fail cases. - Failures for missing source profile, wrong account, missing required binary, missing Premium fidelity dependency, remote k3d unreachable, and module-proxy/image-import mismatch are action-oriented. - The relevant skills require doctor before mutation and explain failure classes. ### Issue #520: Premium proof runner and proof manifest Historical deliverable: Premium proof runner and canonical proof summary artifacts. The runner has since been removed. Acceptance: - The command can wrap existing setup/doctor/up/train/destroy flows. - Successful and failed runs emit `proof-summary.json` and `proof-summary.md`. - Manifest schema includes config digest, commits, host mode, Tilt readiness, train records, created resources, endpoint/SQL evidence, cleanup result, and artifact paths. - At least one successful fixture and one failure fixture validate the schema. - Premium OpenAPI and get-started skills use the manifest as the reporting and debug entrypoint. ### Issue #521: Direct-service cleanup verifier and destroy closeout Make direct service verification the cleanup truth and improve destroy closeout. Acceptance: - Direct-zero verifier covers the session-owned AWS resource classes cp9 can create. - Resource Groups Tagging stale mappings are warnings when direct service APIs prove not-found/zero. - Destroy closeout includes workflow URL/result, failure phase, fallback result, direct-zero summary path, stale-tag warning, and final verdict. - IAM `DeleteConflict` and common VPC dependency races use shared retry/settle behavior with regression coverage. - Skills require direct-zero summary for cleanup claims. ### Issue #522: Layered `cp9 status` Expose lifecycle, host, Tilt, product, proof, and cleanup status separately. Acceptance: - Text and JSON output clearly separate lifecycle, Tilt, product route, proof, and cleanup layers. - Premium status includes public OpenAPI route, Kong/backend route, source/restored TiDB state when available, SQL endpoint readiness when available, last proof summary path, and last failure class. - Status reads persisted artifacts without expensive live refresh by default. - Golden tests cover no session, lifecycle-only, Tilt-not-ready, product-route-failed, proof-passed, and cleanup-direct-zero states. ### Issue #523: Premium local fidelity contract Encode what Premium local mode treats as real, source-built, live, stubbed, or acceptance-critical. Acceptance: - A committed Premium fidelity contract lists real local services, source-built services, live cloud services, stubs, required schemas, required seeds, and diagnostic helpers. - Contract tests fail when Account/Billing/KMS/Kong/metadata proof dependencies are missing. - Doctor surfaces predictable fidelity gaps before product traffic reaches opaque `403`, `404`, or `500` responses. - Skills describe the current fidelity level before running Premium proof trains. ### Issue #524: Remote host-mode abstraction Make remote source sync, module proxy/cache, image build/import, kubeconfig, and tunnel behavior explicit and reusable. Acceptance: - Remote host execution facts are persisted and shown in proof/status output. - Remote source sync and module proxy freshness use a stable stamp contract. - Remote image build/import never silently uses local Docker/k3d for `remote-ec2-k3d`. - Failure classes distinguish source sync, module proxy, image build, image import, k3d reachability, kubeconfig, and tunnel errors. - Local-k3d and remote-ec2-k3d proof runs share the same proof command surface. ## Delivery Order 1. Publish this PRD and create the linked issue train. 2. Implement issue #519. 3. Implement issue #520. 4. Implement issue #521. 5. Implement issue #522. 6. Implement issue #523. 7. Implement issue #524. Issues #519, #520, and #521 are the first milestone because they turn cp9 from a set of expert-interpreted logs into a proof workflow with actionable readiness and cleanup evidence. Issues #522, #523, and #524 deepen the workflow after the main proof and cleanup artifacts exist. ## Open Questions 1. Should the proof runner be a top-level command immediately, or should the first implementation hide behind `cp9 premium proof` until the command vocabulary is proven? 2. Should proof manifests be considered stable enough for external consumers in the first implementation, or marked experimental while the schema is refined? 3. Which resource classes must be included in the first direct-zero verifier before the cleanup closeout can be trusted for Premium full? 4. Should remote proof artifacts always be copied back to the local session root, or should large artifacts stay remote with only a manifest pointer?