# CP9 spec tests

This directory is CP9's product QA surface. Cases judge customer-visible
behavior and must be runnable through their documented entrypoints without
relying on CI to complete the journey.
Repository-wide coding, pre-push, and shared-resource rules remain defined in
[AGENTS.md](../AGENTS.md); this document defines the QA process within them.

## Scope

- **Go functional test (`*_functional_test.go`)** — one executable customer
  journey in native automation, behind the `functional` build tag, named
  `TestFunctional_...`, and run only through `make test-functional`. Its prose
  contract lives in comments beside that test instead of a companion Markdown
  file.

These journeys cross a real product boundary: CP9 commands, the public UI or API,
and real dependencies. Tests built from fabricated `.cp9` state or local
workloads are unit or component integration tests, even if they launch CP9 or
Tilt. CP9 command specs stop at command-visible runtime behavior; Dedicated,
Premium, Essential, and Starter behavior belongs to the Public UI functional
test. Local harness checks follow the root
[non-runtime-check retention policy](../AGENTS.md#runtime-and-non-runtime-check-boundary-mandatory);
they do not substitute for Product evidence.

## QA principles

Agents can misread expectations, miss defects, and repeat expensive work just
as people can. Treat tests as measurement instruments: repeatable execution
and correct acceptance criteria both need validation.

- **Automate understood work.** Use agents to clarify intent, explore boundaries,
  diagnose exceptions, and review evidence. Encode established procedures in the
  existing harness so normal runs reuse them without model interpretation.
- **Validate the judgment.** Derive expected results from the customer contract
  independently of the implementation. For a new or repaired assertion, verify
  a valid behavior it accepts and a plausible defect it rejects. Use asymmetric
  inputs and boundary cases when they expose otherwise hidden mistakes.
  Agreement between checks that share an assumption does not validate it.
- **Make learning reusable.** Deliver verified harness repairs through review
  into the version future runs use; a private workaround alone does not help
  the next run. Fix shared helpers at their common source and reuse existing
  runners and journals before adding infrastructure.
- **Measure cost without reducing coverage.** Separate product execution and
  cloud waits from evidence collection, harness repair, and agent coordination.
  Batch compatible remote reads and reuse observations only while their
  provenance and freshness remain valid. Report state changes and blockers,
  not each poll. Dropping a scenario is a coverage change, not a speedup.

## Case shape

- Give each case and its test or subtest a customer-journey name, never a
  catch-all product area such as `remote-dev-full`.
- Use a subdirectory only when one journey needs several files.
- Group compatible behaviors into ordered phases with a common build/config
  baseline, compatible starting state, and reversible mutations. Reuse one
  session and cluster.
- Use a separate case for a fresh lifecycle, incompatible config/state, or a
  phase that cannot restore a safe starting point.

Go functional journeys use explicit, named `t.Run` subtests for their phases,
not table-driven test cases. Ordinary iteration over observed resources inside
one phase is fine; keep scenario selection and acceptance steps explicit.

## Execution ownership

One parent agent owns each selected case from preparation through verdict and
cleanup. Every case declares one mode in its Go contract comments:

- **single agent** — the parent runs every phase and spawns no sub-agents;
- **parent with sub-agents** — the spec names each independent read-only check,
  its input, evidence path, and join point.

When a task assigns QA and reviewer roles, the parent is QA and owns execution
and diagnosis. The reviewer independently checks changes and evidence, decides
when a fresh session is needed, and approves, merges, or closes work within the
task's granted authority.

The parent alone mutates the checkout, runs session commands, writes the final
summary, and cleans up. Never run `cp9 up`, `cp9 deploy`, or `cp9 destroy`
concurrently against one session, and never let two agents edit one checkout.
Product-created concurrency is allowed: one agent starts it and observers only
read. At the join point, record every declared report or its `fail`/`not_run`.

Run independent selected specs in parallel when agents are available. Each
agent runs at most one active spec; each spec owns its checkout, session,
evidence path, and exact cleanup. A shared AWS account or region is not a reason
to serialize them.

## Authoring contract

Write the journey in execution order: starting condition, user action,
customer-visible observation, verdict, and cleanup. Use natural language for
product intent and put locators, parsers, polling, traps, and evidence layout in
the harness unless they are product or safety boundaries. Configuration remains
subject to the root
[control-format priority](../AGENTS.md#control-format-priority-mandatory), including
when Go orchestrates a journey.

Every spec must declare:

- exact CP9 build plus executable default config, account, region, source, and
  cleanup path;
- prerequisites and only human inputs that cannot be automated;
- mutation ownership, cleanup scope, bounded waits, and stop conditions;
- expected customer-visible results and required raw evidence;
- phase-level Product and Harness criteria, one run-level Cleanup verdict, and
  Security or O11Y criteria when needed;
- execution mode, sub-agent boundaries/join point, and explicit non-claims.

Keep a migrated journey's prose contract as comments beside its
`TestFunctional_...` entrypoint; do not also keep a same-named Markdown file.
Add code under `spec-tests/scripts/` only when it materially improves
repeatability, shared logic, fault injection, raw evidence capture, or exact
cleanup. Helpers must not add undeclared assertions, mutations, or cleanup.

The harness implements the product contract. Selecting a case authorizes
repairing a broken probe without another approval when the assertion, mutation,
and cleanup scope stay unchanged. Preserve the original evidence and repair
diff. A wider account or resource scope still needs explicit authority.

Write for an unfamiliar lower-cost agent: provide real default values, remove
unnecessary choices, name the first command, and make every completion condition
observable.

## Verdicts

Every case reports Product and Harness per phase, one run-level Cleanup
verdict, and Security or O11Y when declared. Go contract comments state these
distinctions alongside their subtest results; remaining Markdown specs end with
`## Verdicts` and the corresponding sections.

| Lane | `pass` | `fail` | `not_run` |
|---|---|---|---|
| Product | Direct evidence proves every assertion. | A customer action fails or produces the wrong result. | The path is unobserved, blocked, or stopped by a prerequisite. |
| Harness | Every required observation is collected correctly. | A locator, parser, script, tool, or evidence write is wrong or incomplete. | A prerequisite stops the case before any probe begins. |
| Cleanup | Every run-owned mutation is restored or its exact handoff is accepted. | Required restoration or an accepted handoff is missing after cleanup. | The case created no run-owned state. |

Classify a wrong or ambiguous expectation as a Spec defect and the affected
Product assertion as `not_run`. Classify an unavailable machine or human input
as a Prerequisite and any unstarted Product assertion as `not_run`.

The run-level Product summary is `pass` only when every phase ran and passed;
any Product `fail` makes it `fail`; otherwise it is `not_run`. `partial` may be
extra context, never `pass`. Direct product evidence keeps its result when a
harness later fails. A harness defect never fills an evidence gap, and internal
logs may explain but never rewrite a customer-visible result.

Bind observations to the executing revision, session, command, exit status,
and time. Distinguish the original failure, repair verification, and unobserved
assertions. Mixed-revision evidence is not a full pass on the latest revision;
filtered or skipped phases do not become passed because the process exits zero.
Preserve raw observations without rewriting values, honoring each case's explicit
credential exclusions. The GitHub App case records only its declared non-secret
credential metadata.

## Execution contract

Use the selected case's declared `make test-functional` entrypoint and opt-in.
The retained `run-markdown-case.sh` runner serves only the separate
`tools/github-credential-residue/README.md` check.

The executor:

1. Reads this README, the selected case, and its executable harness completely.
2. Starts a new run from the latest main unless the task pins another revision.
   Records that baseline, enumerates the selected cases from its tree, and
   verifies prerequisites. Continuations retain their original run identity and
   record any code/config repair instead of silently updating the baseline.
3. Records the session, cluster when present, owner, account, region, TTL, and
   mutation scope before mutation. Case selection authorizes its declared work;
   honor its stop and retention conditions without requesting the same authority
   again.
4. Checks each phase's declared starting state before running it. A prior failure
   does not block an independent safe phase; unsafe or unreadable state makes the
   affected phase `blocked` and Product `not_run`. At the first unexpected
   behavior, freeze new mutation, preserve the scene, and diagnose before
   continuing. Allow an in-flight operation to reach its bounded terminal state
   unless safety requires stopping it.
5. Repairs a confirmed harness defect in the same scene and collects missing
   evidence without repeating proved Product actions. For a Product or uncertain
   defect, create or reuse its task, diagnose and validate a scoped fix, then
   submit it for independent review and return to the case. Rebuild only for an
   uncrossable blocker or required fresh-lifecycle acceptance.
6. Records the verdicts above and follows the cleanup contract below. A retained
   scene is an unfinished run with cleanup pending, not a reason to invent a
   Product failure or a passing Cleanup receipt.

### Cleanup

Preserve review evidence and the journal outside the cleanup scope before
removing session state. Restore only run-owned mutations and perform the case's
exact cleanup when its retention conditions are resolved. Preserve the scene
while diagnosis or reviewer disposition is pending.

Record the exact cleanup receipt; for hosted cleanup, follow the root
[asynchronous handoff rule](../AGENTS.md#asynchronous-destroy-handoff-mandatory).
A replacement session uses a different name; never cancel accepted cleanup to
reuse its name. The run is complete when all phase verdicts and required cleanup
receipts are recorded. An accepted handoff does not prove cloud zero.

### Public UI functional resume

The Public UI functional test persists its non-secret stage journal at
`.cp9/<session>/spec-tests/public-ui-api-state.json`. The journal is bound to
the session and selected scenarios, records the executing commit per stage,
and stores only write-once run-owned Product IDs and connection locators. It
never stores tokens, passwords, keys, request bodies, or credentials.

The Functional test preserves its session by default, including on first red.
Set `CP9_UI_API_DESTROY_SESSION=1` only after a human, or an agent explicitly
authorized in the task thread, decides the exact session is safe to destroy. A
later run sets `CP9_UI_API_RESUME=1`, keeps the exact session and scenario
selection, and uses the documented Make target's `GO_FUNCTIONAL_TEST_RUN` to
select only the next safe Go subtests. For example:

```text
GO_FUNCTIONAL_TEST_RUN='^TestFunctional_PublicUIAPI/premium/(backup-restore|ui-metrics|delete-cluster)' make test-functional
```

A selected passed stage is reused without executing its body. A selected
`running` or `failed` stage fails closed before mutation; select a later stage
only after same-scene diagnosis establishes that doing so is safe. The failed
receipt remains in the journal, so a continuation may preserve direct Product
evidence but cannot turn the run-level verdict into `pass`. A fresh session is
required when the journal is corrupt, belongs to a different session or
scenario set, or lacks an output needed by the selected stage. A pre-Product
CP9 session created outside this functional test may set
`CP9_UI_API_ADOPT_SESSION=1` together with `CP9_UI_API_RESUME=1` only after a
read-only check proves the selected Product mutations have not run. Adoption
requires the journal to be missing and creates only a new empty journal bound
to the exact session, scenario set, and Dedicated cloud; it never imports
Product state or overwrites an existing journal.

Public endpoint, allowlist, and password writes are separate stages. Run-owned
changefeed and migration children likewise use separate create, wait/proof,
delete, and absent stages with journaled exact IDs. Other composite stages stay
fail-closed: they are never automatically rerun after `running` or `failed`.
When a stage or stored value changes meaning incompatibly, bump the journal
version instead of interpreting an older receipt under the new contract.