# E2E Test

Declarative browser tests executed by a small TypeScript agent using the AI SDK and Playwright. Business cases remain JSON; the executor asks the configured OpenAI-compatible model to choose one browser action from a fresh ARIA snapshot. Playwright and the runtime decide whether the declared evidence passes.

## Author cases with AI

Use the repository's [write-e2e-case skill](../.agents/skills/write-e2e-case/SKILL.md) to add or update coverage. For example: `Use $write-e2e-case to extend Dedicated backup settings coverage with hard evidence of the save, restoration to the shared baseline, and reruns after interruption.` The skill explains how to choose hard evidence, constrain mutations, reuse resources, order cleanup, and validate the result with the existing root commands. For tools that do not discover `.agents/skills/`, follow the skill-directory linking instructions in `AGENTS.md` or ask the AI to read the skill file directly.

## Structure

- `test/e2e-agent/cases/`: feature goals and evidence, with explicit contracts for mutations.
- `test/e2e-agent/suites/`: shared variables, resource identity protection, and execution order.
- `test/e2e-agent/tests/browser-executor.ts`: snapshots, typed browser tools, action history, and budgets.
- `test/e2e-agent/tests/agent-runtime.ts`: authentication, request guards, captured resource IDs, hard assertions, and cleanup recovery.
- `test/e2e-agent/tests/agent-cases.spec.ts`: discovery and lifecycle ordering.
- `test/e2e-agent/smoke/provider.json`: an in-memory model/browser check, separate from business suites.

A case inherits its suite from its directory. Read-only cases normally need just a goal and hard evidence:

```json
{
  "id": "starter-backup-list",
  "name": "Open the Starter backup list",
  "goal": ["Open Data → Backup and wait for the backup list to finish loading."],
  "success": {
    "evidence": [
      { "kind": "url", "matches": "/tidbs/[^/]+/backup(?:[/?]|$)" },
      { "kind": "visibleText", "text": "Backup Time" }
    ]
  }
}
```

## Execution and evidence

The runtime logs in, verifies `/api/v1/user/me`, and opens the permitted start page. If an authenticated case unexpectedly returns to login, execution stops before another model decision or queued action; only runtime setup may authenticate. The agent observes and performs one action at a time. Each element reference belongs to one snapshot revision and URL; missing or stale references fail without guessing another locator. Actions respect disabled controls and pointer obstructions. Switches can use the visible label of one scoped checkbox. Uploads are restricted to `test/e2e/serverless/dataPlatform/fixture`, including traversal and symlink checks.

Snapshots have a depth of 12. `[focused]` means keyboard focus; `[checked]` means a switch or checkbox is on. Password inputs and generated-password confirmations are redacted before model calls. The executor retains eight recent action descriptions without typed values. It exposes no arbitrary JavaScript, direct HTTP calls, guessed URL navigation, or force-click tool.

Action count and elapsed time are bounded. Model requests may retry within that deadline; browser mutations are never automatically replayed. The runtime checks whether an earlier mutation completed before executing a queued model action. An agent `stop` or exhausted action budget still requires hard assertions to pass. Independent AI-only assertions are unsupported.

Cases with distinct save/restore stages can declare ordered `phases`, each with its own `goal` and hard `evidence`. The model sees only the active phase's task. `skipIf` is evaluated once on entering that phase, so an unsaved draft cannot change the selected branch during recovery. All phases share the case's action/time budget and at most one phase recovery. Final case evidence and network guards still apply. `testIdChecked` observes a single checkbox through its visible control or label and can require it to be enabled; `acceptedRequest` requires a successful guarded response whose request body matches the declared fields. Dedicated backup settings uses these to distinguish initial state, the saved OFF state, and the final saved ON state.

Browser HTTP caching is disabled by default through the shared `tests/fixtures.ts` context route, before the first navigation. Service workers are blocked in `playwright.agent.config.ts` so they cannot serve cached assets. This applies to login, refreshes, navigation, and new pages in both local and CI runs; use the shared fixture when adding runner tests. Authentication cookies/storage may still be reused. This simulates browser asset cache misses, not a purge of Netlify's CDN or backend caches.

The known “The New Resource Center” onboarding dialog is closed through its visible Close button when it blocks an action, including when it appears late after a reload. This handler stays within the original action timeout and does not dismiss business dialogs. Feature cases may start on a verified detail route using a captured resource ID; Networking does this so its action budget covers endpoint changes instead of repeated sidebar navigation.

Cases whose initial and final UI look identical need `success.beforeEvidence` for an intermediate state. Value evidence can require `enabled: true` to reject disabled loading defaults. Starter Connection must observe loaded Public Endpoint before accepting Private Endpoint. Required mutation guards wait for successful responses. `finalRequestBodyEquals` or `finalResponseBodyEquals` verifies restored state. Data App cleanup also requires a successful unfiltered list response excluding the captured app ID: a temporarily empty sidebar cannot prove deletion.

`responseArrayContains` verifies exactly one matching API list entry and can capture its ID, including when reusing a resource. Branch creation requires the named branch's `READY` state and captures `starterBranchId`; deletion is bound to both cluster and branch IDs. For paginated `responseArrayAbsent` evidence, only the declared pagination query parameters are accepted, and the first page must contain the entire reported `total`. A partial or filtered page cannot prove absence. The disposable Branch lifecycle uses a cluster with a small complete list; successful deletion clears its captured branch ID for the next cycle. Metrics requires observing Database Status before returning to Instance Overview.

AI request deadlines, retries, and timing cover the entire non-streaming response body, not just receipt of HTTP headers. Cancelling the case aborts an in-flight provider response without another retry.

Slow Dedicated and future Premium resources must support repeated functional validation. Creation can reuse the exact named resource discovered through the UI. Backup settings repair an interrupted disabled state and finish enabled. Password tests rotate the generated password once; later cases do not depend on the old value. Cleanup remains last and is limited to the captured resource ID, even after functional failures.

Dependencies and commands use the root `package.json` and lockfile. Root `playwright.agent.config.ts` and `tsconfig.e2e-agent.json` configure the browser runner and test type checks.

## Setup

```bash
# Run from the repository root. Add settings from .e2e-test.env.example
# to your ignored .e2e-test.env without replacing existing login credentials.
pnpm install --frozen-lockfile
pnpm exec playwright install chrome
pnpm typecheck:e2e-agent
```

Set `E2E_AI_BASE_URL`, `E2E_AI_MODEL`, and `E2E_AI_API_KEY` in the root ignored `.e2e-test.env`. Login credentials use `E2E_LOGIN_EMAIL` and `E2E_LOGIN_PASSWORD`, or the existing `EMAIL` and `PASSWORD` shared with other E2E tests.

Set `E2E_TEST_ORG_ID` to pin authenticated cases and recovery navigation to a test organization. The runtime stops agent actions if the application leaves that organization. For prod previews CI pins Premium Preview (`1372813089209271775`), matching the validated lifecycle runs. Staging previews require the repository variable `E2E_STAGING_ORG_ID`; without it, the workflow fails before running business cases.

Authentication and captured IDs are cached under the ignored `test-results/e2e-agent/environments/`, scoped by target origin and resource name. Reused authentication is verified and recovers through a real login when needed. On a fresh preview origin, the login callback may land in the account's default organization; setup restores the configured start URL once and verifies the requested organization before allowing agent actions. Organization drift during a case still fails immediately. Public cases clear authentication. Login and application startup have their own bounded setup allowance, separate from the agent budget.

Read-only cases inheriting the suite start can resume from a verified resource page left by an earlier case. An explicit case `start.path` always wins over that continuation, so a direct feature route behaves the same in isolation and after another case. Validate both entry paths when adding direct navigation.

## Run

```bash
# List business cases without running them
pnpm test:e2e-agent agent-cases.spec.ts --list

# Open the generated report
pnpm exec playwright show-report playwright-report/e2e-agent

# Real model/provider, in-memory page
pnpm test:e2e-agent provider-smoke.spec.ts

# Executor and evidence regressions, no model requests
pnpm test:e2e-agent:unit

# CI's two groups
E2E_SUITES=public,starter E2E_STARTER_CLUSTER_NAME=e2e-starter-unique \
  pnpm test:e2e-agent agent-cases.spec.ts
E2E_SUITE=dedicated E2E_DEDICATED_CLUSTER_NAME=e2e-dedicated-unique \
  pnpm test:e2e-agent agent-cases.spec.ts

# Create/reuse once, run Dedicated functional cases twice, delete once
E2E_STABILITY_CYCLES=2 \
E2E_SUITE=dedicated E2E_DEDICATED_CLUSTER_NAME=e2e-dedicated-unique \
  pnpm test:e2e-agent agent-cases.spec.ts

# Directed rerun on an already captured resource
E2E_CASES=dedicated-backup-settings,dedicated-restore \
E2E_DEDICATED_CLUSTER_NAME=e2e-dedicated-unique pnpm test:e2e-agent agent-cases.spec.ts
```

Mutating cases require explicit case or suite selection. `pnpm test:e2e-agent agent-cases.spec.ts` without selection skips them. `E2E_CASE_RETRIES=0` disables Playwright retries. Use disposable resource names. Dedicated creation has a 65-minute readiness budget; after the accepted create request, the runtime owns the long wait.

Public covers registered-user and SSO entry journeys. Starter covers lifecycle, connection/password, backup/restore, metrics, slow query, import, Data App, networking, and Branch lifecycle. Dedicated covers lifecycle, connection/password, backup settings/list, and restore entry/history. Existing skips remain explicit in JSON: Starter/Dedicated Events, Starter SQL Editor, and Dedicated Changefeed.

## CI and artifacts

The `E2E Test` workflow has two independent jobs: Public + Starter, and Dedicated. Internal PRs targeting `master` require the `e2e-test` label to run both jobs against the prod Netlify preview. Adding that label starts a run; subsequent commits, reopening, or retargeting to master also run while the label is present, regardless of changed paths. PR title/body edits and unrelated label additions do not rerun business tests. There are no group or skip labels. PRs targeting `dev` do not automatically run this workflow; staging previews remain available through manual dispatch.

`Publish Release` explicitly dispatches this workflow from the release branch, using the master PR's prod preview URL. Bot-triggered PR events skip the automatic jobs to avoid duplicating that dispatch. The release script adds `e2e-test` to the master PR. The legacy workflow only handles automatic PR runs against dev, so this label does not start both frameworks on a master PR. The workflow and runner must be present in dev before cutting the release branch. Legacy workflows and the new workflow remain manually dispatchable.

Concurrency is applied to each job, rather than the entire workflow, so the two checks can appear on the PR while waiting. Runs from the same branch serialize Public + Starter separately from Dedicated, including manual runs. A running job is never canceled by a newer commit, allowing resource cleanup to finish. Only the latest pending run for each group is retained. A completed Public + Starter group can start its next run while the previous Dedicated group finishes.

Each job checks out the PR head SHA and waits up to 20 minutes for that SHA's `netlify/<site>/deploy-preview` commit status to succeed. A failed deployment fails the job before resource creation. The shared `.github/actions/e2e-preview` action resolves the deployment ID from Netlify's preceding pending status and tests its immutable URL, so later pushes cannot change the frontend midway through a run. It does not fall back to the live site or an older successful status. The job summary records the tested SHA, fixed URL, PR alias, and deployment record.

For a manual run, select the PR branch under **Use workflow from** and enter its Netlify preview URL in `testUrl`. Both jobs run against the successful deployment for that selected commit. A PR alias or the corresponding immutable deploy URL is accepted; the live site and URLs belonging to another commit are rejected. Authentication uses the matching backend environment, while cleanup remains scoped to that run's owned resources.

The design follows the existing `e2e-test.yml` PR/environment mapping with two changes: it waits for deployment success instead of the `Pages changed` check (which can legitimately conclude `neutral`), and uses a fixed deploy URL instead of a mutable PR alias. [Netlify documents commit-status deployment notifications](https://docs.netlify.com/deploy/deploy-notifications/) and [immutable deploy permalinks](https://docs.netlify.com/deploy/deploy-types/deploy-previews/). A [`deployment_status` trigger](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#deployment_status) would avoid polling, but the current Netlify integration publishes commit statuses/checks, not GitHub deployment records. Using those existing notifications avoids adding a webhook, token, or Netlify plugin just to start E2E.

CI uses `EMAIL_PROD`/`PASSWORD_PROD` for prod previews, `EMAIL`/`PASSWORD` for staging previews, and the shared `E2E_AI_API_KEY`. Repository variables can override the model and endpoint. Each workflow attempt uses unique resource names. Reports and failure traces are uploaded as artifacts. Logs, reports, diagnostics, authentication, and captured IDs must not be committed. Diagnostic mode logs action kinds, boolean switch targets, request sizes, status codes, and timings; it omits prompts, typed values, reasoning, and response bodies.