# Staging Inspection This directory contains a role-first staging inspection system for NexusCRM. Hierarchy: 1. role 2. module 3. function Goals: - prove the selected shared environment can complete baseline online workflows - surface 500/504 style failures with request evidence - keep the report readable for business stakeholders - stay separate from Watchdog naming and case DSL - prefer API execution over UI whenever the business conclusion does not require a UI step - reuse the current production-mirrored staging dataset for read-oriented business probes instead of creating new seed data - use case-local synthetic records and bytes for lifecycle acceptance whose result must not depend on cloned L3 data, including Contract and Order file upload/download/delete proof Connector side-effect isolation: - API-script `POST` creates for `Lead`, `Contact`, and `Account` default `HubSpot_Sync_Enabled__c` to `false`, so ordinary CRM inspection data cannot enqueue external HubSpot traffic in the production-mirrored staging database. - A connector-specific case must opt in explicitly by setting `HubSpot_Sync_Enabled__c` to `true` in its request body. Explicit `false` and `true` values are preserved by the runner. GitHub workflow policy: - every successful staging deployment workflow dispatches a full staging inspection - deploy-triggered and manual inspections run the reviewed workflow from `main`; inspection proves the live target's business behavior and does not accept or validate a backend build SHA - staging inspections share the staging deployment concurrency group so their mutations cannot overlap a deployment - staging inspection binds the installed Salesforce-compatible programmatic OAuth client, quiesces both active HubSpot inbound secret values through the active slot's encrypted database boundary, and restores the real values after success, failure, or cancellation - focused inspections patch migration-gated permissions only for Marketing or SDR scopes; unrelated cases do not depend on those preview-only mutations - functions marked `manualOnly: true` are excluded from default, role-prefix, and module-prefix runs; they execute only when their exact `role/module/function_id` key is selected - scheduled patrol inspections are disabled - manual dispatch can target `staging`, `test1`, `test2`, or `test3` ## Session lifecycle validation `.github/workflows/staging-session-lifecycle.yml` is the manual, staging-only validation surface for proof-bound first-party browser sessions. It shares the staging deploy concurrency group, runs only on the `staging-ec2` runner, and fails closed unless its checkout is the current `origin/main`, the nginx-selected active slot is healthy, and that slot's runtime refs match the committed final pack lineage. The workflow can run all lifecycle cases or one focused case: - `active-renewal`: a near-expiry session rotates its bearer, extends durable expiry, adopts the replacement in browser storage, and keeps one stable proof cookie - `app-client-activity-renewal`: a login-created document uses only SPA navigation and the shipped CRM app client to create and then query one runner-owned Product record; the mutation and query are shortened and proven independently to send the exact session intent, adopt distinct replacement bearers, and extend the matching durable proof-bound session without a host userinfo request - `idle-expiry-ui-logout`: an expired durable session receives `401`, clears browser token storage, and returns to the login UI - `lost-response-recovery`: a committed replacement whose response is discarded is recovered on the next app load by the matching proof cookie - `superseded-bearer-without-proof`: a stale bearer in a same-origin context without the proof receives `401`, no replacement header, and no proof cookie - `concurrent-logout-wins`: renewal racing proof-bound logout leaves the durable session revoked, clears its verifier, rejects every observed bearer, and returns the browser to login This is deliberately not a generic inspection DSL. The runner creates a fresh admin browser session per case and the Python helper exposes only fixed DB operations: bounded snapshot, fixed case expiry shortening, exact session cleanup, exact runner-owned Product cleanup, and nonsecret-ledger readback. Every session mutation is tied to the case JTI, user, User-Agent, UI client, password grant, recent creation window, current bearer, and proof hash as applicable. Product cleanup requires the fixed `Product2` object, exact ID when available, exact run-unique name, exact creating user, and recent creation window; it fails closed on ambiguity. Cleanup never targets all sessions for a user or all records for an object. A pre-create ledger and bounded settle retry also cover a server commit racing a lost response or browser abort. Bearer tokens and the raw proof stay in memory. Node passes them to the DB helper only on stdin; the DB password stays in an allowlisted process environment. Artifacts contain only sanitized JSON/Markdown plus a cleanup ledger limited to the bounded session selector and the exact runner-owned Product selector. The workflow always reruns bounded cleanup and retains uploaded artifacts for seven days. Focused contract tests: ```bash cd staging-inspection/runner npm run test:selection npm run test:redaction npm run test:session-lifecycle ``` Current default roles: - `admin`: baseline platform and cross-object checks - `sales`: sales-specific access and workflow checks - `presales`: pre-sales specific access and workflow checks Runner output: - JSON report - Markdown summary - screenshots on browser failures - per-check request/response artifacts for API failures Artifact security contract: - authenticated API responses stay raw only in memory while assertions run - every persisted JSON result, diagnostic, and final report is sanitized at its write boundary - `Authorization`, cookies, refreshed bearer headers, OAuth token/secret fields, browser control proofs, form-encoded revocation tokens, and JWT-shaped strings are redacted recursively, including retry attempts and error text - request IDs, traceparents, timestamps, and explicit hash/fingerprint prefixes remain available for correlation - multipart file requests persist only filename, media type, byte count, SHA-256, and bounded form-field summaries; raw upload/download bytes and `FormData` never enter artifacts - reusable Playwright storage state is held in memory only; it is never written under the uploaded run directory - screenshots must not intentionally render credentials; add new artifact formats through the same centralized redaction boundary Validate the redaction contract locally with `npm run test:redaction`. ## Public Connected App smoke `admin/setup-security/public-bi-external-records-read-api` is a required default `api_script` case for #5863/#5868. It creates a run-owned Account with connector sync disabled, completes Public BI Authorization Code + S256 PKCE, and uses the issued `external.records.read` token to query/queryAll/read the exact Account and describe Account, Opportunity, Contact, Task and Order. PATCH must return 403 with the D3 `FORBIDDEN` error, and administrator readback independently verifies that the record was not changed. After refresh, the public token must still read the exact record and reject PATCH; another administrator readback checks no mutation. Refresh may rotate the refresh token or retain the original token. Cleanup deletes and purges the Account, revokes case-owned credentials, and verifies that revoked refresh tokens cannot mint credentials again. The auth-only `public-bi-connector` role declares `oauth.grantType`, `loginRole`, `clientId`, `redirectUri` and `scope`. The runner uses the existing Playwright login helper for `loginRole`, then the real authorize/resume/token endpoints. Ordinary API steps keep using the workflow's programmatic OAuth client; that credential is never substituted for the browser or public session. The role has no modules of its own. `refreshAuth: true` explicitly refreshes an already-created public session before a step; an unexpected public 401 fails without reminting. This case needs Chromium, the existing admin login credentials, and the normal `payloads.accountCreate` fixture. Missing prerequisites fail the case; there is no optional enable marker or `requiredFixtures` skip. Tokens, codes and browser storage stay in memory. The API report includes nonsecret authentication checks alongside normal request assertions and cleanup results. A successful administrator setup/readback is never public-access acceptance. Every supported public read expects 200 and checks content. An unexpected 403 on those reads fails the smoke (the #5863 regression), whereas 403 on the two explicit PATCH denial steps is expected. A wrong error payload, changed record, refresh failure, or effective refresh after revocation also fails. On the unfixed Test2 target the expected verdict is **failed**, at the affected public read; only rerunning the same case after repair can establish recovery. Test2 also accepts a manually selected repository branch in Env Inspection, so this case can reproduce a deployed failure before its test PR is merged. In Actions choose the test branch, `target_env=test2`, and the exact case key above. The checkout SHA in the report identifies the test code; selecting that branch does not deploy it to Test2. Other targets retain their main-only policy. Focused Test2 execution from `staging-inspection/runner`: ```bash npm run inspect -- \ --base-url https://nexuscrm-new-test2.pingcap.com \ --backend-base-url https://nexuscrm-new-test2.pingcap.com \ --case admin/setup-security/public-bi-external-records-read-api ``` Use the target's installed programmatic client credentials for ordinary API steps, just as Env Inspection does. `npm run test:public-oauth` executes the real case with Chromium and an HTTP test peer, including public-read/Describe 403, wrong record, wrong scope/state, 401, write acceptance, false denial with a mutation, wrong denial errors, refresh scope widening/read failure/write acceptance, ineffective revocation, and failed cleanup. Healthy rotating and non-rotating refresh responses both pass. The peer validates the complete Account wire payload and the failing public step, not only the overall verdict. CI runs this contract check after installing Chromium. It proves runner behavior; the target case execution is the deployment E2E evidence. Local run: ```bash cd staging-inspection/runner npm install npm run inspect -- --dry-run npm run inspect -- --role admin npm run inspect -- --case admin/crm-core/lead-list ``` Explicit target run: ```bash cd staging-inspection/runner npm install npm run inspect -- \ --base-url http://127.0.0.1:5177 \ --backend-base-url http://127.0.0.1:3103 \ --case admin/crm-core/lead-list ``` Notes: - default with no extra target args is shared staging - passing `--base-url` and/or `--backend-base-url` switches execution to the explicit target you provide - browser flows use `base_url` - API/auth/bootstrap flows use `backend_base_url` - if your local frontend and backend share one host/port, you can pass the same URL to both - `--case` uses the canonical key `role/module/function_id`; prefer it for issue closeout proof because `--function` can match duplicate function ids across roles/modules Useful environment variables: ```bash STAGING_INSPECTION_BASE_URL=https://nexuscrm-new-staging.pingcap.com STAGING_INSPECTION_BACKEND_BASE_URL=https://nexuscrm-new-staging.pingcap.com STAGING_INSPECTION_TARGET_ENVIRONMENT=staging STAGING_INSPECTION_ADMIN_PASSWORD=... STAGING_INSPECTION_ROLE_PASSWORD=... STAGING_INSPECTION_DEFAULT_PASSWORD=... STAGING_INSPECTION_FIXTURES_FILE=../fixtures/staging-fixtures.json STAGING_INSPECTION_FIXTURES_JSON='{"search":{...},"records":{...}}' ``` Role bootstrap: ```bash cd staging-inspection STAGING_INSPECTION_ADMIN_PASSWORD=... STAGING_INSPECTION_ROLE_PASSWORD=... node scripts/bootstrap-role-users.mjs ``` Roster approver permission guard (upstream NexusCRM#4148): ```bash cd staging-inspection STAGING_INSPECTION_BASE_URL=... STAGING_INSPECTION_ADMIN_USERNAME=... STAGING_INSPECTION_ADMIN_PASSWORD=... \ node scripts/check-approver-object-read.mjs ``` Asserts every Region_All__c roster approver/ops user has effective object-level Read on Credits__c (an approver without it can still approve, but terminal approval notifications silently drop). Self-skips on environments without a roster; runs automatically as a workflow step after each inspection. Fixture model: - use current staging data for ordinary read-oriented probes - create and clean up case-local L3 data when the acceptance claim owns a mutation lifecycle; never bind current-product file acceptance to production-cloned records or object bytes - prefer queryable business anchors over raw record ids - keep local fixtures focused on anchor hints, stable keywords, and fallback ids only when needed - keep role login identities in `STAGING_INSPECTION_*_USERNAME`; `security.existingPresalesUsername` is only an existing business-data/role anchor and is never the pre-sales inspection login (`security.presalesUsername` is accepted only as a legacy input alias) - use the runner-provided 16-character `__runToken` instead of the full `__runId` in length-bounded fields such as `LastName` - workflow can also inject the fixture document through `STAGING_INSPECTION_FIXTURES_JSON` - checks that require missing fixtures are reported as `skipped`, not hard-failed - readiness drift should be reported separately from product regression Historical Salesforce file checks are exact-case, `manualOnly` migration verification. They remain skipped unless the injected fixture provides `migration.salesforceFiles.` with an external `authorizationReference`, exact `parentId` and `fileId`, plus expected filename, content type, size, and SHA-256. The download is fully streamed and only its size and digest enter the artifact. These values must come from an authorized migration artifact, never from a checked-in production record or bucket listing. Files: - `checklist.md`: issue coverage checklist - `docs/api-first-r17-integration.md`: API-first integration policy for issue `#2466` - `config/roles.json`: role > module > function inventory - `config/r17-migration-backlog.json`: story / AC migration backlog for issue `#2466` - `config/fixtures.example.json`: sample staging fixture file - `config/fixtures.sample.staging.json`: concrete sample fixture captured from current staging - `config/staging-fixtures.template.json`: fill-in template for real staging data - `runner/`: Playwright + HTTP runner Runner primitives: - `http_probe`: endpoint reachability - `api_create`: single create + optional readback - `api_script`: multi-step API scene using the existing authenticated request pipeline. A step may declare one of `body`, `bodyText`, or the bounded record-file `multipart` shape; multipart fixes the file field to `file`, accepts only `parentId`, `parentObjectApiName`, and optional `attachmentKind`, and lets the HTTP client generate the boundary. A case that creates disposable records can declare `cleanupSteps`; the runner executes every cleanup after the main steps finish or fail and reports cleanup failures. Guard each record-specific cleanup with `whenPathExists`, and use `expectStatuses` when more than one exact response is valid (for example `[204, 404]` for an idempotent delete). Independent cleanup steps continue after a sibling failure by default. A cleanup step that proves whether later destructive steps are safe must declare `cleanupFailurePolicy: "abort"`; if it fails, the runner records all remaining cleanup steps as skipped and performs none of them. State-dependent cleanup must branch from the already-proven response with `whenPathEquals: { "path": "...", "equals": "..." }`; do not issue a second filtered query and treat an empty or malformed result as proof of a terminal state. `eachItemPathExists` can validate every row in a zero-or-more query before conditional per-row cleanup runs. Assertion operators form a closed schema: unknown or misspelled keys fail instead of becoming no-op assertions. Use `length` for exact cardinality and `lengthAtLeast` / `lengthAtMost` for bounded cardinality. Recovery discovery must use case-owned identity (for example the returned ID, run-token filename, or synthetic parent ID), not metadata values under test: a product defect must not filter its own resource out of cleanup. Runtime Files intentionally keeps a `Deleted` metadata tombstone after removing object bytes, so prove that no case-owned file remains outside that terminal state before deleting a synthetic parent; do not treat the tombstone as an orphan or route it through the record Recycle Bin. - `browser_flow`: minimal UI verification Replay-safety contract: - `GET` and `HEAD` are read-only by their HTTP contract and need no additional metadata. - A non-standard read such as a POST search/report request must declare `"replaySafety": "read-only"` on the request. - An operation whose repeated application has the same durable effect may declare `"replaySafety": "idempotent"`; the checked-in use is record cleanup DELETE with both `204` and `404` accepted. - A POST/PATCH/DELETE without one of those endpoint contracts must not declare `retry`, enable `infraRetry`, or use multiple `performanceSamples`. If a write needs time to settle, send it once and poll the resulting state with GET. - The request-level `503 + Retry-After` policy is independent: that exact signal proves the rejected attempt never entered business execution and may replay any method within its fixed bounds. Timeout, connection loss, plain `503`, failed assertions, and other ambiguous outcomes never acquire mutation replay safety. - `infraRetry.safe` and URL-pattern inference are forbidden parallel sources of truth. The runner validates the single `replaySafety` fact before any request is sent.