# Test2 standalone deployment This is the end-to-end operator runbook for the standalone Test2 stack. It joins four independent deployment boundaries without turning them into a new full-stack controller: ```text L0 process group .github/workflows/deploy-test2-l0.yml AppRelease + L2 metadata .github/workflows/manage-test2-apprelease.yml L1 AppService + UI .github/workflows/deploy-test2-l1-application.yml Integration .github/workflows/manage-test2-integration.yml ``` Dispatch all Test2 controllers from reviewed L2 `main`. Product release publishers may select `main` or an exact commit according to their owner workflow; deployment consumes only the resulting immutable release identities. Deployment never builds product code. The L0 controller resolves the selected source identity and pulls its already-published Engine Release; the AppRelease and L1 controllers consume exact artifact digests. Each workflow owns only its component. This path preserves the Test2 database. It does not refresh the database, run a full D3 deployment, apply a patch directory, or retain a previous deployment. ## Existing installation Test2 completed its one-time standalone transition in [run 33716203193](https://github.com/pingcap-inc/nexuscrm-pingcap-pack/actions/runs/33716203193) on 2026-09-03. The one-time transition controller and old Test2 D3 deployment instructions are retired. Use AppRelease `upgrade` for this existing installation. The standalone controllers refuse old full-D3 ownership and share the Test2 operation lock. Keep these guards when changing a component controller. Integration has one process owner: `manage-test2-integration.yml`, with `stop` and `deploy`. Deploy consumes an already-published Integration image digest from the selected Core Engine Release. It reconstructs the process from the canonical environment and Runtime inputs, then proves readiness. It does not select a Core branch, install metadata, or operate Runtime or L1 processes. Stop Integration before an AppRelease metadata upgrade; deploy it after Runtime activation and any separately approved schema/protocol preparation. The first deployment replaces the retained D3-created container and removes its dependency on old D3 credential paths. It does not copy credentials from that container, rotate keys, migrate data, or delete old runtime directories. Independent database maintenance belongs to `refresh-test-database-from-production.yml`. It requires standalone L0, L1 and Integration to be stopped and must not be part of an ordinary application upgrade. Read its current controller before a separately authorized refresh. The existing nginx file and frontend symlink remain in use. The Integration controller owns only the public `/webhooks/` location inside that shared file; under the common Test2 deployment lock it re-reads the live file, updates that location atomically after Integration is ready, reloads nginx, and proves the public route. Component upgrades preserve the database and existing storage authorities. ## Test2 resource mapping | Resource | Test2 value | |---|---| | GitHub environment | `test2-serviceification-d3` | | Runner label | `test2-ec2` | | Concurrency | `deploy-test-multi-slot-environment-test2` | | Environment root | `/home/ec2-user/nexuscrm-deployment/offline/envs/test/test2` | | L0 state / L1 state | `.runtime/l0-standalone` / `.runtime/l1-application` under that root | | Runtime gRPC / TLS name | `127.0.0.1:3312` / L0 `config/runtime-server-name` (existing identity) | | Public backend | Existing Test2 `BACKEND_PORT` (`3302` in the environment map) | | Public `/webhooks/` owner | Standalone Integration (`3403`) | Test2 owns nine work families, including `runtime.permission-authoring-command-lifecycle.v1`, and runs nine Worker processes. The old command-based Layout publication Worker remains retired. CRM 0.72 restores Layout administration through the synchronous `runtime.layout.authoring.v2` capability selected by the signed installation. The L1 handler calls Runtime, which checks authoring permissions and the expected workspace revision before committing the metadata transaction. Successful saves use exact committed readback and refresh Runtime-backed page projections; they do not publish Catalog, enqueue a Layout command, or require another Worker. Deploying this capability requires the selected L0 release to advertise Layout v2 and the matching CRM 0.72 AppRelease to be installed through the normal stopped upgrade sequence. Merely replacing the UI or selecting the capability against the old Runtime is not a supported upgrade. The legacy `layout-authoring` Worker is cleanup-only and must not be recreated. The L0 manager and state preparation share one process list; the signed installation input owns work-family policy and Catalog owns work-ownership activation. No second command table, lease, or ownership model is introduced. ## Source roles in the signed installation The v10 BOM selects frontend and installed metadata independently. Each role must match its own exact source and signed artifact; the two L2 commit fields are not an equality constraint. | Input | Owner and verification | | --- | --- | | `frontend.l2_commit` | Source of the assembled frontend, together with its explicit Core and L1 frontend refs; the frontend publication binds the complete source tuple. | | `l2_metadata.commit` and `tree_sha256` | Source and content of installed metadata; D1 verifies its final-pack lineage against the selected AppRelease source. | | Resolved BOM publisher commit | Source of the desired release selection; its canonical digest binds both roles and the other component selections. | A frontend-only correction can therefore use a newer generated L2 frontend snapshot while retaining the verified AppRelease and metadata snapshot. Do not relabel old metadata lineage or republish unchanged metadata merely to make the commit strings equal. A metadata change still requires a matching AppRelease, D1 proof, and the stopped upgrade sequence below. `IndependentSourceAuthorityTest` in `scripts/test_manage_test2_apprelease.py` exercises the same `load_execution_bom` path used by this controller: distinct source snapshots remain intact, and changing either role without its sealed binding is rejected. Historical release evidence may describe a candidate that used one shared snapshot; that candidate's choice is not a v10-wide constraint. ## Choose the smallest operation | Change | Required operation | |---|---| | L0 Runtime API/internal code | L0 `stop` → L0 `upgrade` | | L0 Worker image changes | L0 `stop` → L0 `upgrade`; the upgrade advances installed Worker ownership | | Missing installed Worker containers | L0 `stop` → L0 `upgrade`; retain the installed role declarations | | L1/L2 metadata or AppRelease changes | L0/L1/Integration `stop` → AppRelease `upgrade` → L0 `activate` → L1 `deploy` → Integration `deploy` with the selected digest | | L1 AppService or UI image changes | L1 Application deploy only | | Integration image changes | Integration `deploy` with its exact published digest after Runtime and schema/protocol prerequisites are ready | | First standalone Test2 installation | Use the first-install sequence below | | Database refresh, schema migration, or complete D3 replacement | Use the operation that owns that change; do not add it here | An AppRelease operation is not required merely because an L0 or L1 process image changed. An L1 Application deployment is not required merely because metadata changed. Changing one boundary must not silently redeploy another. ## Integration deployment and retry Select the Integration digest from the published Core Engine Release chosen for the rollout. For a coordinated all-component upgrade, freeze all component references together; do not resolve moving branches between operations. ```bash gh workflow run manage-test2-integration.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack --ref main \ -f operation=deploy \ -f integration_image='ghcr.io/pingcap-inc/nexuscrm-integration-service@sha256:' ``` Deploy validates the image, canonical inputs, and existing `/webhooks/` route authority before stopping the existing Integration process. It then recreates one container, checks its readiness, and atomically points only `/webhooks/` at the standalone process from a fresh read of the shared nginx file. The common Test2 deployment lock serializes this with the other component controllers. A failure after stopping leaves Integration unavailable until the same deployment is retried. There is no rollback or predecessor state; an absent container and a retry use the same reconstruction path. The process enforces its Runtime, Catalog and schema/protocol startup gates. This workflow never runs a migration. | Fact | Sole source | |---|---| | Desired Integration version | Exact published image supplied to this operation | | Database, named-credential key, ingress tuning, public URL | `env/runtime.env` | | Organization and installation | `.runtime/l0-standalone/config/serving.json` | | Integration workload token | `.runtime/l0-standalone/secrets/workload-identities.json` | | Runtime trust and server name | Existing L0 `secrets/tls/server.crt` and `config/runtime-server-name` | | Test2 process limits and endpoints | Integration controller's environment renderer | `.runtime/integration/integration.env` and `integration.token` are generated, private outputs. Never edit them as inputs. The controller mounts only the Integration token and Runtime certificate. Docker environment values and old D3 run paths are not configuration fallbacks. This is the current Test2 topology; connected-mail provider configuration requires an explicit deployment contract and is rejected before stopping a service rather than silently dropped. Use `stop` before changing metadata or running separately authorized database maintenance. After that work and Runtime activation complete, run `deploy` with the chosen image, including when its version did not change. ## First install ### 1. Publish and deploy L0 in bootstrap mode Publish the selected NexusCRM commit through the L0 owner workflow, then deploy that same source through the L2 controller: ```bash gh workflow run publish-l0-engine-dev-release.yml \ --repo pingcap-inc/NexusCRM \ --ref main \ -f source_commit= ``` NexusCRM `main` is published automatically, so the manual publication command is needed only for an exact non-main commit. ```bash gh workflow run deploy-test2-l0.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack \ --ref main \ -f operation=upgrade \ -f l0_source= ``` Bootstrap starts Runtime API and keeps the matching nine Worker image carriers stopped. It does not install application metadata. ### 2. Install the AppRelease and L2 metadata Package `nexuscrm-app-packs` `main` or an exact commit through its owner workflow: ```bash gh workflow run package-sfcrm-apprelease.yml \ --repo pingcap-inc/nexuscrm-app-packs \ --ref main \ -f source= ``` Then use the exact packaged L1 AppRelease and exact signed L2 installation input: ```bash gh workflow run manage-test2-apprelease.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack \ --ref main \ -f operation=install \ -f apprelease_reference='ghcr.io/pingcap-inc/nexuscrm-sfcrm-apprelease@sha256:' \ -f l2_installation_reference='ghcr.io/pingcap-inc/nexuscrm-serviceification-bom@sha256:' ``` This runs Controlplane, publishes the Catalog, and prepares the canonical L0 serving state. It does not operate the L0 process or deploy L1 binaries. ### 3. Activate L0 ```bash gh workflow run deploy-test2-l0.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack \ --ref main \ -f operation=activate ``` Activation starts the already-selected L0 release against the prepared state. It does not select or pull another L0 image. ### 4. Deploy the L1 AppService and UI ```bash gh workflow run deploy-test2-l1-application.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack \ --ref main \ -f operation=deploy \ -f sfcrm_app_service_image='ghcr.io/pingcap-inc/nexuscrm-sfcrm-appservice@sha256:' \ -f frontend_image='ghcr.io/pingcap-inc/nexuscrm-frontend-assets@sha256:' ``` The workflow proves the exact AppService against the live L0 before replacing the existing AppService. It publishes the matching frontend only after the new AppService is ready. ### 5. Deploy Integration Use the Integration deployment operation above with the digest from the selected Core release. First installation needs no predecessor container or D3 files. ## Upgrade and retry rules - Use the decision table above; do not replay unrelated steps. - Use AppRelease `upgrade`, not `install`, after the first installation. - Reuse the existing signed L2 installation input when neither its L2 content nor its AppRelease binding changed. Publish a new one only when one of those facts changed. - Stop L0, the L1 AppService and retained Integration before an AppRelease upgrade. The AppRelease workflow never stops or restarts either process itself. After activation, deploy the desired exact L1 images; reuse the pre-stop references when they did not change. - A Worker image change uses the stopped L0 upgrade, which advances ownership through the existing control-plane CAS and preserves installed metadata. Changing role declarations belongs to the installation owner and is a separate metadata change; a code-only upgrade never invents new installed roles. - A failed operation retains no rollback or previous-release state. Fix the owning failure and rerun the same exact desired release. - Database refresh is an explicit independent operation and is never implied by install, upgrade, activation, or L1 Application deployment. ## Success criteria The component workflow summaries are the authority for their selected immutable artifacts. After the required operations succeed: 1. L0 Runtime API and all nine Worker processes are healthy and Runtime is ready. 2. The L1 AppService is ready against the live L0. 3. `https://nexuscrm-new-test2.pingcap.com/health` returns HTTP 200. 4. Integration is ready against the new Runtime and its image digest matches the selected release. 5. `https://nexuscrm-new-test2.pingcap.com/version.json` returns HTTP 200 and the selected frontend version. Business-path inspection is separate evidence. It does not change the result of these deployment workflows. ## Component references - [L0 deployment and management](test2-l0-only-deployment.md) - [AppRelease installation and upgrade](test2-apprelease-management.md) - [L1 AppService and UI deployment](test2-l1-application-deployment.md) ### Query release acceptance `scripts/verify_query_release.mjs` is the shared read-only runner for local and Test2 query releases. It covers the original GROUP BY and wide-response cases, relative date literals, selection-field errors, relationships, and complete pagination. It records 64 cases, expected outcomes, and row-ID hashes without retaining successful record bodies. The process is bounded to five minutes. ```sh node scripts/verify_query_release.mjs --base "$ORIGIN" --output "$NEW_EVIDENCE_FILE" ``` Use the existing origin-bound NexusCRM Connect session. For an isolated local session add `--state-dir "$LOCAL_QUERY_AUTH"`. Local fixture setup must supply more than 2,000 Accounts with enough selected text to cross the old 1 MiB response boundary, and Opportunities/Contacts for grouped and relationship queries; case-local records stay outside deployment Seed profiles. Run the same runner before and after the image upgrade. Test2 uses its existing data. The new error-code assertion is `INVALID_FIELD` for root, parent, child and aggregate selection fields. Unknown objects and filter fields remain rejection regression controls. SOSL and list-view scope denials are recorded separately from the query-fix claim; this runner does not claim complete v63 parity.