# Test2 L0 deployment and management This is the standalone runbook for the complete Test2 L0 process group. L0 release publication and L0 process management each have one dedicated CI path. Neither path is part of full-stack deployment. For the complete Test2 sequence and the smallest-operation decision table, see [`test2-standalone-deployment.md`](test2-standalone-deployment.md). ## Boundary This path changes one L0 release only: ```text Runtime API + the nine Runtime Worker roles ``` Runtime API and Runtime Worker are selected from one L0 Engine Release, so they cannot drift to different source versions. Integration Service remains outside L0. This path does not use a deployment BOM, AppRelease, L1/L2 source, D3 active state, pack apply, database initialization, migration, or refresh. ## Deployment ### 1. Publish an L0 version The L0 Engine release CI is owned by the NexusCRM repository. NexusCRM `main` is published automatically. For the exact head of a non-main branch, run: ```text pingcap-inc/NexusCRM .github/workflows/publish-l0-engine-dev-release.yml ``` ```bash gh workflow run publish-l0-engine-dev-release.yml \ --repo pingcap-inc/NexusCRM \ --ref main \ -f source_commit= ``` The release contains the matching immutable subjects: ```text runtime-api@sha256: runtime-worker@sha256: ``` The Engine Release itself is addressable by the exact NexusCRM commit. ### 2. Upgrade L0 The dedicated process-management CI is owned by this L2 repository: ```text pingcap-inc/nexuscrm-pingcap-pack .github/workflows/deploy-test2-l0.yml ``` Use the same source selected by the L0 Engine release publication: ```bash gh workflow run deploy-test2-l0.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack \ --ref main \ -f operation=upgrade \ -f l0_source=main ``` The workflow resolves the exact L0 Engine Release, selects its Runtime API and Runtime Worker images together, replaces the current L0 process group, and verifies every process. Bootstrap mode runs only Runtime API; serving mode runs Runtime API plus all nine Worker roles. This is a stopped replacement, not a rolling update or an in-process patch. Expect a short process-level interruption while the new container starts. Test2 refuses existing full-D3 ownership before operating any standalone process. A failure before the replacement step leaves existing L0 containers untouched; cleanup runs only after a replacement actually started. An upgraded bootstrap is expected to be live and healthy but not ready. Readiness becomes true through the same management workflow's separate `activate` operation after Application Activation prepares the L0 state root. The matching Worker containers exist but remain stopped in bootstrap mode; their immutable images are the Worker half of the deployed L0 release. Activation reads those container images directly. There is no persisted release pointer. For deterministic branch testing, publish and deploy the same exact 40-character commit. For `main`, deploy the commit whose Engine Release has already completed publication. ## L0 process management Normal process management uses the dedicated L0 management workflow. Do not use the full-stack deployment workflow to manage this standalone process group. ### Stop for an AppRelease transition AppRelease pack convergence is an offline Controlplane operation. Stop the standalone Runtime before an AppRelease upgrade: ```bash gh workflow run deploy-test2-l0.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack \ --ref main \ -f operation=stop ``` `stop` stops Runtime API and every Runtime Worker. It does not remove them, change their release, prepare application state, or create a handoff. Repeating it when L0 is already stopped succeeds without doing anything. After the AppRelease operation succeeds, use `activate`. ### Restart Restart the currently deployed L0 process group without changing its release or configuration: ```bash gh workflow run deploy-test2-l0.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack \ --ref main \ -f operation=restart ``` Restart requires an existing Runtime API. In serving mode it restarts all nine workers as the same L0 unit. It does not resolve a source ref, pull an image, remove containers, or run any pack or database operation. ### Application activation boundary Application Activation interacts with L0 through exactly one command: `activate`. It keeps the exact current L0 release and starts it with the prepared Catalog, serving configuration, database connection, and workload identity. `restart` remains an independent L0 operator command; it is not part of the Application Activation interface. Application Activation must not select, build, pull, or upgrade an L0 image. It prepares the existing L0 state root owned by the platform deployer: ```text config/ host-compatibility.json runtime.env runtime-server-name # existing TLS identity serving.json workers/*.json secrets/ workload-identities.json signing-keys.json workers/*.token tls/ data/ catalog/ files/ uploads/ ``` This is the same state-root contract used by NexusCRM's existing standalone L0 deployer. Runtime remains the semantic validator for `serving.json` and the Catalog. There is no activation receipt, digest, `current` pointer, copied release, backup container, or second state machine. Catalog selection and version ownership stay outside L0. Application Activation materializes the selected Catalog into the canonical state root; the L0 manager does not read, store, compare, or report a Catalog revision or digest. It only starts Runtime against the prepared root and verifies readiness. L0 upgrade owns `host-compatibility.json`. It is the selected image's public protocol and capability contract; AppRelease installation reads it without selecting or interpreting an L0 binary version. No later workflow may stop, remove, create, or restart the L0 container. It may prepare this state root and then dispatch the L0 management CI's parameter-free `activate` operation. Activate the current image after Catalog publication: ```bash gh workflow run deploy-test2-l0.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack \ --ref main \ -f operation=activate ``` `activate` never checks out L0 source, pulls an image, or selects an L0 version. It replaces bootstrap or the previous serving process group with the exact current L0 release and prepared state root. Only this L0 management CI may perform that replacement. ### Upgrade Publish the new NexusCRM commit, stop L0, then run the deployment workflow with `operation=upgrade` and that same commit. The upgrade uses the installed role files and the existing control-plane `work-ownership` CAS to move each Worker's release binding and epoch before starting the replacement API and Workers. The API and all Workers must remain stopped throughout this transition. Before reading database credentials, the remote mutation entrypoint validates exact official Worker and Catalog image references and verifies the locally available Worker image against the selected source revision. This operation needs no pack input or pack apply. It preserves the installed Catalog and role metadata. A retry after a partial transition resumes only the same deterministic CAS operation; conflicting ownership fails closed. Completed roles do not advance their epochs again. The shared implementation also backs `local_l0.py upgrade` for local verification. A first bootstrap without installed Worker roles stays in bootstrap with stopped image carriers. If bootstrap already has installed role files, a stopped upgrade refreshes their bindings as well, so later activation needs no reinstall. Missing Worker containers do not change the predecessor Runtime mode. If the Runtime container itself is absent, upgrade conservatively creates bootstrap; prepared installation files do not prove activation. Run the explicit `activate` operation afterward to restore serving, without reinstalling the app. Incomplete or conflicting installed role files must be repaired at their owner before the upgrade can proceed. ### Redeploy the current version Run `operation=upgrade` with the currently deployed commit. Publication does not need to run again when its exact Engine Release already exists. ### Recover by deploying an exact release Run `operation=upgrade` with the exact desired NexusCRM commit. The management CI does not retain a previous container, release pointer, or rollback state. If the desired commit is an older known-good release, it is handled by the same normal replacement operation; installed pack state is preserved and Worker ownership advances through the same guarded transition. ### Observe The management CI summary is the normal source for the selected source commit, immutable images, and process mode. On the Test2 host, the managed group is `nexuscrm-l0-test2-runtime-api` plus `nexuscrm-l0-test2-worker-*`. Runtime health, `/livez`, TLS, and every Worker health check must succeed. Runtime `/readyz` is false in bootstrap mode and true in serving mode. ## When not to use this workflow Use this path only for a process-only L0 code change that remains compatible with the existing database and public protocols. If the change requires a schema migration, it is not a process-only upgrade and must use the migration-owning path. Do not add migration or other component logic to the L0 management workflow.