# Production D3 Release Runbook Prod is a four-slot serviceified D3 environment backed by one shared TiDB database. The only ordinary release path is successful Formal Staging D3, followed by a D3 release Tag and the Prod D3 operator. ## Operator surface The release has exactly two operator workflows: 1. `.github/workflows/emit-staging-d3-release-tag.yml`, with one input: `staging_run_id`; 2. `.github/workflows/deploy-multi-prod-d3-environment.yml`, with one input: `release_tag`. The Classic Tag emitter and Classic multi-prod workflow have been retired. `staging-release-*` Tags, Candidate bundles, Business Acceptance fields, `prod_release_eligible`, release phase ledgers, and per-Tag S3 release directories are not Prod D3 authorities. ## Successful release sequence 1. A reviewed `main` deployment completes `.github/workflows/deploy-staging-blue-green-environment.yml` successfully. 2. The trusted-main D3 Tag emitter reads that exact run. It binds the exported signed BOM, its exact L2 commit, and the immutable Staging evidence artifact into an annotated `staging-d3-release-*` Tag. The Git Tag target is the BOM-selected L2 commit. 3. The Prod workflow accepts only that D3 Tag. It verifies the signed BOM, requires the BOM L2 commit to equal the Tag target, and downloads the exact Staging evidence artifact by its recorded ID. 4. Prod proves that the artifact's immutable dispatch record authorized the same BOM and L2 commit carried by the Tag. 5. Under the `prod-release-tidb-boundary` lock, Prod reconciles the live nginx route and `multi_prod/d3/in-progress.json`, then selects or resumes the target slot. 6. Prod converges the complete target D3 stack against shared TiDB while the previously active runtime remains live. 7. Runtime API, seven Worker roles, Integration, SFCRM, frontend, and the workflow-action companion must all be ready before nginx changes. 8. Prod switches nginx to the ready target and proves the public backend port, `/health`, `/`, and `/index.html`. 9. Prod stops and verifies the pending-authority-bound predecessor, then proves the public target again. Only after these proofs does it publish `multi_prod/d3/current` and `current.json`, and clear the matching in-progress authority. Env Inspection and unrelated business-path workflows remain independent of release admission. The successful Staging deployment conclusion is the Tag eligibility boundary. ## Four-slot rotation The physical backend ports rotate strictly: ```text 3103 -> 3104 -> 3105 -> 3106 -> 3103 ``` Current and target are dynamic blue/green roles; there are four physical slots, not two permanent color slots. All four slots share the production database. Ordinary promotion explicitly permits the previous runtime to remain active during Patch/Catalog convergence and target startup. After cutover and successful public target readback, retirement is automatic and required for release closeout. A retained predecessor is not public traffic or release authority, and its presence alone does not certify an automatic rollback as Catalog-compatible. Any separate migration must establish the runtime-disposition behavior required by that migration's own authority. The retained host authority is: ```text /home/ubuntu/nexuscrm-deployment/online/envs/multi_prod/d3/ current -> releases// current.json in-progress.json # present only during a release releases/ / ``` An exact-Tag retry must resume the target recorded in `in-progress.json`. A different Tag must not advance the slot cycle until that authority is closed. ## Frontend access ACL Prod keeps frontend files under the existing release's `frontend-releases`. Before publishing the frontend link or switching nginx, the publisher backs up access ACLs to `.runtime/frontend-acl-backups/before-*.json` (0600 in a 0700 directory), grants the reviewed nginx user `www-data` (UID 33) traverse-only access on `d3`, `d3/releases`, and the exact Tag directory, and explicitly denies that user access to `.runtime` and the adapter launcher. It changes no other paths, default ACLs, owners, or world permissions. Unknown existing ACLs fail closed rather than being overwritten. The nginx-user probe must read the full immutable frontend while private env/log/run/artifact paths remain inaccessible. Failure restores this attempt's ACL changes before aborting; retry rechecks access and does not rewrite already-correct ACLs. The host must retain its existing `libacl.so.1`, nginx user and non-interactive sudo access. Do not manually restore an old ACL backup while its frontend path is serving; that can reintroduce #5824. No release paths, current/pending identity, or CI workflows change as part of this permission repair. ## Patch Bundle boundary Staging is the only workflow that selects `patch_dir`. Its successful dispatch records the nullable path/digest in the new D3 Tag format. Prod accepts only the Tag, resolves the bound directory from the exact Tag-selected L2 checkout, and replays it after target quiescence and immediately before the stopped-upgrade pack/Flow apply, matching Staging. Catalog publication follows that apply; its entry retains a pre-Catalog fallback but does not replay a patch already completed in the same process. Ownership handoff patches must run before pack/Flow reconciliation, not merely before the later Catalog publisher. Changing this ordering does not repair Flow content already skipped by an older deployment. That persisted state needs a separately reviewed recovery; replaying an ownership-marker-only SQL patch cannot update the Flow payload by itself. After Catalog has begun, an exact-run retry must not replay SQL. It instead verifies the existing run-owned `patch-replay/patch-summary.json` against the same requested path, sealed tree digest, manifest, and complete successful SQL file results. The exact Tag-selected L2 source and promoted digest are still checked before convergence. The verified receipt is retained in release evidence. A missing, incomplete, foreign, or changed receipt/seal fails closed; do not fabricate a receipt, remove the Tag-bound patch, or rewind active state to force a retry. ## Predecessor retirement The stop target comes only from `in-progress.json`'s `previousBackendPort` and `previousRouteKind`, never from the already-switched nginx route. The operator re-reads pending authority and proves nginx still selects the new target before any stop. It operates only on: - D3 containers with the exact `io.nexuscrm.serviceification-d3=prod-slot-` label and matching slot-scoped name, using inspected immutable container IDs; - the exact slot systemd units, plus the reviewed global backend/adapter aliases only for a Classic-to-D3 handoff. Systemd units are disabled and stopped; Docker restart policy is set to `no` before a bounded graceful stop. Containers and release files are retained. Closeout re-reads container/unit state and proves the predecessor's reviewed service ports have no listener. An unknown listener blocks closeout; the operator never kills a process inferred from a port. Stop failure, an active/restartable predecessor, or failed post-stop target readback preserves pending authority and prevents new current publication. Retry the same exact Tag: retirement rechecks the recorded predecessor and is idempotent after a partial stop. Do not rotate slots or automatically roll nginx back. Previously retained services in other historical slots are outside this operation's scope and require separately authorized handling. ## HubSpot ingress An ordinary release converges both production paths to `proxy/proxy`: | Key | Release value | | --- | --- | | `HUBSPOT_ORDINARY_WEBHOOK_MODE` | `proxy` | | `HUBSPOT_WORKFLOW_EXECUTE_MODE` | `proxy` | `block` and `ack_drop` are separately authorized incident states, not release inputs or successful release evidence. The L2 backend contract keeps `NEXUSCRM_CONNECTOR_WEBHOOK_ACK_DROP_ENDPOINTS` explicitly empty and `NEXUSCRM_CONNECTOR_WEBHOOK_ACK_DROP_TIMEOUT_MS=750`. ## Success evidence A successful Prod release summary must bind at least: - release Tag and Tag target; - signed resolved-BOM reference and verified L2 commit; - immutable Staging run/artifact identity; - selected slot and backend port; - target D3 readiness; - nginx and public health/frontend readback; - `previousRuntimeRetirement` with schema `nexuscrm.prod-predecessor-retirement/v1`, the matching previous/target ports, and `verified: true`, followed by public target readback after retirement; - `current.json` publication and in-progress authority removal. Do not report success from artifact verification, target-local readiness, or nginx cutover alone. The current marker and cleared pending authority are part of the successful closeout. ## Retention and supporting assets Retention cleanup must protect both the D3 `current` release and any release named by `in-progress.json`; it may delete only inactive `multi_prod/d3/releases/` directories outside the configured window. Retirement does not delete these files or containers. Retention cleanup does not discover running services: before non-dry-run cleanup, independently prove all cleanup candidates inactive, including any historical slot not covered by the latest predecessor retirement receipt. Handling such services requires separate authorization; their shutdown is not inferred from a cleanup request. The production file-storage IAM policy and HubSpot readback constants are supporting operational contracts, not Classic release authority. They remain available to migration and verification tooling while the old release operator is removed.