# ADR 0003: Runtime-only Blue/Green Prod promotion

- Status: Retired with the Classic lane; retained as monolith history
- Date: 2026-07-28
- Scope: classic monolith Staging-to-Prod promotion only

Serviceified D3 promotion is a separate release lane governed by
[ADR 0007](0007-serviceified-d3-four-slot-prod-promotion.md). Its signed-BOM,
four-slot stopped-upgrade design does not reinterpret this classic runtime-only
decision.

## Context

Ordinary Prod previously mixed two different operations:

1. shared-state setup, pack lifecycle, metadata, policy, and reference-data
   mutation; and
2. runtime replacement and traffic cutover.

That made production the first environment to exercise release-owned database
writes and created recovery states in which Blue, Green, Nginx, setup receipts,
and resume markers could disagree. Per-worker standby flags did not solve the
problem because a candidate process could still run an unguarded background
writer.

The product accepts a brief interruption of an in-flight request during
cutover. It does not accept using Prod as a setup or migration test
environment, or leaving the public service down while release code is repaired.

## Decision

Ordinary Prod is runtime-only.

1. Formal Staging builds the immutable candidate and proves its shared-state
   digest.
2. Prod recomputes the active and candidate digests from their exact retained
   source trees.
3. If the digests differ, Prod blocks before candidate or service mutation.
   The change requires a separately governed compatible migration and a fresh
   Staging candidate.
4. Customer-file migration and historical all-Active file census are separate
   governed data operations. Ordinary Prod does not copy legacy filesystem
   roots, migrate customer-file bytes, or use a historical file census as a
   release gate. It proves only that the candidate runtime identity can
   put/get/delete an exact canary under the fixed S3 `.readiness/` prefix.
5. Green starts with the L0 global runtime switch
   `NEXUSCRM_RUNTIME_STANDBY=true`. In standby, only `GET`/`HEAD /health` is
   served; connector projection, receipt routing, audit capture, scheduled
   work, session cleanup, and every other request or background worker remain
   disabled.
6. Green must report the exact candidate version, `runtime_mode=standby`, and a
   connected database.
7. Inside one cutover lock, the release restarts Green with
   `NEXUSCRM_RUNTIME_STANDBY=false` while Blue remains live, proves the exact
   active health, and starts the Green adapter.
8. The release switches Nginx directly to Green and proves the public route
   and frontend. Only after the public Green route is proved does it stop and
   disable the retired Blue services, resume any release-owned HubSpot
   admission pause, then publish the current marker.

The Nginx switch and Blue stop may interrupt an in-flight request. Uploads and
writes may be retried by the user. Blue and Green background workers may
overlap from Green activation until Blue is stopped after the public route
proof; each worker's own transactional claim/lease/fence contract remains the
duplicate-execution authority. There is no Blue runtime-mode restart, general
drain protocol, database-wide writer lease, maintenance route, warm rollback
protocol, candidate resume marker, or downtime setup fallback in the ordinary
path. This direct-switch protocol is limited to the current single-host
cutover-lock topology; a multi-host release controller requires a separately
designed distributed fencing protocol.

## Failure semantics

- Failure before the cutover lock: Blue remains active and routed.
- Failure before the public Green route is proved: stop Green and retain or
  reinstall the Blue route. Blue remains active throughout this interval and
  does not require a runtime-mode restart.
- Failure while stopping Blue after the Green route is proved: restart any Blue
  services already stopped, reinstall the previous Nginx route, and stop Green
  before the run fails.
- The exact current marker is the only cutover commit point. Release evidence,
  Blue-unit retirement, and current-pointer preparation complete before that
  marker is atomically published. A pre-commit failure restores the previous
  current pointers, re-enables Blue, and reinstalls its Nginx route.
- At the beginning of every ordinary promotion, the marker-selected release is
  reconciled to the live route under the same cutover lock. If a prior runner
  terminated after Green activation, Nginx switching, or Blue stopping but
  before the marker commit, the marker-selected Blue service, pointers, and
  Nginx route are restored and its exact version/mode/DB health is proved before
  any new candidate work. On every entry, all non-current Slot services are
  stopped and their ports proved listener-free before the live route is
  accepted. This incident recovery consumes only the committed current
  marker/runtime facts, never candidate identity or admission. It retains the
  ability to repair an old marker-selected runtime left in standby by a legacy
  release attempt, but new direct-switch attempts never put Blue into standby.
- After the current marker is published, Green remains the active release and
  no runtime mutation or rollback is allowed. The workflow must still validate
  the immutable deploy summary and durably record `service-verified`; failure
  blocks terminal release delivery and is retried as service verification,
  without undoing or reapplying the live candidate.

This restoration is a local safety action for the current run. It is not a
schema rollback or a retained-candidate resume protocol.

## Invariants

- Ordinary Prod performs zero release-owned business/database shared-state
  writes and no customer-file data migration. Its only storage write is an S3
  put/get/delete canary under the reserved customer-file `.readiness/` prefix.
  Release metadata records the canary receipt; inability to read back exact S3
  bytes or delete the visible canary blocks promotion.
- A separately governed all-Active census binds every S3 row to one declared
  authority. The ordinary runtime bucket/prefix remains the only unscoped
  authority; the Salesforce archive bucket/prefix is an additional authority
  restricted to `SourceType=salesforce`, so recovery can validate restored
  history without widening native-file storage or coupling the census to
  ordinary deployment.
- Salesforce history replacement is admitted only from a manifest whose blob
  count is bound to exact-key, `ContentSize`, checksum/source-byte, and
  post-upload SHA-256 readback under `exact-key-size-checksum-v1`. Recursive
  basename discovery and counter-only manifests are not deletion authority.
- Prod binds the Salesforce history authority to the exact archive bucket and
  prefix granted read-only `s3:GetObject` access in the tracked runtime IAM
  policy. A different bucket/region blocks env preparation, and the archive
  statement grants no Put or Delete access. Deployment independently rechecks
  the materialized runtime S3 authority before its canary, preventing stale or
  altered env state from bypassing preparation.
  The Salesforce history recovery entry point independently validates the
  hash-bound snapshot bucket/prefix and target-manifest region against that
  fixed authority before database or S3 access, and binds the normalized tuple
  into its dry-run, backup, and apply receipts.
  Metadata-only ContentVersion modes cannot publish a load-eligible manifest.
- Ordinary Prod never invokes setup, pack lifecycle apply, seeds, reference
  data, database reset, or a migration identity.
- Different release identities are allowed when their exact shared-state
  digests are equal.
- Final-pack-external official dependencies must also match their exact
  read-only `_System_InstalledPack` source revision and payload hashes.
- Blue is never restarted during ordinary cutover. It remains active until the
  public Green Nginx route is proved and is retired immediately afterward.
- `SKIP_BACKEND_START=true` is rejected; standby and active health proofs
  cannot be bypassed.
- A run cannot report BLOCK after publishing the new current marker.
- A release-owned HubSpot admission pause is resumed only after the retired
  Blue slot is stopped, so it cannot reopen work to both active runtimes.
- Prod consumes the exact Staging-built candidate bytes and does not rebuild.
- PASS is bound to the exact tag, tuple, bundle hashes, active baseline,
  standby/active runtime health, Nginx readback, and current marker.

## Dependency

This L2 workflow must not be deployed until the selected NexusCRM L0 version
contains the fail-closed global runtime standby capability. A CI pass in this
repository cannot substitute for that lower-layer capability.