# ADR 0007: Serviceified D3 four-slot Prod promotion

- Status: Accepted; amended 2026-09-06
- Date: 2026-08-23
- Scope: formal serviceified D3 Staging-to-Prod promotion

## Amendment — 2026-09-06

This amendment ends the 2026-09-02 one-promotion no-stop exception. After nginx
cutover and successful public target readback, the operator automatically stops
only the predecessor bound by durable pending authority, disables automatic
restart, and verifies retirement. It re-proves the public target before
publishing current authority and clearing pending authority. Failure retains
the exact target for an idempotent exact-Tag retry, including after partial
retirement; it does not authorize an automatic nginx rollback.

Retirement uses exact D3 slot labels plus validated immutable container IDs and
the reviewed systemd unit allowlist. It retains stopped containers and release
files; retention cleanup is separate. Unknown listeners block closeout, never
authorize killing an arbitrary PID. Historically retained services outside the
recorded predecessor are not swept by this operation. The predecessor still
remains live throughout target Patch/Catalog convergence and startup.

## Historical amendment — 2026-09-02 (superseded)

The one-promotion exception left the predecessor running after public cutover
and required a later manually authorized shutdown. It accepted concurrent old
background work against shared TiDB during observation, without certifying
rollback compatibility. The 2026-09-06 amendment restores automatic retirement.

## Amendment — 2026-08-31

This amendment supersedes the stopped-upgrade ordering from the 2026-08-24
contract. Ordinary Prod promotion explicitly permits the previous runtime to
remain active while the target performs Patch/Catalog convergence against the
shared TiDB. The operator fences that predecessor only after nginx cutover and
successful public target readback, and still requires retirement proof before
publishing current authority. This policy accepts the resulting concurrent
shared-state access risk. The other admission, exact-Tag resumption, and joined
live-state requirements remain in force.

## Activation state

This ADR is implemented by `emit-staging-d3-release-tag.yml` and
`deploy-multi-prod-d3-environment.yml`. Those are the only formal D3 Tag and
Prod promotion surfaces.

## Context

Prod already owns four physical backend slots: `3103`, `3104`, `3105`, and
`3106`. Treating only `3103/3104` as permanent blue/green slots discards half
of that topology and disagrees with the current multi-prod contract. In D3,
blue and green are release-time roles: the nginx-selected runtime is current;
the slot being prepared is target.

The D3 stack is selected by one signed resolved BOM. Patch Bundle remains a
separately usable migration mechanism, but the ordinary release lane has one
closed projection: Staging alone selects the optional Patch, its successful
dispatch records the nullable path/tree digest, the v5 Tag copies that exact
identity, and Prod replays it from the Tag-selected L2 tree after target
quiescence and before stopped-upgrade pack/Flow apply, matching Staging. The
later Catalog publication is not the source-pack reconciliation boundary.
Prod exposes no independent Patch input or override. A separate
Prod migration initiated outside this projection still requires its own
reviewed authority and evidence. This is a distinct lane from the classic
runtime-only promotion in ADR 0003.

This ADR replaces the earlier D3 note that Candidate/release-manifest recovery
had to precede Prod promotion. Those authorities remain classic-lane concepts;
they are not deferred admission gates for the signed-BOM D3 lane.

## Decision

1. A completed/successful
   `deploy-staging-blue-green-environment.yml` run is the complete D3 Tag
   admission condition. The emitter reads that run's exported dispatch identity
   to carry its resolved-BOM reference, exact `l2_metadata.commit`, and
   immutable Staging evidence-artifact ID/name/digest; it
   does not requalify deployment receipts, process counts, Env Inspection, BA,
   Candidate, or S3 state.
2. `emit-staging-d3-release-tag.yml` publishes an annotated
   `staging-d3-release-*` Tag containing the successful Staging run identity,
   Tag target commit, and exact signed resolved-BOM reference. The Tag target
   must equal the signed BOM's exact `l2_metadata.commit`. The Tag also binds the
   immutable Staging evidence artifact that contains `dispatch.json`. Its v5
   payload requires the nullable `patchBundle` copied from that dispatch:
   `null` means no Patch, while a path/tree-digest object requires Prod replay.
3. `deploy-multi-prod-d3-environment.yml` has exactly one release-selection
   input, `release_tag`; it accepts no independent BOM, L2 commit, target slot,
   or Patch override. It accepts only the D3 Tag contract and
   independently verifies the signed BOM before using its exact L0, L1, and L2
   artifact identities. Prod must reject the release unless the verified BOM's
   L2 commit equals the Git Tag target. Prod retrieves the exact unexpired
   Staging evidence artifact by recorded ID, requires its GitHub artifact
   metadata ID/name/digest and source workflow run ID/attempt/head commit to
   match the Tag, and requires its immutable `dispatch.json` BOM/L2 identity to
   equal the Tag. The reviewed controller comes from the workflow's exact
   `github.sha` on `main`; the BOM-selected L2 checkout is release metadata
   authority, not executable operator authority.
4. The D3 workflow uses the GitHub Actions concurrency group
   `prod-release-tidb-boundary`. It holds that release-wide lock before it reads,
   reconciles, selects, or publishes current/pending release authority and
   before it mutates shared Prod TiDB or nginx. It retains the lock through
   current-marker publication and pending-authority removal. The legacy group
   name is retained so an interrupted first handoff cannot overlap another
   release operation.
5. Physical Prod slots rotate strictly:
   `3103 -> 3104 -> 3105 -> 3106 -> 3103`. Before the first shared-state
   mutation, the operator atomically publishes
   `multi_prod/d3/in-progress.json`, binding the exact Tag, previous route,
   target slot/port, Tag target, exact resolved-BOM digest reference, and
   release root. Before publishing that authority or materializing target
   secrets/state, the operator completes the Tag, Staging run, evidence
   artifact, `dispatch.json`, signed-BOM, desired-BOM binding, Tag-target/L2,
   and exact-source admission chain. A fresh release may use the reconciled current
   marker plus live nginx route to select the successor only when no in-progress
   authority exists. If the target release root already contains a runtime env,
   its complete root/Tag/commit/BOM/slot/predecessor identity must match
   before the operator may publish pending authority or create release state.
6. The four slots have disjoint SFCRM, Integration, Runtime API, Worker, and
   workflow-action companion ports. Slot identity is `slot-<backend-port>`;
   blue/green are not persistent slot names. Ingress reconciliation rejects any
   localhost proxy target outside the routed slot's reviewed SFCRM, Integration,
   and companion port tuple; it may not ignore an unknown split-traffic target.
   Durable authority must name different predecessor and target backend ports.
7. All slots share the production TiDB. Before the first Catalog mutation, the
   operator quiesces the selected target. Every Catalog entry, including a
   resumed `catalog-ready` phase, immediately re-proves target-unit and
   target-port quiescence before mutation. Ordinary Prod promotion permits the
   previous runtime to keep serving and writing the shared TiDB throughout
   Patch/Catalog convergence and target startup. The TiDB host and
   least-privilege runtime user must match the separately reviewed
   `PROD_TIDB_HOST` and `PROD_RUNTIME_TIDB_USER` bindings, and the user must not
   belong to any configured non-Prod target. Nginx remains on the previous route
   until the complete target is ready. Candidate source validation, image pulls,
   artifact sealing, and standby-slot quiescence complete before convergence;
   the engine performs at most one automatic exact-Tag/BOM/target state-machine
   resume before requiring operator intervention.
   After nginx selects the target and public readback succeeds, the operator
   stops and verifies only the pending-authority-bound predecessor. It disables
   automatic restart but retains stopped containers and release files.
8. D3 stopped-upgrade/Catalog convergence may update shared Catalog authority
   and applies the exact L2 app-shell release configuration. When the new Tag
   format binds the Patch Bundle used by Staging, Prod replays that directory
   from the exact L2 checkout after target quiescence and immediately before
   stopped-upgrade pack/Flow apply. A successful replay is retained for the
   subsequent Catalog publisher; post-Catalog retries only verify its durable
   receipt and never replay SQL. Prod exposes no
   separate patch input and performs no database reset, TiDB branch reset, Prod
   seed, or governed reference-data apply.
9. Runtime API, seven Worker roles, Integration, SFCRM, frontend, and the
   workflow-action companion must all be ready before nginx switches. After
   the target route and public frontend/health readback succeed, the operator
   retires and verifies the predecessor, re-proves the public target, records
   final serving and retirement evidence, publishes `current.json`, moves the
   `multi_prod/d3/current` convenience symlink, and only then clears pending
   authority. Companion
   executable sources come only from the reviewed activation checkout; the
   BOM-selected L2 checkout is metadata authority, not production
   orchestration-code authority.
   A release is active only when `current.json`, the current symlink, live nginx
   route, public backend/frontend readback, target runtime identity, verified
   predecessor retirement, and cleared matching in-progress authority all agree.
10. The first D3 promotion is an explicit one-way handoff from the Classic lane.
    While holding the shared lock, it validates the classic current marker
    against live nginx, records that live route as the previous owner in D3
    `in-progress.json`, and then performs the normal D3 release. The Classic
    `current/env/runtime.env` and `current/frontend-dist` pointers must both
    resolve inside the routed Classic `ENV_ROOT`; an unrelated or escaping
    frontend pointer blocks the handoff. Once either a D3 pending or current
    authority exists, it is the only release authority. The legacy
    `multi_prod/current.release-tag` then remains historical evidence, not
    active-slot authority. The Classic workflow and Tag emitter are removed;
    any reverse handoff would require a new reviewed migration and operator
    design.
11. Every D3 dispatch reconciles `in-progress.json`, the D3 current marker, and
    live nginx before selecting or resuming a target. While an in-progress
    authority exists, only its exact Tag and resolved BOM may
    run, its recorded target and predecessor remain authoritative, and nginx
    must name either the recorded previous port or target port. The mutating
    runner independently re-reads this authority instead of trusting only its
    runtime env. Any other Tag, BOM, predecessor, or route shape blocks
    before mutation.
12. After successful predecessor retirement and post-stop public target readback,
    the operator publishes the new D3 current marker and then removes the matching
    in-progress authority. Retirement must be freshly verified on every retry. A
    failure after public cutover but before that closeout does not select
    another slot. Recovery resumes the same retained target with the exact Tag
    and resolved BOM.

## Successful release sequence

```text
successful Staging D3 run
  -> immutable Staging evidence artifact binds BOM/L2 deployment identity
  -> annotated D3 Tag targeted at the resolved BOM's exact L2 commit
     and carrying the artifact identity + exact BOM
  -> acquire and retain the release-wide prod-release-tidb-boundary lock
  -> complete Tag/run/artifact/dispatch/signed-BOM/L2/exact-source admission
  -> reject unsafe legacy state or validate the one-time Classic-to-D3 handoff
  -> reconcile or atomically publish the durable in-progress target
  -> validate sources, pull images, seal artifacts, and quiesce the standby target
  -> converge the complete target against shared Prod TiDB
  -> prove target readiness
  -> switch and read back nginx/public traffic
  -> stop the pending-authority-bound predecessor and verify retirement
  -> re-prove nginx/public target traffic
  -> publish D3 current marker
  -> clear the matching in-progress authority
```

## Invariants

- Classic `staging-release-*` Tags and Candidate contracts are historical and
  have no dispatch surface.
- D3 release dispatch exposes no forced slot or forced cutover input.
- The shared lock covers legacy handoff, current/pending authority reconciliation,
  target selection, shared-state mutation, cutover, and closeout.
- No second promotion workflow may overlap D3 authority mutation, shared Prod
  mutation, or cutover.
- Nginx never selects a partially ready target.
- Shared-state mutation and background work may overlap previous-generation
  writers during target convergence, startup, and cutover, until retirement.
- A pending release owns one durable target; a different Tag cannot advance the
  slot cycle until that release is reconciled and closed.
- Ordinary-release Patch authority originates only from the successful Staging
  dispatch, is copied exactly into the v5 Tag, and cannot be overridden by the
  Prod dispatch.
- The current marker names the target only after predecessor retirement and
  post-stop public target readback. Retirement failure retains pending authority.
- Retirement affects only the recorded predecessor, not the target or unrelated
  historical slots, and does not certify rollback compatibility.
- A successful sequence advances exactly one physical slot in the defined
  cycle.