# ADR 0002: Shared Release Operation Protocol

**Status:** Proposed

> **Current-contract note:** This proposed protocol never became the current L2
> promotion authority. The Classic Candidate lane and its
> `release-candidate/v3` publisher have been retired; current D3 promotion uses
> the resolved-BOM authority defined by ADR 0007 and has no Candidate contract.
> Statements below about the "current" v2/v3 lane describe the decision
> baseline, not today's operator surface. Any later operation-protocol
> implementation must retain continuous HubSpot ingress and must not restore a
> release-owned quiescing fence or drain inbound work.

**Version:** `release-operation-protocol/v3`

**Decision date:** 2026-07-24

**Baseline reviewed:** downstream `20f692710aee6ce45c17419a791da05643cc27a6` (`origin/main` at review time)

## Reader and intended action

This ADR is for maintainers of the NexusCRM L0 control plane, L1 packs, and
PingCAP L2 release automation. After reading it, a maintainer can split an
implementation into the named owners and decide whether a staging result makes
a tag eligible for Prod promotion without creating a new Prod-only migration.

## Decision

Adopt one typed, L0-owned release-operation protocol for a candidate's
deterministic transition. Staging and Prod use the same protocol and serialized
operation plan, but each supplies an explicit environment policy adapter.

This is not a unified shell script. L0 owns the state machine, canonical data
types, plan validation, receipt rules, compensation classification, and
activation primitives. L1 provides declarative, capability-bounded operation
profiles. L2 selects a reviewed tuple and environment policy, binds deployment
credentials at runtime, and invokes the L0 protocol; it does not implement a
second lifecycle engine or execute arbitrary pack hooks.

The existing `release-candidate/v2` contract remains the live authority until a
separately reviewed implementation adopts v3. This ADR changes no workflow,
tag, deployment script, runtime, cloud resource, queue, database, ingress, or
Prod state.

## Context and problem statement

The current formal staging path correctly failed closed, but discovered
deterministic prerequisites serially after a candidate began its shared-state
transition: source-convergence ordering, exact setup-binary capability,
namespace-wide ingress fencing, then 103 nonterminal WorkItems without a valid
Event or InstalledPack owner. The final failure had already crossed the
InstalledPack mutation boundary, so inbound admission remained quiesced and no
tag was emitted.

Prod has a separate problem. It can derive an active-Prod-to-candidate
transition that staging never performed, and a slot rollback does not restore
the shared InstalledPack state visible to the still-active runtime. A staging
target-state success is therefore not sufficient proof that a promotion is
deterministic or failure-atomic.

The current code is a migration input, not a blank slate:

- L2 has a candidate checkpoint that binds downstream, upstream, app-pack,
  connector, final-pack, slot, and workflow identities; its release ledger
  already gives a candidate a deterministic identity, catalog digest,
  rollback points, phase records, and redacted evidence.
- The former L2 target-local pack convergence phase treated ingress quiescence as
  mandatory before `apply-source-convergence` mutated shared InstalledPack
  metadata. That behavior is not legal for the active ordinary multi-prod
  contract; the current runner keeps ingress continuous while a future v3
  design may provide isolated/atomic control-plane semantics.
- L0 setup accepts an explicit source-revision advance only with distinct
  `from` and `to` revisions and `verify-pack`. L0 serving admission already
  rejects a capability set that mixes installation, release digest, source
  snapshot, catalog revision, or generation.

Those facts support a common protocol. They do not establish that the present
L2 staging and Prod scripts have one equivalent transition path.

## Canonical candidate and operation-plan identity

### Candidate digest

Every proposed release produces one canonical JSON candidate document and a
SHA-256 `candidate_digest`. Canonicalization sorts object keys, rejects unknown
required-schema fields, and excludes credentials, live URLs with secrets, and
raw customer records. The digest covers at least:

| Identity group | Required fields |
| --- | --- |
| Source | downstream commit, release-tuple digest, upstream L0 commit, L1 app-pack commit, module-version resolution, final-pack digest |
| Executable | exact setup/runtime binary digest, distribution/build tags, capability-report digest, declared protocol and profile versions |
| Payload | per-pack desired payload digest, installed applied/desired payload digests, source revision, `PackInstallId`, generation, and active-pointer expectation |
| Queue and admission | namespace ingress policy, fence scope/epoch expectation, WorkItem ownership classification digest and counts, permitted terminal-quarantine manifest digest when applicable |
| Transition | ordered operation-plan digest, baseline-projection digest, side-effect boundary, approved compensation or forward-recovery class |

`candidate_digest` is not a synonym for a Git SHA. A Git commit is one input;
the digest binds every execution-relevant input that must remain immutable from
staging planning through Prod activation.

### Source projection and operation plan

Before a staging mutation, L0 materializes a redacted control-plane projection
of the active Prod source state. It contains only identities, revisions,
hashes, generations, ownership classifications, required capability facts, and
counts needed to plan a transition. It must not contain connector payloads,
PII, credentials, or unbounded queue rows. The projection has a
`baseline_projection_digest` and retention policy.

L0 then produces an immutable `operation-plan/v1`:

```text
operation_id, ordinal, kind, input_digest, preconditions,
expected_effect, verification, side_effect_class,
compensation_class, owner, profile_id
```

An operation can be a read-only assertion, ingress-fence acquisition,
quiesce/drain, source-revision convergence, generation preparation, isolated
install/apply, verification, activation, or governed terminal quarantine. The
plan must express every deterministic source/payload/install/generation change
that Prod would need. It cannot contain opaque executable hooks, `eval`-style
commands, or an unbounded "run pack script" operation.

The exact setup binary produces and signs its capability report before plan
approval. Source-tree presence, a module version alone, or a different binary
is not capability proof. A missing capability produces a plan that selects the
ordinary verified path or blocks; it may not be emulated in L2.

## Shared protocol state machine

The protocol has one durable operation record and an append-only receipt stream.
The following states are monotonic except for the terminal recovery branch:

```text
PLAN
  -> PRECONDITIONS_PROVEN
  -> PREPARED
  -> MUTATION_STARTED
  -> VERIFIED
  -> ACTIVATED
  -> COMPENSATED | FORWARD_RECOVERY_REQUIRED
```

| State | Required proof | Mutation rule | Failure result |
| --- | --- | --- | --- |
| `PLAN` | canonical candidate, exact binary capability report, baseline projection, deterministic ordered plan | read-only | `BLOCKED_BEFORE_MUTATION` |
| `PRECONDITIONS_PROVEN` | every deterministic precondition and operation dependency passes, including ownership and policy checks | read-only | `BLOCKED_BEFORE_MUTATION` |
| `PREPARED` | lease/serialization authority, retained rollback/forward-recovery targets, and candidate artifact are durable | read-only except artifact retention | `BLOCKED_BEFORE_MUTATION` |
| `MUTATION_STARTED` | intent receipt is durable before the first effect; receipt records the named side-effect boundary | only the approved plan may run | compensate only when its class permits it; otherwise hold for recovery |
| `VERIFIED` | every mutation's postcondition, installed/generation identity, WorkItem policy, and inactive candidate runtime read back exactly | no new deterministic operation may be added | `FORWARD_RECOVERY_REQUIRED` if effects are ambiguous or irreversible |
| `ACTIVATED` | one L0 activation commit and activation receipt bind the candidate digest, generation, and ingress fence epoch | activation is compare-and-swap on the expected active pointer | reconcile from the activation record; never guess by slot alone |
| `COMPENSATED` | compensable effects were reversed and independently read back | terminal | retain receipts and keep a fresh candidate requirement |
| `FORWARD_RECOVERY_REQUIRED` | authoritative state is ambiguous, partially committed, or cannot be safely reversed | terminal until an approved recovery plan is executed | ingress remains at the recorded safe policy; no blind retry or rollback |

`MUTATION_STARTED` is the explicit side-effect boundary. Its receipt is written
before the first ingress-fence write, InstalledPack/generation change, queue
terminalization, provider reset, or active-pointer change. A failed run cannot
claim "preflight only" once that receipt exists.

"Atomic activation" has a precise, limited meaning: L0 atomically compares the
expected active generation/pointer and fence epoch, changes the active pointer,
and records the activation receipt in one authority. Nginx or another transport
adapter acknowledges that committed record; it is not falsely represented as a
cross-system database transaction. If process routing is ambiguous, the record
is reconciled to a single safe active state and the run enters
`FORWARD_RECOVERY_REQUIRED` rather than attempting an unproven rollback.

## Staging proof versus Prod reattestation

Staging is where all deterministic effects are planned and rehearsed. Prod is
allowed to reattest volatile facts read-only; it may not add, reorder, or
parameterize a deterministic mutation after the tag is accepted.

| Requirement | Must be proved in staging before mutation | Prod may reattest read-only | Result on difference |
| --- | --- | --- | --- |
| Candidate, tuple, source, payload, install, generation identities | Exact identities and their digests; plan covers every deterministic transition | Candidate/tag/executor and current baseline still match the recorded values | Invalidate tag; restage. Never synthesize a new plan in Prod. |
| Exact binary capability | Probe the exact staged distribution binary and bind its report digest | Probe the exact Prod candidate binary and compare digest/capability set | Block before mutation; a different binary requires restaging. |
| WorkItem ownership | Every nonterminal authoritative WorkItem is classified as exact Event owner, exact InstalledPack owner, or an approved, immutable quarantine input; unknown is a block | Counts and ownership-class digest still match the accepted plan | Block/invalid tag. The P0 103-WorkItem recovery supplies its own governed manifest and proof. |
| Ingress policy | Profile declares namespace scope, channel semantics, writer/reader, expiry behavior, resume authority, and safe failure policy; staging proves the adapter can enforce it | Fence absent/present state, authority, and current queue/drain condition | Block before new mutation if a required fact differs; do not weaken a retained fence. |
| Side effects and recovery | Every operation has a boundary, postcondition, and compensation or forward-recovery class | Volatile locks, service availability, branch/provider epoch, and active routing are safe now | Wait or block. No new deterministic action. |
| Activation | Isolated candidate generation/runtime verifies before activation and the CAS expectations are recorded | Expected active pointer, generation, and route/fence epoch are still current | Block before activation; reconcile only after an ambiguous committed activation. |

The phrase **staging success implies Prod promotion eligible** means only that a
staging candidate with all mandatory receipts has no unplanned deterministic
Prod migration. It does not promise that volatile Prod conditions will be
healthy; those remain a permitted read-only reattestation gate.

## Tag artifact and receipt contract

`release-candidate/v3` replaces neither the final-pack lineage nor the tag
target. It adds a tag-bound promotion bundle whose fields are content-addressed:

```text
schema_version, candidate_digest, tuple_digest, executor_commit,
final_pack_digest, exact_binary, capability_report_digest,
baseline_projection_digest, operation_plan_digest,
source_payload_install_generation_digest, workitem_ownership_digest,
ingress_policy_digest, staging_receipt_digest, accepted_at,
prod_reattest_allowlist, redacted_receipt_manifest_digest
```

The tag target, `executor_commit`, staging head, and candidate downstream commit
must be identical. The bundle references immutable artifacts for the plan,
capability report, baseline projection, and receipts; a tag must not embed
secrets or mutable host paths. Its receipt manifest contains only redacted
identifiers, digests, counts, phase/state, side-effect class, safe ingress
state, evidence object digest, and next safe action.

The acceptance evaluator is fail-closed:

1. all required deterministic staging receipts must be `PASS` and bind the
   same candidate and plan digests;
2. a `BLOCKED_BEFORE_MUTATION`, `COMPENSATED`, or
   `FORWARD_RECOVERY_REQUIRED` run is never tag-eligible;
3. a tag becomes invalid if a Prod reattestation finds a deterministic identity
   mismatch; and
4. a Prod resume uses the original operation plan and receipts only. It may
   finish volatile reattestation or reconciliation, not derive a new plan.

The current v2 checkpoint and phase ledger remain migration sources. v3 may
mirror their phases initially, but it must not create a second progress SSoT:
the v3 operation record is the eventual shared receipt authority and v2 records
link to it during migration.

## Release acceptance matrix

| Acceptance row | Staging acceptance | Tag requirement | Prod admission/closeout | Owner |
| --- | --- | --- | --- | --- |
| Candidate build | Deterministic final-pack rebuild, tuple, source refs, binary digest, capability report | Candidate and all input digests | Same tag target and executable digest | L2 with L0 verifier |
| Transition planning | Redacted source projection, ordered plan, no unknown operation kinds | Plan and baseline digests | Reattest only approved volatile fields | L0 control plane |
| WorkItem/ingress readiness | Ownership digest complete; namespace policy and failure-safe fence contract pass | Ownership/policy/manifest digests | Reattest counts, fence authority, and drain; mismatch invalidates | L0 connectors; P0 inputs as applicable |
| Prepared recovery | Rollback point or forward-recovery target is durable before effects | Prepared receipt | Retain and compare authority/epoch | L0 control plane plus environment adapter |
| Shared-state mutation | Same ordered plan is executed; every side effect has an intent and postcondition receipt | Staging verification receipt | Execute only the identical plan | L0 executor |
| Candidate verification | Exact install/payload/generation identities, service capability, and no unclassified work | Verified receipt | Verify candidate before activation | L0 runtime; L1 profile checks |
| Activation | Activation contract exercised against the staged candidate | Activation-capable receipt and policy digest | CAS activation plus route acknowledgement and live readback | L0 runtime; L2 adapter |
| Post-activation evidence | Redacted receipt manifest and run identity retained | All receipt digests present | Mandatory read-only runtime, route, and ledger verification | L2; separate canary follows |

## Layer ownership and issue decomposition

| Work item | Owner | Depends on | Acceptance boundary |
| --- | --- | --- | --- |
| ROP-1: typed operation/plan/receipt IR and canonicalization | L0 `nexuscrm-contract` | ADR acceptance | Schemas reject unknown/mutable/secret-bearing fields; canonical digests are stable. |
| ROP-2: deterministic planner, dependency validator, receipt recorder, recovery classifier | L0 `nexuscrm-controlplane` | ROP-1 | Same input projection yields the same plan; no operation starts without a durable intent receipt. |
| ROP-3: exact-binary capability report, isolated generation preparation, active-pointer CAS and reconciliation | L0 `NexusCRM` | ROP-1, ROP-2 | Mixed source/payload/generation states fail; activation ambiguity is reconciled from an authoritative record. |
| ROP-4: namespace ingress policy and WorkItem ownership verifier | L0 `nexuscrm-connectors` | ROP-1, ROP-2 | Unknown nonterminal owner blocks; retained fence never reopens on failure; no pack-specific semantics enter L0. |
| ROP-5: declarative L1 operation profiles | L1 `nexuscrm-app-packs` | ROP-1 through ROP-4 | Profiles name only registered operation kinds, capability requirements, and verification; arbitrary hooks are rejected. |
| ROP-6: tuple/tag v3 adapter, projection transport, receipt publication, staging/Prod policy adapters | L2 `nexuscrm-pingcap-pack` | ROP-1 through ROP-5 | One candidate/plan flows from staging to Prod; L2 does not contain a parallel planner or executor. |
| P0 input A: 103-WorkItem recovery | Parallel P0 owner | ROP-4 interface | Supplies a governed ownership/quarantine manifest and exact proof; this ADR neither modifies nor races that recovery. |
| P0 input B: Prod admission/failure-atomicity repair | Parallel P0 owner | ROP-3, ROP-4 contracts | Supplies Prod-safe admission and InstalledPack recovery semantics; it remains independent of this design document. |

### Migration sequence

1. Publish the v3 schema and an L0 planner in read-only shadow mode. It emits
   a plan and redacted receipt beside the current v2 evidence, but changes no
   release decision.
2. Require the shadow plan to match the currently executed deterministic
   transition for a representative set of candidates, including source advance,
   payload drift, no-op convergence, and orphaned WorkItem cases.
3. Make `PRECONDITIONS_PROVEN` a mandatory staging pre-mutation gate. A failed
   deterministic input must fail before ingress or InstalledPack mutation.
4. Introduce isolated generation preparation and L0 activation CAS, then make
   v3 tag eligibility require the same verified plan for Prod.
5. Enable Prod volatile reattestation and forbid deterministic plan derivation
   in Prod. Migrate v2 ledger/checkpoint readers to link to the v3 operation
   record.
6. Remove duplicated L2 phase logic only after parity, recovery, and SLO gates
   pass. Do not delete legacy evidence or add a broad framework rewrite.

## Test strategy and measurable SLOs

### Tests

- **Schema and unit:** canonical digest stability; secret rejection; unknown
  operation rejection; plan ordering; immutable resume; stale tag invalidation.
- **L0 transition tests:** Catalog candidate no-op/source-revision-advance plans, capability
  absence, payload/install/generation drift, ownership classifications,
  ingress-fence failure behavior, activation CAS conflict, and ambiguous commit
  reconciliation.
- **Profile tests:** each L1 profile admits only registered operation kinds and
  declares complete preconditions, verification, and recovery class.
- **Fault injection:** failure before the intent receipt, after fence acquire,
  after InstalledPack mutation, during verification, at activation CAS, and
  after route acknowledgement. Assert the recorded side-effect boundary and
  only the approved compensation/forward-recovery path.
- **Environment tests:** staging runs the full deterministic plan before
  mutation; Prod performs an identity-preserving dry-run reattestation before
  any write. A shared-environment test uses one candidate WIP lease and a
  deliberate baseline change to prove invalidation.
- **Acceptance readback:** tag target, candidate, runtime binary, final-pack,
  active generation, ingress policy, WorkItem digest, and redacted receipt
  manifest are independently compared after activation.

### Initial SLOs

| Measure | Initial target |
| --- | --- |
| Deterministic Prod-only operations | 0 for every v3 tag |
| Mutation before all deterministic preconditions | 0 |
| Candidate/plan/binary identity mismatch accepted on resume | 0 |
| Required receipt publication and redaction | 100% |
| Shared staging candidate WIP | 1 active mutation candidate per environment |
| First-attempt promotion success | at least 80% over the first rolling 10 v3 candidates; raise to 95% only after the first threshold holds |
| Ambiguous side effect without a recorded recovery class | 0 |

Workflow conclusions, invalid dispatches, cancellations, staging target-state
success, Prod activation, and independent live proof are reported separately;
none is a substitute for another.

## Adversarial design challenges

### 1. High-change incident sample bias

**Challenge:** The recent four-gate sequence is a high-change incident sample.
Designing only for its exact HubSpot and InstalledPack details could miss quiet
pack transitions and turn a single incident into a permanent special case.

**What survives:** immutable candidate identity, exact-binary proof, a
pre-mutation deterministic plan, explicit side-effect boundaries, and
generation-safe activation are provider-neutral invariants.

**What changes:** acceptance cannot use only the incident cohort. The planner
test matrix must stratify Catalog no-op, source revision advance, payload drift, install repair,
generation advance, and no-connector profiles; fault injection supplies cases
that production history has not observed. P0's 103-row manifest remains an
input case, not the schema for all WorkItems.

### 2. Non-hermetic shared environments

**Challenge:** staging shares control-plane state and can change while a
candidate waits. A reset, a healthy target runtime, or a successful staging
apply cannot make it a hermetic copy of Prod.

**What survives:** one mutation WIP lease, durable ingress fence semantics,
baseline projection digest, immutable plan, and exact reattestation preserve a
safe decision boundary.

**What changes:** the system must treat any deterministic baseline mismatch as
tag invalidation and restaging, not a reason for Prod to recalculate the plan.
Staging receives an explicit shared-state policy adapter and reports its
rehearsal result separately from the source projection and Prod promotion
eligibility. Volatile drift may wait or block in Prod; it cannot silently be
absorbed into the candidate.

### 3. Giant-framework risk

**Challenge:** Combining every release check into a new generic engine could
move L1/L2 business semantics into L0 and simply replace long scripts with a
large framework.

**What survives:** a deliberately small L0 kernel: canonical identity, plan,
precondition evaluator, receipt/state recorder, registered operation executor,
compensation classifier, and activation/reconciliation primitive.

**What changes:** L1 profiles are declarative data over registered operations;
L2 policy adapters select profiles and environment credentials but cannot add
arbitrary executable callbacks. Any new operation kind requires an L0 contract,
owner, tests, deterministic inputs, verification, and recovery semantics. This
keeps one generic protocol while preserving L0/L1/L2 ownership.

## Governance and non-goals

The release-acceptance authority is the tag-bound candidate bundle plus its
operation receipts. A dashboard, merged PR, workflow `success`, HTTP 200, or
processed WorkItem is evidence only; it cannot override a missing receipt or an
identity mismatch. A previous full blocker list is superseded when a new plan
digest or candidate digest is accepted; its receipt must point to the newer
identity rather than remain as a competing readiness list.

This ADR does not authorize an implementation, a release, a tag, a queue
replay, a WorkItem rebind, a database write, a HubSpot admission change, a
workflow dispatch, or a Prod operation. It does not decide the P0 recovery
manifests. Those remain separately governed inputs to the protocol.

## Cold-read check

A fresh maintainer can identify the one immutable artifact (`candidate_digest`),
the only allowed Prod change to an accepted plan (read-only volatile
reattestation), the exact point where side effects begin, the no-go path for
ambiguous activation, and the repository that owns each implementation slice.
That is the intended post-read action for this ADR.