# Outbox #3477 rollout gate

Status: **minimal gate**. Issue: #3477.
Companion docs: [outbox-redesign.md](outbox-redesign.md), [outbox-idempotency-audit.md](outbox-idempotency-audit.md), [cascade-delete-lock-scope.md](cascade-delete-lock-scope.md).

`OUTBOX_PULL_WORKER` may flip on only when every immediate item below is true. Until then, the existing drainer remains the fallback. The old and new drainers must be mutually exclusive.

## G1 - Pull Worker Isolation

- [ ] `ProcessOutbox` no longer runs a per-tick fan-out behind one `wg.Wait()` barrier.
- [ ] Long-lived workers independently select, claim, process, and finalize one event at a time.
- [ ] The implementation reuses the existing durable state transition: `ClaimEvent` moves the row to `processing`.
- [ ] Pull workers commit that `processing` claim before handler work and refresh it while the handler is live, so unrelated workers do not pile up behind an uncommitted claim.
- [ ] A blocked handler occupies only its worker; a test proves unrelated events still process while one event is blocked.

## G2 - RecordKey FIFO

- [ ] `_System_OutboxEvent` has additive nullable `RecordKey`.
- [ ] Runtime probes the physical table before using `RecordKey`; pre-apply restarts keep ordinary enqueue on the legacy column set and keep `OUTBOX_PULL_WORKER` on the legacy batch drainer.
- [ ] Enqueue populates `RecordKey` from the same object/record identity currently used by `slotForEvent` only after the physical column exists; malformed new events fall back to event Id behavior.
- [ ] After schema apply, every outbox-writing backend process is restarted before `OUTBOX_PULL_WORKER` is enabled, so a process that cached "RecordKey missing" cannot keep writing null keys and the one-shot backfill worker observes the post-apply table.
- [ ] Existing backlog is drained before cutover or backfilled with `RecordKey` using the same extraction logic, so old and new rows for one record cannot bypass each other.
- [ ] Eligible-event selection uses stable `(CreatedDate, Id)` order and excludes a younger event when an older same-`RecordKey` event is not terminal (`processed` or `failed`), including stale `processing` rows.
- [ ] Tests prove same-record FIFO, stale-head retry before younger same-record work, and cross-record progress under concurrent workers.

## G3 - Failure/Retry Bookkeeping

- [ ] The pull-worker repair does not introduce a generic handler wall-clock timeout.
- [ ] Retry bookkeeping for handler failure and DB lock timeout rolls back handler work to a post-claim savepoint and writes retry/failed state before releasing the attempt; transaction/setup/commit failures are retried through detached bookkeeping while the visible claim remains heartbeat-protected.
- [ ] A just-failed retry row cannot be immediately reclaimed in a tight pull-worker loop.
- [ ] Tests prove a failed or lock-timed-out event increments retry count or reaches `failed` at `MaxRetryAttempts`; it must not reappear forever with the old retry count.

## G4 - Proven DB-Internal Idempotency

- [ ] The proven duplicate create's natural key is confirmed from current metadata/code.
- [ ] A DB `UNIQUE` constraint backs that natural key.
- [ ] Create path is find-or-create/upsert and short-circuits before `Insert` on redelivery.
- [ ] Redelivery and concurrent-redelivery tests prove exactly one created row and no autonumber burn.
- [ ] The named after-save field-capture RHS values are confirmed deterministic, or the non-deterministic case is keyed/reworked.
- [ ] Redelivery tests prove the field values are unchanged on a second delivery.

## G5 - Current External-Effect Surface

- [ ] Derive the current-tree outbox consumer map before flipping `OUTBOX_PULL_WORKER`.
- [ ] If record-triggered flows on this path can execute email, webhook, connector, or other provider I/O, each live consumer is either protected by an existing durable idempotency/attempt contract or explicitly scoped out of the pull-worker rollout.
- [ ] The rollout claim does not say "email exactly once" or "all future external effects are idempotent"; network effects remain at-least-once unless a consumer-specific dedup/idempotency contract proves stronger behavior.
- [ ] New external-effect action types added later must provide their own idempotency classification before they are allowed on the pull-worker path.

## Cascade Track

Cascade lock-scope repair is independent and has its own design/gate in [cascade-delete-lock-scope.md](cascade-delete-lock-scope.md). It is not required before `OUTBOX_PULL_WORKER` can be reviewed, but it remains a separate #3477 follow-up.

## Deferred From This Repair

The following are explicitly out of this implementation:

- `source_event_id`, `handler_id`, `effect_key`, and `attempt_id` plumbing;
- durable send-command rows and separate sender workers;
- owner-token leases, `LockedUntil`, `NextVisibleAt`, and retry backoff scheduling;
- generic handler wall-clock timeout;
- lanes, tenant fairness, candidate spreading, dedicated worker DB pool, and poison-head parking;
- source-scan seals for future external-effect call sites.

These can return when scale evidence requires them or when a future external-effect class lacks its own durable idempotency contract. They should not expand the immediate queue-mechanics fix, but G5 is still a gate for any current external consumers that would actually run behind `OUTBOX_PULL_WORKER`.