# WorkItem Dead-Letter and Replay Authority

## Scope

This contract defines the L0 connector operator path for Event-backed work.
It covers dead-letter list/detail, replay reservation, replay lineage, and the
bounded compatibility gate while Event executor mirrors remain installed.

The contract applies to both `connector_event` and
`workflow_action_apply`. Provider and tenant behavior remains outside this
mechanism.

## Ownership gate

`_System_ConnectorEvent` owns immutable provider, routing, delivery, and audit
context plus the bounded Event executor before cutover.
`_System_ConnectorEvent.ExecutionOwner` records the rolling-version handoff.
`_System_ConnectorWorkItem` owns mutable execution state after its atomic claim
changes that owner to `work_item`; the same statement increments `Attempt`.

The read and replay paths use the same durable gate:

1. A matching Event-backed WorkItem with `Attempt > 0` owns state, failure
   budget, terminal time, replay reservation, and replay lineage.
2. A zero-attempt WorkItem is a compatibility mirror and leaves authority on
   the Event row while still contributing its `WorkId` and `ProjectionKey` to
   replay lineage and source-projection conflict checks. Its WorkItem `State`
   does not decide R1.2 replay eligibility. If the Event executor wins and
   reaches processed, skipped, or dead, claim-time convergence copies that
   terminal classification to prevent duplicate execution and performs any
   source-projection successor handoff; Event remains the replay owner.
3. An Event with no WorkItem uses the legacy Event reservation path. A
   source-projection replay still derives its current `ProjectionKey` and fails
   closed when another nonterminal generation owns it.
4. An installation without the additive WorkItem failure/replay schema uses the
   legacy Event path until that schema is available.
5. Detail and replay inspect multiple Event-backed candidates and fail closed
   on ambiguous ownership. List rejects duplicate candidates that survive its
   dead-row projection; it is an operator view, not a complete ownership-audit
   endpoint.

This gate is durable and process-independent. An in-memory feature flag cannot
select operator authority because the management HTTP service and queue workers
may have different service instances.

## Operator read path

The WorkItem-owned projection joins terminal WorkItem execution state to its
immutable Event context. It exposes:

- WorkItem identity and effective state
- fixed-shape failure counts and timestamps
- terminal and replay timestamps
- `ReplayOfWorkId` and Event audit lineage
- namespace, pack incarnation, event kind, idempotency identity, and delivery
  ordering metadata

The permissioned operator service and public admin response omit payloads,
credentials, lease tokens, worker identity, raw provider response bodies,
`FailureClass`, `FailureSummary`, and other caller-authored or secret-bearing
diagnostics. Existing view and replay permission keys remain the authorization
boundary.

The REST handler preserves the published Event-shaped
`ConnectorEventsAdminService` interface. The in-tree service advertises the
WorkItem list/detail projection through an additive capability; pinned services
and custom mocks continue through the legacy interface without a source
compatibility break.

## Replay transaction

Replay follows the global transaction lock hierarchy: active pack, Event
compatibility row, then WorkItem. Enqueue uses active pack before Event and
WorkItem materialization. Active WorkItem retry and completion mutate only the
WorkItem row; legacy compatibility bridges retain their explicit guarded order.

For a WorkItem-owned generation:

1. Lock the requested active pack incarnation.
2. Lock the Event for immutable replay context and the pre-cut
   `Event.ReplayedAt` compatibility fence. Other Event lifecycle fields do not
   participate in WorkItem-owned replay authority.
3. Inspect both Event-backed WorkTypes because compatibility rows can predate
   the `workflow_action_apply` split. Prefer the Event-derived type and fail
   closed if more than one candidate exists.
4. Require the selected WorkItem to have `Attempt > 0`.
5. Require terminal `dead` state and atomically set the original WorkItem's
   `ReplayedAt`.
6. Leave Event status, retry, claim, error, terminal, and replay fields
   unchanged. A committed pre-cut `Event.ReplayedAt` fails closed so an older
   replay service and the WorkItem-owned path cannot create two children.
7. Derive the original execution class from its WorkItem generation identity.
   Event-only rows with a source lane derive the class from their immutable
   original route snapshot and fail closed if that snapshot is unavailable.
8. Require the caller-resolved current route version to remain live and retain
   the original source-projection versus per-event execution class while
   inserting a new immutable replay Event.
9. Create a distinct WorkItem generation whose `ReplayOfWorkId` names the
   terminal original.

For a legacy generation, the Event `dead` state and `ReplayedAt` compare-and-set
remain the reservation authority. An Event-only generation materializes a fresh
WorkItem without WorkItem lineage. A terminal zero-attempt compatibility mirror
materializes a fresh WorkItem whose `ReplayOfWorkId` names that mirror, even
though the Event remains the replay reservation owner. The zero-attempt
mirror's own WorkItem state can still be pending, retry, blocked, or dead when
the Event-owned reservation begins. The same transaction terminalizes that
mirror, records its `TerminalAt` and `ReplayedAt`, clears projection-active and
lease fields, and only then creates the child. A processing, processed, skipped,
already-replayed, or otherwise conflicting zero-attempt mirror fails closed.
These paths retain explicit replay semantics after materialization derives the
current projection identity.

Every reservation, compatibility-fence check, Event insert, and WorkItem insert
commits in one transaction. Pack rotation, route sweep, duplicate reservation,
or materialization failure rolls the entire transaction back.

## Source-projection replay

`ProjectionKey` is stable projection identity; `WorkId` is immutable generation
identity. Replay preserves the terminal original and creates a distinct
generation. It retains `ProjectionKey` only when the replay target has the same
pack incarnation, route version, companion identity, work type, and source
lane. Re-resolving the current pack or route can create a new projection
identity; `ReplayOfWorkId` preserves lineage across the keys.

Replay also derives an identity-independent source scope from immutable
namespace, canonical event kind, and source lane. If any other generation in
that scope is nonterminal, replay returns a conflict without consuming Event or
WorkItem replay reservation. The check spans pack, route, companion, work-type,
and `ProjectionKey` changes, and includes queued successors whose
`ProjectionActiveKey` is null. It also applies when the requested original is a
zero-attempt compatibility mirror or has no WorkItem at all. Identity-changing
replay can proceed only after every prior generation in the source scope is
terminal.

The target route must remain a settled source-projection route. A change to
per-event invocation returns a conflict and rolls back both replay reservations
and the provisional Event. An Event-only generation with a source lane also
fails closed when its original route snapshot is no longer available to prove
the execution class.

During the compatibility window, a newer nonterminal Event can exist in the
same source scope before its WorkItem is materialized. Replay holds the active
pack lock, locks this Event-only successor, and fails closed before the normal
projection upsert can retarget or skip it. The transaction rolls back the
original reservation and the provisional replay Event together.

## Invariants

1. A WorkItem-owned DLQ row is replayable through the WorkItem reservation path
   regardless of divergent Event lifecycle fields, provided the pre-cut Event
   replay fence is clear.
2. A terminal WorkItem is never reopened.
3. One original WorkItem can produce at most one replay child.
4. Concurrent replay has one winner.
5. Event and WorkItem replay lineage commit together.
6. Old-schema and zero-attempt behavior is explicit compatibility, not a second
   active WorkItem read path. A successful zero-attempt replay leaves no
   claimable original WorkItem generation.
7. Operator responses contain fixed-shape execution metadata and immutable
   context, never provider secrets, raw failure bodies, or caller-authored
   failure class/summary fields.
8. WorkItem-owned replay cannot regress or overwrite Event audit state.
9. Ambiguous Event-to-WorkItem ownership cannot be detailed, reserved, or
   replayed. List rejects duplicate rows present in its dead-row result.

## Validation

Required TiDB evidence covers:

- WorkItem-owned and legacy rows in the same DLQ result
- zero-attempt and old-schema fallback
- zero-attempt replay lineage with Event-owned eligibility, atomic mirror
  terminalization, conflicting-state rejection, and source-projection
  conflicts
- Event-only source-projection replay success and active-successor conflict
- WorkItem-owned replay blocked by an Event-only nonterminal successor
- WorkItem-owned and Event-only source projections rejected after a target
  route changes to per-event invocation
- Event-only source-lane replay rejected when its original route snapshot is
  unavailable
- compatibility WorkType lookup and ambiguous-owner rejection
- divergent pending, processing, failed, processed, and skipped Event lifecycle
  fields ignored by dead WorkItem replay authority
- committed pre-cut Event replay marker fencing WorkItem-owned replay
- legacy-only admin service compatibility
- distinct ordinary and source-projection replay generations
- same-identity replay retaining `ProjectionKey` and identity-changing replay
  preserving lineage under a distinct key
- `ReplayOfWorkId` lineage and preserved terminal history
- one-winner concurrent replay
- replay waiting for the active-pack row before attempting Event or WorkItem
  locks
- replay preserving active-pack-before-Event-before-WorkItem order while active
  retry and completion use WorkItem-only lease compare-and-swap
- active source-projection successor conflict with unconsumed reservations
- rollback on pack, route, Event insert, and WorkItem materialization failure