# L0 Worker Lifecycle and Reliability Contract Status: first mechanism slice. This document does not declare any production work-family migration complete. ## Outcome Every production-reachable asynchronous path declares what a successful producer acknowledgement promises and proves that promise across crash, restart, retry, and ownership transfer. Common process orchestration is reused; domain state and effects retain one owner and one read path. ## Current facts The independently runnable Runtime Worker is a process composition, not one universal worker engine. Its supported work families use several control loops: Audit outbox, record-command lifecycle, file-upload lifecycle, Scheduled Flow, and the metadata worker shared by record events, rollup delta, and platform commands. Monolith and Integration workers add more loops. `_System_RuntimeWorkOwnership` and `_System_RuntimeWorkFence` already own which deployment generation may claim or commit Runtime work. They do not own whether producer acceptance is durable, when work becomes due, a row claim token, heartbeat, retry, or remote-effect outcome. The architecture explicitly does not move all asynchronous work into one generic queue table. Flow, Integration, Bulk, audit, and derived-state stores retain their own canonical state machines. ## Reliability classification Each work family declares one durability class: - `durable-critical`: acknowledgement promises that accepted work survives a process crash and reaches a durable terminal, poison, or uncertain state. - `derived-reconcilable`: canonical source state remains authoritative and a fresh process can converge the derived effect without replaying every event. - `best-effort`: loss is an explicit product contract; the path must not claim completeness. It also declares effect semantics: - `transactional`: effect and disposition commit in one domain transaction; - `idempotent-or-observable`: a stable key or queryable outcome makes replay safe; - `outcome-uncertain`: ambiguous external results are quarantined, never blindly replayed; - `convergent`: repeated derivation converges to canonical source state; - `loss-tolerant`: loss is allowed by the best-effort contract. The declaration grammar rejects incoherent combinations. It is not proof. Real producer/store/worker tests provide proof. ## Ownership | Fact or decision | Sole owner | |---|---| | Producer acknowledgement and atomic durable acceptance | Domain producer transaction/store | | Work identity, queue state, claim token, lease, heartbeat | Domain store | | Claim ordering, tenant/lane fairness, admission policy | Domain source, which has those facts | | Effect execution, retry, poison, external uncertainty | Domain attempt | | Deployment generation and Runtime effect fence | Existing Runtime work ownership authority | | Local wake, exact due timer, continuation, capacity, reconcile cadence, drain | `nexuscrm-worker` controller | | Queue depth, age, poison, uncertain outcomes | Domain observations | | Process activation/claim/completion observations | `nexuscrm-worker` snapshot | The controller never changes a domain row after an attempt error. An attempt returns nil only after its domain has durably handled the outcome. A non-nil attempt result is process-fatal because the controller cannot safely infer whether replay is allowed; the controller becomes permanently unusable and rejects another Run. ### Protocol authority and proof boundary | Surface | Meaning owned by the controller | What it does not prove | |---|---|---| | `CheckReady` | Read-only startup gate | Store durability or schema correctness | | generation probe | Read-only change hint; a changed cursor requests reconciliation | Work existence, ordering, or ownership | | `Reconcile` | Serialized opportunity to materialize derived state or durable acceptance | Safe replay after an error unless no commit is proven | | `Claim` | Sole immediate eligibility and durable-ownership transition | Correct domain SQL, fairness, or cross-process uniqueness | | `ClaimBatch.More` | Sole authority for an immediate bounded continuation | Future eligibility after a terminal batch | | `NextWorkAt` | Read-only, strictly-future execution opportunity after a terminal batch | Immediate work authority or a reason to replay an overdue empty claim | | `Wake` | Coalesced process-local claim hint | Durable work, reconciliation, or delivery guarantee | | periodic reconciliation | Bounded liveness backstop | A business-queue polling cadence | | `Attempt.Run` | One already-claimed heartbeat/effect/disposition lifecycle | Replay safety after a non-nil result | Controller tests prove process-local ordering, capacity, fail-stop behavior, and Source-call budgets. They cannot prove database atomicity, distributed claim uniqueness, durable recovery, effect idempotency, or deployed RU. The shared crash matrix proves that concrete lifecycle objects are created and passed in the required sequence; only each migrated adapter's real store/effect assertions can prove what those callbacks actually did. Until a production adapter passes that conformance and before/after environment evidence, this PR is mechanism, not a reliability or RU result for any work family. ## Controller protocol 1. Validate source readiness. 2. Establish the optional generation cursor. 3. Reconcile once before reporting Ready. 4. Claim no more than immediately free execution capacity. 5. Execute already-claimed attempts; each attempt owns heartbeat and durable disposition. 6. Continue immediately only when the source requested a bounded continuation. 7. Whenever a claim reports no immediate continuation, arm a strictly-future domain-owned `NextWorkAt` timer whether the terminal batch was empty or non-empty. A timestamp that is already due is not re-armed: the authoritative claim just declined immediate continuation, so retrying the same stale or concurrently consumed scheduling fact at zero delay would create an empty query loop. The controller records the overdue signal and waits for `Wake`, reconciliation, or generation change. Attempt completion only frees local capacity; it does not imply more durable work. 8. A process-local `Wake` skips reconciliation and offers a new claim turn. 9. `RequestReconcile`, a generation change, startup, or the bounded periodic backstop reconciles before another claim turn. 10. Cancellation stops new claims, cancels attempts, and waits for the declared shutdown bound. If an attempt ignores cancellation beyond that bound, the controller reports the remaining in-flight work and permanently rejects reuse. The process supervisor must terminate that process before replacing the controller because the abandoned attempt may still be executing. `Reconcile` and `Claim` may mutate durable state. Their errors are therefore outcome-unknown by default: acceptance or ownership may have committed before the response was lost. An outcome-unknown error permanently disables controller reuse, including when cancellation caused the response loss. A Source may wrap `ErrSourceOperationNotCommitted` only when it can prove that no durable transition committed. Returning that marker from `Claim` together with claimed attempts is contradictory and fails closed. Domain conformance tests must prove the rollback/no-commit boundary before using the marker. `CheckReady`, `NextWorkAt`, and generation probes are read-only, so their errors do not poison the controller. All Source operations must honor context cancellation. A controller cannot safely abandon an in-progress durable claim or reconciliation operation; each domain migration must test that its store calls return when cancellation is propagated. Wake and generation are latency hints. Claim remains authoritative. Periodic reconciliation is the liveness backstop for a lost wake, bypassed producer, or drift; it must not become a hot poll of the business queue. ## Module boundary The implementation is the stdlib-only module `github.com/nexuscrm/worker` in the existing NexusCRM repository. This does not create an eighth repository. It does not belong in: - `nexuscrm-api-host`, which owns transport and request/response process probes; - `nexuscrm-contract`, which is the runtime-pure semantic dependency root; - `nexuscrm-persistence`, because the loop owns no storage; - `nexuscrm-connectors`, because connector code cannot own platform-wide work; - `backend/internal/runtimeworker`, which is inaccessible to other release boundaries and remains composition glue. ## Conformance `github.com/nexuscrm/worker/workertest` provides shared crash-point vocabulary. A durable-critical family must inject those points into its real producer, store, claim, effect, heartbeat, disposition, and ownership paths. After each injected crash the matrix constructs a fresh worker against the same durable store fixture and invokes the domain-specific outcome assertion. Required common boundaries are before/after acceptance, after claim, before/after effect, before/after disposition, during heartbeat, and ownership transfer, including lease expiry. The shared matrix requires each boundary to actually inject a crash. It owns the lifecycle proof rather than accepting self-reported identity strings: the matrix receives one non-nil concrete store fixture, calls the domain's `NewWorker` twice with that exact fixture, requires two distinct concrete worker pointers of the same type, passes the first into the crash callback and the second into restart/drain, then passes the same store fixture into the domain recovery assertion. The domain callbacks remain responsible for exercising the real transition and persisted state paths; a wrapper that ignores the supplied worker or store is not conformance. The matrix enforces construction, crash, restart, and recovery-assertion deadlines even when a callback fails to honor context cancellation, so a broken adapter cannot hang the suite indefinitely. Correctness conformance also covers startup ordering/readiness, cancellation of every Source operation, signal coalescing, exact-due scheduling, bounded claim capacity and continuation, attempt panic/error shutdown, graceful drain, shutdown timeout fail-stop behavior, generation change/loss recovery, and snapshot accounting. Passing the controller unit tests is not a substitute for running these cases against each migrated domain adapter. ## Performance contract The controller has no empty-queue poll ticker. After startup reaches an empty claim, another domain read may only be caused by an exact due timer, a coalesced wake, a changed generation, or the configured low-frequency reconciliation backstop. An unchanged generation probe must not invoke reconciliation, claim, or next-work queries. Attempt completion while a future due timer is armed must not invoke claim or next-work queries either. If `NextWorkAt` is already due after a terminal claim batch, the controller must leave the timer disarmed and expose the suppression count; the periodic reconciliation backstop provides bounded recovery instead of an immediate empty-query loop. Controller tests enforce source-call budgets and signal coalescing. Benchmarks cover only process-local CPU/allocation behavior. They do not prove TiDB RU. Every production adapter migration must record, before and after rollout, the surviving statement digest and plan, execution count, total/average read RU, TiDB and resource-control queue latency, examined/processed keys, workload cardinality, and replica count. Claim and `NextWorkAt` SQL must be bounded by the domain eligibility keys. Reconciliation must not hide a high-frequency exact aggregate or compete with the business connection pool. ## Delivery slices 1. Common module, classifications, controller, fault vocabulary, CI and architecture boundary. No production ownership change. 2. Bulk: durable Job claim and idempotent batch/segment progress. 3. Scheduled Flow: per-schedule occurrence cursor and transactional reservation/cursor advancement. 4. Paused Flow: token lease, heartbeat, and fenced completion/recovery. 5. Formula View convergence and explicit audit/telemetry classification. Each migration has one active claim path. An ownership epoch or equivalent cutover fence must prevent the legacy loop and new controller from executing the same family concurrently. Old loops are deleted in their migration PR, not left as fallback.