# Issue 798 Recall FieldUpdate Durable Replay Status: implementation-ready Scope class: approval lifecycle replay substrate Date: 2026-05-15 Issue: https://github.com/pingcap-inc/NexusCRM/issues/798 ## Purpose Recall approval lifecycle FieldUpdate actions must survive a process crash after the recall state commits. The durable source of truth is a replay queue row written in the same transaction that marks the `ProcessInstance`, `ProcessInstanceNode`, and pending work items as removed. This is generic approval runtime substrate. It does not branch on SObject names, profiles, record types, tenant metadata, or field values. ## Current Failure The historical bug was a split-brain window: recall core state could commit as `Removed` while a post-commit FieldUpdate failed or was lost on crash. The current main branch avoids that specific window by executing Recall FieldUpdates inside the recall transaction. That is safe but does not satisfy the durable replay contract from issue #798: - Recall latency still includes linked-record mutation cost. - A transient target-record failure aborts recall instead of letting durable replay complete after recovery. - There is no durable failure shelf, retry budget, or operator-visible escalation for the lifecycle side effect itself. ## Contract - Recall writes one durable replay row for all Recall FieldUpdate actions found on the approval process. - The row is inserted inside the same transaction that commits the recall core state. If the enqueue fails, the recall transaction rolls back. - The row payload captures the concrete FieldUpdate instructions at recall time: action id/name, original field path, field value, process instance id, process definition id, object API name, source target record id, actor session snapshot, and the frozen mutation target: update object API name, update record id, and update field API name. - Relationship paths are resolved before enqueue while the recall transaction is open. Replay must not re-resolve lookup paths later; otherwise a lookup changed after recall could make the worker mutate a different related record than the lifecycle action targeted when recall committed. - The replay worker processes rows at least once. A crash before processing leaves the row due for the next worker. A crash during processing leaves a leased row reclaimable after lease expiry. - FieldUpdate replay runs in system context, preserving the approval runtime rule that metadata-defined lifecycle actions are system-owned effects. - Replays are idempotence-aware: before writing a field, the worker skips the mutation when the target field already has the desired value. This protects the common "target update committed, status mark did not" retry edge. - Retry policy is configurable through environment-backed worker config: attempts, initial backoff, maximum backoff, poll interval, batch size, and lease duration. - Retry exhaustion writes a durable failed row with `FailedAt`, `LastError`, and `AlertedAt`, and emits an operator log/metric signal. - Submit and final-approval FieldUpdates remain synchronous in this issue. They are documented as follow-on candidates for the durable channel. ## Schema Add `_System_ApprovalActionOutbox` as an L0 infrastructure table: - `Id BIGINT AUTO_INCREMENT PRIMARY KEY` - `LifecycleEvent VARCHAR(64)`; currently `Recall` - `ProcessInstanceId VARCHAR(255)` - `TargetObjectId VARCHAR(255)` - `Payload JSON` - `Status VARCHAR(32)`; `pending`, `processing`, `processed`, `failed` - `Attempts INT` - `NextAttemptAt DATETIME(6)` - `LeaseHolderId VARCHAR(128)` - `LeaseExpiresAt DATETIME(6)` - `LastError VARCHAR(1024)` - `ProcessedAt DATETIME(6)` - `FailedAt DATETIME(6)` - `AlertedAt DATETIME(6)` - standard created/modified timestamps Generic CRUD writes are blocked. Read access is restricted to `ViewAllData,ModifyAllData` because payload values may carry record data. ## Worker Design 1. Claim due rows with `Status = pending AND NextAttemptAt <= now`, plus stale `processing` rows whose lease expired. 2. Claim is a conditional `UPDATE` to `processing` with a lease holder and expiry. This is TiDB-safe and avoids relying on `SKIP LOCKED`. 3. For each claimed row, run replay in a fresh transaction: - unmarshal and validate payload - execute only `Recall` FieldUpdate instructions - mark the outbox row `processed` in the same transaction as the target mutation 4. On replay failure, roll back the target transaction, then schedule retry or mark failed in a separate row-state update guarded by the lease holder. 5. On exhausted attempts, set `Status = failed`, `FailedAt`, `LastError`, and `AlertedAt`, increment failure metrics, and log an `OPERATOR_ALERT` line. This two-transaction claim/process shape gives the important crash properties: - Crash after recall commit and before claim: row remains pending. - Crash after claim: row is reclaimed after lease expiry. - Crash during target update: target transaction rolls back; row is reclaimed. - Crash after target update and processed mark commit: both effects are durable. ## Service Changes `ApprovalService.Recall` changes only at the lifecycle action boundary: - Keep resolving Recall actions inside the recall transaction. - Keep updating/deleting approval runtime rows inside the recall transaction. - Replace inline Recall FieldUpdate execution with a durable enqueue. - Continue returning Alert actions for best-effort post-commit notification delivery. - Fail closed if Recall FieldUpdates exist but the durable outbox is not wired. `Approve`, `Reject`, and `Submit` keep their current behavior in this PR. ## Observability Prometheus counters: - `nexuscrm_approval_action_replay_attempts_total{lifecycle_event,outcome}` - `nexuscrm_approval_action_replay_exhausted_total{lifecycle_event}` `/internal/sentry-metrics` adds approval action replay counts grouped by status. The failed row itself is the durable operator record. ## Tests Backend unit tests: - Recall with a FieldUpdate enqueues a replay row and does not call the target record update inline. - Recall with a related-record FieldUpdate freezes the concrete update object, record id, and field in the replay payload. - Replay applies the frozen update target even if the source record's lookup now points somewhere else. - A pending Recall replay row drains after "restart" by constructing a new worker over the same store and running one poll. - Permanent replay failure retries until exhaustion, then writes a durable failed row with `AlertedAt`. - A successful replay marks the row processed in the same worker pass. - Retry backoff calculation respects the configured attempts/backoff policy. Focused command: - `cd backend && go test ./internal/features/automation ./internal/interfaces/rest` Baseline commands after implementation: - `cd backend && go test ./...` - `cd shared && go test ./...` - `cd frontend && npm run build` - `git diff --check` `npm run build` rewrites `frontend/public/version.json`; revert that generated change unless the version file is intentionally in scope. ## Design Iteration Notes 1. Reusing `_System_OutboxEvent` looked attractive, but its worker assumes record-event payloads and immediately decodes them into `RecordEventPayload`. Teaching it approval-specific payloads would broaden a hot shared path and still would not provide per-event backoff without changing the generic table. 2. Keeping Recall FieldUpdates inline avoids split-brain but fails the durable retry and operator-failure requirements. It also keeps target mutation cost on the synchronous recall path. 3. A post-commit in-memory hook was rejected because the issue requires process crash recovery. The row must exist before the recall transaction commits. 4. A dedicated approval action queue is the narrowest substrate: it is generic to approval lifecycle replay, durable, backoff-aware, and leaves the generic record event outbox untouched. 5. Final pre-implementation confidence check: every Recall FieldUpdate outcome has exactly one durable owner, `_System_ApprovalActionOutbox`; successful replay and target mutation commit together; failed replay remains visible to operators; and the design does not introduce object-specific platform code. 6. Implementation validation found one L0 inventory gap: the new `_System_ApprovalActionOutbox.TargetObjectId` column is a runtime-data reference, not a platform-owned schema reference. The inventory classification was added so future L0 checks continue to account for this surface explicitly. ## Follow-On Submit and FinalApproval FieldUpdates should move through the same durable channel only after their ordering requirements are separately designed. Submit FieldUpdates may feed entry/step criteria, and final approval actions may have different response expectations, so this PR intentionally limits the first durable channel to Recall.