# Connector Inbound Scheduler Contract This document records the L0 terms used by the connector inbound pressure isolation work tracked by `pingcap-inc/NexusCRM#3337`. ## Terms - Admission budget: foreground capacity reserved for provider-facing webhook admission, verified receipt materialization, route/capability lookup, and durable enqueue. Admission overload returns bounded retryable `503` with `Retry-After`; accepted work returns `202` only after the durable boundary. - Worker drain budget: background capacity for `_System_ConnectorWorkItem` processing rows. It is capped by `NEXUSCRM_CONNECTOR_INBOUND_MAX_PROCESSING_ROWS` and the claim limiter, not by worker goroutine count. - Scheduler: a process-local, per-pack admission coordinator that owns queue polling, claim permits, queue-class reservations, and DB-pressure checks before work reaches an executor. It is not the distributed ownership authority for a WorkItem row. - Executor: a worker slot that receives an already-claimed WorkItem, loads its immutable connector Event input, and runs event preparation, route-version validation, batch read, flow execution, lease heartbeat, and finalization. - Queue-class reservation: a scheduler-owned permit scoped to a queue class set such as `interactive/default`, `bulk`, or shared all-class capacity. The reservation decides whether a claim scan may run and how many rows that claim unit may admit. ## Claim Ownership For every supervisor-managed worker pool, one `CentralClaimScheduler` is created per connector pack incarnation in each runtime process and injected into every executor for that incarnation, including a one-executor pool. The scheduler serializes `ClaimNext` and claim-limiter admission across that process's executor set. Raising `NEXUSCRM_CONNECTOR_INBOUND_WORKERS_PER_PACK` above one increases execution slots and restart capacity inside the process, but it does not create more concurrent queue scanners in that process. Distributed WorkItem ownership is decided by the WorkItem row compare-and-swap claim together with generation, eligibility, and lane-head predicates. Every runtime process may therefore contribute one process-local claim scanner and its executor capacity. A row can still be claimed by only one runtime, and ordered lanes still expose only their eligible head. WorkItem readiness is self-contained: `State`, `RunAfter`, `LeaseUntil`, lane identity, generation, source-projection activation, and immutable ordering fields decide selection and final claim CAS. A projection generation with `ProjectionKey` set and `ProjectionActiveKey` null is a durable queued successor and cannot participate in readiness or lane blocking until the prior generation activates it. Mutable Event status, retry time, claim timestamp, and terminal mirrors do not participate. Event joins remain limited to immutable delivery ordering and execution input lookup. During the bounded mixed-version window, Event terminal state is inspected only after a WorkItem final-claim CAS miss to converge an attempt-zero compatibility mirror; it never makes work eligible. `_System_ConnectorEvent.ExecutionOwner` is the durable rolling-version arbiter. It defaults to `event`. The first successful WorkItem claim changes the WorkItem lease and Event owner in one multi-table compare-and-swap. An Event-owned pending or failed row becomes `ExecutionOwner=work_item` with legacy `Status=skipped`; older binaries already treat that status as terminal, so their existing claim and finalizer predicates lose without understanding the new column. A fresh Event `processing` lease keeps ownership until it finishes. The WorkItem claimant may transfer a stale Event lease using the same legacy timeout, clearing its claim fields and installing the same skipped fence atomically. Later WorkItem retries accept only the stable `work_item`/`skipped` pair and leave all Event executor fields unchanged. If an older binary completes first, Event ownership remains `event` and the new claimant does not execute the WorkItem. It copies processed, skipped, or dead onto the attempt-zero compatibility WorkItem under Event-before-WorkItem locks. Source projections perform their normal terminal-generation successor handoff in that same transaction. This convergence records the old executor's outcome, advances queue ordering, and never runs business behavior or changes the winning Event owner. Every legacy Event list/claim surface applies a durable WorkItem ownership guard for current binaries. A direct Event-backed WorkItem row fences that Event, including an unclaimed generation, and a source-projection identity recorded by the current schema fences its whole source lane across active and terminal generations. The `ExecutionOwner`/`skipped` claim CAS supplies the old-binary-visible half of the same protocol. Together they keep superseded immutable Event facts from re-entering either queue after ownership transfers. Both guards are evaluated in the database. Every legacy Event claim scan and peek begins with the full durable guard; targeted lease CAS and mutation APIs retain the per-Event `ExecutionOwner` guard. Process memory never selects claim authority. A missing WorkItem table or required projection column is handled in stages: a missing generation column falls back to the historical nonterminal `csp:*` lane fence while the direct Event-to-WorkItem fence remains active; only a genuinely missing WorkItem table permits the unguarded schema-compatibility predicate. The supervisor starts a pool only after the complete WorkItem claim schema, Event `ExecutionOwner`, and historical Event-to-WorkItem mirror coverage are visible. A confirmed schema or coverage gap fails closed and stops an existing worker. A transient readiness-probe error preserves an already-running WorkItem worker and is retried by reconciliation. `SchedulerLease` no longer participates in supervisor-managed WorkItem execution. `NewWorkItemWorker` always uses WorkItem row CAS, skips lease hooks, and marks its claim contexts as distributed. Those contexts continue forwarding the stable owner/fence fields required by the atomic coalescing handoff. The legacy constructor retains its capability-based behavior for published custom claim services until the compatibility cleanup. Inbound claim, peek, and generation-resolution contexts also carry a normalized `ClaimWorkTypes` executor scope for `connector_event` and `workflow_action_apply`. The repository applies that scope before candidate limits and blocked-head promotion. Every supervisor-managed pool enters claim, peek, event-targeted claim, stale reclaim, active-generation resolution, single or batch immutable Event loading, failure, retry, lease, terminal, and atomic coalescing through one `WorkItemService`. Admission requires `HonorsClaimWorkTypes()` so unrelated Integration WorkItems cannot enter the inbound executor or satisfy its Event-coverage readiness gate. A confirmed incomplete-service capability gap fails closed and stops an existing worker, matching the schema and historical-coverage gates. Operational queue statistics use the same executor scope and WorkItem state as the claim path. Exact live statistics apply WorkItem due time, lease expiry, projection activation, Event ownership, and lane-head fences; cheap live statistics retain the same WorkItem ownership while omitting lane eligibility. WorkItem becomes the statistics authority after `Attempt > 0`, or after source- projection activation while the Event is still pending/failed on the bounded legacy path. An inactive attempt-zero compatibility mirror, plus any Event- owned processing or terminal winner, leaves mutable statistics on Event until claim or terminal convergence succeeds. During the rolling-version window, an Event row contributes a compatibility stat only when no authoritative direct or source-projection WorkItem covers that execution source at the selected projection boundary. Exact live compatibility counts reuse the legacy Event delivery/source-lane ordering guard, while cheap live counts retain raw due pressure. The existing Event statistics API and JSON response shape delegate to this unified read path. Before the additive WorkItem and Event `ExecutionOwner` schema is visible, those methods retain the base Event-only statistics query as a bounded schema-compatibility fallback; unrelated query errors still fail closed. Existing custom claim services that have not adopted the row-CAS capability retain their former acquire/renew hook lifecycle during the mixed-version compatibility window, and their claim contexts carry no distributed marker by default. The retained `UseDistributedWorkItemClaims` opt-in continues to skip those hooks and emit the marker for custom services that already implemented lease-free row claims before the capability interface existed. The DB-backed lease repository methods therefore remain operational for pinned/custom callers. Distributed claim contexts never consult SchedulerLease rows. Unmarked legacy compatibility requests retain the conditional lease join and final CAS predicate until that published path is retired, and the schema remains available until all older binaries have drained. The published split `WorkItemClaims` plus `WorkItemEventLoader` construction surface remains available through `NewWorker` during this window. Supplying the compatibility loader keeps those split surfaces authoritative. Its loader-free consolidated path is admitted only when the service-owned WorkItem repository is the exact configured retry/finalization authority. The live supervisor calls `NewWorkItemWorker` with one complete service; its adapter performs no SchedulerLease calls, Event executor mutations, or optional claim, loader, mutation, and coalescing capability probes. Queue contents do not control scheduler lifecycle. A live process-local scheduler is woken on reconcile and is rebuilt only when it is no longer running or its worker configuration fingerprint changes. In particular, blocked or delayed rows must not cause a healthy idle scheduler to be destroyed and recreated; eligibility changes are handled by normal polling and wake-up. Single-worker and fan-out packs both use the central claim-first path. Batch collection and enrichment prefetch occur only after the WorkItem claim CAS has established execution ownership. The immutable connector Event remains the delivery and audit fact. WorkItem is the execution source of truth for claim ownership, source-lane and delivery-lane ordering, stale reclaim eligibility, failure budget, lease renewal, retry, terminal state, and pack-generation scoping. Repository-backed execution loads Event only for payload, route, companion, source, delivery, idempotency, and audit context. Worker-facing processing and claim fields are synthesized in memory from the claimed WorkItem; renew, release, retry, fail, and complete use the WorkItem lease CAS and do not write Event executor mirrors. Ordinary `latest_by_source` coalescing accepts a fresh transferable Event or an Event already frozen under WorkItem ownership for retry. It completes the current WorkItem, claims the eligible successor, and performs any one-time Event ownership transfer in one transaction under lease-token and ownership-epoch CAS. A lost successor CAS rolls back current completion. Strict delivery ordering checks authoritative predecessor WorkItem state after Event ownership has transferred; source-scoped ordering relaxes only that cross-source delivery fence. `csp:` source-projection generations remain on their dedicated terminal-generation handoff. Event payload is read-only throughout, and Event executor fields are immutable after the one-time ownership transfer. Current-binary legacy Event claim, coalesced handoff, heartbeat, release, retry, failure, and terminal APIs apply the durable WorkItem guard at mutation time. Older binaries are fenced by the atomic Event owner/status transition. Exactly one side can acquire durable execution ownership, and only that side can finalize. Event-only rows retain a bounded, durably guarded fallback while they await migration or drain; a missing WorkItem table uses the separate schema-compatibility fallback. Published custom WorkItem claim and bridge interfaces remain available through the legacy constructor during the R1.4 compatibility window. Supervisor-managed workers receive the complete `WorkItemService` contract at construction and use WorkItem mutations directly, including `FailWorkItem` as the sole failure-budget transition. Retirement of the custom bridge methods, Event executor columns, SchedulerLease APIs, and compatibility flags requires the later mixed-version removal gate. WorkItem-owned replay reserves `WorkItem.ReplayedAt` and never writes the Event mirror. A committed pre-cut `Event.ReplayedAt` remains a read-only monotonic fence during the mixed-version window so an older replay service and the WorkItem-owned path cannot create two children. R1.4 removes this compatibility read only after pinned replay callers have converged and the marker has been backfilled or proven drained. Event-only generations that predate WorkItem materialization are reported as a coverage gap. Migration or drain must establish complete WorkItem coverage before the supervisor starts that pack. ### Event-to-WorkItem materialization concurrency record Run `gate1a-v023-local-r1` used production-behavior base `61123185cca08b75b0afb39d14be2d35da5139a2` (`v0.2.3`), the deterministic test seam and fixture in the containing change, local TiDB 8.5.0, and two real Event Service materializers. The fixture began with one historical Event that had no WorkItem. The winner repaired that row while holding the namespace and InstalledPack authority locks. Before the winner committed, the waiter established its pessimistic SI transaction snapshot, then waited for the same locks and attempted an 89-Event / 49-delivery group. The focused run passed 20 consecutive repetitions of the characterization plus three under the Go race detector. The waiter saw the historical Event through its old snapshot but not the winner's newly committed WorkItem. Its insert lost on `uniq_connector_work_item_event_type`; the duplicate handler then used the same nonlocking snapshot, missed the winner again, leaked MySQL 1062, and rolled back the other 88 Events. Durable readback was one Event and one winner WorkItem. This proves that authority-row serialization does not refresh a waiting TiDB SI snapshot. The next repair hypothesis is repository-owned current-read convergence with complete winner identity validation and bounded bulk conflict isolation; changing the whole enqueue transaction to read committed is not selected. ## Queue-Class Selection `QueueClasses` is the executor's eligibility filter. For fresh WorkItem claims, the application configures an optional soft-affinity tier through `Worker.SetWorkItemQueueClassPreference`, which is forwarded as the request-local `workqueue.WithClaimQueueClassPreference` capability. Claim selection applies pack, generation, lease, and lane-head eligibility first, then considers independently claimable preferred heads before fallback classes. Existing `LaneOrdinal`, `RunAfter`, `CreatedDate`, and `WorkId` ordering remains unchanged within each tier, and preference never bypasses source, target-mutation, or delivery-lane ordering. Preference is work-conserving: a shared executor borrows fallback classes when its preferred tier cannot fill the available slots. Empty preference preserves the historical selection order. Stale-only reclaim ignores fresh-class preference and retains its existing ordering. Permit reservation follows the selected queue shape. The scheduler first picks one compatible executor shape, then reserves only that shape's dispatchable slots before peeking candidates. It must not reserve every idle executor and release mismatched shapes afterwards: those phantom reservations consume the processing budget without producing claims and are amplified when several runtime processes share the database. On the WorkItem claim path, a 16-executor `default-heavy` profile is shaped as 4 latency-only executors (`interactive/default`), 8 affinity-free all-class executors, and 4 bulk-preferred all-class executors. The latency reservation prevents an old bulk backlog from starving interactive work, the bulk preference preserves a bulk liveness floor, and the middle half remains genuinely borrowable by whichever eligible queue is oldest. Lane-head eligibility and the global processing budget remain authoritative. ## Pressure Contract - Ingress path pressure is handled before durable admission work and returns `503` plus `Retry-After` when infrastructure capacity is exhausted. - Drain path pressure is handled before queue mutation. The claim limiter may deny fresh claims, grant a smaller `MaxClaims`, or grant stale-reclaim-only capacity when pressure probes are degraded. - Stale-reclaim-only capacity must not fall back to a general fresh claim if the event service lacks stale-only support. - Worker panic recovery remains executor-local: claimed rows are finalized or released by the existing worker recover paths, and the supervisor restarts the executor slot without creating another scheduler. ## Metrics And Logs Current and expected low-cardinality signals for this contract: - Admission decisions and retry reasons. - Ingress and shared/background DB pool stats. - Claim limiter decisions, wait counts, processing rows, stale-processing rows, DB pool pressure, and queue-class scope. - Worker startup logs showing worker count, max processing rows, and DB pool stats. - Connector inbound stage timing without payloads, secrets, tokens, or raw provider identifiers. ## Rollout Flags - `NEXUSCRM_CONNECTOR_INBOUND_WORKERS_PER_PACK`: executor slot count. Values above the conservative default are compatibility-preserved but logged as unsafe until rollout policy changes. - `NEXUSCRM_CONNECTOR_INBOUND_MAX_PROCESSING_ROWS`: explicit drain processing budget. When set, the budget is pack-global across queue classes. - `NEXUSCRM_CONNECTOR_INBOUND_QUEUE_PROFILE`: queue-class allocation strategy. The default `reserved` profile keeps latency, bulk, and shared worker reservations. On the WorkItem path, `default-heavy` keeps one quarter of executors latency-only, one quarter bulk-preferred, and the middle half as an affinity-free all-class pool. - `NEXUSCRM_CONNECTOR_INBOUND_STAGE_TIMING`: enables safe per-event stage timing. - `NEXUSCRM_CONNECTOR_FLOW_DEADLINE_MS`: bounds wedged post-claim BatchRead and Flow execution calls. ## Non-goals - No adaptive budget allocator is defined here. - No target mutation lane metadata is defined here. - This change does not remove the `SchedulerLease` schema or its published source-compatibility shapes. Schema removal requires a later mixed-version upgrade decision. - No provider, HubSpot, tenant, or payload-specific branch belongs in this L0 scheduler contract.