# Connector Root Architecture Contract Date: 2026-07-02 Current-tree revision: 2026-07-26, bound to NexusCRM base `cc92b04b2f8fe1f108009ba38908699a26bf8530`. Both trusted consumer modules pin connectors `v0.2.2`. This revision supersedes the original v0.1 consumer/re-pin and live legacy-Event-selection assumptions; source-current WorkItem-only authority is still separate from deployment, drain, and compatibility-retirement proof. Status: design authority for [#3379](https://github.com/pingcap-inc/NexusCRM/issues/3379), linked root repair for [#3337](https://github.com/pingcap-inc/NexusCRM/issues/3337). The later [#3725 Integration work/effect ADR](issue-3725-integration-work-effect-ownership.md) owns the target distributed scheduler authority and the cross-family Integration work/effect contract. This document is a contract/design PR only. It does not add runtime code, schema, migrations, downstream pack changes, or connector module changes. ## Scope The July 2026 connector slowdown class is not fixed by lowering worker count, adding one more index, isolating the ingress DB pool, hardening a metrics sampler, or wrapping the current event claim path in a process-local scheduler. Those are foundations and mitigations. The root architecture must stop using one connector event row family as all of these at once: - provider admission receipt; - immutable materialized event/fact ledger; - executable queue; - retry and dead-letter state; - worker lease table; - source ordering substrate; - target mutation serialization substrate; - live queue pressure source. The clean boundary is: 1. `IngressReceipt` records that a provider delivery was verified and durably accepted. 2. `ConnectorOutboxEvent` records the immutable materialized event fact derived from the receipt. 3. `ConnectorWorkItem` owns executable queue state. 4. `ConnectorWorkItem` row CAS owns distributed execution authority; the process-local scheduler owns fairness and admission. 5. The connector module `main` supervisor runs mandatory WorkItem execution without `ConnectorSchedulerLease`. NexusCRM's backend and distributions pin connectors `v0.2.2`; the live supervisor calls `NewWorkItemWorker` with a complete service and activation context, and fails closed unless the current controlplane cutover receipt, claim schema, Event owner field, historical Event-to-WorkItem coverage, and mandatory zero-old-claimer/drained-lease gate are verified. It does not select historical Event execution or legacy custom-service admission. Published custom/pinned `NewWorker` callers may retain `ConnectorSchedulerLease` when they do not opt into distributed row CAS. That compatibility surface remains until `test1` mixed-version/drain evidence proves those callers and older binaries exited. 6. Source lane and target mutation lane are separate runtime facts. 7. `ConnectorLaneState` and `ConnectorQueueCounter` are optional phase-2 materialized read models, not mandatory first-cut state. This is L0 substrate. It must stay connector-provider-neutral and must not branch on HubSpot, CRM object names, tenant fields, profile names, or pack payload details. ## Current Foundations The following merged work is foundation, not completion of #3379/#3337: - connector guardrails, scheduler interface, adaptive budget, central per-process claim scheduling, target lane metadata, and cheap stats in `pingcap-inc/nexuscrm-connectors`; - NexusCRM ingress DB pool isolation and live-stats hardening; - L1 app-pack metadata for target mutation lanes; - L2 worker default gates. The source has advanced beyond the original Event-queue baseline. `_System_ConnectorEvent` retains immutable delivery facts plus bounded compatibility fields/projections for older generations, but the pinned live supervisor's executable state and `ClaimNext` authority are `_System_ConnectorWorkItem`. Its process-local scheduler reduces fan-out inside one backend process, while distributed WorkItem row CAS bypasses `_System_ConnectorSchedulerLease`. NexusCRM's backend and distributions pin connectors `v0.2.2`. The live supervisor directly constructs `NewWorkItemWorker` with one complete `WorkItemService` and activation context. It starts only after current controlplane receipt, claim-schema, Event `ExecutionOwner`, historical mirror coverage, zero active Event leases, and zero old claim-capable processes all verify; a confirmed gap fails closed. This WorkItem-only path does not execute historical Events and does not admit the legacy custom-service contract. Event owner/mirror state remains guarded transition and compatibility evidence, not a second live claimant. A published custom or pinned `NewWorker` caller that does not opt into distributed row CAS may retain the SchedulerLease compatibility surface, but it must be stopped and drained for a generation before that generation enters the WorkItem-only cutover receipt. R1 deletes that surface only after `test1` mixed-version/drain proof shows custom, pinned, and older-binary callers have exited, so multiple replicas contribute capacity only through fenced WorkItem row claims. ## Ownership Model ### IngressReceipt `IngressReceipt` is the durable provider-admission receipt. It owns: - provider namespace and delivery identity; - verified signature/admission result; - receipt idempotency; - the verified request envelope needed to resume routing after process restart: HTTP method, connector namespace, route/path key, content type, provider delivery id, event kind when available, received timestamp, safe correlation ids, and the body reference or bounded body snapshot; - durable acceptance timestamp; - payload hash or payload reference; - receipt-router status, attempt count, next attempt time, claim owner/token, lease expiry, enqueued event ids, and sanitized receipt-routing error fields until materialization completes. It does not own: - post-materialization executor claim state; - WorkItem retry/dead state; - source or target lane head; - queue pressure; - Flow execution result. The webhook response contract returns the provider-compatible success status only after durable receipt. Before receipt, the L1 route response contract defines the failure response and redelivery expectation. Generic L0 does not assume that `429` or `5xx` produces provider redelivery; sources without status-based retry require alerting and reconciliation. There is no ack-before-durable-boundary mode. ### Receipt Routing State `IngressReceipt` remains the durable pre-materialization inbox and retry owner. Acknowledged provider deliveries must survive backend restart and continue draining from the receipt row until they materialize into immutable events and WorkItems, or reach a terminal receipt-routing failure. Receipt-router claim/retry state is distinct from executor claim/retry state: - receipt-router state is owned by `IngressReceipt` before materialization; - executor state is owned by `ConnectorWorkItem` after materialization and generation cutover; - no follow-up PR may create a second mutable receipt-router queue beside the receipt row. After materialization succeeds, the receipt records the materialized event ids for audit/readback. It does not continue to own executor retry, DLQ, lane, or lease state for those events. ### ConnectorOutboxEvent `ConnectorOutboxEvent` is an immutable materialized fact ledger. It owns: - event id; - receipt id; - installed pack id and generation/incarnation snapshot; - route id and route version snapshot; - source object kind and id; - idempotency and replay identity; - delivery sequence and enqueue ordering; - payload hash or payload reference; - parser/materialization outcome; - companion snapshot or fail-closed companion decision; - safe correlation ids. It does not own: - `pending`, `processing`, `retry`, or `dead` executor state; - retry attempts or next attempt time; - worker lease owner/token/until; - scheduler generation; - target mutation lane head; - queue class pressure. Old event rows may remain readable for audit, replay, and migration, but the hot executor claim path must move away from broad event-table claim/order scans. ### ConnectorWorkItem `ConnectorWorkItem` is the executable queue source of truth. It owns: - work id; - event id; - installed pack id; - generation; - work type; - queue class; - state; - run-after; - priority if needed by the scheduler; - attempt count; - terminal outcome, failure class, sanitized failure summary, and retry/dead operator state for WorkItem-claimed executions; - rollback execution fence id plus legacy-outcome receipt version and digest when ownership is transferred back to the Event compatibility path; - lease owner; - fencing or lease token; - lease-until; - source lane key; - target mutation lane key; - timestamps. Required first-cut constraints and indexes: - unique `(EventId, WorkType)` to prevent duplicate work after receipt or materializer retry; - `(InstalledPackId, QueueClass, State, RunAfter, WorkId)`; - `(InstalledPackId, SourceLaneKey, State, RunAfter, WorkId)`; - `(InstalledPackId, TargetMutationLaneKey, State, RunAfter, WorkId)`. The target scheduler claims by primary-key CAS on `ConnectorWorkItem`. Executor count controls how many already-claimed items can run; it must not create more queue scanners. Every retry, renewal, completion, and finalization is fenced by the WorkItem lease token. ### Retained Custom/Pinned ConnectorSchedulerLease Compatibility The connector module `main` supervisor has no pack/generation-wide `ConnectorSchedulerLease` predicate: `NewWorkItemWorker` uses distributed WorkItem row CAS and skips lease hooks. NexusCRM's current `v0.2.2` pin uses that WorkItem-only live supervisor after current-receipt, schema, and historical coverage plus zero-old-claimer verification; it has no legacy Event supervisor selection. Separately, published custom and pinned `NewWorker` callers that do not opt into distributed row CAS retain the SchedulerLease compatibility contract during drain, but their generation cannot become WorkItem-claimable until they are stopped and all Event leases/transactions reach zero. R1 deletes this compatibility surface only after: - concurrent claims have one WorkItem winner; - stale WorkItem holders cannot renew or finalize; - source and target lane policies remain correct; - process kill, drain, lease expiry, and reclaim remain live; - queue pressure does not depend on one pack-wide owner; - the trusted consumers remain pinned to `v0.2.2`, and `test1` proves current receipt/schema/mirror readiness, zero old claimers/leases/transactions, and per-pack activation; and - `test1` mixed-version validation and drain evidence show that the retained callers and older binaries have exited the lease path. - rollback fault proof shows zero WorkItem processing rows, `Started` attempts, leases, claim/reclaim transactions, and claim-capable processes before an Event owner receipt can be issued; every remaining unstarted WorkItem is durably fenced as `AwaitingLegacyOutcome`; and re-cutover is blocked until every such row has a confirmed terminal legacy-outcome receipt. After that gate, remove the scheduler-lease predicate, acquire/renew/release APIs, schema row, diagnostics, and lease-holder compatibility proof. Process-local schedulers continue to own fairness and admission. The WorkItem row remains the sole distributed execution authority. ### Source And Target Lanes Source lane and target mutation lane are separate runtime facts. Source lane owns source ordering and replay/idempotency identity, for example a provider object family and source id. Target mutation lane owns serialization of a hot mutation target, for example many source events that write the same remote or local target resource. Target mutation lane is additive. It must not replace source identity, idempotency, delivery FIFO, or route/version safety. Effective claim blocking is the declared scheduler policy over both lane facts. The current compatibility path that normalizes target lane metadata onto `SourceLane*` is not the final runtime model. ### Failure, DLQ, And Replay Read Path Terminal failure and operator replay follow the active claim path for the pack/generation: - legacy event-claim generations keep `_System_ConnectorEvent` as the failure and DLQ read path; - WorkItem-claim generations use `ConnectorWorkItem` as the failure and DLQ read path, joined to `ConnectorOutboxEvent` and `IngressReceipt` for immutable event and receipt context. There is no mixed terminal-failure authority. When a generation is cut over to WorkItem claims, retry/dead state, sanitized error details, and operator replay eligibility move with the WorkItem read path. `_System_ConnectorEvent` remains immutable audit/provenance for that generation and cannot keep a parallel mutable DLQ state. The existing operator surface may preserve its route names and permissions (`ViewConnectorEvents` / `ReplayConnectorEvents`), but its backing read model must switch by generation to the single active failure owner. For WorkItem generations, listing DLQ rows reads terminal WorkItems and projects event/receipt context for compatibility. Replay preserves the current operator intent: replay is an explicit new attempt against the current active pack/route when the route resolver and generation guards allow it. The immutable event snapshot is audit/provenance and a source for replay identity checks; it is not silently mutated into a new historical route decision. If the current route, pack generation, companion snapshot, or idempotency contract cannot be proven safe, replay fails closed. ## One-Active-Claim-Path Rule For any `(InstalledPackId, generation, EventId, WorkType)`, exactly one executor claim path may own execution: - legacy `_System_ConnectorEvent` claim path; or - new `ConnectorWorkItem` claim path. Both paths must never claim the same logical work concurrently. The source-current `v0.2.2` supervisor admits only the WorkItem path. Older binaries/custom compatibility must stop new Event claims and drain every already-owned Event lease to zero before any WorkItem claim for that generation is enabled. Migration or rollback fails closed if a pack/generation has ambiguous claim ownership. Cutover gates: 1. Quiesce Event claim admission for the pack/generation, revoke its rollout selection, and prevent every older/custom binary capable of claiming it from starting or restarting. 2. Drain all already-owned Event leases and write authoritative proof of zero active Event claim tokens, zero Event-owned processing rows, zero claim-capable old processes, and zero active Event claim transactions. WorkItem claiming remains disabled throughout this drain. 3. Only then write the durable, generation-scoped controlplane cutover receipt binding zero-old-claimer proof, complete WorkItem materialization, claim schema, Event ownership field, and historical mirror coverage. Every backend observes that exact receipt before any WorkItem claimant starts. 4. Rollback first disables all WorkItem claim/reclaim admission and prevents every WorkItem-capable process from starting or restarting. It then proves zero active/expired WorkItem leases, zero WorkItem-owned processing rows, zero `Started` effect/execution attempts, zero claim/reclaim transactions, and zero running claim-capable processes. In the same rollback-fence transaction, every remaining eligible `Pending` or pre-dispatch `Retry` WorkItem becomes non-executable `AwaitingLegacyOutcome` and records the `RollbackExecutionFenceId`; a WorkItem with prior dispatch, ambiguous external outcome, or unverifiable attempt history blocks rollback. Only a new durable owner-generation receipt binding that complete zero proof and the complete fenced-WorkItem set may admit a previously approved rollback-bridge-compatible legacy binary. The minimum compatible binary must atomically write a versioned `LegacyOutcomeReceipt` when its Event reaches a terminal outcome. A receipt projector/reconciler moves the matching WorkItem to `ReconciledLegacyTerminal` and stores the receipt version/digest; a crash after the Event/receipt commit but before projection is repaired from that receipt. An unknown or quarantined legacy outcome remains non-executable but blocks re-cutover. 5. Re-cutover to WorkItem ownership is prohibited until there are zero `AwaitingLegacyOutcome` or unknown/quarantined rollback-fenced rows and every `RollbackExecutionFenceId` has a confirmed terminal receipt. A `ReconciledLegacyTerminal` WorkItem is immutable and is never reopened, reset, or reused: its unique `(EventId, WorkType)` row proves that the logical work already executed under legacy ownership. Materialization may create executable WorkItems only for eligible non-terminal Events with no WorkItem row. Any missing, conflicting, or ambiguous receipt fails closed. `v0.2.2` never switches itself to Event execution and the gates never permit both paths. 6. Audit/replay reads may span receipt, event, and work item records, but they cannot become a second executor claim path. 7. Failure/DLQ reads follow the active claim path for the pack/generation and cannot keep mutable terminal state on both event and WorkItem rows. ## Migration And Rollback Requirements The implementation sequence must preserve: - receipt idempotency; - event replay identity; - route version guard; - active pack incarnation/generation guard; - companion snapshot fail-closed behavior; - delivery FIFO; - source lane ordering; - target mutation lane serialization; - stale reclaim semantics; - token or fencing-gated finalizers. Backfill or lazy materialization for existing non-terminal event backlog must be idempotent. The unique `(EventId, WorkType)` constraint is mandatory before any retrying materializer can create WorkItems. A pack/generation must not become WorkItem-claimable until the backlog materialization state is known and the legacy event claim path is fenced for that generation. Rollback must be documented and tested before removing legacy fields or flags. Rollback cannot require manual SQL rescue. The rollback test matrix must cover pending WorkItem -> rollback fence -> legacy success and terminal failure -> reconciliation receipt -> re-cutover; duplicate legacy terminal delivery; a crash after the Event/receipt commit but before the WorkItem projection; an ambiguous legacy outcome that blocks re-cutover; proof that a reconciled terminal row is never reopened; and materialization of a new eligible Event that has no WorkItem row. ## Optional Materialized Read Models `ConnectorLaneState` and `ConnectorQueueCounter` are not mandatory PR2 schema. They may be introduced only after the WorkItem claim path proves that indexed WorkItem reads are still too expensive, or that the scheduler/admission budget needs cheaper due-head or counter reads than WorkItem indexes can provide. If introduced: - `ConnectorLaneState` is rebuildable from WorkItems and cannot become an unverified second source of truth. - `ConnectorQueueCounter` is transactionally maintained on WorkItem transitions and periodically reconciled. - exact full queue stats remain scoped diagnostics, not a hot-loop scheduler or admission dependency. ## Schema Source Of Truth The system-table metamodel and bootstrap/apply read path remain in the existing NexusCRM kernel/controlplane schema authority. Follow-up PRs must not define a second connector-specific table schema source in `nexuscrm-connectors`. `nexuscrm-connectors` owns connector runtime behavior, DAOs, scheduling, and state transitions after the tables exist. NexusCRM owns the single schema definition and setup/apply path for `_System_*` tables until the repository split explicitly moves that authority through a separate design decision. ## Rejected Workarounds This architecture rejects: - lowering `NEXUSCRM_CONNECTOR_INBOUND_WORKERS_PER_PACK` as the root fix; - adding indexes while event rows remain the executor queue and lease table; - relying on ingress DB pool isolation alone; - relying on process-local scheduling without fenced WorkItem CAS; - treating stats sampler hardening as the root fix; - permanently normalizing target mutation lane into source lane; - adding HubSpot-specific L0 branches; - adding payloads, credentials, tokens, or secrets to logs, metrics, issues, or proof artifacts. ## PR Spine PR1 - this contract/design authority PR in `pingcap-inc/NexusCRM`. - Publish the clean root boundary. - Define the one-active-claim-path rule. - Define migration, rollback, and proof gates. - Reference #3379 and #3337. - No runtime, schema, L1, or L2 changes. PR2 - additive schema metamodel and bootstrap/apply path in `pingcap-inc/NexusCRM` (landed foundation). - Add `ConnectorWorkItem`. - Add the transitional `ConnectorSchedulerLease`. - Keep the definitions on the existing kernel/controlplane system-table schema SSoT and setup/apply read path. - Add first-cut constraints and indexes listed above. - Do not add `ConnectorLaneState` or `ConnectorQueueCounter` unless PR1 proof gates are updated to make them mandatory. PR3 - connector runtime DAO/materialization in `pingcap-inc/nexuscrm-connectors`, then consume from `pingcap-inc/NexusCRM` as needed. - Create WorkItems transactionally with immutable event facts. - Prove duplicate receipt/materializer retry does not create duplicate WorkItems. - Backfill or lazily materialize existing non-terminal backlog. - Preserve the NexusCRM-owned schema metamodel as the table-definition SSoT. PR4 - WorkItem claim-path convergence and scheduler-lease retirement. - WorkItem primary-key CAS owns distributed execution. - Process-local schedulers own polling, fairness, admission, and drain. - The connector module `main` supervisor path has no pack-wide scheduler-lease predicate or lifecycle. Trusted consumers already pin the WorkItem-only `v0.2.2`; current-receipt, schema, historical-coverage, and zero-old-claimer `test1` evidence govern per-pack activation. - Remove the custom/pinned compatibility predicate, lifecycle, and schema after multi-process proof plus `test1` mixed-version and drain evidence. - Stop old claim admission and drain every older/custom Event lease and claim transaction to zero before recording the receipt or enabling any WorkItem claim; `v0.2.2` itself never falls back to Event execution. PR5 - runtime source/target lane split. - Preserve source identity and delivery ordering. - Serialize target mutation resources through target lane. - Define old in-flight lane handling. PR6 - optional materialized read models. - Add `ConnectorLaneState` and/or `ConnectorQueueCounter` only if WorkItem proof gates show they are needed. PR7 - cutover and deprecation. - Remove or quarantine legacy claim, lease, and retry fields from the hot executor path. - Keep legacy diagnostics read-only only when needed. - Remove compatibility fallback flags only after the pinned `v0.2.2` activation proof and `test1` mixed-version/custom-caller drain proof. Release/deploy proof may require separate consumer PRs, but those PRs consume this L0 contract rather than redefining it. ## Proof Required Before #3337 Closure Do not call #3337 fixed from a green design PR or from merged foundation PRs. Closure requires implementation and deployment proof, including: - deployed NexusCRM head SHA, connector module version, app-packs ref, downstream release tuple, and runtime health; - two-backend-process proof with `workerCount=16`; - multiple backend processes claiming the same generation through WorkItem CAS; - process kill, GC pause, WorkItem lease expiry, reclaim race, and clock-skew tests with no double claim and no orphaned processing work beyond policy; - no `ConnectorSchedulerLease` predicate or lifecycle in the connector module `main` supervisor WorkItem claim path, trusted consumers pinned to `v0.2.2`, and `test1` evidence for current receipt/schema/mirror readiness, zero old claimers/leases/transactions, and per-pack activation; - `test1` mixed-version and drain evidence before deleting the retained custom/pinned compatibility surface; - claim path `EXPLAIN` showing indexed WorkItem query and primary-key CAS, not broad `_System_ConnectorEvent` correlated `NOT EXISTS` or JSON fallback in the hot executor loop; - separated ingress and shared/background DB pool metrics; - WorkItem queue depth, active claimant count, claim QPS/latency, executor concurrency, lease-reclaim rate, retry/dead rates; - same-target and multi-target mutation burst proof; - SQL digests proving old live-stats or claim-guard amplification is absent from normal operation; - rollback proof with the feature flag disabled, including durable `AwaitingLegacyOutcome` fencing, terminal `LegacyOutcomeReceipt` reconciliation, zero unresolved rollback fences before re-cutover, and proof that `ReconciledLegacyTerminal` rows cannot be reopened or duplicated. ## Authority This document is the repo-local design authority for #3379 PR1. #3379 remains the GitHub tracking epic for the multi-PR repair, and #3337 remains the linked root pressure-isolation authority. If a later review changes scope, acceptance criteria, sequencing, or durable design decisions, update this document and the PR/issue body in the same round.