# Durable source projection input ranges

This supersedes PR #329's historical coverage inference. The outcome is that
every newly admitted settled source input belongs to exactly one durable
WorkItem generation, and the worker consumes that generation's complete input
set. Receipt duplicates neither change membership nor create executable work.

## Ownership and invariant

Events remain immutable receipt/payload facts. Workqueue owns execution and
membership: `InputStartOrdinal` is an inclusive, immutable lower bound;
`LaneOrdinal` is the inclusive upper bound and selected Event's enqueue ordinal.
The projection identity is installed-pack incarnation, canonical Event kind,
route version, companion pack/Flow version, work type and source lane. One
Workqueue helper encodes this identity. There is no second per-Event ledger.

The admitted pack transaction serializes fresh ingress and explicit replay,
allocates positive non-reused ordinals, and commits Event plus WorkItem together.
A fresh batch retains its minimum ordinal for each identity. Only an unclaimed
attempt-zero generation can extend its upper bound. Claimed/retried work keeps
both bounds; later inputs extend a queued successor. A replay starts with its
new replay Event and can absorb later fresh inputs before it is claimed.

The worker adapter passes the WorkId remembered at claim. Payload loading uses
that persisted interval and exact immutable identity, ordered by ordinal. It
does not infer membership from timestamps, selected Event status, delivery
sequence or retained terminal history. Missing membership is an error; there is
no selected-payload-only fallback.

The source-lane Event index and existing projection-order WorkItem index cover
the bounded queries. The ordinal allocator reads the protocol once per batch; duplicate verification
uses a bounded protocol PK read before probing task coverage.
Admission performs work per affected source. Payload reads scale with the new
generation's inputs, not the number of retained generations. Event/WorkItem
retention must not delete live range members or reuse ordinals; no current
production path performs either operation.

## Release boundary

This is one coordinated change across three release repositories:

| Owner | Change |
|---|---|
| Core schema catalog | Nullable WorkItem `InputStartOrdinal`; immutable per-pack `_System_ConnectorProjectionProtocol` |
| Controlplane | Empty-incarnation initialization, drained-history checkpoint and apply/recovery gates |
| Connectors | Fresh admission, duplicate verification, replay, claim validation and worker payload reads |
| Core Integration Service | Activation gate before workers become available |

Controlplane owns protocol version 1 and `LegacyMaxOrdinal`: the last input
admitted under the old protocol. It writes this cutoff exactly once after old
projection work has drained. It never advances an existing cutoff. Runtime
acknowledges receipts at/below it without interpreting historical ranges. This
is an old-protocol boundary, **not** proof of historical business success.
Post-cutoff duplicates must have durable coverage even when their Event is
skipped or terminal; claim changes the selected Event's status before execution
has finished.

### Recovery contract

Event and WorkItem creation share one transaction. A crash after that commit but
before ingress-receipt confirmation leaves both facts intact, so retry remains
idempotent. Ordinary one-Event/one-WorkItem repair is unchanged.

Source-projection duplicates no longer reconstruct missing tasks as PR #329
attempted to do. A retained input does not reliably prove whether a deleted
historical generation already executed. Native missing coverage is an explicit
error, not a successful acknowledgement or an automatic replay. Old missing or
ambiguous responsibility blocks the cutover audit and requires a separately
reviewed recovery decision. The cutoff must never be used to hide that refusal.

## Existing-data cutover

This upgrade adds schema and one checkpoint per pack incarnation. It performs
**no historical Event or WorkItem UPDATE** and requires no range backfill.
Existing completed/failed history stays intact. Old nonterminal projection work
must finish using the old runtime; it cannot be handed to the new range reader.

The L2 delivery owner executes the following ordered transition:

1. Apply the additive canonical schema. Arm a non-expiring namespace-wide `*`
   admission fence with no active overrides. Keep old workers and dependency
   resolvers running so they can claim, retry and finish their durable backlog.
2. Wait for all old projection work to become terminal, including pending,
   blocked, retry and processing states. Waiting only for active leases is not
   sufficient. `RunUntilDrained` stops new claims and waits for in-flight calls;
   do not invoke it as a substitute for draining the durable queue. Retry and
   external dependency waits may prevent a bounded drain; stop and report that
   condition rather than changing task status to proceed.
3. Stop every old writer, receipt materializer and worker, retaining the fence.
   The database cannot prove that an old binary will never resume.
4. Run `connector-projection-cutover -pack-install-id <incarnation>` with
   `CONNECTOR_CUTOVER_DSN` and a bounded `-timeout`, or the same Controlplane
   lifecycle entrypoint. Under namespace/pack locks it audits retained owners,
   refuses live or unresolved old projection responsibility, and writes only
   the cutoff. Concurrent/repeated calls preserve the same checkpoint.
5. Start the matching new host and Connectors, verify readiness and resume
   admission through L2. Durable receipts not yet materialized may be handled
   by the new materializer. Do not restart an old writer after this point.

Native projection activation requires the complete WorkItem schema, including
`ReplayOfWorkId`, for both process-wide and standalone workers. It does not
inherit the ordinary non-projection WorkItem path's optional-column rolling
compatibility. Library consumers must run the activation gate before serving.

The prior WorkItem-only v2 handoff must already exist. This upgrade does not
retire populated Event backlog to manufacture that prerequisite. Greenfield
pack activation writes cutoff zero in its own transaction.

The one-time audit permits ordinary completed coalesced inputs to remain in
historical Event status. A pending Event without a direct task is not by itself
an orphan. A missing owner, input beyond the retained completed boundary, or
indirect input in mixed zero/positive history is refused. This audits retained
responsibility, not historical exactly-once business effects. Do not mark input
processed/skipped, delete it, raise the cutoff, or replay a source to make the
gate pass. Historical recovery and target-environment drain duration require
separate operational decisions.

Keep the new schema/checkpoint on rollback; a binary rollback to old writers is
not supported after new inputs have been admitted. Use the L2 delivery runbook
for process fencing and recovery; L0 does not orchestrate tenant deployments.

## Verification map

- `source_projection_ranges_integration_test.go`: both PR #329 regressions,
  clock/order disagreement, active/successor membership, mixed duplicates,
  enqueue/claim races, retries, replay, interleaved command exclusion.
- `projection_range_integration_test.go` (inbound): real TiDB ingress through
  claim, WorkId adapter, complete Flow payload set and durable completion.
- Existing lifecycle/snapshot/lock tests: duplicate immutability, ownership
  transfer, concurrent completion, stale snapshots and replay conflict fences.
- Controlplane `connector_projection_range_protocol*_test.go`: independent
  drain/ownership counterexamples, unchanged historical rows, fences and idempotency.
- Core schema and Integration Service composition: canonical schema and
  startup refusal before the protocol receipt.
- History scaling and 400-source walking workload: retained index plans and
  correctness counts; measurements recorded in the performance learned log.

Canonical-table integration tests require `NEXUSCRM_REPO_ROOT` to identify the
candidate Core checkout and `NEXUSCRM_TIDB_REQUIRED=1` with the declared TiDB
connection. A skipped database test is not acceptance evidence. Cross-repo
local verification uses an external temporary Go workspace, never checked-in
local replaces or invented published versions.


The canonical CI owner is Core's existing `source-backed-runtime-gate` TiDB job. It runs
`bash tests/integration/connector-projection-ranges.sh`, the same runner used
locally, against Backend's real module pins. This replaces the two separate
Connectors trusted ingress/recovery workflows that lacked access to Core's
schema; it does not introduce another schema snapshot or credential surface.