# Issue 517 Design: Bulk Rollup Parent-VR Projection

## Scope

This PR is design-only for `#517`.

- `D01`: done in this memo.
- `T01`: skipped - reason `out of scope for this PR rollup; runs in separate longrun`.
- `T02`: skipped - reason `out of scope for this PR rollup; runs in separate longrun`.
- `T03`: skipped - reason `out of scope for this PR rollup; runs in separate longrun`.
- `T04`: skipped - reason `out of scope for this PR rollup; runs in separate longrun`.

No production behavior changes are made here.

## Merge Unit

- `primary_fact`: whether a bulk-insert child row is allowed to land when its accepted contribution would make the rollup-induced parent update fail a parent Validation Rule.
- `owner_layer`: `L0` records/rollup runtime.
- `single_resolver`: a future dry-run helper that reuses the records-owned `UpdateDerived -> Update` validation path up to, but not including, physical parent write side effects.
- `primary_surface`: `backend/internal/features/records/persistence_bulk_insert.go`.
- `verification_proof`: docs-only in this PR; future implementation must prove parity with targeted bulk-insert + concurrent-insert tests.
- `implementation_target`: `prerequisite-issue` before execution, because `#582` explicitly leaves rollup storage strategy unresolved and that contract changes what "projected parent image" means.

## Current State

`BulkInsert` already does child-side pass-2 validation in transactional order:

1. `assignAutoNumbersBatch`
2. `computeFormulaFields`
3. `validateRuntimeScopedPicklists`
4. `validateRuntimeScopedLookups`
5. `ValidateRecord`
6. `checkUniqueness`
7. `repo.BulkInsert`

That order lives in `backend/internal/features/records/persistence_bulk_insert.go`.

Rollups are still skipped on the bulk path. The same file documents that skip directly before outbox enqueue. By contrast, single-record create/update/delete call `ps.rollup.ProcessRollups(...)` inside the transaction, and steady-state rollup writes route through `RollupService -> DerivedWriter.UpdateDerived -> PersistenceService.Update`.

The validation engine already supports relationship traversal through the shared formula engine, including multi-hop relation-chain augmentation. `#517` therefore does not need a special one-off depth cap in the projection layer; it needs parity with the existing validator.

## Design Decisions

### 1. Simulation Source Of Truth

Do not reuse any preloaded child snapshot and do not invent a second
in-memory rollup calculator.

The authoritative projection input must be:

- the locked parent row fetched in the same transaction by the records repository lock path;
- a SQL-authoritative projected aggregate over:
  - committed existing child rows for that parent/rollup, and
  - transaction-local candidate batch rows materialized from the already-surviving bulk pass-2 child image;
- the already-surviving batch children from bulk pass-2 (`validRecords` shape, not raw input rows), considered in original batch order for row admission.

Reasoning:

- `FindAffectedRollups` only identifies parent ids; it is not a child-state snapshot.
- Raw inbound rows are not sufficient because bulk pass-2 may already have changed their persisted image through autonumber, formula, normalization, and validator-driven drops.
- Re-querying "existing children + inserted children" after child INSERT would collapse back to the current fail-open/fail-late shape.

Authoritative rollup-value contract for the future execution PR:

- reuse the existing aggregate operator and `RollupConfig.Filter` SQL semantics;
- evaluate projected values in SQL, not Go, by building a derived rowset that unions:
  - committed child rows from the real child table, and
  - candidate batch rows expressed as bound SQL literals / a derived table / a CTE in the same transaction;
- run the same aggregate (`COUNT`, `SUM`, `MIN`, `MAX`, `AVG`) and the same filter text against that derived rowset.

This keeps filtered rollups and non-additive aggregates under one authority. A
plain "accepted subtotal" optimization is only allowed later for the strictly
equivalent subset (`Filter == nil` plus additive operators) after parity tests
prove it matches the SQL-authoritative path byte-for-byte.

### 2. Computed-Field / Half-Projected Gap

Do not add a generic in-memory simulator for autonumbers, formulas, system fields, or defaults.

Instead, future implementation should reuse existing authoritative phases:

- child-side projection input comes from the post-pass-2 `validRecords`, which already include child autonumber/formula/system-field materialization as seen by SQL;
- parent-side projection must call a new dry-run helper extracted from the
  existing `UpdateDerived` path, so parent formula recompute and Validation
  Rule evaluation use the same logic as the eventual real parent update.

That helper should stop before:

- audit writes,
- outbox enqueue,
- `repo.Update`,
- post-update rollup fanout.

It should still run the same parent-side merge, formula recompute,
runtime-scoped validators, Validation Rules, and uniqueness check that the
real derived-update contract will hit.

Explicit prerequisite decision:

- the dry-run helper itself must not dispatch audit, outbox, physical parent
  writes, or any external side effect;
- but the project still needs a separate authority decision for
  rollup-derived `RecordBeforeUpdate` / before-save Flow semantics, because
  today's `UpdateDerived -> Update` path still runs the normal update
  lifecycle;
- `#225` design discussion is evidence that "VR runs, Flow skips" was a
  debated target contract, but `#582` did not land that behavior, so this PR
  must not pretend the decision is already frozen;
- future execution must therefore choose one of these prerequisite options
  before code lands:
  - Option A: keep current `UpdateDerived` semantics and make the dry-run
    helper execute the same before-save transform without persisting side
    effects;
  - Option B: add a rollup-specific derived-write mode that skips before-save
    flows and treat that as an explicit product contract change with its own
    tests and rollout notes;
  - Option C: block `#517` execution until the project pins one of A or B as
    authority.

If the later real parent `UpdateDerived` still fails after a dry-run pass, treat that as an invariant breach and fail closed. Do not add a second "sanity re-fire" projection authority.

### 3. Multi-Hop Validation Rule Formula Scope

Do not introduce a new depth-1-only rule for this slice.

Future projection must inherit whatever the current validation engine already supports:

- single-hop roots through the validation formula relation resolver;
- multi-hop traversal through the shared formula engine fetcher / relation-chain augmentation.

`#517` is not the place to generalize rollup math beyond current rollup declarations, but it also must not silently evaluate a narrower Validation Rule image than `UpdateDerived` uses today. If an existing rule would fail closed on normal parent update because a relation root is unknown, the projected path must fail the same way.

### 4. Bulk-Insert Reparent Edge

Out of scope here. Pure `INSERT` has no authoritative old-parent image.

Known limitation for the eventual execution PR:

- no attempt to solve update/upsert reparent orchestration,
- no attempt to dedupe old-parent/new-parent fanout,
- no attempt to reuse this design for update/delete surfaces.

Those belong to a separate longrun slice. Status: skipped - reason `out of scope for this PR rollup; runs in separate longrun`.

### 5. Concurrency Semantics

Optimistic transaction + ad hoc `FOR UPDATE` is not enough for this path.

Future implementation should switch bulk-insert projection to deterministic pessimistic locking whenever the batch reaches projected parent-VR evaluation:

1. identify affected parent ids from surviving child rows;
2. sort parent ids deterministically;
3. execute inside `WithPessimisticTransaction` (or an equivalent bulk-safe wrapper);
4. lock each parent in sorted order before reading its base aggregate;
5. project accepted child rows in original input order per parent.

This gives a deterministic answer inside one serial lock order. Across
concurrent transactions, outcome resolves by lock acquisition order
(`first-lock-holder wins`), while sorted parent locking prevents deadlocks and
split-brain pass/fail outcomes. Within the winning transaction, the first
candidate in that batch order whose acceptance would make the projected parent
validation fail is the row that gets rejected.

## Recommended Future Algorithm

1. Keep the existing child-side bulk pass-2 unchanged through `validRecords`.
2. Group surviving rows by `(parent object, parent id, rollup field)` while preserving original bulk row order.
3. For each affected parent, lock the parent row and build a SQL projection source made of:
   - committed child rows from storage, and
   - zero or more accepted candidate batch rows.
4. For each candidate child row in original bulk order:
   - tentatively add that row to the projection source;
   - compute the projected rollup value through the SQL-authoritative aggregate path;
   - build the projected parent updates map for the rollup field;
   - call the new dry-run derived-update validator;
   - if it passes, keep the child row accepted and keep it in the projection source;
   - if it fails, reject only that child row, remove it from the tentative projection source, and record a per-row Validation Rule error.
6. After all parents are processed, mutate the survivor set before `repo.BulkInsert`, using the same `validRecords` / index-map partial-success mechanism the bulk path already uses.
7. After survivor INSERT succeeds, perform the chosen per-parent rollup
   materialization step from the prerequisite authority decision:
   - if the project keeps stored rollup writes, run the chosen derived-update
     lifecycle contract from Option A or Option B above;
   - if the project adopts a different storage/materialization strategy, use
     that strategy's authoritative parent-finalization step instead.

Algorithm parameterization:

- step 7 is intentionally not fixed in this design PR;
- it is parameterized by two prerequisite freezes:
  - rollup storage/materialization strategy, and
  - rollup-derived before-save automation semantics;
- execution must not assume "Flow skipped" or "stored UpdateDerived write" until
  those two authority decisions are frozen.

This preserves the documented partial-success contract without inventing a second parent-write authority.

## Why This Must Not Implement Yet

`#582` leaves rollup storage strategy unresolved and explicitly says the remaining rollup follow-ups should not be implemented standalone until that contract is fixed. `#517` depends on that answer because projected-parent correctness changes depending on whether the rollup field is:

- a physical stored column,
- a virtual field with read-time view materialization, or
- a filtered logical write that never persists directly.

The design here is implementation-ready only after the storage-strategy authority exists.

## Proposed Split

### Prerequisite

Freeze the rollup storage strategy as an authority issue under the `#225` / `#582` chain.

### Execution PR After Prerequisite

Implement exactly one runtime slice:

- add the dry-run derived-update validator helper,
- after the prerequisite decision above is frozen, implement either:
  - a dry-run helper that reproduces the current before-save transform without
    side effects, or
  - a rollup-specific derived-write mode that explicitly skips before-save
    automation,
- add bulk parent-VR projection in `BulkInsert`,
- add targeted bulk-insert parity tests,
- add concurrent same-parent pessimistic integration coverage.

### Explicit Non-Goals

- generic `_System_Job` queue work: skipped - reason `out of scope for this PR rollup; runs in separate longrun`
- cross-object formula write-time reactivity: skipped - reason `out of scope for this PR rollup; runs in separate longrun`
- sharing criteria reactivity: skipped - reason `out of scope for this PR rollup; runs in separate longrun`
- outbox metadata rehydration: skipped - reason `out of scope for this PR rollup; runs in separate longrun`

## Future Test Plan

- `backend/internal/features/records/persistence_bulk_insert_test.go`
  - `TestBulkInsert_RollupParentVRProjection_DropsOnlyContributingRows`
  - input: two child rows for one parent, second row breaches parent VR after rollup
  - expected: row 1 commits, row 2 rejected, parent rollup reflects row 1 only
- `backend/internal/features/records/persistence_bulk_insert_test.go`
  - `TestBulkInsert_RollupParentVRProjection_UsesPostPass2ChildImage`
  - input: child autonumber/formula contributes to rollup predicate
  - expected: projection matches actual stored child image, not raw inbound payload
- `backend/internal/features/records/persistence_bulk_insert_integration_test.go`
  - `TestBulkInsert_RollupParentVRProjection_ConcurrentSameParentSerializes`
  - input: two concurrent batches hitting one parent
  - expected: deterministic parent lock ordering and no lost-update / split-verdict drift