# Immutable Worker binding performance

## 2026-09-02 — preparation must follow immutable identity, not work count

- Baseline Core: `5b326134fe0880d1626e4965dd909d3e0c68fd35`; the sibling
  binder is unchanged from serving Core `eba5abc1ef3ff4c7f0c1f9476c72ccb0660f5554`.
- Incident evidence: run `33573192948`, 900-second archived observation.
  Allocation profile: 91.27 GB of 99.11 GB **cumulative allocation** under
  `OpenCommand -> openSiblingProfileFromSource`, not resident memory or CPU share.
- Reproduction: immutable toy Catalog, real DirectoryStore, same command
  binding. Twenty warm calls add 40 projections and 40 decodes despite 60
  verified-view cache hits and zero physical reads. A wider 200-field toy
  schema raises allocation from 12.8 MB to 100.3 MB over those calls.
- Model: the snapshot plus installation/capability uniquely owns the sibling
  identity. The current binder computes its output digest by compiling before
  looking in its digest-keyed cache. A local unchanged-envelope counterfactual
  retaining the derived identities eliminated this repeated work while still
  calling production binding/profile/compatibility checks.
- Ownership: the existing L0 binder owns reuse of verified immutable views;
  Catalog still owns projection publication. See `docs/design/l0-l1-standard.md`
  for the engine/content boundary. No L1/L2 or public command-contract change.
- Consumer map: scheduled Flow and Flow-command binding share
  `openSiblingProfileFromSource`; explicit record-event bindings already use
  `openProfileForCapability`. Original command/Flow authorization remains in
  their existing callers.
- Planned change: resolve the immutable tuple from the existing bounded
  `verifiedViews` entries before compilation, coalesce cold sibling loads,
  and remove the temporary projection decode used only to discover its digest.
  No second cache, index, metadata selector, or publication fallback is needed.
  Initial published-artifact/source verification and per-attempt lineage,
  profile, scope and compatibility checks remain mandatory.
- Regression oracle: warm preparation allocation must not grow with schema
  width; test real command and scheduled entry points, cold concurrent callers,
  cancellation, exact-snapshot isolation, missing publication and recovery.
  Benchmark the same toy widths before and after. Archive diagnostic compiler
  counts as additional evidence, not a substitute for behavioral regressions.
- Evidence limits: whole-batch latency and database cost require a frozen
  local walking slice and later exact-cohort environment acceptance. The old
  canary only required a nonzero final result. No full-batch speedup is claimed.

### Red test / baseline (before implementation)

- Real `OpenCommand` plus authority `Bind`, 20 warm calls, toy schema widths
  0 and 200: regression fails with 2,898 versus 21,851 allocations/op.
- Three 200 ms benchmark samples: small schema 637,607–637,983 B/op;
  wide schema 4,944,644–4,963,196 B/op. Timing was noisy (1.10–4.43 ms
  small, 8.64–10.29 ms wide); allocation scaling is the decisive baseline.
- Local Go/darwin arm64, Apple M4; logs retained at
  `/private/tmp/worker-sibling-fix.wFGAor/{red,baseline-bench}.log`.
- Result supports the planned owning-layer fix; next change moves immutable
  sibling reuse ahead of projection without weakening publication verification.

### Implemented fix / green evidence

- `openSiblingProfileFromSource` now resolves the immutable tuple against the
  existing bounded verified-view cache before projection. Cold misses share
  one load; the projection supplies only an address, never execution authority.
  Removed the temporary projection decode. No protocol or metadata changes.
- Every caller still checks scope, profile, compatibility and complete source
  lineage (including schema and engine generation). Missing publication fails
  retryably; a later publication or process restart can recover normally.
- Original failing command regression: small/wide **67 / 67 allocations/op**.
  Scheduled Flow equivalent: **304 / 304 allocations/op**. Both use real
  DirectoryStore publications and production execution-context installation.
- Same three-sample benchmark: **5,963–5,965 B/op** at either schema width,
  versus approximately 0.64 MB / 4.95 MB before. New timings 24.7–39.8 microseconds
  are diagnostic only; other local tests were running concurrently. This is
  binding preparation, not database/Flow execution or whole-batch throughput.
- Reopened the original probe using a Go overlay that adds only the diagnostic
  test to the **actual fixed source**. Atomic coverage counters, warm 0 versus
  warm 20: `ProjectRuntimeMetadataView` **8 -> 8**, `OpenRuntimeMetadataView`
  **12 -> 12**, sibling address derivation **2 -> 2**, physical reads **+0**.
  Thus the original **40 extra projections / 40 extra decodes become 0 / 0**.
- Cold concurrent test: one canceled caller plus 24 surviving callers, repeated
  five times. Atomic counters total 10 sibling address derivations, i.e. one
  for each of the two sibling capabilities per cold run, independent of caller
  count. All survivors receive exact mutation/Flow views; canceled callers
  return promptly. The ordinary test's file-read counts alone are not used as
  proof of compiler coalescing.
- Regression coverage also includes wrong org/installation/revision/schema/
  engine/profile, incompatible engine on warm entries, missing publication and
  retry, same-numbered-revision/different-digest isolation, and bounded-cache
  eviction/reopen. An independent read-only adversarial review found no blocker.
- Evidence files: `green-regressions.log`, `fixed-bench.log`,
  `fixed-atomic-{warm0,warm20,concurrent5}.{log,cover}` in the directory above.
  Diagnostic probes are not shipping code or replacement implementations.

### Verification commands and limits

- PASS: `go test ./backend/internal/runtimeworker ./backend/internal/runtimeapi
  ./backend/internal/runtimeflowexecution ./nexuscrm-metadata/catalogsnapshot`
  (all four package results from the broader run).
- PASS: `go test -race ./backend/internal/runtimeworker -run
  'Test(CommandBindingWarm|ScheduledBindingWarm|CommandSibling|SiblingReuse|CommandArtifactBinder|ScheduledFlowArtifactBinder)'
  -count=3`. macOS linker emitted an `LC_DYSYMTAB` warning; no race or test failure.
- PASS: `go test ./nexuscrm-automation -run
  '^TestPrepare(Scheduled|Timed)FlowExecutionPlan' -count=1 -timeout=60s`.
- PASS: `go vet ./backend/internal/runtimeworker ./backend/internal/runtimeapi
  ./backend/internal/runtimeflowexecution` and `git diff --check`.
- PASS: `go test ./backend/... -run '^$' -timeout=60s` (compile every backend
  package and its tests; deliberately not a full backend behavioral test run).
- The broad automation-module suite also starts local TiDB integration tests;
  its completion is recorded separately below. It is not part of the timing
  benchmark or evidence for warm-binding reuse.
- No deployment or prod access. The incident's entire batch, queue-drain time,
  and database delay still need exact-cohort end-to-end acceptance after rollout;
  this delivery does not relabel the old nonzero-result canary as full success.

### Broader-suite result (outside the binding patch)

- The broad automation run finished after 331.665 seconds with one failure:
  `TestFencedScheduledRunTiDBCommitsEffectsAndCompletionTogetherAndFencesStaleOwner`,
  `fenced_scheduled_run_reservation_integration_test.go:737`:
  `CheckReady() with legacy active row error = <nil>`.
- This is a legacy-row readiness assertion, not sibling preparation. The module
  and its source dependencies have no local changes; `go list -test -deps
  ./nexuscrm-automation` confirms that the changed runtimeworker package is not
  in its dependency graph. The run used local Docker TiDB v8.5.0 on port 4001,
  not an external database. No unrelated test or readiness code was changed to
  make this suite green. This result remains a separately recorded limitation,
  not a claim that the entire test suite passed or that its cause was fixed.
- Logs: `packages.log`, `automation-deps.txt`, `backend-compile.log`.
  The stronger final eviction assertion (all three artifacts must reopen)
  also passed independently in `eviction-final.log`.

### PR preparation on updated main

- Rebased the isolated five-file fix onto
  `e266f1309cea5ad9f675a4890dd7199b8aa4f8c9` after main advanced by 24 commits.
  The rebase preserved main's new Layout command/snapshot handling; the sibling
  fix itself required no additional production changes.
- PASS on the rebased candidate: the command/scheduled/sibling/mutation binder
  regressions, the same set under `-race -count=3`, all runtimeapi,
  runtimeflowexecution and catalogsnapshot tests, whole-backend compile-only
  checks, and the three-package `go vet` check listed above.
- Warm allocation regressions remain command **67 / 67** and scheduled
  **304 / 304**. The macOS race linker warning remains non-fatal; there were no
  race reports. Full automation-module testing was not repeated on this base;
  its earlier failure remains explicitly recorded above.
- Exact commands are in the PR verification section. Local logs are
  `pr-rebase-{worker,race,neighbors,compile,vet}.log` in the evidence directory.

### Post-merge timeout ownership regression — 2026-09-02

- Review comment `#4759#issuecomment-5509472878` identifies a real regression:
  sibling singleflight added a 30-second overall deadline, although snapshot
  and published-view loaders already own independent 30-second budgets.
- Reopened evidence against reviewed base `5b81671fc1c0b06191c61d88838f1809034533c2`
  and head `0fb6034d0b36c0c2ad69ae731d7e6611228095f1`: real published artifacts,
  cold sibling, two transport delays of 18 seconds. Base succeeded in 36.007s;
  head returned `context deadline exceeded` at 30.001s. Source admission usually
  warms the snapshot, but projection plus the subsequent view load still acquire
  an unintended combined budget. Warm-path performance results remain valid.
- Logs, overlays, and probe: `/private/tmp/sibling-timeout-audit.sImZYf`.
  No production data or deployment was involved.
- Follow-up base: merged `977888a3051c77eec5966dfc48e4a25c61127ac3`.
  Fix only the owning boundary: retain detached shared orchestration, remove its
  extra overall timeout, leave per-artifact budgets and per-caller cancellation
  intact. Add virtual-time regressions over the real publication/verification
  path for combined slow reads, each individual timeout, and a caller deadline.
- RED before the production edit: combined reads failed at 30s instead of
  succeeding at 36s; the second read's deadline fired at total 30s instead of
  48s; a surviving caller also inherited the unintended total timeout. GREEN
  after the edit: all four cases pass using Go 1.25 virtual time (under 0.2s
  test-body wall time), without a production timeout override.
- PASS: `go test ./backend/internal/runtimeworker ./backend/internal/runtimeapi
  ./backend/internal/runtimeflowexecution ./nexuscrm-metadata/catalogsnapshot
  -count=1 -timeout=120s`.
- PASS: `go test -race ./backend/internal/runtimeworker -run
  'Test(CommandBindingWarm|ScheduledBindingWarm|CommandSibling|SiblingReuse|SiblingLoadIndependent|CommandArtifactBinder|ScheduledFlowArtifactBinder|MutationArtifactBinder)'
  -count=3 -timeout=180s`. Only the previously observed macOS `LC_DYSYMTAB`
  linker warning; no race reports.
- PASS: `go vet ./backend/internal/runtimeworker ./backend/internal/runtimeapi
  ./backend/internal/runtimeflowexecution` and `git diff --check`.
- Warm allocation regressions remain command **67 / 67**, scheduled **304 / 304**.
  Full automation-module testing and environment acceptance were not repeated;
  the separately recorded broader-suite limitation above remains unchanged.
- Follow-up logs: `/private/tmp/sibling-budget-fix.yg67Ll/{focused,packages,race}.log`.
- The original real-time diagnostic also passes against the follow-up: two
  18-second reads completed in **36.003s**, returning the exact published sibling
  (`walltime-probe.log`). Only the diagnostic test was injected with a Go overlay;
  the candidate production code and its timeout constant were unchanged.