# Runtime Flow command execution performance

## 2026-09-02 — Test3 2,000-contact non-RU baseline

- Run: acceptance case `admin/hubspot/k-camp-10`, case run
  `k-camp-10-791ed4cb`, Test3. Core/runtime-worker commit
  `015c75c24324b138ca99c55ab889c3acc3c65e6d`; Connectors release
  `v0.2.20-exp.4740.17`. The workload admitted 5,848
  `runtime.flow.execute` commands from 2,000 contact-creation events plus the
  later property-change projections. This investigation did not read command
  payloads or credentials.
- Deployed-worker counters at 19:26 UTC: 2,651 attempts, 2,650 authoritative
  attempts and 2,644 successful finalizations in 5,876.144 seconds of combined
  claim/effect wall time. The worker had one execution lane and no database-pool
  waits. The queue-claim full scan, RU queuing/lease loss and process restart are
  owned by a separate incident track and are not changed here.
- A 30.13-second CPU profile while the backlog was continuously draining
  contained 3.22 CPU-seconds (10.69% of one core). Formula evaluation owned
  1.09 CPU-seconds (33.85%); regexp normalization/rewriting owned 0.89
  CPU-seconds (27.64%). This is the dominant in-process, non-RU stage.
- The allocation profile at the same point contained 99.18 GB cumulative
  allocation since worker start, not resident memory. `FlowExecutor` execution
  accounted for 92.87 GB; validation accounted for 49.87 GB; formula evaluation
  for 40.14 GB; expression program lookup/compilation for 24.04 GB; and formula
  registry cloning for 7.37 GB. A later profile delta added 5.16 GB while the
  same workload continued. Independent runtime counters advanced by roughly
  50 MB allocated per command over the surrounding interval.
- A 100-sample, 200-millisecond goroutine observation found the active command
  in the SQL-driver/TLS wait stack 93 times and in local formula/metadata work
  seven times. A second 50-sample call-site pass attributed eight samples to
  queue claim; effect-side waits were spread across query, record, audit and
  connector durable stores. Thus CPU is not the whole wall-time bottleneck, but
  repeated formula preparation is the largest directly observed local cost and
  magnifies GC pressure.
- Current-tree cause: the Flow executor already reuses one formula engine per
  executor, and `RuntimeFormulaEngineFromContext` reuses an exact-view engine
  inside one admitted execution. The record-mutation validation composition
  bypasses that request cache and calls `RuntimeMetadataView.NewFormulaEngine`
  for every validation pass. One Flow command can create and update records in
  the same owned transaction, so it repeatedly clones the same immutable
  revision's function registry and recompiles the same validation expressions.
  The allocation profile attributes 4.43 GB directly to the deployed
  `records_mutation.go:290` resolver call `view.NewFormulaEngine()`; the
  `RuntimeMetadataView.NewFormulaEngine` stack accumulated 8.04 GB in total.
- Next hypothesis: route validation through the existing request-bound formula
  engine cache, keyed by the already verified `RuntimeMetadataView` pointer.
  This keeps revision isolation and request lifetime unchanged while eliminating
  duplicate engine/regexp/program preparation within one command. Prove same
  view reuse, different-view isolation, concurrent safety and allocation change
  with focused tests/benchmarks before considering any cross-command cache.
- No production change, deployment, GitHub Action, push or pull request was made
  in this baseline loop.

## 2026-09-02 — Exact-view request cache result

- The focused change routes record-mutation validation through the existing
  request-local formula-engine cache while supplying the exact mutation sibling
  view. Cache identity is the `RuntimeMetadataView` pointer: the primary and
  mutation views remain isolated, a fresh request receives a fresh engine, and
  nil or missing authority still fails closed. There is no cross-request,
  cross-process or cross-revision cache.
- Focused semantic tests passed in `github.com/nexuscrm/metadata` and
  `github.com/nexuscrm/backend/internal/runtimeassembly/recordmutation`.
- On the repository's representative Runtime metadata fixture, five 500 ms
  benchmark samples measured a median 409,869 ns/op, 157,971 B/op and 765
  allocations/op for constructing and evaluating with a fresh engine. Resolving
  and evaluating with the request-local exact-view engine measured a median
  2,692 ns/op, 510 B/op and 11 allocations/op. The isolated repeated-call seam
  is therefore 152x faster with 310x fewer bytes and 70x fewer allocations.
  This is a microbenchmark of the proven preparation seam, not an end-to-end
  Test3 throughput claim.
- The exact metadata/cache and record-mutation resolver tests passed three
  times under the race detector. Both changed packages' complete local test
  suites passed, `go vet` passed, and `git diff --check` passed. Keep any
  proposed Prometheus stage observability separate so optimization and
  diagnostics have independent review and rollback surfaces.
- Test3 was not redeployed for this loop, so the benchmark confirms the local
  mechanism and its isolated cost but does not yet prove an end-to-end QPS
  change. Deployment and an equivalent bounded acceptance run are required to
  measure that effect.