# API Queue Observation Performance Learnings

This document records each evidence loop for the API-hosted Outbox and Rollup
Delta queue observers. These observers only publish Prometheus metrics; they do
not own queue admission, claiming, retry, or processing behavior.

## 2026-08-16 — static-audit-2026-08-16

- **Commits:** baseline `90e44b728be19732f76856381b428841179c0c89`
  (`origin/main`); implementation not started.
- **Environment:** current-tree static audit. Production RU attribution remains
  the Test2 deployment follow-up; this loop establishes statement frequency and
  connection ownership from the executable wiring.
- **Workload shape:** every API replica starts both observers immediately and
  then every 15 seconds. Outbox executes two aggregate statements per sample;
  Rollup Delta executes five.
- **Profile result:** 28 aggregate statements per minute per API replica
  (Outbox 8, Rollup Delta 20), including an uncoordinated startup burst. Both
  observers use the shared primary business pool. Cost scales linearly with API
  replica count and table growth.
- **Highest-ROI bottleneck:** sampling frequency and replica multiplication, not
  Prometheus collection. `/metrics` already reads in-memory gauges without a DB
  query.
- **Change hypothesis:** replace the two 15-second observers with one L0-owned,
  fixed hourly scheduler. It must perform no startup read, lazily open one
  independent pool limited to one connection, run sources sequentially with
  independent timeouts, preserve the last successful gauge snapshot, and expose
  per-source freshness/failure/duration telemetry. Existing queue metric names
  and labels remain compatible.
- **Correctness boundary:** do not synthesize a global queue gauge by accumulating
  process-local deltas. Multiple API/worker replicas and restarts make such a
  value incomplete. The hourly database snapshot remains the exact source of
  truth; memory is only the Prometheus read cache.
- **Expected result:** reduce these observers from 28 aggregate statements per
  minute per API replica to seven per hour per API replica (240x fewer), remove
  startup scans, and isolate observation from the business pool.
- **Next hypothesis:** focused lifecycle and refresh tests can prove no startup
  read, one-connection reuse/close, failure isolation, last-good preservation,
  and unchanged queue gauge publication before Test2 measures the RU reduction.

## 2026-08-16 — implementation-loop-1

- **Commits:** working tree on baseline `90e44b728be19732f76856381b428841179c0c89`.
- **Environment/workload:** formatting gate for the two refresh packages, the
  API observation coordinator, runtime wiring, and focused tests.
- **Result:** formatting stopped before compilation because one new sqlmock
  expectation placed the selector dot at the start of the next line
  (`rollup_delta_queue_test.go:285`). No runtime or performance conclusion was
  drawn from this failed gate.
- **Learned / next hypothesis:** fix only the malformed test chaining syntax,
  then rerun formatting and the focused package tests unchanged.

## 2026-08-16 — implementation-loop-2

- **Commits:** working tree on baseline `90e44b728be19732f76856381b428841179c0c89`.
- **Environment/workload:** focused Go tests for the Outbox middleware and API
  runtime packages, followed by the records module if the backend gate passed.
- **Result:** Outbox middleware tests passed. API runtime compilation stopped on
  one unused `records` import left in `runtime.go` after the old direct Rollup
  sampler call was removed; records tests therefore did not run in this loop.
- **Learned / next hypothesis:** remove the stale import and rerun the identical
  focused gate; no behavior or design change is indicated.

## 2026-08-16 — implementation-loop-3

- **Commits:** working tree on baseline `90e44b728be19732f76856381b428841179c0c89`.
- **Environment/workload:** focused tests for
  `backend/internal/host/adapters/apihost/middleware`,
  `backend/pkg/serverruntime`, and the complete `nexuscrm-records` module.
- **Result:** all focused tests passed. Tests cover no startup pool open, fixed
  one-open/one-idle independent pool reuse and close, per-source timeout
  isolation, freshness/failure publication, and last-good gauge preservation.
- **Learned:** the coordinator can remove the 15-second shared-pool samplers
  without coupling the two sources' failure domains. The statement-frequency
  hypothesis remains 28/minute/replica to 7/hour/replica; actual RU reduction
  still requires the same Test2 workload after deployment.
- **Next hypothesis:** a strict current-diff and consumer audit should falsify
  hidden lifecycle, metric-compatibility, stale-path, or business-path changes
  before broad repository tests.

## 2026-08-16 — review-loop-1

- **Commits:** uncommitted implementation on baseline
  `90e44b728be19732f76856381b428841179c0c89`.
- **Environment/workload:** adversarial current-diff, symbol-consumer, startup,
  lifecycle, and Prometheus registration review.
- **Result:** business queue mutation paths remain untouched and the two old
  sampler symbols have no remaining consumers. One metric-compatibility defect
  was found: after removing the immediate startup query, fixed-label GaugeVec
  children would not be created until the first successful hourly observation,
  so existing queue series could be absent from `/metrics` for up to one hour.
- **Learned / next hypothesis:** synchronously initialize every fixed existing
  queue series to zero while freshness is `stale=1`. This preserves metric
  discovery without a startup DB read and makes zero explicitly mean “unknown
  initial cache” until `last_success` becomes positive. Focused tests must prove
  initialization and no-startup-read together.

## 2026-08-16 — implementation-loop-4

- **Commits:** uncommitted implementation on baseline
  `90e44b728be19732f76856381b428841179c0c89`.
- **Environment/workload:** repeated focused tests for the Outbox middleware,
  API runtime, and complete records module after fixed-series initialization.
- **Result:** all focused tests passed. Existing fixed queue series are created
  synchronously with zero values and `stale=1`; the independent pool still does
  not open before the first tick. Successful refreshes set `last_success` and
  clear stale independently per queue.
- **Learned / next hypothesis:** metric discovery and DB-read isolation now hold
  together. Focused race detection and broad backend/shared tests should expose
  any residual lifecycle or cross-package regression before commit.

## 2026-08-16 — verification-loop-1

- **Commits:** uncommitted implementation on baseline
  `90e44b728be19732f76856381b428841179c0c89`.
- **Environment/workload:** focused race tests for Outbox middleware and API
  runtime; complete backend and shared module tests; complete records module
  tests were already repeated in implementation loop 4.
- **Result:** all tests passed. The race build emitted only the local macOS
  linker's malformed `LC_DYSYMTAB` warnings; it reported no data race or test
  failure. No frontend files changed, so no frontend build was needed.
- **Learned:** the lifecycle, metric registration, refresh composition, and
  unchanged business packages compile and test together. Static review still
  needs to confirm generic L0 boundary compliance, clean diff, removed sampler
  references, and final changed-file scope.
- **Next hypothesis:** if the boundary scan and final diff are clean, the change
  is ready to commit; Test2 remains the owner of post-deployment RU measurement.

## 2026-08-16 — verification-loop-2

- **Commits:** uncommitted implementation on baseline
  `90e44b728be19732f76856381b428841179c0c89`.
- **Environment/workload:** final whitespace check, removed-symbol search, full
  public-core boundary scan, and changed-file boundary inspection.
- **Result:** whitespace check passed and no old Outbox/Rollup sampler symbol
  remains. The repository-wide boundary script exited 1 on existing main-tree
  hits outside this change (including `shared/pkg/models/salesforce_flow.go` and
  `backend/internal/runtimeworker/metadata_binding.go`). No added non-test line
  in this change matches the boundary rule; test-only `__c` fixtures in the
  pre-existing records test file remain exempt by the scanner itself.
- **Learned:** the PR introduces only generic L0 observation mechanisms and no
  tenant/product payload. The repository-wide boundary result is a baseline
  limitation, not a regression from this change.
- **Next hypothesis:** stage and inspect the complete diff (including new files),
  then commit if the final scope remains limited to the two metrics observers.