# Runtime Worker Observation Performance Learnings ## 2026-08-16 - Record-event queue observation baseline (#4425) - Run: the retained quiet-window Test2 evidence is recorded in issue #4425; deployment-specific run identifiers remain in the L2/local performance worklog and are not copied into public core. - Source revision: `ddc123007ebd6c3197ac89e2b48ef935357146a5`. - Environment: an independently deployed Runtime Worker reading a resource-controlled TiDB queue partition. Tenant identity and payload are outside this repository. - Workload shape: no selected foreground case was running while the record-event Worker refreshed exact pending/processing depth, poison depth, and oldest active age every 30 seconds. - Profile result: the exact aggregate examined tens of thousands of keys per sample and spent most of its wall time in resource-control queueing. The observation is not an admission, claim, retry, repair, readiness, or effect authority. - Failed/insufficient probe: a range-grouped SQL candidate was much faster on an automatically selected small scope, but that scope touched only one key and therefore does not prove the active heavy-scope result. - Invariant: `HasOutboxWorkHint` remains the bounded negative hint and the deployment/ownership-fenced durable claim remains the sole processing authority. Observation failure must not suppress or authorize work. - Next hypothesis: defer the first exact observation until a fixed one-hour L0 interval, keep Prometheus collection memory-only, expose sample freshness and failures, and preserve the last successful snapshot on sample failure. This reduces scheduled aggregate executions from 120 per hour to one without an L2 tuning surface. - Required next evidence: focused tests must prove no startup observation, hourly sampling, cached metrics, and failure isolation. Test2 must then compare quiet-window statement count, read RU, TiDB latency, resource-control queueing, keys, and the two consecutive focused eight-case runs required by issue #4425. ## 2026-08-16 - Implementation verification loop - Run: local `runtimeworker` package tests plus a race-enabled focused suite on the observation and metrics paths. - Source revision: working tree on `codex/issue-4425-hourly-observation` based on `ddc123007ebd6c3197ac89e2b48ef935357146a5`. - Environment: local Go test process with SQL mocks; this run does not claim a production RU result. - Workload shape: idle 15-lane record-event worker, fixed-cadence construction, successful and failed cached observations, a sampler-owned timeout, and Prometheus collection from the cached snapshot. - Profile/result: the full `internal/runtimeworker` package passed; the focused suite also passed under the race detector. Code-path inspection confirmed that record events are the only production metadata processor implementing the bounded work-hint interface, so the new fixed observation policy does not change rollup-delta or Platform-command admission. - What changed: no startup exact observation, one fixed L0 hourly ticker, a 30-second sampler timeout, last-good cache retention, and explicit last-success/duration/failure/stale metrics. - What failed: no test failure; the race-enabled link emitted a macOS linker warning but completed successfully. - Learned: the collector consumes atomics and database pool counters only; it never calls the queue repository. The remaining resource claim must be established by the deployment Test2 comparison, not inferred from unit tests. - Next hypothesis: remove test-only bookkeeping that is not asserting an invariant, then run the repository-standard backend/shared checks and record the final local result. ## 2026-08-16 - Final local validation - Run: repository-standard Go validation after the review cleanup. - Source revision: final working tree on `codex/issue-4425-hourly-observation`. - Environment/workload: local unit and integration-style test suites; no live TiDB load was generated. - Result: `go test ./...` passed in both `backend` and `shared`; the complete `internal/runtimeworker` package and the focused race-enabled observation suite had already passed in the preceding loop. - Regression review: no frontend, schema, SQL text, work-admission, claim, retry, or L2 configuration path changed. Metrics scrape reads only the cached worker snapshot and `database/sql` pool counters. - Remaining evidence: production-like Test2 is still required to measure the expected statement-frequency and RU reduction and to verify latency and resource-control queueing under the retained workload. ## 2026-08-16 - Pre-PR isolation review - Run: adversarial call-flow review of all three metadata-worker run modes. - Source revision: uncommitted pre-PR working tree on `codex/issue-4425-hourly-observation`. - Evidence: the hourly ticker case called `refreshObservation` synchronously from the serial, coordinated-parallel, and parallel control loops. A slow aggregate could therefore pause polling, refill/claim coordination, effect completion accounting, and maintenance for up to the sampler timeout. - Why the previous evidence was insufficient: unit tests proved timeout and last-good cache behavior but did not prove that the timeout was isolated from durable work processing. - Owning cause: observation scheduling shared the business control-loop goroutine even though observation is diagnostic-only. - Next change: move the fixed hourly sampler into one lifecycle-bound goroutine per worker run, delete the observation cases from business selects, and add a focused test showing that a blocked observation cannot block a processing pass. Re-run the same package, race, backend, and shared verification after the change. ## 2026-08-16 - Observation isolation verification - Run: complete `internal/runtimeworker` package plus the focused observation/metrics suite under the race detector after asynchronous sampler isolation. - Source revision: final pre-commit working tree on `codex/issue-4425-hourly-observation`. - Result: both suites passed. The new regression test holds `Observe` blocked, then proves a business processing pass completes independently. The sampler stop function cancels and joins its goroutine, so worker shutdown does not leak the diagnostic query lifecycle. - Cleanup: removed the obsolete synchronous refresh wrapper and kept the one production scheduling path through the fixed L0 interval and timeout. - Remaining evidence: repeat repository-wide local tests, then use Test2 after deployment for the real RU and quiet-window cadence comparison. ## 2026-08-16 - Final pre-PR validation after isolation - Run: `go test ./...` in `backend` and `shared`, followed by diff and public boundary checks. - Result: both Go suites and `git diff --check` passed. The public-core boundary scan still reports the existing Salesforce-name and test-fixture baseline; this change adds no scan hit and no tenant payload. - Final local conclusion: the implementation now changes diagnostic scheduling and cached metrics only. The exact SQL, schema, hint, claim, retry, ownership, and effect paths remain unchanged, and a blocked observation is proven not to block processing. - Remaining production evidence: deploy the Worker candidate to Test2 and compare quiet-window statement count, RU, latency, processed keys, and resource-control queueing against the retained baseline. ## 2026-08-16 - Shared-pool isolation blocker (#4428 review) - Run: pre-change review of database-pool capacity, record-event repository wiring, independent-pool primitives, and Runtime process lifecycle. - Evidence: the worker pool admits the minimum `2*parallelism+1` capacity for effect lanes, heartbeats, and controller work. The asynchronous sampler still called a repository backed by that same pool, so its 30-second timeout could consume capacity required by claims or ownership verification. - Owning cause: goroutine isolation did not establish database-resource isolation. - Rejected shape: merely increasing the configured shared pool would leave no hard resource boundary and would transfer a diagnostic concern into L2 sizing. Eagerly opening an auxiliary pool during assembly would also need a new cleanup path when a process is assembled but never started. - Next change: give record-event observation a lazy factory backed by `TiDBConnection.OpenIndependentDB` with a fixed one-open/one-idle L0 pool. The observation loop owns closing that factory result after canceling the query and before it reports stopped. Claims, effects, hints, readiness, and ownership remain on the primary pool. - Required evidence: focused tests must prove factory reuse, close/reopen lifecycle, fixed pool bounds, and loop-owned cleanup; existing blocked-query processing isolation and race tests must continue to pass. ## 2026-08-16 - Dedicated-pool focused test loop 1 - Run: focused record-event factory/lifecycle and blocked-observation tests. - Result: implementation compiled and all behavioral assertions reached, but the dedicated-pool test failed because `sqlmock` requires an explicit close expectation; the observed unexpected operation was `database Close`. - Learned: the test already observed the intended pool release. The harness contract, not the production lifecycle, was incomplete. - Next change: declare the close expectation and rerun the same focused suite; do not suppress or discard the close error. ## 2026-08-16 - Dedicated-pool focused test loop 2 - Run: the same focused suite after completing the mock lifecycle contract. - Result: passed. The tests prove one fixed one-open/one-idle pool request, reuse within a worker observation lifecycle, close on loop shutdown, reopen for a later lifecycle, and continued business processing while observation is blocked. - Next evidence: inspect the final call graph for a shared-pool fallback or orphaned close path, then run the full package and focused race suite. ## 2026-08-16 - Dedicated-pool call-graph review - Run: reference scan from `metadataWorker` observation through `recordEventProcessor`, Events, Persistence, and the DB opener. - Result: production exact observation has one path and it is constructed only from `connection.OpenIndependentDB`; no shared-pool fallback remains. The loop cancels the in-flight query, closes the observation source, and closes its `done` channel only after cleanup completes. - Cleanup before broad tests: remove unused call bookkeeping from the observation-source test stub, then run package/race verification. ## 2026-08-16 - Dedicated-pool package and race verification - Run: complete `internal/runtimeworker` tests, focused dedicated-pool and observation tests under the race detector, and `go vet` for the package. - Result: all passed. The race-enabled linker repeated the known macOS `LC_DYSYMTAB` warning but returned success. - Next evidence: run repository-standard backend/shared suites and final diff checks before updating PR #4428. ## 2026-08-16 - Dedicated-pool final local validation - Run: repository-standard `go test ./...` in `backend` and `shared`, plus `git diff --check`. - Result: all passed. Final review found no shared-pool fallback: exact record-event observation lazily opens only the fixed single-connection pool, while hints, claims, effects, heartbeats, readiness, and ownership retain the primary pool. - Remaining production evidence: Test2 must still measure the hourly statement cadence and RU/latency/resource-control results; that evidence is independent of the resolved in-process pool-isolation blocker. ## 2026-08-16 - Record-command lifecycle baseline (#4427) - Run: current-tree static audit on `d5798a07485b2314f7aca1f2e2b05ecc33ea88f7` plus the retained Test2 quiet-window measurements in issue #4427. - Environment/workload: the independently fenced record-command lifecycle Worker polls a shared TiDB queue. The retained approximately two-minute window observed 113 executions of the exact lifecycle aggregate at 81.517 average read RU, approximately 9,211 total read RU. - Current profile: every startup/pass executes `COUNT(*)` plus `MIN(OutcomeExpiresAt)` before the bounded resolve/delete operations. The query filters only by expiry even though the verified surviving index is `(State, OutcomeExpiresAt, Id)`, so it cannot use the leading state key as a bounded eligibility range. Exact observation also shares the business pool. - Consumer classification: the aggregate is both a negative admission hint and a metrics source. It is not the durable authority: `ResolveExpired` and `DeleteExpired` retain the ownership fence, row locks, state/expiry rechecks, bounded limits, and transactions. - Rejected local experiment: an uncommitted worktree outside this branch backs the same aggregate off to 30 seconds after an empty result. It is preserved but not adopted because it keeps exact aggregation at high cadence, retains the shared-pool coupling, and has no explicit freshness contract. - Change hypothesis: add an index-supported `EXISTS (... LIMIT 1)` over the closed `ACCEPTED`/`COMPLETED` states for high-frequency admission. A negative result may defer a racing deadline only until the next bounded poll; a hint error must fail open into the existing fenced work. Move exact count/oldest diagnostics to the fixed one-hour, no-startup observation lifecycle and a lazily opened independent one-connection pool using the startup-captured DB specification. Preserve the last good snapshot and existing freshness metrics. - Schema hypothesis: no table or index change is required. The eligibility hint and bounded resolve/delete queries can use the already verified lifecycle index. The hourly exact aggregate may add the closed state predicate to avoid treating unsupported residue as actionable work. - Expected result: the measured high-RU aggregate digest disappears from poll cadence and runs at most once per hour; the replacement hint remains frequent but examines only the first eligible index entry. Actual RU, latency, resource-control queueing, keys, and plan remain Test2 evidence, not a local unit-test claim. - Next evidence: focused SQL, fail-open admission, no-startup observation, blocked-observation isolation, fixed pool reuse/close, and last-good metrics tests before broad backend/shared validation. ## 2026-08-16 - Record-command implementation loop 1 - Run: focused persistence SQL tests and Runtime Worker record-command/ independent-observation tests on the uncommitted `codex/issue-4427-record-command-observation` working tree. - Result: both focused suites passed. The SQL contract uses the existing `(State, OutcomeExpiresAt, Id)` prefix for the bounded `EXISTS` hint; hint failure falls through to the unchanged fenced resolve/delete path. Exact observation has no startup read, preserves the last good snapshot on failure, reuses and closes a fixed one-connection independent pool, and a blocked observation does not block a business pass. - Connection result: long-running production worker assembly now retains the startup-captured immutable DB specification through `TiDBConnection`; the primary pool size/readiness behavior is unchanged, while record-command exact observation can lazily manufacture its separate pool without rereading the environment. - What was not measured: local mocks do not establish TiDB plan choice, keys, RU, latency, or resource-control queueing. Test2 remains required for those facts. - Next hypothesis: a strict current-diff and state-machine audit should find any unsupported lifecycle state, shutdown, source-factory, or assembly blast-radius defect before expanding the test scope. ## 2026-08-16 - Record-command strict review loop - Run: current-diff, constructor-consumer, lifecycle-state, and connection-opener audit after the focused suites passed. - Result: `ACCEPTED` and `COMPLETED` are the exhaustive persisted command states; the exact observation and indexed hint therefore cover the whole actionable lifecycle. The fenced `ResolveExpired` and `DeleteExpired` transactions are unchanged, and the high-frequency path no longer calls the exact aggregate. Production roles retain their existing primary pool settings while the immutable connection specification is now available to lazily open the diagnostic pool. - Review finding: no business-semantic defect was found. One comment still claimed payload-only roles did not retain the connection specification, which became false when production assembly gained the independent observation pool. Correct that documentation before broader verification. - Remaining uncertainty: only a Test2 plan/RU measurement can establish the physical TiDB benefit; local review and tests establish query shape, isolation, freshness, and fail-open correctness only. - Next evidence: fix the stale ownership comment, then run complete persistence and Runtime Worker suites plus the focused race detector before repository- standard backend/shared verification. ## 2026-08-16 - Record-command package and race verification - Run: complete `nexuscrm-persistence` and `backend/internal/runtimeworker` suites, the record-command Runtime Worker tests under the race detector, and `git diff --check` after formatting the final Go diff. - Result: all commands passed. The race-enabled linker repeated the known macOS `LC_DYSYMTAB` warning but returned success. No test exposed a business-path, pool-lifecycle, shutdown, or metrics-freshness regression. - Next evidence: run the repository-standard backend/shared suites, the records module suite that owns authoritative lifecycle transitions, and the public- core boundary scan before the final current-main comparison. ## 2026-08-16 - Record-command repository validation - Run: repository-standard `go test ./...` in `backend` and `shared`, complete `nexuscrm-records` tests, and `scripts/scan-public-core-boundary.sh`. - Result: all Go suites passed. The boundary scan reported only pre-existing repository hits in files outside this diff; none of the record-command, persistence, worker-database, or performance-record files changed here appear in its output. - Final local claim: durable lifecycle transitions still use the original records transactions and ownership fences. Frequent admission is now a bounded, fail-open existence hint; exact queue count/age is diagnostic-only, no-startup, hourly, last-good, and isolated to one independent connection. - Remaining production evidence: compare the deployed TiDB statement digest, plan, examined keys, RU, latency, and resource-control queueing after rollout. Do not infer those physical results from local tests. - Next evidence: refresh `origin/main`, resolve any newly landed overlap, then perform the final diff review and publish the focused PR. ## 2026-08-16 - Record-command final publication review - Run: refreshed `origin/main`, compared branch ancestry and changed paths, ran `go vet` for persistence and Runtime Worker, rechecked the complete diff, and searched the final high-frequency call path for exact aggregation. - Result: `origin/main` remains `d5798a07485b2314f7aca1f2e2b05ecc33ea88f7`, the branch is directly based on it, and there is no newly landed overlap. Both vet commands and `git diff --check` passed. The only lifecycle `COUNT`/`MIN` call is behind the hourly observation source; `runPass` calls only `HasExpired` before the unchanged fenced operations. - Publication disposition: ready for a focused PR referencing #4427 and the consolidated performance inventory. Production TiDB measurement remains the rollout follow-up and is deliberately not represented as proven here. ## 2026-08-16 - Runtime Rollup Delta baseline and business-authority audit (#4427) - Run: current-tree consumer, state-machine, index, schema-evolution, and L2 deployment-topology audit on `983e3db4c3191081082117407dcb5eb5d92004d8` plus the retained Test2 quiet-window measurement in #4427. - Environment/workload: the fenced `runtime.rollup-delta.v1` worker executes the exact scoped pending/failed/oldest aggregate 106 times in approximately two minutes at 81.471 average read RU, approximately 8,636 total read RU. - Current business path: the aggregate is only a negative admission hint, but it currently gates entry to `ProcessRollupDeltaBatch`. Durable authority remains in the existing path: active-owner verification, a scoped `LIMIT 1` definition-binding peek, exact metadata binding, transactional ownership lock, parent-lane row locks, state/due rechecks, bounded retry, poison disposition, and exact repair. - Root design decision: do not use an hourly cache or process-local counter to suppress work. The producer lives in another process, and memory cannot be a complete queue authority. Poll the existing scoped bounded business peek directly; a concurrent insert may wait only until the next poll, while no cached negative can cause indefinite deferral. Exact count/poison/age moves to the established fixed one-hour, no-startup observation lifecycle and a lazy independent one-connection pool. - Index result: the surviving indexes begin with `Status`, not the complete Runtime scope. A shortened query alone can still scan other scopes. Add an additive `(OrgId, AppInstallationId, RuntimeCapabilityId, Status, NextAttemptAt, CreatedDate, Id)` index through warm schema reconciliation, retain the old status-leading indexes for the legacy unscoped consumer, and do not make the performance index a business-readiness prerequisite. Code and schema rollout order must therefore remain behaviorally safe. - Transitional ownership finding: active API startup still starts the legacy unscoped in-process Rollup worker, while the dedicated Runtime Worker has a separate ownership fence. The tracked L2 repo contains no Rollup role activation proof, and the D2B design explicitly retains legacy production traffic until a later ownership cutover. This performance slice must not remove or modify the legacy worker. Production activation of both consumers against the same queue remains a separate cutover blocker. - What local evidence cannot prove: TiDB index selection, keys examined, RU, latency, resource-control queueing, or actual environment consumer topology. Test2 must verify those facts before production activation. - Next evidence: implement the additive warm index, direct bounded admission, and isolated observation read path; then prove no startup aggregate, no metrics-to-business feedback, ownership/retry/poison preservation, pool isolation, and migration idempotency with focused tests. ## 2026-08-16 - Runtime Rollup Delta implementation loop 1 - Run: focused Records observation-store, Persistence scope-index migration, and Runtime Worker Rollup processor tests on the uncommitted `codex/issue-4427-rollup-delta-observation` worktree. - Result: all focused suites and `git diff --check` passed. High-frequency admission now returns a conservative positive without reading observation; `ProcessOne` immediately enters the pre-existing active-owner verification and scoped bounded binding peek. There is no startup exact read. Exact diagnostics lazily reuse and close the fixed independent one-connection pool, filter to the configured scope and actionable pending/failed states, and reject stale worker ownership before aggregating. - Schema result: greenfield catalog intent adds the complete scope-leading due index. Warm reconciliation retains every legacy index, accepts the exact existing shape, adds a missing index, rejects an incompatible same-name shape, and accepts only an exact result after a concurrent DDL race. - What remains unproven: focused mocks do not prove that the current diff has no stale observation caller, that the metadata Worker never falls back to the aggregate, that schema bootstrap reaches the new reconciler from the catalog, or that real TiDB chooses the index. - Next evidence: perform a strict caller/schema/worker-state review, add any missing structural or failure-path coverage, then expand to complete module and race suites. ## 2026-08-16 - Runtime Rollup Delta strict index review - Run: current diff and physical-key-width review against the complete Rollup table definition and the existing `RuntimeWorkScope.Digest` contract. - Finding: the proposed index over three `VARCHAR(255)` scope columns plus status, time, and row ID can exceed the TiDB/MySQL index-key budget under the table's multibyte collation. The platform already documents a 64-character execution-scope digest as the correct way to keep scoped claim indexes below that limit. Shipping the wide index could fail warm schema apply and block a deployment, which violates this slice's safety goal. - Correction: remove the index and migration from this PR. The surviving high-frequency business read is the pre-existing scoped `LIMIT 1` peek over the current status/due index; the measured exact aggregate still disappears from poll cadence. Test2 must measure that surviving query before any schema change. If cross-scope backlog keeps it expensive, the root follow-up is an additive `ExecutionScopeDigest` projection with a separately proven producer/backfill/cutover protocol, not a wide or partial substitute index. - Business result: this correction changes no producer, row shape, claim, ownership, retry, poison, repair, or legacy API-worker path, and introduces no schema prerequisite that could stop processing. - Next evidence: add direct-admission error/failure isolation coverage, confirm the final diff contains no schema change or old observation caller, then run complete Records/Runtime Worker suites and the focused race detector. ## 2026-08-16 - Runtime Rollup Delta package and race verification - Run: complete `nexuscrm-records` and `backend/internal/runtimeworker` suites, focused Rollup/observation tests under the race detector, and `git diff --check` after removing the rejected schema path. - Result: all commands passed. The race-enabled linker repeated the known macOS `LC_DYSYMTAB` warning but returned success. A failed observation source marks freshness stale and increments its diagnostic failure counter, while the next admission remains conservatively positive and cannot be suppressed by that failure. - Current diff boundary: no table definition, schema migration, producer, legacy API Rollup worker, claim transaction, retry/poison transition, repair, or ownership implementation is changed. The old Worker-bound exact observation method has one replacement read path on the independent pool. - Next evidence: run repository-standard backend/shared suites, vet the changed modules, audit the final call graph and current `origin/main`, then publish the focused PR if no broader regression appears. ## 2026-08-16 - Runtime Rollup Delta admission-metrics review - Run: adversarial review of the generic `metadataWorker.runDrainPass` counters and idle behavior with the Rollup processor's initial conservative-positive `HasWork` implementation. - Finding: always-positive admission cannot lose work, but it makes every idle poll enter `ProcessOne` and increment `attempts_total`. That changes existing alert/diagnostic semantics and performs the binding peek even when a bounded negative test could skip it. The result is safe for rows but not a clean compatibility-preserving optimization. - Correction: add a scoped, due-aware `EXISTS (... LIMIT 1)` hint on the primary business store. A successful negative can defer a racing insert only until the next poll; every positive is rechecked by the existing binding peek and fenced transaction; every hint error falls through through the generic Worker into that same authoritative path. Exact count/poison/age remains isolated and hourly. - Index disposition remains unchanged: use the existing status/due index for this bounded first phase and require Test2 plan/key/RU evidence. Do not add a width-unsafe scope index or a new digest projection in this PR. - Next evidence: implement and test false/positive/error/ownership-loss hint behavior, then repeat focused package and race verification before broad suites. ## 2026-08-16 - Runtime Rollup Delta bounded-admission implementation - Run: replaced the conservative-positive poll admission with an ownership-verified, exact-scope, due-aware `SELECT EXISTS(... LIMIT 1)` on the primary business connection. The diagnostic count/poison/age aggregate remains on the independent hourly observation connection and is not an input to admission. - Correctness boundary: a successful negative skips only the current poll; a racing insert is eligible on the next poll. A positive is still rechecked by the existing binding peek and fenced transaction. Ownership loss stops the hint before the queue read, while ordinary hint errors remain visible to the generic metadata worker, whose existing fail-open path invokes authoritative processing. No producer, claim, lock, retry, poison, repair, or legacy API worker code changed. - Evidence: focused Records tests cover scoped/due positive admission, ownership loss before queue access, and queue-read failure. Runtime Worker tests prove a successful negative never opens the diagnostic source, there is no startup aggregate, and a failed diagnostic refresh cannot suppress a later positive business admission. Both focused suites and `git diff --check` passed. - Limitation: the bounded query uses the existing status/due index and applies the exact scope as residual predicates. This removes the measured per-poll aggregate but does not yet prove low keys/RU under a large cross-scope pending backlog; Test2 must capture the TiDB plan and RU before considering an additive execution-scope digest projection. - Next evidence: run complete package, race, vet, backend/shared, and boundary verification; then audit the final diff and current `origin/main` before publication. ## 2026-08-16 - Runtime Rollup Delta bounded-admission package verification - Run: complete `nexuscrm-records` and `backend/internal/runtimeworker` tests, vet for both changed Go package trees, and the focused Rollup/metadata observation suite under the race detector. - Result: all commands passed. The race link repeated the known macOS `LC_DYSYMTAB` warning and returned success. No concurrency failure appeared in lazy observation-source reuse/release or admission/observation separation. - Interpretation: package-level evidence now covers the changed implementation and its failure boundaries. It does not replace repository-wide integration coverage or live TiDB plan/RU evidence. - Next evidence: run repository-standard backend/shared tests and the public boundary scan, then re-fetch `origin/main` and strictly review the final current-tree diff. ## 2026-08-16 - Runtime Rollup Delta repository regression verification - Run: repository-standard complete backend and shared Go suites plus the public-core boundary scanner. - Result: backend and shared suites passed. The boundary scanner reported only the repository's existing generic/test-fixture review list; none of the six changed files introduced a reported tenant-specific or L1 literal. - Scope check: the final code change remains inside the generic L0 Rollup Runtime mechanism, Runtime Worker composition, tests, and this performance record. Frontend and generated version files were not touched. - Next evidence: fetch and compare current `origin/main`, inspect every final diff hunk and the aggregate/admission call graph, and re-run targeted checks after any integration change before commit and PR. ## 2026-08-16 - Runtime Rollup Delta final call-path review - Run: re-fetched `origin/main`, reviewed every production diff hunk, traced `metadataWorker.runPass` and its serial/parallel fail-open behavior, and enumerated Rollup aggregate and admission callers from the current tree. - Finding: current `origin/main` advanced by one Controlplane dependency-pin commit only; it does not overlap the six changed files. A hint error enters the existing drain path, ownership loss remains terminal through the authoritative processor, and only a successful negative skips one poll. The exact Rollup aggregate has one diagnostic owner and one caller through the independent observation factory; it is no longer reachable from poll admission. - Index review: `idx_rollup_delta_claim(Status, NextAttemptAt, CreatedDate, Id)` supports the bounded status/due candidate walk. Exact Runtime scope remains a residual predicate, so the change is a decisive removal of the measured aggregate rather than a claim that all cross-scope candidate scans are solved. The latter remains a plan/RU-gated follow-up. - Disposition: no correctness or merge blocker found. Integrate the dependency pin from current `origin/main`, repeat targeted verification on that base, then commit and publish the focused PR. ## 2026-08-16 - Runtime Rollup Delta latest-main integration - Run: rebased the focused change onto `origin/main` commit `c2e7bfb75`, then repeated the complete Records and Runtime Worker package suites, the focused race suite, and diff whitespace validation. - Result: the rebase was conflict-free and every check passed. The race linker repeated the known macOS `LC_DYSYMTAB` warning and returned success. - Final evidence boundary: code and automated tests establish preservation of queue ownership, fail-open admission, hourly diagnostic isolation, and no startup aggregate. Live TiDB plan/key/RU comparison is intentionally left as the Test2 rollout measurement and is not claimed by local SQL mocks. - Next action: publish the PR without an index/schema change and keep the cross-scope plan/RU measurement visible as the explicit follow-up gate. ## 2026-08-16 - Audit Outbox observation baseline and authority audit (#4427) - Run: current-tree producer, consumer, state-machine, topology, and exported API audit on `6da6048d8af63c55243a32747bf1e265c63f6087`, plus the retained approximately two-minute Test2 window from #4427. - Measurement: the Audit Outbox exact active/poison/oldest aggregate executed 112 times at 5.227 average read RU, approximately 585 total read RU. It runs synchronously before every post-startup `AcquireLease` on the same business pool even though its result is not an admission, claim, readiness, retry, repair, or ownership input. - Business authority: the producer writes the payload and audit head in the caller's transaction; `AcquireLease` returns the durable lease; `CommitFanout` locks the active owner and lease, writes exact field rows, and consumes the queue row atomically. Known rollback, unknown commit, deterministic poison, retry exhaustion, compare-and-set repair, and ownership transfer remain independently persisted and fenced. - Consumer map: `OutboxWorker.observeQueue` is the only runtime caller of `ObserveQueue`. The legacy API process starts the worker but never reads its snapshot; only the independently composed Runtime Worker exports the queue fields. Organization-wide code search found no other snapshot or observer consumer. The API Outbox sampler owns `_System_OutboxEvent`, not this `_System_AuditOutbox` fact. - Root design: remove exact observation from the business worker loop without adding an `EXISTS` preflight. Keep `AcquireLease` unchanged as the one bounded business read. Give the independently deployed Runtime Worker one lazy, ownership-aware observation source on the fixed one-connection diagnostic pool, sampled hourly with no startup read, timeout isolation, last-good retention, and explicit freshness. Do not infer durable queue depth from process counters because producers, repair, and replacement owners live in other processes. - Scope boundary: no schema/index, producer, initial real claim, lease, fanout, retry, poison, repair, retention, ownership-transfer, or legacy/dedicated topology change belongs in this slice. Test2 must measure the surviving `AcquireLease` plan/RU separately before any physical-schema proposal. - Next evidence: implement the narrow observation source and Runtime Worker wrapper, delete the poll-cadence observation path, and prove startup, failure, shutdown, metrics-freshness, and business-state invariants before broadening tests. ## 2026-08-16 - Audit Outbox observation split implementation - Run: removed `ObserveQueue` from `OutboxWorker.processBatch` and removed durable queue fields from the business worker's process-local snapshot. Added a fenced, narrow observation resolver and an Audit-specific Runtime Worker wrapper that lazily opens the existing fixed one-connection diagnostic pool, samples on the fixed hourly/no-startup cadence, retains the last good result, exposes freshness, and closes the source on shutdown. - Business boundary: `Prepare` still performs the real initial claim and every later pass still enters the unchanged `AcquireLease` directly. Producer, candidate selection, per-row lease acquisition, exact replay, atomic fanout and consume, known/unknown commit handling, retry, poison, repair, retention, and ownership transactions were not changed. The legacy API worker receives no observer and therefore performs no diagnostic aggregate. - Evidence: complete `nexuscrm-audit` tests, focused Runtime Worker Audit tests, and `git diff --check` passed. New tests prove that a business store which implements `ObserveQueue` is never observed by the business loop; fenced observation rejects stale ownership before aggregation; startup performs no observation; last-good values survive a later failure; observation failure does not stop the business controller; and the source is reused and released on the fixed independent pool. - What remains unproven: local mocks do not establish live TiDB plan, keys, RU, latency, resource-control queueing, or the complete crash/transfer topology. The existing business integration suite and Test2 remain the required proof for those facts. - Next evidence: adversarially review exported API consumers, observation lifecycle races, ownership/freshness behavior, and all remaining aggregate references before any correction; then run complete package and race suites. ## 2026-08-16 - Audit Outbox strict lifecycle review - Run: adversarial review of the post-split controller, observer lock and cancellation lifecycle, assembly ownership inputs, legacy API call graph, exported snapshot consumers, and every remaining `ObserveQueue` reference. - Finding: the production boundary remains correct. The business worker enters the unchanged fenced `AcquireLease` directly; the exact aggregate has one read-only Runtime Worker caller and cannot authorize, suppress, or mutate work. No current-tree or organization consumer requires the removed durable fields on `OutboxWorkerSnapshot`. - Evidence gap: focused tests prove fail-open errors and explicit source close, but do not yet prove that a sampler-owned timeout cancels a blocking query or that a business-controller return stops the loop and releases the lazily opened independent source. - Next action: add only those two lifecycle tests, then run focused package, race, vet, complete backend/shared, boundary, and diff verification. Change production code only if either test falsifies the reviewed lifecycle. ## 2026-08-16 - Audit Outbox focused lifecycle verification - Run: complete `nexuscrm-audit` tests; focused Runtime Worker Audit tests; race detection for both surfaces; `go vet` for both surfaces; and whitespace validation on the implementation with the two lifecycle tests added. - Result: all checks passed. A context-aware blocking observer was canceled by the sampler-owned deadline, recorded one diagnostic failure, and stayed stale; an already-open observation source was released exactly once when the business controller returned its own error. Race detection found no data race. The macOS linker emitted its known malformed `LC_DYSYMTAB` warning but the race binary linked and the tests passed. - Learning: the diagnostic timeout and cleanup boundary is demonstrated without a production correction. Observation failure/latency remains fail-open for business processing, while shutdown waits for the context-aware database operation to unwind before closing its dedicated pool. - Next evidence: run complete backend and shared suites, public-core boundary policy, current-tree consumer searches, and final diff review; then refresh against `origin/main` before publication. ## 2026-08-16 - Audit Outbox complete local verification - Run: complete backend and shared tests, final current-tree caller/field searches, `git diff --check`, and the public-core boundary scanner. - Result: all backend and shared packages passed. The caller audit leaves one exact Audit Outbox aggregate implementation and one narrow Runtime Worker invocation; the business worker has none. `OutboxWorkerSnapshot` has no production consumer outside the new wrapper. Diff validation passed. - Boundary result: the scanner still reports its existing generic-code and exempt test-fixture candidates, so its repository-wide exit remains nonzero; none is in a changed file and this slice adds no tenant, CRM, object, field, profile, record-type, or flow payload. - Scope result: frontend code and contracts are untouched, so a frontend build would add no decision-relevant evidence and was not run. - Next action: fetch and compare the latest `origin/main`, rebase if it advanced, rerun the affected proofs on the final candidate, then commit and publish the PR. Live Test2 plan/RU remains post-merge evidence rather than a local claim. ## 2026-08-16 - File-upload lifecycle authority and physical-read baseline (#4430) - Run: current-tree producer, journal, foreground reconciliation, Runtime Worker, object-store authority, poison/repair, Controlplane receipt cleanup, Sentry consumer, canonical schema, and organization-wide exported-consumer audit on `e494dda6494794eaae3cf661ec7f950ee1333fed`. - Business semantics: `_System_FileUploadOperation` is the durable authority across database and object storage, not a metrics queue. Expired `PREPARED` work uses two fenced deletes separated by a quiet period to catch a late object-store commit. `BLOB_STORED` work either reconciles exact metadata and content, deletes an exact orphan, or parks as repairable poison. Claim leases, storage mapping digests, ownership transfer, and exact repair CAS must remain unchanged. - Poll-path problem: `RuntimeUploadLifecycle.Observe` executes an unfiltered `SUM(CASE)` / `MIN(CASE)` aggregate over every journal row at startup and before every Worker pass. Its zero result is a negative admission hint; queue, poison, and age fields are diagnostics. A read error already fails open into the fenced `ProcessExpired` authority. - Physical shape: the canonical table has only the unique idempotency index and `(ExpiresAt)`. The latter cannot exclude retained `COMPLETED` receipts before evaluating state. Controlplane deletes expired completed receipts during apply/repair, not continuously, so steady-state cost must not assume a bounded terminal population. The deprecated unfenced prepared cleanup has no organization production consumer; the Sentry endpoint still recomputes separate state totals every ten minutes and remains a later cutover owner. - Root design: install an additive `(State, ExpiresAt, Id)` lifecycle index before changing Worker reads. It supports the surviving due hint, fenced candidate claim, and completed-receipt cleanup while preserving the generic `(ExpiresAt)` index for the current cross-state ops metric. This schema-first slice changes no producer, state transition, cleanup, object I/O, poison, repair, observation, Worker readiness, or runtime behavior. - Evidence boundary: no live TiDB environment is attached to this workspace, so index selection, keys, RU, latency, online-DDL cost, and resource-control queueing remain Test2 facts. The warm reconciler must add the exact index idempotently, reject an incompatible same-name shape, and converge both greenfield and retained legacy tables before the Worker cutover PR. - Next action: implement and verify only the canonical additive index and warm schema reconciliation, publish it independently, then implement bounded admission plus isolated hourly diagnostics after the schema has landed. ## 2026-08-16 - File-upload lifecycle index unit loop - Run: added the canonical `(State, ExpiresAt, Id)` index, one narrowly owned warm reconciler, catalog shape coverage, and missing/exact/incompatible physical-index tests. Renamed the existing connector-specific physical index inspector to the shared managed-schema name without changing its SQL or validation semantics. - Result: complete Persistence tests, Kernel schemacatalog tests, and diff validation passed. The reconciler adds a missing non-unique index, verifies its exact post-DDL shape, performs no DDL on the next pass, and rejects a same-name index whose column order is incompatible. - Business boundary: `VerifyFileUploadOperationSchema` intentionally does not require the new index in this schema-first release. Existing Runtime Workers therefore keep byte-identical readiness and cleanup behavior while schema deployment converges the additive physical prerequisite. - Next evidence: make the real TiDB catalog integration start from the exact legacy table shape, prove two canonical warm passes add and retain the index, then run race/vet and the broader schema consumers before publication. ## 2026-08-16 - File-upload lifecycle warm-upgrade integration loop - Run: changed the existing real-TiDB canonical FileUploadOperation test to create the retained legacy definition without the lifecycle index, assert its absence, apply the canonical definition twice, and verify the exact final non-unique `(State, ExpiresAt, Id)` shape. - Result: complete Persistence and Kernel schemacatalog suites plus diff validation passed. This workspace has no declared TiDB endpoint, so the integration case compiled and remained environment-gated rather than being misreported as a live DDL execution. - Learning: greenfield catalog ownership and warm-table convergence now share the same canonical definition and migration chokepoint. No runtime readiness or mutation path reads the new index as an authority in this release. - Next evidence: run race and vet on Persistence/Kernel, repository-standard backend/shared tests, boundary and current-tree schema/caller audits, then refresh `origin/main` before publishing the schema-first PR. ## 2026-08-16 - File-upload lifecycle schema race and vet loop - Run: complete Persistence race/vet and Kernel schemacatalog race/vet after the warm-upgrade integration change. - Result: all checks passed. The Kernel race suite completed in approximately three minutes; a cached confirmation then returned the final successful package result. No race, vet diagnostic, or schema-test regression was found. - Interpretation: the additive catalog declaration, generic managed-index inspector rename, reconciler, and integration harness are internally consistent. This still does not prove live TiDB optimizer selection or DDL resource cost. - Next evidence: run complete Kernel, backend, and shared suites, scan the public-core boundary, inspect every changed schema consumer and final diff, then compare with the latest `origin/main`. ## 2026-08-16 - File-upload lifecycle schema repository verification - Run: complete Kernel, backend, and shared tests; public-core boundary scan; changed-file and schema-call-path audit; and final diff validation. - Result: all test suites and diff validation passed. The boundary scanner reports only its existing generic-code and exempt fixture candidates; no hit is in this change. The new index is reachable from the one canonical `CreatePhysicalTable` reconciliation chokepoint and no Runtime business path references it yet. - Strict-review finding: production behavior is correct, but the shared managed index inspector and dispatch function still physically live in the connector-specific migration file, and the new test borrows a connector-named row helper. That ownership would make the additive Files migration unnecessarily depend on an unrelated feature's test/filename. - Next correction: move only the generic dispatcher/inspection/equality helpers to a neutral managed-physical-index file and make the Files test self-owned; then repeat the focused schema suites before the final current-main audit. ## 2026-08-16 - File-upload lifecycle neutral-owner test loop - Run: moved the managed-schema dispatcher and physical-index inspector out of the connector-specific migration file, then reran the complete Persistence suite. - Result: production code compiled, while the new Files unit test failed after DDL because it returned the inspector's four-column row shape to the existing exact-index verifier, whose query deliberately projects three columns. - Learning: the two checks have different contracts: pre-DDL inspection needs `SUB_PART` to reject prefix indexes, while post-DDL exact verification filters prefixes in SQL and scans only uniqueness, position, and column name. The fixture must model each real query instead of sharing an approximate row. - Next correction: add a Files-owned three-column exact-index fixture for the post-DDL verification and rerun the focused and repository-level proof. ## 2026-08-16 - File-upload lifecycle schema-first release candidate - Run: corrected the exact-verifier fixture, reran complete Persistence tests, race, and vet; Kernel catalog tests, race, and vet; diff validation; and a final `origin/main` comparison. - Result: every check passed and `origin/main` remains `e494dda6494794eaae3cf661ec7f950ee1333fed`, the branch base. The release candidate contains only the additive catalog index, warm-schema convergence, exact shape tests, and this evidence record; it changes no lifecycle state or Runtime read path. - Evidence boundary: the real-TiDB warm-upgrade case remains environment-gated in this workspace. Test2 must still measure online-DDL cost and prove the optimizer selects `(State, ExpiresAt, Id)` before the later Worker cutover. - Next action: publish this schema-first PR independently. After it lands, change Worker admission to a bounded due-row hint and move full diagnostic aggregation to an isolated hourly observation path. ## 2026-08-16 - File-upload lifecycle Worker cutover baseline (#4430) - Run: refreshed `origin/main` after #4443 and reconstructed the current Files lifecycle, Runtime Worker, independent observation-pool, metrics, schema readiness, repair, and real-TiDB integration paths at `9c36519ed01ef889c22059f917452987c046af64`. - Authority: `ProcessExpired` remains the only business path. Its ownership lock, bounded ordered claim, per-row CAS, two-pass PREPARED deletion, object authority validation, poison, retry, and repair semantics must not change. Observation is diagnostic only; an admission-read error must fall through to the existing fenced claim so a monitoring failure cannot strand work. - Baseline problem: every startup and poll pass currently calls the unfiltered exact `SUM(CASE)` / `MIN(CASE)` observation before deciding whether to claim. The query accounts for every journal row, including retained terminal history, and shares the business pool. No live TiDB endpoint is attached to this workspace, so baseline RU, optimizer plan, keys examined, and RC queue time remain Test2 evidence rather than inferred measurements. - Cutover design: require the landed exact `(State, ExpiresAt, Id)` index; use only an index-supported `EXISTS (... LIMIT 1)` due-work hint at poll cadence; retain the exact aggregate under a Files-owned DB-only observer; run it after the first fixed one-hour interval on the existing one-connection diagnostic pool with a 30-second timeout; retain the last good snapshot and mark it stale on failure. Prometheus remains memory-only. - Next evidence: implement the split without changing the claim query or any state transition, then prove zero-hint suppression, error fall-through, no-startup observation, pool isolation, timeout/failure freshness, source reuse/release, and existing restart/ownership/repair tests. ## 2026-08-16 - File-upload lifecycle cutover compile loop - Run: implemented the first production cutover slice and ran Files, Runtime Worker, and Persistence tests before updating test contracts. - Result: the complete Files suite passed. Runtime Worker failed to compile only because existing tests still used the old constructor; Persistence schema tests reached the newly required lifecycle-index check and lacked its expected inspection row. No lifecycle behavior test failed. - Learning: the production boundary compiles in the Files owner, while the remaining work is to make tests explicitly model the new two-read split: bounded business hint versus independently scheduled exact observation. - Next correction: update readiness expectations, add a direct index-hint SQL test, and replace Worker tests that treated exact diagnostics as admission with tests for fail-open admission and isolated observation freshness. ## 2026-08-16 - File-upload lifecycle readiness ownership review - Run: passed complete Files, Runtime Worker, and Persistence tests, then audited every caller of `VerifyFileUploadOperationSchema` and the final production diff instead of treating green tests as sufficient evidence. - Finding: adding the lifecycle performance index to the shared journal schema verifier also makes the foreground Runtime API and operator repair startup depend on it. Those paths neither execute the new hint nor need the index for correctness, so the broader readiness dependency could unnecessarily block business upload or repair during a partial rollout. - Owning boundary: the base journal verifier continues to own upload identity, expiry, file-authority, and repair columns. A separate lifecycle-index verifier belongs to Persistence physical shape and is invoked only by `RuntimeUploadLifecycle.CheckReady`, the consumer that issues the indexed high-frequency hint. - Next correction: restore the shared verifier's old contract, add the narrow lifecycle-index check, prove missing/wrong/exact shapes, and rerun the focused suites before reviewing query and shutdown behavior. ## 2026-08-16 - File-upload lifecycle isolated-observation verification - Run: narrowed readiness to the lifecycle consumer; ran complete Files, Runtime Worker, and Persistence suites; ran race and vet across all three; ran complete backend and shared suites; and audited every constructor, observer, and schema-verifier caller. - Result: all checks passed. The base foreground upload and repair readiness path is unchanged. Lifecycle readiness rejects missing, unique, or reordered indexes. Poll passes execute only `HasExpired`; exact observation starts stale, waits for the first interval, reuses one independent connection, retains the last good snapshot on failure, and cannot block a business pass. - State-shape evidence: ordinary FAILED claim/quiet rows always have a finite lease or quiet expiry, while durable repair poison clears `ExpiresAt` and is excluded from the hint. Repair restores an immediately due supported state. Thus the `(State, ExpiresAt, Id)` ranges exclude terminal COMPLETED history and parked poison before the residual ErrorCode check. - Residual evidence boundary: the workspace still has no live TiDB endpoint. The explicit `USE INDEX` plus `LIMIT 1` proves the intended physical path in code, not its measured RU or keys examined; Test2 must capture the actual plan and quiet-window before/after telemetry. - Next evidence: run public-boundary and final diff review, compare the branch with the latest `origin/main`, then publish only if no caller or lifecycle invariant was broadened. ## 2026-08-16 - File-upload lifecycle final acceptance audit - Run: public-core boundary scan, latest-main comparison, changed-file audit, and acceptance-criterion review after all broad tests passed. - Result: `origin/main` remains the branch base. The boundary scan's nonzero result contains only existing test fixtures and unrelated generic-code candidates; no changed file is reported. No high-frequency exact lifecycle aggregate remains in Runtime Worker admission or startup. - Evidence gap: ordinary observation failure and business non-blocking behavior are proven, but the test suite does not yet distinguish sampler-owned timeout from parent shutdown. The production code has the correct timeout context, yet the acceptance claim needs a direct duration/failure/stale assertion. - Next correction: add that one timeout test, rerun the focused race suite and final repository checks, then freeze the release candidate. ## 2026-08-16 - File-upload lifecycle Worker release candidate - Run: added the sampler-owned timeout assertion and reran Runtime Worker unit and race tests, complete Files and Persistence suites, and diff validation. - Result: all checks passed. A timed-out exact observation records duration and failure, remains stale, and releases cleanly; the concurrent business pass test independently proves that the same blocked observer cannot consume or lock the business cleanup path. - Release claim: this candidate removes the unfiltered aggregate from startup and poll admission, retains one hourly exact diagnostic owner on a fixed one-connection pool, keeps Prometheus memory-only, and leaves every durable claim/effect/repair transition byte-for-byte unchanged. - Rollout evidence: after merge, Test2 must repeat the file-upload focused case twice and capture execution count, read RU, TiDB latency, RC queue time, keys examined/processed, and selected plan for the old aggregate digest and new hint. This local candidate does not fabricate those environment facts. ## 2026-08-16 - File-upload lifecycle live-fixture review correction - Run: investigated both the standard and adversarial #4444 review blockers against the exact live-TiDB Files fixture and lifecycle readiness call. - Finding: both comments identify the same concrete gap. The shared handwritten `_System_FileUploadOperation` integration fixture still declares only the legacy expiry index, so the first lifecycle integration test would fail the newly narrow `VerifyFileUploadLifecycleIndex` readiness check when TiDB is enabled. Local suites skipped that environment-gated path. - Root correction: update the one shared Files fixture to the already-landed canonical `(State, ExpiresAt, Id)` index shape. Do not weaken readiness or change production behavior; the failing `CheckReady` remains the regression assertion that protects the physical prerequisite. - Evidence boundary: no live TiDB endpoint is attached locally. After the fixture correction, compile and all local Files/Worker/Persistence suites can be proven here; the environment workflow owns the actual TiDB DDL execution. ## 2026-08-16 - File-upload lifecycle live-fixture correction verification - Run: added the exact non-unique lifecycle index to the shared live-TiDB Files fixture; compared its ordered columns with the canonical Kernel catalog; ran complete Files, Runtime Worker, and Persistence tests plus diff validation. - Result: every local check passed. The fixture now supplies both the retained `(ExpiresAt)` index and canonical `(State, ExpiresAt, Id)` index, so its existing lifecycle `CheckReady` call validates the same physical contract as production rather than failing before the state-machine journey begins. - Scope: test fixture and evidence only. No production SQL, readiness rule, Worker behavior, metrics cadence, claim, disposition, or repair path changed. - Remaining proof: the GitHub environment must rerun the live-TiDB-enabled integration/review gate on the pushed commit; local absence of TiDB remains explicit rather than being represented as a passing live execution. ## 2026-08-16 - Runtime Platform Command baseline and authority audit (#4430) - Run: current-tree producer, acceptance, execution, dependency/FIFO, retry, poison, repair, ownership, schema, Runtime Worker, observation, and metrics audit on `d4a41c9cc7506e4b264165260794fdf9254c06b1`. - Environment/workload: local source and SQL-contract inspection of the independently owned `runtime.platform-command.v1` Worker. No live TiDB endpoint or tenant workload is attached to this workspace, so statement frequency, plan, keys, RU, latency, and resource-control queueing remain Test2 facts rather than inferred measurements. - Business authority: acceptance persists immutable scope and metadata binding. `Claim` locks the active deployment owner, reads one due ACCEPTED, expired PROCESSING, or due BLOCKED row, preserves PROCESSING-before-BLOCKED- before-ACCEPTED ordering, and transactionally establishes the lease. Effect completion, dependency blocking/expiry, retry, poison, exact replay, and compare-and-set repair remain independently fenced durable transitions. - Baseline problem: `platformCommandProcessor` has no bounded work-hint implementation, so `metadataWorker.hasWork` calls the exact scoped queue/poison/oldest aggregate at startup and every poll before the real claim. The aggregate evaluates every retained row in the execution scope, including COMPLETED and FAILED history, and uses the same pool as business ownership, claim, effect, and disposition work. - Physical/index result: `RuntimeWorkScope.Digest` is the canonical bounded identity of the complete `(OrgId, AppInstallationId, RuntimeCapabilityId)` tuple. The verified surviving claim index begins with `(ExecutionScopeDigest, State, AvailableAt, LeaseExpiresAt, Id)`, which can support a scope/state/due `EXISTS (... LIMIT 1)` for ACCEPTED and BLOCKED; PROCESSING cardinality is bounded by the singleton Worker lease path but its lease-time predicate remains a Test2 plan/key measurement. No schema change is justified before that evidence. - Root design: add a negative-only `HasWork` query using the exact claim eligibility and scope predicates on the primary connection. A successful negative skips only one bounded poll; every positive and every hint error is rechecked by the unchanged fenced `Claim`. Move the exact active/poison/age aggregate behind a narrow ownership-aware observer, filter out unrelated terminal history, and run it only after the first fixed one-hour interval on the existing lazy independent one-connection pool. The generic metadata cache retains the last good snapshot and Prometheus remains DB-free. - Expected result: the unbounded aggregate disappears from startup and poll cadence. The replacement hot read returns after the first eligible indexed candidate, while diagnostics cannot consume business pool capacity or authorize, suppress, claim, retry, repair, or execute work. - Next evidence: implement the read-path split and prove hint false/true/error behavior, no startup observation, fixed pool lifecycle, timeout/failure freshness, business non-blocking, ownership-aware observation, unchanged claim SQL/state transitions, and complete package/race/repository checks. ## 2026-08-16 - Runtime Platform Command focused implementation loop - Run: focused Persistence SQL-contract tests and Runtime Worker Platform Command admission/observation tests on the uncommitted `codex/platform-command-observation` worktree. - Result: the bounded scope/state/due hint passed true, false, and query-error cases. Ownership-aware exact observation excludes COMPLETED/FAILED history; ownership loss stops it before the aggregate. Runtime tests prove a negative hint opens neither Claim nor observation, a hint error falls through to the original Claim, startup opens no observation source, one source is reused and closed, and a blocked diagnostic query cannot block a business Claim pass. - First failure: Persistence tests passed, while Runtime Worker `go test` vet rejected a test failure message that formatted a function value. The production code reached no failing assertion. Replacing that diagnostic with a boolean `release_set` field made the same focused suite pass. - Current boundary: production Claim SQL, owner lock, ordering, lease, dependency, effect, retry, poison, and repair methods are unchanged. The old exact aggregate is no longer implemented by the business execution store; its only production owner is the narrow independent-pool observation store. - Remaining uncertainty: local SQL mocks prove statement shape, not TiDB range selection or the PROCESSING lease-range key count under retained backlog. Test2 must collect the selected plan, keys, RU, latency, and RC queue time. - Next evidence: adversarially inspect the complete diff, constructor and shutdown call graph, startup-read classification, claim/hint predicate equivalence, and all exact-aggregate references before expanding to complete package and race suites. ## 2026-08-16 - Runtime Platform Command strict lifecycle review - Run: adversarial review of every production diff hunk, the serial metadata Worker startup/poll path, independent source lifecycle, scope-digest and index contracts, Claim/hint predicates, ownership transfer, and remaining exact-aggregate references. - Result: the hint and Claim select the same due ACCEPTED, expired PROCESSING, and due BLOCKED states in the same complete Runtime scope. A hint result is never cached and cannot order or claim a row. The only aggregate caller is the ownership-aware observation source; the business execution interface no longer exposes observation. Source cancellation completes before its fixed pool is closed. - Startup classification: `CheckReady` still performs durable-authority and dependency validation reads. Those are correctness/readiness gates, not the removed diagnostic aggregate, and weakening them would change the business contract. The serial startup pass now calls only the bounded hint before the unchanged Claim. - Schema disposition: the exact complete-scope digest and existing verified claim index are already the owning physical contract. No additive index, migration, readiness broadening, or L2 configuration is required in this slice. The live optimizer result remains a measurement, not a code-review assumption. - Evidence gap: the blocked-observer test proves business independence during a query, but cancellation currently comes from explicit loop shutdown. It does not directly prove that a sampler-owned timeout increments diagnostic failure, records duration, and leaves the cached snapshot stale. - Next action: add that one timeout/freshness regression test, then run complete Persistence and Runtime Worker suites plus the focused race detector. Change production code only if the new test falsifies the reviewed lifecycle. ## 2026-08-16 - Runtime Platform Command package and race verification - Run: the new sampler-owned timeout test, complete Persistence and Runtime Worker package suites, and focused Platform Command admission/observation tests under the race detector. - Result: every suite passed. The timeout test records one observation failure, positive duration, and stale freshness after the observation context expires. Complete Persistence and Runtime Worker packages passed, and the focused race suites found no data race in lazy source open/reuse/close or concurrent business admission. - Tooling note: the macOS race link repeated the repository's known malformed `LC_DYSYMTAB` warning and still linked and returned success. - Learning: no production correction was required after the strict review. Sampler timeout and worker shutdown both unwind the context-aware observation before closing the independent pool, while primary-pool HasWork/Claim remains available concurrently. - Next evidence: run repository-standard backend/shared suites, vet the changed package trees, public-core boundary and diff checks, then refresh current `origin/main` and inspect the final candidate for overlap or stale callers. ## 2026-08-16 - Runtime Platform Command repository regression verification - Run: repository-standard complete backend and shared tests, Persistence and Runtime Worker vet, public-core boundary scan, and diff validation. - Harness correction: the first parallel wrapper had an invalid nested shell conditional, so none of its four checks started. After correcting only the wrapper, the same commands ran with their real exit statuses preserved. - Result: complete backend and shared suites passed; both vet targets and `git diff --check` passed. The boundary scanner retained its known nonzero repository baseline in unrelated generic code and exempt test fixtures; no changed Platform Command, assembly, test, or performance-record file appears in its output. - Scope result: no frontend, API wire, contract, L1 pack, L2 payload, schema, migration, or generated version file changed. A frontend build would add no decision-relevant evidence and was not run. - Next evidence: refresh `origin/main`, compare ancestry and changed paths, inspect every final diff hunk and exact aggregate/admission caller, then commit and publish only if the latest-main candidate remains conflict-free. ## 2026-08-16 - Runtime Platform Command final publication review - Run: fetched `origin/main`, compared ancestry and changed paths, reviewed the complete staged implementation and tests, and re-enumerated Platform Command exact-observation and high-frequency admission callers. - Result: `origin/main` remains `d4a41c9cc7506e4b264165260794fdf9254c06b1`, the direct branch base, with no newly landed overlap. The exact `SUM/CASE/MIN` statement has one production caller through `RuntimePlatformCommandObservationStore`; poll admission has one production path through `RuntimePlatformCommandExecutionStore.HasWork`. - Final local claim: no exact Platform Command diagnostic aggregate runs at startup or poll cadence, observation uses only the lazy fixed one-connection pool, and every successful positive or hint failure reaches the unchanged fenced Claim. No schema or business transition changed. - Remaining rollout evidence: Test2 must compare old and new statement counts, selected plan, examined/processed keys, RU, TiDB latency, and resource-control queue time. In particular, live evidence—not this local review—must establish the PROCESSING lease-range cost under retained backlog. - Publication disposition: ready for a focused PR linked to #4430. Connector live-queue measurement and Sentry cache ownership remain separate follow-ups. ## 2026-08-16 - Worker observability PR 1A baseline (#4447) - Run ID: `worker-observability-pr1a-static-baseline`; base commit: `57726e67fd6cb5f9c2c0fb8eaf40d70ed7dc17c0`; environment: local current-tree source audit without a live TiDB workload. - Workload shape: traced process assembly, readiness, Prometheus collection, Permission State gating, ownership verification, and Worker startup for both payload and metadata execution classes. - Profile result: `nexuscrm_runtime_process_ready` calls `Process.Ready()` while collecting `/metrics`. Runtime Worker supplies a readiness callback whose Permission State gate performs a strong database read, so every metrics scrape can create correctness-grade database work. Worker identity, startup stages, Permission State checks, and ownership checks also lack one bounded metric contract; loopback diagnostics are supported by the process substrate but Runtime Worker does not opt in. - Change hypothesis: keep `/readyz` as the active correctness probe, make `/metrics` consume only its in-memory observation, add bounded Worker identity/startup/authority metrics, and enable the existing diagnostics only on loopback admin listeners. Do not change queues, claims, fences, polling, ownership semantics, deployment topology, schema, or tenant configuration. - Evidence gap: this static trace does not quantify live RU, scrape rate, or startup latency. Focused tests must first prove that metrics collection cannot invoke the readiness callback and that every new metric is memory-only; rollout should then compare metrics-query statement count and RU with the pre-change baseline. - Next evidence: implement the owning-boundary readiness observation and focused regression tests before adding the Worker-level metric contract. ## 2026-08-16 - Memory-only process readiness metric proof - Run ID: `worker-observability-pr1a-readiness-unit`; candidate based on `57726e67fd6cb5f9c2c0fb8eaf40d70ed7dc17c0`; environment: local Go unit test. - Change: active `Process.Ready()` calls now publish their completed result to a concurrency-safe in-memory observation. The process collector reads only that observation plus local lifecycle/controller state and exports whether a check is known and when it completed. - Result: the complete `backend/internal/runtimeprocess` suite passed. The new regression scrapes `/metrics` before and after `/readyz`: the first scrape performs zero readiness callbacks, `/readyz` performs exactly one, and the second scrape retains exactly one while reporting the cached ready result and timestamp. - Learning: this removes the database feedback loop at the process owner rather than hiding it in Runtime Worker. Active readiness semantics remain unchanged; metrics explicitly distinguish “not ready” from “not checked yet.” - Next evidence: add bounded Worker identity, startup, Permission State, and ownership observations and prove that their collectors read memory only. ## 2026-08-16 - Worker identity, startup, and authority metric proof - Run ID: `worker-observability-pr1a-worker-unit`; candidate remains based on `57726e67fd6cb5f9c2c0fb8eaf40d70ed7dc17c0`; environment: local complete Runtime Process and Runtime Worker package tests. - Change: Worker assembly now records a bounded payload/metadata execution class, seven fixed startup stages, Permission State readiness checks, and periodic work-ownership checks. All collectors consume atomics, immutable identity, startup snapshots, queue snapshots, or `sql.DB.Stats`; loopback Worker admin listeners opt into the process substrate's on-demand pprof handlers without enabling continuous contention profiling. - Result: both package suites passed. Tests verify exact Worker identity labels, startup stage values, authority success/failure counters, preservation of Permission State results, ownership-loss accounting, and the integration surface contract for cached readiness and loopback diagnostics (the live TiDB integration body remains environment-gated). - Learning: the metric contract describes existing correctness activity instead of adding another checker. There are no tenant, org, deployment, digest, row, or arbitrary configuration labels, so cardinality is bounded by two execution classes, the configured work families, two authority kinds, and seven stages. - Next evidence: challenge concurrency, lifecycle, startup-stage ownership, nil assumptions, and metric compatibility under the race detector and complete repository call-site review before broad verification. ## 2026-08-16 - Worker observability race and compatibility review - Run ID: `worker-observability-pr1a-race-review`; environment: local macOS race detector over the complete Runtime Process and Runtime Worker packages. - Review result: no repository dashboard or alert directly consumes `nexuscrm_runtime_process_ready`; the existing integration contract probes `/readyz` before reading metrics. A readiness result produced before Start is now explicitly cleared at the ready-state transition because it did not run the role dependency check and must not make the current observation “known.” - Test result: both package suites passed under `-race`. The linker emitted the repository's known malformed `LC_DYSYMTAB` warning and still returned success. - Learning: sequence-ordered completion prevents a slower, older readiness call from replacing a newer result, but the current suite proves only race freedom, not that ordering invariant deterministically. - Next evidence: add one controlled overlapping-probe test, then run complete backend/shared verification and inspect the frozen diff against latest main. ## 2026-08-16 - Overlapping readiness probe ordering proof - Run ID: `worker-observability-pr1a-overlap-race`; environment: local macOS race detector over Runtime Process and Runtime Worker plus diff validation. - Result: a deterministic test holds the older readiness probe, completes a newer successful probe, then releases the older failed probe. The retained observation remains the newer success. Both package suites passed under `-race`, and `git diff --check` passed. - Learning: metrics can safely read the cached result during concurrent health traffic without database work, stale completion overwrite, or a data race. - Next evidence: repository-standard backend/shared suites, vet, public-core boundary scan, final current-tree consumer audit, and latest-main comparison. ## 2026-08-16 - Worker observability repository regression verification - Run ID: `worker-observability-pr1a-repository`; environment: local complete backend/shared suites plus changed-package vet and boundary/diff checks. - Result: all backend and shared tests passed, changed packages passed `go vet`, and `git diff --check` passed. The boundary scanner retained its known nonzero repository baseline in unrelated generic code and exempt fixtures; none of the changed Runtime Process, Runtime Worker, test, or learned-record paths appears in its output. - Scope result: no queue query, claim, execution, fence, schema, migration, metadata, API wire shape, frontend, L1 pack, L2 configuration, or generated file changed. A frontend build would not exercise this Go-only process boundary and was not run. - Cleanup finding: the cached observation type does not need to be exported, and `startup_complete` would always be one because the collector is inaccessible until successful assembly returns. Keeping either would add API/metric surface without new operational information. - Next evidence: remove those two redundant surfaces, rerun the focused and repository checks, then compare and rebase against latest `origin/main`. ## 2026-08-16 - Worker observability final local candidate - Run ID: `worker-observability-pr1a-final`; candidate base and latest `origin/main`: `57726e67fd6cb5f9c2c0fb8eaf40d70ed7dc17c0`. - Cleanup: the cached readiness type/method remain package-private, and the always-true startup-complete metric was removed. Startup total and fixed-stage durations retain the useful information without redundant surface. - Result: focused Runtime Process and Runtime Worker suites passed under the race detector; complete backend/shared suites, changed-package vet, and diff validation passed after cleanup. Latest `origin/main` is still the direct branch base, so there is no upstream overlap or rebase requirement. - Final local claim: `/metrics` cannot call the role readiness callback; `/readyz` retains its active strong validation; Worker metrics describe bounded identity, completed startup work, Permission State checks, ownership checks, existing queue snapshots, and pool memory only. Loopback diagnostics use the existing process substrate and do not enable block/mutex sampling. - Remaining rollout evidence: compare strong Permission State statement count and RU per metrics scrape before/after deployment, then use the new startup and authority series to evaluate topology under #4447. No topology change is part of this candidate. ## 2026-08-16 - Worker observability PR 1B current-tree baseline (#4447) - Run ID: `worker-observability-pr1b-static-baseline`; base commit: `cf7a7ba9beeeb54f69b63067fd36dc38b6fa3304`; environment: local current-tree source and consumer audit without a live TiDB workload. - Workload shape: traced all seven Worker controllers, admission hints, fenced claims/effects, hourly observation sources, scheduled-Flow discovery/recovery, rejected-save audit pools, and transient object-ID reprobes. - Profile result: most roles expose attempts/processed/failures, but there is no common count or duration for admission checks, confirmed idle checks, hint failures, claimed rows, and empty authoritative attempts. Scheduled Flow still exports an empty Worker snapshot even though it owns discovery, durable acceptance, recovery, execution, and capacity deferral. Primary pool stats are visible, while the lazy hourly observation pool, mutation graph rejected-save pool, and transient object-ID pool openings are not. - Ownership finding: Runtime Worker owns loop coordination and observation-pool construction; `automation.SchedulerService` owns scheduled execution facts; `recordmutation.RecordsMutationRuntime` owns rejected-save and object-ID probe pool facts. Metrics must project those existing owners rather than create a second checker or infer state from durable tables. - Change hypothesis: add process-local counters at those three owning boundaries, compose their immutable snapshots into the existing Worker collector, and close mutation auxiliary pools with the Worker controller. Do not add SQL, queue aggregation, new polling, or change claim/effect/fence outcomes. - Next evidence: implement the common loop observation and exact auxiliary-pool lifecycle first, prove negative hints and failures still retain their current authority behavior, then instrument Scheduler-native facts. ## 2026-08-16 - PR 1B owning-boundary implementation compile proof - Run ID: `worker-observability-pr1b-owning-boundaries`; candidate based on `cf7a7ba9beeeb54f69b63067fd36dc38b6fa3304`; environment: local complete Automation plus focused Runtime assembly/Worker package suites. - Change: Worker coordinators now retain common in-memory poll/idle/hint/claim counts and duration; Audit records the same facts at its own lease boundary. Queue-observation pools have explicit open/failure/active/close state; mutation graphs expose rejected-save pool state and transient object-ID open cost. Scheduler owns discovery, acceptance, recovery, execution, capacity, and phase-duration counters, which the scheduled Worker projects into the common and role-native metric vocabulary. - Lifecycle correction: metadata Worker failure and controller-exit paths now close the rejected-save pool owned by their mutation graph. Runtime execution ordering is unchanged; cleanup occurs only after the controller and its internal workers have joined. - Result: all Automation tests and focused Runtime assembly/Worker tests pass. No durable-table read, new ticker, queue aggregate, claim branch, or effect branch was added. - Evidence gap: existing tests compile the new observations but do not yet prove exact counter semantics, pool close state, Scheduler capacity retention, or metric exposition. - Next evidence: add focused value/lifecycle tests at each owner and run them under the race detector before reviewing the full diff. ## 2026-08-16 - PR 1B metric-semantics review - Run ID: `worker-observability-pr1b-semantics`; candidate based on `cf7a7ba9beeeb54f69b63067fd36dc38b6fa3304`; environment: static owner-to- metric review after the first compile proof. - Finding: Scheduler terminal completion includes both successful and failed executions, so projecting it directly as the common `processed` value would contradict that metric's successful-finalization contract. Unsigned `started - completed` also relied on an unstated ordering invariant. - Correction: common processed is now terminal completion minus terminal failure, and in-flight subtraction is guarded. The Scheduler-native completed and failed counters remain lossless, so operators can see both the aggregate lifecycle and its outcome split without a second source of truth. - Next evidence: prove exact idle/fail-open/claim values, auxiliary-pool close idempotence, Scheduler capacity/execution values, and Prometheus exposition with focused tests and the race detector. ## 2026-08-16 - PR 1B focused-test execution-path finding - Run ID: `worker-observability-pr1b-focused-values-1`; candidate based on `cf7a7ba9beeeb54f69b63067fd36dc38b6fa3304`; environment: focused Backend and Automation unit tests. - Result: idle, hint-failure fallback, common metric exposition, queue- observation pool open/close, and rejected-save/object-probe lifecycle tests pass. The existing Scheduler compatibility test falsified the initial claim that execution counters covered every path: fenced executions were observed, but the retained reservation-store path was not. - Owning-boundary correction: observe legacy execution inside `executeReservedScheduledVersion`, including its terminal reservation write. Count completion only when that durable terminal write succeeds; a failed completion remains in-flight/uncertain rather than being reported as a terminal success or failure. - Next evidence: rerun both Scheduler paths, then add an explicit projection test for successful, failed, and uncertain terminal arithmetic. ## 2026-08-16 - PR 1B Scheduler compatibility acceptance finding - Run ID: `worker-observability-pr1b-scheduler-compat-2`; candidate based on `cf7a7ba9beeeb54f69b63067fd36dc38b6fa3304`; environment: focused Automation compatibility and terminal-outcome tests. - Result: legacy execution start/completion is now correct, including leaving a failed terminal write nonterminal in the metrics. The compatibility journey then exposed the same omission one stage earlier: legacy per-input `Reserve` was not included in acceptance attempts or accepted occurrences. - Owning-boundary correction: record duration/attempt at the existing Reserve call and increment accepted only for a newly reserved durable identity, never for a duplicate. This matches fenced `ClaimBatch` semantics without altering reservation, duplicate, or execution behavior. - Next evidence: rerun the exact compatibility journey and the complete Scheduler suite, then inspect recovery and panic terminal semantics. ## 2026-08-16 - PR 1B common-contract semantic audit - Run ID: `worker-observability-pr1b-common-contract-audit`; candidate based on `cf7a7ba9beeeb54f69b63067fd36dc38b6fa3304`; environment: full diff and owner- vocabulary review after focused tests passed. - Finding: Scheduler acceptance attempts were lossless, but successful duplicate or zero-admission attempts were not projected as common empty claims. Also, Scheduler's role-native discovery-failure count includes incomplete scans such as capacity deferral, so presenting it as an advisory hint failure would give one fact two incompatible meanings. - Correction: retain Scheduler discovery failures only in its role-native metric, leave common hint failures zero because Scheduler has no advisory hint, and add an owner-native empty-acceptance count for successful attempts that admit no new durable run. - Further owner audit: fenced acceptance must count every newly claimed durable lease, even when its canonical payload cannot be decoded for immediate execution. Counting only the decoded in-memory slice would hide the exact durable row operators need to recover or repair. - Next evidence: assert duplicate/empty acceptance projection and rerun focused race tests before the repository-wide candidate. ## 2026-08-16 - PR 1B focused race proof and cleanup target - Run ID: `worker-observability-pr1b-focused-race`; candidate based on `cf7a7ba9beeeb54f69b63067fd36dc38b6fa3304`; environment: focused changed- owner race suites on macOS/arm64. - Result: Backend Runtime Worker/record-mutation and Audit race suites pass. Scheduler compatibility, fenced acceptance, capacity retention, empty/ duplicate acceptance, and durable terminal-outcome tests pass under the race detector. The complete Automation race suite still exposes a pre-existing race in `FlowRuntimeContext.RememberRecordLookupBaselines`; no changed file or Scheduler path appears in that report. - Cleanup finding: six observation factories repeat the same variadic optional- tracker extraction solely to keep focused test constructors source-compatible. Consolidate that mechanical selection in the auxiliary-pool owner before the full normal test candidate; this removes duplicated lifecycle glue without changing a factory contract. - Next evidence: perform that cleanup, run complete normal module/repository tests, vet, boundary/diff checks, and compare with latest `origin/main`. ## 2026-08-16 - PR 1B repository-boundary failure - Run ID: `worker-observability-pr1b-full-normal-1`; candidate based on `cf7a7ba9beeeb54f69b63067fd36dc38b6fa3304`; environment: complete Backend, Shared, Automation, and Audit normal test suites. - Result: Shared, Automation, Audit, and every Backend functional package pass. Backend's architecture gate correctly rejected one new import: `runtimeworker/auxiliary_pools.go` directly imported the metadata-bound record-mutation composition outside an explicit assembly file. - Root cause: the generic pool/metric carrier was made aware of a concrete mutation graph to obtain its snapshots. That is a convenience dependency, not the owning boundary. - Correction: keep the concrete graph import and snapshot projection in `assemble.go`, which already owns composition; pass zero-argument snapshot readers into the generic database-observation carrier. No new registry or second source is needed. - Next evidence: rerun the architecture gate first, then the complete Backend suite and changed-package vet. ## 2026-08-16 - PR 1B fenced terminal-state audit - Run ID: `worker-observability-pr1b-terminal-owner-audit`; candidate based on `cf7a7ba9beeeb54f69b63067fd36dc38b6fa3304`; environment: post-green static review of Scheduler's fenced transaction and separate failure-disposition path. - Finding: the first wrapper could distinguish lease/cancellation/uncertain transaction errors, but could not distinguish a failed separate terminal `Complete` write when an earlier execution error was retained. It could therefore overstate terminal failed executions and understate in-flight recoverable work. - Root fix: observe fenced execution inside `executeFencedScheduledVersion`, the only owner that knows whether the effect transaction committed a success or the separate failed disposition committed. Preparation failures do not start execution; failed terminal writes do not complete it. - Next evidence: add the failed-disposition case to the focused Scheduler race proof, then rerun normal Automation and Backend suites. ## 2026-08-16 - PR 1B final local candidate - Run ID: `worker-observability-pr1b-final`; candidate and latest `origin/main`: `cf7a7ba9beeeb54f69b63067fd36dc38b6fa3304` before the candidate commit; environment: macOS/arm64 local workspace. - Correctness proof: complete Backend, Shared, Automation, and Audit normal suites pass; focused Backend, Audit, and Scheduler paths pass under the race detector; changed Backend/Automation/Audit packages pass `go vet`; the Runtime Worker architecture boundary test and `git diff --check` pass. - Boundary result: the public-core scanner retains its known nonzero baseline in unchanged generic code and test fixtures; no changed path appears in the scan. The temporary concrete mutation-graph import caught by the architecture test was removed from the generic pool owner and is now confined to explicit assembly. - Known unrelated evidence: the complete Automation race suite reports the pre-existing `FlowRuntimeContext.RememberRecordLookupBaselines` race in `executor_record_lookup_batcher_test.go`; focused changed Scheduler race tests pass and none of this candidate's files appears in that report. - Final claim: every new metric is derived from process-local counters or `sql.DBStats`; no SQL, ticker, queue scan, claim/fence decision, durable schema, or business outcome was added. Scheduled terminal counters advance only after the owning durable terminal boundary confirms completion, and auxiliary pools are explicitly accounted and reclaimed. - Rollout evidence: compare per-role poll/empty-claim ratios, phase duration, capacity deferrals, observation-pool waits, and transient opens before any topology consolidation under #4447. This candidate changes observability, not the seven-Worker deployment shape. ## 2026-09-02 - Audit worker stalled network wait: bounded driver I/O - Run: `audit-worker-network-timeout-baseline`; core candidate base `8c60a50780ff59a8667ff4efaa0dc33455338d96`; local macOS/arm64, mysql-tracing over go-sql-driver/mysql v1.7.1. A controlled TCP peer withheld replies from the real driver. Query cancellation returned in 151 ms, but COMMIT continued past its transaction context deadline until the peer replied. A native 150 ms read timeout returned an invalid-connection error in 151 ms. - Production evidence establishes delayed audit materialization and an invalid-connection claim failure; the initiating network/database fault is not established. A later candidate completed 24 original-case replays with 1–2 second materialization and millisecond audit SQL. Deployment-specific evidence stays in the delivery repository's performance record. - Current-tree ownership: runtimeworker assembles the dedicated audit pool; nexuscrm-db constructs its driver connections; sql.DB already discards bad connections; the audit store atomically consumes the outbox with its effects and preserves leases after uncertain commits. No custom reconnect or retry controller is needed. Other worker families have different operation budgets. - Change selected: expose one optional per-pool network timeout, implemented by the driver's native dial/read/write settings. Apply 5 seconds to the audit production pool only. This bounds each network wait, including COMMIT, ROLLBACK and session reset; it is not a whole-transaction or queue-latency SLA. - Verification boundary: real-driver silent-peer query/commit/rollback and pool recovery; real TiDB audit effects and lease recovery for both uncommitted and committed-but-unacknowledged work; existing audit ownership guards and runtime assembly tests. No deployment or CI rerun until local proof. ### First local result: readiness query is a real blocker - Run: `audit-worker-network-timeout-local-1`, same candidate; local TiDB v8.5.0 at port 4001, existing instance with many schemas. Real-driver fault tests passed: query, COMMIT, ROLLBACK and session reset returned in 150–152 ms under a 150 ms network budget, then sql.DB used a new healthy connection. - Real worker assembly failed before fault injection because index readiness exceeded 5 seconds. A direct connection reproduced the same failure, ruling out the fault proxy. Do not relax the gate or hide this behind a larger timer. - EXPLAIN ANALYZE of the existing exact-index check on a fresh one-table scratch database: its anti-join scanned 129,525 rows of global STATISTICS, took 7.57 seconds, and used 178.5 MB at the scan. The outer target-index scan was already bounded. Reading SUB_PART alongside the same target-index rows took 194.2 microseconds total and scanned one row using 1.36 KB. - Owning path: persistence.VerifyExactTableIndex (and its transaction queryer variant) is shared by audit, permission-state, work ownership, and other Runtime store readiness checks. Replace only its redundant global anti-join with a direct target-index read; reject any non-NULL prefix length while checking uniqueness and column order. Keep a single metadata read path. - Scope: this is a measured startup bottleneck blocking the bounded pool. It does not establish the initiating cause of the historical running-worker connection failure. Verification adds real TiDB missing/prefix/ordering/ uniqueness cases before rerunning worker assembly and silent-COMMIT recovery. ### Final local result - Run: `audit-worker-network-timeout-local-final`, same base plus this worktree's changes; no image publication or deployment. The 11 real TiDB index cases pass, including prefixes in either column, absent indexes, uniqueness, column count/order and case-insensitive column names. - The full audit worker assembles with the unchanged 5-second network limit. Both real TiDB COMMIT fault windows pass: disconnect before COMMIT and a successful COMMIT whose acknowledgement is withheld. Each leaves exactly one effect per event, no stranded outbox row, and processes a second event without restarting the worker. The two-case batch takes 11.88 seconds, including two deliberate 5-second stalls. The existing ownership-transfer and lifecycle walking slice also passes (2.55 seconds). - Shared readiness consumers pass against real TiDB: canonical file-upload schema, Permission State promotion, and platform-command identity/fence rejection cases. DB, Audit and Runtime Worker packages pass under `-race`; the Runtime Worker architecture gate and `git diff --check` pass. The macOS linker emits its LC_DYSYMTAB warning for the race binary; tests still exit 0. - Verification commands: `go test -race ./nexuscrm-db ./nexuscrm-audit ./backend/internal/runtimeworker -count=1 -timeout=120s`; new TiDB fault tests use `-tags runtime_worker_integration`, selecting `TestAuditWorkerRealTiDBRecoversSilentCommit` and `TestRuntimeWorkerRealTiDBFenceRecoveryAndLifecycle`. Real database batches declare `TIDB_HOST=127.0.0.1 TIDB_PORT=4001 TIDB_USER=root`. - Claim boundary: native driver I/O now bounds silent waits on the audit pool; the existing durable protocol resolves uncertain commits. No extra pool, watchdog, timer loop, queue query, schema or retry state was introduced. Index readiness now reads only the target index while preserving its strict contract. The historical initiating connection fault remains unproven; original-BOM remote reproduction and post-rollout results are not claimed. ### PR publication candidate - Rebased this single fix onto main `c2bdaddb6037831b035809f7ae1d96792a3928f4`. The production diff is unchanged. Rechecked the assembled real TiDB worker: both silent-COMMIT windows pass in 11.79 seconds total; ownership-transfer and lifecycle pass in 1.95 seconds. The complete Runtime Worker `-race` package passes again (9.99 seconds), with the same macOS linker warning. - The PR contains the generic Core fix, regression tests and this record. Delivery-repository polling configuration remains outside this release boundary. No environment workflow or deployment was dispatched. ## 2026-09-03 - Platform-command parallel root-fix baseline - Run ID: `platform-command-partitioned-parallel-baseline`; base `7ea22feea33a6de56a20f73aa9694f4a34b8d988`; environment: local source audit plus the retained Test3 profile from run `33327561494`. - Workload shape: Integration hands accepted Flow commands to Runtime with a declared source `PartitionKey`; ordinary Integration commands do not carry a trusted `SerializationKeyDigest`. The unchanged 2,000-contact/action workload spent roughly four seconds per Flow and remained behind one global effect lane/ownership-row lock. - Confirmed root cause: production assembly fixes Platform-command parallelism at one, and the complete Flow transaction locks the singleton work-family ownership row. The abandoned parallel branch enabled lanes and fixed the deployment fence, but its later merge replaced PartitionKey admission with trusted aggregate admission while leaving the source-ordering claim in comments only. - Root-fix direction: keep the existing bounded worker and fixed 64-slot deployment-transfer fence; select an effect slot by immutable command row ID; separately enforce declared source-partition single-flight and trusted aggregate single-flight in the durable Claim owner, each through an exact scoped index. Do not add a scheduler, process-local lock registry, provider branch, or second source of truth. - Next evidence: prove same-partition exclusion, independent Integration-shaped overlap, trusted-key exclusion, and ownership-transfer drain against real TiDB; then run the complete Persistence and Runtime Worker owner suites before claiming the candidate locally complete. Environment throughput remains a post-deployment verification and is not claimed by this entry. ### Root-fix query and concurrency evidence - Run: `platform-command-partitioned-parallel-focused`; same base plus the current worktree changes; local TiDB `127.0.0.1:4001`. - Correctness: two Integration-shaped commands with distinct source partitions and no trusted aggregate key entered their effect callbacks concurrently in 0.59 seconds; a same-partition successor remained unclaimable until its predecessor committed. The existing publication test retained trusted-key exclusion across different declared partitions. The generic 64-slot transfer test passed against explicitly declared TiDB in 0.39 seconds. - Query plan: with 20,000 immediately eligible rows in an unrelated execution scope, the exact Claim SQL completed its `EXPLAIN ANALYZE` test in 6.69 seconds including fixture insertion. The executed plan contained the scoped claim, partition, and serialization indexes and no `TableFullScan`. - Learning: the throughput fix does not require a new admission coordinator. The short root-locked Claim remains the single durable selector; exact scoped indexes make its two independent exclusion checks bounded to the requested execution scope/key, while the long Flow transaction uses one immutable-row deployment fence slot. - Next evidence: freeze the code, run the changed owner packages under the race detector, run vet/static boundary checks, and record any unrelated full-suite failures separately rather than weakening this claim. ### Final local root-fix candidate - Candidate: branch `codex/platform-command-partitioned-parallel-root-fix` on base `7ea22feea33a6de56a20f73aa9694f4a34b8d988`. - Final result: all `TestRuntimePlatform*` Persistence tests passed in 12.52 seconds; the explicitly enabled real-TiDB ownership-slot drain passed in 0.65 seconds. The independent/same/blocked partition and publication tests passed under the race detector, as did the Platform parallel assembly path. Runtime Worker's complete package passed in 6.13 seconds and the complete system-table catalog package passed in 19.02 seconds. Changed-package vet and `git diff --check` passed. - Full Persistence classification: the first broad run exposed one transient TiDB history-sequence contention failure and one pack-owned DDL timeout; each passed in isolation. `TestPermissionRepositoryDirectWriterClosureUsesGeneratedWriterIDs` still fails for three RecordType writer IDs on the exact unmodified base as well as the candidate, and none of its files is in this diff. The public-core boundary scanner likewise reports only its existing baseline paths and no changed file. - Claim boundary: Runtime Platform commands now use the role's existing bounded worker parallelism. The long Flow transaction no longer locks the singleton work-family row; it locks one immutable-command deployment slot. Declared source partitions and trusted aggregate keys remain separate durable Claim constraints with exact scoped indexes. No new scheduler, lock service, process-local coordination state, provider branch, or alternate read path was added. - Remaining reality gate: local evidence proves concurrency, exclusion, deployment transfer, and bounded query access. The original Test3 HubSpot throughput improvement still requires deploying an exact image/schema tuple and rerunning the unchanged workload; it is intentionally not claimed here. ## 2026-09-03 - Platform-command prepared-statement round-trip candidate - Run: `test3-5667-100-formula-reuse-profile`; Core base `ffe3a10b477e313ddc7830c19084119c0fc1d75c`. A 15-second platform-command execution trace attributed 13.04 seconds to database network reads, including 4.84 seconds (37.1%) in `database/sql.ctxDriverPrepare` / driver `PrepareContext`. A concurrent 30-second CPU profile accumulated only 2.95 CPU-seconds (9.83%), so CPU saturation does not explain the wait. - Single-variable candidate: expose the driver's `InterpolateParams` pool option and enable it only for the production platform-command worker family. Other worker families retain the default prepared-statement path; worker topology and parallelism are unchanged. - Local proof: the DB module passes complete build, vet and test batches; the Runtime Worker package passes its complete test batch. A disposable local TiDB v8.5.0 instance accepted a typed two-parameter query through the real `mysql-tracing` driver with interpolation enabled and returned the expected result. The test was read-only and the instance was removed afterward. - This is an un-deployed local image candidate. Whole-command and synchronization throughput gains remain unproved until a controlled host-local A/B workload; no environment workflow was used. ## 2026-09-05 - Platform-command silent response stalls: baseline and hypothesis - Historical Test3 evidence: deployment Core `4c0bc9527f04c72ca92f53eb71ce7c868a1f9b81`, Connectors `30e98dbb280498641591e242169986286151b366`, and acceptance `ffc26655c3f7ccb3584c4885e648110bba2c22d3`. The 20-record run `test3-perf20-ffc26655-20260904T134514Z` failed its authored deadline; a uniqueness-check response read lasted 339.786 seconds before EOF. In `test3-perf2000-ffc26655-20260904T140127Z`, one command's Flow lookup stalled until EOF and completed on attempt two after 342.760 seconds. Both retained observations found the server transaction idle while the client waited for a response. This proves delayed network-failure detection, not which peer or intermediary lost the response. Deployment and provider details remain in the delivery repository's execution tracker. - Current-tree diagnosis, base `3230b03c7`: `productionWorkerPool` enables native network timeouts only for audit. The Platform-command primary pool carries both the durable command store and the Records-owned transaction graph. Its controller passes a process-lifetime context; effect execution holds the exact command row and ownership fence until its effect/result transaction ends. Lease expiry therefore cannot interrupt an in-flight read or safely release that transaction. Five-minute connection retirement also does not bound an active driver read. The merged Connectors NOWAIT admission fixes cover a different lock path. - Selected hypothesis: reuse the existing driver's per-pool network budget for Platform commands, with a 30-second bound per dial/read/write. This is deliberately separate from whole-command duration: a command may perform many successful operations and exceed 30 seconds overall. The historical delayed handoff's eventual Runtime execution took about 2.1 seconds; 30 seconds leaves substantial headroom while detecting a silent response well before the observed five-minute stalls. Normal Test3 query latency and the complete functionality/performance walking slices remain rollout gates; this initial budget is not a measured universal SQL latency ceiling. - Preserve existing authority: no lease renewal, reconnect loop, effect replay, unknown-commit classification, global database default, deployment DDL, or other worker-family budget changes. Only durable state may decide whether a transaction that lost its acknowledgement should execute again. - Verification matrix: real-driver silent query, COMMIT, ROLLBACK and session reset; the actual Platform-command production pool with a real TiDB scratch database at silent read, before COMMIT, and after COMMIT before acknowledgement; independent effect/result readback and expired-lease recovery; existing worker composition, rollback, restart and ownership guards. Execute tests only on the authorized Test3 host with isolated endpoints/databases, never an active deployment database. No new performance result is claimed yet. ### Test3 isolated red/green result - Candidate: base `3230b03c7` plus this change; production `database.go` SHA-256 `720b83ded1f649e5045f9f93a24255c94ce9fee5e0ba44e7881585f174aa2ff7`. The four changed Go files matched byte-for-byte between the review worktree and the tested source copy. Environment: Linux/amd64, Go 1.25.0, TiDB 8.5.0, `GOMAXPROCS=2`, build parallelism two. Canonical setup created unique scratch databases, and the final independent schema read found none remaining. No deployed service or application database was changed. - Setup limitation: the shared module cache had a root-owned Git fetch file; the initial check stopped before tests. Validation used a task-private module cache and file proxy containing exact published module archives, retaining module checksum verification. No shared-cache permission was changed. - Red run `platform-command-baseline`: an overlay restored only the base production pool configuration, keeping the new regression intact. Its one silent-read case failed the 40-second network-wait bound as expected; package exit one, 41.305 seconds overall. No other baseline test was run. - Green run `platform-command-fixed`: the real production pool released silent reads in 30.0004 seconds, pre-COMMIT disconnects in 30.0011 seconds, and lost post-COMMIT acknowledgements in 30.0002 seconds. Both COMMIT windows returned the existing uncertain-outcome error. Independent durable readback showed zero effects and an empty result for uncommitted attempts; recovery produced exactly one effect and the exact terminal result. A committed attempt retained that same effect/result and was never reclaimed. The suite's leases were two seconds: the 30-second I/O budget does not promise a 30-second command-recovery deadline when a deployment configures a longer lease. - The same green batch included the unchanged audit pre/post-COMMIT recovery cases and complete Platform-command production composition; exit zero, 105.894 seconds, no skips. `platform-command-driver` covered the real driver's silent SELECT, execution, COMMIT, ROLLBACK and session-reset paths plus DSN configuration in 0.767 seconds. `platform-command-owner` ran the entire Runtime Worker package in 1.215 seconds with no failures or skips. - Full JSON logs, exit files and the consolidated `platform-command-validation` report are retained with the task's Test3 evidence. These runs prove bounded failure detection and existing atomic recovery; no new full live cohort, throughput, deployment or Staging qualification is claimed. The next rollout check remains normal SQL latency and the authored functionality/cohort suites on the exact selected release.