# Runtime Worker observation learned record ## 2026-09-06 — Staging Platform-command capacity diagnosis - Runs and frozen inputs: functionality `33978774591` and Set Campaign performance `33980865706` used Staging deployment `33969552593`, Core `33ad7363666f3018e447045cd1a5747d58686c4c`, and L2 metadata `3e9d882b6245812260aee31391bc156e7d4463c9`. Read-only durable-row and worker metric evidence is diagnostic, not an admitted three-run baseline or a new throughput result. The performance case began without prior Runtime work; it still built its own backlog. The active resolved BOM SHA-256 was `cfc8f9058c46c194263477d4873edcaba89a682576428d41fd9fc15fb05f6118`. - Workload and measurements: all 2,000 action commands have different nonempty source partitions and no trusted aggregate serialization key. Their claim-to-terminal time averaged 1.592 seconds (p50 1.48, p95 3.18), while queue wait ranged from 2,232.83 to 6,913.65 seconds. Every command remained on attempt one with zero repairs. The live role omitted `parallelism`, so Core selected one lane; its primary pool allowed four open connections and recorded no pool wait. Batch size ten is a serial drain limit, not ten concurrent executions. - Source authority: deployed Core already admits independent source partitions concurrently, keeps same-partition and trusted-aggregate exclusion in its durable Claim owner, and uses immutable-command fixed-slot effect fences. The effect and terminal result still share one transaction; deployment transfer drains all effect slots. Core `31d9bb58b1b9663afc9a75b8fb1e5fd60fe003a9`, already selected by this L2 main, retains that implementation. The earlier L2 serial policy was introduced in `bf8fa7610c` for Core `7cd73d9a`; retaining it after the reviewed Core concurrency fix leaves that mechanism unused. - Next bounded change: set only `runtime.platform-command.v1` to eight lanes, seventeen maximum open connections, and eight idle connections. This role is rendered for `runtime.commands.accept.v1`. Preserve source/aggregate ordering, ownership epochs, release selection, retry/lease policy, all other roles, and the existing acceptance deadlines. Older signed serial role documents must remain valid. Do not add processed-row retention to this work family. - Verification contract before rollout: prove desired/resolved BOM and rendered role identity agree, reject invalid lane counts and insufficient pools, and retain the exact activation/Catalog authority. Run the selected Core's real TiDB independent/same/blocked-partition, publication, deployment-transfer, rollback/restart, and Flow execution checks with the candidate capacity. Stop at a decisive failure. Only a subsequent unchanged acceptance workload can establish an environment throughput improvement; no larger timeout is part of this hypothesis. ### Local candidate verification - Candidate capacity is eight lanes with a 17/8 open/idle primary pool; Core remains `31d9bb58b1b9663afc9a75b8fb1e5fd60fe003a9`. Existing L2 validation and component checks passed (31 tests); 17 temporary checks additionally covered invalid capacity, serial-release compatibility, and the complete BOM-to-activation-to-Catalog role projection. No new tests were added to the repository. The signed producer-closure digest was refreshed because its validator changed. - Real local TiDB v8.5.8: the six existing command persistence and ownership tests passed in 2.268 seconds. The seven selected worker tests passed in 8.220 seconds, including actual Flow execution, inbound identity finalization, rollback/replay, restart, and V3 ApplicationResult ownership. A temporary Go overlay supplied the BOM's 8/17/8 capacity to the existing worker fixtures; their deliberate short fault-injection leases/retries remained unchanged. The matching pool metric assertion was adjusted to seventeen. Production source and test deadlines were unchanged. - Local proof artifacts: `/tmp/nexuscrm-5434-runtime-capacity/` contains the temporary checks, exact policy, overlay, and complete worker log. Overlay SHA-256: `b53faeef533008e8f35423c8c20562156539ca463e7586f03e247cabe230f3dd`. The initial run exposed only missing local test credentials and the fixture's old four-connection expectation; both were corrected in the temporary test invocation/overlay. An unrelated local-D3 filesystem fixture also fails on the untouched checkout because macOS lacks the Linux Runtime user identity. - Remaining gate: deploy the newly signed candidate through the existing Staging owner and compare the unchanged workload's terminal correctness and queue wait. Local provider-free proof does not establish Staging throughput. This is the durable evidence log for the Runtime Worker topology decision in NexusCRM issue #4447. Append one record before changing the next performance hypothesis. A run is baseline evidence only when its artifact passes `scripts/compare_runtime_worker_baseline.py`; an otherwise useful observation must be recorded as provisional. ## Baseline acceptance contract Dispatch `Collect Test Backend Logs` from the same selected L2 controller revision and select the same Test2 or Test3 environment for all three repetitions. Select one baseline scenario and case, use the explicit active D3 run ID (never `active`), and use repetition `1`, `2`, then `3`. Non-quiet samples must name a distinct Actions workload run from that same selected controller revision which overlaps the observation window. `Env Inspection` dispatches only from `refs/heads/main`, so non-quiet baselines require dispatching the collector from `main`; arbitrary branch dispatches support generic observation and quiet baselines only. Quiet samples use `queues-empty`, no workload run, and exactly 900 observation seconds. The current v1 gate accepts quiet and mixed-workload evidence only. Mixed runs must be a completed successful full `Env Inspection` of the selected Test environment; the collector reads its official `report.json` artifact and rejects dry-run, staging, a different Test slot, selected-case, failed, incomplete, unrelated-workflow, or in-progress runs. The inspection runner seals the full planned automatic-case inventory before any runtime case can be skipped; every required case must appear and pass, while only cases declared `optional` in the selected metadata may be skipped. Single-family pressure remains closed until one reviewed Worker-to-producer case catalog exists. Failure workloads remain closed until dedicated fault runners own and assert each fault. Neither unsupported scenario may be represented by a free-form label. After downloading the three artifacts, compare them with: ```bash python3 scripts/compare_runtime_worker_baseline.py \ --run \ --run \ --run \ --json-output comparison.json \ --markdown-output comparison.md ``` The comparison fails closed unless the D3 deployment, resolved BOM, effective images, effective Worker runtime policy, Catalog/security epochs, schema/engine generations, frontend, observer revision, scenario, and one supported exact seven- or eight-role Worker shape match. It also rejects incomplete observations, quiet runs which process durable work, non-quiet workload runs without proven time overlap, and any tracked RSS/heap/connection/execution/SQL-RU spread above 10%. ## 2026-08-17 — baseline comparability gap - Run IDs: observer `31990513399`; overlapping Test2 inspection `31990522254`. - Commits: L2 `81bab3f44e1fb3e03070c73ba72299c05da47450`; the active D3 Runtime/App revisions remain to be recovered from the completed observer artifact. - Environment: Test2 D3, active ten-container Runtime composition. - Workload: full environment inspection overlapped the bounded Runtime/Worker observation; this is a mixed-workload candidate, not a quiet run. - Profile/result: the observer was cancelled during its bounded observation and the inspection failed (`234` passed, `5` failed, `16` skipped; three failures classified as infrastructure availability). Neither run is accepted baseline evidence. The observer can capture all seven Worker time series, profiles, pool state, and TiDB statement digests, but this pre-gate artifact does not bind the sample to a frozen Runtime/App/L2/database-shape identity or a named successful workload run. - Failed/learned: matching the L2 workflow commit and wall-clock overlap is not enough to prove equivalent runs. Without an artifact-owned identity and scenario/repetition contract, a three-run 10% variance verdict would be an inference rather than evidence. - Next hypothesis: emit a sanitized frozen baseline identity and explicit workload context from the active D3 authority, then fail closed when three artifacts differ in identity, scenario, role set, duration, completeness, or the cost fields required by the baseline gate. ## 2026-08-18 — baseline evidence authority review - Run IDs: Codex Review `32141981144` and `32167191746`; no Test2 workload or observer run was dispatched. - Commits: reviewed heads `876fbc992cf329afc3741e55556fcadb6e3f3315` and `7bd204a5efb76daa9ec88bbdbdadc90b3e94823f`; L2 base `abc84c74d46f0cbb8c36ea8b8f319f9ff928c8b8`. - Environment: local contract tests plus GitHub PR review; no runtime profile was collected because the failure was in evidence admission, not Worker behavior. - Workload shape: quiet and full Test2 mixed-workload baseline admission. - Profile/result: the live inspection metadata expands to 256 automatic cases, including 13 explicitly optional cases. A full workload may skip those 13 declared optional cases, but every required case must be present and pass. - Failed/learned: a caller-controlled variance threshold, a mutable branch checkout, regex-only scenario/case validation, and summary-only inspection validation each allowed evidence to claim more than it proved. Evidence facts must be sealed at their owner: the comparator owns the fixed 10% threshold and scenario/case map, the verified checkout owns the observer revision, and the inspection runner owns its planned case inventory. - Next hypothesis: collect three quiet and three full mixed-workload samples only after the revised gate passes review; reject pre-contract reports rather than treating their missing inventory proof as historical baseline evidence. ## 2026-09-06 — Test3 acceptance purge validation - Candidate L2: `546104f30f0123127c2b7c86a47d6e1b7b0b9ca1`; Test3 deployment `d3-test3-main-4c0bc952-33de9365`, resolved BOM file digest `sha256:b1f2abf4249fd7e704d6d80a90c555f91c06a33af675e874bae76d4272beda7a`. - Workload: one real scoped HubSpot queue purge through the unchanged candidate runner, with only its environment selectors adapted to native Test3 authority. All 12 exact containers were restored. The operation failed after 114.824 seconds with `InterfaceError(0)`; it produced no successful purge or empty-check result and was not retried. - Read-only evidence: Test3 TiDB v8.5.3-serverless plans the Runtime delete as a hash semi join, not a correlated Apply. The attempt delete scans an estimated 250,038 attempt rows and joins two Event scans; current WorkItem statistics estimate two rows. These plan estimates do not identify the timed-out statement. Cluster statement-summary access was denied (1142), so no server latency attribution is claimed. - Learned: the transaction's unguarded rollback can replace a disconnected client's original error with `InterfaceError(0)`. Preserve the original failure stage and driver code before drawing conclusions from the generic error. - Next hypothesis: compare the attempt scan with disjoint indexed Event-owned and eventless WorkItem-owned reads. Change the query only after the read-only comparison confirms equivalent scope and useful cost reduction. Keep the 30-second driver, 60-second purge, and acceptance deadlines unchanged. Read-only comparison on the same Test3 deployment: the original attempt-key count took 1.051 seconds (about 1990 RU); disjoint Event-owned and eventless-owned branches took 0.538 and 0.641 seconds (about 409 and 1473 RU). All matched zero rows. The eventless branch still scanned 250,313 attempts, and the combined elapsed time did not improve. This does not explain the earlier purge failure or justify that rewrite. Preserve the failure stage first; check indexed lookups before considering any SQL change. These SELECT measurements exclude deletion, index maintenance, and commit latency. A final SELECT-only indexed-join-hint variant took 1.039 seconds and about 2239.68 RU while still scanning 250,313 attempts. It did not improve resource cost; no SQL rewrite or optimizer hint was adopted. The follow-up changes only failure reporting so a subsequent bounded validation can identify the original failed statement and driver error instead of guessing from plan estimates.