# Worker lifecycle performance evidence This log records the performance loop for the shared worker lifecycle. It separates controller call-frequency proof from adapter SQL/RU proof: a Go benchmark cannot establish TiDB Request Unit cost. ## 2026-08-26 — PR 4624 idle-work baseline and test plan - Run/evidence: NexusCRM issue #4430 Test2 evidence at `90e44b728be19732f76856381b428841179c0c89`; controller candidate `14b8c328a4fee429b8202816271af48c7cdaa06b`. - Environment: the historical RU measurements are Test2 statement-digest observations over retained two-minute windows. The controller measurements in this PR are deterministic source-call budgets and local Go benchmarks, not TiDB RU measurements. - Workload shape: quiet/empty durable queues, wake storms, unchanged cross-process generation, exact future due work, and bounded backlog drain. - Baseline: record-command observation executed about 113 times at 81.517 average read RU (about 9,211.421 read RU in the retained window). Rollup observation executed about 106 times at 81.471 average read RU (about 8,635.926 read RU). The Connector diagnostic sampler formerly ran every 15 seconds per API replica; its fixed hourly replacement reduced static scheduled frequency from 240/hour to 1/hour per replica, before measuring actual RU. - Current controller model: startup performs readiness, one reconciliation, one claim pass, and one `NextWorkAt` lookup. After an empty claim it has no queue poll ticker. Further domain reads require an exact due timer, a coalesced wake, a generation change, or the configured low-frequency reconciliation backstop. An unchanged generation performs only the generation probe and must not invoke reconciliation/claim/next-work reads. - Evidence gap: the candidate has behavior tests but no explicit idle source call budget, wake-storm budget, unchanged-generation budget, or benchmark. It also does not yet expose the number of `NextWorkAt` lookups. - Next hypothesis: deterministic call-budget tests will prevent reintroducing wall-clock empty polling; coalesced signal benchmarks will show whether the process-local path allocates or degrades under storms. Each production adapter migration must separately capture its surviving SQL digest, selected plan, executions, total/average read RU, TiDB/RC latency, examined/processed keys, and replica count before and after rollout. - Stop condition: do not claim an RU reduction for PR 4624 itself because no production adapter uses it. Reject any migration whose quiet-window expensive source reads scale with a short poll cadence, Prometheus scrape cadence, or API replica count. ### First controller result - Local run: Apple M4/darwin arm64, Go 1.25, `go test -race -count=20 ./...` and five benchmark samples. - Correctness/call budget: all race repetitions passed. While the initial claim was deliberately blocked, 100,000 Wake requests produced exactly one queued wake, 99,999 coalesced requests, and one additional empty claim plus `NextWorkAt` lookup. An idle controller performed no additional Source read before an explicit reconcile request. Ten unchanged generation probes caused zero additional reconciliation, claim, or next-work operations. - Process-local signal cost: coalesced Wake measured 2.370–2.432 ns/op and coalesced reconciliation signal measured 2.422–2.508 ns/op; both reported 0 B/op and 0 allocations/op. These numbers guard controller overhead only. - Historical scale assessment: if an 81.517-RU record-command aggregate that ran 113 times/two minutes were instead owned by a once-hourly observer, its static frequency would fall from about 3,390/hour to 1/hour and estimated read RU from about 276,342.63/hour to 81.517/hour at the same per-execution cost. The analogous 81.471-RU rollup aggregate would fall from about 3,180/hour and 259,077.78 RU/hour to one execution and 81.471 RU/hour. This is a frequency projection from historical evidence, not a measurement of a future adapter or of PR 4624. - Next step: add the call budgets and benchmark results to PR evidence, then require each migration to demonstrate the actual Test2 before/after digest, RU, plan, keys, latency, queueing, and replica-normalized execution count. ### Follow-up after cancellation and boundary review fixes - Candidate: working tree after `14b8c328a`; review findings fixed locally. - Verification: controller/workertest race suite passed 50 consecutive runs, then 20 additional runs with the allocation guard enabled. - Regression gate: 10,000 repeated pairs of already-coalesced Wake and reconciliation signals report exactly zero allocations per pair. Idle, unchanged-generation, exact-due, and 100,000-wake source-call budgets remain hard test assertions. - Benchmark recheck: Wake 2.143–2.161 ns/op and reconcile signal 2.177–2.331 ns/op across three samples, still 0 B/op and 0 allocations/op. - Learning: the review correction changes shutdown correctness, not the idle read model. Non-nil attempt results observed during cancellation are now surfaced and counted rather than treated as normal drain completion. - Next gate: broad module/boundary verification, then publish this evidence on PR 4624. Production RU remains unmeasured until the first domain migration is deployed to Test2. ### Reconcile ordering and claim-outcome regression - Candidate: working tree after `52d2a7af5`; no production adapter migration. - Workload: queued explicit, periodic, and generation reconciliation signals during `More` continuation, concurrent Wake/reconcile activation, claim cancellation with known and unknown commit outcomes, and the then-current crash-matrix self-reported restart identity validation (superseded below). - Correctness result: controller and workertest suites passed 50 repetitions; the same packages passed 20 race-enabled repetitions. The tests now reject a claim before queued explicit/periodic/generation reconciliation handling, unsafe controller reuse after an outcome-unknown claim error, a contradictory uncommitted marker with returned attempts, any malformed successful claim batch that could strand durable ownership, reuse of the crashed worker identity, a restart against a different durable-store identity, and a conformance callback that ignores its deadline. The identity-string checks were later shown not to bind those claims to actual store/worker instances; see the concrete lifecycle correction below. - Process-local signal recheck: Apple M4/darwin arm64, three samples. Coalesced Wake measured 2.252–2.597 ns/op and reconciliation signal 2.272–2.557 ns/op; both remained 0 B/op and 0 allocations/op. - Learning: prioritizing already-queued control signals adds no polling and does not change the idle Source-read budget. Claim errors need an explicit no-commit proof; cancellation alone is not proof that durable ownership did not change. - Next gate: run the PR's clean-checkout CI/review on the committed candidate. TiDB RU remains intentionally unclaimed until a real domain adapter migrates. ### Exact-due completion regression - Review evidence: adversarial review comment `5424501519` on candidate `f2353b3a2` identified an unconditional claim turn after every successful attempt completion. - Reproduction: three focused tests failed before the fix. An in-flight completion with a one-hour future timer increased Claim calls from 2 to 3 and `NextWorkAt` lookups from 1 to 2; a terminal non-empty batch failed to establish any future timer; the existing exact-due test expected the extra completion-driven empty claim instead of rejecting it. - Root cause: one `more` flag conflated Source-owned immediate continuation with local attempt completion. Completion is not evidence of new durable work and must not become an implicit polling signal. - Fix: only `ClaimBatch.More` and explicit wake/reconcile/generation/due events create another claim turn. Any terminal (`More=false`) claim batch refreshes the exact `NextWorkAt` timer immediately, including a non-empty terminal batch, so removing completion polling does not lose future scheduling. - Result: exact-due, active-completion, terminal-batch scheduling, and bounded continuation tests passed 50 focused repetitions. The complete controller and workertest suites then passed 50 repetitions and 20 race-enabled repetitions; the 100-run boundary scanner check, `go vet`, and diff check also passed. Deployed TiDB RU remains unmeasured. ### Durable reconciliation outcome regression - Review evidence: standard review comment `5424830163` on candidate `26c426277` identified that a `Reconcile` error left the controller reusable even though reconciliation may durably materialize acceptance. - Reproduction: before the fix, three new tests proved that a startup reconciliation error left `Unusable=false`, cancellation hid an unknown reconciliation result by returning nil, and a runtime reconciliation failure drained its active peer but still allowed controller reuse. - Root cause: the controller classified `Claim` as a durable transition but treated `Reconcile` as if it were read-only. Cancellation was incorrectly used as evidence that reconciliation had not committed. - Fix: `Claim` and `Reconcile` now share one `ErrSourceOperationNotCommitted` contract. An error from either mutating Source operation fails closed unless the Source proves that no durable transition committed. Read-only readiness, next-work, and generation errors remain reusable. Runtime reconciliation failures still cancel and fully drain active attempts before returning. - Correctness result: the focused unknown/known outcome matrix passed 50 repetitions, the complete controller and workertest suites passed 50 repetitions, and both packages passed 20 race-enabled repetitions. The module build, vet, race CI slice, 100-run worker boundary scan, and boundary gate passed. - Performance result: no wake, timer, claim-cadence, generation, or Source-read path changed. Coalesced Wake measured 4.232–6.379 ns/op and reconciliation signal 4.489–5.038 ns/op across three local Apple M4 samples, both with 0 B/op and 0 allocations/op. This preserves the controller cost class but is not deployed TiDB RU evidence. ### Overdue exact-time hot-loop regression - Review evidence: standard review comment `5425083411` on candidate `634089371` identified that an already-past `NextWorkAt` was repeatedly armed as a zero-delay timer after an authoritative empty claim. - Reproduction: before the fix, a stale one-hour-overdue timestamp caused 35,913 empty Claim calls and 35,913 `NextWorkAt` lookups in 25 milliseconds. With a 10-millisecond reconciliation backstop, the controller reached only four reconciliations while performing 44,748 empty Claim/next-work pairs in roughly 30 milliseconds. This directly reproduced CPU/database hot looping and reconciliation starvation. - Root cause: `ClaimBatch.More=false` said no immediate continuation existed, but `resetDueTimer` converted the stale scheduling fact into a new immediate continuation. Two authorities contradicted each other and the timer bypassed the bounded retry path. - Fix: only a strictly-future `NextWorkAt` arms a timer. An already-due fact after a terminal claim is counted in `OverdueNextWorkSuppressions` and left disarmed. Wake, generation change, and the existing periodic reconciliation backstop remain the bounded recovery paths; no arbitrary sleep or second poll cadence was added. - Result: the regression now performs exactly one Claim and one next-work lookup during the same 25-millisecond idle window. A second test proves that periodic reconciliation subsequently materializes and completes the work without exceeding one Claim/next-work pass per reconciliation. Overdue, exact-future, terminal-batch, and idle-budget tests passed 50 repetitions; the complete controller/workertest suites passed 50 repetitions and 20 race-enabled repetitions. Deployed TiDB RU remains unmeasured until a domain adapter migrates. ### Concrete crash-matrix lifecycle proof - Review evidence: adversarial review comment `5425173055` on candidate `634089371` showed that the crash matrix accepted domain-supplied store and worker identity strings. Its own test used only invented literals, so a migration could pass without constructing any store or worker. - Reproduction: the old validator accepted `invented-store`, `invented-worker-before`, and `invented-worker-after` without a corresponding fixture or runtime instance. A regression test failed on that false proof before the API was replaced. - Root cause: lifecycle construction was owned by the adapter closure while the harness validated only its narrative. Equality of two unbound strings cannot prove same durable state or a fresh process-local worker. - Fix: the generic case now supplies one non-nil concrete store pointer and a typed `NewWorker`. The harness itself calls `NewWorker` before the crash and again before recovery with the exact same store value, rejects nil/non-pointer fixtures, rejects a reused or type-changed worker pointer, and passes the actual instances into crash/restart callbacks and the store into the durable recovery assertion. The string identity surface was removed. - Result: concrete lifecycle, false-fixture, fresh-instance, and callback deadline tests passed 50 repetitions. The complete controller/workertest suites passed 50 repetitions and a fresh 20-run race suite; the unrelated periodic-ordering test also passed 100 focused race repetitions after one scheduler-timing failure in an earlier combined run. This harness correction changes no controller scheduling, Source-read, or deployed RU path. ## 2026-08-26 — first domain migration: Bulk ingest - Run ID / candidate: `bulk-worker-migration-local-20260826`, based on `18d7634986c74d6a21781a80ef536f234ebfc976`; candidate was still an uncommitted clean-worktree diff when this record was written. - Environment and workload: local Apple M4 workspace tests. The walking slice covers upload admission, close/generation commit, first-claim metadata binding, competing claims, lease-token fencing, effect/checkpoint rollback, retry-safe in-memory checkpoint calculation, malformed-CSV cursor replay, insert/update/upsert/delete dispatch, legacy-row refusal, and storage-authority rollout refusal. The build-tagged TiDB test compiles locally and owns the real claim/fence/effect/checkpoint transaction proof when a declared TiDB test environment is supplied. - Before: Bulk execution was an API-process EventBus subscriber plus goroutine reading process-local files. It had no idle database polling, but an API restart after event publication could strand accepted work; a second process had no durable unique claim; record effects and progress were not one commit; and update/delete jobs accepted by the API were not executed. - After: `_System_BulkJob` and `_System_BulkBatch` remain the only work and checkpoint authority. Close advances a singleton generation in the same transaction; the independent worker claims with ownership epoch, token, and lease fencing; first claim assigns the deployment-frozen immutable metadata binding; each 100-row record effect and cursor update share one transaction. Restarts reopen the committed cursor and exact binding. The old event and goroutine path is removed. - Quiet read budget: unchanged generation is exactly one primary-key query, `SELECT Generation ... WHERE SingletonKey = 1`, proven by a one-query sqlmock budget. With the assembly floor of 15 seconds this is at most 240 point reads per worker-hour, plus the one-minute bounded reconciliation/readiness path. The availability scan is guarded by the exact `(State, AvailableAt, CreatedDate, Id)` index. This is a bounded durability cost, not an RU improvement over the old event-only Bulk path. - Verification so far: full records, shared-worker, schema-catalog, constants, application-service, Runtime-worker, serverruntime, and focused persistence suites passed. Focused tests also prove that a transaction callback replay computes the same checkpoint without mutating the committed base. - Failed/limited evidence: an earlier unfiltered persistence run reached an unrelated environment-backed schema test and failed with a MySQL `unexpected EOF`; the focused persistence suite is green. No Test2 statement digest or RU measurement exists yet, so no deployed RU reduction is claimed. - Next hypothesis/gate: run the real TiDB integration and the frozen candidate locally, then after L2 activation measure the generation and availability digests in Test2: plan, executions/hour, total/average RU, latency, examined/processed keys, queue depth/age, and start latency. Reject activation if pending legacy rows or a different storage authority exist; drain or explicitly migrate them instead of adding compatibility polling. ### Local deployed-TiDB result for the Bulk candidate - Candidate and artifact: NexusCRM `952c27982e5121a636d3e198dbce1ebba7948b2f`. The local-only D3 build produced `nexuscrm-pr4626/runtime-worker:dev-952c27982e5121a636d3e198dbce1ebba7948b2f` with image ID `sha256:3d28a534f5ba857fe4db3ea789255aad29a07444120926a15ccd22fe5637c944`. This is an unsigned local artifact, not a release or Test2 image. - Deployment shape: TiDB 8.5 on port 4001, isolated database `nexuscrm_pr4626_bulk_20260826_a`, filesystem object storage under an isolated `/tmp` root, full L0 `ApplyAll`, Gmail connector dependency, Salesforce CRM standard pack, and the required post-install `apply-if-needed`. The final `ApplyAll` and shallow installed-pack verification completed successfully. - Full Runtime walking slice: a deployment-only integration assembled the production Runtime Worker against the fully bootstrapped database, activated the exact Bulk ownership tuple and Catalog compatibility receipt, reopened the filesystem object by its checksum/location authority, rebuilt the current persisted administrator session, and inserted one real `Account` through the production mutation graph. The Bulk job reached `JobComplete` with one processed row and zero failures. A second run also passed, and cleanup left zero test jobs, ownership rows, Catalog rows, Accounts, and record-event outbox rows. - Restart correctness: a real-TiDB Source/controller test committed the first 100-row checkpoint, injected a transient failure in the next chunk, stopped only after the claim was durably released, then constructed a fresh controller/store holder. Recovery executed only the remaining 150 rows and completed the 250-row job with retry count one. Three race-enabled repetitions passed. This joins the existing claim-token/epoch/fence and atomic effect/checkpoint integration proof instead of testing either half alone. - Orchestration throughput: the same real TiDB path processed 10,000 CSV rows through 100 durable checkpoint transactions in 2.781 seconds (3,596 rows/s) without race instrumentation. Three race-enabled repetitions completed in 2.752-3.029 seconds (3,301-3,633 rows/s). The executor in this measurement is deliberately in-memory: it measures CSV verification, durable claiming, checkpoint transactions, and shared-controller overhead, not object-specific validation, Flow, or record-write throughput. The deployed walking slice is the separate proof that the real mutation graph remains compatible. - Production-cadence idle result: over 65 seconds, an empty Bulk source made exactly five generation reads (startup plus four 15-second probes), two reconciliations (startup plus the one-minute backstop), two empty claims, and two `NextWorkAt` reads. It made no additional short-poll reads. The generation point lookup reported 0.475 average read RU, zero processed keys, 0 ms queued RC time, and 0.512-0.863 ms average latency across the two statement-summary windows. This projects to at most 114 local-TiDB read RU per worker-hour for generation detection. The separate 30-second process ownership guard uses one exact ownership-row read (0.475 read RU in this run), or at most another 57 local-TiDB read RU per worker-hour. The once-minute reconciliation/claim transactions remain an additional bounded durability cost and must be measured as a whole in Test2; they are not hidden in the generation number. - Interpretation: no local RC queueing or processed-key scan appeared, and the availability path used `idx_bulk_job_available`. The orchestration layer is not the measured throughput bottleneck. This still does not claim a Bulk RU improvement over the retired EventBus path, whose idle DB cost was zero; the bounded cost buys durable ownership, replay, and atomic checkpoints. Test2 before/after statement digests remain the release gate for cloud RU and replica-normalized totals. - Unrelated broad-suite limits observed: an unfiltered persistence package run against the local TiDB exposed existing integration-fixture failures: two older Runtime Platform-command scratch schemas omit `_System_RuntimeWorkFence`, and one high-contention history-sequence test can exhaust its pessimistic-lock retry budget. The implicated tests, ownership implementation, and scratch fixture have no diff from this candidate's `origin/main` base. The focused Bulk store/fence/checkpoint integration is green; these failures are recorded rather than misreported as a Bulk regression or silently folded into this PR. ### Records-owned Bulk transaction correction - Run/candidate: 2026-08-26 local correction on PR #4626 head `0c8b8af85` plus the uncommitted Records-owned transaction diff; TiDB 8.5 on port 4406. Port 4001 was rejected as performance evidence after it returned TiDB 8027 (information schema could not update in one lease). - Root cause: the first migration made the Bulk claim store the outer transaction owner. Nested Records calls then skipped their lifecycle-batch setup and source-aware Group/User closure state; Update/Delete/Upsert also converted per-row errors to terminal failures inside that ambient transaction, so post-DML failures could commit with a failed checkpoint. - Correction: Records now owns the effect transaction and Bulk fencing, lease renewal, and checkpoint state join that existing transaction. Insert remains one 100-row bulk mutation per checkpoint. Independent Update/Delete/Upsert operations use one Records-owned effect/checkpoint transaction per row; a classified business rejection is checkpointed only in a fresh transaction after the first transaction reports a typed proven rollback. Unknown outcomes remain process-fatal. - Correctness evidence: the real-TiDB injected post-write validation failure completed with one failed-row checkpoint and zero rows in the business-effect probe table, proving the effect rolled back before the cursor advanced. The 250-row restart test again resumed from the committed 100-row cursor and completed only the remaining work. Focused tests prove lifecycle-batch and Group/User closure state are both installed by the same outcome-returning Records boundary, and that Bulk Store joins it without opening another transaction. - Performance evidence: the unchanged 10,000-row Insert workload completed in 1.735 seconds (5,765 rows/s) through exactly 100 effect/checkpoint transactions. This is above the earlier 3,596 rows/s local orchestration result; environment variance means it is evidence of no observed regression, not a claimed optimization. Normal Insert batching is unchanged. Legacy Upsert already executed one Records transaction per row; Update/Delete were not executable in the retired processor, so there is no earlier supported throughput baseline to regress. - Next gate: race the outcome boundary and the real-TiDB rollback/restart tests, then run the complete affected package suites and the deployed mutation-graph walking slice. Test2 RU/latency remains the cloud release evidence; no local number is substituted for it. ## 2026-08-27 — Outbound-intent safe-retry hot loop - Runs/revisions: Test2 Env Inspection `32585126993` (Saturday, Core `2d54c762`, connectors `v0.2.13`) completed in 899,471 ms; controlled current rerun `33042721152` (Core `4d1c5bd6`, connectors `v0.2.17`) completed the identical 261-case inventory in 1,067,429 ms. Diagnostic runs `33044680420` and `33044914940` supplied TiDB statement-summary evidence; their later D3 journal steps failed because no D3 run was selected, but the SQL collection steps and artifacts completed successfully. - Workload shape: one full inspection followed by a quiet six-minute Test2 window. Both windows used the same deployed Integration owner. The quiet window is the falsification boundary for the hypothesis that inspection itself caused the claim traffic. - Profile result: the dominant statement digest `7461b6c846c4cb08` claims the next `_System_IntegrationOutboundCommand` with lane ordering and `FOR UPDATE`. During the quiet window it executed 711 times from `06:04:23..06:10:27` (about 1.95/s), averaged 111.768 read RU and 8,065.7 processed keys, and consumed an estimated 79,467 read RU. A companion `State=Retry` update had the exact same 711 executions, proving repeatedly processed commands rather than empty claim polling. Resource-control queue, application pool wait, and lock-wait evidence remained zero. - Root cause in current connectors `v0.2.17`: the outbound-intent lifecycle has no common safe-retry policy. Retryable preparation failures, schema/window deferrals, default definitely-not-dispatched outcomes, and Runtime writeback failures all schedule `NextAttemptAt=now+1s`. `AttemptCount` advances only at provider dispatch by design, so it cannot supply a retry-series input for preparation or writeback and is not a correct general retry counter. The exact Test2 `FailureCode` is not present in sanitized statement summaries and still requires an aggregate read-only diagnostic; that trigger classification selects the hottest entry point but does not change the lifecycle failure class shared by all of them. - Ownership decision: retry, poison, and external-outcome semantics remain in the outbound-intent domain Source/Attempt, not in the shared process-local controller. Preserve `AttemptCount` as provider-dispatch count and add a separate durable consecutive safe-retry count shared by preparation, definitely-not-dispatched provider outcomes, association continuation, and Runtime writeback. Compute a capped exponential delay with bounded jitter from that count, start a new series only after durable forward progress, and keep dependency/writeback recovery durable across restart. Authoritative permanent failures remain rejected; retryable and post-success writeback failures remain recoverable without a short hot loop. - Performance gate: deterministic tests must prove no claim before the exact due time, increasing/capped retry delays, restart persistence, reset after a durable forward-progress or terminal transition, and a bounded claim/update count during a sustained unavailable-dependency window. After rollout, repeat the quiet-window digest measurement and the identical full inspection before attributing any share of the 168-second Saturday-to-current wall-clock delta to this fix. - Local result: the connectors candidate passed the deterministic retry-policy, preparation, definitely-not-dispatched, writeback, provider-attempt-budget, and restart-persistence tests for 20 repetitions. The sustained dependency outage case permits at most 10 attempts in six simulated minutes instead of one attempt per second, and the real TiDB persistence paths passed 10 repetitions. The full connectors unit suite also passed with TiDB integration disabled. This proves the state-machine and restart boundary locally; it does not substitute for the post-rollout Test2 RU comparison. - Next hypothesis: with a 1-second initial delay and exponential cap, the first recovery remains responsive while persistent failures rapidly leave the hot path; quiet claim executions and RU should be bounded by the retry series, not scale linearly at roughly two executions per second. ### Test2 rollout and acceptance evidence - Candidate and rollout: Core `867d82a1e0acc39cd6d548b44110c73a2b371a2a` (current `main` plus the Bulk-worker migration), connectors `v0.2.18`, Runtime Worker image `sha256:d24498a41d3bb17f0c39bfda00d22e537fac44a29f3b21ca3283b37c7bbf859d`. Test2 L0 recovery run `33057685021` replaced all seven standard Worker roles, Runtime API, and Integration without a database refresh; Bulk activation and its durable walking slice passed in run `33058011006`. - Correctness workload: full inspection `33058781288` ran 264 cases in 1,090,635 ms: 244 passed, 4 failed, and 16 skipped; all 38 executed performance cases passed. Targeted serial rerun `33060515917` reproduced the same four functional failures. Two are the known Credits active-formula record-UI defect (`#5508/#5509`), one is the known app-services omission of `PageSection.tab` (`nexuscrm-app-packs#746`), and the Lead failure reaches the same formula-view read boundary: its create commits, but the immediately following update's `prelockInitialUpdateParents` lookup through `v_lead` returns no row. None enters the connector or Bulk-worker paths. - Measurement validity: diagnostic run `33058109675` began while Integration was still being replaced, so its statement window straddled old and new owners and is not used as post-rollout acceptance evidence. After the full inspection and a four-case targeted rerun, the environment was left idle for more than six minutes; read-only diagnostic run `33061138554` then captured the TiDB statement history and exact request logs without another observation workload. - Active-window result: from `09:51:36..09:54:38`, the inspection workload produced 336 outbound candidate claims and 219 durable `RetryCount` updates. The seven `_System_IntegrationOutboundCommand` digests consumed an estimated 35,680 read RU in aggregate. This is not a same-fault retry-count comparison: the statement summary cannot identify how many distinct commands contributed those updates. - Quiet-window result: the last outbound-command statement was observed at `09:54:38`; none was recorded through the `10:00:44` collection. The former retained idle failure executed 711 claims plus 711 retry transitions in its comparable six-minute quiet window. The deployed system therefore no longer has that continuous idle hot loop in the current Test2 state. - Acceptance boundary: this proves the observed idle-loop outcome and preserves task/data durability, but it does **not** independently prove the projected 95% reduction for one continuously failing command. That percentage remains gated on a controlled unavailable-dependency run (or a read-only per-command retry aggregate) that demonstrates the persisted exponential series for the same fault shape. Do not infer it from a changed digest or from the absence of a retained failing command. ## 2026-09-05 - Cross-worker queue-plan audit - Commits/environment: Core `3230b03c7`; Connector PR #317 candidate `b1489e0`; local TiDB `v8.5.0` at `127.0.0.1:4001`. Synthetic fixtures placed 1,000 idle or foreign-scope rows ahead of one target-scope row. - Scope: every production Runtime worker family plus Integration inbound and outbound scheduling. The decision boundary was hot claim, hint, due-time, and recovery reads; hourly diagnostics and one-time startup integrity checks were not treated as queue scheduling. - Good paths: Bulk uses one-row generation wake detection, a one-minute reconciliation backstop, and `idx_bulk_job_available`; its empty due claim scanned zero rows (0.475 read RU) and its next-time lookup read one indexed row (0.954 read RU). Audit outbox and the lifecycle-expiry workers also use bounded leading-state/time indexes. Record-event outbox and Runtime platform commands already have scoped claim-plan guards. - Confirmed defect 1: RollupDelta filters exact `(OrgId, AppInstallationId, RuntimeCapabilityId)` only after a global status/time index. With 1,000 due rows for another scope, TiDB selected `TableFullScan`. The same missing scope prefix affects the binding peek and parent-lane load. - Confirmed defect 2: scheduled-run recovery has separate status/time indexes but no scope prefix. Both the hint and bounded candidate read examined all 1,001 status-matching rows in the fixture; the candidate read consumed 5.630 read RU. Acceptance also lacks an index for exact scope plus flow/version/ schedule-window batch identity. - Ownership and next change: keep the existing state machines and add only catalog-owned, scope-leading recovery indexes plus the exact Scheduled batch identity index for these existing read paths. Rewrite the scheduled recovery OR as two disjoint indexed branches so TiDB cannot choose one status index and residual-filter another scope. Add real-TiDB regression plans that reject `TableFullScan` and bound examined rows. - Local result: warm migration added the declared Scheduled and Rollup indexes idempotently and rejected an incompatible existing index before adding any sibling. A proposed Rollup index including all three `VARCHAR(255)` scope columns was rejected by real TiDB at 3,188 bytes versus the 3,072-byte index limit; the final index uses `(OrgId, AppInstallationId, Status, NextAttemptAt, CreatedDate)` and keeps RuntimeCapabilityId as the residual authority check. This is the smallest index that isolates one installation without prefix-index semantics or a second scope identity. - With 1,000 foreign-scope due rows, Scheduled accepted-batch/hint/list and Rollup hint/binding peek each examined zero rows and reported no table full scan. Scheduled list caps each accepted/reserved branch before its final merge, so the outer limit cannot hide an unbounded branch read. The relevant Records package passed in full; focused Scheduled recovery/restart, poison handling, starvation, point lookup, and plan tests passed against real TiDB. - Full-owner context: Core schemacatalog and Records passed. Persistence's one history-sequence contention failure passed immediately in isolation; the other three permission writer-closure failures reproduce on untouched `3230b03c7`. Automation's one legacy-row readiness assertion also reproduces on untouched `3230b03c7`; every changed Scheduled path passed in the focused batch. These baseline failures are recorded as evidence and were not patched into this performance change. - Final lane-read correction: review of the transaction-held Rollup parent-lane load found that it still forced the global parent-lane index after the scoped claim. Runtime-bound loads now retain the installation-scoped claim index; only the retained unbound compatibility path uses the global parent index. A real-TiDB fixture with 1,000 foreign-installation rows sharing the exact target parent identity reports zero scanned rows and no full scan for the hint, binding peek, and lane load. This closes the foreign-row scan/lock window without adding another persisted identity or migration protocol. - Final Scheduled batch correction: the first batch index keyed only Flow, version, and window, so identical schedule tuples in another Runtime scope could still be scanned and locked. `BatchIdentityDigest` now projects the exact Runtime scope plus that batch tuple into one catalog-owned `CHAR(64)` index key; the explicit columns remain read-time authority. Both normal scheduled producers persist it, and the warm migration backfills existing scoped rows with the same eight-byte length-framed SHA-256 grammar (including a non-ASCII scope test). With 1,000 foreign-scope rows using the identical Flow/version/window tuple, the accepted-batch read scans zero rows and does not full-scan. Legacy unscoped rows remain outside the fenced worker and the existing quiescent ownership cutover remains mandatory. - Reopened Rollup scope evidence at Core head `5d67f693a`: the installation-only index still left `RuntimeCapabilityId` and the selected parent lane as residual filters. That interim index and read path are superseded. Rollup now persists the existing generic `RuntimeWorkScope.Digest()` projection and uses compact scope-claim and scope-parent-lane indexes; explicit scope columns remain the authority and a digest disagreement fails closed. The selector reads only one indexed candidate ID, point-loads its binding, and point-locks the same ID after the binder succeeds. This avoids both an over-width composite key and a table-lookup batch across unrelated rows. - Updated real-TiDB profile: 1,000 rows in the same Org/App but another Runtime capability plus 1,000 rows in the exact Runtime scope but another parent lane. The hint examined 0 rows, the covering candidate selection examined TiKV's constant 64-key coprocessor batch, and the target parent-lane load examined 0; none used a table full scan. Warm migration backfilled a Unicode scope with the exact two-byte length-framed `RuntimeWorkScope.Digest()` value. The full Records owner package then passed in 44.5 seconds after its complete queue shape assertion was updated for the additive digest column. ## 2026-09-05 - Review reopened queue evidence and cutover - Baseline: Core `9bbb56f85`, Connector `59159ea`; local TiDB v8.5.0. - A canonical Rollup queue followed by the old producer INSERT after backfill contains one scoped pending row with NULL ScopeDigest. The new hint sees zero rows while schema readiness succeeds. Stop only Integration is insufficient; Runtime API/Worker producers must remain stopped through final backfill. - An uncorrelated EXISTS hint folds to Projection/TableDual. Forcing its inner read to TableFullScan still passes the old scan-node counter. The former zero-scan hint evidence was unobserved, not a measurement. - Next change: fail readiness on unindexed scoped Rollup work, document the stopped-writer cutover, and inspect the exact inner hint reads with a required data-access node rather than accepting folded plans. - The precise cutover contract is now in [worker-queue-digest-cutover.md](../design/worker-queue-digest-cutover.md). Schema-first is not permission for old Runtime producers to keep writing after the final backfill; their activation fence is separate from Integration. - First correction batch: canonical RemoteLink point-read, registration races, interrupted-page resumption and the Rollup late-writer cutover test passed. Exposing the real Rollup hint now reports a 64-row TiKV batch rather than the former unobserved zero. Its index prefix and LIMIT are unchanged; verify the same small bound after doubling the due backlog instead of claiming zero. - Final correction evidence: after exposing the actual inner reads, Rollup hint and candidate selection each read a constant 64-row TiKV batch; the selected parent-lane miss reads zero rows. Doubling the population from 2,000 to 4,000 keeps the hint at 64 with no full scan. Scheduled accepted/reserved inner hint reads and scoped migration tests pass. The late-old-writer test fails readiness for both pending and failed unindexed work, then proves idempotent final schema reapply restores visibility with the original metadata binding. - Full Records passes in 46.956s; full Automation has only the existing `TestFencedScheduledRunTiDBCommitsEffectsAndCompletionTogetherAndFencesStaleOwner` legacy-active-row readiness assertion (also reproduced at `33ad73636`). The focused changed Scheduled tests pass. `scripts/ci/check-no-sqlmock.sh` and `git diff --check` pass. No Test2 deployment or online data mutation was run.