# Field History Sequence Performance Learnings ## 2026-07-31 production incident and emergency containment - Evidence owner: NexusCRM stable PR [#4199](https://github.com/pingcap-inc/NexusCRM/pull/4199); the exact tenant tuple and workload identifiers remain in the downstream incident ledger. - Frozen L0 baseline: `2f2282ea96d3808812dfadd7fe8bdb3d17979523`. - Workload: an inbound connector batch amplified concurrent generic Records writes across a multi-step atomic Flow. - Profile: 23 of 23 sampled cluster lock-wait observations were on `_System_FieldHistorySequence`; one heavily written tracked field's `FieldId` row was a shared hotspot for otherwise independent record writes. The production deployment reported TiDB 1205 failures and loss of useful inbound progress. - Learned: deterministic acquisition order prevents inverse-order deadlocks but cannot provide throughput when every writer still needs the same singleton sequence row. The bottleneck is platform-wide per tracked field; the inbound connector workload was an amplifier, not the ownership boundary. - Accepted containment: an explicit default-on circuit breaker skips all Update history-sequence reservation while preserving Generic Audit V2. Update Field History facts during the window have no sequence and remain permanently hidden. No replay or guessed backfill is allowed. - Verification target: under the same candidate, zero sequence-table DML from the Records update path, continued Generic Audit V2 emission, terminal inbound business progress, and no new sequence-attributable TiDB 1205 errors. - Next hypothesis: replace the singleton per-FieldId allocator with a non-hot ordering authority, then validate mixed-object connector, ETL, Flow, and API concurrency before removing the circuit breaker. ## 2026-08-28 #4639 compatibility implementation - Evidence owner: NexusCRM issue [#4639](https://github.com/pingcap-inc/NexusCRM/issues/4639); implementation branch `codex/issue-4639-field-history-eligibility`. - Frozen Core baseline: `d884b55fbb5ae12689e1d3ad4ad5496f66054fde`. - Workload shape: one Records Update produces field diffs, atomically writes an Audit V2 JSON outbox row, and a restartable worker expands the payload into `_System_AuditLog`; the legacy synchronous writer shares the same diff owner. - Profile/model result: `recordAuditFieldChanges` already resolves the exact request-bound schema once and that field metadata already contains `TrackHistory`. A proposed `_System_Field` query would duplicate authority and add SQL/RU; no new database read is required. - Change in progress: stamp nullable `FieldHistoryEligible` from that existing schema traversal and preserve it through both writers, JSON transport, worker expansion, exact-replay verification, and the history projection. During the rolling compatibility window, sequence reservation remains enabled and only NULL eligibility may use the existing interval predicate. - Not yet claimed: this compatibility revision does not itself remove sequence DML or improve RU. Its acceptance target is zero additional metadata SQL and no silent history loss across old/new producer, reader, and worker pairs. - Next hypothesis: after all replicas carry the new transport, pre-cutover outbox work is drained, and deterministic materialization is complete, disabling sequence reservation will remove the observed lock/replay cost; the same approval workload must then confirm zero sequence DML, zero history-prelock replay, and lower RU/latency before final code deletion. ### 2026-08-28 implementation evidence loop - Targeted TiDB compatibility, Audit, Records, schema-catalog, generated constants, and generated-model tests pass on the frozen branch candidate. - The Runtime producer test resolves eligibility exclusively through a bound `RuntimeMetadataView`; it has no repository or SQL fallback, proving the change adds no metadata query/RU to the record path. - Review found and removed an avoidable allocation shape: the first draft took one escaped boolean address for every schema field. The retained path stores boolean values in the already-required field map and materializes an optional value only for fields that actually changed. - The compatibility revision still intentionally makes no end-to-end latency or RU improvement claim: sequence writes remain until the rollout death conditions are met. Performance acceptance belongs to the boolean-only cutover using the same approval workload and quiet-window RU collection. - Mixed-version falsification found that producer stamping cannot default on in the transport-readiness release. An old worker ignores the new JSON member; if it writes a NULL AuditLog row and crashes before outbox finalization, a new worker sees a non-exact replay, while successful old-worker finalization can permanently lose a stamped `FALSE`. The producer therefore needs a fail-closed, default-off rollout switch until every worker preserves the new field; sequence compatibility remains active during that stage. - Review falsification found a second authority boundary: the legacy ServiceManager/REST graph resolves `GetSchema` through a mutable metadata cache, not an admitted Runtime view. Even after transport readiness, that graph cannot safely stamp an authoritative revision fact. The narrow fix is to permit stamping only when `UsesRequestBoundRuntimeView()` is true; legacy producers remain NULL+sequence compatibility producers until migrated or removed. No new legacy admission mechanism is justified in this PR. ## 2026-08-28 #4639 Runtime cutover loop - Evidence owners: [#4639](https://github.com/pingcap-inc/NexusCRM/issues/4639) for the mechanism and [#4636](https://github.com/pingcap-inc/NexusCRM/issues/4636) for the approval acceptance workload. Frozen Core baseline: `67dad5092` on branch `codex/issue-4639-field-history-cutover`. - Runtime baseline: two test2 terminal-approval runs on a post-#4655 tier each required five business-transaction attempts: four `_System_FieldHistorySequence` ordering rollbacks followed by one commit. The HTTP response was successful, so the authoritative signal is the per-attempt approval action log, not response status. - Reopened topology evidence: the independently deployed L0 uses an exact, stopped-upgrade tuple. Runtime API and Runtime workers are replaced from one frozen Core candidate before the new serving owner is activated; this path has no ordinary rolling old-worker/new-producer window. The temporary default-off transport switch therefore adds no safety to this deployment. - Change hypothesis: a producer backed by the immutable request-bound Mutation view can always persist `FieldHistoryEligible` and must skip field-history sequence reservation. The mutable legacy ServiceManager graph remains a NULL+sequence compatibility producer until it is migrated or retired; it is not used by the serviceified test2 approval path. - Required local proof: request-bound true/false decisions are revision-stable, unresolved fields fail closed, Audit V2 transport/replay preserves the decision, and a panic-on-use transaction double observes zero sequence work. Required environment proof: the #4636 terminal approval commits in exactly one attempt with zero sequence-lock rollback, while the complete approval cascade remains correct. - Performance target: remove every `_System_FieldHistorySequence` write and history-prelock replay from request-bound Runtime mutations without adding a metadata query. Compare test2 request latency and RU against the recorded five-attempt baseline; no improvement is claimed for the legacy monolith compatibility path in this slice. - Falsification result: the first implementation treated one unresolved generic-audit diff as making the whole batch legacy. The real Runtime+TiDB Approval test then incremented the WorkItem Status and FlowMarker sequence rows, disproving that graph-wide rule. The corrected boundary is per fact: every diff with a stable FieldId must carry eligibility; a diff without a FieldId cannot enter field history and cannot force authoritative siblings back onto the sequence path. - Local result: the production Runtime assembly completed submit plus both approval steps with exactly one committed terminal ActionLog, zero terminal rollbacks, correct before-save and notification Flow effects, and no change to the sum of `_System_FieldHistorySequence`. With only the cutover condition reverted, the same test fails on the pre-existing out-of-order conflict (`FlowInterview.InterviewStatus` below the held WorkItem Status high-water), proving that the regression guard detects the removed failure class. ## 2026-08-28 #4639 remaining producer cutover loop - Frozen Core baseline: `069ee9af0` on branch `codex/4639-field-history-legacy-cutover`. - Reopened topology result: the ServiceManager/REST graph is not dead compatibility code. Product and dialect routes still receive its local Records contract, and production activates Audit V2 on that same `PersistenceService`. Deleting sequence compatibility before migrating this producer would therefore lose real Field History rows. - Root cause inside that graph: `Update` already captures one immutable schema pointer before entering its retry loop and uses it for every attempt, but audit emission discarded that admitted schema and called `GetSchema` again at transaction end. The second cache read was the only reason this producer could not carry an authoritative eligibility decision. - Change: thread the already-admitted schema through the one Update audit bridge shared by Audit V2, the synchronous fallback, and exclusive-sibling derived updates. `recordAuditFieldChanges` no longer performs a metadata lookup. Every resolved real field now carries true/false provenance; an unresolved field remains NULL and fail-closed. No SQL, cache, registry, timestamp, or second metadata authority was added. - TiDB correctness result: the enable/disable concurrency matrix now follows admission semantics in all four orderings. A mutation admitted before enable stays untracked; one admitted before disable stays tracked, regardless of later commit order. Target audit rows always have non-NULL `FieldHistoryEligible` and NULL `HistorySequence`. Whole-record rollback still rolls back business and audit facts and leaves the sequence row unchanged. - Local verification: the full `github.com/nexuscrm/records` package passed, including its real-TiDB matrix. The synchronous fallback test also proves two field rows share one transaction timestamp while carrying true/false and issuing no sequence reservation. - Performance conclusion: steady-state ServiceManager Update removes the sequence upsert plus readback and also removes the redundant metadata-cache lookup from audit emission. It adds no query. Environment RU and latency acceptance remains pending deployment of the frozen candidate; no numerical improvement is claimed from local tests alone. - Next boundary: upgrade every producer binary, drain pre-cutover Audit outbox rows, and deterministically materialize historical NULL eligibility. Only then is it safe to remove the compatibility predicate, metadata-transition interval allocation, and history prelock/replay code.