# Runtime Latency Standard (SFDC-comparable)

This standard defines the success runtime for sales-workflow actions so the
staging inspection can flag when NexusCRM is no longer comparable to the
Salesforce experience it replaces. It powers the `sales > latency` module.

## Method

- **Read paths are measured**, not assumed. They are anchored to the real
  PingCAP production Salesforce org (`pingcap.my.salesforce.com`) over the REST
  Data API, read-only (record GET, SOQL query, describe), 9 iterations each.
- **Write/submit paths are norm-based.** We do not issue test writes against the
  production Salesforce org, so save/submit targets follow Salesforce's own UX
  guidance: < 1s feels instant, 1-3s acceptable, > 3s perceived lag, and the
  Lightning EPT (Experienced Page Time) bar of healthy < 3s / poor > 5s for a
  record page load.
- `max` is the hard bound that fails the inspection. Sampled latency cases use
  p90 against that bound, not p50, so slow-tail regressions stay visible.
  `target` is the SFDC-comparable experience.

## Measured SFDC production read latency (ms, 9 iterations)

| Path | min | p50 | p95 |
|---|---|---|---|
| describe Opportunity | 742 | 1124 | 1964 |
| describe Account | 1050 | 1196 | 1256 |
| GET Opportunity record | 491 | 582 | 737 |
| GET Account record | 524 | 567 | 640 |
| GET Contact record | 499 | 558 | 651 |
| SOQL Opportunity list 50 | 445 | 489 | 592 |
| SOQL Account list 50 | 451 | 490 | 537 |
| SOQL related Opps by Account | 443 | 487 | 590 |
| SOQL COUNT Opportunity | 457 | 494 | 553 |

Captured via `bench_sfdc.mjs` (auth-code + PKCE, read-only). Includes client
network RTT, so these are conservative upper bounds on server time.

## Standard

| Action class | latencyClass | Target | Max (FAIL) | Basis |
|---|---|---|---|---|
| Record GET (API) | `record-read` | 1s | **2s** | SFDC prod p95 0.74s (measured) |
| SOQL list / related / count (API) | `soql-query` | 1s | **2s** | SFDC prod p95 0.59s (measured) |
| Describe (API) | `describe` | 2s | **4s** | SFDC prod p95 1.96s (measured) |
| Click record -> detail visible (cold load) | `record-detail-cold-load` | 3s | **5s** | SFDC EPT healthy<3s / poor>5s |
| Stage path click -> stage saved | `stage-path-save` | 2s | **3s** | SFDC interaction norm |
| Edit record -> Save (Opp / Account / forecast) | `record-edit-save` | 2s | **3s** | SFDC interaction norm |
| Add product (line-item save) | `add-product` | 3s | **5s** | norm, multi-row DML |
| Weekly Update / quick action | `quick-action` | 2s | **3s** | SFDC quick-action norm |
| Create Quote from Opportunity (incl. QLI clone) | `create-quote` | 5s | **8s** | norm |
| Submit for Approval | `submit-approval` | 5s | **8s** | norm |
| Lookup typeahead (parameterized search, API) | `lookup-search` | 1s | **3s** | search-read norm; added after prod parameterizedSearch 29s alert |
| Connect notifications poll (API) | `soql-query` | 1s | **2s** | filtered Notification read; added after prod connect/notifications 20s alert |

## How it is enforced

- API cases assert the measured step's selected latency `atMost` the `max`
  (probe-creation/setup steps are not timed against the bound). When
  `performanceSamples` is present, the selected latency is p90 unless the case
  explicitly chooses another supported statistic.
- Browser cases pass `maxLoadMs` on the `goto` step; the runner times navigation
  and fails when the selected load exceeds it. Sampled browser latency uses p90.
  By default the timing waits for UI-stable (network idle). Record-detail cases
  additionally set `readyResponse` to the detail data request
  (`/ui-api/record-ui/<id>`), which times the load to "the content the user
  opened has arrived" instead of full network silence — a real user already has
  the app shell warm and never blocks on the trailing shell requests
  (notifications, approvals/pending, quickActions polling) that `networkidle`
  would otherwise bill to the measurement.
- Write probes are create-first (never mutate shared fixtures); Create Quote and
  Submit for Approval reuse the same patterns as the functional cases.

## Failure triage standard

Latency failures that move between full watchdog runs must not be handled as
one-off red cases. Before changing code, test data, or thresholds, collect
evidence for the shared root cause:

- Compare the latest run against the previous failed and passing runs: refs,
  deploy mode, reset/bootstrap mode, module concurrency, failed-case matrix,
  request IDs, measured durations, and whether the same cases pass in focused
  reruns.
- When the failure count changes between runs, do not treat the newest matrix as
  the whole truth. First group the historical failures by shared path and shared
  evidence, then decide whether this is the same root cause moving under load, a
  newly exposed root cause, a design flaw, or runner/environment pressure.
- Classify the failure before fixing: build/deploy script issue, L0 generic
  mechanism issue, L1 pack payload/metadata/flow issue, or test data,
  permission, runner, or environment issue.
- State the best current hypothesis in a way watchdog can disprove, then run the
  smallest verification that can answer it: focused rerun for reproducibility,
  small grouped rerun for shared contention, or full inspection for final
  equivalence.
- Treat focused pass evidence as necessary but not sufficient when the failure
  only appears under full-run pressure. A final claim needs either a full-run
  pass-rate comparison or a concrete runner/environment explanation with logs.
- Do not relax a latency bound or add a workaround unless the evidence shows the
  bound is no longer the right product standard. The normal fix is to remove the
  slow path, reduce unnecessary work, or isolate runner/environment noise.

## D3 Test2 comparison and expansion standard

Use the following ladder for Test2 serviceification performance work. It keeps
case discovery, optimization proof, and release-level validation as three
different claims.

### 1. Build a comparable discovery set

- Join cases by the canonical `role/module/function_id` key.
- Require the staging reference case to have passed. Exclude staging skips,
  missing fixtures, and cases whose assertions changed between the two
  inspection checkouts.
- Record both inspection checkout SHAs, the selected case-file paths, and any
  runner change that can alter retries, timing, readiness, or failure
  classification. A historical run with different case semantics is context,
  not a baseline.
- Rank by absolute excess wall time, `Test2 - staging`; use the Test2/staging
  ratio only as a tie-breaker. If a historical staging artifact lacks per-case
  wall time, rank provisionally by Test2 wall time and label the result a seed
  list. Establish the real delta with a current paired run before choosing the
  optimization target.
- Keep deterministic contract, fixture, route, permission, and product-semantic
  failures in the 85-case correctness ledger, but do not call their retry time
  a SQL bottleneck without request-level evidence.

### 2. Establish the current Top-20 baseline

- Start and verify the metrics observer before dispatching the workload. Sample
  at least every five seconds until the last case finishes. The time series must
  include Runtime DB pool `open`, `in_use`, `idle`, `max_open`, `wait_count`, and
  `wait_duration`; Runtime admission `in_flight`, limit, and rejection counters;
  RPC observation/SQL statement/SQL duration deltas; and TiDB active sessions,
  running statements, and lock waits when the environment exposes them. A
  post-run gauge snapshot cannot recover the peak and is not evidence for or
  against connection saturation by itself.
- Run the identical canonical Top-20 list against staging and Test2 with
  `module_concurrency=4`. Record the workflow run IDs, environment URLs,
  inspection checkout SHAs, serving tuples, case totals, and per-case wall
  times. Do not compare a staging 255-case wall time directly with a Test2
  85-case wall time.
- Product latency cases continue to use their declared p90 and hard bound. Case
  wall time is only the ranking signal; it does not replace the user-visible
  latency assertion.
- If the two environments cannot use semantically identical case definitions
  and runner timing, retain the historical staging result as a reference only
  and use Test2 before/after A/B for the optimization decision.
- Select the slowest three to five current Test2 cases with a large staging
  excess and a shared runtime path. Capture request IDs and an aligned profile
  window before changing code.

### 3. Prove an optimization on Test2

- Change one owning layer at a time and keep the environment, case list,
  fixtures, module concurrency, Runtime/sfcrm/Worker tuple, DB pool, admission
  limit, and profiling mode fixed. For a Runtime fix, replace Runtime only.
- The profile must connect case -> request ID -> runtime stage -> SQL statement.
  Record SQL call count, cumulative SQL duration, critical-path duration, and
  the selected case latency before and after. A plausible code diff or focused
  pass by itself is not proof.
- Start the bounded CPU/trace/profile capture before the focused case and prove
  that its window overlaps the request IDs being analyzed. Profiles and peak
  connection/admission gauges cannot be reconstructed after the workload.
- Accept the change only when the predicted unnecessary SQL is absent or
  coalesced, target latency improves outside observed run-to-run noise, all
  target assertions pass, and an explicit semantic control still proves any
  required relationship hydration. If the result is noisy or near the bound,
  repeat or bracket the candidate with another baseline run.
- Reject or revise a candidate that moves work elsewhere, changes returned
  data, introduces object/tenant special cases, or regresses an unchanged
  control. Do not raise thresholds to make it pass.

### 4. Expand in gates

1. Re-run the same Top 20. No new availability/timeout/functional failure is
   allowed; every remaining slow case must have a current classification.
2. Run the exact historical 85-case union. This is the first mixed-load
   regression gate and must use the same canonical list and concurrency.
3. Run the full 255-case inspection on the exact serving tuple. Only this run
   can support the final failure count and the 25-minute stop claim.

Do not append a performance-learned result until the live before/after evidence
passes the applicable gate. Preliminary hypotheses and unvalidated local
results stay in the active investigation, not the durable learned record.

## Performance Learned: ambiguity stop-rule

Repeated watchdog failures that change shape are themselves a signal. If the
failed-case count goes down and then back up, moves across cases that share a
runtime path, or survives more than one plausible fix, stop treating the latest
red case as the problem. At that point the working model is not explanatory
enough, and another patch is likely to become whack-a-mole.

Before the next code change, reset the investigation:

- Build a failure matrix across the last passing run, the last failed run, and
  the newest failed run. Include case names, request IDs, target fixtures,
  durations, HTTP status, concurrency, deploy branch/ref, reset/bootstrap mode,
  and whether focused reruns pass.
- Name the shared path before naming the fix. A group such as Quote/Order/
  approval may be one `createQuote`/after-save/parent-lock root cause, not
  separate Quote, Order, approval, and latency bugs.
- Require a direct evidence chain before patching: watchdog case -> request ID
  -> backend/FlowAudit stage -> code path -> recent diff or long-standing
  design gap. If any link is missing, the next action is instrumentation or a
  targeted rerun, not a behavioral patch.
- Challenge the hypothesis explicitly. Ask what evidence would disprove it, and
  check nearby explanations such as runner pressure, shared fixture contention,
  branch drift, stale deploy state, or a previous fix exposing the next layer.
- Prefer removing unnecessary work or narrowing a generic mechanism over
  adding a case-specific bypass. A fix that cannot explain why earlier runs
  passed and later runs failed is not yet a root-cause fix.

The stop-rule is triggered for the current test2 investigation: the apparent
movement from one failed performance case to five failed cases was not evidence
for five unrelated bugs. It was evidence that the remaining shared model was
incomplete. The correct response is grouped root-cause analysis with logs and
branch diffs, then one L0/L1-bounded fix, followed by a targeted watchdog
validation that can falsify the hypothesis.

## 2026-06-28 test2 watchdog concurrency evidence

After the main/main deploy on test2, the same heavy subset was rerun with
different runner module concurrency to separate product regressions from
inspection pressure:

| Run | Selection | module_concurrency | Result | Failed case |
|---|---:|---:|---:|---|
| `28317904610` | orders/quotes, campaign-member, sales latency, SDR latency, lead performance | 4 | 37/38 | `sdr/latency/parameterized-search-lookup-latency-api` at 4117ms/3000ms |
| `28318219907` | same subset | 2 | 37/38 | `sales/latency/record-detail-load-account-latency` at 5362ms/5000ms |
| `28318059972` | same subset | 1 | 38/38 | none; search retried from 3569ms to 2935ms |

Read-only TiDB diagnostics from the test2 runner showed `Lead` has about
393k rows, `Lead` has an available TiFlash replica, and the parameterized
search plan uses `mpp[tiflash]` with a full table scan over the pack-declared
Lead search defaults (`Id`, `Name`, `FirstName`, `LastName`, `Email`). The
same plan varied from roughly 0.99s to 3.55s without metadata changes.

Current classification: test runner/environment pressure exposing a tail
latency edge in the generic L0 search path, not an L1 payload regression. A
formal performance comparison should run with module concurrency 1 unless the
runner grows a two-phase model that runs functional coverage in parallel and
latency probes after background work has settled.

## 2026-06-28 test2 low-concurrency full-run evidence

The first low-noise full Env Inspection after the test2 main/main deploy was
run as `28318331173` with `module_concurrency=1`. It improved functional
coverage to green, but did not yet match the baseline performance pass rate:

| Area | Passed | Failed | Skipped |
|---|---:|---:|---:|
| Functional | 171 | 0 | 12 |
| Performance | 34 | 4 | 1 |
| Total | 205 | 4 | 13 |

All remaining failures were in `sales/latency` and clustered in the same
10:05-10:07 UTC window:

| Case | Evidence |
|---|---|
| `record-detail-load-account-latency` | browser page load 7401ms > 5000ms |
| `create-quote-from-opportunity-latency-api` | request `292d3f8e-617e-4066-ba5d-976fc8a14d88`; CloudFront 504 after ~30s; backend logged Quote child-clone insert failing on lookup validation after `context canceled` |
| `submit-for-approval-latency-api` | request `41fcca6b-3f06-4f75-851d-e1fc1c630c38`; Account create hit auto-number reserve `context canceled`; backend latency ~30023ms |
| `record-read-latency-api` | request `bc6e6dc5-c211-4e86-939c-8ac1266ea906`; first Opportunity GET failed after ~30037ms with `context canceled`, then retried successfully in ~3793ms |

Active test2 backend diagnostics from `28319093183` showed the service was not
restarting during this window. The same log window also contained connector
background deadline symptoms:

- `connector live queue stats sample failed: context deadline exceeded`
- `connector inbound worker reconcile: connector inbound worker projection version query: context deadline exceeded`
- `connector inbound worker reconcile: read projection-enabled profile count for hubspot-crm-connector ... context deadline exceeded`
- later, `refresh connector inbound projection snapshots ... upsert ... context deadline exceeded`

Current classification: functional behavior is now green; the remaining gap is
a shared performance/availability tail-latency window. The evidence rules out a
single L1 case payload regression, but does not yet prove whether the root cause
is the generic L0 connector-background convergence loop, test2 DB pressure, or
their interaction. Next fixes must preserve this distinction: either remove
unnecessary steady-state background work from L0 with code evidence, or classify
the slot as runner/environment noise with DB/service evidence.

## 2026-06-28 after connector polling fix

The L0 branch `codex/watchdog-l0-main-fixes` changed connector inbound runtime
reconcile from a one-second unconditional convergence refresh to a lifecycle
version-gated reconcile with a 30-second default tick. A code-only deploy to
test2 succeeded as `28319345991` in about four minutes.

The follow-up full Env Inspection was `28319439921`:

| Area | Passed | Failed | Skipped |
|---|---:|---:|---:|
| Functional | 169 | 2 | 12 |
| Performance | 36 | 2 | 1 |
| Total | 205 | 4 | 13 |

Failure grouping:

| Group | Cases | Evidence | Current classification |
|---|---|---|---|
| Standard action tail latency | `admin/orders-quotes/order-create-clones-quote-line-items-api`, `admin/orders-quotes/order-create-field-inheritance-api` | `createOrder` request `b72252f4-3436-4173-8114-3e5fbefb4ca0` and `createQuote` request `6aaa47b4-85e6-4824-807e-a96c4a757c0c` both hit CloudFront 504 at about 30s. Other successful `createQuote/createOrder` calls in the same run took roughly 14s-24s. | Not yet classified as L1 or L0. The shared evidence points at the standard action / Flow / record-create path rather than either case's assertion payload. Next step is backend log segmentation for these request IDs before changing payload or platform code. |
| Moving performance tail | `sales/latency/record-detail-load-opportunity-latency`, `sdr/latency/parameterized-search-lookup-latency-api` | Opportunity page load was 6252ms/6000ms. Broad Lead `parameterizedSearch` retried at 3033ms and 6134ms against a 3000ms threshold. | Still consistent with the earlier concurrency-sensitive search/page-load tail. Do not loosen thresholds yet; compare with low-concurrency full run after the standard action root cause is understood. |

Learning: the connector polling fix reduced one source of constant background
DB work, but it did not make the formal full run equivalent to the baseline.
The next root-cause boundary is narrower: if standard action logs show repeated
generic engine overhead, fix L0; if they show inefficient CRM action assembly,
move the correction to L1; if they show slot-level DB deadlines across unrelated
paths, classify as runner/environment with evidence.

## 2026-06-28 standard action Flow audit evidence

The follow-up diagnostic run `28319934111` added `_System_FlowAuditLog`
segmentation for the standard actions that failed in `28319439921`. The
evidence points to the generic record-create path rather than the two watchdog
case payloads:

| Flow element | Type | Status | Rows | Avg ms | Max ms |
|---|---|---|---:|---:|---:|
| `CreateOrder` | `RecordCreate` | success | 25 | 11504.6 | 24897 |
| `CreateQuoteLineItem` | `RecordCreate` | error | 2 | 11207.0 | 22173 |
| `CreateQuote` | `RecordCreate` | error | 5 | 3302.6 | 12635 |
| `CreateQuote` | `RecordCreate` | success | 65 | 3168.3 | 7452 |
| `CreateQuoteLineItem` | `RecordCreate` | success | 581 | 715.6 | 5626 |
| `CreateOrderItem` | `RecordCreate` | success | 212 | 477.3 | 2262 |

The slowest failed `CreateQuoteLineItem` and failed `CreateQuote` rows both
ended with `failed to validate lookup target (caused by: context canceled)`.
Backend request `292d3f8e-617e-4066-ba5d-976fc8a14d88` also showed
`prepareInsertRequest.validateNonPolymorphicLookups` spending 8097ms on
`Quote_Product__c` before the request hit the 30s edge.

Code readback found that `Std_CreateQuoteFromOpportunity` and
`Std_CreateOrderFromQuote` are L1 Flow metadata, but every `RecordCreate`
element calls the L0 single-record `PersistenceService.Insert`. That Insert
runs the same non-polymorphic lookup access/existence owner at multiple points
(`prepareInsertRequest`, before-save prelockless, before-save prelocked), and
atomic standard-action flows can create many related rows in one transaction.

Classification: L0 generic mechanism issue. The correct fix is not to loosen
watchdog thresholds or encode Quote/Order facts in L0. The L0 branch now adds a
request/transaction-scoped success-only memoization for non-polymorphic lookup
validation, with write invalidation and fresh cache per transaction retry. This
keeps `validateNonPolymorphicLookups` as the single owner/read path while
removing repeated DB/RLS proof work inside the same atomic write.

## 2026-06-28 after lookup-validation memoization

The L0 branch `codex/watchdog-l0-main-fixes` was deployed to test2 by
`28320281375` using a code-only deploy (`skip_bootstrap=true`,
`reset_branch=false`, `ensure_indexes_workers=8`). Deploy completed in 3m30s,
so the fix did not reintroduce the earlier build/deploy failures or slow
bootstrap path.

The automatically triggered Env Inspection `28320369247` ran with
`module_concurrency=4` and produced:

| Area | Passed | Failed | Skipped |
|---|---:|---:|---:|
| Functional | 171 | 0 | 12 |
| Performance | 37 | 1 | 1 |
| Total | 208 | 1 | 13 |

The previous standard-action 504 group did not reproduce. The only remaining
failure was `sdr/latency/parameterized-search-lookup-latency-api`, a broad
Lead `parameterizedSearch` probe for `"sony pictures"`:

| Attempt | Request id | Status | Duration | Threshold |
|---:|---|---:|---:|---:|
| 1 | `53725815-5702-4fb8-afa3-c9d09ef9848c` | 200 | 3282ms | 3000ms |
| 2 | `0001a698-2c3a-494e-9001-d8c416555713` | 200 | 5149ms | 3000ms |

Classification: the standard action failure class is fixed or at least no
longer reproduced under the same auto-run pressure. The remaining gap is the
known L0 generic search/read performance tail under module concurrency 4, not an
L1 payload issue. The code path is L1 dialect request assembly into neutral
`SearchWithSpec`, then L0 persistence emits the substring-search
`LOWER(col) LIKE '%term%'` path with `tidb_enforce_mpp=ON` on the base table
when it is safe. Prior test2 evidence shows the same plan on about 393k Lead
rows varying from roughly 0.99s to 3.55s without metadata changes, and the same
subset passed at `module_concurrency=1` (`28318059972`) while failing at
`module_concurrency=4` (`28317904610`).

Follow-up verification: full low-noise Env Inspection `28320741829` ran with
`module_concurrency=1` and produced the same aggregate pass rate, but moved the
single remaining failure:

| Area | Passed | Failed | Skipped |
|---|---:|---:|---:|
| Functional | 171 | 0 | 12 |
| Performance | 37 | 1 | 1 |
| Total | 208 | 1 | 13 |

The SDR search case passed in the low-noise run (`2283ms / 3000ms`), so the
latest evidence no longer supports treating search as the only active failure.
The remaining failure was
`sales/latency/record-detail-load-account-latency`, where the Account detail
browser load took `6315ms / 5000ms`. The screenshot shows the Account page
rendered successfully, and the matching Contact and Opportunity detail probes
passed in the same run, so the current classification is performance-only:
record detail load path or test2/browser-runner pressure, not a functional L1
payload break.

Learning: the lookup-validation memoization removed the repeated standard
action 504 class, and low-noise execution removed the SDR search tail. The next
root-cause step must compare Account detail against the Contact/Opportunity
detail traces from the same run and the earlier failing runs before changing
thresholds or code. The useful question is now narrower: why is Account detail
still over the SFDC EPT-derived 5s bound when sibling detail pages pass?

Follow-up focused proof: `28321501289` ran only
`sales/latency/record-detail-load-account-latency` on test2 with
`module_concurrency=1` and passed. The warm measured Account detail load was
`4400ms / 5000ms`. This means the failure is not a deterministic functional or
case-authoring break; it is an Account record-detail tail-latency path that
shows up under broader inspection pressure.

Read-only endpoint probes using the same sales storage state narrowed the path:

| Endpoint | Account | Contact | Opportunity |
|---|---:|---:|---:|
| `/ui-api/record-ui/<id>` direct probe | 6073 / 5243 / 3305ms | 2058 / 2319 / 2185ms | 3642 / 3679 / 3439ms |
| `/ui-api/layout/<object>` direct probe | 3196 / 2382 / 3028ms | 1080 / 1243 / 1224ms | 2536 / 2342 / 2495ms |
| raw `/sobjects/<object>/<id>` direct probe | 1901 / 1618 / 1639ms | 1738 / 1510 / 1547ms | 2180 / 2267 / 2294ms |

The failed browser timing is therefore not caused by a frontend render wait or
`related-list-counts`; the runner waits for the `record-ui` response, and the
codebase currently has no broad `aria-busy` markers that would add a hidden
3s wait. The best current classification is **L1 dialect/UI API payload
assembly performance tail** over generic mechanisms: Account layout has eight
related lists and the record-ui path serializes record read before layout
resolution because record type comes from the record. A suspected lookup-display
serial bottleneck was checked and deprioritized: Account has only four real
non-null lookup fields, fewer than Contact and Opportunity.

Do not relax the 5s bound for this case. If another code fix is needed, the
likely safe shape is a generic UI API assembly optimization (for example,
overlapping layout resolution with record read when the resolved record type is
the default, or adding a correctly invalidated transformed-layout cache), not an
Account-specific branch.

## 2026-06-28 formal test2 run after Account focused pass

The next formal low-noise full Env Inspection on test2 was `28322087268`. It was
confirmed to target `test2` and produced:

| Area | Passed | Failed | Skipped |
|---|---:|---:|---:|
| Functional | 171 | 0 | 12 |
| Performance | 37 | 1 | 1 |
| Total | 208 | 1 | 13 |

This is better than the target baseline run in aggregate, but the remaining
failure is a case the baseline passed, so it still needs root cause before
claiming equivalence:

| Case | Current evidence | Baseline evidence |
|---|---|---|
| `sales/latency/create-quote-from-opportunity-latency-api` | `createQuote` returned HTTP 200, but request `778886df-1725-430e-9b89-bb981ba2c625` took `24529ms / 18500ms`. The target Opportunity was `0065g000002Hn6gAAC` (`CC-FY24Q4 TiDB Cloud`) with 23 `OpportunityLineItem` rows and 1 `Opportunity_Product__c` row. | Baseline `27805400553` logged the same case passing at `6561ms`. |

Historical comparison prevents treating this as a new isolated case. Earlier
standard-action failures (`28318331173`, `28319439921`) moved between
`createQuote`, `createOrder`, and approval, and diagnostics showed slow
`RecordCreate` elements plus lookup-validation timeouts. The lookup-validation
memoization fixed the 30s/504 class, but this run exposed the next shared
fanout cost: every Flow `RecordCreate` still calls single-record L0 `Insert`.

Code comparison found that the Flow loop and `ChildCloneRules` executor are not
newly divergent from `old-main-for-prod`; the notable main-only cost is #3229's
final-create-image reload. The intent of #3229 is correct: when post-insert
side effects can rewrite the created row, outward create images must come from
the committed DB row. The implementation was too broad because
`hasOutbox => reload` made ordinary record creates do an extra physical
`FindOne`. In createQuote's 23-line-item fanout, that adds one extra DB read per
line item even though `QuoteLineItem` and `Quote_Product__c` have no
`DefaultChildRows`, `ChildCloneRules`, `ExclusiveSiblingFields`, or
record-lifecycle capability in the final L2 object metadata.

Classification: L0 generic mechanism issue in the final-create-image read
condition. The fix on `codex/watchdog-l0-main-fixes` (`bc278e69c`) keeps the
#3229 correctness contract but removes the overbroad outbox trigger:

- reload final create images only when metadata says post-create side effects
  can rewrite the created row (`DefaultChildRows`, `ChildCloneRules`, or an
  after-create lifecycle capability);
- keep ordinary create outbox payloads on the already-prepared image;
- preserve final-image coverage for the mutation-risk case;
- add explicit tests proving outbox alone does not trigger `FindOne` /
  `FindMany`.

Local verification for the fix:

- `go test ./nexuscrm-records`
- `go test ./nexuscrm-automation ./nexuscrm-events ./backend/internal/application/services ./backend/pkg/serverruntime`
- `git diff --check`

Next verification is a code-only test2 deploy from the L0 branch, followed by a
full low-noise Env Inspection. No reset, backfill, or L1 payload change is
needed for this hypothesis.

## 2026-06-28 auto inspection after final-image fix

The follow-up auto Env Inspection was `28323259742` on test2. It produced:

| Area | Passed | Failed | Skipped |
|---|---:|---:|---:|
| Functional | 168 | 3 | 12 |
| Performance | 37 | 1 | 1 |
| Total | 205 | 4 | 13 |

The performance regression itself improved: `createQuote` was `2393ms /
18500ms`, confirming the final-create-image fanout fix removed the 24s tail.
The remaining failures were all in the Quote/Order/approval group:

- `sales/latency/create-quote-from-opportunity-latency-api`
- `sales/opportunity/order-create-clones-quote-products-api`
- `sales/opportunity/quote-approval-submit-recall-api`
- `sales/opportunity/quote-approval-state-propagation-api`

The shared error evidence is not latency: the `createQuote` Flow failed while
assigning `QuoteNumber`:

`autonumber: AssignOne: render: autonumber: counter overflow: field counter 100
exceeds width=2 max=99`

The approval cases showed the same sequence at counters `101` and `102`.
Initial suspicion was an L2 numbering payload problem because PingCAP L2
declares `QuoteNumber` as `{Quote.Account_Country_Code__c}-QT-{YY}{MM}{DD}{00}`
with `per_prefix` scope. Re-checking the Salesforce Auto Number semantics
changed the classification: the zero count is a minimum padding width, not a
maximum representable width. Counter `100` should render as `...100`, not fail.

Classification: L0 platform generic mechanism bug. The L2 format is a tenant
numbering fact and should stay in L2; L0 must render Salesforce-compatible
autonumber display formats generically. The fix on
`codex/watchdog-l0-main-fixes` (`dc31290ec`) removes runtime overflow rejection
and treats counter width as minimum zero padding. It keeps parser validation for
declared token width and keeps negative/invalid counters fail-closed. Local
verification:

- `go test ./nexuscrm-automation/autonumber`
- `go test ./nexuscrm-automation ./nexuscrm-records ./backend/internal/application/services ./backend/pkg/serverruntime`
- `git diff --check`

Next verification is intentionally **not** a reset deploy: the current test2
counter state has already crossed 100, so it is the fastest evidence that the
runtime now handles the real failed condition.

## 2026-06-28 parent-lock root cause after autonumber fix

After the autonumber fix, the formal test2 Env Inspection `28323847024`
improved the QuoteNumber failure class but still failed in the same
Quote/Order/approval group:

| Area | Passed | Failed | Skipped |
|---|---:|---:|---:|
| Functional | 169 | 2 | 12 |
| Performance | 35 | 3 | 1 |
| Total | 204 | 5 | 13 |

The failures clustered around `createQuote` and downstream `createOrder`:

- `sales/opportunity/order-create-clones-quote-products-api`: `createOrder`
  504 after a prior `createQuote` succeeded slowly.
- `sales/opportunity/quote-approval-submit-recall-api`: `createQuote` 504.
- `sales/latency/create-quote-from-opportunity-latency-api`: `createQuote`
  504.
- two read/search tail-latency failures also appeared in the same noisy full
  run.

The low-concurrency sales-only run `28324241305` narrowed the group to two
`createQuote` failures: one functional 504 and one successful but slow
`createQuote` at `24233ms / 18500ms`. The diagnostic run `28324604433` provided
the decisive L0 evidence:

- FlowAudit: `CreateQuote` and `CreateQuoteLineItem` `RecordCreate` had slow
  rows, but the concrete backend stage failures were on `Quote` update.
- Backend logs repeatedly showed `Quote` `Update`
  `prelockInitialUpdateParents` taking about 10.4s and failing with
  `failed to lock parent row Opportunity/0065g000002Hn6gAAC`.
- The L1 metadata showed legitimate Quote self-updates: after-save flows update
  fields on `$Record`, and a before-save Quote flow can run on update. These
  are not reasons for L0 to encode Quote or Opportunity facts.

Code readback found the design mistake in L0 records parent-lock ownership:

- `prelockInitialUpdateParents` asked the generic non-polymorphic lookup path
  to collect targets from both `oldRecord` and the merged candidate.
- For a Quote self-update that did **not** change `OpportunityId`, this still
  locked the existing `Opportunity` parent.
- A separate pre-flow verifier then required every old lockable lookup parent
  to have been prelocked, so even if initial collection were narrowed, the
  retry loop would reintroduce the same over-lock.
- `old-main-for-prod` has the same broad shape in the old directory layout, so
  this is not a missing port to copy back. It is a long-standing generic
  mechanism bug exposed by the new main/test2 workload and stricter full-run
  pressure.

Correct boundary:

- L0 records owns generic lock coordination.
- L1 owns the Quote/Opportunity flows and fields.
- Generic non-polymorphic lookup locking should lock old/new parents only when
  the lookup field actually changes, or when a final before-save image changes
  it and fail-closed retry discovers the missing lock.
- Exclusive sibling and rollup continue to own their own old-parent locks. The
  fix explicitly moved exclusive-sibling partition old-parent locking into the
  exclusive-sibling collector so it no longer depends on the generic lookup
  path as a side effect.
- Recycle-bin restore keeps its restore-specific parent validation owner:
  after a before-save repair repoints a deleted row away from a hard-deleted
  ancestor, the shared update finalizer must not resurrect or lock the
  historical deleted parent.

The L0 branch `codex/watchdog-l0-main-fixes` now implements that boundary:

- non-polymorphic lookup update/prepared-update prelock collects only changed
  lookup parent targets;
- the pre-flow "all old lookup parents must be prelocked" verifier was removed;
- final update validation now verifies changed lookup old/new parents so
  before-save lookup mutations still fail closed and retry with the complete
  lock set;
- exclusive sibling initial prelock still locks the partition parent;
- restore finalization validates only the final repaired lookup image.

Local verification:

- `go test ./nexuscrm-records`
- `go test ./nexuscrm-automation ./backend/internal/application/services ./backend/pkg/serverruntime`
- `git diff --check`

Next verification should be efficient: code-only deploy the L0 branch to test2,
then run the focused sales low-concurrency subset first. If `Quote` update no
longer logs `prelockInitialUpdateParents` waits on Opportunity, proceed to the
formal low-noise full watchdog. If failures move again, trigger the ambiguity
stop-rule and rebuild the failure matrix before another patch.

### 2026-06-28 follow-up: hot source Opportunity fixture

After deploying the changed-lookup L0 fix to test2, Env Inspection run
`28325622616` still failed `6` cases:

- admin `order-create-clones-quote-line-items-api`
- admin `order-create-field-inheritance-api`
- admin `metadata-fanout-latency`
- sales `order-create-clones-quote-products-api`
- sales `record-detail-load-opportunity-latency`
- sales `create-quote-from-opportunity-latency-api`

Diagnostic run `28326158137` showed the common write-path root:

- backend stage timing for request
  `4a310c65-5d1f-4315-8316-0fee15c687df` spent about `10030ms` in
  `Quote Insert prelockFinalInsertParents`, failed to lock
  `Opportunity/0065g000002Hn6gAAC`, retried, and returned HTTP 200 only after
  about `30029ms`; the inspection client had already observed a 504/timeout;
- FlowAudit grouped the same pressure under slow/error `CreateQuote`,
  `CreateQuoteLineItem`, and `CreateOrder` record-create elements;
- the failed and slow cases all used the same seeded source Opportunity name
  (`CC-FY24Q4 TiDB Cloud`) or fixture Opportunity id, so unrelated case modules
  were contending on one hot parent record.

Boundary conclusion:

- L0 parent locking is doing what the current metadata asks it to do; old
  `old-main-for-prod` has the same ControlledByParent/sharing-parent lock
  shape, so this was not a missing repo-split port.
- L1/L2 metadata still needs separate cleanup because the L2 overlay shadows
  some L1 standard relationship facts, but changing `Quote.OpportunityId` from
  `masterdetail` to `reference` alone would not remove the parent lock while
  the object remains `ControlledByParent` with `OpportunityId` as sharing
  parent.
- The immediate watchdog failure class is a test-data/runner isolation issue:
  write cases that create Quote/Order and clone child rows must not share the
  same source Opportunity under module concurrency.

Fix direction now applied in the L2 branch:

- the affected Quote/Order write cases self-seed their own source Opportunity
  from the module-scoped `payloads.opportunityCreate` account pool;
- clone coverage is preserved by creating the required source
  `Opportunity_Product__c` or `OpportunityLineItem` inside the case before
  invoking `actions/standard/createQuote`;
- the latency case now measures `createQuote` against an isolated source
  Opportunity instead of the shared `CC-FY24Q4 TiDB Cloud` parent.

Lesson: when the failed case count grows or moves after a plausible L0 fix,
stop treating each timeout as a separate defect. Build a matrix by request id,
backend stage, parent record id, and case source data. A repeated parent id is a
strong signal for fixture or runner isolation, while a repeated stage without a
shared parent points back to L0 mechanism.

### 2026-06-28 correction: parent-lock optimization overcut

PR `NexusCRM#3236` gate review caught that the first parent-lock L0 fix had
overcut the generic lock contract. The earlier note above correctly recorded
the evidence (`Quote` update blocked on the shared
`Opportunity/0065g000002Hn6gAAC` parent), but the conclusion was too broad:
"unchanged lookup parent" is not enough to skip locking.

Corrected root cause split:

- the watchdog timeout was driven by hot shared source data, fixed in L2 by
  self-seeding Quote/Order source Opportunities;
- the L0 mechanism must still preserve parent-before-child ordering for
  lock-required lookup semantics: master-detail, cascade, ControlledByParent
  sharing parent, exclusive sibling partition, and rollup-feeding parents;
- only unchanged plain optional lookup parents can be skipped safely.

What failed in the earlier design: it used a performance symptom to weaken a
correctness invariant. The local tests were also changed to assert the weaker
behavior (`DoesNotRetryUnchangedLookup...`), while `origin/main` and
`old-main-for-prod` both kept the fail-closed retry expectation when locked old
lookup parents differ from the preload.

Correct fix now applied in the L0 branch:

- update/prepared-update initial prelock includes unchanged lock-required lookup
  parents and still skips unchanged plain optional lookups;
- locked-old lookup parents are verified after the row is locked, so stale
  preload data triggers a retry with the correct parent lock set;
- final before-save validation uses the same update parent collector, keeping
  before-save lookup mutations fail-closed.

Local verification:

- `go test ./nexuscrm-records -run
  'TestPrelockInitialUpdateParents|TestLookupParent.*RetriesWhenLockedOldParentDiffersFromPreload|TestExclusiveSiblingUpdatePrelocksLookupParentWhenLockedOldWinnerDiffersFromPreload'`
- `go test ./nexuscrm-records`
- `go test ./nexuscrm-automation ./backend/internal/application/services
  ./backend/pkg/serverruntime`
- `git diff --check`

### 2026-06-28 after Quote/Order fixture isolation

PR `#3434` isolated the Quote/Order source records and preserved child-clone
coverage by self-seeding each affected case. Focused verification passed all
six touched cases on test2:

- `28326825909`: `sales/latency/create-quote-from-opportunity-latency-api`
- `28326829194`: `sales/opportunity/quote-create-line-item-clone-count-api`
- `28326892159`: `sales/opportunity/order-create-clones-quote-products-api`
- `28326923097`: `admin/orders-quotes/order-create-clones-quote-line-items-api`
- `28326952273`: `admin/orders-quotes/order-create-field-inheritance-api`
- `28326981587`: `sales/opportunity/quote-create-clones-opportunity-products-api`

Formal full-run comparison after the fix:

| Run | Mode | Total | Functional | Performance | Failed cases |
| --- | --- | --- | --- | --- | --- |
| `28327024404` | default module concurrency | 207 passed / 2 failed / 13 skipped | 171 passed / 0 failed / 12 skipped | 36 passed / 2 failed / 1 skipped | `record-detail-load-opportunity-latency`, `parameterized-search-lookup-latency-api` |
| `28327579142` | module concurrency 1 | 208 passed / 1 failed / 13 skipped | 171 passed / 0 failed / 12 skipped | 37 passed / 1 failed / 1 skipped | `record-detail-load-account-latency` |

The functional trend is good: the Quote/Order failures disappeared and the full
suite has zero functional failures. The remaining failures are performance-only
and move inside two latency families:

- `parameterized-search-lookup-latency-api` failed under default concurrency
  (`11693ms / 3000ms`) but passed in the low-noise full run (`2300ms /
  3000ms`), which keeps the classification as concurrency-sensitive generic
  search/DB tail latency rather than L1 payload drift.
- Record-detail browser latency moved from Opportunity (`8056ms / 6000ms`) in
  the default full run to Account (`5288ms / 5000ms`) in the low-noise full
  run. A focused rerun `28328253765` of only
  `sales/latency/record-detail-load-account-latency` still failed at
  `5007ms / 5000ms`, so Account detail is a reproducible near-budget tail, not
  just default-concurrency noise.

Baseline `27805400553` failed four cases overall and one performance case
(`lead-multifield-like-search-latency-api`). Current test2 is better on total
and functional failures, but strict performance pass count still needs care:
the same Lead multi-field red-line probe is now intentionally skipped, while
`record-detail-load-account-latency` is the only hard performance failure left.

Next root-cause rule: do not relax `record-detail-load-account-latency` just
because it misses by `7ms` in the focused run. First collect evidence for where
the second warm `goto` spends time: backend `record-ui`/metadata/query latency,
frontend render/aria-busy wait, or runner timing. Only then decide whether the
correct fix belongs to L0 runtime, frontend detail rendering, or the
performance harness.

### 2026-06-28 Account record-ui latency root cause

The focused Account reruns added browser timing and backend diagnostics:

- `28328491592` passed the focused Account case, but the measured second
  `record-ui` response still spent about `3737ms` in backend readiness.
- `28328639259` failed the same focused case at `5553ms / 5000ms`; browser
  timing showed `goto=201ms`, `ready=5283ms`, `busyWait=20ms`, HTTP `200`.
- Diagnostic run `28328757982` tied the failing request to backend request id
  `9836f63a-a938-4922-aa47-d0a25d7fd792`: the Account
  `/ui-api/record-ui/0015g000002hX2VAAU?type=Account` request returned 200 in
  about `4134ms`, while permission diagnostics recorded
  `perm_effective_access_calls=11`, `perm_effective_access_fields=1323`,
  `perm_field_db_fallbacks=104`, and `perm_field_db_ms=5918ms`.

Classification:

- this is not a frontend render or runner timing problem; `aria-busy` wait was
  only about `20ms`;
- this is not a L1 payload correctness failure; Account simply exercises a
  large standard record-ui surface with many related objects/fields;
- the root mechanism is L0 effective-access cost: page-level `record-ui`
  fans out across object-info/layout/resolved related-list targets, and field
  access falls back to repeated point `LoadEffectiveFieldPermission` calls when
  the request does not already have a complete effective-field snapshot.

L0 fix direction applied in `codex/watchdog-l0-main-fixes`: extend the
request-scoped access-input cache with a successful bulk
`ListEffectiveFieldPermissionsForUser` snapshot. That keeps the old process
cache miss behavior fail-closed, preserves explicit FLS deny semantics, and
turns the Account page's repeated field permission DB fallbacks into one
generic request-level bulk load.

### 2026-06-28 test2 deploy-wrapper divergence

The first verification deploy after the Account record-ui fix did not reach Env
Inspection. Runs `28329078197` and `28329292682` failed during setup apply
before any case execution with a source-backed dependency preflight mismatch:
`pingcap-hubspot-companion` reapply saw the installed `hubspot-crm-connector`
payload hash at an older value than the current app-packs source.

The initial L2 deploy fix added a pre-apply reapply step for active official
dependency packs in `deploy_staging_env.py`, but `28329292682` still did not log
that step. Code/log comparison showed the test2 workflow enters
the former Test application deploy adapter, a test-specific copy of the apply orchestration that
reuses staging helpers but owns its own sequence. The helper existed, but the
test wrapper never called it.

Classification: build/deploy script issue in L2. The root cause is wrapper
sequence divergence, not a L0 pack-hash mechanism bug and not a test case
failure. The fix is to call the same active-official-dependency reapply helper
from the test apply flow after active source rebind and before pre-apply
metadata sync, with an AST regression test that locks this order:

1. `rebind_active_runtime_pack_sources`
2. `reapply_active_official_dependency_packs_before_setup_apply`
3. `sync_runtime_pack_metadata_before_setup_apply`

Lesson: if a submitted fix is expected to produce a deploy log line and the
next run does not show it, first verify the workflow entrypoint and copied
orchestration paths. Do not keep patching the helper until the actual caller is
proven to execute it.

### 2026-06-28 createQuote recurrence and standard-action fanout

Run `28329765578` moved the suite backward from zero functional failures to
`206 passed / 3 failed / 13 skipped`: two functional `createQuote` cases timed
out at the 30s edge, and one performance case failed because
`GET /services/data/v63.0/actions/standard` took `7127ms / 4000ms`.

The `createQuote` failures were:

- `quote-create-tidb-cloud-standard-action-api`, request
  `b8b4556f-711b-4a48-8e2d-b8e74846760b`;
- `quote-approval-submit-recall-api`, request
  `1584ef36-6cd4-4527-bdb2-80d10b0d845f`.

Both used the same source Opportunity `0065g000002Hn6gAAC`
(`CC-FY24Q4 TiDB Cloud`). The diagnostic run `28330163032` showed the matching
backend symptom in FlowAudit: `CreateQuote` record-create errors included
`failed to lock parent row Opportunity/0065g000002Hn6gAAC` with TiDB
`Error 1205: Lock wait timeout exceeded`; successful `CreateQuoteLineItem`
fanout rows also had a high tail. This is the same class as the earlier
standard-action timeout work, but the uncovered callers were different.

Classification: L2 test-data isolation gap. The platform lock/write semantics
are not the first fix target. The earlier self-seed change covered the visible
latency and order paths, but two remaining write paths still used the shared
fixture Opportunity. Correct fix: every watchdog case that invokes
`actions/standard/createQuote` for a write path must create its own source
Opportunity before invoking the action. Read-only related-list cases may keep
using shared fixture rows because they do not contend on parent updates.

The performance failure was `metadata-fanout-latency` step `actions-standard`,
request `b18bf385-7c3e-44b8-bd8a-1d0ac7f39cb0`. Baseline
`27805400553` ran the same step at `2364ms`. The tempting shortcut is to turn
standard-action list caching back on, but release history already marks that as
unsafe: the old cache work was followed by metadata-revision and permission
revision gates, then disabled because cross-request action visibility/schema
facts can go stale. That means the safe root-cause target is the per-request
read shape, not a cross-request TTL.

Code inspection found the L0 generic discovery path doing an N+1 metadata
fanout. After listing active actions and filtering permissions, each permitted
action separately loaded:

- `_System_ActionInputGuard` rows for discovery visibility;
- `_System_ActionSchema` rows for input/output descriptors;
- `_System_ActionFieldDefault` rows for input default summaries.

Classification: L0 platform mechanism performance issue. The fix direction is
a neutral single-request bulk read through the existing StandardActionReader
port implementation: filter permissions first, then load schemas, defaults, and
guards for the permitted action IDs in three batch queries, preserving live
permission checks and the disabled cross-request cache. The implementation also
keeps the single-action readers for invocation/preview paths and for older test
fakes.

Next suspicion if the focused run still misses the threshold: preview support
is still checked per action via `StandardActionCreatePreviewSupported`, which
walks Flow metadata and child flows. Do not re-enable TTL caching as the next
move; first collect request-level evidence that preview-support checks, not the
schema/default/guard reads, dominate the remaining latency.

Lesson: when a recurring failure moves, scan all callers of the shared operation
before changing the operation itself. Repeated failures after one or two fixes
are a signal that the test surface still has a common dependency, or that the
read path has an architectural fanout, not that the latest failed case needs a
bespoke patch.

## 2026-06-28 approval payload and Lead object-info tail

After the L0 standard-action bulk-discovery fix and the remaining createQuote
self-seed changes were deployed to test2, Env Inspection run `28330832503`
reported `206 passed / 3 failed / 13 skipped`:

| Bucket | Failures | Evidence | Classification |
| --- | --- | --- | --- |
| Quote approval functional | `quote-approval-submit-recall-api`, `quote-approval-state-propagation-api` | `createQuote` succeeded in `2152ms` / `2863ms`, but submit approval returned HTTP 200 with `success=false`, `instanceStatus=Error`, and `record does not meet the entry criteria for this approval process`. | L2 test-data / payload correctness |
| Lead metadata performance | `lead-metadata-bootstrap-latency-api` | `Lead/describe` was `2787ms / 4000ms`; `ui-api/object-info/Lead` was `3432ms / 3000ms`, request `5ce13d21-8604-4318-aad6-47a53d637c30`. | L0 cold-start / environment tail under investigation |

What did:

- Compared the failed approval cases with the passing
  `quote-approval-entry-criteria-routing-api` evidence. The active NA OP Service
  criteria require `Opportunity.Region__c = NA`, Quote product family not
  containing `Cloud`, rebate/partner margin null or zero, and referral fee rate
  `<= 5`.
- Read the L2 `createQuote` field mapping. It copies
  `Opportunity.Product_Family__c` into `Quote.Product_Family__c`.
- Found the self-seeded approval Opportunities did not set
  `Product_Family__c`, so the created Quotes matched neither the NA OP
  non-Cloud process nor the NA DBaaS Cloud process.
- Verified the fixed shape directly against test2 with the admin token:
  source Opportunity on an NA account, `Product_Family__c =
  TiDB Enterprise Subscription`, `Opportunity_Type__c = Co-sale`, fixed
  Enterprise pricebook entry. The created Quote had
  `Opportunity.Region__c = NA`, `Product_Family__c = TiDB Enterprise
  Subscription`, `Referral_Fee_Rate__c = 0`, and null rebate/partner margin;
  approval submit returned `success=true`, `instanceStatus=Pending`, with one
  work item.

What failed:

- The earlier L2 fix correctly removed shared Opportunity write contention, but
  it treated "create any source Opportunity" as sufficient. That preserved
  createQuote mechanics but lost the approval-process business preconditions
  that the old shared fixture happened to satisfy.
- The focused local runner could not run the case directly because these cases
  use `authRole: admin` and local shell does not have the GitHub
  `NEXUSCRM_ADMIN_PASSWORD` secret. Direct API simulation with the downloaded
  admin storage state was used only as a fast payload check; official validation
  still needs GitHub Env Inspection.

What we learned:

- Replacing shared fixtures with self-seeded records must preserve the original
  story's business preconditions, not only the object graph. Otherwise we trade
  lock-wait failures for false-negative functional failures.
- For approval-process cases, the test should assert the entry-criteria fields
  immediately before submit. That converts a vague submit failure into a local
  data-shape failure and prevents another round of "why did approval reject?"
  debugging.

What next:

- Keep the two approval cases self-seeded, but fix their L2 payloads to target
  one explicit active process: NA OP Service, non-Cloud. Add a pre-submit query
  asserting `Quote.Product_Family__c`, `Opportunity.Region__c`,
  `Referral_Fee_Rate__c`, `Rebate_Fee_Rate__c`, and `Partner_Margin__c`.
- Run a focused GitHub inspection for the two approval cases after pushing the
  L2 branch, then a full Env Inspection once focused validation passes.

Follow-up focused run `28331660428` validated the value of the pre-submit
assertions: both cases failed before submit because the GitHub fixture-provided
`payloads.opportunityCreate.AccountId` resolved to an APAC account, so
`Opportunity.Region__c` became `APAC`. The correct fix is not to query any
existing NA Account from the environment. Each approval case now self-seeds its
own Account using the already validated Account create required fields,
overrides `Region__c=NA`, reads it back, and then uses that Account for both
the source Opportunity and Contact. The fix deliberately avoids writing
`Country__c` because Account models it as a formula field. Follow-up focused run
`28331869583` proved this still was not enough: createQuote failed before
approval because QuoteNumber's autonumber token reads
`Quote.Account_Country_Code__c`, which is the formula
`Text(Opportunity.Account.BillingCountryCode)`. The final fixture shape now
queries the L1 `Country_Dictionary__c` fact for `United States -> US`, writes
`BillingCountry=United States`, and asserts `BillingCountryCode=US` before
calling createQuote. The entry-criteria assertion stays in place; deleting it
would only hide the data-shape mistake.

The Lead object-info performance failure was analyzed separately instead of
being patched as a threshold issue. Historical evidence shows the same case was
stable on test2 and baseline:

| Run | `Lead/describe` | `Lead object-info` | Result |
| --- | ---: | ---: | --- |
| Baseline `27805400553` | `2506ms` | `584ms` | pass |
| `28304530646` | `913ms` | `516ms` | pass |
| `28314551045` | `1152ms` | `629ms` | pass |
| `28329765578` | `787ms` | `487ms` | pass |
| Failed `28330832503` | `2787ms` | `3432ms` | fail |
| Direct repeat on same test2 after failure | `870-1960ms` | `470-493ms` | pass |

The failed run had module concurrency 4. The presales latency module ran after
the sales/opportunity module completed, so it was not directly concurrent with
the two 15s approval submits. It did run at the start of the presales role in
parallel with presales access/credits modules; shortly afterward another Lead
UI API path (`sales-apac/latency/record-detail-load-lead-apac-sales-latency`)
also showed a 3315ms tail but still passed its budget.

Code evidence points to a generic L0 cold-start race:

- `uiAPIService.getObjectInfo` computes effective object access with
  `IncludeFields=true` and `IncludeRecordTypes=true`, then walks every readable
  field to emit UI API field metadata.
- The permission cache is warmed from OAuth login, but the OAuth handler calls
  only `WarmUserAsync`. The role auth precheck waits for the token response, not
  for object/field permission cache convergence.
- The api-host cache has a synchronous `WarmUser` and singleflight around the
  warm load, but the first business request does not join or await the in-flight
  warm. If `Lead object-info` lands before the async object/field cache is
  published, it computes the heavy Lead permission surface on the request path;
  once the cache is warm, the same endpoint returns at about 0.5s.

Classification: L0 platform mechanism performance tail. This is not an L1
payload failure and not enough evidence for a threshold change. The candidate
fix is a neutral auth/permission convergence improvement: make login-time
permission warm observable to the first heavy metadata request, or add a bounded
sync warm for API login flows before returning the token. The fix must preserve
cache invalidation/epoch safety and must not reintroduce stale permission facts.
If the next full run fails the same case again, implement this in the L0 auth /
api-host permission boundary, not in the watchdog case.

Follow-up focused run `28331967218` separated the quote failures again:
`quote-approval-state-propagation-api` passed, while
`quote-approval-submit-recall-api` failed only after a successful recall API
response. The created self-seeded Account now read back `Region__c=NA`,
`BillingCountry=United States`, and `BillingCountryCode=US`; `createQuote`
completed in `2377ms`; submit returned `success=true` and set Quote
`Status=Pending`; recall returned `success=true` with ProcessInstance
`instanceStatus=Removed`; the final Quote read still showed `Status=Pending`
instead of `Draft`.

Classification: L2 runtime approval metadata / deploy source-of-truth
divergence, not createQuote, not the L0 approval engine, and not a timing
retry. L0 recall executes recall lifecycle actions synchronously before
returning success. If the matching Recall `ProcessAction` exists and targets
`Status=Draft`, the final read should already see `Draft`; a delay would only
hide the missing metadata. The seed data confirmed the mismatch: 24 Quote
approval processes had the four lifecycle actions
`Submit->Pending`, `Recall->Draft`, `FinalApproval->Approved`, and
`FinalRejection->Rejected`, but the four NA 250502 processes carried 28 empty
FieldUpdate rows copied from Salesforce action references (`fieldApiName=""`,
`fieldValue=""`). Test deployment also did not apply the quote approval runtime
Seed, so reset runs could depend on stale ownerless rows or another
incomplete materialization path.

Correct fix direction:

- Make the runtime Quote approval Seed the explicit source for Test runtime
  approval rows while pack-owned approval_process components remain
  intentionally disabled for ownerless prod compatibility.
- Replace the four NA process action sets with the same four concrete Status
  lifecycle FieldUpdates used by the other Quote approval processes.
- Make the seed applier fail closed when any FieldUpdate lacks
  `fieldApiName`/`fieldValue`, or when any process does not have exactly one
  required Status action for Submit, Recall, FinalApproval, and FinalRejection.
- Keep the approval test's pre-submit entry-criteria assertions. They are the
  local guardrail that tells us whether a future failure is data-shape,
  lifecycle metadata, or engine behavior.

Lesson for recurring unclear failures: when the failed case count grows or the
same area alternates between "almost fixed" and "new failure", stop editing the
last failing assertion. Compare at least the last two runs, classify whether
the moving symptom shares a source-of-truth/read-path dependency, and add a
deployment-time or seed-time invariant before rerunning. A repeated failure is
evidence that the system is telling us the design boundary is unstable.

Follow-up full test2 Env Inspection `28332567731` ran after the runtime Quote
approval seed was applied by the test deploy path. Result:
`209 passed / 0 failed / 13 skipped`; functional `171/0/12`; performance
`38/0/1`.

What did:

- Deployed `codex/test2-watchdog-runner-fixes` to test2 in startup-only mode
  against NexusCRM `main` and app-packs `main`.
- Verified the deploy log applied the Quote approval runtime seed:
  `process_definitions=28`, `process_nodes=216`, `process_actions=112`, then
  refreshed the live metadata cache.
- Ran the automatically dispatched full Env Inspection on test2.

What passed:

- `quote-approval-submit-recall-api`: self-seeded Account readback passed,
  createQuote completed in `2348ms`, submit set Quote `Status=Pending`, recall
  returned success in `4668ms`, final Quote read returned `Status=Draft`, and
  ProcessInstance read returned `Removed`.
- `quote-approval-state-propagation-api` passed.
- `create-quote-from-opportunity-latency-api` passed with createQuote
  `2289ms`.
- `lead-metadata-bootstrap-latency-api` passed with Lead describe `900ms` and
  Lead object-info `521ms`.
- `metadata-fanout-latency` passed with `/actions/standard` at `2893ms`,
  effective permissions `614ms`, Account describe `499ms`, and Account
  object-info `258ms`.

What failed:

- No case failed in this run.

What we learned:

- The Quote approval failures were not independent defects. They were a single
  chain of L2 runtime metadata/data-shape assumptions: entry criteria, Account
  region, Account billing country code for QuoteNumber, then Recall lifecycle
  action materialization. The useful move was to stop at each new symptom,
  assert the upstream business facts, and finally make the deployment seed
  enforce the lifecycle invariant.
- The Lead object-info tail did not reproduce once the heavy sales/opportunity
  failures were removed and the metadata caches had a clean run. Keep the L0
  permission-warm hypothesis recorded, but do not implement it without another
  failed run or direct profiling evidence.

What next:

- Treat `28332567731` as the current test2 no-failure evidence for the branch.
- If another validation is needed before merge, prefer a targeted non-reset
  deploy/inspection or focused quote/performance run first. A full reset deploy
  is useful only when we need to prove reset-specific setup behavior, not for
  rechecking the already verified runtime seed and case payload.

## 2026-06-28 final main/main test2 verification

After all L0 and L2 fixes were merged, the formal test2 deploy used explicit
main refs for all three layers:

- L2/downstream `main` at `98099069c10308f3ea35cc1471ce89044787a44e`
- L0/NexusCRM `main` at `e4ae1b6a88db225c10f0a53d1ac2554aa5863aee`
- L1/app-packs `main` at `2c3f7ff8deee1613840c99aa8aab9b541e6229fa`

Deploy run `28335828066` targeted test2 with `bootstrap:verify`,
`reset:false`, and succeeded. The core deploy/apply step ran in about
`10m46s`, then auto-dispatched Env Inspection.

The first full Env Inspection, `28336122854`, had zero functional failures but
one performance-only tail:

- Total: `208 passed / 1 failed / 13 skipped`
- Functional: `171 passed / 0 failed / 12 skipped`
- Performance: `37 passed / 1 failed / 1 skipped`
- Failure: `sales/latency/record-detail-load-opportunity-latency`; the warm
  Opportunity `/ui-api/record-ui` request returned HTTP 200 but took about
  `7515ms` ready time, producing a `7783ms / 6000ms` page-load failure.

The failure was classified before action:

- not build/deploy, because the deploy had completed and triggered inspection;
- not functional, because all e2e functional cases passed;
- not L1/L2 payload, because the same fixture and case passed in the preceding
  no-failure run and in a focused rerun;
- best current classification: generic record-ui performance tail under full
  test2 concurrency/runtime pressure.

Focused run `28336558810` for only
`sales/latency/record-detail-load-opportunity-latency` passed with warm
record-ui ready time about `4099ms` and load `4365ms / 6000ms`. Because the
tail did not reproduce, no threshold or code patch was made.

The final full rerun, `28336672030`, passed:

- Total: `209 passed / 0 failed / 13 skipped`
- Functional: `171 passed / 0 failed / 12 skipped`
- Performance: `38 passed / 0 failed / 1 skipped`

This is the final main/main test2 evidence for the refactor validation. The
important process lesson is to classify a single moving performance failure
before editing code: focused reproduction and one full rerun were enough to
separate a transient record-ui tail from a real L0/L1 regression.

## 2026-06-29 test2 full-run tail and report extraction audit

After the test2 reset/deploy path and role-scoped lookup seed fixes were merged
to main, full Env Inspection run `28399932760` reported:

- Total: `206 passed / 3 failed / 13 skipped`
- Functional: `170 passed / 1 failed / 12 skipped`
- Performance: `36 passed / 2 failed / 1 skipped`

The remaining failures were classified before changing code:

| Case | Evidence | Current classification |
| --- | --- | --- |
| `sales/opportunity/quote-approval-submit-recall-api` | Approval submit returned a 504 after about 30s. Nginx later showed a nearby `/process/approvals` 200, but the exact request id was not found in backend logs. | Infra/runtime tail under full-run pressure. Needs exact request-id evidence before an engine or payload fix. |
| `sales/latency/record-read-latency-api` | REST GET `/services/data/v63.0/sobjects/Opportunity/0065g000002I8QCAA0` returned 404 in 351ms. Earlier cases read or used the same id successfully, and later focused runs read it successfully. | Not permanent missing data. Full-run-only read/access-path inconsistency or transient pressure under investigation. |
| `sdr/latency/parameterized-search-lookup-latency-api` | p90 was `3564ms / 3000ms`, samples `[2303,2398,2340,2392,2391,3019,3564,2412,2352,4097]`. | Real p90 tail, not p50. Focused rerun passed, so keep investigating under full-run scheduling. |

Focused sales latency run `28401772313` used the same resolved Opportunity id
and passed all `17/17` cases. `record-read-latency-api` returned 200 in
`1201ms`; `submit-for-approval-latency-api` also passed in `6237ms`. This is
evidence against a deterministic case-order delete, deterministic sales-latency
module bug, or permanently invisible fixture record. It is not enough evidence
to call the full-run 404 solved.

Code inspection also found an independent L2 report extraction bug. The runner
computed performance summary metrics by testing `Number(step.maxLoadMs)`.
Browser warm-up steps persist `maxLoadMs: null` in artifacts, and JavaScript
converts `Number(null)` to `0`, so the summary could record the warm-up page
load as the performance duration even though the sampled p90 step passed.

The fix extracts performance metrics through a null-aware helper. Recomputing
the focused artifact with the fixed helper produced:

| Case | Old report duration | Fixed report duration | Threshold |
| --- | ---: | ---: | ---: |
| `record-detail-load-opportunity-latency` | `6206ms` | `2993ms` | `6000ms` |
| `record-detail-load-account-latency` | `5877ms` | `2851ms` | `5000ms` |
| `record-detail-load-contact-latency` | `5717ms` | `2825ms` | `5000ms` |
| `record-read-latency-api` | `1201ms` | `1201ms` | `5000ms` |

The API metric staying unchanged is the important guardrail: the fix corrects a
browser warm-up reporting bug and does not hide API runtime failures. Next
runtime work should add evidence, not retries: capture exact backend request-id
logs for approval 504s and compare the same full-run fixture id through SOQL,
REST SObject read, and UI API record-ui in one probe when the record-read 404
reappears.

## 2026-06-29 post-report-fix p90 evidence gap

After the browser warm-up report fix landed, formal test2 Env Inspection run
`28402807947` produced:

- Total: `207 passed / 2 failed / 13 skipped`
- Functional: `171 passed / 0 failed / 12 skipped`
- Performance: `36 passed / 2 failed / 1 skipped`

Both remaining failures were performance-only and must stay classified by p90:

| Case | Evidence | Current classification |
| --- | --- | --- |
| `sales/latency/record-detail-load-account-latency` | sampled browser load selected `5435ms / 5000ms`; all 10 samples reached the record-ui ready response, but the summary still showed `0ms / 5000ms` because the failed sampled step was stored in `detail.failedStep`. | Real browser p90 tail with insufficient request-id evidence. Fix the artifact/report path before changing product code. |
| `sdr/latency/parameterized-search-lookup-latency-api` | sampled API p90 selected `3103ms / 3000ms`; backend logs for request `d7b1c6dc-7f20-4c2c-9530-2115ac59e2e0` showed about `1522ms` SQL time, `1526ms` query repository time, and `3059ms` HTTP latency. | Generic search/query performance tail under full-run conditions; needs current-SQL diagnostics before deciding L0 query rewrite versus L1 case shape. |

The immediate fix is evidence quality, not threshold relaxation:

- `perfMetrics` must inspect `detail.failedStep` as well as `detail.steps`, so a
  failed sampled browser p90 reports the actual selected load.
- browser timing samples must capture the ready-response request id and trace
  context, so future record-detail tails can be joined to backend logs.
- diagnostic SQL must be generated from the current DB and current observed
  query shape. Stale hard-coded inspection user ids or pre-refactor SQL can
  produce misleading conclusions.

Operational rule: documentation and historical issue comments are only leads.
For root cause and fixes, prefer current code, workflow artifacts, exact request
ids, backend/nginx logs, and live DB diagnostics.

## 2026-06-29 full-run p90 tail after request-id browser evidence

Formal test2 Env Inspection run `28405879108` on `main` SHA `8fb65fcd`
produced:

- Total: `206 passed / 3 failed / 13 skipped`
- Functional: `171 passed / 0 failed / 12 skipped`
- Performance: `35 passed / 3 failed / 1 skipped`

All remaining failures were performance-only:

| Case | Evidence | Current classification |
| --- | --- | --- |
| `sales/latency/record-detail-load-opportunity-latency` | selected p90 `6037ms / 6000ms`; samples `2950,6037,2914,2832,2854,4682,3011,2902,5660,32830`; selected request id `dc1da7d2-240e-4073-b6c5-b2e3683f53db`; `readyMs` dominated the selected sample. | Shared record-ui backend readiness tail under full-run conditions. Do not patch Opportunity-specific payload unless backend logs prove an object-specific cause. |
| `sales/latency/record-detail-load-contact-latency` | selected p90 `5481ms / 5000ms`; samples `5566,2682,2694,2731,5481,2728,2689,2755,4356,2638`; selected request id `5ad6c16b-85fe-4468-bd6f-ef6fa972914d`; `readyMs` dominated the selected sample. | Same shared record-ui tail group as Opportunity/previous Account. |
| `sdr/latency/parameterized-search-lookup-latency-api` | selected p90 `3384ms / 3000ms`; samples `2693,3384,2319,2388,2317,2291,3612,2242,2245,2270`; selected request id `e09cf71d-1515-4799-8187-226d2d890b1b`. | Generic Lead substring search tail under SDR RLS. Current code still uses the L0 substring path; L1/L2 field defaults are not the immediate missing fact. |

Focused reruns can be green while the formal full run still fails p90. Treat
focused success as a capability check, not as closure for full-run performance.
The next evidence step is backend/nginx log segmentation for the exact record-ui
request ids and current SQL diagnostics for Lead search. API p90 artifacts now
must keep per-sample request timing details so the slow selected sample can be
joined to backend logs instead of inferred from aggregate durations.

## 2026-06-29 current-code permission root cause

No doc, old comment, or issue note is authoritative for this phase. The current
code and request-id logs showed a generic L0 permission read-path waste:

- `ContractPolicy.Access` ignored the projection shape after either fields or
  record types were requested, so field-only callers still computed record-type
  access and record-type-only callers still computed field access.
- request-scoped effective field permissions had a cache, but no first-load
  coalescing. Concurrent record-ui goroutines could race and issue duplicate
  full-user FieldPermissions bulk loads before the cache was populated.

NexusCRM PR `#3289` fixes those two L0 mechanism issues and adds regression
tests. This is not an L1 payload change and not a threshold relaxation. The
remaining `parameterizedSearch` p90 tail still needs separate current-SQL and
request-id evidence before choosing a query/permission/search fix.

## Recalibration

Re-run `bench_sfdc.mjs` to refresh the measured table when the production org or
network vantage changes, then adjust the `record-read` / `soql-query` /
`describe` bounds. Write/submit bounds are policy, not measured, and should move
only with an explicit product decision.