# HubSpot Inbound Pressure Matrix

## Current contract

The pressure matrix has one source of truth in
`pingcap-inc/nexuscrm-hubspot-acceptance`:
[`config/hubspot-inbound-pressure-matrix.json`](https://github.com/pingcap-inc/nexuscrm-hubspot-acceptance/blob/main/config/hubspot-inbound-pressure-matrix.json).

It defines the full cross-product below. Each cell is a separate run against one
already-deployed Test runtime, and every cell owns an explicit expectation and
description in the JSON contract.

The nine executable cases are first-class HubSpot integration cases under the
acceptance repository's `config/cases/hubspot/pressure/`. They are registered
in its `config/roles.json`, use canonical keys of the form
`admin/hubspot/inbound-pressure-w<workers>-e<events>`, and appear in the normal
integration `report.json` / `report.md`. They are `optional` in a full suite
because one deployed runtime cannot simultaneously satisfy the 8-, 16-, and
32-worker readbacks; selecting a pressure cell explicitly makes it runnable.

| Cell | Expectation | What it proves | Non-PASS meaning |
|---|---|---|---|
| worker8 / 2k | must pass | Safe/default-worker admission, receipt, projection, drain, and business baseline | Blocks release acceptance |
| worker8 / 4k | must pass | Production-like SC400 baseline at the current safe/default worker count | Blocks release acceptance |
| worker8 / 6k | diagnostic | Default-worker stretch above the historical 4k shape | Capacity shortfall after complete drain may be informational; timeout/residue blocks |
| worker16 / 2k | should pass | Worker-16 activation and low-load scale sanity | Does not block worker8 release, but blocks a worker16 scale claim |
| worker16 / 4k | should pass | Worker-16 SC400 comparison against worker8 | Does not block worker8 release, but blocks a worker16 scale claim |
| worker16 / 6k | diagnostic | Worker-16 high-load stretch | Capacity shortfall after complete drain may be informational; timeout/residue blocks |
| worker32 / 2k | diagnostic | Non-production worker-32 activation experiment | Pre-mutation worker32 mismatch or terminal capacity shortfall may be informational |
| worker32 / 4k | diagnostic | Non-production worker-32 SC400 scaling experiment | Pre-mutation worker32 mismatch or terminal capacity shortfall may be informational |
| worker32 / 6k | diagnostic | Maximum worker/event-count experiment | Pre-mutation worker32 mismatch or terminal capacity shortfall may be informational |

"Diagnostic" does not waive correctness. For every executed cell, durable
receipt loss, accepted-but-missing Events, duplicate active CRM identities,
wrong final business values, source-projection contract violations, dead/retry
residue, timeout, or lane/lease correctness failures remain blocking defects.
Only a runtime-profile mismatch proven before mutation, or insufficient
capacity evidence after the exact Event set has fully terminalized, may be
treated as non-release-blocking for a diagnostic cell.
The runtime-profile exception is deliberately narrow: only
`workerCountExpected=false` with every other runtime check true qualifies.
Health, ref, schema, claim-mode, queue-profile, route, receipt-mode, or database
failures always fail the case and cannot be hidden by a capacity observation.

Every cell keeps the other pressure variables fixed:

- 400 candidate-scoped HubSpot Contact source lanes
- the current generic HubSpot provider envelope:
  `subscriptionType=object.propertyChange`, `objectTypeId=0-1`
- the normalized runtime route `contact.propertychange` ->
  `HubSpot_Contact_Lead_Upsert`
- `latest_by_source_full_record_preserve_payload_settled`, with the L0 five
  second source-settle window and route batch size 100
- one lane-major burst per Contact: 5, 10, or 15 ordered `firstname` changes
  for the 2k, 4k, or 6k cell. A source burst never crosses webhook requests,
  and every value carries a bounded run hash so an earlier run cannot satisfy
  the current business oracle.
- four concurrent webhook requests, held constant across all worker cells. This
  is the calibrated admission producer shape, not executor count.
- `max_processing_rows=32`
- `queue_profile=default-heavy`
- `queue_class=bulk`
- a two-hour safety timeout; this is a bounded test timeout, not a signed
  product throughput SLA

The shape follows the actual implementation rather than the old pressure-runner
shortcut. Every webhook request must return durable `202`; any `204` or partial
admission is a BLOCK. Verified-ingress receipts must be enabled. The runner
resolves each `receiptId` to exact internal `EnqueuedEventIds`, reads those Event
rows by primary key, and requires their endpoint-scoped `IdempotencyKey` set to
equal the set derived from this run's generated `eventId` values. Matching
counts or internal IDs from another run are rejected. It does not repeatedly
scan JSON payloads while measuring TiDB.
The object webhook is signed with the pack-owned v1 contract,
`sha256(secret + rawBody)` in `X-HubSpot-Signature`; v3 HMAC remains exclusive
to the separate workflow-action endpoint.
The report separates receipt materialization, ingress admission rate, and
request-start-to-terminal throughput, and verifies the final active Lead or
Contact `FirstName` equals the last ordered change for each of the 400 source
lanes. Queue drain alone is not business acceptance.

`default-heavy` reserves one quarter of the executors for
`interactive/default`. A bulk-only run therefore has 6, 12, or 24 eligible
executors at 8, 16, or 32 deployed workers. Each cell must actually observe that
bulk-eligible lease width; merely reading back the worker environment variable
is not worker-scaling evidence.

This contract was derived on 2026-08-25 from downstream `main` at
`a0a3a2035a8ac38661c76d1dc2fca94fc0251057`, app-packs ref
`39b0e70c899e44414909c00eae13a01d64f2e555`, upstream NexusCRM ref
`99d50b04a0a8a9781c6b6cc6fd4b72268d45ecfb`, and connectors `v0.2.14`
(`78e1d69da90630b334f03298aa0c58c429371ec6`). Those refs document the design
input; every live cell still reads back the deployed route and runtime settings
so a newer deployment cannot inherit a stale PASS.

## Execution boundary

`.github/workflows/integration-test-hubspot.yml` owns the pressure orchestration.
Select `suite_mode=pressure`, `target_env=test1`, a worker count, an event count,
the candidate-scoped fixture manifest, and its candidate id. The workflow maps
that selection to one registered case, checks out exact current `main`, runs on
the Test1 host, and publishes the result through the same HubSpot integration
report and artifact path as the other cases. There is no separate pressure-only
reporting workflow.

Every Test1 HubSpot integration run, including non-pressure custom/full runs,
shares the deployment concurrency group
`deploy-test-multi-slot-environment-test1`. A pressure run therefore cannot
overlap another Test1 integration run or a Test1 deployment on the shared queue
and CRM data.

The standard runner's `hubspot_pressure` case kind delegates to the existing
`scripts/connected_work_matrix_gate.py` `runtime,workitem-scale` gates. This
keeps the pressure algorithm, actual HubSpot route checks, worker readback, and
business oracle shared with the existing connected-work evidence implementation
instead of reimplementing them in case JSON.
Both Python entry points also enforce the Test1-only boundary and require the
candidate manifest plus candidate id for `scale`/`workitem-scale`; invoking the
scripts directly cannot fall back to existing-contact discovery on another
slot.

Before reading fixture Contacts or admitting any event, the pressure runner
calls HubSpot account-info with the selected credential and requires
`accountType=SANDBOX`. The observed portal, `HUBSPOT_EXPECTED_PORTAL_ID`, and
the portal embedded in generated events must all match. A missing expected
portal or any mismatch fails closed before admission.

The workflow does not deploy, edit runtime files, or restart services. Before
admission it requires an idle HubSpot queue, verified-ingress receipt mode, the
exact active route above, and
the deployed
`NEXUSCRM_CONNECTOR_INBOUND_WORKERS_PER_PACK` to equal the requested 8, 16, or
32 value. A mismatch is a BLOCK, not a lower-worker PASS carrying a misleading
profile name.

The 32-worker cell is an explicit non-production experiment. Current production
release policy remains capped at 8 workers, and the current classic Staging
guard remains capped at 16. Run 32 only after an authorized Test deployment
publishes and reads back that exact runtime configuration; this test workflow
does not grant that deployment authority.

Pressure fixtures are case-local and mutation-heavy. They must use a
candidate-scoped manifest under the Test1 pressure-fixture artifact root and
must never be promoted into deploy baseline seed data. The manifest and
candidate identity are required before the first pressure request.

The standard report has four live outcomes for these cases:

- `passed`: every requested runtime, correctness, capacity, and business gate passed;
- `failed`: any required/should-pass non-PASS, or any correctness defect in any tier;
- `diagnostic`: only a pre-mutation runtime-profile mismatch or a terminally
  drained capacity-evidence shortfall; this is visible but does not fail the workflow;
- `skipped`: the usual full-suite optional or dry-run behavior, not a live
  pressure observation.

## Existing `concurrency=4/5` cases

The independently governed Set Campaign performance acceptance is documented
in `docs/hubspot-set-campaign-performance.md`. It is a workflow-action workload,
not an inbound executor-worker cell.

The existing values are not obsolete. They are **request-side admission concurrency**,
not deployed worker count:

- `k-camp-10` keeps `concurrency=4` for its 2,000 Set Campaign action driver.
  It holds provider/admission pressure constant while the backend drains work.
- `k-camp-05` keeps `concurrency=4` as a bounded bulk-integrity smoke. It is not
  worker scalability acceptance.
- `k-camp-13` keeps `concurrency=5` so five actions collide on one Campaign and
  exercise the parent-lock regression. It is not a capacity benchmark.

The runner caps these workflow-action values to 1-8 and uses them only to size
the client-side fan-out promise pool. They neither configure nor prove inbound
executor workers. The pressure matrix's own admission concurrency is also 4,
but that is an independently pinned object-webhook producer control.

Historical evidence also showed that raising replay threads to 16 could
overload receipt verification and produce harness-side `503` responses, while
four replay threads completed the accepted ladder. Therefore the worker matrix
must vary the runtime worker count while keeping the inbound producer shape
stable. If request-side concurrency is studied later, it is a separate matrix
and must not be folded into the 8/16/32 worker comparison.

## PASS boundary

A cell passes only when:

1. runtime health, schema authority, WorkItem claim mode, worker count,
   processing-row bound, and queue profile match the requested cell;
2. the candidate-scoped fixture manifest supplies 400 remotely readable,
   unambiguous HubSpot Contact sources;
3. every requested Event receives durable `202` admission, every receipt-bound
   Event row maps to the exact generated `eventId` identity set, and every Event
   materializes;
4. all Events terminalize with no failed, retry, dead, or nonterminal residue;
5. source projection produces exactly 400 processed WorkItems and skips the
   superseded Events; a successor generation is classified as a settle-window
   fixture split rather than as an architecture failure;
6. the final active Lead/Contact identity is unique and its `FirstName` equals
   the last generated value for all 400 sources;
7. the claim-plan EXPLAIN resolves the run to exactly one active
   `hubspot-crm-connector` pack generation, uses no scheduler-lease read path,
   and contains no broad scan;
8. lane ordering, single-winner lease invariants, and observed bulk capacity of
   6/12/24 pass; and
9. the report and artifact remain bound to the exact downstream commit,
   candidate id, worker count, event count, and runtime readback.

Local unit tests prove only matrix construction and runner/workflow wiring. They
do not prove any live 8/16/32 or 2k/4k/6k cell has passed.