# HubSpot Overload Retry-Loop Contract Status: the generic L0 overload mechanism remains merged, but the L2 production ordinary-webhook `ack_drop` selection is superseded by `#4228` after live data loss evidence; must-complete callback PR B is implemented in `#4049` but merged as `f7ddca9c06b701f6b8699652f8ecad37184c574b`; its staging and production acceptance are not evaluated by `#4228`, 2026-07-21 The original 2026-07-17 sequence was intentionally incomplete: L0 `#138/#140/#139`, L1 `#406`, and L2 `#3937/#3938/#3939` are merged, but the connectors changes must be published and adopted by the NexusCRM runtime before dependent L1/L2 behavior can be released. No release or production runtime deployment is authorized by this document. Published PR map: - L0 bounded SQL cleanup: `nexuscrm-connectors#138` (merged as `3b731b1d49b586d157005c5299487eea39924a89`) - L0 ordinary ingress `ack_drop`: `nexuscrm-connectors#139` (merged as `586e49d5b19712b786c72f96826e819259f4cc2e`) - L0 composite identity substrate: `nexuscrm-connectors#140` (merged as `9b5885cc31da3efc6cc01a5883a33d54f5d2a816`) - L1 HubSpot composite adoption: `nexuscrm-app-packs#406` (merged early as `85a23cf1f6083f898a0661de140500dc52b8eeea`; release remains blocked on publishing and adopting L0 `#140`) - L2 workflow best-effort PR A: `nexuscrm-pingcap-pack#3937` (merged as `a749dcc8e96ee1257dc3c5322fe7b68ca5565c23`) - L2 pressure response consumer: `nexuscrm-pingcap-pack#3938` (merged as `b77264c1acc7a50353cc8e5a1764d44623fb317c`) - L2 prod ingress modes/readback and this contract: `nexuscrm-pingcap-pack#3939` (merged as `e661aac0b2c82bd6aeec54f382ab837563d38959`; release depends on publishing and adopting merged L0 `#138/#139`) - L2 workflow must-complete PR B: `nexuscrm-pingcap-pack#4046` merged only its scaffold and is superseded; `nexuscrm-pingcap-pack#4049` is the implementation authority and merged as `f7ddca9c06b701f6b8699652f8ecad37184c574b`. Merge state is not release evidence; formal staging still must prove a real provider callback leaves `BLOCK` only after the corresponding Connector Event reaches `processed`. ## Target Invariant For PingCAP's normal production ordinary HubSpot channel, a successful HTTP response is valid only after the request is durably admitted. Dependency, admission, or receipt failures must remain retryable; ordinary CRM change events are not disposable. The L0 provider-neutral `ack_drop` mechanism still exists for an endpoint whose owning product explicitly declares its events disposable. `#4228` supersedes that declaration for `hubspot-crm-connector/webhook` in L2 production. HubSpot's webhook error contract retries connection failures, timeouts, and `4xx`/`5xx` responses up to 10 times over 24 hours; a `2xx` is an acknowledgement: . ## Scope Guard This is a design-first umbrella and must not land as one cross-repository PR. The implementation is split into independently reviewable slices: 1. L0 connector response mechanism: provider-neutral, opt-in `ack_drop`, with the current retry behavior remaining the default. 2. L2 workflow action semantics: per-action `FAIL_CONTINUE` or durable-first `BLOCK` plus completion callback. 3. L2 ingress operations: independent ordinary/workflow nginx modes and a pressure gate that rejects retryable overload responses. 4. Event identity dependency and adoption: the current neutral envelope parser accepts only one `idempotencyKeyPath`, so a provider-neutral L0 composite identity contract must land before L1 replaces `.eventId` across its HubSpot rules. These are two ordered successor slices and must not be mixed into the first three implementations or one cross-layer PR. ## Non-Goals - Do not merge the intentional inbound App and workflow App. - Do not change HubSpot property mappings or field ownership in this L0/L2 contract. Subscription changes require separately documented owner approval. - Do not change the NexusCRM permission hot path from the 2026-07-15 outage. - Do not introduce an L0/TiDB schema migration or backfill in the first three slices or the composite-identity metadata dependency. PR B may additively migrate its local SQLite delivery state. - Do not claim the existing single-host SQLite queue is a replicated durable inbox. - Do not change final-pack lineage or deploy production from these local implementation sessions. ## Operational App Controls - The inbound App is an independent HubSpot Project controller. On 2026-07-15, its production project build `#5` was deployed after validation: it lowered `maxConcurrentRequests` from `10` to `6` and removed only the proven-unmapped `mobilephone` subscription. Live provider readback reported `38` total subscriptions, `35` property subscriptions, and no `mobilephone` entry. - `leadsource` and `lead_source_details` remain subscribed. They are two-way fields and cannot be pruned until their owner matrix and echo suppression are approved and implemented. - The Workflow Action App was not altered. It keeps its own deployment, pause/recovery, and `BLOCK`/`FAIL_CONTINUE` action contracts. ## Slice Contracts ### L0 Ordinary Ingress - Default behavior remains the existing retryable response contract. - Explicit `ack_drop` returns `204` without `Retry-After` for internal dependency and overload failures. Failure reasons remain bounded metrics; they do not decide whether an approved disposable endpoint may drop. - Security and protocol failures remain fail-closed. - Policy selection is local and testable and performs no TiDB lookup. - Once the endpoint is selected, an allowlisted request gets one absolute deadline. Direct Go body reads and context-aware pack, secret, verification, admission, routing, companion, and enqueue work share that deadline. It cannot bound the client-to-nginx upload that precedes a buffered proxy request. Nginx `ack_drop` remains the no-body, no-database break-glass path. - Receipt-mode `ack_drop` and default requests use separate pack/verifier singleflight owners while retaining one shared healthy cache. Same-contract loads stay synchronous and request-owned so the verification gate accounts for their full lifetime. A follower does not independently observe an earlier parent cancellation, and a simultaneous cross-contract cold miss may perform two bounded loads; a cache-owned/ref-counted shared-flight lease is separate hardening, not part of this release contract. - Pessimistic transaction session reset uses a fresh bounded `200ms` cleanup context. Timeout or reset failure marks the connection bad before close rather than returning it to the pool. - L0 contains no HubSpot-specific names or tenant policy. - `request_uri_resolve_failed` is the single named fail-closed internal `503` exception. Invalid signature and malformed, missing, or oversized requests retain their existing `4xx` behavior. ### L2 Workflow Actions - Merged `main` contains PR A: exact action policy plus `best_effort` `FAIL_CONTINUE`. PR B source merged in `#4049`; staging acceptance and production runtime proof remain separate and are outside `#4228`. - Policy is selected by action definition from local startup configuration. - `best_effort` returns `FAIL_CONTINUE` after V3 signature verification and does not enqueue work. - `HUBSPOT_WORKFLOW_EXECUTE_MODE=proxy` is the only execution-enable SSoT. It requires the workflow App's exact signing secret and an exact-action policy allowlist. In `block` mode pre-provisioned PR B configuration is ignored and no replayer starts. - `must_complete` durably commits the signed admission before returning HTTP `200` with `BLOCK`. CRM enqueue, Connector Event observation, and HubSpot callback delivery are distinct persisted phases. - NexusCRM webhook `202` is enqueue proof only. It must persist a receipt or event id and must never make `SUCCESS` callback-eligible. Only Connector Event `processed` does so; Event `dead`/`skipped` and receipt `dead`/`config_terminal` produce terminal `FAIL_CONTINUE`. - Result observation never repeats an accepted CRM enqueue. Callback retry never repeats either CRM enqueue or result observation. Phase backlogs are scheduled fairly and use independent retry budgets inside the provider block expiration. - Staging acceptance requires one real HubSpot-created callback id, one exact CampaignMember effect, a local row completed with Connector Event `processed` plus callback `204`/`409`, and HubSpot workflow history no longer blocked. Callback `401`/`403` blocks release because the token lacks required `automation` authority. - Unknown actions fail closed rather than inheriting an unreviewed semantic. - The SQLite queue remains single-host durability, not a replicated inbox. ### L2 Ingress Operations - Ordinary webhook and workflow execute have independent desired-state modes. - Ordinary retains an explicit nginx break-glass `ack_drop` capability, but it is a deliberately lossy incident action and is not the normal production backend contract or a formal release PASS mode. Workflow cannot be blindly acknowledged at nginx because the action policy is in its payload. - The legacy combined block flag is read-only compatibility input. New generated desired state has one owner per channel. - Retryable `503` is not a pressure-gate PASS condition. - Prod runtime fixes `NEXUSCRM_CONNECTOR_WEBHOOK_ACK_DROP_ENDPOINTS` to the explicit empty set and `NEXUSCRM_CONNECTOR_WEBHOOK_ACK_DROP_TIMEOUT_MS` to `750`. A missing key or any non-empty process-env override is rejected. Active runtime and rendered nginx are read back on the prod host before HTTP acceptance. ### Event Identity - L0 supports a provider-neutral, versioned `sha256` composite identity with ordered, typed, length-framed scalar parts and exact numeric canonicalization. Legacy single-path rules remain compatible. - L1 adopts the composite for all 32 ordinary HubSpot rules using stable app/account/subscription/event/time plus event-specific fields. `attemptNumber` is excluded from the business Event identity, so HubSpot retries retain one Event identity. The verified ingress receipt is a separate transport-attempt identity: it includes `attemptNumber` so two retry attempts with different verified bodies do not collide on one receipt. Staging's signed no-business probe must therefore include `attemptNumber` even though its unmatched `subscriptionType` prevents business Event creation. - The workflow rule remains the legacy `.callbackId` contract. L1 cannot be released before a runtime containing the now-merged L0 composite decoder is published and adopted. GitHub Actions merged L1 `#406` first, so repository merge state alone is not evidence that this compatibility gate passed. ## Acceptance - L0 mechanism tests retain proof that an independently owned disposable endpoint stops waiting on direct Go reads and context-aware server dependencies at its configured deadline, then returns `204` without `Retry-After`. That mechanism proof is not L2 production acceptance. - L2 preparation and active readback prove the backend ack-drop endpoint set is explicitly empty. A forced internal-unavailable branch in normal `proxy` mode must return retryable `503`; a healthy signed delivery must return `202` only with a durable receipt or Connector Event correlation. - Transaction session reset retains a separate `200ms` cleanup ceiling. L0 policy lookup itself performs no database call. - Default connectors retain current `503` behavior. - Invalid signatures and malformed or oversized bodies retain their existing rejection behavior. - Current workflow PR A tests cover exact `FAIL_CONTINUE` response bodies, signature verification, action allowlisting, and block/proxy isolation. - PR B acceptance must cover enqueue-before-BLOCK ordering, duplicate callback ids, crash/restart callback phase recovery, callback completion, and transient versus terminal replay classification. - Nginx tests cover independent channel modes and preserve default production isolation. - The pressure report producer and consumer use the same versioned response contract. ## Historical Evidence Snapshot (2026-07-17) This section preserves the evidence state used by the original overload design. Its release/tag blockers are historical and must not be read as current without fresh verification. The `#4228` supersession above is the current ordinary webhook contract. - `nexuscrm-connectors`: `#138/#140/#139` are merged in that mainline order as `3b731b1d`, `9b5885cc`, and `586e49d5`. Full `go test ./...`, `go build ./...`, focused race, diff check, and latest-main integration proof pass. The exact current heads received Standard and Adversarial APPROVE. - `nexuscrm-app-packs`: 32 ordinary composite rules, one unchanged workflow rule, focused tests, L0 wire round trip, and pack validation pass. `#406` merged as `85a23cf1`, but the latest published connectors tag remains `v0.1.16` and NexusCRM `main` still adopts `v0.1.16`; release/deploy therefore remains blocked. One full suite `richtextarea` assertion also fails on the clean L1 head and is unrelated to this slice. - `nexuscrm-pingcap-pack`: focused ingress, prod readback, pressure-gate, and workflow PR A tests pass; `#3937/#3938/#3939` are merged as `a749dcc8`, `b77264c1`, and `e661aac0`. The pressure report producer is external to this repository and still must adopt the new response schema. - The former release acceptance requiring backend `ack_drop` `204` is superseded by `#4228`. New acceptance requires exact empty backend allowlist, active `proxy` readback, one real signed `202` with durable correlation and expected record effect, plus a controlled internal-unavailable probe that is retryable rather than acknowledged. Local tests do not substitute for this live gate. - An automatic endpoint circuit breaker remains a separate P1 hardening successor. HubSpot bounds failed-webhook retries to 10 attempts over 24 hours; capacity and retry behavior still require live pressure acceptance. - The pre-merge PR B live-probe blocker is superseded by the `#4049` merge. This document does not assert current staging or production acceptance for that workflow-action path. ## Stop-Loss Rules Stop and split again if an implementation requires new durable schema, a migration/backfill, an identity protocol outside the ordered L0 metadata plus L1 adoption successors, changes to the two-App boundary, or more than three fail-closed invariants in one slice. Evidence bindings at implementation start: - `nexuscrm-connectors` clean head: `7695f98a004f5c717a556b1d056f8155eec8a2c5` - `nexuscrm-pingcap-pack` head: `c3345269a6ba75a069199f475e8fac13d06e5073` (dirty workspace; preserve all pre-existing changes)