# HubSpot Authored Identity Recovery Operator Tracked by [NexusCRM #3765](https://github.com/pingcap-inc/NexusCRM/issues/3765) and the HubSpot consumer slice [nexuscrm-pingcap-pack #4154](https://github.com/pingcap-inc/nexuscrm-pingcap-pack/pull/4154). `scripts/hubspot_authored_identity_recovery_maintenance.py` is the L2 production boundary for the approved current-cohort historical repair. It never uses Email, `HubSpot_Id__c`, or an operator-supplied evidence/link ID as authority. It invokes the protected L0 setup command `recover-connector-remote-identity-batch`; it contains no SQL or HTTP mutation fallback. Production execution is **BLOCKED** until the deployed runtime tuple includes that L0 command and the #3765 strict recovery service. A merged source branch or tracked source-checkout provenance is not runtime proof. No production manifest, runtime-authority snapshot, backup marker, receipt, or stderr log is tracked in this repository. ## Guarded scope - The manifest is exactly `prod`, NexusCRM #3765, and cohort `prod-legacy-unresolved-20260719T150500Z`: 68 observed contacts split at the read-only snapshot into 31 unique-active-authored-evidence Lead candidates, 23 already-finalized locators, and 14 rows with no active authored evidence. - The snapshot closes at `2026-07-20T06:12:18Z`. The operator pins the SHA-256 of the exact sorted 31 candidate tuples; count-only census data or a cohort name is not membership authority. The manifest must contain exactly those 31 tuples and the exact `31 + 23 + 14 = 68` partition. - Candidate removal, addition, source-pair change, or substitution blocks the wrapper. Natural state changes do not authorize editing the fixed list: L0 must transactionally accept and read back the same 31 exact pairs or block. A materially different cohort requires a newly reviewed snapshot. - Each candidate contains only an opaque HubSpot remote ID plus the expected NexusCRM source object and source record ID. It contains no Email, name, phone, mirror value, evidence ID, link ID, or credential. - The exact manifest file bytes are bound by `--manifest-sha256`. Candidate and non-candidate identifiers are never tracked in this repository; only the reviewed candidate-membership SHA-256 is pinned in code. - The live deployed tuple is independently reconstructed from the active production release's `release.json` and `current/env/runtime.env`. Do not substitute source-checkout provenance or an operator JSON file for these two runtime authorities. - The exact setup binary, `release.json`, and `runtime.env` bytes are bound in the manifest. L2 opens and hashes stable file descriptors, executes the binary through `/proc/self/fd`, and L0 independently verifies the same file descriptors plus `/proc/self/exe` before mutation. - Dry-run is the default, but it is not validation-only: it must execute the real L0 mutation path inside a sentinel-rolled-back transaction. - Apply additionally requires the raw SHA-256 and file for a successful L0 dry-run receipt, plus the raw SHA-256 and file for an exact immutable backup marker. - Receipt and stderr-log files are reserved with `O_EXCL`, mode `0600`, and `fsync` before runtime starts. The same receipt file descriptor first holds an outcome-unknown guard and is then finalized after runtime; a committed mutation can never be followed by first-time receipt creation. - Runtime stderr is written directly to the private host log. It is never copied into the JSON receipt or console summary. ## Manifest contract The top-level and nested field sets are exact. `candidates` must contain the same number of unique entries as `census.eligible`; this one-entry excerpt documents the item shape only and is not an executable manifest. ```json { "schema_version": 1, "operation": "hubspot_authored_identity_recovery", "issue": "https://github.com/pingcap-inc/NexusCRM/issues/3765", "environment": "prod", "approval_ref": "", "deployed_tuple": { "deployment_id": "", "release_tag": "", "downstream_commit": "<40-char-L2-commit>", "upstream_commit": "<40-char-L0-commit>", "official_app_packs_commit": "<40-char-L1-commit>", "final_pack_hash": "<64-char-final-pack-sha256>" }, "runtime_binding": { "release_json_sha256": "", "runtime_env_sha256": "", "setup_binary_sha256": "" }, "selector": { "connector_owner_key": "pingcap-hubspot-companion", "connector_namespace": "hubspot-crm-connector", "destination_slot_id": "hubspot-contact" }, "census": { "cohort_ref": "prod-legacy-unresolved-20260719T150500Z", "observed": 68, "eligible": 31, "already_finalized": 23, "excluded_no_evidence": 14 }, "candidates": [ { "remote_id": "", "expected_source_object": "Lead", "expected_source_record_id": "" } ] } ``` Remote IDs and source record IDs must each be unique. Candidate tuple membership is encoded as the SHA-256 of newline-separated, lexically sorted `remote_id NUL expected_source_object NUL expected_source_record_id` rows, without a trailing newline. The digest must equal the operator's reviewed snapshot constant. The 23 finalized and 14 no-evidence rows are census counts only and are never accepted as mutation inputs. ## Backup marker The backup must be complete, encrypted, immutable, and independently reviewed before apply. The marker has this exact shape: ```json { "schema_version": 1, "operation": "hubspot_authored_identity_recovery", "environment": "prod", "manifest_sha256": "", "deployed_tuple": { "deployment_id": "", "release_tag": "", "downstream_commit": "", "upstream_commit": "", "official_app_packs_commit": "", "final_pack_hash": "" }, "runtime_binding_sha256": "", "candidate_set_sha256": "", "backup_reference": "", "backup_sha256": "", "state": "complete_encrypted_immutable" } ``` The wrapper passes the validated marker file's raw SHA-256 to L0. The marker must bind the exact manifest, deployed tuple, mutation runtime, and candidate set. ## Runtime authorities Use the active release files on the production host: - `/release.json`, with flat keys `release_tag`, `resolved_downstream_ref`, `resolved_upstream_ref`, `resolved_official_ref`, `final_pack_hash`, `tag_target_commit`, and `version_root`, plus the isolated `recovery_release_pack_root` and `recovery_official_app_packs_root` source paths; - `/current/env/runtime.env`, whose symlink target must be exactly `/env/runtime.env` and whose deployment ID and refs must agree with `release.json`; - the setup binary inside that same `version_root`. `release.json` must be physically inside its declared `version_root`, the tag target must equal the resolved downstream commit, and all six tuple values must equal the reviewed manifest. The wrapper re-hashes the already-open file descriptors immediately before launch, so an in-place edit, path replacement, current-release switch, stale binary, or arbitrary wrapper script blocks before mutation. New multi-prod releases create dedicated, detached recovery-source worktrees for the exact downstream and official commits. They are reset and cleaned with ignored files included, then never used for final-pack building, Seed Dispatcher execution, frontend assembly, or other deployment work. The mutable downstream build worktree and ordinary official checkout remain separate release metadata; build `.tmp`, `.build-lineage.json`, and `__pycache__` artifacts can therefore neither block nor enter recovery source authority. New multi-prod releases write a deterministic `NEXUSCRM_SETUP_BIN=/run/bin/backend-setup-` into the release `runtime.env`. Before backup reset, setup apply, or cutover, the deploy builds it once from the clean normal-clone `repos/NexusCRM` checkout at the resolved upstream commit, using the exact resolved app-packs checkout and the CRM distribution `backend/distributions/cmd/setup`. The deploy checks both Git HEADs and tracked cleanliness before and after build, then requires `go version -m` to report the distribution setup main path, the exact upstream revision, and `vcs.modified=false`. A build or identity failure blocks the release; configured paths never fall back to `go run`. The CRM distribution checkout already ignores the required sibling `nexuscrm-app-packs` link. The release build therefore keeps Git VCS stamping clean; the post-build `vcs.modified=false` check remains the final authority. The L2 wrapper validates and hashes the backup marker file. It passes only that verified digest to the protected L0 command as the opaque `--backup-marker` proof; the L0 CLI does not accept the marker path or a second marker hash flag. The protected L0 command also acquires `.nexuscrm-current-cutover.lock` while revalidating the active release and holds the shared lock through the database batch. L2 intentionally does not acquire a second copy of that lock. ## Dry-run Set `NEXUSCRM_SETUP_BIN` to the absolute executable path from the live deployment, or pass the same path with `--setup-bin`. The active `runtime.env` must independently pin the same real executable through `NEXUSCRM_SETUP_BIN`, and its `ENV_ROOT` real path must equal the active `release.json` `version_root`; a caller-provided substitute is rejected. If either pin is absent, stop: publish a reviewed release that carries the immutable setup binary and both runtime pins. Do not edit the active env file or build an operator-selected binary merely to pass this gate. ```bash MANIFEST_SHA="$(shasum -a 256 /secure/3765-manifest.json | awk '{print $1}')" RELEASE_JSON="/release.json" RUNTIME_ENV="/current/env/runtime.env" python3 scripts/hubspot_authored_identity_recovery_maintenance.py \ --manifest /secure/3765-manifest.json \ --manifest-sha256 "$MANIFEST_SHA" \ --release-json "$RELEASE_JSON" \ --runtime-env "$RUNTIME_ENV" \ --setup-bin "$NEXUSCRM_SETUP_BIN" \ --receipt-out /secure/3765-dry-run-receipt.json \ --stderr-log /secure/3765-dry-run.stderr.log ``` The only successful dry-run outcome is `dry_run_verified` with all 31 reviewed exact source pairs, 31 finalized-link transitions, and 31 consumed-evidence transitions observed inside the mutation transaction before the sentinel rollback, where `31 == census.eligible == len(candidates)`. It has `committed=false`, `readback_verified=false`, and `enrollment_allowed=false`. There is no manifest-validation receipt that can authorize apply. ## Apply Re-read the active release authorities immediately before apply. The wrapper requires their raw hashes and setup-binary hash to equal the manifest and the dry-run receipt. If any differ, produce a new reviewed manifest and dry-run rather than editing an existing receipt. ```bash DRY_SHA="$(shasum -a 256 /secure/3765-dry-run-receipt.json | awk '{print $1}')" BACKUP_MARKER_SHA="$(shasum -a 256 /secure/3765-backup-marker.json | awk '{print $1}')" python3 scripts/hubspot_authored_identity_recovery_maintenance.py \ --manifest /secure/3765-manifest.json \ --manifest-sha256 "$MANIFEST_SHA" \ --release-json "$RELEASE_JSON" \ --runtime-env "$RUNTIME_ENV" \ --setup-bin "$NEXUSCRM_SETUP_BIN" \ --apply \ --dry-run-receipt /secure/3765-dry-run-receipt.json \ --dry-run-receipt-sha256 "$DRY_SHA" \ --backup-marker /secure/3765-backup-marker.json \ --backup-marker-sha256 "$BACKUP_MARKER_SHA" \ --receipt-out /secure/3765-apply-receipt.json \ --stderr-log /secure/3765-apply.stderr.log ``` Apply is authorized only by `committed_verified` with all of these exact effects: - 31 finalized links; - 31 consumed authored-evidence rows; - 31 authoritative resolver matches to the manifest's exact Lead IDs; - zero Lead, Contact, or CampaignMember records created by recovery itself; - every already-finalized and no-evidence row outside the candidate list untouched. Only that outcome sets `readback_verified=true` and `enrollment_allowed=true`. Fresh targeted CampaignMember enrollment for the committed 31 contacts is a separate downstream step and must produce its own readback. This script never claims that enrollment occurred. ## Block outcomes `rolled_back_blocked` means the L0 transaction did not commit and all durable effect counts are zero. Correct the reviewed input or runtime blocker, then start again with a new dry-run receipt. `committed_readback_blocked` is materially different: all 31 link/evidence transitions committed, but post-commit authoritative resolver proof did not complete. The receipt deliberately says `committed=true`, `readback_verified=false`, and `enrollment_allowed=false`. Do not describe this state as rolled back, do not re-apply blindly, and do not start targeted enrollment. Investigate and complete exact resolver readback against the same deployed tuple first. If L0 reports `committed=true` but its result violates the receipt contract, L2 persists a sanitized `committed_readback_blocked` receipt with `result_contract_valid=false`. Available non-sensitive counts and exact manifest source pairs are retained; missing values remain JSON `null`. They are never replaced with zero or described as rollback. If runtime starts but produces no parseable commitment result, the already reserved receipt remains a private `runtime_outcome_unknown` block with `committed=null`. It cannot authorize apply or targeted enrollment. Any block receipt uses only a stable machine-code `block_reason`; raw database errors remain only in the explicitly named mode-`0600` stderr log and are not copied into the receipt.