# Deal Document Storage Lifecycle Contract **Status:** implementation candidate; not yet deployed to Prod **Bound to:** L2 base `16e1f57352556853564de635a4e9303921a2adc5` and L0 base `1d39fdf76e3036035d29079476e3f3176fac6ea0` **Last updated:** 2026-08-08 ## Incident and ownership boundary Deal-document metadata is durable database state, but Prod previously wrote the corresponding filesystem bytes below `releases//runtime-data`. Release promotion changed the active root and retention could delete the old root. That violated the required lifetime: a customer file must outlive every release that can reference it. Salesforce history is a separate source-authority path. The daily full export can return a syntactically successful but visibility-collapsed `ContentDocument` result when its source principal loses file-wide visibility. Native storage durability and Salesforce history completeness therefore have separate gates and separate recovery actions. ## Single authorities | Concern | Authority | Lifecycle | |---|---|---| | Native file metadata | `_System_File` row, including per-row storage authority | database lifetime | | New Prod customer-file bytes | `s3://nexus-crm-files-prod/prod/customer-files/v1/` | customer-data lifetime; independent of release retention | | Pre-S3 filesystem rows | `multi_prod/shared/data/object-storage` and `multi_prod/shared/data/uploads` | compatibility lifetime; never release retention | | Public Connect artifacts | `releases//runtime-data/connect-artifacts` | immutable release lifetime | | Public artifact catalog | `NEXUSCRM_DOWNLOAD_ARTIFACTS` plus `NEXUSCRM_DOWNLOAD_ARTIFACT_STORAGE_ROOT` | exact candidate lifetime | | Salesforce file-history source | full snapshot manifests and blobs from an admitted source principal | source-snapshot lifetime | The public `/downloads` handler uses the dedicated artifact root when it is configured. It must not use the customer-file S3 authority. Customer file APIs continue to resolve each `_System_File` row through the generic L0 file-storage authority. ## Ordinary Prod storage gates Before a new Green slot starts, ordinary Prod promotion must use the runtime identity to put random bytes to the exact customer-file S3 prefix, get and compare those bytes, delete the visible canary object, and retain the receipt in release metadata before Connect artifact materialization or backend startup. Ordinary Prod promotion does not copy legacy filesystem roots and does not run the no-limit all-Active historical file census. Those are separately governed data-migration and recovery operations. Inside the cutover lock, promotion activates and proves Green, switches Nginx directly, proves the public Green route, then stops Blue. It does not restart Blue or perform customer-file migration during that cutover interval. ## One-time legacy migration authority The legacy migration primitive accepts only the current marker-selected release-local roots or an already-shared root, compares the lexical absolute roots before any normalization, rejects symlinks in every existing source/target path component via `lstat`, rejects special files, and uses exclusive publication plus exact readback. Contract `prod-legacy-file-storage-migration/v2` fsyncs each temporary file before publication, fsyncs every newly created directory and its parent, and fsyncs the target directory after hard-link publication and temporary-name cleanup. A directory fsync failure blocks before the migration receipt or file census can accept the bytes. A missing source remains visible to a separately invoked all-Active census; it is not treated as a successful recovery. Source bytes remain intact for Blue rollback until ordinary retention is separately authorized. This repository intentionally does not expose this primitive as an ordinary deployment mode. Designing and operating the eventual one-time migration or recovery command is deferred to a separately reviewed data-migration change; ordinary promotion neither requires nor produces its receipt. The bucket is versioned, so canary object versions and delete markers remain an audit trail under `.readiness/`. The runtime role intentionally receives no `s3:DeleteObjectVersion`; lifecycle management for that readiness-only prefix is a separate bucket-owner decision. The least-privilege runtime policy is `deploy/online/runtime/prod_file_storage_iam_policy.json`. Applying it to the actual Prod instance role is an external infrastructure step and must be proved by the canary receipt, not inferred from policy text. That policy binds the exact Salesforce archive authority `s3://nexuscrm-sf-export-604175033442-us-west-2/salesforce/content/blobs/` with read-only `s3:GetObject` permission (the permission used by exact-key `HeadObject` and file reads). The Prod workflow and runtime-env preparer bind the same fixed bucket and region; an override not covered by the tracked policy blocks before runtime materialization. The deployment validator independently re-reads `FILE_STORAGE_ADDITIONAL_S3_AUTHORITIES_JSON` and applies the same IAM binding before running the S3 authority canary, so a stale or altered runtime env cannot bypass prepare-time admission. The archive grant has no Put or Delete permission. Prod declares the Salesforce history bucket as a second, source-scoped S3 authority under `FILE_STORAGE_ADDITIONAL_S3_AUTHORITIES_JSON`. It admits only `SourceType=salesforce` rows below `salesforce/content/blobs/`; it does not change the native upload destination and cannot authorize native rows in the archive bucket. A separately invoked census binds both authorities into its receipt. ## Salesforce history gates The full snapshot config sets a minimum `ContentDocument` cardinality of 5,000, below the last known complete count but far above the observed visibility collapse. The exporter blocks before publishing a manifest or S3 directory when the source result falls below that floor and reports the source principal as the likely authority boundary. The dedicated `ContentDocumentLink` exporter seeds both `Order.csv.gz` and `Contract.csv.gz`; `ContentVersion` then unions base documents and linked documents. It blocks before publishing a manifest unless an S3 blob authority is resolved and `uploaded + already-present == verified == metadata rows` with zero failed or unattempted blobs. Existing objects are checked at the exact `/` key; basename-only recursive listings are not authority. Every accepted object must match `ContentSize` and an exact byte readback against the Salesforce checksum when available, or against fresh source bytes otherwise. New uploads additionally require persisted SHA-256 metadata plus exact-key post-upload readback. The manifest binds these facts to the `exact-key-size-checksum-v1` contract, and replacement projection rejects legacy counter-only manifests before database access. Full-snapshot projection re-reads the actual `ContentDocument.csv.gz` IDs and enforces the same 5,000-row floor immediately before its mutation path; it does not trust an older manifest's self-declared minimum. It also computes candidate and projectable link counts before deleting old projected rows, so empty, collapsed, or partial replacement scopes fail closed. A successful job is not evidence of history completeness unless these gates passed and required blobs were published. Metadata-only ContentVersion manifests are never published: both the direct `--skip-blob-upload` flag and the phase-1 `--skip-content-version-blob` alias fail closed. A diagnostic run must omit the entire ContentVersion layer with `--skip-content-version`. The dedicated Salesforce history recovery command derives its source bucket and prefix from the hash-bound snapshot manifest and combines them with the hash-bound target manifest's source region. Dry-run and apply both require that tuple to equal the fixed read-only archive authority before any database audit or S3 read. The normalized provider, bucket, region, and prefix are embedded in the dry-run, preimage backup, and apply receipts; a hash-bound manifest cannot redirect recovery to another currently accessible S3 location. ## Recovery and rollout order 1. Recover the currently missing native bytes into a stable authority and independently re-run the all-Active readiness census. 2. Restore an admitted Salesforce export principal with file-wide visibility, rerun a full snapshot, and prove `ContentDocument`, Order links, Contract links, ContentVersion metadata, and blobs before projection. 3. Run the separately governed legacy migration or recovery operation needed to place retained filesystem bytes under a durable authority and emit its own receipt. 4. Publish/adopt the L0 artifact-store split and promote the L2 candidate that sets S3 customer storage and binds legacy reads to the stable shared roots. 5. Apply the exact prefix-scoped runtime IAM policy. 6. Run Formal Staging and Prod promotion. The Prod S3 canary must pass before Green starts; ordinary deployment does not rerun legacy migration or the historical all-Active census. 7. Prove an authenticated upload and download across backend restart and a second release promotion, then retire one inactive release and prove the same file still opens. Until all seven steps pass against exact release identities, the correct status is **code candidate**, not **Prod fixed**.