# Deal document history recovery

This runbook restores historical Salesforce Order and Contract document bytes
after the ordinary upload/download durability fix is deployed. It does not
infer bytes from `_System_File` metadata and it does not make an unverified
document downloadable.

## Recovery authorities

The supported authority is a **Salesforce snapshot recovery**: a hash-bound
snapshot with an exact `ContentDocumentLink` for every target parent and a
readable `ContentVersion` blob for every projected file.

An `_System_File` row, filename, expected byte count, old release directory, or
Salesforce metadata CSV alone is not source authority.

## Common safety contract

- Generate manifests and receipts outside the repository. Do not commit
  customer names, record IDs, file paths, or document bytes.
- Set `approval_ref` to the exact approved recovery change. A dry-run-only
  inventory must say that it carries no apply authority.
- Bind every invocation to the literal SHA-256 of its manifest and snapshot.
- Bind the exact `ContentDocumentLink` and `ContentVersion` metadata file sizes
  and SHA-256 values in the dry-run receipt. Apply projects only from a private
  materialized copy of those bound files.
- Write receipt and backup paths that do not already exist. The tools create
  them with mode `0600` and refuse overwrite.
- Run dry-run again immediately before apply. Apply recomputes the live plan
  and requires exact equality with the hash-bound dry-run receipt.
- Do not report recovery complete from an apply receipt. Apply ends at
  `applied_pending_authenticated_readback`; completion requires an
  authenticated backend download with exact byte-size and SHA-256 readback.

## Retired native-file recovery

The native-file incident recovery completed out of band on 2026-08-18. The
former `deal_document_native_recovery.py` entrypoint is retired and must not be
replayed: it persisted a physical S3 object key in `_System_File.StorageKey`,
while the runtime requires `StorageKey` to remain logical and prepends
`FILE_STORAGE_PREFIX` itself. Historical manifests, receipts, preimages, and
authenticated-download evidence remain under their approved private change
records; they are evidence, not reusable apply authority.

Do not restore the retired entrypoint for another incident. A future native
recovery workflow requires a newly reviewed manifest version that separates a
logical `StorageKey` from the physical S3 object key and machine-enforces the
authenticated backend download before reporting completion.

## Salesforce history

Use `deployments/offline/runtime/deal_document_salesforce_recovery.py` once per
parent object (`Order` or `Contract`). The target manifest operation is
`deal-document-salesforce-history-recovery` and binds:

- exact target parent keys and parent object/table/attachment kind;
- snapshot source region;
- the Prod customer-file destination bucket, region, and prefix.

The snapshot manifest must resolve both `ContentDocumentLink.csv.gz` and
`ContentVersion.csv.gz`. Dry-run scopes links to the resolved target ParentIds,
downloads every unique source blob to compute its SHA-256, and checks that each
destination is absent or byte-identical. It blocks when any target has no
complete projectable link or any blob is unreadable/conflicting. Snapshot
`ContentSize` is an exact byte contract, including zero: a non-empty source blob
cannot be projected with missing or zero size metadata. This dedicated recovery
also evaluates the exact snapshot links even when the parent already has a
different healthy file; parent-level coverage is not evidence that the missing
historical document was recovered.

```bash
python3 deployments/offline/runtime/deal_document_salesforce_recovery.py \
  --env-file /secure/prod-runtime.env \
  --target-manifest /secure/order-history-targets.json \
  --expected-target-manifest-sha256 <target-manifest-sha256> \
  --snapshot-dir /secure/salesforce-snapshot \
  --snapshot-manifest-name manifest.json \
  --expected-snapshot-manifest-sha256 <snapshot-manifest-sha256> \
  --receipt-output /secure/order-history.dry-run.json
```

Apply requires `--dry-run-receipt`,
`--expected-dry-run-receipt-sha256`, `--backup-output`, `--receipt-output`, and
`--actor`. It copies and verifies source blobs only after the additive,
create-only `_System_File` rows have been inserted inside an uncommitted
transaction. Exact deterministic projection IDs are also bound in dry-run and
rechecked immediately before the transaction. A duplicate or concurrent row
therefore fails before any S3 mutation; successful blob verification is
followed by the database commit and an exact-ID database readback of every
planned row. The generic parent-level projection audit is retained as an
additional check, but cannot substitute for that exact readback.

## Acceptance and rollback boundary

For every target, retain the manifest, dry-run receipt, preimage backup, apply
receipt, and authenticated-download readback under the approved change record.
The preimage is rollback authority for database metadata; source and
destination blob hashes are byte authority. Never delete source or destination
objects as an automatic rollback action. If post-readback fails, stop, preserve
both blobs and receipts, and restore metadata only through a separately
approved exact-preimage change.