# Salesforce To S3

> Focused scope document for the raw `Salesforce -> S3` snapshot layer only.

## Goal

This document answers only two questions:

1. what the current `Salesforce -> S3` pipeline exports
2. what is still intentionally left out and waiting for business confirmation

It does **not** define the downstream `S3 -> TiDB` load policy.

For business confirmation, use this document together with:

1. the current raw-scope config:
   - `scripts/salesforce_migration/configs/daily_full_export.json`
2. the Salesforce global describe catalog attachment:
   - `docs/migration/attachments/salesforce_describe_objects_2026-05-17.csv`
   - `docs/migration/attachments/salesforce_describe_objects_2026-05-17.json`

## Current Source-Snapshot Contract

Current default raw profile:

- `full_phase1_source_snapshot`

Current normal Bulk CSV scope:

- `48` objects

File-object special paths:

1. `ContentDocument`
   - stays in the normal daily snapshot
2. `ContentDocumentLink`
   - exported separately through filtered chunks from `ContentDocumentId`
3. `ContentVersion`
   - metadata exported into the daily snapshot
   - `VersionData` blob stored separately in the deduplicated blob path

S3 partition shape:

```text
salesforce/full/YYYY-MM-DD/<run-label>/
salesforce/content/blobs/<ContentVersionId>
```

## Included In Current Salesforce -> S3 Scope

### Identity / Access

- `User`
- `UserRole`
- `Group`
- `GroupMember`
- `Profile`
- `PermissionSet`
- `PermissionSetAssignment`
- `ObjectPermissions`
- `FieldPermissions`
- `RecordType`

### Core Business

- `Account`
- `Contact`
- `Lead`
- `Opportunity`
- `Campaign`
- `CampaignMember`
- `Task`
- `Event`
- `Quote`
- `Contract`
- `Product2`
- `Pricebook2`
- `PricebookEntry`

### Activity / Messaging / Relation

- `TaskRelation`
- `TaskWhoRelation`
- `EmailMessage`
- `OpportunityLineItem`
- `QuoteLineItem`
- `Order`
- `OrderItem`
- `OpportunityContactRole`

### Custom Business Objects

- `Opportunity_Product__c`
- `Quote_Product__c`
- `Order_Product__c`
- `PoC__c`
- `POC_Result__c`
- `Approval_Condition__c`
- `org__c`
- `invoice_collection__c`

### History / Share

- `LeadHistory`
- `LeadShare`
- `AccountShare`
- `OpportunityShare`
- `ContactShare`
- `CampaignShare`
- `QuoteShare`
- `ContractShare`

### Files

- `ContentDocument`
- `ContentDocumentLink`
- `ContentVersion`

## Pending Business Confirmation

These objects are **not** in the current default `Salesforce -> S3` scope yet.

Add them only after business confirms they are worth preserving in the first-wave raw snapshot:

- `FeedItem`
- `FlowInterview`
- `interface_log__c`
- `integration_log__c`

## How To Confirm Scope

Business confirmation should use the two-file method:

1. this document:
   - confirms what is already included
   - highlights the pending list
2. the describe attachment:
   - provides the full Salesforce object inventory
   - lets business identify any additional object they still want preserved in S3

Decision rule:

1. if an object is needed for future validation, reconciliation, identity, permissions, sharing, activity history, or business traceability, add it to the raw `Salesforce -> S3` scope
2. if an object is only runtime noise, logs, or platform-internal exhaust, leave it out unless business explicitly asks for it

## Current Program Entry

Current unified entrypoint:

```bash
python3 scripts/salesforce_migration/run_phase1_source_snapshot.py \
  --config ~/.salesforce/readonly_config \
  --output-root /tmp/sf-export \
  --snapshot-date 2026-05-17 \
  --run-label full-phase1-source-access \
  --s3-bucket nexuscrm-sf-export-604175033442-us-west-2
```

Current default behavior:

1. base export uses `full_phase1_source_snapshot`
2. then exports `ContentDocumentLink`
3. then exports `ContentVersion` metadata
4. then uploads `ContentVersion.VersionData` blobs
5. then writes the unified source-snapshot manifest

## EC2 Deployment Shape

Current recommended host-side deployment is intentionally simple:

1. clone or pull this repository onto the EC2 host
2. create the host config file:
   - `scripts/salesforce_migration/.env_config`
   - start from `scripts/salesforce_migration/.env_config.example`
3. keep the Salesforce readonly config on the host:
   - `~/.salesforce/readonly_config`
4. run the wrapper script:
   - `scripts/salesforce_migration/run_sf_to_s3.sh`
5. schedule it with cron once per day

Important runtime assumptions:

1. the EC2 host should use an AWS instance role for S3 access
2. the wrapper script should be the only cron entrypoint
3. the wrapper script already handles:
   - lock file
   - daily date default
   - run label override
   - log file output

Minimal daily cron example:

```cron
15 2 * * * cd /opt/nexuscrm-pingcap-pack && /bin/bash scripts/salesforce_migration/run_sf_to_s3.sh
```

For hourly delta source capture, use the dedicated wrapper and keep loader out of the cron job:

1. copy `scripts/salesforce_migration/.env_hourly_config.example`
2. set the hourly output / log / state paths on the host
3. install cron through `scripts/salesforce_migration/install_hourly_delta_cron.sh`
4. each run exports the previous completed UTC hour to S3

Minimal hourly cron example:

```cron
12 * * * * cd /opt/nexuscrm-pingcap-pack && /bin/bash scripts/salesforce_migration/run_hourly_delta_to_s3.sh
```