# Externally-managed sources load incrementally with inferred, drift-gated contracts [ADR 0017](./0017-ingestion-spans-source-systems-and-object-stores.md) covered loading an object store we fill ourselves via an extract stage (with a run manifest as the handoff). Some buckets are filled by a pipeline we do not own — e.g. a daily BigQuery→GCS export landing at `gs://metadb/metadb/billing/billing2_invoices/run--part-…parquet`. Others are reviewed one-off freezes that are intentionally not promoted into recurring extract stages, such as a historical parity snapshot preserved in an ad-hoc bucket. Loading those needs a different shape, recorded here. The load stage still materializes only the raw layer and owns only the raw-snapshot product contract ([ADR 0016](./0016-load-materializes-only-the-raw-layer.md)). This `billing2_invoices` export is the externally-produced counterpart to the `bq extract` example in ADR 0017: the same logical table, but here a pipeline we do **not** run lands the objects, so we produce nothing and its `source` contract is *provided* (below) rather than owned by an extract stage. The two examples deliberately share a table to make that ownership contrast concrete — `ingestion/extract/bigquery/` (we run the export, extract owns the contract) versus `ingestion/load/gcs_bulk_load/sources/metadb/` (we do not, the contract is provided). ## No extract stage, so no manifest For an externally-managed or one-off provided source there is no extract stage and no run manifest. The loader instead **discovers** objects by listing the bucket prefix and **derives** the schema by inferring a sample object (Databend `INFER_SCHEMA` over an external stage). The same stage is reused for the `COPY`, so the object-store credential lives server-side in the stage rather than inlined per statement. A helper script committed for audit/reproducibility of a one-off freeze does not make the freeze a managed extract stage; a managed extract stage owns source config, setup, cadence, and the source contract it produces. ## The source contract is inferred, solidified, and drift-gated The external `source` contract is "provided" (CONTEXT), but hand-authoring an exporter's schema is error-prone and it drifts. So: infer it once (`infer-source`) and **solidify** it as a committed contract under the load source's `contracts/upstream/`. On every later run the loader re-infers and compares to that committed contract. A match proceeds; a change does **not** silently overwrite the contract or load against a stale one — it writes a **proposed** contract under `contracts/proposed/` and aborts, routing the change through a contract update process. This keeps Git the source of truth for contracts ([ADR 0004](./0004-git-is-source-of-truth-for-contracts.md)) and matches the PR-generating contract-engine stance: tooling proposes contract changes, review promotes them. Promotion is the deliberate human action: once the proposed change is reviewed and accepted, move the `contracts/proposed/` file to `contracts/upstream/` (equivalently, re-run `infer-source`, which re-writes the committed contract). Only these explicit commands write `contracts/upstream/`; the automatic load path never does. If a reviewed one-off freeze needs lifecycle, provenance, retention, freshness, or executable parity gates that differ from the external-incremental defaults, those settings live in the load source's `source.yaml` as source/raw contract overrides. Regeneration must preserve them; the committed source and raw contracts are generated artifacts, not the only copy of the decision. ## A pluggable incremental strategy, delta against the warehouse How "newer" is determined is a **pluggable strategy** (`incremental.strategy` in the source config), because different external producers signal increments differently. Two ship: - **`filename`** — an `extraction_ts` (epoch ms) captured from each object name; objects sharing it are one snapshot, loaded together (e.g. metadb's `run--part-…parquet`). - **`last_modified`** — each object's storage last-modified time as the cursor, each file its own increment (a dltHub filesystem-incremental shape, for buckets where files arrive over time). Every strategy normalises its cursor to a BIGINT epoch-ms **watermark column** plus a derived timestamp, so the delta is uniform: load only increments strictly newer than `max()` already in the raw table. The warehouse is the watermark, so the result is correct after partial failures with no separate state store. Loading an increment is idempotent (clear that increment's rows by the strategy's selector, then a `FORCE` re-`COPY` of its files), and increments accumulate in raw. The strategy decides the increment grouping, watermark column, provenance columns, clear selector, and stamp; the orchestration (list → infer → drift-gate → delta → COPY+stamp) is otherwise identical. ## The mode is shared across object-store loaders The orchestration and strategies live in `ingestion/common/external/`; each loader supplies a small storage adapter (how to list objects-with-metadata and how to build the Databend external stage). So both `s3_bulk_load` and `gcs_bulk_load` get the external-incremental mode — differing only in S3 vs GCS listing and stage credentials — alongside their unchanged manifest-driven path. This closes the gap where the manifest-driven loaders could not ingest a bucket we do not extract. ## The read credential is a load-stage concern here When an extract stage exists it owns the object-store credentials (ADR 0017). With no extract stage, the loader owns its own read credential: a service account + key + an `objectViewer` binding on the external bucket, provisioned by the load unit's `setup/grunt/` (the bucket itself is not created — it is externally owned). This keeps credential provisioning as IaC next to the unit that needs it.