# Ingestion spans multiple source systems and object stores

The first ingestion pipeline (TiDB Cloud → S3 → Cloud Lake raw) could be read as one
monolithic path. Onboarding BigQuery → GCS → Cloud Lake raw forces the general shape, so
this records it.

## Two independent axes

Ingestion is split on two independent axes, not one:

- **Extract families are keyed on the source system.** `ingestion/extract/tidbcloud/`
  introspects TiDB via `information_schema` and exports to S3; `ingestion/extract/bigquery/`
  introspects BigQuery via `INFORMATION_SCHEMA` and exports to GCS with `bq extract`. Each
  owns the `source` contract for the snapshot objects it writes and the run manifest that
  is the physical handoff.
- **Loaders are keyed on the object store they read.** `ingestion/load/s3_bulk_load/`
  reads S3; `ingestion/load/gcs_bulk_load/` reads GCS. A loader is not tied to a source
  system — TiDB→S3 and BigQuery→GCS are just two producers of object-store bulk extracts.

The source system that an extract reads and the object store a loader reads are chosen
independently; a new source system or a new object store adds one unit on its own axis,
not a new end-to-end pipeline.

## Externally-managed sources still get an extract stage

BigQuery is externally managed — we do not own or produce the source table. We still run an
extract stage that snapshots it to GCS, and that stage owns the GCS `source` contract,
because [ADR 0015](./0015-contracts-have-a-kind-and-the-producer-owns-the-contract.md) ties
contract ownership to the producer of the *asset*: we produce the GCS Parquet objects, so we
own their `source` contract, even though we do not own the upstream BigQuery table. BigQuery
has no enforced primary keys, so the snapshot source declares them in `source.yaml`; the
loader needs a key for raw-snapshot dedup detection. The load stage still materializes only
the raw layer and owns only the raw-snapshot `product` contract
([ADR 0016](./0016-load-materializes-only-the-raw-layer.md)).

## Storage-agnostic concerns live in common

Adding a second loader showed which load concerns are storage-agnostic: the Cloud Lake
(Databend) connection, the load config base (target databases, setup files, source-contract
resolution), and table classification are identical across loaders and now live in
`ingestion/common/` (`cloud_lake.py`, `load_config.py`, `classification.py`). Only the
object-store specifics differ per loader — the read credentials and the `COPY INTO`
connection block (S3 access keys vs. a base64 GCS service-account credential, and Databend's
`gcs://` scheme). This is the implementation expression of the
[config-driven clean structure principle](../principles/config-driven-clean-structure.md):
the engine decides *how*, source config decides *what*, and genuinely shared code is shared
rather than duplicated per loader.