# The transform layer is dbt-on-Databend, keyed on source and product

[ADR 0016](./0016-load-materializes-only-the-raw-layer.md) reserved staging and above for "a
transformation tool such as dbt"; the tooling research settled on dbt over SQLMesh
(no Databend engine) and Databend Task Flow (a scheduler, not a framework). Porting the first
model — `salesforce_org_monthly_revrec` and its upstreams from the BigQuery + dbt DWH — forces
the shape, so this records it.

## transform/ is a peer of ingestion/, with three engine-keyed units

`transform/` sits beside `ingestion/`. It holds three engine-keyed units, the way
`s3_bulk_load` and `gcs_bulk_load` are peer loaders:

- **`transform/dbt/`** is the primary engine: dbt running on Databend via `dbt-databend-cloud`.
- **`transform/taskflow/`** is a parallel build of the same pipeline as native Databend Task
  Flow, kept only to compare the two engines while we feel things out. It is not a second source
  of truth; if it earns its keep it can replace or supplement dbt later, otherwise it is dropped.
- **`transform/python/`** runs standalone jobs for models dbt cannot execute on this gateway
  (see below).

## Python models run as standalone jobs, not in dbt

`dbt-databend-cloud` cannot execute dbt **python** models (no DataFrame backend on this gateway),
so a python model in the BigQuery DWH cannot be ported as a dbt model here. Such a model becomes a
standalone job under **`transform/python/`**: it reads its inputs from the curated products dbt
materialized (`transform_dbt_dwh`), runs the same pandas logic, and writes its own products
(`transform_python_dwh`). It is engine-keyed like the others — generic I/O in `engine/`, one file
per job under `jobs/` (a job may emit several products — `monthly_cloud_billing_revrec` writes both
`monthly_revrec_collections` and `monthly_revrec_orders` from one shared revrec pass) — and is
dbt-free so an orchestrator (e.g. Airflow) can run a job with
just the driver and pandas. First instance: `monthly_cloud_billing_revrec` (monthly revrec
collections + orders), downstream of `salesforce_org_monthly_revrec` and `salesforce_order_daily_view`.

Because such a job is an independent producer of its own curated products (not a parity backend), it
**owns a contract** for each product it writes (ADR 0015). dbt's generator does not run python jobs,
so these contracts are hand-authored and committed under `transform/python/contracts/` rather than
generated — but every produced data product still carries one ([ADR 0003](./0003-data-products-require-versioned-observable-contracts.md)).

## Models are keyed on source (staging) and product (warehouse)

Inside `transform/dbt/`, config follows the source it reads and the product it produces, per the
[config-driven clean structure principle](../principles/config-driven-clean-structure.md):

- `sources/<source>/models/staging/` — staging models keyed on the input domain they read
  FROM, whatever its kind: a raw snapshot (`nexuscrm`, `metadb`, `bq_warehouse`), a committed
  seed (`googlesheets`), or a generated utility dimension with no upstream (`core` —
  `stg_dates`, a Databend-native date spine). The staging contract lives beside them.
- `products/<product>/models/` — warehouse models keyed on the product they produce; cross-source
  by nature, so they do not belong under any one source. The curated contract lives beside them.
- `engine/` — generic, source-free dbt config (macros, tests, the run shim).

## The transform layer owns its staging and curated contracts

Per [ADR 0015](./0015-contracts-have-a-kind-and-the-producer-owns-the-contract.md) the producer
owns the contract: the transform layer owns the **staging** and **curated** product
contracts (same schema with the layer profile,
[ADR 0007](./0007-contracts-use-one-schema-with-layer-profiles.md)). It does **not** author the
raw-snapshot contracts — those remain the load stage's ([ADR 0016](./0016-load-materializes-only-the-raw-layer.md));
dbt `sources.yml` references them. dbt **generates** these contracts under `contracts/generated/`; the
python jobs are independent producers too, so they **own** contracts for their products as well, but
hand-authored under `transform/python/contracts/` (the generator does not run python). The Task Flow
build writes no contracts — it is a parity backend reproducing dbt's products, not an independent producer.

## Coexistence with the BigQuery DWH is gradual and per-source

Migration is domain-by-domain, not a cutover. For this first slice: Salesforce CRM objects are read
from the lakehouse's `nexuscrm` instance (`nexuscrm.Org__c` / `nexuscrm.Account`) rather than the
BigQuery Fivetran tables. The historical Revenue__c-shaped rev-rec input is the exception:
NexusCRM has no populated Revenue__c rows, and the managed finance-hub mirrors lack
order/date/exchange and pre-adjust context, so strict #16 parity uses a one-off `salesforce_raw`
Revenue__c snapshot with a future-process placeholder tracked in #19. `cloud_tenants` is
snapshotted from BigQuery into `bq_warehouse_raw` rather than re-porting the GA4 → cloud_users →
cloud_tenants chain, because only its scalar columns are consumed downstream. Snapshotting a built
BigQuery table is the sanctioned shortcut for a branch we are not yet re-porting.

## The Databend adapter is community-tier; expect gaps

`dbt-databend-cloud` is community-tier and lags upstream TiDB Cloud Lake behavior. The transform
extra is pinned to the tested fork branch with gateway `warehouse` and `dbt seed` support until
those fixes land upstream. Native Task Flow on this build has no `AFTER` clause, so its "DAG" is
standalone tasks ordered externally. These are workarounds, not the design — revisit when the
adapter and the Databend build catch up.