# transform/python — Python (pandas) transform jobs

The third transform engine (ADR 0019), for models the dbt adapter **cannot** run: dbt-databend-cloud
has no python-model backend on this gateway, so a python model from the BigQuery DWH is ported here
as a standalone pandas job instead of a dbt model.

Unlike `taskflow/`, this is **not** a parity reimplementation of the dbt pipeline — it is a genuine
downstream step. A job reads curated products dbt already built (`transform_dbt_dwh`), runs its
pandas logic, and writes its own products into `transform_python_dwh`. Because it is an independent
producer (not a parity backend), it **owns a contract** for each product it writes (ADR 0003/0015).

## Layout
- `engine/databend_io.py` — generic, product-free I/O: `read_df()` (relation → typed DataFrame,
  coercing the all-VARCHAR seed amounts / DECIMALs the driver returns) and `replace_table()`
  (DataFrame → `CREATE OR REPLACE TABLE`, write-truncate, over the Rust driver).
- `jobs/<job>.py` — one file per job (a job may emit several products): the ported pandas logic plus
  a `run()`/`main()` that wires read → process → write. Exposes `run(...)` so an orchestrator can import it.
- `contracts/<product>.curated.contract.yaml` — the hand-authored curated contract each product
  carries. dbt's generator does not run python, so these are committed by hand (ADR 0019), not
  generated; validate them against `schemas/data-product-contract.authored.v0.schema.json`.
- `setup/databend/001_databases.sql` — the `transform_python_dwh` output database.

## Jobs
| Job | Reads (`transform_dbt_dwh`) | Writes (`transform_python_dwh`) |
|---|---|---|
| `monthly_cloud_billing_revrec` | `salesforce_org_monthly_revrec`, `salesforce_order_daily_view` | `monthly_revrec_collections`, `monthly_revrec_orders` |

## Run it
The jobs need only pandas/numpy + the driver — no dbt (so an Airflow worker stays light):
```bash
uv sync --extra pyjobs
uv run --extra pyjobs --env-file .env -- \
    python transform/python/jobs/monthly_cloud_billing_revrec.py
# options: --billing-month YYYY-MM (default: latest), --order-date YYYY-MM-DD (default: latest),
#          --revenue recognised|predicted, --dry-run, --verbose
```

## Parity
The output-level parity harness compares the ported Python logic over legacy BigQuery inputs and
Databend inputs before anything is written:

```bash
uv run --extra pyjobs --env-file .env -- \
    python transform/python/parity/monthly_cloud_billing_revrec.py

uv run --extra pyjobs --env-file .env -- \
    python transform/python/parity/monthly_cloud_billing_revrec.py \
    --billing-month all --order-date latest --revenue all --summary-only --allow-empty-matches
```

Use `latest` selectors for the production-equivalent comparable run. Use `--billing-month all`
with a selected order snapshot to prove every common billing month. Full multi-month x multi-date
cartesian sweeps are guarded by `--allow-cartesian` because they are much larger and should be
intentional. Discovery output reports BigQuery-only and Databend-only billing months/order dates
so non-comparable parameter values are visible. `--summary-only` still includes full failing case
details when a case does not pass.

## Status: ad-hoc for now
Run by hand today; the intended home is an orchestrator (Airflow) that calls `run()` per schedule.
The pandas logic is a faithful port of the BigQuery model — only the I/O is Databend instead of
BigQuery.