# transform/taskflow — native Databend Task Flow build

This is the **parallel implementation** of the transform pipelines (`salesforce_org_monthly_revrec`
and `salesforce_order_daily_view`) using Databend's native Task Flow, alongside the dbt build under
`../dbt`. It exists so we can compare
the two execution engines directly (ADR 0019) — same SQL logic, two orchestration models, output
into separate databases for row-by-row parity.

| | dbt (`../dbt`) | Task Flow (here) |
|---|---|---|
| Output databases | `transform_dbt_stg`, `transform_dbt_dwh` | `transform_taskflow_stg`, `transform_taskflow_dwh` |
| Reads from | `nexuscrm_raw`, `salesforce_raw`, `metadb_raw`, `data_foundation_finance_raw`, `bq_warehouse_raw`, `transform_seeds` | same |
| Model = | dbt model file | one `CREATE TASK` per model |
| Dependency ordering | dbt ref() graph | see limitation below |
| Observability | dbt run results / logs | `system.task_history` |

## Layout
- `flows/transform.flow.sql` — **generated** artifact: the `CREATE TASK`
  statements (one per model). Regenerated by `deploy.py`; do not hand-edit.
- `setup/databend/001_databases.sql` — the Task Flow output databases.
- `deploy.py` — generates the flow artifact from the dbt-compiled SQL, creates the tasks, runs
  them in topological order (polling `system.task_history`), then suspends them.

## Run it
```bash
# 1. dbt must have compiled (provides the SQL bodies) and seeds + cloud_tenants fixture must exist:
uv run --extra transform --env-file .env -- \
    python transform/dbt/engine/run_dbt.py compile --project-dir transform/dbt --profiles-dir transform/dbt
uv run --extra transform --env-file .env -- \
    python transform/dbt/engine/run_dbt.py seed --project-dir transform/dbt --profiles-dir transform/dbt

# 2. Deploy + run the Task Flow pipeline:
uv run --env-file .env -- python transform/taskflow/deploy.py

# 3. Parity vs dbt (expects 0 rows):
uv run --extra transform --env-file .env -- \
    python transform/dbt/engine/run_dbt.py test --project-dir transform/dbt --profiles-dir transform/dbt \
    --select parity_vs_taskflow
```

## Why the SQL bodies come from dbt-compiled output
The task bodies are the dbt-compiled SQL, so both engines run identical, dialect-verified SQL. The
goal here is to compare *orchestration and materialization*, not to re-author the SQL. (If you want
a truly independent SQL expression for a stronger parity signal, hand-author the bodies instead.)

## Limitation: no native task DAG on this Databend build
TiDB Cloud Lake's Databend build does not support the task `AFTER` clause, so a native task-DAG
(root + `AFTER` children) cannot be declared. Each model is therefore a standalone task with a
1-year schedule (so it never auto-fires) and `deploy.py` drives them in topological order. When a
Databend build with `AFTER` support is available, the flow can collapse into a real root→children
DAG and `deploy.py` can stop orchestrating order by hand.