# Orchestration on Airflow 3 Assets - Current-State Companion

> **Status: companion / current-state evidence.**
> Date: 2026-06-05.
> Binding authority: [ADR 0020](../adr/0020-orchestration-is-contract-compiled-airflow-assets.md).
> Scope: local v1 orchestration status after ADR 0020, not production deployment
> design.
> This document does not reopen the orchestration engine decision. ADR 0020 already
> settles Airflow 3.2 Assets as the local-first orchestration projection for v1.
> This is not a new or competing design authority; it records the current
> implementation surface, what evidence it has produced, what remains incomplete,
> and which later decisions are still deferred.

## 0. Bottom line

The local v1 orchestration direction is no longer hypothetical: the repo now has
an Airflow 3.2 dev stack under `orchestration/airflow/`, thin DAG registration
modules, contract-to-Asset compiler code, dbt/Cosmos, transform/python, ingestion
load, and ingestion extract slices.

The governing rule remains: **Git-authored contracts are the source of truth;
Airflow Assets are a runtime projection, not a second authority.** DAGs and
orchestration config may describe local schedules, pools, shims, and connection
IDs, but business dependencies and product identity must come from contracts.

The current work proves the shape is viable for local v1. It does not settle
production hosting, the observed-lineage event model, generated documentation
shape, detailed approval workflow, or a bespoke agent permission model.

## 1. Authority and relation to the corpus

This file is a companion to the settled orchestration decision:

| Surface | Authority |
|---|---|
| Orchestration engine shape for local v1 | ADR 0020 |
| Main capability map and orchestration plane responsibilities | `docs/design/v1-lakehouse-architecture.md` |
| Current implementation layout and local runbook | `orchestration/README.md`, `docs/agents/airflow-local-boot.md` |
| Contract source of truth | ADR 0004 and authored contract schemas |
| Agent action API shape | `docs/design/v0-agent-runtime-api.md` |

This document should be read as evidence and runway. If it disagrees with an ADR,
the ADR wins and this file should be corrected.

## 2. Durable read path

The durable local v1 orchestration read path is:

```text
Git-authored contracts
  -> contract-to-Asset compiler / orchestration source config
  -> thin Airflow DAG registration
  -> task wrapper or per-unit shim
  -> data product build or extract/load action
  -> Airflow Asset event metadata
  -> runtime inspection, lineage candidate, and agent action request surfaces
```

Forbidden durable paths:

- Hand-authored DAG dependencies that duplicate product lineage from contracts.
- Source or product literals embedded in reusable orchestration engine code.
- Airflow metadata treated as the source of truth for contract identity,
  ownership, freshness, governance, or lineage.
- Airflow REST endpoints exposed as the agent-facing API without the product-first
  action-request layer defined by the agent runtime API.

Allowed supporting paths:

- `orchestration/airflow/sources/` files for local schedules, pools, connection
  IDs, manifest fixtures, and per-slice registration config.
- Per-unit `uv` shims that keep dbt, Python jobs, load, and extract runtimes out
  of the Airflow service environment.
- Local Docker Compose setup for proving the control-plane shape.

## 3. Current state inventory

| Surface | Current state | Evidence | Disposition |
|---|---|---|---|
| Architecture decision | Airflow 3.2 Assets are settled for local v1 orchestration. | ADR 0020 | Binding. |
| Orchestration unit | `orchestration/airflow/` exists with `engine/`, `dags/`, `sources/`, `setup/`, `bin/`, `logs/`, and `plugins/`. | `orchestration/README.md` | Active local control-plane unit. |
| Contract compiler | `contract_assets.py` discovers ingestion and transform contracts, emits Airflow asset specs, explicit external upstream Assets, graph edges, and stale-asset diff helpers. | `orchestration/airflow/engine/contract_assets.py` | Durable projection compiler. |
| DAG registration | DAG files stay thin and load slice config from `orchestration/airflow/sources/`. | `orchestration/airflow/dags/*.py` | Active pattern. |
| Local Airflow stack | Docker Compose stack, API/UI, worker image, and local boot helper exist. | `orchestration/airflow/setup/`, `docs/agents/airflow-local-boot.md` | Local-only, not production hosting. |
| Hello smoke | Producer emits a hello Asset and consumer is scheduled by the emitted event. | `orchestration/README.md` | Smoke test only. |
| dbt revrec slice | Cosmos loads a committed manifest fixture, selects revrec models, and uses contract `x-datalake://...` asset identities through node converters. | `orchestration/airflow/dags/dbt_revrec_assets.py`, `orchestration/airflow/sources/dbt-revrec.yaml` | Active proof slice; test-gate semantics still need hardening. |
| dbt warehouse shim | Cosmos calls `dbt-databend-shim`, which delegates to `transform/dbt/engine/run_dbt.py` through `uv`. | `orchestration/README.md` | Required local workaround until adapter support is settled. |
| transform/python slice | The monthly revrec Python job runs through `python-asset-job-shim`, consumes dbt-curated products, emits two curated products, and attaches row-count metadata. | `orchestration/README.md` | Active proof slice. |
| ingestion load slice | Manifest-driven S3 loads and the scheduled external-incremental poll are registered from `load-assets.yaml`. | `orchestration/README.md` | Active local slice; event watcher remains future work. |
| ingestion extract slice | BigQuery extract is synchronous; TiDB Cloud extract uses a deferrable path and emits a manifest Asset. | `orchestration/README.md` | Active local slice; production credentials and operations are not settled. |
| Secrets | Runtime values belong in Airflow Connections, Variables, secret backend entries, or local `.env`, not committed source config. | `orchestration/README.md` | Active boundary. |

## 4. Decisions already taken

1. **Airflow is the local v1 orchestration projection.**
   ADR 0020 settles Airflow 3.2 Assets for the local control-plane surface.
   Production hosting remains separate.

2. **Contracts stay the source of truth.**
   Airflow asset names, URIs, groups, producer hints, freshness metadata, and
   dependency edges are compiled from authored and generated contracts. Airflow is
   not allowed to become the product registry.

3. **DAG files stay thin.**
   Product dependencies belong in contracts. DAG modules register slices and bind
   operators, shims, schedules, and local config.

4. **Runtime environments are per unit.**
   The repo already uses shims and `uv` environments so Airflow, dbt, Python jobs,
   extract, and load do not require one shared dependency set.

5. **External-incremental load starts as scheduled polling.**
   The loader already self-watermarks. Queue-backed `AssetWatcher` support is a
   later integration after bucket notifications and compatible queue triggers are
   designed.

6. **Airflow can execute agent-requested work, but it is not the agent API.**
   Agents use the product-first runtime API and action-request model. Airflow may
   be the executor for reruns, clears, backfills, or contract-update handoffs after
   governance, approval, and audit rules approve the request.

## 5. Current gaps and risks

### Test gates and materialization semantics

The design goal is that blocking tests pass before a downstream product Asset is
materialized. The current slices prove Asset registration and event scheduling,
but they still need a precise gate rule per producer family:

- dbt/Cosmos: ensure outlet Asset events happen only after the intended model
  tests have passed.
- transform/python: define which job checks are blocking versus degrading before
  emitting product Assets.
- ingestion load: distinguish successful raw load, schema drift abort, and
  proposed-contract handoff.
- ingestion extract: emit manifest Assets only after the manifest is complete and
  durable.

### Lineage depth

Product-level lineage can be derived from contract upstreams and Airflow inlets /
outlets. The observed-lineage event model is still deferred. OpenLineage remains
a likely target, but this file does not adopt it.

### Freshness and deadline mapping

Contracts declare `freshness.max_age`, `schedule_ref`, and `degrade_after`.
Airflow 3.2 has newer deadline and asset features, but the compiler still needs a
repo-level mapping that says which contract field drives schedule, alerting,
degraded state, and runtime-served contract view status.

### Stale asset cleanup

ADR 0020 requires stale-asset cleanup when a contract is removed or renamed. The
compiler exposes a stale-name diff helper, but the operational procedure and gate
for retiring metadata are not yet a completed design.

### Production operations

The Docker Compose stack is a local proof surface. It does not decide managed
Airflow versus self-hosted Airflow, production secret backend, worker topology,
network boundaries, audit retention, or disaster recovery.

### Agent action execution

Airflow can run operational commands, but approval workflow and a specialized
agent permission model remain deferred. The durable boundary is still:

```text
agent -> product runtime API action request -> governance / approval decision
  -> approved orchestration execution -> audit and runtime-served status
```

## 6. Runway and gates

| Phase | What lands | Gate |
|---|---|---|
| Current-state documentation | This companion doc and the design index link. | It names ADR 0020 as authority and no longer presents Airflow adoption as hypothetical. |
| Compiler hardening | Corpus compile command or test fixture for all current contracts. | It proves one Asset per accepted v0 contract profile, explicit external upstream Assets, no source literals in engine code, and deterministic manifest output. |
| Local stack smoke | Hello Asset smoke plus documented boot/shutdown path. | `docs/agents/airflow-local-boot.md` can be followed on a clean local stack. |
| Revrec proof | dbt/Cosmos slice triggers the transform/python slice through Asset events. | Downstream scheduling is caused by product Asset events, not hand-authored ordering. |
| Test-gated materialization | Blocking tests gate product Asset emission. | A failing blocking test prevents the downstream Asset-scheduled run; degrading tests mark state without blocking. |
| Ingestion proof | Extract manifest Assets schedule manifest-driven loads; external-incremental load remains scheduled poll. | Manifest metadata resolves at runtime; schema drift writes a proposed contract/action request and aborts without silent overwrite. |
| Lineage decision | OpenLineage or another event model is adopted deliberately. | New ADR updates `CONTEXT.md`, v1 architecture, and runtime-view expectations together. |
| Agent execution bridge | Approved product action requests can dispatch orchestration operations. | Airflow is behind the product runtime API, and approval/audit semantics are visible before mutation. |

## 7. Open questions

1. Should production use managed Airflow or a self-hosted deployment?
2. Should the compiler register dynamically from contracts, from a precompiled
   manifest, or from generated DAG files for debuggability?
3. Should the dbt warehouse shim remain a local wrapper, or should the
   Databend/TiDB Cloud Lake adapter be forked or upgraded so the wrapper can die?
4. Which observed-lineage event model should be adopted when runtime lineage
   integration begins?
5. What exact freshness-to-deadline mapping should runtime-served contract views
   expose?
6. What stale-asset cleanup operation proves a rename or removal is complete?
7. When, if ever, should external-incremental load move from scheduled polling to
   queue-backed event watching?

## 8. References

- [ADR 0020: Orchestration is contract-compiled Airflow Assets](../adr/0020-orchestration-is-contract-compiled-airflow-assets.md)
- [V1 lakehouse architecture](./v1-lakehouse-architecture.md)
- [Orchestration README](../../orchestration/README.md)
- [Local Airflow boot helper](../agents/airflow-local-boot.md)
- [V0 agent runtime API](./v0-agent-runtime-api.md)