# Local Airflow Boot Helper

Use this runbook when an agent needs to boot the local Airflow stack, validate
asset scheduling, or shut the stack down after a PR check.

## Start

```bash
cd orchestration/airflow/setup
cp .env.example .env  # only when .env is missing
docker compose up airflow-init
docker compose up -d --wait
docker compose ps
```

The UI is at `http://localhost:8080`. The default local user from `.env.example`
is `airflow` / `airflow`.

If Docker is not reachable, start the local Docker runtime first, then rerun the
commands above.

## Runtime Volumes

The compose stack persists two important `uv` runtime paths:

- `/opt/airflow/.cache/uv` for downloaded and built artifacts.
- `/opt/airflow/.venvs` for shim project environments.

`docker compose down` stops containers and preserves those volumes. Use it for a
normal shutdown. `docker compose down --volumes --remove-orphans` deletes the
volumes and creates a cold cache plus cold venvs on the next boot.

## Prewarm Shims

When volumes are cold, prewarm shims one at a time before triggering a fan-out
DAG. This avoids many parallel Airflow tasks trying to create the same venv.

```bash
cd orchestration/airflow/setup
docker compose exec airflow-worker \
  /opt/datalake/orchestration/airflow/bin/dbt-databend-shim --version

docker compose exec airflow-worker bash -lc '
  UV_PROJECT_ENVIRONMENT=/opt/airflow/.venvs/pingcap-datalake-pyjobs \
  uv run --project /opt/datalake --extra pyjobs --env-file /opt/datalake/.env \
    -- python - <<PY
import numpy
import pandas
import databend_driver
print("pyjobs imports ok")
PY
'
```

Only run the Python asset job itself when it is acceptable to execute product
logic against the configured local credentials.

## Trigger Revrec Assets

Airflow does not have Dagster-style `asset+` selection syntax. Trigger the
upstream DAG and let Airflow's asset-aware scheduler create downstream runs from
emitted asset events.

```bash
cd orchestration/airflow/setup
run_id="manual__codex_revrec_plus_$(date -u +%Y%m%dT%H%M%SZ)"
docker compose exec airflow-apiserver \
  airflow dags trigger -r "$run_id" orchestration_dbt_revrec_assets
```

Monitor the dbt DAG and downstream Python DAG:

```bash
docker compose exec airflow-apiserver python - <<'PY'
from collections import Counter
from airflow.models.dagrun import DagRun
from airflow.models.taskinstance import TaskInstance
from airflow.utils.session import create_session

dbt_run_id = "<dbt-run-id>"

with create_session() as session:
    dr = session.query(DagRun).filter(
        DagRun.dag_id == "orchestration_dbt_revrec_assets",
        DagRun.run_id == dbt_run_id,
    ).one()
    rows = session.query(TaskInstance.task_id, TaskInstance.state).filter(
        TaskInstance.dag_id == dr.dag_id,
        TaskInstance.run_id == dr.run_id,
    ).all()
    print("dbt run:", dr.state, dr.start_date, dr.end_date)
    print("dbt task states:", dict(Counter(state or "none" for _, state in rows)))

    py_runs = session.query(DagRun).filter(
        DagRun.dag_id == "orchestration_python_revrec_assets",
    ).order_by(DagRun.run_after.desc()).limit(5).all()
    print("python runs:", [(r.run_id, r.state, r.run_after) for r in py_runs])
PY
```

Asset-triggered downstream DAGs can start before the upstream DAG run ends if the
required asset-emitting tasks finish earlier than the final upstream task. If
tests must gate downstream scheduling, make sure the outlet event is emitted only
after the tests that are meant to gate it.

## Inspect Logs

Logs are mounted under `orchestration/airflow/logs`.

```bash
find ../logs -maxdepth 6 -type f | rg '<run-id>'
tail -n 200 ../logs/dag_id=<dag-id>/run_id=<run-id>/task_id=<task-id>/attempt=1.log
```

Worker process checks are useful for distinguishing dependency sync from actual
job execution:

```bash
docker top setup-airflow-worker-1 aux | \
  rg 'dbt-databend-shim|run_dbt.py|python-asset-job-shim|run_python_asset_job|uv run|COMMAND'
```

## Stop

For a normal shutdown that keeps warm `uv` cache and venv volumes:

```bash
cd orchestration/airflow/setup
docker compose down
```

For a fully cold reset:

```bash
cd orchestration/airflow/setup
docker compose down --volumes --remove-orphans
```