# AGENTS.md

## What this repo is

Apache Airflow DAG repository for TiDB Cloud (PingCAP) infrastructure automation — creating/destroying dedicated TiDB regions, o11y stacks, BYOC regions, and next-gen regions on AWS and Azure. DAGs are served to a Kubernetes-hosted Airflow 2.9.2 cluster via **git-sync from the `main` branch** — there is no build step; merging to `main` is the deployment.

---

## Developer commands

```bash
make fmt    # autoflake → isort → black on dags/**/*.py (run before committing)
make check  # flake8 on dags/**/*.py (ignores E501, W291, W503)
```

There is no `make install`. Manually install lint/format tools into a Python 3.9.7 venv:
```
flake8 autoflake isort black
```

**There are no tests.** No test runner, no test files.

---

## Repository layout

```
dags/
  trigger/          # auto_trigger.py — polls MySQL every minute, emits Dataset events
  deploy/
    dedicated/
      common.py     # shared @task functions used by all env DAGs
      dev/          # DAG files + .env for dev
      staging/      # DAG files + .env for staging
      prod/         # DAG files + .env for prod
    o11y/           # o11y create/destroy DAGs + .env
  utils/            # github.py, jenkins.py, lark.py, database.py, tasks.py, url.py, aws_az_check.py
  examples/         # example DAGs (not deployed as real workflows)
values.yaml         # Helm chart overrides (tracked); chart/ directory is gitignored
```

DAG files import from `utils.*` and `deploy.dedicated.common` as bare module paths — Airflow's DAG folder is the Python path root.

---

## Environment configuration

Each environment directory contains a **committed `.env` file** read at DAG parse time via `python-dotenv dotenv_values()`. The `.env` determines `ENV`, bucket names, Lark chat IDs, `AUTO_DEPLOY_REPO_BRANCH`, etc. There are no runtime environment variables for these — the directory a DAG lives in defines its environment.

- `dev/` and `staging/` → `AUTO_DEPLOY_REPO_BRANCH=main`
- `prod/` → `AUTO_DEPLOY_REPO_BRANCH=prod`

---

## Runtime secrets

All Airflow connections and variables are fetched from **AWS Secrets Manager** (not the Airflow DB):
- Connections prefix: `autonomous/airflow/connections`
- Variables prefix: `autonomous/airflow/variables`

Key variables required at runtime:
- `git_action_bot_token` — GitHub API token for workflow dispatch
- `auto_bot_config` — JSON `{"appId": "...", "accessToken": "..."}` for Lark/Feishu notifications

---

## Framework quirks

- **TaskFlow API throughout:** all tasks use `@dag` / `@task` / `@task_group` decorators; data passes via XCom (`ti.xcom_push` / `ti.xcom_pull`).
- **Dataset-driven scheduling:** downstream DAGs do not run on cron; they are scheduled on Airflow Dataset events emitted by `auto_trigger.py` after MySQL row polling.
- **Jinja2 custom delimiters:** templates use `[% %]` blocks and `[[ ]]` variables (not `{% %}` / `{{ }}`), to avoid conflicts with YAML syntax. Template files live under `templates/`.
- **Jenkins tasks use `@task.virtualenv`:** `jenkinsapi==0.3.13` is installed into an ephemeral venv at execution time — it is not in the Airflow image.
- **`max_active_runs=1`** on all DAGs — concurrent runs would cause MySQL race conditions.
- **Prod-only logic is inline:** several DAGs gate behavior on `ENV == "prod"` rather than using separate files.

---

## Gotchas

- **Python version:** `.python-version` pins 3.9.7; match this to stay consistent with the Airflow 2.9.2 base image. The local `.venv` on disk was built with 3.14 — do not rely on it.
- **`chart/` is gitignored:** only `values.yaml` at root is tracked for Helm config.
- **`.venv/` is not gitignored** — avoid staging or committing `.venv` changes.
- **EKS unsupported AZs:** `use1-az3`, `usw1-az2`, `cac1-az3` do not support EKS subnet creation. `utils/aws_az_check.py` filters these; keep filtering logic in sync with that file.
- **Single-request guard:** `get_create_region_requests` raises `AirflowFailException` if more than one region-create request is pending — this is intentional.
- **Azure ABFS URL conversion:** `utils/url.py blob_url_to_abfs()` drops the storage account hostname; the Airflow connection supplies the account. Do not add the hostname back.
- **GitHub workflow dispatch is fire-and-poll:** tracks run count vs. batch size — brittle if GitHub triggers unrelated background runs in the same window.
- **No CI on this repo** runs lint or tests automatically. `make fmt && make check` before pushing.