# DAX Dataset Inventories

This directory stores repo-managed JSON inventories for DAX catalog seeding and
BigQuery apply scope. The runtime catalog projection path is
`server/scripts/seed_data_access_resources.py`; the provider IAM boundary is
managed by `infra/gcp-dax`.

## `gcp-digital-bi-dbt.json`

`gcp-digital-bi-dbt.json` is the current SSOT for BigQuery datasets projected
into DAX from the `/Users/ywang/projects/work/dbt-dev` dbt project. Entries
remain in the inventory when they are not currently requestable so their
catalog identity, owner, and history are preserved.

Derivation on 2026-06-24:

- collected enabled schema targets from `dbt_project.yml`
- collected model, seed, and snapshot output schemas from `target/manifest.json`
- intersected those schemas with live BigQuery datasets in `gcp-digital-bi`
- excluded dbt test audit datasets
- excluded configured/manifest targets that were not present in BigQuery:
  `default`, `stg_mixpanel`, `stg_temp`, `tidbcloud`
- excluded dbt-project disabled targets:
  `dwh_temp`, `model_of_monthly_cloud_revrec_confirm`

Requestability and descriptions were refreshed on 2026-08-19 from a read-only
production DAX catalog readback, live BigQuery object inventory, and dbt
`main@5139557ce1276adb5b07d740d5a71ed8cfe636b2` lineage, plus the
user-approved exact addition `gcp-digital-bi.metadb_new`. Exactly 23 entries
are `requestable=true`; 20 historical entries stay active but are not offered
for new requests. Every requestable entry has a single-line
business description (at most 500 characters) suitable for the request UI and
Feishu approval payload.

On 2026-08-21, source projection was narrowed to canonical dbt
`resource_type=source` nodes reachable upstream from an active/requestable dbt
result dataset already admitted by Helix. These are externally populated source
datasets, not dbt outputs: traversal stops at `source`, so a source-only dataset
is never a new origin, and unused/unreachable declarations plus all `model`,
`seed`, and `snapshot` nodes are never admitted by this rule. Added rows are
active, default to `xiaopeng.zhu@pingcap.com`, and carry `dbt_generated=false`.
The user subsequently made every referenced source dataset requestable instead
of discoverable only inside a downstream lineage graph. The 32 referenced
source locators (23 added plus 9 already present) union with the prior 23
business-requestable locators. After seven overlaps and the Provider-missing
`usage_us_west2` suspension, the result is 47 active/requestable rows among 66
total inventory rows. On 2026-08-25, the user approved the historical Alibaba
Cloud billing landing dataset `gcp-digital-bi.alibaba_cloud` as active owner
metadata with `requestable=false`; the inventory therefore contains 67 rows
while the active/requestable count remains 47. Existing source owners remain
unchanged.

The deployed lineage projection canonicalizes duplicate dbt aliases by exact
`project.dataset.table` locator before source-dataset synchronization. The
artifact contains 38 unique source-declaration datasets, of which 32 are
referenced by a dependency edge. Under the result-reachable rule, 31 remain in
the effective projection; the already missing/suspended `usage_us_west2` row is
not reachable and exits the authoritative feed. The dry-run result is 67 rows
with 47 active/requestable locators. A stale lineage-derived row is omitted,
while a manually governed row and its owner are retained. A physical relation
that is also a dbt-produced model/snapshot/seed is not counted as an external
source root.

Terraform deliberately reads every **active** catalog row, including
`requestable=false`, and explicitly allows only the `gcp-digital-bi` and
`gcp-digital-bi-dev` projects. This wider controller scope preserves the
ability to observe and revoke historical grants after new-request admission is
closed; it does not make a dataset requestable. Adding another project to the
inventory fails validation until the allowlist is deliberately reviewed.
Catalog seed and Terraform apply are separate Production gates.

The dbt-derived entries retain `xiaopeng.zhu@pingcap.com` as owner;
`gcp-digital-bi.metadb_new` is owned by `xukai@pingcap.com`, and the manual
`gcp-digital-bi.alibaba_cloud` owner record is owned by
`fanfei.kong@pingcap.com`.

After regenerating the lineage catalog, this command remains available for a
local review or for committing the assembled inventory back to the repo:

```bash
cd server
.venv/bin/python scripts/sync_data_access_lineage_roots.py \
  app/domains/data_access/lineage_catalogs/gcp-digital-bi-dbt.json \
  ../deploy/data-access/datasets/gcp-digital-bi-dbt.json
# Review the dry-run list, then repeat with --commit.
```

Dry run from a backend checkout/container:

```bash
cd server
/app/.venv/bin/python scripts/seed_data_access_resources.py \
  ../deploy/data-access/datasets/gcp-digital-bi-dbt.json \
  --lineage-catalog app/domains/data_access/lineage_catalogs/gcp-digital-bi-dbt.json \
  --integration-slug dax-bigquery-prod \
  --source-version repo-gcp-digital-bi-dbt-20260818-descriptions
```

Commit the projection:

```bash
cd server
/app/.venv/bin/python scripts/seed_data_access_resources.py \
  ../deploy/data-access/datasets/gcp-digital-bi-dbt.json \
  --lineage-catalog app/domains/data_access/lineage_catalogs/gcp-digital-bi-dbt.json \
  --integration-slug dax-bigquery-prod \
  --source-version repo-gcp-digital-bi-dbt-20260818-descriptions \
  --commit
```

The catalog feed is authoritative for the target integration. Removing a row
from the seeded inventory suspends existing catalog-owned rows for that
integration if they are omitted from the feed.

The Terraform feed is also authoritative for provider controller scope. Every
active seed row becomes a dataset-level custom-role binding for the DAX service
account, while the catalog's separate `requestable` flag still controls new
request admission. Moving a row away from active removes the service account's
permission to observe/apply/revoke that dataset on the next Terraform apply.

## GitHub Actions

Use the manual **Seed DAX Dataset Catalog** workflow for explicitly requested
catalog imports. Runs default to dry-run. There is no scheduled catalog rewrite:
production admission settings must not be reset from the repository on a timer
or during deployment. A manual committed import remains authoritative and can
overwrite `requestable`; review its input and dry-run before committing. The workflow
validates both the selected inventory and the
version-bound lineage catalog, automatically derives referenced source roots,
uploads both artifacts plus the EC2 helper to S3, then runs the backend seed
script through SSM on the configured EC2 instance. Operators do not discover
or add individual source datasets by hand. The workflow does not list every
BigQuery dataset, fetch schemas, or apply Terraform/IAM; provider metadata stays
lazy and controller scope remains a separate gate.

The workflow projection is the only lineage assembly step. The EC2 helper
copies the content-addressed inventory and lineage evidence to the backend but
seeds the already assembled inventory without passing `--lineage-catalog` to
the running image. This prevents an older Production image from reinterpreting
a newer workflow artifact with stale root-selection code.

Defaults:

- inventory: `deploy/data-access/datasets/gcp-digital-bi-dbt.json`
- lineage catalog: `server/app/domains/data_access/lineage_catalogs/gcp-digital-bi-dbt.json`
- integration slug: `dax-bigquery-prod`
- dry run: `true`
- unresolved owners: rejected

Committed writes (`dry_run=false`) are blocked unless the workflow is run from
`main`, so production catalogue state follows the merged repo artifact.

Use the manual **Deploy DAX BigQuery Integration** workflow after Terraform
apply. It reads the Terraform-generated service account JSON from SSM, upserts
the `data_access/bigquery` integration secret, and writes `apply_mode_enabled`.
It also dry-runs by default and blocks committed writes unless run from `main`.