# Project Glossary ## TiDB Cloud Lake-first lakehouse The target lakehouse architecture for this project. TiDB Cloud Lake is the product and platform anchor; design choices should optimize for TiDB Cloud Lake semantics, operations, and user experience first. ## Databend engine context Databend is treated as the underlying engine context for TiDB Cloud Lake. Databend Cloud documentation and behavior are useful supporting references, but Databend is not a separate first-class target unless a design explicitly says so. ## Lakehouse scope The project scope spans the full analytical data platform surface, from ingestion through warehouse-facing consumption. Governance and orchestration are in scope rather than later add-ons. ## First version artifact The first version should be a capability map plus design principles and a reference architecture skeleton. It should define each layer's purpose, boundaries, key interfaces, non-goals, and unresolved decisions, without pretending to be a detailed implementation spec. ## V1 capability map The v1 capability map covers ingestion, storage and table management, compute and SQL, transformation and warehouse modeling, orchestration, governance, observability and quality, data products, agents, and consumption. The data product plane is the organizing plane that cuts across the rest. ## V1 non-goals V1 excludes production implementation design, Databend Cloud as a peer product target, a full autonomous operation model, final vendor/tool selections not explicitly settled by ADRs, an exhaustive connector catalog, a full governance policy language, unverified benchmark claims, migration strategy, and detailed UI/UX design. ## Observability as design surface Observability is baked into the lakehouse design rather than added after implementation. Documentation generation, lineage, run history, freshness, quality, ownership, and operational signals should be first-class outputs of ingestion, transformation, orchestration, governance, and data product workflows. ## Documentation and lineage shapes The design should favor proven, machine-readable shapes for documentation and lineage. Examples include dbt-style generated docs for modeled assets and OpenLineage-style event models for orchestration lineage, but the lineage event model decision is deferred. ## Deferred lineage event model V1 does not decide the observed-lineage event model. OpenLineage is a strong candidate, especially if Airflow or OpenLineage-compatible tooling is used, but the final choice should wait for orchestration and runtime integration decisions. ## Observable contract Every data product must publish a machine-readable contract that includes ownership, domain, purpose, consumer promises, schema and semantic definitions, freshness expectations, tests and status, lineage, access policy, masking notes, orchestration or run history, and agent permissions by risk tier. ## Contract versioning Data product contracts are versioned. The contract upgrade process is a core part of the design and must define compatibility, deprecation, migration, validation, and consumer or agent impact. ## Draft pre-1.0 data product contracts Data product contract versions below 1.0.0 are draft and not live. They may change shape, tests, meanings, or guarantees during normal Git review without major-version promotion ceremony, migration notes, or a deprecation window. Version 1.0.0 is the first live compatibility baseline for a data product contract. ## Contract schema versioning The authored contract schema and runtime-served contract view are themselves versioned formats. Schema version changes are separate from individual data product contract version changes. ## Contract schema version change strategy Contract schema version changes follow standard semantic versioning. Pre-1.0 schema versions may introduce breaking changes. After 1.0, patch and minor versions must remain backward-compatible, while major version bumps may break compatibility and require migration tooling, a deprecation window, validation behavior, runtime projection behavior, and agent/client support for old and new schema versions during transition. ## Contract compatibility Data product contracts use semantic versioning, with draft pre-1.0 versions treated as unstable/non-live. For live contracts at 1.0.0 and later, patch-compatible changes include clarifications and non-breaking metadata updates. Minor-compatible changes are additive and non-breaking. Major-breaking changes include removed or renamed fields, changed meanings, freshness or SLO changes, access or masking changes, metric definition changes, widened agent permissions, or changed upstream dependency assumptions. ## Contract promotion Contract promotion requires compatibility checks. Draft pre-1.0 product contract changes can be promoted as reviewed draft revisions when validation passes, even if they would be breaking after 1.0.0. For live contracts at 1.0.0 and later, patch and minor versions can be promoted automatically when validation passes. Major versions require owner approval, consumer impact analysis, migration notes, and a deprecation window. Agents should prefer the latest compatible live contract version, not blindly the latest version. ## Git-authored catalog-served contracts Git is the true source of truth for data product contracts. The runtime catalog or control plane serves promoted projections of approved Git contract revisions for discovery, enforcement, orchestration, and agent use. ## Authored contract schema The authored contract schema is the Git source of truth. It contains declared intent: identity, version, layer, owner, domain, purpose, intended and prohibited usage, declared assets, schema, semantics, freshness expectations, embedded tests, governance intent, agent permission intent, and compatibility rules. ## Universal contract schema Authored contracts use one universal schema with layer-specific validation profiles rather than separate raw, staging, curated, and service schema variants. This keeps tooling, CI, runtime projection, and agent parsing consistent while allowing each layer to have different required and optional fields. A top-level `kind` discriminates `product` (a layered data product) from `source` (an upstream artifact such as exported S3 objects); products carry a layer, sources do not. Absent `kind` means `product`. ## Contract serialization Authored contracts are YAML files in Git. Runtime-served contract views are JSON returned by runtime APIs. Both authored and runtime-served formats should be defined by JSON Schema for validation and generated tooling. ## Contract location Authored contracts use a hybrid Git layout. Canonical product contracts live under a central contracts directory, while asset-local metadata and fragments may live beside dbt models, Dagster assets, dlt sources, SQL files, or Python asset definitions. ## Contract engine The contract engine collates canonical contracts, asset-local fragments, generated metadata, embedded tests, layer validation profiles, and runtime projection rules. It validates authored contracts, compiles effective contracts, and publishes runtime-served contract views. ## PR-generating contract engine The contract engine may generate pull requests that update contracts, docs, fragments, schemas, or tests. It must not directly commit to protected branches. Git review remains the authority for changing the authored contract source of truth. ## Runtime-served contract view The runtime-served contract view is a promoted projection of the authored contract enriched with runtime facts. It contains effective lineage, effective governance policy, effective agent permissions, current observability status, run history, test results, freshness state, observed schema state, and promotion provenance. ## Runtime view snapshot The runtime-served contract view is an agent-friendly denormalized JSON snapshot, not a warehouse for every detail. It includes enough current state for discovery, query, explanation, and action decisions, with links to deeper lineage, policy, run history, test history, documentation, and audit surfaces. ## Agent runtime API The agent runtime API is the product-facing interface for agents to discover products, fetch runtime-served contract views, inspect lineage/runs/tests/docs/policies, run governed queries, and request controlled actions. Mutating operations are action requests rather than direct production mutation. ## Governance intent and effective policy Authored contracts declare governance intent. The runtime control plane resolves that intent into effective engine, catalog, masking, retention, audit, and agent-permission policies. ## Classification-driven governance Classifications should be sufficient to derive effective policy in common cases. Explicit policy declarations are overrides or refinements, not the default way to hand-author every rule. ## Classification inheritance Classifications are inheritable through the data product hierarchy, such as domain to product to asset to field. More specific classifications may refine or strengthen inherited classifications, but weakening inherited governance requires an explicit approved exception. ## Starter governance classifications V0 starts with these classifications: public, internal, confidential, restricted, pii, financial, customer_data, and operational_sensitive. ## Contract version clocks Contract design distinguishes three version clocks: the data product contract version, the authored contract schema version, and the runtime-served contract view schema version. These versions may change independently and should not be conflated. ## Contract provenance Every runtime-served contract must carry provenance linking it back to the approved Git revision, including repository, path, commit SHA, contract version, promotion time, promoter, and validation result. ## Catalog split The design treats catalog responsibilities as separate logical concerns: SQL/table catalog, data product registry, governance registry, and discovery index. These may later be backed by one physical control plane, but v1 names them separately. ## Logical not physical split Architectural splits in v1 describe responsibilities and interfaces, not necessarily separate deployed systems. Logical planes may later be implemented by one physical control plane or by multiple services. ## SQL/table catalog The engine-facing catalog for databases, schemas, tables, views, and table metadata. ## Data product registry The product-facing registry for contracts, owners, domains, docs, test status, lineage summaries, freshness, layer, and status. ## Governance registry The registry for policies, classifications, masking, grants, audit bindings, and agent action permissions. ## Discovery index The search and semantic index optimized for humans and agents to find and understand data products. ## Warehouse level The curated analytical layer where data products expose reliable consumption shapes: modeled tables or views, marts, metrics, semantic definitions, dimensions, tested transformations, BI-ready interfaces, and agent-ready query interfaces. The warehouse level is not a separate platform from the lakehouse. It is a governed data product layer built on TiDB Cloud Lake tables and views, contracts, transformations, semantic definitions, and tests. ## Python-first ingestion, SQL-first warehouse Ingestion assets may be Python-native because source onboarding often needs API handling, authentication, pagination, retries, incremental state, schema drift handling, and connector-specific logic. Warehouse-level data products should prefer SQL models, views, tables, tests, semantic definitions, and metric definitions where practical. ## Config-driven ingestion engine Ingestion implementation is split into reusable engine code and source-specific configuration. Source onboarding should usually add or change Git-authored source YAML and generated contracts, not create bespoke scripts. ## Ingestion stage layout Ingestion lives under `ingestion/` and is split by stage, each stage using the same `engine/` + `sources/` split. It is split on two independent axes (ADR 0017): **extract families are keyed on the source system**, **loaders are keyed on the object store they read**. The extract stage uses `ingestion/extract//{engine,sources}` — the TiDB Cloud family is `ingestion/extract/tidbcloud/` (→ S3), the BigQuery family is `ingestion/extract/bigquery/` (→ GCS). The load stage uses `ingestion/load//{engine,sources}`; `ingestion/load/s3_bulk_load/` and `ingestion/load/gcs_bulk_load/` are generic "load object-store bulk extracts into the warehouse" engines, not tied to a source system (TiDB→S3 and BigQuery→GCS are just two producers of such extracts). Shared, storage-agnostic code lives in `ingestion/common/` so neither stage — and neither loader — imports the other: the load config base (`load_config.py`), the Cloud Lake connection (`cloud_lake.py`), table classification (`classification.py`), raw-snapshot product-contract generation (`raw_product.py`), the external-incremental engine + strategies (`external/`), object-store listing (`gcp.py`, `aws.py`), and models/env/SQL helpers. Each loader supplies only its object-store specifics (S3 vs GCS credentials, listing, and stage). ## Standalone load sources inherit from contracts A load source under `ingestion/load//sources//source.yaml` is standalone: it declares where to write (Cloud Lake DSN), how to read the object-store bulk extract (S3 or GCS), and warehouse setup files. The load stage materializes only the raw layer, so it generates and owns the raw-snapshot product contract, deriving its schema, keys, and classifications from the source contract of the object-store snapshot it reads rather than re-declaring them. Staging and above are transform-layer concerns (dbt/airflow), not load. This supports three cases: an externally-managed bucket whose source contract is provided — hand-authored, or bootstrapped by inference and committed (the external-incremental mode below); a source system we snapshot ourselves with an extract stage we own (the TiDB→S3 extract writes the S3 objects and owns their source contract; the BigQuery→GCS extract writes the GCS objects and owns theirs, even though the BigQuery table is externally managed — per ADR 0017 the producer of the *objects* owns the contract); and the load source consumes that source contract either way. The run manifest remains the physical handoff (column schema, object list with per-table prefix, runtime log destination). When the bucket is filled by a pipeline we do not own, or by a reviewed one-off freeze that is not promoted into a recurring extract stage, there is no extract stage and no manifest, so the load runs in an **external-incremental** mode (ADR 0018) — available to both `s3_bulk_load` and `gcs_bulk_load`, with the orchestration shared in `ingestion/common/external/` and each loader supplying a small storage adapter (how to list objects + build the Databend stage). It lists the bucket, infers the schema from a sample object (`INFER_SCHEMA`) and solidifies it as a committed `source` contract under `contracts/upstream/`, and on later runs re-infers — a schema change writes a *proposed* contract and aborts (a contract update process), never a silent overwrite. A committed one-off helper script may document how the frozen object was produced, but it is not a managed extract unit unless it owns source config, setup, cadence, and contract production. Any reviewed one-off lifecycle/provenance/freshness/retention settings or raw parity gates that differ from defaults must live in the load source's `source.yaml` contract overrides so `infer-source` and `generate-contracts` preserve them. External-incremental loads increment by a **pluggable strategy** (`filename`: an `extraction_ts` epoch-ms parsed from each object name, as metadb uses; or `last_modified`: each object's storage mtime, a dltHub filesystem-incremental shape), loading only increments newer than the strategy's `max()` in the raw table and stamping the cursor + provenance onto every row. With no extract stage to own them, the read credentials are provisioned by the load unit's own `setup/grunt/`. ## Source-local ingestion config Each source owns its source YAML, its generated contracts (the extract stage owns its object-store source contract — S3 for TiDB, GCS for BigQuery; the load stage owns its raw-snapshot product contract), ignored secrets, and ignored run artifacts, under `ingestion///sources//`. Connection strings, tokens, access keys, manifests-in-progress, and other local runtime artifacts must not become repository-root scratch dependencies or engine code. Setup that a unit owns lives under that unit's `setup/` (the load's Databend database creation under `ingestion/load//setup/databend//`, the extract's object-store infrastructure as terragrunt under `ingestion/extract//setup/grunt/` — S3 buckets + IAM for `tidbcloud`, the GCS bucket + lifecycle + read-only loader service account for `bigquery`, both reading bucket/prefix/retention from `source.yaml`). A load unit also owns a `setup/grunt/` when it reads an externally-owned bucket with no extract stage to provide credentials — there it provisions just the reader service account + key + an `objectViewer` binding on that bucket (not the bucket itself), per ADR 0018. A repo-root `setup/` holds only cross-cutting setup orchestration that spans units or engines — currently the loader-neutral `setup/run.py` runner that applies a source's declared warehouse-setup files — not per-source SQL or values. See the config-driven clean structure principle. ## Transform stage layout Transform lives under `transform/`, a peer of `ingestion/`, and holds three engine-keyed units (ADR 0019): `transform/dbt/` is the primary engine (dbt on Databend via `dbt-databend-cloud`); `transform/taskflow/` is a parallel build of the same pipeline as native Databend Task Flow, kept only to compare the two while feeling things out; and `transform/python/` holds standalone pandas jobs for models dbt can't run on this gateway (no python-model backend), each reading dbt's curated products and writing its own under `transform_python_dwh`. Inside `transform/dbt/`, models are keyed on what varies: `sources//models/staging/` for staging (keyed on the input domain read FROM, whatever its kind — a raw snapshot `nexuscrm` / `metadb` / `bq_warehouse`, a committed seed `googlesheets`, or a generated utility dimension `core` such as `stg_dates`), `products//models/` for warehouse models (keyed on the product produced, cross-source by nature), and `engine/` for generic source-free config (macros, tests, the run entrypoint). The transform layer is the producer of — and owns — its staging and curated contracts (ADR 0015/0009): dbt generates them beside each model under `contracts/generated/`, and because the python jobs are independent producers of their own curated products they also own contracts for those — hand-authored under `transform/python/contracts/`, since the dbt generator does not run python (every produced data product still carries a contract, ADR 0003). All reference but do not author the load stage's raw-snapshot contracts; Task Flow writes none, being a parity backend rather than an independent producer. dbt's `schema` maps to a Databend database (relations render `schema.identifier`), so layers route to `transform_dbt_stg` / `transform_dbt_dwh` (and seeds to a neutral `transform_seeds`). The `dbt-databend-cloud` adapter is community-tier: the transform extra is pinned to a tested fork branch that carries TiDB Cloud Lake `warehouse` and `dbt seed` support until those fixes land upstream; `run_dbt.py` is the transform-env dbt entrypoint used by local scripts and Airflow, and `load_seeds.py` remains only a manual fixture-loading fallback. Native Task Flow on this build has no `AFTER` clause (standalone tasks ordered externally) — see ADR 0019. ## Metadata-first assets Data product outputs must be contract-native and observable regardless of implementation language. Python and SQL assets must emit sufficient metadata for contracts, lineage, quality, freshness, ownership, provenance, and agent use. ## Orchestration control plane Orchestration is a control plane workflow, not merely a scheduler. It coordinates asset dependencies, freshness policies, partitions, incremental state, retries, backfills, test gates, contract validation, layer publication, lineage/run event emission, and agent-triggered operational actions. Run cadence is config, not code: each extract `source.yaml` declares its own `schedule` block (`cron` canonical, alongside human-readable `cadence`/`days_of_month`), which the orchestrator reads rather than hardcoding a timetable. Onboarding a source on a different cadence — e.g. `data_foundation_finance` runs on the finance close cycle (days 1/3/5 of each month), not daily — is a source-config change, not an orchestrator change. The source's `freshness` thresholds bound the acceptable gap that cadence implies. ## Orchestration stage layout Orchestration lives under `orchestration/`, peer to `ingestion/` and `transform/`. The local v1 control plane uses Airflow 3.2 Assets compiled from the contract corpus (ADR 0020): `orchestration/airflow/engine/` holds reusable compiler and operator code; `orchestration/airflow/dags/` stays thin registration only; `orchestration/airflow/sources/` holds orchestration-only config such as local schedules, pools, and connection declarations; `orchestration/airflow/setup/` owns the local Docker Compose stack. Contracts remain the source of truth. Airflow Assets are a runtime projection with asset names, URIs, groups, freshness metadata, producer hints, and dependency edges derived from authored contracts and generated contracts. The v0 orchestration compilation profile supports exactly one declared asset per contract with `produced_by`; contract-backed upstream refs missing from the current corpus are explicit external upstream Assets rather than silent `None` edges. ## Orchestration reference shapes Airflow 3.2 Assets are the local-first orchestration projection for v1. OpenLineage-style events remain a useful interoperability reference for observed lineage, but the observed-lineage event model is still deferred until runtime lineage integration work. ## Data layers The lakehouse uses logical layers rather than fixed physical bronze/silver/gold zones or a mandatory movement order: raw, staging, curated, and service. ## Raw layer Source-faithful landed data, platform-owned, with limited semantic promise. ## Staging layer Normalized, typed, deduped, and schema-managed data that remains mostly source-aligned. ## Curated layer Governed data product assets with contracts, tests, freshness, ownership, and consumer promises. ## Service layer Consumption-specific interfaces such as marts, semantic metrics, BI views, APIs, and agent-facing query surfaces. ## Contract class Contracts declare the layer, guarantee level, intended usage, and prohibited usage for a data product. Raw, staging, curated, and service contracts provide different levels and types of promise. ## Contract kind and ownership A contract is either a `product` (a layered data product) or a `source` (an upstream artifact the platform consumes, such as exported S3 or GCS objects). The stage that produces an asset owns and generates its contract: the extract stage owns the source contract for the object-store objects it writes (S3 for TiDB, GCS for BigQuery); the load stage materializes only the raw layer and owns the raw-snapshot product contract; staging and above are produced and owned by the transform layer (dbt/airflow). The one exception is an externally-managed bucket with no extract stage: the platform produces no objects, so the source contract is *provided* — inferred and committed under `contracts/upstream/` as a consumed input, not generated as something a stage owns (ADR 0018). Each references its immediate upstream's contract. ## Contract design non-goal V1 contract design should not add a separate source-aligned versus domain-aligned product taxonomy. Layer and contract guarantees already express the relevant distinction. ## Executable contract tests Contracts may embed executable tests, such as SQL tests, expectation definitions, or dbt-style tests that run when the underlying asset is built. V1 may define the embedded test shape even if full execution is lax or deferred. ## Contract test severity Contract tests use severity-based failure behavior. Blocking tests prevent promotion or service exposure. Degrading tests allow publication but mark the data product unhealthy or degraded. Informational tests record signals without changing status. Experimental tests are visible only to owners or agents with appropriate scope. ## Agent respect for product status Agents must respect contract test status. For example, a query agent should not use a degraded service product for authoritative reporting unless the user explicitly accepts that risk. ## Source contract Describes an upstream artifact the platform consumes (e.g. the Parquet objects an extract job writes to S3 or GCS): source system identifiers, the object-store asset, source-faithful schema, and governance. When the platform produces the objects via an extract stage, that stage owns the contract (ADR 0015 — the producer owns). When the bucket is externally managed and there is no extract stage, no one in the platform produces the objects, so the contract is *provided* rather than owned — bootstrapped by inference and committed under `contracts/upstream/` as a consumed input, with drift routed to `contracts/proposed/` (ADR 0018). Either way it carries no layer and is the declared upstream the load stage builds raw products from. ## Raw contract Source-faithful access with minimal semantic guarantees, strong provenance, schema capture, retention rules, and access constraints. ## Staging contract Typed, normalized, deduped, source-aligned data with caveats, schema/version guarantees, freshness, quality basics, and usage for exploration, data science experiments, and upstream validation. ## Curated contract Business semantics, quality thresholds, ownership, SLOs, stable compatibility promises, and governed production consumption. ## Service contract Consumption-specific interface contract for marts, metrics, BI views, APIs, or agent-facing tools, with strong user-facing semantics and compatibility expectations. ## Primary persona The first-class users are data platform engineers and analytics engineers. The design should optimize for their ability to build, operate, model, govern, and evolve lakehouse data products. ## Downstream persona Data agents are the main downstream consumers. The lakehouse should expose governed, discoverable, machine-readable data products and metadata that agents can use safely. ## Data product The primary unit of architecture and ownership. A data product groups analytical assets, contracts, governance policies, lineage, test expectations, freshness expectations, and operational responsibilities around a coherent business or analytical capability. Data products may exist at any layer when they have an owner, contract, policy, observability, and declared usage. ## Data-agent mesh The target architecture: a mesh of governed data products that can be safely discovered, reasoned over, and consumed by data agents. The mesh should make ownership, semantics, access policy, lineage, quality, and allowed agent actions explicit. ## Agent capability levels The target lakehouse design includes agents that can discover data products, run governed queries, explain data behavior using metadata and lineage, act through managed workflows, and eventually autonomously operate parts of the lakehouse under guardrails. ## Deferred agent permission model V1 does not define a bespoke agent permission model. Agent permissions should follow standard permission models such as RBAC/ABAC for now, using governance policy, domain ownership, product health, and action risk tier as inputs where applicable. ## Deferred approval workflow V1 does not define the detailed approval workflow for major live contract changes or approval-gated agent actions. It records that approval is required in those cases, but the exact approval states, approver resolution, workflow engine, and audit implementation are deferred. ## V1 agent scope The first version should include autonomous operation as an explicit in-scope target capability, but only define its boundaries, prerequisites, and guardrails. V1 should not try to fully specify or implement the autonomous operating model. ## Agent action risk tiers Agent actions are classified by operational risk: observe, reversible workspace actions, controlled operational actions, approval-gated production changes, and forbidden or break-glass actions. ## Domain ownership boundary Autonomous agent scope is bounded by domain ownership as well as action risk. An agent's allowed actions must be evaluated against the data product or domain it is authorized to operate within. ## Platform-mediated ownership The ownership model for the lakehouse. Domains own meaning, contracts, quality expectations, access intent, and consumer promises; the data platform owns shared ingestion, storage, orchestration, policy enforcement, observability, and runtime reliability. Analytics engineers can own modeled data products within domain boundaries.