# V1 Lakehouse Architecture ## Purpose This document defines the first-version architecture for a TiDB Cloud Lake-first lakehouse that supports a data-agent mesh. It is a capability map, design-principles document, and reference architecture skeleton. V1 is not an implementation spec. It names the system, draws responsibility boundaries, defines the first durable concepts, and records the decisions that should constrain future implementation work. ## Scope The lakehouse scope spans ingestion through warehouse-level consumption. Governance, orchestration, observability, and data-agent operation are in scope from the beginning. The core storage and execution posture is TiDB Cloud Lake first. Databend is treated as the underlying engine context for TiDB Cloud Lake, so Databend behavior and documentation can inform design where applicable, but Databend Cloud is not a separate first-class product target in v1. ## Non-goals V1 explicitly excludes: - Production implementation design - Databend Cloud as a peer product target - A full autonomous operation model - Final vendor/tool selection for planes not explicitly settled by ADRs - Exhaustive connector catalog design - Full governance policy language design - Unverified benchmark or cost claims - Migration strategy from other warehouses or lakes - Detailed UI/UX design ## Personas The primary users are: - Data platform engineers - Analytics engineers The main downstream consumers are data agents. The architecture should make metadata, policy, lineage, contracts, test state, and operational history explicit enough for agents to discover, query, explain, act, and eventually operate under guardrails. ## Design Principles ### TiDB Cloud Lake first, Databend-informed Optimize the architecture for TiDB Cloud Lake as the managed product surface. Use Databend as engine context where it helps explain SQL behavior, execution behavior, or table/storage semantics. ### Data products are the contract boundary Data products are the primary unit of architecture and ownership. Ownership, contracts, governance, lineage, tests, freshness, documentation, semantics, and agent permissions attach to data products first. ### Agent-ready by default Data products and platform state should be machine-readable enough for agents to safely discover, query, explain, and act. Agent readiness is not a late integration layer. ### Governance is in the path Access policy, masking, audit, lineage, retention, and agent permissions must be designed into ingestion, modeling, orchestration, and consumption rather than bolted on after the fact. ### Platform-mediated decentralization Domains own meaning, contracts, test expectations, access intent, and consumer promises. The platform owns shared ingestion, storage, orchestration, policy enforcement, observability, and runtime reliability. ### Observability is a product surface Generated documentation, lineage, run history, freshness, test results, ownership, and operational signals are first-class outputs of the lakehouse. dbt-style docs are a useful generated-documentation reference. OpenLineage-style events are a useful lineage reference, but the observed-lineage event model is deferred. ### Python-first ingestion, SQL-first warehouse Ingestion assets may be Python-native because source onboarding often requires API handling, auth, pagination, retries, incremental state, schema drift handling, and connector-specific logic. Warehouse-level data products should prefer SQL models, views, tables, tests, semantic definitions, and metric definitions where practical. ### Metadata-first assets Data product outputs must be contract-native and observable regardless of implementation language. Python and SQL assets must emit enough metadata for contracts, lineage, tests, freshness, ownership, provenance, and agent use. ### Open interfaces where lock-in hurts The design can use TiDB Cloud Lake-specific behavior where it matters, but should preserve stable boundaries around contracts, metadata, table/catalog concepts, orchestration events, and agent action protocols. ### Logical, not physical, splits Architectural planes describe responsibilities and interfaces. They do not require separate deployed systems in v1. ## Reference Architecture V1 uses the following logical planes. ### Ingestion Plane Responsible for source onboarding and raw data capture. Capabilities: - Batch ingestion - Incremental ingestion - CDC-like ingestion where supported - File ingestion - API ingestion - Source schema capture - Source state tracking - Raw landing - Ingestion metadata emission Typical asset shape: - Python-first assets - dlt-style loading patterns - Dagster-style asset definitions - Source-local schemas and state Primary outputs: - Raw layer assets (staging and above are transform-layer concerns; see [ADR 0016](../adr/0016-load-materializes-only-the-raw-layer.md)) - Source metadata - Provenance - Initial schema observations - Ingestion run events ### Storage and Table Plane Responsible for durable analytical storage, table metadata, and table/view access through TiDB Cloud Lake. Capabilities: - Table and view management - Schema/version metadata - Storage layout metadata - Object/table abstraction - Engine-facing catalog integration - Table-level access surfaces Primary outputs: - Engine-readable tables and views - Table metadata for the SQL/table catalog - Observed schema state ### Compute and SQL Plane Responsible for execution through TiDB Cloud Lake and Databend engine behavior where applicable. Capabilities: - SQL execution - Query routing - Workload isolation - Materialization execution - Query observability - Engine-specific optimization hooks Primary outputs: - Query results - Query history - Execution metadata - Cost/performance signals when available ### Transformation and Warehouse Plane Responsible for staging and curated analytical modeling and warehouse-level consumption shapes, on dbt-on-Databend (with a native Task Flow parity build and standalone python jobs) — see [ADR 0019](../adr/0019-transform-layer-is-dbt-on-databend-keyed-on-source-and-product.md). Capabilities: - SQL models - Views and materialized tables - Marts - Metrics and semantic definitions - Dimensions and entities - Model tests - Documentation generation Primary outputs: - Staging layer data products - Curated layer data products - Service layer data products - Modeled assets - Semantic metadata - Test results - Generated docs ### Orchestration Plane Responsible for coordinating assets, runs, tests, promotions, and operational actions. Orchestration is a control plane workflow, not merely a scheduler. Capabilities: - Asset dependencies - Freshness policies - Partitions - Incremental state - Retries - Backfills - Test gates - Contract validation - Layer publication - Run state - Lineage/run event emission - Agent-triggered operational actions Run cadence is config, not code: each extract `source.yaml` declares its own `schedule` block (`cron` canonical), which the orchestrator reads rather than hardcoding a timetable — so a source on a non-daily cadence (e.g. a monthly finance close cycle) is a config change, not an orchestrator change. Reference shapes: - Airflow 3.2 Assets compiled from the contract corpus - Thin Airflow DAG registration modules, with business dependencies coming from contracts - Explicit external upstream Assets for contract-backed lineage refs that are not yet present in the compiled corpus - OpenLineage-style events, pending a later lineage event model decision ### Governance Plane Responsible for policy intent, effective enforcement, audit, and access semantics. Capabilities: - Classifications - Access policy - Masking policy - Retention policy - Grants and approvals - Audit requirements - Agent action permissions - Policy evaluation - Classification inheritance Primary outputs: - Governance intent in authored contracts - Effective governance in runtime-served contract views - Audit trails - Policy evaluation results Governance should be classification-driven by default. Authored contracts declare classifications and optional policy refinements; the runtime control plane resolves inherited classifications into effective access, masking, retention, audit, and agent-permission policies. The v0 starter classification hierarchy is defined in `docs/design/v0-governance-classifications.md`. ### Observability and Test Plane Responsible for making product and platform state inspectable by humans and agents. Capabilities: - Generated documentation - Lineage - Run history - Test results - Freshness state - Volume and drift signals - Incident/degradation signals - Ownership and on-call metadata Primary outputs: - Product status - Test status - Freshness status - Lineage graph or events - Documentation views - Agent-readable operational state ### Data Product Plane Responsible for product ownership, contracts, docs, status, layer, and discoverability. This is the organizing plane that cuts across the rest of the architecture. Capabilities: - Product identity - Domain ownership - Contract authoring - Contract promotion - Documentation - Layer classification - Compatibility and deprecation - Consumer promises - Product health/status Primary outputs: - Authored contracts - Runtime-served contract views - Product registry entries - Documentation - Product status ### Agent Plane Responsible for data-agent discovery, query, explanation, action, and autonomous operation guardrails. Target capabilities: - Discover data products - Query governed assets - Explain data behavior using metadata, lineage, tests, run history, and contracts - Act through managed workflows - Eventually operate parts of the lakehouse autonomously under guardrails V1 includes autonomous operation as a target capability, but only defines boundaries, prerequisites, and guardrails. ### Consumption Plane Responsible for human and machine consumption interfaces. Capabilities: - BI-ready views - Semantic metrics - Notebooks - APIs - Reverse ETL interfaces - Agent-facing query and discovery interfaces ## Data Layers The lakehouse uses logical layers rather than fixed physical bronze/silver/gold zones or a mandatory movement order. ### Raw Source-faithful landed data, platform-owned, with limited semantic promise. Raw contracts emphasize: - Source provenance - Schema capture - Retention - Access constraints - Minimal semantic guarantees ### Staging Normalized, typed, deduped, and schema-managed data that remains mostly source-aligned. Staging contracts emphasize: - Source alignment - Type normalization - Deduplication rules - Caveats - Schema/version guarantees - Freshness - Basic tests - Exploration and data science usage ### Curated Governed data product assets with contracts, tests, freshness, ownership, and consumer promises. Curated contracts emphasize: - Business semantics - Quality thresholds - SLOs - Stable compatibility promises - Governed production consumption ### Service Consumption-specific interfaces such as marts, semantic metrics, BI views, APIs, and agent-facing query surfaces. Service contracts emphasize: - Interface compatibility - Consumer-specific semantics - Metrics and dimensions - BI/agent usage - Strongest user-facing guarantees ## Data Product Model A data product is the primary unit of architecture and ownership. A data product may exist at any layer when it has: - Owner - Domain - Contract - Policy - Observability - Declared usage - Operational responsibility Data products are not synonymous with curated assets. Raw, staging, curated, and service products are all valid when their contract class makes their guarantees and caveats explicit. V1 contract design should not add a separate source-aligned versus domain-aligned product taxonomy. The layer and contract guarantees already express that distinction. ## Contract Model Contracts are mandatory for data products. Contracts are: - Authored in Git - Served through runtime APIs - Versioned per data product - Governed by versioned contract schemas - Validated by the contract engine - Enriched into runtime-served contract views ### Authored Contract Schema The authored contract schema is the Git source of truth. Authored contracts are YAML files. It contains declared intent: - `id` - `name` - `version` - `schema_version` - `layer`: raw | staging | curated | service - `owner` - `domain` - `purpose` - `intended_usage` - `prohibited_usage` - `assets` - `schema` - `semantics` - `freshness` - `tests` - `governance_intent` - `agent_permission_intent` - `compatibility` Layer-specific validation profiles decide which fields are required, optional, or not applicable for each layer. ### Runtime-Served Contract View The runtime-served contract view is a promoted projection of the authored contract, enriched with runtime facts. Runtime views are served as JSON. It contains: - Contract identity - Contract version - Runtime view schema version - Layer - Effective assets - Effective lineage - Effective governance policy - Effective agent permissions - Current product status - Current test status - Current freshness state - Observed schema state - Run history summaries - Documentation links - Promotion provenance Runtime-served views are what agents and platform runtime clients should normally consume. ### Version Clocks The design distinguishes three version clocks: - Data product contract version - Authored contract schema version - Runtime-served contract view schema version These versions may change independently and must not be conflated. Version strategy follows standard semantic versioning: - Pre-1.0 schema versions may break - Pre-1.0 data product contract versions are draft and not live; they may break during normal Git review - After 1.0, patch and minor versions must remain backward-compatible - Major version bumps may break compatibility with migration tooling, deprecation windows, validation behavior, runtime projection behavior, and agent/client support windows ### Contract Compatibility Contract changes use semantic versioning. Version 1.0.0 is the first live compatibility baseline for a data product contract. Before 1.0.0, data product contracts are draft/non-live and may change shape, tests, meanings, or guarantees without major-version promotion ceremony, migration notes, or a deprecation window. Patch-compatible changes include: - Clarifications - Documentation updates - Non-breaking metadata changes Minor-compatible changes include: - Additive schema fields - Additive semantic definitions - Additive tests - New lineage annotations - New allowed agent actions that do not expand data exposure Major-breaking changes include: - Removed fields - Renamed fields - Changed meanings - Freshness or SLO changes - Access or masking changes - Metric definition changes - Widened agent permissions - Changed upstream dependency assumptions Draft pre-1.0 product contract revisions can be promoted as reviewed draft revisions when validation passes, even if they would be breaking after 1.0.0. For live contracts at 1.0.0 and later, patch and minor versions can be promoted automatically when validation passes. Major versions require owner approval, consumer impact analysis, migration notes, and a deprecation window. Agents should prefer the latest compatible live version, not blindly the latest version. ### Executable Tests Contracts may embed executable tests, such as SQL tests, expectation definitions, or dbt-style tests that run when the underlying asset is built. V1 may define the embedded test shape even if full execution is lax or deferred. Test severities: - `blocking`: prevents promotion or service exposure - `degrading`: allows publication but marks the product unhealthy or degraded - `informational`: records signal without changing status - `experimental`: visible only to owners or agents with appropriate scope Agents must respect product status. For example, a query agent should not use a degraded service product for authoritative reporting unless the user explicitly accepts that risk. ## Contract Engine The contract engine is the compiler for the contract system. Inputs: - Canonical product contracts - Asset-local fragments - Generated metadata - Embedded tests - Layer validation profiles - Runtime projection rules - Compatibility rules Outputs: - Validation results - Compiled effective contracts - Runtime-served contract views - Generated documentation - PRs for generated or suggested changes The contract engine may generate pull requests that update contracts, docs, fragments, schemas, or tests. It must not directly commit to protected branches. Git review remains the authority for changing the authored source of truth. ## Git and Runtime Source of Truth Git is the true source of truth for authored contracts. The runtime catalog/control plane serves promoted projections of approved Git revisions. Runtime projections are not an independent source of truth. Every runtime-served contract view must carry provenance: - Repository - Path - Commit SHA - Contract version - Promotion time - Promoter - Validation result ## Contract Location Authored contracts use a hybrid Git layout. Canonical product contract: ```text /contracts///contract.yaml ``` Asset-local metadata and fragments may live beside: - dbt models - Dagster assets - dlt sources - SQL files - Python asset definitions The contract engine collates these inputs into one validated authored contract and one runtime-served contract view. ## Logical Catalog Split V1 names separate logical catalog responsibilities. These may later be backed by one physical control plane. ### SQL/Table Catalog Engine-facing catalog for databases, schemas, tables, views, and table metadata. ### Data Product Registry Product-facing registry for contracts, owners, domains, docs, test status, lineage summaries, freshness, layer, and status. ### Governance Registry Registry for policies, classifications, masking, grants, audit bindings, and agent action permissions. ### Discovery Index Search and semantic index optimized for humans and agents to find and understand data products. ## Governance and Agent Autonomy Agent operation is governed by two boundaries: - Action risk - Domain ownership ### Risk Tiers Tier 0: observe - Read metadata - Read docs - Read lineage - Read run history - Read test and freshness state Tier 1: reversible workspace actions - Draft SQL - Create temp views - Run sampled queries - Create analysis notebooks - Propose data product definitions Tier 2: controlled operational actions - Trigger idempotent jobs - Rerun failed tasks - Request access - Create tickets - Open PRs - Propose backfills Tier 3: approval-gated production changes - Schema changes - Production model changes - Policy changes - Large backfills - Schedule changes Tier 4: forbidden or break-glass - Destructive deletes - Privilege escalation - Irreversible retention changes - Cross-boundary data exposure Agent permissions should be declared as intent in authored contracts and served as effective permissions in runtime-served contract views. V1 does not define a bespoke agent permission model. Agent permissions should follow standard RBAC/ABAC-style resolution for now, using governance policy, domain ownership, product health, and action risk tier as inputs where applicable. The v0 agent runtime API is defined in `docs/design/v0-agent-runtime-api.md`. V1 does not define the detailed approval workflow for major live contract changes or approval-gated agent actions. It records that approval is required in those cases, but the exact workflow is deferred. ## Open Questions 1. Which layer-specific validation profiles should be enforced first? 2. Which contract test executors are supported in v0: SQL only, dbt-style tests, expectation definitions, or all three? 3. How should generated docs be shaped for humans and agents? 4. Which observed-lineage event model should be adopted when runtime lineage integration begins? 5. Which implementation track should be explored first: contract engine prototype, TiDB Cloud Lake integration, or orchestration integration?