# V0 Contract Engine ## Purpose The contract engine is the compiler for the data product contract system. It reads Git-authored contracts and asset-local fragments, validates them, compiles an effective authored contract, produces runtime-served contract views, generates documentation, and opens pull requests for suggested changes. The contract engine keeps Git as the source of truth while making contracts usable by runtime systems and data agents. ## Non-goals V0 does not implement: - A full policy engine - A full lineage graph store - A full orchestration system - A full documentation portal - Direct commits to protected branches - Autonomous production mutation - Exhaustive connector metadata extraction - Final TiDB Cloud Lake enforcement integration ## Inputs ### Canonical Product Contract The canonical product contract lives at: ```text contracts///contract.yaml ``` It is the human-authored Git source of truth for the product. ### Asset-Local Fragments Asset-local fragments may live beside: - dbt models - Dagster assets - dlt sources - SQL files - Python asset definitions Fragments can contribute metadata such as: - Asset references - Column metadata - Test definitions - Descriptions - Freshness hints - Source references - Lineage hints - Documentation fragments ### Generated Metadata The engine may ingest generated metadata from tools such as: - dbt manifests/catalogs - Dagster asset metadata - dlt schema output - SQL parser output - Lineage events from the chosen observed-lineage event model Generated metadata can enrich validation and docs, but it does not override Git-authored intent without producing a pull request. ### Runtime Observations Runtime observations can be used to produce runtime-served contract views: - Latest test results - Freshness state - Observed schema state - Latest run summary - Effective lineage - Effective governance decisions - Effective agent permissions Runtime observations do not mutate authored contracts directly. ## Pipeline ### 1. Discover Find canonical contracts and eligible asset-local fragments. Minimum discovery rules: - Scan `contracts/**/contract.yaml` - Scan configured dbt, Dagster, dlt, SQL, and Python asset locations - Ignore generated build output unless explicitly configured - Preserve source path for every discovered input Output: - Input inventory - Source paths - Input type classification ### 2. Parse Parse YAML, JSON, SQL-adjacent metadata, and tool manifests into normalized intermediate objects. Minimum parsing rules: - YAML authored contracts must parse cleanly - Unknown top-level contract fields fail validation unless placed under `extensions` - Fragments must declare their target product or asset - Parser errors are reported with file path and location where available Output: - Parsed contracts - Parsed fragments - Parse diagnostics ### 3. Collate Merge canonical contracts, fragments, and generated metadata into an effective authored contract candidate. Precedence: 1. Canonical product contract owns declared intent 2. Asset-local fragments may fill missing metadata 3. Generated metadata may fill observed or derived metadata 4. Conflicts are diagnostics unless an explicit merge rule exists The engine should avoid silent overwrites. If a generated source disagrees with authored intent, produce a diagnostic and optionally a PR suggestion. Output: - Effective authored contract candidate - Merge diagnostics - Suggested edits ### 4. Validate Validate the effective authored contract candidate. Validation layers: - JSON Schema validation - Layer profile validation - Semantic version validation - Compatibility validation against prior promoted live versions; draft pre-1.0 product contracts may break while still being reviewed and validated - Embedded test shape validation - Governance intent shape validation - Agent permission intent shape validation - Classification inheritance validation V0 validation should use: - `schemas/data-product-contract.authored.v0.schema.json` - Layer profile rules from the schema and contract engine configuration Output: - Validation result - Errors - Warnings - Compatibility classification ### 5. Evaluate Tests V0 may validate test definitions without executing every test. Supported v0 test handling: - Validate embedded SQL test shape - Validate dbt test references - Validate expectation object shape - Record unsupported test executors as warnings or blocking errors based on config Execution behavior: - Blocking tests should block promotion when execution is available and failing - Degrading tests should mark product status degraded when failing - Informational tests should record status only - Experimental tests should be visible only to owners or scoped agents Output: - Test definition validation - Latest test result ingestion when available - Test status summary for runtime view ### 6. Project Runtime View Compile the runtime-served contract view. Inputs: - Effective authored contract - Promotion metadata - Runtime observations - Effective lineage - Effective governance - Effective agent permissions Output: - JSON runtime-served contract view - Validation against `schemas/data-product-contract.runtime-view.v0.schema.json` The runtime view is a denormalized snapshot with links to deeper systems. It must not become the full lineage store, policy store, or run log store. ### 7. Generate Docs Generate docs from the effective authored contract and runtime observations. Minimum docs output: - Product identity - Layer - Owner/domain - Purpose - Usage and prohibited usage - Assets - Schema - Semantics - Freshness expectation and current state - Test definitions and latest status - Lineage summary - Governance summary - Agent permissions summary - Contract version and provenance Docs should be useful to both humans and agents. dbt docs are a useful reference shape. ### 8. Publish Publish successful runtime views and docs to configured runtime surfaces. V0 publish targets may be: - Local generated files - Static docs directory - Metadata API - Data product registry - Discovery index Publishing requires: - Authored contract validation passed - Runtime view validation passed - Promotion provenance recorded ### 9. Generate Pull Requests The contract engine may generate pull requests for: - Missing descriptions - Generated schema updates - New asset-local fragments - Test suggestions - Documentation updates - Compatibility migration notes - Drift reconciliation The engine must not directly commit to protected branches. PRs should include: - Summary of generated changes - Validation diagnostics - Source observations that motivated the PR - Compatibility impact - Reviewer/owner hints ## Outputs Required outputs: - Effective authored contract candidate - Validation report - Runtime-served contract view - Generated documentation - Diagnostics Optional outputs: - Pull request branch - Pull request body - Discovery index update - Product registry update - Local static site output ## Diagnostics Diagnostics should be structured enough for humans, CI, and agents. Each diagnostic should include: - Severity: error | warning | info - Code - Message - Source path - Field path where available - Suggested fix where available - Whether it blocks promotion ## Promotion Rules Promotion requires: - Contract parses successfully - JSON Schema validation passes - Layer profile validation passes - Compatibility validation passes or has required approval metadata. Draft pre-1.0 data product contract changes are not live and may be breaking without major-version approval metadata. - Blocking tests pass when execution is available - Runtime view schema validation passes - Provenance can be recorded Promotion may be blocked by: - Missing required layer fields - Breaking live contract change without approval or migration notes - Failed blocking test - Invalid governance intent - Invalid or unapproved classification weakening - Invalid agent permission intent - Unresolved merge conflict between authored and generated metadata ## Compatibility Rules The engine classifies product contract changes as: - Draft pre-1.0 - Patch-compatible - Minor-compatible - Major-breaking Draft pre-1.0 changes are reviewed and validated, but they are not live compatibility promises and may introduce breaking changes without major-version ceremony. Major-breaking changes require: - Owner approval - Consumer impact analysis - Migration notes - Deprecation window The engine should compare against the latest promoted compatible version, not only the latest Git file. ## Contract Engine Modes ### CI Mode Used in pull requests. Behavior: - Parse - Collate - Validate - Classify compatibility - Validate test definitions - Report diagnostics - Do not publish runtime views ### Promotion Mode Used after merge or approved release. Behavior: - Parse - Collate - Validate - Ingest runtime observations - Project runtime view - Validate runtime view - Generate docs - Publish - Record provenance ### Suggestion Mode Used by scheduled jobs, platform operators, or agents. Behavior: - Discover drift or missing metadata - Generate suggested edits - Open pull requests - Do not directly mutate protected branches ## Minimal V0 CLI Shape The contract engine should expose a small CLI: ```bash contract-engine discover contract-engine validate --contract contracts///contract.yaml contract-engine project --contract contracts///contract.yaml --out .generated/contracts/ contract-engine docs --contract contracts///contract.yaml --out .generated/docs/ contract-engine suggest --product ``` Exact flags can change before v1.0. ## Open Questions 1. Should v0 execute SQL tests directly against TiDB Cloud Lake, or only validate their shape? 2. What tool manifests should be supported first: dbt, Dagster, dlt, or one narrow happy path? 3. What is the first publish target: local files, static docs, registry API, or discovery index? 4. How should compatibility history be retained across the first live 1.0.0 baseline and later promoted contract versions? 5. What approval metadata is required for a major-breaking contract change? 6. Which observed-lineage event model should be adopted when runtime lineage integration begins? 7. How should the v0 classification mapping be translated to TiDB Cloud Lake grants, masking, retention, and audit features? 8. Which standard permission model should be used first for agent permissions: RBAC, ABAC, or a hybrid?