# Agent Guide

## What This Repo Is

This repo is a design workspace for a TiDB Cloud Lake-first data lakehouse and data-agent mesh.

The current goal is not to build production services. The goal is to make the architecture crisp: principles, boundaries, contracts, schemas, governance, orchestration, and agent-facing surfaces.

Start with:

1. `README.md`
2. `CONTEXT.md`
3. `docs/design/v1-lakehouse-architecture.md`
4. `docs/design/README.md`
5. `docs/adr/README.md`
6. `docs/principles/README.md`

## Current Architecture Posture

- TiDB Cloud Lake is the product and platform anchor.
- Databend is underlying engine context, not a peer product target for v1.
- Data products are the primary unit of ownership and architecture.
- Raw, staging, curated, and service are logical layers, not a mandatory lifecycle progression.
- Contracts are mandatory, versioned, Git-authored, and runtime-served.
- Authored contracts are YAML; runtime-served contract views are JSON.
- The contract engine compiles contracts, fragments, generated metadata, and runtime facts into served views.
- Governance is classification-driven with inheritance.
- Agent autonomy is in scope as a target capability, but detailed agent permission and approval workflows are deferred.

## Terms To Preserve

Use these terms exactly unless the user changes the model:

- Use `layer`, not `lifecycle_stage`.
- Use `tests`, not `quality_checks`.
- Use `runtime-served contract view` for the agent/runtime JSON projection.
- Use `authored contract schema` for the Git-authored YAML source.
- Use `data product`, not table/pipeline/project, as the main ownership unit.
- Use `data-agent mesh` for the target architecture.

Do not introduce a separate source-aligned/domain-aligned product taxonomy. Layer plus contract guarantees already carry that meaning.

## How To Work

When exploring or changing design:

1. Read `CONTEXT.md` first.
2. Check `docs/adr/` for prior decisions and `docs/principles/` for implementation-shape rules before reopening a topic or adding ingestion, transform, or orchestration engine code.
3. Ask one focused question at a time when the design is still ambiguous.
4. Give a recommended answer with the question.
5. When a term is settled, update `CONTEXT.md`.
6. When a hard-to-reverse decision is settled, add or update a short ADR.
7. When a design surface is settled enough, update or add a doc under `docs/design/`.
8. Before merging a change to ingestion, transform, or orchestration engine code, the contract schemas, or any doc under `docs/adr/`, `docs/principles/`, `docs/design/`, or `CONTEXT.md`, run the design conformance review in `docs/agents/design-conformance-review.md` to catch doc-vs-doc, doc-vs-code, and stale-doc drift. When guidance has drifted, fix the guidance **and its example** in every copy of the invariant, not just the code.

Keep docs small, direct, and decision-oriented. Avoid turning v1 into an implementation spec.

## Validation

When changing schemas, run:

```bash
jq empty schemas/data-product-contract.authored.v0.schema.json schemas/data-product-contract.runtime-view.v0.schema.json
```

Also scan for stale vocabulary:

```bash
rg -n "lifecycle_stage|quality_checks|quality checks|contract checks" CONTEXT.md docs schemas
```

When changing ingestion, transform, or orchestration engine code, run the
engine/config boundary scan — it fails on a source/product literal leaking into
reusable engine code
(see [`docs/principles/engine-config-red-lines.md`](docs/principles/engine-config-red-lines.md)):

```bash
python scripts/boundary_scan.py
```

## Deferred Decisions

These are intentionally deferred. Do not accidentally decide them while editing nearby docs:

- Observed-lineage event model
- Generated docs rendering shape
- Bespoke agent permission model
- Detailed approval workflow
- Full governance policy language
- Production implementation design
- Final orchestration/vendor choices

## Repo-Local Skills

Matt Pocock's skills are vendored under `.agents/skills/mattpocock` from `mattpocock/skills` at commit `b8be62ffacb0118fa3eaa29a0923c87c8c11985c`.

Codex does not automatically discover those repo-local skills. If the user names a slash-style skill, or the request clearly matches one, read the relevant `SKILL.md` before acting.

Most relevant skills for this repo:

- `/grill-with-docs`: use for design grilling that updates `CONTEXT.md` and ADRs.
- `/grill-me`: use for focused design interrogation without docs.
- `/to-prd`: use after a design thread is settled enough to become a PRD.
- `/to-issues`: use to break a PRD/design into local markdown issues.
- `/handoff`: use to summarize state for another session.

Skill configuration:

- Issue tracker: local markdown under `.scratch/`; see `docs/agents/issue-tracker.md`.
- Triage labels: see `docs/agents/triage-labels.md`.
- Domain docs: single-context repo; see `docs/agents/domain.md`.
- Design conformance: check changes against the design corpus; see `docs/agents/design-conformance-review.md` (also exposed as the `design-conformance-review` skill).
- Local Airflow boot, prewarm, trigger, monitor, and shutdown: see `docs/agents/airflow-local-boot.md`.