# Mixed-Language Development Model and Project Organization (Rust Core + Go Emulator with Embedded Mini-CDC)

## 1. Scope

This document defines only two things:

1. Mixed-language development model (`Rust` for production core, `Go` for emulator with embedded Mini-CDC).
2. Mixed-language project organization (source/documentation layout, module mapping, dependency boundaries, and engineering workflow).

Out of scope:

- Single-language alternatives.
- Generic organization-option comparison.
- Non-structural architecture discussions that do not affect mixed-language development and repository structure.

## 2. Design Anchors

The mixed-language model keeps canonical service and plane boundaries unchanged.

- Plane boundaries: `docs/specs/core/search-fabric-api.md` Section `3.1`, Section `4`, Section `5.1`, Section `6.1`
- Canonical gRPC services: `api/proto/searchfabric/v1/search_fabric.proto` service definitions (`QueryGatewayService`, `RelationalQueryService`, `IngestGatewayService`, `CatalogControlService`, `IdAllocatorService`, `InternalDebugService`)
- Topology and stateful collaboration chain: `docs/specs/core/search-fabric-architecture.md` Section `3.1`, Section `3.2`, Section `6.7`, Section `6.9`
- Emulator boundaries: `docs/specs/core/emulator.md` Section `5`, Section `6.1`, Section `6.2`, Section `9`, Section `10`, Section `11`
- Mini-CDC runtime and fail-fast constraints: `docs/specs/core/embedded-mini-cdc.md` Section `3.11`, Section `5.2`, Section `5.4`, Section `6`, Section `8.1`, Section `8.2`

## 3. Mixed-Language Development Model

### 3.1 Language Ownership

1. `rust/` owns all production paths: `query`, `ingest`, `control`, `stateful`, `storage`, `background`, and production protocol adapters.
2. `go/` is limited to emulator domain, and Mini-CDC is implemented as an embedded emulator submodule.
3. Contract governance is domain-scoped with one normative owner per domain:
   - machine-readable public/cross-language protobuf source: `api/proto/searchfabric/v1/search_fabric.proto`
   - machine-readable public protobuf-set manifest for the checked-out revision: `api/proto/searchfabric/public-proto-set.txt`
   - machine-readable system-owner protobuf-set manifest for the checked-out revision: `api/proto/searchfabric/system-proto-set.txt`
   - machine-visible protobuf namespace boundary for the checked-out revision: public contract stays in `package searchfabric.v1`, internal-only owner files use `package searchfabric.v1.internal`
   - normative semantic/wire specification: `docs/specs/core/search-fabric-api.md`
   - canonical error-code registry: `docs/specs/core/error-codes.md`
4. `gen/rust` and `gen/go` are reserved locations for checked-in generated artifacts. In the current repository revision they may contain placeholders only; once generated bindings are committed, they must be derived from `api/proto/searchfabric/v1/search_fabric.proto` and must never be hand-edited.

### 3.2 Development Flow

1. Contract-first:
   - change `api/proto/searchfabric/v1/search_fabric.proto`, any impacted internal protobuf owner files under `api/proto/searchfabric/v1/`, `docs/specs/core/search-fabric-api.md`, and `docs/specs/core/error-codes.md` in the same change whenever public wire semantics, internal RPC semantics, validation rules, or canonical `SF_*` mappings evolve.
   - `[Rule P-201]` `api/proto/searchfabric/public-proto-set.txt` is the machine-readable manifest for the public/cross-language protobuf contract set, and `api/proto/searchfabric/system-proto-set.txt` is the machine-readable manifest for the full checked-out system owner set; `api/proto/searchfabric/v1/search_fabric.proto` remains the public/cross-language source for code generation and public protobuf validation, while the system manifest covers every checked-in internal/public protobuf surface in the current revision; `docs/specs/core/search-fabric-api.md` remains the normative semantic companion specification for public behavior; `docs/specs/core/error-codes.md` remains the canonical error registry.
   - `[Rule P-202]` package namespace is the authoritative public/internal protobuf boundary inside the shared `api/proto/searchfabric/v1/` tree: the public owner stays in `package searchfabric.v1`, and internal-only owner files use `package searchfabric.v1.internal`.
   - other documents may reference or summarize these domains, but they MUST NOT redefine them.
2. Current mandatory validation step:
   - `bash scripts/validate-api-contract.sh`
   - explicit manifest-driven `protoc` sweep over `api/proto/searchfabric/public-proto-set.txt` for the public/cross-language contract
   - explicit manifest-driven `protoc` sweep over `api/proto/searchfabric/system-proto-set.txt` for checked-out system-contract health
3. When checked-in generated bindings are introduced beyond placeholders, the repository MUST add one documented generation entry point (for example `make proto` or an equivalent script) that regenerates both Rust and Go outputs from the same `api/proto` source.
4. Mixed-language validation target state after codegen lands:
   - Rust local suite.
   - Go local suite (emulator + embedded Mini-CDC).
   - Cross-language conformance suite using shared vectors.
5. Merge gate:
   - current required gate for contract changes is protobuf validation plus same-change sync across `.proto`, API spec, and error-code registry,
   - once checked-in generated bindings and consumers exist, the gate expands to generation freshness plus mixed-language conformance.

### 3.3 Boundary of Semantic Ownership

1. Canonical semantic decisions remain in the production core model.
2. Emulator replays canonical semantics; it does not define an independent semantic branch.
3. Embedded Mini-CDC provides source-side normalization/streaming semantics for emulator flows only; it does not introduce a second control/query semantic path.

## 4. Mixed-Language Repository Organization

### 4.1 Recommended Source Tree

```text
search-fabric/
  api/
    proto/searchfabric/v1/               # machine-readable canonical protobuf contract home
    openapi/

  rust/                                  # production workspace
    Cargo.toml
    crates/
      kernel/
      ingest/
      query/
      control/
      stateful/
      storage/
      background/
      adapters-es/
      adapters-tp/
      adapters-tidb-cdc/
      observability/
    bins/
      sf-query-gateway/
      sf-ingest-gateway/
      sf-catalog-control/
      sf-commit-coordinator/
      sf-visibility-barrier/
      sf-manifest-publisher/
      sf-background-coordinator/
      sf-compaction-worker/
      sf-gc-worker/
      sf-allinone/

  go/                                    # emulator workspace only
    go.work
    emulator/
      cmd/
        sf-emulator/
      internal/
        normalizer/
        singlewriter/
        queryengine/
        snapshot/
        tokens/
        readiness/
        faultprofile/
        minicdc/                         # embedded Mini-CDC module
          runtime/
          subscription/
          dml/
          ddl/
          schema/
          checkpoint/
          lifecycle/

  gen/
    rust/
    go/

  tests/
    conformance/
    integration/
    emulator/
      minicdc/
    performance/
    chaos/

  deploy/
    profiles/
      cloud/
      private/
      allinone/

  scripts/
  docs/
```

### 4.2 Plane-to-Module Mapping

- Ingest plane -> `rust/crates/ingest/*` + edge adapters
- Query plane -> `rust/crates/query/*`
- Control plane -> `rust/crates/control/*`
- Stateful collaboration -> `rust/crates/stateful/*`
- Storage + background -> `rust/crates/storage/*`, `rust/crates/background/*`
- Emulator model -> `go/emulator/internal/*`
- Mini-CDC model (embedded) -> `go/emulator/internal/minicdc/*`

### 4.3 Documentation Tree Conventions (Active Layout Parallel to `plans`)

```text
docs/
  plans/                                 # dated design drafts and implementation plans
  specs/                                 # stable, implementable source-of-truth specs
    core/                                # canonical platform specs
  architecture/                          # topology, module boundary, deployment architecture docs
  compatibility/                         # ecosystem compatibility specs
  testing/                               # stable test-strategy and integration-test design docs
  governance/                            # roadmap, scope, milestone, and release-governance docs
  articles/                              # customer-facing or explanatory technical articles
  adr/                                   # architecture decision records
  runbooks/                              # ops/release/incident playbooks
```

Document placement rules:

1. Stable core specs live under `docs/specs/core/`.
2. Compatibility specs live under `docs/compatibility/`.
3. Repository/module-boundary and integration-architecture docs live under `docs/architecture/`.
4. Stable test-strategy docs live under `docs/testing/`.
5. Roadmap/scope governance docs live under `docs/governance/`.
6. Customer-facing technical explainers live under `docs/articles/`.
7. `docs/plans/` is for dated design/process documents, not final SoT specifications.

Document status vocabulary for stable design docs:

- `Draft`:
  - working or exploratory design material,
  - not eligible as launch-authoritative or milestone-closing contract,
  - lower-tier docs and closeout records MUST NOT present it as frozen current-state authority.
- `Frozen`:
  - accepted current-state authority for the document's declared scope,
  - launch-critical implementation, roadmap closeout, and lower-tier references may rely on a frozen revision as the active contract.

Usage rules:

- Core specs under `docs/specs/core/` may qualify the frozen status with domain wording such as `Frozen (Normative API contract)` or `Frozen (Normative error registry)`.
- Steady-state component/integration notes under `docs/architecture/` should use `Frozen architecture note` once they are part of the accepted runtime owner stack.
- Governance notes under `docs/governance/` should use `Frozen governance note` once roadmap gates are allowed to cite them.
- If a document is not `Frozen`, any document that depends on it MUST describe it as non-final and MUST NOT use it as a launch-authoritative closure dependency.

## 5. Dependency and Integration Rules

### 5.1 Hard Dependency Rules

1. Rust production crates must not import Go internals.
2. Go emulator modules (including embedded Mini-CDC) must not import Rust internals.
3. Cross-language interaction is allowed only through the proto-defined gRPC boundary:
- wire shape/codegen are defined in `api/proto/searchfabric/v1/search_fabric.proto`
- wire semantics are normatively constrained by `docs/specs/core/search-fabric-api.md`
- canonical error semantics in `docs/specs/core/error-codes.md` govern that same gRPC boundary; they are not an independent peer protocol boundary
Any snapshot/chunk object encoding may exist only as an internal persistence format behind the gRPC boundary. It MUST remain subordinate to the canonical protobuf contract and MUST NOT be treated as an allowed peer cross-language interaction boundary.
4. Every protocol domain has exactly one normative owner:
- wire shape/codegen owner: `api/proto/searchfabric/v1/search_fabric.proto`
- semantic/wire behavior owner: `docs/specs/core/search-fabric-api.md`
- error-code registry owner: `docs/specs/core/error-codes.md`
Runtime code must generate or mechanically derive protocol-impact constants from the normative owner of that domain. Other documents may reference or summarize those domains, but they MUST NOT redefine them.

### 5.2 Hot-Path Safety Rules

1. Do not add cross-language FFI into ingest/query hot paths.
2. Keep canonical digest/idempotency/token semantics centralized and contract-driven.
3. Keep emulator single-writer behavior aligned with canonical ingest ordering and admission rules.
4. Keep embedded Mini-CDC fail-fast/non-blocking callback constraints explicit in emulator runtime tests.

## 6. Test, CI, and Release Organization

### 6.1 Test Layout Responsibilities

1. `tests/conformance/*`: cross-language contract equivalence.
2. `tests/integration/*`: end-to-end adapter -> canonical -> stateful -> query flow.
3. `tests/emulator/*`: emulator parity with canonical contract behavior.
4. `tests/emulator/minicdc/*`: embedded Mini-CDC DML/DDL ordering, checkpoint monotonicity, lifecycle constraints.
5. `tests/performance/*`, `tests/chaos/*`: concurrency, replay, failover, manifest recovery.

### 6.2 CI Gates

1. Current required contract gate is `bash scripts/validate-api-contract.sh`, explicit manifest-driven `protoc` compilation over `api/proto/searchfabric/public-proto-set.txt` for the public/cross-language contract, and explicit manifest-driven `protoc` compilation over `api/proto/searchfabric/system-proto-set.txt` for checked-out system-contract health; reviewers MUST also verify same-change coherence with `docs/specs/core/search-fabric-api.md` and `docs/specs/core/error-codes.md`.
2. When generated bindings are checked in beyond placeholders, CI MUST run the documented generation entry point and fail when generated output is stale (`git diff` after generation is non-empty).
3. Required checks in the mixed-language generated-binding state:
- Rust lint/test.
- Go lint/test (emulator workspace, including embedded Mini-CDC packages).
- Cross-language conformance.
4. Go emulator binaries are excluded from production release jobs.

### 6.3 Release Policy

1. Use one release version line for mixed Rust+Go artifacts.
2. Publish only after contract compatibility checks pass across both workspaces.
3. Rollback artifacts by version set, not by single-language partial rollback.

## 7. Practical Operating Rules

1. Keep production query/ingest/control/stateful/background logic in Rust only.
2. Keep Go restricted to emulator domain; keep Mini-CDC under `emulator/internal/minicdc`.
3. Do not fork the contract: `api/proto/searchfabric/v1/search_fabric.proto`, `docs/specs/core/search-fabric-api.md`, and `docs/specs/core/error-codes.md` are the normative owners for their respective domains. When contract semantics evolve, update the owning domain(s) in the same change, and require all other documents to reference or summarize rather than redefine message/service definitions or `SF_*` semantics.
4. Prefer shared contract vectors over cross-language internal code reuse.
5. Treat cross-language conformance regressions as release blockers.
6. Place all new stable semantic specifications under `docs/specs/core` (or `docs/compatibility` for compatibility specs), but keep machine-readable protocol definitions in `api/proto`; Markdown specs explain and constrain the contract, they do not replace the canonical `.proto`.

## 8. Summary

The repository is organized as a mixed-language, contract-first system:

- Rust carries production semantics and runtime hot paths.
- Go carries emulator semantics, with Mini-CDC embedded as an emulator submodule.
- `api/proto/searchfabric/public-proto-set.txt` and `api/proto/searchfabric/system-proto-set.txt` carry the machine-readable public/system protobuf-set manifests for the checked-out revision; `api/proto/searchfabric/v1/search_fabric.proto` remains the canonical public/cross-language protobuf source; `docs/specs/core/search-fabric-api.md` carries the normative semantic/wire specification for public behavior, and `docs/specs/core/error-codes.md` carries the canonical error registry that every compatibility and runtime document must reuse.
- `docs/plans` vs `docs/specs/core` vs `docs/compatibility` are already separated by lifecycle purpose in the current repository layout.

This keeps language boundaries and documentation governance explicit while preserving one canonical behavior model.