# L0 Platform Substrate — Architecture and Module Design Status: living architectural specification Purpose: invariants, layer responsibilities, target architecture, and known violations of the L0 platform substrate Date: 2026-05-14 Related: [`nexuscrm-platform-architecture.md`](nexuscrm-platform-architecture.md), [`frontend-l0-l1-ownership-inventory.md`](frontend-l0-l1-ownership-inventory.md), [`salesforce-v63-api-contract.md`](salesforce-v63-api-contract.md), [`l0-module-relocation-audit.md`](l0-module-relocation-audit.md) ## 1. Purpose and Position NexusCRM is a metadata-driven, Salesforce-compatible SObject platform. The platform binary is not a CRM application; it is a generic interpreter that hosts any number of application packs. L0 is the platform binary. Everything in this document describes only what the binary itself owns and contracts. Application schema, tenant overlays, and record data are out of scope and live in L1, L2, and L3 respectively. The strongest design statement governing L0: > Salesforce compatibility is a runtime and protocol contract over generic engine primitives, not a baked-in CRM object catalog. The platform should interpret Salesforce-shaped metadata; it should not know what a CRM is. This rule shapes every module boundary in L0. A module either implements a generic interpreter for a Salesforce-shaped primitive (objects, fields, sharing, flows, validations, approvals, layouts, permissions) or it provides cross-cutting infrastructure (audit, events, storage, transactions). No module owns application vocabulary. ## 2. Layer Position L0 sits at the bottom of a four-layer model: ``` +----------------------------------------------------------------+ | L3 Record Data user-created business rows | | L2 Tenant Overlay admin-authored configuration | | L1 Application Pack declarative metadata for one app | | L0 Platform Substrate the platform binary <-- THIS DOC | +----------------------------------------------------------------+ ``` The relationship between L0 and the layers above it is asymmetric: - L0 reads metadata that originates in L1 or L2 and interprets it. L0 never authors L1 or L2 facts. - L0 is blind to L1 and L2 instances. A generic engine can know that a field exists, that it has a type, that it has a Reference target. It cannot know that the field is called `Industry` or that its value should belong to a specific picklist. Those are L1 or L2 facts. - L0 changes are scoped to engine behavior and substrate shape. Changes to L1 or L2 metadata content are not L0 changes. Four invariants from the platform architecture govern every L0 decision: 1. **Author decides layer.** Engineer-authored facts live in L0 or L1. Admin-authored facts live in L2. User-authored facts live in L3. 2. **L0 is blind to L2 instances.** Generic runtime inspects schema shape, never the specific custom field name, profile literal, picklist value, RecordType DeveloperName, or ApprovalProcess name. 3. **One fact, one SSoT, one read path.** If Salesforce API v63 already models a fact, L0 reuses that representation. 4. **Exhaust reuse before inventing.** Salesforce primitive → existing platform primitive → generic extension of an existing primitive → new primitive. A new substrate is admitted to L0 only if it would serve a Salesforce-compatible CRM pack and a non-CRM pack (Jira-like, ITSM, ERP) without any code-level branch on application vocabulary. ## 3. Internal Architecture L0 **targets** a hexagonal layering: dependencies flow inward toward the domain port boundary and outward toward adapters. The target shape is below; HEAD has documented violations of this rule that are being retired (see "Current-state caveats" at the end of this section). ``` +--------------------------------------------------------+ | interfaces/ HTTP wire and middleware stack | +--------------------------+------------------------------+ | +--------------------------v------------------------------+ | application/services/ orchestration and DI root | +--------------------------+------------------------------+ | +--------------------------v------------------------------+ | features/ generic capability modules | | (CRUD, permissions, metadata, | | automation, audit, events, ...) | +--------------------------+------------------------------+ | (depends only on ports) +--------------------------v------------------------------+ | domain/ interfaces and pure rules | | ports/ schema/ rls/ | +--------------------------^------------------------------+ | (implements ports) +--------------------------+------------------------------+ | infrastructure/ adapter implementations | | database/ persistence/ email/ storage/ | +---------------------------------------------------------+ Cross-cutting machinery: bootstrap/ schema and class-invariant assertions controlplane/ pack lifecycle and consistency verification layoutrouting/ generic layout selection shared/pkg/ primitives mirrored to the frontend backend/pkg/ platform-internal public packages ``` The target dependency rule: - `interfaces` depends on `application` - `application` depends on `features` and `domain` - `features` depends on `domain` only; never on `infrastructure` directly - `infrastructure` implements interfaces declared in `domain/ports` The contract `features → domain → infrastructure` is the property that lets feature code be unit-tested without a database and lets the backing store change without touching feature logic. Any code path that bypasses it is a layering violation. ### Current-state caveats HEAD does not yet satisfy the target rule above. Two classes of violation exist and are tracked as architectural debt: - **`features/*` importing `infrastructure/persistence` directly.** At HEAD `grep -rln '"github.com/nexuscrm/backend/internal/infrastructure" backend/internal/features` shows production imports in `features/records/persistence_{service,insert,delete,bulk_insert}.go`, `features/records/{group_membership_closure,polymorphic_discriminators,service_test_helpers}.go`, `features/automation/autonumber/reserve.go`, `features/events/outbox_service.go`, `features/metadata/{metadata_objects,helpers}.go`, and `features/analytics/{query_service,query_helpers,picklist_resolver,crt_column_path,report_service}.go`. The target is that each of these calls the adapter through a `domain/ports` interface; the cleanup is incremental and not yet sequenced into the refactor program. - **`domain/ports` importing `features/*`.** `domain/ports/composite.go` imports `features/composite`, which inverts the dependency direction. The composite handler's interface needs to be promoted into `domain/ports` standalone (without referencing the feature package) for the dependency direction to recover. - **`controlplane/packs` exposing raw `*sql.DB` to pack apply.** `ApplyDeps.DB *sql.DB` (`controlplane/packs/deps.go`) is consumed by seed writers — `ApplyAutoNumberFormatSeeds` (`controlplane/packs/platform_config.go`) and the apply closures in `source_backed_metadata_components.go` — which construct SQL directly. The target is to route pack-apply writes through a dedicated port (or extend `ports.MetadataRepository`) and remove `*sql.DB` from `ApplyDeps`. - **`features/automation` constructing HTTP directly — closed by PR #1621.** `executeCallWebhook` in `backend/internal/features/automation/action_service.go` previously built requests through `http.Client` rather than through a `ports.HTTPClient`. **Closed by PR #1621 (NCRM-INT-10 AC6)**: the webhook caller now goes through `ports.HTTPClient`, with two adapter implementations sharing a single `infrastructure/url_safety` SSoT (allowlist + `*url.Error` redaction): `infrastructure/http_client/NewWebhookClient` (proxy-aware via `http.ProxyFromEnvironment`, 30s `Timeout`, no IP-pinned dial) is wired into `ActionService.executeCallWebhook` for admin-configured webhook targets that may traverse a corporate egress proxy; `infrastructure/http_client/NewOutboundClient` (direct-only, dial-time IP-pinned with Happy Eyeballs) is reserved for `connector_callout.Engine` traffic to vendor APIs over public internet. Proxy support + dial-time IP pinning are mutually exclusive on the same `*http.Client` (proxy CONNECT hides the final hop), so transport policy is per-consumer while the URL allowlist + redact contract is SSoT. Both adapters refuse cross-host redirects and HTTPS→HTTP scheme downgrades unconditionally — any part of the initial URL (path / query / userinfo) and any request header can carry credentials in conventions that cannot be enumerated; the only provably-complete defense is to refuse the redirect class. These violations do not refute the target. They are the gap between the target and the current implementation, and reading code as SSoT is the only reliable way to know which paths still bypass the port boundary. ## 4. Module Decomposition L0 contains a small number of modules. Each module has one responsibility and a single SSoT-or-port-shaped boundary with its peers. ### bootstrap Owns the platform's DDL and the class-level invariants that must hold for the platform to be functional at all. Responsibilities: - Materialize `_System_*` schema from a single declarative source - Reconcile column shapes incrementally (idempotent ALTER paths) - Seed managed metadata (system administrator profile, default theme, system tables) - Stamp one-shot migrations through `_System_Migration` so each migration runs once per deployment, not on every boot - Execute the class-invariant assertion battery at the end of each apply The migration-stamp substrate is the canonical mechanism for any historical cleanup, schema rename, or data backfill that is genuinely one-shot. A migration that runs on every boot without either a stamp or an idempotent self-heal fast path is a defect. bootstrap does not interpret application metadata. It establishes the substrate on which interpretation runs. ### controlplane Owns the lifecycle of installed application packs. Responsibilities: - Resolve pack sources (currently `official_repo:` and `local_path:`) - Drive the six-phase install plan (ObjectMetadata → Security → Rules → Automation → UISurfaces → Reporting) - Compute and verify canonical pack payload hashes - Detect drift between expected and live metadata, distinguishing immutable surfaces from admin-overlayable ones - Provide one-stop verification entry points (`verify`, `verify-pack`) **假设预期 (Target):** controlplane reads pack manifests and writes through `domain/ports`; it does not embed pack-specific Go code paths. **现状 (Current):** the read path conforms — pack source resolution and manifest parsing route through `MetadataRepository`. The write path partially violates the target. `ApplyDeps.DB *sql.DB` (`controlplane/packs/deps.go`) exposes the raw connection to pack apply, and seed writers such as `ApplyAutoNumberFormatSeeds` (`controlplane/packs/platform_config.go`) and the apply closures in `source_backed_metadata_components.go` construct SQL directly against `deps.DB`. **不符合预期 (Gap):** pack-apply seed writers bypass `domain/ports`. Migration target: route pack-apply writes through a dedicated port (or extend `ports.MetadataRepository`) and remove `*sql.DB` from `ApplyDeps`. Sequenced alongside the other §3 known boundary violations. ### domain Owns the cross-layer contracts and pure rules that have no environmental dependency. - `domain/ports/` declares every interface across which features and infrastructure communicate. **Target:** every cross-module call in L0 traverses one of these interfaces. **Current:** the target holds for most paths; known violations are enumerated in §3 ("Known boundary violations at HEAD") and in the controlplane / infrastructure subsections below. - `domain/schema/` declares the DDL-shaped types and the managed permission set. - `domain/rls/` implements sharing-model and access evaluation as a pure function, returning SQL fragments that downstream repositories compose into queries. domain contains no business logic and no I/O. It is the spec the rest of the binary implements against. ### features The capability layer. Each subdomain implements one generic engine or one cross-cutting service. - `metadata` is the single read path for object, field, relationship, validation rule, autonumber, and describe data. It is the most important SSoT in L0. - `records` is the generic CRUD path over any SObject. Every record write threads through it. - `permissions` is the single read path for object, field, profile, permission set, role, and group access decisions. - `automation` is the generic flow, validation rule, approval, action, autonumber, and scheduler runtime. It contains three internal subpackages: `autonumber/` (declaration, reservation, and runtime formatting), `triggerprojection/` (the canonical extraction of trigger order from flow metadata), and `triggersub/` (the `_System_TriggerSubscription` registry that materializes subscriptions and reconciles them with flow definitions). - `audit` is the platform's append-only history surface for record-level events. Flow-element audit lives in `automation` as a private subsystem (`_System_FlowAuditLog`). The retention, taxonomy, and read-path relationship between these two surfaces is future work — to be specified in a dedicated audit-subsystem design. - `events` is the outbox and event bus that connect synchronous record writes to asynchronous consumers. It also carries the AfterSave chain-id lineage and tuple-count cap that bound recursive trigger expansion. - `uiapi` adapts metadata into the Salesforce UI API wire shape. - `auth` mints, validates, and revokes user sessions. - `mru`, `streaming`, `files`, `composite` are smaller cross-cutting services for most-recently-used tracking, push notifications, attachments, and composite-tree request handling. - `compat` is a transitional registry layered on top of `metadata`. It is retired through the refactor program; new facts go directly to `metadata`. A capability module never owns an SObject-shape registry of its own. If it needs object or field facts, it asks `metadata`. ### infrastructure Adapter implementations of the ports declared in `domain`. The dominant subpackage is `persistence`, which implements every repository against the TiDB-compatible storage backend. **假设预期 (Target):** Adapters are the only place in L0 that constructs SQL or speaks to a network endpoint. Features call adapters only through ports. **现状 (Current):** the target holds for L0 read paths and most write paths. One material gap remains: pack-apply seed writers documented under the `controlplane` subsection above. The previous second gap — the webhook action constructing HTTP directly — was closed by PR #1621 via two `infrastructure/http_client` adapters behind a single `ports.HTTPClient`: `WebhookClient` (proxy-aware) for Flow webhook, `OutboundClient` (direct-only, IP-pinned) for `connector_callout.Engine`. Both share the same `infrastructure/url_safety` allowlist + redact contract and the same redirect policy (cross-host refused + HTTPS→HTTP scheme downgrade refused unconditionally); transport policy is per-consumer because proxy mode and dial-time IP pinning are mutually exclusive on the same `*http.Client`. **不符合预期 (Gap):** pack-apply SQL writers. Migration target: pull pack-apply seed writers behind a dedicated port (or extend `ports.MetadataRepository`) and remove `*sql.DB` from `ApplyDeps`. ### interfaces The HTTP boundary. Middleware (request ID, tracing, auth, permission load, rate limiting, error mapping) plus the REST handlers that translate the wire to feature calls. Handlers do not author business decisions; they translate request shape to feature method calls and translate feature responses back to v63-shaped wire. The translation contract is documented in `salesforce-v63-api-contract.md`. ### application The dependency-injection root and the platform-level operations layer. `ServiceManager` wires every feature service together at startup; `SystemManager` exposes administrative operations (bootstrap reapply, repair, rebuild, verify) that mirror the `setup` CLI. application does not contain business logic. It establishes the instance graph the rest of the binary uses. ## 5. The Single Source of Truth Catalog L0 holds many facts. Each fact has exactly one authoritative read path. The discipline is the property that prevents parallel registries from drifting against live metadata. | Fact class | SSoT | Read path | |------------|------|-----------| | Object, field, relationship shape | `_System_Object` / `_System_Field` / `_System_Relationship` | `metadata.MetadataService` snapshot | | RecordType and layout | `_System_RecordType` / `_System_PageLayout` | `metadata.MetadataService`; layout selection via `layoutrouting` | | Profile, permission set, role, group identity | `Profile` / `PermissionSet` / `UserRole` / `Group` | `permissions.PermissionService` | | Object, field, custom, and system permission grants | `ObjectPermissions` / `FieldPermissions` / `_System_ProfilePermission` / `_System_PermissionSetPermission` / `ProfileCustomPermission` / `PermissionSetCustomPermission` | `permissions.PermissionService` | | Transitive group membership | `_System_GroupMembershipClosure` | `permissions.PermissionService` + `domain/rls` | | Sharing and ownership | `SharingRule`, `OwnerId` columns | `domain/rls.Result.ToSQLFilter` | | Validation rules | `_System_ValidationRule` | `metadata.MetadataService` | | Flow definitions | `FlowDefinition` | `metadata.MetadataService` | | Approval process metadata (runtime) | `ProcessDefinition` / `ProcessNode` / `ProcessAction` | `features/automation/approval` via `query.QueryWithFilter` | | Standard actions and their renderers | `_System_StandardAction` (renderer id, required permissions) | `interfaces/rest/standard_action_dispatcher.go` queries `_System_StandardAction` directly via `db.QueryRowContext` / `listActiveActions`; rendererId is opaque to L0 | | Autonumber declarations | `_System_Field.auto_number_*` columns (format, scope, start, token_refs, canonical_hash, counter_reset_policy) + `_System_AutoNumberCounter` for per-(field, scope_key) sequence state | `features/automation/autonumber` (`MetadataSource.Refresh` reads `_System_Field`; the runtime engine reserves and assigns the next value). The legacy `_System_AutoNumber` table and the retired `MetadataRepository.GetAutoNumbers` / `GetAllAutoNumbers` / `GetAllIdentifierRules` compatibility shims have been removed. | | Trigger subscription registry | `_System_TriggerSubscription` | `features/automation/triggersub` | | Trigger ordering | derived from flow metadata | `features/automation/triggerprojection` | | Scheduled run idempotency | `_System_ScheduledRunReservation` | `features/automation` scheduler | | Formula `$User` / `$Profile` / `$UserRole` projections | the user's `UserSession` (frozen at login) | `backend/pkg/formula.BuildFormulaUserContext` | | Field-level permissions effective at runtime | `FieldPermissions` plus object-level fallback | `permissions.PermissionService` with `shared/pkg/fieldpermissions` matching | | Navigation visibility | `ProfileAppVisibility` and `PermissionSetAppVisibility` | `permissions` effective-visibility helper | | Polymorphic discriminator (`_type`) | derived from the FK target row | `features/records.polymorphic_discriminators` | | Installed pack health | `_System_InstalledPack` | `controlplane` | | Pack payload identity | manifest plus canonicalized component payloads | `controlplane` `PackPayloadHash` (`nexus-pack-sha256-v1:`) | | One-shot migration stamps | `_System_Migration` | `bootstrap/migration_stamp` | | Pack-owned field ownership | `_System_Field.OwnerKind` and `OwnerPack` columns | `controlplane` pack-owned field operations | | Connector namespace ingress admission | `_System_ConnectorIngressControl` global namespace row `(Namespace, PackInstallId='', Channel='*')`, with a monotonic `FenceEpoch` per active-to-quiescing transition; a legacy pre-epoch row upgrades to `0` and is never receipt-safe | `nexuscrm-connectors` `admission.SQLGate.RequireActiveTx` locks the row inside the same pessimistic transaction as Event + WorkItem materialization; `TransitionNamespaceFence` is the sole arm/resume writer; missing schema is unavailable/fail-closed and `SQLGate.Decide` is a non-authoritative fast rejection only | | Flow element-level audit | `_System_FlowAuditLog` | `features/automation` private audit subsystem | | Record-level audit | `_System_AuditLog` | `features/audit` | | Outbox events | `_System_OutboxEvent` | `features/events.OutboxService` | | AfterSave event lineage | `chain_id`, `chain_depth` on record event payload | `features/events.LineageFromContext` | | Salesforce v63 wire shape | `docs/design/salesforce-v63-api-contract.md` | `interfaces/rest` handlers | | Rollup field declarations | `_System_Field.RollupConfig` | `features/analytics.RollupService` via `ports.RollupProcessor` | | State machine transition graphs | `_System_Field.Validator` + `_System_Field.ValidatorConfig` | `features/metadata.ValidationService` dispatch on `state_transition` | | Custom Report Type declarations | `_System_ReportType` | `features/analytics.CRTResolver` via `CRTRepository` | A registry violates L0 discipline if it caches or declares any of these facts on top of the SSoT and then accepts writes from a different source. Caches that read from the SSoT and refresh on a documented causal boundary are legitimate; parallel authorities are not. ## 6. The Generic Engines L0 ships a small set of generic engines. Each engine interprets metadata and produces behavior; none branches on application vocabulary. ### Metadata Service The metadata service is the SSoT for everything describable. It maintains a lock-free copy-on-write snapshot of the schema graph plus an inner core graph. Reads acquire the current snapshot pointer and traverse in-memory maps. Cache rebuilds happen out of band and atomically swap the pointer. Decorators run after raw schema load to synthesize virtual fields (for example, the `Permissions` boolean fields on `PermissionSet`). Decorators are the only legitimate way to add computed fields to the snapshot; feature code cannot maintain a parallel shape map. The pack lifecycle refreshes the snapshot at three causal boundaries during install: after ObjectMetadata, after Security, and after Automation phases. Outside these boundaries, the snapshot is immutable. ### Records Engine The records engine is the generic write path over any SObject. A record update threads through this pipeline: 1. Permission preparation: the operation is authorized via `PermissionService`. 2. Validation: validation rules and runtime-scoped lookup validators are evaluated against the proposed write. 3. Field synthesis: defaults, lookups, formulas, and compound fields are applied. 4. Persistence: the write is applied through `RecordRepository`, with RLS injected as a SQL `WHERE` fragment. 5. Audit: change events are emitted. 6. Session invalidation: if the write changes a security-relevant fact, the appropriate session revocation is invoked synchronously within the transaction. 7. Post-update hooks: outbox publication and rollup processing run. Session invalidation triggers are co-located with the write that causes them. The current set of triggers is documented in §8. ### Permission Engine Permission decisions for an `(user, object, record)` triple have exactly one read path. The permission service maintains a snapshot keyed by `(userID, profileID)` and answers system, custom, object, and field permission queries against it. The middleware layer loads the snapshot at request entry. The snapshot is JWT session-bound: profile and role identity come from the JWT and are not refreshed mid-session. Any administrative change that mutates a session-bound fact triggers session revocation at the write site; the affected user re-logs in and receives a fresh JWT. ### RLS Engine Row-level security is a pure function. Given an object, a user, and the sharing model in metadata, the RLS evaluator produces a `Result` describing direct ownership, sharing-rule projections, group transitive closure, parent filters, and SQL criteria fragments. The result is composed into the query path as a `WHERE` clause. The transitive group membership is precomputed into `_System_GroupMembershipClosure` and rebuilt under advisory lock when group, group-member, or user assignments change. The RLS evaluator queries the closure table; it does not recompute closure on every query. ### Formula and Expression Engines Formula and expression evaluators share a generic AST and evaluator core. Formula is the Salesforce-shaped surface used by validation rules, formula fields, flow expressions, and approval criteria. Expression is the lower-level core they share. Formula evaluation has one rule about identity: the `$User`, `$Profile`, and `$UserRole` globals are projected from the user's session exactly once per session, by `BuildFormulaUserContext`. No formula evaluation reads identity from the database. This is the property that guarantees a single save transaction sees byte-identical user context across every validation rule, flow formula, approval criterion, and action expression it executes. ### Validation Rule and Flow Runtimes Validation rules and flows are declarative metadata interpreted by L0. The flow runtime supports record-triggered, scheduled, and invocable execution and threads its state through a typed `FlowRuntimeContext` carrying the interview, the evaluation cache, the persistence layer, and the trigger lineage. The flow runtime communicates with the rest of L0 through exactly four narrow interfaces: `PersistenceProvider` (writes), `QueryProvider` (reads), `SchemaLookup` (metadata), and `FieldMappingApplier` (declarative bulk copy). No SObject name appears in flow runtime code. ### Trigger Projection and Subscription Registry Two adjacent substrates govern trigger ordering and subscription bookkeeping. `triggerprojection` is the canonical extraction of trigger order from flow metadata — it answers "for object X with action Y, what flows fire in what order". `triggersub` materializes a `_System_TriggerSubscription` registry that records which flows subscribe to which (object, trigger) combinations, and provides a reconcile path that detects and repairs drift between the registry and live flow definitions. The two are deliberately separate: projection is a pure derivation from metadata; subscription is a materialized index used by the dispatcher. Drift between them is a verify-gate fail. There is no third party authoring trigger order or subscriptions. ### Before-Save Effects Classification A before-save flow may produce in-place mutations to the record being saved, but it may not perform DML against other records during the before-save phase. The substrate that enforces this is the before-save effects classifier: it inspects a flow definition at deploy time and classifies its effects as in-place-mutation, deferred-after-save, or unsupported. Classification informs both runtime dispatch and pack-verify gating. ### Flow Preview The flow runtime supports a no-DML preview mode used by the frontend for form prefill and validation feedback. Preview shares the runtime engine with live execution but routes through a preview persistence shim that records intended writes without committing them. Preview is fully isolated; live execution observes nothing from a preview run. ### Scheduled Run Reservation Scheduled flows reserve their execution slot in `_System_ScheduledRunReservation` before running. The reservation row carries an idempotency key derived from the flow id, scheduled time, and reservation scope. The reservation is the substrate that makes scheduled runs idempotent and safe against concurrent schedulers. ### AfterSave Chain Lineage AfterSave recursion is bounded by a chain-id lineage. Each record event carries a chain id and depth; new events inherit and increment depth; events whose depth exceeds the cap are dropped. The substrate is the property that prevents an AfterSave flow from triggering itself recursively without bound. The lineage is also surfaced in the audit subsystem so operators can trace the full chain that originated from a user write. ### Automation Audit Subsystem The automation engine maintains its own audit subsystem at element granularity. A flow execution writes one row per element to `_System_FlowAuditLog`, including the resolved values, the elapsed time, and the propagated chain id. The retention policy is per-feature; the read path is future work — to be specified in a dedicated audit-subsystem design. This subsystem is private to the flow runtime. The platform-wide audit surface (`features/audit`, `_System_AuditLog`) handles record-level events. The two surfaces share PII scrub and FLS-aware projection but are otherwise independent. ### Approval Engine The approval runtime supports submission, recall, approval, rejection, and reassignment, multi-step processes, and multiple approver modes (first response, all-must-approve, serial). Step advancement and action execution support FieldUpdate, Alert, and the native approval-notification Flow bridge. FieldUpdates execute in-transaction. Alerts and approval-notification Flows are buffered for post-commit dispatch so notification failures do not split the approval state transaction. The approval engine reads `ProcessDefinition` / `ProcessNode` / `ProcessAction` rows directly through `query.QueryWithFilter` in `approval_service.go`, not through `metadata.MetadataService`. It parses each definition's `Metadata` blob itself via `ParseApprovalProcessMetadata` in `approval_metadata.go`. Metadata-service surfaces are involved only for cross-engine concerns such as merge-field projections and validation-rule reuse during approval criteria evaluation; approval-process metadata authority lives inside the approval engine. ### Action Dispatcher Two dispatchers coexist: - **REST standard-action dispatcher** (`interfaces/rest/standard_action_dispatcher.go`): `POST /services/data/v63.0/actions/standard/:name` keys on the action's `Name` column, looks up the matching `_System_StandardAction` row via `lookupAction`, fails closed on missing `RequiredPermissions`, and executes the row's `FlowDeveloperName`. `RendererId` is read out for the `List` response but is not part of the dispatch decision — it is an opaque hint frontend code uses to pick a renderer component. - **Automation action dispatcher** (`features/automation/action_service.go`): switches on the `_System_Action.Type` column (the runtime enum in `shared/pkg/constants/actions.go`: `CreateRecord`, `UpdateRecord`, `DeleteRecord`, `SendEmail`, `CallWebhook`, `Composite`, `Action`, `SubmitForApproval`); only a subset of the constants currently have case branches and the rest fall through to "unsupported action type". REST standard actions enforce per-action permissions via the `RequiredPermissions` declaration on the `_System_StandardAction` row, checked before dispatch (fail-closed on missing). The automation engine does **not** have first-class per-action permission declarations today — `_System_Action` carries no permission column and `ActionMetadata` carries no permission field; the automation dispatcher's only gate is the downstream record-engine path (FLS, RLS, DML) that each action's effects flow through. First-class per-action permission declarations for automation actions are a **target**, not current behavior. ### Autonumber Engine Autonumber declarations live on `_System_Field.auto_number_*` columns (format, scope, start, token_refs, canonical_hash, counter_reset_policy); per-(field, scope_key) sequence state lives in `_System_AutoNumberCounter`. The legacy `_System_AutoNumber` table has been dropped, and the retired `MetadataRepository.GetAutoNumbers` / `GetAllAutoNumbers` / `GetAllIdentifierRules` accessors (kept briefly as empty compatibility shims) are removed. The engine reserves the next value, formats it according to the field's display format, and assigns it before the record is persisted. AutoNumber initialization is part of the platform's fail-closed boot sequence: an autonumber inconsistency aborts the boot. ### Generic SQL Builder The platform's repository layer composes SQL through a single fluent builder in `backend/pkg/query`. The builder exposes `From`, `Insert`, `Update`, `Delete` entry points (`query/builder.go`) and lets repositories add `WHERE`, `JOIN`, `GROUP BY`, `ORDER BY`, `LIMIT`, and `OFFSET` clauses incrementally. Row scanning into `models.SObject` is handled by `ScanRowsToSObjects` (`query/scanner.go`). The builder is consumed across `infrastructure/persistence/`, `features/records/runtime_scoped_*`, and `features/analytics/report_join_exec.go`. The builder is the preferred SQL composition path, but it is not universal at HEAD: 33 of the 58 production files in `infrastructure/persistence/` import `pkg/query`, while 25 still hand-author SQL strings (`metadata_repository.go`, `oauth_repository.go`, `repo_dashboard.go`, `group_membership_closure.go`, `mru_repository.go`, and others). Reducing the hand-authored set toward zero is incremental work; for each migration, switching an over-builder repo to the builder is a one-update-site change. `ParseFormulaQuery` (`query/parser.go`) is the narrow exception that lifts simple formula-search expressions into builder-shaped predicates. ### SOQL Wire Interpreter The SOQL wire interpreter parses Salesforce v63 SOQL queries into the platform's internal `models.QueryRequest` shape. It is an L1 (Salesforce query LANGUAGE) concern and lives ABOVE the L0 contract in the standalone `github.com/nexuscrm/dialect-sf-crm-soql` module (extracted from `backend/pkg/soql` in #2015 PR2.5; seal-clean — requires `shared` only, no backend dependency). Entry points are `ParseSOQL` and `ParseSOQLWith` (`translator.go`); the parser is split into `lexer.go`, `parser.go`, and `ast.go`; semi-join projection has its own resolution pass (`ResolveSemiJoins` in `resolver.go`) so the records engine never observes nested-query AST. The interpreter is invoked exclusively at the REST boundary: `query_handler.go`, `bulk_query.go`, and `tooling_query_handler.go`. After parsing, the request is handed to the records engine and `permissions.Service` for execution and authorization. The interpreter is the SSoT for v63 SOQL grammar conformance; the contract it implements is documented in `salesforce-v63-api-contract.md`. ### State Machine Substrate A `_System_Field` row with `Validator = "state_transition"` and a `ValidatorConfig` JSON body of the shape `{Initial: [...], Transitions: {: [...]}}` configures a per-field state machine. The validator is dispatched at record save through `metadata.ValidationService` (`validation_service.go`); on insert the initial-state set is consulted, on update the `(oldValue → newValue)` transition is checked against the declared graph, and writes that violate the graph fail the save. Parsing of the config is centralized in `parseStateTransitionConfig` so the graph shape is validated once. The substrate is the property that lets an L1 application declare a workflow without writing Go code, on top of the same record-engine save path that drives validation rules and flows. ### Rollup Engine The rollup engine recomputes derived parent fields when child records change. Its substrate-level surface is two ports in `domain/ports/analytics.go`: `RollupProcessor` (used by the records engine to fan out child-change events) and `DerivedWriter` (used by the rollup service to write parent updates back through the full save lifecycle, including validation, audit, and outbox). The `DerivedWriter` port exists to resolve the circular dependency that would otherwise tie `features/analytics` directly to `features/records`. Implementation lives in `features/analytics/rollup_service.go`; in-transaction dispatch happens at four sites in `features/records/persistence_{service,insert,delete}.go`. The SSoT for rollup declarations is `_System_Field.RollupConfig`. Bulk-insert deliberately skips rollup recompute to bound write latency; the trade-off is documented at the call sites and a separate `RecalculateAll` entry point exists for catch-up. ### Custom Report Type (CRT) Engine The CRT engine resolves declarative report-type metadata into executable cross-object join plans. `_System_ReportType` is the SSoT for CRT declarations; the engine's read path is `crt_loader.go` (load report-type rows through `CRTRepository`), `crt_resolver.go` (resolve report-column requests against join templates), and `crt_synthesizer.go` (synthesize the SQL projection). Polymorphic relationships are handled separately in `crt_polymorphic_joins.go` so the resolver path stays linear for the common case. The engine is consumed at three boundaries: pack install (`controlplane/source_pack_ops.go`, `controlplane/packs/deps.go`), REST analytics surfaces (`dashboard_handler.go`, `formula_view_handler.go`, `analytics_handler.go`), and service-init wiring (`application/services/service_manager_init.go`). The substrate lets a pack declare reports without authoring SQL. ## 7. Pack Lifecycle L0 owns the generic mechanics of pack installation. Pack payload — the actual object, field, validation rule, and flow definitions — lives outside the platform binary in external pack repositories. ### Source Resolution Pack sources name an installable artifact. Two resolvers are implemented: - `official_repo:` — resolved against a configured external pack repository. This is the canonical resolver since pack payload was externalized from the platform binary. - `local_path:` — resolved against a local filesystem path. Primarily used for development and test fixtures. The legacy `in_repo:` prefix that resolved to in-tree pack payloads is accepted at the parser boundary for backward compatibility and rewritten to `official_repo:` internally. New code should use `official_repo:`. Named pack sources (currently only `official_repo`) enforce that the manifest's `name` field equals the source URI; the resolver fails closed when they disagree via `isNamedPackSourceKind` in `pack_source.go`. `local_path` sources are unnamed — the manifest `name` is taken as authoritative and the directory name is not checked. Both resolver kinds require explicit `sharingModel` declaration on every owned object. L0 never infers sharing model. ### Payload Hash Pack identity is a SHA-256 over a canonicalized representation of the manifest plus its component payloads. Component order is deterministic. Source URI, source revision, file timestamps, and filesystem paths are not part of the hash. Two installations of the same pack from different sources produce the same hash. ### State Machine `_System_InstalledPack` carries the lifecycle state per pack: - `installing` is entered before the install plan begins - `active` is entered after the install plan completes and the applied payload hash matches the desired hash - `failed` is entered on install error, with the last error preserved - The `IsDeleted` flag supports retry of soft-removed failed installations The install plan runs six phases in order: object metadata, security, rules, automation, UI surfaces, reporting. Each phase is a unit of work; the metadata snapshot refreshes between phases that depend on each other's outputs. ### Verify Verification answers a single question per installed pack: does the live state match the expected state derived from the manifest? The answer respects the component's mutability: - Locked components must match expected exactly - Defaults-only components are allowed to drift from expected only when the row carries the `IsCustomized` marker indicating legitimate admin override **Enforcement reality (per the comment block in `pack_component.go` next to the `Manageability` constants):** the `defaults_only` contract is fully wired today only for `object_spec` (via `specdrift.CustomizationPolicyAllow`). Other `defaults_only` kinds carry the `IsCustomized` marker for contract intent, but per-kind override-preservation enforcement varies and some kinds rely on per-kind verifiers rather than the central specdrift path. Treat the universal version of this rule as **target**; the source of truth for per-kind status is the enforcement-reality comment in `pack_component.go`. Verify is fail-closed: an unverifiable installed pack blocks server readiness. ### Pack-Owned Field Operations A pack declares custom fields against objects it owns within the same pack. The custom-field component kind carries the target object reference; the platform writes the field with `OwnerKind = "pack"` and `OwnerPack` set to the declaring pack. Cross-pack custom-field extension (declaring a field against another pack's object) is not supported today: `backend/internal/controlplane/packs/source_backed_custom_field_component.go` rejects the pattern with `custom_field … requires declaring object_spec in the same pack`. The substrate gap is tracked in §17. Once a field is owned by a pack, four operations are defined on it: - **Writer** creates the field's `_System_Field` row and physical column with full pack attribution. - **Verifier** asserts the live column matches the declared spec; admin overrides on `defaults_only` surfaces are preserved. - **Repairer** reconciles drift detected by the verifier, restoring the declared shape without disturbing legitimate admin overrides. - **Remover** drops the field when the pack uninstalls or the declaration is removed in a new manifest version. All four operations share a single writer through the controlplane shim. Bootstrap, install-pack, and repair routines route through the shared writer; there is no second path for creating pack-owned fields. ## 8. Session and Security Model The session is the unit across which L0 promises consistent identity. ### Session Construction The session is created at login. The JWT carries the user id, name, email, profile id, profile name, user role id, user role name, connected app id, and OAuth scope (the `UserSession` struct in `backend/pkg/auth/jwt.go`). These values are stamped once and frozen for the session's lifetime. Permission set assignments are not part of the JWT; they are read from the database by the permission middleware on each request. The permission middleware reads the JWT, validates it, and loads the effective permission snapshot keyed by `(userId, profileId)` into the request context. No mid-session database read refreshes the JWT-baked identity. ### Session Revocation Any administrative change that mutates a session-bound fact triggers session revocation in the same transaction as the change. The seven trigger conditions are: - `User.IsActive` transitions `true → false` - `User.ProfileId` is reassigned - `User.UserRoleID` is reassigned - `Profile.Name` is renamed - `UserRole.Name` is renamed - `ConnectedApp` security fields (`IsActive`, `RunAsUserID`, `PermittedUsers`) change - `ConnectedAppAccess` is modified After revocation, the affected user's next request fails JWT validation. The user re-authenticates and receives a fresh JWT carrying the updated identity. This is the property that guarantees a single save transaction sees identity facts that are mutually consistent and have not split mid-transaction. ### Field-Level Security FLS is enforced at runtime, not only at metadata declaration. The enforcement points are record CRUD, record creation, and SOQL compilation. The matching logic is case-insensitive and accepts both `Field` and `Object.Field` qualification. When no explicit FLS row exists for a field, the engine falls back to the object-level CRUD permission. ### Polymorphic Discriminator Immutability A polymorphic lookup field's `_type` discriminator is system-managed. User writes to the discriminator column are scrubbed; the discriminator is derived from the FK target at write time. This is the property that prevents an inconsistency between the foreign key and the type it points at. ### Transitive Group Membership Sharing rules and role-based access frequently need to ask "is user U a member of group G, possibly through nested groups". L0 materializes the transitive closure of group membership in `_System_GroupMembershipClosure` and consults the closure table at query time. A recursive CTE on every query was the previous approach and was both slower and harder to reason about; the materialized closure is the design choice. The closure is rebuilt under an advisory lock whenever group, group-member, or user assignments mutate. Rebuilds run inside the transaction that triggered the mutation so the closure is always consistent with the source rows the same transaction wrote. ### Navigation Visibility Authority App and tab visibility for a user is determined by `ProfileAppVisibility` and `PermissionSetAppVisibility` rows. These are the sole authority for navigation visibility decisions. An earlier dual-source model (where `_System_App.AssignedProfiles` was also consulted) is retired; the legacy column is not a runtime fallback. The cutover is documented in the navigation visibility SSoT contract. ## 9. HTTP Entry The HTTP entry is shaped by a strict, ordered middleware stack. The globally-registered middleware runs in this order: request id, audit entrypoint, request capture, request logger, tracing, recovery, error mapping, security headers, CORS, CSRF, Prometheus, rate limiting. Per-route middleware adds authentication, scope enforcement, app-level rate limiting, permission loading, and a final permission check. The order is load-bearing. Authentication runs before any permission-aware code path. Permission loading caches the snapshot per request. Rate limiting protects the database before authentication does any work. REST handlers translate the v63 wire shape to feature calls and back. The contract for what each endpoint family looks like on the wire is documented in `salesforce-v63-api-contract.md`. Handlers do not embed application logic. ## 10. Extension Model L0 is extended only through metadata. The extension points are: - **Application packs** install through the controlplane lifecycle. A pack declares objects, fields, record types, layouts, validation rules, flows, permissions, actions, related lists, report types, field mappings, autonumber formats, and standard flows in a manifest. All component kinds are interpreted by generic L0 engines; no pack ships executable code that runs inside the platform binary. - **Pack-owned custom fields** through the custom-field component kind, attributing ownership to the declaring pack (`OwnerKind = "pack"`, `OwnerPack`). Today the custom-field declaration must live in the same pack as its target object_spec; cross-pack extension is a known gap tracked in §17. - **Renderer hosts** in the frontend register opaque renderer ids declared by pack metadata. The platform's generic renderer host looks up the component by id and renders it. No generic frontend code infers renderer choice from action name or panel name. - **Invocable actions** declared in metadata bind to generic platform handlers. A pack-specific Go bridge is acceptable as a temporary adapter only when a metadata-only path is being designed; the bridge does not become the long-term extension model. The frontend's current L0/L1 ownership map is documented in `frontend-l0-l1-ownership-inventory.md`. The model is identical in shape to the backend's pack lifecycle: declared metadata interpreted by a generic host. ## 11. Verification L0 enforces its invariants at three levels. ### Class Invariants `bootstrap.RunAssertions` runs as the substrate-readiness check at the top of the verify path (`controlplane/verify.go` and `verify_l0.go`). It is not invoked inline from `ApplyAll`; the apply pipeline applies substrate, repairs, builds the service graph, applies platform packs, rebuilds derived state, and only then runs verify (which invokes `RunAssertions`). It checks facts that must hold for every member of a class — every managed table has the mandatory system fields, every relationship's parent and child object exists, every object name maps to a physical table, every formula parses, the system administrator profile and user exist, every managed permission and access policy is consistent, every required connected app is seeded, every derived artifact is materialized. Class invariants live in `RunAssertions` and not in per-table tests. Per-table tests enumerate one row at a time; the property "every X has Y" silently degrades when a new X is added without a corresponding test. Class invariants iterate the class. ### Pack Verification The controlplane verify family runs against the database and the manifest catalog: - L0 verify checks that the platform's own substrate is sound - Pack verify checks each installed pack against its declared expected state, respecting mutability - Spec drift verify compares declared component shape against live `_System_Field` rows - Flow integrity verify checks that flow definitions are coherent and reachable - Trigger subscription consistency verify checks the trigger registry against flow metadata All verify gates are fail-closed. An unverifiable substrate or pack blocks server readiness. ### L0 Completeness Probes The `tests/integration/l0-completeness/` family exercises three install-mechanic dimensions against reference packs of arbitrary object shape: - **Idempotency** (`idempotent-and-source-missing.sh` dim #1): re-installing the same pack source produces an identical `AppliedPayloadHash`, leaves the install in `Status=active`, and creates no `_System_Field` row drift. - **Source-missing fail-closed verify** (`idempotent-and-source-missing.sh` dim #2): moving the pack source aside and running `verify-pack` exits non-zero with a source-related diagnostic. - **Locked-component drift** (`admin-override-drift.sh`): flipping `_System_Field.Type` on a locked component triggers a verify-pack fatal; the `IsCustomized` row-level escape hatch is exercised separately. These dimensions run against a Salesforce v66-shaped CRM pack and a Jira-like single-object probe (the `jira-like-probe` fixture in the external `nexuscrm-controlplane` module's `testdata/packs/`, containing one `Issue` object with eight fields — `Subject`, `Description`, `Status`, `Priority`, `IssueType`, `ResolutionDate`, `EstimatedHours`, `ParentIssueId` — and three picklists, no flows, layouts, or permissions). Passing across both packs is the multi-app neutrality contract **for install mechanics**. Behavioral multi-app neutrality is not yet covered by the probe family. State-machine transitions, cross-object rollup recompute, transitive permission inheritance over non-CRM data, flow execution on non-CRM records, and approval submission on non-CRM records are gaps in **probe coverage** of existing substrate; their closure plan is the "Future scripts" backlog at `tests/integration/l0-completeness/README.md`. Cross-pack custom-field extension is a different class — it is a **missing substrate** gap tracked in §17, not a probe-coverage gap. ## 12. Bootstrap The bootstrap path materializes the L0 substrate. It is resumable, idempotent, and stamped. A bootstrap run progresses through twelve ordered phases. Each phase persists its state to `_System_BootstrapState`. If a run fails partway, the next run resumes from the persisted phase rather than from the beginning. Each phase is idempotent: re-running a completed phase is a no-op. The phases divide into column reconciliation (`phase0_columns_reconcile`, `phase0_schema_reconcile`), core table creation (`phase1_core_tables`, `phase1_batch_tables`), object metadata population (`phase2_object_metadata`, `phase2_read_object_ids`, `phase2_field_metadata`), field reconciliation (`phase3_field_reconcile`), name field designation (`phase4_name_fields`), field labels (`phase6_field_labels`), spec defaults (`phase7_spec_defaults`), and a boolean fallback (`phase7b_boolean_fallback`). The original `phase5_auto_numbers` step was intentionally removed (see comment in `backend/internal/bootstrap/schema.go`); autonumber registration now happens through the platform-packs apply step rather than as a bootstrap phase. After all bootstrap phases complete, `ApplyMetadata` returns to `ApplyAll` (in `controlplane/operations.go`), which runs the remaining controlplane stages — repair, pack apply, pack-source reconciliation, derived-state rebuild, index ensure, flow-orphan reconcile, and the terminal verify gate — before the apply returns. The verify step is fail-closed and invokes `bootstrap.RunAssertions`. Container auto-start has one authoritative read-and-converge path: `apply-if-needed` under the control-plane apply lock. The shell never treats a read-only marker probe as permission to bypass that path. The full `ApplyAll` completion marker (`StateKey=control_plane_apply_all`, written only at the end of `ApplyAll`) proves that the baseline full sequence ran to completion, but it is not a release-versioned receipt for physical removals or other migrations added after the marker was stamped. On every `auto` boot, `apply-if-needed` classifies once under the lock, consults the distribution source-revision plan, converges its bounded completed-marker migrations, runs the final distribution verify, and only then starts the server. A fully converged restart is an idempotent no-op; a warm upgrade may perform those explicitly released bounded migrations or promote an authorized source revision into `ApplyAll`. This prevents replicas from replaying first boot, prevents read errors from becoming a mutating default, and prevents an old completed marker from bypassing a later destructive cutover receipt. Three not-completed cases are handled distinctly: - **Greenfield** (no bootstrap state) or an **incomplete current-release apply** (a `running`/`failed` `control_plane_apply_all` row — the marker is stamped *before* any work, so a partial/failed apply leaves such a row): `auto` runs first-boot/recovery `apply-if-needed`. `apply` is advisory-locked and idempotent, so concurrent first boots serialize safely. - **Legacy / unmigrated** (no `control_plane_apply_all` row at all, but a completed `schema_initialize` left by a *prior* release): the lock-held `apply-if-needed` classifier **fails closed with a one-time-migration instruction** before any `running` marker is stamped — it neither auto-applies (which would recouple restart to a full apply for an already-serving deployment, the regression `#1590` removes) nor auto-starts (this release's apply may include repairs the database has never run). The operator runs an explicit `setup apply` once, which stamps the marker; later restarts still enter `apply-if-needed` and no-op after all current completed-marker receipts and checks pass. Explicit `setup apply` always runs the full sequence regardless of marker. For the marker to be an *honest* convergence proof, `ApplyAll` upholds three invariants. (1) It stamps `completed` only after its terminal writer-safe `Verify` passes. This is the online apply completion gate, not the offline `VerifyDeep` audit: `VerifyDeep` includes live-data scans such as autonumber V5/V7 that assume record writers are quiesced, while the control-plane apply lock serializes control-plane mutation only. (2) Convergence-critical `RunRepair` steps that `Verify` does **not** cover are made fatal in the apply path so their failure cannot be silently swallowed past the marker. Specifically `RepairOptions.FailClosedOnConvergenceRepairs` (set only by `ApplyAll`) makes the permission SSoT backfill (`_System_PermissionSetPermission`) and the setup-object OP/FP backfill fatal — `VerifyProfilePermissionCoverage` scopes to business objects (`TableType NOT LIKE 'system%'`) and never reads the SSoT, so without this a swallowed backfill failure would leave authorization metadata missing while the marker still said completed. They no-op on a greenfield/clean DB (nothing migrated to backfill; packs seed permissions later), so fail-closed is safe there; standalone `setup repair` keeps them best-effort. The polymorphic-discriminator backfill is similar: when its operator skip flag is set the apply is partial and `completed` is **not** stamped (the CLI exits non-zero). (3) Every *mutating* `ApplyAll` attempt stamps `running` before doing work and records `failed`/`completed` for that attempt; it never preserves a prior completed marker across a reapply, so a re-apply that fails after partial changes leaves a `failed` marker (the next auto restart re-applies) rather than a stale `completed`. The auto path's skip-when-completed check runs under the lock *before* the `running` stamp, so a no-op restart never overwrites a completed marker. **Design principle — the marker proves the baseline apply; `apply-if-needed` is the startup authority; the *pod-readiness* `controlplane.Verify` is the online safety gate, not a complete offline drift detector.** Restart correctness does not re-run the full convergence sequence or writer-quiesced live-data audits on every boot. Instead, the lock-held authority trusts the completed marker only after the current distribution plan, explicitly registered completed-marker migrations, and final distribution verify pass. This preserves the `#1590` no-full-reapply restart contract without pretending an older unversioned marker attests migrations introduced later. Verify still runs fail-closed on startup and catches much (schema readiness, L0 assertions, pack drift via `specdrift`, applied-vs-desired pack hashes, per-pack source-backed derived-artifact compile-proof, **formula-view readability for every active-formula object — pack-owned, `__c`, and tenant — via `verifyActiveFormulaViewsUsable`**, orphan-field / flow-integrity / trigger-subscription integrity), but it is deliberately **not** a complete substitute for apply: some convergence apply performs is not pod-readiness-gated. External post-apply drift on an un-probed surface (a snapshot restore predating the apply, a manually dropped column/table) is an operational event recovered by an explicit `setup apply`, **not** something restart is expected to detect; bounded completed-marker migrations are release-authored convergence, not a general drift-healing scan. The convergence-critical best-effort repairs whose silent failure would make the marker dishonest are now gated: `RunRepair`'s permission SSoT / setup-object OP/FP backfills, field-capability clamping, and legacy plural-label canonicalization fail the apply (via the `FailClosedOnConvergenceRepairs` end-of-repair gate) so the marker is not stamped if they failed, and the related-list layout backfill (`runLayoutMaintenance`) propagates its error to the apply-packs phase for the same reason. `BackfillFieldCapabilities` was also fixed to stop swallowing its scan/UPDATE errors. Every non-`ApplyAll` baseline control-plane mutator invalidates the marker before its first write while holding the same control-plane apply lock. This includes explicit partial/source convergence, standalone metadata/repair/packs/rebuild/index commands, L0 init, and source-backed pack install/reapply/removal flows. Those paths never stamp `completed`; the next `auto` restart must run a full `ApplyAll` to restore the marker. Tenant overlay apply is intentionally separate: the marker scopes only baseline convergence, while tenant overlay convergence is achieved by re-applying the overlay on every opt-in boot. Named residual (accepted, tracked as follow-up): autonumber counter live-max absorption — verifying a counter is not behind the live-data max requires the V5/V7 full-table scan, which is deep-only and assumes writers are quiesced; a counter that drifts behind live rows after apply is recovered by an explicit `setup apply` plus an operator-run `setup verify` in a maintenance window. Tenant-overlay surfaces (overlay autonumber/record-types/action/related-list) are **not** a residual — they are converged by the tenant overlay's idempotent re-apply on every opt-in boot, not by a restart-skip. **Tenant overlay does not skip on restart**, and this asymmetry is deliberate. `controlplane.Verify` is intentionally tenant-agnostic, and the overlay's own readiness check (`runtimeOverlay.Verify` at server startup) only covers object/field *shape* — not the record types, auto-number formats/counters, action and related-list overlays, insert-inheritance / child-clone rules, object `SharingModel`, or tenant flows that `apply-tenant` also writes. There is therefore no *complete* fail-closed backstop that could make an overlay restart-skip safe (a marker-trusting skip could serve a drifted overlay with no readiness failure). Mirroring Salesforce — which trusts deployed metadata on restart *only because* its deploy-time readiness gate is total — the overlay instead **re-applies on every boot that opts into overlay mutation via `NEXUSCRM_APPLY_TENANT_OVERLAY=1`**. `ApplyTenant` is an idempotent reconcile to the selected overlay's desired state, so re-applying converges every overlay surface and heals drift rather than risking a silent serve; it never gates on a stale completion marker. With the opt-in flag unset, the baseline still enters `apply-if-needed`; a converged same-release restart performs no persistent control-plane mutation, while a warm upgrade may run its bounded released convergence work. Follow-up (named boundary): building a *total* tenant-overlay readiness verify — desired-vs-live checks over every surface `apply-tenant` writes (object shape/`SharingModel`, physical columns, record types, action/related-list overlays, auto-number, insert/child-clone closure, tenant flow versions), plus a tenant formula-view usability probe if tenant formula views are ever introduced — would let the overlay use a bounded receipt-and-verify no-op path instead of re-applying on restart. Until that gate exists, re-apply is the correct convergence path; a partial verify is not, because any uncovered surface silently reopens the gap. **Lifecycle invariants (not the full step list — the step list lives in `ApplyAll` and changes as stages are added or reordered):** - `RunAssertions` is the **terminal substrate-readiness check**; it runs after every other apply stage has committed, not inline between stages. - The verify gate is **fail-closed**: a verify error blocks server readiness. - A `RunAssertions` failure surfaces *after* the database mutations have taken effect — writes are not rolled back by the verify gate. - Moving the assertion gate earlier (so that substrate violations fail before pack apply) is a target for the substrate-lifecycle work in §17, not current behavior. Historical cleanups must be either stamped through `_System_Migration` (run once per deployment) or written as self-healing fast-paths with no work to do when their target condition has been met. A historical cleanup that runs on every boot without an idempotent fast-path is a defect — it conflates one-time migration with permanent runtime behavior, in violation of L0 boundary discipline. ## 13. Invariants The full set of invariants L0 enforces: 1. **One fact, one SSoT, one read path.** Every fact in §5 has exactly one authoritative read path. 2. **L0 is blind to L1 and L2 instances.** No SObject-name branch, no custom-field literal, no profile-name literal, no record-type DeveloperName literal, no approval-process DeveloperName literal appears in generic engine code. 3. **Generic engines have no application branches.** The records, query, automation, permissions, and RLS engines work identically for any pack. 4. **One-time cleanup is not permanent runtime.** Historical migrations stamp through `_System_Migration` or self-heal idempotently. They do not silently run on every boot. 5. **Declaration conflicts fail closed.** When a declaration (pack manifest, registry, spec) disagrees with live metadata, the verify path returns an error. "Warn and continue" is not a steady state. 6. **Class invariants materialize in `RunAssertions`.** Per-table enumeration of class properties is not acceptable. 7. **Refactor discipline.** A refactor either removes the wrong-shape piece atomically or lands the new mechanism, switches all read paths, removes the old, and verifies the cutover in one PR. "Land now, fix the shape later" is not acceptable. 8. **Session-bound identity.** Profile id, profile name, user role id, and user role name are stamped at login and frozen for the session. Any administrative change that mutates them revokes the session at the write site. 9. **Formula context is built once.** `$User`, `$Profile`, and `$UserRole` projections are produced by `BuildFormulaUserContext` from the session and never re-read from the database during evaluation. 10. **Pack payload identity is canonical.** The same pack from any source produces the same hash. Source identity is not part of the hash. 11. **L0 ships no application Go code.** Pack-specific Go handlers inside the platform binary are tolerated only as transitional adapters with a documented migration path to declarative metadata. ## 14. Decision Tests Apply to every new L0 substrate or contract: 1. **Greenfield.** Does this work in a brand-new organization with no installed packs? 2. **Admin self-service.** Can an admin change this rule without a code change? 3. **Two-path consistency.** Does every caller read this fact through the same helper or SSoT? 4. **Multi-app neutrality.** Can the same substrate host CRM and a Jira-like pack with no app-object branches in L0? 5. **Salesforce primitive reuse.** If Salesforce v63 already models this fact, are we using that representation? 6. **Override safety.** Will pack verify preserve legitimate L2 overrides on overlayable surfaces? 7. **Write-path co-location.** If a write changes a session-bound or cache-invalidating fact, is the invalidation co-located with the write? 8. **Class-invariant test.** Is the property "every X has Y" expressed in `RunAssertions`, not in per-X tests? A proposal that fails any of these tests is L1 application policy or L2 tenant configuration, not L0 substrate. ## 15. Out of Scope The following are not L0 concerns: - Application schema. Object catalogs (CRM's `Account`, `Lead`, `Opportunity`; Jira-like's `Issue`, `Project`) belong in L1 packs. - Tenant overlays. Profile grants, custom fields with tenant business semantics, tenant-specific flows and validation rules belong in L2. - Record data. User-created business rows are L3. - Frontend ownership and remaining host/runtime surfaces are documented in `frontend-l0-l1-ownership-inventory.md`. - REST wire shape. The v63 endpoint contract — what each endpoint accepts, returns, and how errors map — is documented in `salesforce-v63-api-contract.md`. - L2 patch authority. The contract for how tenant overlays attach to installed packs is documented in `l2-apply-tenant-patch-contract.md`. - Specific application behavior (lead conversion, order activation, opportunity stage progression) is application policy and lives in pack metadata. L0 will not grow to host these concerns. Substrate gaps that surface while building an application are resolved by extending generic L0 primitives, not by branching on application vocabulary. ## 16. Transitional L1 Inside L0 The L0 binary ships four "platform packs" under `backend/internal/controlplane/packs/platform_{core,policy,config,ux}.go`. These files exist as Go-coded packs that the controlplane apply pipeline (`ApplyAll` in `controlplane/operations.go`) runs as the `ApplyPacks` step, before the final `Verify` step that invokes `bootstrap.RunAssertions`. Their content needs care to keep Invariant 11 ("L0 ships no application Go code") honest. The current platform-pack boundary is: | File | Verdict | Reason | |------|---------|--------| | `platform_core.go` | **L0 substrate** | Applies `standard_objects_spec.json`, which contains exactly one entry: `RecordType`. RecordType is a platform-universal primitive. | | `platform_policy.go` | **L0 substrate** | Seeds two profiles (System Administrator + Standard User), the bootstrap admin user, and the bootstrap connected app. The profile shapes are platform-universal; the former CRM/Salesforce vocabulary leaks in `profiles.json` and the duplicate-profile `(Salesforce)` rename branch have been removed. | | `platform_config.go` | **L0 substrate** | Machinery for seeding roles, record types, and standard validation rules. Embedded `roles.json` is empty `{"roles": []}`. No CRM-shape literals in code. | | `platform_ux.go` | **L0 substrate** | Machinery for seeding themes, UI components, setup pages, standard actions. Embedded `themes.json` (two themes: Default active + Midnight inactive), `ui_components.json` (8 components: 7 pages — Dashboard, ObjectView, Flows, ObjectManager, Setup, AppManager, UserManager — and 1 widget — LayoutEditor), `setup_pages.json` (admin UX surfaces: Users & Profiles, Sharing Rules, etc.). All platform-universal. | The previous transitional L1-inside-L0 debt was the in-binary autonumber seed list for 21 Salesforce-shape application objects. That debt is resolved: - `platform_config_autonumber_formats.go` and `platform_payload/autonumber/formats.json` are removed. - `AutoNumberFormatSeeds` is empty by construction in L0. - L0 keeps only the generic seed types, writer, parser, conflict detection, and verifier plumbing. Source-backed L1 packs and tenant overlays pass their own autonumber component payloads through the same generic writer and register their own declared seed keys. **Other potential leaks**: `platform_core.go` mentions `Opportunity` in a comment about pluralization handling (NX#492 historical incident); not active behavior. The platform-packs *mechanism* is L0 substrate; the transitional debt is the application-object autonumber *data* shipped inside the binary. ## 17. Known Substrate Gaps L0 owns substrate today; the gaps below are substrate L0 should own but either does not yet own, or owns only through a narrower partial lane. Each entry names the missing or partial primitive, the first L1 application shape that fails against the gap, the current workaround, and the promotion criteria for first-class substrate status. Each is verifiable through the check named in the entry; for older entries the check may be an absence grep, while partially promoted entries record the current code-backed substrate and the remaining question. These gaps are intentionally distinct from §15 Out of Scope. §15 lists concerns that belong to L1/L2/L3 by design and L0 will never own. §17 lists concerns that meet the §14 decision tests for L0 substrate but are missing, incomplete, or still narrower than the general substrate they may need to become. ### 17.1 Multi-Tenancy Primitive There is no tenant identity column on any `_System_*` table, no tenant context in any request, and no per-tenant data path. The platform is operationally single-tenant: a NexusCRM binary serves one organization's data, and a second tenant requires a separate deployment. - **Verification**: `grep -cE '"name":\s*"(OrgId|TenantId|OrganizationId)"' backend/internal/bootstrap/system_tables.json` returns 0. - **First-failure example**: any second tenant cannot coexist on the same binary. - **Workaround today**: deploy a separate binary and database per tenant. - **Promotion criteria**: tenant identity propagates through session and is enforced at the records-engine and query boundary; all `_System_*` tables either carry a tenant column or are explicitly platform-wide; multi-tenancy decision documented in a dedicated design doc. ### 17.2 Saga / Long-Running Workflow Substrate The longest-lived built-in workflow primitive is `ApprovalProcess`, which is single-record-bound and single-transaction-completion-bound. Flow execution is bounded by `AfterSave` chain depth. There is no `_System_Saga` table and no checkpointable substrate for workflows that span weeks, multiple records, or external callouts. `_System_BulkJob` exists for Bulk API v2 (async DML), but it is not a saga: it has no compensation step, no cross-record workflow state, and no resume semantics for arbitrary user workflows. - **Verification**: `find backend/internal -name '*saga*' -o -name '*long_running*'` returns 0; `grep -rn 'type Saga\|LongRunningJob' backend/internal` returns 0. - **First-failure example**: a sprint-tracking workflow that opens at sprint start, accumulates work-log entries over two weeks, and closes when all child issues reach `Done`. - **Workaround today**: per-pack Go bridge — a transitional adapter, not the long-term extension model. - **Promotion criteria**: a `_System_Saga` substrate with declared steps, compensation actions, persistent state, and resume-after-crash semantics; integrated with `events` so external completion notifications drive state transitions. ### 17.3 Inbound Event / Callback Substrate `_System_Webhook` exists but is **outbound** — `Url`/`Method`/`AuthConfig` describe how L0 POSTs out to an external URL, paired with `events.OutboxService`. The old "no inbound substrate" claim is no longer true. Current code contains a connector-pack scoped inbound substrate: - `_System_ConnectorInboundRoute` and `_System_ConnectorInboundHandlerConfig` carry pack-authored route and handler configuration. - `nexuscrm-connectors/features/connector/inbound` implements generic connector inbound runtime pieces: signature verification, envelope parsing, queue-envelope carriage, batch enrichment, route-to-Flow dispatch, and worker processing. - `nexuscrm-connectors/features/connector/inboundrouting` provides the route resolver used to map `(namespace, eventKind)` to a Flow. - `backend/internal/controlplane/packs` validates and applies connector inbound route / handler config components. - `nexuscrm-connectors/application/connectors` wires connector inbound workers and reconciliation. - Boundary tests assert the connector inbound provider remains L0-neutral. That is real substrate. The remaining gap is the broader architectural question: is the connector-scoped route/event model the general inbound callback primitive, or does L0 still need a more universal `_System_InboundEndpoint` surface for arbitrary app callbacks outside connector packs? The connector root queue architecture is governed by [`connector-root-architecture.md`](connector-root-architecture.md): provider admission, immutable materialized event facts, executable work items, scheduler leases, and source/target lane semantics must remain separate facts with one active executor claim path per installed pack generation. - **Verification**: the previous `find backend/internal -name '*inbound*' returns 0` check is obsolete; current code has connector inbound runtime, routing, application wiring, pack components, and tests. - **First-failure example**: a non-connector pack that needs a generic third-party callback endpoint (Stripe payment confirmations, GitHub PR sync, Slack slash commands) and cannot naturally model the source as a connector inbound event. - **Workaround today**: connector packs should use the connector inbound substrate; non-connector callbacks still need a per-pack bridge or must first justify promoting the connector route/event model into the general inbound substrate. - **Promotion criteria**: either document `_System_ConnectorInboundRoute` / `_System_ConnectorInboundHandlerConfig` as the authoritative generic inbound primitive, or introduce a broader `_System_InboundEndpoint` with HTTP path, auth mechanism, idempotency-token field, and dispatch into a generic action handler. The decision should also settle whether outbox and inbound share one event envelope so inbound replays can drive the same handler model as internal events. ### 17.4 Pack-Authored Formula UDFs The formula engine has a `RegisterFunction(name, fn)` API in `backend/pkg/formula/engine.go` — but it is used only by L0-internal code at engine construction (BCRYPT, ISNULLX, ISBLANKX, ISCHANGEDX, SAFE comparisons). There is no `formula_function` component kind and no `_System_FormulaFunction` table in the pack lifecycle, so an L1 pack cannot ship its own UDF declarations. - **Verification**: `grep -E 'ComponentKind[A-Za-z]+\s*=' backend/internal/controlplane/packs/pack_component.go | grep -i 'formula\|function\|udf'` returns 0. - **First-failure example**: a CRM pack declaring `distance_km(lat1, lng1, lat2, lng2)` for territory routing; an ITSM pack declaring `sla_remaining(record)`. - **Workaround today**: every reusable function must be added to L0 and ship in the platform binary. - **Promotion criteria**: new component kind `formula_function` plus `_System_FormulaFunction` table; pack-owned lifecycle parallel to custom-field cross-pack semantics (`OwnerKind = "pack"`, `OwnerPack`); uninstall removes the function from the formula registry and existing references fail closed at next evaluation. ### 17.5 Translation Workbench (Metadata-Level i18n) User-level locale primitives exist — `User.LocaleSidKey` and `User.LanguageLocaleKey` carry the active user's locale — but metadata strings (object labels, field labels, picklist value labels, help text, validation rule error messages) are single-locale. There is no `_System_Translation` table, no `$Label` resolver, no per-locale renderer override. - **Verification**: `grep -E '"tableName":\s*"_System_(Translation|Label|LocalizedField)"' backend/internal/bootstrap/system_tables.json` returns 0. - **First-failure example**: any pack shipping to a non-English locale; a multi-language admin UI. - **Workaround today**: English-only metadata strings. - **Promotion criteria**: `_System_Translation(scope, key, locale, value)` covering object labels, field labels, picklist values, help text, validation error messages; formula `$Label` operator resolves against user locale at evaluation time. ### 17.6 Feature Flag Substrate `_System_Config` exists as a key-value table (`KeyName`, `Value`, `IsSecret`, `Description`) but is unscoped — it has no `Scope`/`OrgId`/`ProfileId`/`UserId` columns to express "this flag is on for these users / this profile / this tenant". No `_System_FeatureFlag` table provides a scoped toggle primitive. - **Verification**: `grep -E '"tableName":\s*"_System_(FeatureFlag|Flag|Toggle)"' backend/internal/bootstrap/system_tables.json` returns 0. - **First-failure example**: a pack with experimental features gated per tenant; gradual rollout of a new flow to 10% of users. - **Workaround today**: unscoped `_System_Config` keys with all-or-nothing semantics, or hardcoded per-deployment configuration. - **Promotion criteria**: `_System_FeatureFlag(name, scope ∈ {org, profile, user}, target, value, expires_at)` queried through the permission engine so the same engine that gates record access also gates feature access. ### 17.7 Backup / Restore Primitive `_System_RecycleBin` provides record-level soft-delete recovery, but there is no `_System_Snapshot` table, no platform-level snapshot, no scoped export API for tenant migration, and no point-in-time restore primitive. The platform relies on underlying TiDB backup infrastructure rather than a substrate-level capability. - **Verification**: `grep -E '"tableName":\s*"_System_(Snapshot|Backup|Export)"' backend/internal/bootstrap/system_tables.json` returns 0. - **First-failure example**: tenant migration to another binary; GDPR data-subject export; cross-environment promotion of admin-authored configuration. - **Workaround today**: external infrastructure backup of the TiDB instance. - **Promotion criteria**: `_System_Snapshot` with scope (tenant / object / record-set), checksum, retention policy; export-by-scope API; verify-restore probe. ### 17.8 Cross-Pack Custom Field Extension A pack cannot declare a custom field against an object owned by another pack. `backend/internal/controlplane/packs/source_backed_custom_field_component.go` rejects the manifest at validate time with `custom_field … requires declaring object_spec in the same pack`, and additionally requires the same-pack `object_spec` to be `locked`-manageable and listed under the custom_field's `dependsOn`. The substrate that an L1 extension pack would need — declaring a field against a peer pack's object with `OwnerKind = "pack"` and `OwnerPack` set to the *declaring* pack while the underlying object remains owned by the *target* pack — does not exist. - **Verification**: `grep -n 'requires declaring object_spec in the same pack' backend/internal/controlplane/packs/source_backed_custom_field_component.go` returns the rejection at line 47. - **First-failure example**: a Jira-like extension pack adding `SprintId` to a base pack's `Issue` object without forking the base pack; any horizontal-extension pattern across two packs that share an object. - **Workaround today**: the consumer pack must fork the target object and ship its own `object_spec`, or the target pack must accept upstream changes and ship the new field itself. - **Promotion criteria**: drop the same-pack constraint in `source_backed_custom_field_component.go`; add a cross-pack declared-dependency check that fails closed when the target object's owning pack is absent or version-incompatible; verify the `_System_Field` row writes with `OwnerPack = ` while the object's `_System_Object` row keeps its original owner; add an integration probe under `tests/integration/l0-completeness/cross-pack-field-extension.sh`. ## 18. Document Conventions This document describes L0 at the architecture and module-design level. It does not enumerate files, line counts, or struct fields; those belong in code. It does not enumerate every transitional debt; those are tracked in the refactor program. It does not adjudicate specific PRs; those have their own decision records under `docs/design/issue-*.md`. When the architecture changes, this document changes with it. When a substrate is added, removed, or has its SSoT moved, the changes appear here. When an L1 pack ships a new component kind, that is documented in the application pack contract, not here.