# NexusCRM Serviceification Target Architecture > **Online configuration boundary (2026-09-07):** > [Configuration and execution lifecycle](configuration-and-execution-lifecycle.md) > supersedes blanket Catalog ownership of editable metadata in this document, > including §§8.10–8.11. Stable requests and durable definitions remain required; > configuration save/activation/readback must not require release publication. > Existing coupling is migration work, not a prerequisite to restore authors. > > **Current-tree reading (2026-09-07):** The runnable monolith and its unused > route composition have been retired. See the follow-up > [legacy connector execution retirement](legacy-connector-execution-retirement.md) > for the surviving recovery boundary. The dated delivery narrative below is > historical, not a current production status. See > [current owners and remaining authoring work](metadata-authoring-current-paths.md). > Capability-specific ownership, including > [Runtime Layout V2](layout-authoring-runtime-domain.md), takes precedence over > the older blanket description of Catalog metadata ownership. > > **Historical status:** Two independent review rounds concluded **Architecture direction: > PASS**. The contract, security, App authority, Catalog snapshot, and first > record-access walking-slice foundations are implemented. Capability work now > proceeds through the D1-D3 test deliveries in Section 11.2. Phase-one > production replacement remains **BLOCKED** until the complete new distribution > passes those executable gates; production then changes owner once in D4, not > through incremental per-route cutovers. D5 deletes the legacy distribution > only after the bounded rollback gate. > > **Authority order:** L0/L1 ownership remains governed by > `docs/design/l0-l1-standard.md` and the binding sequence in > `docs/architecture/repo-split.md`. This document defines service boundaries, > runtime contracts, consistency, and delivery DoD. It cannot override the > ownership standard. A capability ADR linked from this document may own a > more-specific exact contract. In that case this document retains the > cross-capability invariant and routes the exact fact to that ADR; it does not > preserve a competing inline copy. > > **Historical execution alignment (2026-07-15):** C0 selected the real CRM > record-access route; A1c, P7b, R1b, P6/F1-I, and the dormant V1a Runtime/L1 > walking slice are complete. D1 builds the independently deployed test stack > and performs the capability-scoped P7c/S1 ownership flip only inside that new > Runtime composition. D2A/D2B/D2C then complete the synchronous-read, > mutation/worker, and external/async surfaces in test. D3 proves the complete > product without a legacy process. D4 performs the single-org production > replacement during a controlled maintenance window, and D5 retires the legacy > distribution after the rollback gate. The live non-deployment execution > status remains `pingcap-inc/NexusCRM#3555`; this document owns the target > boundaries and binding gates rather than a competing progress counter. --- ## 0. TL;DR NexusCRM targets a generic, tenant-isolated, metadata-driven L0 engine. Multiple L1 Apps run as independent services and consume L0 through coarse neutral APIs. L1 executable code is neither compiled into L0 nor loaded through Go plugins as the primary extension mechanism. The first milestone creates independent binaries, images, configuration, identity, readiness, and deployment units and proves the first real vertical slice in a separate test distribution. Additional capabilities migrate into that test distribution incrementally, but the legacy monolith remains the only production deployment until the complete new product passes D3. D4 stops and drains the legacy deployment before starting the exact test-approved release tuple. The new distribution never calls the legacy deployment as a fallback, and one environment has one active owner for every route and work family. The target boundaries are: 1. **Catalog / Control Plane**: the authority for immutable App releases, installation ownership, namespaces and release publication. Editable configuration and its versions belong to their domain owners. 2. **L0 Runtime**: one logical service with independently deployable `runtime-api` and `runtime-worker` roles that share engine semantics and runtime data ownership. 3. **Integration Service**: the webhook trust boundary, durable receipt owner, and connector delivery owner. 4. **L1 Apps**: every App has an AppPackage; only Apps that need an independent public dialect/UI host or post-commit executable capability have an AppService. `sfcrm` is the first service-backed App. `issue-tracker` is a CI-only non-CRM conformance fixture, not a product App or deployable service. Internal synchronous communication uses protobuf/gRPC. Public and edge protocols use HTTP. All post-commit and asynchronous behavior uses durable outbox/inbox delivery. Runtime hot paths do not call Catalog publication. They use admitted release context and configuration from the relevant domain owners, with request/run consistency independent of release revision. --- ## 1. Governing Requirements ### R1 — Generic L0, Multiple L1 Apps, and Clear L2 Ownership - L0 owns only generic mechanisms for metadata, records, policy, query, automation, storage, and integration. - The same L0 supports CRM and future real non-CRM Apps. The issue-tracker fixture tests that new L0 seams remain App-neutral, but it is not evidence of a second production App. - L1 owns App metadata, public API/wire protocol, UI, dialect, and product behavior. - L2 owns org/tenant overlays, administrative policy, and deployment configuration. ### R2 — Metadata-Driven App Construction A new L1 App is built primarily through AppPackage, metadata, neutral contracts, and registered capabilities. Adding an App must not require concrete object names, status values, Salesforce DTOs, tenant names, or product branches in L0. ### R3 — Independently Deployable in the First Phase Catalog, Runtime roles, Integration, and every executable L1 App have independent build and deployment artifacts from the first milestone. A metadata-only App deploys as an independent AppPackage release and is not forced to own an empty service. Independent deployment does not imply arbitrary version compatibility; every boundary declares its compatibility window. ### R4 — Independent L1 Evolution - A metadata-only App release does not rebuild L0. - An L1 service-code release does not rebuild L0. - Coordinated upgrades are required only when a neutral protocol or L0 mechanism changes outside the supported compatibility window. ### R5 — Performance and Seamless Evolution - The split must meet approved end-to-end latency, throughput, and error-budget SLOs. - RPC counts have hard per-use-case bounds and cannot grow linearly with records, fields, lookups, or related lists. - Upgrade, rollback, metadata apply, schema migration, drain, and recovery have executable protocols. - A control-plane maintenance window is acceptable. Unverified zero-downtime claims cannot replace correctness. ### R6 — One Source of Truth Every fact has one owner, one authoring path, and one runtime read path. Ownership, namespace, schema, and dependency conflicts fail closed at write time. ### R7 — Tenant and Security Isolation Identity, records, metadata, queues, files, caches, secrets, and permissions are isolated by trusted OrgContext. A caller cannot elevate OrgContext or SecurityContext through an ordinary request field or header. ### R8 — Recoverable Cross-Boundary Behavior Every synchronous call and asynchronous work item defines deadlines, idempotency, retry, deduplication, ordering, leases, dead-letter behavior, repair, and observability. ### R9 — Incremental Test Construction Without Permanent Dual Paths Capabilities move into the separate test distribution as complete vertical slices, each with one new-stack read/write path and no fallback to the legacy deployment. Production does not follow those per-capability moves. It remains entirely legacy until D3 proves the complete new product, then changes owner once in D4. Compatibility is a bounded build, acceptance, and rollback state, not a second permanent read or write path. --- ## 2. First-Principle Facts ### F1 — The Engine Transaction Core Cannot Be Cut by a Network Boundary Record mutation, permission enforcement, before-save automation, synchronous rollups, and transaction hooks are one transaction unit. Product behavior that must participate in this unit is expressed as an L0 generic primitive plus metadata. It cannot synchronously call L1. ### F2 — A Network Call Is Not a Function Call Networks add latency, timeouts, duplication, reordering, and ambiguous outcomes. Synchronous RPCs are coarse, bounded, and deadline-aware. Post-commit side effects are made durable before delivery. ### F3 — Process Separation Immediately Creates Distributed State Even one replica per role creates different metadata snapshots, caches, sessions, registries, clocks, lifecycles, and failure states. ### F4 — An Independent Binary Is Not Necessarily an Independent Logical Service Service boundaries follow business/data authority and contracts, not the number of `main` packages. `runtime-api` and `runtime-worker` share engine semantics and runtime state, so they are deployment roles of one L0 Runtime logical service. ### F5 — Independently Deployable Does Not Mean Arbitrarily Composable Services can roll out and roll back independently only within declared protocol, engine-generation, and capability compatibility windows. ### F6 — A Shared Database Becomes an Implicit API When separate logical services read or write each other's internal tables, they bypass contracts, ownership, and rollout gates. The first phase may share a physical TiDB cluster, but logical databases/schemas, credentials, and table ownership are mandatory. ### F7 — Metadata Is Executable System State Flow, permission, schema, layout, and lifecycle metadata change runtime behavior. TTL-based or best-effort cache invalidation is insufficient. Metadata needs domain-owned commit/activation and version-binding semantics. These do not require Catalog publication; a stable request or immutable run definition must not turn an online edit into a release operation. ### F8 — Security and Tenant Context Cannot Trust Caller Declarations OrgContext, principal, workload identity, and SecurityContext originate from an authenticated chain and operator-controlled routing. An org ID in a payload is only a projection of signed context, never the authority. ### F9 — Performance Is Proven Only with the Same Workload Service separation is not evidence of speed or isolation. End-to-end p99/p999, throughput, database contention, and queue lag must be compared under identical data, concurrency, request distribution, and environment. --- ## 3. Explicit Design Choices 1. **L1 executable code is an optional independent service.** AppPackage is mandatory. AppService exists only when executable code exists. Go plugins, dynamic linking, and compiling dialect code into L0 are not primary paths. 2. **Declarative extension is metadata.** Publisher-authored schema, policy, Flow, Layout and action defaults enter through AppPackage installation. Subsequent online edits use the domain owner, not AppPackage republication. 3. **Transaction-critical behavior is an L0 primitive.** L1 provides only post-commit or asynchronous executable behavior. 4. **Internal sync uses protobuf/gRPC.** Public/browser/webhook/file surfaces use HTTP. Asynchronous behavior uses durable outbox/inbox. 5. **Catalog publishes release projections.** Runtime and L1 never query Catalog implementation tables. Configuration reads use their domain owner; derived snapshots/caches cannot create another authority. 6. **One route or work family has one owner.** Migration uses gateway routing and worker ownership fencing, not a general L1-to-legacy semantic proxy. 7. **Per-org deployment first, pooled deployment later.** Contracts, keys, and work envelopes use trusted OrgContext from day one so pooling does not change the identity shape. 8. **No broker on day one.** Service-owned outbox/inbox plus idempotent delivery comes first. Kafka, NATS, or another transport can replace delivery later without changing work envelopes or delivery semantics. --- ## 4. Target Architecture ```mermaid flowchart LR Client["Clients / Browser"] --> Gateway["Generic Edge Router"] Gateway --> SF["L1 sfcrm service"] SF -->|"coarse gRPC + delegated identity"| API["L0 Runtime API + Identity capability"] Fixture["CI-only non-CRM conformance fixture"] -->|"test-only coarse gRPC"| API Admin["Configuration administrator"] -->|"Product UI / API"| SF Deploy["App install / deployment operator"] -->|"Release job API"| Catalog["Catalog / Control Plane"] Hook["External Webhook"] -->|"HTTP"| Integration["Integration Service"] API --> RuntimeDB[("Runtime DB")] Worker["L0 Runtime Worker role"] --> RuntimeDB Catalog --> CatalogDB[("Catalog + Snapshots")] Catalog -->|"Install / migration preparation"| Schema["L0 Schema preparation owner"] API -->|"Online schema command"| Schema Schema -->|"Fenced DDL"| RuntimeDB Integration --> IntegrationDB[("Receipts / Delivery State")] Integration -->|"idempotent command delivery"| API ``` ### 4.1 Catalog / Control Plane Catalog owns: - App package/release identity, install, upgrade and removal; - released component ownership/dependencies and namespace reservation; - release install/migration journals and their desired deployment state; - immutable release publication artifacts and the active release-selection pointer; - engine/App capability and installation admission. Catalog is the authority for desired and published **release selections**, not for administrator configuration versions. Installation/migration delegates physical preparation to the L0 Schema owner. An online schema edit uses that owner without a Catalog release job. See the [configuration lifecycle boundary](configuration-and-execution-lifecycle.md). Catalog `ACTIVE` establishes control-plane release activation. Runtime `ServingState` separately establishes replica readiness against the admitted release, security lease, physical preparation and engine compatibility. These are different facts, not competing serving authorities. A configuration commit or activation does not advance either release identity as its visibility token. Publication availability is not a prerequisite for an already-admitted app's online configuration save/readback. Configuration, security or schema owners can still fail closed when their own required evidence is unavailable. Keep security leases and workload admission; do not interpret this independence as permission to bypass them. Release cold-start preparation remains separately subject to the deployment contract. #### 4.1.1 Consumer-Specific Snapshot Projections Catalog release publications have an org-scoped CatalogRevision and verifiable content-digest lineage. Their projections disclose only release/installation facts admitted for the requesting consumer. Domain configuration versions are independent; there is no single CatalogRevision for all editable org facts. Current implementation carriers include RuntimeMetadataView, AppMetadataView and IntegrationBindingView. Several still embed editable definitions or validate current schema through Catalog. Those fields/readers are the migration inventory in the [configuration boundary](configuration-and-execution-lifecycle.md), not an extension of Catalog's target ownership. Release-owned templates and capability declarations may remain in release projections; current customized values must come from the domain owner with one effective read path. A projection or cache cannot become an authoring authority. Preserve workload, installation and tenant scoping when migrating a fact. Published wire formats and old durable references require an explicit compatible transition; this architecture correction does not claim that transition is already implemented. ### 4.2 L0 Runtime Logical Service Runtime owns: - domain-owned online configuration authoring/read/activation and its versions; - runtime records and L3 data; - query and search execution; - permission and RLS enforcement; - formula and flow interpretation; - mutation lifecycle and transaction boundaries; - runtime outbox and inbox; - asynchronous work lifecycle; - identity/session authority for JWT/session validation, revocation, Connected App state, scopes/audience, and user-to-service delegation. Deployment roles: - `runtime-api`: foreground request and transaction path; - `runtime-worker`: outbox, after-save, scheduler, bulk, asynchronous rollup, approvals, and related background work. The roles scale, restart, and deploy independently but share EngineGeneration compatibility. A worker-only change may ship independently. Interpreter-semantic changes require joint API/worker verification, and incompatible engine generations cannot run together indefinitely. Identity is a first-class capability inside the Runtime logical service in the first phase; it is not another process. All L1 and Gateway requests use the same Runtime Identity issuer and authoritative interceptor. Ordinary public API credentials enter through the existing `IdentityContext.Exchange`. The [independent L1 MCP Worker](https://github.com/pingcap-inc/nexuscrm-app-packs/blob/49e9998af867edfc3f5715eee2bfc8e8e0b8c2f7/docs/design/mcp-l1-worker.md) reuses that published authority; the earlier MCP `ExchangeResource`/resource-profile prerequisite is superseded. L1 does not create a parallel session/revocation read path and cannot construct trusted context from user-provided fields. ### 4.3 Integration Service Integration owns: - the webhook HTTP trust boundary; - provider signature verification; - durable receipts; - connector delivery state, retry, and dead-letter handling; - idempotent delivery into the Runtime command inbox. Integration does not write Runtime tables. It persists a receipt before calling Runtime `AcceptCommand`. When Runtime is unavailable, Integration retries from its own delivery state. ### 4.4 Optional L1 App Service An L1 AppService owns: - the App public API and wire protocol; - App-specific errors, envelopes, and projection; - dialect parsing and compilation; - UI, frontend, and static assets; - product orchestration outside L0 transactions; - AppPackage source; - post-commit idempotent action handlers when required. A metadata-only App does not need an AppService. A service is deployed only for a public dialect/UI host or post-commit executable capability. L1 does not access platform databases, mutate Catalog, register in-transaction callbacks, or declare an elevated SecurityContext. #### 4.4.1 Authorization Locality and Coarse Use-Case Operations Caller-effective authorization is an L0 Runtime decision. An AppService may validate public wire syntax, bound request size and item count, and map neutral Runtime outcomes into App-specific HTTP status and error envelopes. It does not compute authoritative access from exported permission rows, cache a parallel caller-permission truth, or decide that a caller may execute an L0 operation. This preserves the logical ownership of the in-process system after process separation: ```text public App request -> L1 wire parsing and bounded validation -> one coarse Runtime use-case operation -> L0 identity, Catalog binding, authorization, and execution -> L1 public-wire projection ``` Moving an in-process permission interface across gRPC method by method is not a valid service boundary. In particular, a complete effective-permission export must not become the general authorization primitive for actions, record UI, describe, navigation, or mutation guards. A request that needs a few system, custom, object, or field decisions cannot load and serialize every caller- effective object, field, App, navigation, and record-type permission in the App installation. The approved use-case shapes are: | Public use case | L0 operation | Caller-effective authority returned to L1 | |---|---|---| | Execute one Catalog action | `ExecuteCatalogAction` | Final neutral result or bounded failure only | | Execute a bounded action batch | one bounded batch form of `ExecuteCatalogAction` | Ordered per-item results and receipts; no per-item RPC loop | | List actions for a target | `ListEffectiveCatalogActions` | Only caller-visible, invocable neutral action descriptors and guard verdicts | | Preview an action | `PreviewCatalogAction` | One authorized, no-effect neutral preview | | Describe one object | object-scoped describe operation | Only that object's authorized object, field, relationship, and record-type facts | | Compose record UI | record-UI/read-graph operation | One revision-bound record, layout, access, and display projection | | List Apps/navigation | App/navigation projection | Only effective App and navigation facts | | Inspect or administer permissions | `ReadEffectivePermissions` | The complete bounded permission export | `ReadEffectivePermissions` remains a valid explicit inspection and administration capability. It is not deleted or weakened. Its complete response shape is precisely why ordinary hot paths must not reuse it. ##### Catalog action execution For execution, L1 sends the public action identity, canonical bounded inputs, and idempotency identity. Runtime binds the request to the authenticated `AppInstallationID`, selects the action from one immutable AppMetadata view, derives the exact Flow closure, and owns every authoritative permission check. The current AppMetadata view is a locally served Catalog projection. Installed action declarations remain release facts; selection of an administrator-edited Flow definition must move to its domain owner under the configuration boundary. The resulting execution closure binds the exact domain definition versions and admitted handler identity. The hot path does not call Catalog publication. The target contract does not require L1 to prefetch a Catalog component ID or caller permission snapshot before execution. Component identity and exact definition/version are Runtime-derived facts and appear in the execution receipts. A bounded public batch remains one use case and therefore one L1-to-Runtime RPC. Runtime binds every item to the same request Catalog view and SecurityEpoch, preserves request order, accepts an independent idempotency key per item, and returns the existing per-item success/failure semantics. Batch transport does not turn separate atomic Flow executions into one database transaction unless the public contract explicitly requests and Runtime supports that stronger semantic. The Runtime authorization plan is action-specific: 1. canonicalize and validate the bounded input; 2. derive the required system/custom permission IDs and conditional object operations from the immutable action declaration; 3. evaluate that bounded fact set in one L0 permission-engine operation; 4. prepare the exact action/Flow closure with domain definition versions, admitted release context, SchemaGeneration, EngineGeneration, SecurityEpoch and permission floor; 5. inside the owning transaction, re-evaluate only the same security facts and fence the permission floor/SecurityEpoch before committing effects; 6. write the invocation receipt in the same transaction. Transaction revalidation is mandatory. Reconstructing the complete effective- permission export inside the transaction is not. The narrow evaluator must be batched; replacing one broad projection with one SQL/RPC call per guard is an N+1 regression and is rejected. Records, Query, Files, Approval, and other engines continue to enforce their own row, field, sharing, and mutation invariants inside the operation they own. An action-level verdict does not bypass or duplicate those engine checks. ##### Discovery, concealment, and public projection Action discovery is advisory and revision-receipted. Execute and Preview always authorize again, so a permission change between list and invocation fails closed. Runtime returns neutral bounded failure categories that allow L1 to preserve the public dialect's concealment contract without learning or owning the underlying permission state. For example, an App may intentionally project both an absent action and a caller-hidden action as the same public 404 while Runtime logs the internal reason with the request identity. L1 may cache immutable release presentation by release identity; editable presentation must use its domain version. It must not cache caller-effective authorization independently of Runtime. Reused action plans must bind every authoritative input, including domain definition versions and admitted handler identity; CatalogRevision alone cannot invalidate edited definitions. Runtime may use a SecurityEpoch-bound authorization optimization only if cancellation, tenant isolation, and permission-floor correctness are proven. Cross-request caching is not the first repair for an incorrectly broad use-case contract. ##### Migration and proof Migration is additive and does not require a new service or network layer: 1. stop the action execute path from reading the complete permission export; 2. add the bounded internal action-authority evaluator and transaction variant; 3. add an ActionRef-based request profile on the existing Runtime endpoint so Runtime, not L1, resolves the AppMetadata component; 4. replace per-item action RPC loops with the bounded batch profile while preserving idempotency, order, partial results, and timeout behavior; 5. move list and preview to their coarse effective operations; 6. replace complete permission exports in describe, record UI, and App navigation with their object/use-case-scoped projections; 7. remove the obsolete L1 permission-derived authorization code after the compatibility window. Required proof includes: - an unauthorized action is concealed exactly as its public App contract requires while Runtime records the correct internal denial; - permission or SecurityEpoch change between prepare and commit fails closed; - one action execution performs no complete effective-permission projection; - a maximum-size action batch uses one L1-to-Runtime action RPC and preserves per-item receipts and partial-result semantics; - action list, execute, and preview bind to one immutable view per request and reject mixed receipts; - object describe, record UI, and App navigation never load permissions outside their authorized use-case projection; - explicit permission inspection still returns the complete, revision-bound response; - RPC and repository-query counts remain fixed for each use case and do not grow with unrelated objects, fields, Apps, navigation items, or record types. ### 4.5 Generic Edge Router The edge owns only TLS, service routing, coarse admission, and the active static route configuration. It owns no Salesforce or App semantics. In phase one the test environment routes only to the new distribution while production remains entirely on the legacy ingress. At D4 the legacy ingress is stopped and drained before the exact test-approved Gateway configuration becomes public. The new Gateway never proxies an unsupported path to the legacy deployment. The Gateway is an independently deployable generic transport component owned and released by `nexuscrm-api-host` at `cmd/gateway`. It is not part of an L1 release and cannot import or encode App-specific semantics. --- ## 5. Data and Credential Ownership The first phase may share a physical TiDB cluster, but logical ownership exists from the first step: | Logical store | Owner | Access by others | |---|---|---| | Release Catalog, release operation journal, release projections | Catalog service | Consumers use the admitted release contract, never table access | | Domain configuration, records, Runtime outbox/inbox | Respective L0 Runtime domain | Other services use domain contracts; no competing configuration writes | | Physical schema preparation | L0 Schema/Persistence owner | Install/migration and online authors use the same fenced owner; no record DML | | Integration receipts and delivery attempts | Integration service | Runtime does not read directly; it receives idempotent commands | | L1 local state | Corresponding L1 | Never a second source of truth for platform records or metadata | Target credential shape: - Catalog: release-catalog DML and authorized install/migration preparation commands; no runtime-record DML or online configuration authority. - Runtime API/worker: domain-owned runtime DML, no Catalog writes or unrestricted DDL. Only the designated Schema preparation owner receives fenced DDL access; command identity, authorization and recovery are part of its contract. - Existing deployment DDL credentials require an explicit owner/credential migration; this target policy does not silently grant ordinary Runtime roles new database privileges. - Integration: integration-store DML; no catalog or runtime-table writes. - L1: no platform database credentials. Before credential cutover, real restricted users must prove that TiDB logical database/schema grants cover future dynamic record tables. If not, the physical schema design changes; the system does not fall back to multiple writers. ### 5.1 Control-Fact Authorities | Control fact | Single owner and authoring path | Runtime read path | |---|---|---| | Phase-one ingress release tuple and static route configuration | L2 deployment/release authority consumes AppPackage declarations and reviewed service artifacts | Gateway reads only the exact immutable configuration selected by the active release tuple; there is no per-route production `RouteEpoch` in phase one | | Minimum SecurityEpoch and SecurityLease issuance | Runtime Identity/Security authority allocates and writes it; Catalog activation may request an advance only through a fenced command | Runtime authoritative interceptor and lease validator | | Runtime ServingState | Runtime logical service; admitted only after preparation, compatibility, and lease checks | Gateway/readiness admits traffic only to replicas serving the required revision and epochs | | Serving Runtime replica membership and prepared revision | Runtime logical-service lease registry in Runtime DB | Catalog waits and queries only through the Runtime preparation API | | AppService handler/service binding | Catalog HandlerBinding authority selects trusted workload identity and exact ServiceRelease digest; deployment authority resolves endpoints | Runtime reads active CatalogSnapshot/HandlerBinding; App metadata never supplies endpoints | | Session, revocation, and Connected App tables | Runtime Identity authority | Single authoritative auth interceptor | | Work-family owner and OwnershipEpoch | Runtime logical service, through a fenced Runtime DB record | Claimer/renew/completion conditional updates | These control facts cannot degrade into parallel authorities spread across operator config files, environment variables, or independent service caches. --- ## 6. Communication Contracts ### 6.1 Protobuf/gRPC for Internal Synchronous Calls The first version defines use-case families rather than mirroring the current 33 Go interfaces: - `IdentityContext.Exchange` - `CatalogSnapshot.Get/Watch`, returning only the caller-authorized projection - `Runtime.ReadSelectedRecord` - `Runtime.Query` - `Runtime.ReadGraph` - `Runtime.Mutate` - `Runtime.ExecuteFlow` for an explicit external synchronous entry point; mutation hooks remain inline - `Runtime.AcceptCommand` - `Runtime.SubscribeEvents` with durable cursor and replay - `Catalog.Apply/GetJob/GetActiveRevision` Every RPC carries: - authenticated OrgContext, workload identity, and AppInstallationID; - a deadline; - an idempotency key for mutation and command operations; - protocol and capability metadata; - bounded request and response sizes; - documented retry classification. `IdentityContext.Exchange` is owned only by Runtime Identity. It does not accept a caller-selected principal. The same authority validates the presented user's session, revocation and credential validity, derives the applicable Runtime API grant, then issues the signed short-lived delegation specified by [`runtime-trusted-delegation-contract.md` Section 4](runtime-trusted-delegation-contract.md#4-delegation-facts). That section is the sole delegation-field inventory. Connected App checks apply when the existing credential is bound to one; they do not require a native user to register a Connected App for MCP. MCP adds no separate business permission, scope or resource profile. Data RPCs enforce the user's current permissions. Every Runtime RPC checks the intersection of workload identity, App installation, user delegation, and requested capability. Any missing element fails closed. The first approved and implemented neutral Runtime record capability is the selected-root, single-record read specified by [`runtime-selected-record-read-contract.md`](runtime-selected-record-read-contract.md). It is one complete Runtime RPC after delegation: Runtime binds one metadata and security view and performs CRUD, RLS, FLS, and root projection internally. Lookup labels, relationships, graphs, and partial results are excluded from v1. P2 IDL and generated-code work are complete and consumed through the separately versioned `nexuscrm-contract/wire-go` module. That completion validates the neutral wire boundary. The Runtime adapter, dormant sfcrm client, and CI-only issue-tracker conformance fixture are also complete. None of those facts selects a production CRM route: the current-tree C0 audit found no behaviorally equivalent CRM consumer. The first real CRM walking slice in the new distribution is the UI API record-access operation selected by [`c0-first-real-crm-consumer.md`](../refactor/serviceification/c0-first-real-crm-consumer.md): optional object-type resolution plus a bounded batch of caller-effective Read, Update, Delete, and Locked decisions. This is a different coarse Runtime use case, not an extension or composition of selected-root. Its approved neutral semantic/wire contract, non-CRM CI conformance consumer, and request-bound serving view prove the D1 path without moving production traffic. The existing selected-root fixture proves only selected-root; it does not transfer proof to this seam. Production route ownership changes only with the complete D4 release tuple after D2 and D3 close the remaining product surface. ### 6.2 HTTP for Public and Edge Protocols - Salesforce and public App APIs; - browser and frontend traffic; - OAuth and public identity exchange; - webhooks; - file upload/download; - an optional Catalog REST gateway. MCP runs as an **independent L1 Worker**, under the [current L1 product design](https://github.com/pingcap-inc/nexuscrm-app-packs/blob/49e9998af867edfc3f5715eee2bfc8e8e0b8c2f7/docs/design/mcp-l1-worker.md). It owns MCP protocol, tools and lifecycle, and calls only published Runtime CRM APIs. It does not depend on SFCRM AppService or a connector service. The old embedded-process recovery amendment is superseded. Runtime retains identity and data authorization; user permissions apply equally to MCP reads and writes. Workload registration, artifact publication and deployment still use their existing authorities; process health or a protocol test is not user-permission or release acceptance. ### 6.3 Durable Outbox/Inbox for Asynchronous Work A service persists an outbox item or receipt in its own transaction before delivering it through idempotent gRPC to the target inbox. A successful response means the target durably accepted the message, not that business execution completed. The formal state model is: ```text MessageKey = source-service + org + message-id [+ app-installation when installation changes message semantics] Producer: PENDING -> LEASED -> DELIVERED | RETRYABLE | DEAD Consumer: ACCEPTED -> EXECUTING -> SUCCEEDED | RETRYABLE | OUTCOME_UNKNOWN | DEAD ``` Each message family defines: - producer outbox commit boundary; - consumer inbox insert and dedupe boundary; - atomic boundary between inbox, business DML, and completion ledger; - safe retry after lost acknowledgement; - payload schema version; - ordering partition key; - dedupe and handler retention; - poison/dead-letter/repair behavior; - uncertain external outcome handling. Runtime API-to-worker work belongs to one logical service and should use a transactional queue inside Runtime DB. It does not add a cross-service delivery hop merely to resemble microservices. Forbidden behavior: - fire-and-forget as a business-delivery guarantee; - RPC to another logical service inside an active mutation transaction; - automatic mutation retry without an idempotency key; - arbitrary endpoint URLs declared by metadata. Operator-controlled deployment bindings resolve service endpoints by trusted workload identity. AppPackage declares only capabilities and handler references. --- ## 7. Metadata-Driven App Contract `AppPackage` is mandatory for every App. `AppService` is an optional executable artifact. A metadata-only App adds no process or service failure domain. ### 7.1 AppManifest The normative v1 identity, canonicalization, digest, publisher, dependency, namespace, capability, and compatibility rules live in the [App authority contract](app-authority-contract.md). AppManifest is a closed, publisher-authored behavior contract. It contains `AppID`, `PublisherID`, a digest-covered semantic version, the MetadataBundle digest, App dependencies, owned namespaces, protocol/engine requirements, provided capabilities, and frontend compatibility requirements. `AppRelease` is `(AppID, AppReleaseDigest)`, where the digest covers the canonical AppManifest and transitively every L1 metadata behavior byte. The digest is not a field inside the manifest. Exact service and frontend artifact digests, endpoints, installation IDs, and L2 revisions are separate bindings. ### 7.2 MetadataBundle The first version covers: - objects, fields, and relationships; - permissions and PolicySchemaBindings; - layouts, navigation, and UI metadata; - flow, validation, and formula declarations; - lifecycle profiles; - action declarations; - search and query configuration; - integration declarations. Unknown component kinds, missing required capabilities, and missing security bindings fail closed. MetadataBundle alone owns component identity, component dependencies, override/ removal policy, and content hashes. It cannot repeat App identity, semantic version, App dependencies, namespaces, protocol/engine compatibility, or provided capabilities. Cross-App component references use an AppManifest dependency alias and cannot author another version range. The closed policy vocabulary and its exact semantics are owned by the [App authority contract](app-authority-contract.md); this architecture summary does not define a second policy shape. ### 7.3 OwnershipContract Every component records: ```text BusinessKey DefinitionVersion OverridePolicy RemovalPolicy ``` The containing AppPackage supplies the one `OwnerAppID`; the component does not repeat it. This makes ownership `AppID + ComponentID` without a second owner field that can disagree. Rules: - ownership is write-once; another App cannot silently take over; - cross-App references declare dependencies; - an App extends another only through explicit extension points; - uninstall validates reverse dependencies; - downstream overrides require separate component-kind authorization and cannot change the owner; `OverridePolicy` only controls replacement of an already authorized override during release replacement. ### 7.4 AppService Contract When present, an L1 executable service may: - expose the public App API and UI; - call neutral L0 Runtime APIs; - consume neutral durable events; - provide post-commit idempotent action handlers. It may not: - directly read or write Catalog or Runtime databases; - register arbitrary in-transaction callbacks; - pass opaque engine handles; - elevate OrgContext or SecurityContext through headers or payloads; - put vendor DTOs in the neutral L0 contract. ### 7.5 Non-CRM Proof Before a new L0 seam enters the new distribution, C0-style evidence must identify an exact current-tree product consumer and prove the required behavior and side effects. A separate non-CRM conformance consumer must exercise the same neutral seam in CI. Selected-root currently has a dormant sfcrm client adapter and the CI-only issue-tracker fixture; neither fact proves that a production CRM route is selected-root-equivalent. The fixture proves only that the seam does not encode CRM semantics. It is not a product App, AppService, production deployment, or evidence that NexusCRM already supports a second real App in production. That product claim requires an independently owned and deployed real L1 App; a planned future App is not evidence. Conformance proof is capability-specific. The selected-root fixture does not prove a later access-evaluation, mutation, query, worker, or Integration seam. Each such seam must add its own public-neutral non-CRM vectors before production replacement can be approved. Reusing the CI-only issue-tracker vectors is allowed; adding a product route, service, image, installation, or deployment to that fixture is not. P1x removed the older release-shaped issue-tracker pack, dialect, distribution, frontend, install, deploy, and publication paths. Only the sealed CI fixture and its reviewed NexusCRM snapshot remain. Regression guards make this classification physical: no production command, image, distribution, route, manifest, or dependency closure may acquire the fixture. ### 7.6 AppInstallation Binding `AppID` identifies a product. `AppInstallationID` identifies a controlled installation of an AppRelease in one org. The exact lifecycle, projection, and conflict rules are fixed by the [App authority contract](app-authority-contract.md). Catalog binds: ```text OrgID + AppInstallationID -> AppID + active AppReleaseDigest -> exact resolved dependency AppReleaseDigests -> AuthorizedWorkloadIdentities -> GrantedCapabilities -> effective OwnedNamespaces -> HandlerBindings + frontend binding ``` Catalog is the only AppInstallation author. Runtime owns only a fenced serving projection; delegation and deployment lockfiles are carriers or desired state, not authority. Operator deployment bindings resolve endpoints by canonical workload and exact ServiceRelease digest. Runtime authorization always validates `OrgID + AppInstallationID`, but AppInstallation is an authorization and audience dimension, not the identity of every shared fact. Keys follow fact ownership: - shared record: `OrgID + ObjectID + RecordID`; - org-global configuration: org plus domain-owned resource identity/version; - release publication: `OrgID + CatalogRevision`; - installation-owned state: `OrgID + AppInstallationID + LocalKey`; - request/work/idempotency: includes AppInstallationID when caller or handler installation changes semantics. This prevents a compromised L1 from crossing its authorization boundary without duplicating an org's shared records per App installation. ### 7.7 Interim Capability Exceptions (ADR-Gated) The AppService target (§7.4) assumes the Runtime serving substrate a capability needs is production-live. When a capability's substrate prerequisites are not yet landed and the business need cannot wait for them, a capability ADR linked from this document may record a **time-boxed interim exception** allowing that one capability to serve in-process in the legacy distribution, under all of the following binding conditions: - **Existing seams only.** The interim path may use only already-shipping product seams (for example the legacy distribution's product route surfaces) and the neutral engine contract ports. It may not add an L0 seam, scope-floor tables, or any direct Catalog/Runtime database access — the §7.4 "may not" list applies to the interim path in full. - **Default-off activation.** The interim path must be gated behind an explicit operator opt-in and mount nothing when unset, so a default build carries no second serving path. - **Recorded exit condition.** The owning ADR must name the exit condition that ends the exception: the substrate prerequisites the successor needs, **and** the successor AppService being admitted and accepted as the capability's production owner — prerequisites landing alone never authorizes removing the interim path (that would orphan live traffic). The interim path must be deletable in one commit once the successor serves (shared grammar/mapping code lives in its L1 library, so cutover to the AppService form is a re-wire, not a rewrite). - **Fixed expiry, named owner, self-disable.** The ADR must record a fixed calendar expiry date (at most two quarters from the amendment merging) and a named accountable owner. Renewal happens only through a new reviewed docs amendment merged before expiry. The implementation must enforce the expiry itself: past the recorded date the interim path refuses to mount even with its activation flag set, so an unrenewed exception disables itself rather than lingering. An expired, unrenewed exception is not part of this contract. - **No gate distortion.** The exception must not add requirements to, or block, the D-gate migration sequence (§11.2); it is retired by that sequence, never the reverse. An exception under this section is part of the trusted contract only once the amendment recording it has merged on `main` through the required review gate as a documentation change, distinct from the implementation change it authorizes. The current recorded exceptions are: - **SCIM identity sync (push-only receiver)** — [scim-l1-rebuild.md](scim-l1-rebuild.md) §6.1: interim in-process **push-only** receiver in the CRM distribution. Exit condition: a SCIM AppService successor — enabled by the Runtime serving path being production-deployed and the run-as and AppInstallation admission substrate landing (scim-l1-rebuild.md §5 A0, C, E) — is admitted and takes production ownership of SCIM provisioning. A reconcile worker is not part of the push-only form and not covered by this exception. --- ## 8. Metadata, DDL, and Work Consistency A single `MetadataVersion` cannot represent every fact. The first version uses the following axes. ### 8.1 ProtocolVersion Protobuf/gRPC wire compatibility. At least N/N-1 is supported. Breaking changes create a new major/service instead of changing existing field semantics. ### 8.2 EngineGeneration L0 interpreter semantics. Runtime API and worker report generation and capabilities. Catalog cannot activate an AppRelease whose required engine capabilities are unavailable. ### 8.3 AppRelease An immutable `(AppID, AppReleaseDigest)` used for dependencies, upgrade, rollback, and ownership provenance. Canonical semver is a digest-covered selection label and has one immutable mapping to a digest; it is not a second release identity. ServiceRelease, FrontendRelease and Catalog publication are independent release/deployment axes. Online configuration versions belong to their domain owners and do not require a new AppRelease or CatalogRevision. ### 8.4 CatalogRevision A monotonically increasing revision of Catalog release publication per org. It is not the version of all org metadata or an online-save completion token. Requests retain stable domain configuration/definition bindings independently of the admitted release context; see §8.8 and the [configuration lifecycle boundary](configuration-and-execution-lifecycle.md). ### 8.5 SchemaGeneration The completed generation of physical runtime schema preparation, proved by its Schema owner. Both release activation and online schema adoption must reject unprepared physical requirements. This axis is not automatically incremented for every configuration edit; the Schema contract defines its meaning. Online DDL recovery/preparation must not require Catalog publication. ### 8.6 DefinitionVersion Flows, validations, lifecycle definitions, and other executable definitions are immutable. Activation pointers can change, but queued work uses exact definition references. The definition owner selects active versions; activation of an online definition does not require a Catalog revision. ### 8.7 Security-Critical Changes Permission bindings, session revocation, Connected App disablement, credential revocation, and secret revocation do not depend on ordinary metadata TTLs. - Runtime Identity provides the sole authoritative read/epoch for session, user, and Connected App revocation. - Permission-policy snapshot activation uses SecurityEpoch fencing. - A serving Runtime holds a short-lived, verifiable `SecurityLease{OrgID, SecurityEpoch, ExpiresAt}`. - A Runtime partitioned from the revision authority cannot prove that no newer revision exists. It fails closed when the lease expires. - Security apply succeeds only after publication, authoritative minimum-epoch advancement, and serving-replica acknowledgement/renewal at the new epoch. - Secret values never enter ordinary CatalogSnapshot; snapshots contain only controlled secret references. `MaxSecurityStaleness` must be an explicitly approved security SLO because it is a correctness/availability tradeoff; it cannot be hidden behind best-effort Watch. P6-D owns the numeric decision, and D4 production replacement remains blocked until that decision is approved. ### 8.8 Synchronous Request Binding The request entry point creates immutable execution context. P6 owns the contents of `TrustedCall`; this target architecture does not repeat its delegation-field inventory: ```text TrustedCall (validated delegation, current principal/session, and SecurityLease) Admitted release context (CatalogRevision where required by the release protocol) Domain configuration / exact definition version bindings SchemaGeneration EngineGeneration RequestID / Deadline ``` ### 8.9 Asynchronous Work Envelope ```text WorkID TrustedOrgContext AppID / AppInstallationID / AppReleaseDigest ExactDefinitionRefs[] MinSchemaGeneration RequiredEngineGeneration / RequiredEngineCapabilities[] PayloadSchemaVersion HandlerBindingID / HandlerRef / ServiceReleaseDigest / AppReleaseDigest WorkFamily / OwnershipEpoch CatalogRevisionAtEnqueue (audit only) ActorReference / SecurityMode IdempotencyKey Payload ``` Execution semantics: - flow and automation use exact definition versions; - consumers reach at least `MinSchemaGeneration`; - permission, secret, and revocation use current authoritative state; - claim validates payload schema, engine capability, handler binding, and ownership epoch; incompatible work waits for a compatible worker/handler, follows an explicit drain/block rule, or enters a defined repair state; - old L1 handlers and AppReleases remain available until pending work drains or completes an explicit migration; - CatalogRevisionAtEnqueue is audit/diagnostic data, not an automatic execution snapshot; - old definitions are retained until no work references them and retention gates pass. ### 8.10 RuntimeMetadataView Serving Activation Runtime must expose one effective configuration read path per fact, with the request/run lifetime promised by its domain. Catalog publication supplies release admission context; it does not select current UI configuration. The deployed RuntimeMetadataView still combines some configuration with release lineage. Treat that as a migration boundary, not the target authoring protocol. For each migrated domain: 1. enumerate API, worker, compiler, cache and durable-definition consumers; 2. select the domain's committed version and bind a stable execution view; 3. replace the Catalog content authority across that complete consumer closure; 4. remove the replaced read path; never add live fallback to Catalog content; 5. prove online save/activation/readback with publication disabled, alongside security, schema readiness, restart, concurrency and old-work retention. Existing artifacts, wire receipts and old work remain subject to their explicit compatibility/retention contract until that cutover. The earlier P7 snapshot migration is historical implementation evidence, not a rule that future configuration must be published through Catalog. ### 8.11 Publication and Activation State Machine This state machine describes release installation/publication, not online configuration authoring. Its initial release schema operations are additive: ```text DRAFT -> VALIDATED -> RESERVED (org-scoped writer lease + fencing token) -> DDL_RECONCILED -> CATALOG_COMMITTED -> SNAPSHOT_PUBLISHED (content-addressed + checksum) -> RUNTIME_PREPARED -> ACTIVE (CAS) ``` Every step is journaled and idempotent. Rules: - only the current fencing token advances apply state for an org; - DDL carries an operation ID and desired/actual schema fingerprints; after a timeout or ambiguous result, observe and reconcile before retrying; - DDL must reconcile before ACTIVE; - a release-publication job cannot report completion before its artifacts are published; online configuration completion is defined by the domain owner; - the active pointer advances monotonically through CAS and is Catalog's sole published-revision truth; - Runtime replicas preload at `RUNTIME_PREPARED` and acknowledge capability, schema, and security compatibility; - Runtime then admits the revision into `ServingState`; only `ServingState` is authoritative for whether a data-plane replica may receive traffic; - apply returns `CatalogRevision + SchemaGeneration + SecurityEpoch`; - release-publication read-your-write requests carry `MinCatalogRevision` (not online configuration save receipts); an old Runtime waits, forwards to a caught-up replica, or fails closed; - rollback creates a newer revision that points to compatible content; it never decrements generation or active pointers; - apply resumes from the journal after a crash; - every state and crash point has a recovery matrix and executable test; - rename, drop, type narrowing, and ownership transfer are unsupported in phase one and later use expand/migrate/contract. --- ## 9. Tenant Model ### First Phase - Phase one runs exactly one org per deployment, with deployment-specific database resources. - Protocol, snapshot, work, event, repository, cache, idempotency, and ownership keys are classified as org-global, shared-record, or installation-owned. Every access first validates TrustedOrgContext and AppInstallation authorization. Only installation-owned or semantics-dependent facts include AppInstallationID in identity keys. - OrgContext comes from authenticated session, workload identity, or operator routing, never an ordinary caller field. ### Future Pooled Deployment One service instance may serve multiple orgs without changing App, Runtime, or Catalog contracts. Tenant routing, resource quotas, and pooled storage are added without changing the established key and identity shape. ### Required from Day One - cross-org record queries fail closed; - cache keys cannot collide across orgs; - workers cannot claim or execute across orgs; - Catalog revisions, namespaces, and ownership remain org-scoped; - an L1 service cannot forge another org; - one App workload cannot access an unauthorized AppInstallation in the same org; - Integration receipts and secret references cannot cross orgs. --- ## 10. Measurable Definition of Done ### 10.1 Architecture DoD - Every deployment unit has an independent binary, image, configuration schema, workload identity, readiness, metrics, and version endpoint. - L0 does not import L1; L1 depends only on generated neutral contracts. - The neutral semantic/domain contract remains stdlib-only. Protobuf/gRPC generated bindings live in a separate wire module and adapters map them to neutral IR. - Logical services do not import each other's implementation packages. - Runtime API and worker share only the L0 engine kernel and contracts, not service composition. - Every table/state family has one logical owner and credential policy. - Every route, worker family, and event consumer has one runtime owner. - Identity/session/revocation/Connected App has one Runtime authority and one interceptor read path. - Every configuration fact has one effective domain read path; request/run consistency is preserved without a Catalog/live fallback. Release admission is not a second source for current configuration content. ### 10.2 Performance DoD Before D4 production replacement, reviewed evidence fixes the workload, dataset, environment, and approved numbers for at least: - Describe/object model; - query; - record retrieval/read graph; - record mutation plus before-save; - global search; - outbox bursts; - webhook bursts; - metadata apply plus immediate read; - multiple Apps in one org; - cross-org negative workload. Hard gates: - every use case has a fixed RPC-count budget; - RPC count cannot grow linearly with records, fields, lookups, or related lists; - every hop has a latency budget and end-to-end p99/p999 meets product SLOs; - worker and ingress shock load stays within API isolation budgets; - aggregate connection budgets stay within approved TiDB capacity; - retries cannot amplify exponentially and every mutation retry is protected by idempotency; - comparisons use the same workload and never substitute averages for p99/p999. Numbers come from baseline evidence. This document does not invent a threshold such as 10 percent. ### 10.3 Upgrade DoD - Catalog, Integration, each L1, and Runtime roles roll out and roll back independently within declared windows. - Protocol supports at least N/N-1. - Runtime API/worker EngineGeneration compatibility fails closed in readiness. - Pending work validates payload, engine, handler, and AppRelease compatibility at claim and rollout gates; old-handler retention and drain rules are explicit. - AppRelease uses capability admission rather than exact build-SHA equality. - Phase-one schema changes are additive; destructive changes use expand/migrate/contract. - Catalog publication unavailability must not by itself prevent online configuration save/activation/readback for an already-admitted app. Its domain and schema authorities must still satisfy their own availability and consistency contracts. Security-protected traffic requires a valid SecurityLease; affected traffic fails closed beyond `MaxSecurityStaleness`. A release installation/publication job may remain unavailable. - One L1 upgrade does not rebuild L0 or other L1 Apps. - The rollback matrix covers every rollout order and persistent-state stage. ### 10.4 Failure-Recovery DoD - Within the durability fault model of the owned store, acknowledged work and receipts survive a single process or node crash. AZ, region, and storage-loss RPO/RTO belong to a separate backup/DR contract; this document does not claim unconditional RPO=0. - Integration responds successfully only after the receipt is durable. - Work recovers under lease/retry policy after worker termination. - Durable acceptance/dedupe is exactly-once inside the owned store; execution is at-least-once. - Effect-once is guaranteed only when the downstream accepts a stable idempotency key or exposes a queryable outcome. Non-idempotent ambiguous effects enter `OUTCOME_UNKNOWN` for query, reconciliation, or manual repair. - Ordering is guaranteed only within a declared partition key. - Catalog apply can continue or repair safely after termination at every state. - L1 failure affects only that App, not L0 or other L1 Apps. - Invalid snapshot checksum/version is rejected while retaining the last verified snapshot. - Every boundary has kill, timeout, duplicate, reorder, stale-version, and incompatible-version tests. --- ## 11. Phase-One Delivery: Test-First Full-Stack Replacement ### 11.1 Target Directories and Release Boundaries Retain the completed seven-repository release topology. Do not add repositories merely to resemble microservices: ```text NexusCRM/ backend/cmd/runtime-api/ backend/cmd/runtime-worker/ nexuscrm-api-host/ cmd/gateway/ nexuscrm-contract/ *.go # existing stdlib-only semantic Go module proto/common/v1/ proto/runtime/v1/ proto/catalog/v1/ wire-go/ # separate generated Go module nexuscrm-controlplane/ cmd/catalogd/ nexuscrm-connectors/ cmd/integration-ingress/ nexuscrm-app-packs/ cmd/sfcrm/ tests/conformance/issue-tracker/ ``` Every delivered command has an independent image, configuration schema, health/readiness, metrics, version/capabilities, and deployment manifest. `role=all` means exactly `runtime-api + runtime-worker`: two deployment roles of the same L0 Runtime logical service. It never embeds Catalog, Integration, `sfcrm`, or another L1 AppService into L0. Test fixtures are never embedded in the production Runtime. A complete test environment runs independently released service artifacts through the L2-selected immutable release tuple. An empty Runtime worker or Integration command does not complete a delivery; each is deployed only when it takes real, durable work ownership. ### 11.2 Binding Delivery Sequence The contract, process, security, App authority, snapshot, and dormant record-access foundations preceding D1 are complete. Routes, metadata, mutation, workers, and Integration still move as reviewable vertical slices, but those slices build one all-new test distribution rather than taking production routes independently. `pingcap-inc/NexusCRM#3555` owns live status; the binding dependency order is: 1. **D1 — test-stack foundation** - publish and pin the generic Gateway, sfcrm AppService, serving Runtime API, Catalog reconcile, frontend, AppRelease, and exact L2 metadata as immutable artifacts; - activate P7c/S1 only inside the new Runtime composition, with one `TrustedCall`, one request-bound `RuntimeMetadataView`, and zero process-global/repository metadata fallback; - exercise the real record-access walking slice through Gateway, AppService, Runtime Identity, SecurityAdmission, ServingAdmission, one Records RPC, and TiDB; - keep production entirely legacy and provide no new-to-legacy fallback. 2. **D2A — synchronous read surface** - migrate supported record read/query/search, describe/metadata/UI API, access-decision, file-read, and other synchronous read families through bounded coarse L0 contracts; - preserve public behavior and add capability-specific CRM parity plus non-CRM conformance evidence. 3. **D2B — mutation plus Runtime worker** - migrate create/update/delete together with validation, synchronous automation, transactionally written queue/outbox state, worker ownership, retry, idempotency, kill/restart, and repair; - validate exact definition, payload schema, engine/handler compatibility, and `OwnershipEpoch` at claim, renewal, and completion. 4. **D2C — external and asynchronous surface** - migrate real webhook, connector, streaming, file-upload, inbox/outbox, dead-letter, and repair behavior; - return success only after durable receipt and never count an empty Integration process as progress. Its final delivery consumes D2B's durable ownership and idempotency rules. 5. **D3 — full-stack acceptance** - run the complete supported product in test with no legacy process and no source-checkout serving dependency; - pass parity, security, metadata upgrade, performance, failure, recovery, immutable promotion, and whole-release rollback gates; - prove a machine-checked production-surface manifest has no unowned route, worker, scheduled job, EventBus subscriber, webhook/connector, file, streaming, or frontend dependency. 6. **D4 — single production replacement** - during a controlled maintenance window, stop ingress, drain the legacy deployment, and start the exact D3-approved release tuple; - expose one static versioned all-new Gateway route configuration and one identity/security/metadata path. No per-route production canary, proxy, or fallback is introduced. 7. **D5 — legacy retirement** - retain the previous whole release tuple only for a bounded rollback window; - after state-compatibility and observation gates pass, physically delete the embedded sfcrm distribution, in-process L1-to-L0 paths, legacy worker startup, source-backed CRM serving, deployment roots, compatibility guards, and obsolete documentation. D2A and D2B implementation may proceed in parallel after D1 is stable, but they must converge through one L2-owned launch-candidate release train. They do not own competing full BOMs or independently chase each new repository `main`. D2C may prepare in parallel, but its final inbox/outbox ownership depends on D2B. A real second App proof is independent and does not block the CRM replacement. #### 11.2.1 Launch-Candidate Convergence Once the launch-required D2A and D2B implementations exist, delivery changes from capability expansion to release convergence: 1. restore a green, publishable owner revision for every selected repository; 2. select one exact immutable tuple in the L2 delivery repository; 3. freeze that tuple while functional acceptance runs; 4. accept a newer owner revision only when it fixes a demonstrated blocker in the declared launch surface; 5. update the tuple atomically and rerun the failed proof plus the complete local acceptance path. The launch candidate does not automatically follow the tip of any repository. Unrelated features merged after the freeze belong to a later release. D2A and D2B can retain separate engineering owners, but only the L2 release authority may select the combined candidate and declare it ready for D3. Functional and project-level discovery is local-first. Before immutable publication, the selected source tuple must run the real release lifecycle against an ephemeral TiDB and separate Runtime API/Worker/AppService processes: ```text apply-metadata -> repair -> apply-packs -> Catalog activation -> Gateway -> sfcrm AppService -> Runtime API -> TiDB -> Runtime Worker durable processing and recovery ``` Local execution is the fast feedback loop for lifecycle, behavior, ownership, restart, fencing, and no-fallback failures. Repository CI remains authoritative for compilation, tests, boundary policy, deterministic generation, exact-source identity, and publication admission. Signed test-environment execution confirms the already-proven candidate; it must not be the first place ordinary functional incompatibility is discovered. One capability-family change does not republish a complete release tuple merely to record progress. The launch tuple is published and run when all currently selected changes have passed the local convergence path. Evidence-only, status-only, and repin-only changes are folded into the owning implementation or L2 candidate change whenever repository ownership permits. ### 11.3 Single-Org Replacement and Durable Work Ownership - The separate test environment acquires capabilities incrementally, but each migrated capability has only the new owner/read path inside that environment. The new distribution never invokes the legacy deployment as fallback. - Production remains entirely on the legacy deployment through D1-D3. D4 stops and drains that ingress before the new Gateway becomes public, so production never has route-by-route dual ownership. - Phase one uses one static, versioned all-new route configuration selected by the immutable L2 release tuple. It does not require a dynamic per-route `RouteEpoch` or a generic always-on rollout controller. - Rollback during the bounded window replaces the whole release tuple; it does not route selected paths back to a live legacy proxy. D5 starts only after the rollback and state-compatibility gates close. - Every durable work family still has an authoritative owner record and monotonically increasing `OwnershipEpoch` in Runtime DB. A stale worker can mutate state even when public ingress is stopped, so this fence is not removed by the single-org route simplification. - Claim, lease renewal, completion, and side-effect ledger conditionally update the epoch. Work-family transfer stops new claims by the old owner, drains or expires in-flight leases, increments `OwnershipEpoch`, then permits the new owner. Stable downstream idempotency protects already-sent effects; uncertain non-idempotent effects enter `OUTCOME_UNKNOWN`. - Startup assertions are diagnostic only and never provide mutual exclusion. --- ## 12. Explicit Rejections - Generate remote RPCs for the current 33 Go interfaces. - Compile L1 executable code into L0 or use Go plugins as the primary boundary. - Give L1 direct access to platform databases. - Make Runtime hot paths synchronously depend on Catalog. - Call L1 or Integration inside an active mutation transaction. - Use fire-and-forget as business delivery. - Allow metadata to declare arbitrary service endpoints. - Treat API and worker as independently evolving microservices with incompatible interpreter semantics. - Write another logical service's internal tables directly. - Allow dual owner, dual read, or dual write for a route or work family. - Introduce a phase-one per-route production canary, dynamic `RouteEpoch`, or new-to-legacy routing fallback instead of the D4 whole-release replacement. - Claim performance success without a baseline, SLO, and RPC budget. - Count a CI conformance fixture as a real L1 product, production AppService, or proof of multi-App production deployment. --- ## 13. Incremental Implementation and Full-Stack Replacement Gates The technical gates remain capability-scoped so each new-stack slice can be built and tested independently. Production activation is not capability-scoped: phase one admits the complete distribution only through the D3 acceptance and D4 whole-release replacement gates. ### 13.1 Skeleton Gates — Foundation Complete These decisions were required before the service skeleton could be created and remain regression gates for every new executable role: 1. the logical-service, release, and data-ownership matrix; 2. the exact command locations and the rule that `role=all` contains only the two L0 Runtime roles; 3. configuration, workload identity, health/readiness, version, and capability endpoint conventions; 4. package/version rules separating the stdlib-neutral contract from proto/generated wire modules; 5. the golden behavior/performance baseline and current consumer/read-path inventory used to prove later activations. Passing these gates does not make an empty command a completed service, approve data-plane serving, transfer durable work ownership, or authorize production traffic. A role becomes part of the distribution only with a real capability and its capability-specific proof. ### 13.2 Capability and Production-Replacement Gates Before a capability enters the new test distribution, its applicable ADR and contract artifacts must be approved and backed by executable tests. Before D4, all items applicable to the complete supported production surface must pass as one release decision: 1. logical-service and data-ownership matrix; 2. Identity/session/revocation authority, delegation, and AppInstallation authorization contract, specialized for the first Runtime capability by [`runtime-trusted-delegation-contract.md`](runtime-trusted-delegation-contract.md); 3. package/version/capability rules separating the stdlib-neutral contract from proto/generated wire modules; 4. [AppManifest, MetadataBundle, ownership, installation, and binding contract v1](app-authority-contract.md); 5. one effective configuration read path per domain, stable request/run versions, online adoption and replacement-path deletion contract; 6. Catalog release writer lease/fencing, install/migration preparation, publication/activation and release read-your-write state machine; 7. SecurityEpoch/lease, maximum staleness, and serving-replica fencing; 8. Runtime request context, outbox/inbox state model, and asynchronous work implementation-compatibility envelope; 9. immutable phase-one ingress/release-tuple activation and durable-worker `OwnershipEpoch` transfer protocol; 10. benchmark matrix, data scale, environment, and SLO approval process; 11. N/N-1 upgrade, rollback, and handler-retention matrix; 12. tenant/security threat model and cross-org/cross-App negative tests; 13. failure-domain-specific delivery, RPO/RTO, and external `OUTCOME_UNKNOWN` repair contract. Items are not a blanket prerequisite for unrelated construction work. Identity and security gates precede the identity/security slice; App contract gates precede an L1 slice; snapshot/publication gates precede metadata serving in the new composition; durable delivery and ownership fencing precede worker or Integration ownership. Capability evidence can therefore close incrementally through D1-D2, but it authorizes test operation only. D3 must prove the complete supported surface without the legacy process, and all thirteen items applicable to that surface must pass before D4 changes the single production owner. Component-kind expansion, pooled multi-tenancy, brokers, fine-grained snapshot deltas, and destructive schema migration can be added compatibly later without changing these authority and communication boundaries.