# L0 independent deployment working notes Status: implementation baseline, 2026-08-05 ## Goal Deploy and maintain the app-blind NexusCRM L0 independently. L1 applications consume one private Runtime gRPC endpoint and publish their metadata through the L0 Controlplane/Catalog mechanism. L2 chooses environment values but owns no L0 mechanism. The first delivery optimizes for a clean deployment followed by stopped activation. It does not add rolling-upgrade, hot-reload, fallback, dual-read, or compatibility machinery. ## Desired deployment shape ```text one-shot l0-controlplane init | v external TiDB Runtime API bootstrap (live, never ready) | L1 publishes through L0 Catalog | stopped activation v external TiDB <-> Runtime API <-> immutable Catalog snapshot store ^ | one Runtime Worker image x seven role configurations L1 sfcrm / Integration -> one private TLS gRPC Runtime endpoint ``` L0 contains three release images and four deployment roles: 1. `runtime-api`: the only private L0 network endpoint. The same image carries the one-shot `l0-controlplane` command for `init-l0`. 2. `runtime-worker`: one image activated with seven role-scoped configs. 3. `nexuscrm-controlplane-catalog-reconcile`: the existing signed, one-shot Catalog publication job. It is not a serving process or network endpoint. The four roles are one-shot substrate initialization, one-shot Catalog reconcile, long-running Runtime API, and long-running Runtime Worker. Neither Controlplane role is a network service. TiDB is external to the Compose project. Local development may use TiUP/TiDB without TLS. Remote TiDB uses TLS 1.2+ with the image's system CA bundle and hostname verification. The deployment path must never default a remote connection to `TIDB_TLS_DISABLE=1`. ## Ownership boundary - L0 owns Catalog publication, immutable revisions, AppInstallation identity, authorization, revision binding, generic metadata interpretation, and the Runtime/Worker mechanisms. - L1/L2 own metadata **content**. L0 does not author, predeclare, or branch on an application's object names, Flow names, capability values, or tenant values. - The deployment script treats the Catalog-published Runtime serving artifact as opaque. It mounts the artifact; Runtime's existing config owner validates it. The script must not parse an AppInstallation ID or capability list and become a second policy owner. - Catalog is part of L0 but is not a separate serving process or network hop. Controlplane publishes; Runtime reads immutable results and exposes neutral consumer views through the existing endpoint. ## Clean deployment and stopped-activation sequence ```text 1. l0 init - connect to an empty or retained TiDB database - initialize only the generic L0 substrate 2. l0 deploy - start the final private TLS endpoint from the same Runtime API image - require only L0-owned TLS transport material - remain live but not ready; reject every business RPC 3. l1 install/publish - L1 supplies metadata content - L0 Controlplane/Catalog validates and publishes an immutable revision - publication produces the opaque Runtime serving artifact 4. l0 activate (stopped transition) - replace bootstrap with one Runtime API and all seven Worker roles - Runtime validates Catalog, security, generation, and receipt authority - readiness remains fail closed until all authority is valid ``` Production uses the same sequence in a maintenance window. No long-running old/new dual topology is required. ## Pits found while implementing ### 1. The checked-in Compose represents an obsolete D1 shape The previous `infra/runtime/compose.yaml` starts only `runtime-api`, exposes its admin port, omits every Worker role, defaults to mutable `dev` images, and defaults `TIDB_TLS_DISABLE` to `1`. It is not the current L0 topology and must not remain as an alternative deployment path. ### 2. Reusing the downstream D3 bundle as the permanent L0 input is wrong The first working prototype consumed `nexuscrm-pingcap-pack/.runtime/serviceification-d3-local`. That proved the process split, but it tied a generic L0 deployer to one L2 repository's private directory layout. The upstream deployer instead consumes a neutral L0 state root. L2 may populate it, but does not define the mechanism or path grammar. ### 3. The deployer started interpreting L1/L2 values An early draft decoded `serving.json`, required a non-empty capability list, and reported the AppInstallation ID. That duplicated Runtime's config owner and made deployment orchestration aware of application values. The deployer now checks only that the opaque artifact can be mounted; Runtime owns semantic validation. ### 4. "Serving revision" was an ambiguous term The immutable Catalog revision/snapshot is metadata authority. The Runtime serving artifact is an activation selection that binds a Runtime replica to Catalog/security/release receipts. They are related but not interchangeable. Neither is authored by the deployment script. ### 5. Business serving authority was incorrectly promoted into a deployment prerequisite `runtime-api` intentionally requires complete serving authority. This is correct for business RPC readiness: missing or stale authority must fail closed. The mistake was requiring that L1-produced artifact before the L0 release could even be deployed. The root fix separates two states already represented by Runtime: - **deployed:** the TLS endpoint is live, readiness is false, and all business RPCs reject; - **activated:** immutable Catalog/security/release authority has been supplied and the full serving composition passes readiness. The explicit `runtime-api bootstrap` mode uses the same image and existing rejecting composition. It opens no database, reads no Catalog artifact, and does not provide a fallback path. Activation is a stopped replacement, not dynamic reload or a second long-running API. ### 6. Database TLS authority can be silently contaminated by old env files Prepared runtime environments may carry an earlier `TIDB_TLS_DISABLE=1`. Simply appending a remote database env file that omits the key does not unset the earlier value. The Compose must explicitly override it: - local mode: `TIDB_TLS_DISABLE=1`; - remote mode: empty/false, allowing `nexuscrm-db` to enable verified TLS for the non-loopback hostname. No second SQL/TLS implementation belongs in the deployment script. ### 7. Historical CI assertions are evidence, not architecture authority The Runtime boundary gate still asserts the old D1 one-service Compose. That assertion must follow the proven deployment shape; it must not force retention of the obsolete topology. Functional deployment proof comes first. The stale assertion is then removed or updated in the same delivery without preserving a compatibility Compose. ### 8. Private-module credentials are a build concern, not a runtime concern The existing Runtime Dockerfile uses repository-scoped deploy keys to fetch private Go modules. A developer's ordinary SSH key may not have those grants, so a clean local image build can fail even though all three binaries compile from the checked-out source and module cache. Do not copy `gh` login, SSH keys, or source-fetch logic into a running L0 container. CI/release image publication continues to supply build-only deploy keys; deployment consumes already-built images and requires no GitHub identity. ### 9. Clean InitL0 needs explicit L0 identity secrets in remote mode On a clean database, `InitL0` creates only generic platform substrate and platform-owned baseline identity rows. Without `JWT_SECRET` and `NEXUS_ADMIN_PASSWORD`, the current command generates ephemeral values and may print the generated bootstrap password in local init logs. Remote init must therefore receive an owner-protected L0 Controlplane env file. This is L0 identity/crypto configuration, not L1/L2 metadata, and does not justify binding the deployment to an app pack. ### 10. Activation evidence must use one aligned authority tuple A real transition test initially combined a Catalog/serving bundle from the D3 database with a newly initialized L0-only database. Runtime correctly rejected the combination because the referenced Catalog and Worker ownership rows did not exist. A second attempt combined that bundle with a source binary that did not contain the bundle's newer external-effect capability and was also correctly rejected. These are not reasons to couple deployment back to L1/L2. They show where the strict check belongs: **activation** must align the exact Runtime image, Catalog artifacts, database publication state, and Worker ownership tuple. Bootstrap deployment requires none of them. ### 11. Release identity belongs to the L0 release, not to the deployer Runtime API, Runtime Worker, and Integration Service are separate images, but they are roles of one L0 Engine release. Their compatibility must be established before deployment and recorded as one immutable release identity: ```text L0 release Catalog reconcile -> exact OCI digest, Controlplane source revision C Runtime API -> exact OCI digest, source revision R Runtime Worker -> exact OCI digest, source revision R Integration Service -> exact OCI digest, source revision R ``` The human-readable release name is shared by every L0 image and is precise to one UTC day: ```text dev-YYYYMMDD--<8-hex-source-prefix> staging-YYYYMMDD--<8-hex-source-prefix> prod-YYYYMMDD--<8-hex-source-prefix> ``` The L0 release number has one source of truth in `release/l0-release-version`; the first release train starts at `0.0.1`. It is a version of the assembled L0 release, not a component module version. The development date is derived from the immutable Git commit date, not the workflow execution clock, so rerunning one source revision cannot manufacture a different development release. `staging` and `prod` are later promotion identities for an already-proven set of digests; promotion does not rebuild any component image. The complete source SHAs and all four OCI digests remain the real immutable identity. The short hash and date are only the operator-facing name. The exact-source `go-ci` run is the sole development release coordinator. It calls the app-blind same-commit `l0-release-ci` gate and the single independently reviewed, fixed-SHA `trusted-publish-l0-engine-release` reusable workflow. The gate binds the already-published Catalog reconcile job selected by the pinned Controlplane module graph. The publisher builds each Engine role once, verifies its image contract and supply-chain evidence, signs the exact digest, commits the shared version tag, and emits one release descriptor containing every digest. A D3 or later whole-stack BOM consumes that L0 release identity; it must not select arbitrary component image references independently. The L0 release DAG consumes only L0-owned evidence. CRM app-packs, sfcrm source, product dialects, CRM distributions, and any future L1 compatibility suites run as independent application gates against an already selected L0 release; they do not authorize or block publication of the app-blind Engine. Adding another L1 therefore adds another compatibility result, not another prerequisite to the L0 publisher. The reviewed reusable workflow is pinned by immutable commit SHA and is the security boundary. Candidate source can change Dockerfiles and release data, but cannot change the workflow that receives OIDC/KMS signing authority in the same release. This removes the old `workflow_run` receipt handoff and the former publisher-activation cycle without weakening identity: the trusted workflow SHA is both the OIDC identity and the builder recorded in signing provenance, while the caller's exact main commit remains the release source identity. Candidate, signer, and verifier remain separate least-privilege jobs. For automatic main publication, the push event's exact SHA is the source authority. The checkout, release-version suffix, gate inputs, image labels, signatures, and descriptor must all retain that SHA even if another commit merges while the run is building. The publisher refreshes main and requires that the selected source remain an ancestor of its current head; a rewrite that removes the source fails closed. Requiring equality with the current main tip would incorrectly reject ordinary concurrent merges. Manual development publication has no source-branch push event, so it continues to require exact equality with the selected non-main branch head. `scripts/ci/l0-engine-release-source.test.js` executes the workflow's actual admission shell against real local remotes and shallow source checkouts. It covers main advancement, removed source history, mismatched caller/source, failed fetches, and moved or deleted development branches. The L0 release boundary gate runs this test batch. Trusted publisher changes require two reviewed steps: merge the controller implementation first, then pin that immutable merge SHA in both callers and the signing policy and align the boundary guard. Until the activation pin changes, the old reviewed controller continues to execute. Any required OIDC trust update must retain the exact reviewed workflow identity; neither a same-commit relative workflow call nor a mutable workflow ref is an activation shortcut. `deploy.py` is a generic composition runner. It receives the already-resolved image digests and must not become a second release resolver, compare mutable tag strings, infer compatibility, or manufacture a release version. Local development may pass source-built tags, but production and candidate evidence must name the owning release manifest and its resolved digests. No individual role tag is a completed L0 release. The release is committed only after all roles pass the same run and the descriptor binds their exact digests. Do not compensate for a partial release by adding version-selection policy to the deployer. ### 12. L1 publication is a data input, never an L0 build dependency `l0-controlplane init-l0`, Runtime API, Runtime Worker, and Integration Service must build and start without sfcrm, connector installations, official app packs, tenant packs, or any other L1 implementation. L0 cannot know which or how many L1 applications will be installed. An activated business-serving Runtime does consume a neutral immutable Catalog artifact produced after an L1 installation. This is a contract-shaped data dependency, not a source, module, image, or deployment dependency. Runtime may validate revision, scope, receipts, and neutral capability bindings; it must not import L1 code or interpret product-specific values. Current-tree dependency evidence shows no app-pack, Salesforce dialect, sfcrm, Integration composition, or issue-tracker fixture in the production dependency closure of `cmd/runtime-api`, `cmd/runtime-worker`, or `cmd/setup`. The Runtime image and Compose boundary gates additionally reject those owners. `init-l0` dispatches directly to the generic Controlplane `InitL0` operation. The audit found one unnecessary build-context leak: the Runtime Dockerfile and its deny-by-default allowlist still copied `dialect-toy`, an external dialect conformance fixture, even though none of the three production commands depends on it. An isolated exact-main build of Runtime API, Runtime Worker, and setup succeeded with that directory absent. The current delivery removes the two Dockerfile copies and its context allowlist, and the boundary gate now rejects its return. This was not a final-image dependency, but retaining it would make the stated L0-only build boundary misleading and fragile. ### 13. The current Runtime publication receipt is still gated by one L1/D3 composition The exact-main `source-backed-runtime-gate` is not only an L0 real-TiDB gate. It checks out pinned official app-pack and sfcrm sources, runs CRM source-backed install/recovery, Integration Service composition, and an L1 separate-process walking slice alongside valid Runtime API/Worker, persistence, restart, and fencing tests. The Runtime publication receipt currently waits for that entire mixed job. This is a release-ownership leak. The tests are valuable, but their approval owners differ: - generic Runtime/TiDB/Worker authority tests gate the L0 release; - sfcrm, official app-pack, Integration, and cross-layer walking-slice tests gate the resolved D3 whole-stack candidate that selects an L0 release. An app-blind L0 image publication must not require one particular L1 source checkout or Integration composition. Do not delete the cross-layer tests and do not move their logic into `deploy.py`. Split the publication approval DAG at the release boundary: issue the L0 release receipt from L0-owned evidence, then let the D3 BOM/candidate gate consume the resulting L0 release together with its selected L1/Integration releases. This does not invalidate the current images or deployment mechanism; both Runtime binaries already have L1-free production dependency closures. It is a remaining release-pipeline correction and explains why a nominally L0 image publication currently waits for CRM source material. ## V1 acceptance criteria - A clean TiDB can be initialized using the L0 Controlplane artifact. - The same database configuration surface supports explicit local non-TLS and secure-by-default remote TLS modes. - L1 metadata publication is a separate operation and no L1/L2 source is present in the L0 image or Compose dependency closure. - L0 deployment succeeds with only its TLS transport material; it does not require `serving.json`, AppInstallation, capabilities, Worker metadata scope, or a Catalog snapshot. - Before activation the endpoint is live, readiness is false, and business calls fail closed. - The running project contains exactly one Runtime API and seven role-scoped instances of one Runtime Worker image. - Only the loopback-bound private TLS gRPC port is host-published; admin ports stay container-internal. - TiDB, sfcrm, Integration, frontend, ingress, and a generic Gateway are absent from the L0 Compose project. - Stopped activation replaces bootstrap with Runtime/Worker serving from the Catalog-published opaque artifact and fails closed on invalid authority; there is no fallback or dual API. - Local deployment reaches healthy state and an actual TLS gRPC handshake is proven before review/CI is used as evidence. - No production route, metadata, database, BOM, or deployed service is changed. ## Current local evidence - `l0-controlplane init-l0` completed against a new empty local TiDB database in 10.8 seconds with zero L1 Flow definitions. - The standalone Compose project started one Runtime API and all seven Worker roles; all eight containers reported healthy. - Docker published only `127.0.0.1:39090 -> runtime-api:9090`; Worker/admin ports remained container-internal. - The published Runtime endpoint completed a verified TLS 1.3 handshake. - The focused deploy suite and `check-runtime-process-boundary.sh` passed. - The complete `backend` Go test suite passed after the command/config split. - Remote TLS mode is configuration-tested and fail-closed; no TiDB Cloud credential was used in this local proof. - The current source-built Runtime API was packaged over the existing release base and deployed from a TLS-only state root: one container, `livez=200`, `readyz=503`, no TiDB or metadata input, verified TLS 1.3. - The same Compose project then performed a stopped activation with an aligned published D3 image/state/database tuple: bootstrap was removed and exactly one Runtime API plus seven Worker roles became healthy. ## Exact-identity clean-install proof: PASS The closing proof ran on 2026-08-05 using the successful exact-main receipt from Actions run `31016581111` and the two trusted publishers. Both images have OCI revision `9eda7cf057a6ab946112b49cfdfb0c76afc54b5d`: ```text Runtime API ghcr.io/pingcap-inc/nexuscrm-runtime-api @sha256:ad20c808562c73f9298cfa9089943300c7beb916445b55454df5594d1bc1c2c9 Runtime Worker ghcr.io/pingcap-inc/nexuscrm-runtime-worker @sha256:0baa0b52bb976c5ba461cca474d05ca4068324cdf3ff97eb9b7c0794c0b6363a ``` The API image's embedded `l0-controlplane` initialized the newly created empty TiDB database `nexuscrm_l0_release_9eda7cf0`. The database started with zero tables; `InitL0` completed successfully, produced 177 L0 tables/objects, and left `FlowDefinition` empty. No L1 pack or application installation was provided. The same exact API digest then started project `nexuscrm-l0-release-proof` in bootstrap mode. The running composition contains one healthy `runtime-bootstrap` container, publishes only `127.0.0.1:39091 -> 9090`, mounts only its TLS certificate/private key, and has no TiDB, database, Catalog, serving revision, or L1 environment input. Observed boundary results: - the container-internal `livez` returned HTTP 200 with `{"status":"ok"}`; - the container-internal `readyz` returned HTTP 503; - TLS 1.3 completed with certificate verification success; - TLS 1.2 was rejected with a protocol-version alert; - a real `RecordsService.EvaluateRecordAccess` call reached the registered endpoint and failed closed with gRPC `FailedPrecondition`; - there was no second endpoint, legacy process, fallback, or L1 metadata path. This proves the standalone L0 clean-install/bootstrap deployment boundary. It does not claim business readiness: activation still requires a separately published neutral Catalog authority and starts the API plus seven Worker role instances through a stopped replacement.