# Serviceified deployment version and compatibility design **Status:** normative architecture for serviceified deployment **Owner:** `nexuscrm-pingcap-pack` (L2 delivery) Test2 is the first proving environment. Its operational procedure lives in [`test2-standalone-deployment.md`](test2-standalone-deployment.md), but this design applies to every target environment. ## Outcome Deployment has three independent release units: 1. **L0 Engine release** — the generic Engine service; 2. **L1 Application release** — one executable application, including its AppService and frontend; 3. **AppRelease** — the L1 base packs assembled with the L2 overlays for one application. Each release unit has its own version and can be deployed, upgraded, and rolled back independently. Compatibility has only two one-way checks: ```text AppRelease --requires L1 Application version range--> L1 Application L1 Application --requires L0 API version range-----> L0 Engine ``` The dependent release declares the supported range. The provider never binds itself to consumers: - L0 does not know or check any L1 or L2 version; - L1 does not know or check any AppRelease or L2 version; - AppRelease does not check L0 directly. Both relationships are many-to-many. No cross-layer commit or hash equality is part of compatibility. ## Release units and owners | Release unit | Single owner | Version facts | Contents | |---|---|---|---| | L0 Engine release | L0 | Engine release version and L0 API version | one signed L0 release descriptor; L0 owns any role-specific artifact identities inside it | | L1 Application release | L1 | Application version and supported L0 API version range | AppService and frontend | | AppRelease | L2 delivery | AppRelease version and supported L1 Application version range | exact signed L1 base-pack releases, L2 overlays, migrations, and activation material | L1 remains the authority for each L1 base-pack release. L2 delivery selects an immutable L1 base-pack release, applies the L2 overlays, and becomes the single owner and publisher of the resulting AppRelease. Preserving the L1 source and version as provenance does not make the final AppRelease a dual-owned fact. L2 must verify and bind the exact signed L1 base-pack descriptor and artifact; it does not copy, mutate, or re-sign the L1 release, and it does not republish the separately selected executable L1 Application. ### L0 Engine release L0 is independently built and deployed. Its signed release descriptor may bind one image or several role-specific image digests, but that is an internal L0 packaging fact. Runtime API, Worker, Integration, scheduler, and other generic roles do not become separately selected L1 or L2 release axes. L0 publishes its Engine release version and L0 API version. It does not import L1, contain an L1 allowlist, or block an upgrade because of a downstream application version. ### L1 Application release An L1 Application release contains the complete executable application. For SFCRM this means AppService and frontend under one Application version. The frontend is not a separate L2-selected release and is not rebuilt during L2 deployment. L1 declares the range of L0 API versions it supports. L1 is the sole owner of this compatibility decision and checks the running L0 API version at startup before serving traffic. Deployment observes L1 readiness; it must not reimplement the same range decision in L2. Failure blocks only that L1 activation; it does not make L0 unhealthy or block unrelated applications. ### AppRelease For the PingCAP SFCRM application, one AppRelease contains the selected official L1 SFCRM base packs plus the PingCAP L2 overlays, migrations, and activation material. The AppRelease declares the range of L1 Application versions it supports. Controlplane checks that range before activation. Failure leaves the currently active AppRelease and serving application unchanged. An AppRelease never names an allowed L1 source commit and never compares its L1 base-pack commit with the running L1 Application commit. ## Compatibility rule The compatibility rule is deliberately small: ```text L1 is compatible with L0 iff selected L0 API version is inside L1.supported_l0_api_range AppRelease is compatible with L1 iff selected L1 Application version is inside AppRelease.supported_l1_application_range ``` Version ranges are the only cross-release compatibility mechanism. This design does not add feature-by-feature negotiation, hash allowlists, build-lineage equality, or a second compatibility authority. Each version denotes one complete owner-defined conformance contract, not an arbitrary package label. L0 may publish an L0 API version only after owner CI proves every required primitive and route in that version; L1 applies the same rule to its Application version. A release missing part of its claimed contract is an invalid release and must fail owner publication/readiness. It is not made safe by adding an L2 capability negotiation path. Ranges use two canonical semantic-version fields, not a free-form expression: ```text supported_l0_api_range = { minimum_inclusive: 0.0.1, maximum_exclusive: 0.1.0 } ``` Versions are canonical SemVer without build metadata. The lower bound is inclusive and the upper bound is exclusive. This is the first SFCRM range; L0 may publish any independently built release inside it without requiring a new L1 or L2 release. Example: ```text L0 Engine dev-20260821-0.0.42-deadbeef publishes L0 API version 0.0.42 L1 SFCRM 0.40.0 supports L0 API [0.0.1,0.1.0) AppRelease 18.5 supports L1 SFCRM [0.39.0,0.41.0) ``` This tuple is compatible. Other L0, L1, and AppRelease versions are also valid when they fall inside the declared ranges; their commits and build dates do not need to match. ## Artifact identity is not compatibility Artifact identity and compatibility answer different questions: ```text digest/signature = which immutable artifact is this, and who published it? version range = can these independently released artifacts run together? ``` Digests, signatures, source commits, and SBOMs remain valid for artifact integrity, provenance, audit, and exact rollback. They must not become cross-layer compatibility checks. The following checks are forbidden: - L0, L1, or L2 source commits must be equal; - an L1 or AppRelease source commit must appear in a hash allowlist; - final-pack lineage must match an L0 or L1 repository hash; - L0, L1, frontend, and L2 must share a build revision or release date; - rebuilding a signed artifact is required to prove compatibility. ## AppRelease BOM boundary The AppRelease BOM authorizes the immutable L1/L2 application release. It must bind the L1 Application artifacts, the assembled metadata and migrations, the exact L2 metadata source/tree used to assemble them, and their signatures and digests. Removing L0 from this BOM does not make L2 provenance optional. The AppRelease BOM must not contain or derive any L0 fact, including: - an L0 release version, source commit, tag, digest, image repository, or signature; - Runtime API, Worker, Integration, Catalog, or other L0 subject identities; - an L0 API version or a copied L1-to-L0 compatibility result; or - a whole-stack hash whose value changes when only L0 changes. Historical whole-stack BOMs remain audit records only. No current writer, resolver, validator, planner, or deployment path may accept them as the active authority after cutover. ## Deployment principles The operator may request changes to the desired L0 Engine, L1 Application, and AppRelease in one operation, but these remain independent selections. The exact L0 release identity is carried by the L0 deployment request and active L0 release state, never by the AppRelease BOM. Plan, prepare, apply, checkpoint, and resume are internal phases of that one deployment action, not separate operator procedures. One signed, authenticated deployment authorization binds the operation ID and every selected release to an exact signed digest for that operation. The release gate signs this authorization before mutation, and Prepare and Apply must consume those same bytes, so neither L0 nor AppRelease can be substituted after approval. This authorization is an operation receipt, not a reusable whole-stack release or compatibility authority: changing only L0 creates no new AppRelease BOM, and changing only AppRelease creates no new L0 release. Before mutation, each selected release owner must: 1. resolve its exact signed artifacts; 2. verify their identities and signatures; and 3. validate its target-environment and migration preconditions. L1 alone authoritatively evaluates `supported_l0_api_range`. Before a candidate L0 serves traffic, deployment presents its signed descriptor and API version to the L1-owned evaluator and consumes the resulting allow/deny verdict. L1 also evaluates the running provider at startup and when its observed version changes, then publishes the result through readiness. L2 must not parse the range, produce the verdict, or copy it into its own compatibility state. Controlplane likewise owns the AppRelease-to-L1 decision. Before a candidate L1 serves with an unchanged active AppRelease, deployment presents the signed L1 descriptor and version to Controlplane's evaluator and consumes its verdict. An unchanged release unit does not need to be reinstalled or restarted merely to perform its owner-native pre-activation check. A denied candidate never becomes serving, and the current compatible tuple remains active. Apply changes only the selected release axes: - an L0-only change rolls L0 roles without reinstalling L1 or AppRelease; - an L1-only change rolls AppService and frontend together; - an AppRelease-only change runs its Controlplane lifecycle without restarting unchanged L0 or L1 services; - an unchanged release unit performs no work. ## Operator usage during the migration The target architecture above is introduced through the existing trusted operator surfaces. Exact commits and digests in these commands select immutable inputs; they are not cross-layer compatibility rules and do not require a PR whose only purpose is to copy a new hash into L2. ### Test validation Remote Test full deployment uses the exact controller ref selected by an authorized manual dispatch and one already-signed resolved BOM. Controller selection and product release authority remain independent. The same selected-ref serviceification workflow owns production-to-Test database refresh and may replace only SFCRM AppService with one already-published exact owner-repository digest. A standalone L0 candidate also crosses its owning repository's publication boundary before selected-ref Test execution. Neither image diagnostic accepts a product source ref or performs a build or publication, and neither claims a signed L1 Application or AppRelease transition. Neither diagnostic changes the database, Catalog, application metadata, or frontend. The next full signed-BOM deployment, including the same signed BOM under a new deployment run ID, discards the transient process-image override. No raw source ref becomes a serving identity, and Staging and Production inherit no diagnostic exception. After success, prove the public surface independently: ```bash curl -fsS https://nexuscrm-new-test2.pingcap.com/health curl -fsS https://nexuscrm-new-test2.pingcap.com/ >/dev/null curl -fsS https://nexuscrm-new-test2.pingcap.com/index.html >/dev/null gh workflow run staging-inspection.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack \ --ref main \ -f target_env=test2 \ -f role=sales \ -f case_id=sales/account/account-related-list-count-parity-api \ -f module_concurrency=1 \ -f dry_run=false \ -f notify_feishu=false ``` Run the focused case first when it proves the changed claim directly. Run the full inspection only after the focused proof passes. Deployment and inspection remain separate actions so either can be rerun without manufacturing a new release identity. ### Transitional Test release path Until the independent descriptors and active-state cutover below land, formal Test D3 deployment uses the exact manually selected controller ref and one already-signed resolved-BOM digest: ```bash gh workflow run deploy-test-serviceification.yml \ --repo pingcap-inc/nexuscrm-pingcap-pack \ --ref \ -f slot=test2 \ -f operation=deploy \ -f resolved_bom_reference='ghcr.io/pingcap-inc/nexuscrm-serviceification-bom@sha256:' ``` This is a migration source, not the target authority. Existing signed-BOM fields remain enforced until their owner-native replacements are live and the old readers are deleted in the same cutover. New work must not extend the legacy whole-stack BOM or add another hash-compatibility gate. ### Safe multi-axis upgrades A compatible final tuple does not guarantee that every intermediate tuple is compatible. Deployment must preserve compatibility for every tuple that serves traffic. If no safe sequential order exists, it must prepare the complete candidate and use one coordinated activation cutover after the candidate is ready. The exact ordering, staging, routing, and rollback algorithm belongs to the implementation design, not this architecture document. ### Active-state invariant Each release unit has one authoritative active state: L0 owns the active L0 release, L1 owns the active L1 Application release, and Controlplane owns the active AppRelease generation. A displayed three-part tuple is a joined read model, not another mutable authority and not a whole-stack BOM. Prepared, applying, and active states must be distinguishable, and a candidate may become active only after successful health verification and public readback. A partial or failed deployment must not be reported as active. The exact state owner, storage schema, and commit protocol require a separate implementation design based on the chosen deployment topology. Retries resume the same deployment checkpoint. Deployment never resets tenant data and never runs whole-platform `ApplyAll` merely because one release unit changed. ## Required legacy deletion The new deployment path replaces the old deployment model. The completed implementation must delete, not preserve as fallback: - cross-repository commit and hash equality checks; - every L0 field, projection, receipt, and compatibility result in current AppRelease BOM writers, resolvers, validators, signatures, and deploy readers; - BOM fields whose purpose is to bind L0, L1, frontend, and L2 source commits; - whole-stack active-state and successor-BOM predicates that make an L0-only release look like an AppRelease change; - reverse L0-to-L1 or L1-to-L2 version checks; - obsolete Test deployment entrypoints; - reset-and-deploy application workflows; - whole-stack `ApplyAll` upgrade behavior; - obsolete CI tests and compatibility aliases for those paths. The exact file-by-file deletion ledger belongs to the implementation plan so it can be derived from the live tree rather than frozen in this long-lived design. Each compatibility cutover must remove the replaced old read path in the same change; old and new authorities must not remain live together. Each release boundary has one remote Test mutation owner. The standalone Test1 walking slice proves that split: its L0 management CI alone replaces or restarts its L0 process, while its AppRelease CI alone applies L1/L2 packs and publishes the Catalog during that walking slice. The existing full-D3 workflow remains the formal Test owner until the complete Test1 L0/L1/AppRelease path is validated and explicitly cut over; operators must not mix the two paths in one Test1 experiment. Test2/Test3 remain on full D3. At cutover the composed full deployment must call the owner surfaces instead of duplicating their container or Controlplane logic, and the replaced Test1 mutation path must be removed. ## Incremental optimization ledger This ledger is updated as implementation PRs land. Every milestone must name its owning fact, new read path, real verification evidence, and the replaced path deleted in the same cutover. | Milestone | Required outcome | Completion evidence | Status | |---|---|---|---| | Test topology integrity | Full D3 uses an exact selected controller ref plus a signed BOM; standalone Test1 L0 and AppRelease operations have disjoint owners and no duplicated process or Controlplane path; selected-ref SFCRM diagnostics consume one prepublished exact digest with no build or release claim | Exact-dispatch evidence plus standalone Test1 L0/AppRelease and retained-database Test D3 deployment evidence | In progress | | Phase visibility | Build, source-set apply, and start expose distinct phase/timing evidence while still using one runner and one mutation state machine | A failure can be assigned to one phase without replaying completed work | Next | | Owner-native descriptors | L0 publishes release/API/artifact identity; L1 publishes Application version and supported L0 range; AppRelease publishes exact L1/L2 application material and supported L1 range | Signed descriptor schemas and conformance vectors from each owner | Planned | | L1-to-L0 cutover | Only L1 evaluates its supported L0 range; deployment consumes the L1 verdict before activation and observes L1 readiness afterward | In-range and out-of-range pre-activation/startup/readiness tests; L2 protocol/capability/hash compatibility readers deleted | Planned | | AppRelease-to-L1 cutover | Only Controlplane evaluates the AppRelease-supported L1 range and deployment consumes that verdict before L1 activation | Activation retains the previous generation on mismatch; duplicate L2 evaluator deleted | Planned | | Independent authorization and active state | One signed operation authorization binds exact selected artifacts; L0, L1, and AppRelease keep separate active authorities; AppRelease BOM contains no L0 fact | Signature, restart/resume, and readback tests for each axis; all L0 BOM writers/readers/signatures deleted | Planned | | Delta apply | An unchanged axis performs no build, install, restart, or `ApplyAll`; every intermediate serving tuple remains compatible | L0-only, L1-only, AppRelease-only, multi-axis, failure, and resume walking slices | Planned | | Formal-path cutover | Reviewed formal deployment consumes the independent release model and no alternate Test topology entrypoint exists | Test2 signed deployment, public health, focused inspection, full inspection, and legacy-entrypoint deletion | Planned | Do not mark a milestone complete from document or mock-only evidence. Record the actual run or test that proves it and update the deletion column before moving to the next milestone. ## Out of scope for this architecture document Implementation work must follow these principles, but this document does not prematurely choose: - descriptor JSON schemas or parsing libraries; - the internal Plan/Prepare/Apply API shape; - the multi-axis cutover algorithm; - active-state storage and transaction details; - the file-by-file migration sequence. Those decisions must be made from the current implementation and target environment, without reintroducing hash compatibility, duplicate authority, or legacy fallback paths.