# NexusCRM independent deployment — agent runbook

This is the executable V1 runbook for deploying the app-blind L0 composition,
installing independently released L1 applications, and activating sfcrm and
Integration without bringing back the legacy process or a generic Gateway.

Read [`README.md`](README.md) for the L0 composition details. Use this document
for ordering, ownership, version selection, collision handling, and the current
stopped-install procedure.

## 1. Keep the three operations separate

```text
L0 bootstrap
  init-l0 → empty neutral database and system authority

L1 application installation
  validate/install exact app pack
  → publish signed AppRelease into an immutable Catalog revision
  → produce AppInstallation-scoped serving authority

L1 Integration activation
  publish real provider binding metadata
  → project IntegrationBindingView and command authority
  → start Integration Service with its own workload credential
```

They share an L0 database and Catalog authority, but they do not share release
ownership:

- L0 owns Runtime API, Runtime Worker, neutral contracts, TiDB schema,
  immutable Catalog mechanics, workload admission, and Worker ownership.
- sfcrm owns its Salesforce-compatible HTTP API, app metadata, frontend, and
  sfcrm service release.
- Integration Service is L1. It owns provider webhooks, provider credentials,
  external HTTP effects, and provider retry semantics. L0 only owns neutral
  `IntegrationBindingView`, `AcceptCommand`, and command execution machinery.
- L2 owns tenant overlays and concrete provider binding values. L2 may depend
  on an L1 capability, but neither L0 nor L1 depends on an L2/D3 BOM.

The serving path is:

```text
client → sfcrm (L1) → private L0 gRPC
provider → Integration Service (L1) → L0 AcceptCommand
L0 Runtime Worker → TiDB
```

## 2. Select versions before changing state

Never use a mutable tag or “whatever is currently checked out” for a release
or shared test deployment.

### L0 release

Select Runtime API and Runtime Worker from one reviewed L0 release manifest.
They normally have different OCI digests but the same source/release identity.
Do not independently mix an API image from one L0 release with a Worker image
from another.

```text
runtime-api@sha256:<exact digest>
runtime-worker@sha256:<exact digest>
```

The release process establishes that the two images are one component version;
`deploy.py` only composes the images selected by the operator.

### Temporary Controlplane policy

Until the online installation path in
[NexusCRM issue #4255](https://github.com/pingcap-inc/NexusCRM/issues/4255) is
complete, use the latest reviewed and published Controlplane Catalog reconcile
release for an L1 install. Resolve it to its immutable image digest before
execution:

```text
ghcr.io/pingcap-inc/nexuscrm-controlplane-catalog-reconcile@sha256:<exact digest>
```

“Latest” never means passing `:latest`. The installer combines the exact
Runtime and Catalog host-compatibility documents and fails before publication
if the selected AppRelease is unsupported. Controlplane is a one-shot
administration job; it is not a long-running service and adds no network hop.

### L1 release

Select these independently released facts:

- signed AppRelease directory and digest;
- exact sfcrm service image digest;
- exact frontend release and compatibility-receipt digests;
- the app-owned install profile approved for that exact AppRelease;
- the pack source that deterministically rebuilds to the signed AppRelease.

The standalone L1 installer must not accept an L2/D3 BOM, a NexusCRM source
commit, or Runtime database credentials for the serving sfcrm process.

## 3. Prepare L0-owned state

Run L0 commands from the NexusCRM repository root and use
`infra/runtime/compose.yaml`. Do not use the repository-root legacy
`docker-compose.yml`.

Create environment files outside source control:

- [`database.local.env.example`](database.local.env.example) for disposable
  local TiDB;
- [`database.remote-tls.env.example`](database.remote-tls.env.example) for a
  remote TLS TiDB;
- [`controlplane.env.example`](controlplane.env.example) for L0 identity and
  crypto inputs.

Use one unique state root and Compose project per deployment. Before bootstrap,
the state root needs only:

```text
<l0-state>/
  secrets/tls/server.crt
  secrets/tls/server.key
```

Do not copy `serving.json`, Catalog files, Worker receipts, or workload tokens
from another database or state root.

Define the deployment inputs once and keep them unchanged for the whole
install. The examples below use these names:

| Variable | Meaning |
|---|---|
| `DB_ENV` | Exact database environment file shared by `init-l0`, `setup`, Catalog reconcile, ownership publication, and activation |
| `L0_CONTROLPLANE_ENV` | L0 bootstrap identity and crypto environment file |
| `L0_STATE` | Unique state root for this database and L0 deployment |
| `L0_PROJECT` | Unique Compose project name |
| `L0_GRPC_PORT` | Host port for the one private Runtime gRPC endpoint |
| `RUNTIME_API_IMAGE` | Exact Runtime API image digest |
| `RUNTIME_WORKER_IMAGE` | Exact Runtime Worker image digest from the same L0 release |
| `CONTROLPLANE_IMAGE` | Exact reviewed Catalog reconcile image digest |
| `APP_PACKS_REPO` | Exact released `nexuscrm-app-packs` checkout |
| `APP_RELEASE_VERSION` | Signed sfcrm AppRelease version in that checkout |
| `L1_TOOLS` | Private local directory for one-shot install binaries |
| `SETUP_BINARY` | CRM-aware one-shot pack lifecycle binary |
| `L1_EVIDENCE` | New output directory for this installation receipt set |

The remaining command variables are release facts or deployment identities:
`ORG_ID`, `INSTALL_IDEMPOTENCY_KEY`, `RUNTIME_REPLICA_ID`,
`RUNTIME_HOST_COMPATIBILITY`, `CATALOG_HOST_COMPATIBILITY`,
`SFCRM_SERVICE_DIGEST`, `FRONTEND_RELEASE_DIGEST`, and
`FRONTEND_COMPATIBILITY_RECEIPT_DIGEST`. Resolve them from reviewed release
artifacts; do not infer them from a running process or an old state root.
The complete Integration publication example additionally uses
`CATALOG_CONFIG` and `APP_RELEASE_DIR`, both produced or selected by its
reviewed deployment renderer.

The L0 release owner must also provide the version-bound serving-state
substrate: `runtime.env`, signing/workload identity documents, seven Worker
role templates, and seven role tokens. `deploy.py` validates and composes these
files; it does not invent credentials or render release authority. If this
bundle is absent, the serving deployment is incomplete. Do not reconstruct it
from a stale D3/L2 BOM or another cluster.

## 4. Initialize and deploy an empty L0

Initialize a new database exactly once:

```bash
python3 infra/runtime/deploy.py init \
  --database-mode local \
  --database-env-file "$DB_ENV" \
  --controlplane-env-file "$L0_CONTROLPLANE_ENV" \
  --runtime-api-image "$RUNTIME_API_IMAGE"
```

For a remote database use `--database-mode remote`; TLS and non-empty stable
identity secrets are mandatory.

Then start the metadata-independent endpoint:

```bash
python3 infra/runtime/deploy.py deploy \
  --state-root "$L0_STATE" \
  --runtime-api-image "$RUNTIME_API_IMAGE" \
  --project "$L0_PROJECT" \
  --runtime-grpc-port "$L0_GRPC_PORT"
```

Expected bootstrap state:

- TLS endpoint is live;
- readiness is false;
- business RPCs fail closed;
- no Worker claims or external effects occur;
- no L1 token exists yet.

`init-l0` creates only neutral L0 schema and system metadata. It does not
install sfcrm, an Integration binding, a tenant pack, or any public route.

## 5. Install the sfcrm L1 application

### 5.1 What runs during the current V1 install

The app-owned installer performs two mutations against the same database:

1. the current L1-aware `setup` binary validates, installs, and verifies the
   exact sfcrm pack through the formal Controlplane pack lifecycle;
2. the exact Controlplane Catalog reconcile image imports the signed
   AppRelease and publishes one immutable Catalog revision.

This is transitional composition, not a second metadata authority. `setup`
does not remain running, and Catalog publication remains the only serving
authority. The installer proves the selected pack rebuilds byte-for-byte to
the signed AppRelease before either install mutation begins.

The database environment is a single input. On Docker Desktop,
`host.docker.internal` is used by the container and is normalized to
`127.0.0.1` only for the host-side `setup` process; database name, port, user,
password, and TLS policy remain identical.

Do not use the installer's `--skip-pack-install` option for a clean or
release-class installation. It is only valid when separate evidence proves
that the exact pack payload is already installed in the same database.

Until `setup` is published as a standalone installer artifact, build its
CRM-aware distribution from the reviewed compatible NexusCRM checkout:

```bash
GOWORK=off go build -tags crm \
  -o "$L1_TOOLS/setup" \
  ./backend/distributions/cmd/setup
```

This is an install-time adapter only. sfcrm does not link it or depend on a
NexusCRM source revision at runtime; sfcrm depends on the neutral L0 API and
capabilities.

### 5.2 PREPARE a new AppInstallation

Run the installer from the exact `nexuscrm-app-packs` release checkout:

```bash
python3 tools/l1-installer/l1_installer.py install \
  --release-dir "releases/io.nexuscrm.sfcrm/$APP_RELEASE_VERSION" \
  --profile "releases/io.nexuscrm.sfcrm/$APP_RELEASE_VERSION/l1-install-profile.json" \
  --runtime-host-compatibility "$RUNTIME_HOST_COMPATIBILITY" \
  --catalog-host-compatibility "$CATALOG_HOST_COMPATIBILITY" \
  --org-id "$ORG_ID" \
  --runtime-audience nexuscrm-runtime-api \
  --service-release-digest "$SFCRM_SERVICE_DIGEST" \
  --frontend-release-digest "$FRONTEND_RELEASE_DIGEST" \
  --frontend-compatibility-receipt-digest "$FRONTEND_COMPATIBILITY_RECEIPT_DIGEST" \
  --idempotency-key "$INSTALL_IDEMPOTENCY_KEY" \
  --operation-kind PREPARE \
  --expected-catalog-revision 0 \
  --expected-security-epoch 0 \
  --schema-generation 1 \
  --engine-generation 1 \
  --output-config "$L1_EVIDENCE/catalog-request.json" \
  --output-receipt "$L1_EVIDENCE/catalog-output.json" \
  --catalog-store "$L0_STATE/data/catalog" \
  --database-env "$DB_ENV" \
  --catalog-reconcile-image "$CONTROLPLANE_IMAGE" \
  --docker-network host \
  --setup-binary "$SETUP_BINARY" \
  --pack-source "$APP_PACKS_REPO/salesforce-crm-standard" \
  --pack-name salesforce-crm-standard \
  --replica-id "$RUNTIME_REPLICA_ID" \
  --runtime-catalog-store-root /var/lib/nexuscrm/catalog \
  --serving-config-output "$L0_STATE/config/serving.json" \
  --installation-receipt-output "$L1_EVIDENCE/installation-receipt.json"
```

For the first app publication into a clean L0 database, both expected epochs
are zero. When preparing another app in an already used L0 authority, use the
current authoritative Catalog revision and SecurityEpoch; never reset them to
zero. A `PREPARE` never supplies an AppInstallation ID. Record these outputs as
one inseparable receipt set:

- new `appInstallationId`;
- `catalogRevision` and `securityEpoch`;
- `catalogSnapshotDigest`;
- AppRelease, service, frontend, and metadata source digests;
- generated Runtime serving-config digest.

Do not generate a token by inventing an identity. Workload credentials are
L0-owned secrets created for the workload IDs authorized by the publication.

### 5.3 Base Integration state is intentionally empty

The base sfcrm install declares two explicit L1 workload identities:

```text
sfcrm-app-service
integration-service
```

It publishes a revision-bound `IntegrationBindingView`, which may be empty.
When there is no real Integration binding, the Runtime serving artifact must
not contain `runtime.commands.accept.v1`.

This distinction is required:

- `required_engine_capabilities` says the selected L0 release supports a
  capability;
- a serving grant authorizes one workload at one immutable revision.

Turning the former into the latter creates ambient authority. Runtime correctly
rejects an `AcceptCommand` grant whose Catalog snapshot has no corresponding
installation/capability projection.

## 6. Install and activate Integration

Integration Service is an independently deployed L1 consumer, not part of the
L0 Compose project. Installing it has two parts.

### 6.1 Publish real binding authority

The tenant/application owner supplies a bounded neutral Integration binding
document. Each binding identifies:

- `authorizedWorkloadId` (normally `integration-service`);
- neutral binding kind;
- owner pack, exact installed-pack ID, and pack payload digest;
- business key and definition version;
- credential references, never credential values;
- for an inbound route, the exact Runtime command kind, payload-format
  version, definition reference, and definition version.

Publish the binding through a reviewed Controlplane Catalog operation. Catalog
must prove that every reference belongs to the exact active installed-pack
generation and must emit a new immutable snapshot. Do not patch `serving.json`
or Catalog JSON by hand.

The current standalone app-packs installer deliberately renders an empty
Integration binding set. It is enough to prove the sfcrm L1 boundary, but it is
not a complete D3 Integration install. For a fresh candidate with Integration,
a reviewed deployment renderer must combine the exact L1 release/profile facts
and the bounded L2 binding document into one schema-v4 Catalog reconcile input
before the initial `PREPARE`. The exact Controlplane image then executes that
input as a one-shot job:

```bash
docker run --rm \
  --network host \
  --user "$(id -u):$(id -g)" \
  --env-file "$DB_ENV" \
  --mount type=bind,src="$CATALOG_CONFIG",dst=/config.json,readonly \
  --mount type=bind,src="$APP_RELEASE_DIR",dst=/release,readonly \
  --mount type=bind,src="$L0_STATE/data/catalog",dst=/var/lib/nexuscrm/catalog \
  --mount type=bind,src="$L1_EVIDENCE",dst=/output \
  "$CONTROLPLANE_IMAGE" \
  -config /config.json -output /output/catalog-output.json
```

The renderer is deployment orchestration: L2 supplies values and depends on
the published L1/L0 schemas. It does not make L0 or the standalone L1 release
depend on an L2 BOM. The rendered input and output are immutable evidence and
must be retained together.

The current V1 does not support attaching bindings to an already installed
same AppRelease by pretending that it is a same-release `UPGRADE`. Use one of
the two supported stopped paths:

- for a fresh D3 candidate, include the reviewed L2 binding document in the
  same initial neutral Catalog `PREPARE` that creates the AppInstallation;
- for an existing installation, publish a genuinely new AppRelease through
  `UPGRADE` and include the reviewed bindings in that complete revision.

The fresh D3 orchestrator may compose L1 release facts with L2 binding values
for this one publication. That does not make the standalone L1 installer read
or depend on a D3 BOM.

The resulting serving authority may grant:

```text
catalog.consumer.integration-binding-view.v1
runtime.commands.accept.v1
```

only when the same snapshot contains the corresponding binding and command
acceptance projection. An empty binding set remains fail closed and carries no
`AcceptCommand` grant.

Concrete PingCAP/provider binding values are L2 inputs. Their publication may
select an L1 capability, but the base L1 installer remains independent of the
L2/D3 BOM.

### 6.2 Start the Integration Service

Provision the Integration process with:

- the one private L0 gRPC target and TLS CA;
- the exact AppInstallation ID;
- its L0-issued `integration-service` workload token;
- its own provider ingress and provider credential store;
- the exact released Integration Service image digest.

Do not give Integration Runtime database, Catalog-store, or Worker credentials.
The request path must be:

```text
provider webhook
  → Integration Service
  → read its scoped IntegrationBindingView
  → L0 AcceptCommand
  → Runtime command Worker
```

Provider external effects remain in Integration. The Runtime Worker must not
import provider-specific code or credentials.

## 7. Activate Worker ownership and start serving L0

Before activation, publish ownership for all seven work families with the same
exact Controlplane image. Each role uses its own reviewed activation input and
receives an ownership receipt:

```bash
docker run --rm \
  --env-file "$DB_ENV" \
  --add-host host.docker.internal:host-gateway \
  --mount type=bind,src="$ROLE_ACTIVATION",dst=/config.json,readonly \
  --mount type=bind,src="$ROLE_OUTPUT_DIR",dst=/output \
  "$CONTROLPLANE_IMAGE" \
  work-ownership -config /config.json -output "/output/$ROLE"
```

The seven work families are:

1. `runtime.audit-history-fanout.v1`
2. `runtime.file-upload-lifecycle.v1`
3. `runtime.platform-command.v1`
4. `runtime.record-command-lifecycle.v1`
5. `runtime.record-event-dispatch.v1`
6. `runtime.rollup-delta.v1`
7. `runtime.scheduled-flow.v1`

Use one Runtime Worker image for all roles. Worker configs must carry the exact
Worker release identity and ownership epoch. App-scoped roles must carry the
new AppInstallation ID; scheduled Flow must also carry the compatibility
receipt for the exact `io.nexuscrm.sfcrm.flow-actions.v1` selection.

The L1 installer does not mint Worker tokens or own generic Worker templates.
Those remain L0 release artifacts. The installation operation only supplies
the new AppInstallation and capability receipts needed to close the
app-scoped fields in those templates.

Once the complete state root is ready, activate:

```bash
python3 infra/runtime/deploy.py activate \
  --database-mode local \
  --database-env-file "$DB_ENV" \
  --state-root "$L0_STATE" \
  --runtime-api-image "$RUNTIME_API_IMAGE" \
  --runtime-worker-image "$RUNTIME_WORKER_IMAGE" \
  --project "$L0_PROJECT" \
  --runtime-grpc-port "$L0_GRPC_PORT" \
  --timeout 180
```

Current V1 activation is intentionally stopped: `activate` removes bootstrap,
starts Runtime API and all seven Worker containers, and waits for health. It is
not rolling activation. [Issue #4255](https://github.com/pingcap-inc/NexusCRM/issues/4255)
owns the future online transition; do not build a second hot-reload path here.

## 8. Restart and upgrade rules

### Restart with no metadata or version change

Use `activate` again with the same state root, project, port, and exact image
digests. Do not call raw Compose with an incomplete environment. Runtime
revalidates the immutable state and all roles start against the same authority.

### Restart only an L1 service

Restart sfcrm or Integration independently when only its process changes and
its declared L0 contract/capability range remains compatible. That does not
require replacing Runtime API or Worker images.

### Metadata or AppRelease upgrade

For the current stopped V1:

1. stop/drain L1 ingress and the L0 serving composition;
2. run a Controlplane `UPGRADE` for a genuinely new AppRelease using the exact
   current AppInstallation ID, Catalog revision, and SecurityEpoch, and include
   any reviewed binding changes in that complete new revision;
3. generate a complete new receipt set, serving config, and affected Worker
   configs;
4. activate the unchanged or intentionally upgraded exact L0 release;
5. start L1 services with the new AppInstallation/revision authority.

Do not use `UPGRADE` to re-run the same AppRelease merely because a local
profile or command changed. The current Catalog transition rejects this, and
bypassing it would mix two authority definitions under one release identity.

### L0 binary upgrade

During the maintenance window, drain and stop Runtime/Workers, apply the
release-owned L0 migration, then activate the new Runtime API and Worker images
from one exact L0 release. Do not force an L1 rebuild when its neutral contract
range remains compatible.

## 9. Catalog, database, and AppInstallation collision rules

The TiDB authority rows, Catalog artifact store, serving config, Worker
receipts, and workload identities are one deployment state. Mixing any of them
must fail closed.

| Situation | Correct action |
|---|---|
| New empty database | `init-l0`, then `PREPARE` a new AppInstallation |
| Existing AppInstallation, new AppRelease | stopped `UPGRADE` with exact current revision/epoch |
| Same release already installed | reuse it; do not issue another `PREPARE` or fake `UPGRADE` |
| Stale expected revision/epoch | stop; reread authoritative receipt; never guess or increment manually |
| Database reset but old Catalog store remains | invalid mixed state; reset/archive both as one local operation |
| Catalog store reset but database authority remains | invalid mixed state; restore matching store or use a reviewed upgrade |
| Receipt/token copied from another database | reject; provision authority for this deployment |
| Empty Integration bindings | keep IntegrationBindingView scoped but omit `AcceptCommand` |
| Runtime reports missing installation/capability projection | serving grant and immutable snapshot disagree; republish correctly, do not suppress validation |

For a disposable local clean install only:

1. `deploy.py down` the exact Compose project;
2. verify the exact database name before dropping it;
3. archive or remove the matching Catalog store together with that database;
4. recreate the database and run `init-l0`;
5. run a new `PREPARE` and regenerate all ownership/serving receipts.

Never reset a shared test, staging, or production database this way. Production
uses a reviewed stopped upgrade or whole-deployment rollback.

Preserving L0 signing keys during a disposable metadata reset is valid only
when it is intentionally the same L0 deployment. Never copy secrets between
clusters or expose them in evidence logs.

## 10. Verify the running topology

```bash
python3 infra/runtime/deploy.py status --project "$L0_PROJECT"
```

The activated L0 project must contain exactly:

- one Runtime API container;
- seven role-scoped containers using one Runtime Worker image;
- one private TLS gRPC endpoint;
- no sfcrm, Integration, frontend, Gateway, legacy process, or TiDB container.

Then test through the owning L1 surfaces:

- sfcrm readiness and OAuth;
- a selected-root/User query;
- Account Describe;
- Account create/read/update/query/delete and update visibility;
- for a real Integration binding, binding-view read, webhook acceptance,
  `AcceptCommand`, command completion/replay, and external-effect ownership.

Also verify exact image digests, AppInstallation ID, Catalog revision,
SecurityEpoch, snapshot digest, and all seven ownership epochs. A health check
alone is not sufficient evidence for an L1 journey.

Stop the local L0 project with:

```bash
python3 infra/runtime/deploy.py down --project "$L0_PROJECT"
```

## 11. Common failure meanings

- `serving config v4 requires ... multiple explicit workloads`: the base
  serving authority omitted one of the explicit sfcrm/Integration workload
  identities. Restore the reviewed workload set; do not grant extra business
  capability merely to satisfy the shape.
- `installation/capability scope is absent`: a serving grant (commonly
  `AcceptCommand`) points at a snapshot that did not publish that projection.
  Remove the premature grant for an empty binding set or publish the real
  binding through Controlplane.
- setup connects to `host.docker.internal` from the host and fails DNS: use the
  L1 installer that loads the same database env and normalizes only this Docker
  Desktop host alias. Do not maintain two unrelated database env files.
- Catalog rejects same-release `UPGRADE`: the operation is wrong. Reuse the
  existing release, publish a genuinely new AppRelease that includes the
  binding changes, or reset the exact disposable local database and Catalog
  store together.
- Runtime/Worker is healthy but sfcrm returns 503: test the L1 consumer against
  the exact Catalog projection. Do not infer business readiness from container
  health.

## 12. Prohibited shortcuts

- no L2/D3 BOM input to the standalone L1 installer;
- no L1 implementation module in Runtime;
- no Salesforce or provider semantics in neutral L0 contracts;
- no direct Catalog/metadata row edits;
- no repository/global metadata fallback;
- no stale receipt, token, snapshot, or Worker config reuse across databases;
- no mutable image tags for a candidate/release;
- no generic Gateway or second L0 serving endpoint;
- no long-lived dual-read, dual-owner, or rolling compatibility path for this
  stopped-upgrade V1.