# Staging Blue-Green Deployment Plan

> Historical planning record. The current topology authority is
> `deployments/deployment-config.json`: Formal Staging uses `bluegreen`, Test
> uses the fixed `test1`/`test2`/`test3` `multi_slot` topology, and the retired
> regression-only deployment target is absent. The tables and rollout steps below
> describe earlier plans and must not be used as current operator instructions.

## Purpose

This document records the target directory model and deploy boundaries for the
shared EC2 host deployment framework. Blue-green deployment is one supported
mode, not a mandatory mode for every environment. The immediate goal is to stop
mixing live runtime state, deploy artifacts, repo checkouts, and one-off ops
fixes in ad hoc locations.

## Scope

This plan currently focuses on the offline EC2 environments:

1. `staging`
2. `exp`
3. `test`

It assumes:

1. one shared EC2 host
2. one shared Nginx process on that host
3. per-environment deployment mode selection
4. slot-local runtime assets when blue-green is enabled
5. environment-level switch-over through Nginx when blue-green is enabled

## Deployment Modes

The framework should support more than one environment deployment mode.

### Mode A: single-slot environment

Use this when an environment does not need blue-green switch-over.

Characteristics:

1. one runtime slot
2. one backend service
3. one database target
4. direct in-place deploy

Example shape:

```text
~/nexuscrm-deployment/offline/envs/<env>/
  single/
    env/
    repos/
    frontend-dist/
    frontend-dist-build/
    logs/
    run/
    runtime/
    artifacts/
      deploys/<deploy-id>/
```

### Mode B: blue-green environment

Use this when an environment needs standby deploy plus switch-over.

Characteristics:

1. `blue` and `green` slots
2. active-slot state
3. standby deploy
4. switch-over after verification
5. rollback-to-previous-slot capability

Hard rule:

1. blue-green is an option in the framework
2. not every environment is required to use it
3. each environment must declare which mode it uses

## Environment Mapping Table

This table is the planning contract for how each environment maps to machine
class, deployment mode, and database isolation.

| Environment | Machine Class | Deployment Mode | Runtime Targets | Database Rule | Notes |
| --- | --- | --- | --- | --- | --- |
| `staging` | `offline` | `single` | `single` | TiDB branch-backed, one branch target | main validation environment |
| `test` | `offline` | `bluegreen` | `blue`, `green` | TiDB branch-backed, two branch targets, one per slot | integration validation with low-disruption switch-over |
| `ci` | `offline` | `single` | `single` | TiDB branch-backed, one branch target | pack gate environment, not a business-demo surface |
| `exp` | `offline` | `single` | `single` | local TiDB, one dedicated database | sandbox-style fast visual verification |
| retired regression environment | `offline` | `single` | `single` | TiDB branch-backed, one dedicated branch target | historical automated regression target |
| `<env>` | `offline` | `single` | `single` | one DB/branch target | for environments that do not need switch-over |
| `<env>` | `online` | `single` or `bluegreen` | mode-specific | environment-specific | decided by online host requirements |

Planning rules:

1. this table is authoritative for whether an environment uses `single` or `bluegreen`
2. offline blue-green means two runtime targets and two database targets
3. an environment must not implicitly switch mode without updating this contract
4. offline blue-green is used only where business testing continuity matters
5. current offline policy:
   - `test` uses blue-green to avoid integration-test interruption
   - `staging`, `ci`, `exp`, and the regression environment use single-slot deployment
6. current offline database policy:
   - `staging`, `test`, `ci`, and the regression environment use TiDB branch-backed targets
   - `exp` uses local TiDB with a dedicated per-environment database

## Current Model

Today each environment is effectively a single-root deployment.

Example:

```text
~/nexuscrm-deployment/staging.ec2.env
~/nexuscrm-deployment/offline/envs/staging/
  NexusCRM/
  nexuscrm-pingcap-pack/
  nexuscrm-app-packs/
  frontend-dist/
  frontend-dist-build/
  logs/
  run/
```

Current staging deploy characteristics:

1. one env file per environment
2. one env root per environment
3. one backend port per environment
4. one systemd service per environment
5. one Nginx vhost per environment
6. no active/standby slot state
7. no isolated standby repo/runtime directory

## Current Staging Bindings

Current staging binds these values directly:

```text
env file:        ~/nexuscrm-deployment/staging.ec2.env
env root:        ~/nexuscrm-deployment/offline/envs/staging
systemd service: nexuscrm-staging-backend.service
nginx conf:      /etc/nginx/conf.d/nexuscrm-staging.conf
frontend root:   ~/nexuscrm-deployment/offline/envs/staging/frontend-dist
backend port:    3101
```

Operational consequence:

1. deploy updates the same repo directories every time
2. deploy overwrites the same frontend root every time
3. deploy restarts the live backend service directly
4. no clean way exists to prepare a standby slot first and cut over later

## Target Model

The target model should support both runtime shapes:

1. blue-green environments with `blue` and `green`
2. single-slot environments with `single`

Target layout:

```text
~/nexuscrm-deployment/offline/envs/
  nginx/
    conf.d/
      staging.conf
      exp.conf
      test.conf
    state/
      staging.active-slot
      exp.active-slot
      test.active-slot

  staging/
    blue/
      env/
      repos/
        NexusCRM/
        nexuscrm-pingcap-pack/
        nexuscrm-app-packs/
      frontend-dist/
      frontend-dist-build/
      logs/
      run/
      runtime/
      artifacts/
        deploys/<deploy-id>/
    green/
      env/
      repos/
        NexusCRM/
        nexuscrm-pingcap-pack/
        nexuscrm-app-packs/
      frontend-dist/
      frontend-dist-build/
      logs/
      run/
      runtime/
      artifacts/
        deploys/<deploy-id>/
    shared/
      deploy/
        history/
      systemd/

  exp/
    single/
      env/
      repos/
      frontend-dist/
      frontend-dist-build/
      logs/
      run/
      runtime/
      artifacts/
        deploys/<deploy-id>/
    shared/

  test/
    blue/
    green/
    shared/

  ci/
    single/
      env/
      repos/
      frontend-dist/
      frontend-dist-build/
      logs/
      run/
      runtime/
      artifacts/
        deploys/<deploy-id>/
    shared/

  regression/
    single/
      env/
      repos/
      frontend-dist/
      frontend-dist-build/
      logs/
      run/
      runtime/
      artifacts/
        deploys/<deploy-id>/
    shared/
```

## Shared vs Isolated Resources

### Shared across the host

These should remain single-instance host resources:

1. Nginx process
2. `/etc/nginx/conf.d/` as the live load path
3. system package/toolchain installation
4. shared Docker TiDB instance for local-db environments, if still used

### Shared per environment

These should be shared only at the environment level:

1. environment host/domain contract
2. active slot marker
3. rendered Nginx env config source
4. deploy history index
5. optional environment-wide rollback metadata

Example:

```text
~/nexuscrm-deployment/offline/envs/nginx/state/staging.active-slot
~/nexuscrm-deployment/offline/envs/nginx/conf.d/staging.conf
~/nexuscrm-deployment/offline/envs/staging/shared/deploy/history/
```

### Isolated per slot

These must be isolated between slots when an environment uses blue-green:

1. downstream repo checkout
2. upstream repo checkout
3. official app packs checkout
4. frontend build output
5. backend runtime worktree/runtime dir
6. logs
7. pid/run files
8. per-deploy artifacts
9. slot env file
10. slot backend port
11. slot systemd unit

## Nginx Arrangement

The host should run one shared Nginx instance.

It should not run one Nginx process per environment or per slot.

Recommended structure:

```text
~/nexuscrm-deployment/offline/envs/nginx/conf.d/staging.conf
~/nexuscrm-deployment/offline/envs/nginx/conf.d/exp.conf
~/nexuscrm-deployment/offline/envs/nginx/conf.d/test.conf

~/nexuscrm-deployment/offline/envs/nginx/state/staging.active-slot
~/nexuscrm-deployment/offline/envs/nginx/state/exp.active-slot
~/nexuscrm-deployment/offline/envs/nginx/state/test.active-slot
```

Live Nginx should still load from:

```text
/etc/nginx/conf.d/nexuscrm-staging.conf
/etc/nginx/conf.d/nexuscrm-exp.conf
/etc/nginx/conf.d/nexuscrm-test.conf
```

But those files should no longer be treated as the source of truth. The source
of truth should live under `~/nexuscrm-deployment/offline/envs/nginx/`, and the `/etc/nginx`
files should be synced from there.

For blue-green environments, per-environment switch-over should only change:

1. frontend root path
2. backend upstream port

Example idea:

```nginx
root ~/nexuscrm-deployment/offline/envs/staging/blue/frontend-dist;
proxy_pass http://127.0.0.1:3101;
```

Switching to green changes only those environment-local targets.

## Systemd Arrangement

Systemd should remain a host-level service manager.

For blue-green environments, each slot should own a separate backend unit.

Recommended naming:

1. `nexuscrm-staging-blue-backend.service`
2. `nexuscrm-staging-green-backend.service`
3. `nexuscrm-exp-blue-backend.service`
4. `nexuscrm-exp-green-backend.service`
5. `nexuscrm-test-blue-backend.service`
6. `nexuscrm-test-green-backend.service`

This is necessary so the standby slot can be deployed and verified without
restarting the live slot.

For single-slot environments, one backend unit is sufficient.

## Environment File Model

Current env files are single-instance:

```text
~/nexuscrm-deployment/staging.ec2.env
~/nexuscrm-deployment/exp.ec2.env
~/nexuscrm-deployment/test.ec2.env
```

Target model should move to mode-specific env files.

For blue-green:

```text
~/nexuscrm-deployment/offline/envs/staging/blue/env/runtime.env
~/nexuscrm-deployment/offline/envs/staging/green/env/runtime.env
~/nexuscrm-deployment/offline/envs/exp/blue/env/runtime.env
~/nexuscrm-deployment/offline/envs/exp/green/env/runtime.env
```

For single-slot:

```text
~/nexuscrm-deployment/offline/envs/<env>/single/env/runtime.env
```

The env file should carry mode-specific runtime values such as:

1. deployment mode
2. slot name
3. backend port
4. resolved repo refs
5. database or branch target
6. deploy id

## Database Modes

The deploy model needs to support two different environment types.

### Local database environments

Examples:

1. `exp`
2. possibly a regression sandbox

These currently use local TiDB and separate databases inside one shared TiDB
instance.

If the environment uses blue-green:

1. each slot must use its own database name
2. blue and green must not share one database
3. offline blue-green is defined as dual-environment plus dual-database, not app-only isolation

This is a hard rule for the offline deployment model.

If the environment uses single-slot:

1. one environment maps to one database target
2. no standby slot is required

### Branch-backed database environments

Examples:

1. `staging`
2. `test`

If the environment uses blue-green:

1. each slot should eventually own its own TiDB Cloud branch
2. deploying the standby slot should reset or reconcile only that slot branch
3. blue and green must not share one branch if we want true data isolation

For the offline deployment framework, the intended rule is stricter:

1. blue and green are two different environments
2. each environment owns its own database target
3. shared-database blue-green is not an accepted target state

If the environment uses single-slot:

1. one environment maps to one branch/database target
2. branch reset or reconcile is applied in place

## Deploy Identity and Artifacts

Each deployment should emit a stable deploy id and write artifacts under the
runtime target that was deployed.

Recommended location for blue-green:

```text
~/nexuscrm-deployment/offline/envs/staging/green/artifacts/deploys/<deploy-id>/
```

Recommended location for single-slot:

```text
~/nexuscrm-deployment/offline/envs/<env>/single/artifacts/deploys/<deploy-id>/
```

Recommended records:

1. requested refs
2. resolved refs
3. target slot
4. database/branch target
5. start time
6. end time
7. health result
8. pack verification result
9. pack missing-items report

This is important because current troubleshooting is too dependent on mixed logs
and prior failed runs in shared locations.

## Pack Verification Requirement

Post-deploy verification should not stop at:

1. backend `/health`
2. Nginx `/health`

It also needs a deploy artifact that records:

1. pack apply/import outcome
2. objects or metadata that failed to load
3. missing items that match the current downstream pack issue set

That report should be written into the slot-local deploy artifact directory so
old failed runs do not pollute new analysis.

## Current Gaps

### Gap 1: directory model

Current scripts assume one `ENV_ROOT` per environment, not one root plus
`blue|green` slots.

### Gap 2: live-service coupling

Current deploy restarts the live environment service directly. There is no
standby-first workflow.

### Gap 3: no deployment-mode state

There is no current concept of:

1. environment deployment mode
2. active slot
3. standby slot
4. switch-over
5. rollback target

### Gap 4: Nginx source of truth

Current deploy writes straight to `/etc/nginx/conf.d/nexuscrm-${ENV_NAME}.conf`
instead of treating an env-local rendered config as the authoritative source.

### Gap 5: systemd service shape

Current service name is per environment, with no mode-aware layout.

### Gap 6: database isolation

Current branch/db selection is per environment, not per runtime target.

### Gap 7: deploy records

Current deploy does not persist a clear deploy id and per-run artifact bundle.

### Gap 8: pack validation

Current deploy health checks do not produce a standardized missing-items report
for pack import/apply failures.

## Recommended Framework Split

Do not revive or replace current runtime entrypoints with new special cases in the retired `deploy_ec2_env.sh` shape.

Recommended new deployment framework area:

```text
deployments/
  offline/
    modes/
      bluegreen/
        lib.sh
        detect_standby_slot.sh
        prepare_slot_env.sh
        init_slot_repos.sh
        deploy_slot.sh
        verify_slot_pack.sh
        switch_env_slot.sh
        status.sh
      single/
        prepare_env.sh
        deploy_env.sh
        verify_env.sh
        status.sh
    nginx/
    systemd/
    env/
    runbooks/
  online/
    nginx/
    systemd/
    env/
    runbooks/
```

Responsibility split:

1. detect environment deployment mode
2. prepare runtime-target-local env and deploy id
3. update runtime-target-local repo checkouts to requested refs
4. for blue-green, deploy only into the standby slot
5. verify runtime health and pack completeness
6. for blue-green, switch Nginx only after standby verification passes
7. preserve rollback-to-previous-slot information when blue-green is enabled

## Repo-Managed Framework vs Host-Local State

The deployment framework itself should be versioned in the repo.

That is required so:

1. remote hosts can refresh deploy behavior with `git pull`
2. deployment logic stays reviewable and reproducible
3. machine replacement does not require rebuilding ad hoc shell scripts by hand

### Should live in the repo

These should be committed and reviewed:

1. blue-green deploy scripts
2. Nginx templates
3. systemd templates
4. env examples
5. deploy workflow definitions
6. pack verification scripts
7. runbooks and directory contracts

Recommended areas:

```text
deployments/offline/
deployments/online/
docs/
.github/workflows/
```

Boundary rule:

1. `deployments/offline/` is for staging/exp/test style environments on the offline EC2 host
2. `deployments/online/` is for prod-like or other online hosts
3. keep these trees separated because they may be deployed and refreshed on different machines
4. deployment mode is a per-environment choice inside each tree, not a global default

### Should stay local on the host

These should not be committed:

1. real env files with secrets
2. TiDB passwords and API private keys
3. GitHub tokens
4. active-slot state files
5. rendered runtime logs
6. per-deploy artifacts
7. `/etc/nginx/conf.d/` live symlink/copy outputs
8. `/etc/systemd/system/` installed unit files

Examples:

```text
~/nexuscrm-deployment/offline/envs/nginx/state/*.active-slot
~/nexuscrm-deployment/offline/envs/<env>/<slot>/env/runtime.env
~/nexuscrm-deployment/offline/envs/<env>/<slot>/artifacts/deploys/<deploy-id>/
```

### Recommended remote operational model

Keep one repo checkout on the remote host whose job is to hold the latest
deployment framework code.

Example:

```text
~/nexuscrm-deployment
```

That checkout should not be treated as the live runtime directory for any slot.
Its job is to expose the repo-managed `deployments/` framework on the remote
host.

Practical note:

1. Git does not literally check out only one directory by itself
2. the remote host should either keep a dedicated framework checkout of this repo
3. or use sparse-checkout so the working tree is centered on `deployments/`

Its job is:

1. pull the latest deploy framework code
2. render templates
3. prepare slot-local env/runtime directories
4. run deploy and switch commands

The actual runtime state should live under the environment directories:

```text
~/nexuscrm-deployment/offline/envs/staging/blue/
~/nexuscrm-deployment/offline/envs/staging/green/
~/nexuscrm-deployment/offline/envs/exp/blue/
~/nexuscrm-deployment/offline/envs/exp/green/
~/nexuscrm-deployment/offline/envs/<env>/single/
```

### Operational consequence

After the framework is repo-managed, remote refresh should become:

```bash
cd ~/nexuscrm-deployment
git fetch origin
git checkout origin/main
```

Then deploy through the versioned `deployments/` entrypoints instead of editing
remote one-off scripts in place.

### Workflow alignment target

The target state is:

1. each remote machine keeps the deployment-framework checkout it needs
2. offline-host environment build/switch logic is entered through `deployments/offline/`
3. online-host environment build/switch logic is entered through `deployments/online/`
4. GitHub Actions workflows call the matching `deployments/...` entrypoint path
5. local manual ops and remote workflow ops use the same framework location

This means the workflow contract should eventually look like:

```text
repo path: deployments/offline/... or deployments/online/...
remote framework checkout: ~/nexuscrm-deployment
workflow entrypoint: same deployments/... path inside that checkout
```

So a framework change should be publishable by:

1. merge framework changes
2. remote host refreshes the framework checkout
3. workflow or operator invokes the same `deployments/...` entrypoint

## Evolution Path

The deployment framework should evolve in phases instead of combining runtime
re-architecture and deployment-structure cleanup in one step.

### Phase 1: deployment framework cleanup

This is the current priority.

Goals:

1. move the framework under `deployments/`
2. separate `offline/` and `online/`
3. make deployment mode explicit per environment
4. make `staging` and `test` blue-green
5. keep `ci`, `exp`, and the regression environment single-slot
6. separate framework checkout from runtime directories
7. define env-local state, deploy artifacts, and switch-over boundaries
8. make workflow entrypoints align to `deployments/...`

Non-goals:

1. full containerization
2. full online-host redesign
3. replacing every legacy script immediately

### Phase 2: runner contract normalization

After the framework shape is stable, normalize the runtime contract so every
deployment mode uses the same high-level lifecycle.

Suggested lifecycle:

1. prepare
2. build
3. start
4. verify
5. switch
6. rollback

This step should reduce how much mode-specific logic is embedded in ad hoc
shell flow.

### Phase 3: selective containerization trial

Containerization is a later optimization, not the first migration step.

Recommendation:

1. do not couple blue-green rollout with immediate full Docker migration
2. try containerization first on simpler single-slot offline environments
3. prefer `exp` or a regression sandbox as the first trial target

Reason:

1. current pain is framework sprawl, not only process startup style
2. `staging` and `test` already carry blue-green plus branch-backed complexity
3. `exp` and a regression sandbox are safer places to validate a container runner model

### Phase 4: optional broader runtime migration

If the selective trial works, then decide whether:

1. more offline environments should adopt containers
2. online environments should share the same runner contract
3. systemd-managed and container-managed modes should coexist long-term or converge

## Migration Direction

Suggested implementation order:

1. introduce the directory model and deployment mode declaration
2. introduce runtime-target-local env files and systemd units
3. introduce env-level active-slot state and Nginx render/switch logic for blue-green environments
4. run deploys into the correct runtime target for each mode
5. add runtime-target-local pack verification artifacts
6. move branch/db ownership from env-level to runtime-target level

This allows us to improve deploy safety first, then close the data-isolation gap.

## Phase 1 Checklist

This checklist is the concrete implementation scope for the first phase.

### 1. Framework skeleton

1. create `deployments/offline/` as the authoritative offline framework root
2. split offline modes into:
   - `deployments/offline/modes/bluegreen/`
   - `deployments/offline/modes/single/`
3. create shared offline support areas:
   - `deployments/offline/env/`
   - `deployments/offline/nginx/`
   - `deployments/offline/systemd/`
   - `deployments/offline/runbooks/`

### 2. Environment contract

1. define offline environment mode mapping
2. define branch-backed vs local-TiDB mapping
3. define runtime target naming:
   - `blue`
   - `green`
   - `single`
4. define host-local state file layout

### 3. Blue-green implementation scope

This phase should implement blue-green only for:

1. `staging`
2. `test`

Required blue-green capabilities:

1. detect current active slot
2. compute standby slot
3. prepare standby env/runtime target
4. deploy only into standby
5. verify standby health
6. switch Nginx to standby
7. preserve rollback metadata

### 4. Single-slot implementation scope

This phase should keep single-slot deployment for:

1. `ci`
2. `exp`
3. the regression environment

Required single-slot capabilities:

1. prepare env/runtime target
2. deploy in place to `single`
3. verify health
4. emit deploy artifacts

### 5. Workflow alignment

1. keep existing workflows working during migration
2. introduce new `deployments/offline/...` entrypoints first
3. move workflow calls to the new entrypoints after the framework skeleton exists
4. do not duplicate business logic in workflow YAML

### 6. Legacy compatibility

1. do not delete legacy `scripts/` entrypoints in the first step
2. allow legacy scripts to remain as wrappers during migration
3. move logic gradually into `deployments/`
4. remove legacy entrypoints only after workflow cutover and host verification

### 7. Explicit non-goals for phase 1

1. full Docker migration
2. full online-host migration
3. replacing every old offline script in one pass
4. redesigning pack build logic end to end