# Helix Deployment Guide > **Shared Staging is a separate deployment path.** Its daily/manual Alibaba > Cloud workflow, required bootstrap and operational boundaries are documented > in [Shared Staging deployment](staging.md). It is disabled until that > bootstrap completes and must not reuse the production AWS instructions below. ## Architecture Overview ``` ┌──────────────┐ Internet ──────┤ AWS ALB │ (TLS termination, ACM cert) │ port 443 │ └──────┬───────┘ │ port 80 ┌──────┴───────┐ │ EC2 Host │ │ │ │ ┌─────────┐ │ │ │frontend │ │ ← nginx, serves React app │ │ :80 │ │ proxies /api/v1 → backend:8000 │ └─────────┘ │ │ ┌─────────┐ │ │ │backend │ │ ← FastAPI (uvicorn) │ │ :8000 │ │ │ └─────────┘ │ │ ┌─────────┐ │ │ │worker │ │ ← Background task processor │ └─────────┘ │ └──────┬───────┘ │ ┌────────────┼────────────┐ │ │ │ ┌──────┴──────┐ ┌──┴───┐ ┌──────┴──────┐ │ TiDB Cloud │ │Redis │ │ AWS SSM │ │ Dedicated │ │(EC) │ │ Parameters │ │ PrivateLink │ │ │ │ │ └─────────────┘ └──────┘ └─────────────┘ ``` ## Prerequisites 1. AWS account with Terraform state bootstrap completed (`infra/aws/bootstrap/`) 2. TiDB Cloud Dedicated cluster in ap-southeast-1 3. Domain name with Route53 hosted zone 4. GitHub repository with Actions enabled 5. GitHub production deployment controls in place (see "Production deployment controls" below): `main`-only deployment branch policy + `Protect production main` ruleset (PR review gate) + PR #2314 IAM trust boundary ## Infrastructure Setup ### 1. Bootstrap Terraform Backend ```bash cd infra/aws/bootstrap terraform init terraform apply # Note the outputs: state_bucket, lock_table ``` The commands above are for a genuinely new account only. If the bucket or lock table already exists, do not apply bootstrap from local or empty state. Adopt the existing controls into the dedicated bootstrap key by following the [AWS Terraform state adoption runbook](../ops/aws-terraform-state-adoption.md). ### 2. Deploy AWS Infrastructure ```bash set -euo pipefail cd infra/aws/environments/prod cp terraform.tfvars.example terraform.tfvars # Edit terraform.tfvars with real values # Required before any backend initialization, normal plan, or apply. This is # read-only and fails closed when canonical state or a protection is absent. python3 ../../../../scripts/terraform_state_backend_preflight.py \ --bucket \ --lock-table terraform init \ -backend-config="bucket=" \ -backend-config="key=infra/aws/environments/prod/terraform.tfstate" \ -backend-config="region=ap-southeast-1" \ -backend-config="dynamodb_table=" \ -backend-config="encrypt=true" terraform plan terraform apply ``` Do not continue to `terraform plan` or `terraform apply` when the preflight fails. In particular, an absent state object is an adoption incident, not a new environment. Follow the [AWS Terraform state adoption runbook](../ops/aws-terraform-state-adoption.md); the preflight never creates an empty state object. ### 3. Deploy TiDB Cloud Cluster + PrivateLink ```bash cd infra/tidb-cloud # Set required variables: tidb_project_id, tidb_root_password, vpc_id, # private_subnet_ids, db_security_group_id (from AWS infra outputs) terraform init -backend-config=... terraform plan terraform apply # Creates: TiDB cluster → PrivateLink service → AWS VPC endpoint → # connection acceptance → Route53 private DNS (fully automated) ``` ### 4. Set SSM Secrets After infrastructure is up, populate secrets via AWS Console or CLI: ```bash # These are created by Terraform with placeholder values — update them: aws ssm put-parameter --name /helix/prod/DATABASE_URL --value "..." --type SecureString --overwrite aws ssm put-parameter --name /helix/prod/JWT_SECRET_KEY --value "..." --type SecureString --overwrite aws ssm put-parameter --name /helix/prod/GOOGLE_CLIENT_ID --value "..." --type SecureString --overwrite ``` ## EC2 First-Time Setup SSH into the EC2 instance (via SSM Session Manager): ```bash aws ssm start-session --target # Run bootstrap (installs Docker, creates directories, configures systemd) sudo /opt/helix/scripts/bootstrap-ec2.sh # Fetch secrets and render .env sudo /opt/helix/scripts/render-env.sh ``` Deploy scripts and compose files are delivered to EC2 via S3 sync during each deployment. The `deploy-ec2.yml` workflow uploads `deploy/` to S3, then SSM syncs them to `/opt/helix/` before running `deploy.sh`. For the initial deployment, trigger the workflow or manually sync: ```bash aws s3 sync s3:///deploy-assets/scripts/ /opt/helix/scripts/ aws s3 sync s3:///deploy-assets/compose/ /opt/helix/compose/ chmod +x /opt/helix/scripts/*.sh sudo /opt/helix/scripts/deploy.sh main ``` ## Deployment (CI/CD) ### Production deployment controls (human gate) Production deploys are gated by defence-in-depth. The human-approval gate is enforced at **merge-to-`main`**, not at the GitHub Environment layer, because environment-level *required reviewers* are a paid-plan feature the current GitHub billing plan does not provide (`PUT .../environments/production` returns HTTP 422 "billing plan does not support the required reviewers protection rule"). The effective, verified controls are: 1. **Merge review gate** — repository ruleset `Protect production main` (`enforcement: active`) requires, for `refs/heads/main`: a pull request with ≥1 approving review, `require_last_push_approval`, `dismiss_stale_reviews_on_push`, required status checks (Backend, Frontend, CLI), squash-only merges, linear history, and no deletion/force-push. Nothing reaches `main` without a human approval on the PR. 2. **Deploy-branch policy** — the `production` environment allows deployments from **`main` only** (one exact `branch`-type policy named `main`, no wildcard). Feature branches cannot deploy. 3. **IAM trust boundary** (PR #2314) — the AWS production roles are assumable only when the OIDC claim is `environment:production` **and** `ref:refs/heads/main` **and** the exact workflow claim. This blocks any non-`main` or unapproved workflow identity from obtaining prod credentials, independent of GitHub settings. Because (1) is the approval gate and only reviewed `main` commits can deploy, the production release path cannot run un-reviewed code. If the org later moves to a GitHub plan that supports environment required reviewers, add them as a second, environment-layer gate (self-review allowed for the solo operator); until then (1)+(2)+(3) are the auditable control set. See issue #2418. ### Image publication is a manual dispatch (changed 2026-08-13) Publication is **no longer automatic on push to `main`**. Two separate steps: 1. `go-ci.yml` runs on every push to `main` and validates that immutable SHA (backend, frontend, CLI, Docker context). It publishes nothing and holds no cloud credential — see DECISIONS.md, 2026-08-13. 2. `image-release.yml` is dispatched manually (Actions → "Build & Push Images", run from `main`) and pushes ECR images tagged `sha-` and `main`. It refuses to run off `main`, because the floating `main` tag would otherwise label a feature build as production. So a green `main` does **not** mean a deployable image exists. Check ECR (or the Actions run) before dispatching a deploy; asking `Production Deploy` for a tag that was never pushed fails with `manifest unknown`. > This gap has bitten once: #2646 removed publication from `go-ci.yml` while > `image-release.yml` was still `workflow_call`-only, leaving no trigger at all. > Three `main` commits had no image and nothing reported it. ### Manual production deploy Use the manual workflow for every production deployment, including rollback, retry, or deployment of a newly published image: 1. Go to Actions → "Production Deploy" 2. Enter the image tag. It must be the **full 40-character** commit SHA — `sha-1a2b3c4d5e6f...` (40 hex chars), not a short SHA. A short SHA is not a published tag and the deploy fails with `manifest unknown`. `main` is accepted but resolves to whatever was published last, so prefer the SHA 3. Optionally enter `git_ref` when the deploy scripts/compose files must match an older release commit 4. Click "Run workflow" 5. Deployment runs only for the `main`-tagged image built from a PR-reviewed `main` commit; the approval gate is the merge-to-`main` review (see "Production deployment controls"), and the IAM trust boundary blocks any non-`main` identity from obtaining prod credentials 6. Deployment executes via SSM Run Command ### Manual Deploy via SSH ```bash sudo /opt/helix/scripts/deploy.sh sha-abc1234 ``` ## Rollback ```bash # Rollback to previous release tag sudo /opt/helix/scripts/rollback.sh sha-previous # Or check current release: cat /opt/helix/releases/current ``` **Important**: Migrations are forward-only. Do not write destructive migrations that break the previous application version. If a migration must be rolled back, write a new forward migration. ## Database Migrations Migrations run as a one-off container during deployment (not at app startup): ```bash sudo /opt/helix/scripts/run-migrations.sh sha-abc1234 ``` ## Health Checks ```bash sudo /opt/helix/scripts/healthcheck.sh ``` Checks: - Backend: `GET http://localhost:8000/api/v1/health` - Frontend: `GET http://localhost:80/` ## Monitoring - **Application logs**: CloudWatch Log Groups (`/helix/backend`, `/helix/worker`, `/helix/frontend`) - **EC2 metrics**: CloudWatch default metrics (CPU, network, disk) - **ALB metrics**: Request count, latency, 5xx rate - **Audit trail**: GitHub Actions history + AWS CloudTrail + in-app audit logs ## Production Auth Constraints Production environment **only allows Google SSO**: - `AUTH_PASSWORD_LOGIN_ENABLED=false` in SSM → backend rejects `/auth/login` with 403 - Frontend still renders the password form, but any password login attempt is rejected by the backend - Google OAuth client configured for production domain only - Callback URL: `https:///auth/callback` ## Future: Migration to EKS When the time comes to migrate to Kubernetes: 1. **Images**: Already in ECR — no change needed 2. **Secrets**: Move from SSM to Kubernetes ExternalSecrets operator 3. **Compose → K8s manifests**: Convert docker-compose.prod.yml to Deployments + Services 4. **ALB**: Replace with AWS Load Balancer Controller + Ingress 5. **Redis**: ElastiCache stays as-is, or migrate to Redis on K8s 6. **CI/CD**: Replace `deploy-ec2.yml` with `kubectl apply` or ArgoCD 7. **Migrations**: Run as Kubernetes Job instead of one-off container