# Shared Staging deployment This is the deployment contract for the shared public Staging host. It is a separate Alibaba Cloud release path; it does not call any production AWS ECR, SSM, Terraform, compose files, database, credentials, or worker process. > **Activation state:** the workflow source is present but intentionally > disabled until the one-time bootstrap below is complete. Set the repository > Configuration variable `HELIX_STAGING_AUTO_DEPLOY_ENABLED=true` only after a > successful manual dry run. A disabled workflow emits an explicit green > `Staging deployment disabled` job and makes no cloud call. ## Release contract `.github/workflows/staging-deploy.yml` is **`workflow_dispatch` only**. A run may name one full, 40-character `main` SHA; otherwise it deploys the workflow's exact default branch SHA. > **There is deliberately no daily schedule** (removed 2026-09-06). A run fails > closed unless the isolated Staging database is already at the release's > Alembic head, and `main` lands 3–13 migrations on every working day, so a > nightly trigger was red on every day a migration merged: 4 of its 24 runs > succeeded, all inside one migration-free weekend, and it then failed > identically for 13 consecutive nights while nobody acted on it. **Staging is > only as fresh as the last time somebody dispatched this workflow.** Restoring > a daily cadence is a safety-boundary decision, not a cron change — see the > 2026-09-06 ADR in [DECISIONS.md](../architecture/DECISIONS.md). Before it receives a cloud identity, it proves all of the following: 1. The selected SHA is an ancestor of the current `main` branch. 2. The SHA has exactly one successful GitHub Actions `go-ci / Main` check. 3. The target is the one configured Staging ECS instance and Cloud Assistant is online. It builds `server` and `web` with the immutable local tag `sha-`, saves them as compressed Docker archives, and publishes each archive to a private OSS key derived from its SHA-256 digest. A per-release immutable manifest binds the source SHA to both archive keys and digests. The workflow sends only that bound metadata plus the current reviewed deployment script and non-secret compose file to Cloud Assistant. The host downloads through its ECS RAM role, verifies each archive digest before `docker load`, verifies the OCI revision label equals the source SHA, retains and health-checks the isolated Staging Redis service, then runs exactly `backend` and `frontend`. It also publishes the exact release inventory under a content-addressed `helix-staging/catalogs/sha256/` key. After local readiness, the host verifies that artifact and runs the backend catalog seed with the release image's bundled dbt lineage catalog. This automatically adds every actually referenced non-dbt source dataset to the requestable catalog, defaults missing source owners to `xiaopeng.zhu@pingcap.com`, and preserves explicit existing owners. If an explicit Production owner is absent from the sanitized Staging identity cohort, Staging alone resolves that row to `xiaopeng.zhu@pingcap.com`; the Production workflow never enables this fallback. The seed is transactional and deployment fails if the enabled BigQuery integration is not unique, the Staging fallback identity cannot resolve, or commit evidence is absent. It checks local readiness routes, then writes `/opt/helix/releases/staging-current.json`. The workflow finally checks the public `/api/v1/ready` route. `worker` is deliberately absent. This keeps scheduled jobs and provider effects disabled on shared Staging. Production remains a separately authorized manual AWS path. The deployment seed is metadata-only: it neither calls BigQuery nor reads table or column structure. Catalog-owned dataset rows are idempotently refreshed; Staging-only bootstrap test rows are not deleted. Table lists continue to use the DAX lineage/managed shortlist and bounded Provider pagination, while schema is fetched only after a user clicks a concrete table. ## Activation blocker: sanitised Staging data The current `helix-staging` TiDB Branch began as a full Production snapshot. Being a write-isolated Branch does **not** make copied people, tokens, credentials, integrations or pending executable work safe for authenticated Staging use. Before any deployment is enabled, a separately reviewed, idempotent sanitizer must reset the Branch and write a non-secret receipt below `/opt/helix/releases/staging-sanitization-.json`. The host refuses to pull or replace an image unless the receipt is valid JSON with schema version 1 and proves all of these booleans: scheduled jobs, integrations and identity targets disabled; provider credentials removed; tokens revoked; real provider locators removed; people pseudonymised; and a synthetic cohort seeded. It also binds a full sanitizer source SHA, timestamp, and a database object marked `sanitized_from_snapshot=true`. A hand-written or incomplete JSON file is not an acceptance substitute; verify its database and storage readback against the sanitizer's own contract before naming it in `deploy.env`. The receipt's `database.alembic_revision` must equal the live Branch revision. A receipt from before a migration is stale even when every boolean still says `true`: migrations can seed enabled jobs or other executable state. After that one-time sanitizer, user-approved DAX validation metadata is allowed to persist across deployments through a separate schema-version-2 receipt at `/opt/helix/releases/staging-dax-retained-.json`. This receipt does not pretend the Provider locator or credential was removed. It must explicitly prove `receipt_type=dax_retained_validation`, `retention_policy=preserve_across_deploys`, and `cleanup_required_before_next_deploy=false`; bind the live schema and runtime SHA; retain exactly one metadata-only BigQuery integration; and prove worker, apply mode, Personal apply, Service Account apply, scheduled jobs, and identity targets remain disabled. The legacy `STAGING_SANITIZATION_RECEIPT` variable may point to either the original sanitizer receipt or this reviewed retained-DAX receipt. Any other path or partial receipt fails before artifact download or runtime replacement. Create that receipt only with the reviewed [`reattest-dax-retained-on-ecs.sh`](../../deploy/staging/reattest-dax-retained-on-ecs.sh) helper through Cloud Assistant. It reads only credential presence and bounded safety counts, never the credential value; verifies the exact backend/frontend/ Redis service allowlist, schema, scheduled jobs, identity targets, integration count, and all three apply gates; then writes a new immutable receipt. Supply a new `HELIX_STAGING_DAX_RECEIPT_ID` each time and independently read back the result before pointing `deploy.env` at it. ## Migration and rollback semantics A run never applies database migrations unless it was dispatched with `apply_migrations=true`. It fails closed when the isolated Staging database is not already at the release's Alembic head, and reserves exit code `43` for exactly that state so the workflow can report an awaited-migration precondition instead of a generic deployment failure. A manual dispatch can set `apply_migrations=true`; that is the only path which runs `alembic upgrade head`, and it targets the independent Staging database configured on the host. A migration run writes a successful migration-only receipt and intentionally stops before replacing the application containers after it reaches the new head. The host reserves exit code `42` for this one state; every unexpected command failure is normalized to `1`, and the workflow recognizes `42` from Cloud Assistant without parsing human-readable output. **Re-run the sanitizer** against that exact revision, update `STAGING_SANITIZATION_RECEIPT`, then dispatch again with migrations off. This two-step path prevents a migration that seeds enabled jobs from inheriting a pre-migration safety receipt. > **Re-attestation is not an alternative here.** This document previously > offered "re-sanitize *or* re-attest"; only the first works after a migration. > [`reattest-dax-retained-on-ecs.sh`](../../deploy/staging/reattest-dax-retained-on-ecs.sh) > copies the source receipt's `database` block verbatim, so it cannot attest a > revision the sanitizer never saw: it compares the source receipt's > `alembic_revision` against the live schema and exits `8` whenever Staging has > moved. Re-attestation re-reads live safety state at an **unchanged** schema — > that is its entire purpose. Only a full sanitizer run produces a valid > `staging-sanitization-*.json` at a new revision. If image startup or a local readiness check fails, the host restores the two previous image digests from its receipt. This is a **runtime-only rollback**: it never attempts to reverse a completed database migration. A failed command leaves the workflow failed even if the previous runtime becomes healthy again. ## Identities and trust boundary The GitHub-hosted runner uses a short-lived GitHub OIDC token with audience `sts.aliyuncs.com` to assume the Alibaba RAM role `HelixStagingDeploy`. It has no Alibaba AccessKey, SSH key, application secret, GitHub App key, or production credential. The ECS instance has a separate `HelixStagingRuntime` RAM role. It may read only the private `helix-staging/` OSS prefix. It cannot write objects or call ECS APIs. The deploy script receives no secret: the existing `/etc/helix/staging/runtime.env` stays on the host, is required to be mode `0600`, and must contain only Staging (never production) application values. OSS is intentionally used instead of ACR Enterprise Edition: this account has no existing ACR instance, while Enterprise is a separately subscribed fixed service. For this single-host internal environment, two content-addressed private archives avoid that standing cost without introducing a static cloud key. The objects are uploaded with server-side AES-256 encryption and `x-oss-forbid-overwrite`; a retry may reuse an existing object only after a full download and digest comparison. ## One-time bootstrap (cloud and repository admins) These are the prerequisites for turning the source workflow on. They are not performed by a normal deployment run. 1. Create the GitHub Environment `staging`, with a deployment branch policy for `main` only. No required reviewer is configured. Add the repository Configuration variable `HELIX_STAGING_AUTO_DEPLOY_ENABLED=false`; it is repository-scoped so the job's pre-start `if` can fail closed before an Environment is entered. Add these non-secret **Staging Environment** variables: ```text ALICLOUD_STAGING_REGION=ap-southeast-1 ALICLOUD_STAGING_INSTANCE_ID= ALICLOUD_STAGING_OIDC_PROVIDER_ARN=acs:ram:::oidc-provider/ ALICLOUD_STAGING_DEPLOY_ROLE_ARN=acs:ram:::role/HelixStagingDeploy ALICLOUD_STAGING_OSS_BUCKET= ALICLOUD_STAGING_OSS_ENDPOINT=oss-ap-southeast-1.aliyuncs.com STAGING_GOOGLE_CLIENT_ID= STAGING_PASSWORD_LOGIN_ENABLED=false STAGING_PUBLIC_ORIGIN=https://staging.helix.pingcap.net ``` `STAGING_GOOGLE_CLIENT_ID` is public build configuration, not a client secret. Do not put a provider secret, database URL, API key, or RAM credential in GitHub variables. 2. In the `it-infra` Alibaba account, create an OIDC provider for `https://token.actions.githubusercontent.com`, client ID `sts.aliyuncs.com`. `HelixStagingDeploy` must trust only the GitHub `sub` `repo:pingcap-inc/Helix:environment:staging` and audience `sts.aliyuncs.com`; do not trust a repository wildcard, branch name, or a reusable workflow by itself. 3. Give `HelixStagingDeploy` only these abilities, scoped to the configured region, account, instance and OSS prefix where the action supports resource ARNs: - ECS: `ecs:RunCommand` against the exact Staging instance; read-only `ecs:DescribeInstances`, `ecs:DescribeCloudAssistantStatus`, and `ecs:DescribeInvocations` for polling. - OSS: read/write objects only below `helix-staging/` in the dedicated private bucket. It does not need bucket administration. It must not receive RAM administration, ECS instance lifecycle/network permissions, Secrets Manager, or any production/AWS permission. 4. Create a private Standard/LRS OSS bucket in `ap-southeast-1`. Keep public access disabled and use the `helix-staging/` prefix exclusively for this workflow. Attach `HelixStagingRuntime` to the Staging ECS instance; that role needs only `oss:GetObject` below that prefix. Lifecycle cleanup may expire old archives only after the required rollback window is agreed. 5. On the ECS instance, install Docker Compose, `zstd`, the Alibaba CLI, and the reviewed files: ```text /etc/helix/staging/deploy.env (no secret; mode 0644 or stricter) /etc/helix/staging/runtime.env (Staging-only secrets; mode 0600) /opt/helix/releases/ ``` The workflow delivers `docker-compose.staging.yml` on every deployment; `deploy.env` must set `STAGING_OSS_REGION`, `STAGING_OSS_BUCKET`, `STAGING_OSS_ENDPOINT`, `STAGING_ECS_RAM_ROLE`, `STAGING_DOCKER_NETWORK`, `STAGING_REDIS_VOLUME`, and `STAGING_SANITIZATION_RECEIPT`. The value names the current safety receipt: either the original sanitization receipt or the reviewed schema-version-2 retained-DAX receipt described above. The network and volume must name the existing Caddy-attached Staging network and adopted Redis volume; the host script inspects both and fails rather than creating a disconnected replacement. Do not install the production `deploy.sh`, AWS SSM renderer, production compose file, or worker unit on this host. 6. Run the Staging Branch sanitizer and independently verify its receipt before setting `STAGING_SANITIZATION_RECEIPT`. If DAX validation metadata is later retained, create and independently read back a new version-2 retained-DAX receipt; never rename or weaken the sanitizer receipt. Only then confirm database and Redis isolation, rotate any historically exposed Staging database credential, and separately prove OAuth login plus the business smoke contract with external effects denied. `/api/v1/ready` alone proves only service readiness. 7. Keep `HELIX_STAGING_AUTO_DEPLOY_ENABLED=false` while running a dry preflight. Then manually dispatch `Staging Deploy` for a current green `main` SHA with migrations off. Verify the receipt, public route, image digests, and that no worker is running. Only then set the flag to `true`; every subsequent deployment is an explicit dispatch. ## Operational response - **Staging is behind `main`:** trigger a manual run with no SHA. Do not edit a floating image tag or invoke the host script by hand. Because there is no daily schedule, this is the normal way Staging advances at all. - **Schema mismatch** (the run fails with *"Staging awaits an authorized migration"*, host exit `43`): inspect the Staging migration plan; after authorization, dispatch with `apply_migrations=true`. That run ends as a migration-only phase (exit `42`) and stops before runtime replacement. Then **re-run the sanitizer** against the new exact revision — re-attestation cannot advance a revision, see above — update `STAGING_SANITIZATION_RECEIPT` in `/etc/helix/staging/deploy.env`, and dispatch once more with migrations off. Expect to repeat this whenever Staging has been left behind: `main` lands migrations on essentially every working day. - **`go-ci / Main` is not green** (the run fails with *"main is not verified"* before any cloud identity is issued): this is the ancestor gate working, not a Staging defect. Fix `main`'s own post-merge CI, then dispatch again. Roughly half of this workflow's historical failures were this, not Staging. - **The host output is unreadable:** it should not be. Cloud Assistant returns `Output` Base64-encoded and the workflow decodes it before printing. If you are looking at a Base64 blob in the Actions log, the decode step regressed. - **Deployment failed after replacement:** inspect the Cloud Assistant command output and local receipt. The former images are automatically re-created, but the failed workflow stays red. Do not call this a database rollback. - **Cloud credentials fail:** repair the OIDC trust or ECS RAM role, not by adding long-lived access keys to GitHub or the host.