# pause-stacks

Restore snapshots for the EKS freetier cost-pause tool (`hack/eks-pause-resume.sh`).

Each `*.json` here is the **restore source** for a cluster: per-nodegroup
`{min, max}` to put back on resume. The staging snapshot also contains a
`lifecycle` object for its current state. The append-only staging audit history
is stored separately in `staging-us-east-1-f01.history.json`.

The snapshot is generated by `snapshot` from the Pulumi stack config
via `hack/derive-nodegroup-bounds.mjs`, with live values as fallback for
nodegroups the config doesn't describe (serverless TiKV, `tiw-*` test-infra).
Why committed here instead of a local file: pulumi does **not** restore nodegroup
bounds (managed nodegroup `scalingConfig` is `ignoreChanges`), so a durable,
versioned restore source is required — a `/tmp` file would be a single point of
failure.

## Workflow

```sh
# 1) snapshot (read-only) + commit it BEFORE pausing
hack/eks-pause-resume.sh --profile qa-sso --cluster dev-us-east-1-f02 \
    --config pool/eks-freetier/Pulumi.dev-us-east-1-f02.yaml snapshot
git add pause-stacks/dev-us-east-1-f02.json && git commit -m "pause dev-us-east-1-f02"

# 2) pause (dry-run, then --apply)
hack/eks-pause-resume.sh --profile qa-sso --cluster dev-us-east-1-f02 pause
hack/eks-pause-resume.sh --profile qa-sso --cluster dev-us-east-1-f02 pause --apply

# 3) resume (dry-run, then --apply); unlocks max, seeds default-burstable
#    desired=1 so CA can start, then CA refills the rest
hack/eks-pause-resume.sh --profile qa-sso --cluster dev-us-east-1-f02 resume
hack/eks-pause-resume.sh --profile qa-sso --cluster dev-us-east-1-f02 resume --apply
```

Precondition: TiKV must be on EBS (instance-store i7ie/i8g drained) before pausing,
and pulumi/GitOps must not be reconciling the stack.

## Staging GitHub Actions workflow

`staging-us-east-1-f01` is operated through the **staging EKS lifecycle**
workflow, not by running the script locally. Select `pause` or `resume`, then
enter `staging-us-east-1-f01` as the confirmation value.

The workflow runs on the same self-hosted runner and `DBAAS_DEV_AWS_ROLE`
as `eks-freetier.yaml` (account `385595570414`). Those credentials need
read access to ASGs plus permission to update ASG bounds and clear
scale-in protection.

`resume` requires manual approval through the `staging-resume-approval`
environment. Configure that environment with required reviewers:
`baiyuqing`, `iosmanthus`, and `zuowei593`. GitHub environment protection allows
any one required reviewer to approve before the resume job starts. The
`resume_reason` workflow input is mandatory for resume requests and is stored in
the durable lifecycle audit record.

After a successful operation, the workflow commits the state, UTC timestamp,
GitHub actor, and workflow-run URL to
`pause-stacks/staging-us-east-1-f01.json`. The snapshot remains the restore
source after resume. Every successful pause/resume is also appended to
`pause-stacks/staging-us-east-1-f01.history.json`; history entries include the
operation, UTC timestamp, GitHub actor, workflow-run URL, and (for resume) the
required `resumeReason`. Its
initial configuration snapshot is derived from
`release-0.62:pool/eks-freetier/Pulumi.staging-us-east-1-f01.yaml` and is
stored on `staging-lifecycle-snapshot`, not on the PR-protected `main` branch.
Before every staging pause, the workflow refreshes this snapshot from the latest
`release-0.62` configuration and live ASG discovery, then commits the refreshed
snapshot and lifecycle record to `staging-lifecycle-snapshot`.