--- name: tidb-onebox description: "How to deploy, validate, pause/resume, and destroy a TiDB-X on EKS dev environment using the one-box repo. Use when: creating a TiDB-X cluster on AWS EKS; running SQL validation against TiKV; pausing or resuming a one-box environment; destroying the one-box dev environment; refreshing EKS kubeconfig after SSO expiry." --- # TiDB one-box Terraform-first dev/test environment: VPC + EKS + TiDB Operator v2 (TiDB-X) on AWS. All resources are managed by Terraform. The Python script wraps every operation and handles known failure modes automatically. ## Prerequisites - `uv` installed - `terraform`, `aws`, `kubectl` CLIs in PATH - AWS SSO session active for the target account - `dev.tfvars` configured (see Config below) ## Script location ``` tidb-onebox.py (repo root) ``` Run all commands from inside the skill directory (no `--repo-root` needed since the script auto-detects its own location): ```bash uv run tidb-onebox.py [options] ``` > Override `--repo-root` only if running the script from a different working directory. ## Config All TiDB-X configuration lives in `terraform/environments/dev/dev.tfvars`. On first use, `bootstrap` creates it from the example template automatically. Key fields to set before deploying: | Field | Description | |-------|-------------| | `aws_region` | Deployment region (default: `us-west-2`) | | `project_name` + `environment` | EKS cluster name = `${project_name}-${environment}` (default: `tidb-onebox-dev`) | | `cluster_admin_principal_arn` | IAM principal ARN for EKS admin access. Any IAM role/user ARN works (e.g. the ARN from `aws sts get-caller-identity`, or an `AWSReservedSSO_*` role ARN). Leave empty to default to the caller's ARN. | | `s3_bucket_name` | S3 bucket for TiFlash disaggregated storage. The script auto-creates it on `deploy` if missing (errors out if creation fails — usually a wrong AWS profile). | | Image URIs | `pd_image`, `tikv_image`, `tidb_image`, etc. — default registry: `385595570414.dkr.ecr.us-west-2.amazonaws.com` | ### Advanced `tidb_x` fields - `pd_tso_enabled = true` **and** `pd_scheduling_enabled = true` together switch PD to microservice mode (`mode: ms`). Both must be toggled together. - `tiflash_write_enabled` / `tiflash_compute_enabled` enable disaggregated TiFlash write/compute groups. - `tikv_extra_spec` and `tiflash_*_extra_spec` are raw YAML overlays used to inject the IRSA `ServiceAccount` and S3 env vars. - `tiflash_write_config` / `tiflash_compute_config` are TOML blocks and must include `[flash] disaggregated_mode`. - `enable_ingress_nginx = true` deploys ingress-nginx (NLB mode); default is `false`. ### Default resource sizing | Component | Replicas | Storage | |-----------|----------|---------| | EKS nodes | `m6i.large`, desired=2, min=2, max=4 | — | | PD | 1 | 20Gi gp3 | | TiKV | 1 | 100Gi gp3 | | TiDB | 1 | — | | TiFlash write / compute | 1 each | 100Gi gp3 | Adjust via `node_desired_size`, `node_min_size`, `node_max_size`, `tidb_x._replicas`, etc., then re-run `deploy`. For one-off runtime scaling, use `uv run tidb-onebox.py scale ...` instead of editing `dev.tfvars`. The script passes temporary Terraform `-var` overrides for replica counts, so the change applies to that operation without rewriting the config file. ## Commands ### bootstrap — init + validate + plan (no apply) Initialises Terraform, validates the config, and generates a plan so you can review what will be created before deploying. ```bash uv run tidb-onebox.py bootstrap ``` Creates `dev.tfvars` from the example template if it does not exist yet. --- ### deploy — single-phase apply Creates the full stack (VPC + EKS + add-ons + TiDB Operator + TiDB-X cluster) in one `terraform apply`. Idempotent: re-run after a transient failure to resume. ```bash uv run tidb-onebox.py deploy ``` The script: 1. Auto-creates the configured S3 bucket if missing (aborts on failure — usually wrong AWS profile). 2. Runs `terraform apply -auto-approve` for the full stack. 3. Updates kubeconfig for the new EKS cluster. If apply fails (e.g. EKS API timeout, ECR throttling), simply re-run `deploy` — Terraform resumes from the current state. The script streams Terraform output to stdout and also tees it to a log file under `/tmp`, printing the log path before the long-running `plan` / `apply` step. Use that log if your terminal or UI truncates output. > The `--auto-approve` flag exists only for symmetry with `destroy` and is a no-op. --- ### destroy — full teardown with automatic remediations ```bash uv run tidb-onebox.py destroy [--auto-approve] [--force-cleanup-elb] [--sso-role-region REGION] ``` Handles all known failure modes automatically (up to 3 retry attempts): | Failure | Auto-remediation | |---------|-----------------| | `BucketNotEmpty` | Empties S3 bucket before destroy if Terraform owns it (detected via `terraform state list`); pre-existing buckets are left intact | | Classic ELBs blocking VPC deletion | Deleted when `--force-cleanup-elb` is set | | Helm release uninstall timeout | `terraform state rm` + orphaned pod cleanup | | Namespace stuck `Terminating` on TiDB-X CR finalizers | `kubectl patch` finalizers off all `*.core.pingcap.com` CRs in `tidb-x-cluster` | | Kubernetes provider `Unauthorized` / `forbidden` | Restores EKS access entry + policy, then retries | Flags: | Flag | Description | |------|-------------| | `--auto-approve` | Skip the `'destroy'` confirmation prompt | | `--force-cleanup-elb` | Delete classic ELBs that would block VPC deletion | | `--sso-role-region` | Override SSO role region used if the script cannot resolve the stable IAM role ARN directly (default: same as `aws_region`) | | `--also-delete-external-bucket` | After destroy, also empty + delete the configured `s3_bucket_name` even if it was auto-created outside Terraform (default: external buckets are kept) | On failure the full Terraform output is preserved in a log file under `/tmp`, and the script prints that path before the `plan` / `apply` step starts. --- ### validate — cluster health check ```bash uv run tidb-onebox.py validate [--namespace NS] [--cluster CLUSTER] [--status-service SVC] ``` Checks all group CRDs, waits for all pods to become Ready (20 min timeout), then hits the TiDB status endpoint on port 10080. Defaults: `--namespace tidb-x-cluster`, `--cluster tidb-x-onebox`, `--status-service tidb-tidb` --- ### verify-sql — SQL pushdown verification ```bash uv run tidb-onebox.py verify-sql [--namespace NS] [--service SVC] [--port PORT] [--user USER] [--password PWD] ``` Port-forwards to the TiDB service, connects via `pymysql` (no `mysql` client required), creates a temporary table, runs `EXPLAIN ANALYZE` with `READ_FROM_STORAGE(TIKV[t])`, and verifies `cop[tikv]` appears in the plan. Defaults: `--port 4000` (direct TiDB). Use `--port 6000` to connect through TiProxy instead. Note: `pymysql` avoids the `mysql_native_password` error that MySQL 9.x clients report on port 4000. --- ### stop — pause the environment ```bash uv run tidb-onebox.py stop [--cluster CLUSTER] [--nodegroup-name NAME] ``` Scales the EKS managed node group down to `desired=0,min=0`, which pauses the one-box environment without destroying Terraform-managed resources. This is intentionally an infra-level pause, not a graceful TiDB-X store drain. Defaults: - `--cluster`: inferred from `project_name` + `environment` in `dev.tfvars` - `--nodegroup-name`: auto-detected if exactly one managed node group exists If multiple node groups exist, pass `--nodegroup-name` explicitly. --- ### start — resume the environment ```bash uv run tidb-onebox.py start \ [--cluster CLUSTER] \ [--nodegroup-name NAME] \ [--desired-size N] [--min-size N] [--max-size N] ``` Restores the EKS managed node group to the default size from `dev.tfvars`: - `node_desired_size` - `node_min_size` - `node_max_size` You can override those values with flags for one-off resumes. --- ### scale — scale TiDB-X replicas and/or node group size ```bash uv run tidb-onebox.py scale \ [--cluster CLUSTER] \ [--nodegroup-name NAME] \ [--desired-size N] [--min-size N] [--max-size N] \ [--wait-timeout-seconds N] [--stall-timeout-seconds N] \ [--pd N] [--tikv N] [--tidb N] [--tso N] [--scheduling N] [--tiproxy N] \ [--tiflash-write N] [--tiflash-compute N] ``` Examples: ```bash # Scale TiKV and TiFlash compute, and add one EKS node first uv run tidb-onebox.py scale --tikv 2 --tiflash-compute 2 --desired-size 3 # Scale TiFlash down without changing nodegroup size uv run tidb-onebox.py scale --tiflash-write 0 --tiflash-compute 0 ``` Behavior: - Replica changes are applied through temporary Terraform `-var` overrides; `dev.tfvars` is not rewritten. - one-box also reconciles PD replication `max-replicas` to the effective TiKV replica count, which avoids stuck TiKV scale-in on small clusters. - When expanding the node group, the script scales nodes first, waits for them to register Ready, then applies the workload scale. - When shrinking the node group, the script applies the workload shrink first, waits for the namespace to settle, then scales nodes down. - The command waits for the requested Group CR replica counts to land and for all pods in the TiDB-X namespace to become Ready. - By default it waits up to 15 minutes total and fails early if the scale status is unchanged for 3 minutes. Override with `--wait-timeout-seconds` and `--stall-timeout-seconds`. --- ### refresh-auth — fix EKS access after SSO expiry ```bash uv run tidb-onebox.py refresh-auth \ [--aws-profile PROFILE] \ [--region REGION] \ [--cluster-name NAME] \ [--sso-role-region REGION] \ [--expected-account-id ACCOUNT_ID] ``` Runs `aws sso login` if needed, resolves the stable IAM role ARN from the current caller (preferring `aws iam get-role` over hand-built SSO role ARNs), upserts the EKS access entry + admin policy association, and updates kubeconfig. `--expected-account-id` causes the script to fail fast if the resolved AWS account ID does not match, preventing accidental operations against the wrong account. Defaults: `--aws-profile default`, `--region us-west-2`, `--cluster-name tidb-onebox-dev` ## Typical workflow Run all commands from inside the skill directory (the one containing `tidb-onebox.py`): ```bash # 1. Configure: edit terraform/environments/dev/dev.tfvars # (run `bootstrap` once if dev.tfvars doesn't exist yet — it copies the template) # 2. Deploy (single-phase, idempotent) uv run tidb-onebox.py deploy # 3. Validate cluster health uv run tidb-onebox.py validate # 4. Verify SQL pushdown to TiKV uv run tidb-onebox.py verify-sql # 5. Optionally scale / pause / resume the environment uv run tidb-onebox.py scale --tikv 2 --desired-size 3 uv run tidb-onebox.py stop uv run tidb-onebox.py start # 6. Tear down everything uv run tidb-onebox.py destroy --auto-approve ``` ## Troubleshooting / design notes - **Single-phase apply works** because the gp3 StorageClass uses the typed `kubernetes_storage_class_v1` resource (no plan-time API calls) and `helm_release.tidb_cluster` sets `disable_openapi_validation = true` (EKS `/openapi/v2` is ~12 MB and frequently exceeds the helm provider's 32 s download timeout). Do not switch back to `kubernetes_manifest` or remove the validation flag. - **AWS auth is checked early** in `validate`, `verify-sql`, `stop`, and `start`; on failure they fail fast instead of waiting for long-running AWS or kubectl errors. - **`scale` uses temporary Terraform overrides** (`-var scale_*_replicas=...`) so it can adjust TiDB-X component counts without rewriting `dev.tfvars`. A later plain `deploy` still reconciles back to the values in `dev.tfvars`. - **PD replication tracks TiKV replica count in one-box** so a 1-2 node dev/test cluster does not inherit PD's default `max-replicas = 3` and get stuck with TiKV stores permanently `Offline/Removing` during scale-in. - **`refresh-auth` only runs `aws sso login`** when the profile actually has SSO configured (`sso_start_url` set). For profiles that use externally-managed temporary creds (e.g. `default`), it fails fast and asks you to refresh credentials yourself. - **`stop` / `start` are nodegroup operations**. They do not mutate TiDB-X CR replica counts, which avoids the single-node safe-drain dead-end that made the old `stop` ineffective for one-box clusters.