# tidb-onebox — Agent Guide

Terraform-first dev/test environment: VPC + EKS + TiDB Operator v2 (TiDB-X) on AWS.
All resources — including K8s add-ons and TiDB CRDs — are managed by Terraform. There is no `kubectl apply` workflow.

## Skill

This repo ships as a self-contained agent skill. `SKILL.md` (repo root) is the skill descriptor. `tidb-onebox.py` (repo root) is the single entry point for all operations.

```bash
uv run tidb-onebox.py <command> [options]
```

`--repo-root` defaults to the directory containing `tidb-onebox.py`, so no extra flags are needed when running from inside the repo.

| Command | Purpose |
|---------|---------|
| `bootstrap` | `terraform init` + `validate` + `plan` (no apply; creates `dev.tfvars` from example if absent) |
| `deploy` | Single-phase `terraform apply` for the full stack; auto-creates the S3 bucket if missing; idempotent on retry |
| `destroy` | Full teardown with automatic remediations (S3 emptying, ELBs, Helm uninstall timeout, stuck finalizers, Unauthorized) |
| `scale` | Temporarily scale TiDB-X component replicas and/or EKS node group size |
| `validate` | kubectl health check + TiDB status endpoint |
| `verify-sql` | SQL pushdown verification via `pymysql` (no `mysql` client required) |
| `stop` | Pause the environment by scaling the EKS managed node group to 0 |
| `start` | Resume the environment by restoring the EKS managed node group size |
| `refresh-auth` | Refresh AWS SSO, upsert EKS access entry, update kubeconfig |

## Directory layout

```
SKILL.md                      # skill descriptor (agent entry point)
tidb-onebox.py                # Python skill script (uv run tidb-onebox.py <command>)
terraform/environments/dev/   # sole Terraform root; dev.tfvars is the config entry point
terraform/modules/            # network, eks, vpc_endpoints, access, addons, tidb_x
```

## Deploy model

`tidb-onebox.py deploy` runs a single `terraform apply -var-file=dev.tfvars -auto-approve` for the full stack (VPC + EKS + add-ons + TiDB Operator + TiDB-X cluster). It is idempotent: if a transient failure occurs (EKS API timeout, ECR throttling, etc.), simply re-run `deploy` and Terraform resumes from the current state.

Before applying, `deploy` ensures the configured `s3_bucket_name` exists (creating it if missing) and aborts early if creation fails — that is almost always a wrong AWS profile or a globally-unique-name conflict.

The script now tees long-running Terraform `plan` / `apply` output to a log file under `/tmp` while still streaming to stdout. When a Terraform step fails, the script prints the log path so the full output remains available even if the terminal/UI truncates it.

> The historical two-phase apply (Phase 1 EKS → Phase 2 add-ons) is no longer required; the `helm` / `kubernetes` providers now resolve EKS credentials lazily.

## Config entry point

All TiDB-X configuration lives in a single `tidb_x` object in `terraform/environments/dev/dev.tfvars`.
Copy from the example on first use (or run `uv run tidb-onebox.py bootstrap`):

```bash
cp terraform/environments/dev/dev.tfvars.example terraform/environments/dev/dev.tfvars
# Edit image URIs (ECR), s3_bucket_name, and cluster_admin_principal_arn before applying
```

Non-obvious fields:

- `pd_tso_enabled = true` **and** `pd_scheduling_enabled = true` → PD switches to microservice mode (`mode: ms`). Both must be toggled together.
- `s3_create_bucket` no longer drives bucket lifecycle in the script: `deploy` auto-creates the bucket if missing, and `destroy` decides whether to empty it based on `terraform state list` (i.e. whether `module.tidb_x.aws_s3_bucket.tidb_x[0]` is in state).
- `node_desired_size`, `node_min_size`, `node_max_size` define the default running size of the EKS managed node group. `start` restores to these values unless flags override them.
- `scale` uses temporary Terraform `scale_*_replicas` override vars for one-off TiDB-X replica changes. It does not rewrite `dev.tfvars`; a later plain `deploy` reconciles back to the `tidb_x.*_replicas` values in config.
- `tikv_extra_spec` / `tiflash_write_extra_spec` are raw YAML overlay blocks injecting the IRSA ServiceAccount and S3 env vars into pods.
- `tiflash_write_config` and `tiflash_compute_config` must include a `[flash] disaggregated_mode` key.

## Gotchas

The `destroy` command in `tidb-onebox.py` handles all of the following automatically. Manual steps are only needed if running `terraform destroy` directly.

- **Destroy + S3**: If Terraform owns the S3 bucket (`module.tidb_x.aws_s3_bucket.tidb_x[0]` appears in `terraform state list`), it must be emptied before destroy or Terraform errors with `BucketNotEmpty`. The script does this automatically; for a manual destroy:
  ```bash
  aws s3 rm s3://<bucket-name> --recursive
  ```
- **Destroy + classic ELBs**: K8s-provisioned ELBs leave requester-managed ENIs that block subnet/IGW deletion. Use `uv run tidb-onebox.py destroy --force-cleanup-elb`, or manually delete before destroying.
- **Destroy + EKS access entry**: `terraform destroy` removes the EKS access entry early, causing the Kubernetes provider to get `Unauthorized` on subsequent destroy steps. The script detects this and restores access automatically. It first tries to resolve the stable IAM role ARN via `aws iam get-role`, then falls back to reconstructing the SSO role ARN with `--sso-role-region` if needed. Manual recovery:
  ```bash
  PRINCIPAL_ARN="arn:aws:iam::<account-id>:role/aws-reserved/sso.amazonaws.com/<region>/<role-name>"
  aws eks create-access-entry --cluster-name tidb-onebox-dev --principal-arn "$PRINCIPAL_ARN" --region us-west-2
  aws eks associate-access-policy --cluster-name tidb-onebox-dev --principal-arn "$PRINCIPAL_ARN" \
    --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
    --access-scope type=cluster --region us-west-2
  ```
- **Destroy + Helm release timeout**: `module.tidb_x.helm_release.tidb_cluster` uninstall may report `context deadline exceeded` after 5 minutes even though the release was successfully removed. The script proactively `terraform state rm`s this entry before each destroy attempt and force-deletes leftover pods. Manual fallback:
  ```bash
  terraform state rm module.tidb_x.helm_release.tidb_cluster
  kubectl delete pods --all -n tidb-x-cluster --force --grace-period=0
  ```
- **Destroy + stuck TiDB-X finalizers**: if `tidb-x-cluster` is stuck `Terminating`, the script dynamically enumerates all namespaced `core.pingcap.com` resources and strips finalizers so namespace deletion can finish.
- **Terraform state is local** by default. Multi-user setups need an S3 + DynamoDB backend added to `versions.tf`.
- **SSO region**: `refresh-auth` defaults to `--region us-west-2`; `--sso-role-region` falls back to `--region` for both `refresh-auth` and `destroy`. Override if your SSO role lives in a different region than the EKS cluster.
- **Pause/resume semantics**: `stop` and `start` operate at the EKS node group level, not by scaling TiDB-X CR replicas. Use them as a cost-saving pause/resume mechanism for one-box dev/test clusters.
- **Scale semantics**: `scale` is the supported way to do one-off runtime TiDB-X replica changes without editing `dev.tfvars`. It scales nodes first for expansions, and workloads first for contractions, so capacity changes are applied in a safer order.
- **MySQL 9.x client**: Reports `mysql_native_password` errors connecting on port 4000. Use `verify-sql` (pymysql-based) or the status port: `kubectl port-forward svc/tidb-tidb -n tidb-x-cluster 10080:10080 && curl http://127.0.0.1:10080/status`.

## Related Documentation

- [README](README.md)
- [SKILL.md](SKILL.md)
- [tfvars example](terraform/environments/dev/dev.tfvars.example)