# Troubleshooting A Fresh Remote Session

These are reproducible infrastructure, timing, and config failures seen while bringing up a fresh `remote-ec2-k3d` Dev full Session. Match the evidence before classifying a product bug.

## `cp9 up` bring-up

- **Terraform provider install fails in `substrate_plan` (`terraform init`)** — `connection reset` / `unexpected EOF` pulling `hashicorp/aws` (and `archive`/`random`) from the HashiCorp release CDN. The provider binaries are large; the download flakes even when general network is fine. Copying `.terraform/providers` is not enough (`cp9 up` re-runs `init`). Workaround: set an explicit `TF_PLUGIN_CACHE_DIR` (pre-populated with the locked versions) before `cp9 up`, then rerun. No cloud resources are created on this failure.
- **The local Tilt UI shows stale or stopped state while EC2 workloads continue** — remote mode has exactly one EC2-owned, session-scoped systemd Tilt controller. The local URL is only an SSH forward. Read the authoritative state with `cp9 status config=cp9.yaml`; rerun the same `cp9 up` to repair a missing local forward or restart an inactive owned unit. Do not start `tilt up` locally, stop arbitrary PIDs, or edit internal state.
- **`ImagePullBackOff` on a default (image-pull) service** — a pinned staging image tag has been auto-deleted by the staging ECR 14-day lifecycle policy, so image-pull consumers fail. Confirm with `aws ecr describe-images` (the tag is gone; newer tags exist). The pinned staging tags in `internal/sourcebacked/contracts.json` need periodic refresh. Tracking: `tidbcloud/controlplane9#393`. Workaround: refresh the stale pin(s) to a current tag, or source-build the affected service.
- **`cp9 doctor` on a brand-new remote session** reports that remote host metadata is not expected until the first `cp9 up`; this expected state passes doctor. Other warnings and failed checks remain blocking.
- **`cp9 doctor` reports `remote-controller-tools-missing`** — remote bootstrap owns the pinned Tilt/Helm installation. Rerun the same `cp9 up` so `host_ready` repairs the EC2 tools; do not create a local controller as a workaround.
- **Remote source sync is interrupted by a poor connection** — rerun the same
  `cp9 up` or deploy command. CP9 publishes one Git-aware, complete snapshot per
  selected repository root only after its atomic upload finishes; do not mutate
  the remote staging directory manually.
- **Transient `serverless-service` brworker leader-election lease errors during first boot** — the `serverless-service` leader-election RBAC is a manifest-only Tilt resource, so workloads may start before that RBAC has fully converged. Client-go leader election retries continuously; treat brief lease acquisition errors as a startup race, not a proof failure, unless the brworker stays unavailable or the errors do not self-heal.
- **An explicit Tilt port is occupied** — preflight names the owning process and stops before lifecycle mutation. Remove the explicit port and let cp9 reserve and persist a collision-free port, or stop the named owner and retry. Do not edit internal session state.

## Readiness Gates

Real remote provisioning starts with lifecycle and continues through later product convergence. Follow the actionable watch from its initial lifecycle baseline; do not treat the first not-ready as a failure:

- **Dev full runtime handoff** — `cp9 up` begins watching before lifecycle work and continues the same stream until top-level `ready` (exit 0) or `failed` (nonzero). On failure, use its named bounded-log/repair action. Ctrl-C while lifecycle runs preserves the resumable lifecycle path; after lifecycle finishes it only detaches the watch. Reattach with `cp9 status --watch config=cp9.yaml`.
- **`starter activate` readiness** now waits for rollouts and service hosts concurrently under one activation deadline. If activation fails, use the per-resource messages plus `cp9 starter status` to identify the missing rollout or LoadBalancer host; do not treat a first timeout as automatically normal.
- **Premium public SQL** — enabling public access is mandatory, and the public endpoint DNS/NLB take time to become reachable after the API reports the cluster active. Sequence: enable public connection → allowlist the client egress IP (/32) → poll until the public DNS resolves and the NLB accepts a connection → then run the MySQL handshake.
- **Restored-cluster SQL after backup/restore** — the restored public endpoint likewise needs a MySQL retry until its NLB is ready.

## Route product and cleanup failures

- Premium/Essential v2 API, customer-hostname, console, backup, or restore symptoms -> read `skills/premium/SKILL.md`, the retained Operation Map, and the exact current response/trace supplied by the task.
- Dedicated route, API-returned hostname, SQL, or delete symptoms -> read `skills/dedicated/SKILL.md` and the retained Operation Map. Remote mode must not replace a bad API hostname with a discovered NLB.
- Starter/Essential v1 create, branch, SQL, PD topology, readiness, PrivateLink, or cleanup symptoms -> read `skills/starter/SKILL.md`, its retained Operation Map, and current status/log evidence.
- Destroy, purge, inventory, stale-row, ownership, or delegated-cleanup symptoms -> read `skills/get-started/references/safety-and-ownership.md` before any additional mutation.

Use human-facing Wiki pages for onboarding only. Shipped code, the root config reference, current status/logs, and retained API maps own agent decisions.