# ADR: Remote EC2 Owns the Tilt Controller

Date: 2026-07-24
Status: Accepted
Issue: [#900](https://github.com/tidbcloud/controlplane9/issues/900)

## Context

`remote-ec2-k3d` originally placed Docker, k3d, image import, and Kubernetes on
EC2 but kept the Tilt controller on the user's laptop. A local interruption
could therefore stop reconciliation while EC2 workloads remained alive. A
second `cp9 up`, `cp9 status`, or local Tilt UI could then observe a stale local
PID/resource graph instead of the live remote session.

## Decision

Controller placement follows host placement:

| Host mode | Tilt/Helm controller | Workload runtime |
| --- | --- | --- |
| `local-k3d` | local process | local k3d |
| `remote-ec2-k3d` | session-scoped EC2 systemd unit | EC2 k3d |

For a remote session:

1. `host_ready` installs pinned Tilt and Helm binaries on EC2.
2. `tilt_ready` syncs the CP9 checkout, selected source repositories, and a
   remote-path version of `CP9_SESSION_CONTEXT`.
3. CP9 installs one loopback-only `cp9-tilt-<session>.service` and waits for its
   remote Tilt API.
4. The local Tilt URL is an SSH forward only. Losing it does not stop or replace
   the controller.
5. `cp9 status`, `logs`, `deploy`, resume, `down`, and `destroy` address the
   remote unit/API directly. `cp9 deploy` refreshes checkout/source inputs
   before triggering the selected resource.
6. Remote evidence is mirrored into the existing local session artifact path
   for status and activation consumers. Static controller metadata remains
   local; it is never used as the live health source.

Tilt binds only to EC2 loopback. No Tilt port is added to the security group,
and no new daemon or public protocol is introduced.

## Recovery Contract

- Interrupting the local `cp9 up` command does not terminate the systemd unit.
- Re-running the same `cp9 up` observes the exact unit. If active, it resumes
  incomplete lifecycle steps and repairs the disposable local forward. If
  inactive, it marks `tilt_ready`/`session_ready` pending and restarts the unit.
- `cp9 status` reports `running`, `not_ready`, `failed`, `stopped`, or
  `unreachable` from remote systemd plus the remote Tilt resource graph. A live
  SSH tunnel alone is never controller-health evidence.
- `cp9 destroy` stops the unit and runs remote `tilt down` before host cleanup
  when the host still exists. An accepted asynchronous cleanup handoff keeps
  the existing destroy boundary.
- Each local controller or kube-API tunnel role has at most one exact-session
  supervisor. A retry serially replaces any matching supervisor and gives its
  child a unique PID file; stop and destroy also discover pre-contract
  duplicates by their exact session-owned supervisor path.
- A supervisor resets its failure count after 30 seconds of stable operation.
  Eight consecutive failures exhaust the bounded retry budget and leave the
  role `quiescent` instead of launching AWS/SSH again every 30 seconds.
  `cp9 status` reports the role, supervisor PID, attempts, last exit, and
  `action_required`; rerunning `cp9 up` resumes it after credentials or the
  target are repaired.

## UI Login Evidence Boundary

UI delivery and login are separate:

- session-owned static image/runtime config;
- shared Auth0 client plus seed-user lookup;
- account identity/org membership;
- real browser callback and authenticated-shell acceptance.

The first three can be inspected without credentials and are reported
separately. Successful static delivery is not a claim that browser login passed.
The final acceptance remains a real browser login through
`/auth_redirect`; a destroyed customer scene is not enough evidence to assign
its prior login failure to a product service. After the callback,
authenticated shell, and direct history-path refresh are exercised, the
operator records only outcome, bounded stage, normalized same-origin path, and
timestamp with `cp9 browser-acceptance`. Passwords, tokens, cookies, headers,
queries, and response bodies are outside the evidence contract.