# Design: Source Build and Live Update — who builds what, and who restarts it

How a code change in a product repo reaches a running container in a CP9 session.
There are two mechanisms with different owners, and the difference between them is
the difference between a five-second edit loop and a multi-minute one. Getting them
confused is how a deploy comes back green while the pod runs last week's binary.

Decisions and their rationale live in ADR 0011. This document is the mechanics.

## The one-line version

**Live update** replaces a binary inside a container that is already running.
**Source build** produces a new image and rolls a new container. Live update is the
fast path for a code edit; source build is what happens when the change is one a
running container cannot absorb, such as a new dependency.

Which one a service gets is not a user choice. `source_build_services` is the only
opt-in; whether that service can additionally live-update is derived from its
contract in `internal/sourcebacked/contracts.json`.

## Selection

`_live_update_supported()` requires all three:

- the contract says `sourceBuildSupported`
- `backend.kind == "k3d-control-plane"`
- the service has live-update wiring in the persona Tiltfile

The backend split is the load-bearing one. Today the registry holds 24
`k3d-control-plane` services and 11 `real-eks-data-plane` services (all
`starter-regional`). Only the k3d ones live-update.

The reason is control, not capability. k3d control-plane pods are created by Tilt,
so Tilt has their full lifecycle. Starter data-plane pods live on a Terraform-managed
EKS cluster that Tilt did not create and does not own. Syncing a binary into a pod
nobody is watching is not fast iteration, it is an unauditable mutation: the next
rescheduling, node rotation or scale event silently discards it and no resource goes
red. Data-plane services therefore use the image-level path, which is slower and
leaves a record.

## Live update (k3d control plane)

Three resources per service, and the ordering between them is the whole story.

| Resource | Kind | Produces |
|---|---|---|
| `<svc>-live-binary` | `local_resource` | a bare Go binary at `.cp9-build/<svc>/live/service` |
| `<svc>-image` | `local_resource` | the container image, imported into the k3d node |
| `<svc>` | `k8s_custom_deploy` | the applied manifests, and the `live_update` steps |

`<svc>-live-binary` runs `personas/dev-full/build/build-live-binary.sh`. It cross-compiles
for the container architecture against a persistent GOCACHE and touches no Docker.

`<svc>` carries the `live_update` steps:

```python
sync(binary_path, "/workspace/service")
run("touch /tmp/cp9-live-update.restart", trigger=[binary_path])
```

Tilt performs the `sync` natively, copying the file into the **running** container.
No image build, no pod reschedule.

### Why the entrypoint is `/workspace/service`

The live binary does not land at the image's canonical path (`/dataflow`, `/manager`,
`/config-center`). It lands at `/workspace/service`, because the runtime user is
non-root and must be able to overwrite it.

### Who restarts the process

Not Tilt. Tilt only touches a marker file. The restart is performed by a shell
supervisor CP9 writes into the image at build time (`build-local-image.sh`), which runs
as PID 1:

```sh
while :; do
  /workspace/service "$@" &          # the real service, as a child
  child=$!
  while kill -0 "$child"; do
    if [ "$(marker_stamp)" != "$last_stamp" ]; then
      kill "$child"; break           # marker mtime moved -> exec the new binary
    fi
    sleep 1
  done
done
```

This is why live update has a sub-second-to-one-second restart latency: the supervisor
polls once per second.

**Tilt's own `ext://restart_process` is deliberately not used.** It restarts by setting
the derived image's `ENTRYPOINT`, and a Kubernetes pod-spec `command:` overrides image
`ENTRYPOINT`. Four live-update services pin `command:` in their dev-full overlay
(`serverless-service`, `serverless-global-service`, `account-provider` — a
volume-mounted path — and `dedicated-ticdc-service`), so for them the Tilt wrapper would
never execute and a synced binary would never be picked up. CP9's supervisor instead
stages itself **at the canonical path the manifest names**, keeps the live binary at
`/workspace/service`, and execs `"$@"` through, so it survives a pinned `command:` and
forwards `args:`. See the ADR 0011 amendment; this is a settled decision, not an
oversight.

## Source build (k3d control plane)

One pinned tag, known to everyone, and nothing ever pulls.

`<svc>-image` is a `local_resource` running `build-local-image.sh` (or
`build/remote/build-remote-image.sh` when host is remote-ec2-k3d and the controller is
local). The Tiltfile hands it a fixed tag — `cp9.local/<session>-<service>:local` — and
the script builds under that tag, writes CP9's restart wrapper into the rootfs, then
`k3d image import`s it straight into the node's containerd. The kustomize override in
the deploy points at the same fixed string. No registry is involved, because the image
is pushed *into* the node rather than pulled by it.

The cost of a tag that never changes is that a rebuilt image leaves the pod template
byte-identical, so Kubernetes sees nothing to roll. That is what the stamp exists for:
the build writes `.cp9-build/<svc>/image.stamp` (`<image_id> <build_time>`), the file is
in `<svc>`'s deps, and the apply command carries it into the pod-template annotation
`cp9.tidbcloud.com/source-image-stamp` — changing the template hash and forcing the
rollout. Nobody "restarts" the container; a new one is rolled.

> **Tilt owning the image build was attempted and reverted.** `custom_build` gives Tilt
> tag ownership, which would make the stamp unnecessary — but Tilt deploys a
> content-addressed tag computed *after* the build, while the script can only import the
> tag it was handed beforehand, so the pod lands in `ImagePullBackOff`. Pushing instead
> fails because `cp9.local` is a naming convention, not a host. Both were measured on a
> live session. The two designs — pinned tag with a stamp, or Tilt-owned tag with a
> registry the cluster can pull from — are each coherent; the prerequisite for the second
> is the registry, which is ADR 0011's own unimplemented item. See the ADR amendment.

### Which changes take which path

`_source_image_build_deps` narrows the image build's watched paths once live update is
enabled, to the fallback set only — `Dockerfile`, `go.mod`, `go.sum`, and per-service
extras such as `migration`. Ordinary `.go` edits are deliberately invisible to the image
resource; they are meant to reach the container through `<svc>-live-binary` and the sync.
`fall_back_on()` in the live-update steps is what escalates a dependency change back to a
full rebuild.

This is the correct design, and it has a sharp edge worth stating plainly: when live
update is on, an ordinary code change has **exactly one** route to the container. If that
route does not fire, the change vanishes with no resource reporting an error.

## Source build (Starter data plane, real EKS)

Different topology, different mechanism, and no live update.

| Resource | Does |
|---|---|
| `<svc>-image` | builds and **pushes to ECR** (repository from the terraform output `starter_source_image_repositories`) |
| `<svc>-remote-rollout` | `rollout-starter-remote-image.sh` |

The rollout script:

1. `aws eks update-kubeconfig` for the real cluster
2. if the namespace or workload is absent, prints `waits for activation` and exits 0 —
   the data plane is Terraform-owned and may legitimately not exist yet
3. `kubectl set image` when the image differs
4. patches the `cp9.tidbcloud.com/source-image-stamp` annotation
5. **`kubectl rollout status --timeout=300s`**

Step 5 matters: this path waits for convergence and fails if it does not converge. It
also declares a real `resource_deps` edge from the rollout resource to the image
resource, so ordering is expressed in the graph rather than implied by a shared file.

## `cp9 deploy`

Tilt has no concept of a "service" — it has resources. "Service" is CP9's abstraction,
so mapping one service onto the set of resources that constitute its deploy is CP9's job
and cannot be delegated.

`cp9 deploy` owns exactly four things:

1. resolve config and ensure the Tilt controller is running
2. **transport local source to the remote host** (`RefreshRemoteControllerInputs`) — the
   only path by which an edit on a laptop reaches a remote session
3. resolve the service to its **set** of Tilt resources and trigger them in order
4. wait for Tilt to finish, then report

Compiling, image building, applying, live-updating and readiness are Tilt's. CP9 must
not reimplement any of them — but *waiting and verifying* is not reimplementing. It is
the conversion between Tilt's asynchronous, UI-oriented model and a CLI that returns an
exit code, and only CP9 can perform it.

Steps 3 and 4 are why this section exists. `tilt trigger` only enqueues work. A deploy
that triggered only `<svc>` skipped the compile in `<svc>-live-binary` entirely and
returned as soon as the trigger was accepted, so Tilt redeployed the previously built
binary in under a second and CP9 reported success (#977). Measured on a live session:
after three deploys the host source, the compiled binary and the running pod sat three
generations apart, `rc=0` every time, with nothing pending and no resource red — it would
never have converged on its own.

Convergence is read from Tilt's own `UpdateStatus` (`pending` / `in_progress` / `ok` /
`error`). One subtlety: immediately after a trigger Tilt has not yet moved the resource
off `ok`, and that stale `ok` describes the *previous* build, so it is not accepted
inside a short pickup window.

## Reading the state yourself

The fast loop requires no cluster:

```bash
tilt alpha tiltfile-result --file <tiltfile>
```

Against a live session, the questions worth asking Tilt directly:

```bash
tilt get uiresource <svc>-live-binary -o json   # did the compile actually run?
tilt get uiresource <svc> -o json               # buildHistory, pendingBuildSince
```

`pendingBuildSince: null` with an old `buildHistory` entry means nothing is queued and
nothing will happen without a trigger. When a change appears to have been lost, compare
three things before concluding anything: the source on the host, the compiled binary at
`.cp9-build/<svc>/live/service`, and the binary inside the pod at `/workspace/service`.
They can and do disagree, and which pair disagrees tells you which half of the chain
broke.