# 11. Fast tilt re-iteration via live_update on Tilt-native source-build services

Status: Accepted (amended 2026-07-03 — unified source-backed semantics; amended 2026-07-31 — Tilt cannot own the k3d image build without a registry; see the Amendment sections)

## Context

When a developer or agent changes a source-build service's code, the inner loop today is
cold: the service is deployed as a raw kustomize `local_resource` (so Tilt cannot hot-reload
it), each build does a cold `go build` (no persistent GOCACHE) + plain `docker build` (no
layer cache) + a globally-serialized `k3d image import` + a full pod reschedule. For agent
auto-loops this per-iteration cost is a primary blocker. The target: ~5 changed services
rebuilt+reloaded in ~3 min.

## Decision

- Restructure the source-build services onto **Tilt-native image objects**
  (`custom_build`/`docker_build` + `k8s_yaml`) so Tilt owns the image→deploy edge, then attach
  **`live_update`**: on a code change, incrementally cross-compile the binary on the host
  (persistent GOCACHE), **sync the new binary into the running pod, and restart the entrypoint
  process** — skipping image build, k3d import, and pod reschedule. Only cp9-built binaries are
  synced; no service-source changes.
- Add a persistent, shared **GOCACHE** and **Docker BuildKit layer cache** on the build host
  (incremental compiles; stop forcing `--no-cache`) for the cases that still need a full image
  build.
- **De-serialize the k3d image import** (today a global filesystem mutex) — e.g. push once to
  an in-cluster registry that k3d pulls from — for the full-rebuild path (non-live-update
  changes).
- **No separate "fast-redeploy / skip proof chain" mode.** The ~7 min `session_ready` proof
  chain (metadb seed → global-service → gateway → portal) is a **one-time `cp9 up` cost**; it
  is not re-run on iteration (a code change is just Tilt rebuilding/rolling that one resource),
  so skipping it is automatic. This item was dropped from the plan.

Scope: cp9-internal — the Tiltfile and build scripts are cp9's, and live_update syncs the
cp9-built binary. No upstream business-repo changes.

## Consequences

- A changed source-build service reloads in seconds (binary sync + process restart) instead of
  a multi-minute rebuild → import → reschedule; ~5 changed services fit well under the ~3 min
  target.
- Full image rebuilds (non-live-update changes — config, new deps) are faster via persistent
  GOCACHE + layer cache + de-serialized import.
- Restructuring the source-build services' deploy onto Tilt-native objects is the main up-front
  cost.
- live_update requires the container to tolerate binary swap + process restart; the existing
  alpine + standalone-binary image already fits.

## Amendment (2026-07-03): unified source-backed semantics across the merged Premium runtime

The task #140 grill (PRD #664) unified the product semantics this ADR's first implementation
had split into two opt-in lists. The mechanics above are unchanged; what changes is how a
service is selected and how far the model reaches.

- **Source-build and live-update are one product mode.** A product service either runs its
  staging-aligned image, or it is **source-backed**: CP9 watches, builds, publishes, and
  applies source changes for it. The only user-facing opt-in is **`source_build_services`**.
  `live_update_services` is no longer a second product opt-in; any remaining second-list
  handling in the Premium Session Tiltfile is an implementation compatibility detail to
  converge (PRD #664 Tiltfile-refactor slice), not a product distinction.
- **Live-update capability is a CP9-owned gate, not a user choice.** Update behavior derives
  as `_contains(source_build_services, service) && _live_update_supported(service)`, where
  supported means CP9 ships a pre-tested internal rollout/readiness contract for the service
  (rollout target, image writer, readiness check, status/proof image target — one canonical
  registry shared by the Tilt writer, status, and proof). Selecting a service without that
  contract fails closed at session graph / Tiltfile validation with a clear message, before
  build, publish, summary write, or activation.
- **Scope is the whole merged Premium runtime.** This model covers all staging-image product
  services under the merged Premium persona: existing Premium services, imported
  Starter/Essential services, and imported Dedicated services — Dedicated services must join
  this same model (same opt-in, contract registry, and image-source summary) when they enter
  the merged runtime, rather than introducing a separate source mode.
- **Backend mechanics split by runtime location.** k3d control-plane services (across all
  families) use this ADR's mechanics: host incremental build, binary `sync()` into the running
  pod, and a **process-level restart wrapper** — never a container restart, which would
  re-create the container, pull the image again, and discard the synced binary. Real EKS
  data-plane services use a v1 **image-level** backend instead (remote build → session-scoped
  ECR unique tag → patch Deployment image → wait rollout; ADR 0014/0015): this ADR's pod-level
  binary sync must not be assumed there until a per-service restart contract exists.
- **Proof semantics are unchanged by this model.** In the merged proof closeout (ADR 0018),
  source-backed mismatch/readiness/provenance findings are advisory evidence and never fail
  proof directly or skip product API trains; only proof infrastructure failures (missing or
  malformed required artifacts/schema) can fail closeout, because proof cannot report without
  them.

## Amendment (2026-07-31): Tilt cannot own the k3d image build without a registry

The Decision above said Tilt should own the image→deploy edge, and it still does not. Moving
the k3d control-plane source-build services onto `custom_build` was attempted and **reverted**
after live testing; recorded here so the same attempt is not repeated.

- **Why it fails.** `custom_build` hands tag ownership to Tilt: the build command is given
  `$EXPECTED_REF`, but Tilt then derives a content-addressed tag from the built image and
  deploys *that*, passing it to the apply command as `$TILT_IMAGE_0`. Measured on a live
  session, the build command received `…:tilt-build-1785491165` while the pod was told to pull
  `…:tilt-1f62f1b2c94318d2`. The build script `k3d image import`s the only tag it knows — the
  first — so the pod lands in `ImagePullBackOff`. The script cannot close the gap, because the
  deploy tag does not exist until after the build returns.
- **Neither delivery route is available.** With pushing enabled Tilt tries to `docker push` to
  `cp9.local`, which is a naming convention rather than a host, and fails DNS resolution. With
  `disable_push=True` Tilt treats delivery as the build command's responsibility and does
  nothing. Both were measured; no combination of `custom_build` options works against a local
  k3d cluster with no registry.
- **The existing mechanism is retained because it is internally consistent.** One pinned tag
  (`cp9.local/<session>-<service>:local`) is known to the build script, the `k3d image import`
  and the kustomize override alike, so nothing ever pulls and no registry is needed. Its cost
  is that a rebuilt image leaves the pod template byte-identical, which is what the
  `image.stamp` file and the `cp9.tidbcloud.com/source-image-stamp` annotation exist to force
  around.
- **The real prerequisite is this ADR's own unimplemented item: a registry the cluster can pull
  from.** With one, tag ownership stops mattering, and the stamp annotation and the manual
  `k3d image import` both retire together. Until then the two designs — pinned tag with a stamp,
  or Tilt-owned tag with a registry — are each coherent, and taking half of each is not.
- **The process restart wrapper stays CP9-owned; `ext://restart_process` was evaluated and
  rejected separately.** The extension restarts by setting the derived image's `ENTRYPOINT`
  (`/tilt-restart-wrapper --watch_file=/tmp/.restart-proc <entrypoint>`), and a Kubernetes
  pod-spec `command:` overrides image `ENTRYPOINT`. Four live-update services pin `command:` in
  their dev-full overlay — `serverless-service`, `serverless-global-service`, `account-provider`
  (a volume-mounted path) and `dedicated-ticdc-service` — so for them the wrapper would never
  execute and `live_update` would sync a new binary that no restart ever picks up. CP9's wrapper
  instead stages itself **at the canonical path the manifest names**, keeps the live binary at
  `/workspace/service`, and execs `"$@"` through, so it survives a pinned `command:` and forwards
  `args:`. Adopting the extension for only the services that do not pin `command:` was also
  rejected: two restart mechanisms are harder to reason about than one that covers every service.