# Change TiDB Kernel Component Images Use this when a kernel build (tidb, tikv, pd, tiflash, tiproxy, ticdc, br) must run inside a cp9 environment — for example when a kernel team delivers a candidate image and wants it under test. For a cp9-owned control-plane service image, stop here and use `source_build_services` in the Session Tiltfile instead; the source-backed path owns that rollout by `docs/adr/0011-cp9-tilt-fast-reiteration-live-update.md`. Two scenarios, three lanes. Read the scenario contract, then jump to your lane. ## Scenario contract **Catalog swap.** Change what the control plane hands out, then create a cluster. Persistent, survives operator reconciliation, scoped per tenant on the metadb lanes. **Live swap.** Patch the running dataplane object and restart pods now. Immediate, scoped to one cluster, and reverted by lane-specific events. A live swap is a legitimate delivery path, not only a diagnostic hack — but it is only honest if you know what will revert it, so each lane below states that explicitly. ### Registry constraint, and why it differs between the scenarios On the Premium and Dedicated lanes the catalog path does **not** use `base_image` as written. `cluster-service-ng` builds the final reference as: ``` //: ``` The same format is built on both the create path (`cluster-service-ng` `domain/workflow/action/apply_component_group.go:372-377`) and the update path (`update_component_group.go:178-183`); cp9's BYOC mirroring derives the same repository at `internal/byoc/images.go:271`. One difference worth knowing: the create path wraps the tag in `strings.TrimSpace` and the update path does not, so a `tag` column with trailing whitespace creates a working cluster and then breaks on the next update. For cp9 Premium those come from `personas/app-delivery/envs/premium/cluster-ng-server/config.yaml:27-30`: registry `385595570414.dkr.ecr.us-west-2.amazonaws.com`, repoPrefix `tidbcloud`. Consequences to plan around: - Everything before the last path segment of `base_image` is discarded. Editing `base_image` to point at a different registry does nothing. A catalog swap on these lanes can only load images that are already in the cp9 ECR. - `basename` collapses components onto shared repositories: `pd`, `pd_tso`, `pd_scheduling`, `pd_scheduler` all resolve to repository `pd`; `tikv`, `tikv_worker`, `coprocessor_worker` to `tikv`; `tiflash`, `tiflash_write`, `tiflash_compute` to `tiflash`. Push each distinct binary to the repository matching its basename. - The **tag** is still selected per `(cluster_type, component_name, version, tenant_id)`, so components sharing a repository can still run different tags. - A live swap writes a full image reference into the object, so it bypasses this collapse entirely. Any registry the dataplane can pull from works, including a registry the kernel team owns, as long as it grants cross-account pull to the runtime account. ### Registry access `terraform/cp9-developer/ecr-merge.md` splits pull from publish: - `CP9RuntimePull` covers every repository in the account-414 registry (`repository/*`). A **new repository or tag needs no pull-policy change**. - `CP9DeveloperPublish` (`:179-189`) covers only `us_west_2_publish_repositories` (`:28-34`), which does **not** include those four. The `cp9-developer` role therefore cannot push a kernel image to them; whoever publishes must use an identity that can. A kernel team publishing a candidate image uses its own publishing identity, which is outside cp9's IAM and needs nothing from cp9. The gap above only matters if a **cp9 operator** wants to push an ad-hoc image with the `cp9-developer` role; that would require adding those repositories to the publish set, which is an account-level shared dependency and needs real cross-account proof rather than a configuration review (`AGENTS.md`, shared CP9 dependencies). Leave the shared ECR policy alone when the goal is only to move a kernel image. ### Verifying before you point a cluster at an image Starter activation preflight verifies the four cp9-owned kernel inputs that have a single source of truth — the pooled serving TiDB image and the manager's `activate-tidb-image`, `serverless-lightning-image`, and `dumpling-image` — and blocks activation before any mutation if one of those ECR tags does not exist (`starterKernelImagePreflightChecks` in `internal/cmd/starter_cmd.go`). A reference pointing at a non-ECR registry cannot be checked and passes with an explicit "existence not verified" message. Everything else is still unverified, and a bad tag there surfaces only as `ImagePullBackOff`: - the Premium, Essential V2, and Dedicated catalog rows in the metadb; - the Starter `TidbCluster` CR images and the standalone serverless workload images. So confirm existence yourself before pointing a cluster at an image: ```bash aws ecr describe-images --profile --region us-west-2 \ --repository-name tidbcloud/ --image-ids imageTag= >/dev/null ``` `../scripts/set-kernel-image.sh` and `../scripts/patch-live-kernel-image.sh` both run this check. ### Metadb access Both metadb lanes live in the same MySQL instance. `premium-metadb` is in the preserved `cp9-system` namespace, not the session namespace: ```bash KUBECONFIG=.cp9//kubeconfig.yaml kubectl --context k3d- \ -n cp9-system port-forward svc/premium-metadb 3306:3306 ``` Record the exact statements you ran in the PR or run notes. Keep the password out of any output, and select the specific rows you need rather than dumping tables. --- ## Lane 1 — Premium / Essential V2 (nextgen) Catalog database `config_center_regional`, seeded by `personas/app-delivery/envs/premium/premium-metadb/30-config-center-regional-20260521.sql` and `68-premium-config-center-staging-defaults-20260723.sql`. Resolution order: `default_versions` selects the `version` string for a component, then `component_images` maps `(cluster_type, component_name, version, tenant_id)` to `base_image` + `tag`. `cluster_type` is `SHARED_POOL` or `LOGICAL_CLUSTER`. ### Catalog swap Config Center reads this with a direct query per call and **no cache**, and a `tenant_id` miss falls back to tenant 0 (`config-center` `internal/db/component.go:212-226`). Two consequences: - An upsert against the running metadb takes effect on the **next cluster creation**. No config-center restart, no `cp9 deploy`, no `cp9 up`. - Writing rows under a specific `tenant_id` scopes the change to that tenant and leaves every other cluster in the environment on the global defaults. Prefer this over editing tenant 0. The seeded CP9 Premium tenant is `1369847559692509642` (`68-premium-config-center-staging-defaults-20260723.sql:11`). For a scope narrower than a whole tenant, use a meta profile instead — see "Per-cluster scope: meta profiles" below. Minimum change — retag an existing version for one tenant: ```sql USE config_center_regional; INSERT INTO component_images (cluster_type, component_name, version, base_image, tag, GA, tenant_id) VALUES ('SHARED_POOL', 'tidb', 'v8.5.4', 'gcr.io/pingcap-public/dbaas/tidb', '', 0, ) ON DUPLICATE KEY UPDATE base_image=VALUES(base_image), tag=VALUES(tag); ``` The unique key is `(cluster_type, component_name, version, tenant_id)`, so this is idempotent. `base_image` only needs a correct final path segment. If you introduce a **new** version string rather than retagging an existing one, also point `default_versions` at it, otherwise nothing selects it. **`default_versions` has no unique key** on `(cluster_type, component_name, tenant_id)` — only `PRIMARY KEY(id)` and a non-unique `KEY(cluster_type, component_name)` (`30-config-center-regional-20260521.sql:107-118`). An `ON DUPLICATE KEY UPDATE` there never fires and silently inserts a second selector row, leaving which row wins undefined. Update in place when a row exists, and insert only when it does not: ```sql -- check first; if this returns more than one row, resolve that by hand before writing SELECT id, version FROM default_versions WHERE cluster_type='SHARED_POOL' AND component_name='tidb' AND tenant_id=; -- exactly one row UPDATE default_versions SET version='' WHERE cluster_type='SHARED_POOL' AND component_name='tidb' AND tenant_id=; -- no rows INSERT INTO default_versions (cluster_type, component_name, cloud_provider, region, tenant_id, version, cluster_version) VALUES ('SHARED_POOL', 'tidb', '', '', , '', ''); ``` `scripts/set-kernel-image.sh --set-default-version` implements exactly this sequence. `component_name=''` is the cluster-wide default row; a per-component row overrides it. Verify by reading back, then create a cluster and assert the pod image: ```bash kubectl --kubeconfig -n get pods \ -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}' ``` To change the registry or prefix for **all** kernel components at once, edit `server.image.registry` / `repoPrefix` in `personas/app-delivery/envs/premium/cluster-ng-server/config.yaml:27-30` and `cp9 deploy config=cp9.yaml service=cluster-ng-server`. This is a whole-environment switch and requires that registry to hold the full component closure under one prefix; use it for "test an entire kernel build", not for one component. #### Per-cluster scope: meta profiles Editing `component_images` changes the catalog for a whole tenant. A **meta profile** overrides component image tags for the clusters that reference it, which is the narrowest scope available and leaves the tenant defaults untouched. A meta profile is not a separate table. It is a row in the existing `templates` table with `config_type = 'CC_META_PROFILE'`, plus `template_values` rows keyed `component_versions.` and `component_tags.` (`config-center` `internal/service/meta_profile.go:14-24`, `:56-66`). cp9's seeded `config_center_regional` already has `templates`, `template_values`, and `default_templates` with the required `config_type` and `content_hash` columns (`30-config-center-regional-20260521.sql`), so the mechanism is available in a cp9 environment today. No cp9 seed ships a `CC_META_PROFILE` row, so nothing uses it until you create one. Resolution: `cluster-ng-server` reads `sharedPool.ConfigCenterMetaProfileId`, then a non-empty `metaProfile.GetComponentImageTags()[component]` overrides the tag that `component_images` would have supplied (`cluster-service-ng` `pkg/clusterconfig/impl.go:134-139`). The component version can be overridden the same way. Two constraints to plan around: - **A tag requires a version.** `CreateMetaProfile` rejects an image tag for a component with no version for that same component (`meta_profile.go:50-54`). Always set both. - **Create it through the Config Center API, not raw SQL.** `CreateMetaProfile` goes through `CreateConfigurationTemplate`, which owns `content_hash` and region sync. A hand-written `INSERT` into `templates`/`template_values` skips that; the config-center migration `2026062500_metaprofile_component_version.up.sql` has to set `content_hash = NULL` precisely to force a recompute after touching those rows directly. Writing them by hand is why `../scripts/set-kernel-image.sh` covers only the `component_images` path and deliberately does not create meta profiles. Use a meta profile when two kernel builds must coexist in one environment, or when one cluster must be pinned without disturbing anyone else's clusters in the same tenant. Use the `component_images` upsert when every new cluster for that tenant should pick the build up. ### Live swap Patch the dataplane group CR for the component. The images live at `.spec.template.spec.image` on the `core.pingcap.com` group CRs — `PDGroup`, `TiKVGroup`, `TiDBGroup`, `TiFlashGroup`, `TiCDCGroup`, `TiProxyGroup`, `TSOGroup`, `SchedulerGroup` — installed by `personas/app-delivery/envs/premium/dataplane-manager/k8s-dedicated-templates.yaml:10-24` (tidb-operator v2, `image.tag: v2.2.0-alpha.4` at `:35`). ```bash kubectl --kubeconfig -n patch tidbgroup \ --type merge -p '{"spec":{"template":{"spec":{"image":""}}}}' ``` tidb-operator rolls the pods itself; wait on the owning StatefulSet rollout rather than deleting pods by hand. **Patch the group CR — it is the only patch point that holds here.** tidb-operator owns the Deployment/StatefulSet and reverts a `kubectl set image` on them almost immediately. **What reverts a CR patch:** `cluster-ng-server` recomputes the image from the catalog only when a workflow actually requests `OpsUpdateImage` (`update_component_group.go:177-183`). That happens for the `VERSION` and `ALL` config-version scopes (`domain/server/set_component_config_version.go:101-106`) — in other words a version change or a full reconcile. Scale, quota, label, audit-log and config-only changes do **not** pass `OpsUpdateImage`, so they leave a patched image alone (`scale_sharedpool.go`, `set_component_quota.go`, `set_component_labels.go`, `set_auditlog_setting.go`). So the patch survives ordinary cluster operations and is reverted by a version change or a full reconcile. If the build must survive those too, use catalog swap. --- ## Lane 2 — Dedicated classic Catalog database `config_center_staging`, read by the **global** `config-center` in namespace `cp9-global` (`personas/app-delivery/envs/dev-full/config-center/config.toml:14`), consumed by `cluster-controller` (`personas/app-delivery/base/dedicated-cluster-controller/config.yaml:81`). `cluster_type` is **`DEDICATED_TIDB`**, not the Premium values. Kernel image rows are seeded **only** by `personas/app-delivery/envs/premium/premium-metadb/62-dedicated-classic-config-center-20260610.sql` (`component_images` at `:202`, `default_versions` at `:198`). `67-dedicated-config-center-aws-staging-20260722.sql` carries templates only, so `62-` is the file to edit. ### Catalog swap Same table shapes and same no-cache behaviour as Lane 1, with the different database and `cluster_type`. One asymmetry to keep in mind: the tenant-0 fallback exists only for `GetComponentImage`. `GetDefaultVersion` (`config-center` `internal/db/component.go:115-135`) matches every column exactly with no fallback, so a tenant-scoped `default_versions` row is not a drop-in analogue of a tenant-scoped `component_images` row. ```sql USE config_center_staging; INSERT INTO component_images (cluster_type, component_name, version, base_image, tag, GA, tenant_id) VALUES ('DEDICATED_TIDB', 'tidb', 'v8.5.2', 'gcr.io/pingcap-public/dbaas/tidb', '', 0, ) ON DUPLICATE KEY UPDATE base_image=VALUES(base_image), tag=VALUES(tag); ``` Seeded baseline for orientation: `default_versions` has `('DEDICATED_TIDB','tidb',…,'v8.5.2')`, and `component_images` carries `tidb`, `tikv`, `pd`, `tiflash`, `ticdc`, `tiproxy`, backup/restore, and lightning rows at `v8.5.2`. If a redeploy is needed because you changed seeded input rather than live rows, the owning services on this lane are `config-center`, `regional-server`, and `cluster-controller` — not the Premium set. ### Live swap This lane has a control-plane surface that carries images, which is a better patch point than the dataplane object: - `cloud.pingcap.com` `ConfigRevision` and `ClusterConfig` (both cluster-scoped) carry per-component `baseImage` + `tag` for `pd`, `tidb`, `tikv`, `tiflash`, `tiflashLearner`, `tiproxy`, `ticdc`, `backup`, `restore`, `lightning`, `tiflowMaster`, `tiflowExecutor`, plus a cluster-wide `spec.version` (`personas/app-delivery/base/dedicated-infra-api-crds/cloud.pingcap.com_clusterconfigs.yaml:41-329`, `cloud.pingcap.com_configrevisions.yaml:44-225`). - Effective image is `baseImage + ":" + (tag ?? spec.version)` per the CRD field semantics. Because `baseImage` here is a full repository reference, this path does **not** apply the basename collapse and can point at any pullable registry. - `Cluster` carries no image or version, only topology (`cloud.pingcap.com_clusters.yaml:64-610`); the observed version surfaces at `.status.db.version`. Prefer patching `ConfigRevision` and letting `cluster-controller` propagate. Note that `ClusterConfig` carries a `spec.selector`, so patching it affects **every** cluster the selector matches, not one — use `ConfigRevision` when the intent is a single cluster. Patching the dataplane group CR directly also works and behaves as in Lane 1 — `use_tidb_operator_v2` is globally enabled (`base/dedicated-cluster-controller/config.yaml:79-80`), so the dataplane runs the same `core.pingcap.com` group CRs, at operator `v2.0.0-beta.8` (`envs/dev-full/regional-server/k8s-dedicated-templates.yaml:68,82`). **What reverts it:** `cluster-controller` re-renders the revision from Config Center on the next config or version workflow, replacing your values with catalog values. Not provable from this repo: the exact dataplane group-CR jsonpath for this lane is inferred from the installed v2 CRD set plus the Premium precedent, because the rendering logic lives in `tidbcloud/infra-provider` `cluster-controller`. Confirm once with `kubectl get tikvgroups -o yaml` on a live Dedicated dataplane before relying on it in automation. --- ## Lane 3 — Starter / serverless No metadb involvement. Kernel images are in kustomize `base/` only; the Starter overlay patches scheduling and storage but contains **no image patches**. Reconciled by a namespaced serverless tidb-operator, `Deployment/tidb-operator-controller-manager` in namespace `tidb-serverless` (`personas/app-delivery/base/tidb-serverless/tidb-operator-controller.yaml:4-5`, `-cluster-scoped=false` at `:35`). ### Where each Starter kernel image lives 1. **The pooled serving TiDB image has one home.** The TiDB image a tenant cluster actually serves does not come from any CR: pooled TiDB pods are created by the starter manager from a template that deliberately has **no** `image` field for the `tidb` container (`personas/app-delivery/envs/dev-full/starter-regional/tidb-pools/default/pod-template.yaml:57-58`). The value comes from the ConfigMap key `tidb-image`, which is defined once in `tidb-pools/components/pool-tidb-image/tidb-image-patch.yaml` and pulled into all nine serving lanes (`default`, `canary`, `vip`, `canary-vip`, `super-vip`, `canary-super-vip`, `export`, `export-vip`, `export-super-vip`) through that component. **Edit the component, not a lane.** Applied by `coreWorkloadsScript` in `internal/cmd/starter_cmd.go`, which joins `tidb-pools/` per serving lane. 2. **The manager image flags have one home.** `--activate-tidb-image` and `--serverless-lightning-image` are defined in `personas/app-delivery/envs/dev-full/starter-regional/manager/runtime-patch.yaml` (`:32`, `:33`, with `--dumpling-image` at `:40`). `starterKernelImagePinsFromAppDelivery` (`internal/cmd/starter_cmd.go`) reads all three back out of that same overlay when it builds the remote workload activation plan, and fails closed if a flag is missing or empty — so there is no second copy to keep in sync and no flag can go silently missing. 3. **TiKV is the one image that is genuinely spread out, and one copy of it is inert.** The `tikv` block in `personas/app-delivery/base/tidb-serverless/serverless-cluster.yaml` pins `replicas: 0` (`:93`), so editing its image changes nothing that serves data. Real storage runs in the **six** `TidbCluster` partition objects in `serverless-tikv-cluster.yaml` (`:2/4`, `:246/248`, `:480/482`, `:714/716`, `:959/961`, `:1203/1205`), each with its own `spec.tikv.baseImage` + `version` at `:29-30`, `:273-274`, `:507-508`, `:741-742`, `:986-987`, `:1230-1231`. A complete TiKV kernel bump touches fifteen places: those six CRs plus `tikv-worker`, `cop-worker-sts`, `tikv-api` (`serverless-cse-workers.yaml:50`, `:211`, `:389`), `schema-manager.yaml:68`, `cse-backup.yaml:32`, `cse-dfsgc.yaml:73`, `load-data-worker.yaml:49`, `replication-worker.yaml:60`, `cse-archive.yaml:88`. Whether the inert `tikv` block in `serverless-cluster.yaml` can simply be removed is not decidable from this repo, because tidb-operator may read it for discovery or config derivation and its source is not vendored here. Until that is proven on a live environment, treat that block as load-bearing-unknown: treat it as load-bearing until a live environment proves otherwise, and leave it in place. Only `tidb-pools/` is applied, by `coreWorkloadsScript` in `internal/cmd/starter_cmd.go`. Note that the bare string `tidb-pool` (`starterSelectedServingLane`, and the `starterServingLanes` list) is a serving-lane name that maps to the `default` overlay, not a directory. Two unapplied decoy overlays (`starter-regional/tidb-pool/` and `starter-regional/super-vip-tidb-pool/`) used to sit alongside these and carried live-looking `tidb-image` keys; they have been deleted, so if you find them again in an old checkout, do not edit them. Other kernel-bearing standalone workloads: `tiflash-columnar-readnode` is the only TiFlash in Starter (`tiflash-columnar.yaml:74`); PD-derived services are `pd-tso-server.yaml:68`, `pd-resource-manager-server.yaml:75`, `pd-scheduler-server.yaml:34`, `pd-ctl-runner.yaml:39`. ### Catalog swap Edit the manifests above, then re-apply through the product path (`cp9 up`, or `cp9 starter activate` on an activated substrate). Unlike the metadb lanes there is no tenant scoping: the change is environment-wide. ### Live swap For CR-managed components, patch the `TidbCluster` CR; the operator rolls pods itself because `configUpdateStrategy: RollingUpdate` is set (`serverless-cluster.yaml:14`): ```bash kubectl --kubeconfig -n tidb-serverless patch tidbcluster \ --type merge -p '{"spec":{"tikv":{"baseImage":"","version":""}}}' ``` Apply it to all six partition objects when changing TiKV, per trap 1. For pooled TiDB, patch the lane's `serverless-pool-config` ConfigMap key `tidb-image`, then delete the pooled pods so replacements pick it up. Pooled pods are ephemeral by design (`pod-template.yaml:9` `restartPolicy: Never`, plus GC and reschedule flags in `manager/runtime-patch.yaml:27,68-69`), so patching a running pod's image is pointless — the pod is replaced from the template. For the standalone StatefulSets and Deployments listed above, patch the workload directly; no operator owns their image field. **What reverts it:** no cp9 controller rewrites these continuously, and the manager's version updater is off (`--enable-version-updater=false`, `manager/runtime-patch.yaml:72`). The stomp is `cp9 up` and `cp9 starter activate`, which apply the rendered overlay with `kubectl apply --server-side --force-conflicts` (`personas/dev-full/starter/apply-remote-kustomize-images.sh:515`, invoked through the `applyScript` resolved in `coreWorkloadsScript`). That takes field ownership back and reverts a hand-patched image. So a live Starter patch is stable until the next activation, not until the next cluster operation. Not provable from this repo: whether per-component `version` or top-level `spec.version` wins in `TidbCluster` (tidb-operator source is not vendored; upstream semantics are `baseImage:(component.version ?? spec.version)`), whether the manager prefers pool-config `tidb-image` or `--activate-tidb-image` for a serving pod (both carry the same tag today), and where `tidb-worker` pod images come from. --- ## Recording the change Whichever scenario you use, capture in the PR or run notes: the exact object or rows changed, the full image reference, the command output proving the running pod uses it, and — for live swap — which revert event the environment is now exposed to. A live patch that nobody wrote down looks identical to drift the next time someone inspects the environment.