# BYOC Customer Pull Image Delivery Design ## Summary This design adds a side-by-side image delivery path for prod AWS BYOC regions. The existing push-to-customer-ECR workflow remains enabled, while customers can also pull the exact images required by a prod AWS BYOC deployment from PingCAP prod ECR repositories. The implementation is intentionally config-driven: - `image-sync-byoc-prod.yaml` is the source of truth for image names that should be available through the customer pull path. - `app-delivery` deployment configs are the source of truth for DBAAS tags that are actually used by prod AWS BYOC control-plane deployments. - `observability` new-region rendering, the managed-by-ops deploy plan, and `auto-deploy` regional-server inputs are the source of truth for observability image and chart tags that are actually used by prod AWS BYOC observability deployments. - Kernel image tags are maintained manually in `byoc-pull-images-aws-prod.yaml` until there is a reliable kernel release source of truth. - `docker-image-controller` owns the generated customer pull manifest, the customer principal list, and the ECR repository policy input. - Existing sync configs are updated to ensure required public mirrors exist and BYOC entries point at PingCAP prod ECR sources. ## Problem The current BYOC prod flow pushes images into each customer's ECR. Some customers want the inverse ownership model: PingCAP hosts the images in PingCAP ECR, and the customer pulls those images from PingCAP. That requires four things: 1. Every image listed for prod AWS BYOC customer pull must exist in a PingCAP-owned prod ECR. 2. The customer-facing image list must use exact tags, not broad regex tag filters. 3. When deployment image tags change, the customer-facing list and sync configs must be refreshed automatically. 4. PingCAP must grant repository-level pull permission to customer IAM principals. ## Goals - Generate the DBAAS and observability sections of the prod AWS BYOC pull manifest from the image names in `image-sync-byoc-prod.yaml` and the tags resolved from their respective prod AWS BYOC deployment sources. - Keep kernel images in the same pull manifest file, but leave them manually maintained. - Use `name`, `source`, and `tag` in the pull manifest. Do not use `repository` or `owner`. - Keep source images in PingCAP prod ECR accounts: - `380838443567.dkr.ecr.us-west-2.amazonaws.com` for DBAAS images. - `557537366020.dkr.ecr.us-west-2.amazonaws.com` for observability images. - Add missing public upstream source/tag mirrors to `image-sync-prod.yaml`. - Normalize public third-party sources in `image-sync-byoc-prod.yaml` to PingCAP ECR sources when an exact deployed tag is resolved. - Ensure resolved tags are present in `image-sync-byoc-prod.yaml` without pruning old tags, except for public-source normalization where the entry is rewritten to the exact resolved tag. - Let customer IAM role ARNs and root ARNs be configured as principals. - Reuse the existing ECR policy updater for repo-level pull permissions. - Keep the current push-to-customer-ECR workflow unchanged. ## Non-goals - Removing or replacing the push-to-customer-ECR BYOC workflow. - Supporting dev BYOC, Azure BYOC, Alibaba Cloud, GCP, or Harbor pull delivery. - Granting customer identity-side permissions. Customers must grant their own principals `ecr:GetAuthorizationToken` on `*`. - Enforcing image removal or cleanup for images no longer used by deployments. - Discovering new customer-pull image names from deployment sources. New non-kernel image names must be added to `image-sync-byoc-prod.yaml` before they can appear in the generated pull manifest sections. - Automatically resolving kernel image tags. Kernel tags are manual until a supported BYOC kernel version catalog is available. ## High-level Design ```mermaid flowchart TD A["app-delivery prod AWS BYOC configs"] --> G["generate-byoc-pull-images.py"] B["observability new-region render and managed-by-ops plan"] --> G R["auto-deploy AWS BYOC regional-server inputs"] --> G K["manual kernel image tags"] --> E["byoc-pull-images-aws-prod.yaml"] C["image-sync-prod.yaml"] --> G D["image-sync-byoc-prod.yaml"] --> G G --> E G --> C G --> D H["byoc-pull-customers-aws-prod.yaml"] --> P["update_ecr_policy.py"] E --> P I["ecr-byoc-pull-prod.yaml"] --> P P --> J["PingCAP ECR repository policies"] ``` The generator reads prod AWS BYOC deployment tag sources from: - `tidbcloud/app-delivery/nextgen-regional-control-plane` - `tidbcloud/app-delivery/dedicated-cluster-ng-service` - `tidbcloud/observability/deployments` - `tidbcloud/auto-deploy/terraform/aws/byoc` It writes: - generated `dbaasImages`, `o11yImages`, and `o11yCharts` sections in `byoc-pull-images-aws-prod.yaml` - `image-sync-prod.yaml` - `image-sync-byoc-prod.yaml` It preserves and validates the manually maintained `kernelImages` section in `byoc-pull-images-aws-prod.yaml`. ECR pull grants are controlled by: - `byoc-pull-customers-aws-prod.yaml` - `ecr-byoc-pull-prod.yaml` - `.github/scripts/update_ecr_policy.py` ## Data Model ### BYOC Image Name Source `image-sync-byoc-prod.yaml` is the allowlist for customer-pull image names. Deployment configs do not add new names to the customer pull manifest. If a team adds a new image to prod AWS BYOC deployment and that image must be available through customer pull, the team must add the image name to `image-sync-byoc-prod.yaml` in advance. The generator then resolves the tag for that known name from the generated image set's source scope. Deployment source images that are not listed in `image-sync-byoc-prod.yaml` are ignored silently by the customer-pull generator, because the generator cannot know whether those images should be customer-pull images. ### Pull Image Manifest `byoc-pull-images-aws-prod.yaml` is the customer/shareable image and OCI artifact manifest. It has four top-level sets: - `kernelImages`: manually maintained kernel component images. - `dbaasImages`: generated from `app-delivery` deployment configs. - `o11yImages`: generated observability container images from managed AWS BYOC deployment sources. - `o11yCharts`: generated observability OCI Helm chart artifacts from managed AWS BYOC deployment sources. ```yaml kernelImages: - name: tidbcloud/pd source: 380838443567.dkr.ecr.us-west-2.amazonaws.com/tidbcloud/pd tags: - v8.5.4-nextgen.202606.1 - v8.5.4-nextgen.202606.2 dbaasImages: - name: tidbcloud/audit-ng source: 380838443567.dkr.ecr.us-west-2.amazonaws.com/tidbcloud/audit-ng tag: v0.11.0 o11yImages: - name: observability-ops-server source: 557537366020.dkr.ecr.us-west-2.amazonaws.com/observability-ops-server tag: e5a584a o11yCharts: - name: observability-charts/ops-server source: 557537366020.dkr.ecr.us-west-2.amazonaws.com/observability-charts/ops-server tag: 0.2.7 ``` Fields: - `name`: repository or OCI artifact name under the PingCAP ECR registry. - `source`: fully qualified PingCAP prod ECR repository. - `tags`: one or more exact literal tags for a manually maintained kernel image. - `tag`: one exact literal tag for a generated image or chart. Mutable tags such as `latest` are not allowed in the customer pull manifest. If a deployment source resolves a BYOC-listed image to a mutable tag, the generator reports the source input and omits that image from the generated pull list until the deployment side provides an immutable tag. Generated sections contain only the final resolved tag for each BYOC-listed image entry. If an old tag is no longer the resolved deployment tag, it is pruned from `dbaasImages`, `o11yImages`, or `o11yCharts` on the next generator run. Kernel tags are added and removed manually and are not pruned by the generator. `kernelImages` is not generated. The generator preserves it exactly as written except for schema validation. ### Kernel Images Most kernel image tags are not currently available from a deployment source of truth. They are maintained manually in `kernelImages`. `tidbcloud/dm` is configured in app-delivery deployment configs, so it is not a manual kernel image for this generator. It is resolved and emitted in `dbaasImages`. The allowed kernel image names are: ```text tidbcloud/pd tidbcloud/tidb tidbcloud/tiflash tidbcloud/tikv tidbcloud/ticdc tidbcloud/br tidbcloud/tiproxy ``` Kernel validation rules: - every `kernelImages` row must use one of the allowed kernel image names; - every `kernelImages` row must have only `name`, `source`, and `tags`; - `source` must be a PingCAP prod ECR source; - `tags` must be a non-empty list of unique, exact, immutable literal tags; - every kernel image name must appear only once; - the kernel image name must exist in `image-sync-byoc-prod.yaml`. The generator does not add, remove, or update `kernelImages` rows and does not use them to update `image-sync-prod.yaml` or `image-sync-byoc-prod.yaml`. ### Customer Principals `byoc-pull-customers-aws-prod.yaml` is manually reviewed through PRs. ```yaml customers: - name: customer-name accountId: "123456789012" principals: - arn:aws:iam::123456789012:role/customer-eks-node-role - arn:aws:iam::123456789012:root ``` Validation rules: - `accountId` must be a 12-digit AWS account ID. - Each principal must be either: - `arn:aws:iam:::root` - `arn:aws:iam:::role/` - The principal account ID must match the customer `accountId`. ### ECR Policy Input `ecr-byoc-pull-prod.yaml` maps pull-image repositories and customer principals into the existing ECR policy updater. Repository queries read all pull-list image sets and select repositories by PingCAP ECR source prefix. Conceptually: ```text (.kernelImages[]?, .dbaasImages[]?, .o11yImages[]?, .o11yCharts[]?) | select(.source matches the target PingCAP ECR account) | .name ``` The policy grants only repository-level pull actions: - `ecr:BatchCheckLayerAvailability` - `ecr:BatchGetImage` - `ecr:GetDownloadUrlForLayer` Customers still need an identity policy that allows `ecr:GetAuthorizationToken` on `*`. ## Generator Design The generator is implemented in `hack/generate-byoc-pull-images.py`. ### Inputs Required source checkouts: - `--app-delivery-root` - `--observability-root` - `--auto-deploy-root` Docker image controller root: - `--repo-root` Modes: - default: update generated files and sync configs in place. - `--check`: fail if generated outputs are not current. - `--check --report-only`: report generated diffs and deployment-derived tag conflicts, but return success so unrelated PRs are not blocked by app-delivery or observability drift. - `--validate-local-only`: validate local customer and pull image files without reading source repos. ### Source Scope The generator uses different tag-source models for `app-delivery` and `observability`. #### App-delivery Source Scope For `app-delivery`, the generator reads non-env deployment config layers in priority order. Env files are excluded because they may be stale and region-specific. Priority order: 1. BYOC-specific variant files. 2. Prod AWS variant files. 3. General variant files. 4. Base configs. BYOC-specific variant files include AWS BYOC variants such as: - `prod-aws-byoc-*` - `aws-byoc-*` - `byoc-aws-*` Prod AWS variant files include AWS prod variants such as: - `prod-aws-*` - `aws-prod-*` General variant files include: - files under `variants/` that are not provider-specific. Base configs are the lowest-priority fallback when the image tag is not present in any selected variant file. The app-delivery source scan excludes: - env files; - dev and staging inputs; - Azure, Alibaba Cloud, and GCP variant inputs. #### Observability Source Scope For `observability`, the app-delivery base/variant priority model is not used. AWS BYOC observability new-region setup is assembled by auto-deploy workflows that call local `deploy-new-region.sh` scripts, and the managed BYOC runtime deploys only the subset selected by `deploy-with-ops.sh`. The generator should render or dry-run the same local new-region roots used by auto-deploy for AWS BYOC: - `deployments/infra-v2` - `deployments/cloud-addon` - `deployments/dataplane` - `deployments/regional-server` - `deployments/drhouse` Those generated files are candidate inputs only. The generator must scan only the artifacts in the AWS BYOC managed observability component list. The initial implementation keeps this list static in `docker-image-controller`, derived from the current `deploy-with-ops.sh` scripts: - infra: `ops-server`, `kube-prometheus-stack`, `cert-manager`, `o11y-mixin`, `victoria-metrics-operator`, `eventrouter`, `volume-autoscaler`, `vector`, and `velero` when the AWS BYOC managed path enables it; - cloud addon: AWS load balancer controller, `external-dns`, `cluster-autoscaler`, `ingress-nginx-write`, and `ingress-nginx-read`; - dataplane: `vmcluster`, `loki`, and `alertmanager-regional`; - regional server: the regional-server Helm chart plus `observability-regional-server` and `observability-regional-data-server` image tags injected by auto-deploy; - DrHouse: `diagnosis-query` and `o11ydiagnosis-deltalake`. The generator must not scan all generated observability output wholesale. Some new-region scripts can generate artifacts that are not deployed in the managed AWS BYOC path, such as `deltalake-ops-server` or logstats components. Those artifacts must not appear in `o11yImages` or `o11yCharts` until the static managed component list is intentionally updated. This static list is an implementation tradeoff. It avoids parsing shell scripts or adding new observability plan-mode support in the first PR, but it means observability deploy changes that add or remove managed AWS BYOC components must also update the generator's component list. ### Image Extraction And Tag Resolution The generator extracts images from YAML structures and templated text, then uses those references only to resolve tags for non-kernel image names that already exist in `image-sync-byoc-prod.yaml`. Image set assignment is based on the source repository: - images resolved from `app-delivery` belong to `dbaasImages`; - container images resolved from `observability` belong to `o11yImages`; - OCI Helm charts resolved from `observability` belong to `o11yCharts`; - images in `kernelImages` are ignored by tag resolution. Supported YAML patterns: - full image string: ```yaml image: 380838443567.dkr.ecr.us-west-2.amazonaws.com/tidbcloud/audit-ng:v0.11.0 ``` - repository plus tag: ```yaml image: repository: observability-ops-server tag: e5a584a ``` - image plus tag: ```yaml image: tidbcloud/audit-ng tag: v0.11.0 ``` - kustomize image override: ```yaml newName: tidbcloud/audit-ng newTag: v0.11.0 ``` Some deployment templates contain raw placeholder lines and are not valid YAML. For those files, the generator falls back to text scanning for full image references instead of failing the refresh. Observability extraction has additional inputs: - Helm chart OCI artifacts are resolved from the static AWS BYOC managed observability component list. Local charts use `deployments/charts//Chart.yaml` for chart `name` and `version`. External charts pinned in deploy scripts use those pinned chart versions. - `observability-regional-server` and `observability-regional-data-server` use the image tag passed by the AWS BYOC auto-deploy Terraform chain, currently the `regional_server_img_tag` input in `auto-deploy`. - DrHouse component image tags use `image-tag.yaml` overlays. The resolver first checks an env-specific override under `envs/--/image-tag.yaml`, then falls back to the component-level `image-tag.yaml`. - `o11ydiagnosis-deltalake/image-tag.yaml` also carries `taskCollector.image.tag`, which resolves the `observability-regional-task-collector` image tag. If a deployment image or chart is not listed in `image-sync-byoc-prod.yaml`, it is not included in `dbaasImages`, `o11yImages`, or `o11yCharts` and is not reported by default. If a non-kernel BYOC-listed image or chart has no resolved tag in its generated set's source scope, the generator reports it in CI for manual review. ### Source Normalization The generator maps BYOC-listed image names to PingCAP prod ECR sources based on their generated image set: - `dbaasImages` maps to `380838443567.dkr.ecr.us-west-2.amazonaws.com/`. - `o11yImages` maps to `557537366020.dkr.ecr.us-west-2.amazonaws.com/`. - `o11yCharts` maps to `557537366020.dkr.ecr.us-west-2.amazonaws.com/`. Deployment source references are used to resolve tags and detect public third-party sources for BYOC-listed names. They do not decide whether an image name belongs in the pull manifest. `image-sync-byoc-prod.yaml` remains the image-name allowlist. If a BYOC-listed image has a public third-party source in its deployment source scope, and an exact tag is resolved, the generator ensures that public source/tag is present in `image-sync-prod.yaml` so PingCAP prod ECR can mirror it. The same image name may appear in both `dbaasImages` and `o11yImages`. That is not a conflict because the image sets map to different PingCAP ECR accounts. Observability Helm charts are treated as OCI artifacts in `o11yCharts` using the `observability-charts/` repository name. Chart publishing owns publishing those artifacts into PingCAP ECR; the BYOC pull generator validates and refreshes the exact chart tags in `byoc-pull-images-aws-prod.yaml` and `image-sync-byoc-prod.yaml`, but it does not add chart artifacts to `image-sync-prod.yaml` as public image mirrors. ### Public Source Normalization If an entry in `image-sync-byoc-prod.yaml` for a generated image set has a public third-party source and the generator resolves an exact deployed tag for that image, the generator does all of the following: 1. Add the public source and exact resolved tag to `image-sync-prod.yaml`. 2. Rewrite the `image-sync-byoc-prod.yaml` source to the PingCAP prod ECR source for that repository set. 3. Rewrite the `image-sync-byoc-prod.yaml` `tags.allow` list to contain only the exact resolved tag. Example: ```yaml # image-sync-byoc-prod.yaml before - name: third-party/foo/bar source: docker.io/foo/bar tags: allow: - "v1\\..*" ``` If the deployment source scope resolves `v1.2.3`, the generator writes: ```yaml # image-sync-prod.yaml - name: third-party/foo/bar source: docker.io/foo/bar tags: allow: - v1.2.3 ``` ```yaml # image-sync-byoc-prod.yaml after - name: third-party/foo/bar source: 380838443567.dkr.ecr.us-west-2.amazonaws.com/third-party/foo/bar tags: allow: - v1.2.3 ``` If a public-source BYOC entry for a generated image set has no resolved tag, the generator reports it in CI and does not rewrite the source or tags, because it cannot safely choose the exact mirrored tag. Public source normalization does not apply to `kernelImages`; manual maintainers must keep kernel sources as PingCAP prod ECR sources. ### BYOC Sync Config Updates For generated image set entries that already use PingCAP ECR sources, the generator ensures the resolved tag is present in `image-sync-byoc-prod.yaml`. If the entry exists but lacks the resolved tag, the tag is added to its `allow` list. The generator does not prune old tags from `image-sync-byoc-prod.yaml` for entries that already use PingCAP ECR sources. This keeps the existing push-to-customer-ECR workflow stable. The generator never writes `allow: []`. ### Tag Precedence And Conflict Handling The generator resolves one final tag per generated image set and BYOC-listed image entry. For `app-delivery`, precedence rules are: 1. If a BYOC-specific variant resolves a tag, use that tag. 2. Otherwise, if a prod AWS variant resolves a tag, use that tag. 3. Otherwise, if a general variant resolves a tag, use that tag. 4. Otherwise, if a base config resolves a tag, use that tag. 5. Otherwise, report the image as unresolved in CI. Higher-priority tags override lower-priority tags. For example, a BYOC-specific variant tag wins over a prod AWS variant tag, a prod AWS variant tag wins over a general variant tag, and a general variant tag wins over a base config tag. If multiple app-delivery files at the same priority resolve different tags for the same generated image set and BYOC-listed image, the generator fails with file-level diagnostics. This avoids silently choosing between equally authoritative tags. In `--check --report-only` mode, the generator reports the same diagnostics, skips that generated image entry, and exits successfully. For `observability`, the generator does not apply the app-delivery priority order. It resolves tags from the selected managed AWS BYOC deploy inputs: - rendered or dry-run output from the selected new-region roots, filtered by the static AWS BYOC managed observability component list; - DrHouse `image-tag.yaml` overlays for `diagnosis-query`, `o11ydiagnosis-deltalake`, and the nested task collector image tag; - AWS BYOC auto-deploy `regional_server_img_tag` for `observability-regional-server` and `observability-regional-data-server`; - managed Helm chart versions from local `Chart.yaml` files or pinned external chart versions in the deploy scripts. If those selected observability inputs produce more than one tag for the same `o11yImages` or `o11yCharts` BYOC-listed name, the generator reports a conflict with the source files or source inputs that produced each tag. In strict mode this fails; in `--check --report-only` mode it is reported without blocking unrelated PRs. The conflict key includes the manifest set. For example, `tidbcloud/vector` in `dbaasImages` and `tidbcloud/vector` in `o11yImages` are independent entries and do not conflict. Chart names are routed to `o11yCharts`, so `observability-charts/*` entries cannot conflict with observability container images. ### Idempotency The generator is deterministic: - It preserves `kernelImages` as written. - It sorts generated `dbaasImages`, `o11yImages`, and `o11yCharts` rows. - It updates YAML files only when content changes. - `--check` prints a diff and exits non-zero when generated files are stale. - `--check --report-only` prints the same diff but exits zero. ## Workflows ### Refresh Pull Images `.github/workflows/refresh-byoc-pull-images-prod.yml` Triggers: - manual `workflow_dispatch` - scheduled run - `repository_dispatch` event `refresh-byoc-pull-images-prod` Behavior: 1. Check out `docker-image-controller`. 2. Check out `tidbcloud/app-delivery`. 3. Check out `tidbcloud/observability`. 4. Check out `tidbcloud/auto-deploy`. 5. Install Python dependencies. 6. Run the generator. 7. Create a PR with refreshed generated files and sync config updates. Source repositories can add lightweight dispatch steps later. For example, an observability chart publishing workflow can dispatch this refresh after it updates chart image tags. ### Customer Principal Changes Customer principals are maintained manually in `byoc-pull-customers-aws-prod.yaml` through reviewed PRs. The file contains a commented sample customer entry. Local validation and CI run: ```bash python hack/generate-byoc-pull-images.py --repo-root . --validate-local-only ``` ### Update ECR Pull Policy `.github/workflows/update-byoc-pull-ecr-policy-prod.yml` Triggers: - manual `workflow_dispatch` with `action`: `check` or `sync` - `push` to `main` when pull-image, customer, policy config, policy updater, or policy workflow files change Behavior: 1. Resolve repositories from all image sets in `byoc-pull-images-aws-prod.yaml`. 2. Resolve principals from `byoc-pull-customers-aws-prod.yaml`. 3. Use `.github/scripts/update_ecr_policy.py` with `ecr-byoc-pull-prod.yaml`. 4. For manual runs, use the selected `check` or `sync` action. 5. For `push` events on `main`, always run `sync`. The policy config should query all four pull-list sets by ECR source prefix. That lets `kernelImages`, `dbaasImages`, `o11yImages`, and `o11yCharts` participate in the same permission workflow while still routing permissions to the correct PingCAP ECR account. `check` is a dry run: it reads current ECR repository policies and reports the changes that would be made, but it does not call `set_repository_policy`. ## CI Design CI now validates the BYOC pull path by: - checking out `tidbcloud/app-delivery` and `tidbcloud/observability`; - checking out `tidbcloud/auto-deploy`; - installing Python dependencies; - validating local customer and pull-image config; - running unit tests for the generator; - running the generator in `--check --report-only` mode; - keeping the existing no-empty-allow-list checks. The deployment-derived generated-current check is report-only in PR CI. It surfaces drift from app-delivery or observability without blocking unrelated docker-image-controller changes. Local file validation, unit tests, YAML lint, and no-empty-allow-list checks remain blocking. The generator CI output reports BYOC-listed images that need manual review: - non-kernel images in `image-sync-byoc-prod.yaml` with no resolved tag from their generated image set's source scope; - generated-set entries in `image-sync-byoc-prod.yaml` that still use public third-party sources; - generated-set BYOC-listed images whose deployment source is public third-party; - generated-set BYOC-listed images whose resolved deployment source tag is mutable, such as `latest`; - invalid `kernelImages` rows. For public third-party sources with resolved tags, the generator also updates `image-sync-prod.yaml` and normalizes `image-sync-byoc-prod.yaml`. In `--check` mode this fails if those generated updates were not committed. In PR CI, `--report-only` prints the diff without failing the job. ## ECR Permission Model ECR authorization has two sides: 1. PingCAP repository policy, owned by this repo: ```json { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::123456789012:role/customer-eks-node-role" ] }, "Action": [ "ecr:BatchCheckLayerAvailability", "ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer" ] } ``` 2. Customer identity policy, owned by the customer: ```json { "Effect": "Allow", "Action": "ecr:GetAuthorizationToken", "Resource": "*" } ``` Without the customer identity-side `GetAuthorizationToken`, Docker or containerd cannot authenticate to ECR even when the PingCAP repository policy allows image pull. ## Security Considerations - Customer principals are explicit IAM ARNs, not whole organizations. - Principal account IDs must match the customer account ID in config. - Policies are repository-scoped, not registry-wide. - The customer manifest exposes only exact image tags from PingCAP prod ECR. - The ECR policy workflow supports `check` mode before applying changes. - Existing manually added principals are preserved in non-`sync` behavior by the shared policy updater. ## Failure Modes ### Generated Files Are Stale PR CI reports the generator `--check --report-only` diff without failing. The scheduled/manual refresh workflow or a local strict `--check` run still catches the stale output, and the fix is to run the generator and commit the output. ### Customer Principal Is Malformed `--validate-local-only` fails and blocks CI or the add-customer workflow. ### Customer Principal Belongs to Another Account Validation fails because the ARN account ID does not match `accountId`. ### Public Source Is Missing From Prod Sync The generator adds an exact source/tag entry to `image-sync-prod.yaml`. ### BYOC Entry Uses Public Source If an exact tag is resolved, the generator adds the public source/tag to `image-sync-prod.yaml`, rewrites the BYOC entry source to PingCAP ECR, and rewrites its `tags.allow` to the exact resolved tag. If no exact tag is resolved, CI reports the image for manual review and the generator does not rewrite it. ### BYOC-listed Image Has No Resolved Tag CI reports the non-kernel image or chart for manual review. The entry is omitted from the generated `dbaasImages`, `o11yImages`, or `o11yCharts` set until a tag can be resolved from the generated set's source scope. ### Deployment Resolves Mutable Tag CI reports the BYOC-listed image or chart, mutable tag, and source input. The entry is omitted from the generated `dbaasImages`, `o11yImages`, or `o11yCharts` set until the deployment source is changed to an immutable tag. ### Same-priority Tag Conflict Strict generator runs fail with file-level diagnostics when multiple files at the same priority resolve different tags for the same generated image set and BYOC-listed image. PR CI reports the conflict in `--report-only` mode and does not block the PR. ### Invalid Kernel Image Row Local validation or CI fails if a `kernelImages` row has an unknown name, non-PingCAP source, an empty or duplicate tag list, a mutable or non-literal tag, or fields other than `name`, `source`, and `tags`. ### ECR Policy Query Resolves No Principals The policy updater skips the assignment. This allows the policy config to exist before any customers are added. ## Test Coverage Unit tests cover: - extracting full image strings and structured repository/tag pairs; - resolving tags only for image names listed in `image-sync-byoc-prod.yaml`; - applying BYOC variant, general variant, and base config precedence; - assigning `app-delivery` images to `dbaasImages`; - assigning `observability` container images to `o11yImages`; - assigning observability Helm chart artifacts to `o11yCharts`; - filtering observability generated output to the static AWS BYOC managed observability component list; - excluding observability artifacts generated by new-region scripts but not deployed by the managed AWS BYOC path; - resolving observability Helm chart artifact tags from local `Chart.yaml` files and pinned external chart versions; - resolving regional-server and regional-data-server image tags from `auto-deploy` AWS BYOC inputs; - resolving DrHouse `image-tag.yaml` overlays, including the nested `taskCollector.image.tag`; - preserving multiple manually maintained tags for each `kernelImages` entry; - rejecting empty, duplicate, mutable, or non-literal kernel tag values; - allowing the same image name to appear in both generated sets; - adding public sources to `image-sync-prod.yaml`; - rewriting public BYOC sources to PingCAP ECR sources when an exact tag is resolved; - pruning old tags from the generated pull list; - preserving old tags in `image-sync-byoc-prod.yaml` for entries that already use PingCAP ECR sources; - ignoring deployment images that are not listed in `image-sync-byoc-prod.yaml`; - accepting bare observability repository/tag pairs; - rejecting principals from a different AWS account; - same-priority tag conflict failure; - report-only same-priority tag conflict diagnostics. CI and local validation cover: - generated file drift is reported; - generated pull manifest rows have `name`, `source`, and one literal `tag`; - kernel pull manifest rows have `name`, `source`, and a non-empty list of unique literal `tags`; - pull manifest sources are PingCAP prod ECRs; - `kernelImages` names are in the allowed kernel image set; - no `allow: []` is introduced; - prod BYOC sync entries still have at least one allowed tag. ## Rollout Plan 1. Merge the BYOC pull image delivery PR. 2. Manually populate or review `kernelImages` for the kernel versions customers must be able to pull. 3. Wait for image sync to populate any newly added PingCAP prod ECR tags. 4. Add customer principals through a reviewed PR. 5. Merge manifest or customer principal changes to `main`; the BYOC pull ECR policy workflow automatically runs `sync` for relevant changes. 6. Optionally run the BYOC pull ECR policy workflow manually with `check` before merging a sensitive policy change. 7. Share `byoc-pull-images-aws-prod.yaml` and the customer-side `ecr:GetAuthorizationToken` requirement with the customer. ## Future Work - Add repository dispatch calls from source repos after deployment image tag updates. - Replace the static AWS BYOC managed observability component list with a dynamically derived deploy plan from `observability`, or migrate the observability-specific generator logic into the `observability` repo. - Replace manually maintained `kernelImages` with a generator input once a reliable BYOC kernel version catalog exists. - Add removal workflow for customer principals when a customer offboards. - Add optional smoke checks that selected manifest images exist in PingCAP prod ECR before policy sync. - Add a dedicated CI summary artifact for unresolved BYOC-listed images and public-source normalization actions.