# TiDB X Kernel Image Release and Multi-Cloud Delivery | Item | Value | | --- | --- | | Status | Draft for review | | Date | 2026-08-05 | | Scope | Current TiDB X image-delivery chain, DevOps approval, multi-cloud delivery status, M x N image-tag selection | | Related repositories | `tidbcloud/devops`, `PingCAP-QE/ci`, `PingCAP-QE/ee-ops`, `tidbcloud/docker-image-controller`, `PingCAP-QE/ee-apps`, `tidbcloud/app-delivery` | > Implementation scope update (2026-08-06): the standard > `DeliverKernelImage` Operation and its approval are now included in the > DevOps-only phase. Its immediate auto action posts the existing `/fid` > command after approval, polls the request reaction, and transitions the > release to `AVAILABLE` or `SYNC_FAILED`. Notify Task cutover and Publisher > integration remain deferred. The active > implementation contract is specified in > [2026-08-06-kernel-image-release-mxn.md](./2026-08-06-kernel-image-release-mxn.md). > The future-flow sections below remain design input and are not the current > implementation contract. ## Table of Contents - [Introduction](#introduction) - [Motivation or Background](#motivation-or-background) - [Detailed Design](#detailed-design) - [Verified current implementation](#verified-current-implementation) - [Recommended future flow](#recommended-future-flow) - [DevOps data and APIs](#devops-data-and-apis) - [Implementation plan](#implementation-plan) - [Test Design](#test-design) - [Functional Tests](#functional-tests) - [Scenario Tests](#scenario-tests) - [Compatibility Tests](#compatibility-tests) - [Impacts & Risks](#impacts--risks) - [Unresolved Questions](#unresolved-questions) ## Introduction This document has two purposes: 1. Record the **current, code-verified** path from `ng-image-release` to AWS/Aliyun/Azure delivery and `UpdateNgGlobalComponentImage` creation. 2. Recommend a **small-change future design** that adds environment-local DevOps approval and M x N visibility without moving FID or Publisher logic into DevOps. The central recommendation is deliberately simple: ```text Keep the existing build-success CloudEvent and its EventListener/Trigger. Change the triggered Notify Task from direct /fid fan-out to dev/prod DevOps callbacks. Keep the existing FID -> GitHub result comment -> Tekton -> Publisher -> DevOps path. Insert one environment-local DevOps approval before /fid is issued. Add one correlated result callback after Publisher finishes. ``` DevOps does not implement registry copy commands, cloud credentials, image-to-Component scanning, or direct creation of every `UpdateNgGlobalComponentImage`. Those responsibilities already exist in FID and Publisher and should remain there. ### Decisions | Topic | Decision | | --- | --- | | Image build | Continue using TiBuild and the existing Tekton build chain. | | Build-success event | Keep `delivery.images.success`, `el-notify`, and the current Trigger. Only change the triggered Task's action. | | Cloud copy | Continue using FID in `docker-image-controller`. | | Component discovery | Continue using Publisher's `base_image == image repository` scan. Remove the duplicate DevOps `componentMappings` concept. | | Config Center update | Continue letting Publisher create `UpdateNgGlobalComponentImage` Operations. | | Approval | Add one `DeliverKernelImage` Operation in each target DevOps environment; its standard Operation approval gates the local `/fid` request. | | dev/prod ownership | Build once, but notify dev DevOps and prod DevOps separately. Each database creates and controls its own Operation. | | Correlation | Use the approved delivery's UUID as `/fid cid=` and carry it through FID `META`, Tekton, Publisher result, and the DevOps result callback. | | Formal release scope | The automatically created approval always delivers to all targets configured by that environment's FID policy. | | Manual scope | A later manual request may select one provider; this requires a small provider-filter extension in FID and does not affect formal-release M x N. | | Storage | Retain `kernel_image_releases` and `kernel_image_sync_targets`; the second table holds repeated cloud/Operation results, not a new workflow engine. | | M x N | Keep M logical versions, each with up to N concrete image tags. Recalculate after the environment's FID delivery and Publisher ticket creation both succeed. | | Cleanup | Out of scope; another service owns image deletion. | ## Motivation or Background ### Terminology | Term | Meaning | | --- | --- | | Version | Logical Config Center version, for example `v26.3.99` or `v8.5.4`. | | Image tag | Concrete TiBuild output, for example `v26.3.9-nextgen` or `v8.5.4-nextgen.202510.35`. | | M x N | At most M logical versions, with at most N eligible image tags under each version. | | FID | Fast Image Delivery, implemented by GitHub Actions and scripts in `docker-image-controller`. | | Publisher | Service in `PingCAP-QE/ee-apps/publisher`; for this flow it maps an image repository to Components and calls DevOps CloudConfig APIs. | | Delivery correlation ID | UUID linking one local `DeliverKernelImage` Operation to its `/fid` request, FID result, Publisher tickets, and final result callback. | | Local environment | The DevOps database and delivery stage receiving the record: `dev` or `prod`. Staging is used for validation but is not a substitute for either database. | ### Why the current chain was confusing The work is spread across several repositories and event systems. In particular: - `PingCAP-QE/ci` is a repository containing Tekton YAML; the Git repository itself does not listen to Issue comments. - `PingCAP-QE/ee-ops` deploys an EventListener and selects/deploys those Tekton Trigger/Task definitions. - FID writes its result to an Issue in `tidbcloud/docker-image-controller`, not to `PingCAP-QE/ci`. - Publisher does not copy images. It runs only after FID writes a result comment. - The final CloudConfig changes are new Operations; they are not completion callbacks on the original Release Change. ### Environment topology DevOps dev, staging, and prod have separate databases. IDs in one database are not globally meaningful in another. The currently observed TiBuild/EE endpoint configuration is: | DevOps deployment | Current TiBuild/EE endpoint | Database | | --- | --- | --- | | dev | `https://do2.pingcap.net/tibuild` | dev-only | | staging | `https://internal2-do.pingcap.net` | staging-only | | prod | `https://do2.pingcap.net/tibuild` | prod-only | Dev and prod sharing one TiBuild endpoint does not merge their DevOps state. A built artifact can be shared, but delivery approvals and records must be written separately to dev and prod. ## Detailed Design ### Verified current implementation Everything in this section describes the implementation observed in the repositories on 2026-08-05. Proposed behavior is kept in the next section. #### Current chain at a glance ```mermaid sequenceDiagram autonumber actor User participant Origin as Origin DevOps participant TiBuild participant Build as Tekton build/delivery participant Queue as docker-image-controller Issue participant FID as FID GitHub Actions participant Clouds as AWS / Aliyun / Azure participant EL as ee-ops EventListener participant Notify as PingCAP-QE/ci Task participant Publisher participant DevOps as dev or prod DevOps User->>Origin: Create and approve ng-image-release Origin->>TiBuild: bump-tag-for-tidbx(repo, branch, commit, author, release/change) TiBuild-->>Origin: repo, tag, commit accepted TiBuild->>Build: Git tag/build event Build->>Queue: /fid env=dev + image list Build->>Queue: /fid env=prod + image list Queue->>FID: issue_comment event dispatches workflow FID->>Clouds: Copy to the stage's configured registries FID->>Queue: Result comment + hidden META Queue->>EL: GitHub issue_comment webhook EL->>Notify: Start update-ops TaskRun Notify->>Publisher: POST stage + source image Publisher->>Publisher: Scan Components by base_image loop Every matching Component Publisher->>DevOps: Create UpdateNgGlobalComponentImage DevOps-->>Publisher: Operation instance ID end Publisher-->>Notify: Component/version/ticket list ``` #### Step 1: `ng-image-release` asks TiBuild to create the tag After the existing Release Change approval, DevOps calls TiBuild's `bump-tag-for-tidbx` API with repository, branch, commit, applicant/author, and source Release/Change metadata. TiBuild returns values such as `repo`, `tag`, and `commit`. The current Release deployment treats HTTP 200/201 as a successful deploy call. That means **TiBuild accepted tag creation**; it does not mean the image has been built, copied to clouds, or applied to Config Center. The later build and delivery steps are asynchronous and are not represented as child states of the original change. #### Step 2: build success creates two `/fid` requests The TiDB X successful-image event reaches the Tekton Trigger `notified-successful-image-delivery-tidbx`, which starts: ```text PingCAP-QE/ci/tekton/v1/tasks/delivery/ pingcap-notify-to-deliver-images-to-cloud-tidbx.yaml ``` The Task explicitly loops over: ```bash for stage in dev prod ``` For each stage it: 1. Reads the stage-specific GitHub repository and token. 2. Finds the newest open queue Issue with label `delivery-queue-tidbx/`. 3. Rotates the Issue after the configured comment limit. 4. Writes a comment containing `/fid env=` followed by the source images. The current queue repository is `tidbcloud/docker-image-controller`. At the time of verification the active Issues were #778 for dev and #779 for prod. These numbers are not permanent because the Task rotates queue Issues. This is why a release originating in either dev or prod can still produce both a dev FID request and a prod FID request: the fan-out is hard-coded in the post-build Tekton Task, not controlled by the originating DevOps database. #### Step 3: the FID dispatcher starts the copy workflow `docker-image-controller/.github/workflows/fast-image-delivery-dispatch.yml` listens for Issue comments whose first line starts with `/fid`. Supported syntax is: ```text /fid env= [cid=] ``` The dispatcher parses `env`, optional `cid`, and images, and dispatches `fast-image-delivery.yml` with: - `images` - `env_stage` - `reply_issue_number` - `reply_comment_id` - `reply_actor` - `correlation_id` The workflow selects the stage-specific image-sync YAML and runs `hack/nextgen-image-sync.sh`. The script resolves all target registries configured for the source repository set and calls `regctl image copy` for every destination. The current TiDB X policies contain AWS ECR, Aliyun ACR, and Azure ACR targets for both dev and prod. FID, not Publisher or DevOps, owns registry credentials, GitHub OIDC/cloud federation, destination routing, and copy commands. #### Step 4: FID writes the result to the same queue Issue The dispatched workflow receives the request Issue number and writes its result back to that same Issue. Its hidden metadata currently has this shape: ```json { "images": ["us.gcr.io/pingcap-public/tidbx/tikv:v26.3.9-nextgen"], "target_images": ["", "", ""], "correlation_id": "", "stage": "dev" } ``` It is embedded as: ```html ``` The visible comment title says either success or failure and links to the GitHub Actions run. ##### Important current correctness gap The result-comment step uses `if: always()`. The `META` object does not include a success boolean, and `target_images` is the planned/generated destination list rather than proof that every copy succeeded. The downstream Tekton filter currently checks only that: - `META` exists; - `images` is non-empty; and - source image tags match the accepted format. It does not test the FID job conclusion. Therefore, based on the code, a failed or partially failed FID run can still produce metadata that satisfies the Publisher trigger. This is a code-level risk and must be fixed before approval-driven delivery is considered reliable. #### Step 5: GitHub webhook reaches the deployed Tekton EventListener The result comment generates a GitHub `issue_comment` webhook. In `ee-ops`, the deployed EventListener configuration is: ```text PingCAP-QE/ee-ops/apps/gcp/tekton/configs/triggers/event-listeners/el-github.yaml ``` It accepts created comments containing a `META` block. `ee-ops` also deploys Trigger resources from `PingCAP-QE/ci` through its Tekton/Flux configuration. The matching Trigger is: ```text PingCAP-QE/ci/tekton/v1/triggers/triggers/env-gcp/_/notify/ update-ops-when-tidbx-image-distributed-to-clouds.yaml ``` It extracts `images`, `target_images`, and `stage` from `META`, then creates a TaskRun using `pingcap-notify-to-update-ops-tidbx`. So the precise relationship is: ```text GitHub sends the webhook -> ee-ops hosts the EventListener -> the EventListener selects a Trigger definition deployed from PingCAP-QE/ci -> the Trigger starts a Task definition deployed from PingCAP-QE/ci ``` `PingCAP-QE/ci` does not directly poll or subscribe to the Issue. The exact GitHub-side webhook/GitHub App configuration could not be inspected with the available token because repository hook administration permission was unavailable. The downstream EventListener and Trigger routing above are verified from code. #### Step 6: the Tekton Task calls Publisher For each source image, `pingcap-notify-to-update-ops-tidbx.yaml` calls: ```http POST /tidbcloud/devops/cloudconfig/versions/component Content-Type: application/json { "stage": "dev", "image": "us.gcr.io/pingcap-public/tidbx/tikv:v26.3.9-nextgen" } ``` The Task does not call the DevOps CloudConfig endpoint directly. It calls Publisher once per source image and prints the returned tickets. #### Step 7: Publisher discovers Components and creates Operations The implementation is in: ```text PingCAP-QE/ee-apps/publisher/internal/service/impl/tidbcloud/ops.go ``` Publisher: 1. Parses the source image into repository and tag. 2. Rejects unsupported non-nextgen tags. 3. Loads the Publisher configuration for `stage`. 4. Scans every configured Component and selects those whose `base_image` exactly equals the source image repository. 5. Derives the logical version from the image tag and currently normalizes `v26.3.*` to `v26.3.99`. 6. Queries TiBuild tag metadata to recover author, source Release ID, and source Change ID. 7. Calls the stage-specific DevOps CloudConfig API once for every matching Component. 8. Returns a ticket containing Component, logical version, Operation ID, and Operation URL. For example, the same `tidbx/tikv` image repository is used by multiple configured Components, including `tikv`, `tikv_worker`, and other workers. Publisher therefore creates separate Operations for all matching Components. The calls are sequential and are not one atomic transaction. This existing scan is the reason DevOps should not maintain a second `componentMappings` configuration. #### Step 8: DevOps creates `UpdateNgGlobalComponentImage` Publisher calls the existing stage-specific CloudConfig endpoint with: - `cluster_type` - logical `version` - source `base_image` - concrete image `tag` - `policy=immediate` - TiBuild metadata `author`, `release_id`, and `change_id` The API creates an `UpdateNgGlobalComponentImage` Operation and returns its instance ID. Publisher converts it into the stage-specific Operation URL and returns all tickets to the Tekton Task. There is no final callback to the original `ng-image-release` Change, no durable parent object covering both FID and all Publisher tickets, and no automatic update of the new kernel-image tables in this current chain. #### Current responsibility map | Responsibility | Current owner | | --- | --- | | Approve source tag creation | Origin DevOps Release Change | | Create Git/image tag and retain tag metadata | TiBuild | | Start build and initial dev/prod notification | Tekton definitions from `PingCAP-QE/ci` | | Find stage queue Issue and write `/fid` | Tekton delivery Task | | Copy to AWS/Aliyun/Azure | FID in `docker-image-controller` | | Emit result event | FID result Issue comment + `META` | | Receive Issue webhook | Tekton EventListener deployed by `ee-ops` | | Map image repository to Components | Publisher | | Derive logical version | Publisher | | Create `UpdateNgGlobalComponentImage` | Publisher calling the stage's DevOps API | | Track the whole delivery | No current owner | ### Recommended future flow #### Design objective Add an approval before cloud delivery and make the result visible in each DevOps database while retaining the working execution chain. The proposed change is: ```mermaid flowchart TB B[TiBuild/Tekton build succeeds] --> CE[Existing delivery.images.success CloudEvent] CE --> EL[Existing el-notify EventListener and Trigger] EL --> NT[Changed Notify Task] NT --> DCB[Callback dev DevOps] NT --> PCB[Callback prod DevOps] DCB --> DA[dev DeliverKernelImage Operation] PCB --> PA[prod DeliverKernelImage Operation] DA -->|approved| DQ[Post /fid env=dev cid=delivery_uid] PA -->|approved| PQ[Post /fid env=prod cid=delivery_uid] DQ --> FID[FID existing dispatcher and copy workflow] PQ --> FID FID --> META[Result comment with success + META] META --> TK[Tekton existing result trigger] TK -->|success only| PUB[Publisher existing Component scan] PUB --> OPS[Existing UpdateNgGlobalComponentImage Operations] TK --> RES[Callback corresponding DevOps with FID result and Publisher tickets] RES --> MXN[Update local release and M x N] ``` The build-success CloudEvent, its EventListener/Trigger, FID, the result Issue, Publisher scanning, and CloudConfig Operations already exist. The new work changes the Notify Task behind the existing build-success event, then adds build callbacks, approvals, correlation, a truthful result field, and a final correlated status callback. #### What changes and what stays unchanged | Area | Future behavior | Change size | | --- | --- | --- | | Build | Existing build and `delivery.images.success` CloudEvent remain unchanged. The current triggered Notify Task calls dev and prod DevOps instead of directly issuing both `/fid` commands. | Small change to one post-build Task. | | Approval | Each callback creates a release record and local `DeliverKernelImage` Operation. The Operation framework creates its approval. | DevOps work. | | FID request | The future Operation action posts the same `/fid env= cid=` command to the existing queue after approval. | Extend DevOps GitHub client or add a very small queue adapter. | | Cloud copy | Existing FID workflow and cloud configuration remain. | No redesign. | | Result event | Add explicit `success`, workflow URL/run ID, and keep `correlation_id` in `META`. | Small FID workflow change. | | Component mapping | Existing Publisher scan remains authoritative. | No DevOps mapping and no new resolver API. | | Operations | Publisher continues to create Operations and returns tickets. | Add idempotency/correlation metadata if available. | | Completion | Tekton calls the local DevOps result API after FID/Publisher finishes. | Small Task and DevOps API change. | #### Environment-local approvals The Notify Task triggered by the existing build-success CloudEvent sends the same artifact facts to two endpoints: ```text dev callback -> dev DB -> dev DeliverKernelImage Operation -> /fid env=dev prod callback -> prod DB -> prod DeliverKernelImage Operation -> /fid env=prod ``` It must not ask one DevOps database to create or approve work in another database. The callbacks share a global artifact identity, image, commit, tag, and digest. They differ in `target_devops_env`. Each DevOps generates its own local delivery UUID and approval ID. If dev acknowledges but prod is temporarily unavailable, the callback producer retries only prod. Duplicate callbacks are idempotent by `artifact_uid` within each database. #### Concrete future example Assume TiBuild produces: ```text repo: pingcap/tiflash commit: ff685313539da0046e0951a02f8de2f0e2791602 git/image tag: v26.3.8-nextgen logical version: v26.3.99 source image: us.gcr.io/pingcap-public/tidbx/tiflash:v26.3.8-nextgen source digest: sha256:abc... source applicant: user@pingcap.com ``` ##### 1. The existing build-success event causes callbacks to both DevOps environments The dev payload is: ```http POST /api/v1/devops/kernel-images/build-callback { "artifact_uid": "6d402aed-2d21-4ee8-8b30-c4a9ff61121d", "target_devops_env": "dev", "source_devops_env": "prod", "source_applicant": "user@pingcap.com", "source_release_id": "30082", "source_change_id": "62440", "cluster_type": "DEDICATED_TIDB", "component": "tiflash", "version": "v26.3.99", "repo": "pingcap/tiflash", "branch": "release-nextgen-202603", "commit_sha": "ff685313539da0046e0951a02f8de2f0e2791602", "git_tag": "v26.3.8-nextgen", "source_repository": "us.gcr.io/pingcap-public/tidbx/tiflash", "source_tag": "v26.3.8-nextgen", "source_digest": "sha256:abc..." } ``` Prod receives the same artifact with `target_devops_env=prod`. `version` must be supplied explicitly by the build-result adapter using the same normalization already implemented by Publisher. DevOps stores it and does not introduce another tag parser. ##### 2. Each DevOps creates its own delivery Operation The callback transaction stores the artifact first, then creates one `DeliverKernelImage` Operation in that same environment. The existing Operation framework creates and tracks the associated approval. The approval displays: - source applicant and source Release/Change link; - repository, branch, commit SHA, Git tag, image tag, and digest; - local delivery environment; - the formal rule that FID will deliver to all targets in its own policy for that environment; - the fact that Publisher will update every Component whose configured `base_image` matches this repository. The user cannot narrow the target scope on this automatically created formal delivery. DevOps neither stores nor presents its own copy of FID's provider list. ##### 3. The approved Operation posts one existing `/fid` command When the dev approval passes, DevOps writes: ```text /fid env=dev cid=0f934a92-69da-4ea7-8924-947a98d098af us.gcr.io/pingcap-public/tidbx/tiflash:v26.3.8-nextgen ``` The UUID is the dev row's `formal_delivery_uid`. Prod has a different UUID and issues its own `env=prod` command only after prod approval. This reuses the current dispatcher. DevOps does not need AWS, Aliyun, or Azure credentials. ##### 4. FID delivers and emits a truthful result Extend `META` additively: ```json { "images": ["us.gcr.io/pingcap-public/tidbx/tiflash:v26.3.8-nextgen"], "target_images": ["", "", ""], "correlation_id": "0f934a92-69da-4ea7-8924-947a98d098af", "stage": "dev", "success": true, "workflow_run_id": "30524708090", "workflow_run_url": "https://github.com/tidbcloud/docker-image-controller/actions/runs/30524708090" } ``` The result Trigger must never call Publisher when `success != true`. On failure it only reports the failed FID result to the corresponding DevOps row. In the minimal first version, one FID workflow remains the execution unit. DevOps creates no cloud target rows in advance or after completion. It records only the aggregate release status and the FID request/result URLs in Operation outputs. FID remains the source of truth for actual cloud targets. ##### 5. The existing Publisher path runs after success For a successful result, Tekton calls Publisher exactly as it does today. Publisher finds `tiflash_compute` and `tiflash_write` from its own configuration and creates the Operations. No `componentMappings` is read from DevOps. The returned tickets are the authoritative set for this delivery: ```json [ { "component": "tiflash_compute", "component_version": "v26.3.99", "id": "38735", "url": "https://ops-dev.tidbcloud.com/operations/38735" }, { "component": "tiflash_write", "component_version": "v26.3.99", "id": "38736", "url": "https://ops-dev.tidbcloud.com/operations/38736" } ] ``` ##### 6. Tekton reports the result to the matching DevOps database After Publisher returns, the Task calls only the DevOps environment named by `stage`: ```http POST /api/v1/devops/kernel-images/delivery-results { "correlation_id": "0f934a92-69da-4ea7-8924-947a98d098af", "stage": "dev", "fid": { "success": true, "target_images": ["", "", ""], "workflow_run_id": "30524708090", "workflow_run_url": "..." }, "publisher": { "success": true, "tickets": ["..."] } } ``` The callback is authenticated and idempotent by `correlation_id`. DevOps verifies that `stage` equals its configured local environment and that the source images match the stored release. If FID succeeds but Publisher fails, the release remains `CONFIG_UPDATE_FAILED`. Retrying should re-run only the Publisher notification, not copy the images again. This directly addresses the observed class of failure where prod Operations were created but the dev Operations were missing. ##### 7. DevOps updates M x N The local release becomes `AVAILABLE` when: ```text FID workflow succeeded AND Publisher returned all created Operation tickets successfully ``` Phase one tracks successful Operation **creation**, matching the current chain. Waiting for every Operation to finish would require an additional polling/reconciliation project and is not necessary to insert the approval gate. When the row becomes `AVAILABLE`, the selection API immediately recalculates the environment-local M x N result. Dev and prod can become available at different times. #### Why this proposal is smaller than the previous design The previous draft made DevOps a per-target dispatcher and a Component mapping owner. This proposal removes those responsibilities: | Removed complexity | Reused implementation | | --- | --- | | DevOps cloud-specific executor/client | FID GitHub Actions and sync script | | DevOps `componentMappings` | Publisher's existing `base_image` scan | | DevOps directly creating every Component Operation | Publisher's existing CloudConfig calls | | One workflow per cloud provider | One existing FID workflow per local formal delivery | | DevOps cross-environment calls | Separate build callbacks and approvals in dev/prod | | New global orchestration service | Existing Issue correlation plus local DevOps rows | The target table is retained because one release still produces multiple repeated results that the UI must show. It is storage, not a requirement to dispatch every row independently. #### Manual one-cloud synchronization Manual single-cloud delivery is useful but should be implemented after formal delivery works. Proposed request: ```json { "artifact_uid": "6d402aed-2d21-4ee8-8b30-c4a9ff61121d", "provider": "aliyun", "reason": "Repair the local Aliyun copy", "related_ticket": "https://..." } ``` Rules: - The current DevOps environment is implicit. A dev request can select only a target allowed by dev FID policy. - It requires its own approval. - The source image must already exist in `kernel_image_releases` in phase one. - FID must add an allowlisted provider input/filter; DevOps still does not implement the copy. - It does not invoke Publisher and does not make a formal release `AVAILABLE`. - It does not change M x N. ### DevOps data and APIs #### Existing first-slice status The dev branch already contains the image-release foundation: - `kernel_image_releases` and `kernel_image_sync_targets` models/migration; - authenticated `POST /api/v1/devops/kernel-images/build-callback`; - release list/detail APIs; - M x N selection API; - standard `DeliverKernelImage` Operation creation with an immediate auto action; - frontend API contracts already generated for these endpoints. This foundation does not yet change the real `ng-image-release` Notify Task or consume Publisher results. It creates a real environment-local Operation and approval, dispatches FID after approval, and consumes FID completion through the existing request-comment reactions. The backend foundation is aligned with the simplified ownership model: `componentMappings` and DevOps provider configuration are absent, and `build-callback` creates no target rows. The representation of future FID and Publisher results remains deferred rather than being fixed prematurely. #### Release storage and the legacy target table ##### `kernel_image_releases` One row stores immutable artifact facts and aggregate local state: | Important field | Purpose | | --- | --- | | `artifact_uid` | Idempotency/correlation of the same built artifact across dev and prod. | | `local_devops_env` | Prevents accepting a callback for another database. | | `cluster_type`, `component`, `version` | M x N grouping and Config Center lookup dimensions. | | `commit_sha`, `git_tag`, `source_tag`, `source_digest` | Immutable commit-to-image mapping. | | `source_applicant`, source Release/Change IDs | Audit and approval applicant context. | | `delivery_operation_instance_id` | Local `DeliverKernelImage` Operation instance. Its Approval ID stays inside the Operation model. | | `formal_delivery_uid` | UUID passed as `/fid cid`; final callback lookup key. | | `status`, `last_error_*`, timestamps | Aggregate UI state. | Multiple rows may have the same `cluster_type + component + version`, because N represents multiple image tags under one version. `artifact_uid` identifies one build. ##### `kernel_image_sync_targets` The existing table is not populated or read. DevOps does not pre-create or backfill cloud-provider or Component rows from FID results. Its eventual schema removal is separate from this non-destructive rollout. #### Configuration Minimal DevOps configuration: ```yaml kernelImage: enabled: true localDevOpsEnvironment: dev selectionPolicy: versionCount: 2 imageTagCount: 4 fid: repositoryOwner: tidbcloud repositoryName: docker-image-controller queueLabelPrefix: delivery-queue-tidbx queueCommentLimit: 2000 ``` `componentMappings` and provider/required-target configuration are intentionally absent. Dispatch integration configures only how to reach the FID queue; it does not duplicate FID's target registry policy. Only M and N are local DevOps selection policy. FID's stage YAML remains the sole authority for target registries, credentials, and repository routing. The UI displays actual targets returned by FID rather than a configured expected list. #### API surface ##### Existing APIs | Method and path | Purpose | | --- | --- | | `POST /api/v1/devops/kernel-images/build-callback` | Idempotently persist one environment-local build result. | | `GET /api/v1/devops/kernel-images/releases` | List releases; supports Component/version filtering and returns multiple image tags under one version. | | `GET /api/v1/devops/kernel-images/releases/:artifact_uid` | Release facts and its delivery Operation instance ID. | | `GET /api/v1/devops/kernel-images/selection` | Backend-computed M x N grouped as `versions[].images[]`. | ##### Minimal new APIs | Method and path | Purpose | | --- | --- | | `POST /api/v1/devops/kernel-images/delivery-results` | Authenticated correlated FID + Publisher result from Tekton/adapter. | | `POST /api/v1/devops/kernel-images/releases/:artifact_uid/retry-config-update` | Re-run Publisher only after FID success and Publisher failure. | Approval callbacks should use the repository's existing approval/event mechanism rather than a new public endpoint. Manual sync APIs are deferred until FID accepts a provider filter. #### M x N calculation Eligibility in the recommended minimal design is: ```text release.status == AVAILABLE AND masked == false ``` In the implemented first slice, `AVAILABLE` means the local FID workflow succeeded. Publisher completion is not yet part of local availability and may add a separate readiness dimension when that integration is implemented. The algorithm remains: 1. Load releases for `cluster_type + component`. 2. Group by exact logical `version`. 3. Sort versions semantically descending and retain M groups. 4. Within each version, sort concrete image tags by `available_at` descending and retain N. 5. Read the existing Config Center default tag for the same Component/version and mark `is_default`. 6. Protect an eligible default tag inside N when it would otherwise be pushed out by newer tags. 7. Return selected groups plus excluded rows and reason codes. Example response shape: ```json { "versions": [ { "version": "v26.3.99", "images": [ {"release": {"source_tag": "v26.3.9-nextgen"}, "is_default": true}, {"release": {"source_tag": "v26.3.8-nextgen"}, "is_default": false} ] } ] } ``` There is no cross-database selection transaction. Dev and prod expose their own result. #### State model ```mermaid stateDiagram-v2 [*] --> BUILT: build callback persisted BUILT --> DELIVERY_APPROVING: local Operation created DELIVERY_APPROVING --> SYNCING: approved and /fid posted DELIVERY_APPROVING --> BUILT: rejected/cancelled SYNCING --> CONFIG_UPDATING: FID success SYNCING --> SYNC_FAILED: FID failure CONFIG_UPDATING --> AVAILABLE: Publisher tickets returned CONFIG_UPDATING --> CONFIG_UPDATE_FAILED: Publisher failure SYNC_FAILED --> SYNCING: reissue /fid after authorization CONFIG_UPDATE_FAILED --> CONFIG_UPDATING: retry Publisher only ``` This state model distinguishes the two failure boundaries observed in the current chain without creating a generic workflow engine. ### Implementation plan #### Phase 0: correct the current result signal Before adding approval dispatch: 1. Add `success`, workflow run ID, and workflow URL to FID `META`. 2. Change the result Trigger so Publisher is called only when `success == true`. 3. Preserve failed-result notification so the future DevOps callback can show an error. This is an independent correctness fix for the existing chain. #### Phase 1: align the existing DevOps foundation 1. Remove `ComponentMappings` and formal provider/required-target configuration from `KernelImageServiceConfig`. 2. Create no `REGISTRY_SYNC` or `OPS_CONFIG` rows during `build-callback`. 3. Add the real local `DeliverKernelImage` Operation creation path. 4. Completed: replace the temporary manual action with the immediate FID auto action. #### Phase 2: put approval before FID 1. Keep the existing `delivery.images.success` CloudEvent, EventListener, and Trigger; change only their Notify Task from direct `/fid` fan-out to separate dev and prod DevOps callbacks, with independent retries. 2. Completed: implement the Operation action so that, after local approval, it finds the local FID queue Issue and posts `/fid env= cid=`. 3. Completed: persist the request comment URL as an Operation output and mark the release `SYNCING` only after GitHub accepts the comment. Poll FID's reaction and transition to `AVAILABLE` or `SYNC_FAILED`. 4. Keep the old automatic `/fid` fan-out behind a cutover flag to avoid duplicate delivery. #### Phase 3: correlated completion 1. Carry `correlation_id` and `success` from FID `META` through the Trigger and Task. 2. Keep the existing Publisher call for successful FID results. 3. Capture Publisher's returned tickets as JSON instead of only formatting them into a text file. 4. Call the corresponding environment's DevOps `delivery-results` API in both success and failure cases. 5. Correlate Publisher Component tickets without adding a DevOps-owned component mapping. 6. Transition to `AVAILABLE`, `SYNC_FAILED`, or `CONFIG_UPDATE_FAILED` idempotently. #### Phase 4: frontend and optional manual sync 1. Finish the Kernel Images page under Config Center using existing selection and release APIs. 2. Show M x N groups, commit/image mapping, approval, three cloud statuses, and returned Component Operation links. 3. After FID supports a provider filter, add manual one-cloud approval and dispatch. #### Repository changes | Repository | Minimal required changes | | --- | --- | | `tidbcloud/devops` | `DeliverKernelImage` Operation, FID queue comment dispatch, result callback, state transitions, APIs/UI data. | | `PingCAP-QE/ci` | Keep the build-success CloudEvent consumer and change its Notify Task from direct `/fid` fan-out to two DevOps callbacks; pass correlation/success and send Publisher results to local DevOps. | | `tidbcloud/docker-image-controller` | Add truthful result fields to `META`; later add optional provider filter for manual sync. | | `PingCAP-QE/ee-apps` Publisher | Keep current scan and Operation creation; optionally accept/pass correlation and idempotency metadata. No new Component resolver is required. | | `PingCAP-QE/ee-ops` | Deploy updated Trigger/Task/EventListener definitions and callback secrets/routes. | | `tidbcloud/app-delivery` | Define `DeliverKernelImage` per enabled DevOps environment and configure its local name, callback credentials, M, and N. Do not configure a provider list in DevOps. | ## Test Design ### Functional Tests #### DevOps - Identical build callbacks are idempotent by `artifact_uid`. - A callback for the wrong `target_devops_env` is rejected. - Commit, Git tag, source tag, and digest cannot change on a repeated callback. - Build callback creates no target rows; FID and Publisher result handling creates rows from actual output. - Real approval is created with the original applicant and exact artifact information. - Approval success posts exactly one `/fid` request with the correct local env and delivery UUID. - Duplicate approval events do not create duplicate Issue comments. - Delivery result rejects the wrong stage, unknown correlation ID, or mismatching image. - Successful Publisher tickets dynamically create unique `OPS_CONFIG` rows. - FID failure produces `SYNC_FAILED` and never `AVAILABLE`. - FID success plus Publisher failure produces `CONFIG_UPDATE_FAILED`. - Retrying config update does not post another `/fid` request. - FID and Publisher success produces `AVAILABLE` and changes M x N. - M versions and N concrete image tags per version are selected deterministically. Use Ginkgo/Gomega for new service tests and generated mocks for reusable GitHub/approval clients. #### FID and Tekton - Successful workflow emits `success=true` with the input correlation ID. - Failed validation or any failed copy emits `success=false`. - Failure metadata cannot enter the Publisher execution branch. - Result Trigger preserves `stage`, `correlation_id`, source images, target images, and workflow URL. - Publisher response tickets are preserved as structured JSON. - DevOps result callback is attempted for Publisher success and failure. #### Publisher - Existing repository-to-Component scan remains unchanged. - One repository shared by multiple Components returns every expected ticket. - Unsupported tags return no tickets and are treated as an explicit non-success for formal release. - Partial sequential Operation creation returns a clear failure and enough ticket data to diagnose/retry idempotently. ### Scenario Tests 1. **Happy path:** one build calls dev and prod; dev and prod approvals pass separately; each FID and Publisher path succeeds; both local M x N results include the image. 2. **Only dev approved:** dev completes and becomes available while prod remains `SYNC_APPROVING`. 3. **Prod callback temporarily unavailable:** dev row is created once; only prod callback retries; no duplicate dev approval. 4. **FID copy fails:** FID emits `success=false`; Publisher creates no Operation; the local release is visible as `SYNC_FAILED`. 5. **Publisher fails only in dev:** prod becomes available; dev becomes `CONFIG_UPDATE_FAILED`; retrying dev does not copy clouds again. 6. **Multiple Components:** one TiKV source image creates the same Component set Publisher currently discovers, without DevOps configuration changes. 7. **Lost DevOps result callback:** replaying the same correlation payload completes the same row without duplicate tickets. 8. **Repeated version builds:** two image tags under `v26.3.99` occupy two N slots and keep their own commits/digests. 9. **Manual single provider, later phase:** approval dispatches only the selected provider and does not invoke Publisher or alter M x N. ### Compatibility Tests - Existing `ng-image-release` tag creation is unchanged. - Before cutover, the current automatic `/fid` path still operates. - During cutover, a feature flag prevents old and new paths from dispatching the same artifact. - Existing manual `/fid` comments without `cid` remain accepted. - Existing result consumers tolerate additive `META` fields. - Existing Publisher clients and CloudConfig Operation payloads remain compatible. - dev and prod use only local database state and local API credentials. - Staging can test the flow without creating dev/prod records. - Database migration and application rollback are additive and do not affect existing Release/CloudConfig tables. ## Impacts & Risks ### Impacts - Cloud delivery waits for a separate approval in both dev and prod. - DevOps gains a durable commit-to-image record and an environment-local delivery view. - The working FID and Publisher execution paths remain operationally recognizable. - Publisher remains the only image-to-Component mapping owner. - M x N becomes queryable without changing Config Center's existing role as the current default-image store. ### Risks and mitigations | Risk | Impact | Mitigation | | --- | --- | --- | | Current `META` does not prove success | Failed copy may create Config Center Operations | Phase 0 adds `success` and gates Publisher before approval cutover. | | Old and new paths both post `/fid` | Duplicate delivery and Operations | Per-artifact cutover flag and stable correlation/idempotency keys. | | DevOps GitHub comment identity is not authorized by dispatcher | Approval cannot start FID | Validate the service account as repository collaborator in staging before rollout. | | Queue Issue rotates | Hard-coded Issue number becomes stale | Discover the newest open Issue by the existing stage label; never configure #778/#779 directly. | | FID succeeds but Publisher fails | Cloud images exist but M x N is incomplete | Separate `CONFIG_UPDATE_FAILED` state and Publisher-only retry. | | Publisher creates some Operations before failing | Retry could duplicate Operations | Add/pass an idempotency key derived from delivery UUID + Component before automatic retry. | | FID returns only unstructured target image strings | Provider labels may be ambiguous in the UI | Use the exact target repository as the stable key initially; add an additive structured target identifier to FID `META` instead of a DevOps mapping table. | | One environment callback is lost | Only dev or prod sees the artifact | Durable independent callback retries and alerts. | | Operation creation succeeds but execution later fails | Phase-one `AVAILABLE` can overstate final application | Clearly define phase-one availability as delivery plus Operation creation; add Operation polling only if product semantics require execution success. | ## Unresolved Questions 1. Which existing Approval template/type should represent `RELEASE_DELIVERY`, and who are the dev/prod approvers? 2. Should the approved delivery's GitHub queue comment be posted directly by DevOps using its GitHub client, or by a small authenticated Tekton queue adapter? Direct DevOps posting is the fewest runtime hops if the bot identity satisfies the dispatcher check. 3. Can Publisher/CloudConfig accept an idempotency key before Publisher-only automatic retry is enabled? 4. Is `AVAILABLE` after successful Operation creation sufficient, or must it wait for Operation execution? This document recommends creation success for the minimal first rollout because it matches the current chain. 5. Can FID add a structured target identifier beside each returned target image? Until then, DevOps uses the exact target repository as `target_key` and does not own a host-to-provider mapping. 6. When manual single-cloud sync is implemented, should FID expose an allowlisted target ID or provider name? The value must be defined and validated by FID; target URLs remain FID-owned configuration.