# Kernel Image Releases, Approval-Gated Delivery, and the M × N Image View

## Metadata

| Item | Value |
| --- | --- |
| Status | In progress |
| First drafted | 2026-08-06 |
| Last updated | 2026-08-10 |
| Scope | Notify callback, DevOps image records, `DeliverKernelImage`, FID, multi-cloud availability, and M × N |
| Related design | [NG image cloud sync management](./2026-08-05-ng-image-cloud-sync-management.md) |

## Table of Contents

- [Introduction](#introduction)
- [Motivation or Background](#motivation-or-background)
- [Detailed Design](#detailed-design)
  - [Design Principles and Responsibility Boundaries](#design-principles-and-responsibility-boundaries)
  - [End-to-End Flow](#end-to-end-flow)
  - [Notify Callback Contract](#notify-callback-contract)
  - [Version and Stable Identifiers](#version-and-stable-identifiers)
  - [Data Model](#data-model)
  - [Approval-Gated FID Delivery](#approval-gated-fid-delivery)
  - [Dynamic M × N View](#dynamic-m--n-view)
  - [Multi-Cloud Availability Checks](#multi-cloud-availability-checks)
  - [API](#api)
  - [Configuration](#configuration)
- [Test Design](#test-design)
  - [Functional Tests](#functional-tests)
  - [Scenario Tests](#scenario-tests)
  - [Compatibility Tests](#compatibility-tests)
- [Impacts & Risks](#impacts--risks)
  - [Impacts](#impacts)
  - [Risks and Mitigations](#risks-and-mitigations)
- [Rollout Plan](#rollout-plan)
- [Unresolved Questions](#unresolved-questions)

## Introduction

This design separates the NextGen Kernel Image lifecycle into three clearly bounded concerns: build facts, approval-gated delivery, and component-selectable images. After an image build succeeds, Notify combines TiBuild metadata with the build result and calls back each environment's DevOps service. DevOps stores `repository + version + tag` at the image level, creates a `DeliverKernelImage` Operation, and calls the local environment's FID after approval. When the M × N page is queried, DevOps dynamically matches image records against the component's `base_image + version` in Config Center.

DevOps does not store duplicate configuration such as component, cluster type, provider, required targets, or component mappings. This design also does not introduce a general-purpose workflow engine, Catalog, `REGISTRY_SYNC`, `OPS_CONFIG`, or any other unnecessary abstraction.

The implementation centers on the DevOps repository: the new callback contract, image-scoped releases, Operation parameters and Actions, dynamic M × N queries, and availability scans. TiBuild metadata, Notify Task, Publisher, frontend, and deployment configuration are described where they define integration contracts or rollout order.

## Motivation or Background

After receiving `delivery.images.success`, the old Notify Task wrote one `/fid` comment to each of the dev and prod FID Queue Issues. That flow offered no independent approval per environment and did not give DevOps a unified record of builds, delivery, and multi-cloud availability.

The first DevOps implementation required the callback to provide `cluster_type/component/version/artifact_uid/target_devops_env` and bound each release to a component. This caused several problems:

- Notify does not know which components Publisher will ultimately match to the image.
- One image can serve multiple components, so persisting the component duplicates image records and delivery state.
- If Publisher fails to create `UpdateNgGlobalComponentImage`, the M × N page cannot discover a newly delivered tag.
- DevOps can derive `version` and `artifact_uid` deterministically from image facts, so upstream callers should not provide arbitrary values for them.
- Scheduled availability checks should not depend on component selection groups.

The final model is therefore image-centric. A component is only a dynamic filter when the M × N view is queried.

Terminology:

| Term | Meaning |
| --- | --- |
| Source image | The image produced by Tekton, identified by `source_repository:source_tag` |
| Logical version | The product version derived consistently by DevOps from the tag, for example `v8.5.4` or `v26.3.99` |
| Artifact UID | A stable UUID generated by DevOps from the source repository and source tag; identical in dev and prod |
| Delivery CID | A stable UUID generated by DevOps from the local environment and Artifact UID; different in dev and prod |
| M × N | At most M versions per component and at most N delivered tags per version |

## Detailed Design

### Design Principles and Responsibility Boundaries

| System | Responsibility |
| --- | --- |
| DevOps ng-image-release | Stores the original applicant, release/change, Git repository/branch, and other build-request context, then asks TiBuild to create a Git tag |
| TiBuild | Creates a Git tag, although some build paths may not have one; preserves applicant context and exposes a metadata query API |
| Tekton build/delivery | Builds the image and emits the existing `delivery.images.success` CloudEvent |
| Notify Task | Reads image facts from the event, queries TiBuild tag metadata once, and calls back dev and prod DevOps separately; it no longer writes `/fid` directly |
| DevOps Kernel Image | Derives the version, Artifact UID, and local Delivery CID; stores the release; creates and tracks the Operation; and exposes the M × N and availability APIs |
| `DeliverKernelImage` Operation | Reuses the standard Operation approval and Action lifecycle; resolves the source digest after approval and then triggers FID |
| FID | Determines target clouds from `env` and the image, performs synchronization, and writes the result back to the docker-image-controller Queue Issue |
| Publisher | Continues to scan its own component configuration and create `UpdateNgGlobalComponentImage`; this only updates the default Config Center tag and does not determine whether DevOps records the image |
| Config Center | Stores `base_image + tag` for every `cluster_type + component + version` and is the source of truth for component queries and default tags |

### End-to-End Flow

```mermaid
flowchart TD
    A[DevOps ng-image-release] -->|repo/branch/applicant/release/change| B[TiBuild creates Git tag]
    B -->|Git tag| C[Tekton builds image]
    C -->|delivery.images.success: image + Git facts| D[Notify Task]
    D -->|Query tag metadata| B
    D -->|Same build facts| E[dev DevOps build-callback]
    D -->|Same build facts| F[prod DevOps build-callback]
    E --> G[dev DeliverKernelImage Operation]
    F --> H[prod DeliverKernelImage Operation]
    G -->|Resolve source digest after approval| I[/fid env=dev cid=stable UUID]
    H -->|Resolve source digest after approval| J[/fid env=prod cid=stable UUID]
    I --> K[FID multi-cloud synchronization]
    J --> K
    K -->|GitHub reaction/result comment| G
    K -->|GitHub reaction/result comment| H
    C --> L[Publisher]
    L -->|Scan component mappings| M[UpdateNgGlobalComponentImage]
    N[Config Center component images] --> O[DevOps dynamic M × N query]
    E --> O
    F --> O
```

The existing image build, CloudEvent, EventListener, and Trigger remain unchanged. The only Notify cutover is changing the final side effect from writing two FID comments to making two DevOps callbacks.

### Notify Callback Contract

Endpoint:

```text
POST /api/v1/devops/kernel-images/build-callback
x-api-key: <environment-specific key>
Content-Type: application/json
```

Request body:

```json
{
  "source_applicant": "xieyujie@pingcap.com",
  "source_release_id": "123",
  "source_change_id": "456",
  "repo": "tidbcloud/cloud-storage-engine",
  "branch": "release-nextgen-20251015",
  "commit_sha": "ff685313539da0046e0951a02f8de2f0e2791602",
  "git_tag": "v8.5.4-nextgen.202510.1",
  "source_repository": "us.gcr.io/pingcap-public/tidbx/tikv",
  "source_tag": "v8.5.4-nextgen.202510.1"
}
```

Required fields and their sources:

| Field | Source |
| --- | --- |
| `source_repository/source_tag` | Tekton build/delivery result |
| `repo/commit_sha` | Delivery event and TiBuild metadata; `commit_sha` must be the source commit SHA, not an annotated-tag object SHA |
| `source_applicant` | Prefer `meta.ops_req.applicant`; fall back to the top-level `author` for legacy tags |

`source_release_id/source_change_id/branch/git_tag/source_digest` are optional. `branch` and `git_tag` are independent source-traceability fields: both, either, or neither may be present. `source_digest` supports callers that can already supply a digest. When Notify omits it, DevOps resolves it asynchronously after approval. `source_devops_env` is removed from the callback, domain model, and database because the target environment is determined by the DevOps endpoint being called.

Notify does not send these derived fields:

- `artifact_uid`
- `target_devops_env`
- `cluster_type`
- `component`
- `version`
- `delivery_correlation_id`

The target environment is determined by the DevOps endpoint called by Notify. Notify retries the TiBuild query and callbacks that fail because of network errors or HTTP 5xx responses. If either target environment ultimately fails, the Task fails. Retrying a Task after partial success is safe because the DevOps callback is idempotent.

The callback uses the existing authenticated `/api/v1/devops` route rather than exposing an unauthenticated internal endpoint. The caller no longer sends `Idempotency-Key`; the server generates the idempotency key so callers do not need to duplicate the Artifact UID algorithm.

### Version and Stable Identifiers

DevOps version derivation matches Publisher:

1. Support legacy tags such as `vX.Y.Z-nextgen.YYYYYY.N`.
2. Support calendar-semver tags such as `v26.0.0-nextgen` and `v26.3.1`, with a base version no lower than `v26.0.0`.
3. Use the substring before the first `-` as the base version.
4. During the transition, normalize every `v26.3.*` tag to `v26.3.99`.
5. Reject tags outside the supported formats so they do not create records that the M × N view can never use.

Stable UUIDs:

```text
artifact_uid = UUIDv5("tidbcloud/devops/kernel-images/artifacts/" + source_repository + ":" + source_tag)
fid cid      = UUIDv5("tidbcloud/devops/kernel-images/" + local_devops_env + "/" + artifact_uid)
```

Artifact UID does not include the digest. If a mutable tag is repushed with a different digest, DevOps reports an immutable conflict instead of silently creating another release. The FID CID includes the local environment, so approvals and FID requests for the same image are independent in dev and prod.

### Data Model

`kernel_image_releases` stores records at the image level:

| Category | Fields |
| --- | --- |
| Identity | `artifact_uid`, `local_dev_ops_env`, `version` |
| Git facts | `repo`, `commit_sha`, plus optional `branch` and `git_tag` |
| Image facts | `source_repository`, `source_tag`, plus `source_digest`, which may be resolved after approval |
| Audit | `source_applicant`, `source_release_id`, `source_change_id` |
| Delivery | Operation instance, stable CID, status, FID target images, and error details |
| Availability | Aggregate status, per-platform snapshot, check UID/time, and error details |

New records no longer store or use `cluster_type/component`. Environments that ran the earlier prototype must remove the legacy component-scoped indexes and columns before rollout. The code, APIs, and selection algorithm use only the final image-scoped schema.

A callback with the same `artifact_uid` and the same required facts returns the existing record. Optional metadata may fill previously empty values during a retry; if both the stored and incoming values are non-empty but differ, DevOps returns a conflict. Once the source digest is resolved or provided by the callback, a different value cannot overwrite it. The callback persists the release first and then idempotently creates one `DeliverKernelImage` Operation. If Operation creation fails, the release retains `DELIVERY_OPERATION_CREATE_FAILED`, and an upstream retry can continue creating the Operation without duplicating the release.

### Approval-Gated FID Delivery

The `DeliverKernelImage` Operation requires only these parameters:

- `artifact_uid`
- `local_devops_environment`
- `source_repository`
- `source_tag`
- `delivery_correlation_id`

The Operation definition contains an optional, immediate `ResolveKernelImageDigest` Action followed by the required, immediate `DeliverKernelImage` Action. When the callback omits the digest, both Actions are instantiated in order. When the callback provides a valid digest, only the delivery Action is instantiated. Neither path performs an external query or triggers FID before approval.

The Operation reason uses the full source image and no longer refers to a component. When the callback does not provide a digest, the Operation runs two sequential automated Actions after approval:

1. `ResolveKernelImageDigest` dispatches the existing `query-image-tag.yml` workflow in docker-image-controller, persists the workflow run ID, and polls it through the Operation Result lifecycle.
2. After the workflow succeeds, the Action reads the `sha256` digest from `result.json` and writes it back to the release.
3. `DeliverKernelImage` verifies that the digest exists and then sends the FID request.

The workflow handle is persisted as Operation instance outputs in the `operation_instances.outputs` JSON field. It contains the queried image reference, workflow run ID, and run URL, so the kernel image release table does not need dedicated workflow columns.

If the callback supplies a valid digest, DevOps skips the first Action. Before approval, neither path triggers an external query or FID. `DeliverKernelImage` sends:

```text
/fid env=<local_devops_environment> cid=<delivery_correlation_id>
<source_repository>:<source_tag>
```

The existing implementation continues to handle FID Queue Issue discovery and capacity rotation, stable-CID deduplication, polling for `github-actions[bot]` reactions, and result-comment parsing. After FID succeeds, the release moves to `AVAILABLE` and stores the target images returned by FID. On failure, it moves to `SYNC_FAILED`.

### Dynamic M × N View

The endpoint still requires:

```text
GET /api/v1/devops/kernel-images/selection?cluster_type=SHARED_POOL&component=tikv
```

`cluster_type` cannot be omitted because a component with the same name can exist under multiple cluster types in Config Center. It is a query condition, not a release attribute.

The view is calculated as follows:

1. Call Config Center `ListComponentImages(cluster_type, component, tenant_id=0)`.
2. Read `(base_image, version, tag)` from each configuration entry.
3. Match local releases where `source_repository == base_image && version == configured version`.
4. Only `AVAILABLE`, unmasked releases are selectable.
5. Sort matching versions by semver descending and keep at most M.
6. For each version, retain the current Config Center default tag first, then sort by availability/build/creation time descending and keep at most N tags.
7. Return selected items and excluded items with explicit reasons without deleting historical data.

With this approach, discovering a new tag under an existing component version does not depend on Publisher or `UpdateNgGlobalComponentImage` succeeding. As long as Config Center already contains the matching `base_image + version`, a new tag appears after FID succeeds. A completely new version that has never entered Config Center does not appear as an upgrade candidate for that component, which satisfies the goal of showing only versions with a safe component-configuration basis before an upgrade.

Publisher remains responsible for scanning component mappings and creating `UpdateNgGlobalComponentImage` to make a tag the Config Center default. DevOps does not copy Publisher configuration or create component targets during the callback.

### Multi-Cloud Availability Checks

Manual check:

```text
POST /api/v1/devops/kernel-images/releases/{artifact_uid}/availability-check
```

The API uses Artifact UID because the release uniquely stores the source digest and FID target images. Callers do not need to repeat the tag and cannot overwrite a record with an incorrect tag.

A scheduled check runs once per day. It directly scans every local release that is `AVAILABLE`, unmasked, has FID target images, and is older than the freshness window. It does not discover candidates through component or M × N groups. For each target registry, the check queries the digest, multi-architecture manifest, and OCI platforms; compares them with the source digest; and stores both a per-platform snapshot and an aggregate status.

The default cron expression, `0 0 * * *`, uses the service process's local time zone. Because the current container runs in UTC, it executes every day at 08:00 China Standard Time. If the deployment time zone changes, the cron configuration must be adjusted accordingly; `0 0` alone must not be interpreted as a fixed China Standard Time value.

### API

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/v1/devops/kernel-images/build-callback` | Store a successfully built image and create the local environment's delivery Operation |
| `GET` | `/api/v1/devops/kernel-images/releases` | List releases with pagination and filters such as image version and status |
| `GET` | `/api/v1/devops/kernel-images/releases/{artifact_uid}` | Get one image release |
| `GET` | `/api/v1/devops/kernel-images/selection` | Dynamically calculate M × N from a Config Center component |
| `POST` | `/api/v1/devops/kernel-images/releases/{artifact_uid}/availability-check` | Asynchronously refresh the image's availability on every target platform |

The release APIs no longer expose `cluster_type/component`; the selection request still retains both.

### Configuration

```yaml
services:
  kernelImage:
    enabled: true
    localDevOpsEnvironment: dev
    selectionPolicy:
      versionCount: 2
      imageTagCount: 4
    availabilityCheck:
      cronSpec: "0 0 * * *"
      concurrency: 2
    fid:
      repositoryOwner: tidbcloud
      repositoryName: docker-image-controller
      queueLabelPrefix: delivery-queue-tidbx
      queueCommentLimit: 2000
```

`localDevOpsEnvironment` remains required. It determines the FID `env`, the stable CID namespace, database-isolation validation, and list scope. It is not an upstream callback field.

## Test Design

### Functional Tests

- Derive versions correctly from legacy and calendar tags, including `v26.3.* -> v26.3.99`, and reject invalid tags.
- Generate a stable Artifact UID from repository and tag. The same image has the same UID in dev and prod, and retries in one environment do not change it.
- Retrying a callback with identical facts creates only one release and one Operation. Optional metadata can fill empty values but cannot overwrite different existing values.
- Reject caller-provided target environment, component, cluster type, or version fields.
- Use the original ng-image-release applicant as the Operation applicant instead of the user associated with the Notify API key.
- When the source digest is missing, resolve and persist it asynchronously after approval and then send FID. When a digest already exists, proceed directly to the delivery Action.
- Make Operation Action inputs depend only on the Artifact UID, local environment, source image, and stable CID.
- Match one or more release tags using a Config Center `base_image + version`.
- Query Config Center separately for components with the same name but different cluster types.
- Enforce both the version and tag limits in M × N while retaining the current default tag first.
- Scan every stale `AVAILABLE` release daily without depending on component selection.

### Scenario Tests

1. Simulate a Notify callback using a real build result without branch, Git tag, or digest. Verify that DevOps derives the version and UID and creates a `DELIVERY_APPROVING` Operation.
2. Before approval, verify that docker-image-controller receives no new `/fid` comment.
3. After approval, verify that a source-image query workflow runs before the `/fid` comment appears, that the digest is written back, that the comment environment matches the local DevOps environment, that the CID is stable, and that the release eventually becomes `AVAILABLE`.
4. If Config Center already contains a component's base image and version but `UpdateNgGlobalComponentImage` creation fails, verify that the new tag still appears in M × N and is not marked as the default.
5. Send the same callback to dev and prod. Verify that the Artifact UID is identical, the CIDs differ, and approvals are independent.
6. Repush the same tag with a different digest and verify that DevOps refuses to mutate the immutable release.
7. Verify that both manual and scheduled availability checks store per-cloud digest and platform results.

### Compatibility Tests

- An environment that never ran the earlier prototype can create the final image-scoped table directly. An environment that did run it must remove the legacy component-scoped indexes and columns before deploying the final model.
- When an old TiBuild tag lacks the new metadata, Notify can fall back to the top-level `author`; a missing branch or Git tag does not block the callback.
- Publisher continues to use the existing TiBuild query response; additional metadata fields do not break deserialization.
- EventListener and Trigger retain the existing event type and routing; only the Notify Task parameters and behavior change.
- Old Notify remains functional before cutover. After cutover, it must not write FID directly while also calling DevOps, which would cause duplicate delivery.

## Impacts & Risks

### Impacts

- DevOps releases change from component scope to image scope. The frontend release list removes component and cluster type, while the selection filters retain both.
- Notify needs TiBuild and dev/prod DevOps endpoint and credential configuration, and it supplies the applicant, Git repository and commit, and source image repository and tag.
- DevOps reuses the existing docker-image-controller `query-image-tag.yml` workflow to resolve the source digest, so the FID repository does not need a new workflow.
- Publisher behavior is unchanged. `UpdateNgGlobalComponentImage` is no longer a prerequisite for M × N to discover a new tag under an existing version.
- FID needs no changes for environment-level multi-cloud synchronization and result reporting. Synchronizing to one explicitly selected provider remains outside this phase.

### Risks and Mitigations

| Risk | Mitigation |
| --- | --- |
| Notify succeeds for one environment and fails for the other | Fail and retry the Task; stable UIDs and immutable validation make the DevOps callback idempotent |
| TiBuild metadata is incomplete | Prefer the applicant from operation metadata and fall back to the top-level author; make branch/Git tag optional and return a clear error when repository/commit is missing |
| The same tag is overwritten | Because Artifact UID excludes the digest, immutable comparison rejects the conflict |
| Publisher and DevOps version rules drift | Cover agreed tag cases in DevOps unit tests; a shared library can be extracted later, but this phase does not add a general abstraction |
| Config Center calls fail | Fail the selection request without mutating releases; release history and delivery are unaffected |
| A new version has not entered Config Center | Do not expose it as an upgrade candidate for a component; it enters the view naturally after Publisher updates Config Center successfully |
| Approval accidentally triggers an external synchronization | Enable dev first; approval remains the sole gate for the FID side effect; cut over prod only after its formal approval policy is configured |
| Scheduled scan volume grows | Scan only stale releases that are `AVAILABLE`, unmasked, and have targets, using bounded concurrency |

## Rollout Plan

1. Confirm that TiBuild and Notify can provide the original applicant, repository, and actual commit SHA; pass branch and Git tag when available.
2. Deploy the new DevOps callback contract, asynchronous digest resolution, image-scoped schema, and image-scoped M × N calculation. In any environment that ran the earlier prototype, clean up its legacy component-scoped indexes and columns first.
3. Simulate Notify with the API in dev and verify the complete flow: no FID before approval, synchronization after approval, and availability checks.
4. Update the dev frontend Swagger/client and page fields.
5. Configure the TiBuild and dev/prod DevOps credentials required by Notify, then deploy the new Notify Task.
6. Cut over Notify in dev and verify that it no longer writes FID directly.
7. Enable the prod callback only after its approval policy, credentials, and monitoring are ready.

When rolling back Notify, ensure that only one path—direct FID or the DevOps callback—is active at a time. Disabling `services.kernelImage.enabled` only blocks new callbacks; it does not cancel FID requests that have already been approved and sent.

## Unresolved Questions

1. Which Secret Manager keys will hold the TiBuild API credential and the dev/prod DevOps API keys used by Notify?
2. What approval stages and approver rules should the prod `DeliverKernelImage` Operation use?
3. Should historical component-scoped releases be backfilled and merged into image-scoped records, or retained only as audit history?