# Kernel Image Batch Build Callback and Delivery

Date: 2026-08-21

## Context

The currently enabled TiDBX delivery task submits one `/fid` comment containing
all images produced by a notification batch. The replacement Publisher flow is
not enabled yet, so there is no compatibility requirement for its proposed
single-image callback contract.

DevOps should preserve the existing batch boundary instead of reconstructing it
from multiple callbacks. Publisher will eventually submit one callback with the
complete image list. DevOps will persist one release per image, create one
`DeliverKernelImage` Operation for the batch, and submit one `/fid` command with
all source images.

## Existing FID batch provenance

The existing direct-to-FID flow does not pass through Publisher. Its batch is
created by one `pingcap-build-package-linux` PipelineRun:

1. The PipelineRun accepts one `git-url`, `git-ref`, `git-revision`, and
   `component`.
2. Its single checkout produces one commit, which is passed to
   `pingcap-build-images` as `git-sha` together with the same `git-url` and
   `git-ref`.
3. `pingcap-build-images` may build several image artifacts from that checkout.
   For example, the `tidb` NextGen package definition can produce `tidb`, `br`,
   `dumpling`, and `tidb-lightning` images from the `pingcap/tidb` repository.
   Every artifact receives the same OCI source, revision, and ref labels.
4. The build result contains one entry per image repository plus its primary,
   additional, and multi-architecture tags. The delivery generator flattens all
   of those entries and aliases into `delivery-targets.yaml.images` after
   applying the standard delivery rules.
5. After delivery succeeds, `pingcap-deliver-images` sends that file as the
   `net.pingcap.tibuild.notify.delivery.images.success` CloudEvent body. The
   TiDBX notification Trigger passes the complete event data to one TaskRun,
   which emits one `/fid` comment per stage followed by every `.images[]` item.

Consequently, under the normal Pipeline path, all images and aliases in one old
`/fid` comment share one source repository, commit SHA, and git ref even when
their image repositories differ. For example, `tidb`, `br`, and `dumpling`
image repositories can all come from one `pingcap/tidb` checkout. A fixed alias
such as `master-<short-sha>-nextgen` and its moving `master-nextgen` alias are
also generated from that same build context.

This invariant is structural rather than encoded in the CloudEvent:
`delivery-targets.yaml.images` contains only image references and does not carry
or validate their source repository, commit SHA, or git ref. The reusable
`pingcap-deliver-images` Task can technically be called with an arbitrary
`images-info` list, including a mixed-provenance list, even though the normal
Pipeline does not do that.

When Publisher is changed to call the batch callback, it must therefore read
the OCI source, revision, and ref labels from every image and verify that they
are identical before submitting the batch-scoped `repo`, `commit_sha`,
`branch`, and `git_tag` fields. Publisher must reject a mixed-provenance batch
instead of copying the first image's metadata to every release. DevOps treats
one accepted callback as an already validated source-build boundary.

Relevant upstream definitions:

- [Build Pipeline](https://github.com/PingCAP-QE/ci/blob/afbb58e9cc16c0ea6a05c9995525d8a074284a8b/tekton/v1/pipelines/pingcap-build-package-linux.yaml)
- [Image build Task](https://github.com/PingCAP-QE/ci/blob/afbb58e9cc16c0ea6a05c9995525d8a074284a8b/tekton/v1/tasks/pingcap-build-images.yaml)
- [Image delivery Task and success CloudEvent](https://github.com/PingCAP-QE/ci/blob/afbb58e9cc16c0ea6a05c9995525d8a074284a8b/tekton/v1/tasks/delivery/pingcap-deliver-images.yaml)
- [TiDBX delivery Trigger](https://github.com/PingCAP-QE/ci/blob/afbb58e9cc16c0ea6a05c9995525d8a074284a8b/tekton/v1/triggers/triggers/env-gcp/_/notify/notified-successful-image-delivery-tidbx.yaml)
- [Direct FID notification Task](https://github.com/PingCAP-QE/ci/blob/afbb58e9cc16c0ea6a05c9995525d8a074284a8b/tekton/v1/tasks/delivery/pingcap-notify-to-deliver-images-to-cloud-tidbx.yaml)
- [Delivery image-list generator](https://github.com/PingCAP-QE/artifacts/blob/main/packages/scripts/gen-delivery-image-commands.ts)

## Goals

- Treat one build callback request as one explicit delivery batch.
- Persist image-level release facts so Config Center M x N selection continues
  to operate on individual images.
- Create exactly one local `DeliverKernelImage` Operation for a callback batch.
- Resolve every missing source digest before delivery.
- Submit exactly one `/fid` comment containing every source image in request
  order.
- Apply the FID result back to every image release in the batch.
- Make retries idempotent without a timer, collection window, or background
  aggregation job.

## Non-goals

- Supporting the unenabled single-image callback request.
- Changing Publisher or PingCAP-QE/ci in this change.
- Changing the docker-image-controller `/fid` command or result format.
- Changing the `DeliverKernelImage` definition in app-delivery.

## Callback API

`POST /api/v1/devops/kernel-images/build-callback` accepts only a non-empty
`images` array. Source build metadata is batch-scoped because all aliases and
components in one TiDBX notification come from the same source build.

```json
{
  "source_applicant": "xieyujie@pingcap.com",
  "source_release_id": "30082",
  "source_change_id": "62440",
  "repo": "pingcap/tidb",
  "branch": "master",
  "commit_sha": "66f307f000000000000000000000000000000000",
  "images": [
    {
      "source_repository": "us.gcr.io/pingcap-public/tidbx/tidb",
      "source_tag": "master-66f307f-nextgen"
    },
    {
      "source_repository": "us.gcr.io/pingcap-public/tidbx/br",
      "source_tag": "master-66f307f-nextgen"
    }
  ]
}
```

The fields have these rules:

- `images` is required and must contain at least one item.
- Each image requires `source_repository` and `source_tag`.
- `source_digest` is optional per image. DevOps resolves it after approval when
  absent.
- `source_applicant` remains optional and defaults to
  `devops-internal@pingcap.com` when absent.
- `repo` and `commit_sha` are required batch facts.
- `branch` and `git_tag` remain optional, mutually exclusive source ref facts.
- Duplicate `source_repository + source_tag` entries in one request are
  rejected.

The response contains every release and the single Operation instance ID:

```json
{
  "errcode": "OK",
  "releases": [],
  "instance_id": 32535
}
```

There is intentionally no legacy top-level `source_repository`, `source_tag`,
or `source_digest` fallback.

## Identity and persistence

Each image remains a row in `kernel_image_releases`. A new indexed
`delivery_batch_uid` column associates all rows from one callback.

The identities are deterministic:

- `artifact_uid = UUIDv5(source_repository + source_tag + commit_sha)`
- `formal_delivery_uid = UUIDv5(local environment + artifact_uid)`
- `delivery_batch_uid = UUIDv5(local environment + sorted artifact_uids)`

Including `commit_sha` in `artifact_uid` makes moving aliases such as
`master-nextgen` reusable across builds. The database uniqueness constraint is
changed from `source_repository + source_tag` to
`source_repository + source_tag + commit_sha`.

For existing rows, migration sets `delivery_batch_uid = formal_delivery_uid`;
every historical delivery was a one-image batch.

An identical retry derives the same artifact and batch UIDs, returns the same
release rows, and reuses their Operation. If any artifact is already associated
with a different batch, the callback fails with an immutable artifact conflict.

## Operation creation

DevOps creates all releases first, then creates one `DeliverKernelImage`
Operation and links every release to its instance ID in one transaction.

The existing app-delivery Operation definition remains unchanged. Its existing
parameters are populated as follows:

- `artifact_uid`: the first release artifact UID, retained as a representative
  value for the existing definition.
- `local_devops_environment`: the local DevOps environment.
- `source_repository` and `source_tag`: the first image, retained as
  representative values.
- `delivery_correlation_id`: the `delivery_batch_uid`.

The Kernel Image actions use `delivery_correlation_id` to load the complete
batch. The representative parameters are not used to decide which images are
delivered.

The Operation includes `ResolveKernelImageDigest` when at least one release is
missing a digest, followed by `DeliverKernelImage`.

## Digest resolution

`ResolveKernelImageDigest` loads all releases by `delivery_batch_uid` and starts
the existing image availability query for every release without a digest. The
query handles are serialized into the action output so subsequent Result polls
can resume all queries.

The action succeeds only after every query succeeds and each digest is stored on
its release. If a query fails, the whole batch is marked `SYNC_FAILED`; no
`/fid` request is sent.

## FID submission and result handling

The FID client accepts an ordered `Images []string` request and builds:

```text
/fid env=<local_env> cid=<delivery_batch_uid>
<source_repository_1>:<source_tag_1>
<source_repository_2>:<source_tag_2>
...
```

`DeliverKernelImage` verifies that all releases have digests, submits the batch
once, and marks all releases `SYNCING`.

docker-image-controller already returns both `images` and `target_images` in
the FID result META block. DevOps parses both fields. It first verifies that the
source image set equals the release set, then assigns target images to a release
by the source image's final repository component and exact tag. For example,
targets ending in `/tidb-lightning:master-nextgen` belong to the
`tidb-lightning:master-nextgen` source release. Ambiguous or unmatched results
fail the whole batch instead of silently attaching targets to the wrong image.

On success, each release receives only its own target images and availability
snapshot, while all releases transition to `AVAILABLE` together.

## Failure and retry semantics

- Validation failure creates no releases or Operation.
- Release persistence or Operation creation failure fails the callback. Created
  releases retain a retryable failure status when Operation creation fails.
- A callback retry with the identical batch resumes Operation creation without
  duplicating releases.
- Digest resolution failure fails the batch before FID submission.
- FID failure marks every release in the batch `SYNC_FAILED`.
- Operation and FID idempotency use the deterministic `delivery_batch_uid` as
  the correlation ID.

## Test plan

1. Callback validation rejects an empty image list and duplicate image refs.
2. A multi-image callback creates N releases and one Operation.
3. Retrying the same batch in a different image order reuses the same Operation.
4. Reusing a moving tag with a new commit creates a new artifact and batch.
5. Missing applicants default to `devops-internal@pingcap.com` for every release.
6. Digest resolution starts and resumes one query per missing digest.
7. FID command rendering emits one command with all image lines in order.
8. FID result parsing reads `images` and `target_images` and maps targets to the
   correct release.
9. One failed digest or FID result marks the entire batch failed.
10. Release-tag versions still enter M x N selection; branch build versions stay
    empty and remain excluded.