# Component Template Versioning Design

## Overview

This document defines component template versioning in Cluster Service NG.

Template version is defined as **template content hash** (`sha256`) rather than business-entity version fields. The system stores all historical component template versions in MySQL, selects a default available version per template, and enforces version availability during apply.

All templates loaded from `deployments/templates/` are version-registered in `template_registries`.
Apply API template version selection and usage-detector cron focus on dataplane component templates (TiDB/TiKV/PD/TiProxy/TiFlash/Workers/PD-TSO/PD-Scheduling/ResourceManager).

## Goals

- Use `sha256(normalized_template_content)` (64 hex chars) as template version.
- Keep only latest template files in `deployments/templates/`; store historical versions in DB.
- Re-register current templates on service startup.
- Support optional `template_version` in apply APIs.
- Support version status management: `AVAILABLE` / `DISABLED` / `EXPIRED`.
- Block apply for `DISABLED` and `EXPIRED` versions.
- Persist actual applied template metadata (name + version) into CR annotations and expose version via `Component.template_version`.
- Detect components using unavailable versions via cron + metrics.
- Mark components without template annotation as `UNKNOWN` in checks and emit dedicated metrics.

## Non-Goals

- Do not use legacy `TemplateVersion(int64)` in `SharedPool/TiDBGroup/TiDBCluster` as template-version source.
- Do not implement template diff API in this phase.

## Concepts

### Template Name

Template name is mapped from component type (via existing component-template mapping function) and persisted on CR annotation for audit/read-back.

### Template Version

Template version is `sha256` of normalized template content.

Normalization rule:
- Convert `CRLF` (`\r\n`) to `LF` (`\n`) before hashing and persisting content.

### Template Registry

`template_registries` stores all historical template versions per template name and their status.

> **Note:** The GORM model `schema.TemplateRegistry` maps to the default pluralized table name `template_registries`.

Data consistency invariant:
- For every row, `version` MUST equal `sha256(normalized(Content))`.
- `normalized(Content)` uses the same normalization rule (`CRLF -> LF`) as bootstrap hashing.

### Default Version

For each template name, default version is:
- newest record with status `AVAILABLE` by order `created_at DESC, id DESC`.

### Expired Version

For each template name, versions are ordered by `created_at DESC, id DESC`.
If `lag >= template_version_max_lag`, status is system-marked as `EXPIRED`.

## Data Model

### TemplateRegistry (MySQL)

Unique key: `(template_name, version)`

```go
type TemplateRegistry struct {
    ID           uint64         `gorm:"primaryKey;autoIncrement"`
    TemplateName string         `gorm:"type:varchar(128);not null;uniqueIndex:uk_template_version,priority:1;index:idx_template_status,priority:1;index:idx_template_created,priority:1"`
    Version      string         `gorm:"type:char(64);not null;uniqueIndex:uk_template_version,priority:2"`
    Content      string         `gorm:"type:longtext;not null"`
    Status       string         `gorm:"type:varchar(16);not null;index:idx_template_status,priority:2"` // AVAILABLE|DISABLED|EXPIRED
    CreatedAt    time.Time      `gorm:"index:idx_template_created,priority:2"`
    UpdatedAt    time.Time
    DeletedAt    gorm.DeletedAt `gorm:"index"`
}
```

Status values:
- `AVAILABLE`
- `DISABLED`
- `EXPIRED`

Status constraints:
- Manual set is allowed only for `AVAILABLE` and `DISABLED`.
- `EXPIRED` is a pure system-derived status (computed from lag), not user-overridable.
- `EXPIRED -> AVAILABLE` is forbidden (`FailedPrecondition`).
- `EXPIRED -> DISABLED` is forbidden (`FailedPrecondition`).
- `AVAILABLE -> DISABLED` is forbidden when the target version is the last `AVAILABLE` version for that template (`FailedPrecondition`).

### Status Transition Validation Matrix

`SetComponentTemplateVersionStatus` validates both the requested status and the current→target transition:

| Requested Status | Rule | Error Code |
|------------------|------|------------|
| `UNSPECIFIED` | Rejected (invalid input) | `InvalidArgument` |
| `EXPIRED` | Rejected (system-managed only) | `InvalidArgument` |
| `AVAILABLE` with current = `EXPIRED` | Rejected (irreversible) | `FailedPrecondition` |
| `AVAILABLE` with current = `DISABLED` | Allowed | — |
| `AVAILABLE` with current = `AVAILABLE` | No-op, success | — |
| `DISABLED` with current = `AVAILABLE` and other `AVAILABLE` versions exist | Allowed | — |
| `DISABLED` with current = `AVAILABLE` and this is last `AVAILABLE` version | Rejected | `FailedPrecondition` |
| `DISABLED` with current = `DISABLED` | No-op, success | — |
| `DISABLED` with current = `EXPIRED` | Rejected (expired is irreversible) | `FailedPrecondition` |
| Version not found | Rejected | `NotFound` |

Concurrency rule for `AVAILABLE -> DISABLED`:
- To enforce "cannot disable the last `AVAILABLE` version", validation and update MUST run in one transaction.
- The implementation MUST lock all rows for the same `template_name` (for example: `SELECT ... FOR UPDATE`) before counting `AVAILABLE` rows and writing status.
- If the locked snapshot shows only one `AVAILABLE` row and it is the target version, return `FailedPrecondition`.

## Annotation

Annotation keys:

- `tidbcloud.pingcap.com/template-name: <template_name>`
- `tidbcloud.pingcap.com/template-version: <sha256>`

Usage:
- `executeTemplate` injects both annotations into rendered CR before apply.
- tioperator read path parses this annotation and fills `Component.template_version`.
- Missing annotation is treated as legacy component and marked as `UNKNOWN` in version-dependent checks.

## Component-Template Mapping

- `TIDB -> tidb_group`
- `TIKV -> tikv_group`
- `PD -> pd_group`
- `TIPROXY -> tiproxy_group`
- `TIFLASH_WRITE/TIFLASH_COMPUTE -> tiflash_group`
- `TIKV_WORKER/COPROCESSOR_WORKER -> tikv_worker_group`
- `SCHEMA_MANAGER -> tikv_worker_group`
- `TIDB_WORKER -> tidb_worker_group`
- `PD_TSO -> tso_group`
- `PD_SCHEDULING -> scheduling_group`
- `RESOURCE_MANAGER -> resource_manager_group`

Current constraint:
- `TIFLASH_WRITE` and `TIFLASH_COMPUTE` intentionally share one template name (`tiflash_group`).
- If these two component types are split into different templates in future, the change MUST include:
  1. mapping update in component-template resolver;
  2. migration/backfill plan for historical template metadata (`template_registries` usage and CR annotations) to keep version history and observability consistent.

## API Interface List

### ApplySharedPool (updated)

- gRPC: `SharedPoolService.ApplySharedPool`
- HTTP: `POST /sharedPools/{shared_pool_id}:apply`

`ApplySharedPoolRequest` is single-component apply and does not reuse `TargetComponent`:

```protobuf
message ApplySharedPoolRequest {
  string shared_pool_id = 1 [(google.api.field_behavior) = REQUIRED];
  string id = 2 [(google.api.field_behavior) = OPTIONAL];
  ComponentType component_type = 3 [(google.api.field_behavior) = REQUIRED];
  string template_version = 4 [(google.api.field_behavior) = OPTIONAL];
  bool force = 5 [(google.api.field_behavior) = OPTIONAL];
}
```

Constraints:
- Exactly one target component per request.
- If `id` is empty, server resolves default component id by existing shared-pool logic.

### Component (updated)

```protobuf
message Component {
  // existing fields ...
  string template_version = <new_no> [(google.api.field_behavior) = OUTPUT_ONLY];
}
```

API visibility rule:
- `Component.template_version` is output-only (read-only for clients).
- `template_name` MUST NOT appear on `Component` response fields; cluster-facing APIs expose only `template_version`. However, `template_name` IS intentionally exposed only in TemplateService admin APIs that operate by `template_name` (e.g. `GetTemplate`, `ListTemplateVersions`, `SetTemplateVersionStatus`, `ListTemplates`) where it is required for template management operations.

### TemplateService
TemplateService provides two parallel access patterns:
1. **By ComponentType** (`/componentTemplates/...`): for templates mapped from `ComponentType` enum.
2. **By template_name** (`/templates/...`): for all templates, including those without a `ComponentType` mapping (e.g. `tidb_cluster`, `cluster_agent`, `dm_*`, `tibr_*`).

Each pattern has Get / ListVersions / SetStatus RPCs, plus a discovery List endpoint.

```protobuf
service TemplateService {
  // --- By ComponentType ---
  rpc GetComponentTemplate(GetComponentTemplateRequest) returns (GetComponentTemplateResponse) {
    option (google.api.http) = {get: "/componentTemplates/{component_type}/versions/{version}"};
  }
  rpc ListComponentTemplateVersions(ListComponentTemplateVersionsRequest) returns (ListComponentTemplateVersionsResponse) {
    option (google.api.http) = {get: "/componentTemplates/{component_type}/versions"};
  }
  rpc SetComponentTemplateVersionStatus(SetComponentTemplateVersionStatusRequest) returns (TemplateVersionInfo) {
    option (google.api.http) = {post: "/componentTemplates/{component_type}/versions/{version}:setStatus" body: "*"};
  }
  rpc ListComponentTemplates(ListComponentTemplatesRequest) returns (ListComponentTemplatesResponse) {
    option (google.api.http) = {get: "/componentTemplates"};
  }

  // --- By template_name ---
  rpc GetTemplate(GetTemplateRequest) returns (GetTemplateResponse) {
    option (google.api.http) = {get: "/templates/{template_name}/versions/{version}"};
  }
  rpc ListTemplateVersions(ListTemplateVersionsRequest) returns (ListTemplateVersionsResponse) {
    option (google.api.http) = {get: "/templates/{template_name}/versions"};
  }
  rpc SetTemplateVersionStatus(SetTemplateVersionStatusRequest) returns (TemplateVersionInfo) {
    option (google.api.http) = {post: "/templates/{template_name}/versions/{version}:setStatus" body: "*"};
  }
  rpc ListTemplates(ListTemplatesRequest) returns (ListTemplatesResponse) {
    option (google.api.http) = {get: "/templates"};
  }
}
```

#### By-ComponentType Messages

```protobuf
message GetComponentTemplateRequest {
  ComponentType component_type = 1 [(google.api.field_behavior) = REQUIRED];
  string version = 2 [(google.api.field_behavior) = REQUIRED];
}

message GetComponentTemplateResponse {
  string template_name = 1;
  string template_version = 2;
  string content = 3;
}

message ListComponentTemplateVersionsRequest {
  ComponentType component_type = 1 [(google.api.field_behavior) = REQUIRED];
}

message ListComponentTemplateVersionsResponse {
  string default_version = 1;
  repeated TemplateVersionInfo versions = 2;
}

message SetComponentTemplateVersionStatusRequest {
  ComponentType component_type = 1 [(google.api.field_behavior) = REQUIRED];
  string version = 2 [(google.api.field_behavior) = REQUIRED];
  TemplateVersionStatus status = 3 [(google.api.field_behavior) = REQUIRED];
}

message ListComponentTemplatesRequest {}

message ListComponentTemplatesResponse {
  repeated ComponentTemplateSummary component_templates = 1;
}

message ComponentTemplateSummary {
  ComponentType component_type = 1;
  string template_name = 2;
  string default_version = 3;
}
```

#### By-template_name Messages

```protobuf
message GetTemplateRequest {
  string template_name = 1 [(google.api.field_behavior) = REQUIRED];
  string version = 2 [(google.api.field_behavior) = REQUIRED];
}

message GetTemplateResponse {
  string template_name = 1;
  string version = 2;
  string content = 3;
}

message ListTemplateVersionsRequest {
  string template_name = 1 [(google.api.field_behavior) = REQUIRED];
}

message ListTemplateVersionsResponse {
  string default_version = 1;
  repeated TemplateVersionInfo versions = 2;
}

message SetTemplateVersionStatusRequest {
  string template_name = 1 [(google.api.field_behavior) = REQUIRED];
  string version = 2 [(google.api.field_behavior) = REQUIRED];
  TemplateVersionStatus status = 3 [(google.api.field_behavior) = REQUIRED];
}

message ListTemplatesRequest {}

message ListTemplatesResponse {
  repeated TemplateSummary templates = 1;
}

message TemplateSummary {
  string template_name = 1;
  string default_version = 2;
}
```

#### Shared Types

```protobuf
message TemplateVersionInfo {
  string template_name = 1;
  string version = 2;
  string created_at = 3;
  TemplateVersionStatus status = 4;
}

enum TemplateVersionStatus {
  TEMPLATE_VERSION_STATUS_UNSPECIFIED = 0;
  TEMPLATE_VERSION_STATUS_AVAILABLE = 1;
  TEMPLATE_VERSION_STATUS_DISABLED = 2;
  TEMPLATE_VERSION_STATUS_EXPIRED = 3;
}
```

#### API Semantics

| RPC | Semantics |
|-----|-----------|
| `GetComponentTemplate` | Resolve `component_type` to `template_name` internally; return rendered content for specified version. |
| `ListComponentTemplateVersions` | Query by `component_type` (mapped to template name). Return all versions. `default_version` = newest `AVAILABLE`. |
| `SetComponentTemplateVersionStatus` | Change version status by `component_type`. Only `AVAILABLE` / `DISABLED` accepted. |
| `ListComponentTemplates` | List all `ComponentType` values that have a template mapping, with each type's `template_name` and `default_version`. Note: multiple component types may share one template (e.g. TIKV_WORKER/COPROCESSOR_WORKER → tikv_worker_group). |
| `GetTemplate` | Get template content by `template_name` + `version` directly. For templates without `ComponentType` mapping. |
| `ListTemplateVersions` | List all versions of a template by `template_name`. Same semantics as `ListComponentTemplateVersions`. |
| `SetTemplateVersionStatus` | Change version status by `template_name`. Same constraints as `SetComponentTemplateVersionStatus`. |
| `ListTemplates` | List all distinct `template_name` values in `template_registries` with their default version. |

## Startup Bootstrap Flow

Startup flow runs before serving traffic:

Startup dependency order (must):
- `RunDBMigrate` -> `Template Bootstrap` -> `RegisterServer`
- `Template Bootstrap` MUST run only after DB migrations are successfully applied.
- `RegisterServer` (start serving gRPC/HTTP) MUST run only after template bootstrap succeeds.

```text
acquire global bootstrap lock (MySQL advisory lock)
  -> begin transaction
  -> scan deployments/templates/*.templ
  -> normalize content (CRLF->LF)
  -> compute sha256
  -> upsert template_registries(template_name, version, content)
  -> verify persisted row hash consistency (`sha256(normalized(content)) == version`)
  -> recompute EXPIRED by lag per template_name
  -> validate each scanned template_name has at least one AVAILABLE version
  -> commit transaction
  -> release global bootstrap lock
```

If validation fails (no available default for any template), startup fails.

Recompute trigger rule:
- `EXPIRED` recomputation runs only during startup bootstrap.
- New template versions are introduced only by service restart (loading latest files from `deployments/templates/`), so no runtime recomputation is required.

Bootstrap idempotency requirement:
- Every bootstrap step MUST be idempotent.
- `upsert template_registries(template_name, version, content)` is naturally idempotent by unique key `(template_name, version)`.
- `recompute EXPIRED by lag` MUST be implemented as deterministic idempotent recalculation from current registry state.
- If lock ownership changes mid-process (for example lock session interruption), the new lock-holder MUST be able to rerun the whole bootstrap flow safely without semantic drift.

Hash consistency enforcement:
- Implementation MUST enforce `sha256(normalized(content)) == version` at least in one layer:
  1. repository write path (strict validation before/at upsert), or
  2. bootstrap upsert + immediate read-back verification.
- If both are implemented, repository validation is the primary guard and bootstrap read-back is a defense-in-depth check.

Validation scope rule:
- Startup availability validation MUST only cover template names scanned from current deployment (`deployments/templates/*.templ`) in this bootstrap run.
- Historical template names not present in current deployment are out of this startup validation scope.

Concurrency guarantees:
- `(template_name, version)` unique key makes registration idempotent under concurrent startup.
- only lock holder performs expiration/default recomputation.
- recomputation + validation are atomic in one transaction.

Multi-service restart compatibility (`cluster_ng_server` + `executor`):
- Both binaries may execute template bootstrap during startup and may restart at the same time.
- A single global MySQL advisory lock serializes bootstrap across all service processes.
- `cluster_ng_server` MUST wait for lock release and complete bootstrap before serving traffic.
- `executor` MUST wait for lock release and complete/bootstrap-confirm before starting cron or workflow execution.
- Non-lock-holder instance MUST NOT perform recomputation/write path concurrently.
- Lock contention itself MUST NOT be treated as bootstrap failure; only lock-holder execution errors (for example: no `AVAILABLE` default after recomputation) fail startup.

## Apply Flow

```text
resolve template_name by component_type
  -> call resolveTemplate(template_name, request.template_version)
  -> call executeTemplate(templateInfo, templArgs, remoteCli)
```

Template apply is split into two layers to keep responsibilities clear:
- `resolveTemplate(template_name, templateVersion) -> templateInfo`
  - version resolution (`templateVersion` or default `AVAILABLE`)
  - DB lookup/status validation
  - template content loading
- `executeTemplate(templateInfo, templArgs, remoteCli)`
  - annotation injection
  - render/unstructured conversion
  - Kubernetes `CreateOrUpdate`

```text
resolveTemplate(template_name, templateVersion):
  -> resolve selected_version:
       - templateVersion if non-empty
       - otherwise template default AVAILABLE version
  -> validate selected_version exists in template_registries
  -> validate status == AVAILABLE
  -> load template content from template_registries(selected_version)
  -> return templateInfo{template_name, selected_version, template_content}

executeTemplate(templateInfo, templArgs, remoteCli):
  -> render yaml with templArgs
  -> GenerateUnstructuredFromYaml(yaml) to resourceCr
  -> merge annotations into resourceCr.metadata.annotations:
       tidbcloud.pingcap.com/template-name=templateInfo.template_name
       tidbcloud.pingcap.com/template-version=templateInfo.version
     (preserve existing annotations; only overwrite these two keys)
  -> apply CR
```

Callers MUST NOT perform version resolution or template loading themselves; they only pass the raw `template_version` from the request (or empty string for default) to `resolveTemplate`.

Annotation injection timing rule:
- Annotation merge MUST happen after `GenerateUnstructuredFromYaml` and before entering `CreateOrUpdate`.
- `CreateOrUpdate` should then use the merged annotations from `resourceCr` when syncing to `toCr`.

Error mapping:
- from `resolveTemplate`:
- version not found: `NotFound`
- status is `DISABLED`/`EXPIRED`: `FailedPrecondition`
- from `executeTemplate`:
- render/unstructured/apply errors: `Internal` with wrapped context

Rollback rule:
- No one-step restriction. Any existing `AVAILABLE` version is allowed.

## Read Component Flow (tioperator)

For each `cvt*ComponentToEn` conversion path:

```text
read CR metadata.annotations[tidbcloud.pingcap.com/template-name]
read CR metadata.annotations[tidbcloud.pingcap.com/template-version]
  -> fill Component.template_version (template_name used for audit/consistency checks)
```

If annotation missing:
- leave `Component.template_version` empty.
- version-dependent services (TemplateService/cron) classify this component as `UNKNOWN` instead of mapping to default version.

## Cron: Unavailable/Unknown Version Usage Detector

New cron job checks components using unavailable or unknown versions:

```text
list shared pools
  -> discover components via tioperator
  -> for each component:
       current_version = component.template_version
       if empty -> emit unknown metric and continue
       load status(current_version)
       if status in {DISABLED, EXPIRED} -> emit metric
```

Metrics:
- `cluster_ng_template_unavailable_usage{template_name,status,component_type}` (Gauge)
- `cluster_ng_template_unknown_usage{template_name,component_type}` (Gauge)

Observability constraints:
- Both metrics are snapshot Gauges: each cron run recomputes and overwrites the current value.
- Do not expose per-component metric labels (`component_id`, etc.) to avoid high-cardinality explosion.
- Component-level details are emitted as structured logs / alert events, not metrics.

## Configuration

Add configuration:

- `server.biz.templateVersionMaxLag` (default `3`)
- `server.cronJob.enableTemplateVersionUsageDetector` (default `false`)
- `server.cronJob.templateVersionUsageDetectorCron` (default `0 */5 * * * *`)

Used by:
- startup expiration recomputation
- apply availability validation
- TemplateService status response
- cron detector (`unavailable` + `unknown`)

## Migration And Compatibility

- `ApplySharedPool` is a new internal API with no existing callers in production; **no backwards compatibility is required**. The old `apply_components` repeated field has been removed without deprecation.
- Other API changes are additive for existing callers.
- Legacy components without annotation remain supported.
- Legacy DB `TemplateVersion(int64)` fields remain unchanged but are no longer semantic source.

## Failure Handling

- Bootstrap failure -> service startup failure.
- Missing default AVAILABLE for template -> startup failure.
- Invalid manual status transition (including `EXPIRED -> DISABLED` and disabling last `AVAILABLE`) -> `FailedPrecondition`.
- `EXPIRED` status is always system-derived and cannot be changed through manual status APIs.
- Any detected hash inconsistency (`sha256(normalized(content)) != version`) -> bootstrap failure.
- `resolveTemplate` errors:
- version not found -> `NotFound`
- status is `DISABLED`/`EXPIRED` -> `FailedPrecondition`
- `executeTemplate` render/unstructured/apply errors -> `Internal` with wrapped context.

## Testing

### Unit Tests

- content normalization and hash stability
- template row hash invariant validation (`sha256(normalized(content)) == version`)
- default version selection
- lag expiration computation
- status transition constraints
- apply version selection and error mapping
- `resolveTemplate` table tests: default resolve / explicit version / status reject / not found
- `executeTemplate` tests: annotation merge timing and key overwrite-only behavior

### Integration Tests

- startup bootstrap idempotency
- concurrent startup of `cluster_ng_server` and `executor` only allows one bootstrap writer (global lock)
- `cluster_ng_server` lock contention path waits and eventually completes bootstrap
- `executor` lock contention path waits and only starts jobs after bootstrap is complete
- bootstrap detects hash/content mismatch after upsert (or repo rejects it) and fails fast
- apply with explicit AVAILABLE historical version
- apply rejection for DISABLED/EXPIRED
- apply path uses `resolveTemplate` + `executeTemplate` split and keeps behavior compatible
- annotation write (`template-name` + `template-version`) + tioperator read-back
- missing annotation path is classified as `UNKNOWN` and does not fallback to default
- reject disabling last `AVAILABLE` version
- concurrent disable requests cannot bypass "last `AVAILABLE`" protection (transaction + lock)
- reject `EXPIRED -> DISABLED`

### Regression

- apply without `template_version` uses default
- non-component templates are version-registered but not targeted by component apply/version-usage detection flows

## Code References

| Concern | File |
|---------|------|
| API proto | `api/cluster-service-ng/v1beta/services.proto`, `resources.proto` |
| Startup bootstrap hook | `cmd/cluster_ng_server/main.go` |
| Apply handlers | `domain/server/apply_sharedpool.go` |
| Apply action/render | `domain/workflow/action/apply_component_group.go` |
| Template loading/render | `pkg/templates/*` |
| Component conversion | `pkg/tioperator/*.go` |
| MySQL schema | `pkg/storage/mysql/schema/*` |
| DB migration | `pkg/storage/mysql/migrate/db/*` |
| Repo interfaces | `pkg/storage/repo/*` |
| Cron job | `domain/app/cron/*` |