Simplified backend design

Multi-cluster, multi-component Premium upgrade

One immutable upgrade plan, one approval request, and ordered Apply Groups across one or many clusters. TiFlash, PD and BR use one managed multi-component ConfigurationChange per group. TiCDC is out of scope for the first version.

Draft One plan approval Fixed component order Three new tables

Key decisions

The first version deliberately favors predictable orchestration over a generic workflow engine.

One plan, one approval

All selected clusters and components are frozen into one manifest and authorized together.

Hard-coded order

The server owns one fixed component order. A partial selection keeps only the relative order.

Managed child changes

Each step uses one ConfigurationChange with a null approval_id. A step may contain one component or an SOP-defined atomic component group.

Reuse conflict checks

No Premium-specific resource lock table. Existing unfinished-change validation remains authoritative.

ⓘ

Out of scope for v1No dependency graph, compatibility matrix, cross-stage dependency inclusion, unselected-component version checks, one-day idle gate, user-defined Dependency Policy, Failure Policy, or custom order.

End-to-end execution

Child ConfigurationChanges are created just in time after approval, so a pending plan does not occupy every target locator.

1. PreviewResolve cluster IDs, components, actual locators and versions.

→

2. Create planPersist immutable Plan, Target and Step records.

→

3. Approve onceOne approval contains the complete upgrade manifest.

→

4. Claim stepCAS on status + version prevents duplicate workers.

→

5. Check conflictRead unfinished Changes; BR also checks every shared-pool cluster for Restore/PITR.

→

6. Create & applyManaged Change starts at wait_for_apply; no child approval.

→

7. ReconcileCheck result, finish the stage, then advance sequence.

Fixed component order

The order and atomic groups follow the first table in the 2510 → 2603 Premium upgrade SOP.

TiFlash ×2→ PD ×3→ TiKV workers→ TiKV→ TiDB→ BR ×5→ Compatibility

Selecting: PD + PD TSO → One Apply: PD + PD TSO; PD Scheduling remains unselected

↳

Sequence means execution stageA stage may contain several independent Apply Groups. Every Apply Group creates its own ConfigurationChange on each actual locator; the next stage begins only after all Changes in the current sequence are terminal.

Approval boundary

The plan approval is the only authorization boundary. Child changes are implementation records, not new requests.

Premium upgrade plan · PUP-20260812-0042

Frozen manifest presented to approvers

Pending approval

Cluster ID	Actual target	Components	Version
clu-001, clu-002	SHARED_POOL / pool-17	TiProxy · separate Change	v8.1.2
clu-001, clu-002	SHARED_POOL / pool-17	DM · separate Change	v8.1.2
clu-001, clu-002	SHARED_POOL / pool-17	Resource Manager · separate Change	v8.1.2
clu-001, clu-002	SHARED_POOL / pool-17	Schema Manager · separate Change	v8.1.2
clu-001	LOGICAL_CLUSTER / clu-001	TiDB · separate Change	v8.1.2
clu-001	LOGICAL_CLUSTER / clu-001	TiProxy · separate Change	v8.1.2
clu-001	LOGICAL_CLUSTER / clu-001	Dumpling · separate Change	v8.1.2

Parent approvalpremium_upgrade_plans.approval_id is the sole approval controlling execution.

Managed ConfigurationChangeCreated only when its Step is executable. It starts in wait_for_apply and skips SendApproval.

Approval routingThe desired-version Base Repo selects a named approver group. Component routes are exceptions; the default group is the fallback.

Audit linkagepremium_upgrade_step_id links the child Change through Step to Target and Plan. Its approval_id stays null.

API protectionPublic Apply/Cancel routes reject managed changes; orchestration uses explicit internal service methods.

Data model

Three orchestration tables plus additive linkage fields on the existing ConfigurationChange table.

premium_upgrade_plans

Approval, immutable request snapshot, plan state and aggregate progress.

name, reason, applicant
approval_id, approval_status
desired_version, cluster/component snapshots
status, idempotency_key, timestamps

premium_upgrade_targets

One normalized actual CloudConfig locator, with source cluster IDs for user-facing aggregation.

plan_id, cluster_type, cluster_id
source_cluster_ids + tenant_ids snapshots
status, last_error
unique(plan_id, cluster_type, cluster_id)

premium_upgrade_steps

One atomic Apply Group on one actual locator.

target_id, apply_group, sequence
components[] with from/to versions
status, version, attempt_count
configuration_change_id, last_error

Alter cloudconfig_configuration_changes

Add nullable unique premium_upgrade_step_id and a failed finished phase. Failure details remain only on the Upgrade Step, which already links to Target and Plan.

Forward migration for existing feature environments

Legacy single-component Steps are backfilled as one-member Apply Groups before the old component columns and index are removed. Existing plans keep their original sequence and are never merged during migration.

Not added: premium_upgrade_step_dependencies and premium_upgrade_resource_locks. Relative order is derived from server code, and conflicts reuse ConfigurationChange validation.

Normalize the actual target locator

A user selects logical cluster IDs, but some components physically belong to a shared resource pool.

User selection

clu-001 + clu-002Both clusters use shared pool pool-17.

→

Execution targets

SHARED_POOL / pool-17 once
LOGICAL_CLUSTER / clu-001
LOGICAL_CLUSTER / clu-002Shared-pool steps are deduplicated by actual locator.

Failure propagationIf an actual-locator Step fails, targets with overlapping source cluster IDs skip their later Steps. Unrelated clusters continue.

Step states

The orchestrator reconciles durable state; a process restart must not lose progress or create a second Change.

State	Meaning	Next action
pending	Waiting for plan approval or an earlier sequence.	Do nothing until eligible.
preparing	CAS claim acquired; state and conflicts are being checked.	Create or recover the managed Change.
waiting_conflict	An unfinished Change exists, or BR found an active Restore/PITR (including status lookup failure).	Retry with bounded backoff.
applying	The managed Change is running through its existing applier.	Poll existing CheckResult behavior.
succeeded	The component reached the requested version.	Contribute to sequence completion.
failed	The step hit a hard error or fixed apply deadline; its managed Change is closed as failed.	Release the locator, skip later affected steps, and continue unrelated targets.
skipped / cancelled	Execution stopped by upstream failure or user cancellation.	No automatic rollback.

Proposed API

These endpoints are design proposals, not claims about currently available backend APIs.

Method	Endpoint	Purpose
GET	`/api/v1/devops/clusters?tenant_id=...&deploy_type=DEPLOY_TYPE_NEXTGEN`	Reuse the existing paginated endpoint for exact tenant-ID discovery; exact cluster-ID lookup is also supported.
GET	`/api/v1/devops/premiumUpgradePlans/componentPolicies`	Return the backend-owned cluster type, component, priority, apply strategy and order revision used by the component picker.
POST	`/api/v1/devops/premiumUpgradePlans/preview`	Resolve tenant-scoped cluster IDs, actual locators, selected/all components, versions and fixed order. Selected components use cluster_type + component to disambiguate shared and logical targets.
POST	`/api/v1/devops/premiumUpgradePlans`	Revalidate preview, persist the immutable plan and asynchronously create one approval request.
GET	`/api/v1/devops/premiumUpgradePlans/{plan_id}`	Return approval, target, step and child Change progress.
POST	`/api/v1/devops/premiumUpgradePlans/{plan_id}/cancel`	Stop scheduling new Steps and request cancellation of safe in-flight work.
POST	`/api/v1/devops/premiumUpgradePlans/{plan_id}/steps/{step_id}/retry`	Retry only a failed Step that has not created a managed Change; it never creates another approval.

Primary risks and controls

Approval create uncertainty

The current Approval Service has no business idempotency key. A dispatcher CAS prevents concurrent sends; an uncertain response moves the Plan to reconcile-required instead of silently creating another approval.

Conflict check race

The existing “check unfinished Change, then insert” flow is not atomic. V1 reuses it as agreed; a future generic CloudConfig locator reservation should solve the race for every caller, rather than adding a Premium-only lock table.

Order becomes business logic

The fixed order must be versioned, covered by unit tests and confirmed by component owners. Plans persist the resolved sequence so a deployment cannot reorder an approved plan.

Shared-pool blast radius

Preview and approval must expose the actual shared-pool target and every affected source cluster before submission.

No multi-cluster atomicity

One approval does not imply one distributed transaction. The final plan state must distinguish completed, partially succeeded, failed and cancelled outcomes.

Companion visual for 2026-08-11-premium-multi-cluster-component-upgrade-orchestration.md · Simplified design · 2026-08-14