# Design Documents ## Table of Contents - [Design Documents](#design-documents) - [Table of Contents](#table-of-contents) - [Introduction](#introduction) - [Motivation or Background](#motivation-or-background) - [Detailed Design](#detailed-design) - [Test Design](#test-design) - [Functional Tests](#functional-tests) - [Scenario Tests](#scenario-tests) - [Compatibility Tests](#compatibility-tests) - [Impacts \& Risks](#impacts--risks) - [Unresolved Questions](#unresolved-questions) ## Introduction This design adds a DevOps API for searching cluster component versions through config center's new `SearchClusterComponents` RPC. DevOps will wrap the raw config center API with tenant, cluster, and all-region search planning so the UI can query component versions without knowing how to discover cluster IDs or regional config center routing details. ## Motivation or Background Config center now exposes `SearchClusterComponents(cluster_type, component_name, cluster_ids, component_version, image_tag, page_size, page, logical_region)`. For non-regional cluster types, DevOps can call config center directly. For regional cluster types such as `SHARED_POOL` and `LOGICAL_CLUSTER`, config center global mode requires `logical_region`; omitting it returns `InvalidArgument`. The DevOps UI needs three user-facing workflows: - Tenant page: show all component versions for all `LOGICAL_CLUSTER` and `SHARED_POOL` resources under one tenant. - Cluster page: show all component versions for a single cluster or resource pool. - Global component page: search by `cluster_type` and `component_name`. For `SHARED_POOL` and `LOGICAL_CLUSTER`, users want one operation that covers every region instead of manually repeating the same search region by region. The key design problem is the global regional search case. A single flat table across all regions sounds simple, but exact global pagination is unreliable when each regional config center only supports page-based local pagination. The proposed design keeps the backend responsible for regional fan-out, while presenting all-region results grouped by region with per-region totals and partial failure information. ## Detailed Design ### External API Add one authenticated HTTP endpoint under the existing cloudconfig controller: ```text GET /api/v1/devops/cloudconfig/cluster-components/search ``` Request query fields: | Field | Required | Notes | | --- | --- | --- | | `tenant_id` | Scope dependent | Tenant page scope. Mutually exclusive with `cluster_id`. | | `cluster_id` | Scope dependent | Cluster page scope. Mutually exclusive with `tenant_id`. | | `cluster_type` | Yes | Required for tenant, cluster, and global scope. Tenant scope searches exactly this cluster type; no implicit `LOGICAL_CLUSTER` plus `SHARED_POOL` wrapper. | | `component_name` | Scope dependent | Required for global scope. Optional for tenant and cluster scope. | | `component_version` | No | Passed through to config center. | | `image_tag` | No | Passed through to config center. | | `logical_region` | No | Narrows a regional search to one region. | | `region_mode` | No | `auto` by default. `all` forces all-region fan-out for regional global scope. `single` requires `logical_region`. | | `page` / `per_page` | No | DevOps response pagination. Single-region and tenant searches use normal pagination. All-region grouped searches use `per_page` as the first-page size per region; follow-up paging is per region. | Response shape: ```go type SearchClusterComponentsResp struct { Total int64 `json:"total"` Items []*ClusterComponentVersion `json:"items"` RegionResults []*ClusterComponentRegionResult `json:"region_results,omitempty"` PartialFailure bool `json:"partial_failure"` } type ClusterComponentRegionResult struct { LogicalRegion string `json:"logical_region"` ClusterType ccent.ClusterType `json:"cluster_type"` Total int64 `json:"total"` Page int32 `json:"page"` PerPage int32 `json:"per_page"` HasMore bool `json:"has_more"` NextPage int32 `json:"next_page,omitempty"` Items []*ClusterComponentVersion `json:"items"` Error string `json:"error,omitempty"` } type ClusterComponentVersion struct { ClusterType ccent.ClusterType `json:"cluster_type"` ClusterID string `json:"cluster_id"` ComponentName string `json:"component_name"` Version string `json:"version"` CloudProvider string `json:"cloud_provider"` NodeInstance string `json:"node_instance"` OverwriteImageTag string `json:"overwrite_image_tag"` BaseImage string `json:"base_image"` EffectiveImageTag string `json:"effective_image_tag"` UpdateStrategy string `json:"update_strategy"` LogicalRegion string `json:"logical_region,omitempty"` } ``` For tenant and cluster views, `Items` is the primary list. For global all-region views, `RegionResults` is the primary list and `Items` may be empty or contain the flattened first page only if the frontend explicitly asks for flat output later. ### Search Planning Implement the wrapping logic in `pkg/service/cloudconfig`, not directly in the controller. The controller should only bind/validate HTTP parameters and call the service. Internal plan model: ```go type clusterComponentSearchPlan struct { ClusterType ccent.ClusterType LogicalRegion string ClusterIDs []string } ``` DevOps builds one or more plans, then executes each plan by calling config center: ```go ccReq := &ccproto.SearchClusterComponentsRequest{ ClusterType: cvt.To.ConfigCenter.ClusterType(plan.ClusterType), ComponentName: req.ComponentName, ClusterIds: plan.ClusterIDs, ComponentVersion: pointer if set, ImageTag: pointer if set, PageSize: int32(req.PageSize), Page: int32(req.Page), } if plan.LogicalRegion != "" { ccReq.LogicalRegion = &plan.LogicalRegion } ``` For non-regional cluster types, there is exactly one plan without `LogicalRegion`. For regional cluster types, every plan must have `LogicalRegion`. If DevOps cannot resolve it, return `InvalidArgument` with the cluster ID or tenant ID that caused the unresolved route. ### Pagination Strategy Single-plan searches should pass `page` and `per_page` directly to config center and return the config center `total`. Tenant scope is finite because DevOps first resolves a bounded set of cluster IDs. For tenant scope with multiple plans, DevOps should fetch each plan in config center pages of 500, merge the results, sort by `(cluster_type, logical_region, cluster_id, component_name)`, then apply the requested DevOps `page/per_page` to the merged list. Add a conservative internal cap, for example 10,000 component rows, and return an explicit truncation error if the cap is exceeded rather than silently returning partial tenant data. Global all-region scope should not fake global offset pagination. It should return grouped region results: - The first all-region request fetches page 1 from every region with `per_page` as the per-region sample size. - A region with 3 total rows returns all 3 rows with `has_more=false`. - A region with 300 total rows returns the first `per_page` rows with `has_more=true`. - To continue paging one large region, the frontend calls the same endpoint with `logical_region=`, `region_mode=single`, and `page=2`. This keeps the all-region call useful as an overview while preserving exact pagination where it is actually meaningful: inside one region. ### Tenant Page Scope When `tenant_id` is present: 1. Check tenant scope with the existing controller authorization helpers. 2. Require `cluster_type`. Tenant scope does not query both `LOGICAL_CLUSTER` and `SHARED_POOL` in one request. 3. For `LOGICAL_CLUSTER`, discover logical clusters: - Call `Mgmt.ListClusters` with `TenantIDs=[tenant_id]` and `DeployType=DEPLOY_TYPE_NEXTGEN`. - Collect logical cluster IDs from `cluster.GetClusterId()`. - Resolve each logical cluster's logical region. Prefer a direct field from mgmt if available; otherwise use the cluster's `NextGenCluster.SharedPoolId` and `Mgmt.GetResourcePool` to obtain `rp.GetRegion().GetLogicalName()`. 4. For `SHARED_POOL`, discover shared pools: - Call `ResourcePool.ListOrgResourcePools(OrganizationId=tenant_id)` to collect tenant-owned resource pool IDs and logical regions. - De-duplicate by resource pool ID. 5. Build plans grouped by `(cluster_type, logical_region)`. - `LOGICAL_CLUSTER`: cluster IDs are logical cluster IDs. - `SHARED_POOL`: cluster IDs are resource pool/shared pool IDs. 6. Execute plans concurrently with a small fixed limit, for example 4, using `pkg/infra/utils/workqueue`. The tenant page still gets one DevOps request per selected cluster type and does not need to know region routing rules. ### Cluster Page Scope When `cluster_id` is present: 1. Require `cluster_type`. 2. For non-regional cluster types, call config center directly with `cluster_ids=[cluster_id]`. 3. For `LOGICAL_CLUSTER`: - Load the mgmt cluster by ID. - Verify it is `DEPLOY_TYPE_NEXTGEN`. - Resolve logical region from the cluster itself or its shared pool. - Call config center with `logical_region` and `cluster_ids=[cluster_id]`. 4. For `SHARED_POOL`: - Treat `cluster_id` as resource pool ID. - Load the resource pool with `Mgmt.GetResourcePool`. - Resolve `rp.GetRegion().GetLogicalName()`. - Call config center with `logical_region` and `cluster_ids=[cluster_id]`. This gives the cluster page the simple contract requested by the UI: it supplies `(cluster_type, cluster_id)`, and DevOps handles the regional details. ### Global Component Scope When neither `tenant_id` nor `cluster_id` is present: 1. Require both `cluster_type` and `component_name`. 2. For non-regional cluster types, call config center once. 3. For regional cluster types: - If `logical_region` is provided, call config center once. - If `region_mode=single`, require `logical_region`. - Otherwise, fan out across all active NextGen regions from `Mgmt.Region.ListRegions(DeployType=DEPLOY_TYPE_NEXTGEN)`. Recommended UI behavior: - Default region selector value: `All regions`. - Render all-region results grouped by logical region, with each region showing `total`, first-page items, `has_more`, and a retry action when that region fails. - For a large region, use a "Load more" action that switches to `region_mode=single&logical_region=&page=2` for exact region-local pagination from the all-region overview. - Keep a region dropdown or tabs so users can narrow to one region when they need exact paging and faster response. - Show a global summary count as the sum of successful region totals, plus a partial failure badge when any region fails. This is more reliable than a single global flat table because each regional config center owns its own pagination and availability. A flat global table can be added later only if config center supports a true global cursor or DevOps persists an indexed aggregate. ### Error Handling For tenant and cluster scopes, unresolved cluster metadata or a failed config center call should fail the whole request because the requested target is specific. For global all-region scope, use partial failure semantics: - A failed region produces one `RegionResult` with `error` set. - Successful regions still return their data. - `PartialFailure=true` when at least one region failed. - Return HTTP 200 if at least one region succeeded; return an error only when all regions fail or region discovery fails. ### Compatibility The config center dependency is updated to commit `749fa8edf829`, which resolves to pseudo-version `v0.0.0-20260527080838-749fa8edf829` for the direct `gitopsfile` and `proto/config_center` modules. The existing generated mock for `ConfigCenterServiceClient` must be regenerated because the upstream interface gained new methods, including `SearchClusterComponents`. This is mechanical and does not change runtime behavior. No database migration is required. The feature reads cluster metadata from mgmt/core services and component data from config center. ## Test Design ### Functional Tests - Controller binding and validation: - Reject requests with both `tenant_id` and `cluster_id`. - Reject tenant search without `cluster_type`. - Reject global search without `cluster_type` or `component_name`. - Reject cluster search without `cluster_type`. - Reject `region_mode=single` without `logical_region` for regional cluster types. - Service plan building: - Tenant `LOGICAL_CLUSTER` scope groups logical clusters by region. - Tenant `SHARED_POOL` scope groups resource pools by region. - Cluster scope resolves logical region for `LOGICAL_CLUSTER` and `SHARED_POOL`. - Non-regional cluster types produce a single non-regional plan. - Config center request conversion: - Optional `component_version`, `image_tag`, and `logical_region` are set only when provided. - `page/per_page` are passed through for single-region calls. ### Scenario Tests - Tenant `LOGICAL_CLUSTER` search with two logical clusters in different regions returns results from both regions. - Tenant `SHARED_POOL` search with multiple resource pools returns only shared pool component results. - Cluster page for a logical cluster only requires `(cluster_type, cluster_id)` from the frontend and still calls config center with `logical_region`. - Global `LOGICAL_CLUSTER + tidb` search with `All regions` returns one first-page result group per active region. - All-region search where one region has 3 rows and another has 300 rows returns `has_more=false` for the 3-row region and `has_more=true,next_page=2` for the 300-row region. - One region failing during all-region search returns successful regions plus `PartialFailure=true`. - All regions failing returns an API error. ### Compatibility Tests - Existing cloudconfig tests compile and pass after regenerating `pkg/infra/clients/configcentercli/client_mock.go`. - Existing config and tenant-init APIs continue to use current config center calls unchanged. - Dedicated TiDB and TiCDC component searches do not require `logical_region` and continue to route to the central config center client. ## Impacts & Risks Impacts: - Tenant and cluster pages get a simpler frontend contract; DevOps owns cluster discovery and regional routing. - Global component search is operationally safer because each region is isolated in the response. - All-region searches add extra calls to mgmt region discovery and regional config center clients. Risks: - Region resolution for logical clusters depends on mgmt/resource-pool metadata being complete and current. - Large tenants may produce many cluster IDs. DevOps should cap internal page fetches and log when truncation is necessary. - A grouped all-region UI is slightly more complex than a flat table, but it avoids misleading global pagination. - If config center later supports true cross-region aggregation, this DevOps fan-out layer may be simplified. ## Unresolved Questions - Does mgmt cluster data expose a reliable logical region directly for NextGen clusters? If not, the implementation should resolve it via shared pool/resource pool lookup and cache by shared pool ID per request. - Should DevOps offer a later export API for full all-region data? The interactive API should remain grouped and region-paged.