# Ops Server Design
## Background
### Status Quo
Currently, the deployment and fundamental lifecycle management of Observability (O11y) components, such as version upgrades and resource adjustments, are primarily handled through a GitOps workflow. This approach involves maintaining the Helm charts and Kubernetes manifests for various O11y components in a GitHub repository. GitHub Actions are then configured to interact with the respective Kubernetes cluster APIs, automating component deployment and basic operational tasks.
However, certain complex operation and maintenance (Ops) procedures lack API standardization and often require manual execution via a jump server with direct access to the Kubernetes cluster. Examples of such operations include:
* **Advanced Troubleshooting**: Executing commands within specific Pods for diagnosis, retrieving detailed status information for particular resources, etc.
* **Granular Scaling Control**: Implementing multi-step scaling operations (scale-down/scale-up) that necessitate execution in a precise sequence.
* **Data Management**: Performing tasks like data migration, backup, or recovery.
## Challenges
The introduction of the Bring Your Own Cloud (BYOC) scenarios presents new requirements for the deployment and operation of O11y components:
* **BYOC - PingCAP-managed**: In this model, where PingCAP manages the O11y components within the user's cloud environment, there is a clear need for standardized API interfaces to facilitate troubleshooting and complex operational tasks remotely and programmatically.
* **BYOC - Self-managed**: For users managing O11y components independently in their own cloud environment, O11y must provide capabilities for bootstrapping component deployment *without* relying on the GitOps workflow managed by PingCAP.
## Goals
Implement Ops server:
* Ops server provides O11y component Ops API.
* **Day-1 Ops**: Provides Bootstrap deployment capability for components
* **Day-2 Ops**: Provide standardized operation and maintenance operation API, including version upgrade, resource allocation, etc
* Ops Server implements and collaborates with K8s Controller (Operator) : In order to reliably and automatically perform complex operation and maintenance operations, it is necessary to implement or collaborate with necessary K8s Controller (Operator), Controller can monitor K8s resource status changes, and automatically adjust resources according to the solidified logic, so as to complete the complex DevOps flow.
## Design
> **Note**: This design is based on K8s native mode
### Overview
* **Ops Server**: Operations portal, deployed in K8s, handles API requests and executes some logic
* **Ops API Server**: Provide standard HTTP API, interact with K8s APIServer, complete `Operation` `HelmRelease` and other CR CRUD.
* **HelmRelease**: CR abstraction of Helm Release. Using Helm provides the ability of application packaging and unified management, more flexible template configuration capability (go template), and more concise configuration interface (values). In addition, many components of O11y are currently deployed using Helm.
* **Helm Controller**: Based on `HelmRelease` CR, call Helm SDK to manage the lifecycle of components.
* **Ops Controller**: located inside the Ops Server, based on specific CR, perform operation and maintenance tasks
* **Operators**: such as VM-Operator, based on CR, perform the actual deployment and operation of components
## Helm Controller
### Helm Chart & Release
Based on HelmRelease CR for deployment and operation, the basic CR definition is as follows:
```yaml
iam:
spec:
repoURL: https://helm.vector.dev # open-sourced repo url
chart: vector # chart name
version: 0.17.1 # chart version
values: # values provided to override the default values in chart.
replicaCount: 2
---
spec:
targetNamespace: observability # target namespace for installing the chart.
repoURL: oci://us-west1-docker.pkg.dev/pingcap-public/helm/vm-chart # oci-compatible repo url
chart: vm-distributed # chart name
version: 0.1.1 # chart version
values: # values provided to override the default values in chart.
image:
tag: v1.102.1
---
spec:
targetNamespace: observability # target namespace for installing the chart.
chart: /etc/charts/vm-distributed # chart path
version: 0.1.1 # chart version
```
* **Release management**:
* Controller is based on helm SDK and implements release lifecycle management.
* The Controller periodically performs reconcile to determine the diff of actual manifests and rendered manifests, ensuring that the actual resources in the cluster are consistent with the state defined by the `HelmRelease` CR.
* **Chart Repo Management**:
* Using the open source helm chart repository
* Based on gcp artifact repository, package the helm chart and manage it in the mirroring repository in a way similar to docker image
* Directly package the charts into the container mirroring where the controller is located
### Alternatives
The helm controller and source controller based on flux opencd -source are deployed in the ops server pod in a sidecar manner.
* **Source Controller**: Data source abstraction, support helm repository, github repository, s3 bucket, etc
* **Helm Controller**: Complete helm release lifecycle management, exception handling mechanism, Release dependency management
### Comparison
| | Pros | Cons |
| :--- | :--- | :--- |
| **Community** | ✓ complete helm release lifecycle management, including install, upgrade, rollback, diff detection, unisntall, deployment exception handling, etc
✓ Deployment Dependency Management of Helm Release
✓ apache 2.0 open source license
✓ only one deployment component, easy to deploy | ✗ introduce additional API and component dependencies |
| **self-develop**| ✓ code is autonomous and controllable | ✗ development and testing costs (about 5d to implement a basic deployment, upgrade, and deletion)
✗ Insufficient functional completeness |
| **conclusion**| **Tentatively based on community version** | |
## Ops Controller
Taking the cluster backup operation and maintenance scenario as an example, implement specific controller logic
### VM Cluster Backup
1. Call ops server API to create ClusterBackup CR, return 202 accepted
2. Ops Controller reconciles ClusterBackup CR
a. Call the instant backup API of different vmstorage
b. The Ops Controller records the backup metadata of each vmstorage to the ClusterBackup CR.
c. Ops Controller queries different vmstorage backup statuses until all replica backups are complete
d. The Ops Controller records the backup status to ClusterBackup CR.
3. Call ops server API to query ClusterBackup status to obtain real-time progress and final results of backups.
## Ops API Server
### Authn & Authz
To reuse regionalserver
### API Design
Ops API Server provides a standard RESTful API interface for receiving operational requests from users or systems. It converts these requests into management operations for CRs such as `HelmRelease` or `Operation`.
* **Domain**: `observability.tidbcloud.com`
* **Base Path**: `/api/v1/observability-ops`
* **Resources**: `vmclusters`
---
* `POST /vmclusters` **:deploy victoriametrics cluster**
* **Description**: Deploys a new VictoriaMetrics cluster by creating a new Helm release. The cluster configuration is provided via Helm values.
* **Request Body**:
```json
{
"name": "new-vm-cluster", // Resource name includes collection and ID
"values": {
"ha-mode": "true",
"vmselect": {
"service": {
"type": "LoadBalancer"
}
}
// ... desired initial Helm values
}
// Other fields like state, create_time, endpoints are server-managed
}
```
* **Response Body**:
```json
{
"name": "vmclusters/my-cluster-1",
"id": "",
"status": "",
"endpoints": {},
"values": {
"ha-mode": "true",
"vmstorage": {
"resources": {
"limits": {
"cpu": "500m",
"memory": "2Gi"
}
}
},
"vmselect": {
"affinity": {
"nodeAffinity": {
// ... node affinity configuration
}
},
"service": {
"type": "ClusterIP"
}
// full applied Helm values
},
"helm_release_info": {
"release_name": "my-cluster-1",
"release_namespace": "observability-ns",
"chart": "victoria-metrics/victoria-metrics-cluster",
"version": "0.0.1" // Chart version
}
// More detailed status information can be added here if needed
}
}
```
* **Code**: `202 Accepted`, `400 Bad Request`, `409 Conflict`.
---
* `GET /api/v1/observability-ops/vmclusters` **: List VM Clusters.**
* **Description**: Lists all deployed VictoriaMetrics clusters managed by the Ops API Server.
* **Request Body**: None
* **Responses**:
```json
{
"vmclusters": [
{
"name": "vmclusters/my-cluster-1",
"id": "",
"status": "RUNNING",
"endpoints": {
"prometheus": "http://my-cluster-1-vmselect:8481/prometheus",
"grafana": "http://my-cluster-1-grafana"
}
// values are typically not included in list responses
},
{
"name": "vmclusters/another-vm",
"id": "",
"status": "UPDATING",
"endpoints": {} // Endpoints might be unavailable during update
}
]
// "next_page_token": "..." // Included if pagination is supported (AIP-158)
}
```
* **Status code**: `200 OK`
---
* `GET /api/v1/observability-ops/vmclusters/{vmclusterId}` **: Retrieves the details of a specific VictoriaMetrics cluster.**
* **Description** : Gets details of a single deployed O11y component.
* **Path parameters**: `{vmcluster}` vmcluster uuid.
* **Response**: Individual component details.
```json
{
"name": "vmclusters/my-cluster-1",
"id": "",
"status": "RUNNING",
"endpoints": {
"prometheus": "http://my-cluster-1-vmselect:8481/prometheus",
},
"values": {
"replicaCount": 1,
"vmstorage": {
"resources": {
"limits": {
"cpu": "500m",
"memory": "2Gi"
}
}
},
"vmselect": {
"affinity": {
"nodeAffinity": {
// ... node affinity configuration
}
}
},
"service": {
"type": "ClusterIP"
}
// ... full applied Helm values
},
"helm_release_info": {
"release_name": "my-cluster-1",
"release_namespace": "observability-ns",
"chart": "victoria-metrics/victoria-metrics-cluster",
"version": "0.0.1" // Chart version
}
// More detailed status information can be added here if needed
}
```
* **Status code**: `200 OK`, `404 Not Found`.
---
* `PUT /api/v1/observability-ops/vmclusters/{vmclusterId}` **:change vm cluster**
* **Description** : Updates the configuration of an existing VictoriaMetrics cluster by performing a Helm upgrade. The new configuration is provided via Helm values within the full vmcluster resource representation.
* **Path parameter**: `{vmclusterId}` :cluster id.
* **Request Body**:
```json
{
"values": {
"vmstorage": {
"resources": {
"requests": {
"cpu": "500m", // Example change
"memory": "2Gi" // Example change
},
"limits": {
"cpu": "1000m", // Example change
"memory": "4Gi" // Example change
}
}
},
// ... updated desired Helm values
}
// helm_release_info is server-managed
}
```
* **Response**: Returns the updated `HelmRelease` CR basic information.
* **Status code**: `202 Accepted`, `400 Bad Request`, `404 Not Found`.
---
* `DELETE /api/v1/observability-ops/vmclusters/{vmclusterId}` **: Uninstall the VM cluster.**
* **Description**: Deletes a VictoriaMetrics cluster by uninstalling the associated Helm release.
* **Path parameters**: `{ vmclusterId }` : cluster uuid
* **Response**:
```
...
```
* **Status Code**: `202 Accepted`, `404 Not Found`.
## Cluster Upgrade
Cluster upgrade operations create `ClusterUpgradeOperation` CRs in the `observability` namespace.
| Method | Path | Description |
| --- | --- | --- |
| `POST` | `/ops/api/v1/upgradeOperations` | Create a cluster upgrade operation. |
| `POST` | `/ops/api/v1/upgradeOperations/dry-run` | Build and return the CR that would be created, without creating it. |
| `GET` | `/ops/api/v1/upgradeOperations` | List cluster upgrade operations. Optional query: `k8sClusterName`, `status`, pagination fields. |
| `GET` | `/ops/api/v1/upgradeOperations/{id}` | Get upgrade operation details. |
| `PUT` | `/ops/api/v1/upgradeOperations/{id}/pause` | Pause an upgrade operation. |
| `PUT` | `/ops/api/v1/upgradeOperations/{id}/resume` | Resume a paused upgrade operation. |
| `DELETE` | `/ops/api/v1/upgradeOperations/{id}` | Delete or cancel an upgrade operation. |
| `GET` | `/ops/api/v1/nodeGroups` | List node groups. Required query: `k8sClusterName`, `region`, `cloudProvider`. |
| `GET` | `/ops/api/v1/addons` | List cluster addons. Required query: `k8sClusterName`, `region`, `cloudProvider`. |
Upgrade request:
```json
{
"k8sClusterName": "my-eks-cluster",
"region": "us-west-2",
"cloudProvider": "eks",
"targetVersion": "1.30",
"schedule": "2026-05-06T12:00:00Z",
"clusterAddons": [
{
"name": "vpc-cni",
"version": "v1.18.5-eksbuild.1",
"resolveConflicts": "PRESERVE"
}
],
"nodeGroupUpgrades": [
{
"nodeGroupName": "observability-ng",
"strategy": "in-place",
"requiresVMStorageRollout": false
}
]
}
```
`cloudProvider` must be one of `eks`, `gke`, `aks`, `ack`, or `mock`. `strategy` must be `blue-green` or `in-place`. Cluster addons are supported for `eks`, `ack`, and `mock`.
VMStorage node group upgrades have additional constraints:
* `requiresVMStorageRollout: true` is only valid with `strategy: "blue-green"`.
* `vmClusterHelmValues` is a partial `VMCluster.spec` overlay applied to `VMCluster observability/vmcluster`; it is not HelmRelease values and is not applied during in-place upgrades.
* `vmClusterHelmValues` requires `requiresVMStorageRollout: true`, and VMStorage rollout requires the `vmClusterHelmValues` field to be present. Use `{}` when no VMCluster spec change is needed.
* `vmstorageSTSName` and `vmstorageSTSNamespace` are used only to observe rollout status and pod placement. If omitted for VMStorage rollout, ops-server defaults them to `vmstorage-vmcluster` and `observability`.
* The vmstorage StatefulSet is not patched directly to trigger rollout because it is managed by the VM operator.
Example VMStorage blue-green node group entry:
```json
{
"nodeGroupName": "vmstorage",
"strategy": "blue-green",
"requiresVMStorageRollout": true,
"newNodeGroupName": "vmstorage-130",
"vmClusterHelmValues": {
"vmstorage": {
"replicaCount": 3
}
}
}
```
## User scenarios
* [Design: Deploy Connected BYOC region]
### BYOC - PingCAP-managed
* **Scenario description** : Users manage their own infrastructure and K8s clusters.
* **How to**:
* PingCAP developed based on ops-server Ops API to implement troubleshooting and operation and maintenance operations
### BYOC - self-managed
* **Scenario description** : Users manage their own infrastructure and K8s clusters.
* **How to**:
* **Day-1/Day-2 Ops** : After Ops Server deployment is completed, users can directly manage Helm Releases through K8s API or implement operation and maintenance through Ops Server API using their own CI/CD system.
### TiDBCloud
* **Scenario description** : Pingcap internal, deploy O11y service to internal or tenant
* **How to**:
* ...