# Ops Server Cluster Upgrade Skill

Use this workflow when a user asks for a runnable `curl` request to upgrade a Kubernetes cluster through ops-server, especially when addons and VMStorage node groups are involved.

## Required User Inputs

Ask the user to provide:

- Ops-server address, for example `http://127.0.0.1:8180`
- Kubernetes cluster name, for example `o11y-prod-regional-us-west-2-staging`
- Vendor/cloud provider, using the ops-server `cloudProvider` value: `eks`, `gke`, `aks`, or `ack`
- Region, for example `us-west-2`
- Target Kubernetes version, for example `1.35`

If the user does not provide addon target versions, query or confirm them before generating the final request. The ops-server addons API returns the currently installed addon versions, not necessarily the correct target versions for the requested Kubernetes version.

## Read Local API Docs First

Start by reading `cmd/opsserver/README.md`. The cluster upgrade API is:

- Dry-run: `POST /ops/api/v1/upgradeOperations/dry-run`
- Create: `POST /ops/api/v1/upgradeOperations`
- List node groups: `GET /ops/api/v1/nodeGroups?k8sClusterName=...&region=...&cloudProvider=...`
- List addons: `GET /ops/api/v1/addons?k8sClusterName=...&region=...&cloudProvider=...`

Use `--noproxy '*'` for localhost ops-server requests if the environment has proxy variables set.

## Discovery Commands

Replace variables before running:

```bash
OPS_SERVER='http://127.0.0.1:8180'
CLUSTER='o11y-prod-regional-us-west-2-staging'
CLOUD_PROVIDER='eks'
REGION='us-west-2'
```

Check ops-server health and service info:

```bash
curl --noproxy '*' -sS "${OPS_SERVER}/health"
curl --noproxy '*' -sS "${OPS_SERVER}/ops/api/v1/"
```

List installed addons:

```bash
curl --noproxy '*' -sS "${OPS_SERVER}/ops/api/v1/addons?k8sClusterName=${CLUSTER}&region=${REGION}&cloudProvider=${CLOUD_PROVIDER}"
```

List node groups:

```bash
curl --noproxy '*' -sS "${OPS_SERVER}/ops/api/v1/nodeGroups?k8sClusterName=${CLUSTER}&region=${REGION}&cloudProvider=${CLOUD_PROVIDER}"
```

Check existing upgrade operations before creating a new one:

```bash
curl --noproxy '*' -sS "${OPS_SERVER}/ops/api/v1/upgradeOperations?k8sClusterName=${CLUSTER}"
```

## Build The Upgrade Request

Request fields:

- `k8sClusterName`: cluster name from the user
- `region`: region from the user
- `cloudProvider`: vendor/cloud provider from the user
- `targetVersion`: target Kubernetes version
- `clusterAddons`: addons to upgrade, each with `name`, target `version`, and optional `resolveConflicts`
- `nodeGroupUpgrades`: node groups to upgrade, each with `nodeGroupName`, `strategy`, and `requiresVMStorageRollout`

For EKS addons, `resolveConflicts: "PRESERVE"` is a conservative default when the goal is to keep existing custom settings. Valid values are `NONE`, `OVERWRITE`, and `PRESERVE`.

## Upgrade Step Ordering

The service builds steps in a fixed order:

1. `upgrade-controlplane`, one step per intermediate Kubernetes version
2. `upgrade-cluster-addons`, one step per addon in the request order
3. `upgrade-node-group`, one step per node group in the request order

Because addons are upgraded after the control plane, addon target versions in a combined request should be compatible with the final target Kubernetes version. If the request contains only addon changes and no control plane upgrade, choose versions compatible with the current control plane version.

For Kubernetes minor-version upgrades on EKS, `kube-proxy` may be included in the same operation when its target version matches the final Kubernetes target version, because ops-server runs control plane steps before addon steps.

Do not include already-current addons just to mirror a template. If the installed addon version is already the intended target version, omit it from `clusterAddons` to avoid unnecessary addon update operations.

## Vendor-Specific Restrictions

The request model accepts `eks`, `gke`, `aks`, `ack`, and `mock`, but the real provider registry initializes only `eks`, `gke`, `aks`, and `ack`. Treat `mock` as a test-only value unless a mock provider has been explicitly registered in the process.

### All Vendors

- Control plane upgrades are expanded into one-minor-version steps, for example `1.32 -> 1.33 -> 1.34`.
- Cross-major upgrades are not supported by this workflow.
- Do not generate downgrade requests. The version planner only steps forward; a lower target version will not produce a useful control plane upgrade plan.
- The service checks node group compatibility before creating the operation.
- For non-ACK providers, node groups must be at most 3 minor versions behind the target control plane version.
- For ACK, node groups must be at most 1 minor version behind the target control plane version; an empty ACK node group version is ignored because ACK may not return it.

### EKS

- Cluster addons are supported.
- Use addon target versions that are compatible with the target Kubernetes version. Query AWS EKS addon versions or ask the user to provide them; do not assume the currently installed addon versions are target versions.
- For same-operation control plane upgrades, choose addon versions compatible with the final target Kubernetes version. The service schedules addon updates after all control plane steps.
- `resolveConflicts` is forwarded to EKS as `NONE`, `OVERWRITE`, or `PRESERVE`.
- Node group `in-place` and `blue-green` strategies are implemented.
- If an EKS node group uses a launch template with a custom AMI, use `launchTemplateVersion` for `strategy: "in-place"`; `launchTemplateId` or `launchTemplateName` can be included when you need to override the current template identifier. If the launch template does not specify an AMI ID, the node group can be updated through the normal `version` path without changing the launch template.
- VMStorage node groups should use `blue-green`, not `in-place`, when the user expects the VMStorage-specific migration flow.

### ACK

- Cluster addons are supported.
- ACK control plane requests may upgrade at most one minor version per operation. If the calculated intermediate version list has more than one item, dry-run fails with `maxVersionDiff is 1 for ACK`; create sequential operations instead.
- ACK node group `blue-green` is rejected by request building. Use `strategy: "in-place"`.
- ACK node group `in-place` may include `runtimeType` and `targetRuntimeVersion`; these are forwarded to ACK as runtime upgrade options.
- ACK `resolveConflicts` handling differs from EKS. Through the current API validation, only `OVERWRITE` is actually forwarded as an ACK addon policy. `PRESERVE` and `NONE` are accepted by request validation but are not forwarded as ACK policies.
- ACK `CreateNodeGroup` and `DeleteNodeGroup` are not implemented, which is why blue-green should not be used.

### GKE

- Do not include `clusterAddons`; buildUpgradeSteps rejects addon upgrades for GKE.
- The direct addons listing endpoint returns an empty list because GKE addons are not modeled as named addon versions in this workflow.
- Control plane upgrades and node pool `in-place` upgrades are implemented.
- Node pool `blue-green` uses create, cordon, and delete logic and is implemented through the generic node group flow, but verify dry-run and provider behavior for the specific cluster before giving a final command.
- GKE node pool upgrades may return a cloud operation ID; the reconciler polls that operation until it completes.

### AKS

- Do not include `clusterAddons`; buildUpgradeSteps rejects addon upgrades for AKS.
- The direct addons listing endpoint returns an empty list because AKS addons are not modeled as named addon versions in this workflow.
- Control plane upgrades and agent pool `in-place` upgrades are implemented.
- Agent pool `blue-green` uses create, cordon, and delete logic and is implemented through the generic node group flow, but verify dry-run and provider behavior for the specific cluster before giving a final command.

## VMStorage Node Group Guidance

If a node group is the VMStorage node group and the user expects the VMStorage-specific migration flow, use the blue-green strategy and include both the VMStorage rollout observation fields and a VMCluster spec overlay. The VMCluster CR is the source of truth for vmstorage configuration; do not rely on direct StatefulSet patching to trigger the rollout.

```json
{
  "nodeGroupName": "vmstorage",
  "strategy": "blue-green",
  "requiresVMStorageRollout": true,
  "newNodeGroupName": "vmstorage-135",
  "vmstorageSTSName": "vmstorage-vmcluster",
  "vmstorageSTSNamespace": "observability",
  "vmClusterHelmValues": {
    "vmstorage": {
      "replicaCount": 3
    }
  }
}
```

This is only an example; adapt the node group name, new node group name, StatefulSet name, namespace, and VMCluster overlay to the target cluster. Do not copy the example `vmClusterHelmValues` unless the requested VMCluster spec change is actually desired.

Do not use `requiresVMStorageRollout: true` with `strategy: "in-place"` when the user expects the VMStorage-specific migration flow. Ops-server rejects that combination; the VMStorage rollout, cordon, and old-nodegroup deletion flow is implemented only in the `blue-green` branch. If the VMStorage workload is light enough for regular node rotation, consider an in-place upgrade instead and leave `requiresVMStorageRollout` false.

Choose a deterministic `newNodeGroupName` when generating a runnable request. If omitted, the controller generates a random name, which is less convenient for review and rollback discussions.

Be careful with "VMStorage migration only" requests where the user does not want to change VMCluster config. The VMStorage rollout path no longer supports direct StatefulSet patching, so `requiresVMStorageRollout: true` must include `vmClusterHelmValues`. If no VMCluster spec change is needed, set `vmClusterHelmValues` to `{}`.

### VMCluster Config During VMStorage Upgrade

Ops-server can change VMCluster configuration during a node group upgrade, but only through the VMStorage blue-green path.

Request-side behavior:

- The request field is `nodeGroupUpgrades[].vmClusterHelmValues`.
- `buildNodeGroupUpgradeOptions` only forwards `vmClusterHelmValues`, `vmstorageSTSName`, and `vmstorageSTSNamespace` when `requiresVMStorageRollout: true`.
- `requiresVMStorageRollout: true` is rejected unless `strategy` is `blue-green`.
- `vmClusterHelmValues` is rejected unless `requiresVMStorageRollout` is true, and `requiresVMStorageRollout` requires the `vmClusterHelmValues` field to be present.
- When rollout is needed but no VMCluster config should change, use `vmClusterHelmValues: {}`.
- If `vmstorageSTSName` or `vmstorageSTSNamespace` is omitted for a VMStorage rollout, ops-server fills `vmstorage-vmcluster` and `observability`.
- VMStorage rollout fields are omitted from normal node group upgrades where `requiresVMStorageRollout` is false.
- `strategy: "in-place"` does not run the VMStorage migration flow, so do not rely on it to apply VMCluster changes.

Controller-side behavior:

- The VMCluster change is applied only in the blue-green `vmstorage-rollout` sub-step.
- Despite the field name, `vmClusterHelmValues` is not patched into HelmRelease values. The controller treats it as a partial `VMCluster.spec` overlay and patches the VM operator resource `VMCluster observability/vmcluster`.
- The overlay may be either a bare spec object, for example `{"vmstorage": {...}}`, or a full-resource-shaped object with `{"spec": {"vmstorage": {...}}}`.
- The merge is recursive for JSON objects. Scalar values and arrays overwrite existing values. `null` deletes the target key.
- The controller annotates the VMCluster with the cluster-upgrade timestamps after patching it.
- The vmstorage StatefulSet is still read to observe rollout status and pod placement, but it is no longer patched to trigger the rollout.
- If `requiresVMStorageRollout: true` is set and `vmClusterHelmValues` is omitted, ops-server rejects the request before creating the operation.

Example overlay:

```json
{
  "vmstorage": {
    "replicaCount": 3
  }
}
```

Important edge case: if `requiresVMStorageRollout: true` is set and the request omits `vmClusterHelmValues`, dry-run/create fails before the operation is created. When the intent is to modify VMCluster, pass the desired overlay object. When the intent is only to move vmstorage pods without changing VMCluster config, pass `{}`.

## Dry-Run Before Final Curl

Always dry-run the final payload first:

```bash
curl --noproxy '*' -sS -X POST "${OPS_SERVER}/ops/api/v1/upgradeOperations/dry-run" \
  -H 'Content-Type: application/json' \
  -d '<REQUEST_JSON>'
```

Confirm that the response has `code: 0` and that the generated steps match the intent:

- One or more `upgrade-controlplane` steps, depending on current and target versions
- One `upgrade-cluster-addons` step per requested addon, after the control plane steps
- One `upgrade-node-group` step per requested node group
- VMStorage node groups should show `strategy: "blue-green"` and `requiresVMStorageRollout: true`

## Monitor An Upgrade Operation

After the create request succeeds, list operations and then fetch the specific operation:

```bash
curl --noproxy '*' -sS "${OPS_SERVER}/ops/api/v1/upgradeOperations?k8sClusterName=${CLUSTER}"
curl --noproxy '*' -sS "${OPS_SERVER}/ops/api/v1/upgradeOperations/${OPERATION_ID}"
```

The ops API summary may not include `currentStep`, `nextSteps`, or failure details. When more detail is needed, read the `ClusterUpgradeOperation` CR directly:

```bash
kubectl get clusterupgradeoperation "${OPERATION_ID}" -n observability -o yaml
```

## Final Create Curl Template

After dry-run succeeds, switch the path from `/upgradeOperations/dry-run` to `/upgradeOperations`. Example:

```bash
curl --noproxy '*' -sS -X POST "${OPS_SERVER}/ops/api/v1/upgradeOperations" \
  -H 'Content-Type: application/json' \
  -d '{
    "k8sClusterName": "o11y-prod-regional-us-west-2-staging",
    "region": "us-west-2",
    "cloudProvider": "eks",
    "targetVersion": "1.35",
    "clusterAddons": [
      {
        "name": "coredns",
        "version": "v1.13.2-eksbuild.4",
        "resolveConflicts": "PRESERVE"
      },
      {
        "name": "vpc-cni",
        "version": "v1.21.1-eksbuild.7",
        "resolveConflicts": "PRESERVE"
      }
    ],
    "nodeGroupUpgrades": [
      {
        "nodeGroupName": "vmstorage",
        "strategy": "blue-green",
        "requiresVMStorageRollout": true,
        "newNodeGroupName": "vmstorage-135",
        "vmstorageSTSName": "vmstorage-vmcluster",
        "vmstorageSTSNamespace": "observability",
        "vmClusterHelmValues": {
          "vmstorage": {
            "replicaCount": 3
          }
        }
      }
    ]
  }'
```

This template omits `kube-proxy` only because the exact target addon set depends on the cluster. For a `1.34 -> 1.35` EKS upgrade, a `kube-proxy` `v1.35.x` target can be included in the same operation when it is the intended final addon version. Treat the addon versions in this template as an example from one investigation. For a different cluster, Kubernetes target version, region, or cloud provider, refresh the addon target versions before producing the final command.