--- title: "TiProxy Bandwidth Utilization Custom Metric (ext_avg_bandwidth_utilization)" updated: 2026-04-07 watches: - pkg/metricsprovider/ - cmd/autoscale_stub/main.go - pkg/k8s/utils.go - domain/entity/entity.go - domain/entity/utils.go - domain/server/get_hpa_spec.go - domain/server/set_hpa_spec.go - domain/server/set_scale_out_config.go - domain/server/set_hpa_scale_behavior.go - deployments/templates/tiproxy_group.templ - internal/dataplane-manager/helm/ng-autoscaler/ --- ## Summary This proposal adds a TiProxy bandwidth-based object metric, `ext_avg_bandwidth_utilization`, to `autoscale_stub` and also defines the missing prerequisite TiProxy HPA plumbing required to consume that metric. The key design changes are: 1. Add AWS-gated TiProxy HPA support end-to-end. 2. Compute bandwidth utilization from node-exporter data reached directly from `autoscale_stub`. 3. Derive the TiProxy pod set from the target HPA's `scaleTargetRef` selector, not from a namespace-wide `tiproxy` label scan. 4. Avoid `MatchingFields{"spec.nodeName": ...}` for node-exporter lookup in `autoscale_stub`, because the shared `k8s.NewClient` path is cache-backed for `Pod` reads. 5. Add a small in-memory counter cache so the adapter can convert node-exporter byte counters into byte-per-second rates. After these changes, the proposal is implementation-ready. ## Deployment Topology `ng-autoscaler` (`autoscale_stub`) runs inside the dataplane cluster, not the control plane: - It is deployed remotely by dataplane-manager into the dataplane Kubernetes cluster. - `cmd/autoscale_stub/main.go` builds an in-cluster Kubernetes client from the pod ServiceAccount. - TiProxy pods and node-exporter pods are in the same dataplane cluster. - Existing RBAC in `internal/dataplane-manager/helm/ng-autoscaler/templates/rbac.yaml` already grants cluster-wide `pods` get/list/watch. Consequence: - `autoscale_stub` can reach node-exporter directly over plain HTTP. - No cluster-agent or clustergateway hop is required. ## Prerequisite: TiProxy HPA Enablement This branch does not support TiProxy HPA today. The metric cannot ship alone. Missing support that must be added in the same implementation: - `domain/entity/entity.go` Add `apiv1.ComponentType_TIPROXY` to `EnableHAPComponent`. - `domain/entity/utils.go` Make `GetComponentHPATemplateName` return a new `tiproxy_hpa` template for `TIPROXY`. - `deployments/templates/tiproxy_hpa.templ` New HPA template with: - `kind: HorizontalPodAutoscaler` - `scaleTargetRef.kind: TiProxyGroup` - `scaleTargetRef.name: {{!ComponentGroup}}` - `pingcap.com/group: {{!ComponentGroup}}` - `pingcap.com/component: tiproxy` - object metric `ext_avg_bandwidth_utilization` - `domain/server/get_hpa_spec.go` Recognize TiProxy HPA objects when listing HPA specs. - `domain/server/set_hpa_spec.go` Allow `TIPROXY` HPA patching using selector: - `pingcap.com/component=tiproxy` - `pingcap.com/group=db--tiproxy` - `domain/server/set_scale_out_config.go` Allow TiProxy HPA min/max/threshold updates using the same selector. - `domain/server/set_hpa_scale_behavior.go` Allow TiProxy HPA behavior updates using the same selector. - `pkg/autoscale/modules/common/types.go` Add `HPAComponentTypeTiProxy` and `BuildLabels` support for group-scoped TiProxy HPA lookup. Provider gate: - Initial rollout is AWS only. - TiProxy HPA rendering must be gated in the HPA-apply path so non-AWS dataplanes do not receive an HPA that depends on this metric before provider-specific validation exists. - Implement that gate in `domain/workflow/action/apply_hpa.go` before template execution. ## How the Metric Finds the Correct TiProxy Pods Do not list all `app.kubernetes.io/component=tiproxy` pods in the namespace. That is not group-safe when a namespace contains multiple TiProxy groups. Use the same selector derivation pattern as `ext_cpu_scaled_ratio`: 1. The request is for a single HPA object. 2. Load that HPA. 3. Read `hpa.Spec.ScaleTargetRef`. 4. Resolve the scale subresource with `getScale(...)`. 5. Parse `scaleObj.Status.Selector`. 6. List only the pods matching that selector in the HPA namespace. This gives the exact TiProxy pod set for that HPA and keeps the implementation aligned with existing HPA-object metric logic in `pkg/metricsprovider/provider.go`. ## Node-Exporter Discovery Reuse the same constants already used by cluster-agent: ```go nodeExporterNamespace = "monitoring" nodeExporterInstanceKey = "app.kubernetes.io/instance" nodeExporterInstanceValue = "node-exporter" nodeExporterPortName = "metrics" ``` Do not copy cluster-agent's `MatchingFields{"spec.nodeName": ...}` lookup literally. That works there because cluster-agent builds a direct `client.New(...)` client, while `autoscale_stub` uses `k8s.NewClient(...)`, which is cache-backed for `Pod` reads and therefore would require a field index. Instead, for each metric evaluation: 1. List all `monitoring` namespace pods with label `app.kubernetes.io/instance=node-exporter`. 2. Keep only `Running` pods. 3. Build an in-memory map `nodeName -> nodeExporterPod`. 4. For each TiProxy pod's `Spec.NodeName`, pick the co-located node-exporter pod from that map. 5. Read the node-exporter port named `metrics`. 6. `GET http://{nodeExporterPod.Status.HostIP}:{port}/metrics`. This avoids any cache-index dependency and keeps lookup deterministic. ## Rate Computation: Required Cache The node-exporter metrics used here are cumulative byte counters. A single scrape is not enough to compute utilization. Therefore `cmProvider` needs an in-memory cache of the previous sample: ```go type networkCounterSample struct { Timestamp time.Time RxBytes float64 TxBytes float64 SpeedBps float64 } ``` Suggested cache key: - `namespace/hpaName/nodeName/device` Evaluation flow: 1. Scrape current counters and interface speed from node-exporter. 2. Load the previous sample from cache. 3. Compute `deltaBytes / deltaSeconds` for receive and transmit. 4. Compute `rxRate / speed` and `txRate / speed` for each eligible interface. 5. Track the max receive utilization and max transmit utilization for the node. 6. Store the new sample back into cache. Cold-start and reset handling: - If there is no previous sample for an interface, store the sample and mark it unusable for this round. - If `deltaSeconds <= 0`, `deltaBytes < 0`, or the previous sample is older than 2× the expected scrape interval (60 seconds), treat it as a reset and re-baseline without contributing a value. - If no TiProxy pod has at least one usable interface sample in the current round, return an error. This causes one warm-up scrape after adapter startup, which is acceptable and safer than returning `0`. ## Metric Definition **Name**: `ext_avg_bandwidth_utilization` Per interface: ```text rx_bytes_per_second / interface_speed_bytes_per_second tx_bytes_per_second / interface_speed_bytes_per_second ``` Interface eligibility: - `speed > 0` - exclude loopback and common container/bridge interfaces by denylist: - `lo` - `veth.*` - `docker.*` - `cni.*` - `cali.*` - `flannel.*` - `br-.*` - `virbr.*` - `kube-ipvs0` **AWS device assumption — must validate before production**: the `eni` prefix is the standard AWS VPC CNI naming for ENI interfaces on EKS nodes. This has not been validated with a live node-exporter scrape from a dataplane cluster. Before deploying, an operator must confirm on a real dataplane node that: (a) `node_network_receive_bytes_total` and `node_network_transmit_bytes_total` carry `device` labels matching `eni*` for the primary network interface, and (b) `node_network_speed_bytes` is non-zero for those interfaces. If the naming differs, update the eligibility filter accordingly. If no interface survives eligibility filtering on a node, the implementation must return an error for that node: `"no eligible network interface found on node %s"`. The HPA will treat the missing metric as a stale value and not act on it. Per node: - max receive utilization across usable interfaces on that node - max transmit utilization across usable interfaces on that node Per pod: - the receive utilization of the node hosting that TiProxy pod - the transmit utilization of the node hosting that TiProxy pod Final HPA metric value: - `avgRx`: average of the per-pod receive utilization values across the HPA-selected TiProxy pods - `avgTx`: average of the per-pod transmit utilization values across the HPA-selected TiProxy pods - final metric: `max(avgRx, avgTx)` - returned as `int64` percentage `0..100` Why per-pod averaging instead of per-node averaging: - the HPA target is a pod group - using the HPA-selected pod set preserves normal HPA semantics - if two TiProxy pods share one saturated node, that saturation is reflected twice, which is appropriate for a pod-scaling decision ## HPA Template Shape New AWS-only `deployments/templates/tiproxy_hpa.templ`: ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: tiproxy-{{!ComponentGroup}} namespace: tidb{{!ClusterId}} labels: pingcap.com/group: {{!ComponentGroup}} pingcap.com/component: tiproxy pingcap.com/cluster: db spec: scaleTargetRef: apiVersion: core.pingcap.com/v1alpha1 kind: TiProxyGroup name: {{!ComponentGroup}} minReplicas: {{!MinNodeCount}} maxReplicas: {{!MaxNodeCount}} metrics: - type: Object object: metric: name: ext_avg_bandwidth_utilization describedObject: apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler name: tiproxy-{{!ComponentGroup}} target: type: Value value: 50 ``` ## TiProxy Pod Labels Relevant to This Metric From `deployments/templates/tiproxy_group.templ`: - object label: `pingcap.com/group={{!ComponentGroup}}` - pod label: `app.kubernetes.io/instance={{!ComponentGroup}}` - pod label: `app.kubernetes.io/component=tiproxy` This confirms the HPA's scale selector can stay group-scoped. ## Implementation Scope | File | Change | ~LoC | |------|--------|------| | `deployments/templates/tiproxy_hpa.templ` | New AWS-only TiProxy HPA template | ~60 | | `domain/entity/entity.go` | Enable TiProxy HPA support | ~5 | | `domain/entity/utils.go` | Map TiProxy to `tiproxy_hpa` | ~5 | | `domain/workflow/action/apply_hpa.go` | AWS gate for TiProxy HPA rendering | ~10 | | `domain/server/get_hpa_spec.go` | Read TiProxy HPA | ~20 | | `domain/server/set_hpa_spec.go` | Patch TiProxy HPA | ~20 | | `domain/server/set_scale_out_config.go` | Patch TiProxy HPA min/max/thresholds | ~20 | | `domain/server/set_hpa_scale_behavior.go` | Patch TiProxy HPA behavior | ~20 | | `pkg/autoscale/modules/common/types.go` | Add TiProxy HPA type/labels | ~15 | | `pkg/metricsprovider/ext_avg_bandwidth_utilization.go` | New metric implementation + cache + scrape parsing | ~220 | | `pkg/metricsprovider/ext_avg_bandwidth_utilization_test.go` | New table-driven tests | ~220 | | `pkg/metricsprovider/provider.go` | Metric constant, dispatch, `ListAllMetrics` | ~30 | | **Total** | | **~645 LoC** | No RBAC change is needed. No cluster-agent change is needed. ## Test Plan Unit tests: - HPA -> scale selector -> pod selection returns only the target TiProxy group pods. - node-exporter pod discovery uses label-only lookup and nodeName matching. - first scrape returns warm-up error when no previous counter sample exists. - second scrape computes utilization correctly from deltas. - counter reset or stale cache re-baselines cleanly. - interface denylist excludes loopback and container virtual devices. - multiple pods on the same node yield duplicated per-pod node utilization before averaging. - `ListAllMetrics` exposes the new HPA object metric. Template / server tests: - TiProxy HPA template renders with `TiProxyGroup` scaleTargetRef. - HPA server CRUD surfaces recognize `TIPROXY`. - AWS gate prevents TiProxy HPA template application on non-AWS dataplanes. ## Code Locations | What | Where | |------|-------| | Existing HPA-object metric precedent | `pkg/metricsprovider/provider.go` (`ext_cpu_scaled_ratio`) | | autoscale_stub kube client construction | `cmd/autoscale_stub/main.go` | | cache-backed shared client behavior | `pkg/k8s/utils.go` | | cluster-agent node-exporter constants | `internal/cluster-agent/service/node_exporter.go` | | TiProxy group template and labels | `deployments/templates/tiproxy_group.templ` | | HPA apply action | `domain/workflow/action/apply_hpa.go` | | HPA read/update server flows | `domain/server/get_hpa_spec.go`, `domain/server/set_hpa_spec.go`, `domain/server/set_scale_out_config.go`, `domain/server/set_hpa_scale_behavior.go` |