--- title: "Proposal: Node-Exporter Metrics Proxy in Cluster-Agent" --- # Proposal: Node-Exporter Metrics Proxy in Cluster-Agent ## Background TiKV's own `/metrics` endpoint exposes application-level metrics (RocksDB, Raft, gRPC) but does not expose raw disk IOPS or throughput. Those live in `node_disk_reads_completed_total` / `node_disk_writes_completed_total` / `node_disk_read_bytes_total` / `node_disk_written_bytes_total` on the **node-exporter** daemon running as a DaemonSet on every node. Today there is no way to query node-exporter metrics for a specific TiKV pod through the cluster-agent API. Consumers must know the dataplane K8s topology to reach node-exporter directly, which is not feasible for external callers going through Cluster Gateway. ## Goal Expose node-exporter metrics for a given TiKV pod through the existing cluster-agent metrics proxy, maintaining the same URL shape as the TiKV metrics endpoint. ## Non-Goals - Filtering or aggregating metrics (raw passthrough only). - Supporting component types other than TiKV in the first version. - Replacing the monitoring stack (vmagent / VictoriaMetrics) for long-term storage. ## Proposed API New route, following the existing dual-prefix pattern: ``` GET /api/v1/clusters/:cluster_id/components/:component_type/:pod_name/node-metrics GET /cluster-agent/:tcns/api/v1/clusters/:cluster_id/components/:component_type/:pod_name/node-metrics ``` - `:component_type` is validated (only `tikv` supported initially). - `:pod_name` identifies which pod to use for node lookup — the response contains **node-scoped metrics** (all disks on the host), not pod-scoped metrics. Consumers must filter by device label in PromQL. - Port is resolved at runtime from the node-exporter pod co-located on the same node. - Response is the raw Prometheus text from `http://:/metrics`. The path suffix `/node-metrics` (not `/metrics`) avoids collision with the existing TiKV application metrics endpoint and makes the source explicit. ## Design ### How Node-Exporter Is Reached Node-exporter runs as a DaemonSet in the `monitoring` namespace. In our deployment (`k8s-infra-templates.yaml`, Helm release `node-exporter`, chart `prometheus-node-exporter`) it is configured with `hostNetwork: true`. The live DaemonSet binds `containerPort: 9100` on the host network interface directly (via `hostNetwork`, not `hostPort`). > **Assumption**: the cluster-agent pod can reach node IPs on the node-exporter port over > plain HTTP. This holds for our VPC-local deployment. Environments with strict pod-to-node > network policies would require a different approach (e.g. Service-based routing). The host IP is available as `pod.Status.HostIP` on the TiKV pod. The port is discovered at runtime by looking up the node-exporter pod co-located on the same node and reading the named container port `"metrics"`. **Why not use the node-exporter Service ClusterIP?** The chart does define a ClusterIP Service for node-exporter, and it can route to pod endpoints. However, using it would send the request to any node-exporter pod in the cluster rather than the instance on the same node as the TiKV pod. Since the goal is to obtain disk metrics for the specific host running that TiKV pod, we must target the node-exporter pod on that node directly via `pod.Status.HostIP`. Same-node affinity is the design constraint, not a routing limitation of the Service. **Why not hard-code port 9100?** The port is a deployment-time decision owned by the monitoring stack. Reading it from the pod spec keeps the code correct across environments where the port may differ. Request flow: ``` caller → cluster-agent :8080 (HTTPS, Gin) → GET tikv pod from dataplaneCli (namespace = "tidb" + cluster_id) → validate pod.Labels["app.kubernetes.io/component"] == "tikv" → validate pod.Status.Phase == Running → LIST pods in "monitoring" ns, labels app.kubernetes.io/name=node-exporter AND app.kubernetes.io/instance=node-exporter, field spec.nodeName = tikv pod.Spec.NodeName → assert exactly one Running pod returned (503 if zero, 502 if List error) → read containerPort from named port "metrics" on that pod → plain HTTP GET http://:/metrics → stream response body to caller ``` No mTLS is needed: node-exporter listens on plain HTTP inside the cluster network. ### Node-Exporter Pod Lookup ```go // errNodeExporterListFailed is a sentinel wrapping control-plane List errors. // The handler uses errors.Is(err, errNodeExporterListFailed) to return 502 (infrastructure // failure) rather than 503 (missing deployment resource). var errNodeExporterListFailed = errors.New("node-exporter list failed") func (s *AgentService) getNodeExporterPort(ctx context.Context, nodeName string) (int32, error) { var podList corev1.PodList if err := s.dataplaneCli.List(ctx, &podList, client.InNamespace(nodeExporterNamespace), client.MatchingLabels{ nodeExporterLabelKey: nodeExporterLabelValue, // "app.kubernetes.io/name": "node-exporter" nodeExporterInstanceKey: nodeExporterInstanceValue, // "app.kubernetes.io/instance": "node-exporter" }, client.MatchingFields{"spec.nodeName": nodeName}, ); err != nil { return 0, fmt.Errorf("%w: on node %s: %w", errNodeExporterListFailed, nodeName, err) } // Collect only Running pods; a DaemonSet should yield exactly one. var running []corev1.Pod for _, p := range podList.Items { if p.Status.Phase == corev1.PodRunning { running = append(running, p) } } if len(running) == 0 { return 0, fmt.Errorf("no running node-exporter pod found on node %s", nodeName) } if len(running) > 1 { return 0, fmt.Errorf("ambiguous: %d running node-exporter pods found on node %s", len(running), nodeName) } pod := running[0] for _, c := range pod.Spec.Containers { for _, p := range c.Ports { if p.Name == nodeExporterPortName { // "metrics" return p.ContainerPort, nil } } } return 0, fmt.Errorf("node-exporter pod on node %s has no port named %q", nodeName, nodeExporterPortName) } ``` New constants (alongside existing ones in `common.go` or a new `node_exporter.go`): ```go const ( nodeExporterNamespace = "monitoring" nodeExporterLabelKey = "app.kubernetes.io/name" nodeExporterLabelValue = "node-exporter" // live DS label; see note below nodeExporterInstanceKey = "app.kubernetes.io/instance" nodeExporterInstanceValue = "node-exporter" // Helm release name nodeExporterPortName = "metrics" ) ``` > **Label source**: the live EKS DaemonSet (`kubectl get ds/node-exporter -n monitoring`) carries > `app.kubernetes.io/name=node-exporter` and `app.kubernetes.io/instance=node-exporter`. These > values differ from the chart's compiled-in default (`prometheus-node-exporter`) and the exact > cause is not fully traced in this repo — the vendored chart is version 4.23.2 while the live > cluster runs 4.47.3, and deploy-time overrides outside this repo's static config may also > apply. The constants above reflect the **runtime contract verified on the live cluster**; the > live contract, not the vendored chart, is the selector source of truth for implementation. **Field selector prerequisite**: controller-runtime's `List` with `MatchingFields` on `spec.nodeName` requires the field to be indexed on the cache. Since `dataplaneCli` is a non-cached `client.Client` (created directly with `client.New`, not a cached manager client), field selectors are passed through as server-side field selectors — no local index registration needed at runtime. **Unit-test note**: tests that use `fake.NewClientBuilder()` from `sigs.k8s.io/controller-runtime/pkg/client/fake` must call `.WithIndex(&corev1.Pod{}, "spec.nodeName", func(o client.Object) []string { ... })` on the builder, because the fake client evaluates field selectors locally and does not delegate to the API server. Without this index registration, `MatchingFields{"spec.nodeName": nodeName}` will return zero results even when matching pods are present in the fake store. ### Key Metrics Available | Metric | Description | |---|---| | `node_disk_reads_completed_total` | Cumulative read IOPS counter (rate → IOPS) | | `node_disk_writes_completed_total` | Cumulative write IOPS counter | | `node_disk_read_bytes_total` | Cumulative read throughput counter | | `node_disk_written_bytes_total` | Cumulative write throughput counter | | `node_disk_io_time_seconds_total` | Disk utilization | | `node_disk_io_now` | In-flight I/Os at scrape time | Consumers compute IOPS with `rate(node_disk_reads_completed_total[5m])` in PromQL. ### Code Changes #### 1. `service/service.go` — new field and option ```go // nodeExporterHTTPClient is the plain HTTP client for reaching node-exporter // on the host network (port 9100). Never nil after NewAgentService; defaults // to a 10-second timeout client. nodeExporterHTTPClient *http.Client ``` ```go func WithNodeExporterHTTPClient(c *http.Client) Option { return func(as *AgentService) { as.nodeExporterHTTPClient = c } } ``` In `NewAgentService`, initialize the default: ```go as.nodeExporterHTTPClient = &http.Client{Timeout: 10 * time.Second} // then apply opts (opts may override it) ``` Unlike `tikvHTTPClient` (which is nil when `/client-ssl/` is absent), this client requires no external secrets, so it is always initialized and never nil. #### 2. `service/component_metrics.go` — new handler and case Add a new exported handler `HandleComponentPodNodeMetrics`: ```go // HandleComponentPodNodeMetrics proxies raw Prometheus metrics from the // node-exporter instance co-located with the given component pod. // Route: GET /clusters/:cluster_id/components/:component_type/:pod_name/node-metrics func (s *AgentService) HandleComponentPodNodeMetrics(c *gin.Context) { componentType := c.Param("component_type") clusterID := c.Param("cluster_id") podName := c.Param("pod_name") switch componentType { case TiKVComponentType: s.handleTiKVPodNodeMetrics(c, clusterID, podName) default: c.JSON(http.StatusNotImplemented, gin.H{"error": fmt.Sprintf("node metrics proxy not implemented for component: %s", componentType)}) } } ``` Implementation of `handleTiKVPodNodeMetrics`: ```go func (s *AgentService) handleTiKVPodNodeMetrics(c *gin.Context, clusterID, podName string) { ctx := c.Request.Context() ns := buildNamespace(clusterID) var pod corev1.Pod if err := s.dataplaneCli.Get(ctx, client.ObjectKey{Namespace: ns, Name: podName}, &pod); err != nil { // same 404 / 502 error handling as handleTiKVPodMetrics } if pod.Labels[K8sComponentLabelKey] != TiKVComponentType { c.JSON(http.StatusBadRequest, gin.H{"error": "pod is not a tikv component"}) return } if pod.Status.Phase != corev1.PodRunning { c.JSON(http.StatusServiceUnavailable, gin.H{"error": "pod is not running"}) return } port, err := s.getNodeExporterPort(ctx, pod.Spec.NodeName) if err != nil { tilog.Error(ctx, "failed to resolve node-exporter port", zap.String("pod", podName), zap.String("node", pod.Spec.NodeName), zap.Error(err)) // Distinguish infrastructure errors (502) from missing-resource errors (503). // getNodeExporterPort wraps List failures with errNodeExporterListFailed so the // handler can return the correct status without inspecting error strings. if errors.Is(err, errNodeExporterListFailed) { c.JSON(http.StatusBadGateway, gin.H{"error": "failed to list node-exporter pods"}) } else { // not-found, no-running-pod, ambiguous, or no named port c.JSON(http.StatusServiceUnavailable, gin.H{"error": "node-exporter unavailable"}) } return } metricsURL := fmt.Sprintf("http://%s:%d/metrics", pod.Status.HostIP, port) s.proxyNodeExporterMetrics(c, podName, metricsURL) } ``` ```go func (s *AgentService) proxyNodeExporterMetrics(c *gin.Context, podName, metricsURL string) { ctx := c.Request.Context() req, err := http.NewRequestWithContext(ctx, http.MethodGet, metricsURL, http.NoBody) if err != nil { ... } //nolint:gosec // G107: URL is built from validated pod.Status.HostIP + fixed path. resp, err := s.nodeExporterHTTPClient.Do(req) if err != nil { tilog.Error(ctx, "failed to reach node-exporter", zap.String("pod", podName), zap.Error(err)) c.JSON(http.StatusBadGateway, gin.H{"error": "failed to reach node-exporter"}) return } defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("node-exporter returned %d", resp.StatusCode)}) return } c.Header("Content-Type", resp.Header.Get("Content-Type")) c.Status(http.StatusOK) if _, err = io.Copy(c.Writer, resp.Body); err != nil { tilog.Error(ctx, "failed to stream node-exporter metrics", zap.String("pod", podName), zap.Error(err)) } } ``` #### 3. `cmd/agent/grpc.go` — route registration Extend `registerComponentMetricsRoutes`: ```go func registerComponentMetricsRoutes(router gin.IRoutes, agentSvc *service.AgentService) { componentMetricsPath := "/clusters/:cluster_id/components/:component_type/:pod_name/metrics" componentNodeMetricsPath := "/clusters/:cluster_id/components/:component_type/:pod_name/node-metrics" router.GET(baseAPIPath+componentMetricsPath, agentSvc.HandleComponentPodMetrics) router.GET(baseAPIPathWithPrefix+componentMetricsPath, agentSvc.HandleComponentPodMetrics) router.GET(baseAPIPath+componentNodeMetricsPath, agentSvc.HandleComponentPodNodeMetrics) router.GET(baseAPIPathWithPrefix+componentNodeMetricsPath, agentSvc.HandleComponentPodNodeMetrics) } ``` No other startup wiring is needed — `nodeExporterHTTPClient` is always initialized in `NewAgentService` without external dependencies. #### 4. `pkg/clustergateway/clustergateway.go` — new interface method Add after `GetComponentRawMetrics` in the `Client` interface: ```go // GetComponentNodeMetrics proxies raw Prometheus metrics from the node-exporter // instance co-located with the given component pod. // Returns the raw Prometheus text body as bytes. Only "tikv" is currently supported. GetComponentNodeMetrics(ctx context.Context, sharedPool *entity.SharedPool, componentType, podName string) ([]byte, error) ``` #### 5. `pkg/clustergateway/impl.go` — new operationMap entry Add after the `"GetComponentRawMetrics"` line in `operationMap`: ```go "GetComponentNodeMetrics": fmt.Sprintf( "%s/{%s}/api/v1/clusters/{%s}/components/{%s}/{%s}/node-metrics", clusterAgentApiPrefix, dataplaneNamespaceParameter, sharedPoolIdParameter, componentTypeParameter, podNameParameter, ), ``` #### 6. `pkg/clustergateway/clusteragentops.go` — new implementation Mirror `GetComponentRawMetrics` exactly, changing only the operation name: ```go func (c *clusterGatewayImpl) GetComponentNodeMetrics(ctx context.Context, sharedPool *entity.SharedPool, componentType, podName string) ([]byte, error) { resp, err := c.callClusterAgentNoRespLog(ctx, sharedPool, http.MethodGet, operationMap["GetComponentNodeMetrics"], func(req *resty.Request) *resty.Request { return req.SetPathParam(dataplaneNamespaceParameter, sharedPool.GetDataPlaneNamespace()). SetPathParam(sharedPoolIdParameter, sharedPool.GetId()). SetPathParam(componentTypeParameter, componentType). SetPathParam(podNameParameter, podName) }) if err != nil { return nil, fmt.Errorf("call GetComponentNodeMetrics failed: %w", err) } if resp.StatusCode() != http.StatusOK { return nil, fmt.Errorf("GetComponentNodeMetrics response not expected: status=%d body=%s", resp.StatusCode(), resp.String()) } return resp.Body(), nil } ``` #### 7. `pkg/clustergateway/mocks/mocks.go` — regenerate Run `make generate-mocks` after the interface change. Do not hand-edit the mock file. #### 8. `pkg/clustergateway/clusteragentops_test.go` — new tests Add two tests mirroring the existing `GetComponentRawMetrics` pair: - `TestGetComponentNodeMetrics_URLConstruction` — assert that the exact path `.../components/{componentType}/{podName}/node-metrics` is sent to the server. - `TestGetComponentNodeMetrics_NonOKStatus` — assert non-200 is wrapped as `"GetComponentNodeMetrics response not expected: status=… body=…"`. ### Error Responses | Status | Condition | |---|---| | `404 Not Found` | Pod not found | | `400 Bad Request` | Pod is not a tikv component | | `503 Service Unavailable` | Pod is not running; or no node-exporter pod found on the node; or node-exporter pod has no `"metrics"` port | | `502 Bad Gateway` | Control-plane `List` call failed; or node-exporter HTTP request failed; or node-exporter returned non-200 | | `501 Not Implemented` | component_type not supported | The distinction between 502 and 503 is meaningful to callers: 503 indicates a missing deployment-side resource (retrying later may help after a pod restart), while 502 indicates an infrastructure communication failure. ## Security Considerations - **Label guard**: pod must have `app.kubernetes.io/component=tikv`. Prevents using an arbitrary pod name to proxy to any host IP in the cluster. - **Fixed path**: the URL path is always `/metrics` — no user-controlled path component. - **HostIP scope**: `pod.Status.HostIP` is an internal node IP, not reachable from outside the VPC. Exposure is bounded to the cluster network. - **No TLS**: node-exporter listens on plain HTTP within the cluster. Acceptable given the traffic stays inside the VPC on the pod network. ## Testing Plan - **Unit**: table-driven tests in `component_metrics_test.go` using a Gin test router and an `httptest.Server` as the fake node-exporter, injected via `WithNodeExporterHTTPClient`. Cover: 200 passthrough, pod not found, wrong component type, pod not running, upstream error, no node-exporter pod on node, node-exporter pod has no named port. - **Integration**: `grpc_test.go` — verify the `/node-metrics` routes are handled by the Gin handler and not forwarded to grpc-gateway fallback (same assertion pattern as line 490-492). ## Effort Estimate ~140-170 net LoC (production code only, excluding tests and generated mocks): Cluster-agent (`internal/cluster-agent/`): - `service.go`: ~10 LoC (field + option + default init) - `component_metrics.go`: ~70 LoC (two handlers + proxy helper + `getNodeExporterPort`) - `common.go` or `node_exporter.go`: ~8 LoC (6 new constants) - `cmd/agent/grpc.go`: ~4 LoC (two route registrations) Cluster Gateway (`pkg/clustergateway/`): - `clustergateway.go`: ~4 LoC (new interface method) - `impl.go`: ~4 LoC (new `operationMap` entry) - `clusteragentops.go`: ~12 LoC (new implementation method) - `mocks/mocks.go`: regenerated (not counted)