---
title: Proposal - Component Pod Metrics Proxy API on Cluster-Agent
---

# Proposal: Component Pod Metrics Proxy API

## Background

Cluster Service (CS-NG) and other internal callers need raw Prometheus metrics from individual
component pods (initially TiKV) for use cases like autoscaler decisions, performance diagnosis,
and observability dashboards.

Currently there is no direct path to fetch per-pod metrics without going through Prometheus
(which has its own scrape cycle delay). Since cluster-agent runs in the same K8s cluster as
TiKV pods and already has proven TLS client code to call TiKV's status port (`topology.go`),
it is the natural proxy point.

## Goal

Add an HTTP API to cluster-agent that proxies raw Prometheus metrics from a specific component
pod's `/metrics` endpoint back to the caller. TiKV is implemented first; other component types
return `501 Not Implemented` as stubs.

## Architecture Context

Cluster-agent runs two servers concurrently:

- **`:8081`** — internal gRPC server (`AgentService` registered here via proto)
- **`:8080`** — public-facing Gin HTTP server; acts as grpc-gateway by dialing `:8081`

Kong calls `:8080`. The Gin router strips the Kong prefix via `ServePathStartFromAny`
middleware and dispatches to grpc-gateway's `mux`. All existing routes go through
`gin.WrapH(mux)`.

### Transport Shape Decision: Direct Gin vs grpc-gateway + HttpBody

grpc-gateway v2 (v2.27.1, the version used here) supports `google.api.HttpBody` **by
default**. `runtime.makeMarshalerMIMERegistry()` seeds `MIMEWildcard` with
`&HTTPBodyMarshaler{...}` as the default marshaler
(`runtime/marshaler_registry.go:20-29`). Cluster-agent's `NewServeMux(...)` call inherits
this — no `WithMarshalerOption` needed. Both approaches are therefore equally viable.

We compared them on the tradeoffs that actually matter:

| | grpc-gateway + HttpBody | Direct Gin handler |
|-|-------------------------|-------------------|
| Proto contract | Yes — RPC in services.proto | No — explicit contract exception |
| Streaming to HTTP writer | No — grpc-gateway materializes the full proto before writing | Yes — `io.Copy` streams chunk by chunk |
| Content-Type passthrough | Yes, via `HttpBody.content_type` | Yes, forwarded directly |
| Test surface | Proto + gateway round-trip | Gin routing + middleware |
| Consistency | Consistent with all other cluster-agent routes | Exception alongside `/healthz` |

**Decision: Direct Gin handler.**

Rationale:
1. **True streaming**: Prometheus metrics responses can be several hundred KB per pod.
   grpc-gateway materializes the full `HttpBody` in memory as a proto message before
   writing to the HTTP response writer. Direct Gin streams via `io.Copy` without
   intermediate buffering — lower peak memory, lower latency to first byte.
2. **Semantic fit**: The response is an opaque byte blob owned by TiKV, not a domain
   object. Wrapping it in a proto message adds no semantic value and forces an artificial
   proto RPC boundary for a passthrough operation.
3. **Contract exception is bounded**: The only callers are CSNG (via `clustergateway`)
   and future direct HTTP clients. The exception is explicit and documented below.

### Contract Exception Justification

All existing cluster-agent HTTP routes are generated from `services.proto` via grpc-gateway.
This endpoint intentionally does not follow that pattern. The justification:

- The API surface is operational/diagnostic, not a domain operation. It is analogous to
  `/healthz` — also registered directly on the Gin router outside grpc-gateway.
- The response format (Prometheus text) is not representable as a proto message without
  buffering overhead that defeats the purpose.
- The route is still governed by the same Kong authN/authZ and URL structure conventions.

**Intended callers**: This endpoint is designed for CSNG, accessed via
`pkg/clustergateway`. The `clustergateway.Client` interface and `operationMap` are the
authoritative contract **for CSNG callers**. Direct HTTP callers outside CSNG may use
the route but must discover it through cluster-agent's own documentation — it will not
appear in the generated OpenAPI spec from `services.proto`.

### HTTP Contract Table

This endpoint exists outside `services.proto` and grpc-gateway. Its contract is
defined here and must be kept in sync with `service/component_metrics.go` manually.

| Condition | Status | `Content-Type` | Body |
|-----------|--------|----------------|------|
| Success | `200 OK` | Forwarded from pod (`text/plain; version=0.0.4; charset=utf-8`) | Raw Prometheus text |
| Pod not found (true absence) | `404 Not Found` | `application/json` | `{"error": "pod not found"}` |
| Pod is not a tikv component | `400 Bad Request` | `application/json` | `{"error": "pod is not a tikv component"}` |
| TiKV TLS client not initialized | `503 Service Unavailable` | `application/json` | `{"error": "TiKV TLS client unavailable; check /client-ssl/ mount"}` |
| Pod not running | `503 Service Unavailable` | `application/json` | `{"error": "pod is not running"}` |
| Control-plane error (timeout, partition) | `502 Bad Gateway` | `application/json` | `{"error": "control plane unavailable: <detail>"}` |
| TiKV pod unreachable (network/TLS) | `502 Bad Gateway` | `application/json` | `{"error": "failed to reach tikv pod: <detail>"}` |
| TiKV pod returned non-200 | `502 Bad Gateway` | `application/json` | `{"error": "tikv returned <N>"}` |
| Component type not implemented | `501 Not Implemented` | `application/json` | `{"error": "metrics proxy not implemented for component: <type>"}` |
| Internal invariant failure (URL build or request construction) | `500 Internal Server Error` | `application/json` | `{"error": "<detail>"}` |

**Note**: `200` responses stream the body via `io.Copy`; partial content may be
written before a copy error occurs. If `io.Copy` fails mid-stream, the error is
logged but no HTTP error can be sent (headers already written).

## API Design

```
GET /clusters/{cluster_id}/components/{component_type}/{pod_name}/metrics
```

| Parameter | Description |
|-----------|-------------|
| `cluster_id` | TiDB cluster ID (e.g. `3841732`), derives K8s namespace `tidb{cluster_id}` |
| `component_type` | Component type: `tikv` (implemented), others return 501 |
| `pod_name` | Exact K8s pod name (e.g. `db-tikv-0`) |

**Response**: Raw Prometheus text format streamed directly from the pod's `/metrics`.
`Content-Type` is forwarded as-is from the pod response.

## CSNG Integration (ClusterGateway)

CSNG calls cluster-agent via `pkg/clustergateway/` using a resty HTTP client with
`InsecureSkipVerify: true` (TLS terminated at Kong). A new entry is needed in:

```go
// pkg/clustergateway/impl.go — operationMap
"GetComponentRawMetrics": fmt.Sprintf(
    "%s/{%s}/api/v1/clusters/{%s}/components/{componentType}/{podName}/metrics",
    clusterAgentApiPrefix, dataplaneNamespaceParameter, sharedPoolIdParameter,
)

// pkg/clustergateway/clustergateway.go — Client interface
GetComponentRawMetrics(ctx context.Context, sharedPool *entity.SharedPool,
    componentType, podName string) ([]byte, error)

// pkg/clustergateway/clusteragentops.go — implementation
// Returns resp.Body() as raw []byte — no protojson, it's Prometheus text
```

## Implementation Plan

### 1. AgentService Refactor (`cmd/agent/grpc.go` + `main.go`)

**Problem**: `AgentService` is currently created inside `setupAndRunGRPCServer` and is
inaccessible to `setupAndRunGRPCGateway`. The two goroutines in `main.go` share no state.

**Solution**: Extract `AgentService` construction before the goroutine split, pass it
into both functions as a parameter:

```go
// cmd/agent/main.go
agentSvc := buildAgentService(ctx, cfg)  // new shared constructor

wg.Add(1)
go func() {
    defer wg.Done()
    setupAndRunGRPCServer(ctx, cfg, agentSvc)
}()

wg.Add(1)
go func() {
    defer wg.Done()
    setupAndRunGRPCGateway(ctx, cfg, agentSvc)
}()
```

#### TLS Client Startup Failure Mode

`NewAgentService` currently does not return an error — it accepts optional deps via
the `Option` pattern, and missing deps degrade gracefully (see `metricsCli == nil` guard
in `resource_metrics.go:167`). The TiKV TLS client follows the **same graceful degradation
pattern**:

- `buildTiKVTLSClient()` is called during `buildAgentService`. If it fails (missing or
  unreadable `/client-ssl/*` files), the error is logged as a **Warn** and
  `tikvHTTPClient` is left `nil` on `AgentService`.
- Cluster-agent startup is **not** failed. This matches the `metricsCli` precedent:
  metrics-server unavailability is a `Warn` at startup, not a `Fatal`.

```go
// cmd/agent/grpc.go — buildAgentService
tikvClient, err := buildTiKVTLSClient()
if err != nil {
    tilog.Warn(ctx, "failed to build TiKV TLS client, metrics proxy will be unavailable",
        zap.Error(err))
    tikvClient = nil
}
opts = append(opts, service.WithTiKVHTTPClient(tikvClient))
```

#### Nil Client Behavior per Endpoint

**New metrics proxy handler** — return `503 Service Unavailable`:
```go
// service/component_metrics.go
if s.tikvHTTPClient == nil {
    c.JSON(http.StatusServiceUnavailable,
        gin.H{"error": "TiKV TLS client unavailable; check /client-ssl/ mount"})
    return
}
```

**Existing `GetClusterTopology(include_tikv_usage=true)`** — **no change**. Per
Decision #3/#8, `topology.go` is left entirely untouched. `fetchTiKVUsage` retains
its own per-call TLS client construction and existing error propagation behavior.
The nil-client concept does not apply to this path.

### 2. Plain Gin Handler Registration (`cmd/agent/grpc.go`)

Register on **both** path patterns (direct and Kong-prefixed) **before** the wildcard
`/*any` routes, so they take precedence over grpc-gateway:

```go
// inside setupAndRunGRPCGateway
metricsPath         := baseAPIPath + "/clusters/:cluster_id/components/:component_type/:pod_name/metrics"
metricsPathPrefixed := baseAPIPathWithPrefix + "/clusters/:cluster_id/components/:component_type/:pod_name/metrics"

apiRouter.GET(metricsPath,         agentSvc.HandleComponentPodMetrics)
apiRouter.GET(metricsPathPrefixed, agentSvc.HandleComponentPodMetrics)

// existing wildcard routes follow
apiRouter.Any(fmt.Sprintf("%s/*any", baseAPIPath), gin.WrapH(mux))
apiRouter.Any(fmt.Sprintf("%s/*any", baseAPIPathWithPrefix), gin.WrapH(mux))
```

`ServePathStartFromAny` is a global middleware — for the new plain Gin routes,
`ctx.Param("any")` is empty so it becomes a no-op (`ctx.Next()` immediately). No conflict.

### 3. Gin Handler (`service/component_metrics.go`, new file)

```go
func (s *AgentService) HandleComponentPodMetrics(c *gin.Context) {
    componentType := c.Param("component_type")
    clusterID     := c.Param("cluster_id")
    podName       := c.Param("pod_name")

    switch componentType {
    case "tikv":
        s.handleTiKVPodMetrics(c, clusterID, podName)
    default:
        c.JSON(http.StatusNotImplemented,
            gin.H{"error": fmt.Sprintf("metrics proxy not implemented for component: %s", componentType)})
    }
}

func (s *AgentService) handleTiKVPodMetrics(c *gin.Context, clusterID, podName string) {
    ctx := c.Request.Context()
    ns  := buildNamespace(clusterID)

    if s.tikvHTTPClient == nil {
        c.JSON(http.StatusServiceUnavailable,
            gin.H{"error": "TiKV TLS client unavailable; check /client-ssl/ mount"})
        return
    }

    var pod corev1.Pod
    if err := s.dataplaneCli.Get(ctx, client.ObjectKey{Namespace: ns, Name: podName}, &pod); err != nil {
        if apierrors.IsNotFound(err) {
            c.JSON(http.StatusNotFound, gin.H{"error": "pod not found"})
        } else {
            // Transient API-server failure (timeout, network partition, etc.)
            tilog.Error(ctx, "failed to get pod from control plane",
                zap.String("pod", podName), zap.Error(err))
            c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("control plane unavailable: %v", err)})
        }
        return
    }
    if pod.Labels[K8sComponentLabelKey] != "tikv" {
        c.JSON(http.StatusBadRequest, gin.H{"error": "pod is not a tikv component"})
        return
    }
    if pod.Status.Phase != corev1.PodRunning {
        c.JSON(http.StatusServiceUnavailable, gin.H{"error": "pod is not running"})
        return
    }

    host       := fmt.Sprintf("%s.db-cluster.%s.svc", pod.Name, pod.Namespace)
    metricsURL, err := buildTiKVMetricsURL(host)  // reuse from topology.go
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, metricsURL, http.NoBody)
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    resp, err := s.tikvHTTPClient.Do(req)
    if err != nil {
        c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("failed to reach tikv pod: %v", err)})
        return
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("tikv returned %d", resp.StatusCode)})
        return
    }

    c.Header("Content-Type", resp.Header.Get("Content-Type"))
    c.Status(http.StatusOK)
    if _, err = io.Copy(c.Writer, resp.Body); err != nil {
        tilog.Error(ctx, "failed to stream tikv metrics", zap.String("pod", podName), zap.Error(err))
    }
}
```

### 4. TLS HTTP Client (`service/service.go` + `service/component_metrics.go`)

Add `tikvHTTPClient *http.Client` to `AgentService`, initialized via `WithTiKVHTTPClient`
option. Extract TLS builder from `topology.go` into a shared `buildTiKVTLSClient()` helper:

```go
func buildTiKVTLSClient() (*http.Client, error) {
    cert, err := tls.LoadX509KeyPair("/client-ssl/tls.crt", "/client-ssl/tls.key")
    if err != nil {
        return nil, fmt.Errorf("failed to load client cert: %w", err)
    }
    caCert, err := os.ReadFile("/client-ssl/ca.crt")
    if err != nil {
        return nil, fmt.Errorf("failed to read CA cert: %w", err)
    }
    pool := x509.NewCertPool()
    pool.AppendCertsFromPEM(caCert)
    return &http.Client{
        Timeout: 30 * time.Second,
        Transport: &http.Transport{
            TLSClientConfig: &tls.Config{
                Certificates: []tls.Certificate{cert},
                RootCAs:      pool,
                MinVersion:   tls.VersionTLS12,
            },
        },
    }, nil
}
```

## TLS Details

| Item | Value |
|------|-------|
| TiKV status port | `20180` (`opv2.DefaultTiKVPortStatus`) |
| TiKV DNS | `{pod-name}.db-cluster.{namespace}.svc` |
| CA cert | `/client-ssl/ca.crt` (already mounted in cluster-agent pod) |
| Client cert | `/client-ssl/tls.crt` + `/client-ssl/tls.key` |
| Secret | `db-cluster-client-secret` — same used by all existing internal ops |

No new K8s secrets, volume mounts, or RBAC required.

### Rollout Preflight Checklist

Before enabling in any environment, verify:

| Check | Command / Signal | Expected |
|-------|-----------------|----------|
| Cert files readable | `ls -la /client-ssl/` from agent pod | `ca.crt`, `tls.crt`, `tls.key` present, world-readable |
| CA cert valid | `openssl x509 -noout -dates -in /client-ssl/ca.crt` | `notAfter` in the future |
| Client cert valid | `openssl x509 -noout -dates -in /client-ssl/tls.crt` | `notAfter` in the future |
| Pod `get` permission | `kubectl auth can-i get pods --as=system:serviceaccount:<ns>:cluster-agent` | `yes` |
| Namespace reachability | `curl -k https://<tikv-pod-dns>:20180/metrics` from agent pod | HTTP 200 |

**Degraded-mode observability**: If `buildTiKVTLSClient()` fails at startup, a `Warn`
log line is emitted (`"failed to build TiKV TLS client, metrics proxy will be unavailable"`).
Callers will receive `503 Service Unavailable` until the cert mount is fixed and the pod
is restarted. No silent failure — errors are explicit at both startup and request time.

## RBAC

No new K8s RBAC needed. Pod metadata reads (`get pods`) are already covered by the
existing cluster-agent ClusterRole.

## Security

- **Caller auth**: Kong gateway handles authN/authZ at the edge.
- **Pod scope**: Pod validated to exist in `tidb{cluster_id}` namespace with `tikv`
  component label — prevents proxying to arbitrary pods.
- **URL injection**: `pod_name` used only as a K8s object key. `buildTiKVMetricsURL`
  rejects inputs containing scheme/port/slash.

## Effort Estimate

| Component | LoC |
|-----------|-----|
| `service/component_metrics.go` (new) | ~90 |
| `service/component_metrics_test.go` (new) | ~110 |
| `cmd/agent/grpc_test.go` (new — router + middleware tests) | ~80 |
| `service/service.go` (add field + option) | ~15 |
| `cmd/agent/grpc.go` (route registration, accept agentSvc param) | ~25 |
| `cmd/agent/main.go` (extract shared AgentService construction) | ~15 |
| `service/topology.go` (add comment only, no functional change) | ~2 |
| `pkg/clustergateway/` (interface + impl + operationMap + mock regen) | ~40 |
| **Total** | **~385 LoC** |

## Notes on Secondary Concerns

### Router tests

The dual-path registration and `ServePathStartFromAny` middleware interaction must be
covered by router-level tests in `cmd/agent/grpc_test.go`:
- `GET /api/v1/clusters/{id}/components/tikv/{pod}/metrics` — direct path
- `GET /cluster-agent/{ns}/api/v1/clusters/{id}/components/tikv/{pod}/metrics` — Kong path
- Wildcard precedence: the above must not be swallowed by `/*any`
- `ServePathStartFromAny` no-op behavior for the new exact routes

### ClusterGateway `[]byte` vs streaming

`GetComponentRawMetrics` returns `[]byte` — the entire response body is buffered in CSNG
before returning to the upstream caller. This is intentional: CSNG is not a streaming proxy
and all existing cluster-agent calls follow the same buffered pattern. The "streamed directly"
wording in the proposal refers to streaming *within cluster-agent* (from TiKV pod to the
HTTP response writer via `io.Copy`). End-to-end streaming (TiKV → cluster-agent → CSNG →
caller) is not a current requirement.

If streaming end-to-end becomes a requirement in future, it would require a different
transport (e.g. server-side streaming gRPC or a dedicated proxy layer) — out of scope here.

### Mock regeneration

Adding `GetComponentRawMetrics` to the `clustergateway.Client` interface requires
regenerating `pkg/clustergateway/mocks/mocks.go` via `make generate-mocks`. This is
included in the implementation checklist.

## Decisions

| # | Question | Decision | Rationale |
|---|----------|----------|-----------|
| 1 | Generalize to other components? | Yes — `/{component_type}/` path, TiKV implemented, others return 501 | Path already supports it; zero extra cost now |
| 2 | HTTP client lifecycle | Create once at startup in `buildAgentService` | Strictly better than per-call; follows existing pattern for other clients |
| 3 | Refactor topology.go? | No — leave `fetchTiKVUsage` unchanged; add a comment only | Original author's design intent is unclear; don't change unrelated code; add a `// TODO: consider reusing s.tikvHTTPClient` comment |
| 4 | Response size cap? | Skip — no cap | TiKV owns what it exposes; passthrough as-is |
| 5 | Transport: grpc-gateway + HttpBody vs direct Gin? | Direct Gin — explicit contract exception | grpc-gateway v2 supports HttpBody by default but materializes full response; `io.Copy` streams without intermediate buffering; response is an opaque blob with no proto value |
| 6 | TLS client startup failure mode? | Graceful degrade — Warn + nil client | Matches `metricsCli` precedent; cert rotation windows should not crash the pod |
| 7 | nil client behavior — new metrics handler | Return 503 | Endpoint is unavailable, fail explicitly |
| 8 | nil client behavior — existing GetClusterTopology(include_tikv_usage=true) | No change — topology.go is left entirely untouched (Decision 3); `fetchTiKVUsage` keeps its own per-call TLS client; `s.tikvHTTPClient` is not referenced from topology.go | Nil-client concept does not apply to this path |
| 9 | End-to-end streaming (TiKV → CSNG → caller)? | Not required — `[]byte` in clustergateway is acceptable | All existing cluster-agent calls are buffered; streaming is out of scope |