# Component Resource Metrics API Design

## Overview

This document introduces a new Cluster Agent API to retrieve real-time CPU and Memory usage metrics for TiDB cluster components (TiKV, TiDB, PD, TiProxy, TiFlash, TSO, Scheduling, etc.), providing data support for Cluster Service's scale-down protection decisions.

## Motivation

To implement the scale-down protection feature (Step 2 of safety protection), Cluster Service needs to:
1. Obtain the current total CPU/Memory usage of components
2. Calculate whether resources after scaling down can meet current load demands
3. Prevent resource shortages caused by scaling down

## Design

### API Design

Add a new RPC method `GetComponentResourceMetrics`:

```protobuf
// GetComponentResourceMetrics returns CPU and Memory usage metrics for specified component(s)
rpc GetComponentResourceMetrics(GetComponentResourceMetricsReq) returns (GetComponentResourceMetricsResp) {
  option (google.api.http) = {get: "/clusters/{cluster_id}/components:resource-metrics"};
}
```

**Query Parameters:**
| Parameter | Required | Description |
|-----------|----------|-------------|
| `component_type` | **Required** | Filter by component type (e.g., "tikv", "tidb", "tiflash-compute") |
| `component_id` | **Required** | Filter by specific component group ID |

**Example:**
- `GET /clusters/{cluster_id}/components:resource-metrics?component_type=tikv&component_id=default` - Get metrics for specific TiKV group

### Request/Response

```protobuf
message GetComponentResourceMetricsReq {
  string cluster_id = 1 [
    (google.api.field_behavior) = REQUIRED,
    (buf.validate.field).required = true
  ];
  // Component type to filter metrics (e.g., "tikv", "tidb", "pd", "tiflash", "tso", "scheduling")
  string component_type = 2 [
    (google.api.field_behavior) = REQUIRED,
    (buf.validate.field).required = true
  ];
  // Component ID (group ID) to filter specific component group
  string component_id = 3 [
    (google.api.field_behavior) = REQUIRED,
    (buf.validate.field).required = true
  ];
}

message GetComponentResourceMetricsResp {
  repeated ComponentResourceMetrics components = 1;
  int64 timestamp_ms = 2; // Metrics collection timestamp
  // Indicates whether metrics-server is available and usage data is populated.
  // If false, CpuCoresUsed and MemoryBytesUsed in components will be 0.
  bool metrics_server_available = 3;
}

message ComponentResourceMetrics {
  string component_type = 1; // "tikv", "tidb", "pd", "tiflash", etc.
  string component_id = 2;   // Component group ID
  string pod_name = 3;       // Pod name
  string node_name = 4;      // K8s node name
  
  // CPU metrics
  double cpu_cores_total = 5;  // CPU limit/request in cores
  double cpu_cores_used = 6;   // Current CPU usage in cores
  
  // Memory metrics
  uint64 memory_bytes_total = 7; // Memory limit/request in bytes
  uint64 memory_bytes_used = 8;  // Current memory usage in bytes
}
```

### Data Collection Strategy

Use **Kubernetes Metrics API** to get Pod-level resource usage:

```go
// Query metrics.k8s.io API
metrics, err := metricsClient.MetricsV1beta1().PodMetricses(namespace).Get(ctx, podName, metav1.GetOptions{})

// Extract CPU/Memory usage
cpuUsage := metrics.Containers[0].Usage.Cpu()
memoryUsage := metrics.Containers[0].Usage.Memory()
```

**Pros:**
- Standard Kubernetes approach, no additional dependencies
- Accurate data, collected by kubelet
- Supports all components (TiKV/TiDB/PD/TiProxy/TiFlash/TSO/Scheduling/ResourceManager)

**Cons:**
- Requires metrics-server to be deployed
- metrics-server data retention is short (usually 15 minutes)

**TiDB Cloud Environment**: dataplane-manager automatically installs metrics-server when deploying K8s clusters (see `internal/dataplane-manager/helm/metrics-server/` and `pkg/controllers/appbox/task/common/metrics_server.go`)

If metrics-server is unavailable (e.g., still deploying), the API should return a clear error (such as `codes.Unavailable`) to let the caller know that metrics cannot be retrieved.

## Implementation Status

### Completed
- ✅ Proto definitions added to `api/tidbcloud/cluster-agent/v1/services.proto` and `resources.proto`
- ✅ RBAC permissions added to `deployments/templates/cluster_agent_role.templ` for `metrics.k8s.io`
- ✅ Service implementation created in `internal/cluster-agent/service/resource_metrics.go`
- ✅ Metrics client created in `internal/cluster-agent/metrics/client.go`
- ✅ Service initialization updated in `internal/cluster-agent/cmd/agent/grpc.go`
- ✅ Unit tests added in `internal/cluster-agent/service/resource_metrics_test.go`

### Implementation Details

#### 1. Proto Definition Update

**Files Modified:**
- `api/tidbcloud/cluster-agent/v1/services.proto` - Added RPC and request/response messages
- `api/tidbcloud/cluster-agent/v1/resources.proto` - Added `ComponentResourceMetrics` message

Run `make proto` to regenerate Go code (already done).

Generated Go files:
- `api/tidbcloud/cluster-agent/v1/services.pb.go`
- `api/tidbcloud/cluster-agent/v1/services_grpc.pb.go`
- `api/tidbcloud/cluster-agent/v1/services.pb.gw.go`
- `api/tidbcloud/cluster-agent/v1/resources.pb.go`

**Proto Definitions:**

File: `api/tidbcloud/cluster-agent/v1/services.proto`

```protobuf
rpc GetComponentResourceMetrics(GetComponentResourceMetricsReq) returns (GetComponentResourceMetricsResp) {
  option (google.api.http) = {get: "/clusters/{cluster_id}/components/{component_type}/{component_id}/resource-metrics"};
}

message GetComponentResourceMetricsReq {
  string cluster_id = 1 [
    (google.api.field_behavior) = REQUIRED,
    (buf.validate.field).required = true
  ];
  string component_type = 2 [
    (google.api.field_behavior) = REQUIRED,
    (buf.validate.field).required = true
  ];
  string component_id = 3 [
    (google.api.field_behavior) = REQUIRED,
    (buf.validate.field).required = true
  ];
}

message GetComponentResourceMetricsResp {
  repeated ComponentResourceMetrics components = 1;
  int64 timestamp_ms = 2;
  // Indicates whether metrics-server is available and usage data is populated.
  // If false, CpuCoresUsed and MemoryBytesUsed in components will be 0.
  bool metrics_server_available = 3;
}

message ComponentResourceMetrics {
  string component_type = 1;
  string component_id = 2;
  string pod_name = 3;
  string node_name = 4;
  double cpu_cores_total = 5;
  double cpu_cores_used = 6;
  uint64 memory_bytes_total = 7;
  uint64 memory_bytes_used = 8;
}
```

### 2. Cluster Agent Implementation

File: `internal/cluster-agent/service/resource_metrics.go` (new file)

```go
package service

import (
    "context"
    "fmt"
    "time"
    
    apiv1 "github.com/tidbcloud/cluster-service-ng/api/tidbcloud/cluster-agent/v1"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/labels"
    "k8s.io/metrics/pkg/apis/metrics/v1beta1"
    metricsclient "k8s.io/metrics/pkg/client/clientset/versioned"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

func (s *AgentService) GetComponentResourceMetrics(ctx context.Context, req *apiv1.GetComponentResourceMetricsReq) (*apiv1.GetComponentResourceMetricsResp, error) {
    ns := buildNamespace(req.ClusterId)
    
    // List pods for specified component type and ID
    podList, err := s.listComponentPods(ctx, ns, req.ComponentType, req.ComponentId)
    if err != nil {
        return nil, fmt.Errorf("failed to list pods: %w", err)
    }
    
    if len(podList.Items) == 0 {
        return nil, status.Errorf(codes.NotFound, "no pods found for component %s/%s", req.ComponentType, req.ComponentId)
    }
    
    // Get resource metrics from metrics API
    metricsList, err := s.fetchPodMetrics(ctx, ns, podList)
    if err != nil {
        return nil, fmt.Errorf("failed to fetch metrics: %w", err)
    }
    
    // Build result from pod metrics
    result := s.buildComponentMetrics(podList, metricsList, req.ComponentType, req.ComponentId)
    
    // Check if metrics server is available
    metricsServerAvailable := s.metricsCli != nil

    return &apiv1.GetComponentResourceMetricsResp{
        Components:             result,
        TimestampMs:            time.Now().UnixMilli(),
        MetricsServerAvailable: metricsServerAvailable,
    }, nil
}

func (s *AgentService) listComponentPods(ctx context.Context, namespace, componentType, componentId string) (*corev1.PodList, error) {
    // Use standard Kubernetes labels to select pods
    // Based on deployments/templates/*.templ, all components use:
    //   app.kubernetes.io/component: {{!ComponentType}}
    //   app.kubernetes.io/instance: {{!ComponentGroup}}
    labelSelector := labels.SelectorFromSet(labels.Set{
        "app.kubernetes.io/component": componentType,
        "app.kubernetes.io/instance":  componentId,
    })
    
    podList := &corev1.PodList{}
    err := s.dataplaneCli.List(ctx, podList, 
        client.InNamespace(namespace), 
        client.MatchingLabelsSelector{Selector: labelSelector})
    if err != nil {
        return nil, fmt.Errorf("failed to list pods: %w", err)
    }
    
    return podList, nil
}

func (s *AgentService) fetchPodMetrics(ctx context.Context, namespace string, pods *corev1.PodList) (*v1beta1.PodMetricsList, error) {
    metricsList := &v1beta1.PodMetricsList{}
    
    for _, pod := range pods.Items {
        podMetrics, err := s.metricsClient.MetricsV1beta1().PodMetricses(namespace).Get(ctx, pod.Name, metav1.GetOptions{})
        if err != nil {
            // If metrics server returns error, check if it's unavailable
            if apierrors.IsServiceUnavailable(err) || apierrors.IsNotFound(err) {
                return nil, status.Errorf(codes.Unavailable, "metrics-server unavailable: %v", err)
            }
            return nil, fmt.Errorf("failed to get metrics for pod %s: %w", pod.Name, err)
        }
        metricsList.Items = append(metricsList.Items, *podMetrics)
    }
    
    return metricsList, nil
}

func (s *AgentService) buildComponentMetrics(pods *corev1.PodList, metricsList *v1beta1.PodMetricsList, componentType, componentId string) []*apiv1.ComponentResourceMetrics {
    // Build map of pod name to metrics
    metricsMap := make(map[string]*v1beta1.PodMetrics)
    for i := range metricsList.Items {
        metricsMap[metricsList.Items[i].Name] = &metricsList.Items[i]
    }
    
    result := make([]*apiv1.ComponentResourceMetrics, 0, len(pods.Items))
    
    for _, pod := range pods.Items {
        podMetrics, ok := metricsMap[pod.Name]
        if !ok {
            continue // Skip pods without metrics
        }
        
        // Get resource limits/requests from pod spec
        cpuTotal := getPodCPULimit(&pod)
        memoryTotal := getPodMemoryLimit(&pod)
        
        // Get actual usage from metrics
        cpuUsed := getPodCPUUsage(podMetrics)
        memoryUsed := getPodMemoryUsage(podMetrics)
        
        cpuUsagePercentage := float64(0)
        if cpuTotal > 0 {
            cpuUsagePercentage = (cpuUsed / cpuTotal) * 100
        }
        
        memoryUsagePercentage := float64(0)
        if memoryTotal > 0 {
            memoryUsagePercentage = (float64(memoryUsed) / float64(memoryTotal)) * 100
        }
        
        result = append(result, &apiv1.ComponentResourceMetrics{
            ComponentType:         componentType,
            ComponentId:           componentId,
            PodName:               pod.Name,
            NodeName:              pod.Spec.NodeName,
            CpuCoresTotal:         cpuTotal,
            CpuCoresUsed:          cpuUsed,
            CpuUsagePercentage:    cpuUsagePercentage,
            MemoryBytesTotal:      memoryTotal,
            MemoryBytesUsed:       memoryUsed,
            MemoryUsagePercentage: memoryUsagePercentage,
            RestartCount:          getPodRestartCount(&pod),
        })
    }
    
    return result
}

func getPodCPULimit(pod *corev1.Pod) float64 {
    // Sum CPU limits from all containers
    // If no limit set, use request as fallback
    totalMilliCPU := int64(0)
    for _, container := range pod.Spec.Containers {
        if limit := container.Resources.Limits.Cpu(); limit != nil {
            totalMilliCPU += limit.MilliValue()
        } else if request := container.Resources.Requests.Cpu(); request != nil {
            totalMilliCPU += request.MilliValue()
        }
    }
    return float64(totalMilliCPU) / 1000.0
}

func getPodMemoryLimit(pod *corev1.Pod) uint64 {
    // Sum memory limits from all containers
    // If no limit set, use request as fallback
    totalBytes := int64(0)
    for _, container := range pod.Spec.Containers {
        if limit := container.Resources.Limits.Memory(); limit != nil {
            totalBytes += limit.Value()
        } else if request := container.Resources.Requests.Memory(); request != nil {
            totalBytes += request.Value()
        }
    }
    return uint64(totalBytes)
}

func getPodCPUUsage(podMetrics *v1beta1.PodMetrics) float64 {
    // Sum CPU usage from all containers (convert nanocores to cores)
    totalNanoCPU := int64(0)
    for _, container := range podMetrics.Containers {
        totalNanoCPU += container.Usage.Cpu().Value()
    }
    return float64(totalNanoCPU) / 1e9
}

func getPodMemoryUsage(podMetrics *v1beta1.PodMetrics) uint64 {
    // Sum memory usage from all containers
    totalBytes := int64(0)
    for _, container := range podMetrics.Containers {
        totalBytes += container.Usage.Memory().Value()
    }
    return uint64(totalBytes)
}

func getPodRestartCount(pod *corev1.Pod) uint32 {
    count := int32(0)
    for _, containerStatus := range pod.Status.ContainerStatuses {
        count += containerStatus.RestartCount
    }
    return uint32(count)
}
```

### 3. Dependencies

Add `k8s.io/metrics` client dependency to `internal/cluster-agent/go.mod`:

```go
require (
    k8s.io/metrics v0.31.0
)
```

### 4. Initialization

Create the metrics client during initialization in `internal/cluster-agent/service/service.go`:

```go
// Create metrics client
metricsConfig := rest.CopyConfig(k8sConfig)
metricsClient, err := metricsclient.NewForConfig(metricsConfig)
if err != nil {
    return fmt.Errorf("failed to create metrics client: %w", err)
}

agentService := &AgentService{
    dataplaneCli:  dataplaneClient,
    metricsClient: metricsClient,
    // ... other fields
}
```

## Testing

### Unit Tests

```go
func TestGetComponentResourceMetrics(t *testing.T) {
    // Mock metrics client
    // Test various scenarios:
    // - Normal case: return metrics for multiple pods
    // - Metrics server unavailable: return codes.Unavailable
    // - No pods found: return codes.NotFound
    // - Single pod component (like TiFlash)
}
```

## Supported Components

Based on `deployments/templates/*.templ`, supports all components in SharedPool:

| Component Type | Component ID Example | Template File |
|----------------|----------------------|---------------|
| tikv | default | tikv_group.templ |
| tidb | default | tidb_group.templ |
| pd | default | pd_group.templ |
| tiflash-compute | default | tiflash_group.templ |
| tiflash-write | default | tiflash_group.templ |
| tiproxy | default | tiproxy_group.templ |
| tso | default | tso_group.templ |
| scheduling | default | scheduling_group.templ |
| resource-manager | default | resource_manager_group.templ |
| tidb-worker | default | tidb_worker_group.templ |
| tikv-worker | default | tikv_worker_group.templ |
| coprocessor-worker | default | tikv_worker_group.templ |

### Component Label Mapping

The API uses `GetComponentTypeLabel()` and `GetComponentInstanceLabel()` functions to accept both CSNG component names and Kubernetes label names, then map them to the labels used by TiDB Operator pods.

**Component Type Label** (`app.kubernetes.io/component`):

| Input Type | Label Value |
|------------|-------------|
| `tikv`, `tidb`, `pd`, `tiproxy`, `resource-manager` | Same as input (lowercase) |
| `TIFLASH_COMPUTE` / `tiflash-compute` | `compute-tiflash` |
| `TIFLASH_WRITE` / `tiflash-write` | `write-tiflash` |
| `TIDB_WORKER` / `tidb-worker` | `worker-tidb` |
| `TIKV_WORKER` / `default-tikv-worker` / `tikv-worker` | `tikv-worker` |
| `COPROCESSOR_WORKER` / `coprocessor-tikv-worker` / `coprocessor-worker` | `coprocessor-worker` |
| `SCHEMA_MANAGER` / `schema-manager-tikv-worker` / `schema-manager` | `schema-manager` |
| `PD_TSO` / `pd-tso` | `tso` |
| `PD_SCHEDULING` / `pd-scheduling` | `scheduling` |

**Component Instance Label** (`app.kubernetes.io/instance`):

| Component Type | Label Pattern |
|----------------|---------------|
| `tikv`, `tidb`, `pd`, `tiproxy`, `tso`, `scheduling`, etc. | `db-{component_id}` |
| `tiflash-compute` | `db-{component_id}-compute` |
| `tiflash-write` | `db-{component_id}-write` |
| `tidb-worker` | `db-{component_id}-worker` |
| `tikv-worker`, `coprocessor-worker`, `schema-manager` | `db-{component_id}` |

## API Usage Example

### Example 1: Get metrics for specific TiKV group

**Request:**
```
GET /clusters/db-xxx/components:resource-metrics?component_type=tikv&component_id=default
```

**Response:**
```json
{
  "components": [
    {
      "component_type": "tikv",
      "component_id": "default",
      "pod_name": "db-tikv-0",
      "node_name": "ip-10-0-1-100",
      "cpu_cores_total": 8.0,
      "cpu_cores_used": 3.5,
      "memory_bytes_total": 17179869184,
      "memory_bytes_used": 8589934592
    },
    {
      "component_type": "tikv",
      "component_id": "default",
      "pod_name": "db-tikv-1",
      "node_name": "ip-10-0-1-101",
      "cpu_cores_total": 8.0,
      "cpu_cores_used": 4.2,
      "memory_bytes_total": 17179869184,
      "memory_bytes_used": 9663676416
    }
  ],
  "timestamp_ms": 1710830400000,
  "metrics_server_available": true
}
```

### Example 2: Get metrics for all TiKV groups

**Request:**
```
GET /clusters/db-xxx/components:resource-metrics?component_type=tikv
```

### Example 3: Get metrics for all components in cluster

**Request:**
```
GET /clusters/db-xxx/components:resource-metrics
```

## RBAC Configuration

To access the Kubernetes Metrics API, cluster-agent requires additional RBAC permissions. The following permissions have been added to `deployments/templates/cluster_agent_role.templ`:

```yaml
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  verbs:
  - get
  - list
```

This allows cluster-agent to:
- Query Pod resource usage via `metrics.k8s.io/v1beta1`
- Get CPU and Memory metrics for all component pods

## References

- [Kubernetes Metrics API](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/)
- [metrics-server](https://github.com/kubernetes-sigs/metrics-server)