# Scale Protection by Realtime Load Design Document ## Overview This document describes the design for protecting TiDB cluster scaling operations based on realtime component load metrics. ## Architecture ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ Cluster Service (CSNG) │ │ │ │ ┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────┐ │ │ │ Scale API │ |Scale Protection │ │ Metrics │ │ │ │ │───▶│ Controller │◄───│ Aggregator │ │ │ └─────────────────────┘ └─────────────────────┘ └─────────────────┘ │ │ │ │ ▲ │ │ │ │ │ │ │ ▼ ▼ │ │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ Scale Decision Engine │ │ │ │ (Combines RU metrics + Load metrics + Protection rules) │ │ │ └─────────────────────────────────────────────────────────────────────┘ │ │ │ │ └─────────────────────────────────────┼────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ Cluster-Agent (per cluster) │ │ │ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ │ Metrics Client │ │ Resource Metrics │ │ │ │ (metrics-server) │◄───│ API │ │ │ │ │ │ (/components: │ │ │ └─────────────────────┘ │ resource-metrics)│ │ │ └─────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` ## API Design ### Resource Metrics API (Cluster-Agent) ``` GET /clusters/{cluster_id}/components:resource-metrics?component_type={type}&component_id={id} Response: { "components": [ { "componentType": "tikv", "componentId": "c9baaa3c", "podName": "db-c9baaa3c-tikv-0", "nodeName": "ip-10-0-1-10.ec2.internal", "cpuCoresTotal": 7, "cpuCoresUsed": 0.081, "memoryBytesTotal": "16106127360", "memoryBytesUsed": "1050196992" } ], "timestampMs": "1742341200000", "metricsServerAvailable": true } ``` ## Load Calculation ### Current Total Usage ``` current_cpu_used = sum(components[].cpuCoresUsed) current_memory_used = sum(components[].memoryBytesUsed) ``` `memoryBytesUsed` comes from the Kubernetes Metrics API and represents the container working set. For TiKV this includes RocksDB block cache, which is reclaimable and partly disappears when TiKV pods are removed. ### TiKV Horizontal Scale-In For a same-spec TiKV count reduction to a non-zero target, Cluster Service queries each current TiKV pod through the existing cluster-agent raw metrics proxy. A single TiKV `/metrics` response provides both `process_resident_memory_bytes` and `tikv_engine_block_cache_size_bytes`. The Metrics Server working set may be older than the TiKV scrape. To avoid subtracting a newer cache value from an older, smaller memory sample, each pod's calculation uses: ``` pod_memory_baseline = max(metrics_server_working_set, same_scrape_resident_memory) pod_reclaimable_cache = min(same_scrape_block_cache, same_scrape_resident_memory) ``` Modern TiKV reports a shared block cache as `cf="all"`. For older TiKV versions without that sample, Cluster Service sums the independent `db="kv"` per-CF cache values. The scale operation does not identify the exact pods that will be deleted. Therefore, the safety calculation subtracts only the smallest `current_count - target_count` per-pod cache values: ``` removed_count = current_count - target_count reclaimable_block_cache = sum(smallest(pod_reclaimable_cache, removed_count)) memory_demand = sum(pod_memory_baseline) - reclaimable_block_cache ``` This keeps all non-cache memory in the projected demand and retains the cache of every possible surviving pod. Missing pod metrics, an unavailable raw metrics proxy, or malformed Prometheus data uses the Metrics Server working set for that pod and contributes zero reclaimable cache, preserving the previous fail-closed behavior. The cache adjustment does not apply to: - CPU protection - TiKV scale-to-zero - TiKV node-spec changes - Other components ### Target Capacity Using `effectiveCPU` and `effectiveMem` helpers for consistent capacity calculation: ``` // CPU: reserve 1 core per node (minimum 0) effective_cpu(spec, count) = max(spec.cpu - 1, 0) * count // Memory: use ComputeAvailableMemory (accounts for system reservation) effective_mem(spec, count) = ComputeAvailableMemory(spec.memory_gi * 1024) * 1024 * 1024 * count ``` ### Protection Logic ```go func checkScaleInSafety(components []*apiv1.ComponentResourceMetrics, currentNodeSpec, targetNodeSpec *entity.NodeSpec, currentNodeCount, targetNodeCount int32, force bool) error { if force { return nil // Skip protection when force=true } // Use effective capacity (after system reservation) for both trigger and enforcement currentCpuCapacity := effectiveCPU(currentNodeSpec, currentNodeCount) currentMemCapacity := effectiveMem(currentNodeSpec, currentNodeCount) targetCpuCapacity := effectiveCPU(targetNodeSpec, targetNodeCount) targetMemCapacity := effectiveMem(targetNodeSpec, targetNodeCount) // Skip if target capacity is not decreased (handles count-down + spec-up correctly) if targetCpuCapacity >= currentCpuCapacity && targetMemCapacity >= currentMemCapacity { return nil } // Calculate current total usage var currentCpuUsed, currentMemUsed float64 for _, comp := range components { currentCpuUsed += comp.GetCpuCoresUsed() currentMemUsed += float64(comp.GetMemoryBytesUsed()) } memoryDemand := currentMemUsed if isSameSpecTiKVHorizontalScaleIn() { rawMemory := fetchSameScrapeResidentMemoryAndBlockCache() memoryBaseline := sum(max(metricsServerWorkingSet, rawMemory.resident)) memoryDemand = memoryBaseline - guaranteedReclaimableBlockCache(rawMemory) } // Enforce protection if currentCpuUsed > targetCpuCapacity || memoryDemand > targetMemCapacity { return fmt.Errorf("scale blocked: current load exceeds target capacity") } return nil } ``` ## Example **Scale TiDB from 2 nodes to 1 node** Current state (2 TiDB): - TiDB-0: 1.5 cores used / 4 cores - TiDB-1: 1.2 cores used / 4 cores - Total CPU used: 2.7 cores - Total Memory used: 0.9 GB Target state (1 TiDB): - Node spec: 4 cores, 8 GB memory - Target CPU capacity: 4 × 1 = 4 cores - Target Memory capacity: 8 × 1 = 8 GB Decision: - 2.7 cores (used) < 4 cores (capacity) - 0.9 GB (used) < 8 GB (capacity) - Result: **ALLOW scale-in** If current usage was higher: - Total CPU used: 5 cores - 5 cores > 4 cores (capacity) - Result: **BLOCK scale-in** ## RU Validation Before applying scale protection, the system validates that the requested node specification can support the cluster's **minimum RU** (Request Unit) requirement. ### Why Use Min RU? The validation uses `minRU` to ensure the selected node specification can handle the cluster's baseline capacity requirement: ```go // Get the required node spec that can support min RU requiredNodeSpec, err := ruConfig.GetNodeSpec(componentType, minRU) // Validate requested spec meets requirement if requestedNodeSpec.GetCpu() < requiredNodeSpec.GetCpu() || requestedNodeSpec.GetMemoryGi() < requiredNodeSpec.GetMemoryGi() { return fmt.Errorf("node spec cannot support min RU %d", minRU) } ``` ### Example **RU Config:** ```yaml tidb: 4c8g: min: 0 max: 50000 8c32g: min: 50000 max: 200000 ``` **Scenario:** - Cluster min RU = 50000 - User requests 8c16g (custom spec) - Validation: GetNodeSpec(TIDB, 50000) → requires 8c32g minimum - 8c16g < 8c32g → **REJECT** (cannot support 50000 RU) This prevents users from selecting undersized node specifications that cannot handle the cluster's minimum RU load. ## Affected APIs **Trigger Condition**: Scale in/down operations (reduce node count or downgrade node spec), when target capacity < current capacity. ### 1. ScaleSharedPool (`POST /sharedPools/{shared_pool_id}:scale`) Add load protection check in `applyNodeQuotaChanges` function (single-check with resolved target spec): ```go func (s *serverImpl) applyNodeQuotaChanges(ctx context.Context, sharedPool *entity.SharedPool, scaleComponent, currComp *entity.Component, nodeSpecMap map[string][]*entity.NodeSpec, optionsMap map[string][]string, force bool) error { // Save original values before any mutation for scale-in safety check origNodeCount := currComp.GetNodeQuota().GetNodeCount() origNodeSpec := currComp.GetNodeQuota().GetNodeSpec() targetNodeCount := scaleComponent.GetNodeQuota().GetNodeCount() // Step 1-3: Deterministic validations and resolve target spec // ... validation logic ... targetNodeSpec := resolvedSpec // from getNodeSpec() // Step 4: Run checkScaleInSafety exactly once with resolved target spec // This prevents false rejection on count-down + spec-up (e.g., 3×4c → 2×16c) if err := s.checkScaleInSafety(ctx, sharedPool, currComp, origNodeCount, origNodeSpec, targetNodeCount, targetNodeSpec, force); err != nil { return err } // Step 5: Mutate state // ... apply changes ... return nil } ``` ### 2. UpdateTiDBGroup (`PATCH /clusters/{cluster_id}/tidbGroups/{tidb_group_id}`) Add load protection in `updateTiDBGroupNodeQuota` (single-check with resolved target spec): ```go func (s *serverImpl) updateTiDBGroupNodeQuota(ctx context.Context, tidbGroup, currTiDBGroup *entity.TiDBGroup, sharedPool *entity.SharedPool, currTiDBComp *entity.Component, optionsMap map[string][]string, force bool) error { // Save original values before any mutation origNodeCount := currTiDBComp.GetNodeQuota().GetNodeCount() origNodeSpec := currTiDBComp.GetNodeQuota().GetNodeSpec() // Determine target node count (0 means "not provided" in proto) targetNodeCount := tidbGroup.GetNodeQuota().GetNodeCount() if targetNodeCount == 0 { targetNodeCount = origNodeCount } // Step 1-3: Deterministic validations and resolve target spec // ... validation and spec resolution ... targetNodeSpec := resolvedSpec // from getNodeSpec() // Step 4: Run checkScaleInSafety exactly once with resolved target spec if err := s.checkScaleInSafety(ctx, sharedPool, currTiDBComp, origNodeCount, origNodeSpec, targetNodeCount, targetNodeSpec, force); err != nil { return err } // Step 5: Mutate state // ... apply changes ... return nil } ``` Both entry points share one implementation. See [Protection Logic](#protection-logic) under Load Calculation for the authoritative pseudo-code, including the TiKV block cache adjustment. #### Special Cases **Scale to 0**: When scaling any component (including TiFlash) to 0 nodes, target capacity becomes 0. Any residual load (> 0) would trigger the block. To scale to 0, users must explicitly set `force=true` to bypass the protection. This ensures the scale-down action is intentional and acknowledged. ## Metrics ``` # Counter scale_protection_blocks_total{component_type, reason} # Gauge current_cpu_used{component_type, component_id} current_memory_used{component_type, component_id} target_cpu_capacity{component_type, component_id} target_memory_capacity{component_type, component_id} ```