# Coprocessor Worker HPA Evaluation

Component-specific evaluation methodology and report page design for TiKV
remote coprocessor worker HPA scaling.

## Component Identity

| Attribute | Value |
|-----------|-------|
| HPA name pattern | `horizontalpodautoscaler=~".*-coprocessor"` |
| Namespace pattern | `tidb{TIDB_CLUSTER_ID}` (e.g., `tidb2013516099403907072`) |
| Pod regex | `pod=~".*coprocessor.*"` |
| Target component | TiKV remote coprocessor worker pods |

## HPA Configuration Context

The coprocessor worker HPA uses **dual metrics** (from `copr_worker_hpa.templ`):

```yaml
metrics:
- type: Object          # PRIMARY DRIVER
  object:
    metric:
      name: ext_cpu_scaled_ratio    # Custom metric: returns 0, 1, or 100
    target:
      type: Value
      value: 1
- type: Resource        # SECONDARY
  resource:
    name: cpu
    target:
      type: Utilization
      averageUtilization: 55
```

HPA uses **whichever metric suggests more replicas** (max of the two).

### HPA Behavior Configuration

```yaml
behavior:
  scaleDown:
    stabilizationWindowSeconds: 180
    policies:
    - type: Percent
      value: 30
      periodSeconds: 180
    - type: Pods
      value: 5
      periodSeconds: 180
    selectPolicy: Min  # Most conservative
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
    - type: Percent
      value: 100
      periodSeconds: 15
    selectPolicy: Max  # Most aggressive
```

## K8s Infrastructure Metrics

Query these using the K8s infra O11Y UUID (obtained via `applicationID=k8s-infra`).

Set `NAMESPACE="tidb${TIDB_CLUSTER_ID}"`.

### HPA Current Replicas

```promql
kube_horizontalpodautoscaler_status_current_replicas{
  namespace="${NAMESPACE}",
  horizontalpodautoscaler=~".*-coprocessor"
}
```

Purpose: Actual running pod count.

### HPA Desired Replicas

```promql
kube_horizontalpodautoscaler_status_desired_replicas{
  namespace="${NAMESPACE}",
  horizontalpodautoscaler=~".*-coprocessor"
}
```

Purpose: HPA's calculated target pods. Compare with current to detect scaling direction.

### HPA Observed CPU Utilization

```promql
kube_horizontalpodautoscaler_status_target_metric{
  namespace="${NAMESPACE}",
  horizontalpodautoscaler=~".*-coprocessor",
  metric_name="cpu",
  metric_target_type="utilization"
}
```

Purpose: HPA's CURRENT OBSERVED CPU utilization percentage (from
`Status.CurrentMetrics[].Resource.Current`, NOT the target threshold).

**CRITICAL**: The label `metric_target_type="utilization"` is MANDATORY. Without it,
the query may match Object metric series (like `ext_cpu_scaled_ratio`) instead of
the native Resource CPU metric. This was the root cause of a major evaluation error
in the V1 methodology.

### HPA CPU Target Threshold

```promql
kube_horizontalpodautoscaler_spec_target_metric{
  namespace="${NAMESPACE}",
  horizontalpodautoscaler=~".*-coprocessor",
  metric_name="cpu",
  metric_target_type="utilization"
}
```

Purpose: Configured CPU target (typically 55). Used to calculate:
- Scale-up zone: `target * 1.1` (e.g., 60.5%)
- Scale-down zone: `target * 0.9` (e.g., 49.5%)

### CPU Throttling

```promql
rate(container_cpu_cfs_throttled_periods_total{
  namespace="${NAMESPACE}",
  pod=~".*coprocessor.*"
}[5m])
/
rate(container_cpu_cfs_periods_total{
  namespace="${NAMESPACE}",
  pod=~".*coprocessor.*"
}[5m]) * 100
```

Purpose: Resource starvation indicator. High throttling suggests CPU limits are too tight.

## Application Metrics (TiKV)

Query these using the TiDB cluster O11Y UUID (obtained via `applicationID=${TIDB_CLUSTER_ID}`).

### P99 Latency

```promql
histogram_quantile(0.99,
  sum(rate(tikv_worker_remote_cop_process_duration_seconds_bucket{
    tidb_cluster_id="${TIDB_CLUSTER_ID}"
  }[5m])) by (le)
)
```

Purpose: Primary user-impact metric. This is the end-to-end processing time for
remote coprocessor requests.

### Request Rate

```promql
sum(rate(tikv_worker_remote_cop_request_duration_seconds_count{
  tidb_cluster_id="${TIDB_CLUSTER_ID}"
}[5m]))
```

Purpose: Workload demand indicator (requests per second).

### Concurrent Processing Requests

```promql
sum(tikv_worker_remote_cop_processing_requests_counter{
  tidb_cluster_id="${TIDB_CLUSTER_ID}"
})
```

Purpose: Current in-flight load.

### Queue Depth (Waiting Requests)

```promql
sum(tikv_worker_limiter_waiting_requests_counter{
  tidb_cluster_id="${TIDB_CLUSTER_ID}",
  type="coprocessor"
})
```

Purpose: Backpressure indicator. Non-zero = requests are queuing.

### Error Rate

```promql
sum(rate(tikv_worker_remote_cop_failed_requests_counter{
  tidb_cluster_id="${TIDB_CLUSTER_ID}"
}[5m])) by (type)
```

Purpose: Failure impact.

### Memory Usage

```promql
sum(tikv_worker_memory_limiter_current_used{
  tidb_cluster_id="${TIDB_CLUSTER_ID}"
})
```

Purpose: Resource usage tracking.

## Application Metrics (TiDB)

Query these using the same TiDB cluster O11Y UUID as the TiKV metrics above.

### TiDB Query P99 Latency

```promql
histogram_quantile(0.99,
  sum(rate(tidb_server_handle_query_duration_seconds_bucket{
    tidb_cluster_id="${TIDB_CLUSTER_ID}"
  }[5m])) by (le)
)
```

Purpose: TiDB server-side query handling latency. Plotted as the secondary (right Y-axis)
latency line in the Workload Timeline chart. This measures end-to-end query time at the
TiDB layer, complementing the coprocessor-specific P99 on the left Y-axis.

**Unit**: The metric returns seconds. Convert to milliseconds (`* 1000`) when populating
`workloadTimeline[].tidbP99Latency`, since the chart axis is labeled "TiDB P99 (ms)".

**Cache file**: `/tmp/tidb_p99_15s.json` (queried at `step=15s`, same as other raw metrics).

## Scaling Event Detection

### Two Separate Concerns (CRITICAL DISTINCTION)

Event detection and lag measurement use DIFFERENT metrics for DIFFERENT purposes:

| Concern | Metric | Purpose |
|---------|--------|---------|
| **Event detection** | `desired_replicas` value transitions | Identify WHEN and WHAT scaling decisions HPA made |
| **Scaling lag** | `kube_pod_status_ready` vs `desired_replicas` | Measure HOW LONG until pods actually become ready |

**DO NOT conflate these.** Events come from `desired_replicas` changes.
Lag comes from comparing `desired_replicas` against `kube_pod_status_ready`.

### Method: desired_replicas Value Transitions

A scaling event occurs when `desired_replicas` changes value between consecutive
scrape points. This captures the exact moment HPA makes a scaling decision.

**Direction** is determined by comparing the new value to the previous value:

| Condition | Direction |
|-----------|-----------|
| `desired[t] > desired[t-1]` | Scale-OUT (UP) |
| `desired[t] < desired[t-1]` | Scale-IN (DOWN) |

**Why desired_replicas (not current_replicas)?**
- `desired_replicas` reflects HPA's INTENT at decision time
- `current_replicas` lags behind (pods take time to start/terminate)
- Using `current_replicas` produces 1-2 extra events due to intermediate steps
  (e.g., a 3->5 scale-up may show current as 3->4->5 = 2 transitions vs 1 desired change)

### Why NOT delta()

Using `delta(desired_replicas[5m]) > 0` is LOSSY:
- A replica change `3->2->3` in one window shows `delta=0`, missing both events
- Phantom events appear when delta crosses 0 in a window with no actual change

Point-by-point value comparison captures every HPA decision.

### Event Extraction Logic

```bash
# Extract scaling events from desired_replicas value changes
jq -r '
  .data.result[0].values as $v |
  [range(1; $v | length)] |
  map(select($v[.][1] != $v[. - 1][1])) |
  map({
    time: ($v[.][0] | strftime("%Y-%m-%d %H:%M:%S")),
    epoch: $v[.][0],
    from: ($v[. - 1][1] | tonumber),
    to: ($v[.][1] | tonumber),
    direction: (if ($v[.][1] | tonumber) > ($v[. - 1][1] | tonumber) then "UP" else "DOWN" end)
  })
' /tmp/desired_replicas_15s.json > /tmp/scaling_events.json
```

### Pod Ready Count (for Lag Measurement)

Query pod readiness to measure how long scaling takes to complete:

```promql
sum(kube_pod_status_ready{
  namespace="${NAMESPACE}",
  pod=~".*coprocessor.*",
  condition="true"
})
```

**Lag calculation**: For each scaling event at time T:
1. Read `desired_replicas` value at T (the target)
2. Scan `kube_pod_status_ready` forward from T
3. Lag = first timestamp where `pod_ready >= desired_replicas` minus T
4. If pod_ready never reaches desired within 120s, lag = timeout value

### Thrashing Detection (Demand-Normalized)

**Definition**: An unjustified reversal -- a scaling event in the opposite direction of the
previous event, where the **real workload demand** never entered the zone that justified it.

**Why demand-normalized**: Raw CPU utilization is affected by scaling itself. When HPA scales
UP (2->3 pods), CPU drops because the same work spreads across more pods. If we check raw
CPU, it always appears to justify the subsequent DOWN. We must reconstruct the original
workload demand to remove this confounding:

```
demand_cpu = observed_cpu * current_replicas / pre_scaling_replicas
```

**Algorithm**:
```
For each consecutive opposite-direction event pair [i-1, i]:
  1. orig_replicas = replicas BEFORE event[i-1] changed them (events[i-1].from)
  2. For each 15s CPU sample between the two events:
     demand_cpu = observed_cpu * current_replicas / orig_replicas
  3. Check: did demand_cpu ever enter the zone that justifies event[i]?
     - If event[i] is DOWN: did demand_cpu < scale_down_threshold (49.5%)?
     - If event[i] is UP: did demand_cpu > scale_up_threshold (60.5%)?
  4. If demand NEVER entered the justifying zone (max continuous stretch = 0s):
     -> UNJUSTIFIED REVERSAL (thrashing)
```

**Rate**: `unjustified_rate = unjustified_reversals / total_opposite_direction_pairs * 100`

### Trigger Validation

For each scaling event, validate the trigger by checking CPU at event time:
- Scale-UP valid if: CPU > `spec_target * 1.1` (e.g., > 60.5% for target 55%)
- Scale-DOWN valid if: CPU < `spec_target * 0.9` (e.g., < 49.5% for target 55%)

## Metric Transparency (MANDATORY)

Every evaluation report MUST include a "Metrics Used" section documenting:

1. Which metric was queried to measure CPU utilization
2. Which metric actually drives HPA scaling decisions
3. Whether these are the same metric -- and if not, the implications

### Mandatory Report Section Template

```markdown
## Metrics Used for Evaluation

### CPU Utilization Source
- **Metric queried**:
  kube_horizontalpodautoscaler_status_target_metric{
      metric_name="cpu",
      metric_target_type="utilization",
      horizontalpodautoscaler=~".*-coprocessor"
  }
- **What it measures**: HPA's CURRENT OBSERVED CPU utilization percentage
  (from Status.CurrentMetrics[].Resource.Current, NOT the target threshold).
  The label metric_target_type="utilization" selects the native Resource CPU
  metric specifically, filtering out any Object or Pods metric series.
- **Limitation**: This metric only reflects the native Resource CPU metric.
  The ext_cpu_scaled_ratio custom metric (which is the primary scaling driver)
  is NOT stored in O11Y and cannot be queried retrospectively.

### Scale-Out / Scale-In Event Detection
- **Event source metric**: `kube_horizontalpodautoscaler_status_desired_replicas`
- **Method**: Detect VALUE TRANSITIONS in desired_replicas between consecutive 15s
  scrape points. When desired_replicas[t] != desired_replicas[t-1], that is a
  scaling event. Direction = UP if new > old, DOWN if new < old.
- **Why desired_replicas (not current_replicas)**: desired reflects HPA intent at
  decision time. current_replicas lags behind and can produce spurious intermediate
  transitions (e.g., 3->5 shows as 3->4->5 = 2 events instead of 1).
- **Why NOT delta()**: delta(desired_replicas[5m]) > 0 is lossy --
  a replica change 3->2->3 in one window shows delta=0, missing both events.
- **Lag measurement metric**: `kube_pod_status_ready` (separate from event detection).
  Lag = time from desired_replicas change until pod_ready count reaches desired value.
- **Validation**: CPU utilization at event time compared against
  spec_target_metric * 1.1 (scale-up threshold) and spec_target_metric * 0.9
  (scale-down threshold).

### Known Evaluation Gaps
- ext_cpu_scaled_ratio values are computed live and not persisted to O11Y
- Transient CPU spikes (< scrape interval) may trigger ext_cpu_scaled_ratio
  but not appear in status_target_metric
- Unready pod handling in ext_cpu_scaled_ratio can inflate utilization ratios
  during pod churn, causing scale-ups that appear anomalous from kube-state-metrics
```

## Metrics NOT Available in O11Y

| Metric | What it does | Why unavailable |
|--------|-------------|-----------------|
| `ext_cpu_scaled_ratio` | Primary HPA scaling signal (0/1/100) | Computed live by autoscale-stub, not exported to Prometheus |
| `dxf_hscale_tikv_worker_hint` | DXF kernel scaling hint | Computed live, not exported |
| Per-event decision log | Why HPA chose to scale | No audit trail exists |

### ext_cpu_scaled_ratio Details

The `ext_cpu_scaled_ratio` custom metric (`pkg/metricsprovider/provider.go:426-578`):
1. Reads live pod CPU via K8s Metrics API (`container_cpu_usage_seconds_total`)
2. Computes `utilization = (sum_pod_cpu_usage * 100) / sum_cpu_requests`
3. Compares against target with 10% tolerance: `|1.0 - usageRatio| <= 0.1`
4. Handles unready/missing pods specially (missing pods = 0% on scale-up)
5. Returns `100` (scale-up), `1` (do-nothing), or `0` (scale-down)

## Common Issues and Recommended Fixes

| Issue | Indicator | Recommended Fix |
|-------|-----------|-----------------|
| Thrashing | >10 events/day | Increase `stabilizationWindowSeconds` to 600 |
| High baseline latency | P99 > 10s with low CPU | CPU is not the bottleneck; consider queue-based scaling |
| Reactive scaling | Latency spikes before scale-out | Add predictive scaling or lower CPU target |
| Over-provisioning | CPU avg << target, replicas stay high | Review `minReplicas` setting |
| CPU mismatch | Low CPU + high latency | Use custom metrics (request rate, queue depth) |

## Report Generation

For report UI setup, page layout, data schema, calculation logic, design tokens,
and step-by-step report generation instructions, see `references/report-ui.md`.

### Coprocessor-Specific Report Fields

When generating the `DATA` JSON object for `generate-report.sh`, use these coprocessor-specific values:

| Field | Value |
|-------|-------|
| `metadata.hpaTarget` | `"copr-worker"` (or actual HPA target name) |
| `metadata.componentLabel` | `"Coprocessor Worker Autoscaling Evaluation"` |
| `workloadTimeline[].coprP99Latency` | Copr P99 latency in ms (primary chart line) |
| `workloadTimeline[].tidbP99Latency` | TiDB P99 latency in ms (secondary chart line) |

See `references/report-ui.md` for the full data schema and all required fields.