---
name: autoscaler-dashboard
description: |
  Use when evaluating HPA autoscaling quality or generating scaling report dashboards.
  Covers scoring methodology, event detection algorithms, and report page generation
  for any HPA-managed component (coprocessor worker, TiFlash, TiDB, etc.).
  Trigger phrases: "evaluate autoscaling", "scaling report", "HPA score",
  "how is the HPA performing", "generate scaling dashboard", "scaling quality",
  "coprocessor scaling", "autoscaler evaluation".
  Do NOT use for general HPA configuration questions or Kubernetes autoscaling concepts.
---

# HPA Autoscaler Dashboard

Evaluate HPA scaling quality and generate report dashboards for TiDB Cloud components.

## How to Run an Evaluation

1. **Authenticate**: Get O11Y token via `o11y-metrics-api` skill
2. **Discover**: Resolve O11Y cluster UUIDs (application + K8s infra) via Global Server
3. **Query**: Fetch all metrics at `step=15s` for 24h using component-specific PromQL
4. **Process**: Extract statistics and detect scaling events with jq (do NOT load raw arrays into context)
5. **Score**: Apply the deterministic scoring rubric (below) to produce 5 category scores + overall
6. **Generate**: Run `generate-report.sh` to inject data into the HTML template (see below)
7. **Open**: View the generated HTML file in a browser -- no build step needed

Each component has its own evaluation methodology and PromQL queries in `components/`.
The shared report UI template and data schema are documented in `references/report-ui.md`.

## Skill File Map

```
autoscaler-dashboard/
  SKILL.md                          # THIS FILE: scoring rubric, query strategy, orientation
  assets/
    copr-worker-report-template.html  # Self-contained HTML report template (CDN Tailwind + Chart.js)
    copr-worker-report.html           # Example concrete report (reference implementation with sample data)
  generate-report.sh                # Injects JSON data into HTML template to produce a report
  components/
    copr-worker.md                  # Coprocessor worker: HPA config, PromQL queries, event detection
  references/
    report-ui.md                    # Shared: data schema, page layout, template placeholder docs
```

## Component Registry

| Component | File | HPA Target | Status |
|-----------|------|------------|--------|
| Coprocessor Worker | `components/copr-worker.md` | TiKV remote copr worker pods | Active |

To add a new component: create `components/{name}.md` with component identity,
PromQL queries, HPA config, and page design. Add a row to this table.

## Query Time Range and Resolution

### Step Size

Use `step=15s` for single-day evaluations. This matches the typical Prometheus scrape
interval and provides maximum fidelity for event detection.

| Step | Points/Day | When to Use |
|------|-----------|-------------|
| `15s` | ~5760 | Single-day evaluation (DEFAULT) |
| `60s` | ~1440 | Only if API returns errors on 15s |
| `300s` | ~288 | Multi-day trend analysis (7-30 days) |

### Handling Large Responses

At `step=15s`, each metric returns ~5760 data points (~150-200KB JSON).
With ~11 metrics, total payload is ~2MB. The O11Y API handles this fine.

**If the API truncates or errors** (some Prometheus-compatible APIs cap at 11000 points),
split the 24h range into 4 x 6h chunks and process sequentially:

```bash
# Split 24h into 4 x 6h ranges
for i in 0 1 2 3; do
  CHUNK_START=$((START_TIME + i * 6 * 3600))
  CHUNK_END=$((START_TIME + (i + 1) * 6 * 3600))
  curl -s "${BASE_URL}/query_range" \
    -G --data-urlencode "query=${PROMQL}" \
    --data-urlencode "start=${CHUNK_START}" \
    --data-urlencode "end=${CHUNK_END}" \
    --data-urlencode "step=15s" \
    -o "/tmp/metric_chunk_${i}.json"
done
# Merge chunks: concatenate .data.result[0].values arrays
jq -s '.[0].data.result[0].values = [.[].data.result[0].values[]] | .[0]' \
  /tmp/metric_chunk_*.json > /tmp/metric_full.json
```

### Processing Strategy (IMPORTANT)

Do NOT load raw 5760-point arrays into AI context. Process with jq/bash first:

```bash
# Extract statistics from raw metric data
jq -r '
  .data.result[0].values | map(.[1] | tonumber) |
  {
    count: length,
    min: min,
    max: max,
    avg: (add / length),
    p50: sort[length/2 | floor],
    p99: sort[(length * 0.99) | floor]
  }
' /tmp/metric.json

# Extract scaling events (only state changes, not raw time series)
jq -r '
  .data.result[0].values as $v |
  [range(1; $v | length)] |
  map(select($v[.][1] != $v[. - 1][1])) |
  map({
    time: ($v[.][0] | strftime("%Y-%m-%d %H:%M:%S")),
    from: $v[. - 1][1],
    to: $v[.][1]
  })
' /tmp/hpa_replicas.json
```

Feed only the **summarized statistics and detected events** into the AI for scoring
and report generation. Raw arrays stay in /tmp/ files for jq processing only.

## Scoring Rubric

All scores are **deterministic step functions** -- same inputs always produce the same
score. No manual overrides. Each category owns exactly one signal dimension to prevent
double-counting. All bracket thresholds are v0 heuristics subject to empirical tuning.

### Signal Ownership (No Double-Counting)

| Category | Weight | Owns This Signal | Does NOT Include |
|----------|--------|-----------------|------------------|
| Scale-Out | 25% | P99 during scale-up x trigger validity | thrashing, overall latency |
| Scale-In | 25% | Unjustified reversal rate (demand-normalized) | latency impact (owned by Workload) |
| Workload | 25% | Scaling harmlessness rate (per-event P99 delta) | scale-up-specific latency (owned by Scale-Out), overall avg P99 |
| Efficiency | 15% | Actionable right-sizing rate | event counts, overall CPU avg |
| Stability | 10% | Calm period ratio (CPR) | event counts, thrashing (owned by Scale-In) |

### Scale-Out Accuracy (25%)

**Input**: avg P99 latency during scale-up events (seconds), valid trigger percentage

**Formula**: `score = bracket(avg_p99_during_scaleup) * (valid_trigger_pct / 100)`

*v0 heuristic brackets -- subject to empirical tuning*

| avg_p99_during_scaleup | bracket score |
|------------------------|---------------|
| < 1s | 95 |
| 1s - 5s | 80 |
| 5s - 10s | 60 |
| 10s - 30s | 40 |
| > 30s | 20 |

The trigger validity multiplier: at 100% valid triggers, score is unchanged. At 50%
valid, score is halved. This captures how many scale-ups were justified.

### Scale-In Accuracy (25%)

**Input**: unjustified reversal rate (demand-normalized thrashing)

**Concept**: For each opposite-direction event pair, reconstruct the original workload
demand (`demand_cpu = observed_cpu * current_replicas / pre_scaling_replicas`).
If the demand never entered the zone that justified the reversal, it's unjustified --
the HPA scaled based on its own feedback loop, not real workload change.

**Formula**: `score = bracket(unjustified_rate)`

*v0 heuristic brackets -- subject to empirical tuning*

| unjustified rate | score |
|-----------------|-------|
| < 5% | 95 |
| < 15% | 75 |
| < 30% | 50 |
| < 50% | 30 |
| >= 50% | 15 |

### Workload Impact (25%)

**Input**: scaling harmlessness rate (percentage of events that did not degrade P99 latency)

**Concept**: Measures whether scaling events harm the workload. For each event,
compare P99 latency before vs after. If latency stays the same or improves, the
event is harmless. This isolates HPA's causal impact from baseline latency levels.

**Definitions**:
```
For each scaling event at time T:
  before(T)  = avg P99 in [T-120s, T-15s]     // pre-event baseline
  after(T)   = avg P99 in [T+60s, T+180s]      // post-event settled
  delta(T)   = after(T) - before(T)

  harmless   = events where |delta| <= 2s       // no meaningful change
  improved   = events where delta < -2s         // latency got better
  degraded   = events where delta > +2s         // latency got worse

harmlessness_rate = (harmless + improved) / total * 100
```

**Formula**: `score = bracket(harmlessness_rate)`

*v0 heuristic brackets -- subject to empirical tuning*

| harmlessness rate | score |
|-------------------|-------|
| >= 95% | 95 |
| >= 85% | 80 |
| >= 70% | 55 |
| >= 50% | 35 |
| < 50% | 15 |

### Efficiency (15%)

**Input**: actionable right-sizing rate (percentage of actionable samples where HPA
is correctly sized or within stabilization forgiveness)

**Concept**: Only score HPA on samples where it **had a choice**. Exclude samples at
minReplicas with low CPU -- that is low workload, not HPA inefficiency.

**Definitions**:
```
sd   = cpu_target * (1 - tolerance/100)     // scale-down threshold (e.g. 49.5%)
su   = cpu_target * (1 + tolerance/100)     // scale-up threshold (e.g. 60.5%)
stab = stabilizationWindowSeconds           // e.g. 180s

For each 15s sample:
  excluded    = (replicas == minReplicas) AND (cpu < sd)
  actionable  = NOT excluded

Among actionable samples:
  right_sized = (sd <= cpu <= su)
  forgiven    = (replicas > minReplicas AND cpu < sd)
                AND (within stab seconds after a scale-up event)
  over_prov   = (replicas > minReplicas AND cpu < sd) AND NOT forgiven
  under_prov  = (cpu > su)

actionable_efficiency = (right_sized + forgiven) / actionable * 100
```

**Safeguard**: if actionable < 5% of total samples, score defaults to 95
(insufficient data to evaluate).

**Formula**: `score = bracket(actionable_efficiency)`

*v0 heuristic brackets -- subject to empirical tuning*

| actionable efficiency | score |
|----------------------|-------|
| >= 90% | 95 |
| >= 75% | 80 |
| >= 60% | 55 |
| >= 40% | 35 |
| < 40% | 20 |

### Stability (10%)

**Input**: Calm Period Ratio (CPR) -- fraction of the day with no scaling activity

**Concept**: Raw event count penalizes HPA for correctly responding to volatile workload.
CPR measures what fraction of the day the autoscaler was at rest. This is time-based,
not event-count-based, and cleanly separates from Scale-In (which owns thrash pairs).

**Definitions**:
```
  bucket       = 5-minute window (288 per day)
  calm_bucket  = bucket with zero desired_replicas transitions
  CPR          = calm_buckets / total_buckets * 100
```

**Formula**: `score = bracket(CPR)`

*v0 heuristic brackets -- subject to empirical tuning*

| CPR | score |
|-----|-------|
| >= 90% | 95 |
| >= 80% | 75 |
| >= 65% | 50 |
| >= 50% | 30 |
| < 50% | 15 |

### Overall Score Calculation

```
overall = (scale_out * 0.25) + (scale_in * 0.25) + (workload * 0.25)
        + (efficiency * 0.15) + (stability * 0.10)
```

### Score Interpretation

| Range | Assessment | Action |
|-------|------------|--------|
| 90-100 | Excellent | No changes needed |
| 70-89 | Good | Minor tuning optional |
| 50-69 | Fair | Review configuration |
| 30-49 | Poor (WARNING) | Changes required |
| 0-29 | Critical | Immediate action needed |

### Score Display Policy

All bracket thresholds are **v0 heuristics** -- hand-crafted initial estimates, not
calibrated against multi-cluster empirical data. The report UI reflects this:

- **Overall Scaling Performance** (gauge rings): Shows "TBD" only, no numeric values
- **Scoring Breakdown**: Shows numeric scores alongside an orange "TBD" badge on EVERY
  category (not just categories with `status: "tbd"`)
- **Evaluation Criteria**: Shows numeric scores alongside TBD badges
- **Dashboard Header**: Shows numeric score alongside TBD badge

This communicates that scores are directional signals, not absolute grades. The TBD
badge will be removed per-category as bracket thresholds are validated with empirical
data from multiple clusters.

### Design Rationale (Why Each Metric Was Chosen)

**Scale-Out (25%) -- TBD**: Uses P99 latency during scale-up events. Marked TBD because
we cannot establish a causal relationship between CPU-driven scaling and P99 latency.
The P99 may be high regardless of scaling (CPU is not the bottleneck for this workload).
Needs a metric that proves scale-out decisions help users.

**Scale-In (25%) -- Unjustified Reversal Rate**: The key insight is that raw CPU
utilization is **confounded by scaling itself**. When HPA scales UP (2->3 pods), CPU
drops because the same work spreads across more pods. The subsequent scale-DOWN appears
justified by raw CPU, but the workload demand never actually decreased. We reconstruct
the original demand via `demand_cpu = observed_cpu * current_replicas / pre_scaling_replicas`.
If demand never entered the justifying zone, the reversal was unjustified -- the HPA
was chasing its own feedback loop.

**Workload Impact (25%) -- Scaling Harmlessness Rate**: Instead of using overall average
P99 (which conflates baseline latency with scaling impact), we measure per-event deltas.
For each event, compare P99 before vs after. This isolates the causal effect of scaling
on latency.

**Efficiency (15%) -- Actionable Right-Sizing Rate**: The naive approach (|avg_cpu - target|)
penalizes the HPA for low-workload periods when replicas are at minimum and CPU is low.
That's not the HPA's fault. We exclude those samples and only score the HPA on samples
where it had a choice. We also forgive over-provisioned samples within the stabilization
window after a scale-up.

**Stability (10%) -- Calm Period Ratio**: Raw event count penalizes the HPA for correctly
responding to volatile workload. CPR measures what fraction of the day the autoscaler
was at rest, giving credit for responsive-but-not-noisy behavior.

## Examples

### Example 1: Evaluate coprocessor worker scaling for a cluster

User says: "Evaluate the autoscaling quality for cluster 2013516099403907072"

Actions:
1. Authenticate with O11Y (`o11y-metrics-api` skill)
2. Resolve O11Y UUIDs (application + K8s infra) via Global Server
3. Query all metrics from `components/copr-worker.md` at `step=15s` for 24h
4. Process with jq: extract statistics, detect scaling events, compute thrashing
5. Apply scoring rubric (this file) to produce 5 category scores + overall
6. Create JSON data object matching the schema in `references/report-ui.md`
7. Run `generate-report.sh` to inject data into the HTML template

Result: A self-contained HTML report with timeline charts, event table, score breakdown,
and recommendations. Score: 72 (Good) with 3 thrashing events flagged.

### Example 2: Investigate high thrashing rate

User says: "The coprocessor worker is thrashing, can you evaluate it?"

Actions:
1. Same evaluation flow as Example 1
2. Focus on Scale-In Accuracy score and unjustified reversal rate
3. Check if `stabilizationWindowSeconds` is too low
4. Report shows demand-normalized analysis proving HPA is chasing its own feedback loop

Result: Report highlights 45% unjustified reversal rate. Recommendation:
increase `stabilizationWindowSeconds` from 180 to 600.

### Example 3: Add a new component type

User says: "Add TiFlash evaluation to the autoscaler dashboard"

Actions:
1. Create `components/tiflash.md` with TiFlash-specific HPA config and PromQL queries
2. Add row to Component Registry table in this file
3. Create a TiFlash HTML template (copy `assets/copr-worker-report-template.html`, adjust chart labels)
4. Create a new eval command `commands/eval-tiflash-hpa.md` referencing the new component
5. Use `generate-report.sh` with the TiFlash template to produce report HTML

## Troubleshooting

| Issue | Cause | Solution |
|-------|-------|----------|
| O11Y query returns empty `{"clusters":[]}` | Wrong vendor/region for the cluster | Use `clinic-api` to look up correct `clusterProviderName` and `clusterRegionName` |
| K8s infra UUID not found (no `dp-` entries) | Tenant has no K8s infra O11Y registration | Verify cluster is NextGen (`prod-tidbcloud-nextgen`); legacy clusters use different discovery |
| Metrics return all zeros | Cluster has no workload during evaluation period | Choose a different date with known workload activity |
| Score seems wrong | Bracket thresholds are v0 heuristics | All scores are directional signals (TBD badges). Check the formula derivation in the report |

## Report Generation

Use `generate-report.sh` to inject evaluation data into the HTML template and produce
a self-contained report file. No build step or Node.js project needed.

### Usage

```bash
# Coprocessor worker report
./generate-report.sh /tmp/eval-data.json assets/copr-worker-report-template.html /tmp/feb14-report.html

# Or with the skill directory resolved automatically
SKILL_DIR="${HOME}/.config/opencode/skills/autoscaler-dashboard"
"${SKILL_DIR}/generate-report.sh" /tmp/eval-data.json \
  "${SKILL_DIR}/assets/copr-worker-report-template.html" \
  /tmp/feb14-report.html
```

### Arguments

| Arg | Required | Description |
|-----|----------|-------------|
| `data-file` | Yes | Path to JSON file containing the report data object |
| `template` | Yes | Path to the HTML template (e.g., `assets/copr-worker-report-template.html`) |
| `output` | Yes | Path for the generated HTML report file |

### What It Does

1. Reads the HTML template file
2. Reads the JSON data file
3. Replaces the `/*__REPORT_DATA__*/` placeholder with the JSON data
4. Writes the output HTML file

The agent's job is: create the JSON data file (matching the schema in `references/report-ui.md`),
then run the script to produce the report HTML.