# Offline evaluation MVP This package evaluates autoscaling policies locally in two modes: point-in-time decision cases and continuous historical replay. It does not start the operator, connect to Kubernetes, or execute a scaling action. The MVP provides: - a strict YAML `CaseSpec` for committed cases; - versioned shared metrics profiles referenced by O11Y cases; - a normalized JSON `CaseBundle` used at evaluation time; - deterministic catalog loading and raw-response replay without O11Y access when a matching archive exists; - embedded metric samples for manual cases; - fixed O11Y request parameters and one committed raw-response archive per historical case; - direct evaluation of the production `pkg/policy.Policy` contract; - trivial policies that establish the lower bounds a candidate must beat; - suite metrics for useful ScaleOut recall, unnecessary ScaleOut rejection, and avoided replicas; - interval and sparse replay schedules over committed historical metrics; - point-in-time production policy inputs with no future samples; - deterministic JSON and Markdown replay reports; - automatic replay dataset discovery and acquisition on a local archive miss; - request-keyed private caching for downloaded O11Y responses; - bounded case-level O11Y acquisition with deterministic aggregate results; - guards against partial, missing, gapped, incomplete, and post-decision data; - committed responses, so the catalog outlives O11Y metrics retention. ## Directory layout | Path | Purpose | |---|---| | `test/evaluation/` | Go library for case loading, O11Y replay, Policy evaluation, continuous replay, and reporting. | | `test/evaluation/cmd/` | Local command entry points built on the evaluation library. | | `test/evaluation/cmd/autoscaling-eval/` | Go CLI for running experiments, fetching O11Y data, and seeding the local cache. | | `test/evaluation/cases/` | Component-grouped historical evidence and reusable metric definitions. | | `test/evaluation/cases//` | Boundary for one component's decision cases, metric profiles, and replay datasets. | | `test/evaluation/cases//point-in-time/` | Independent decision case YAML and same-ID raw O11Y JSON archives. | | `test/evaluation/cases//metrics/` | Shared versioned PromQL profiles referenced by point-in-time and replay definitions. | | `test/evaluation/cases//replays/` | Continuous dataset YAML and adjacent raw O11Y JSON archives. | | `test/evaluation/experiments/` | User-editable replay experiment configurations. | | `test/evaluation/experiments//` | Policy and replay settings for one component. | The policy returns capacity, not an execution action. The evaluator derives `ScaleOut`, `ScaleIn`, or `NoScale` by comparing desired and current replicas. `ScaleUp` and `ScaleDown` remain reserved for vertical changes. Run the local-only evaluation from `internal/capacity-planner`: ```sh make test/evaluation ``` This target reloads and replays all committed cases on every run. The aggregate `make test` target includes this suite. It also runs the default continuous experiment and writes: - `build/bin/evaluation/offline-evalution-report.md`: combined conclusions, point-in-time decision results, and replay results; - `build/bin/evaluation/report.md`: continuous replay details; - `build/bin/evaluation/report.json`: continuous replay decisions. Use a different experiment or report directory without changing the Makefile: ```sh make test/evaluation \ EVALUATION_EXPERIMENT=test/evaluation/experiments/tidb/cost-optimized-horizontal-v1.yaml \ EVALUATION_OUTPUT=/tmp/tidb-replay ``` ## Continuous replay The experiment YAML is the user-facing run configuration. It selects the component, metric profile, logical cluster, inclusive replay window, raw sample interval, evaluation interval, HPA reference metric, production Policy configuration, and mapping from archived series to Policy resource metrics. It does not select a dataset file. Every field in the committed experiment has an inline English comment. The runner searches `cases//replays/` for one dataset with the same component, metric profile, cluster ID, and raw sample interval whose archived UTC window covers the requested experiment window. Its adjacent JSON file contains the raw O11Y responses. If no archive matches, the runner discovers the cluster placement and O11Y data sources, fetches the requested window, and writes a generated dataset YAML and adjacent raw JSON archive. If a matching YAML exists but its JSON archive is missing, the runner fetches that defined dataset without repeating placement discovery. No cluster ID, replay window, or dataset path is hardcoded in Go. For Premium clusters, Clinic resolves the parent Resource Pool, O11Y discovery resolves the application and infrastructure data sources, and an application `up` probe resolves the target TiDBGroup from its instance names. The generated HPA query uses that exact `tidbgroup-...` name; it does not aggregate every HPA in the shared Resource Pool namespace. The committed MVP replays one week for one Premium TiDB cluster at a five-minute decision interval. Its single fixed tier limits recommendations to horizontal ScaleOut, ScaleIn, or NoScale. CPU and RSS are policy inputs; the remaining profile series are preserved as observations. Run the CLI directly with: ```sh go run ./test/evaluation/cmd/autoscaling-eval run \ --experiment test/evaluation/experiments/tidb/cost-optimized-horizontal-v1.yaml \ --output /tmp/tidb-replay ``` A local archive hit requires no credentials, VPN, cache, or O11Y request. An archive miss requires `CLINIC_API_KEY` for cluster placement discovery and `O11Y_AUTH0_TOKEN` for O11Y discovery and metric queries. A missing JSON next to an existing dataset YAML requires only `O11Y_AUTH0_TOKEN`. The default request cache is `/cache`; use `--cache-dir` to preserve it when report output directories change. The generated YAML and raw JSON archive are durable historical evidence and should be committed before O11Y retention expires. The private request cache must not be committed. Each decision uses only the latest samples at or before its timestamp. A point is skipped when the observed replica count is stale or when CPU and memory do not contain the same complete instance set. Policy errors are recorded and do not stop later decisions. The replay is open-loop: historical replica changes remain the observed state, and a recommendation does not change later metrics or replicas. The report is therefore suitable for recommendation coverage, stability, capacity, and cost diagnostics, but it is not a counterfactual workload simulation. The report includes: - evaluated, skipped, and policy-error points; - Policy versus HPA target changes, direction reversals, and reversal rates; - comparable observed, HPA-desired, and Policy-recommended replica-hours; - explicit timeline restart reasons after skipped inputs or Policy errors; - experiment, archive, and policy revision identities. Continuous replay does not load or report point-in-time cases. They are evaluated separately as described below. ## Combined offline report `offline-evalution-report.md` starts with separate conclusions for Point-in-Time Decision Evaluation and Continuous Replay. Point-in-time cases keep their own timestamps and are not restricted to the replay window. The report does not calculate one mixed score. ## Case format Every committed case stores common decision metadata and selects one metric source under `metrics.type`: - `manual`: `metrics.series` contains the complete labeled samples. - `o11y`: the case stores a versioned profile reference, logical target, exact metrics endpoint, fixed UTC window, and step. The profile owns the reusable PromQL, required-series, label, and gap rules. The only supported template is `${cluster_id}`. The repository contains 112 TiDB ScaleOut decision cases under `cases/tidb/point-in-time/`: 81 expect the current replica count and 31 expect the historical target replica count. Their historical evidence does not establish a tier, so the expected tier is omitted and ignored. All cases reference the immutable `cases/tidb/metrics/horizontal-v1.yaml` profile. Only TiDB application metrics are requested. The historical target is recorded in `baseline` and is not fetched as policy input. ## Fetch O11Y data Authenticate using the O11Y workflow, then run: ```sh export O11Y_AUTH0_TOKEN='...' output_dir=$(mktemp -d "${TMPDIR:-/tmp}/autoscaling-eval.XXXXXX") cache_dir=$(mktemp -d "${TMPDIR:-/tmp}/autoscaling-eval-cache.XXXXXX") go run ./test/evaluation/cmd/autoscaling-eval fetch-o11y \ --case test/evaluation/cases/tidb/point-in-time/tidb-scale-out-20260613T133430Z-32fc79a71fdd.yaml \ --output "$output_dir" \ --cache-dir "$cache_dir" ``` The command writes one mode `0600` archive at `//point-in-time/.json`. It contains every exact request, unmodified Prometheus response, count, annotation, and data-quality result. Use `--case-output ` only when a derived `CaseBundle` is useful for local inspection; that file is not committed. Responses are cached as mode `0600` files under the mode `0700` cache directory. The cache key covers the metrics endpoint and complete encoded Prometheus request parameters. Reusing the same cache directory with the same case performs no O11Y API request and does not require a token. Any endpoint, PromQL, window, step, or instant-query time change produces a cache miss and requires authentication. The runtime flag `--cache-dir` selects the cache directory; it is not part of the case. If the flag is omitted, the CLI uses `/cache`, so later runs must reuse the same `--output` directory. Caching happens as soon as a response decodes as a non-partial Prometheus success, before the data-quality checks. Those checks describe the fixed historical window itself, so re-fetching cannot change their outcome; a rejected case must not re-hit O11Y on every rerun. Partial responses are not cached, because they can be a transient backend timeout. Tokens come from the environment and are never serialized. `fetch-o11y` does not execute a policy. The `run` command automatically discovers and fetches a continuous archive when its cluster ID and window are not available locally. The lower-level command below remains available to refresh an existing dataset definition explicitly: ```sh go run ./test/evaluation/cmd/autoscaling-eval fetch-o11y-replay \ --dataset test/evaluation/cases/tidb/replays/tidb-continuous-20260801T000000Z-20260808T000000Z-a3ffdd609441.yaml \ --output test/evaluation/cases \ --cache-dir /tmp/autoscaling-eval-replay-cache ``` The command writes the archive next to the dataset YAML under `cases//replays/`. ## Data quality A response must clear three checks to produce a valid `CaseBundle`: | Check | Rejects | |---|---| | non-partial Prometheus success | backend timeouts and query errors | | no gap longer than `maxGapSeconds` **inside** a series | a break in collection | | instances reporting within `maxGapSeconds` of `decisionAt` >= `current.replicas` | missing coverage at the decision point | A series that starts or ends part-way through the window is deliberately not a gap. Pods are created and removed during a scaling event, so a new pod's series begins mid-window and a removed pod's series ends mid-window. Rejecting those would discard exactly the cases that carry the most signal: of the 112 committed cases, 18 contain instance churn and none contain an actual break in collection. Coverage is therefore checked only where the decision reads it, at `decisionAt`. An instance that stopped reporting earlier in the window was removed before the decision and does not count towards it. Its history remains available to the quality checks, but only per-instance series fresh within `maxGapSeconds` enter the Policy-visible `CaseBundle`. ## Committed archives `cases//point-in-time/.json` is the durable source for every historical case, next to its same-ID YAML definition. Each archive contains all seven raw responses and their request identity. The 112 TiDB archives therefore retain 784 responses. Normalized `CaseBundle` files are derived and are not committed. A case whose metrics have aged out of O11Y retention and has no archive cannot be replayed, so it is removed from the catalog rather than kept as an unusable entry. Committing the responses removes the dependency on O11Y retention: the archive is replayable long after the source data expires. `LoadCatalog` validates each YAML/archive pair and request digest, then recomputes normalized metrics and data quality from the raw responses. It does not read the cache or contact O11Y. The cache remains useful for rerunning the acquisition path. Rebuild it from the committed archives and rerun without a token or VPN: ```sh go run ./test/evaluation/cmd/autoscaling-eval seed-cache \ --cases-dir test/evaluation/cases \ --cache-dir /tmp/eval-cache go run ./test/evaluation/cmd/autoscaling-eval fetch-o11y-suite \ --cases-dir test/evaluation/cases \ --output /tmp/eval-out \ --cache-dir /tmp/eval-cache ``` The archive is stored under the component's `point-in-time/` directory by case ID; the cache is keyed by request digest. Each archived query carries both the digest and response, so `seed-cache` is a mechanical copy. A normalization change can then be replayed and diffed without O11Y access. Fetch the complete catalog with bounded case-level concurrency: ```sh output_dir=$(mktemp -d "${TMPDIR:-/tmp}/autoscaling-eval-suite.XXXXXX") cache_dir=$(mktemp -d "${TMPDIR:-/tmp}/autoscaling-eval-cache.XXXXXX") go run ./test/evaluation/cmd/autoscaling-eval fetch-o11y-suite \ --cases-dir test/evaluation/cases \ --output "$output_dir" \ --cache-dir "$cache_dir" \ --workers 4 ``` Workers fetch different cases concurrently; queries within one case remain sequential. Each case writes to `//point-in-time/.json`, all cases share the cache, and one failure does not stop the remaining downloads. Results are reported in case ID order and any failure makes the command exit nonzero. Policy evaluation remains sequential because the production policy's concurrency contract is not part of this MVP. ## Point-in-Time Decision Evaluation Each case fixes one `decisionAt` and its available metrics, calls the Policy once, and checks the target capacity. Cases may use historical O11Y data or manually constructed metrics. They do not simulate state changes before or after the decision. For TiDB historical cases, the framework reconstructs the node spec active at `decisionAt` from per-instance `GOMAXPROCS` and memory-quota metrics. It then selects the one matching historical request-capacity tier and passes only that tier to `CostOptimized`. Missing, stale, unknown, or conflicting tier evidence fails the evaluation instead of silently removing the case. `LoadCatalog` produces deterministic `CaseBundle` inputs, and `EvaluateCatalog` passes them to `EvaluateSuite` with the supplied `pkg/policy.Policy`. The result contains individual decisions plus these aggregate metrics: - useful ScaleOut cases accepted and their recall; - unnecessary ScaleOut cases rejected and their rejection rate; - avoided replicas from correctly rejected historical ScaleOuts; - total, passing, and failing cases. Action rate and direction reversals are deliberately absent because each case contains one independent decision point. Continuous Replay records target changes and direction reversals over its fixed interval. `Result.AvoidedReplicas` is nonzero only for a rejected `UnnecessaryScaleOut` case, so `SuiteMetrics.AvoidedReplicas` is exactly the sum of the per-case values. ### Trivial lower bounds The catalog is unbalanced by construction (81 unnecessary, 31 useful), so a single headline number is easy to manufacture: | Policy | Exact passes | Recall | Rejection rate | Avoided replicas | |---|---:|---:|---:|---:| | `AlwaysNoScale(tier)` | 82/112 | 1/31 | 81/81 | 86 | | `AlwaysScaleOut(tier)` | 29/112 | 31/31 | 0/81 | 0 | Both use the decision-time `tidb_up_replicas` metric; manual fixtures without that metric fall back to per-instance series cardinality. One committed case records `current.replicas: 1` while its final raw replica sample is 2, which is why `AlwaysNoScale` has one accepted useful ScaleOut. The framework preserves that observed mismatch instead of rewriting the historical evidence. Report both baselines beside a candidate. A candidate is useful only if it improves on both bounds together. There is no "follow the historical baseline" policy. Production policies receive only metrics, not `Baseline` or `Oracle`, so they cannot see what HPA did. Historical ScaleIn cases remain out of scope.