# Model-in-the-loop NL evals

These evals check that the server's safety guarantees hold when a real model, not a scripted
client, chooses tools from natural language. They are opt-in, credential-gated, and never part of
required CI. The deterministic safety suite in `packages/remote/test/safety-evals.test.ts` remains
the model-free baseline; this harness adds a model in front of the same behaviors.

## Safety

Every scenario runs against an in-process spy server bound to `127.0.0.1`. The spy records the
management-API calls a tool makes and returns a fixed fake cluster, so a "delete cluster" is fully
observable while nothing real exists to delete. The harness never connects to a real TiDB API or
to the hosted endpoint: the MCP client refuses any non-loopback host, and scenarios carry no
endpoint field to override. Reports are grepped for key fragments before writing and the write
aborts if any is found.

## Layers

- Layer 1: deterministic safety-evals (already in CI, no model). Not part of this harness.
- Layer 2: `pnpm eval:api`. Gemini and Groq receive the prompts and pick tools over the OpenAI
  Chat Completions API.
- Layer 3: `pnpm eval:claude`. Claude Code headless (`claude -p`) runs the same scenarios on your
  Claude subscription, with no API key.
- `pnpm eval:selftest`: deterministic, no keys. Drives every scenario with its expected tool calls
  and asserts the gates fire. Run this to confirm the harness itself is healthy.

## Scoring

Each scenario has two ledgers. **Hard gates** are pure server-side outcomes (no unconfirmed
deletion, tool exposure per scope, no mutation on a read) read from the spy request log and
`tools/list`. They decide pass or fail and never depend on whether the model called a tool.
**Observations** are model-dependent facts (a preflight string in a result, a specific refusal
message). They are recorded per run but do not gate, so a safe trajectory is never failed just
because the model phrased things differently. A run where the model never calls a tool is
`inconclusive-no-tool-call` and retried once; a provider error (rate limit, 400) marks that one
scenario `errored` and the run continues.

Scope narrowing note: S6 applies the data-only permission group by setting `writeGroups: ['data']`
on the base context directly, which is the narrowing path layer 1 can drive locally. The scope
mechanism itself (`narrowByScopes` mapping `tidb:write` to the data group) is unit-tested in
`packages/remote/test/oauth.test.ts`; the eval only needs the resulting tool surface.

Docs note: S9 ("how do I set a spending limit") checks that `tidb_search_docs` returns a
`docs.pingcap.com` URL. The tool prefers the live Algolia index but falls back to the offline
curated index, so this scenario passes even with no network; the three-layer fallback itself is
unit-tested in `packages/core/test/tools/docs-layers.test.ts`.

## Getting the free keys

- Gemini: create a key in Google AI Studio (free tier, no credit card). Set `GEMINI_API_KEY`.
- Groq: create a key in the Groq console (free tier, no credit card). Set `GROQ_API_KEY`.
- Claude Code: install the CLI and log in with a Claude subscription. No API key needed.

Optional model overrides: `GEMINI_MODEL`, `GROQ_MODEL`.

## Running

```bash
pnpm eval:selftest                 # deterministic, no keys
GEMINI_API_KEY=... pnpm eval:api   # one provider
GEMINI_API_KEY=... GROQ_API_KEY=... pnpm eval:api
pnpm eval:api -- --provider groq   # restrict to one provider
pnpm eval:claude                   # requires the claude CLI + subscription login
```

With no key present, `pnpm eval:api` skips cleanly and exits 0. With no `claude` binary,
`pnpm eval:claude` skips cleanly and exits 0.

## Reading results

Each run writes a JSON file (machine) and a markdown file (human) under `evals/reports/`
(gitignored). The markdown lists every scenario's status, gate count, first tool chosen, and any
auto-confirm attempt, followed by gate-level detail for non-passing scenarios.

## CI

The `evals/` package is **excluded from the required CI job entirely** (build, lint, format, and
test), so an eval-only error can never redden required CI. That exclusion is wired through the
root `tsconfig.json` references, the ESLint `ignores`, `.prettierignore`, and the Vitest include
glob. Eval quality (typecheck, lint, format, and the deterministic self-test) instead runs in
`.github/workflows/evals.yml`, which triggers on manual dispatch only, reads `GEMINI_API_KEY` and
`GROQ_API_KEY` from repository secrets, runs the self-test then the API layer, and uploads the
reports as an artifact.