# Recent Evaluation Dry-run

## Current Design

- The daily report identifies recent ticket signals without creating an optimization Issue.
- `POST /v1/tickets/analyze` already runs one Jira case through the shared
  dry-run/live analysis contract and writes a per-run workspace.
- The first implementation added a small command for explicitly listed Jira
  keys, but it only produced per-ticket dry-run output. It did not define the
  Active Focus, group repeated cases across daily windows, or judge one shared
  acceptance contract.

## Risks

- An evaluation command must never write Jira comments or perform production
  operations.
- A failed case must not prevent the remaining selected cases from running.
- Jira is mutable. This first iteration reuses the existing historical reply
  boundary; it does not claim to be an immutable regression suite.
- Missing local skills or runtime capabilities are evaluation-environment
  limitations and must not be misreported as answer-quality failures.
- Broad automatic case mining would duplicate the daily report and introduce a
  second source of truth.
- Mixing unrelated daily findings in one run would make the result a ticket
  recap rather than a focused evaluation.

## Target Contract

- The one active GitHub Focus Issue is the canonical contract. It contains one
  problem statement, shared acceptance criteria, and up to ten recent cases.
- `npm run evaluate:recent` loads that Issue by default. An explicit focus file
  is accepted only as the controller's credential-free execution input.
- The command delegates each case to the existing single-ticket dry-run with
  Jira write mode set to `none`.
- The hidden historical L1 target reply is retained only as evaluation
  reference; it is never added to the support-agent input.
- A read-only judge applies the shared focus criteria to all current dry-run
  outputs and references. Human replies are evidence, not automatic goldens.
- Output is `recent-evaluation-v3` JSON with the Active Focus, execution
  summary, per-case output, case-level criteria findings, and one focus-level
  `criteria_met`, `gaps_found`, or `inconclusive` finding.
- Failures are recorded per case and the batch continues.
- The process exits non-zero when any selected case fails to execute.
- Daily analysis remains the signal source; the Focus Issue is the only durable
  optimization source.

## Implementation Steps

1. Replace free-form ticket arguments with a validated Active Focus manifest.
2. Preserve the hidden historical L1 target reply as judge-only reference.
3. Add a read-only structured judge and focus-level result.
4. Add focused contract tests and keep Jira write mode fixed to `none`.
5. Run the current non-CDC performance-diagnosis focus across recent days.

## Verification Checklist

- [x] Missing, malformed, duplicate, empty, and over-limit focus cases are covered.
- [x] A failed case is recorded while later cases still run.
- [x] Jira internal comment mode is always `none`.
- [x] Historical L1 reference is excluded from support-agent input and used only by the judge.
- [x] Judge output covers every focus criterion and every case exactly once.
- [x] Build, focused tests, and full tests pass.
- [x] One real focus-level batch completes without Jira, Feishu, or GitHub writes.

## First Focus Run

- Focus: performance evidence routing and conclusion convergence.
- Cases: three recent targets plus one recent control.
- Execution: four succeeded; two `internal_only`, two `human_required`.
- Assessment: `gaps_found`. ST-2814 had a provider-start gap and
  APID-29173 passed. APID-29166 and the APID-29178 control were inconclusive
  because the local evaluation environment lacked the required Wetech skill.
- Next action: fix the Cloud provider-start contract before changing answer
  wording, then rerun Wetech cases in a staging-parity evaluation environment.

## Not In Scope

- CDC diagnosis or CDC configuration retrieval.
- Regression suites or new database tables.
- A second bad-case selection pipeline.
- Jira writes, deployment, merge, or production rollout.