# Optional Evaluation Control

<!-- auto-oncall-optimization-record
{
  "schemaVersion": "auto-oncall-optimization-record-v1",
  "optimizationId": "optional-evaluation-control-2026-07-20",
  "problems": [
    {
      "problemId": "focus-control-absence-blocks-selection",
      "summary": "A missing control blocks an otherwise valid repeated target Focus.",
      "status": "optimized_unverified",
      "issueKeys": [],
      "sourceRefs": [],
      "signalTextIncludesAny": [],
      "observedThrough": "2026-07-20",
      "effectiveAt": "2026-07-20T17:04:09+08:00",
      "verifiedAt": null,
      "attemptedAction": "Allow target-only Focus selection when no useful control exists.",
      "evidenceUrl": "https://github.com/tidbcloud/aisupport/pull/374"
    },
    {
      "problemId": "focus-control-gap-blocks-creation",
      "summary": "A control gap blocks Focus creation even when target gaps prove the problem.",
      "status": "optimized_unverified",
      "issueKeys": [],
      "sourceRefs": [],
      "signalTextIncludesAny": [],
      "observedThrough": "2026-07-20",
      "effectiveAt": "2026-07-20T17:04:09+08:00",
      "verifiedAt": null,
      "attemptedAction": "Gate Focus creation on target findings instead of control status.",
      "evidenceUrl": "https://github.com/tidbcloud/aisupport/pull/374"
    },
    {
      "problemId": "focus-control-failure-aborts-evaluation",
      "summary": "A control execution failure aborts baseline or challenger evaluation.",
      "status": "optimized_unverified",
      "issueKeys": [],
      "sourceRefs": [],
      "signalTextIncludesAny": [],
      "observedThrough": "2026-07-20",
      "effectiveAt": "2026-07-20T17:04:09+08:00",
      "verifiedAt": null,
      "attemptedAction": "Exclude control failures from failed-target counting.",
      "evidenceUrl": "https://github.com/tidbcloud/aisupport/pull/374"
    },
    {
      "problemId": "focus-deployed-revision-cannot-create-focus",
      "summary": "The deployed optimizer cannot create a new Focus from recent signals.",
      "status": "optimized_verified",
      "issueKeys": [],
      "sourceRefs": [],
      "signalTextIncludesAny": [],
      "observedThrough": "2026-07-20",
      "effectiveAt": "2026-07-20T17:04:09+08:00",
      "verifiedAt": "2026-07-20T17:04:09+08:00",
      "attemptedAction": "Deploy optional-control selection and rerun the optimizer.",
      "evidenceUrl": "https://github.com/tidbcloud/aisupport/issues/375"
    }
  ]
}
-->

- Optimization ID: `optional-evaluation-control-2026-07-20`
- Owner: auto-oncall
- Source type: repository change
- Source reference: aisupport PR
  https://github.com/tidbcloud/aisupport/pull/374; merge revision
  `950a394bda1e9994aea0e19f10f00f9c9e732560`
- Problem: A missing, unstable, or unavailable control could prevent a valid
  repeated target problem from becoming a Focus or continuing through replay.
- Capability: Recent Focus selection, Focus creation baseline, daily baseline,
  challenger, and candidate evaluation.
- Implementation status: implemented and deployed to staging
- Effective status: available to staging since 2026-07-20

## Problem validation

| Problem signal | Ticket keys | Status | Evidence |
| --- | --- | --- | --- |
| Control absence blocks Focus selection | None; workflow contract | `optimized_unverified` | Contract accepts two targets with zero controls; unit test passed locally on 2026-07-20. |
| Control gap blocks Focus creation | Recent rejected Focus candidate | `optimized_unverified` | `shouldCreateRecentFocus` now depends on target objective and `gaps_found`, not control status; unit test passed locally on 2026-07-20. |
| Control execution failure aborts baseline/challenger | None; workflow contract | `optimized_unverified` | Control failures are excluded from failed-target counting; unit test passed locally on 2026-07-20. |
| Deployed revision cannot create a new Focus | Recent Optimizer rerun | `optimized_verified` | Staging rerun created Focus https://github.com/tidbcloud/aisupport/issues/375 from four target cases and one control case; all five executions succeeded. |

The deployed Focus creation path is verified. The three narrower control
conditions remain unverified in live evaluation because this rerun selected a
control that met its criteria; it did not exercise a missing, failed, or gapped
control.

## Verification performed

- TypeScript build passed.
- 45 related Evaluation, Focus, Optimizer, judge, and notification tests passed.
- `git diff --check` passed.
- Full `npm test` was attempted but the existing `app.test` process retained a
  long-lived handle; the related bounded suites completed successfully.
- Staging Deployment and Recent Optimizer CronJob both used image revision
  `950a394bda1e9994aea0e19f10f00f9c9e732560` after rollout.
- App-delivery PR https://github.com/tidbcloud/app-delivery/pull/74973 merged.
- Manual Job `auto-oncall-recent-optimizer-rerun4-20260720` completed once from
  2026-07-20T08:33:09Z through 2026-07-20T09:04:09Z.
- The Job returned `focus_created` for
  https://github.com/tidbcloud/aisupport/issues/375 with four targets, one
  control, `targetCasesMet=0/4`, and `controlCasesMet=1/1`.
- Focus #375 is active and contains durable `case_refresh` and `baseline`
  checkpoints.

## Unresolved gaps

- A live rerun with a missing, failed, or gapped control is still needed to
  upgrade the three control-specific rows to `optimized_verified`.
- Problem-signal backlog and durable rejected-candidate evidence remain separate
  follow-up work documented in `docs/AUTO_ONCALL_EVALUATION.md`.

## Next validation action

Let Focus #375 run through its normal lifecycle. When a later candidate naturally
has no usable control, verify that target-only evidence can still create or
continue the Focus and update the three control-specific statuses.