# LLM Auto Coverage Classifier

## Current Design

- The worker runs the L1 Oncaller Gate, then `evaluateAutoCoverageGate`.
- `classifySupportScenario` currently derives `scenarioClass`, `resolutionMode`, evidence requirements, and risk flags from ordered regular expressions over the ticket summary, description, components, and reply-safe comments.
- `evaluateAutoCoverage` compares that classification with the configured enabled scenarios and resolutions. The DB gate-decision row is the canonical record of the live decision.
- If coverage is enabled, the worker creates a live workspace, runs the full Codex support analysis, and may write an internal Jira comment.

## Risks

- Keyword matching cannot reliably distinguish a question about an operation from a request to execute an operation. ST-2809 was classified as `ticdc_or_data_pipeline/evidence_assisted` even though establishing VPC peering is a privileged cloud-network operation.
- Letting an LLM directly return `enabled` would combine semantic classification and rollout policy, creating a second uncontrolled source of truth.
- Classifier timeouts, malformed output, or uncertainty must not fall through to full analysis or Jira writes.
- A second classification inside the live workspace must not overwrite or disagree with the classification used by the gate.

## Target Contract

- The worker Auto Coverage Gate uses a dedicated, read-only Codex classifier with structured JSON output and no web or network access.
- The classifier decides semantic fields: scenario, resolution mode, privileged-operation intent, evidence requirements, missing context, risks, signals, reason, and confidence.
- Requests that create, modify, delete, restart, scale, provision, approve, or otherwise operate real customer/cloud resources must be `operation_request + human_required`, including indirect wording such as “建立 VPC Peering”. Questions asking how a customer can perform an operation may remain Q&A when no support-side action is requested.
- Deterministic rollout policy remains canonical in `evaluateAutoCoverage`: configured scenario/resolution allowlists and hard stops decide `enabled/disabled`; the LLM never enables a scenario by itself.
- Classifier failure, invalid output, or uncertainty fails closed as a disabled Coverage Gate. It must not create a workspace, invoke full analysis, or write Jira.
- The exact gate classification is passed into the live workspace and persisted in `input/scenario-classification.json`; live analysis does not recompute it from regexes.
- Cheap poller gates and the L1 Oncaller Gate remain before the LLM call.

## Non-Goals

- Do not change polling, Jira visibility/write behavior, L1 ownership, production operations, or enabled coverage configuration.
- Do not use web search, Clinic, Investibook, or other diagnostics during classification.
- Do not add a regex fallback that silently enables automation when the classifier fails.
- Do not deploy, merge, or enable production automation as part of the local implementation.

## Implementation Steps

1. Define an asynchronous scenario-classifier interface and a Codex implementation with a strict output schema, low reasoning effort, bounded timeout, and read-only/no-network execution.
2. Keep rollout policy evaluation deterministic and inject the classifier into the auto-analysis runtime for testability.
3. Make the Coverage Gate asynchronous and return an explicit fail-closed result on classifier errors or low confidence.
4. Pass the accepted classification into live workspace creation and persist it as the single classification used by downstream analysis.
5. Add contract tests for ST-2809, Q&A-vs-operation distinction, malformed/timeout failure, no full runner invocation on failure, gate persistence, and workspace classification reuse.
6. Run typecheck, build, focused tests, and the full local test suite.

## Verification Checklist

- ST-2809-like input becomes `operation_request/human_required` and Coverage Gate is disabled.
- A documentation or “how do I” VPC peering question is not automatically treated as a support-side operation request.
- Invalid JSON, schema mismatch, timeout, and low-confidence classification all fail closed before workspace creation and Jira writes.
- Enabled Q&A/diagnostic classifications still pass only when both scenario and resolution are configured.
- The DB gate record contains the LLM classification and failure reason without storing secrets or large payloads.
- `input/scenario-classification.json` equals the classification used by the live Coverage Gate.
- Existing poller and L1 Gate ordering tests remain green.