# 10% Autonomous Resolution Roadmap

## Business Objective

Reach a four-week rolling state where at least 10% of initial tickets that pass
the L1 Oncaller Gate are handled end to end by Auto-oncall:

- Auto-oncall sends the customer-visible answer.
- The customer does not raise or challenge the answer.
- L1 does not need to send a subsequent substantive public reply.
- L1 quality feedback continues to identify the answer as acceptable.

The quality north-star metric is L1 `/ao accept` rate. Coverage is a separate
business outcome and must not be increased by shrinking the denominator or by
leaving candidate replies unreviewed.

## Current Design

- Staging polls and analyzes tickets, writes internal comments, collects `/ao`
  feedback, creates daily/weekly quality reports, and projects ticket facts to
  Base.
- Customer-visible reply code already has a deterministic gate, scenario and
  label allowlists, an independent fail-closed validator, and idempotent
  delivery state.
- Both staging and prod deployment templates currently keep customer-visible
  replies disabled and the scenario allowlist empty.
- Existing ticket analytics can show job outcome, gate category, human
  feedback, and generated AI-vs-human verdict.
- The first Phase 0 implementation now persists the L1-gate-eligible business
  population and direct-reply observations: exact reply/run identity, next
  substantive L1 public reply, conservative customer event, feedback, and the
  seven-day observation result. Cohort assignment and immutable metric
  snapshots remain pending.
- Existing `/ao` feedback is prompted on internal comments. A successful direct
  customer reply currently skips the internal comment, so the north-star
  feedback path is not complete for the future autonomous cohort.

## Risks

- `accept` rate can look high when only easy or successful tickets receive
  feedback. Feedback coverage and cohort composition must be visible.
- Defining the denominator from the current Auto Coverage gate would allow the
  system to improve the percentage merely by excluding difficult L1 tickets.
- Any customer follow-up is not necessarily a raise; acknowledgements and
  closure confirmations must not be treated as failures.
- Conversely, the absence of a customer reply does not prove the answer was
  correct.
- Human answers are reference evidence, not automatic golden answers.
- A broad scenario label is insufficient authorization for a customer-visible
  reply. Evidence-assisted diagnosis, operations, incidents, defects, security,
  account-specific conclusions, and uncertain context remain fail-closed.
- Small cohorts can produce misleading percentages. Promotion requires minimum
  sample sizes and confidence-aware reporting.
- Enabling customer-visible writes changes the approved automation boundary and
  therefore requires an explicit staged rollout decision, kill switch, and live
  verification.

## Target Contract

### 1. Business denominator

`l1_gate_eligible_tickets` is the count of initial tickets whose persisted
`l1_oncaller_gate` decision is `allowed` during the measurement window.

The canonical measurement point is exactly between the two worker gates:

```text
L1 Oncaller Gate --allowed--> business denominator
                              -> Auto Coverage Gate
```

- `l1_oncaller_gate=skipped` does not enter the denominator.
- A failed L1 gate remains a data-quality failure and does not enter either the
  numerator or denominator until the gate can be resolved.
- Once the L1 gate has allowed an initial ticket, later Auto Coverage skips,
  `human_required`, analysis failures, unsupported scenarios, and missing
  context remain in the denominator.
- The Auto Coverage optimizer cannot shrink or rewrite this population.
- Continuation jobs do not create additional denominator entries.

### 2. Autonomous-success numerator

An `autonomous_success` outcome requires all of the following:

1. A customer-visible Auto-oncall reply was written successfully and recorded
   by the idempotent delivery table.
2. The reply passed the deterministic customer-reply gate and independent
   validator under a versioned policy.
3. No substantive L1 public reply followed before the ticket reached Done or
   the seven-day observation window ended.
4. No confirmed customer raise occurred in that window.
5. There is no `/ao revise`, `/ao wrong`, or `/ao human` assessment for the
   reply. During the initial rollout, an explicit `/ao accept` is required to
   count a ticket as confirmed autonomous success.

After a scenario has graduated from full review to stratified audit, unreviewed
replies in that versioned cohort may count from the hard outcome conditions;
the sampled AO Accept Rate and Feedback Coverage remain cohort-level quality
constraints. Reviewed negative feedback always overrides an inferred success.

Customer acknowledgements, thanks, and closure confirmations do not invalidate
success. A material correction request, contradiction, escalation, reopen,
priority escalation, or L1-confirmed complaint does.

The business coverage metric is:

```text
Autonomous Resolution Rate
  = confirmed autonomous_success tickets / l1_gate_eligible_tickets
```

The target is a four-week rolling rate of at least 10%.

### 3. Quality north star and guardrails

```text
AO Accept Rate
  = accept / (accept + revise + wrong)
```

Keep the established contract that `/ao human` is excluded from the accept-rate
denominator, but show it as an independent `Human Required Rate` guardrail.

Always publish these adjacent metrics:

- Feedback Coverage = reviewed direct replies / all direct replies.
- Wrong Rate = wrong / reviewed direct replies.
- Human Required Rate = human / all direct replies with feedback.
- L1 Follow-up Rate = direct replies with a subsequent substantive L1 public
  reply / observed direct replies.
- Customer Raise Rate = confirmed raises / observed direct replies.
- Autonomous Resolution Rate over all L1-gate-eligible initial tickets.
- Median and p95 time to first useful reply.

Initial promotion policy:

- AO Accept Rate at least 90% for the promoted cohort.
- Wrong Rate and Customer Raise Rate equal to 0 during the canary.
- Feedback Coverage at least 80% while the cohort is small.
- At least 30 reviewed shadow/canary cases for a scenario before broadening it.
- No deterministic safety, visibility, idempotency, or evidence-contract
  failure.

These are conservative initial thresholds. They may be revised only through a
recorded business-policy decision, not by an optimizer trying to pass a cohort.

### 4. Canonical outcome model

Add DB-owned facts rather than deriving business success from reports or Base:

- `auto_oncall_direct_reply_cohorts`: cohort, scenario, policy version, traffic
  allocation, and eligibility decision.
- `auto_oncall_reply_outcomes`: direct reply delivery, observation deadline,
  next L1 reply, customer outcome, ticket resolution, final success state, and
  outcome reason.
- `auto_oncall_reply_reviews`: exact run/reply review result, reviewer, source,
  and timestamp. Legacy `/ao` feedback may be linked, but new direct-reply
  reviews must identify the exact reply/run.
- `auto_oncall_customer_events`: normalized acknowledgement, clarification,
  correction, raise, escalation, and uncertain events with evidence references.
- `auto_oncall_business_metric_snapshots`: immutable weekly metric version,
  numerator, denominator, exclusions, and policy version.

Jira is the event source. MySQL is canonical analytical state. Daily reports,
weekly reports, GitHub, and Base are projections.

### 5. Customer-raise determination

Use a two-stage fail-closed classifier:

1. Deterministic signals: reopen, priority escalation, escalation labels,
   explicit L1 correction, and configured complaint/escalation workflow events.
2. Read-only semantic classifier for public customer comments, returning
   `acknowledgement`, `clarification`, `correction`, `raise`, or `uncertain` with
   bounded evidence.

`uncertain` is never silently treated as success during rollout. It enters the
review queue.

### 6. Agent iteration loop

```text
Every ticket
  -> persist business outcome events
  -> calculate cohort metrics
  -> mine bad cases and repeated blockers
  -> create/deduplicate optimization proposal
  -> freeze replay case
  -> root-cause agent
  -> bounded candidate change + regression test
  -> baseline/challenger suites
  -> Draft PR with evidence
  -> approved staging rollout
  -> canary outcome observation
  -> keep, expand, or rollback
```

The optimizer follows the separate control-plane contract in
`2026-07-15-continuous-agent-optimization-loop.md`. It may prepare changes and
Draft PRs. It must not directly broaden customer-reply scenarios, traffic, or
production permissions.

## Rollout Plan

### Phase 0: Measurement baseline (Weeks 1-2)

1. Implement the canonical outcome/event tables and exact run-level review
   identity.
2. Add next-L1-reply and customer-event collection without changing any reply
   behavior.
3. Extend Base with weekly business denominator, feedback coverage, accept,
   follow-up, raise, and confirmed autonomous-success panels.
4. Backfill the recent four weeks where stable internal-comment and next-L1
   reply pairing is available; label unavailable fields instead of inferring
   success.
5. Publish the baseline cohort distribution and select the first two candidate
   scenarios.

Exit: every candidate ticket has a traceable denominator membership, exact
review identity, and observable post-reply outcome.

### Phase 1: Shadow autonomy (Weeks 3-5)

1. Keep customer-visible writes disabled.
2. For candidate direct-answer tickets, generate the exact reply that would have
   been sent and pair it with the next substantive L1 public reply.
3. Require L1 `/ao` review for shadow candidates and measure feedback coverage.
4. Seed evaluation suites from recent real cases, not old T0/T1 corpora.
5. Let the optimization agent address repeated gaps and create evaluated Draft
   PRs.

Recommended first cohort: narrow pure knowledge Q&A with stable documentation,
no real-time evidence, no account-specific conclusion, no operation, no known
incident/defect, no missing context, and no risk flag.

Exit: at least 30 reviewed cases, accept rate at least 90%, zero wrong/safety
failures, and a stable deterministic eligibility policy.

### Phase 2: Human-confirmed delivery canary (Weeks 6-7)

1. Add a time-bounded one-click approval mode for the exact prevalidated reply.
2. L1 approves or rejects without rewriting; rejection reason becomes structured
   optimization feedback.
3. Exercise customer-visible delivery, idempotency, event collection, and kill
   switch with no autonomous traffic yet.
4. Verify the real Jira comment ID, DB delivery/outcome rows, logs, and Base
   projection for every canary.

Exit: delivery and observation mechanics are proven without relying on L1 to
compose the answer.

### Phase 3: Autonomous customer-reply canary (Weeks 8-10)

This phase requires an explicit rollout approval because it enables
customer-visible automation.

1. Enable only one versioned scenario/policy cohort.
2. Start at approximately 1% of L1-gate-eligible initial tickets, during staffed hours, with a
   global kill switch and per-scenario circuit breaker.
3. Require retrospective `/ao` review for every autonomous reply.
4. Automatically pause the cohort on any wrong, confirmed raise, unsafe reply,
   duplicate write, or delivery-state inconsistency.
5. Observe each reply for seven days before marking confirmed success.

Exit: at least 30 reviewed autonomous replies, accept rate at least 90%, zero
wrong/raise, and L1 follow-up rate within the approved threshold.

### Phase 4: Scale from 1% to 10% (Weeks 11+)

Expand by evidence, not by a fixed calendar:

```text
1% -> 3% -> 5% -> 10%
```

At each step:

- Add traffic only within proven scenario policies or promote another shadowed
  scenario that independently passes the gate.
- Hold for at least one complete observation window.
- Compare cohort composition and four-week rolling outcomes.
- Keep feedback coverage high through full review initially, then move to a
  statistically valid stratified audit only after stable performance.
- Roll back automatically to the previous traffic policy on a circuit-breaker
  event.

The 10% goal is achieved only when the four-week rolling confirmed Autonomous
Resolution Rate is at least 10%, not when 10% merely receive an Auto-oncall
draft or public reply.

## Continuous Operating Cadence

### Per ticket

- Persist exact inputs, gate decisions, reply, delivery, review, customer
  events, next L1 reply, and final observed outcome.
- Route failures into answer, skill, tool/data, gate, evaluation, infrastructure,
  or non-actionable categories.

### Daily

- Reconcile pending seven-day outcomes.
- Update business metrics and identify new/repeated bad cases.
- Generate or update deduplicated optimization proposals.
- Run allowlisted evaluation jobs and prepare Draft PRs when evidence passes.

### Weekly

- Agent produces a compact operating review: progress toward 10%, accept rate,
  feedback coverage, raise/follow-up guardrails, cohort changes, proposed
  experiments, merged changes, rejected ideas, and rollback status.
- Human governance chooses or rejects major scenario/traffic changes. Routine
  reproduction, implementation, testing, and evidence assembly remain Agent
  work.

### Monthly

- Reassess whether the best route to 10% is deeper quality in existing cohorts
  or adding another safe scenario.
- Review threshold drift, customer segments, costs, latency, and false-negative
  exclusions.
- Update the versioned business policy explicitly.

## Initial Backlog

1. **Implemented locally:** business denominator and direct-reply outcome
   schema, with insertion exactly after an allowed L1 Oncaller Gate.
2. **Partial:** exact direct-reply `/ao` feedback can bind to the direct-reply
   run; a dedicated review prompt/queue and feedback-coverage metric remain.
3. **Implemented locally:** read-only next substantive L1 public reply
   collector and seven-day outcome reconciliation.
4. **Partial:** deterministic acknowledgement/clarification/correction/raise
   classification is fail-closed; semantic classification and the uncertain
   review queue remain.
5. Base and report projections for the full metric contract.
6. Four-week read-only baseline/backfill.
7. First pure-Q&A shadow cohort and 30-case suite.
8. Optimization proposal state and evaluated Draft-PR automation.
9. One-click delivery canary.
10. Explicit approval package for the first autonomous 1% customer-reply cohort.

## Invariants

- The business denominator cannot be changed by the coverage optimizer.
- Missing feedback or an unexpired observation window cannot count as success.
- Human/public answers never enter the replayed agent input.
- Customer-visible delivery remains independently gated, idempotent, and
  fail-closed.
- Continuations remain internal-only unless separately redesigned and approved.
- The system does not transition or close tickets in the initial 10% plan.
- No diagnostic, operation, incident, defect, security, or account-specific
  scenario enters customer-direct reply merely to increase coverage.
- Staging and prod are never simultaneously active pollers.
- Any circuit-breaker event pauses the affected cohort before further writes.
- Deployment proof includes app-delivery source, live image/env, logs, DB state,
  Jira comment readback, and outcome projection.

## Verification Checklist

- Business denominator reconciles with the L1 queue independently of Auto
  Coverage decisions.
- Direct reply, review, customer event, next L1 reply, and observation deadline
  join to the exact run/reply.
- Acknowledgement does not count as raise; correction/escalation does.
- Uncertain, missing feedback, and pending outcomes do not count as success.
- Repeated polling is idempotent and does not duplicate events or reviews.
- Shadow mode performs no customer-visible write.
- One-click and autonomous delivery reuse the same validated body and delivery
  idempotency contract.
- Any wrong/raise/safety/idempotency failure trips the cohort circuit breaker.
- Weekly reports show numerator, denominator, exclusions, feedback coverage,
  and cohort composition.
- Baseline/challenger evaluation and protected suites pass before Draft PR.
- Four-week rolling rate, rather than a one-day spike, is used for the 10% goal.

## Current Progress (2026-07-15)

- Added canonical `auto_oncall_l1_population` membership for every initial
  ticket allowed by the L1 Oncaller Gate, before Auto Coverage is evaluated.
- Added direct-reply outcome and customer-event persistence keyed to the exact
  run and Jira public comment.
- Added idempotent, read-only Jira reconciliation for next L1 reply, customer
  event, `/ao` feedback, ticket Done state, and the observation deadline.
- Added local contract tests for denominator boundaries, repeat-safe outcome
  selection, accepted success, L1 follow-up, correction, missing feedback,
  missing reply, and conservative customer-event classification.
- Customer-visible replies remain disabled. No deployment or live
  configuration has changed.

## Non-Goals

- Do not count internal comments, drafts, or generated answers as autonomous
  resolutions.
- Do not optimize only for similarity to the next human reply.
- Do not hide unsupported tickets from the business denominator.
- Do not infer success from silence alone.
- Do not enable customer-visible replies as part of the measurement work.
- Do not auto-transition, close tickets, execute production operations, or
  broaden beyond L1 without a separate design and approval.