# Continuous Agent Optimization Loop

## Current Design

The online path already persists most of the observation data needed for a
learning loop:

```text
Jira ticket
  -> immutable ticket snapshots
  -> job / gate / run / evidence / feedback
  -> deterministic AI-human pairing
  -> Codex daily or weekly report artifact
  -> daily GitHub optimization issue + Feishu report
  -> ticket-level Base dashboard
```

- MySQL is canonical for ticket facts, runs, gate decisions, feedback, answer
  evaluations, reports, and delivery state.
- Daily report Codex can identify structured `prompt`, `skill`, `context`,
  `gate`, `tooling`, and `evaluation` optimizations, but it is read-only.
- A daily window can create one idempotent GitHub issue. The issue is an output
  channel; there is no canonical optimization lifecycle behind it.
- Base exposes weekly ticket and AI-effectiveness trends, but it is also only a
  projection.
- Historical Codex work shows that the remaining loop is manual: inspect a bad
  case, separate report/gate/tool/prompt failures, edit code or skills, add
  tests, run a bounded replay, open a PR, deploy staging, rerun the case, and
  verify DB/Jira/runtime state.

The missing chain is therefore:

```text
trusted bad-case signal
  -> reproducible frozen case
  -> root-cause classification
  -> candidate change
  -> baseline/challenger evaluation
  -> reviewable PR
  -> staged promotion and rollback evidence
```

### Existing concept and compatibility audit

- **Retain:** immutable ticket/run facts, replay-safe visible-input selection,
  report answer pairing, `/ao` feedback, answer verdicts, report artifacts, and
  ticket-level Base projection. These are useful observation inputs.
- **Retain but do not treat as evaluation:** the current dry-run
  `replayTarget`/`replayContextMarkdown` path simulates a support reply and is a
  useful execution primitive, but it is not a versioned suite runner and does
  not own baseline/challenger results.
- **Must not become canonical:** daily GitHub issues, Feishu reports, Base rows,
  generated report prose, and Codex thread history. They remain projections or
  operator views.
- **Read-only compatibility with an exit plan:** issue-level feedback fallback
  is currently used only when a ticket has exactly one comparison. New
  optimization signals must use exact `run_db_id`; backfill legacy feedback and
  delete the fallback after the migration window.
- **Historical only:** completed plan documents describe prior migrations and
  fixes; they may seed regression cases but must not be queried as runtime
  policy.
- **Do not copy stale API assumptions:** provider endpoints such as the legacy
  Investibook evidence path must be verified against the deployed integration
  before becoming an evaluation dependency. The optimizer may record a broken
  integration as a tool/data signal; it must not silently fall back to another
  provider and claim the candidate passed.

## Risks

- A human reply is useful reference evidence, not an automatic golden answer.
- `context_changed`, missing attachments, report pairing bugs, auth failures,
  and gate-policy bugs must not be mislabeled as answer-quality failures.
- Optimizing from one case can overfit a prompt or skill and regress unrelated
  scenarios.
- Replaying a mutable Jira ticket or current external metrics changes the input
  and makes the comparison invalid.
- Letting an optimizer edit the online worker workspace or deploy directly can
  couple experimentation to the production-affecting path.
- LLM-as-judge alone can reward style changes while missing factual, evidence,
  safety, latency, or tool-routing regressions.
- Daily issues can duplicate the same underlying problem across windows unless
  proposals have a stable fingerprint and lifecycle.

## Target Contract

### Architecture boundary

Keep the online serving path stable and introduce one isolated Optimization
Control Plane. It may be one deployment with several job types; it should not
be split into many microservices initially.

```text
Online Auto-oncall Service
  writes immutable facts and run artifacts
             |
             v
Optimization Control Plane
  1. Signal Miner
  2. Root-cause Agent
  3. Candidate/Patch Agent
  4. Replay and Evaluation Runner
  5. Promotion Controller
             |
             +-> GitHub Issue / Draft PR
             +-> Feishu daily/weekly summary
             +-> Base trend projection
```

The online service must not depend on the optimizer being healthy. Optimizer
failure never blocks Jira polling, analysis, internal comments, feedback,
reports, or Base synchronization.

### Canonical entities

Add DB-owned state; do not make GitHub labels, issue text, Base records, or
Codex threads the source of truth.

- `optimization_signals`: one normalized observation from feedback, answer
  comparison, job failure, gate mismatch, or trend regression.
- `optimization_proposals`: deduplicated problem statement, category, affected
  scenario, confidence, evidence keys, owner agent, and lifecycle.
- `evaluation_cases`: immutable replay manifest referencing the original run,
  frozen customer-visible input, attachments, evidence snapshot, model/config,
  and deterministic assertions.
- `evaluation_suites`: versioned cohorts such as Wetech diagnosis, Clinic
  diagnosis, pure Q&A, attachment handling, gates, and report accounting.
- `optimization_candidates`: repository base SHA, isolated branch/worktree,
  bounded files, intended contract change, and generated tests.
- `evaluation_runs`: baseline and challenger outputs, deterministic checks,
  judge results, latency/cost, safety findings, and artifact references.
- `promotion_decisions`: draft-PR, approved, merged, staging, rejected,
  rolled-back, with actor and evidence.

Every entity needs a stable idempotency key. A repeated daily signal should
update an existing proposal rather than create a new optimization task.

### Bad-case taxonomy

The Signal Miner must classify before any code change:

1. `answer_quality`: wrong, incomplete, unsupported, or less useful than the
   stable human answer.
2. `knowledge_or_skill`: missing domain rule, evidence check, or diagnosis
   workflow.
3. `tool_or_data`: provider not called, missing metric, attachment mapping, or
   evidence retrieval failure.
4. `gate_or_policy`: wrong scope, scenario, resolution, or safety decision.
5. `report_or_evaluation`: bad pairing, denominator, eligibility, or judge
   classification.
6. `runtime_infrastructure`: auth, timeout, deployment, DB, or external API
   failure.
7. `non_actionable`: context changed, no stable reference, one-off customer
   preference, or insufficient evidence.

Only categories 1-5 may produce a code/prompt/skill candidate. Infrastructure
creates an operational proposal; non-actionable signals are retained for trend
analysis but do not trigger edits.

### Evaluation contract

- Evaluation input is frozen at the original run boundary. There is no fallback
  to current Jira content, current external metrics, or later support comments
  as agent input.
- Human answers and `/ao` feedback are evaluation-only references and never
  enter the replayed support-agent context.
- Baseline and challenger use the same case manifest, model, reasoning effort,
  tools, timeouts, and evidence snapshot. The candidate change is the only
  intended variable.
- A case can have deterministic assertions in addition to a reference answer:
  expected gate/provider, required evidence checks, prohibited claims, required
  clarification, no external write, and output-shape constraints.
- Judge output is structured and evidence-linked. It cannot override a failed
  deterministic safety or contract assertion.
- Candidate acceptance requires both targeted improvement and suite-level
  non-regression. Single-case improvement is insufficient.
- Replays run with Jira/Feishu/GitHub writes disabled and production-operation
  tools unavailable.

### Autonomy levels

1. **Observe**: existing reports, comparisons, feedback, and dashboards.
2. **Propose**: existing daily GitHub optimization issues.
3. **Prepare change**: agent reproduces, edits in an isolated worktree, adds
   tests, evaluates baseline/challenger, and opens a Draft PR with evidence.
4. **Safe merge**: only allowlisted low-risk prompt/skill changes may merge
   after repeated successful evaluations and required checks.
5. **Staging promotion**: controller deploys and reruns bounded cases, then
   verifies DB, artifacts, live image/env, and Jira idempotency.
6. **Production promotion**: remains human-approved. It also remains subject to
   the single-active-environment and no-customer-visible-write rules.

The first delivery target is level 3. Levels 4-5 require observed reliability
and explicit policy approval; level 6 is not part of the initial design.

## Invariants

- MySQL and immutable run/evaluation artifacts are canonical; GitHub, Feishu,
  and Base are projections.
- Online serving and optimization use separate roles, queues, workspaces, and
  credentials.
- The optimizer never changes the online worker checkout in place.
- Read and evaluation paths never write Jira comments, transition issues, send
  Feishu reports, create production resources, or run production operations.
- Customer-visible replies and production automation are not broadened.
- A generated candidate must declare its bounded change category and files.
- Gate, deployment, external write, credential, and destructive-operation
  changes always require human approval.
- No candidate is accepted without a reproducible baseline and recorded
  challenger result.
- Failed and rejected candidates remain queryable; the agent must not repeatedly
  rediscover the same failed idea.
- Promotion and rollback use repository/app-delivery sources and verify live
  runtime state, not only PR or log status.

## Implementation Steps

### Phase 1: Reproducible evaluation foundation

1. Define the canonical entities and proposal state machine.
2. Materialize immutable `evaluation_cases` from selected historical runs.
3. Build a write-disabled replay runner that can execute baseline and
   challenger from the same manifest.
4. Add deterministic assertions for the recurring historical failure classes:
   provider/tool routing, OP non-Wetech gate, attachment mapping, evidence
   closure, timezone clarification, and report answer pairing.
5. Create versioned scenario suites and a machine-readable evaluation report.

Exit condition: a known historical fix can be demonstrated as failing on the
pre-fix SHA and passing on the fixed SHA without reading current Jira state.

### Phase 2: Agent-generated Draft PR

1. Convert daily report optimizations and direct feedback into deduplicated
   proposals.
2. Let the Root-cause Agent inspect the frozen run, code, tests, and relevant
   historical proposals, then choose one taxonomy category.
3. For allowlisted categories (`prompt`, repo-owned `skill`, narrow `gate`,
   `report/evaluation`), create an isolated worktree and candidate branch.
4. Require the Patch Agent to update the plan/contract first, add a regression
   case, implement the bounded change, and run targeted plus full local tests.
5. Run baseline/challenger suites and open a Draft PR only when acceptance
   policy passes. The PR contains proposal id, evidence cases, metric delta,
   regressions, risks, and rollback instructions.

Exit condition: routine prompt/skill/gate bad cases reach a reviewable Draft PR
without a person reproducing, editing, or assembling evidence manually.

### Phase 3: Staging learning loop

1. Add an allowlisted promotion controller for approved/merged candidates.
2. Publish via the existing app-delivery source workflow.
3. Verify rendered/live image and env, logs, DB state, frozen replay results,
   and bounded live/dry-run cases.
4. Observe the next report windows and compare post-change cohorts with the
   baseline; automatically recommend keep or rollback.
5. Feed accepted and rejected outcomes back into future proposal retrieval.

Exit condition: the agent can prove that a merged optimization improved its
target cohort in staging and did not regress the protected suites.

### Phase 4: Policy-based low-risk autonomy

Only after enough Phase 2-3 evidence, define an explicit allowlist for automatic
merge/staging. Start with repo-owned prompt text or skill guidance that has no
credential, API, gate-boundary, deployment, or write-path changes. Keep all
other categories human-approved.

## Verification Checklist

- Repeated signals deduplicate into one proposal and preserve all evidence.
- Empty/missing fields and context-changed cases do not create edit candidates.
- Frozen replay does not call current Jira or live evidence providers.
- Baseline/challenger differ only by the declared candidate.
- Deterministic contract failures cannot be overruled by the semantic judge.
- Target case improves and protected suites do not regress.
- Repeated evaluation is idempotent and does not duplicate PRs or comments.
- Optimizer crash does not affect the online worker or reporting loops.
- Draft PR contains reproducible commands, artifacts, metric delta, and rollback.
- Staging verification checks app-delivery source, live Deployment/Pod, DB, and
  bounded case artifacts.
- Production remains disabled and customer-visible Jira writes remain absent.

## Non-Goals

- Do not let the daily report Agent directly edit or deploy code.
- Do not treat every `human_better` verdict or human wording preference as a
  product defect.
- Do not use GitHub issues, Base, or Feishu messages as canonical state.
- Do not replay from mutable current Jira tickets or current monitoring data.
- Do not auto-merge gate, credential, deployment, external-write, or production
  operation changes.
- Do not introduce a dependency on the parent repository or Investibook
  runtime/code; integrations remain explicit external APIs.
- Do not expand auto-oncall beyond the approved L1 Q&A/internal-comment scope.