# Auto Oncall Optimizer

An independently deployable internal engineering agent. It receives authorized Feishu `@bot` messages, runs an isolated Codex task against `tidbcloud/aisupport`, and limits all writable tasks to `auto-oncall-services/`.

It is not part of the auto-oncall Jira poller or worker, and `auto-oncall-services` has no runtime dependency on it.

## Commands

- `@bot <request>`: read-only diagnosis. Codex runs in a disposable network-enabled worktree so it can query read-only evidence providers; file changes are not published and the completed worktree is removed.
- `@bot /pr <request>`: create an isolated branch/worktree, let Codex modify and test only `auto-oncall-services/`, then let the orchestrator commit, push, and open a PR.
- `@bot /ticket <ISSUE-KEY> [problem]`: investigate one human-nominated bad case using its latest live answer and a fresh read-only replay. It keeps independent findings separate and may select at most one stable-knowledge, deterministic-query, or locally reproduced fallback-contract finding for bounded repair, automatic merge, staging deployment, replay, Jira refresh, and verified optimization record.
- `@bot /fb-reminder`: in the configured `Auto-Oncall Feedback Hub` group, immediately post the same reminder used by the weekday schedule. It lists successful Auto-oncall tickets from the last 14 days that have no ticket-level `/ao` feedback, groups linked Jira keys by current Assignee, and mentions every Assignee.

Persisted Jira `/ao wrong|revise`, LLM Judge `human_better`, successful-run
Provider degradation/failure, and Router/Final timeout notices are automatic
single-ticket triggers. This same Deployment polls the internal unified feed by
durable candidate ID; `/ticket` remains the explicit path for old cases and reruns.

The controller owns GitHub merge, the existing staging release workflow, and
authorized stable Jira internal-comment replay. It never writes a customer-visible
comment, transitions an issue, uses an admin merge bypass, or performs production operations.

## Runtime contract

- Feishu ingress: outbound long connection; no public callback endpoint is required
- Health: `GET /healthz`, `GET /readyz`
- Admin job lookup: `GET /api/v1/jobs/:id` with `Authorization: Bearer <admin token>`
- State is persisted atomically at `AUTO_ONCALL_OPTIMIZER_STATE_PATH`. Run one replica and mount that path on persistent storage.
- Duplicate Feishu `message_id` values return the existing job instead of starting a second task.
- Interrupted `running` jobs are requeued after restart.
- Every terminal single-ticket job sends a Feishu result, including failures and
  non-repair outcomes. Failed sends remain `pending` and are retried without
  rerunning the optimizer job.
- Failed worktrees are preserved and recorded on the job. Completed worktrees are removed and pruned.
- Before every investigation or repair, the controller explicitly refreshes
  `refs/remotes/origin/main`, verifies the new worktree HEAD against that exact
  remote SHA, and persists it as `repositoryBaseSha` on the job.
- A single-ticket investigation whose JSON shape is valid but whose
  classification, finding, or selected-repair fields conflict gets one read-only correction in the same
  investigator thread. The second invalid result fails closed before repair.
- Provider-execution repairs follow the backend recorded by the fresh replay.
  An Agent9 Clinic replay is owned by the Managed Clinic source under
  `auto-oncall-services/tools/agent9-auto-oncall/managed-clinic/`; it cannot be
  repaired by changing the local Codex SDK compatibility executor
  `auto-oncall-services/src/clinic-evidence-batch.ts`.
- A selected repair whose first controller-owned full test fails gets one
  same-repair-thread validation continuation with the exact bounded failure
  output. It may fix only directly affected tests/contracts; the second failure
  fails closed and preserves the worktree.

## Required configuration

| Variable | Purpose |
| --- | --- |
| `AUTO_ONCALL_OPTIMIZER_LARK_APP_ID` | Self-built Feishu app ID |
| `AUTO_ONCALL_OPTIMIZER_LARK_APP_SECRET` | Feishu app secret |
| `AUTO_ONCALL_OPTIMIZER_LARK_BOT_OPEN_ID` | Exact bot Open ID used for structured mention matching |
| `AUTO_ONCALL_OPTIMIZER_ALLOWED_CHAT_IDS` | Groups where an exact bot mention authorizes any sender |
| `AUTO_ONCALL_OPTIMIZER_ALLOWED_SENDER_OPEN_IDS` | Users allowed to invoke the bot in p2p chats |
| `AUTO_ONCALL_OPTIMIZER_PR_SENDER_OPEN_IDS` | Smaller p2p subset allowed to invoke `/pr` |
| `AUTO_ONCALL_OPTIMIZER_ADMIN_TOKEN` | Bearer token for job lookup |
| `GH_TOKEN` | GitHub token used only by the orchestrator for clone/push/PR |
| `CODEX_HOME` | Codex authentication/config directory |
| `JIRA_USERNAME`, `JIRA_API_TOKEN` | Manually managed read-only Jira identity exposed to Codex |
| `AUTO_ONCALL_DATABASE_URL` | Manually managed URL for a database identity with only `SELECT` on Auto-oncall staging; exposed to Codex |
| `CLINIC_API_KEY` | Manually managed Clinic key exposed to the installed Clinic skill |
| `KUBECONFIG` | Generated observer kubeconfig for staging and optional production contexts |
| `AUTO_ONCALL_AGENT9_BASE_URL`, `AUTO_ONCALL_AGENT9_PROJECT_ID`, `AUTO_ONCALL_AGENT9_API_KEY` | Exact Agent9 Console scope used by the read-only runtime-evidence helper to inspect a correlated Session or Operation |
| `AUTO_ONCALL_OPTIMIZER_NOTIFY_CHAT_ID` | Dedicated outbound group receiving automatic single-ticket terminal results when auto trigger is enabled |
| `AUTO_ONCALL_OPTIMIZER_FEEDBACK_REMINDER_ENABLED` | Enables the shared scheduled/manual AI feedback reminder |
| `AUTO_ONCALL_OPTIMIZER_FEEDBACK_REMINDER_CHAT_ID` | Exact `Auto-Oncall Feedback Hub` chat ID; it must also be an allowed command chat |
| `AUTO_ONCALL_OPTIMIZER_FEEDBACK_REMINDER_TIME_ZONE` | Reminder schedule time zone; staging uses `Asia/Shanghai` |
| `AUTO_ONCALL_OPTIMIZER_FEEDBACK_REMINDER_HOUR` | Local weekday hour for the reminder; staging uses `17` |
| `AUTO_ONCALL_OPTIMIZER_FEEDBACK_REMINDER_LOOKBACK_DAYS` | Successful Auto-oncall run lookback; staging uses `14` |
| `AUTO_ONCALL_OPTIMIZER_CORE_USER_BASE_URL` | Internal Core User endpoint used to resolve Jira Assignees to Feishu open IDs by exact email, with exact-name fallback |

Optional variables include `AUTO_ONCALL_OPTIMIZER_REPO_PATH`, `AUTO_ONCALL_OPTIMIZER_WORKSPACE_ROOT`, `AUTO_ONCALL_OPTIMIZER_STATE_PATH`, `AUTO_ONCALL_OPTIMIZER_CODEX_MODEL`, `AUTO_ONCALL_OPTIMIZER_JOB_TIMEOUT`, and `AUTO_ONCALL_OPTIMIZER_GITHUB_REPO`.

Single-ticket jobs also use `AUTO_ONCALL_OPTIMIZER_ANALYSIS_BASE_URL`. In
staging it points across namespaces to the Agent9-backed main AOS Service,
`http://auto-oncall-services-agent9.auto-oncall-services-agent9.svc.cluster.local`;
the optimizer itself remains in the legacy Kubernetes namespace. They also use
`AUTO_ONCALL_OPTIMIZER_SINGLE_TICKET_TIMEOUT`,
`AUTO_ONCALL_OPTIMIZER_AUTO_TRIGGER_ENABLED`,
`AUTO_ONCALL_OPTIMIZER_FEEDBACK_POLL_INTERVAL`, and
`AUTO_ONCALL_OPTIMIZER_FEEDBACK_BATCH_SIZE`. They run in this same Deployment,
state store, PVC, repository clone, worker, and Codex home; no second service or
CronJob is created. The full contract is
`auto-oncall-services/docs/SINGLE_TICKET_OPTIMIZER.md`.
Staging additionally enables `AUTO_ONCALL_OPTIMIZER_AUTO_MERGE_ENABLED` and sets
the deployment workflow, rollout timeout, and polling interval. The live AOS
metadata `buildSha` is the required deployment proof before post-change replay.

The Codex subprocess receives a minimal environment. Add read-only provider variables explicitly with `AUTO_ONCALL_OPTIMIZER_AGENT_ENV_ALLOWLIST`. GitHub and Feishu orchestration secrets are always removed even if named in the allowlist.

For an Agent9-backed replay, the canonical correlation remains in the replay
payload: `runId` identifies the Auto Oncall run and the platform-neutral
`runtimeTrace.calls[]` records identify every Agent thread used by that run.
When `runtimeTrace.backend=agent9`, each non-null `threadId` is an exact Agent9
Session ID; `result.codexThreadId` is only a legacy/direct-path fallback. The
runtime-evidence helper uses those exact Session IDs, and an exact Operation ID
when needed, against display-safe Console GET endpoints. This operational
correlation is not Provider evidence and does not require platform markers in
portable evidence payloads.

The staging app-delivery template is under `deploy/app-delivery/`. It deploys one replica into `auto-oncall-services-staging`, references the existing manually managed Lark/GitHub Secrets, and requires `auto-oncall-optimizer-secret` containing `AUTO_ONCALL_OPTIMIZER_ADMIN_TOKEN`, `auto-oncall-optimizer-readonly` containing `JIRA_USERNAME`, `JIRA_API_TOKEN`, `AUTO_ONCALL_DATABASE_URL`, and `CLINIC_API_KEY`, plus `auto-oncall-agent9-runtime` containing `AUTO_ONCALL_AGENT9_API_KEY`. The database URL must use a dedicated identity with only `SELECT` on `auto_oncall_staging.*`; do not reuse the service's write-capable database URL. `CODEX_HOME` lives on the optimizer data PVC and must be initialized with a dedicated `codex login`; never seed it by copying another active Codex login. No secret value is stored in this repository.

When the feedback reminder is enabled, this same Deployment schedules it at the configured local hour on Monday through Friday. Startup does not backfill a missed reminder. Scheduled sends use one stable Feishu UUID per local day; exact `/fb-reminder` commands use a stable UUID derived from the source message ID. If there are no targets, no group message is sent. If any current Assignee cannot be resolved to one Feishu open ID, the reminder fails closed instead of silently omitting an `@` mention.

The Pod uses a namespace-scoped observer ServiceAccount for staging. Production is a separate cluster: apply the self-contained `deploy/observer/devops-prod.yaml` namespace and observer RBAC, construct a kubeconfig from that observer token, and store it as key `config` in the optional staging Secret `auto-oncall-optimizer-prod-kubeconfig`. Both identities permit resource reads and Pod logs but not Secrets, exec, or mutations.

The image installs the runtime dependencies for local `clinic-api` and `wetech-op-grafana` skills. Synchronize those private skill directories into `$CODEX_HOME/skills` on the PVC operationally; do not commit their authentication assets. The bundled `runtime-evidence` skill is refreshed by the entrypoint on each start.

## Feishu setup

Subscribe the self-built app to `im.message.receive_v1` through long connection and grant only the message receive/reply/create scopes required by the app. No public callback URL is needed. In an allowed group, an exact structured mention authorizes any sender, including `/pr` and `/ticket`; other groups are ignored and p2p messages retain the sender and PR-sender allowlists. The outbound notification chat is independent from this command-ingress allowlist, so a status-only group does not become a bot command surface.

## Local verification

```bash
go test ./... -race
go vet ./...
go build ./cmd/auto-oncall-optimizer
```