# AGENTS.md

## PingCAP AWS SSO and Feishu Reminders

When expired PingCAP AWS SSO blocks AWS or Kubernetes work on this host:

- First run `aws sso login --sso-session pingcap` without `--use-device-code`
  or `--no-browser`, and allow up to two minutes for automatic browser login to
  complete.
- Retry the original AWS or Kubernetes operation after login succeeds. Do not
  send a reminder merely because login started.
- If standard login fails, has no success signal after two minutes, or requires
  explicit user interaction, send the user a Feishu message with the required
  action and any safe login link or device code. Do not rely only on a Codex
  commentary message, because the user may not see it in time.
- Use device-code or no-browser login only after standard login cannot complete
  automatically or when the user explicitly requests it. Continue the original
  workflow and verify its live result after authentication succeeds.

## Isolation Rule

This directory is a standalone subrepo for auto-oncall services.

- Do not import code from the parent repository.
- Do not depend on parent `packages/`, `services/`, `scripts/`, `go.work`, or workspace configuration.
- Keep contracts, deployment files, runtime code, tests, and documentation inside this directory.
- Integrate with Jira, DevOps, Lark, and other systems only through explicit external APIs.
- Treat Investibook only as a design reference; do not add code, config, deployment, or runtime dependencies on it.

## TypeScript

- Use Node.js 20+.
- Keep runtime dependencies minimal.
- Build with `tsc`.
- Tests should run with `node --test`.

## Minimal Defensive Design

- Do not add defensive machinery by default. Every validation, fallback,
  synchronization contract, finalizer, and fail-closed branch must protect a
  concrete boundary or observed failure mode that cannot be handled more
  simply.
- Keep hard validation at real trust boundaries: authorization, tenant/org
  isolation, external side effects, and the structural integrity needed to
  parse or route data. Do not treat normal semantic differences between agents
  as trust-boundary violations.
- An agent with broader scope may independently interpret and combine child
  results with its other context. Do not require its output, status, completion
  judgment, or evidence set to equal or be a subset of a child handoff merely
  to keep two agent outputs synchronized.
- Give each fact one owner. Pass child results to the coordinating agent as
  inputs; do not create parallel representations and then add cross-agent
  validation to force them to agree.
- A validation failure must be handled in proportion to the invalid field. Do
  not discard otherwise valid evidence or reject a whole result when the
  conflicting field can be ignored, normalized, or reported directly.
- Prefer the smallest observable design. Add retries, fallback paths, schemas,
  checkpoints, or compatibility layers only when current evidence shows they
  are necessary, and remove obsolete defensive paths when their boundary no
  longer exists.

## Deployment

`deploy/app-delivery/` is the source template for app-delivery changes.

- Make deployment changes here first, then use the repository GitHub Action to
  create the `tidbcloud/app-delivery` PR.
- The staging app-delivery sync workflow is
  `.github/workflows/open-auto-oncall-app-delivery-registration-pr.yml`.
- Do not hand-edit `tidbcloud/app-delivery` as the only source of a deployment
  change; if a hotfix is made there, mirror it back into this directory.
- Current staging has two Kubernetes applications. The
  `auto-oncall-services-agent9` application is the sole active poller, worker,
  report, Jira-write, notification, and Base-sync owner. The
  `auto-oncall-services` application retains only the Recent Optimizer and
  Codex-history auxiliary jobs plus their ConfigMap and PVCs; it must not
  recreate the retired legacy AOS Deployment or Service.
- Jira, Clinic, Agent9, and Codex credentials are manually-created Kubernetes
  Secrets in the applicable `auto-oncall-services-agent9` or
  `auto-oncall-services-staging` namespace.
- The app-delivery template should reference these Secret names and keys from
  workloads, but should not include real Secret values.
- Do not add `ExternalSecret` resources for these credentials unless the user
  explicitly changes the secret-management model.

### Agent9 Auto-oncall Skill and Managed Tool ownership

- This repository owns the Auto-oncall Skills and the Managed Clinic, Wetech,
  and Manus package source, immutable bundle/catalog pins, publication, exact
  Agent installation orchestration, and replay verification.
- A Skill is versioned reasoning or query-planning guidance; a Managed Tool is
  an executable Provider boundary. Installing a Skill does not grant Provider
  execution authority. Keep the Agent9 installations exact by role:
  - Evidence Agent: `evidence-query-planning` plus `clinic-api`,
    `wetech-op-grafana`, and `manus-op-grafana`, with matching Provider Managed
    Tools. The shared Skill has no Managed Tool and grants no execution
    authority. Evidence Agent must not install or load `op-diagnosis`, `tidb-latency-diagnosis`, or `tidb-query-tuning`.
  - Scenario Diagnostic Agent: only `op-diagnosis`,
    `tidb-latency-diagnosis`, and `tidb-query-tuning`. It may read the
    applicable diagnosis Skill instructions and references, but must not have
    a Provider Skill or Managed Tool and must never query Provider data.
  - Scenario Router and Final Response Agent: no Evidence or Scenario
    Diagnostic Skill installation. They receive only their contract-owned
    inputs and keep their existing role boundaries.
- Skill publication and Agent installation are separate operations. The
  AOS-owned release must first publish every immutable Skill version, ensure
  each owning Agent is pinned to its exact set, and then remove stale
  cross-role installations. Do not preserve a diagnosis Skill on Evidence or
  a Provider Skill on Scenario Diagnostic as a compatibility fallback.
- For `agent9-internal-prod`, this repository owns the Kubernetes ConfigMap
  named `agent9-managed-tool-loader` and only the StatefulSet Pod-template
  annotation `auto-oncall.pingcap.com/managed-tool-bundle-sha256`. Keep their
  source under `deploy/agent9-managed-tools/` and update them through the
  AOS-owned release command.
- Agent9 owns the generic loader implementation and fixed ConfigMap reference,
  sandbox execution, Organization Auth injection, installation authorization,
  and audit. Registration or catalog publication must not grant installation
  or execution authority.
- After the one-time ownership migration, publishing a new Auto-oncall Skill
  or Managed Tool version must not modify Agent9 source, copy package pins into
  Agent9, or require an Agent9 pull request. Reloading an unchanged Agent9 image
  from the AOS-owned annotation is an AOS package rollout, not an Agent9 code
  release.
- Do not give both repositories ownership of the same Kubernetes field. AOS
  release automation may change only its ConfigMap and annotation; every other
  Agent9 workload field remains Agent9-owned.

### Wetech Managed Tool local verification

Before publishing or deploying a Wetech Managed Tool login or navigation
change, run the current source in the pinned Playwright Docker image from this
directory:

```bash
npm run simulate:wetech-managed-tool
```

- The default input is one bounded five-minute `vector(1)` query for Wetech
  `SaaS01`. Use `-- --args <absolute-path>` only for another bounded Tool
  `args` object.
- Treat the check as passed only when stdout contains a successful compact
  Provider result with actual query data. Container startup, browser startup,
  or a successful Turn without Provider data is not a pass.
- Reuse the simulator's private mode-`0600` browser state under
  `~/.cache/wetech-op-grafana/managed-tool-local/` for ordinary iterations so
  P81 is not repeatedly logged in and rate-limited. When authentication or
  login navigation changes, run one controlled validation with an isolated
  empty state, and do not blindly retry a failed login.
- This local check exercises the exact `index.mjs`, stdin payload, pinned
  browser, login flow, Grafana API, and compact result. It does not verify
  Agent9 authorization or audit, E2B lifecycle or restricted egress, Drive9,
  installation, or the live immutable package pin.
- Use the local check to reject bad hypotheses before a release. After a local
  pass, publish once and perform one bounded post-deployment Managed Tool
  canary; do not deploy each debugging iteration.

## Auto Analysis Automation

The automatic Jira poller, worker, Jira internal comment writer, and Feishu
notification path are production-affecting automation. Treat config changes to
this path as rollout changes, not local-only toggles.

### Required Core Contract Reading

Before changing scenario classification, resolution routing, provider
selection, missing-context handling, diagnostic execution state, final answer
completion, risk handling, or customer-visible publication, read
`docs/AUTO_ONCALL_CORE_CONTRACT.md`. It is the canonical field-ownership and
stage-boundary contract for these paths.

Before changing Final Response output, customer/internal message parsing,
answer fingerprinting or persistence, Jira response delivery, replay or
continuation output, Daily/Recent Evaluation response consumption, or the
Single-ticket Optimizer response context, also read
`docs/DUAL_MESSAGE_CONTRACT.md`. It is the canonical lifecycle contract for the
paired `customerReplyMarkdown` customer message and
`internalHandoffMarkdown` internal message.

- Keep that document and the applicable contract tests synchronized with every
  core behavior or field change.
- Do not introduce a second field for a fact already owned by the documented
  contract. Derive stage decisions from the owning field instead.
- Historical persisted payloads may be read compatibly, but obsolete fields
  must not remain in newly generated classifier, workspace, Gate, execution,
  evaluation, or publication payloads.

- This system is scoped to L1 oncall support for Q&A-style tickets. Do not
  broaden automation to non-Q&A tickets, non-L1 owners, customer-visible
  replies, issue transitions, or production operations without an explicit new
  design and user approval.
- Automatic polling is enabled only when `AUTO_ONCALL_AUTOMATION_ROLE=active`
  and `AUTO_ONCALL_POLL_ENABLED=true`.
- Do not run staging and prod as active automation at the same time. Before
  enabling prod automation, disable staging automation first and confirm there
  are no new `queued` or `running` jobs.
- Prod must remain `AUTO_ONCALL_AUTOMATION_ROLE=disabled` and
  `AUTO_ONCALL_POLL_ENABLED=false` unless the user explicitly asks for the
  prod cutover.
- Automatic mode must only write or update Jira internal comments. It must not
  write customer-visible comments, transition or close issues, or execute
  production operations.
- The Jira internal comment idempotency key for live initial analysis is
  `auto-live:v1:<issueKey>:initial` and intentionally does not include the
  environment.

### Gate Boundaries

Core structure:

- Poller: only non-business gates, then enqueue.
- Worker: business gates, then live analysis.

Initial poller gates:

- Done Gate
- Created Cursor Gate
- Minimum Issue Age Gate
- Duplicate Initial Job Gate
- Optional Jira Issue Allowlist

Worker gates:

- L1 Oncaller Gate
- Jira Context Gate
- Auto Coverage Gate

Auto Coverage Gate internals:

- Scenario Classifier
- Enabled Coverage Gate

Auto Coverage decisions:

- `enabled`
- `disabled`

Rules:

- Scenario Classifier is only inside Auto Coverage Gate.
- Done Gate and Optional Jira Issue Allowlist belong only in the initial poller
  path. Continuation discovery must include recently updated Done issues and
  must not apply the autonomous issue allowlist.
- L1 Oncaller Gate must run before Auto Coverage Gate, workspace creation,
  Codex execution, or Jira comment writes for `initial` jobs. An exact internal
  ADF `@bot` continuation is an explicit human invocation and bypasses Done,
  the autonomous issue allowlist, L1 Oncaller, and Enabled Coverage filtering
  after trigger validity checks pass.
- Jira Context Gate runs after L1 Oncaller Gate for `initial` jobs and before
  Auto Coverage, workspace creation, or Codex execution. It skips only when the
  JSM request-comment API returns HTTP 404 with the exact
  `sd.customerview.error.vpOriginMissing` key. Project, issue type, request
  channel, and an empty Request Type field are not sufficient to skip a ticket;
  other 4xx, 5xx, network, and unknown failures retain their existing failure
  and retry semantics.

### Analysis Contract

- Jira project keys `APID` and `NAID` are authoritative OP environment
  boundaries and take precedence over organization- or cluster-shaped ticket
  fields. Other project keys do not select an environment: a non-empty typed
  `orgId` means Cloud and a missing typed `orgId` means OP.
- OP TiCDC diagnostics never require TiDB Cloud Dataflow changefeed metadata.
  Exact `Wetech` or `apac-test` Account Alias may use the read-only Wetech
  metrics/log route; Manus and other OP accounts have no automatic TiCDC
  Provider. Do not fall back between OP and Clinic.
- Dry-run and live analysis must share the same support-analysis skills,
  reply-style expectations, and output semantics. Do not create a separate live
  prompt path that produces a different answer style.
- `customerReplyMarkdown` means the simulated L1/oncall Jira reply draft for
  human review. Live automation still writes it only as an internal note.
- Dry-run/replay workspaces must filter prior L1/support/internal/bot comments
  and preserve the historical replay cutoff so future answers cannot leak into
  the simulated reply.
- Live initial workspaces use the complete current paginated Jira comment
  timeline. Continuation workspaces use that same ordered timeline only through
  the exact triggering comment inclusive; comments created after the trigger
  must not be included.
- An exact internal ADF mention is an explicit continuation request, including
  on a resolved or closed issue. Done, the autonomous issue allowlist, L1
  Oncaller, and Enabled Coverage may not suppress the internal continuation
  response. Safety classification still controls whether the response is
  `internal_only` or `human_required`.
- Continuation resumes the persisted Codex thread when present. If the initial
  run completed through diagnostic preflight without creating a thread, it may
  start one bounded continuation thread from the canonical Jira timeline and
  persist that thread for later turns.

### Diagnostic Agent Architecture

Use these names and ownership boundaries consistently in code, prompts, tests,
traces, and documentation:

1. **Scenario Router** has one persistent thread and two phases. Before evidence,
   it selects bounded scenarios and fixes the canonical entity, incident window,
   and provider. After every selected scenario returns, it owns the one
   ticket-level diagnosis: deduplication, conflict resolution, evidence
   precedence, leading cause, material alternative, mitigation, severity,
   requested context, and human handoff. It never queries evidence or reopens
   provider output.
2. **Evidence Agent** plans neutral, read-only evidence acquisition for exactly
   one scenario and provider. It does not judge identity, causality, mechanism,
   alternatives, severity, safety, scenario status, or human handoff. It loads
   `evidence-query-planning` and the selected Provider query Skill, and never
   loads a diagnosis Skill.
3. **Provider Executor** is runner-owned deterministic code, not an agent. It
   executes the Evidence Agent's validated plan and materializes compact facts,
   query coverage, errors, and canonical evidence references.
4. **Scenario Diagnostic Agent** receives the runner-owned Evidence Agent
   handoff and owns the diagnosis for exactly one scenario. It decides whether
   one precise, load-bearing refinement could still change that diagnosis. It
   may load the applicable diagnosis Skills owned by this repository, but it
   never calls a provider, loads a Provider Skill, or receives a Managed Tool.
   It may request original cutoff-filtered ticket images by their supplied refs
   for same-scenario, same-replay visual verification. The runner supplies only
   those selected images in the same session; this is not Provider acquisition.
5. **Final Response Agent** receives only the Scenario Router's canonical
   ticket-level decision and improves its wording, organization, language, and
   length. It does not receive scenario packets or provider output and must not
   diagnose, deduplicate, resolve conflicts, change causal strength, choose a
   mitigation, alter severity, or change human ownership.

After the Final Response Agent, only structural result parsing and true safety
or publication boundaries may run: secret or internal-data disclosure,
cross-tenant content, destructive or privileged-action promises, unauthorized
commercial or policy commitments, and customer-publication eligibility. SQL
Digest handling, OP identifier handling, transient fault-domain completeness,
workload attribution, monitoring semantics, capacity diagnosis, mitigation,
severity, and handoff are diagnostic semantics and must be settled by the
Scenario Router before the Final Response Agent. Do not add a post-final Agent
correction or a second wording/diagnosis turn.

Clinic, Wetech, and Manus use this same orchestration. Their differences are
limited to the query interface and its trust boundary: exact target shape,
provider skill/helper, plan and result schema, supported operations,
authentication/readiness, and execution concurrency. Independent Clinic
operations may run concurrently inside one deterministic batch; Wetech and
Manus operations run strictly serially. These transport differences must not
change agent ownership or introduce provider-specific diagnosis flows.

For one scenario, reuse the same Evidence Agent thread across authorized
refinement rounds so it retains its provider skill and already supplied compact
context. Reuse the same offline, read-only Scenario Diagnostic Agent session
within that scenario and replay; retain already read diagnosis Skills and load
only newly applicable references as needed. Never reuse a session across replays.
A refinement round is allowed only for one newly discovered
load-bearing question with an observable stop condition; helper retries,
restating existing facts, and already-known small supporting queries remain in
the current batch.

Authentication/bootstrap and provider execution are runner operations. Supply
the Evidence Agent with the exact current-run target, absolute window, auth
reference, and bounded objective before it starts. It may read only
`evidence-query-planning`, its selected provider skill, and the current scenario
workspace; it must not scan repositories,
other runs, sibling `/tmp` workspaces, runner source, schemas, validators, or
tests.

Do not introduce alternate role names such as Evidence Acquisition Agent,
Evidence Planner, evidence child, scenario parent, scenario coordinator, or
scenario judge. `Judge` is reserved for evaluation. Existing versioned wire
identifiers such as `auto-oncall-diagnostic-child-evidence-v2` and
`child-evidence-handoff.json` are compatibility names only; they do not define
an agent role or ownership boundary.

### Isolated Full Replay

For operator/Codex full ticket replays, including post-deployment verification,
use the one-off runner from this directory:

```bash
npm run replay:isolated -- --issue TCID-13132
```

- Full replay means the production analysis entrypoint, canonical Jira cutoff,
  real Agent9/Clinic/Wetech/Manus execution, diagnosis, Final Response, and Judge
  in `mode=read`. It is not frozen-evidence Scenario Diagnostic simulation.
- For a quality-first read replay, `--no-diagnostic-timeout` sets the API's
  `noDiagnosticTimeout=true`. Diagnostic deadlines log `diagnostic_would_timeout`
  without aborting; default analysis is unchanged. The isolated Job and collector
  still have a one-hour limit, and Provider/auth/network operation limits remain.
  Classifier and Judge remain outside this diagnostic-only policy.
- Run it in an independent Kubernetes Job, pinned to the Ready deployed image
  digest. Do not submit a long `/v1/tickets/analyze` request to the rotating
  business Deployment for a new operator replay, or copy its Service labels,
  background workers, pollers, or probes to the replay Job.
- See `docs/ISOLATED_REPLAY.md` for context selection, collection and publication.
  On client disconnect, use `--collect <job-name>` to collect the original run;
  do not submit a duplicate analysis. Never auto-retry an unknown outcome.
- Export the full run, frozen `read-publication.json`, summary, and logs before
  cleanup. Keep failed or uncollected Jobs for inspection. A completed process
  is not a passed diagnostic criterion; inspect execution, query attempts,
  answer fingerprint, and evaluation for the exact case.
- If publication is authorized after verification, restore the exact exported
  run to the current service's runs directory and use `POST /v1/tickets/publish-read`.
  Do not launch `mode=write` analysis to recreate a lost result.
- The existing automatic optimizer HTTP caller is not migrated by this script.
  Its isolation requires a separate controller change; do not report automatic
  optimizer jobs as isolated until that integration is implemented.

### Local Scenario Diagnostic Regression

Before deploying a Scenario Diagnostic Agent prompt or SOP change, prefer a
local real-model regression when a known replay has both its frozen
`diagnostic-scenario-route.json` and runner-owned
`child-evidence-handoff.json`. Run from this directory:

```bash
npm run simulate:scenario-diagnostic -- \
  --route <diagnostic-scenario-route.json> \
  --handoff <child-evidence-handoff.json> \
  --scenario <scenario-id> \
  --trials 3
```

- The simulator builds the production Scenario Diagnostic finalization prompt,
  uses the production response schema and parser, and starts a fresh tool-free,
  read-only model execution for each trial. Use `--include-decisions` only when
  the full validated decisions are needed; the default output is the compact
  status and refinement summary.
- For image-context regressions, use `--ticket-images --initial-context` to
  offer originals from the paired replay and run the production initial
  evidence-request prompt before same-thread finalization. Images are delivered
  only on request. The initial request is not executed against a Provider; the
  original frozen handoff remains the evidence input. Use `--output <path>` to
  retain all trials, including failures. A finalization-only trial does not
  recreate the initial turn's ticket context or timezone.
- Treat the frozen route and handoff as one paired artifact from the same
  scenario round. Do not edit them to manufacture a pass or combine artifacts
  from different runs, scenarios, or refinement rounds.
- Use the local result to test only the Scenario Diagnostic Agent's diagnosis
  or refinement choice. It does not verify Router selection, Evidence Agent
  planning, Provider execution, runner window expansion, end-to-end budget, or
  final ticket synthesis, and it must not query a Provider or write Jira,
  database, or Feishu state.
- Require the trials to satisfy the bounded source-case criterion before
  deployment. A successful local regression reduces prompt-iteration deploys;
  it does not replace one post-deployment `mode=read` replay that verifies the
  changed owner and its downstream integration on the live merged-main image.

### Optimization Agent Ownership

- Every quality optimization must first identify the earliest failing stage
  and the Agent or deterministic component that owns the missing fact or
  decision. Change that owner; do not repair an upstream evidence, product
  knowledge, applicability, diagnosis, mitigation, severity, or ownership gap
  by adding instructions to a downstream formatting or publication stage.
- Bind a Provider-execution repair to the backend used by the fresh replay.
  For `backend=agent9`, the selected Managed Tool source is the Provider
  Executor owner; the local Codex SDK compatibility helper is not an
  implementation target for that live failure. A runner materialization or
  handoff defect may still belong to its exact downstream owner. Do not modify
  both executors merely to keep duplicate implementations visually aligned.
- Preserve the Diagnostic Agent Architecture above. Scenario Router owns
  ticket-level applicability and synthesis; Evidence Agent owns neutral plans;
  Provider Executor owns execution; Scenario Diagnostic Agent owns one local
  diagnosis; Final Response Agent owns wording and organization only. An
  optimization must not expand one role to redo another role's decision.
- Repository-owned product knowledge belongs under `product-knowledge/`, not in
  response-format instructions. Its `index.ndjson` has one searchable topic
  record per line. Primary Analysis uses bounded `rg` results; the runner gives
  the Scenario Router a bounded keyword-recalled candidate set, and the Router
  alone selects semantically applicable topics. No Agent may be required to
  read the full index or all topic files.
- Contract tests for an optimization must prove both sides of the boundary:
  the owning Agent receives the required input and downstream non-owners do not
  receive or reinterpret it.
- Primary Analysis is the semantic owner for the direct path selected by a
  `direct_answer` classification or a passed direct-artifact gate. It may apply
  applicable product knowledge, official documentation, supplied ticket
  evidence, and direct-artifact reasoning, but it must not query a diagnostic
  Provider or take over the Scenario Router pipeline.
- Response-format instructions own presentation only: response language,
  organization, length, Jira-safe Markdown, and audience projection. They may
  keep one presentation rule for an already-decided customer-versus-PingCAP
  responsibility boundary, such as omitting Support-queryable data requests
  from the customer message and placing the Support action in the internal
  message. They must not decide evidence ownership, Provider capability,
  diagnosis, mitigation, severity, `human_required`, or production approval.

### Continuous Improvement and Evaluation

- Before changing report comparison, problem selection, Recent Evaluation,
  Focus lifecycle, candidate gates, or optimization attribution, read
  `docs/AUTO_ONCALL_EVALUATION.md`. It is the canonical lookup for the current
  evaluation data flow, state transitions, criteria, and failure semantics.
- Before changing the feedback-triggered or manual single-ticket bad-case
  investigation, durable terminal notification, deterministic repair gate, or its integration with the external optimizer
  Deployment, read `docs/SINGLE_TICKET_OPTIMIZER.md`. Keep this lane independent
  from Recent Focus and do not turn semantic or cross-case questions into a
  one-ticket engineering repair.
- In a single-ticket investigation, `chain` and every non-historical finding
  describe current main plus the fresh replay. Keep independent knowledge,
  Provider-query, orchestration-fallback, diagnostic-strategy, external-transient,
  and historical-only findings separate; at most one finding may be selected for
  automatic repair. Stable official product knowledge, deterministic local query
  contracts, and currently reproduced local fallback contracts may be selected.
  Diagnostic strategy, external transient failures, roadmap/commercial claims,
  and global timeout/model/reasoning/retry changes may not. The controller may ask the same read-only
  investigator for one classification-only correction when those fields
  conflict; a second mismatch must fail before any write Agent or PR action.
- Before creating any optimizer repair worktree, explicitly fetch the remote
  default branch into `refs/remotes/origin/main`, create the worktree from that
  exact ref, verify its `HEAD`, and persist the selected base SHA. A cached
  local `main` branch is never a repair base.
- Runtime candidate deduplication uses final `failureCodes` only to decide whether
  to start the optimizer. The investigator must receive and inspect the full ordered
  `providerQueryAttempts`; a corrected intermediate failure, supplemental-only
  failure, or historical failure absent from the fresh replay is not a current finding.
- Before tracing that chain, require one explicit independent review of the exact
  nominated AI answer, the complete same-stage human answer, the version-bound
  human feedback, and the fresh Judge result. The controller must bind the nominated
  and fresh fingerprints. Treat Judge as non-authoritative for diagnosis and chain
  classification, but preserve its verdict as the current answer's evaluation result.
  If the versions differ, old feedback remains only on the nominated historical
  answer; record independently whether that problem is reproduced by the fresh answer.
- After a selected single-ticket repair, the controller may return one exact
  full-test failure to the same repair thread. That continuation may only finish
  tests, snapshots, inventories, or generated contracts directly affected by
  the selected diff. It may not change the finding, owner, behavior scope, or
  implementation strategy. A second validation failure must fail closed and
  preserve the worktree; never dismiss a diff-caused failure as an unrelated
  environment problem without proving the same failure on untouched main.
  The continuation must retain every admitted changed path and may add only
  directly affected files below `auto-oncall-services/test/`.
- Single-ticket automatic candidates include exact-answer `/ao wrong|revise`,
  LLM Judge `human_better`, successful-run Provider degradation/failure, and
  Scenario Router or Final Response timeout notices. `/ticket` selects the
  current latest answer; feedback and automatic signals remain bound to their
  persisted job/run and answer fingerprint. All entries share the same fresh
  replay, investigation, repair, and completion state machine after selection.
- Keep `/ao wrong|revise: reason` as the canonical Jira feedback prompt. The
  parser also accepts `:`, `：`, `.`, `。`, and legacy whitespace-only reason
  separators, but the command must remain anchored to one complete trimmed line
  beginning with an exact `/ao` and action token.
- An admitted single-ticket PR requires no human review: wait for CI, merge
  without an admin bypass, deploy a main SHA containing the repair, and verify
  that exact SHA from the live analysis service before replay. Run one
  post-deployment `mode=read` verification. On pass, publish that exact frozen
  read result through `POST /v1/tickets/publish-read`; do not start a second
  `mode=write` analysis replay merely to persist it. Verify the same issue key,
  read run id, answer fingerprint, stable public=false Jira internal comment,
  database run, evaluation, and applicable Base write, then advance the exact
  problem-level optimization record to `optimized_verified` through a separate
  merged status PR. A missing or invalid frozen read artifact is a publication
  failure and must not silently fall back to a new analysis replay. One
  deployed-replay continuation is allowed only for the same finding, earliest
  owner, and implementation targets; a new stage/owner, broader scope,
  external uncertainty, or second failure must notify a human.
- A Recent Focus represents one bounded, reusable problem to improve. It does
  not represent complete resolution of every problem present in its tickets.
- Every newly created Focus must have exactly one blocking criterion. Other
  findings are preservation checks or later Focus candidates, not permission
  to broaden the current attempt.
- Automatic Focus repair is limited to simple shared problems: one common
  earliest break, one evidence-chain stage, and ordinary unprotected
  implementation targets. An independently verifiable prerequisite may become
  another Dependency Focus, including below an existing Dependency Focus, only
  when the same current target replays share its earliest evidence-chain break
  and it makes monotonic product progress: either a forward runtime stage
  transition or one reusable per-case trace/output artifact that replaces a
  generic failure with a stable bounded owner category. It must bind to the
  parent's blocking criterion and use a distinct Focus and criterion id from
  every ancestor. Do not impose an arbitrary dependency-depth limit.
- A prerequisite is not independent when it uses the same current target set,
  evidence-chain stage, and implementation owner as the parent repair. A
  diagnostic or observability step needed only to choose that repair stays in
  the parent Focus: use the ordinary one-stage repair path when it can satisfy
  the parent criterion, otherwise persist a parent `human_handoff`. Do not
  create a Dependency Focus merely because the same owner needs a more precise
  failure category before it can be repaired.
- A unique earlier blocker on exactly one target is not a Dependency Focus.
  Reuse the Focus Optimizer's read-only planning session as the bounded manager:
  it may request one inline repair only when current source proves one exact
  ordinary unprotected owner at that stage. The patch Agent must change only
  that owner and may not add ticket-specific data, target guessing, fallback
  discovery, credentials, deployment, optimizer changes, or a simultaneous
  parent repair. If no safe direct repair exists, or the same case/stage attempt
  did not make the required transition, persist a controller-bound case blocker.
  On the same base and failure signature, do not retarget that case or count it
  as passed. Continue the Focus only when at least two other target replays are
  unblocked; otherwise hand the Focus to a human.
- A prerequisite that only changes optimizer planning, validation,
  protected-path admission, authorization, scheduling, or another workflow
  mechanism is not product progress. Persist it as `human_handoff`; do not
  create an automatic meta-Dependency Focus or start the write Agent. Apply the
  same handoff to a repair plan that requires a protected implementation target
  or spans evidence-chain stages. The handed-off Issue remains open for a human
  and no longer occupies the one-active-Focus slot.
- When a Dependency Focus reaches verified `ready_to_close` or `closed`, resume
  its direct blocked parent as Active and run a fresh parent baseline. Do not
  infer that the parent criterion passed, and do not close the parent merely
  because the dependency completed.
- Three consecutive complete rejected/inconclusive attempts with no confirmed
  blocking improvement that rotate across at least three implementation
  targets are implementation churn, not permission for more automatic
  attempts. Persist them as `human_handoff`. This is not early retirement: the
  Issue remains open and its evidence is retained for a human owner.
- Run an Active Focus in up to three scheduled attempt slots per Shanghai day,
  while keeping new Focus selection to one post-report slot per day.
- Do not retire a reproducible Focus merely because three attempts failed.
  Long-no-progress retirement requires at least 7 days, 10 complete
  challenger/gate/attempt-review attempts, and 6 consecutive attempts without
  confirmed blocking improvement. A multi-blocking legacy scope or a fresh baseline that
  no longer reproduces on two targets may retire earlier.
- One bad ticket may expose multiple independent problems. Meeting a Focus's
  acceptance criteria proves only that the selected problem was improved; it
  must not mark the whole ticket as fully optimized or suppress unrelated gaps
  found in the same ticket.
- Focus consumption and deduplication must operate at problem-signal
  granularity, not only at ticket, report-case source reference, or issue-key
  granularity. A solved problem from a ticket may be reused as evidence while a
  different unresolved problem from the same ticket remains eligible later.
- A Focus candidate rejected by its fresh creation baseline must not consume
  its source problem signals. Persist bounded rejection evidence and supply it
  to later selection runs.
- When the target problem refs and control issue keys are unchanged, later
  selection must not repeat the same rejected group. New problem refs or a
  different target/control group remain eligible.
- After an optimization becomes effective in the auto-oncall runtime, rerun
  every Jira case used as evidence for the changed problem with the applicable
  dry-run/replay cutoff. If the replay produces the corrected analysis, write
  that draft through the historical owner job's stable auto-oncall internal
  comment idempotency key. When an internal mapping already exists, update that
  exact comment in place. When no internal mapping exists, including when the
  historical answer being evaluated is a customer-visible `public=true`
  comment, `mode=write` must still create one independent `public=false`
  internal comment and persist the mapping; later write replays update that
  internal comment. Never use or overwrite the public comment ID, create a
  duplicate internal comment, or turn this refresh into a customer-visible
  reply.
- Generating a dry-run preview and refreshing Jira are separate completion
  steps. A successful replay does not prove that Jira was updated: use the
  authorized write path and verify the existing internal comment by comment ID
  or idempotency key. If the Jira refresh cannot be completed, record it as an
  unresolved follow-up instead of claiming the case refresh is complete.
- For a user-authorized one-time case recovery or correction, when a
  case-specific human-reviewed or otherwise verified draft/body already exists
  and the existing internal comment ID or stable mapping is known, update that
  exact Jira internal comment in place with a direct Jira API `PUT`. Do not add
  a publication endpoint, ledger, schema, or atomic transaction for this
  recovery, and do not rerun analysis to regenerate the verified body.
- Before the direct recovery write, verify the issue key, exact comment ID, and
  `public=false`. Do not create a duplicate comment, write a customer-visible
  comment, or transition the issue. After the write, read back the same comment
  ID and verify `public=false` and the expected body. Preserve existing DB run,
  evaluation, and Base history without rewriting it or claiming atomic
  consistency; record the direct Jira recovery and any resulting view
  differences in the case-refresh evidence.

### Optimization Records

- Recent Focus is not the only source of optimization work. Changes made by a
  separate skill, repository, agent, manual workflow, or other external process
  may be used as optimization evidence, but this repository must not import or
  depend on that external implementation.
- Every optimization intended to affect auto-oncall quality status must have
  one dedicated record under `docs/opts/`, named
  `YYYY-MM-DD-<short-slug>.md`. Do not combine unrelated optimizations.
- Pure replay and evaluation mechanics are not optimizations and do not require
  a `docs/opts/` record. This includes case reruns, historical input filtering,
  replay request parameters or endpoints, evaluation-only controls, and
  validation-only executions when they do not change live auto-oncall analysis
  quality or capability. Keep their evidence in the replay or evaluation
  result. If one task also contains a live quality optimization, create a record
  only for that bounded optimization.
- The agent or workflow performing the optimization owns creating the record
  when work starts and updating it after implementation, deployment, replay,
  or later verification. Commit history, chat history, weekly-report inference,
  and Recent Focus alone are not substitutes for this handoff.
- Each record must include the optimization id, owner and source reference; one
  bounded problem or capability; implementation/effective status; candidate
  problem signals and ticket keys when known; per-problem status (`candidate`,
  `optimized_unverified`, `optimized_verified`, or `ineffective`); verification
  evidence and timestamp; unresolved gaps; and the next validation action.
- Every `docs/opts/*.md` file must contain exactly one
  `auto-oncall-optimization-record` HTML comment with strict
  `auto-oncall-optimization-record-v1` JSON. The human-readable prose explains
  the change; the JSON is the machine-readable source consumed by Focus. Each
  problem entry must have a stable `problemId`, status, applicability, signal
  matchers, `observedThrough`, timestamps, attempted action, and evidence URL.
  Do not derive automation state by scraping prose or Markdown tables.
- Registering a change is not proof of improvement. Use
  `optimized_unverified` only when the change is available to auto-oncall, and
  `optimized_verified` only after a case-specific post-change replay or
  equivalent evaluation passes the stated problem criterion.
- Choose verification by the bounded problem, not by the fact that a PR was
  merged. A change that can alter diagnosis reasoning, evidence use, answer
  semantics, or a known case outcome requires case-specific replay and the
  applicable Jira internal-comment refresh before `optimized_verified`.
- A pure runtime or operational change that does not alter diagnosis capability
  or case conclusions—such as CPU/memory reservation, scheduling, process
  startup, transport implementation, or cache reuse—does not require a Jira
  case replay or comment refresh. It may become `optimized_verified` from
  equivalent runtime evidence that directly passes its bounded criterion, such
  as successful scheduling under the real constraints or a live request that
  proves the new transport path. PR, merge, deploy, or Ready state alone is not
  sufficient evidence.
- Split mixed diagnostic and operational work into separately verifiable
  problems or records. Operational records must use narrow matchers and may not
  suppress unrelated diagnosis-quality signals.
- Record status at problem-signal granularity. One optimization may affect many
  tickets, and one ticket may contain problems with different statuses. Never
  promote the whole ticket because one problem was verified.
- Jira internal comments and optimization records serve different purposes.
  The refreshed Jira comment is the current case-level analysis for humans and
  later live continuations; `docs/opts/` is the durable source for whether each
  bounded problem is `candidate`, `optimized_unverified`,
  `optimized_verified`, or `ineffective`. Keep the Jira body concise and keep
  detailed status, evidence, timestamps, and remaining gaps in the optimization
  record.
- Focus selection, deduplication, and weekly optimization attribution must
  consume `docs/opts/` at problem-signal granularity. An
  `optimized_verified` problem may suppress only matching stale signals, never
  the whole ticket; newer evidence that reproduces the problem remains eligible
  as a regression. `optimized_unverified` remains eligible for verification,
  while `ineffective` must prevent repeating the same failed action without
  suppressing the unresolved problem.
- A newly generated Focus must cite the selected target problem-signal refs in
  `sourceIssueUrls`. Exact problem refs, not the parent report-case ref or Jira
  key, are the durable consumption unit. Legacy report-case refs may remain as
  historical evidence, but must not be emitted for new target selections.
- If an external optimization is outside the current task scope, only define
  or consume its `docs/opts/` handoff record. Do not inspect, modify, deploy, or
  claim results for the external system unless the user explicitly places it
  in scope.

### Rollout Verification

For any poller, gate, worker, Jira write, or notification config change:

1. Update `deploy/app-delivery/` first and create/merge the corresponding
   `tidbcloud/app-delivery` PR through the repository workflow.
2. If a temporary live `kubectl` patch is needed, mirror the same change back
   into `deploy/app-delivery/` and remove temporary Deployment env overrides
   once the template PR is merged.
3. Verify the rendered/live pod environment, not only the ConfigMap source.
4. Verify logs for poller and worker outcomes.
5. Verify DB state for job counts, latest jobs, and poll cursor movement.
6. For Jira write behavior, verify the target internal comment by comment ID or
   idempotency key.

Logs alone are not sufficient proof that automation is healthy.