# Agent9 runner and main-cutover runbook

## Runtime boundary

```text
Jira poller / manual caller
    |
    | automatic job or POST /v1/tickets/analyze
    v
AOS main deployment                   Agent9
(separate namespace)                  (separate runtime/tool sandboxes)
  - build bounded Jira workspace  --> session files
  - existing analysis pipeline    --> turns --> Codex Agent
  - existing publication guards   <-- JSON  <-- managed tools
  - shared staging DB
  - sole poller, worker, Jira internal
    writer, reports, Base sync, Lark
```

Agent9 is a transport/runtime backend, not a replacement for AOS business
rules. `CodexTicketAgentRunner` remains the single analysis pipeline. The
adapter translates its thread calls to Agent9 sessions and turns, uploads only
the current bounded run workspace, and maps Agent9 non-LLM operations back to
tool-call items so existing AOS safety audits still see tool use.

Every linked Agent9 Agent is explicit; the workspace default Agent is never
selected implicitly. `AUTO_ONCALL_AGENT9_AGENT_ID` owns Primary Analysis and
non-diagnostic compatibility work. The diagnostic pipeline uses four distinct
Agent IDs for Scenario Router, Evidence Agent, Scenario Diagnostic Agent, and
Final Response Agent. The Evidence Agent has the shared `evidence-query-planning` Skill plus the three Provider query
Skills (`clinic-api`, `wetech-op-grafana`, and `manus-op-grafana`) and matching
Managed Tools. The Scenario Diagnostic Agent has only the three diagnosis
Skills (`op-diagnosis`, `tidb-latency-diagnosis`, and `tidb-query-tuning`) and
no Provider Skill or Managed Tool. Scenario Router and Final Response have
neither set. A diagnosis Skill is read-only reasoning guidance and does not
grant data-query authority. Agent9 owns Codex auth,
runtime sandboxing, managed-tool code, managed-tool credentials, concurrency,
and scaling. AOS receives neither Codex subscription auth nor managed-tool
secrets.

The Agent9-backed AOS Pod exports `CODEX_HOME=/tmp/auto-oncall-codex` on a dedicated
writable PVC. Its entrypoint installs the image-bundled Skills there, and the
AOS runner loads local routing inputs from that exact path before it creates an
Agent9 Session. The same source directories are published by the unified
staging release workflow as immutable Agent9 Skill versions and the linked
Agent's pins are advanced before either Kubernetes application is bumped.
Provider credentials remain inside Agent9. The local report/Judge Codex SDK is
not an Agent9 turn and uses a separate `auth.json` on this PVC; it must be
bootstrapped from a new Codex login and must not reuse Agent9 Organization Auth.

## Compatibility and failure behavior

- `AUTO_ONCALL_AGENT_RUNNER` defaults to `codex-sdk`. Existing deployments do
  not change unless they explicitly set it to `agent9`.
- A newly created Agent9 Session receives its bounded text workspace through
  one batch upload before the first Turn. Resumed Sessions retain per-file
  overwrite semantics for changed text. Files larger
  than 1 MiB and non-image binary files in the synchronized workspace fail
  closed instead of being silently omitted. A Turn may select at most four
  PNG, JPEG, or static WebP local-image inputs: the adapter uploads each image
  through the Agent9 User File API, verifies the ready file, and passes its
  `userFileId` on that exact Turn. Multipart-required uploads and unsupported
  image formats fail explicitly.
- Agent9 HTTP/stream failures fail the analysis. There is no automatic fallback
  to the local Codex runner because that would change the trust boundary.
- Structured turns pass the existing Codex SDK `outputSchema` to Agent9 as the
  native Turn `input.outputSchema`; the schema is not copied into the prompt.
  An Agent9 clarification response remains a clarification and fails that
  structured turn explicitly instead of being parsed as the requested JSON.
- An Agent9 Agent used by AOS must be configured with the intended Codex runtime
  and the minimum managed-tool set before quality or diagnostic parity is
  claimed.
- Before its first Turn, the adapter aligns the Agent9 Session model with the
  requested AOS model through the Session model API. Codex reasoning effort is
  frozen from the linked Agent's runtime config by Agent9; it is not inferred
  from the AOS request or from a trace default.
- Provider-backed evidence is exposed as three independent Tool packages:
  `managed_auto_oncall_clinic`, `managed_auto_oncall_manus`, and
  `managed_auto_oncall_wetech`. They use separate Tool Sandboxes, Auth
  bindings, and network policies. A generic Tool with a model-selected
  `provider` argument is not supported.

## Active staging deployment

`deploy/app-delivery/staging/devops-service/auto-oncall-services-agent9/` is a
separate app-delivery application in namespace
`auto-oncall-services-agent9`. It uses a distinct ConfigMap, Secret, Service,
and Deployment. It is the sole active owner of:

- automatic Jira polling, worker execution, and continuation discovery;
- Jira internal writes;
- reports, Lark notifications, ticket Base sync, and oncaller reconciliation.

Customer-visible Jira writes and Wetech OP are enabled only through their
existing guarded cohort, scenario, Provider-execution, validation, and safety
gates. Manus OP and Investibook remain disabled. Optimizer, Focus, PR bot, and
history cleanup stay in the auxiliary `auto-oncall-services-staging` namespace;
they are not additional poller owners, and that application no longer contains
an AOS Deployment or Service.

The Agent9-backed deployment uses the shared staging database. `mode=write`
may therefore refresh the shared job, run, gate, and
evaluation state. Diagnostic multi-turn is enabled so eligible evidence-assisted tickets
exercise the Scenario Router, Evidence Agent, Managed Tool Provider Executor,
Scenario Diagnostic Agent, and Final Response Agent path.

The legacy staging AOS Deployment and Service are retired and absent from the
auxiliary application. Rollback uses the Agent9 application's previous image
and configuration; recreating a local `codex-sdk` owner requires a separate
reviewed design and cutover.

Manual tests use a local port-forward:

```bash
kubectl -n auto-oncall-services-agent9 port-forward svc/auto-oncall-services-agent9 18080:80
curl -sS http://127.0.0.1:18080/v1/tickets/analyze \
  -H 'content-type: application/json' \
  -d '{"mode":"write","issueKey":"APID-XXXXX"}'
```

The manually-created `auto-oncall-services-agent9-secret` contains:

- `AUTO_ONCALL_AGENT9_API_KEY`
- `AUTO_ONCALL_JIRA_USERNAME`
- `AUTO_ONCALL_JIRA_API_TOKEN`
- `AUTO_ONCALL_DATABASE_URL` (the same value used by existing staging)
- `CLINIC_API_KEY` (used only for bounded Cloud target inventory before Agent9 execution)
- `AUTO_ONCALL_GITHUB_TOKEN`
- `AUTO_ONCALL_LARK_APP_ID`
- `AUTO_ONCALL_LARK_APP_SECRET`

Codex report auth is not stored in that Secret. Bootstrap the dedicated PVC's
`auth.json` after its first rollout and never print or commit it. Deploy through the manual
`deploy-auto-oncall-services-agent9-staging.yml` workflow.

## Deployment and cutover runbook

The Agent9 runtime remains a separate Kubernetes application, but the release
action intentionally updates it together with active staging and publishes the
same seven Skill source directories to Agent9. It then pins the shared query-planning Skill and the three Provider
query Skills only to Evidence, pins the three diagnosis Skills only to Scenario
Diagnostic, and removes stale cross-role installations before either staging
application is bumped.

| Item | Agent9 main staging |
| --- | --- |
| GitHub workflow | `deploy-auto-oncall-services-agent9-staging.yml` |
| app-delivery application | `auto-oncall-services-agent9` |
| Kubernetes namespace | `auto-oncall-services-agent9` |
| Deployment / Service | `auto-oncall-services-agent9` |
| Agent9 endpoint | `https://internal.us-west-2.agent.mem9.ai` |
| Agent9 project | `proj_536577250ef74d50bacf5bd9a5877e27` |
| Agent9 Primary Analysis Agent | `agent_b29c9fea570f432ebe0311bdc8384a45` |
| Agent9 Scenario Router Agent | `agent_b0f561f8fa3bb508083325d3ace4dc98` |
| Agent9 Evidence Agent | `agent_8a187d1f3645a482ed8861274d25f5fd` |
| Agent9 Scenario Diagnostic Agent | `agent_0f75560bc77477b5fc9c25c2876b4adb` |
| Agent9 Final Response Agent | `agent_9aa6d1d0bc0ec6a7695bf29dc4dc26b4` |

The main cutover has three separately deployed phases. Do not combine the stop
and start flags in one app-delivery sync:

1. **Readiness:** deploy the Agent9 image, aligned business configuration, and
   Secret references while it remains `automationRole=disabled`, poll/report/
   continuation/recheck/write/Lark/Base all false. Verify Ready and health.
2. **Stop and drain:** disable the legacy staging poller and all legacy AOS
   background/write owners. Keep the optimizer/Focus workloads running, point
   their analysis API at the Agent9 Service, and after the zero-work drain
   remove the legacy AOS Deployment and Service from the auxiliary application.
   Confirm the shared database has zero `queued` or `running` jobs and zero
   running analyses.
3. **Activate Agent9:** only after the zero-work drain, enable the Agent9
   automation role, poller, reports, continuation discovery, recheck, Jira
   internal writes, Lark notifications, and Base sync. Customer-visible Jira
   writes, Wetech OP, Manus OP, and Investibook remain disabled. Verify exactly
   one active poller owner from live Pod environment, logs, cursor movement,
   and DB job state.

Persisted local Codex thread ids are not migrated. An explicit continuation
against one fails before Agent9 execution, writes one idempotent internal Jira
notice, and records `continuation_thread_backend_incompatible`. New initial
analysis and continuations without an old conversation use Agent9 normally.

The normal release path is:

```text
aisupport commit
  -> manual unified staging workflow
  -> build and push image tagged with the aisupport commit SHA
  -> append/no-change the seven Agent9 Skill packages and advance exact Agent pins
  -> app-delivery version-bump PRs for the auxiliary auto-oncall-services jobs and Agent9 main
  -> for manifest/config changes, app-delivery registration sync PR
  -> Argo CD sync
  -> auto-oncall-services-agent9 Deployment rollout
```

Deploy the merged `main` commit:

```bash
gh workflow run deploy-auto-oncall-services-agent9-staging.yml \
  --repo tidbcloud/aisupport \
  --ref main

gh run list \
  --repo tidbcloud/aisupport \
  --workflow deploy-auto-oncall-services-agent9-staging.yml \
  --limit 5
```

An image version bump does not copy ConfigMap or Deployment template changes.
Whenever this repository changes an app-delivery manifest, also run and merge
the staging template-sync workflow after the version bump:

```bash
gh workflow run open-auto-oncall-app-delivery-registration-pr.yml \
  --repo tidbcloud/aisupport \
  --ref main \
  -f environment=staging
```

The release workflow creates app-delivery version-bump PRs and enables staging
automerge; the template-sync workflow creates a separate manifest PR. Do not
treat the image build or either PR alone as deployment completion. Verify all
applicable app-delivery PRs merged and then verify the live image and runtime
environment:

```bash
KUBE_CONTEXT='arn:aws:eks:us-west-2:552185537300:cluster/devops-staging'
NAMESPACE='auto-oncall-services-agent9'

kubectl --context "$KUBE_CONTEXT" -n "$NAMESPACE" \
  rollout status deployment/auto-oncall-services-agent9

kubectl --context "$KUBE_CONTEXT" -n "$NAMESPACE" \
  get deployment/auto-oncall-services-agent9 \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

kubectl --context "$KUBE_CONTEXT" -n "$NAMESPACE" \
  get configmap/auto-oncall-services-config-agent9 \
  -o jsonpath='{.data.AUTO_ONCALL_AGENT_RUNNER}{"\t"}{.data.AUTO_ONCALL_AUTOMATION_ROLE}{"\t"}{.data.AUTO_ONCALL_POLL_ENABLED}{"\t"}{.data.AUTO_ONCALL_ENABLE_JIRA_INTERNAL_COMMENT_WRITE}{"\t"}{.data.AUTO_ONCALL_DIAGNOSTIC_MULTITURN_ENABLED}{"\t"}{.data.CODEX_HOME}{"\t"}{.data.AUTO_ONCALL_CODEX_MODEL}{"\n"}'
```

For the active main deployment, the final line must be:

```text
agent9 active true true true /tmp/auto-oncall-codex gpt-5.6-sol
```

Also verify `/healthz` and `/readyz` through a local port-forward before a
diagnostic request:

```bash
kubectl --context "$KUBE_CONTEXT" -n "$NAMESPACE" \
  port-forward service/auto-oncall-services-agent9 18080:80

curl --fail --silent --show-error http://127.0.0.1:18080/healthz
curl --fail --silent --show-error http://127.0.0.1:18080/readyz
```

## Trace one analysis end to end

There are two correlation roots:

```text
Jira issue key
  -> AOS runId
     -> AOS DB run / run workspace / evaluation
     -> result.codexThreadId == Agent9 sessionId
        -> Agent9 Turn
           -> AgentRun
              -> managed-tool Operation
```

Start with `mode=read` unless the test explicitly needs shared staging DB
state. Save the complete response because it is the easiest correlation
artifact:

```bash
curl --fail --silent --show-error \
  http://127.0.0.1:18080/v1/tickets/analyze \
  -H 'content-type: application/json' \
  -d '{"mode":"read","issueKey":"APID-XXXXX"}' \
  | tee /tmp/auto-oncall-agent9-response.json

jq '{
  runId,
  issueKey,
  mode,
  scenarioClassification,
  execution,
  agent9SessionId: .result.codexThreadId,
  publication
}' /tmp/auto-oncall-agent9-response.json
```

When Agent9 actually executed a thread, `result.codexThreadId` is the Agent9
Session id (`sess_...`). A null value is valid only when a preflight path
finished before creating an Agent9 Session; it does not prove Agent9 runtime or
managed-tool execution.

Use the Session id in the Agent9 Console at
`https://internal.us-west-2.agent.mem9.ai/`. Select the configured project and
inspect the Session's Turns, AgentRuns, and Operations. For API-level debugging,
use the same Agent9 API key as the main deployment without printing it:

```bash
AGENT9_BASE_URL='https://internal.us-west-2.agent.mem9.ai'
AGENT9_PROJECT_ID='proj_536577250ef74d50bacf5bd9a5877e27'
AGENT9_SESSION_ID='sess_...'

curl --fail --silent --show-error \
  "$AGENT9_BASE_URL/api/sessions/$AGENT9_SESSION_ID/turns" \
  -H "Authorization: Bearer $AUTO_ONCALL_AGENT9_API_KEY" \
  -H "x-agent9-project-id: $AGENT9_PROJECT_ID" \
  | jq .
```

For `mode=write`, the AOS response also returns `publication.runDbId`; the
latest stored job/run can be read without rerunning analysis:

```bash
curl --fail --silent --show-error \
  http://127.0.0.1:18080/v1/auto-analysis/issues/APID-XXXXX/latest \
  | jq '{issueKey, job, latestRun, gateDecisions}'
```

The AOS run workspace is ephemeral Pod storage under
`/tmp/auto-oncall-services/runs/<runId>`. Inspect it before the Pod restarts:

```bash
POD="$(kubectl --context "$KUBE_CONTEXT" -n "$NAMESPACE" \
  get pod -l app.kubernetes.io/instance=agent9-main \
  -o jsonpath='{.items[0].metadata.name}')"
RUN_ID="$(jq -r .runId /tmp/auto-oncall-agent9-response.json)"

kubectl --context "$KUBE_CONTEXT" -n "$NAMESPACE" exec "$POD" -- \
  find "/tmp/auto-oncall-services/runs/$RUN_ID" -maxdepth 2 -type f -print

kubectl --context "$KUBE_CONTEXT" -n "$NAMESPACE" exec "$POD" -- \
  sh -c "sed -n '1,240p' '/tmp/auto-oncall-services/runs/$RUN_ID/output/result.json'"
```

Useful files are:

- `run-metadata.json`: issue, mode, runner, and prompt/schema versions;
- `input/scenario-classification.json`: selected scenario and evidence provider;
- `input/diagnostics/manifest.json`: runner-owned bounded provider context;
- `trace/context-trace.json`: LLM turn timing, thread/session ids, requested
  settings, and `executionConfig` read from the Agent9 `turn_started` event.
  The execution projection records the frozen model and reasoning effort for
  successful and post-start failed Turns. If Agent9 did not report them, both
  values are null with `source=unavailable`; requested settings are never
  substituted as Agent9 execution facts;
- `trace/diagnostic-multiturn-summary.json`: route, provider evidence, and
  synthesis outcomes when the diagnostic pipeline ran;
- `output/result.json`: final normalized diagnosis and publication-safe draft.

Do not copy ticket contents, credentials, cookies, or raw provider responses
into an issue or chat while debugging.

## Managed Tool and Skill boundary

The similarly named Skills and managed Tools are different objects:

| Provider | Skill (instructions only) | Agent9 managed Tool (execution) |
| --- | --- | --- |
| Clinic | `clinic-api` | `managed_auto_oncall_clinic` |
| Wetech | `wetech-op-grafana` | `managed_auto_oncall_wetech` |
| Manus | `manus-op-grafana` | `managed_auto_oncall_manus` |

Skills describe how to form a bounded query and interpret the result. They do
not contain provider credentials. Managed Tools run in Agent9's isolated Tool
Sandbox; Agent9 injects the Tool credential there, and the Agent never receives
the raw value.

When diagnostic multi-turn is enabled with the Agent9 backend, the Evidence
Agent invokes only the Tool selected for its frozen provider. The Tool call is
the runner-owned Provider Executor boundary: Agent9 AOS validates and executes
it, persists the Operation, and returns a bounded `provider-result-v1`.
Auto-oncall maps that persisted compact result into canonical reports and the
Scenario Diagnostic Agent handoff. It does not run the legacy provider helper,
does not bootstrap Wetech credentials in the auto-oncall Pod, and does not use
the Evidence Agent's final prose as evidence. Wetech Tool turns use no AOS
runner-level provider lock. Within each Wetech Evidence Agent Turn, Metrics
calls do not consume Dashboard slots and may run independently, while at most
two Dashboard calls may be in flight. Manus retains its runner-level provider
lock; Clinic uses no cross-scenario provider lock.

Agent9 Core Tools are not Provider calls. The Evidence Agent may use the
current Session Drive and shell for file reads and writes, directory
inspection, local code, and `curl` when the Agent9 Runtime network policy
permits it. Agent9 owns those operations and their sandbox and audit policy;
auto-oncall neither allowlists their names nor materializes their output as
Provider evidence. Auto-oncall checks only the recognized Managed Provider
calls, still rejects a cross-provider call, and still requires at least one
valid result from the selected Managed Tool.

Managed Tool installations do not own per-ticket authorization. The runner
freezes the current ticket, target, and absolute incident window into the
Evidence Agent Turn. The selected Tool script independently rejects malformed
ticket keys, unsupported operations, unknown bundled OP targets, and excessive
query windows before provider access; Clinic additionally verifies target
ownership through the Clinic API. Installation policy may remain empty and an
old test-ticket policy is ignored by these packages.

The Tool packages are built from `tools/agent9-auto-oncall/` and published to
the immutable S3 release prefix. AOS owns the internal-production bundle and
catalog pins in
`deploy/agent9-managed-tools/prod-internal-us-west-2-pingcap/`, applies the
`agent9-managed-tool-loader` ConfigMap, and changes only its dedicated
Pod-template rollout annotation. Agent9 owns the unchanged generic init
container that loads and verifies those objects. A new AOS package release must
not copy pins into Agent9 or require an Agent9 pull request. The packages are
not copied into the auto-oncall Kubernetes image.

The `skills/` directory is the single source for both runtimes: Docker installs
it into the Kubernetes Pod's `CODEX_HOME`, while the unified staging release
publishes those same seven Git-tree packages through the Agent9 Skill API and
advances the linked Agent's exact installations. Publishing
`wetech-op-grafana` does not enable Wetech OP; the AOS support gate and Managed
Tool installation remain authoritative.

## Debug checklist

1. **No AOS response:** check Deployment readiness, Service endpoints, then
   `kubectl logs deployment/auto-oncall-services-agent9`.
2. **`agent9_http_error:401` or `403`:** verify the namespace Secret key,
   Agent9 project, Agent id, and API-key scope. Never print the Secret value.
3. **Session creation or file upload fails:** inspect the AOS error code. The
   adapter rejects text workspace files over 1 MiB, non-image binary workspace
   files, more than four Turn images, unsupported image content, and images for
   which Agent9 requires multipart upload. Transient Agent9 Drive/session
   errors are retried only by the bounded adapter retry policy.
4. **Session exists but no managed-tool Operation:** first inspect
   `scenario-classification.json`. A provider Tool is not expected when the
   selected evidence provider is `none` or `investibook`, or when preflight
   ended the run. Then verify the Agent has the three required Skill and Tool
   installations.
5. **Managed-tool Operation failed:** inspect that Operation in Agent9 before
   changing AOS. Separate package-load/runtime failure, missing Tool auth,
   sandbox/network policy rejection, provider authentication, and bounded query
   validation failures.
6. **Agent9 succeeded but AOS rejected the run:** inspect AOS
   `diagnostic-multiturn-summary.json`, `context-trace.json`, and
   `output/result.json`. Publication/schema/safety rejection is owned by AOS,
   not by the Agent9 transport.
7. **`mode=write` looks successful:** additionally verify `publication`, the
   stored DB run, and the exact `public=false` Jira internal comment mapping.
   HTTP 200 alone is not publication proof.

## Rollback

Rollback preserves the same single-owner boundary. Roll back the Agent9 image
or configuration while keeping one active Agent9 owner, then verify the Pod,
health endpoints, cursor, logs, and DB job state. The retired local `codex-sdk`
Deployment is not a rollback target. Recreating it requires a separate reviewed
design, a zero-work drain, and explicit single-owner cutover.

Reverting only an image tag does not revert ConfigMap ownership. Use the
repository app-delivery template sync workflow for flag changes, then verify
the live Pod environment, health endpoints, cursor movement, logs, and DB job
state.