# Auto Oncall Services Architecture Design

Date: 2026-06-30

Status: staging MVP design.

Related boundary document:
[`2026-06-30-auto-oncall-services-boundary.md`](./2026-06-30-auto-oncall-services-boundary.md).

## 1. Summary

`auto-oncall-services` is an independently deployable TypeScript service for
internal oncall ticket analysis.

The staging MVP is intentionally small:

- expose one single-ticket API: `POST /v1/tickets/analyze`;
- accept either one Jira `issueKey` or one caller-provided ticket snapshot;
- let the backend read and normalize Jira context;
- create one UUID-named run workspace per analysis;
- run a Codex-backed dry-run analyzer inside that workspace;
- return internal markdown plus a minimal safety decision.

The MVP is not a full oncall automation platform. It does not write Jira,
execute production operations, scan tickets in bulk, fetch metrics/config/logs,
or use skills.

This session is staging-only. Production deployment, release ticket flow, and
production app-delivery registration are out of scope.

## 2. Current Baseline

The current repository already contains a minimal staging-deployed prototype:

- standalone subrepo: `auto-oncall-services/`;
- TypeScript Node.js service;
- own package manifest, lockfile, tests, Dockerfile, and deployment manifests;
- endpoints:
  - `GET /`;
  - `GET /healthz`;
  - `GET /readyz`;
  - `POST /v1/tickets/analyze`;
- Jira issue-key path;
- ticket snapshot path;
- internal-only analysis markdown;
- staging deployment in `devops-staging`.

Verified staging baseline:

- cluster: `devops-staging`;
- namespace: `auto-oncall-services-staging`;
- deployment: `auto-oncall-services`;
- service: `auto-oncall-services`.

The current code still returns a structured rule-based analysis shape with
fields such as `tLevel`, `dispatchLane`, `scenario`, and `riskFlags`. That is
prototype information, not the target MVP public API contract.

## 3. MVP Non-Goals

These are explicitly not part of the MVP:

- production deployment work;
- bulk ticket scan, backfill, scheduler, queue, or async job system;
- database persistence;
- real workspace backend;
- Jira write, Jira transition, Jira assignment, or Jira closure;
- customer-visible reply generation or posting;
- production operation execution;
- metrics, config, logs, Clinic, Kubernetes, or other skills;
- agent direct access to Jira, credentials, production systems, or network
  resources;
- provider-agnostic agent framework abstraction.

The only MVP input context is the single Jira ticket context prepared by the
backend.

## 4. Isolation Boundary

`auto-oncall-services` is a fully isolated subrepo/service.

Required isolation:

- source lives under `auto-oncall-services/`;
- the service owns its own dependency manifest and lockfile;
- tests are local to the service;
- Docker image is built from the service directory;
- Kubernetes manifests are owned by the service;
- runtime config is explicit environment variables;
- external systems are accessed through service-owned integration code;
- no runtime imports from sibling services or parent repo packages.

Allowed repository-level sharing:

- GitHub Actions conventions;
- ECR repository naming conventions;
- app-delivery registration shape;
- Kubernetes labels and kustomize style;
- documentation under `docs/design/`.

Forbidden runtime coupling:

- importing code from `ticket-manager`, `oncall`, `investibook`, or any other
  deployable service;
- depending on parent `packages/`, `services/`, `scripts/`, `go.work`, or
  workspace config;
- using another service database directly;
- treating another service internal API as an implicit contract;
- using Investibook code, config, API, environment variables, or domains.
- reading another service's secret path. Shared secret values are acceptable
  only when copied into an auto-oncall-owned secret path.

## 5. Investibook Boundary

Investibook is design reference only.

Allowed:

- compare interaction patterns;
- inspect high-level ideas such as workspace, case context, and internal
  analysis flow.

Forbidden:

- no code import;
- no runtime API call;
- no `INVESTIBOOK_*` env var;
- no Investibook domain fallback;
- no response field named `investibookContext`;
- no internal wording that says the workspace is Investibook-owned.

If a real workspace backend is needed later, it must be exposed through an
auto-oncall-owned interface.

## 6. HTTP API

### 6.1 Endpoints

```text
GET  /
GET  /healthz
GET  /readyz
POST /v1/tickets/analyze
```

`POST /v1/tickets/analyze` analyzes exactly one Jira ticket. It is not a bulk,
backfill, or full-scan API.

### 6.2 Request

The request accepts either `issueKey` or `ticket`.

Issue-key mode:

```json
{
  "issueKey": "TCOC-123"
}
```

Snapshot mode:

```json
{
  "ticket": {
    "key": "TCOC-123",
    "fields": {
      "summary": "Slow query latency increased for cluster 123456",
      "priority": { "name": "Medium" },
      "description": "..."
    }
  }
}
```

Flattened snapshot mode may be accepted for tests and dry runs:

```json
{
  "ticket": {
    "key": "TCOC-124",
    "summary": "Backup failed after nightly maintenance",
    "description": "Customer reports backup job failed after maintenance.",
    "priority": "Medium",
    "status": "Open",
    "labels": ["backup"],
    "components": ["TiDB Dedicated"],
    "clusterId": "789"
  }
}
```

`createWorkspace` is not part of the MVP request contract.

### 6.3 Response

The MVP response is a dry-run result, not a full structured analysis schema.

```json
{
  "service": "auto-oncall-services",
  "status": "ok",
  "runId": "018f4f9d-8b8b-7c2a-9a5d-123456789abc",
  "issueKey": "TCOC-123",
  "mode": "dry_run",
  "result": {
    "decision": "internal_only",
    "summaryMarkdown": "## Internal analysis\n...",
    "simulatedCustomerReplyMarkdown": "Thanks for reaching out. ...",
    "workspaceUrl": "/runs/018f4f9d-8b8b-7c2a-9a5d-123456789abc"
  },
  "safety": {
    "humanRequired": false,
    "blockedReason": null,
    "requestedContext": []
  }
}
```

For high-risk cases:

```json
{
  "service": "auto-oncall-services",
  "status": "ok",
  "runId": "018f4f9d-8b8b-7c2a-9a5d-123456789abc",
  "issueKey": "ST-99",
  "mode": "dry_run",
  "result": {
    "decision": "human_required",
    "summaryMarkdown": "## Internal intake\n...",
    "simulatedCustomerReplyMarkdown": "Thanks for reaching out. A support engineer will review this case before we provide a final technical answer.",
    "workspaceUrl": "/runs/018f4f9d-8b8b-7c2a-9a5d-123456789abc"
  },
  "safety": {
    "humanRequired": true,
    "blockedReason": "high_risk",
    "requestedContext": ["cluster_id", "time_window"]
  }
}
```

MVP response fields:

| Field | Meaning |
|-|-|
| `runId` | UUID for this analysis run. It is not derived from ticket id. |
| `issueKey` | Jira issue key after normalization. |
| `mode` | Always `dry_run` in MVP. |
| `result.decision` | `internal_only` or `human_required`. |
| `result.summaryMarkdown` | Human-readable internal analysis generated by the runner. |
| `result.simulatedCustomerReplyMarkdown` | Dry-run preview of expected customer-facing wording. It is not posted automatically. |
| `result.workspaceUrl` | Link to the run/workspace view. It may be a planned path in MVP. |
| `safety.humanRequired` | Backend safety gate result. |
| `safety.blockedReason` | Null or a coarse reason such as `high_risk`, `production_operation`, or `missing_context`. |
| `safety.requestedContext` | Minimal list of missing context the human or caller should provide. |

The MVP API should not expose these prototype internals as stable contract:

- `tLevel`;
- `dispatchLane`;
- `scenario`;
- `confidence`;
- detailed `rationale`;
- detailed `nextActions`;
- detailed `riskFlags`;
- `workspaceContext`;
- `automatable`.

Those concepts may exist inside prompts, traces, or implementation details, but
callers should not depend on them yet.

## 7. Backend And Agent Boundary

The backend owns integrations, credentials, side effects, and final safety.
The agent runtime owns analysis inside a run workspace.

Backend responsibilities:

- validate HTTP requests;
- fetch Jira issue by `issueKey`;
- normalize Jira or snapshot input into a service-owned ticket context;
- redact data that should not be visible to the agent;
- create a UUID run workspace;
- write input files for the runner;
- start the Codex runner;
- parse runner output;
- apply backend safety gates;
- serialize the API response;
- own all future side effects, including any Jira write if that is added later.

Agent runtime responsibilities:

- read the prepared files in the run workspace;
- reason over the single ticket context;
- optionally run local scripts inside the run workspace;
- generate internal markdown;
- write a machine-readable result file;
- write optional trace/artifact files.

Agent runtime must not:

- call Jira directly;
- hold Jira credentials;
- call metrics/config/log/Clinic/Kubernetes skills;
- hold production credentials;
- write Jira;
- post customer-visible content;
- execute production operations;
- read or write outside the run workspace.

## 8. Run Workspace

Every analyze request creates a new run workspace keyed by UUID.

Do not use ticket id in the directory path. The same ticket may be analyzed many
times for retries, evaluation, prompt version changes, and runner experiments.

Default local shape:

```text
/var/tmp/auto-oncall-services/runs/
  <runId>/
    run-metadata.json
    input/
      ticket.json
      comments.md
    prompt/
      analyze-ticket.md
      result.schema.json
    work/
      scratch/
      scripts/
    output/
      result.json
      summary.md
      artifacts/
    trace/
      codex-events.jsonl
      runner.log
```

`run-metadata.json` records ticket association and runner configuration:

```json
{
  "runId": "018f4f9d-8b8b-7c2a-9a5d-123456789abc",
  "issueKey": "TCOC-123",
  "mode": "dry_run",
  "runner": "codex-sdk",
  "promptVersion": "analyze-ticket-v1",
  "schemaVersion": "ticket-agent-result-v1",
  "createdAt": "2026-06-30T00:00:00.000Z"
}
```

MVP isolation is path isolation, not container isolation. That is acceptable for
staging MVP as long as:

- runner `cwd` is the run directory;
- generated scripts and artifacts stay under the run directory;
- no production credentials are passed to the runner;
- no skills or production network access are exposed to the runner.

Harder isolation, such as per-run containers or Kubernetes jobs, can be added
after the MVP proves useful.

## 9. Codex Runner

The MVP runner should be Codex-backed.

Primary path:

```text
Codex SDK runner
  -> create one Codex run for this request
  -> cwd = <run workspace>
  -> sandbox = workspace-write
  -> approval policy = never
  -> network access = disabled
  -> prompt = prompt/analyze-ticket.md
  -> output schema = prompt/result.schema.json
  -> output = output/result.json + output/summary.md
```

Fallback path if SDK authentication or deployment is not ready:

```text
codex exec
  -> run from <run workspace>
  -> use JSON/structured output
  -> capture JSONL trace
  -> write final result files
```

Do not build a broad provider abstraction for MVP. The service can define a
small internal runner boundary for tests, but the only planned production runner
is Codex.

Minimum internal result type:

```ts
type TicketAgentResult = {
  decision: "internal_only" | "human_required";
  summaryMarkdown: string;
  simulatedCustomerReplyMarkdown: string;
  requestedContext?: string[];
  blockedReason?: string | null;
};
```

Codex output must be treated as untrusted until the backend validates it:

- result JSON must match the expected schema;
- markdown must be internal-only;
- decision must be rechecked by backend safety rules;
- any artifact paths must stay inside the run workspace.

Codex authentication for staging is an open deployment decision. Acceptable
early-stage options are a service API key, a Codex access token, or a temporary
mounted Codex auth cache. Any auth material must stay out of run directories,
logs, and the Codex prompt.

Codex auth must use an auto-oncall-owned Kubernetes Secret in namespace
`auto-oncall-services-staging`. It must not read Investibook provider secrets.
If the initial value is shared with another service for staging, copy that value
into the auto-oncall-owned Kubernetes Secret.

For temporary personal-subscription staging smoke tests, the accepted short-term
shape is:

- Kubernetes Secret name: `auto-oncall-services-codex-auth`;
- Kubernetes Secret key: `auth.json`;
- mount path: `/tmp/auto-oncall-codex/auth.json`;
- runtime `CODEX_HOME`: `/tmp/auto-oncall-codex`.

The staging Deployment mounts a writable `emptyDir` as `CODEX_HOME` and mounts
the Secret key directly at `$CODEX_HOME/auth.json` with `subPath`. This is
temporary and must not be used for production.

## 10. Jira Boundary

Jira access belongs to the backend, not the agent.

Read path:

```text
POST /v1/tickets/analyze
  -> backend validates request
  -> backend fetches Jira issue if issueKey is provided
  -> backend normalizes ticket context
  -> backend writes ticket.json/comments.md
  -> Codex runner analyzes local files
```

Jira write is not part of MVP.

If Jira write is added later:

- agent may only propose internal comment markdown;
- backend owns idempotency, permission checks, audit, and write execution;
- customer-visible comments, ticket closure, and status transitions remain out
  of scope unless a separate human-approved workflow is designed.

## 11. Safety Model

The MVP safety model is intentionally coarse and backend-owned.

Hard-stop examples:

- P0/P1 or critical incident;
- data loss, data corruption, or consistency risk;
- customer escalation;
- suspected product bug or regression;
- destructive or non-standard operation;
- production operation request.

For hard-stop cases:

- return `result.decision=human_required`;
- return `safety.humanRequired=true`;
- do not return customer-ready text;
- do not write Jira;
- do not execute operations.

For missing context:

- return requested context in `safety.requestedContext`;
- the runner may still produce an internal intake summary;
- the output must not claim final resolution.

`summaryMarkdown` is internal-only. `simulatedCustomerReplyMarkdown` is a
dry-run customer-facing draft preview, but it is not posted automatically and
must stay behind human review.

## 12. Staging Deployment

Staging target:

| Item | Value |
|-|-|
| GitHub workflow | `.github/workflows/deploy-auto-oncall-services-staging.yml` |
| ECR repository | `aisupport/staging/auto-oncall-services` |
| app-delivery path | `staging/devops-service/auto-oncall-services/envs/staging` |
| Kubernetes cluster | `devops-staging` |
| Kubernetes namespace | `auto-oncall-services-staging` |
| Kubernetes deployment | `auto-oncall-services` |
| Kubernetes service | `auto-oncall-services` |

Additional staging config needed for the Codex MVP:

- Codex auth secret or mounted auth cache;
- explicit Codex model setting;
- local run workspace root, for example an `emptyDir` volume;
- cleanup policy to prevent unbounded disk growth.

Staging Jira secret ownership:

- ConfigMap: non-sensitive `AUTO_ONCALL_JIRA_BASE_URL`;
- Kubernetes Secret name: `auto-oncall-services-secret`;
- Kubernetes Secret keys: `AUTO_ONCALL_JIRA_USERNAME`,
  `AUTO_ONCALL_JIRA_API_TOKEN`, `CLINIC_API_KEY`.

The values may be initialized from the same operational source as Investibook
dev, but the manifest must not reference Investibook Secret names or paths.

Production remains out of scope for this session.

## 13. Verification Matrix

Minimum staging checks:

| Area | Check | Expected result |
|-|-|-|
| API health | `GET /healthz` | `status=ok` |
| single-ticket request | one `issueKey` or one `ticket` | accepted |
| no bulk mode | batch/full-scan input | not supported |
| run identity | analyze response | UUID `runId`, not ticket-derived path |
| low-risk dry run | standard snapshot | `decision=internal_only` |
| high-risk dry run | P1/data-risk/production-operation snapshot | `decision=human_required` |
| output contract | response JSON | simple dry-run shape only |
| Jira boundary | runner environment | no Jira credentials exposed to runner |
| no skills | runner environment | no metrics/config/log/Clinic/k8s skills |
| Investibook boundary | source/deploy grep | no runtime Investibook dependency |
| staging namespace | Kubernetes lookup | `auto-oncall-services-staging` |

Suggested tests:

- `POST /v1/tickets/analyze` accepts one ticket snapshot;
- response includes UUID `runId`;
- response includes `mode=dry_run`;
- response includes `result.summaryMarkdown`;
- response includes `result.simulatedCustomerReplyMarkdown`;
- response includes `safety.humanRequired`;
- high-risk cases force `human_required`;
- production operation cases force `human_required`;
- `createWorkspace` is not required;
- response does not include `investibookContext`;
- response does not expose `automatable`, `tLevel`, or `dispatchLane` as MVP
  contract fields.

## 14. Follow-Ups After Design Acceptance

Implementation should be split into small PRs:

1. Replace the public analyze response with the simple dry-run contract.
2. Add UUID run workspace creation.
3. Add Codex SDK runner for staging.
4. Add `codex exec` fallback only if SDK is not viable in staging.
5. Add backend safety gate tests for high-risk and production-operation cases.
6. Update README examples to remove complex prototype response fields.

Future work that is not part of the MVP:

- real workspace backend;
- Jira internal comment write;
- persisted run history;
- evaluation datasets and replay;
- container or Kubernetes-job isolation;
- async jobs and queueing;
- metrics/config/log/Clinic/Kubernetes integrations;
- production deployment.

## 15. References

References checked for this design:

- Codex SDK: https://developers.openai.com/codex/sdk
- Codex non-interactive mode: https://developers.openai.com/codex/noninteractive
- Codex authentication: https://developers.openai.com/codex/auth
- Codex sandboxing: https://developers.openai.com/codex/concepts/sandboxing