# auto-oncall-services

Standalone TypeScript services for oncall automation.

This subrepo is intentionally isolated from the parent monorepo. It owns its own
package manifest, lockfile, source code, tests, Dockerfile, and deployment
manifests.

Goal and boundary source of truth:
[`docs/design/2026-06-30-auto-oncall-services-boundary.md`](../docs/design/2026-06-30-auto-oncall-services-boundary.md).
If this README or the current prototype conflicts with that document, the design
document wins.

Detailed MVP architecture:
[`docs/design/2026-06-30-auto-oncall-services-architecture.md`](../docs/design/2026-06-30-auto-oncall-services-architecture.md).

## Current Service

The service exposes an internal oncall ticket analysis API.

Endpoints:

- `GET /` returns service metadata.
- `GET /healthz` returns liveness status.
- `GET /readyz` returns readiness status.
- `POST /v1/tickets/analyze` analyzes and evaluates one Jira ticket case.
- `GET /v1/auto-analysis/jobs/:jobId` and
  `GET /v1/auto-analysis/issues/:issueKey/latest` expose read-only job status.
- `GET /v1/auto-analysis/optimizer-candidates` exposes the durable internal
  single-ticket signal feed (`wrong|revise`, `human_better`, Provider
  degradation/failure, and Router/Final timeout) by monotonic candidate ID.
- `GET /v1/auto-analysis/feedback-reminder-targets?since=<timestamp>` exposes
  successful `initial`/`bot_assignment` tickets since the timestamp that have
  no ticket-level `/ao` feedback. Skipped and permanently failed jobs are not
  returned.

`POST /v1/tickets/analyze` is the manual analysis API. It accepts
`mode=read` (the safe default) or `mode=write`:

For quality-first diagnostic testing, add `"noDiagnosticTimeout": true` only
with `mode=read`. Diagnostic stage/model-turn deadlines become operational
`diagnostic_would_timeout` logs; normal requests and automatic oncall retain
their existing limits. Provider/auth/network operations, classifier and Judge
limits remain. Isolated replay still has a one-hour Job/collection limit; use
`npm run replay:isolated -- --issue TCID-13132 --no-diagnostic-timeout`.

```json
{
  "issueKey": "ST-2798",
  "mode": "read"
}
```

`read` fetches the case, automatically chooses the historical L1/support reply
when one exists, filters that answer and all later/support/internal/bot context
from the analysis input, runs the analysis and single-case judge, and performs
no Jira, database, or Feishu writes.

`write` runs the same input selection, analysis, and judge, then performs the
complete rerun publication transaction:

- reuse the ticket's existing initial job, or create it only when absent;
- append the new run and evaluation to the database without adding another
  ticket to funnel denominators;
- update the stable Jira internal comment keyed by
  `auto-live:v1:<issueKey>:initial` (or create it when absent);
- refresh that ticket's Feishu Base dashboard row.

```json
{
  "issueKey": "ST-2798",
  "mode": "write"
}
```

`mode=write` requires `issueKey`, the auto-analysis database, enabled Jira
internal-comment writes, Jira credentials, and configured Feishu Base sync.
The API response reports each publication result separately under
`publication`.

After a successful `mode=read` verification, the optimizer can publish that
exact frozen run without a second analysis or Judge call:

```http
POST /v1/tickets/publish-read HTTP/1.1
Content-Type: application/json

{
  "issueKey": "ST-2798",
  "readRunId": "018f4f9d-8b8b-7c2a-9a5d-123456789abc"
}
```

The endpoint loads `output/read-publication.json` from the server-owned run
workspace, verifies the issue identity and answer fingerprint, updates the
stable Jira internal comment, persists the same run and evaluation, and
refreshes the Feishu Base row. It does not accept answer text from the caller
and does not invoke the analysis or Judge agents. Because run workspaces are
runtime-local, callers must publish while the read run is still retained on
the same single-replica runtime; an unavailable run fails instead of falling
back to another replay.

Tests and trusted callers may provide a ticket snapshot. This remains read-only
unless a matching `issueKey` and `mode=write` are also present:

```json
{
  "mode": "read",
  "ticket": {
    "key": "TCOC-123",
    "fields": {
      "summary": "Slow query latency increased for cluster 123456",
      "priority": { "name": "Medium" },
      "description": "..."
    }
  }
}
```

Every workspace writes `input/input-manifest.json`. The same manifest is
returned by the API and persisted for live runs. It records the input mode,
anchor, cutoff, included and excluded comments, ticket field policy, and
backend-derived comparison eligibility. The judge receives the selected human
reply as reference evidence, but the analysis runner never sees it.

Target MVP response:

```json
{
  "service": "auto-oncall-services",
  "status": "ok",
  "runId": "018f4f9d-8b8b-7c2a-9a5d-123456789abc",
  "issueKey": "TCOC-123",
  "mode": "read",
  "result": {
    "decision": "internal_only",
    "decisionReason": null,
    "internalHandoffMarkdown": "Internal Only content...",
    "customerReplyMarkdown": "Thanks for reaching out. ...",
    "commentBodyAdf": {
      "type": "doc",
      "version": 1,
      "content": []
    },
    "clarificationSimulation": {
      "required": false,
      "waitWindowMinutes": 5,
      "l1FollowUpMarkdown": null,
      "supplementReceived": false,
      "supplementMarkdown": null,
      "nextDraftMarkdown": null,
      "nextClarificationMarkdown": null
    },
    "replayTarget": {
      "mode": "before_support_reply",
      "targetCommentIndex": 0,
      "targetAuthor": "L1 Oncall",
      "targetCreated": "2026-06-29T10:05:00.000+0000",
      "cutoffCommentIndex": 0,
      "visibleCommentIndexes": []
    },
    "workspaceUrl": "/runs/018f4f9d-8b8b-7c2a-9a5d-123456789abc"
  },
  "safety": {
    "humanRequired": false,
    "blockedReason": null,
    "requestedContext": []
  },
  "jiraInternalComment": {
    "requested": false,
    "status": "not_requested",
    "endpoint": null,
    "commentId": null,
    "commentUrl": null,
    "bodyMarkdown": null,
    "skippedReason": "analyze_read_mode",
    "error": null
  },
  "evaluation": {
    "status": "evaluated",
    "eligibility": "comparable",
    "verdict": "equivalent",
    "targetHumanCommentId": "23122799",
    "keyDifferences": [],
    "aiStrengths": [],
    "aiGaps": [],
    "humanStrengths": [],
    "evidenceSummary": "The answers are materially equivalent.",
    "error": null
  },
  "publication": {
    "mode": "read",
    "runDbId": null,
    "database": "not_requested",
    "jira": "not_requested",
    "evaluation": "not_requested",
    "feishuDashboard": "not_requested",
    "errors": []
  }
}
```

The default runtime runner uses Codex SDK and writes per-run files under a UUID
run directory. Unit tests inject a fake runner so tests do not require Codex
auth or spend model calls.

To run a bounded, write-disabled Recent Evaluation for the one active GitHub
Focus Issue:

```bash
npm run evaluate:recent
```

The complete current Evaluation data flow, Focus lifecycle, gate semantics,
status meanings, and code lookup map are documented in
[`docs/AUTO_ONCALL_EVALUATION.md`](docs/AUTO_ONCALL_EVALUATION.md).

The command reads the versioned Focus contract from GitHub, removes the GitHub
credential before case execution, runs target/control cases through the shared
dry-run contract, and applies one acceptance contract with a read-only judge.
It prints `recent-evaluation-v3` JSON with `gaps_found`, `criteria_met`, or
`inconclusive`. A gap is a normal finding; only execution failures make the
command exit non-zero.

Staging also defines one Daily Recent Optimizer CronJob after the daily report
window. When no Active Focus exists, the controller reads bounded recent daily
report artifacts from the canonical database, splits target comparisons into
stable problem signals, and excludes only problem refs already consumed by
historical Focus Issues or matched by verified `docs/opts` records. A verified
match newer than the record cutoff remains eligible as a regression. The
read-only structured selector chooses one narrow cross-case root cause and must
cite one problem ref per target. A candidate requires at least two
target cases and up to two optional controls, then runs a fresh baseline on
current `main`. The controller creates the GitHub Focus Issue only when every
target executes and at least two targets still have a blocking gap. A missing,
failed, or unstable control remains visible evidence but does not block Focus
creation. The creation run stores baseline and `case_refresh`
checkpoints and stops; the next run starts candidate optimization.

The GitHub Focus Issue is the optimizer's only durable execution state and explicitly names
the blocking criteria for the current objective. Fresh case results, baseline
provenance, candidate branch, full-test, challenger, candidate-gate, and Draft
PR checkpoints are persisted as Issue comments. Every scheduled run first
executes a fresh baseline on current `main` and fetches Jira again while
preserving the historical replay cutoff. Only candidates created from that same
base SHA may resume. Every baseline and challenger appends a `case_refresh`
checkpoint, and the newest bounded case results are included in the next
candidate analysis.
The same multi-case reproducibility gate applies to every Active Focus refresh.
If fewer than two targets retain a confirmed blocking gap, a strict majority of
targets fully meet the objective, and no target execution failed, the controller
writes a `focus_close` checkpoint, closes matching Draft PRs, and closes the
Focus Issue. This retires a stale Focus without attributing the result to its
candidate. Unmet or inconclusive target problem refs are released for later
selection, and the next run establishes a new selection baseline.
Retirement eligibility is a terminal lifecycle transition: every run checks
persisted complete baseline checkpoints before replaying the old cases. Once a
baseline proves the Focus no longer reproducible, a later variable replay cannot
revive it; newly observed gaps must re-enter selection as new problem signals.
An existing Draft PR prevents duplicate candidate creation only after that
day's fresh baseline and case refresh have completed. A material candidate is
always retained as a Draft PR after its branch is pushed, including when path
policy, full tests, challenger execution, or the candidate gate reports a
needs-work signal. A strict minority of inconclusive target cases does not block
an accepted gate when another blocking target has a confirmed improvement and
no evaluable blocking target has a gap. Every inconclusive case and rejected
gate remains visible in the Draft PR; no Draft PR is merged or deployed
automatically. Focus selection cannot broaden business
gates, deployment, production operations, or customer-visible automation. The
task never merges, deploys, or writes Jira. It sends
one idempotent Feishu summary for each terminal optimizer result. The summary
links the Draft PR when one exists, marks needs-work PRs as not mergeable yet,
and partitions current fully-met cases from unmet cases with improvement-only,
mixed improvement/regression, regression-only, or unchanged outcomes. Run the same controller
manually with `npm run optimize:recent:daily`; it is disabled unless
`AUTO_ONCALL_RECENT_OPTIMIZER_ENABLED=true`.

The API does not send customer-visible replies, close tickets, or execute
production actions. The optional Jira internal comment path is a backend-owned
review artifact only. `internalHandoffMarkdown` is the `Internal Only` message
for the selected historical reply point, not a final ticket-state summary. The
field name remains for wire compatibility; `Internal Only` does not by itself
mean that another support engineer must take ownership.
`customerReplyMarkdown` is a dry-run preview of the next L1/oncall Jira comment
and must remain behind human review. The service converts this Jira-safe
Markdown subset into `commentBodyAdf`, because Jira Cloud REST API v3 comments
are stored as Atlassian Document Format. New responses expose only the two
audience-explicit Markdown fields; historical field names are read-only
compatibility inputs.

For underspecified tickets, `clarificationSimulation` records the L1 follow-up
question, the simulated 5-minute wait for a user/customer supplement, and either
the resulting next draft or the next clarification needed.

The MVP runner is intended to be Codex-backed. The backend owns Jira access,
run workspace creation, and safety gates. Codex analyzes only the prepared
single-ticket files inside a UUID run directory.

Not in MVP:

- customer-visible Jira writes;
- arbitrary JQL, webhook ingestion, or full historical backfills;
- metrics/config/log/Clinic/Kubernetes skills;
- real workspace backend;
- production actions such as transitions, closes, or operational commands.

## Automatic Initial Analysis

Classification, Provider execution, final-answer completeness, risk handling,
and customer-publication fields follow
`docs/AUTO_ONCALL_CORE_CONTRACT.md`. Read that contract before changing this
core path.

The service can run an internal-only automatic initial analysis loop:

1. Poll Jira with a service-owned JQL query for active ST/TCID/APID/NAID/EMID
   tickets.
2. Apply poller-only non-business gates, then enqueue initial-analysis jobs.
3. Store gate decisions, initial-analysis jobs, and run state in a MySQL-compatible
   TiDB Cloud database.
4. Claim jobs with a worker, apply L1 oncaller and auto-coverage gates, run the
   live Codex analysis workspace, and write or update one Jira internal comment.
5. Send a Feishu notification for completed jobs to the configured support
   intelligence group.

The automatic Jira comment idempotency key is:

```text
auto-live:v1:<issueKey>:initial
```

The polling lane writes only the internal review comment. An independently
configured Bot Assignment lane may publish a complete `direct_answer` or a
complete, successfully executed `evidence_assisted` answer after cohort,
scenario, safety, validator, idempotency, and Jira-publication gates pass.
Neither lane transitions or closes issues or executes production operations.

Live automatic analysis and manual dryrun use the same support-analysis output
contract. Both generate reviewer context plus a simulated L1/oncall Jira reply
draft in `customerReplyMarkdown`; live mode differs only in the input slice
because it uses the current ticket state and complete paginated Jira comment
timeline, including customer, L1/support, internal, and prior bot comments. A
continuation includes that timeline only through the exact triggering `@bot`
comment. The polling worker wraps the shared result in a Jira internal comment;
only the separately triggered Bot Assignment lane can admit the normalized
draft to customer publication.

The L1 oncaller gate is conservative and represents the Support ownership
boundary, including explicitly approved L2 escalation teams. The canonical Jira oncaller source is
`assignee`; additional Jira person custom fields are used only when their field
IDs are listed in `AUTO_ONCALL_JIRA_ONCALLER_FIELD_IDS`. `reporter` is
normalized for context, but it is not treated as the oncaller. If the worker
cannot confirm that the oncaller email appears in a configured L1 Feishu
department or an explicitly allowed user list, it marks the job `skipped`
before creating an initial live analysis workspace. A skipped gate does not call Codex,
does not write a Jira internal comment, and is never notified as `succeeded`.

When Feishu app credentials are configured, the worker uses the Feishu Contact
API to recursively list configured L1 departments and their child departments,
then checks whether the Jira oncaller email appears in those department member
lists. Missing Contact API permission, missing configured department IDs, or
Contact API request failures are treated as worker failures so they can retry
and alert. Successful directory lookups that still do not find the oncaller
email in an allowed department are treated as unconfirmed L1 checks.

Automatic poll enqueue only performs non-business filtering: Jira Done issues,
the minimum issue age window, duplicate initial jobs, and optional poll JQL
narrowing through `AUTO_ONCALL_JIRA_ISSUE_ALLOWLIST` for per-ticket staging
smoke tests.

Initial-job business gates run in the worker. The L1 oncaller gate runs before auto
coverage. Auto coverage runs the scenario classifier internally and derives an
`enabled` or `disabled` execution decision from the `scenarioClass` and
`resolutionMode` combination. Jira labels are not coverage inputs. Disabled
coverage combinations are marked as skipped worker jobs before live analysis,
workspace creation, or Jira internal comments.

Every initial ticket allowed by the L1 oncaller gate is persisted in the
business measurement population before auto coverage runs. Auto coverage
skips, unsupported scenarios, and later analysis failures therefore remain in
the denominator; non-L1 tickets and continuation jobs do not enter it.

Written customer replies are reconciled read-only against Jira on the feedback
poll cadence. The stored outcome is anchored to the exact run and public
comment and records the seven-day observation deadline, next substantive L1
public reply, conservative customer event, `/ao` feedback, and final outcome.
Pending, uncertain, or unreviewed replies fail closed and do not count as
confirmed autonomous success.

An exact internal structured ADF `@bot` mention is an explicit continuation and
bypasses L1 oncaller and Enabled Coverage filtering. Trigger validity still
requires an internal, non-bot-authored, non-empty, bounded comment with the
authenticated bot account ID. Continuation remains internal-only: classification
may return `human_required`, but it cannot execute production operations or send
a customer-visible reply. It resumes the persisted Codex thread when available;
if an earlier diagnostic preflight created no thread, it starts one bounded
thread from the canonical Jira timeline and persists it for later turns.

When per-ticket Feishu notifications are enabled, they are sent only for
completed worker outcomes:

- `succeeded`
- `skipped`
- `permanently_failed`

`skipped` jobs with `oncaller_not_confirmed_l1` and `retryable_failed` jobs are
not notified. The former is a conservative L1 boundary skip, and the latter will
be retried by the worker.

Two system-failure outcomes are always sent as actionable alerts while the
global Feishu notification gate is enabled, even when ordinary per-ticket
messages are disabled:

- a scenario classifier exception that fails closed before automatic analysis;
- a final analysis failure after the worker exhausts all retries.

These alerts identify the failed stage, preserve the classifier or worker error,
show the retry outcome, and tell the oncaller to fix the dependency or input and
rerun the ticket. Intermediate retryable failures are not alerted.

User-facing per-ticket messages use the same mutually exclusive flow taxonomy
as the Feishu dashboard. They show `处理阶段` plus either `未进入原因` or
`处理结果`; raw `Decision`, legacy auto-handling labels, and gate skip codes
remain available in DB and the Job API instead of being exposed as competing
message classifications. Messages triggered by an exact internal ADF `@bot`
mention add `[@bot continuation]` to the title, including terminal failure
alerts, so they remain visibly distinct from autonomous initial-analysis
messages. The marker comes from the persisted job `triggerKind`, not from a
publication-gate reason such as `bot_assignment_required`.

When post-answer oncaller reconciliation is enabled, the service rechecks
terminal initial jobs whose current L1 gate is still allowed during the first
60 minutes after Jira ticket creation. If the current oncaller is confirmed
non-L1, a succeeded job first overwrites the exact mapped internal comment with
a short invalidation note; a coverage-skipped or permanently failed job has no
answer to invalidate. Both paths transactionally change current job/gate state
to `skipped/oncaller_not_confirmed_l1`, remove the ticket from the canonical L1
population, and immediately refresh its Feishu Base row. Missing oncaller data
and directory failures do not invalidate a ticket.

Successful live internal comments include a compact Chinese feedback prompt.
Reviewers can reply with one Jira internal comment using `/ao accept`,
`/ao revise: 原因`, `/ao wrong: 原因`, or `/ao human: 原因`. The service stores
parsed feedback in the database. Report scoring maps `accept` to 5, `revise` to
3, and `wrong` to 1; `human` is counted separately and excluded from the score
denominator.

Read-only job status APIs:

```text
GET /v1/auto-analysis/jobs/:jobId
GET /v1/auto-analysis/issues/:issueKey/latest
```

Both read APIs include `gateDecisions` so poller and worker gate decisions can
be inspected without relying on logs.

## Configuration

Jira read credentials are optional when callers pass a full `ticket` snapshot.
They are required for `issueKey` mode:

| Variable | Purpose |
|-|-|
| `AUTO_ONCALL_JIRA_BASE_URL` | Jira base URL. Falls back to `JIRA_BASE_URL`, then `https://tidb.atlassian.net`. |
| `AUTO_ONCALL_JIRA_USERNAME` | Jira username. Falls back to `JIRA_USERNAME`. |
| `AUTO_ONCALL_JIRA_API_TOKEN` | Jira API token. Falls back to `JIRA_PASSWORD`. |
| `AUTO_ONCALL_ENABLE_JIRA_INTERNAL_COMMENT_WRITE` | Must be `true` before `mode: "write"` can create an internal Jira Service Management comment. |
| `AUTO_ONCALL_CUSTOMER_REPLY_ENABLED` | Independent customer-visible reply switch. Defaults to `false`; when false, the validator and public Jira writer are never called. |
| `AUTO_ONCALL_CUSTOMER_REPLY_ENABLED_SCENARIOS` | Comma-separated scenario allowlist for customer replies. Empty by default and required even when the switch is enabled. |
| `AUTO_ONCALL_CUSTOMER_REPLY_PROJECT_KEYS` | Comma-separated Jira project cohort eligible for customer replies. Staging config uses `APID` for APAC OP tickets. |
| `AUTO_ONCALL_CUSTOMER_REPLY_CLOUD_ORG_REGIONS` | Comma-separated `TiDB Cloud Org Region` cohort. A match is eligible only when the ticket also has a non-empty typed Cloud org ID. Staging config uses `APAC`. |
| `AUTO_ONCALL_CUSTOMER_REPLY_ACCOUNT_ALIASES` | Comma-separated OP Account Alias cohort eligible for customer replies. Defaults to `Wetech,apac-test`. |
| `AUTO_ONCALL_CUSTOMER_REPLY_ORG_IDS` | Comma-separated TiDB Cloud org IDs eligible for customer replies, including test-user org IDs. |
| `AUTO_ONCALL_CUSTOMER_REPLY_MAX_BYTES` | Maximum customer reply body size. Defaults to `30000`. |
| `AUTO_ONCALL_BOT_ASSIGNEE_ACCOUNT_ID` | Jira account ID for the Auto Oncall Bot assignee. A non-empty value automatically enables the independent Bot-assignment lane. |
| `AUTO_ONCALL_BOT_ASSIGNMENT_TIMEOUT_MS` | End-to-end deadline measured from the Jira assignee changelog event. Defaults to 25 minutes (`1500000`). |
| `AUTO_ONCALL_JIRA_STATUS_FIELD_ID` | Jira select field used only for the terminal Auto Oncall Status. The field stays empty while work is queued/running/retryable. |
| `AUTO_ONCALL_JIRA_STATUS_SUCCEEDED_OPTION_ID` | Jira option ID for terminal `Succeeded`. |
| `AUTO_ONCALL_JIRA_STATUS_FAILED_OPTION_ID` | Jira option ID for terminal `Failed`. |
| `AUTO_ONCALL_JIRA_STATUS_SKIPPED_OPTION_ID` | Jira option ID for terminal `Skipped`. |
| `AUTO_ONCALL_JIRA_INTERNAL_COMMENT_MAX_BYTES` | Optional internal comment size limit. Defaults to `30000`. |
| `AUTO_ONCALL_WORKSPACE_BASE_URL` | Optional base URL used to render absolute run/workspace links. When unset, links are relative paths. |
| `AUTO_ONCALL_RUNS_ROOT` | Optional absolute root for UUID run workspaces. Defaults to the OS temp directory. |
| `AUTO_ONCALL_ENV` | `staging` or `prod`. Defaults to `staging`. |
| `AUTO_ONCALL_AUTOMATION_ROLE` | `disabled` or `active`. Defaults to `disabled`. |
| `AUTO_ONCALL_POLL_ENABLED` | Must be `true` in addition to active role before the automatic poller/worker starts. Defaults to disabled. |
| `AUTO_ONCALL_REPORTS_ENABLED` | Set to `true` to collect `/ao` feedback and generate Codex daily/weekly quality reports for both `initial` and `bot_assignment` jobs. Daily reports lead with unique ticket totals, automatic-handling counts, blocker categories, and compact per-ticket details; weekly reports retain the summary-first quality format and group AI gaps as optimized-but-unverified, optimized-and-verified, or not-yet-optimized from Recent Focus evidence. Defaults to disabled. |
| `AUTO_ONCALL_POLL_INTERVAL_MS` | Jira poll interval. Defaults to `60000`. |
| `AUTO_ONCALL_ONCALLER_RECHECK_ENABLED` | Enables current L1 ownership reconciliation for terminal initial jobs. Confirmed non-L1 jobs are reclassified and removed from L1 analytics without modifying Jira comments. Defaults to disabled. |
| `AUTO_ONCALL_ONCALLER_RECHECK_INTERVAL_MS` | Reconciliation scan interval. Defaults to `3600000` (1 hour). |
| `AUTO_ONCALL_ONCALLER_RECHECK_WINDOW_MINUTES` | Rolling Jira ticket-created window eligible for reconciliation. Defaults to `20160` (14 days); tickets outside the window are never revisited by the runtime scanner. |
| `AUTO_ONCALL_FEEDBACK_POLL_INTERVAL_MS` | Jira `/ao` feedback collection interval. Feedback may follow the published Auto-oncall Public answer or its internal fallback; a Public feedback command is accepted only from a Support-authored comment. Defaults to `14400000` (4 hours). |
| `AUTO_ONCALL_REPORT_CHECK_INTERVAL_MS` | Report due-time check interval. Defaults to `600000` (10 minutes). |
| `AUTO_ONCALL_FEEDBACK_LOOKBACK_DAYS` | Number of recent succeeded runs scanned for Jira feedback. Defaults to `14`. |
| `AUTO_ONCALL_DAILY_REPORT_HOUR` | Beijing-time hour for sending the daily Feishu report. The daily window ends at this hour, so `18` reports yesterday 18:00 through today 18:00. Defaults to `18`. |
| `AUTO_ONCALL_WEEKLY_REPORT_DAY` | Beijing-time weekday for sending the previous week's Feishu report, where Monday is `1`. Defaults to `1`. |
| `AUTO_ONCALL_WEEKLY_REPORT_HOUR` | Beijing-time hour for sending the weekly Feishu report. Defaults to `10`. |
| `AUTO_ONCALL_CODEX_REPORT_MODEL` | Model used for structured daily/weekly quality report generation. Falls back to `AUTO_ONCALL_CODEX_MODEL`, then `gpt-5.5`. |
| `AUTO_ONCALL_CODEX_REPORT_TIMEOUT_MS` | Timeout for Codex report generation. Defaults to `180000` (3 minutes). |
| `AUTO_ONCALL_GITHUB_OWNER` | GitHub repository owner for the canonical Recent Focus Issue. Defaults to `tidbcloud`. |
| `AUTO_ONCALL_GITHUB_REPO` | GitHub repository name for the canonical Recent Focus Issue. Defaults to `aisupport`. |
| `AUTO_ONCALL_RECENT_FOCUS_LABELS` | Optional comma-separated labels applied when a Focus Issue is created. Defaults to `enhancement`. |
| `AUTO_ONCALL_GITHUB_API_BASE_URL` | Optional GitHub API base URL. Defaults to `https://api.github.com`. |
| `AUTO_ONCALL_GITHUB_TOKEN` | GitHub credential for the target repository. A fine-grained token needs repository Metadata read and Issues read/write; a Classic PAT needs the `repo` scope and, when required by the organization, SAML SSO authorization. Store only in a Kubernetes Secret. |
| `AUTO_ONCALL_RECENT_OPTIMIZER_ENABLED` | Enables the isolated daily Recent Optimizer role. Defaults to disabled; staging sets it only on the CronJob and prod does not define the CronJob. |
| `AUTO_ONCALL_RECENT_FOCUS_AUTO_CREATE_ENABLED` | When the optimizer finds no Active Focus, enables bounded automatic selection from canonical recent daily report artifacts followed by a fresh baseline gate. Defaults to disabled; staging enables it only on the optimizer CronJob. Requires database configuration. |
| `AUTO_ONCALL_RECENT_FOCUS_LOOKBACK_DAYS` | Number of completed Beijing-time daily report windows considered for Focus selection. Defaults to `14`. |
| `AUTO_ONCALL_RECENT_FOCUS_MAX_SIGNALS` | Maximum bounded report-case signals supplied to the read-only Focus selector. Defaults to `40`. |
| `AUTO_ONCALL_RECENT_FOCUS_SELECTION_TIMEOUT_MS` | Timeout for structured Focus selection. Defaults to `180000` (3 minutes). |
| `AUTO_ONCALL_OPTIMIZATION_RECORDS_DIR` | Optional directory containing strict `docs/opts/*.md` optimization records. Runtime images bundle them at `/opt/auto-oncall-optimization-records`; local runs default to `./docs/opts`. |
| `AUTO_ONCALL_RECENT_OPTIMIZER_BASE_BRANCH` | Repository base branch cloned by the optimizer. Defaults to `main`. |
| `AUTO_ONCALL_RECENT_OPTIMIZER_MODEL` | Codex model used only by the bounded patch Agent. Falls back to `AUTO_ONCALL_CODEX_MODEL`, then `gpt-5.5`. |
| `AUTO_ONCALL_RECENT_OPTIMIZER_MAX_CHANGED_FILES` | Maximum candidate file count before the controller rejects the patch. Defaults to `16`. |
| `AUTO_ONCALL_RECENT_OPTIMIZER_AGENT_TIMEOUT_MS` | Timeout for the patch Agent. Defaults to `1800000` (30 minutes). |
| `AUTO_ONCALL_GITHUB_REPOSITORY_URL` | Optional HTTPS clone URL used only by the daily optimizer. Defaults to the configured GitHub owner/repository. |
| `AUTO_ONCALL_POLL_INITIAL_LOOKBACK_MINUTES` | First-run Jira lookback window. Defaults to `60`. |
| `AUTO_ONCALL_POLL_OVERLAP_MINUTES` | Cursor overlap used to avoid missed Jira updates. Defaults to `5`. |
| `AUTO_ONCALL_MIN_ISSUE_AGE_MINUTES` | Minimum Jira issue age before poll enqueue so customer supplements can arrive. Defaults to `5`. |
| `AUTO_ONCALL_POLL_BATCH_SIZE` | Jira search batch size. Defaults to `20`. |
| `AUTO_ONCALL_CODEX_MODEL` | Model for full dry-run and live support analysis, both using `high` reasoning effort. Staging pins `gpt-5.6-sol`; prod remains pinned to `gpt-5.5` instead of inheriting a mutable Codex default. |
| `AUTO_ONCALL_CODEX_TIMEOUT_MS` | Codex analysis timeout for dry-run and live worker runs. Defaults to `1320000` (22 minutes). |
| `AUTO_ONCALL_DIAGNOSTIC_MULTITURN_ENABLED` | Enables one bounded provider follow-up and a no-tool synthesis turn for eligible initial diagnostic runs. Enabled in staging for validation; production remains disabled. |
| `AUTO_ONCALL_CODEX_PIPELINE_TIMEOUT_MS` | Shared absolute Codex pipeline deadline used only when `AUTO_ONCALL_DIAGNOSTIC_MULTITURN_ENABLED=true`. Hard-capped at `1320000` (22 minutes). Stage allocation reserves synthesis and publication windows from the existing pipeline rather than extending the total deadline. The 25-minute worker/Bot deadline remains unchanged, leaving three minutes for writeback and terminal status. |
| `AUTO_ONCALL_DIAGNOSTIC_INITIAL_TIMEOUT_MS` | Feature-specific diagnostic initial-turn cap. Hard-capped at `840000` (14 minutes); does not change the legacy timeout. |
| `AUTO_ONCALL_DIAGNOSTIC_AUTH_TIMEOUT_MS` | Diagnostic provider authentication/bootstrap stage cap. Defaults to `180000` (3 minutes); a positive configured value is accepted but remains constrained by the shared pipeline deadline and downstream reserve. The provider readiness command remains capped separately. |
| `AUTO_ONCALL_DIAGNOSTIC_GAP_TIMEOUT_MS` | Feature-specific diagnostic scenario cap. Hard-capped at `960000` (16 minutes) and still constrained by the shared pipeline deadline and downstream reserve. |
| `AUTO_ONCALL_DIAGNOSTIC_SYNTHESIS_TIMEOUT_MS` | Legacy setting retained in existing manifests but ignored. Router reconciliation has a fixed code cap of `210000` (3.5 minutes). A timeout preserves the scenario packets and returns a deterministic human-review fallback instead of failing the whole run. |
| `AUTO_ONCALL_CODEX_PUBLICATION_CORRECTION_TIMEOUT_MS` | Maximum for the single fresh-thread, low-reasoning, read-only, no-network publication guard correction on both diagnostic and legacy analysis paths. Hard-capped at `60000` (60 seconds) while the shared diagnostic pipeline remains capped at 22 minutes. |
| `AUTO_ONCALL_CODEX_CLASSIFIER_MODEL` | Model for the read-only Auto Coverage semantic classifier. Staging pins `gpt-5.6-sol`; prod remains pinned to `gpt-5.5`, and neither inherits a mutable global Codex default. |
| `AUTO_ONCALL_CODEX_CLASSIFIER_TIMEOUT_MS` | Timeout for one Auto Coverage classification. Defaults to `60000` (1 minute); failures disable coverage for that job. |
| `AUTO_ONCALL_CODEX_NORMALIZE_TIMEOUT_MS` | Codex reply-normalization timeout after analysis. Defaults to `120000` (2 minutes). |
| `AUTO_ONCALL_DIAGNOSTIC_NORMALIZE_TIMEOUT_MS` | Legacy setting retained in existing manifests but ignored. Diagnostic Final Response has a fixed code cap of `75000` (75 seconds); allocation or execution timeout falls back to the already-valid Router draft. |
| `AUTO_ONCALL_WORKER_ANALYSIS_TIMEOUT_MS` | Eligible live-initial diagnostic deadline measured from worker job claim through publication pre-checks. Hard-capped at `1500000` (25 minutes); continuations, ineligible scenarios, and feature-off runs retain the legacy path. Provider/materialization and Jira clients that do not accept an abort signal are checked between calls rather than forcibly cancelled in flight. |
| `AUTO_ONCALL_CODEX_CUSTOMER_REPLY_VALIDATOR_MODEL` | Independent read-only model used for the final customer-reply validator. Defaults to `gpt-5.5`. |
| `AUTO_ONCALL_CODEX_CUSTOMER_REPLY_VALIDATOR_TIMEOUT_MS` | Timeout for the final customer-reply validator. Defaults to `60000`. Validator failures fail closed to the existing internal-comment path. |
| `AUTO_ONCALL_ATTACHMENT_INPUT_MAX_FILES` | Maximum Jira attachment inputs per run, shared by images and text. Defaults to and is hard-capped at `10`. |
| `AUTO_ONCALL_ATTACHMENT_INPUT_MAX_BYTES` | Per-attachment download limit. Defaults to `5242880` (5 MiB). |
| `AUTO_ONCALL_ATTACHMENT_INPUT_MAX_TOTAL_BYTES` | Total materialized attachment byte budget per run. Defaults to `20971520` (20 MiB). |
| `AUTO_ONCALL_ATTACHMENT_INPUT_MAX_TOTAL_TEXT_BYTES` | Total normalized TXT/HTML byte budget per run. Defaults to `524288` (512 KiB). |
| `AUTO_ONCALL_JIRA_ISSUE_ALLOWLIST` | Optional comma-separated issue keys. When set, the poller only searches those issues for staging smoke; it also forces auto coverage to `enabled` for those issues. |
| `AUTO_ONCALL_WETECH_OP_ENABLED` | Kill switch for exact OP tickets whose canonical Account Alias is `Wetech` or `apac-test`. Defaults to enabled. When set to `false`, initial jobs are disabled inside Auto Coverage Gate before workspace creation, issue allowlists cannot bypass it, and live/continuation/replay workspaces prohibit the Wetech provider. |
| `AUTO_ONCALL_MANUS_OP_ENABLED` | Kill switch for exact OP tickets whose canonical Account Alias is `Manus`. Defaults to enabled. When set to `false`, initial jobs are disabled inside Auto Coverage Gate before workspace creation, issue allowlists cannot bypass it, and live/continuation/replay workspaces prohibit the Manus provider. |
| `AUTO_ONCALL_AUTO_COVERAGE_ENABLED_SCENARIOS` | Optional comma-separated scenario classes that may run in automation when paired with an enabled resolution mode. When unset, auto coverage is unconfigured and does not block worker execution. |
| `AUTO_ONCALL_AUTO_COVERAGE_ENABLED_RESOLUTIONS` | Optional comma-separated resolution modes allowed for enabled scenarios. Defaults to `direct_answer`. |
| `AUTO_ONCALL_INVESTIBOOK_ENABLED` | Set to `true` to call the configured Investibook evidence API for eligible evidence-assisted Cloud scenarios. Defaults to disabled. When an ST/TCID SQL-performance or cluster-runtime case has an exact cluster ID and Investibook is disabled or unavailable, the workspace persists a bounded `clinic` / `agent_read_only` fallback route instead of stopping before analysis. |
| `AUTO_ONCALL_INVESTIBOOK_BASE_URL` | Investibook API base URL. Required only when Investibook is enabled. |
| `AUTO_ONCALL_INVESTIBOOK_EVIDENCE_PATH` | Evidence endpoint path under the base URL. Defaults to `/api/v1/evidence`. |
| `AUTO_ONCALL_INVESTIBOOK_API_TOKEN` | Optional bearer token for the Investibook API. Wire through Kubernetes Secret, not ConfigMap. |
| `AUTO_ONCALL_INVESTIBOOK_TIMEOUT_MS` | Investibook request timeout. Defaults to `10000`. |
| `AUTO_ONCALL_INVESTIBOOK_MAX_SUMMARY_BYTES` | Maximum normalized evidence summary size written into `input/diagnostics/`. Defaults to `16000`. |
| `AUTO_ONCALL_INVESTIBOOK_DRY_RUN_CACHE_ENABLED` | Reuses successful Investibook summaries for byte-identical dry-run/replay inputs through the shared auto-oncall database. Live initial and continuation analysis never use this cache. Defaults to disabled. |
| `AUTO_ONCALL_INVESTIBOOK_DRY_RUN_CACHE_TTL_SECONDS` | Lifetime for successful dry-run Investibook cache entries. Defaults to `604800` (7 days) and is capped at 30 days. |
| `AUTO_ONCALL_INVESTIBOOK_DRY_RUN_CACHE_EPOCH` | Manual cache namespace. Change it when an Investibook rollout must invalidate otherwise identical replay inputs. Defaults to `v1`. |

| `AUTO_ONCALL_JIRA_ONCALLER_FIELD_IDS` | Optional comma-separated Jira custom field IDs that contain oncaller user objects. These fields are fetched from Jira and considered by the L1 oncaller gate after `assignee`. |
| `AUTO_ONCALL_L1_ONCALLER_GATE_ENABLED` | Set to `false` only for an explicit emergency bypass. Defaults to enabled. |
| `AUTO_ONCALL_L1_ONCALLER_DIRECTORY_ENABLED` | Set to `false` to disable Feishu Contact API lookup. Defaults to enabled when Feishu app credentials exist. |
| `AUTO_ONCALL_L1_ONCALLER_ALLOWED_DEPARTMENT_IDS` | Comma-separated Feishu department IDs to enumerate for L1 membership checks. Child departments are included recursively. Required when Feishu directory lookup is enabled. |
| `AUTO_ONCALL_L1_ONCALLER_DEPARTMENT_ID_TYPE` | Feishu department ID type used by the configured department IDs. `open_department_id` by default; set to `department_id` only when the configured values are department IDs. |
| `AUTO_ONCALL_L1_ONCALLER_ALLOWED_DEPARTMENT_PATHS` | Optional explicit department path allowlist for injected or Jira-provided department paths. No default is applied. The live Feishu path uses department IDs instead. |
| `AUTO_ONCALL_L1_ONCALLER_ALLOWED_EMAILS` | Optional comma-separated allowed oncaller email addresses for explicit exceptions. |
| `AUTO_ONCALL_L1_ONCALLER_ALLOWED_OPEN_IDS` | Optional comma-separated allowed Feishu open IDs for explicit exceptions or injected directory clients. |
| `AUTO_ONCALL_JOB_MAX_ATTEMPTS` | Worker retry limit per job. Defaults to `3`. |
| `AUTO_ONCALL_DATABASE_URL` | MySQL/TiDB Cloud connection URL for automatic job state. Do not use the `sys` database. |
| `AUTO_ONCALL_DATABASE_HOST`, `AUTO_ONCALL_DATABASE_PORT`, `AUTO_ONCALL_DATABASE_USER`, `AUTO_ONCALL_DATABASE_PASSWORD`, `AUTO_ONCALL_DATABASE` | Split database connection settings. `AUTO_ONCALL_DATABASE` defaults to `auto_oncall`. |
| `AUTO_ONCALL_LARK_NOTIFY_ENABLED` | Global gate for Feishu notifications, including per-ticket outcomes, reports, and Recent Evaluation. Defaults to disabled. |
| `AUTO_ONCALL_LARK_TICKET_NOTIFY_ENABLED` | Set to `false` to suppress ordinary per-ticket worker outcome messages. Classifier exceptions and final analysis failures still alert while the global gate is enabled. Daily/weekly reports and Recent Evaluation notifications remain enabled. Defaults to enabled when the global gate is enabled. |
| `AUTO_ONCALL_LARK_CHAT_ID` | Feishu target chat ID. `Auto Oncall 通知` is `oc_6eeb20e998a6e75b99cb16872b9fd269`. |
| `AUTO_ONCALL_LARK_APP_ID` | Feishu app ID used for bot message sending. Store it in Kubernetes Secret. |
| `AUTO_ONCALL_LARK_APP_SECRET` | Feishu app secret used to fetch tenant access tokens. Store it in Kubernetes Secret. |
| `AUTO_ONCALL_LARK_BASE_URL` | Optional Feishu OpenAPI base URL. Defaults to `https://open.feishu.cn`. |
| `AUTO_ONCALL_SERVICE_BASE_URL` | Optional public base URL for rendering job API links in notifications. |
| `AUTO_ONCALL_TICKET_BASE_ENABLED` | Enables the independent ticket-granularity Feishu Base projection. Defaults to disabled. Base failures do not affect jobs, Jira writes, or reports. |
| `AUTO_ONCALL_TICKET_BASE_TOKEN` | Target Base token. This is resource metadata, not a credential. The Feishu app must have record read/write scopes and access to the Base. |
| `AUTO_ONCALL_TICKET_BASE_TABLE_ID` | Target `Tickets` table ID. |
| `AUTO_ONCALL_TICKET_BASE_SUMMARY_TABLE_ID` | Optional `Auto-oncall Funnel Summary` helper table ID. When set, each ticket reconciliation also upserts the ordered cumulative funnel stages aggregated from the full canonical ticket population, independently of the bounded row-refresh batch. |
| `AUTO_ONCALL_PROVIDER_BASE_TABLE_ID` | Optional `Provider Reliability Daily` table ID. When set, periodic Base sync publishes Shanghai-day and weighted-overall Clinic/Wetech/Manus logical-query success rates; Clinic also includes exact HTTP attempt counts for 200/400/500/502/other. |
| `AUTO_ONCALL_TICKET_BASE_SYNC_INTERVAL_MS` | Periodic Base reconciliation interval. Defaults to `600000` (10 minutes). |
| `AUTO_ONCALL_TICKET_BASE_START_AT` | Optional fixed ticket-created lower bound as an ISO 8601 timestamp. When set, periodic sync and manual Base backfill/sync use this fixed start instead of a rolling window. |
| `AUTO_ONCALL_TICKET_BASE_LOOKBACK_DAYS` | Rolling ticket-created lookback used to select Ticket rows for periodic incremental refresh. Defaults to `30`; it does not remove older Base records or limit the Funnel Summary population. |
| `AUTO_ONCALL_TICKET_BASE_BATCH_SIZE` | Maximum ticket facts reconciled per pass. Defaults to `500`. |

Ticket analytics are DB-canonical. Jira metadata is persisted as immutable
normalized snapshots before worker business gates; Base, daily reports, and
weekly reports are independent projections. Periodic Base sync refreshes only
the configured recent Ticket-row window, while Funnel Summary counts are
aggregated across the full canonical ticket population without re-fetching
Jira or rewriting every historical Base row. In a deployed container,
`npm run backfill:ticket-base` performs the bounded, idempotent Jira metadata
backfill and Base reconciliation used for rollout. It defaults to the latest 14
days; pass `-- --days <n>` to override the window. For a local source checkout,
use `npm run backfill:ticket-base:local` to build first.
When canonical Jira snapshots already exist, `npm run sync:ticket-base -- --days
14` performs only the lightweight MySQL-to-Base projection; the local-source
equivalent is `npm run sync:ticket-base:local -- --days 14`.
`npm run reconcile:oncallers -- --days 14` performs an explicit bounded
historical oncaller reconciliation before a Base backfill; the local-source
equivalent is `npm run reconcile:oncallers:local -- --days 14`. Repeat
`--issue-key ST-2815` to constrain an explicit repair to audited tickets, or use
`--since 2026-07-09T00:00:00+08:00` for an exact historical lower bound. Reconciliation
reads candidates in deterministic pages of 100 and fetches Jira issues serially. Base stores one
row per Ticket Key; the optional funnel helper table contains only aggregated
stage counts and definitions. Ticket-level Scenario, Coverage, Run, and evaluation
facts come from the Ticket's owner job: bot-assignment when it owns the Ticket,
otherwise initial. Both entry mechanisms participate in the same Auto-oncall main
funnel; bot-assignment itself satisfies the entry stage because the Ticket was
explicitly assigned to Auto-oncall. `Public Candidate Value` marks a bot-assignment
customer-reply gate with `publicCandidate=true`; `Public Answer Value` marks a
successfully written Jira customer reply. The summary table projects both as
the separate `public` lane, counted once per Ticket Key. Neither Base table writes business state back to
MySQL. The ticket table never
stores descriptions, comments, attachments, full AI/human answers, raw Jira
payloads, or infrastructure identifiers.

Tool and provider usage is persisted in `auto_oncall_tool_usage_events` in the
same transaction as the corresponding run. `run_id` counts individual analysis
executions, while `codex_thread_id` counts resumable Codex sessions. Events
separate `skill_load` from actual `skill_tool_call`, `mcp_tool_call`,
`web_search`, and backend `provider_call` operations. Investibook records one
high-level `diagnosis` event per attempted run. Wetech query commands are
normalized into operations such as `metrics`, `topsql`, `statements`, and
`slow_query`. The table stores only normalized fields, hashes, status, and
bounded metadata; it does not store raw commands, provider arguments, PromQL,
web queries, or command output.

Provider reliability uses separate run-transactional facts in
`auto_oncall_provider_query_events` and `auto_oncall_provider_http_attempts`.
One logical Clinic/Wetech/Manus query is successful when its final batch status
is `complete` or `empty`; `failed` is unsuccessful, and transport retries do
not expand that denominator. Clinic HTTP attempts are counted separately so
200, 400, 500, 502, and other responses remain distinguishable. The Base
projection writes one Shanghai-natural-day row per Provider plus one weighted
overall row, and starts only from events captured after this contract is
deployed; incomplete historical 5xx-only data is not used to invent success
rates.

For example:

```sql
SELECT
  provider,
  skill_name,
  operation,
  COUNT(*) AS invocation_count,
  COUNT(DISTINCT run_id) AS run_count,
  COUNT(DISTINCT codex_thread_id) AS codex_session_count
FROM auto_oncall_tool_usage_events
WHERE occurred_at >= ? AND occurred_at < ?
GROUP BY provider, skill_name, operation;
```

The staging Deployment requires `AUTO_ONCALL_GITHUB_TOKEN` from the existing
`auto-oncall-services-secret`, and the staging ConfigMap enables daily GitHub
issue delivery. Provision or rotate the secret key operationally; never commit
the token or copy a developer's local `gh` credential into deployment files.
The staging Daily Recent Optimizer additionally needs that token to clone,
push one generated branch, and create a Draft PR, so its fine-grained token
must also have Contents and Pull requests read/write for this repository.

Report generation is intentionally split into deterministic data selection and
Codex-authored content. The backend selects successful AI drafts and stable
customer-visible support replies, marks pairs with intervening customer input
as `context_changed`, and sends only bounded comparison input to a read-only
Codex thread. Codex writes the important Summary and per-ticket comparison only;
it does not create a repair plan. The backend stores the report artifact and
sends Feishu independently of GitHub. Long-lived repair planning and resumable
execution live only in the canonical Focus Issue.

Historical report membership is anchored to the job's immutable `created_at`
window. Both `initial` and `bot_assignment` jobs participate in daily/weekly
totals, comparisons, feedback, and persisted automatic evaluation. Regeneration
selects exactly the latest successful run for each member job, even when that
rerun completed after the original window. The comparison anchor is the actual
published Auto-oncall comment: a written Public customer reply takes precedence,
with the written/updated internal comment as the fallback. A substantive
public support reply may be paired either before or after the selected AI run;
the timing is recorded as `before_ai` or `after_ai`, while material intervening
customer context still makes the pair `context_changed`.

For a read-only local preview of the daily window ending at 18:00 Beijing time,
use `npm run preview:daily-report -- YYYY-MM-DD`. The preview reads the configured
database and Jira, calls the Codex report generator, and writes only to stdout;
it does not claim a report artifact, create a GitHub issue, or send Feishu.

After changing historical report membership or evaluation rules,
`npm run backfill:report-evaluations -- --days 14` regenerates existing ready
daily/weekly artifacts in the bounded window, replaces their persisted report
evaluations, recollects eligible `/ao` feedback, and refreshes affected Ticket
Base rows plus the summary. It does not run ticket analysis, write Jira comments,
or resend Feishu reports. Historical feedback collection scans up to 500
published answer versions in the bounded window; the periodic runtime collector
retains its normal 100-version batch. From a local source checkout, use
`npm run backfill:report-evaluations:local -- --days 14` to build first.

The default Feishu dashboard quality refresh is
`npm run refresh:dashboard-evaluations -- --days 14`. It selects the rolling
window by the canonical `AI Evaluated At` value and selects only poor cases with
`Feedback=human`. It reuses the shared AI-vs-human Judge, writes answer
evaluations directly, and syncs the affected Base rows and summary. It does not
read or rewrite daily/weekly report artifacts, run ticket
analysis, or write Jira comments. The default window is 14 days; from source,
use `npm run refresh:dashboard-evaluations:local -- --days 14`.

Feishu setup required before enabling notifications:

1. Add the service bot to `Support Intelligence Campaign`.
2. Enable the bot message scope `im:message:send_as_bot` for the Feishu app.
3. Configure `AUTO_ONCALL_L1_ONCALLER_ALLOWED_DEPARTMENT_IDS` with the Feishu
   L1 department ID(s) that should be treated as allowed.
4. Enable Contact API read scopes needed by the L1 oncaller gate. The directory
   membership lookup uses `GET /contact/v3/departments/:department_id/children`
   and `GET /contact/v3/users/find_by_department`. The app needs Contact API
   read access for departments/users, app visibility over the configured
   department range, and `contact:user.email:readonly` so member emails are
   returned for matching.
5. Store `AUTO_ONCALL_LARK_APP_ID` and `AUTO_ONCALL_LARK_APP_SECRET` in
   `auto-oncall-services-secret`.
6. Set `AUTO_ONCALL_LARK_NOTIFY_ENABLED=true` only after a staging smoke test.

Deployment reads runtime credentials from the environment provided by the
service deployment manifests and Kubernetes Secret wiring. Do not commit secret
values to this repository or to app-delivery.

Temporary staging Codex auth is stored in the writable
`auto-oncall-services-codex-home` PVC mounted at `$CODEX_HOME`. Do not copy a
developer workstation's active `auth.json` into staging because ChatGPT refresh
tokens rotate during use and must not be shared by two environments. For smoke
testing, exec into the staging pod and run a dedicated Codex login there so the
pod writes and refreshes its own `$CODEX_HOME/auth.json`.

## Local Development

```bash
npm install
npm run build
npm test
npm run dev
```

The service listens on `PORT`, defaulting to `8080`.

```bash
curl http://localhost:8080/
curl -X POST http://localhost:8080/v1/tickets/analyze \
  -H 'content-type: application/json' \
  -d '{"issueKey":"TCOC-123","mode":"read"}'
```

## Docker

```bash
docker build -f Dockerfile.runtime-base -t auto-oncall-services-runtime-base:local .
docker build \
  --build-arg AUTO_ONCALL_RUNTIME_BASE_IMAGE=auto-oncall-services-runtime-base:local \
  -t auto-oncall-services:hello \
  .
docker run --rm -p 8080:8080 auto-oncall-services:hello
```

Deployment workflows tag the runtime base image from the SHA-256 digest of
`Dockerfile.runtime-base` and build it only when that tag is absent in the
target ECR repository. Normal application releases therefore reuse the pinned
Playwright, Chromium, Python, and OS dependency layers.

## Deployment

This service follows the same deployment model as the existing ops services:

- GitHub Actions builds and pushes an image to ECR.
- The workflow dispatches `tidbcloud/app-delivery` to bump the image tag.
- ArgoCD syncs to the target Kubernetes cluster.

Targets:

- Agent9 main staging: ECR `aisupport/staging/auto-oncall-services`,
  app-delivery path
  `staging/devops-service/auto-oncall-services-agent9/envs/staging`, cluster
  `devops-staging`, namespace `auto-oncall-services-agent9`.
- staging auxiliary jobs: the same image, app-delivery path
  `staging/devops-service/auto-oncall-services/envs/staging`, cluster
  `devops-staging`, namespace `auto-oncall-services-staging`. This application
  retains the Recent Optimizer, Codex-history cleanup, ConfigMap, and their
  PVCs; it no longer renders a legacy AOS Deployment or Service.

See `deploy/app-delivery/README.md` for the required app-delivery registration.