# Codex Quality Report And Daily GitHub Issue

## Current Design

- `sendReportsOnce()` reads jobs and `/ao` feedback from MySQL, then
  `buildAutoOncallReportMarkdown()` deterministically concatenates counters.
- Codex generates each ticket's normalized `commentDraftMarkdown`, but does not
  participate in daily or weekly report generation.
- Jira request comments are currently used only to collect internal `/ao`
  feedback. The report path does not fetch public support replies.
- `auto_oncall_report_deliveries` owns one combined Feishu delivery status. It
  cannot persist the Codex generation separately from external channel writes.
- There is no GitHub Issues runtime client or delivery state.

## Risks

- Public Jira comments can be written by either customers or support. Codex
  must not guess author roles.
- A customer supplement between the AI draft and the human reply makes a direct
  quality comparison unfair.
- Report retries must not regenerate different Codex content or duplicate a
  GitHub issue.
- GitHub and Feishu can fail independently. One successful channel must not be
  rolled back or repeated because the other failed.
- Full ticket bodies and full AI/human answers may contain customer data and
  must not be copied into GitHub issues.
- Existing dirty workspace changes own the replay-safe Jira identity contract;
  this implementation must preserve them and avoid a second conflicting
  comment model.

## Target Contract

```text
DB jobs/runs/feedback
  + run-time frozen customer question context
  + paginated Jira request comments
  -> deterministic answer pairing
  -> immutable report input
  -> Codex structured report generation
  -> DB canonical report artifact
  -> idempotent GitHub daily issue delivery
  -> Feishu report containing the issue URL
```

### Data and author boundaries

- AI answer source: `auto_oncall_runs.comment_draft_markdown` only.
- Customer question source: `auto_oncall_run_review_context` captured from the
  same `SupportCase` used by the live run. The report never refetches the
  current Jira description as a fallback.
- Human answer source: customer-visible Jira request comments whose stable Jira
  author account type identifies an Atlassian/support user.
- Internal comments, `/ao` feedback, customers, apps, automation and unknown
  authors are never human answers.
- Eligible human comments are assigned to the nearest preceding successful AI
  run for that Jira issue. One human answer block belongs to at most one run.
- If a customer public comment appears after the AI run and before the human
  answer, mark the pair `context_changed`; Codex may describe it but may not
  declare one answer better.
- Report-window membership uses the human answer completion time. New runs in
  the window without a human answer are counted as pending.

### Codex report contract

- Input contains deterministic stats and bounded answer pairs. Codex has no
  network and no write tools.
- The generated artifact records total/included/truncated sampling counts, and
  both Feishu and GitHub render those deterministic counts.
- Output is strict JSON: important summary, comparison narrative, per-case
  verdicts and structured optimization items.
- The renderer always places `## 重要 Summary` first, immediately after the
  report title. The summary and core analysis are Codex content; the template
  only enforces order, metrics and links.
- Non-comparable pairs must return `not_compared`.
- Optimization evidence keys must come from the input.
- If Codex fails, keep the artifact failed and retry generation; do not publish
  a template-only report as if it satisfied the Codex report contract, and do
  not create an optimization issue.

### Report and delivery state

- A new DB report artifact is canonical for a period. Once ready, retries reuse
  the exact generated JSON and do not invoke Codex again.
- GitHub and Feishu use separate channel delivery rows with retryable state.
- Daily reports create at most one issue when Codex returns actionable
  optimizations. Weekly reports use Codex content but do not create duplicate
  issues.
- GitHub idempotency uses both a DB unique key and a stable hidden marker.
- For an actionable daily report with GitHub enabled, Feishu delivery waits
  until the independently retryable GitHub channel succeeds so the one report
  always contains the issue URL. Feishu sends use a stable API `uuid` to
  deduplicate ambiguous retries.
- If GitHub delivery is disabled, an actionable daily artifact remains pending
  instead of sending a permanently URL-less Feishu report. Weekly and daily
  reports without optimization items remain independent of GitHub.

## Invariants

- Read paths do not create issues or send reports.
- Codex never classifies author identity and never writes external state.
- Human answers never enter the original live analysis context.
- Customer supplements are never silently treated as comparable context.
- A report generation is immutable after `ready`.
- External issue bodies contain Jira keys, short gap summaries and acceptance
  criteria, never full questions/answers/raw payload.
- GitHub-bound model fields are reduced to single-line Markdown-safe text and
  redact credentials, tokens, email, phone, IP, UUID, resource IDs, URLs and
  mentions before the API call.
- Prod remains disabled unless separately approved.

## Implementation Steps

1. Add a paginated Jira request-comment reader and deterministic comparison
   selector using stable identity/visibility metadata.
2. Add bulk report targets from succeeded runs, including AI drafts and Jira
   comment anchors.
3. Add the structured Codex report generator, schema/parser, bounded prompt and
   deterministic renderer with Summary first.
4. Add canonical report artifacts and per-channel delivery state.
5. Add an idempotent GitHub Issues REST client and daily publisher.
6. Replace the report loop with collect -> generate once -> GitHub -> Feishu.
7. Add README/config/app-delivery wiring. GitHub credentials come only from a
   manually-created Kubernetes Secret; no value or ExternalSecret is committed.

## Local Contract Tests

- Jira pagination, stable ordering, missing fields, public customer, public
  support, internal, app/bot and unknown authors.
- Same issue with multiple runs; one human block binds only to the nearest run.
- Customer supplement produces `context_changed`; no human reply is pending.
- Human reply time controls the daily `[start,end)` window.
- Codex schema parsing, invalid output, non-comparable verdict enforcement,
  evidence-key validation and bounded payload.
- Summary is before metrics and comparison sections.
- Report generation persists once and is reused after retries.
- GitHub marker reconcile, create, API errors and no-action no-op.
- GitHub success/Feishu failure retries only Feishu; weekly never creates an
  issue; prod/disabled config never calls GitHub.
- Existing `/ao`, poller/worker, Jira write and deterministic fallback behavior
  remain green.

## Verification Checklist

- `npm run typecheck`
- `npm test`
- Inspect the complete diff without including unrelated dirty changes.
- Render staging/prod app-delivery templates; staging can be enabled only after
  the manually-created GitHub credential exists, prod remains disabled.
- After deployment verify pod env, report artifact rows, channel delivery rows,
  the GitHub issue marker/URL and the Feishu body readback.

## Non-Goals

- No automatic code/prompt edits, PR creation or merge.
- No customer-visible Jira writes or issue transitions.
- No production operations and no prod cutover.
- No GitHub issue for a report with no trusted optimization item.
- No legacy workspace or Jira-body fallback as a source of truth.