# Retrospective Report Reruns

## Current Design

- Daily membership uses `auto_oncall_jobs.updated_at`, so re-running a job can
  move it out of its original report window.
- Comparison targets are successful runs created before the report cutoff, so
  a later successful rerun cannot replace the original AI answer in a
  regenerated historical report.
- Human pairing only scans comments after the AI Jira comment and before the
  period end. A valid human reply before the AI run, or after the daily cutoff,
  is rendered as `no_human_reply`.
- Slack-bridged acknowledgement bodies retain their transport prefix during
  substantive-content checks, which can incorrectly look like new context.
- Ready report artifacts and channel deliveries are immutable until an
  explicit operational reset.

## Risks

- Selecting every run would double-count one job and reuse the same human
  answer across obsolete AI attempts.
- Pairing arbitrary support comments from the whole ticket can compare answers
  for different customer questions or later context.
- Using a rerun timestamp for membership can rewrite historical ticket totals.
- Reprocessing a sent artifact without preserving delivery state can duplicate
  GitHub issues or Feishu messages.

## Target Contract

- A job belongs to the daily or weekly window containing its immutable
  `created_at`; later retries and reruns do not change membership.
- A regenerated report uses the latest successful run for each member job,
  even when that run completed after the report period.
- Human reference answers may be earlier or later than the selected AI run.
  Prefer the first valid post-AI support answer; if none exists, use the nearest
  valid pre-AI support answer.
- Reply timing is recorded as `before_ai` or `after_ai` but timing alone does
  not make a pair ineligible. Material customer-context changes retain the
  existing `context_changed` boundary.
- Only public comments with a stable support identity are eligible. Internal,
  automation, customer, unknown-author, and acknowledgement-only comments are
  not human reference answers.
- DB report artifacts remain canonical. Historical regeneration still requires
  a bounded snapshot/reset and reuses stable GitHub issue markers.

## Implementation Steps

1. Change report membership from job `updated_at` to job `created_at`.
2. Select exactly the latest successful run for every member job.
3. Add bidirectional human pairing and explicit reply-timing metadata.
4. Pass reply timing into the bounded Codex report input.
5. Normalize bridge transport prefixes before acknowledgement checks.
6. Add local contracts for pre-AI replies, post-cutoff replies, latest-run
   selection semantics, acknowledgements, and unchanged visibility boundaries.
7. Run typecheck and the complete test suite, then deploy through the existing
   staging/app-delivery workflows.
8. Snapshot, reset, and regenerate the affected staging daily artifacts; verify
   APID-29167 uses its rerun and APID-29170/APID-29173 use real human replies.

## Verification Checklist

- [x] Job membership remains in the original `created_at` window after rerun.
- [x] Only the latest successful run for a job reaches the report generator.
- [x] A substantive pre-AI support reply is comparable.
- [x] A substantive post-period support reply is comparable on regeneration.
- [x] Internal, bot, customer, unknown, and acknowledgement comments remain excluded.
- [x] `npm run typecheck` passes.
- [x] `npm test` passes.
- [x] Staging runs the merged image and remains ready.
- [x] Regenerated artifacts contain the expected APID cases and latest run IDs.
- [x] GitHub issue delivery is idempotent and Feishu delivery is verified.
- [x] Prod config and state remain unchanged.

## Verification Outcome

- Source PRs: `tidbcloud/aisupport#297` and `#298`; staging image:
  `4e3ed9162a7229d311bc364b7256bd430ac97948`.
- app-delivery PRs: `tidbcloud/app-delivery#74525` and `#74528`; the
  rendered Deployment runs the same image and is ready `1/1`.
- The 2026-07-10 artifact is `124`. It selects APID-29167 run `60007`,
  compares APID-29170 with a `before_ai` support reply, and compares
  APID-29173 with an `after_ai` support reply.
- The 2026-07-11 artifact is `125`; APID-29173 no longer carries into this
  period because membership is anchored to the original job window.
- Existing GitHub issue `#283` retained its stable marker and was reconciled
  to the regenerated content. The regenerated second report has no actionable
  optimization and therefore does not link the prior issue `#286`.
- Feishu messages were read back from `Support Intelligence Campaign`:
  `om_x100b6a7e8fd7d0acc49cdebd249ef43` and
  `om_x100b6a7ee4f840f0dfa140cd6f4aba7`.
- Only staging app-delivery image paths changed; prod manifests and automation
  state were not modified.

## Non-Goals

- No change to live analysis input, Jira comment visibility, customer replies,
  gates, polling, or production automation.
- No cross-ticket human-answer matching.
- No automatic mutation of already-sent report artifacts without an explicit
  operational reset.
- No use of support/internal reply bodies as model input during live analysis.