# `cmd/replay` — Audit Log Replay Tool

> Deep-dive reference for the `cmd/replay` CLI binary. Complements the short
> "Using the replay script" section in [`runbook.md`](./runbook.md). Design
> reference: [`design.md`](./design.md) §14.

## 1. What problem it solves

The audit write path (`_System_AuditLog` + `_System_AuditOutbox`) is a
complex distributed transaction layered under `PersistenceService`. When we
refactor it — retire the legacy `generateAuditLogs` path, change field-diff
semantics, or move to a new outbox store — we need a mechanical way to prove
**the new code produces the same audit trail as the old one for the same input**.

The replay tool operates on the premise that a production/staging system
already captures every incoming HTTP request into `_System_RequestCapture`
(with PII scrubbed per §14.3). Given a source database with both capture rows
and their resulting audit rows, the tool can:

- **Dry-run** — load capture + audit, report what an audit trail looks like
  for a given RequestId or time window. No writes. Useful for forensics and
  for verifying that `RequestId` actually flows from capture to audit (the
  primary correlation invariant of PR #1850).
- **Active replay** — re-POST each captured request to a separate *target*
  backend, then read the audit rows the target produced (keyed by the same
  `RequestId`), and diff them against the source's expected rows.

The active mode answers: "would the target system, faced with the exact same
traffic, emit the same audit trail?" An all-match result is behavior-
equivalence proof across a refactor boundary. A mismatch is an explicit
categorized diff.

## 2. Pipeline

```
┌────────────────────────────────────────────────────────────────────┐
│ Source DB                                                          │
│   _System_RequestCapture            _System_AuditLog               │
│   • RequestId (PK)           ─┐     • RequestId                    │
│   • Body (scrubbed)           │     • ObjectApiName                │
│   • Method / Path / Query     │     • FieldName (__record__ etc.)  │
│   • ReceivedAt                │     • Action / Old / New           │
└───────────────────────────────┼────────────────────────────────────┘
                                │ RequestId
                                ▼
┌────────────────── cmd/replay Run() ───────────────────────────────┐
│  1. LoadCaptures(source, filter{Since,Until,RequestID,UserID,Limit}) │
│  2. for each capture:                                             │
│       expected := LoadAuditRowsForRequest(source, cap.RequestID)  │
│       if len(expected)==0: SkippedNoAudit (orphan); continue      │
│                                                                   │
│       if dry-run (no --target-dsn):                               │
│         diff := {Match:true}   # self-compare placeholder         │
│                                                                   │
│       else:                                                       │
│         status := postCapture(targetURL, cap)                     │
│           # POST cap.Body → cap.Path                              │
│           # Header: X-Replay-RequestId = <cap.RequestID>          │
│           # Target writes new AuditLog rows under the same rid    │
│         replay := LoadAuditRowsForRequest(target, cap.RequestID)  │
│         diff   := DiffAuditTrails(expected, replay, mode)         │
│                                                                   │
│       append to report                                            │
│  3. WriteReport(JSON → --diff-output | stdout)                    │
│     Summary: Captures / Matched / Mismatched / SkippedNoAudit /   │
│              HTTPErrors                                           │
└───────────────────────────────────────────────────────────────────┘
```

## 3. Modes

### Dry-run (default, read-only)
- Invoked by **not** passing `--target-dsn`.
- No writes to any DB; no HTTP traffic.
- `Match` is always `true` in the per-entry diff (the trail is compared
  against itself); the actual signal is **what rows exist** under a
  `RequestId` and how they aggregate across a time window.
- Use cases:
  - **Correlation check** — verify `_System_RequestCapture.RequestId` joins
    cleanly to `_System_AuditLog.RequestId`. If `SkippedNoAudit > 0`, the
    audit writer isn't propagating RequestId (the PR #1850 bug that C17.3
    + `0967a8a3` fixed).
  - **Forensic export** — `--diff-output` produces a JSON dossier per
    request for compliance review.
  - **Regression safety net** — run the same window before and after a
    deploy; `Captures` and per-request `Expected` size should be stable.

### Active replay (read + write to target)
- Requires ALL of:
  - `--target-dsn` (a separate MySQL-compatible DSN — must be non-prod)
  - `--target-url` (base URL of the target backend)
  - `--i-understand-this-writes` (explicit boolean gate)
- For every capture: `postCapture()` re-issues the captured HTTP request
  against `target-url`, attaching `X-Replay-RequestId: <rid>`. The target's
  request-id middleware (C17.3 +0967a8a3) honors the header and threads the
  rid into `_System_AuditLog.RequestId`. The tool then reads the target DB
  by that rid and diffs.
- Use cases:
  - **Refactor boundary** — run prod-captures against a staging build of
    the refactor branch. Zero `Mismatched` is the acceptance bar.
  - **Cross-version regression** — same tape, old build vs new build; diff
    shows where behavior drifted.

### Single-DB mode (active replay)

When the baseline and the replay target are literally the SAME database
(e.g. you're validating a refactor on your dev TiDB without cloning the
schema), set `--replay-rid-suffix=-replay` (or any non-empty string) along
with `--target-dsn` equal to `--db-dsn`. The tool will:

- Leave baseline audit rows under their original `RequestId`.
- Re-POST captures with `X-Replay-RequestId: <rid><suffix>`, so the
  refactored backend writes new audit rows under `<rid><suffix>`.
- Load baseline rows by `<rid>`, replay rows by `<rid><suffix>`, and diff
  normally.

Baseline and replay never collide in the same table.

**Diff under different target RecordIds.** With `--replay-rid-suffix` + active
replay, CREATE operations on the target allocate a fresh RecordId. The default
differ keys on `ObjectApiName|RecordId|FieldName|Action`, so baseline and
replay rows are treated as disjoint and `Match=false` even when behavior is
identical. When you only care about "did the same audit SHAPE get produced",
pass `--diff-ignore-record-id` and the differ will blank RecordId on COPIES of
both sides before comparing. The JSON report's Expected / Replay lists still
show the real RecordIds (for debugging); only the diff Key is degraded.

Requirements:
- `--target-dsn` must equal `--db-dsn` OR be distinct with empty suffix
  (for the classic dual-DB mode).
- Empty suffix + equal DSNs is rejected at validation to prevent
  accidentally overwriting baseline rows.
- Empty suffix + distinct DSNs keeps the original behavior.

Example:
```bash
replay \
  --db-dsn='root:@tcp(127.0.0.1:4000)/nexuscrm?parseTime=true' \
  --target-dsn='root:@tcp(127.0.0.1:4000)/nexuscrm?parseTime=true' \
  --target-url=http://localhost:3855 \
  --i-understand-this-writes \
  --replay-rid-suffix=-refactor-2026-04-24 \
  --replay-auth-bearer="$TOKEN" \
  --diff-ignore-record-id \
  --since=2026-04-24T00:00:00Z --until=2027-01-01T00:00:00Z \
  --diff-output=/tmp/refactor.json
```

(`$TOKEN` is a short-lived Bearer issued by the target backend; see §8.a.)

## 4. Diff algorithm (`internal/features/audit/replay/differ.go`)

Compares two `[]AuditRow` slices. Each `AuditRow` is a narrow projection
`{ObjectApiName, RecordId, FieldName, Action, OldValue, NewValue, Entrypoint}`.

### Multiset key
```
Key = ObjectApiName | RecordId | FieldName | Action
```
Id and ChangedAt are excluded — they change per write by design. The tool
uses **counted multiset equality**: if Expected has K=3 rows under one Key
and Replay has K=2, that's a `count-mismatch`, not a silent collapse.
(C17.3 fix; verified by `TestDiffAuditTrails_Multiset_DuplicateRowsNotCollapsed`.)

### Difference kinds
| Kind | Meaning |
|------|---------|
| `missing` | Key exists in Expected, not in Replay |
| `added` | Key exists in Replay, not in Expected |
| `field-changed` | Key matches but OldValue or NewValue differs |
| `count-mismatch` | Key matches on both sides but multiplicity differs |

### Modes
| Mode | Semantics |
|------|-----------|
| `tolerant` (default) | Ignore `EntrypointDetail` and other informational fields; Old/New value equality is normalized (trim, basic JSON comparison when applicable). |
| `strict` | Byte-identical OldValue/NewValue on every row. |

Tolerant is the right default for refactor-equivalence checks because
informational fields are allowed to evolve; strict is the right default for
forensic integrity audits.

## 5. CLI reference

| Flag | Required | Default | Purpose |
|------|----------|---------|---------|
| `--db-dsn` | **yes** | — | Source DB DSN (`user:pass@tcp(host:4000)/nexuscrm?parseTime=true`) |
| `--since` | no | unbounded | ISO-8601; `ReceivedAt >= since` |
| `--until` | no | `now` | ISO-8601; `ReceivedAt < until`. **Capture table stores local TZ today** — pass UTC explicitly (e.g. `2027-01-01T00:00:00Z`) to cover recent rows safely; see §14.1 item below. |
| `--request-id` | no | — | Replay only this RequestId (debug) |
| `--user-id` | no | — | Restrict to captures by this UserId |
| `--limit` | no | 100 | Maximum captures processed |
| `--mode` | no | `tolerant` | `tolerant` or `strict` |
| `--target-dsn` | no | — | Target DB (enables active replay) |
| `--target-url` | no | — | Target backend base URL (required with `--target-dsn`) |
| `--i-understand-this-writes` | conditionally | `false` | Second-gate acknowledgment; required with `--target-dsn` |
| `--replay-rid-suffix` | conditionally | `""` | Suffix appended to `RequestId` for replay rows. **Required** when `--target-dsn == --db-dsn` (single-DB mode). Rejected without `--target-dsn`. |
| `--replay-auth-bearer` | no | — | Bearer token for the `Authorization` header on replayed requests. Required when target backend requires auth. Captures have scrubbed Authorization per §14.3, so it must be supplied externally. Rejected without `--target-dsn`. |
| `--diff-ignore-record-id` | no | `false` | Ignore RecordId when comparing. Use for single-DB active replay where CREATE allocates a new target RecordId. Diff keys degrade to `ObjectApiName \| FieldName \| Action`. Report's Expected / Replay still carry the real RecordIds. **Precondition:** sound only for captures that produce one row per RecordId per `(Action, FieldName)`. For bulk / composite requests that emit multiple CREATE rows with the same `(ObjectApiName, FieldName, Action)` but different RecordIds, this flag collapses them into one bucket; `sortBucket` byte-compares OldValue/NewValue arrays, so ordering-divergence across multi-record regressions can silently pass. For multi-record equivalence, use explicit dual-DB mode with a schema-cloned target and a known RecordId mapping. |
| `--diff-output` | no | stdout | Path to write the JSON report (or `-` for stdout) |

## 6. Report schema

```jsonc
{
  "generated_at": "2026-04-24T13:02:11Z",
  "mode": "tolerant",
  "dry_run": true,
  "summary": {
    "captures": 35,
    "matched": 7,
    "mismatched": 0,
    "skipped_no_audit": 28,
    "http_errors": 0
  },
  "entries": [
    {
      "request_id": "matrix-1777013348-A2-update",
      "received_at": "2026-04-24T06:49:59Z",
      "method": "PATCH",
      "path": "/services/data/v63.0/sobjects/Account/<id>",
      "expected": [ /* AuditRow[] from source DB */ ],
      "replay":   [ /* AuditRow[] from target; absent in dry-run */ ],
      "replay_http_code": 204,
      "replay_skipped":   "",
      "diff": {
        "match": true,
        "mode": "tolerant",
        "expected_size": 2,
        "replay_size": 2,
        "differences": [ /* Difference[] */ ]
      }
    }
  ]
}
```

## 7. Typical invocations

### a) Verify capture → audit correlation on localhost
```bash
replay \
  --db-dsn='root:@tcp(127.0.0.1:4000)/nexuscrm?parseTime=true' \
  --since=2026-04-24T00:00:00Z \
  --until=2027-01-01T00:00:00Z \
  --diff-output=/tmp/correlation.json
# Expect: Summary.SkippedNoAudit == 0 for any RequestId that reached
#         the V2 emit path. Non-zero = the request-id propagation bug
#         is back (regression of PR #1850 fix 0967a8a3).
```

### b) Single-request forensic snapshot
```bash
replay \
  --db-dsn='...' \
  --request-id=verify2-1777012973-update \
  --until=2027-01-01T00:00:00Z \
  --diff-output=/tmp/one.json
# Expect: captures=1 matched=1 mismatched=0 exit 0
```

### c) Refactor-boundary replay against a staging target
```bash
replay \
  --db-dsn='user:pw@tcp(prod-replica:4000)/nexuscrm?parseTime=true' \
  --target-dsn='user:pw@tcp(staging:4000)/nexuscrm?parseTime=true' \
  --target-url=https://staging.nexuscrm.example.com \
  --i-understand-this-writes \
  --since=2026-04-24T00:00:00Z --until=2026-04-24T01:00:00Z \
  --mode=strict \
  --diff-output=/tmp/refactor.json
# Expect Summary.Mismatched == 0 before approving the refactor PR.
```

## 8. Known limitations (as of PR #1850)

All tracked in the design doc §14.8 or the PR's deferred list.

1. **Body is already scrubbed at capture time.** Any field the PII scrub
   redacted cannot be faithfully replayed — the target receives
   `"***SCRUBBED***"` as the field value. Consequence: replay cannot
   reproduce auth flows or anything gated on secret inputs. A future
   secrets-vault RFC (§14 follow-up) would let privileged replay mirror
   un-scrubbed bodies.
2. **Non-JSON bodies skipped.** `postCapture` returns an error for
   non-JSON content types (form-urlencoded, XML, multipart). These
   captures are reported with `replay_skipped`. §14.8 RecordId-rewrite /
   multi-step tape support for them is Phase 2B.
3. **No RecordId / auth / idempotency rewrite.** The captured request is
   replayed verbatim. If the target DB has different Account Ids from
   source, the replay POST may succeed but the resulting audit will
   reference a different record. For refactor-equivalence against a
   cloned DB this is fine; for cross-environment replay it is not.
4. **Target must be non-production.** Active replay writes real data;
   `--i-understand-this-writes` is a required second gate, but it does
   not prevent operator error — use a dedicated branch / sandbox.
5. **`X-Replay-RequestId` requires the target to honor the header.** PR
   #1850 C17.3 + `0967a8a3` added this behavior to the request-id
   middleware. A target running pre-PR code will allocate a fresh rid
   and the diff join will be empty.
### 8.a Auth on replay (mitigation for §8.1)

Because the captured `Authorization` header is scrubbed at ingest, the
original Bearer token cannot be recovered from the capture. Use
`--replay-auth-bearer <token>` to supply a valid token for the target
backend. The token is NOT logged by the tool and is NOT persisted
anywhere — it lives only in the process memory of the replay run.

Recommended pattern:
```
TOKEN=$(login-to-target-via-oauth)  # external
replay --replay-auth-bearer="$TOKEN" ...
```

Operator responsibility: the bearer must be short-lived and must have
the perms required by the captured operations on the target. Using a
long-lived or over-privileged token is a security risk.

This is an interim mechanism. A §14 follow-up would let the target
backend trust `X-Replay-RequestId` headers for pre-authorized replay
tapes, removing the need for operator-supplied Bearer tokens.

6. **Capture `ReceivedAt` is stored in local TZ today.** The `--since` /
   `--until` default comparison uses UTC; if you see apparent-empty
   windows on a freshly-ingested row, pass `--until=<far-future-UTC>`
   to widen the window. Fix tracked as a follow-up (one-line conversion
   at capture-write time).
7. **`_System_RequestCapture.Path` is VARCHAR(512).** The InnoDB index-key-
   length limit with utf8mb4 on the `idx_path` secondary index caps Path
   at 512 chars. Requests with URL paths longer than that are WARN-logged
   and the capture row is dropped (the request itself still succeeds).
   Current route vocabulary in `backend/internal/interfaces/rest/**` maxes
   at ~150 chars, so this is theoretical today — worth documenting for
   deployments that register custom endpoints with long path templates.

## 9. How the tool relates to the rest of the system

| Dependency | Where |
|------------|-------|
| `_System_RequestCapture` shape | seed `backend/internal/bootstrap/system_tables.json` |
| `_System_AuditLog.RequestId` propagation | `backend/internal/features/audit/v2_hook.go` + `audit_v2_wiring.go` (0967a8a3) |
| `X-Replay-RequestId` header honored | `backend/internal/interfaces/middleware/request_id.go` (C17.3 `d9f3cdce`) |
| Differ algorithm | `backend/internal/features/audit/replay/differ.go` |
| Reader (SQL) | `backend/internal/features/audit/replay/reader.go` |
| CLI | `backend/cmd/replay/main.go` |
| Unit tests | `replay/differ_test.go`, `replay/reader_test.go`, `cmd/replay/main_test.go` |

## 10. Live evidence (PR #1850 verification runs)

Archived under `~/Documents/nexus/loop-state/audit-log-impl/evidence/`:

| File | What it proves |
|------|---------------|
| `local-verify/replay-run-v2.log` | Single-request replay `captures=1 matched=1 mismatched=0` after the RequestId propagation fix |
| `matrix/replay-R2-window.json` | Time-window replay over 35 captures; schema complete, no crash |
| `matrix/replay-R1-per-case.md` | Per-case replay result table (13 rows) |
| `matrix/replay-R3-duplicate-handling.*` | Multiset duplicate handling (injected dup + differ verdict) |
| `matrix/replay-R5-nonexistent.log` | Graceful exit on unknown RequestId |

See also [`triple-test-aggregate.md`](./triple-test-aggregate.md) for the
schema-level triple-tests that back this tool's underlying tables.