# Production HTTP report

The daily HTTP report and the before/after comparison use
`daily_prod_backend_log_report.py --source prod-nginx`. The database access
report is a separate artifact; Flow execution counts and unavailable
`_System_RequestCapture` rows cannot establish external HTTP coverage.

## Source authority

The external-request counting boundary is Prod nginx, including requests
rejected before reaching a service. Collect both
`/var/log/nginx/nexuscrm-prod-d3-*.access.log*` and
`/var/log/nginx/nexuscrm-prod.access.log*`, including numbered and gzip
rotations. The Classic nginx stream is necessary for cutover windows; the old
Classic **backend** log is never an automatic fallback. The collector reads
`multi_prod/d3/current.json` as a collection-time authority snapshot. This
snapshot is not historical attribution for every request in the window.

The reviewed D3 Prod nginx publisher installs `nexuscrm.prod.http.v1` JSON
logging during its normal publication. Its format is enclosed in a dedicated
owned block; the Prod server-replacement adapter removes the previous owned
block before replacing the server. Retries and slot reuse therefore keep one
format definition. The D3 routing marker remains immediately before the server
so the existing live-route parser recognizes D3 ownership. Its fields are `msec`, `method`, `uri`,
`status`, `request_time`, `nginx_request_id`, `upstream_request_id`, and `release`.
`uri` uses nginx `$uri` and excludes query arguments. The collector also strips
queries from legacy input before generating samples or report artifacts. Search
diagnostics retain term-shape/fieldset aggregates but omit raw search samples.
`msec` is completion time in Unix seconds; the report uses the half-open UTC
completion window `[start, end)`. `request_time` is nginx end-to-end seconds,
converted to milliseconds, not Runtime-only duration. The release field is
the selected release-root basename. An absent upstream request ID remains
absent; a generated nginx ID is not substituted as a service correlation ID.

Old nginx combined records remain usable for volume and status, including
malformed client requests rejected with HTTP 400. They lack durations and
stable request IDs. Latency percentiles and slow rates for any partially
measured population are `null`/`n/a`, not zero or percentiles computed over an
undisclosed subset. Known measured slow samples remain available separately.
Historical latency cannot be recovered by deploying the new format later.

Retained D3 Runtime API, SFCRM AppService, and Integration containers supply
`DEBUG_SEARCH_SQL`, `Permission diagnostics`, and `Permission read phase`
events through `docker logs --since --until --timestamps`. Both running and
stopped Prod-labelled instances are included. Restarts within one container
use that same container's retained log. Internal HTTP calls are **not** added
to ingress volume. No raw container environment is written to the artifact.
An empty search diagnostic sample is inconclusive, not proof of zero searches.
Permission HTTP correlation is possible only where the actual upstream
response ID matches the diagnostic request ID; the existing permission gate
remains inconclusive on missing evidence.

## Coverage and retention

The collector opens the entire discovered rotation set before reading, filters
by event timestamps (not file mtime), resolves aliases by device/inode, and
deduplicates versioned events by nginx request ID and completion timestamp.
Distinct requests with identical routes/status/time remain distinct. Legacy
records have no stable event ID: overlapping legacy segment ranges are
reported as inconclusive, not silently deduplicated by text.

The artifact includes each source path, inode/device, size at open, mtime,
first/last event and in-window event, per-release ranges, parsed/malformed/duplicate counts,
missing duration count, and observed release identities. Docker evidence
includes immutable container/image IDs, role, run ID, creation/state times,
restart count, logging driver/path, and observed event range. The report also
records global first/last request times and observed retained-history bounds.

`OBSERVED` means usable samples were collected; it is not a completeness or
health verdict. `INCONCLUSIVE` and a prominent warning are emitted for no
interactive/integration samples (including health-only and background-only
windows), missing durations, malformed logs, Classic-only samples, missing
diagnostic roles, overlapping legacy ranges, or history that does not bracket
the requested window. Bracketing is checked independently for every observed
D3 slot/release across its rotated segments; Classic, another slot, or a
previous release cannot lend it coverage. The union of all sources is retained
only as a diagnostic, not as a completeness gate. Until a durable cutover
timeline establishes the exact routing intervals, every window with multiple
observed D3 slot/release streams remains inconclusive, even when each stream
independently brackets the entire window. Partial single-release windows are
also inconclusive. Unreadable/missing primary sources, unknown current
authority, and Docker read failures fail collection rather than falling back.
Warnings still produce artifacts and a red Feishu card. Comparisons refuse
incomplete windows and different counting boundaries.

At the September 7 investigation, Prod nginx used daily rotation with 14
rotations, compression and delaycompress; the service containers used Docker
`json-file`. This is an observed host configuration, not a retention guarantee
enforced by this PR. Rotation/removal beyond that retention, container removal,
copytruncate, or historical logging outages cannot be reconstructed from
surviving samples. Preserve nginx segments and container logs across the
report/recovery window before retention removes them. For stronger durability,
a separately operated archive or centralized ingestion must carry the same
source/event identities. Do not claim CloudWatch coverage without proving its
ingestion and time range.

## Operation and rollout

The existing `prod-ec2` runner needs read access to nginx logs and Docker.
The observed runner user `ubuntu` belongs to `adm` and `docker`. The report
does not reload nginx, restart processes, replay traffic, or mutate data.

```bash
python3 scripts/daily_prod_backend_log_report.py \
  --source prod-nginx --report-date 2026-09-07 \
  --output-json "$REPORT_DIR/report.json" \
  --output-markdown "$REPORT_DIR/report.md" \
  --output-feishu-text "$REPORT_DIR/report.txt"
```

For offline fixtures only, use `--prod-root`, repeated `--nginx-log-glob`, and
`--skip-container-diagnostics`; skipped diagnostics are visibly inconclusive.
Explicit `--source backend` remains available for historical backend-file
investigations. It cannot be combined with the production ingress source.

Merging enables D3 volume/status recovery for the next scheduled report. Full
latency becomes available only after the existing authorized Prod D3 publisher
installs the new nginx format and a whole report window uses it. No live
configuration was changed by this PR. Do not close #5877 solely on a merged
collector: after normal rollout, verify one known request's route/status/time
and latency in the report, source identities/ranges, search availability, and
permission coverage. The isolated nginx regression sends a synthetic 503
request and verifies the real emitted format; it is not live Prod acceptance.

Rollback uses the existing deployment rollback for nginx configuration. The
collector tolerates mixed combined/JSON history across that boundary and
marks unmeasured latency inconclusive. Reverting to the Classic-only collector
would reintroduce the original blind spot.