# Public HTTP retry semantics

Status: accepted design for incremental rollout.

## Goal

Define one small public HTTP contract that lets a client distinguish a request
that is safe to replay from one whose outcome is unknown.

This document governs NexusCRM-originated public HTTP responses. Internal RPC
scheduling hints may be richer, but an HTTP adapter must obey this contract
when it projects them to a public response.

## Protocol rules

### `503` with `Retry-After`

A NexusCRM public API response may contain both `503 Service Unavailable` and a
valid `Retry-After` header only when the responding boundary can prove that:

1. the request was rejected before business execution began; and
2. no durable command or business effect was accepted.

This combination means that replaying the same request cannot duplicate an
effect from the rejected attempt. Admission control, standby, draining, and
other pre-handler gates may emit it when they satisfy both conditions.

For this contract, `Retry-After` is a positive whole number of seconds. It tells
the client how long to wait before another attempt. It does not tell the client
how many attempts to make.

An error mapper that runs after the business handler, or cannot prove whether
the handler ran, must not add `Retry-After`. Proxies and gateways must preserve
this meaning and must not synthesize the header for an ambiguous upstream
failure.

### `503` without `Retry-After`

A plain `503` has no platform-wide replay meaning. The request may have failed
before execution, during execution, or after an effect was committed.

The owning API may define more specific behavior. Without such an API contract:

- a mutation client must not infer that direct replay is safe;
- a mutation that has a command identity or status API should reconcile the
  original command before deciding whether to submit more work; and
- a read client may apply its own bounded transient-error retry policy.

`UNAVAILABLE` and other outcome-unknown internal failures therefore project to
public HTTP without `Retry-After`, even if an internal transport carries a delay
hint for scheduling or reconciliation.

## Client policy

The client owns retry policy. The server supplies only the pre-execution fact
and the minimum delay when it emits `503 + Retry-After`.

Each client chooses limits appropriate for its request and latency budget. A
retry loop must have both:

- a maximum number of attempts; and
- a total deadline or cancellation signal.

Clients should add bounded jitter when many callers may retry together. They
must stop when the limit or deadline is reached.

### Mutations

A client may directly replay a mutation after the exact response
`503 + valid Retry-After`, subject to its limits. If the API uses a stable
business command ID or idempotency key, the replay keeps that identity.

Other failures do not become replay-safe merely because they are transient.
Timeouts, connection loss, `503` without `Retry-After`, and an ambiguous
`UNAVAILABLE` outcome require the API's reconciliation or idempotency contract.

### Reads

A client may retry an operation that its API contract identifies as read-only.
It chooses which transient statuses to handle, the backoff, maximum attempts,
and total deadline. This read policy is independent of the mutation replay
signal above.

## Retry boundary

Retry only the failed HTTP request or durable command boundary. Do not restart
an entire workflow, UI journey, test case, or create-then-read sequence when an
earlier mutation already succeeded.

Examples:

- A create returns `201`, then its readback GET returns `503`. Retry only the
  GET according to the read policy; never repeat the create.
- A create is rejected with `503 + Retry-After: 1`. The rejected create did not
  execute, so the client may wait and retry that create within its limits.
- A mutation times out or receives plain `503`. Do not directly replay it;
  reconcile its command identity when the API supports that operation.
- A read-only GET receives a transient `429` or plain `503`. The client may
  retry it if its bounded read policy includes that response.

## Ownership

- The pre-execution gate owns whether `503 + Retry-After` is emitted and must
  prove that the business handler was not invoked.
- The business API owns any endpoint-specific meaning of plain `503` and any
  mutation reconciliation contract.
- The client owns which eligible operations it retries and all retry limits.

There is no additional retry-disposition HTTP header. The public signal remains
the standard `503 + Retry-After` pair.

## Required verification

The rollout must prove these invariants:

1. A response with `503 + Retry-After` has a handler invocation count of zero.
2. Replaying an admission-rejected mutation produces exactly one business
   effect.
3. Outcome-unknown `UNAVAILABLE` failures never expose public `Retry-After`.
4. Plain `503` does not trigger generic mutation replay.
5. A successful create followed by a failed readback retries only the read.
6. Read and mutation retry loops stop at their configured attempt and deadline
   limits.

## Rollout order

1. Remove `Retry-After` from ambiguous post-handler and `UNAVAILABLE` HTTP
   projections.
2. Retain it only at gates that prove rejection happened before execution.
3. Add bounded client retry at the failed-request boundary.
4. Verify focused mutation/read cases, then run the same Test2 Full Inspection
   workload used for the baseline.

## Non-goals

This contract does not make every mutation idempotent, define one retry count
for all clients, or treat all `503` responses as equivalent. It also does not
replace command reconciliation for ambiguous mutation outcomes.