# Manifest Publisher

Status: Frozen architecture note
Owner: Search Platform Architecture

## 1. Scope

This document defines the execution design of `Manifest Publisher`, the stateful high-availability component that owns committed-manifest publication for one collection at a time.

This document is subordinate to the canonical owner stack:
- machine-readable contract: `api/proto/searchfabric/v1/search_fabric.proto`
- public ingest/query semantics: `docs/specs/core/search-fabric-api.md`
- canonical error registry: `docs/specs/core/error-codes.md`
- system topology and state ownership: `docs/specs/core/search-fabric-architecture.md`
- manifest JSON schema: `docs/specs/core/manifest-format.md`
- internal owner files for the currently pinned caller boundaries: `api/proto/searchfabric/v1/manifest_publisher.proto` and `api/proto/searchfabric/v1/manifest_publisher_snapshot.proto`

It must not redefine:
- public `MutationBatch`, `MutationAck`, or snapshot-sync RPC semantics,
- public query consistency, `safe_ts`, `effective_read_ts`, or continuation-token rules,
- collection/catalog ownership already assigned to `Catalog Control Service`,
- compaction policy, lifecycle policy, or GC policy owned elsewhere,
- source-facing ingest protocols or signed public manifest intents.

## 2. Canonical Position in the System

`Manifest Publisher` sits after write durability aggregation and before derivative CCS head advancement:

```text
Write path
  Ingest Gateway
    -> Ingest Broker
    -> Write Planner
    -> Commit Coordinator
    -> Stream / Batch Writers
    -> Visibility Barrier
    -> Manifest Publisher
    -> Catalog Control Service

Background / cutover path
  Background Compute Coordinator ----------> Manifest Publisher -> Catalog Control Service
  CommitSnapshotSync owner ----------------/
```

Current checked-out source anchors fix the following placement:
- `Visibility Barrier` emits publish-ready intents and does not mutate manifests directly.
- `Manifest Publisher` accepts publish-ready work and serializes per-collection manifest version progression.
- committed-pointer CAS on `manifest_committed_head.json` is the atomic read-visibility boundary.
- `Catalog Control Service` advances `manifest_head_version` only after committed-pointer CAS succeeds.

Primary source anchors:
- `docs/specs/core/search-fabric-architecture.md` Section `3.2`, Section `6.4`, Section `6.9`, Section `10.2`
- `api/proto/searchfabric/v1/commit_coordinator.proto`
- `docs/architecture/commit-coordinator.md`
- `docs/architecture/catalog-control-service.md`
- `docs/architecture/query-broker.md`

`Manifest Publisher` is not:
- a public API surface,
- a second query snapshot owner,
- a collection-name resolver,
- a writer durability owner,
- a background scheduler,
- a GC authority.

## 3. Design Anchors

The design below is constrained by the following current-source facts:

1. `Manifest Publisher` owns manifest version progression and committed-pointer CAS; upstream components do not preassign manifest versions.
2. Query visibility is defined by `manifest_committed_head.json`, not by CCS `manifest_head_version`.
3. `Query Broker` pins `(committed_manifest_version, manifest_checksum_or_etag)` for the full request lifetime and any continuation/search-context chain, so head advancement must not invalidate already pinned manifests.
4. Published manifests are the durable place where object references and publish metadata become authoritative, including:
   - segment/object pointers,
   - schema metadata,
   - `applied_ts_*`,
   - `cdc_resolved_ts`,
   - `wal_replay_start_anchor`,
   - `wal_durable_anchor`,
   - lifecycle window bounds and `DataTier` annotations when lifecycle is enabled.
5. Streaming writes may become query-visible after committed-manifest publish without waiting for immutable micro-segment flush; therefore a manifest publish may advance watermarks and replay anchors even when the immutable object set changes little or not at all.
6. `INGEST_MODE_BATCH` is admitted only with `VISIBILITY_POLICY_WAIT_FOR_VISIBLE` plus `DURABILITY_POLICY_INGEST_ACK`, so batch publication must still preserve one truthful final ACK per original admitted batch scope.
7. Background lifecycle/compaction rewrites become query-visible only through the same manifest-CAS discipline; there is no second visibility surface for tier transitions or compaction cutover.
8. Data-plane recovery and query execution are pointer-driven. Steady-state correctness must not depend on object-store `LIST`.

## 4. Responsibilities and Non-Responsibilities

### 4.1 Responsibilities

`Manifest Publisher` is responsible for:

1. Accepting trusted manifest-publication candidates from the write path and other architecture-owned internal cutover paths.
2. Serializing manifest version progression per collection.
3. Reading the current committed-manifest head and constructing the next immutable manifest version from that state plus one or more compatible publish candidates.
4. Persisting canonical publish metadata inside the new manifest:
   - object references that become part of the committed snapshot,
   - `applied_ts_*` and other publish frontier metadata,
   - replay anchors,
   - lifecycle window/tier metadata when admitted by collection policy.
5. Writing immutable manifest version `N+1`.
6. CAS-advancing `manifest_committed_head.json` from `N` to `N+1`.
7. Returning the committed manifest identity (`version`, `etag/checksum`) to the trusted caller that waited for visibility.
8. Sending CCS one post-commit `head_advance_intent` so derivative head metadata can converge.
9. Micro-batching compatible updates for one collection when doing so does not weaken public ACK, atomicity, or visibility semantics.

### 4.2 Non-Responsibilities

`Manifest Publisher` is not responsible for:
- source-facing write admission,
- `commit_ts` assignment or write-order arbitration,
- generating durable modality anchors,
- deciding when a write becomes publish-ready across modalities,
- public `MutationAck` reconstruction,
- computing public `safe_ts` or query readiness outcomes,
- validating or minting continuation/search-context tokens,
- authoring collection lifecycle policy or schema policy,
- deciding compaction schedules,
- deleting old manifests or data objects,
- turning object-store caches into durable truth.

## 5. Input Boundaries and Publish Scope

### 5.1 Write-Path Internal Boundary

The concrete checked-out internal RPC boundary for ordinary write publication is:

```text
VisibilityBarrierInternalService
  -> ManifestPublisherInternalService.PublishCommittedWrite
```

`api/proto/searchfabric/v1/manifest_publisher.proto` defines the current trusted input:
- `PublishCommittedWriteIntent.slice`
  - canonical admitted write scope (`CommitBatchSlice`)
  - collection identity
  - source ordering span
  - authoritative `commit_ts`
  - schema-generation token
- `PublishCommittedWriteIntent.modalities[]`
  - one durable modality delta per touched modality
  - resulting `applied_ts`
  - durable anchor for that modality
  - any immutable committed artifacts and produced coverage ranges already needed for batch-lane publish/conflict handling
- `PublishCommittedWriteIntent.watermarks`
  - collection watermarks after the final touched modality completed
- `PublishCommittedWriteIntent.barrier_epoch`
  - barrier-side fenced epoch for diagnostics/fencing
- `PublishCommittedWriteIntent.atomicity` / `continue_on_error`
  - original admitted batch atomicity information needed to keep publisher-side micro-batching subordinate to canonical batch visibility rules

The current write-path response is intentionally small:
- `committed_manifest_version`
- `committed_manifest_etag_or_checksum`

The key ownership split is therefore explicit:
- `Visibility Barrier` decides when a write scope is publish-ready,
- `Manifest Publisher` decides which committed manifest version makes it visible.

### 5.2 Other Internal Publish Sources

The broader architecture also sends manifest-mutation work here from:
- compaction workers that rewrite immutable objects and need cutover,
- lifecycle transition workers that change effective `DataTier` placement through manifest advancement,
- snapshot-import cutover after `CommitSnapshotSync` validates the admitted session.

Those paths are still subordinate to the same rules:
- raw upload or object rewrite is not visibility by itself,
- no caller may bypass committed-pointer CAS,
- no public client submits a signed manifest intent directly.

### 5.3 Publish Scope and Micro-Batching Rules

The serialization key is collection identity. One committed manifest version may include more than one publish candidate only when all of the following remain true:
- every included candidate was already admitted and publish-ready upstream,
- the resulting manifest still preserves canonical winner order and batch atomicity,
- all included wait-for-visible callers may truthfully complete on the shared CAS boundary,
- no included candidate requires a different committed-head precondition than the one being advanced.

Micro-batching is therefore an implementation optimization, not a second commit model. The publisher must not:
- merge work across collections,
- invent writer-local visibility groups,
- publish a later batch chunk ahead of an unresolved earlier chunk under `ATOMICITY_POLICY_FAIL_FAST_PARTIAL`,
- partially expose an `ATOMICITY_POLICY_ALL_OR_NOTHING` batch,
- silently rebase stale background candidates onto a newer head when doing so would change the canonical result.

## 6. State Model

### 6.1 Durable Truth

The durable manifest-publish state for one collection is pointer-driven:

```text
s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/manifest_committed_head.json
s3://bucket/catalog/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/manifest_head.json
```

`manifest_committed_head.json` is the committed visibility pointer. Current checked-out CCS documentation gives it the following payload fields:
- `committed_manifest_version`
- `previous_committed_manifest_version`
- `manifest_checksum_or_etag`
- `publisher_epoch`
- `updated_at_unix_ms`

`manifest_head.json` is derivative CCS state:
- `manifest_head_version`
- `expected_previous_head`
- `last_update_epoch`
- `reconciler_status`

Local queues, caches, and inflight batching windows are disposable. Durable truth lives in:
- the current committed-head pointer,
- immutable manifest objects referenced by that pointer,
- the immutable data objects and replay anchors referenced by those manifests.

### 6.2 Manifest Delta Shapes

The publisher should be understood as handling three common manifest-delta shapes:

1. Streaming visibility publish:
   - advances `applied_ts_*`, `cdc_resolved_ts`, and WAL anchors,
   - may leave the immutable object set unchanged while making the admitted NRT window query-visible under the new committed frontier.
2. Immutable-object introduction:
   - batch writers or snapshot cutover add newly built immutable objects to the committed snapshot,
   - visibility still begins only at committed-pointer CAS.
3. Immutable-object replacement:
   - compaction or lifecycle transition rewrites the committed object set,
   - old objects become logically dereferenced only after the new manifest commits.

These shapes all share one publication rule: no state is query-visible until `manifest_committed_head.json` advances successfully.

## 7. Publish Pipeline

### 7.1 Happy Path

```text
publish-ready candidate
  -> load current committed head N
  -> build immutable manifest version N+1
  -> write manifest_vN+1 object
  -> CAS manifest_committed_head.json from N to N+1
  -> return committed_manifest_version / checksum
  -> emit CCS head_advance_intent(expected_head=N, new_head=N+1, checksum)
```

Visibility begins at the committed-pointer CAS step, not earlier and not later.

### 7.2 Manifest Assembly Rules

When constructing `manifest_vN+1`, the publisher must:
- start from the current committed manifest selected by `manifest_committed_head.json`,
- apply only trusted, already-admitted candidate deltas,
- include only object references that already exist durably,
- preserve canonical manifest metadata required by query execution and replay,
- publish lifecycle window bounds and `DataTier` annotations when those are part of the admitted collection state,
- avoid any dependence on prefix scans or ambient object discovery.

The publisher must not require every publish to rewrite the full immutable object set. For streaming visibility publication, the essential state change may be:
- advancing admitted frontiers,
- advancing replay anchors,
- preserving the previous immutable object references,
- making the corresponding NRT delta window legal for readers under the new committed frontier.

### 7.3 Conflict Handling

Conflict handling must stay subordinate to the canonical conflict model already defined elsewhere.

For ordinary write-path publishes:
- upstream components already fixed canonical write ordering and publish readiness,
- the publisher serializes those scopes and must not invent a second row-order winner rule.

For batch, compaction, and other replacement-style candidates:
- the candidate must remain compatible with the latest committed manifest state,
- stale expected-head or stale coverage assumptions must fail closed,
- retry/rebuild is preferred over silent rebasing that could change canonical visibility.

The intended external/public projection remains the existing canonical error families:
- transient publish inability or expired wait budget -> retryable backend-unavailable path,
- stale conditional advancement or incompatible replacement candidate -> canonical conflict path when surfaced outside the publisher's private caller set.

## 8. CCS Handoff and Derivative Metadata

After successful committed-pointer CAS, the publisher sends CCS one internal `head_advance_intent` containing:
- `namespace_id`
- `collection_id`
- `expected_head_version`
- `new_head_version`
- `manifest_checksum_or_etag`

CCS documentation fixes the replay identity for this intent as the same tuple above. This means:
- the handoff is replay-safe without any public `RequestMeta.idempotency_key`, and the machine-readable boundary for snapshot cutover is separately pinned in `api/proto/searchfabric/v1/manifest_publisher_snapshot.proto`,
- CCS can deduplicate or repair replays deterministically,
- `manifest_head_version` remains derivative metadata rather than a second commit boundary.

Correctness rule:
- readers resolve `manifest_committed_head.json`,
- CCS `manifest_head_version` remains private derivative governance metadata for repair/observability and is not required by the reader/query runtime contract,
- if CCS lags the committed pointer, query serving still executes against the committed manifest.

## 9. Query Pinning, Continuations, and GC Consequences

`Query Broker` pins `(committed_manifest_version, manifest_checksum_or_etag)` for first-page execution and reuses that exact pin for continuation/search-context execution.

That creates a hard contract for the publisher and the background plane:
- head advancement from `N` to `N+1` must not make manifest `N` unreadable immediately,
- delayed physical deletion must preserve pinned manifests and their referenced objects for at least the maximum admitted query token lifetime,
- lifecycle retention and generic GC horizons must both be satisfied before old objects become physically deletable,
- query-time tier pruning must use manifest-published `DataTier` metadata from the pinned committed snapshot rather than CCS hot projections.

The publisher therefore participates in logical dereference, not immediate physical deletion:
- new manifest commit can stop making an older manifest current,
- actual object reclamation remains delayed-GC work under the canonical GC rules.

## 10. Failure Handling and Recovery

### 10.1 Failure Cases

If immutable manifest write or committed-pointer CAS fails:
- visibility does not advance,
- the previous committed head remains authoritative,
- upstream wait-for-visible callers see a retryable publish failure path rather than a partial success,
- already written immutable artifacts may remain staged/orphaned until later retry or GC, but they are not committed truth.

If committed-pointer CAS succeeds but CCS head advancement fails:
- the write is already committed and query-visible,
- wait-for-visible ACK does not need to wait for CCS convergence,
- CCS reconciliation repairs `manifest_head_version` asynchronously.

If a write-side wait budget expires before publish completes:
- the canonical public error family is retryable `SF_BACKEND_UNAVAILABLE`,
- `SF_VISIBILITY_TIMEOUT` remains reserved for query/readiness waiting.

### 10.2 Recovery Rules

Publisher restart must be pointer-driven:
- read the current `manifest_committed_head.json`,
- reopen the referenced immutable manifest,
- resume per-collection serialization from committed state rather than from local cache,
- rely on caller retry/replay for uncommitted inflight work.

Recovery must not:
- scan S3 prefixes to rediscover live data,
- trust local batching state as durable truth,
- rebuild CCS derivative head metadata as if it were the commit source.

The broader system recovery loop remains aligned:
- manifests retain canonical replay anchors for writer/NRT recovery,
- CCS rebuilds committed-head state from checkpoints plus `manifest_committed_head.json`,
- CCS repairs `manifest_head` from committed-head truth rather than from orphan control objects.

## 11. Observability and Backpressure

Useful publisher-local signals include:
- per-collection publish-queue depth,
- per-collection batching delay and effective batch size,
- immutable manifest write latency,
- committed-pointer CAS success/conflict/retry rate,
- CCS head lag behind committed head,
- publish backlog age contributing to ingest wait-for-visible latency,
- object-store throttling or retry pressure on the publish path.

These signals remain internal observability. Public projection should continue to use canonical families owned elsewhere:
- sustained backlog may eventually feed upstream admission throttling (`SF_RATE_LIMITED`),
- a caller whose wait-for-visible budget expires before publication completes receives retryable `SF_BACKEND_UNAVAILABLE`.