# Search Fabric Manifest Object Format

Date: 2026-04-17
Status: Frozen (Normative manifest format)
Authors: Search Platform Architecture

## 1. Scope

This document defines the **durable JSON payload** of the immutable manifest objects:

```text
s3://bucket/manifests/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}/manifest_v{version}.json
```

The manifest format is part of the canonical owner stack:
- storage topology and visibility semantics: `docs/specs/core/search-fabric-architecture.md`
- manifest publication owner: `docs/architecture/manifest-publisher.md`

This document does **not** redefine:
- public query or ingest semantics (owned by `docs/specs/core/search-fabric-api.md`)
- canonical error codes (owned by `docs/specs/core/error-codes.md`)
- committed-pointer payload shape (owned by `docs/architecture/catalog-control-service.md`)

## 2. Design Goals

The manifest object must be:
- **Immutable**: a given `manifest_v{version}.json` is never modified in place.
- **Pointer-driven**: readers discover the current committed snapshot by resolving
  `manifest_committed_head.json` (not by listing manifest prefixes).
- **Deterministic and portable**: field names and representations are explicit and stable across languages.
- **Sufficient for recovery**: manifests carry the minimal replay anchors and watermarks needed for restart without object-store prefix scans.

## 3. Versioning and Compatibility

Every manifest object includes:
- `manifest_format_version` (u32)
- `manifest_version` (u64)

Compatibility rules:
- Readers MUST reject a manifest whose `manifest_format_version` is greater than the highest supported version.
- Writers MUST only emit `manifest_format_version=1` for this repository revision.
- New optional fields MAY be added without changing `manifest_format_version` as long as:
  - old readers can safely ignore them, and
  - the meaning of existing fields does not change.

## 4. Manifest Identity and Checksums

The committed-pointer payload (`manifest_committed_head.json`) and query pinning use
`manifest_checksum_or_etag` as a stable identity token.

For `manifest_format_version=1`, the recommended identity is:
- **Algorithm:** BLAKE3
- **Input bytes:** the exact UTF-8 JSON bytes written to object storage
- **Encoding:** lowercase hex string (64 chars)

The system MAY instead use an object-store ETag when it is stable and appropriate for the backend,
but BLAKE3 is recommended for portability.

## 5. JSON Schema (Manifest Format v1)

Top-level manifest object (all fields are required unless marked optional):

```json
{
  "manifest_format_version": 1,
  "namespace_id": 7,
  "collection_id": 11,
  "manifest_version": 42,
  "previous_manifest_version": 41,
  "schema_generation_token": "token",
  "publish_unix_ms": 1713379200000,
  "watermarks": {
    "safe_ts": 123,
    "applied_ts_metadata_search": 123,
    "applied_ts_semantic_search": 123,
    "applied_ts_metadata_aux": 123
  },
  "cdc_resolved_ts": 123,
  "modalities": [
    {
      "modality": "metadata_search",
      "wal_replay_start_anchor": { "object_key": "...", "object_etag_or_checksum": "", "min_commit_ts": 1, "max_commit_ts": 123 },
      "wal_durable_anchor":      { "object_key": "...", "object_etag_or_checksum": "", "min_commit_ts": 1, "max_commit_ts": 123 },
      "produced_artifacts": [
        {
          "object_key": "...",
          "object_etag_or_checksum": "",
          "key_ranges": [
            { "first_doc_id_u64": 1, "last_doc_id_u64": 10, "max_commit_ts": 123 }
          ],
          "window_start_unix_ms": 1700000000000,
          "window_end_unix_ms_exclusive": 1700003600000,
          "data_tier": "warm"
        }
      ]
    }
  ],
  "commit_event": { "kind": "committed_write", "...": "..." }
}
```

### 5.1 Field Definitions

#### 5.1.1 Identity and lineage

- `manifest_format_version` (u32): current revision uses `1`.
- `namespace_id` (u64): stable namespace identifier.
- `collection_id` (u64): stable collection identifier.
- `manifest_version` (u64): monotonic manifest version.
- `previous_manifest_version` (u64, optional):
  - When present, MUST equal `manifest_version - 1`.
  - Included for debugging and recovery tooling only; it is not a second visibility boundary.
- `schema_generation_token` (string):
  - Fence token for the active schema generation at the time the visible data frontier was published.
  - For background rewrites, this SHOULD remain equal to the base manifest token unless the rewrite is itself schema-fenced by a higher-level owner.
- `publish_unix_ms` (i64): publish timestamp in Unix milliseconds.

#### 5.1.2 Watermarks

`watermarks` is a required object:
- `safe_ts` (u64): MUST equal `min(present enabled modality applied_ts_*)`.
- `applied_ts_metadata_search` (u64, optional): present only when the collection exposes metadata-search state.
- `applied_ts_semantic_search` (u64, optional): present only when the collection exposes semantic-search state.
- `applied_ts_metadata_aux` (u64, optional): present only when the collection exposes metadata-aux state.

Presence semantics:
- omission means the modality is structurally absent.
- a present `applied_ts_*` value MUST be non-zero.

#### 5.1.3 CDC progress and replay anchors

- `cdc_resolved_ts` (u64):
  - For collections whose truth model includes Flow-A CDC ingestion, this field MUST be non-zero and monotonic.
  - For collections that do not use CDC ingestion, it MAY be `0`.
  - This value is a recovery/GC coordination signal and does not redefine `safe_ts`.

Per modality, the manifest stores two replay anchors:
- `wal_replay_start_anchor` (required): earliest retained WAL anchor required to rebuild the unflushed window.
- `wal_durable_anchor` (required): latest durable WAL anchor confirmed for the modality.

Anchor object:
- `object_key` (string): immutable durable-anchor object key (bucket-relative); current revision uses WAL chunk keys for streaming lanes and MAY use immutable descriptor objects for batch-lane anchors.
- `object_etag_or_checksum` (string): optional integrity / identity token.
- `min_commit_ts` (u64): MUST be non-zero for row-bearing anchors.
- `max_commit_ts` (u64): MUST be >= `min_commit_ts`.

Phase note:
- In early revisions, producers MAY set `wal_replay_start_anchor == wal_durable_anchor` when no retained unflushed window is modeled yet; later revisions move replay start forward based on flush/retention policy.

#### 5.1.4 Produced artifacts (immutable objects)

`produced_artifacts[]` is a per-modality list of immutable committed objects that are part of the
**committed snapshot described by this manifest version**.

Assembly rules:
- For ordinary write-path publication, Manifest Publisher typically *extends* the prior committed
  artifact set by adding newly produced immutable artifacts. In that common case the artifact list
  is monotonic across manifest versions.
- For replacement-style publication (compaction / lifecycle rewrite / snapshot cutover), Manifest
  Publisher MAY replace or drop previously referenced artifacts as part of the new committed
  snapshot. Old objects become logically dereferenced only after the new manifest commits.

The list MAY be empty for modalities that have no immutable committed artifacts yet, or for
frontier-only publishes that advance watermarks/anchors without introducing new immutable coverage.

Each artifact contains:
- `object_key` (string): immutable query-serving object key or immutable descriptor-object key.
- `object_etag_or_checksum` (string): optional identity token.
- `key_ranges[]` (required, non-empty): ordered non-overlapping coverage spans.
  - `first_doc_id_u64` (u64): inclusive.
  - `last_doc_id_u64` (u64): inclusive; MUST be >= `first_doc_id_u64`.
  - `max_commit_ts` (u64): MUST be non-zero.
- `window_start_unix_ms` / `window_end_unix_ms_exclusive` (optional):
  - When either is present, both MUST be present and `window_end_unix_ms_exclusive > window_start_unix_ms`.
- `data_tier` (optional): effective tier annotation for lifecycle-managed windows.

#### 5.1.5 Commit event metadata (optional)

`commit_event` is an optional debug/idempotency carrier describing why this manifest version exists.
It is intended for:
- idempotent replay detection (duplicate publish requests),
- operational debugging.

Current recommended `kind` values:
- `committed_write` (Visibility Barrier -> Manifest Publisher)
- `background_rewrite` (Background Compute Coordinator -> Manifest Publisher)
- `snapshot_cutover` (`CommitSnapshotSync` owner -> Manifest Publisher)

The exact sub-fields are implementation-defined as long as:
- they do not create a second visibility boundary, and
- they do not require readers to interpret them for correctness.

## 6. Invariants

- A manifest object MUST be self-contained and parseable without reading CCS projection tables.
- A reader MUST never rely on object-store `LIST` for correctness.
- Manifest objects MUST remain readable as long as they may be pinned by admitted query token lifetimes.