# Elasticsearch Compatible API Profile

Date: 2026-02-27
Status: Draft (Normative)
Owner: Query Gateway / Metadata Search / Semantic Search

## 1. Scope

This document defines Elasticsearch-compatible API behavior for read/write operations exposed by Search Fabric in Unified Search Platform.

Compatibility commitment model:
- Layer 1 (`Wire Parity @ Gateway`): Elasticsearch HTTP compatibility is implemented and validated at gateway boundary.
- Layer 2 (`Canonical Contract @ Search Fabric`): internal gRPC contract after gateway normalization.
- “Compatible” in this document means deterministic lowering from Layer 1 to Layer 2 within declared profile scope, not direct wire passthrough inside canonical services.
- This document therefore has two equally normative jobs:
  - describe the Elasticsearch behavior that is intentionally supported at the gateway boundary,
  - define how that behavior lowers to canonical Search Fabric APIs.
- If an Elasticsearch behavior cannot be lowered deterministically to canonical APIs, it is out of scope in this profile and MUST be called out explicitly as a limitation or unsupported branch. Claiming support for such a behavior requires canonical API evolution first.

In scope:
- Compatibility profile definitions and roadmap phase binding
- Endpoint compatibility (`supported` / `subset` / `unsupported`)
- Option and payload compatibility
- Batch semantics and error behavior
- Query feature compatibility
- Profile-scenario-to-error-code mapping (canonical error definitions are in `docs/specs/core/error-codes.md`)
- Ownership applicability boundary (`collection_truth_mode`) for Elasticsearch-compatible write path
- Cross-profile restrictions and non-goals

Subset interpretation rule:
- `subset` means only explicitly listed included behavior is in scope.
- Any behavior not listed under `included` for that subset is out of scope and treated as unsupported.

## 2. Compatibility Profiles and Phase Binding

Terminology convention:
- `Profile` defines capability boundary and conformance scope.
- `Milestone` and schedule commitments are defined by `docs/governance/roadmap.md` phases, not by separate numbering in this document.

### 2.1 Profile 1: Core Read/Write Compatibility Profile

- Roadmap binding: `docs/governance/roadmap.md` `Phase 1` (`2026-04-01` to `2026-06-30`).
- Governance note: Phase 1 is the integrated architecture, TiDB federated-table, and pre-validation phase of the broader `10^8` logical-table program whose checkpoint closes on `2026-09-30`. This document owns only the Elasticsearch core read/write compatibility slice of that wider acceptance package.
- Scope: includes all currently documented Profile 1 capabilities in Sections `3` to `6`.
- Restriction model: inherits and enforces all compatibility boundaries in Section `9`.
- Primary objective: production-ready subset for core `_doc` read/write compatibility (`index/create/update/delete` plus read-by-id), `_bulk` subset, `_search` subset, and `/{index}/_mapping` subset.
- Delivery posture: integrated Phase-1 delivery slice; preview expansion and broader isolation hardening remain Phase-2 work.

### 2.2 Profile 2: Expanded Query Compatibility Profile

- Roadmap binding: `docs/governance/roadmap.md` `Phase 2` (`2026-07-01` to `2026-07-31`).
- Scope: strict superset of Profile 1.
- Adds expanded Elasticsearch query/API compatibility, including:
  - `MATCH ...` multi-field style compatibility via `query.multi_match`
  - `GET /{index}/_search`
  - `GET /_search` (subset alias)
  - `POST /{index}/_count`
  - `GET /{index}/_count` (method alias)
  - `POST /{index}/_update_by_query` (subset)
  - `POST /{index}/_delete_by_query` (subset)
  - `aggs` / `aggregations` subset
  - `query.nested` with `inner_hits` subset
  - nested-field `sort` subset
  - `nested` / `reverse_nested` aggregation subset
  - `highlight` subset
  - `search_after` (gateway token-continuation subset)
  - `if_seq_no` optimistic-write precondition subset (lowered to canonical `RowMutation.expected_commit_ts`)
- Keeps Profile 1 behavior and restrictions unless explicitly lifted in this profile.

### 2.2.1 Profile 3: Planned Parity Uplift Track

- Roadmap posture: planning track from Profile 2 toward higher-parity semantics; activation requires an explicit phase-scoped conformance addendum in `docs/governance/roadmap.md`.
- Scope objective: strict superset of Profile 2 with priority on:
  - deep paging uplift (`search_after` + PIT-equivalent lifecycle),
  - bulk semantic uplift (deterministic non-atomic per-item outcomes),
  - remaining high-frequency write-surface parity gaps.
- Delivery objective: keep deterministic fail-closed behavior while reducing source-parity gaps with measurable conformance gates.

### 2.3 Phase-to-Profile Binding and Auditable Acceptance Boundary

Phase/profile mapping:
- `Phase 1` (integrated Phase-1 delivery): Elasticsearch Compatibility `Profile 1`.
- `Phase 2` (public preview expansion): Elasticsearch Compatibility `Profile 2`.
- `Phase 3` (`2026-09-01` to `2026-10-31`, GA Readiness): hardening and scale validation of delivered Profile 1/2 scope; Profile 3 admission is optional and MUST be explicitly declared by addendum.
- `Phase 4A` (`2026-11-01` to `2027-01-31`, internal engineering qualification window): delivered Profile 1/2 hardening MAY continue in the same window, but it is same-window continuation by default rather than certification-feeding critical-path scope unless the accepted Phase-4A claim matrix explicitly promotes a regression-blocking subset; the `2026-12-15` Phase-4A foundation checkpoint freezes that claim-matrix baseline before the final `2027-01-31` qualification close; same-date Phase-4B post-GA expansion work still does not expand Elasticsearch profile scope, and Profile 3 rollout remains addendum-gated.

Auditability requirements:
- Each phase MUST declare profile-scoped conformance suite entry/exit criteria in `docs/governance/roadmap.md`.
- Acceptance MUST be evaluated against explicitly listed included/excluded subset boundaries in this document, not inferred behavior.
- Features outside mapped profile scope MUST fail closed and MUST NOT be counted as delivered scope.

### 2.4 Profile-Level Gate Criteria (Functional / Reliability / Economics)

All gates in this section are normative for the ownership-applicability scope declared in Section `9`.

#### 2.4.1 Phase-1 Gate (Profile 1, Integrated Phase-1 Delivery)

Functional gate:
- Profile 1 conformance suite passes for the declared core read/write subset inside the broader Phase-1 acceptance package.

Reliability gate:
- Snapshot consistency, replay/idempotency, and no-filter-leakage checks pass on the Phase-1 integrated validation path.

Economics gate:
- Phase-1 profile-scoped metrics stay within the milestone-closing Phase-1 workload bundle and the Phase-1 engineering-validation envelope declared by `docs/governance/roadmap.md` Sections `3.2` and `3.3`.

#### 2.4.2 Phase-2 Gate (Profile 2, Public Preview Expansion)

Functional gate:
- Profile 2 conformance suite passes for declared expanded-query subset.

Reliability gate:
- Replay/failover/chaos checks for expanded preview cohorts pass with no release-blocking defects.

Economics gate:
- Elasticsearch Profile-2 public-preview metrics stay within the profile-scoped subset of the milestone-closing Phase-2 public-preview workload bundle and the Phase-2 public-preview envelope declared by `docs/governance/roadmap.md` Sections `3.2` and `3.3`.

#### 2.4.3 Phase-3 Hardening Gate (Delivered Profile 1/2 Scope, GA Readiness)

Functional gate:
- Delivered Profile 1/2 scope remains conformant under phase-declared hardening and scale-validation campaigns.
- Retrieval-affecting routes delivered by the same milestone that can affect retrieval quality MUST be represented in the milestone-closing benchmark bundle, and those routes satisfy the roadmap Phase-3 search-quality gate defined by `docs/testing/search-quality-evaluation.md`.

Reliability gate:
- Production-readiness validation (SLO/error-budget/runbook/rollback readiness) passes for delivered Profile 1/2 scope.

Economics gate:
- Delivered Profile 1/2 scope stays within the profile-scoped subset of the milestone-closing Phase-3 multi-tenant workload bundle and the Phase-3 GA-Readiness envelope declared by `docs/governance/roadmap.md` Sections `3.2` and `3.3`.

#### 2.4.4 Phase-4A Same-Window Continuation Gate (Delivered Profile 1/2 Scope, No New Profile)

Functional gate:
- No regression is introduced in delivered Profile 1/2 endpoint/option/subset behavior during same-window hardening.

Reliability gate:
- Scale-validation and failure-path campaigns pass for any continued delivered-scope hardening work with no release-blocking defects.

Economics gate:
- Hardening/scale-validation overhead stays within the accepted same-window evidence package, the accepted `2026-12-15` Phase-4A foundation package when that package names an Elasticsearch regression-blocking subset, and the Phase-4A planning / scale-out envelope declared by `docs/governance/roadmap.md` Sections `3.2` and `3.3`; this work is non-blocking to the certification-feeding checkpoint unless the accepted Phase-4A claim matrix explicitly promotes a regression-blocking subset into the checkpoint-closing set.

## 3. Profile 1 Endpoint Compatibility

### 3.1 Write Endpoints (Supported / Subset)

- `POST /{index}/_doc` (supported; auto-generated id)
- `PUT /{index}/_doc/{id}` (supported)
- `POST /{index}/_doc/{id}` (supported)
- `POST /{index}/_create/{id}` (supported)
- `PUT /{index}/_create/{id}` (supported; same create-only contract as `POST /{index}/_create/{id}`)
- `DELETE /{index}/_doc/{id}` (supported)
- `POST /{index}/_update/{id}` (subset)
- `POST /{index}/_bulk` (subset)
- `PUT /{index}/_bulk` (subset; same request/response contract as `POST /{index}/_bulk`)
- `POST /_bulk` (subset; each item MUST include `_index`)

#### 3.1.1 Auto-ID Generation Contract (`POST /{index}/_doc`)

- For requests without caller-supplied `{id}`, gateway-generated `_id` MUST be a decimal string losslessly parseable to canonical `uint64`.
- Gateway MUST NOT generate Elasticsearch-style base64-like UUID `_id` values for this route in the active profile.
- Generated `_id` values MUST satisfy the same canonical ID parser contract used for explicit `{id}` tokens.

#### 3.1.2 Regular Single-Document Index (`PUT/POST /{index}/_doc/{id}`)

Included:
- full-document replace semantics for one explicit `{id}` target.
- optional Profile-2 OCC query params from Section `8.8`.

Deterministic canonical lowering:
- gateway MUST parse `{id}` with the active decimal-uint64 contract before canonical emission.
- gateway MUST canonicalize the source body into one full-replace `RowMutation.operation.upsert`.
- when `if_seq_no` is present, gateway MUST lower it to `RowMutation.expected_commit_ts`.
- when `if_seq_no` is absent, the route remains unconditional replace and `expected_commit_ts` MUST be omitted.

Response reconstruction:
- if the target document is absent at the gateway point-read snapshot, response MUST reconstruct HTTP `201` with `result=created`.
- if the target document already exists, response MUST reconstruct HTTP `200` with `result=updated`.
- success payload MUST project `_id`, `_index`, `_seq_no` from canonical `QueryHit.version_commit_ts`, and static `_primary_term=1`.

#### 3.1.3 Create-Only Single-Document Index (`POST/PUT /{index}/_create/{id}`)

Deterministic canonical lowering:
- gateway MUST lower create-only semantics to `RowMutation.operation.upsert` plus `RowMutation.expected_commit_ts=0`.
- canonical absent-document version is `latest_commit_ts=0`; any existing target therefore fails deterministically as `SF_VERSION_CONFLICT`.

Response reconstruction:
- admitted success reconstructs HTTP `201` with `result=created`.
- conflict rejects reconstruct HTTP `409` with Elasticsearch-compatible conflict body projection.

#### 3.1.4 Single-Document Delete (`DELETE /{index}/_doc/{id}`)

Deterministic canonical lowering:
- gateway MUST perform deterministic point-read before mutation emission.
- existing targets lower to exactly one `RowMutation.operation.delete`.
- absent targets emit no canonical mutation.
- Profile-2 `if_seq_no` lowers to `RowMutation.expected_commit_ts` when present.

Response reconstruction:
- existing target -> HTTP `200`, `result=deleted`.
- absent target -> HTTP `404`, `result=not_found`.
- success payload MUST project `_id`, `_index`, `_seq_no` when a canonical delete mutation was emitted, and static `_primary_term=1`.

### 3.2 Read, Query, and Mapping Endpoints (Supported / Subset)

- `GET /{index}/_doc/{id}` (supported)
- `POST /{index}/_search` (subset)
- `POST /_search` (subset; target index MUST be specified)
- `POST /{index}/_msearch` (subset)
- `POST /_msearch` (subset; each header item MUST include explicit `index`)
- `GET /{index}/_mapping` (subset; concrete single-index target only)
- `PUT /{index}/_mapping` (subset; canonicalized to control-plane schema apply path)
- Every Elasticsearch-profile read-only unary route that lowers to canonical query/control reads (`GET /{index}/_doc/{id}`, `/_search`, `/_msearch`, `/_count`, `GET /{index}/_mapping`) MUST carry canonical `RequestMeta`; for these read-only unary RPCs, `RequestMeta.idempotency_key` MAY be omitted, but gateway `RequestMeta.client_id` and `RequestMeta.client_kind=CLIENT_KIND_ES_QUERY_GATEWAY` remain mandatory.
- `PUT /{index}/_mapping` is a mutating control-plane route and MUST use explicit gateway write identity: replay-stable non-empty `RequestMeta.idempotency_key`, explicit `RequestMeta.client_id`, and `RequestMeta.client_kind=CLIENT_KIND_ES_WRITE_GATEWAY`.
- Every Elasticsearch-profile route that lowers to canonical query execution MUST emit the full canonical `QueryRequest` envelope, not only query-tree payloads:
  - `collection=<resolved CollectionRef>`
  - `consistency`
  - `search_payload`
  - `fetch_spec`
  - `meta=<RequestMeta>`
- Route legality is validated against the resolved descriptor `collection_truth_mode`; the canonical `QueryRequest` does not carry a public truth-mode field.

#### 3.2.1 Single-Document Read by ID (`GET /{index}/_doc/{id}`)

Deterministic canonical lowering:
- gateway MUST lower the route to one canonical point lookup:
  - exact `doc_id` filter,
  - `fetch_spec.limit=1`,
  - deterministic `_source` projection under the active schema generation.
- this route MUST NOT be implemented by widening to a normal search response and heuristically picking the first hit after response assembly.

Response reconstruction:
- hit present -> `found=true`, `_source` from the canonical projected row, `_seq_no` from canonical `version_commit_ts`, `_primary_term=1`.
- hit absent -> `found=false` and no synthetic placeholder `_source`.

### 3.3 Profile 1 Unsupported Endpoints

- `PUT /{index}` (index lifecycle via control plane, not ES API)
- `DELETE /{index}` (index lifecycle via control plane, not ES API)
- `POST /{index}/_update_by_query`
- `POST /{index}/_delete_by_query`
- `POST /_reindex`
- `POST /_update_by_query/{task_id}/_rethrottle`
- `POST /_reindex/{task_id}/_rethrottle`
- `GET /{index}/_search`
- `GET /_search`
- `POST /{index}/_count`
- `GET /{index}/_count`
- `GET /_count`
- `POST /_count`
- `POST /{index}/_pit`
- `DELETE /_pit`
- `POST /_search/scroll`
- `GET /_search/scroll`
- `DELETE /_search/scroll` (including `/_all` and `/{scroll_id[,scroll_id...]}` clear forms)
- `POST /{index}/_knn_search` (use `/_search` + supported `query.knn` subset)

### 3.4 Representative (Non-Exhaustive) Out-of-Scope Elasticsearch API Families (Current Docs Corpus)

For clarity and auditability, the following Elasticsearch REST API families appear in `docs/elasticsearch` but are outside Profile 1/2 compatibility scope unless explicitly added by a future profile/phase update:

- Security/auth management APIs (`/_security/*`)
- Cluster/node/ops diagnostics APIs (`/_cluster/*`, `/_nodes/*`, `/_cat/*`, `/_stats/*`, `/_tasks/*`)
- Data-stream/template/migration lifecycle APIs (`/_data_stream/*`, `/_index_template/*`, `/_migration/*`, `/_create_from/*`)
- Ingest and inference model-management APIs (`/_ingest/*`, `/_inference/*`)
- Query-rules management APIs (`/_query_rules/*`)
- Relevance-evaluation and analysis APIs (`/{index}/_rank_eval`, `/{index}/_termvectors`)
- Vector tile search API (`/{index}/_mvt/...`)
- Explicit refresh endpoints (`/_refresh`, `/{index}/_refresh`)
- Index maintenance/diagnostics APIs outside profile scope (for example `/{index}/_disk_usage`, `/{index}/_cache/clear`, `/{index}/_settings`)

Behavior rule:
- Requests to these out-of-scope families MUST fail closed with deterministic mapping in Section `10.1` (`400 SF_UNSUPPORTED_ENDPOINT`) unless a profile/phase section explicitly lifts them into scope.

## 4. Profile 1 Option and Payload Compatibility

### 4.1 Write Options (Supported)

- `refresh=false`
- `refresh=true`
- `refresh=wait_for`
- `timeout=<duration>`
- `op_type=index`
- `op_type=create`

### 4.2 Write Options (Unsupported)

- `if_seq_no` (Profile 1 only; lifted in Profile 2 Section `8.8`)
- `if_primary_term`
- `version`
- `version_type`
- `routing`
- `pipeline`
- `wait_for_active_shards`
- `require_alias`

### 4.3 `/_update` Body (Supported Subset)

Included:
- required `doc` object
- optional exactly one admitted upsert branch:
  - `upsert` object, or
  - `doc_as_upsert=true`
- optional `detect_noop` boolean
- Profile-2 OCC query params (`if_seq_no`, `if_primary_term=1`) on the route defined in Section `8.8`

Not included:
- `script`
- `scripted_upsert`
- `_source`
- `retry_on_conflict`
- request bodies without `doc`
- bodies that combine explicit `upsert` and `doc_as_upsert=true`

Deterministic canonical lowering:
- gateway MUST perform deterministic point-read/evaluate/canonicalize under the active schema-generation fence before emitting a canonical mutation.
- if the target document is absent and there is no admitted upsert branch, gateway MUST fail deterministically as not found and emit no canonical mutation.
- if the target document exists, gateway MUST treat `doc` as patch intent and compute merged post-update row state before mutation emission.
- if `detect_noop=true` and the merged row is unchanged after canonical field normalization, gateway MUST emit no canonical mutation and MUST reconstruct Elasticsearch `result=noop`.
- if the target document exists and the update is not a noop, gateway MUST emit exactly one `ROW_MUTATION` with `RowMutation.operation.patch`.
- if the target document is absent and an admitted upsert branch is present, gateway MUST emit exactly one `ROW_MUTATION` with `RowMutation.operation.upsert`; `doc_as_upsert=true` uses `doc` as the upsert payload, while explicit `upsert` uses the `upsert` object.
- for emitted patch mutations, gateway MUST bind the gateway-local read-modify-write decision with `RowMutation.expected_commit_ts`:
  - when `if_seq_no` is supplied, gateway MUST validate the point-read version and lower the same value.
  - when `if_seq_no` is omitted, gateway MUST lower the point-read latest version so the single-document update remains compare-and-set stable.
- for absent-document upsert/create branches, gateway MUST lower `RowMutation.expected_commit_ts=0`.
- response reconstruction MUST be deterministic:
  - emitted patch -> `result=updated`
  - emitted upsert on absent document -> `result=created`
  - noop branch -> `result=noop`

### 4.4 Query Features (Supported / Subset)

- `query.bool` (`must`, `filter`, `must_not`, `should`) subset
- `query.term`
- `query.terms`
- `query.range`
- `query.match` subset
- `query.knn` subset
- `size` (`0..1000`; `size=0` is allowed only when request has canonical lowering intent for zero-hit fetch (`aggs` non-empty, Profile 2, and/or `track_total_hits` is explicitly configured via `enabled` or numeric `threshold`); otherwise gateway MUST fail closed with `400 SF_BAD_REQUEST`)
- `from` (`>=0`, with `from + size` constrained by Elasticsearch `index.max_result_window`, canonically enforced via the resolved `effective_max_result_window` returned by the authoritative `GetCollectionSchema` control-plane descriptor)
- `_source` include/exclude subset
- `sort` subset (field sort + `_score`)
- `track_total_hits` (`true`/`false`/numeric threshold; when omitted, gateway MUST apply canonical default `TrackTotalHitsSpec{threshold: 10000}`)

### 4.5 Query Features (Unsupported in Profile 1)

- `query.multi_match`
- `aggs` / `aggregations`
- `highlight`
- `rescore`
- `pit`
- `search_after`

### 4.6 Profile 1 Subset Definitions (Included vs Excluded)

#### 4.6.1 `POST /{index}/_update/{id}` Boundary

Included:
- route support is limited to the admitted body subset and deterministic lowering rules in Section `4.3`.
- each admitted request emits at most one canonical row mutation, or no mutation for deterministic noop.

Not included:
- full Elasticsearch scripting/update-engine parity
- source behaviors that require gateway-managed retry loops or hidden `retry_on_conflict`
- combinations outside Section `4.3`

#### 4.6.2 `POST/PUT /{index}/_bulk` and `POST /_bulk` Subset

Included:
- NDJSON bulk format with action + source line pairs.
- Action types: `index`, `create`, `update`, `delete`.
- Per-item status/error response with non-atomic commit semantics.
- `update` action uses the same admitted body subset and point-read/evaluate/canonicalize rules as Section `4.3`.
- Gateway canonical lowering MUST use deterministic non-atomic mode:
  - `MutationBatch.policy.continue_on_error=true`
  - per-item reconstruction from canonical `MutationAck.success.item_outcomes[]`
  - wire response MUST follow Elasticsearch bulk shape (`errors` + `items[]` + per-item `status`/`error`)
  - admitted `update` items emit at most one canonical `ROW_MUTATION` or one gateway-local noop outcome
- Dynamic-field evolution consistency subset:
  - when undeclared fields trigger `INFER_AND_ADD` async infer path, gateway MAY enable bounded blocking wait (`IngestPolicy.schema_evolution_wait_ms`) before returning bulk response.
  - for one admitted bulk request that introduces multiple undeclared fields, gateway MUST aggregate those new fields and apply them in one fenced schema update before emitting item writes, so client retry/idempotency assumptions are preserved.
  - that aggregated schema apply MUST use the same outer replay-stable write identity as the surrounding bulk request and MUST emit canonical `PutCollectionSchema` metadata (`meta.idempotency_key`, `meta.client_id`, `meta.client_kind=CLIENT_KIND_ES_WRITE_GATEWAY`) plus canonical gateway-managed control scope (`scope.ddl_commit_ts=0`) before any item mutations are admitted.
  - this mode is optional and policy-driven; `0` keeps default non-blocking DLQ behavior.
- `PUT /{index}/_bulk` is accepted as method alias with the same subset contract.
- `/_bulk` requires `_index` per item.

Not included:
- `PUT /_bulk` pathless alias.
- Unsupported action metadata options such as `routing` and `pipeline`.
- Scripted `update` forms.
- multiple `update` actions targeting the same concrete `(_index, _id)` within one admitted bulk request; later duplicates MUST be reconstructed as deterministic item-level rejects.
- Bulk request shapes that are not valid NDJSON.

#### 4.6.3 `POST /{index}/_search` and `POST /_search` Subset

Included:
- Request body query DSL using Section `4.4` supported query features.
- Pagination controls: `size`, `from`.
- Pagination window MUST satisfy `from + size <= index.max_result_window`, canonically enforced against the collection-level resolved `effective_max_result_window` returned by `GetCollectionSchema.success.schema` rather than raw optional `CollectionSettings.max_result_window`.
- Source filtering via `_source` include/exclude subset.
- Sort via Section `4.6.9` sort subset.
- Concrete single-index target via request path (or equivalent explicit single-index routing for `POST /_search`).

Not included:
- URI `q`-string query mode.
- PIT/scroll related modes.
- Aggregation/highlight/rescore/search_after features in Profile 1.
- Multi-target index expressions (`_all`, wildcard `*`, comma-separated index lists, remote-cluster selectors, failure-store selectors using `::`).

#### 4.6.4 `POST /{index}/_msearch` and `POST /_msearch` Subset

Included:
- NDJSON multi-search format with header/body pairs.
- Each body uses the same query subset as `POST /{index}/_search`.
- Explicit index targeting via request path or per-item header.
- For `POST /_msearch`, each header item MUST include explicit `index`.
- For active profile subset, each resolved target MUST be a concrete single index.

Not included:
- Per-item features outside Profile 1 query subset.
- Scroll/PIT-specific options.
- Multi-target index expressions in path/header (`_all`, wildcard `*`, comma-separated index lists, remote-cluster selectors, failure-store selectors using `::`).

#### 4.6.5 `query.bool` Subset

Included:
- Core clauses: `must`, `filter`, `must_not`, `should`.
- Clause values using supported leaf queries (`term`, `terms`, `range`, `match`, `knn`) plus recursive admitted `bool` / `nested` branches.
- Scalar `should`/`constant_score` boost branches are lowered as retrieval-tree leaves and retain scoring intent through node-local `RetrievalNode.boost`; their boolean role is determined by placement inside `RetrievalGroup.must_clauses[]` or `ShouldMatchSpec.clauses[]`.
- Elasticsearch defaulting rule is preserved: when `bool` has `should` clauses but no `must`/`filter`, and source payload omits `minimum_should_match`, gateway MUST normalize the owning `ShouldMatchSpec.threshold` to `absolute_count=1` and enforce that as hard-filter eligibility.
- For bool lowering, gateway MUST emit one canonical `RetrievalGroup`:
  - source `must` -> `RetrievalGroup.must_clauses[]`
  - source `filter` -> predicate-tree branches under canonical hard filters (`search_payload.global_filter` at top level, `NestedQueryPayload.child_filter` inside nested contexts)
  - source `must_not` -> predicate-tree negation (`LogicalNot`) under canonical hard filters
  - source `should` -> `ShouldMatchSpec.clauses[]`
- source `minimum_should_match` MUST lower to `ShouldMatchSpec.threshold`:
  - integer form -> `MatchThreshold.absolute_count`
  - percentage form -> `MatchThreshold.percentage_ratio`
  - omitted -> canonical fallback (`absolute_count=1` when `must_clauses[]` is empty, otherwise `absolute_count=0`)

Not included:
- Advanced bool tuning fields not explicitly listed here.

#### 4.6.6 `query.match` Subset

Included:
- Single-field analyzed text matching.
- Basic analyzer-driven lexical retrieval.
- `operator` and `minimum_should_match` lowering to canonical plain-match options.

Deterministic canonical lowering:
- gateway MUST lower source `match` to one `TextClause` leaf with `fields=[\"<field>\"]`.

Not included:
- Multi-field match expansion (`query.multi_match`).
- Advanced per-query analysis tuning options not explicitly listed.

#### 4.6.7 `query.knn` Subset

Included:
- Vector nearest-neighbor retrieval with explicit top-K (`size`).
- Combined use with supported metadata filters via `query.bool`.
- query-vector dimension contract is the canonical MRL prefix-compatible rule: if schema declares `Dschema`, decoded query dimensions `Dq` must satisfy `0 < Dq <= Dschema`.
- `Dq < Dschema` is admitted and executes against the stored-vector prefix of length `Dq`; `Dq = 0` or `Dq > Dschema` is malformed and MUST fail closed.

Not included:
- Dedicated `/{index}/_knn_search` endpoint in Profile 1.
- Advanced ANN tuning parameters not explicitly listed.

#### 4.6.8 `_source` Include/Exclude Subset

Included:
- Include list and exclude list for source field projection.

Not included:
- Dynamic/scripted source transforms.

#### 4.6.9 `sort` Subset

Included:
- Sort by regular fields.
- Sort by `_score`.

Not included:
- Script-based sort.
- Geo-distance sort and advanced sort modes not explicitly listed.

### 4.7 `GET/PUT /{index}/_mapping` Subset (Profile 1)

Included:
- `GET /{index}/_mapping` returns mapping projection derived from active canonical control-plane schema snapshot read through `CatalogControlService.GetCollectionSchema`.
- `PUT /{index}/_mapping` accepts mapping updates that can be deterministically normalized into canonical schema payload:
  - `DocIDMapping`
  - `FieldSchema[]` (type/nullability/semantic capabilities/text-analysis/vector contracts)
  - optional `CollectionSettings` (collection-level index-sort, `max_result_window`, and supplemental options)
- External `{index}` MUST first resolve to canonical `CollectionNameBinding{namespace_id=<resolved>, collection_name=<index>}` before the gateway reads or mutates the target collection.
- If that name binding does not yet have an authoritative `CollectionRef`, the gateway MUST first ensure collection identity through `CatalogControlService.PutCollectionMetadata.name_binding=CollectionNameBinding{namespace_id=<resolved>, collection_name=<index>}` and use the returned `CollectionDescriptor.collection` as the concrete `CollectionRef` before any schema apply. Because this ensure leg is state-mutating, it MUST carry the same replay-stable non-empty outer `RequestMeta.idempotency_key`, explicit gateway `RequestMeta.client_id`, and `RequestMeta.client_kind=CLIENT_KIND_ES_WRITE_GATEWAY` that the gateway uses for the subsequent schema-apply leg.
- Mapping updates are then applied via `CatalogControlService.PutCollectionSchema` on that resolved `CollectionRef`, with canonical fencing/idempotency semantics, that same replay-stable non-empty outer `RequestMeta.idempotency_key`, the same explicit gateway `RequestMeta.client_id`, `RequestMeta.client_kind=CLIENT_KIND_ES_WRITE_GATEWAY`, gateway-managed control scope `scope.ddl_commit_ts=0`, authoritative monotonic `schema_version` derived from the current schema lineage of the resolved collection, and explicit non-`UNSPECIFIED` `update_policy`.
- Brand-new collection schema creation MUST use `SCHEMA_UPDATE_POLICY_INITIAL_CREATION`; after the gateway has ensured collection identity, that initial schema apply MUST use the full canonical zero-state tuple (`schema_version=1`, `fence.expected_prev_schema_version=0`, `fence.schema_generation_token="__initial__"`, `update_policy=SCHEMA_UPDATE_POLICY_INITIAL_CREATION`, `fence.enforce_monotonic_apply=true`). Later mapping updates MUST classify into additive vs rebuild policy according to canonical schema-diff rules before dispatch.
- `index.max_result_window` is part of the same control-plane schema contract as `CollectionSettings.max_result_window`; when omitted on apply, Search Fabric control plane resolves and persists a deterministic effective value, and `_mapping` readback MUST use `GetCollectionSchema.success.schema.effective_max_result_window`. `GetSchemaSyncStatus` may be used only for convergence/status polling, never as a schema-descriptor substitute.
- Concrete single-index target is required.

Not included:
- Multi-target mapping updates (`_all`, wildcard, comma-separated index list, remote selectors, failure-store selectors).
- Mapping features that cannot be represented in canonical schema model.
- Dynamic runtime schema mutation branches that violate rebuild-class evolution constraints.

### 4.8 Index Sorting Settings Equivalence (Control-Plane Path)

Because `PUT /{index}` is outside Elasticsearch endpoint scope in this profile, index-sort settings are accepted via control-plane schema APIs with canonical normalization.

Included equivalence:
- Elasticsearch `index.sort.field` / `index.sort.order` / `index.sort.mode` / `index.sort.missing`.
- Elasticsearch `index.max_result_window`.
- Adapter/Gateway normalization to canonical `IndexSortSpec` in Integration API control-plane payload.
- Adapter/Gateway normalization of `index.max_result_window` to canonical `CollectionSettings.max_result_window`, with query admission later enforced against the resolved `effective_max_result_window` returned by Search Fabric control-plane snapshots.
- Deterministic validation of field/type/mode compatibility at schema-apply time.

Not included:
- Dynamic runtime mutation of index-sort settings without schema rebuild.
- Scripted/computed index-sort expressions.
- Applying index-sort settings to pure vector index kind.

## 5. Profile 1 Batch and Error Semantics

1. `/_bulk` behavior:
- Batch is non-atomic.
- Valid request returns per-item results.
- Partial failure does not rollback successful items.
- For syntactically valid bulk requests that pass top-level admission/capacity limits, HTTP status MUST be `200`; partial failure is encoded by `errors=true` and per-item `items[].<action>.status`.
- Global admission rejects (for example auth/ownership/rate-limit failures, payload too large, max batch operation overflow) MUST abort the request as top-level `4xx/5xx` and MUST NOT be reconstructed as per-item bulk outcomes.
- Per-item status reconstruction MUST map canonical outcomes to Elasticsearch-compatible status classes (`400`/`404`/`409`/`429`/`503`/`504` as applicable).

2. Unsupported feature behavior:
- Unsupported endpoint/option/query feature MUST fail closed with deterministic errors.
- Canonical `SF_*` definitions are sourced from `docs/specs/core/error-codes.md`; this section only applies those canonical codes to Profile-1 scenarios.
- Deterministic mapping baseline:
  - unsupported endpoint -> `400 SF_UNSUPPORTED_ENDPOINT`
  - unsupported option -> `400 SF_UNSUPPORTED_OPTION`
  - unsupported query feature -> `400 SF_UNSUPPORTED_QUERY_FEATURE`

3. Timeout behavior:
- Query timeout returns `504 SF_QUERY_TIMEOUT`.
- Wait-for-visible timeout returns `504 SF_VISIBILITY_TIMEOUT`.

## 6. Profile 1 Visibility and Consistency Semantics

- Accepted writes are assigned globally ordered `commit_ts`.
- `refresh` maps to deterministic durability/visibility policy pair:
  - `false`: `visibility_policy=async` with `durability_policy=wal_chunk_durable`.
  - `true` / `wait_for`: `visibility_policy=wait_for_visible` with `durability_policy=wal_chunk_durable` (bounded by timeout).
- Lower durability tier (`durability_policy=ingest_ack`) is outside this profile's default conformance scope unless a phase-scoped addendum explicitly allows it.
- Query path executes under snapshot semantics.
- Standard Elasticsearch query paths without explicit point-in-time context MUST be normalized by gateway to canonical `CONSISTENCY_MODE_BEST_EFFORT` (`read_ts=0` on input; first-page execution uses engine-chosen `effective_read_ts=safe_ts`) for low-latency default behavior.
- Writes with `refresh=true` or `refresh=wait_for` wait for safe-ts advancement; subsequent BEST_EFFORT reads observe those writes through normal safe-ts progression without requiring explicit PIT contexts.

### 6.1 Hybrid Fusion and Dedup Contract (Commitment-B Acceptance)

For requests that include both metadata-search predicates and semantic vector retrieval in active profile subset:
- Query execution MUST use one aligned admitted snapshot boundary across metadata-search and semantic paths (`effective_read_ts` for default BEST_EFFORT first pages, or the exact admitted strict/PIT snapshot when that path is active).
- Metadata predicates are hard constraints for final results; candidates violating active metadata filters MUST NOT appear in final hits.
- Final cross-modality broker fusion for this admitted Elasticsearch compatibility path MUST use deterministic Reciprocal Rank Fusion (RRF) over participating modality candidate lists.
- This compatibility rule governs the broker step that merges metadata-search and semantic candidate lists; it does not override canonical request-level `RetrievalGroup.fusion_strategy` semantics inside the admitted retrieval tree.
- Gateway MUST explicitly set canonical `search_payload.dedup_by_doc_id=true` for this admitted hybrid Elasticsearch lowering path.
- Final hit set MUST therefore be deduplicated by canonical `doc_id` before response encoding.
- Phase/Profile conformance for Commitment B MUST include hybrid acceptance tests that verify deterministic fusion and no post-filter leakage.

## 7. Profile 2 Additional Endpoint Compatibility Scope

Profile 2 extends Section `3` endpoint support with:
- `GET /{index}/_search`
- `GET /_search` (subset alias; same constraints as `POST /_search`)
- `POST /{index}/_count`
- `GET /{index}/_count` (method alias)
- `POST /{index}/_update_by_query` (subset)
- `POST /{index}/_delete_by_query` (subset)
- `POST /{index}/_knn_search` (compatibility alias to supported KNN search path)

Profile 2 still keeps this endpoint out of scope:
- `GET /_count` (pathless all-target count); callers MUST use explicit `/{index}/_count`.
- `POST /_count` (pathless all-target count); callers MUST use explicit `/{index}/_count`.

Rationale:
- Pathless all-target count is excluded to keep target resolution, ownership validation, and admission behavior deterministic at profile scope.
- Callers MUST provide explicit index targets; missing explicit targets map to `400 SF_INDEX_TARGET_REQUIRED` per Section `10.2`.

### 7.1 `POST/GET /{index}/_count` Subset

Included:
- Query selection using the admitted Profile-2 query subset.
- Concrete single-index target only.
- Deterministic success-payload reconstruction from canonical count aggregation response with empty hits.

Not included:
- Pathless `GET /_count` and `POST /_count`.
- Multi-target index expressions (`_all`, wildcard `*`, comma-separated index lists, remote-cluster selectors, failure-store selectors using `::`).

Success-payload reconstruction:

| Elasticsearch field | Reconstruction rule |
|---|---|
| `count` | MUST be reconstructed from the canonical count aggregation result (`AGGREGATE_FUNCTION_COUNT`) after gateway lowering. |
| `_shards.total` | gateway-local execution envelope field; deterministic profile value for the routed concrete target. |
| `_shards.successful` | gateway-local execution envelope field; MUST equal total successful routed shard groups for the admitted concrete target. |
| `_shards.skipped` | `0` in the active profile subset. |
| `_shards.failed` | `0` on success; non-zero belongs to error projection, not success payload reconstruction. |

Gateway lowering rule:
- gateway MUST lower `/{index}/_count` to canonical query execution with empty `hits` and one deterministic count aggregation; it MUST NOT infer `count` from `hits.total` or any non-canonical side channel.

### 7.2 `POST /{index}/_update_by_query` Subset

Included:
- Query selection using Profile 2 query subset.
- Restricted script update mode:
  - `script.lang = painless`
  - assignment-style updates to `ctx._source` fields via `params`.
- Synchronous execution mode.
- Concrete single-index target only.
- Materialized-row count is strictly bounded by system `max_entries_per_batch` policy.
- Gateway MUST execute a probe query with `limit=min(caller_requested_limit, max_entries_per_batch, effective_max_result_window)+1` and `track_total_hits=true` before canonical mutation emission.
- `caller_requested_limit` is the explicit source `size` when provided; when omitted, `caller_requested_limit` defaults to `+inf`.
- If the probe hits this clamped cap, the request MUST be rejected synchronously as `400 SF_TOO_MANY_OPERATIONS` with details bound to the effective limit scope.

Not included:
- Full script-engine capability parity.
- Asynchronous task orchestration and rethrottle workflows.
- Advanced slicing/throttling controls.
- Multi-target index expressions (`_all`, wildcard `*`, comma-separated index lists, remote-cluster selectors, failure-store selectors using `::`).

Success-payload reconstruction:

| Elasticsearch field | Reconstruction rule |
|---|---|
| `took` | gateway-local wall-clock execution time in milliseconds. |
| `timed_out` | `false` on success payload; timeout branches project as top-level error instead of success-with-timeout. |
| `total` | number of documents selected by the gateway probe and admitted into canonical mutation emission. |
| `updated` | count of canonical mutation outcomes reconstructed as successful updates. |
| `deleted` | `0` in this route. |
| `batches` | deterministic gateway batch count for the admitted request; `1` when execution stays within the single synchronous batch cap. |
| `version_conflicts` | count of rejected item outcomes with canonical code `SF_VERSION_CONFLICT`. |
| `noops` | count of gateway-local noop outcomes produced by admitted subset logic. |
| `retries.bulk` | `0` in the active synchronous subset. |
| `retries.search` | `0` in the active synchronous subset. |
| `throttled_millis` | `0` in the active subset (no async throttling path). |
| `requests_per_second` | `-1` in the active subset (rethrottle/throttle controls out of scope). |
| `throttled_until_millis` | `0` in the active subset. |
| `failures` | array reconstructed only from per-item canonical rejects that are not absorbed by top-level admission failure. Each element MUST carry deterministic source `doc_id`, canonical code, and projected ES error shape. |

### 7.3 `POST /{index}/_delete_by_query` Subset

Included:
- Query selection using Profile 2 query subset.
- Conflict behavior controls: `abort` / `proceed`.
- Synchronous execution mode.
- Concrete single-index target only.
- Materialized-row count is strictly bounded by system `max_entries_per_batch` policy.
- Gateway MUST execute a probe query with `limit=min(caller_requested_limit, max_entries_per_batch, effective_max_result_window)+1` and `track_total_hits=true` before canonical mutation emission.
- `caller_requested_limit` is the explicit source `size` when provided; when omitted, `caller_requested_limit` defaults to `+inf`.
- If the probe hits this clamped cap, the request MUST be rejected synchronously as `400 SF_TOO_MANY_OPERATIONS` with details bound to the effective limit scope.

Not included:
- Asynchronous task orchestration and rethrottle workflows.
- Advanced slicing/throttling controls.
- Multi-target index expressions (`_all`, wildcard `*`, comma-separated index lists, remote-cluster selectors, failure-store selectors using `::`).

Success-payload reconstruction:

| Elasticsearch field | Reconstruction rule |
|---|---|
| `took` | gateway-local wall-clock execution time in milliseconds. |
| `timed_out` | `false` on success payload; timeout branches project as top-level error instead of success-with-timeout. |
| `total` | number of documents selected by the gateway probe and admitted into canonical delete mutation emission. |
| `updated` | `0` in this route. |
| `deleted` | count of canonical mutation outcomes reconstructed as successful deletes. |
| `batches` | deterministic gateway batch count for the admitted request; `1` when execution stays within the single synchronous batch cap. |
| `version_conflicts` | count of rejected item outcomes with canonical code `SF_VERSION_CONFLICT`. |
| `noops` | `0` in the active subset. |
| `retries.bulk` | `0` in the active synchronous subset. |
| `retries.search` | `0` in the active synchronous subset. |
| `throttled_millis` | `0` in the active subset (no async throttling path). |
| `requests_per_second` | `-1` in the active subset (rethrottle/throttle controls out of scope). |
| `throttled_until_millis` | `0` in the active subset. |
| `failures` | array reconstructed only from per-item canonical rejects that are not absorbed by top-level admission failure. Each element MUST carry deterministic source `doc_id`, canonical code, and projected ES error shape. |

## 8. Profile 2 Additional Query/Option Compatibility Scope

Profile 2 extends Section `4` feature support with:
- `query.multi_match` (subset)
- `query.nested` with `inner_hits` (subset)
- `aggs` / `aggregations` (subset)
- `nested` / `reverse_nested` aggregation subset
- nested-field `sort` (subset)
- `highlight` (subset)
- `search_after`
- composite aggregation pagination (`aggs.*.composite` + `after`)
- `if_seq_no` optimistic-write precondition (subset; lowered to canonical `RowMutation.expected_commit_ts`)
- `if_primary_term=1` OCC-compatibility pass-through acceptance for write routes that admit `if_seq_no`
- `track_total_hits` numeric threshold mode remains part of the already-supported Profile-1 query subset and is not newly introduced here

### 8.1 `query.multi_match` Subset

Included:
- Multi-field lexical matching across an explicit field list.
- Common modes: `best_fields`, `most_fields`, `cross_fields`.
- `minimum_should_match` lowering to canonical `PlainMatchOptions.minimum_should_match`.

Deterministic canonical lowering:
- `best_fields` MUST lower to one `RetrievalGroup{fusion_strategy=GROUP_FUSION_STRATEGY_DIS_MAX}` whose `should_match.clauses[]` are one-field `TextClause` leaves; source `tie_breaker` lowers to `RetrievalGroup.tie_breaker` (default `0.0` when omitted).
- `most_fields` MUST lower to one `RetrievalGroup{fusion_strategy=GROUP_FUSION_STRATEGY_WEIGHTED_SUM}` whose `should_match.clauses[]` are one-field `TextClause` leaves.
- `cross_fields` MUST lower to one `TextClause` leaf whose `fields[]` contains the admitted source field list in normalized order, so token statistics are computed over the joint field set instead of via gateway-side per-field fusion.

Not included:
- Modes and option combinations not explicitly listed here.

### 8.2 `query.nested` Subset

Included:
- `nested.path` targeting a schema-declared nested path.
- nested-local `query` using admitted bool/term/terms/range/match/knn/nested subset branches valid for that nested scope.
- `score_mode` lowering to canonical `NestedScoreMode`.
- optional `inner_hits` with deterministic lowering to canonical `InnerHitsSpec`:
  - `name`
  - `from` / `size`
  - `sort`
  - `_source`
  - `track_total_hits`
  - `highlight` under the same subset constraints as Section `8.4`
  - `size=0` only when `track_total_hits.enabled=true`; omitted / `false` / threshold-based nested hit counting with `size=0` MUST fail closed

Deterministic lowering rules:
- gateway MUST lower each source nested branch to one canonical `NestedClause` leaf inside the surrounding `RetrievalNode` tree, with `path`, `NestedQueryPayload.root_retrieval`, `score_mode`, and optional `inner_hits`.
- `inner_hits.sort` on Elasticsearch virtual `_id` MUST lower to canonical nested `doc_id` sort target, which is resolved as `parent_array_offset` in nested-hit order.
- nested-local payloads MUST stay nested-local; gateway MUST NOT smuggle top-level-only fusion/search-context controls into `NestedQueryPayload`.

Not included:
- parent/child join-family queries.
- unsupported nested query rewrites that require cross-path correlation outside canonical `NestedClause`.

### 8.3 `aggs` / `aggregations` Subset

Included:
- Bucket aggregations: `terms`, `range`, `date_histogram`, `composite`, `nested`, `reverse_nested`.
- Metric aggregations: `avg`, `min`, `max`, `sum`, `value_count`.
- Leaf collector aggregation: `top_hits` (including highlight controls under the same subset constraints as Section `8.4`).

Deterministic metric lowering:
- `avg` / `min` / `max` / `sum` map to canonical `MetricAggregation.metric_type.builtin_function` with required non-empty `MetricAggregation.field`.
- `value_count(field=X)` maps to canonical `MetricAggregation{builtin_function=AGGREGATE_FUNCTION_COUNT, field=\"X\"}`.
- `value_count` with missing/empty `field` is malformed and MUST fail closed (`400 SF_BAD_REQUEST`).
- canonical execution semantics for lowered `value_count`: count rows where target field `X` is present and non-null.

Deterministic `terms` missing-value lowering:
- source `terms.missing` MUST lower to canonical `TermsAggregation.missing_value_replacement`.
- when source omits `terms.missing`, missing/null docs are dropped from terms buckets.
- source `terms.missing` value incompatible with target field type MUST fail closed (`400 SF_BAD_REQUEST` or `422 SF_SCHEMA_VIOLATION` by precedence).

Deterministic nested-aggregation lowering:
- source `nested` MUST lower to canonical `AggregationRequest.aggregation_type.nested` with the same declared path.
- source `reverse_nested` MUST lower to canonical `AggregationRequest.aggregation_type.reverse_nested`; omitted source path returns to the root parent context.
- recursive `sub_aggregations` under `nested` / `reverse_nested` MUST preserve declaration order and canonical context legality.

Not included:
- Pipeline aggregations.
- Scripted metric aggregations.
- Geo/spatial aggregation families.

### 8.4 `highlight` Subset

Included:
- Field-level highlight with default highlighter mode.
- Basic pre/post tags.
- Per-aggregation `top_hits.highlight` lowering to canonical `TopHitsAggregation.text_options` (with fallback to top-level highlight options when omitted in `top_hits`).

Not included:
- Highlighter-specific advanced tuning modes not explicitly listed.

### 8.5 Nested-Field `sort` Subset

Included:
- nested-field sorts that provide explicit `sort[].<field>.nested.path`.
- optional `sort[].<field>.nested.filter` lowering to canonical `FieldSortTarget.nested_context.filter`.
- `_score` / scalar field ordering rules inherited from the admitted base sort subset.

Deterministic lowering rules:
- gateway MUST lower nested sort context to canonical `FieldSortTarget.nested_context`.
- nested-context is required for nested leaf fields and forbidden for non-nested fields.
- when caller sort omits final deterministic tie-break, gateway MUST still append canonical trailing `doc_id asc`.

Not included:
- scripted nested sort branches.
- nested sort rewrites that require unsupported join or parent/child semantics.

### 8.6 `search_after` Subset

Included:
- Token-mediated deep paging with stable explicit sort keys.
- Use with supported `sort` subset only.
- Continuation MUST use gateway-issued page token (canonical `continuation_page.next_page_token`) bound to admitted sort/query context.

Not included:
- Use without deterministic sort keys.
- Native source `search_after` sort-values array passthrough as canonical continuation input.
- Combinations requiring PIT/scroll parity.

### 8.7 Composite Aggregation (`after`) Subset

Included:
- `composite.sources[]` using supported source strategies:
  - `terms`
  - `histogram`
  - `date_histogram`
- deterministic source declaration order preservation in gateway lowering.
- `after` object continuation mapped to canonical typed `CompositeAggregation.after_key`.
- response `after_key` reconstruction from canonical `BucketResult.composite.after_key.fields`.

Not included:
- source strategies outside declared subset.
- scripted source expressions.
- source dimensions exceeding route-policy guardrail (`max_composite_sources`).

Deterministic continuation contract:
- for no-gap/no-dup multi-page traversal, gateway MUST bind each non-first composite `after` request to the same snapshot context lineage (`search_context.token`) used for the chain.
- `after` key-set mismatch against declared `sources[].name` (or source-incompatible value typing) is malformed and MUST fail closed (`400 SF_BAD_REQUEST`).
- `sources[]` dimension overflow beyond route guardrail is out of subset and MUST fail closed (`400 SF_UNSUPPORTED_QUERY_FEATURE`).

### 8.8 `if_seq_no` Optimistic-Write Subset

Included:
- `if_seq_no` is accepted for Profile-2 admitted write routes and lowered as canonical row precondition `RowMutation.expected_commit_ts`.
- Gateway MUST parse `if_seq_no` as non-negative uint64 before canonical lowering; malformed/negative/out-of-range values are invalid.
- Gateway MUST accept `if_primary_term=1` on the same admitted write routes as a compatibility pass-through for common Elasticsearch SDK OCC loops.
- Requests carrying `if_primary_term>1` MUST be rejected (no silent downgrade).
- On precondition mismatch (`latest_commit_ts != expected_commit_ts`), canonical write path MUST fail closed as `409 SF_VERSION_CONFLICT`.
- For OCC read-modify-write loops, Query Gateway MUST project canonical `QueryHit.version_commit_ts` to Elasticsearch hit `_seq_no` field in read/search responses.
- To avoid common Elasticsearch SDK parse failures, Query Gateway MUST also inject static `_primary_term=1` on read/search hit responses whenever `_seq_no` is projected.

Not included:
- `if_primary_term` values other than `1`.
- Source semantics that require full Elasticsearch primary-term evolution beyond static compatibility value `1`.

Profile 2 still keeps these as non-goals:
- Full script engine compatibility
- Full scroll parity
- Full reindex API parity
- Profile-2 keeps PIT lifecycle out of scope (`POST /{index}/_pit`, `DELETE /_pit`); Profile-3 planning track adds PIT-equivalent lifecycle target.
- Scroll endpoints remain out of scope: `GET /_search/scroll`, `DELETE /_search/scroll`
- Task rethrottle endpoints remain out of scope: `POST /_update_by_query/{task_id}/_rethrottle`, `POST /_reindex/{task_id}/_rethrottle`

## 9. Cross-Profile Restrictions, Guardrails, and Limitations

1. Index lifecycle boundary:
- `PUT /{index}` and `DELETE /{index}` are outside Elasticsearch compatibility scope and remain control-plane operations.
- `GET/PUT /{index}/_mapping` are in Profile-1 scope via canonical control-plane schema path; unsupported mapping branches still fail closed.
- Elasticsearch index-sort settings (`index.sort.*`) are supported only through control-plane equivalence mapping, not via direct `PUT /{index}`.

2. Document ID contract:
- `{id}` is treated as a wire-level string token and MUST be losslessly parseable as base-10 unsigned int64 (Go-equivalent `strconv.ParseUint(id, 10, 64)`).
- Parse failure or out-of-range IDs MUST return `400 SF_INVALID_DOC_ID`.
- `POST /{index}/_doc` auto-ID generation MUST emit decimal-string uint64 tokens only (for example Snowflake-style monotonic IDs); Elasticsearch base64-like UUID auto-IDs are out of profile.
- Gateway MUST cast canonical internal `DocumentID.id_value.u64` back to decimal JSON string form for Elasticsearch `_id` fields in query responses and bulk item responses.
- Gateway MUST project canonical `QueryHit.version_commit_ts` to Elasticsearch `_seq_no` in hit responses and MUST inject static `_primary_term=1` for client-side OCC CAS-loop/SDK compatibility.
- Gateway MUST accept write-side `if_primary_term=1` on routes that admit `if_seq_no`; values `>1` are out of active-profile subset and MUST be rejected (`409 SF_VERSION_CONFLICT`).
- For collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`, Search Fabric owns the canonical `uint64` `doc_id` domain and the active profile admits only `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_UINT64`.
- Caller-supplied or gateway-generated Elasticsearch `_id` tokens MUST normalize directly to canonical `DocumentID.id_value.u64`; TiKV/TiCDC do not provide a second authoritative `doc_id` truth source in this profile.
- Schema/configuration states that attempt to weaken the active canonical `uint64` identity contract or introduce a conflicting external `doc_id` truth source are out of profile conformance scope and MUST be rejected during onboarding/configuration validation.

3. Idempotency contract:
- Clients SHOULD send `Idempotency-Key`.
- Same key + same payload MUST return replayed success semantics.
- Same key + different payload MUST return `409 SF_IDEMPOTENCY_CONFLICT`.
- Gateway MUST derive canonical Flow-B stream-partition identity from this retry-stable idempotency identity (`Idempotency-Key` when provided); if absent, gateway MUST derive deterministic payload-hash-based `batch_id` and use that stable identity instead of transport request IDs.
- Every Elasticsearch-profile route that emits canonical mutating ingest RPCs MUST also carry canonical `RequestMeta` on those RPCs: emitted `MutationBatch.meta.idempotency_key` MUST be non-empty and replay-stable for the admitted outer write identity, and emitted `MutationBatch.meta.client_id` plus `MutationBatch.meta.client_kind=CLIENT_KIND_ES_WRITE_GATEWAY` MUST explicitly identify the stable Elasticsearch gateway caller role rather than relying on route-local idempotency semantics alone; per-instance identity belongs in tracing/telemetry rather than `client_id`.

4. Ownership guardrails:
- Writes MUST satisfy the authoritative `collection_truth_mode` policy and any route/caller rules derived from it.
- Violations MUST return `409 SF_OWNERSHIP_CONFLICT`.

5. Hard limits:
- Max payload: `16 MiB` (`413 SF_PAYLOAD_TOO_LARGE`)
- Max batch operations: `10,000` (`400 SF_TOO_MANY_OPERATIONS`)
- Top-K limit: `1000` (`400 SF_TOPK_TOO_LARGE`)
- Composite source-count limit: route policy `max_composite_sources` (`400 SF_UNSUPPORTED_QUERY_FEATURE` when exceeded)
- Deep paging window limit: `from + size` MUST NOT exceed the collection-level resolved `index.max_result_window` (canonical `effective_max_result_window`, derived from `CollectionSettings.max_result_window` plus control-plane default resolution), else reject with `400 SF_TOO_MANY_OPERATIONS` and details:
  - `limit_scope=max_result_window`
  - `max_result_window`
  - `requested_offset`
  - `requested_limit`
  - `requested_window`
- This Top-K cap is Elasticsearch-profile API scope only; TiDB SQL pushdown limits are governed separately by `docs/specs/core/search-fabric-architecture.md` Section `11.4`.

6. Compatibility evolution rule:
- Any option/feature outside declared profile support MUST fail closed (`400 SF_UNSUPPORTED_OPTION` or `400 SF_UNSUPPORTED_QUERY_FEATURE`), not silently ignored.

7. Index-target form boundary:
- For endpoints that require explicit index targeting, profile-conformant requests MUST resolve external `{index}` to canonical `CollectionNameBinding{namespace_id=<resolved>, collection_name=<index>}`, then to one authoritative `CollectionRef`.
- Multi-target index expressions are outside profile scope, including `_all`, wildcard `*`, comma-separated index lists, remote-cluster selectors, and failure-store selectors using `::`.
- Violations MUST fail closed with deterministic mapping in Section `10.2`.

8. Ownership applicability boundary:
- Elasticsearch compatibility commitments are normative for collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`.
- For collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`, Elasticsearch compatibility (read and write) is out of scope; requests MUST be rejected with `409 SF_OWNERSHIP_CONFLICT` unless a phase-scoped conformance addendum explicitly expands scope per `docs/governance/roadmap.md` Section `3.1`.
- If exposed as convenience routing, such behavior is non-normative and MUST NOT be counted in phase/profile acceptance unless a phase-scoped conformance addendum explicitly includes it per `docs/governance/roadmap.md` Section `3.1`.

### 9.1 Canonical Lowering Limitations

- `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_STRING` and `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_BYTES` are not supported in the active Elasticsearch compatibility profile. The gateway may accept Elasticsearch wire `_id` as a JSON string, but admitted values MUST still lower to canonical `DocumentID.id_value.u64`.
- Elasticsearch mapping branches that require schema semantics not representable by `PutCollectionSchema` / `GetCollectionSchema` are unsupported in this profile and MUST fail closed instead of being approximated gateway-locally.
- Any Elasticsearch read behavior that would require a control-plane read other than `GetCollectionSchema` or `GetCollectionMetadata` is outside the current canonical API surface and therefore outside this compatibility profile until the canonical API is extended. `GetSchemaSyncStatus` remains convergence/status polling only and MUST NOT be treated as a steady-state schema-descriptor substitute.

## 10. Deterministic Error Mapping

Determinism rule:
- For unsupported or out-of-scope Elasticsearch compatibility behavior, the system MUST fail closed with deterministic error code mapping; no silent fallback is allowed.
- Canonical `SF_*` code definitions (status class, retryability, semantic contract) are centralized in `docs/specs/core/error-codes.md` as the source of truth.
- This section only defines Elasticsearch compatibility scenario-to-code mapping and MUST reuse canonical codes from `docs/specs/core/error-codes.md`.
- Error-body shape and mandatory detail-key registry are grounded in `docs/specs/core/error-codes.md`, while canonical precedence application and Elasticsearch-native projection rules remain normative in `docs/specs/core/search-fabric-api.md`.
- Elasticsearch compatibility errors MUST always expose canonical correlation metadata via headers `X-SF-Code` and `X-SF-Correlation-Id`; gateways MAY additionally mirror the same values into Elasticsearch body extension fields, but headers are the required carrier.

### 10.1 Endpoint and Subset Boundary Errors

- Unsupported endpoint path/method for active profile: `400 SF_UNSUPPORTED_ENDPOINT`
- Supported endpoint with unsupported subset branch (for example unsupported option on an otherwise in-scope endpoint): `400 SF_UNSUPPORTED_OPTION`
- Unsupported endpoint-level feature in query path (outside declared profile subset): `400 SF_UNSUPPORTED_QUERY_FEATURE`
- For complex-query unsupported branches (`SF_UNSUPPORTED_QUERY_FEATURE` / `SF_UNSUPPORTED_OPTION`), error `details` MUST follow the canonical detail-key registry in `docs/specs/core/error-codes.md`.
- Aggregation defensive rejection projection on ES routes:
  - admission-time parse/shape/budget reject -> Elasticsearch `parsing_exception`
  - execution-time failure after admission -> Elasticsearch `search_phase_execution_exception`

### 10.2 Option and Payload Errors

- Unsupported request option/parameter: `400 SF_UNSUPPORTED_OPTION`
- `search_after` request branch outside active token-continuation subset (for example raw sort-values array passthrough): `400 SF_UNSUPPORTED_QUERY_FEATURE`
- Composite `after` object key-set mismatch vs declared `sources[]` (or source-incompatible value typing): `400 SF_BAD_REQUEST`
- Composite source-count overflow beyond route policy `max_composite_sources`: `400 SF_UNSUPPORTED_QUERY_FEATURE`
- `size=0` request without canonical zero-hit lowering intent (`aggs` non-empty and/or explicit `track_total_hits` configuration via `enabled` or numeric `threshold`): `400 SF_BAD_REQUEST`
- `aggs.*.value_count` with missing/empty `field`: `400 SF_BAD_REQUEST`
- Invalid `/_bulk` NDJSON payload shape: `400 SF_INVALID_NDJSON_BULK`
- Missing required `_index` in `POST /_bulk` item: `400 SF_INDEX_TARGET_REQUIRED`
- Missing required `index` in `POST /_msearch` pathless header item: `400 SF_INDEX_TARGET_REQUIRED`
- Missing required explicit index target for subset endpoints that require it: `400 SF_INDEX_TARGET_REQUIRED`
- Unsupported multi-target index expression for endpoints requiring concrete single-index target (`_all`, wildcard `*`, comma-separated list, remote selector, failure-store selector): `400 SF_INDEX_TARGET_REQUIRED`
- Malformed `if_seq_no` optimistic precondition (non-integer, negative, or uint64 overflow): `400 SF_BAD_REQUEST`
- Unsupported `if_primary_term` optimistic precondition value (`>1`): `409 SF_VERSION_CONFLICT`
- `PUT /{index}/_mapping` payload branch not representable in canonical schema model: `400 SF_UNSUPPORTED_OPTION`
- `PUT /{index}/_mapping` payload violates canonical schema contracts (type/semantic/tuning/index constraints): `422 SF_SCHEMA_VIOLATION`
- Invalid/unsupported index-sort setting combination during control-plane normalization:
  - not representable in canonical control-plane schema/settings model: `400 SF_UNSUPPORTED_OPTION`
  - representable but violates canonical schema contract: `422 SF_SCHEMA_VIOLATION`
- Optional blocking schema-evolution wait budget expires after an otherwise-admitted additive schema change is waiting for infer-replay convergence: `503 SF_CATALOG_UNAVAILABLE` (retryable)

### 10.3 Contract, Ownership, and Limits

- `{id}` token not losslessly parseable by base-10 uint64 parser (`strconv.ParseUint(id, 10, 64)` equivalent): `400 SF_INVALID_DOC_ID`
- Gateway auto-generated `_id` (for `POST /{index}/_doc`) violating canonical decimal-uint64 contract: `500 SF_INTERNAL` (implementation bug; fail closed).
- Canonical doc-id schema branch requesting `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_STRING` or `SCALAR_FIELD_TYPE_BYTES`: `400 SF_DOC_ID_CONTRACT_VIOLATION`
- Canonical doc-id schema branch requesting malformed source-field enum outside the admitted set: `400 SF_INVALID_SCHEMA`
- Search-Fabric-truth collection configuration violating the active canonical `doc_id` contract (`SCALAR_FIELD_TYPE_UINT64` and one authoritative Search-Fabric-owned `uint64` identity domain): `400 SF_DOC_ID_CONTRACT_VIOLATION`
- Ownership policy violation derived from authoritative `collection_truth_mode`: `409 SF_OWNERSHIP_CONFLICT`
- Catalog/descriptor unavailable for ownership/schema validation: `503 SF_CATALOG_UNAVAILABLE`
- Payload exceeds hard limit: `413 SF_PAYLOAD_TOO_LARGE`
- Batch operation count exceeds hard limit: `400 SF_TOO_MANY_OPERATIONS`
- Requested Top-K exceeds hard limit: `400 SF_TOPK_TOO_LARGE`

### 10.4 Timeout Errors

- Query timeout: `504 SF_QUERY_TIMEOUT`
- Wait-for-visible timeout (only when caller requests `refresh=true` or `refresh=wait_for`, which maps to `visibility_policy=wait_for_visible`): `504 SF_VISIBILITY_TIMEOUT`

### 10.5 Common Access, State, and Backend Errors

- Authentication failure or missing credentials: `401 SF_UNAUTHORIZED`
- Authenticated caller lacks required namespace/collection/API permission: `403 SF_FORBIDDEN`
- Target index/table does not exist: `404 SF_RESOURCE_NOT_FOUND`
- Target document does not exist for read/update/delete (when upsert path is not enabled): `404 SF_DOC_NOT_FOUND`
- Create-only write conflicts with existing document identity or version precondition: `409 SF_VERSION_CONFLICT`
- Same idempotency key reused with different payload: `409 SF_IDEMPOTENCY_CONFLICT`
- Tenant/API throttling or admission quota rejection: `429 SF_RATE_LIMITED`
- Payload/schema/type mismatch: `422 SF_SCHEMA_VIOLATION`
- Vector payload dimension mismatch: `422 SF_VECTOR_DIMENSION_MISMATCH`
- Required backend dependency temporarily unavailable: `503 SF_BACKEND_UNAVAILABLE`

### 10.6 Generic and Internal Fallback Errors

- Malformed request envelope/body that does not match any narrower deterministic mapping: `400 SF_BAD_REQUEST`
- Query structure invalid for active Elasticsearch compatibility profile (after subset/feature checks): `400 SF_INVALID_QUERY`
- Unclassified internal server failure after deterministic mapping rules are exhausted: `500 SF_INTERNAL`

## 11. Quantified Parity Roadmap (Profile 2 -> Profile 3)

Near-term objective (`Profile 2 Hardening`):
- Positioning: high-determinism subset compatibility (not blanket “fully compatible” claim).
- Deterministic conformance gate: `100%` pass on replay/pagination/hash corpus for delivered Profile-2 subset.
- Unsupported-branch gate: `>=99.9%` correct machine-readable rejection payloads per `docs/specs/core/error-codes.md` detail-key registry.

Mid-term objective (`Profile 3 Expansion`):
- Priority 1 (deep paging): PIT-equivalent lifecycle parity (`open`/`keep_alive`/`close`) with `search_after` continuity under refresh and rolling-upgrade scenarios.
- Priority 2 (bulk hardening): preserve the already-delivered canonical `continue_on_error + item_outcomes` baseline across replay/failover/SDK edge cases with `100%` per-item reconstruction accuracy.
- Priority 3 (remaining write-surface parity): close high-frequency write-path gaps after deep paging and bulk semantics are stable.

Long-term objective (`Profile 3+ Continuous Parity`):
- Move compatibility testing from document-driven cases to spec-driven generation:
  - Elasticsearch API-spec/docs -> generated endpoint/option/query conformance corpus.
  - Canonical-lowering assertions auto-generated with deterministic replay validation.
- CI gate requires regenerated conformance cases for any upstream spec delta to prevent semantic drift.