# TiDB / Elasticsearch / Turbopuffer <-> Search Fabric Integration API Design Date: 2026-03-09 Status: Frozen (Normative API contract) Owner: TiDB Compute Integration / Search Fabric API ## 1. Scope This document defines the system-level interaction API between upstream clients (TiDB compute for MySQL-compatible queries, Elasticsearch Query Gateway, and Turbopuffer Query Gateway) and a decoupled Search Fabric service. Compatibility commitment model (normative): - Layer 1 (`Wire Parity @ Gateway`): Elasticsearch and Turbopuffer HTTP/OpenAPI compatibility is implemented and validated at gateway boundaries. - Layer 2 (`Canonical Contract @ Search Fabric`): `api/proto/searchfabric/v1/search_fabric.proto` is the machine-readable public/cross-language gRPC/Protobuf contract used after gateway normalization, and this document is the normative semantic companion specification around that public contract. Internal service-local protobuf contracts that refine the same system behavior live alongside it under `api/proto/searchfabric/v1/` and remain subordinate machine-readable system-owner files rather than a second semantic authority. - “Compatibility” in this document always means deterministic lowering from Layer 1 to Layer 2 within the declared profile scope; it does not imply one-to-one source wire protocol passthrough at this boundary. Normative references: - `docs/architecture/tidb-search-fabric-integration.md` - `docs/compatibility/turbopuffer.md` - `docs/compatibility/elasticsearch.md` - `docs/compatibility/mysql-fulltext.md` - `docs/compatibility/tidb-enhancement.md` - `docs/testing/tidb-emulator-integration-test-plan.md` - `docs/specs/core/emulator.md` - `docs/specs/core/error-codes.md` Primary goals: - gRPC is the only cross-boundary protocol. - A single canonical query request/response supports scalar filtering, vector KNN, full-text match, and hybrid fusion. - TiDB distributed execution can push down local partial aggregation, TopN, and Limit/Offset. - CDC-driven low-latency DML sync and snapshot-based full sync are both first-class. - DDL/schema changes are synchronized through control-plane APIs with explicit fencing and `RequestMeta.idempotency_key` idempotency contracts. - Canonical semantics are ecosystem-neutral and support compatibility requirements for MySQL, Elasticsearch, and Turbopuffer through gateways. Non-goals: - Search Fabric joins TiDB 2PC. - TiDB internal AST/operator/protobuf payloads cross the API boundary. - TiDB-facing canonical API itself does not provide full wire-level parity of ES/Turbopuffer/MySQL external APIs (wire parity remains gateway-owned). - Source-brand query syntax handling/parsing inside Integration API (this belongs to Query Gateway normalization layer). - Elasticsearch/Turbopuffer wire protocols are not implemented here; gateways remain responsible for wire parity. ## 2. Revision Note This document is the normative current-state contract. - Historical review ledgers, resolved defect lists, and iterative change logs are intentionally kept outside this source-of-truth spec. - Only currently active protocol, semantic, validation, and interoperability rules belong in the sections below. ## 3. Interaction Architecture ### 3.1 Planes and Service Boundaries 1. Ingest plane (write/sync): - `TiDB -> TiCDC -> CDC Adapter -> Ingest Gateway (IngestGatewayService)` for authoritative aux-index projection ingest - `TiDB Federated SQL Write Gateway -> Ingest Gateway (IngestGatewayService)` for direct remote DML against Search-Fabric-owned federated collections - `Elasticsearch Write Gateway -> Ingest Gateway (IngestGatewayService)` - `Turbopuffer Write Gateway -> Ingest Gateway (IngestGatewayService)` 2. Query plane: - `TiDB Planner/Executor -> Query Gateway (RelationalQueryService as the current TiDB mapping for projected and primary query routes; QueryGatewayService for shared canonical query clients plus shared readiness/statistics helpers)` - `Elasticsearch Query Gateway -> Query Gateway (QueryGatewayService)` - `Turbopuffer Query Gateway -> Query Gateway (QueryGatewayService)` 3. Control plane: - `TiDB DDL/Schema Manager -> CatalogControlService` - `Elasticsearch/Turbopuffer schema managers -> CatalogControlService` (when gateway-managed schemas are enabled) - `Namespace Provisioner / Admin -> CatalogControlService` (namespace create/update/ensure) - `Elasticsearch write gateway -> IdAllocatorService` when the resolved collection/profile admits allocator-backed create/insert identity authority; this is a pre-ingest control-plane step and does not relax the canonical `RowMutation.doc_id` requirement on `StreamMutations` Only two public collection truth modes are normative: - `COLLECTION_TRUTH_MODE_EXTERNAL` - `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` Compatibility applicability baseline (aligned with compatibility profiles): - MySQL FTS compatibility commitments are normative for `COLLECTION_TRUTH_MODE_EXTERNAL`. - Elasticsearch compatibility commitments are normative for `COLLECTION_TRUTH_MODE_SEARCH_FABRIC`. - `COLLECTION_TRUTH_MODE_EXTERNAL` is out of Elasticsearch compatibility scope for direct Elasticsearch-native routes and MUST fail closed with ownership/profile precedence. - Turbopuffer wire-parity commitments are gateway-surface contracts for `COLLECTION_TRUTH_MODE_SEARCH_FABRIC`; requests entering this API MUST already be canonicalized to this contract's typed payloads. - `COLLECTION_TRUTH_MODE_EXTERNAL` is out of Turbopuffer wire-parity scope for direct Turbopuffer-native routes and MUST fail closed with ownership/profile precedence. Note (TiDB terminology mapping): - TiDB `aux index table` maps to `COLLECTION_TRUTH_MODE_EXTERNAL`. - TiDB `federated table` maps to `COLLECTION_TRUTH_MODE_SEARCH_FABRIC`. - Requests outside active profile applicability MUST fail closed with `SF_OWNERSHIP_CONFLICT` (or `SF_UNSUPPORTED_ENDPOINT` by precedence when route itself is out of scope). ### 3.2 gRPC Transport Choices - Unary RPC: - query execution (`ExecuteQuery`) - TiDB projected-route execute-or-fallback (`ExecuteProjectedQueryOrFallback`) - TiDB primary-route execute (`ExecutePrimaryQuery`) - readiness check (`GetReadiness`) - collection statistics fetch (`GetCollectionStatistics`) - catalog metadata/schema/lifecycle reads and mutations (`CatalogControlService`) - pre-ingest document-ID allocation (`IdAllocatorService.AllocateIds`) - Bidirectional stream: - real-time mutation ingest with incremental ACK - snapshot chunk upload with resumable ACK - Server stream: - large query result streaming (`StreamQuery`) Transport constraints: - mTLS + service identity authz. - Explicit deadlines per request. - Request-level trace id propagation. - Fail-closed for unknown options/unsupported subset branches. ### 3.3 Performance and Latency Targets (Design Budgets) These are API-level budgets for planning and admission, not release SLO commitments: - `ExecuteProjectedQueryOrFallback` p95 added RPC latency budget (TiDB projected-query fast path): <= 2 ms over TiDB local baseline. - `GetCollectionStatistics` collection-only fast path (`target_fields=[]`) warm-cache p95 budget: <= 1 ms at nominal load. - `StreamMutations` acknowledgment budget for `policy.durability=wal_chunk_durable` + `policy.visibility=async`: <= 20 ms p95 under nominal load. - Snapshot chunk upload ACK interval: configurable by chunk count/size, default every 64 chunks or 4 MiB. - Query payload hard cap and Top-K cap are enforced by profile-specific gateways first, then by canonical API limits. ### 3.4 Versionless Full-Service Contract This API is versionless at runtime and does not use capability-handshake RPCs. Deterministic governance rules: - `api/proto/searchfabric/v1/search_fabric.proto` is the machine-readable canonical public/cross-language contract, and this document is the normative semantic/compatibility companion specification for that public surface. - other documents may explain lowering, compatibility, rollout, and operational behavior, but MUST reference this contract rather than restating message/field definitions. - internal service-local protobuf contracts checked into `api/proto/searchfabric/v1/` are part of the machine-readable system owner set for the current revision, but they remain subordinate to the public contract and this document for public semantics, compatibility behavior, and canonical `SF_*` mapping. - `[Rule P-201]` manifest-backed protobuf governance: `api/proto/searchfabric/public-proto-set.txt` is the machine-readable public/cross-language protobuf-set manifest, and `api/proto/searchfabric/system-proto-set.txt` is the machine-readable checked-out public/internal protobuf-set manifest for the current revision. - `[Rule P-202]` package-boundary governance: `api/proto/searchfabric/v1/search_fabric.proto` remains in `package searchfabric.v1`, while internal-only protobuf owner files under `api/proto/searchfabric/v1/` use `package searchfabric.v1.internal`; the file layout stays shared, but the package namespace boundary is now machine-visible and launch-authoritative. - limits, retry windows, token TTL defaults, and feature gates are service-static policy and are documented in this contract. - if a request uses unsupported semantics, server MUST fail closed with canonical `SF_*` errors (for example `SF_UNSUPPORTED_OPTION` / `SF_UNSUPPORTED_QUERY_FEATURE`). - rolling upgrades MUST preserve deterministic behavior through additive-only schema evolution and fail-closed handling for unknown/invalid inputs. ### 3.5 Security and Tenancy Scope Contract - mTLS identity MUST be mapped to allowed `namespace_id`/`collection_id` scope before any plane-specific admission. - gateways use `CollectionRef` as the stateless routing/authz/tenant-throttling key; downstream execution nodes MUST still re-verify scope/session binding and MUST NOT trust gateway routing alone. - `CollectionRef` is a stable catalog identity, not a loose tuple carrier: `namespace_id` and `collection_id` MUST both be non-zero, and the canonical proto machine-closes that invariant so zero-valued collection identities are malformed contract drift rather than tolerated placeholders. - if caller scope and `CollectionRef` mismatch, reject at access tier (`SF_FORBIDDEN`) before feature/limit checks. - for snapshot session-scoped operations (`UploadSnapshotChunks.init`, `CommitSnapshotSync`, `AbortSnapshotSync`, `GetSnapshotSyncProgress`), execution node MUST resolve `sync_session_id` to its bound collection and verify exact equality with request `CollectionRef`; mismatch MUST fail closed (`SF_FORBIDDEN` or `SF_BAD_REQUEST` by precedence). - if `RequestMeta.principal_subject` is provided, server MUST verify it matches authenticated caller identity or an admitted delegated-subject policy; mismatch fails closed (`SF_FORBIDDEN`). - `RequestMeta.client_id` and enum-typed `RequestMeta.client_kind` MUST be present for admitted gateway and internal service clients. - empty `RequestMeta.client_id` is malformed and, when no higher-precedence violation applies, MUST fail closed (`SF_BAD_REQUEST`). - admitted mutating-request replay identity is derived from server-authenticated caller identity plus non-empty `RequestMeta.client_id`; owner-defined replay keys MAY add stricter RPC-specific discriminators such as allocator `client_kind`, while caller-supplied `principal_subject` remains audit/authz context only and MUST NOT participate in replay-key derivation. - authenticated-caller / `client_id` / `client_kind` mismatches against mTLS role bindings fail closed (`SF_FORBIDDEN`). - requests carrying `client_kind=CLIENT_KIND_UNSPECIFIED` are malformed; when no higher-tier violation is selected by canonical precedence, emitted code MUST be `SF_BAD_REQUEST`. - rate-limit and quota enforcement is per namespace and per API plane: - query QPS/Top-K budget, - mutation ingest bytes/entries, - snapshot upload bytes/chunks. - overload or runtime cost-governor branches selected as `SF_RATE_LIMITED` MUST include the canonical diagnostic keys defined in `docs/specs/core/error-codes.md` (`retry_after_ms`, `throttled_dimension`, and `server_limit`) in `ErrorStatus.diagnostics`. ### 3.6 Token Security and Replay Guardrails For mutation `StreamCursor.resume.token`, snapshot `SnapshotChunkAckSuccessPayload.upload_resume.token`, query `PageCursor.next_page.token`, query `PageCursor.search_context.token`, query `PageCursor.composite_page.token`, and control-plane catalog `ListCollections.next_page_token` / `ListNamespaces.next_page_token`: - token payload MUST be signed (server-held key, explicit `key_id`) and include: - server-authenticated caller identity binding (service identity hash), - stable logical `client_id`, - `client_kind`, - `principal_subject` when that field is present on the bound request, - token-purpose binding (`mutation_stream_resume`, `snapshot_upload_resume`, `query_page`, `query_search_context`, `query_composite_page`, `catalog_list_collections_page`, or `catalog_list_namespaces_page`), - issued-at and expiry timestamp. - token-family-specific scope bindings: - mutation resume tokens MUST bind `namespace_id` and `collection_id` - snapshot upload resume tokens MUST bind `namespace_id` and `collection_id` - query page / search-context / composite-page tokens MUST bind `namespace_id` and `collection_id` - `ListCollections.next_page_token` MUST bind `namespace_id` - `ListNamespaces.next_page_token` has no namespace/collection object scope beyond its caller identity and catalog method binding in this revision - `query_search_context` tokens are self-contained snapshot carriers: - token payload MUST include the immutable snapshot-binding tuple needed to continue the admitted query lineage (`canonical_query_digest`, `effective_read_ts`, `applied_schema_generation_token`, committed manifest pin, collection identity, continuation lineage identity), - token payload MUST also carry its own expiry as token-issuance state; expiry is not part of the immutable lineage tuple because keep-alive may mint a later-expiring token in the same lineage without changing the pinned snapshot, - when the admitted projected query route also carries trusted relational snapshot-overlay state through `ProjectedQueryRequest`, token payload MUST additionally bind `snapshot_read_ts` plus a canonical overlay digest derived from the normalized columnar `snapshot_overlay`, - ordinary query execution MUST reconstruct continuation semantics from the validated token payload rather than by looking up PIT state in CCS or another stateful query-plane registry, - keep-alive MAY mint a fresh token in the same lineage with a later expiry without changing the pinned snapshot. - `query_page` tokens are self-contained continuation carriers: - token payload MUST include the same immutable snapshot-binding tuple used by `query_search_context` tokens (`canonical_query_digest`, `effective_read_ts`, `applied_schema_generation_token`, committed manifest pin, collection identity, continuation lineage identity), - token payload MUST also carry its own expiry as token-issuance state; when one unary broker execution emits both `query_page` and `query_search_context` tokens, both carriers MUST share the same execution-local expiry horizon, - when the admitted projected query route also carries trusted relational snapshot-overlay state through `ProjectedQueryRequest`, token payload MUST additionally bind `snapshot_read_ts` plus a canonical overlay digest derived from the normalized columnar `snapshot_overlay`, - token payload MUST additionally include the last-emitted canonical sort cursor, including the forced final `doc_id asc` tie-break, so continuation remains no-gap/no-dup even when the caller changes `fetch_spec.limit`, - ordinary continuation execution MUST reconstruct page-resume semantics from the validated token payload rather than from hidden server-side page state. - `query_composite_page` tokens are self-contained composite-continuation carriers: - token payload MUST include the same immutable snapshot-binding tuple used by `query_search_context` tokens (`canonical_query_digest`, `effective_read_ts`, `applied_schema_generation_token`, committed manifest pin, collection identity, continuation lineage identity), - token payload MUST also carry its own expiry as token-issuance state; when one unary broker execution emits both `query_page` and `query_composite_page` tokens, both carriers MUST share the same execution-local expiry horizon, - when the admitted projected query route also carries trusted relational snapshot-overlay state through `ProjectedQueryRequest`, token payload MUST additionally bind `snapshot_read_ts` plus a canonical overlay digest derived from the normalized columnar `snapshot_overlay`, - token payload MUST additionally include composite page ordinal plus the canonical emitted composite cursor (`BucketResult.composite.after_key.fields`), and any compatibility-layer `CompositeAggregation.after_key` echo on a later request MUST exactly match that token-bound cursor, - composite continuation execution MUST reconstruct resume semantics from the validated token payload rather than from hidden server-side page state. - catalog page tokens MUST use this same signing format, `key_id` governance, TTL policy, and key-rotation overlap window; they are not a separate ad hoc token family. - catalog page tokens MUST additionally bind: - `rpc_method`, - list scope (`namespace_id` for `ListCollections`), - literal prefix filter (`collection_name_prefix` or `namespace_name_prefix`) after canonical NFC validation, - effective page size after default resolution, - last-emitted canonical sort key (`collection_name` or `namespace_name`) under binary ordering. - mutation resume tokens MUST additionally bind `source_partition` and `source_epoch`. - mutation resume tokens MUST additionally bind acknowledged cursor tuple (`commit_ts`, `source_epoch`, `source_seq`). - snapshot upload resume tokens MUST additionally bind `sync_session_id` and `next_expected_seq`. - tokens MUST be rejected if signature validation fails (`SF_BAD_REQUEST` for malformed token envelope). - tokens MUST be rejected if principal or scope mismatches (`SF_FORBIDDEN`). - expired query-page/search-context token -> `SF_INVALID_QUERY` on query execution/continuation paths. - expired snapshot upload resume token -> `SF_INVALID_QUERY`. - mutation-stream reconnect and snapshot-upload reconnect still keep their own path-specific recovery rules after expiry classification. - query-page, composite-page, and search-context tokens have no public early-close control in the active revision; invalidation is by natural expiry or by the token-bound serving lineage becoming non-retained/non-admissible. - canonical token/identity conflict classification: - token signature/format errors and structural token corruption -> `SF_BAD_REQUEST`; - query-page/search-context token expiry or semantic mismatch with token-bound query/page contract -> `SF_INVALID_QUERY`; - catalog page-token expiry, semantic mismatch with token-bound list contract, or request-field drift against bound pagination inputs -> `SF_BAD_REQUEST`; - caller identity (`principal_subject` when present, mandatory stable logical `client_id`, `client_kind`, or server-authenticated caller identity) or namespace/collection authorization mismatch -> `SF_FORBIDDEN`; - mutation/snapshot reconnect token expiry-independent payload-binding mismatches (cursor/session/sequence mismatch) -> `SF_IDEMPOTENCY_CONFLICT`. - key rotation MUST allow a bounded overlap window: - current signing key issues new tokens, - previous key remains verify-only until overlap window ends, - tokens signed by keys outside verify window are rejected deterministically. `AllocateIdsSuccessResponse.allocator_receipt` and `MutationBatch.allocator_receipt` reuse the same `IssuedToken` carrier shape but are not query/catalog continuation tokens: - allocator receipts are single-hop signed follow-on attestations for allocator-backed create/insert lowering, not resumable cursor/state tokens, - they MUST bind server-authenticated caller identity, `client_id`, `client_kind`, resolved `CollectionRef`, token purpose `allocator_mutation_receipt`, `route_intent=ID_ALLOCATION_ROUTE_INTENT_CREATE_INSERT`, the canonical ordered realized-ID result, and the returned `schema_generation_token`, - `MutationBatch` callers MUST propagate the exact receipt returned by `AllocateIds` unchanged when constructing the follow-on allocator-backed batch, - ingest MUST reject missing, forged, caller-mismatched, scope-mismatched, realized-ID-mismatched, route-intent-mismatched, schema-token-mismatched, or expired allocator receipts before row admission. - canonical allocator-receipt failure mapping: - `receipt_validation_reason=caller_mismatch` -> `SF_FORBIDDEN` - `receipt_validation_reason in {missing_required_receipt, malformed_receipt, signature_invalid, expired_receipt, scope_mismatch, realized_doc_ids_mismatch, route_intent_mismatch, schema_generation_token_mismatch}` -> `SF_BAD_REQUEST` - machine-readable consumers MUST treat `ErrorStatus.context.allocator.receipt_validation.reason` as authoritative when that typed branch is present. ### 3.7 Request Metadata and Typing Symmetry - All unary requests in Query/Data/Control planes MUST carry `RequestMeta` for deterministic caller identity, replay governance, and policy auditing. - `RequestMeta.idempotency_key` contract: - the canonical proto models this field as `optional string` with machine-checkable non-empty validation when present, so implementations can distinguish omission from malformed empty-present input. - RPCs that define durable replay MUST set it non-empty (`PutCollectionMetadata`, `PutCollectionSchema`, `DropCollection`, `PutNamespaceMetadata`, `AllocateIds`, `StartSnapshotSync`, `StreamMutations`, `UploadSnapshotChunks`, `CommitSnapshotSync`, `AbortSnapshotSync`). - the canonical proto machine-closes that route family through message-level invariants (`required_non_empty:meta.idempotency_key`) on each replay-governed request carrier rather than leaving the requirement as comment-only intent. - helper/query/catalog-read RPCs (`ExecuteProjectedQueryOrFallback`, `ExecutePrimaryQuery`, `ExecuteQuery`, `StreamQuery`, `GetReadiness`, `GetCollectionStatistics`, `GetSnapshotSyncProgress`, `GetCollectionSchema`, `ListCollections`, `GetCollectionMetadata`, `ListNamespaces`, `GetNamespaceMetadata`, `GetSchemaSyncStatus`, `DebugDigestProbe`) MAY omit it; when present on one of these RPCs it MUST still be non-empty, and it remains caller-chosen correlation metadata only rather than durable replay state. - `idempotency_key` is an externally supplied replay identity string and MUST be at most 128 Unicode scalar values. - RPCs with durable replay semantics in the active revision are `PutCollectionMetadata`, `PutCollectionSchema`, `DropCollection`, `PutNamespaceMetadata`, `AllocateIds`, `StartSnapshotSync`, `StreamMutations`, `UploadSnapshotChunks`, `CommitSnapshotSync`, and `AbortSnapshotSync`. - replay of the same owner-defined durable-replay key with different canonical payload MUST fail closed (`SF_IDEMPOTENCY_CONFLICT`) for those durable-replay RPCs. - for durable-replay RPCs other than `AllocateIds`, the key in this revision is `(server-authenticated caller identity, client_id, rpc_method, scope, idempotency_key)`. - `AllocateIds` replay scope is the resolved `CollectionRef`, and its replay key is `(server-authenticated caller identity, client_id, client_kind, rpc_method, collection, idempotency_key)` because allocator authority is granted by caller-kind policy; same replay key plus identical canonical payload (`collection`, `route_intent`, and the ordered allocation plan payload) MUST return the original realized-ID result verbatim, and the server MUST NOT advance allocator state again. - `StreamMutations` idempotency keys are scoped per admitted batch attempt. - byte-identical retransmit MAY reuse the same `idempotency_key`. - any retransmit that changes canonical batch content, including failover retransmit with higher `source_epoch`, MUST use a new `idempotency_key`. - `RequestMeta.correlation_id` contract: - the canonical proto models this field as `optional string` with machine-checkable non-empty validation when present. - optional on all RPCs and never participates in replay-key derivation. - when present, it is the caller-preferred trace/audit correlation identifier for logs, receipts, and emitted error correlation. - `correlation_id` MUST be at most 128 Unicode scalar values. - `RequestMeta.client_kind` MUST be explicit and MUST NOT be `CLIENT_KIND_UNSPECIFIED` for admitted requests. - `RequestMeta.client_id` is modeled as `optional string` with machine-checkable non-empty validation when present; admitted gateway/internal-service requests MUST set it present and non-empty. - `RequestMeta.principal_subject` is modeled as `optional string` with machine-checkable non-empty validation when present; omission is allowed, but empty-present values are malformed. - emitted `ErrorStatus.correlation_id` is machine-checked as non-empty on the canonical protobuf contract; services MUST NOT emit an empty correlation identifier. - the canonical proto uses machine-checkable `sf_non_zero_enum` annotations on enum fields whose zero/sentinel value is semantically invalid on the admitted public contract. In the current query/ingest/control contract this includes: - query/read-side request enums and enum elements: `RequestMeta.client_kind`, `ReadConsistencySpec.mode`, `ComparisonPredicate.op`, `SortKey.order`, `FieldSortTarget.missing_value_policy` when present, `FieldSortTarget.mode` when present, `VectorProjectionSpec.format` when present, `MetricAggregation.builtin_function`, `AggregationSortKey.builtin` when selected, `AggregationSortKey.order`, `CompositeSource.order`, `RoutingPushdownSpec.target_tiers[]`, and `NestedClause.score_mode` - internal/public diagnostics request enums: `DebugDigestProbeRequest.operation_kind` - ingest/snapshot request enums: `IngestPolicy.ingest_mode`, `AllocateIdsRequest.route_intent`, `SnapshotChunk.checksum_algorithm`, `SnapshotChunk.payload_encoding`, `SnapshotChunk.compression_codec`, and `CommitSnapshotSyncRequest.expected_root_hash_algorithm` - typed public value/schema enum elements: `AtomicValue.null_value` when the null branch is selected, and `IndexOverflowProtection.behavior` when present - schema/control request enums and enum elements: `DocIDMapping.source_field_type`, `DocIDMapping.generation_policy`, `AllocatorAdmissionPolicy.caller_profile`, `VectorTypeSpec.element_type`, `VectorTypeSpec.distance_metric`, `FieldConfig.field_definition.scalar_type`, `IndexSortFieldSpec.order`, `IndexSortFieldSpec.mode`, `IndexSortFieldSpec.missing_value_policy`, `TierTransition.target_tier`, `LifecyclePolicy.initial_tier`, `CollectionSchemaDefinition.collection_truth_mode`, and `PutCollectionSchema.update_policy` - canonical error-context enums: `ErrorStatus.context.allocator.limit_rejection.limit_scope`, `ErrorStatus.context.allocator.reservation_conflict.conflict_reason`, and `ErrorStatus.context.allocator.receipt_validation.reason` - implementations and validators MUST enforce those annotations before business admission or public error emission. - the canonical proto carries machine-readable `sf_required_field` annotations on mandatory public request-visible carriers whose presence proto3 alone cannot force. In the current revision this includes: - Query/helper unary carriers: `QueryRequest.meta`, `QueryRequest.collection`, `QueryRequest.consistency`, `QueryRequest.fetch_spec`, `ProjectedQueryRequest.query`, `PrimaryQueryRequest.query`, `StreamQueryRequest.meta`, `StreamQueryRequest.collection`, `StreamQueryRequest.consistency`, `StreamQueryRequest.fetch_spec`, `ReadinessRequest.meta`, `ReadinessRequest.collection`, `ReadinessSuccessResponse.applied_schema_version`, `ReadinessSuccessResponse.applied_schema_generation_token`, `GetCollectionStatisticsRequest.meta`, `GetCollectionStatisticsRequest.collection`, `GetCollectionStatisticsSuccessResponse.stats_commit_ts`, `GetCollectionStatisticsSuccessResponse.collected_at_unix_ms`, `GetCollectionStatisticsSuccessResponse.applied_schema_version`, `GetCollectionStatisticsSuccessResponse.applied_schema_generation_token`, `DebugDigestProbeRequest.meta`, and `DebugDigestProbeRequest.collection`. - Snapshot/control carriers: `AllocateIdsRequest.meta`, `AllocateIdsRequest.collection`, `AllocateIdsRequest.plan`, `StartSnapshotSyncRequest.meta`, `StartSnapshotSyncRequest.collection`, `StartSnapshotSyncRequest.snapshot_ts`, `StartSnapshotSyncRequest.schema_version`, `StartSnapshotSyncRequest.schema_generation_token`, `StartSnapshotSyncRequest.client_session_token`, `CommitSnapshotSyncRequest.meta`, `CommitSnapshotSyncRequest.sync_session_id`, `CommitSnapshotSyncRequest.collection`, `AbortSnapshotSyncRequest.meta`, `AbortSnapshotSyncRequest.sync_session_id`, `AbortSnapshotSyncRequest.collection`, `SnapshotSyncProgressRequest.meta`, `SnapshotSyncProgressRequest.collection`, `SnapshotSyncProgressRequest.sync_session_id`, `SnapshotUploadInit.meta`, `SnapshotUploadInit.sync_session_id`, `SnapshotUploadInit.collection`, `PutCollectionSchemaRequest.meta`, `PutCollectionSchemaRequest.scope`, `PutCollectionSchemaRequest.schema_version`, `PutCollectionSchemaRequest.fence`, `PutCollectionSchemaRequest.definition`, `PutCollectionSchemaRequest.update_policy`, `DropCollectionRequest.meta`, `DropCollectionRequest.collection`, `SchemaSyncStatusRequest.meta`, `SchemaSyncStatusRequest.collection`, `GetCollectionSchemaRequest.meta`, and `GetCollectionSchemaRequest.collection`. - Catalog unary carriers and typed lookup/binding scalars: `CollectionNameBinding.namespace_id`, `CollectionNameBinding.collection_name`, `CollectionDescriptor.collection`, `CollectionDescriptor.collection_name`, `CollectionDescriptor.created_at_unix_ms`, `CollectionDescriptor.updated_at_unix_ms`, `CollectionDescriptor.encryption`, `CollectionDescriptor.completeness`, `ListCollectionsRequest.meta`, `ListCollectionsRequest.namespace_id`, `GetCollectionMetadataRequest.meta`, `GetCollectionMetadataRequest.lookup`, `GetCollectionMetadataSuccessResponse.descriptor`, `PutCollectionMetadataRequest.meta`, `PutCollectionMetadataRequest.name_binding`, `PutCollectionMetadataRequest.definition`, `PutCollectionMetadataSuccessResponse.descriptor`, `PutNamespaceMetadataRequest.meta`, `PutNamespaceMetadataRequest.namespace_name`, `PutNamespaceMetadataRequest.definition`, `PutNamespaceMetadataSuccessResponse.descriptor`, `NamespaceLookupKey.namespace_id`, `NamespaceDescriptor.namespace_id`, `NamespaceDescriptor.namespace_name`, `NamespaceDescriptor.created_at_unix_ms`, `NamespaceDescriptor.updated_at_unix_ms`, `NamespaceDescriptor.encryption`, `NamespaceDescriptor.completeness`, `ListNamespacesRequest.meta`, `GetNamespaceMetadataSuccessResponse.descriptor`, `GetNamespaceMetadataRequest.meta`, and `GetNamespaceMetadataRequest.lookup`. - Ingest streaming carriers: `RowMutation.doc_id`, `MutationBatch.meta`, `MutationBatch.batch_id`, `MutationBatch.collection`, `MutationBatch.stream_identity`, `MutationBatch.schema_generation_token`, and `MutationBatch.batch_fingerprint`. - implementations MUST enforce those annotations or equivalent generated validation before business admission. - the canonical proto additionally carries machine-readable `sf_non_empty_string` annotations for string carriers whose empty proto3 default is semantically invalid even when the field is present. - this revision uses that non-empty-string validation on `RequestMeta.idempotency_key`, `RequestMeta.client_id`, `RequestMeta.principal_subject`, `RequestMeta.correlation_id`, emitted `ErrorStatus.correlation_id`, `MutationBatch.batch_id`, `MutationBatch.schema_generation_token`, `SchemaFencePrecondition.schema_generation_token`, `CollectionNameBinding.collection_name`, `PutNamespaceMetadataRequest.namespace_name`, `NamespaceLookupKey.namespace_name`, `StartSnapshotSyncRequest.schema_generation_token`, `StartSnapshotSyncRequest.client_session_token`, and `StreamAttemptId.value`; empty-present request-metadata identity strings are malformed canonical contract drift, empty emitted error correlation IDs are malformed service output, empty ingest/catalog identity binders are malformed public contract drift, empty snapshot binders are malformed public contract drift, and empty `StreamAttemptId.value` strings are malformed internal contract drift that MUST be rejected before reconciliation logic runs. - the canonical proto additionally carries machine-readable `sf_non_empty_repeated` annotations for repeated carriers whose empty proto3 default is semantically invalid. - this revision uses that non-empty-repeated validation on `TextClause.fields`, `TextMatchPredicate.fields`, `ShouldMatchSpec.clauses`, `RangeAggregation.ranges`, `CompositeAggregation.sources`, `AllocateIdPlan.items`, `DescriptorPublishing.missing_fields`, and the reviewed internal owner-set repeated carriers whose owner files already declare the same invariant; empty repeated values on those carriers are malformed checked-out contract drift and MUST be rejected before business admission or internal reconciliation. - the canonical proto additionally carries machine-readable `sf_non_empty_map` annotations for map carriers whose empty proto3 default is semantically invalid. - this revision uses that non-empty-map validation on `ExplicitTopHitsVectorProjection.vector_projections`; empty explicit top-hits vector projection maps are malformed canonical contract drift and MUST be rejected as `SF_BAD_REQUEST`. - the canonical proto additionally carries machine-readable `sf_positive_integral` annotations for integral scalars whose zero value is semantically invalid when the field is present or emitted. - this revision uses that positive-integral validation on `SearchContextSpec.keep_alive_ms`, `CompositeAggregation.limit`, `CrossEncoderRescoreSpec.topk`, `SnapshotOverlayRow.commit_ts`, `ExecuteQuerySuccessResponse.effective_read_ts`, `ExecuteQuerySuccessResponse.safe_ts`, `ExecuteQuerySuccessResponse.applied_schema_version`, `StreamQueryFrame.effective_read_ts`, `StreamQueryFrame.safe_ts`, `StreamQueryFrame.applied_schema_version`, `ProjectedQueryResponse.effective_read_ts`, `ProjectedQueryResponse.safe_ts`, `ProjectedQueryResponse.applied_schema_version`, `ReadinessSuccessResponse.applied_schema_version`, `GetCollectionStatisticsSuccessResponse.stats_commit_ts`, `GetCollectionStatisticsSuccessResponse.applied_schema_version`, `StartSnapshotSyncRequest.snapshot_ts`, `StartSnapshotSyncRequest.schema_version`, `AllocatorAdmissionPolicy.max_plan_items_per_request`, `AllocatorAdmissionPolicy.max_forward_reservation_jump`, `IndexOverflowProtection.max_chars`, `VectorTypeSpec.dimensions`, `CollectionSettings.max_result_window`, `CollectionNameBinding.namespace_id`, `ListCollectionsRequest.namespace_id`, `NamespaceLookupKey.namespace_id`, and `NamespaceDescriptor.namespace_id`; zero-valued namespace identity binders, replay/session binders, admitted sizing controls, and success-path snapshot observability fields are malformed checked-out contract drift and MUST be rejected before business admission or emission. - the canonical proto additionally carries machine-readable `sf_strictly_positive_integral` annotations for integral scalars whose value is semantically invalid when the field is present or emitted as `<= 0`. - this revision uses that strictly-positive-integral validation on `GetCollectionStatisticsSuccessResponse.collected_at_unix_ms`, `CollectionDescriptor.created_at_unix_ms`, `CollectionDescriptor.updated_at_unix_ms`, `NamespaceDescriptor.created_at_unix_ms`, `NamespaceDescriptor.updated_at_unix_ms`, and the reviewed signed internal timestamp fields `BackgroundTaskLease.lease_expiry_unix_ms` and `QuotaLeaseGranted.expires_at_unix_ms`; zero or negative emitted timestamps on those carriers are malformed checked-out contract drift and MUST be rejected before business admission or emission. - current revision additionally machine-closes the reviewed critical internal positive-scalar inventory on `background_compute_coordinator.proto`, `catalog_control_private.proto`, `commit_coordinator.proto`, and `query_broker.proto`: `CommittedManifestBaseline.committed_manifest_version`, `BackgroundTaskLease.lease_epoch`, `BackgroundTaskLease.lease_expiry_unix_ms`, `CollectionServingReadiness.ready_generation_version`, `ResolveCollectionNameBindingResponse.collection_name_binding_revision`, `CollectionAdmissionState.descriptor_revision`, `CollectionAdmissionState.collection_name_binding_revision`, `CollectionAdmissionState.active_schema_version`, `QuotaLeaseScope.namespace_id`, `AcquireQuotaLeaseRequest.requested_units`, `AcquireQuotaLeaseRequest.caller_observed_quota_epoch`, `QuotaLeaseGranted.granted_units`, `QuotaLeaseGranted.quota_epoch`, `QuotaLeaseGranted.lease_epoch`, `QuotaLeaseGranted.expires_at_unix_ms`, `QuotaLeaseDenied.quota_epoch`, `ReturnQuotaLeaseRequest.lease_epoch`, `CommitBatchSlice.commit_ts`, `DurableAnchor.min_commit_ts`, `WriteIntent.coordinator_epoch`, `RegisterWriteIntentResponse.barrier_epoch`, `ReportModalityDurabilityRequest.applied_modality_ts`, `DurableModalityState.applied_ts`, `AdmittedQueryContext.effective_read_ts`, `AppliedSchemaGeneration.version`, `ManifestPin.committed_manifest_version`, and `SnapshotLineageIdentity.effective_read_ts`. - the canonical proto additionally carries machine-readable message-level `sf_invariant_rule` annotations for structural/cardinality constraints that cannot be expressed by field-local presence/non-empty annotations alone. - this revision uses that message-level invariant inventory on: - `CollectionRef` for `non_zero_fields:namespace_id,collection_id` - `MutationBatch`, `AllocateIdsRequest`, `StartSnapshotSyncRequest`, `SnapshotUploadInit`, `CommitSnapshotSyncRequest`, `AbortSnapshotSyncRequest`, `PutCollectionSchemaRequest`, `DropCollectionRequest`, `PutCollectionMetadataRequest`, and `PutNamespaceMetadataRequest` for `required_non_empty:meta.idempotency_key` - `RetrievalGroup` for `at_least_one_of:must_clauses,should_match` and `empty_only_if:must_clauses,should_match` - `ManifestPublishedArtifact` for `ordered_non_overlapping:key_ranges:first_doc_id,last_doc_id` - `CommitBatchSlice` for `gt_fields:end_entry_index,first_entry_index` - validators MUST treat those rule strings as part of the checked-out owner-set contract and reject changes that remove or weaken the corresponding machine-readable invariants without updating the canonical validation policy. - the strongest machine-readable validation inventory is defined on the canonical public contract in `api/proto/searchfabric/v1/search_fabric.proto`. - internal owner-set validation remains intentionally mixed in the current revision: selected internal carriers already use the same annotation vocabulary where the owner proto declares it, current revision additionally machine-closes the reviewed critical internal positive-scalar inventory on `background_compute_coordinator.proto`, `catalog_control_private.proto`, `commit_coordinator.proto`, and `query_broker.proto`, and broader internal conditional/state-machine invariants still mix owner-proto annotations with implementation/conformance enforcement rather than repository-wide machine-closed guarantees. - externally supplied request-metadata strings that participate in replay or authorization context have fixed canonical size limits: - `RequestMeta.idempotency_key`, when present, MUST be at most 128 Unicode scalar values, - `RequestMeta.client_id` MUST be at most 256 Unicode scalar values, - `RequestMeta.correlation_id`, when present, MUST be at most 128 Unicode scalar values, - `RequestMeta.principal_subject`, when present, MUST be non-empty and at most 256 Unicode scalar values. - overlength request-metadata strings are malformed and MUST fail closed as `SF_BAD_REQUEST`; services MUST NOT silently truncate, hash, or normalize them before replay-key derivation, authorization checks, token binding, or persistence. - canonical caller-role matrix in this revision: - `CLIENT_KIND_TIDB_COMPUTE` for TiDB planner/executor aux-index fast-path, query, statistics, and readiness callers, - `CLIENT_KIND_TIDB_WRITE_GATEWAY` for TiDB federated-table remote-write callers targeting `COLLECTION_TRUTH_MODE_SEARCH_FABRIC`, - `CLIENT_KIND_TIDB_DDL_ADAPTER` for TiDB DDL/schema-sync control-plane callers, - `CLIENT_KIND_TIDB_CDC_ADAPTER` for TiCDC-derived ingest callers on `COLLECTION_TRUTH_MODE_EXTERNAL` collections, - `CLIENT_KIND_ES_QUERY_GATEWAY` / `CLIENT_KIND_TP_QUERY_GATEWAY` for query gateways, - `CLIENT_KIND_ES_WRITE_GATEWAY` / `CLIENT_KIND_TP_WRITE_GATEWAY` for Search-Fabric-truth write gateways, - `CLIENT_KIND_SNAPSHOT_IMPORT_WORKER` for admitted offline snapshot/import workers operating through the canonical snapshot-sync RPC lifecycle only. - `CLIENT_KIND_CONTROL_ADMIN` for namespace provisioning/admin clients and other direct CCS administrative callers that are not TiDB/gateway/snapshot actors. - Requests that operate on an already resolved concrete collection instance MUST carry `CollectionRef`; name-based collection catalog lookup/ensure RPCs MAY instead use typed collection name bindings (`CollectionLookupKey.name_binding` or `PutCollectionMetadata.name_binding`) before a `CollectionRef` is known. - Requests that operate on an already resolved concrete namespace instance MUST carry `namespace_id`; name-based namespace catalog lookup/ensure RPCs MAY instead use typed namespace names (`NamespaceLookupKey.namespace_name` or `PutNamespaceMetadata.namespace_name`) before a stable `namespace_id` is known. - all session-control requests carrying `sync_session_id` (including stream-init and unary session RPCs) MUST include `CollectionRef` for execution-node binding verification; per-chunk stream frames may omit `CollectionRef` only after a validated init frame establishes binding context. - Search Fabric scope is strictly two-level for logical data organization: `namespace -> collection`. - The canonical protobuf contract in this revision uses strongly typed protocol status/kind/op carriers (enum + oneof structural typing) to avoid cross-language parser drift, including typed predicate-tree branches, `CompareOp`, fusion algorithms, query sort order, and `RequestMeta.client_kind`. - Unknown enum values MUST be treated as unsupported/invalid request input and handled by canonical precedence (`docs/specs/core/error-codes.md` Section `2.1`). - enum `UNSPECIFIED` handling model: - default rule: action/request-parameter enums MUST fail closed when carrying `*_UNSPECIFIED` (for example `SortOrder`, `CompareOp`). - explicit exception branches are allowed only when this spec documents deterministic fallback semantics and validation coverage. - request-path exception branches in this revision: - `MutationBatch.policy.atomicity=ATOMICITY_POLICY_UNSPECIFIED` means deterministic fallback to the route default (`ATOMICITY_POLICY_FAIL_FAST_PARTIAL` on generic ingest routes, `ATOMICITY_POLICY_ALL_OR_NOTHING` on the explicit TiDB federated-table remote-write route). - `MutationBatch.policy.undeclared_field=UNDECLARED_FIELD_POLICY_UNSPECIFIED` means deterministic fallback to `UNDECLARED_FIELD_POLICY_REJECT`. - `RetrievalGroup.fusion_strategy=GROUP_FUSION_STRATEGY_UNSPECIFIED` means deterministic fallback to `GROUP_FUSION_STRATEGY_WEIGHTED_SUM`. - `CompositeSource.missing_bucket=MISSING_VALUE_POLICY_UNSPECIFIED` means deterministic fallback to `MISSING_VALUE_POLICY_LAST`. - config-property enums MAY use `*_UNSPECIFIED` only when explicitly documented as “inherit schema/default policy”. ### 3.8 Compatibility/Auth Ingress Contract and Auth Mapping Compatibility/auth ingress requirements (Elasticsearch/Turbopuffer edges): - compatibility/auth ingress MUST expose deterministic HTTP status mapping from canonical `SF_*` codes: - exhaustive code-to-status mapping is defined only in `docs/specs/core/error-codes.md`; this section defines ingress-specific body/header projection requirements and MUST NOT be treated as a second status registry. - compatibility/auth ingress error body contract: - canonical protobuf/gRPC error projection MUST carry `ErrorStatus` using the canonical `docs/specs/core/error-codes.md` envelope fields (`code`, `message`, `retryable`, `correlation_id`, optional typed `context`, optional string `diagnostics`). - depending on the owning RPC family, `ErrorStatus` is projected either through a declared protobuf response error branch or through non-`OK` gRPC status details; lower-tier docs MUST NOT redefine that transport split. - when both `ErrorStatus.context` and `ErrorStatus.diagnostics` are present for the same semantic branch, machine-readable protobuf consumers MUST treat `context` as authoritative; `diagnostics` remains the compatibility/text projection layer. - `ErrorStatus.code` is the canonical protobuf `ErrorCode` enum and MUST carry the exact `SF_*` selection projected from the error registry; `ERROR_CODE_UNSPECIFIED` is a reserved invalid sentinel, not a canonical public error code, and MUST NOT be emitted. - Elasticsearch-compatibility routes MUST emit Elasticsearch-native error JSON (`error.type`, `error.reason`, `error.root_cause[]`, top-level `status`) and MUST expose canonical `SF_*` code + `correlation_id` through mandatory response headers (`X-SF-Code`, `X-SF-Correlation-Id`). - `X-SF-Code` header projection MUST use the exact `ErrorCode` symbolic identifier string (for example `SF_RESOURCE_NOT_FOUND`), not the numeric enum value or implementation-local formatting. - compatibility/auth ingress MAY additionally mirror the same canonical diagnostics into Elasticsearch body extension fields such as `error.caused_by`, but headers are the required interoperability carrier. - aggregation defensive rejects on Elasticsearch routes MUST project to Elasticsearch-compatible exception classes: - admission-time parse/shape/budget rejection -> `parsing_exception`, - execution-phase failure after admission -> `search_phase_execution_exception`. - Turbopuffer OpenAPI-parity routes MUST use Turbopuffer `ErrorResponse` shape (`status`, `error`) without extra JSON body fields, and MUST project canonical `SF_*` code + `correlation_id` through deterministic headers (`X-SF-Code`, `X-SF-Correlation-Id`). - detail-key requirements and precedence are defined only in `docs/specs/core/error-codes.md`. 429 and quota signaling: - for `SF_RATE_LIMITED`, compatibility/auth ingress MUST follow the required compatibility-header projection defined in `docs/specs/core/error-codes.md` Section `2.3`. - for route families whose error body shape admits structured details, response body `details` SHOULD repeat deterministic retry hints (`retry_after_ms`, `throttled_dimension`, `server_limit`). Auth mapping flow: 1. external bearer/API-key auth is terminated at the compatibility/auth ingress. 2. the compatibility/auth ingress resolves tenant/account scope and maps it to canonical `namespace_id`/`collection_id` access scope for the downstream Search Fabric service owner. 3. the compatibility/auth ingress calls Search Fabric over mTLS using service identity. 4. the compatibility/auth ingress MUST populate `RequestMeta`: - `principal_subject`: authenticated caller subject (or delegated subject policy token), - `client_id`: stable logical caller/client identity that remains constant across replica retries/resume/pagination lifetimes for the same caller, - `client_kind`: strict enum matching gateway role. - instance-level ingress identity belongs in transport tracing/telemetry (gRPC metadata, logs, metrics), not in `client_id`. 5. internal service clients that call Search Fabric directly (for example TiDB compute, TiDB DDL adapter, TiDB CDC adapter, or a snapshot import worker) MUST likewise populate non-empty `RequestMeta.client_id` plus role-specific `RequestMeta.client_kind`; `principal_subject` remains optional unless the client is explicitly acting on behalf of a delegated subject. 6. Search Fabric re-validates server-authenticated caller identity + `client_id/client_kind` + collection scope before plane-specific admission, and verifies `principal_subject` separately when present. Gateway digest probe endpoint (internal-only): - digest probe MUST be an internal/high-priv diagnostics surface only; it MUST NOT be exposed to regular external API clients. - protocol surface is `InternalDebugService.DebugDigestProbe` (not part of `QueryGatewayService`). - access MUST be restricted to SRE/gateway development operators through dedicated authz policy (mTLS identity allowlist + privileged role binding). - endpoint purpose: verify client-side payload serialization/canonicalization against gateway + service expected digest without mutating data. - request MUST support: - `operation_kind` (`query` or `mutation`), - raw source payload, - optional already-canonical payload (when caller performs local canonicalization). - `caller_canonical_payload` contract: - when absent, probe compares gateway-normalized digest and service-attested digest computed from `source_payload`. - when present, it is advisory comparison input only and MUST NOT replace `source_payload` as canonicalization source of truth. - when present and structurally invalid for declared `operation_kind`, request is malformed (`SF_BAD_REQUEST`). - when present and structurally valid but different from gateway-normalized canonical payload, mismatch classification still follows gateway-vs-service digest pair; caller-vs-gateway deltas MAY be reported via `normalization_diffs[]` only on success outcome. - request-metadata strategy for this RPC: - `DebugDigestProbe` is read-only diagnostics and MAY omit `RequestMeta.idempotency_key`. - when `RequestMeta.correlation_id` is present, it MUST remain traceable in logs/audit metadata. - response MUST include: - `digest_algorithm` (current revision: `HASH_ALGORITHM_BLAKE3`; `HASH_ALGORITHM_UNSPECIFIED` MUST NOT be emitted), - `gateway_normalized_digest`, - `service_attested_digest`, - optional `normalization_diffs[]` (field-path-level normalization notes; success-branch only). - mismatch branch: - MUST return `409 SF_DEBUG_DIGEST_MISMATCH` when digest differs but payload is well-formed. - mismatch error MUST include machine-readable details: - `diagnostics["gateway_normalized_digest"]` - `diagnostics["service_attested_digest"]` - `diagnostics["digest_algorithm"]` - malformed debug payload still returns `400 SF_BAD_REQUEST`. - this endpoint MUST be side-effect-free (`dry_run=true` semantics): no write apply, no visibility change, no schema change. Gateway aggregation-depth intercept configuration: - gateway MUST support hard pre-admission depth guardrail `max_aggregation_depth` (recommended default `3`). - gateway MUST run defensive aggregation-shape validation before canonical emission (reject malformed or adversarial trees instead of forwarding): - recursive node-count ceiling (`max_aggregation_nodes`), - per-level fanout/sibling ceiling (`max_aggregation_siblings`), - aggregate `top_hits.limit`/`terms.size`/`composite.limit`/`inner_hits.limit`/bucket budget ceilings by route policy, - composite-dimension ceiling (`max_composite_sources`) and `CompositeSource.name` uniqueness/shape validation, - cycle/duplicate-name/pathological nesting detection in request JSON shape. - requests failing these checks MUST NOT be forwarded to Query Broker. - if request aggregation tree depth exceeds configured guardrail: - strict mode: fail closed (`400 SF_UNSUPPORTED_QUERY_FEATURE`) with machine-readable details defined in `docs/specs/core/error-codes.md` for the strict aggregation-depth branch, - truncation mode: gateway truncates tree to `max_aggregation_depth` before canonical emission and MUST attach prominent truncation warnings in response payload (see Section `5.7`). ### 3.9 Common Data Structures `DocumentID` is the canonical strongly typed identity container for all document-level read/write/query contracts. To keep the hot path deterministic at EB scale, the protocol uses explicit oneof typing instead of runtime string parsing in core RPC payloads. Phase-1 admission rules: - only `DocumentID.id_value.u64` is part of the current canonical contract. - protobuf field numbers `2` and `3` are reserved for possible future non-`u64` identifier expansion and MUST NOT be reused as active branches in this revision. ## 4. Ingest Plane API Design ### 4.1 Real-Time DML Synchronization #### Canonical mutation stream model Mutation stream uses ordered entries in a batch: - `ROW_MUTATION` (`upsert` / `patch` / `delete`) - `PROGRESS_MARK` (`resolved_ts`, modality progress) - `BARRIER_MARK` (ordering fence, optional for test/emulator and controlled flows) Deterministic apply order: - Primary key: `commit_ts` - Tie-break: `source_partition`, `source_epoch`, `source_seq` - `source_seq` is carried by `MutationEntry.source_seq` and is mandatory for every entry kind (`ROW_MUTATION`/`PROGRESS_MARK`/`BARRIER_MARK`). - source scope: `source_partition` (e.g., TiCDC partition or stream shard identity), compared by raw UTF-8 byte order. #### MutationEntry framing rules To keep mixed entry batches deterministic: - `ROW_MUTATION` entries MUST appear before any `PROGRESS_MARK`/`BARRIER_MARK` in a batch. - at most one `PROGRESS_MARK` is allowed per batch; if present, it MUST appear after all `ROW_MUTATION` entries. - at most one `BARRIER_MARK` is allowed per batch; if present, it MUST be the final entry in the batch. - if both `PROGRESS_MARK` and `BARRIER_MARK` are present, order MUST be `... ROW_MUTATION*, PROGRESS_MARK, BARRIER_MARK`. - framing violations are rejected as malformed batch (`SF_BAD_REQUEST`). Source epoch and sequence invariants (per `source_partition`): - stream scope is represented by mandatory `MutationBatch.stream_identity` (`source_partition`, `source_epoch`) for every admitted batch. - generation scope is represented by mandatory `MutationBatch.schema_generation_token`: - token MUST be non-empty. - token MUST resolve to a catalog-known schema-generation lineage record for `(namespace_id, collection_id)`; unknown token MUST fail closed with `SF_VERSION_CONFLICT`. - Flow B strict fence (stateless API ingress with `RowMutation.commit_ts=0`): - token MUST match the active catalog schema generation at admission time. - stale/mismatched token MUST fail closed before entry apply with `SF_VERSION_CONFLICT`. - Flow A MVCC fence (CDC ingress with `RowMutation.commit_ts>0`): - token MAY be active or historical. - let `max_entry_commit_ts` be the maximum `RowMutation.commit_ts` in this batch (if batch has no row mutations, only token existence/scope checks apply). - catalog lineage MUST expose a generation invalidation boundary (`ddl_commit_ts` that activated the next generation; active generation uses `+inf` boundary). - admission MUST allow the batch only when `max_entry_commit_ts <= token_boundary_commit_ts` for the presented token. - if `max_entry_commit_ts` crosses token boundary, request MUST fail closed with `SF_VERSION_CONFLICT`. - when this boundary-crossing branch is selected, returned `ErrorStatus.context.ingest_schema_boundary_invalidation.invalidated_at_commit_ts` MUST carry `` so CDC adapters can split/retry deterministically at the exact boundary. - `ErrorStatus.diagnostics` MAY additionally mirror `invalidated_at_commit_ts=` (decimal string of the token-invalidating `ddl_commit_ts`) for compatibility/string-only consumers. - this Flow-A/Flow-B split prevents CDC lag deadlock across DDL while preserving strict active-generation fencing for stateless API writes. - every `MutationEntry` MUST carry `source_seq > 0`. - progress cursor is represented by `StreamCursor` and surfaced through: - request reconnect hint: `MutationBatch.resume_cursor`, - response acknowledged cursor: `MutationAck.success.acked_cursor` (success outcome only). - `MutationBatch.stream_identity.source_partition` MUST be non-empty. - `MutationBatch.stream_identity.source_epoch` MUST be > 0. - when `MutationBatch.resume_cursor` is present: - `resume_cursor.source_partition` MUST equal `stream_identity.source_partition`, - `resume_cursor.source_epoch` MUST equal `stream_identity.source_epoch`, - resume token, when present, MUST be scoped to that cursor and stream identity. - `source_epoch` MUST be monotonic non-decreasing. - within same `source_epoch`, `MutationEntry.source_seq` MUST be strictly increasing. - reconnect or producer failover that resets sequence MUST increment `source_epoch`. - Stateless Gateway Stream Rule (Flow B): for stateless REST ingests (for example Elasticsearch `_bulk`), gateway lacks persistent stream state. - gateway MUST construct a deterministic Flow-B stream identity without using `RequestMeta.idempotency_key` as data-plane stream scope. - `source_partition` MUST be derived from retry-stable idempotency identity: - when source provides client idempotency key (for example Elasticsearch `Idempotency-Key`), gateway MUST use its canonicalized value, - otherwise gateway MUST deterministically derive `batch_id` from admitted payload hash and use that stable value (or a stable derivation of it) as `source_partition`. - `source_epoch` MUST be `1` for stateless Flow-B requests, and `source_seq` MUST be generated as the 1-based array index within the admitted batch. - Ingest Plane deduplication remains batch-key + fingerprint based, and `batch_fingerprint` MUST remain stable across source failover retries (including `source_epoch` bumps). - TiDB federated-table SQL remote-write rule (Flow B): - applies only to collections whose authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`, - TiDB remote-write callers MUST use `RequestMeta.client_kind=CLIENT_KIND_TIDB_WRITE_GATEWAY` and stable logical `RequestMeta.client_id=tidb-write-gateway`, - one admitted federated SQL DML statement maps to one canonical `MutationBatch`, - `MutationBatch.stream_identity.source_partition` MUST be derived from a retry-stable logical SQL statement identity for that remote DML statement rather than from a process-local connection identifier, - `MutationBatch.stream_identity.source_epoch` MUST be `1`, - `MutationEntry.source_seq` MUST be generated as the deterministic 1-based row order within the lowered statement plan, - caller MUST set `MutationBatch.policy.atomicity=ATOMICITY_POLICY_ALL_OR_NOTHING`, - caller MUST set `MutationBatch.policy.continue_on_error=false`, - all lowered `RowMutation.commit_ts` values MUST be `0`, and `Commit Coordinator` injects canonical TSO before persistence/apply ordering, - this route is statement-atomic in the active revision: if any lowered row mutation rejects, the whole admitted statement batch MUST fail with no row-mutation visibility, - explicit multi-statement transactions that write federated tables are unsupported in the active revision and therefore MUST NOT attempt to encode one SQL transaction as one multi-batch remote mutation stream. - violating these invariants is a deterministic state conflict (`SF_VERSION_CONFLICT`) and MUST surface the first violating cursor in ACK. - stream scope remains single-partition by contract for one logical stream lifecycle; scope changes require a new stream chain. Idempotency key: - `ROW_MUTATION` entries: `(namespace_id, collection_id, stream_identity.source_partition, stream_identity.source_epoch, doc_id, commit_ts, mutation_entry.source_seq)`. - `PROGRESS_MARK` / `BARRIER_MARK` entries: `(namespace_id, collection_id, stream_identity.source_partition, stream_identity.source_epoch, entry_kind, mutation_entry.source_seq)`. Entry-level payload equivalence: - same idempotency key + semantically identical mutation payload (`operation` branch + branch payload, `expected_commit_ts`) => replay-safe duplicate. - same idempotency key + different payload => deterministic conflict (`SF_IDEMPOTENCY_CONFLICT`). Batch-level idempotency: - key: `(namespace_id, collection_id, stream_identity.source_partition, batch_id)` - `batch_id` MUST be non-empty. - the canonical proto models `MutationBatch.batch_id` with machine-checkable non-empty validation, so empty-present input is malformed before replay-key evaluation. - `batch_fingerprint` MUST be present and exactly 32 bytes (`HASH_ALGORITHM_BLAKE3` digest width in this revision). - same key + same `batch_fingerprint` => treated as replay; no duplicated visibility (including legal failover retransmit where `source_epoch` is higher). - same key + different `batch_fingerprint` => reject deterministically with `SF_IDEMPOTENCY_CONFLICT`. `batch_fingerprint` canonicalization contract: - hash algorithm is fixed to `HASH_ALGORITHM_BLAKE3` in this revision. - fingerprint input is canonical serialization of admitted mutation business content: - collection scope (`namespace_id`, `collection_id`), - ordered `MutationEntry[]` business payload content (`doc_id`, operation payload branch, `expected_commit_ts`, mark payloads, and deterministic entry order metadata). - serialization MUST be deterministic across languages: - `RowMutation.commit_ts` is runtime concurrency metadata and MUST NOT participate in `batch_fingerprint` hashing (Flow B sends `0`; Commit Coordinator may inject canonical TSO later). - when `RowMutation.operation=upsert`, `UpsertPayload.fields` is a map and therefore structurally guarantees per-row field-key uniqueness, - when `RowMutation.operation=patch`, `PatchPayload.fields` is a map and therefore structurally guarantees per-row field-key uniqueness, - when `RowMutation.operation in {upsert, patch}`, `fields` map keys MUST be canonicalized in byte-ascending order before hashing/signing, - for cross-language hash stability, any float/double fields (including values carried by `AtomicValue` and `FloatVector`) MUST be canonicalized using strictly ordered IEEE 754 integer bit-patterns before hashing (for example Go `math.Float32bits` / `math.Float64bits`), - `-0.0` MUST be normalized to positive `0.0` before IEEE 754 bit-pattern canonicalization, - all NaN payloads MUST be collapsed to one canonical NaN bit-pattern per width before hashing, - default-valued optional fields use explicit canonical defaults, - transport/stream envelope fields (`MutationBatch.stream_identity.source_partition`, `MutationBatch.stream_identity.source_epoch`, `MutationBatch.schema_generation_token`, `meta`, reconnect cursor/token fields) are excluded. #### Batch execution modes and first-error contract `MutationBatch` admission and apply MUST be deterministic: - generation-fence check (`MutationBatch.schema_generation_token`) MUST run before entry-level apply so invalid-generation batches are rejected atomically (Flow B: active-generation strict match; Flow A: commit-ts-bounded historical-token admissibility). - when the selected rejection branch is Flow-A DDL boundary crossing, returned `ErrorStatus.context.ingest_schema_boundary_invalidation.invalidated_at_commit_ts` MUST carry the invalidating boundary commit TS, and `ErrorStatus.diagnostics` MAY mirror `invalidated_at_commit_ts`. - `ROW_MUTATION` entries are evaluated in canonical order (`commit_ts`, `source_partition`, `source_epoch`, `source_seq`) after framing validation. - `PROGRESS_MARK` and `BARRIER_MARK` are evaluated in framing order after all admitted `ROW_MUTATION` entries. - `MutationBatch.policy.atomicity` selects the batch atomicity contract: - `ATOMICITY_POLICY_UNSPECIFIED` uses the route default in this revision. - generic ingest routes default to `ATOMICITY_POLICY_FAIL_FAST_PARTIAL`. - TiDB federated-table remote DML is the explicit stronger route and MUST override to `ATOMICITY_POLICY_ALL_OR_NOTHING`. - mode switch is controlled by `MutationBatch.policy.continue_on_error`: - `ATOMICITY_POLICY_ALL_OR_NOTHING` requires `continue_on_error=false`; `continue_on_error=true` with all-or-nothing atomicity is malformed (`SF_BAD_REQUEST`). - authoritative Flow-A MUST set `MutationBatch.policy.continue_on_error=false`. - degraded/non-authoritative lane may use `continue_on_error=true` only with `ATOMICITY_POLICY_FAIL_FAST_PARTIAL` as apply-progress semantics and must not treat the resulting ACK/checkpoint as completeness/freshness proof. - `ATOMICITY_POLICY_FAIL_FAST_PARTIAL` + `continue_on_error=false`: first non-retryable violation at entry `N` stops the batch; entries after `N` MUST NOT be applied, but earlier accepted rows remain applied. - `ATOMICITY_POLICY_FAIL_FAST_PARTIAL` + `continue_on_error=true`: per-entry admission continues after a row-level rejection; accepted entries still apply in deterministic order. - `ATOMICITY_POLICY_ALL_OR_NOTHING` + `continue_on_error=false`: any rejecting row causes the whole admitted batch to fail with no row-mutation visibility. - these atomicity rules apply to the original admitted `MutationBatch` even when `policy.ingest_mode=INGEST_MODE_BATCH`; internal chunking, staging, or multi-object artifact build MUST NOT weaken batch-level visibility semantics. - under `ATOMICITY_POLICY_ALL_OR_NOTHING`, no row from the original admitted batch may become query-visible before the entire batch has succeeded. - under `ATOMICITY_POLICY_FAIL_FAST_PARTIAL`, any incremental publish MUST remain a canonical accepted prefix only; rows after the first unresolved or rejected canonical position MUST NOT become visible ahead of that decision point. - `MutationAck.success.metrics` accounting MUST be exhaustive: - `accepted_entries + duplicate_entries + rejected_entries + skipped_entries == total entries in admitted batch`. - when `ATOMICITY_POLICY_FAIL_FAST_PARTIAL` and `continue_on_error=false` and first rejection occurs at index `N`, that entry increments `rejected_entries` and every later entry increments `skipped_entries`. - when `continue_on_error=true`, `skipped_entries` MUST be `0`. - when `ATOMICITY_POLICY_ALL_OR_NOTHING` rejects a row-mutation batch, row-level visibility remains empty for the batch and the rejection is reported as a whole-batch failure rather than as a partially applied prefix. - `MutationAck` uses `oneof outcome`: - `outcome=success` carries `replayed_batch`, `metrics`, `acked_cursor`, `applied_watermarks`, optional `rejection_context`, and `item_outcomes` whenever the admitted batch contains `ROW_MUTATION` entries. - `outcome=error` is reserved for stream-level fatal branches (for example auth/rate-limit/session/transport failures) where success payload is not emitted. - when `continue_on_error=true`, batch payload MUST contain only `ROW_MUTATION` entries; `PROGRESS_MARK`/`BARRIER_MARK` are malformed in this mode (`SF_BAD_REQUEST`). - `MutationAck.success.rejection_context` MUST carry first-rejected canonical cursor (`first_rejected_source_epoch`, `first_rejected_source_seq`) and canonical typed `ErrorCode` (`first_rejected_error_code`) when rejections occur; `first_rejected_commit_ts` is present only when the rejected entry is `ROW_MUTATION`. - `MutationAck.success.item_outcomes[]` is the authoritative per-entry result carrier for deterministic response reconstruction: - when the admitted batch contains one or more `ROW_MUTATION` entries, it MUST be populated for every admitted entry in canonical entry order, regardless of `continue_on_error`. - when the admitted batch contains only `PROGRESS_MARK` / `BARRIER_MARK` entries, it MAY be omitted. - this completeness and ordering rule applies to both `INGEST_MODE_STREAMING` and `INGEST_MODE_BATCH`; any batch-lane executor MUST preserve the original admitted `entry_index` / `source_seq` mapping rather than inventing writer-local ACK units or reordered outcome arrays. - mutation-ack outcome coverage invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-INGEST-002`. - server MUST echo both `MutationItemOutcome.entry_index` and `MutationItemOutcome.source_seq` for every populated item outcome: - `entry_index` preserves source-array positional mapping for ecosystem wire-protocol reconstruction. - `source_seq` preserves absolute stream cursor mapping for deterministic sliced-retry/replay workflows. - emitted `source_seq` MUST exactly equal `MutationBatch.entries[entry_index].source_seq`. - authoritative post-apply result kind for successful `ROW_MUTATION` entries: - `UPSERT_INSERTED`: row was absent immediately before apply and now exists because of this mutation. - `UPSERT_REPLACED`: row existed immediately before apply and was replaced by this upsert. - `PATCH_APPLIED`: patch was applied to an existing row. - `DELETE_APPLIED`: delete removed an existing row version. - `DELETE_ALREADY_ABSENT`: delete was admitted but the target row was already absent at apply time. - successful replay of a prior `ROW_MUTATION` MUST preserve the original authoritative post-apply result kind in the replayed outcome. - retry from caller MUST start at first rejected cursor or later; replaying already-acked prefix MUST remain idempotent. - retry contract for `batch_id` with first-error slicing: - re-sending the exact same admitted batch payload (same entries/order) MAY reuse the original `batch_id`; `batch_fingerprint` must remain identical. - when client retries by slicing payload (for example dropping already-acked prefix and starting from `first_rejected_source_seq`), client MUST generate a new `batch_id` for that retry request. - reusing the original `batch_id` with a sliced payload changes `batch_fingerprint` and MUST be rejected as `SF_IDEMPOTENCY_CONFLICT`. #### Row-level optimistic precondition (`expected_commit_ts`) `RowMutation.expected_commit_ts` is an optional optimistic-lock assertion aligned with ES `_seq_no` style guardrails: - when absent, mutation admission follows normal deterministic ordering/idempotency rules without commit-ts precondition. - when present, server MUST read the latest admitted commit marker for `doc_id` and compare it with `expected_commit_ts` before applying mutation. - a physically absent document MUST be treated as logical `latest_commit_ts=0`. - `expected_commit_ts=0` is the canonical create-only assertion (`doc_id` MUST NOT already exist); existing document rows MUST fail this precondition with `SF_VERSION_CONFLICT`. - precondition evaluation MUST use accumulated apply-state visibility within the same admitted batch: if an earlier row mutation in this batch updates the same `doc_id`, later rows in that batch MUST observe that updated commit marker. - mismatch (`latest_commit_ts != expected_commit_ts`) MUST fail closed with `SF_VERSION_CONFLICT`. - conflict details MUST include canonical detail keys `doc_id`, `expected_commit_ts`, and `latest_commit_ts` for deterministic client retry logic. - in `ATOMICITY_POLICY_FAIL_FAST_PARTIAL` + `continue_on_error=false`, first precondition mismatch stops the batch. - in `ATOMICITY_POLICY_ALL_OR_NOTHING`, any precondition mismatch rejects the whole row-mutation batch with no row-mutation visibility. - in `continue_on_error=true` mode, mismatched entries are reported through `MutationAck.success.item_outcomes[]` with `MutationItemOutcome.rejected.code=\"SF_VERSION_CONFLICT\"`, and accepted entries continue deterministically. #### Row commit timestamp ownership (`RowMutation.commit_ts`) - `commit_ts` MUST be populated by Flow A (CDC) clients. - for Flow B gateways targeting Search-Fabric-owned collections (Elasticsearch, Turbopuffer, and TiDB federated-table remote-write gateway), `commit_ts` MUST be sent as `0`. - Ingest Plane `Commit Coordinator` assigns canonical TSO and injects it before persistence/apply ordering. - `batch_fingerprint` computation MUST remain stable across this injection step because `RowMutation.commit_ts` is excluded from fingerprint canonicalization by contract. #### Row operation payload contract (`RowMutation.operation`) - `RowMutation.operation` is a discriminated oneof and MUST set exactly one branch. - row-level schema lineage contract: - `schema_version` MUST be non-zero on every `upsert` / `patch` row. - all `upsert` / `patch` rows inside one admitted batch MUST carry the same `schema_version`. - `delete`, `progress`, and `barrier` entries do not carry row-level `schema_version`. - progress-only / barrier-only batches are fenced only by `schema_generation_token`, not by a row-level `schema_version`. - `operation=upsert`: - payload is `UpsertPayload` (`fields`, `schema_version`). - semantics are full-row replace: - `fields` represents the authoritative post-mutation row state for schema-declared fields carried in this row payload. - schema-declared fields omitted from `fields` are treated as removed/cleared in the resulting row version. - lifecycle guardrail (when `CollectionSettings.lifecycle_policy` is enabled): `UpsertPayload.fields` MUST explicitly include non-null `lifecycle_policy.time_partition_field`; omission is malformed (`SF_SCHEMA_VIOLATION`) because late/future-arrival checks are payload-driven in this revision. - split-brain guardrail for canonical ID source: - Gateway MUST strip the declared `DocIDMapping.source_field` from `UpsertPayload.fields` before canonical emission. - Ingest Plane MUST reject any admitted row where `DocIDMapping.source_field` still appears in `UpsertPayload.fields` (`SF_SCHEMA_VIOLATION`), so payload-shape/fingerprint remains deterministic across gateways/retries. - field-level undeclared handling (`MutationBatch.policy.undeclared_field`) and `FieldData.list_value` homogeneity checks apply to this branch. - `operation=patch`: - payload is `PatchPayload` (`fields`, `schema_version`). - semantics are merge/partial update: - fields absent from `PatchPayload.fields` remain unchanged in storage. - lifecycle guardrail (when `CollectionSettings.lifecycle_policy` is enabled): `PatchPayload.fields` MUST explicitly include `lifecycle_policy.time_partition_field`; omission is malformed (`SF_SCHEMA_VIOLATION`) because late/future-arrival checks are payload-driven in this revision (no read-before-write dependency). - for any schema field type (`scalar` / `vector` / `list` / `object` / `nested`), `FieldData.atomic_value.null_value = NULL_VALUE_NULL` is the canonical explicit-clear marker (set field to null/remove value). - callers MUST NOT encode clear intent through empty vector/list/nested surrogate payloads. - split-brain guardrail for canonical ID source: - Gateway MUST strip the declared `DocIDMapping.source_field` from `PatchPayload.fields` before canonical emission. - Ingest Plane MUST reject any admitted row where `DocIDMapping.source_field` still appears in `PatchPayload.fields` (`SF_SCHEMA_VIOLATION`). - field-level undeclared handling (`MutationBatch.policy.undeclared_field`) and `FieldData.list_value` homogeneity checks apply to this branch. - vector write-dimension contract (`operation in {upsert, patch}`): - when payload carries `FieldData.vector_value` for a schema-declared vector field, write dimensions are evaluated in logical decoded dimensions `Dw` and MUST exactly equal schema dimensions `Dschema`. - decoded `Dw` rules by `VectorValue.data` branch: - `float_vector`: `Dw = len(FloatVector.values)`. - `f16_vector`: `Dw = len(f16_vector_bytes) / 2`; odd byte-length is malformed and MUST fail closed (`SF_BAD_REQUEST`). - `int8_vector`: `Dw = len(int8_vector_bytes)`. - `binary_vector`: `Dw = len(binary_vector_bytes) * 8`. - query-side MRL prefix truncation (`0 < Dq <= Dschema`) is read-path-only and MUST NOT be applied on write admission. - `Dw != Dschema` MUST fail closed as `SF_VECTOR_DIMENSION_MISMATCH`. - `operation=delete`: - payload is `DeletePayload` and MUST NOT carry row field values. - delete semantics are keyed by external `doc_id` in `RowMutation.doc_id`. - absent-target deletes are still admitted unless a stronger optimistic precondition rejects them; authoritative outcome classification is emitted through `MutationAck.success.item_outcomes[]` as `DELETE_APPLIED` or `DELETE_ALREADY_ABSENT`. #### Source-side document identity generation contract - canonical ingest is identity-complete: every admitted `ROW_MUTATION` MUST carry explicit `RowMutation.doc_id`, regardless of collection `DocIDMapping.generation_policy`. - `DOC_ID_GENERATION_POLICY_CALLER_REQUIRED`: - every admitted write path MUST provide source-side identity before canonical mutation construction, - source-side omission is malformed for create/insert-style routes and MUST be rejected before allocator/ingest interaction, - patch/delete/update-style routes also require explicit identity, as usual. - `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`: - this policy is legal only when the authoritative collection descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`, - only create/insert-style routes on explicitly admitted caller profiles MAY use allocator authority, and those profiles plus their allocator ceilings are owned by `CollectionSettings.allocator_admission_policy` (`caller_profile`, `max_plan_items_per_request`, `max_forward_reservation_jump`), - source-side omission MUST be resolved by a pre-ingest `IdAllocatorService.AllocateIds` call after collection resolution and before canonical mutation construction, - caller-supplied create/insert identity on the same collection/caller path MUST also be resolved through that same allocator authority before canonical mutation construction, so allocator-issued and caller-supplied create/insert IDs share one authoritative reservation space, - the follow-on canonical `MutationBatch` MUST carry the signed `allocator_receipt` returned by `AllocateIds`, so later ingest can verify that allocator-backed explicit `RowMutation.doc_id` values came from allocator authority rather than caller injection, - patch/delete/update-style routes still require explicit identity and MUST NOT use allocator-backed omission. - mixed explicit/omitted source rows are admitted only when the owning route contract already allows per-row create/insert semantics: - gateway MUST lower the source batch into one ordered `AllocateIdsRequest.plan.items[]` sequence after collection resolution, - explicit rows lower to `AllocateIdPlanItem.explicit_doc_id`, - omitted rows lower to `AllocateIdPlanItem.issue`, - allocator executes that ordered plan under one replay-governed request lineage and one returned `schema_generation_token`, - realized identities MUST be bound back to source-input order deterministically before canonical mutation emission. - `StreamMutations` and downstream ingest components MUST NOT infer, backfill, or silently allocate missing document identity. #### `FieldData.list_value` homogeneity admission (gateway hard-check) - Ingest Gateway MUST traverse every `RowMutation.operation.{upsert,patch}.fields` value and inspect `FieldData.list_value.values[]` whenever the list branch is used. - if target field schema is established and `FieldSchema.config.capabilities.multi_valued=false`, `FieldData.list_value` is invalid for that field and gateway MUST reject that row with `SF_SCHEMA_VIOLATION`. - mixed atomic element types in one `ListValue` (for example `int64` + `string`) are malformed for write admission; gateway MUST reject that row with `SF_SCHEMA_VIOLATION`. - Presence of `AtomicValue.null_value` elements within a `ListValue` does NOT violate homogeneity rules; gateway/engines MUST treat inner nulls as skipped/absent elements during indexing while preserving the array's primary atomic type. - if field schema is not established yet, the list payload still targets an undeclared field and gateway MUST reject that row with `SF_SCHEMA_VIOLATION` unless the active undeclared-field mode is `IGNORE`. - if field schema is already established and list payload is schema-compatible, empty `ListValue.values[]` is valid and represents a zero-length collection. - in `ATOMICITY_POLICY_FAIL_FAST_PARTIAL` + `continue_on_error=false`, first such row-level violation stops the batch with first-error semantics. - in `ATOMICITY_POLICY_ALL_OR_NOTHING`, any such row-level violation rejects the whole row-mutation batch with no row-mutation visibility. - in `continue_on_error=true` mode, gateway/executor MUST drop only the offending row, report it in `MutationAck.success.item_outcomes[]` with `MutationItemOutcome.rejected.code=\"SF_SCHEMA_VIOLATION\"`, and continue applying other valid rows. #### `FieldData.object_array_value` depth admission (gateway hard-check) - In this revision, write-path object-container payload depth is intentionally capped at one structural level to keep ingress data shape aligned with admitted schema contract. - Gateway MUST traverse every `RowMutation.operation.{upsert,patch}.fields[*].object_array_value.documents[].fields[*]` value. - if any nested child value under that scope selects `FieldData.object_array_value` again (effective depth `> 1`), row admission MUST fail closed with `SF_SCHEMA_VIOLATION`. - this depth guard applies to both `object` and `nested` container fields because both use the same `ObjectArrayValue` carrier in this revision. - in `ATOMICITY_POLICY_FAIL_FAST_PARTIAL` + `continue_on_error=false`, first such violation stops the batch with first-error semantics. - in `ATOMICITY_POLICY_ALL_OR_NOTHING`, any such violation rejects the whole row-mutation batch with no row-mutation visibility. - in `continue_on_error=true` mode, only offending rows are rejected (`MutationItemOutcome.rejected.code=\"SF_SCHEMA_VIOLATION\"`) and other rows continue deterministically. #### Write durability and visibility policies These two fields are the canonical write-envelope gates; compatibility gateways MUST normalize source-specific write options into them before admission. `policy.durability`: - `DURABILITY_POLICY_UNSPECIFIED` is an invalid sentinel and MUST be rejected as `SF_BAD_REQUEST`. - `DURABILITY_POLICY_INGEST_ACK`: - ACK after admission into ingest control path and buffered - durability is not guaranteed yet - `DURABILITY_POLICY_WAL_CHUNK_DURABLE`: - ACK after WAL chunk is durably persisted and anchored - this gate is defined only for WAL-backed streaming execution in the current revision; `INGEST_MODE_BATCH` MUST NOT use it because no separate pre-visibility batch-durable anchor is part of the admitted public contract yet `policy.visibility`: - `VISIBILITY_POLICY_UNSPECIFIED` is an invalid sentinel and MUST be rejected as `SF_BAD_REQUEST`. - `VISIBILITY_POLICY_ASYNC`: - ACK does not wait for visibility - `VISIBILITY_POLICY_WAIT_FOR_VISIBLE`: - ACK after the visibility barrier confirms publish eligibility and `Manifest Publisher` completes manifest CAS publish (durable + query-visible) Effective ACK timing is the stricter of the two gates when both are specified. - If the write-side `wait_for_visible` budget expires before publish completes, the stream-level failure MUST be projected as retryable `SF_BACKEND_UNAVAILABLE` on `MutationAck.error`; `SF_VISIBILITY_TIMEOUT` is reserved for query/readiness visibility waiting only in this revision. Cross-field write-policy rules in the current revision: - `INGEST_MODE_BATCH` is the canonical batch-build lane for admitted `MutationBatch` execution on `StreamMutations`; it is distinct from the snapshot-session lifecycle (`StartSnapshotSync -> UploadSnapshotChunks -> CommitSnapshotSync`). - `INGEST_MODE_BATCH` MUST set `policy.visibility=VISIBILITY_POLICY_WAIT_FOR_VISIBLE`; `VISIBILITY_POLICY_ASYNC` is malformed for batch mode (`SF_BAD_REQUEST`) because batch execution still owes one truthful `MutationAck` with final `item_outcomes[]`, `rejection_context`, and `applied_watermarks`. - `INGEST_MODE_BATCH` MUST set `policy.durability=DURABILITY_POLICY_INGEST_ACK`; `DURABILITY_POLICY_WAL_CHUNK_DURABLE` is malformed with batch mode (`SF_BAD_REQUEST`) until a canonical batch-durable anchor is defined. - batch-lane policy / atomicity invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-INGEST-001`. TiDB federated-table SQL remote-write default: - For statements admitted through the TiDB direct-write gateway against `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` collections, caller MUST set `policy.ingest_mode=INGEST_MODE_STREAMING`, `policy.durability=DURABILITY_POLICY_WAL_CHUNK_DURABLE`, and `policy.visibility=VISIBILITY_POLICY_WAIT_FOR_VISIBLE`. - The same route MUST set `policy.atomicity=ATOMICITY_POLICY_ALL_OR_NOTHING` and `policy.continue_on_error=false`. - This route is statement-scoped remote DML and MUST NOT depend on TiKV-backed delta overlays for read-your-write semantics. - Implementations MAY expose stricter engine-internal defaults, but they MUST NOT weaken this TiDB-facing default contract in the active revision. `policy.ingest_mode`: - `INGEST_MODE_UNSPECIFIED` is an invalid sentinel and MUST be rejected as `SF_BAD_REQUEST`. - `INGEST_MODE_STREAMING`: - routes writes to Stream Writer path and NRT delta store, - optimizes near-real-time visibility, - remains subject to online throttling/backpressure controls (`SF_RATE_LIMITED`). - `INGEST_MODE_BATCH`: - routes writes to Batch Writer path for bulk segment build on already-admitted `MutationBatch` work, - bypasses NRT delta-store pressure path, - data remains query-invisible until manifest commit/visibility publication, - still returns one canonical `MutationAck` per original admitted batch rather than a writer-local background-job handle. - gateways MUST use `ingest_mode` (not ad-hoc boolean hints) to express physical write-lane intent, and MUST materialize an explicit non-zero enum before canonical admission. #### Undeclared-field policy hint `MutationBatch.policy.undeclared_field` controls deterministic handling of `RowMutation.operation.{upsert,patch}.fields` keys that are outside the admitted schema snapshot: - `UNDECLARED_FIELD_POLICY_UNSPECIFIED` defaults to strict behavior (`UNDECLARED_FIELD_POLICY_REJECT`) in this revision. - `UNDECLARED_FIELD_POLICY_REJECT`: first undeclared field rejects the batch (`SF_SCHEMA_VIOLATION`) and applies first-error stop semantics. - `UNDECLARED_FIELD_POLICY_IGNORE`: undeclared fields are dropped before apply; only schema-declared fields are admitted. - the canonical contract does not auto-create schema from data-plane writes in this revision. - routes that need new fields MUST complete explicit control-plane schema mutation before retrying the write path. - gateways MUST NOT quarantine, infer, or replay undeclared-field rows as an implicit side effect of data-plane ingest admission. #### Progress watermark contract Ingest plane ACK responses expose the derived collection frontier after the admitted write/progress scope has been applied: - `MutationAck.success.applied_watermarks MUST carry every modality enabled on the target collection`. - enabled metadata-search collections MUST return `MutationAck.success.applied_watermarks.metadata_search_ts`. - enabled semantic-search collections MUST return `MutationAck.success.applied_watermarks.semantic_search_ts`. - enabled metadata-aux collections MUST return `MutationAck.success.applied_watermarks.metadata_aux_ts`. - untouched enabled modalities remain present on ACK/readiness carriers with their last derived collection frontier; presence is about collection topology, not about whether the latest admitted write touched that modality. - `MutationAck.success.applied_watermarks.safe_ts = min(emitted enabled modality applied ts)`. Request-side progress input is narrower: - `ProgressMark.watermarks` carries only per-modality applied-progress inputs for the modalities advanced by that mark. - omission of one modality field in `ProgressMark.watermarks` means that this progress mark does not advance that modality; it does not imply that the collection lacks that modality. - collections that do not expose a modality MUST omit that modality from `ProgressMark.watermarks`. - callers MUST NOT send `safe_ts` through `ProgressMark`; collection `safe_ts` is derived only by the server from admitted per-modality progress. ### 4.2 Batch Build and Full Snapshot Synchronization Session lifecycle: 1. `StartSnapshotSync` (creates sync session bound to `snapshot_ts`, schema, collection, generation token) 2. `UploadSnapshotChunks` (bidirectional stream; first request frame MUST be `SnapshotUploadInit`, followed by `SnapshotChunk` frames, and terminated by one `SnapshotUploadComplete` EOF frame before commit) 3. `CommitSnapshotSync` (atomic cutover validation) 4. `AbortSnapshotSync` (explicit caller abort; idempotent) 5. `GetSnapshotSyncProgress` (state, chunk progress, lag, errors) Outcome-presence and success-state rule for public response RPCs: - `MutationAck`, `SnapshotChunkAck`, and `StreamQueryFrame` MUST carry exactly one `outcome` branch on every emitted response. Proto-level unset `oneof outcome` is semantically invalid in this API revision. - `ProjectedQueryResponse` is a transport-`OK` normal-result envelope rather than a success/error envelope and therefore MUST carry exactly one `result` branch (`success` or `fallback`) on every emitted `OK` response. Proto-level unset `oneof result` is semantically invalid in this API revision. - `ExecuteQuery`, `GetCollectionStatistics`, `GetReadiness`, `DebugDigestProbe`, `AllocateIds`, `StartSnapshotSync`, `CommitSnapshotSync`, `AbortSnapshotSync`, `GetSnapshotSyncProgress`, `PutCollectionSchema`, `DropCollection`, `GetSchemaSyncStatus`, `GetCollectionSchema`, `ListCollections`, `GetCollectionMetadata`, `PutCollectionMetadata`, `PutNamespaceMetadata`, `ListNamespaces`, and `GetNamespaceMetadata` no longer expose public success/error envelopes: transport `OK` returns their typed success response bodies directly, while failures use non-`OK` gRPC status plus canonical `ErrorStatus`. - public exact-one response-shape invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-RESP-001`. - `ResultFetchSpec.page_window` and `StreamResultFetchSpec.page_window` MUST each carry exactly one admitted branch on every request; proto-level unset `page_window` is semantically invalid and MUST be rejected as `SF_BAD_REQUEST`. - `QuerySuccessPayload` and `StreamQuerySuccessFrame` MUST set exactly one success-state branch (`ok` or `partial`); `partial` is the only admitted degraded/non-exact success branch. - `SnapshotChunkAckSuccessPayload`, `CommitSnapshotSyncSuccessResponse`, `AbortSnapshotSyncSuccessResponse`, `PutCollectionSchemaSuccessResponse`, `DropCollectionSuccessResponse`, `PutCollectionMetadataSuccessResponse`, and `PutNamespaceMetadataSuccessResponse` MUST each set exactly one success-state branch. - `StartSnapshotSyncSuccessResponse` is acceptance-only and therefore MUST NOT carry a redundant status enum. - `SnapshotSyncProgressSuccessResponse.state` MUST NOT carry `SNAPSHOT_SESSION_STATE_UNSPECIFIED`. - `SchemaSyncStatusSuccessResponse.active_schema_update_policy` and `CollectionSchemaDescriptor.active_schema_update_policy` MUST NOT carry `SCHEMA_UPDATE_POLICY_UNSPECIFIED`. - `IndexDescriptor`, `DescriptorCompleteness`, `SchemaSyncStatusSuccessResponse.state`, and `NamespaceDescriptor` MUST set exactly one public state/lifecycle branch. - `NamespaceDescriptor` MUST set exactly one public lifecycle branch (`active` or `quiescing`). - the canonical proto now carries machine-readable `sf_required_oneof` annotations on the public exact-one carriers above and on request/schema/error-visible typed exact-one carriers whose semantics already declare present-but-empty payloads malformed. This annotation-backed inventory includes: - identity/value carriers such as `DocumentID.id_value`, `AtomicValue.value`, `VectorValue.data`, `FieldData.value`, and `MetadataValue.value` - query/retrieval carriers such as `PredicateNode.node`, `MatchThreshold.threshold_type`, `RetrievalNode.clause`, `SortKey.target`, `MetricAggregation.metric_type`, `AggregationSortKey.target`, `DateHistogramAggregation.interval`, `CompositeSource.strategy`, `TopHitsVectorProjectionIntent.intent`, `AggregationRequest.aggregation_type`, and `BucketAggregation.strategy` - result/state carriers such as `Bucket.key_type`, `BucketResult.result_shape`, `TopHitsResult.result_type`, `AggregationResponse.result`, and `FieldStatistics.stats_type` - ingest/snapshot carriers such as `RowMutation.operation`, `MutationEntry.entry`, `MutationItemOutcome.outcome`, `MutationAck.outcome`, `SnapshotUploadRequest.frame`, and `SnapshotChunkAck.outcome` - error/schema/helper carriers such as `QueryVisibilityTimeoutContext.mode`, `QueryCatalogUnavailableContext.reason`, `FieldConfig.field_definition`, `AttributeProjectionSpec.intent`, `TrackTotalHitsSpec.tracking`, `RescoreSpec.strategy`, `SimilarityModel.model`, `EncryptionDescriptor.mode`, `CollectionLookupKey.target`, `NamespaceLookupKey.target`, and `AllocateIdPlanItem.item_kind` Plain proto3 code generation still leaves unset branches wire-representable, so implementations MUST run annotation-driven validation (or equivalent generated validators) before business logic, and conformance tests MUST keep those annotations pinned. `StartSnapshotSync` idempotency contract: - `RequestMeta.idempotency_key` MUST be non-empty for retry-safe mutation semantics; empty idempotency key is malformed (`SF_BAD_REQUEST`). - idempotency key is `(namespace_id, collection_id, client_session_token)`. - immutable admitted session-binder tuple is `(collection, snapshot_ts, schema_version, schema_generation_token)`. - `snapshot_ts` MUST be > 0. - `schema_version` MUST be > 0. - `schema_generation_token` MUST be non-empty and <= 64 Unicode scalar values. - `client_session_token` MUST be non-empty; empty token is malformed (`SF_BAD_REQUEST`). - same idempotency key + same admitted session parameters (`snapshot_ts`, `schema_version`, `schema_generation_token`) MUST return the original `sync_session_id` (replay-safe). - same idempotency key + different admitted session parameters MUST be rejected (`SF_IDEMPOTENCY_CONFLICT`). - if previous session under same token is terminal (`aborted`/`expired`/`failed`) and caller needs a fresh session, caller MUST use a new `client_session_token`. - `sync_session_id` is immutably bound to the admitted `CollectionRef`; all follow-up session requests MUST present the same collection binding. - snapshot session binder / resume invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-SNAPSHOT-001`. Snapshot session state machine (normative): - `preparing -> streaming -> committing -> committed` - terminal states: `aborted`, `expired`, `failed` - allowed transitions: - `preparing -> aborted|expired|failed` - `streaming -> committing|aborted|expired|failed` - `committing -> committed|failed` - any request that implies an invalid transition MUST be rejected with `SF_VERSION_CONFLICT`. Atomicity model: - Upload path is incremental and retryable. - Upload completion uses explicit EOF (`SnapshotUploadComplete`) and MUST be acknowledged before commit admission. - Visibility cutover is atomic at commit stage only. - Repeated commit with same session root hash is idempotent. - root hash algorithm is explicit and versioned (`HASH_ALGORITHM_BLAKE3` in this revision). Lease and orphan-session model: - every snapshot session has server-issued lease TTL. - `UploadSnapshotChunks` and `CommitSnapshotSync` refresh lease on successful progress. - lease expiry moves session state to `expired`; further upload/commit requests fail deterministically with state-conflict semantics. - `AbortSnapshotSync` is idempotent and MUST release temporary resources even when retried. #### Snapshot upload stream-init and resume contract `UploadSnapshotChunks` request stream framing is normative: - first request frame MUST be `SnapshotUploadInit`. - after a valid `SnapshotUploadInit`, request stream MAY carry `SnapshotChunk` frames and MUST finish with exactly one `SnapshotUploadComplete` frame; additional init frames are malformed (`SF_BAD_REQUEST`). - `SnapshotUploadInit` MUST include: - `RequestMeta` for caller identity/replay governance on stream admission, - `CollectionRef` (for execution-node session-binding verification), - `sync_session_id`, - optional `resume_token` and optional `resume_from_chunk_seq` for reconnect. - when `resume_from_chunk_seq` is present, it MUST be `>= 1`. - for fresh upload (first stream of a session), `resume_token` SHOULD be empty and `resume_from_chunk_seq` SHOULD be omitted or set to `1`. - for reconnect, caller SHOULD pass previous `SnapshotChunkAck.success.upload_resume.token` and set `resume_from_chunk_seq = SnapshotChunkAck.success.next_expected_seq`. - stream-attempt lower bound is fixed by init value: - define `attempt_resume_floor = resume_from_chunk_seq` when present, else `1`. - once init is accepted for one stream attempt, any subsequent `SnapshotChunk.chunk_seq < attempt_resume_floor` is malformed (`SF_BAD_REQUEST`) within that same attempt. - replaying older chunks requires starting a new stream attempt with a lower `resume_from_chunk_seq` (subject to token/session validation), not sending backward chunk indices mid-stream. - if `SnapshotChunk.sync_session_id` is populated, it MUST equal init `sync_session_id`; mismatch is malformed (`SF_BAD_REQUEST`). - `SnapshotUploadComplete.total_chunks_sent` is the explicit client EOF declaration and MUST equal the final accepted chunk sequence count for this stream session. - once `SnapshotUploadComplete` is sent, no further `SnapshotChunk` frames are allowed on that stream; post-complete chunk frames are malformed (`SF_BAD_REQUEST`). - server MUST validate `resume_token` with Section `3.6` and verify it is scoped to init `sync_session_id`. - server MUST resolve init `sync_session_id` to bound collection and compare with `SnapshotUploadInit.collection`; mismatch MUST fail closed (`SF_FORBIDDEN` or `SF_BAD_REQUEST` by precedence). - server MUST acknowledge accepted chunk progress with `next_expected_seq` and new `upload_resume.token`. - on accepted `SnapshotUploadComplete`, server MUST return the `SnapshotChunkAck.success.completed` branch; callers MUST treat this ACK as commit-admission prerequisite. ### 4.3 Ingest Plane Failure and Retry Rules - Duplicate chunk with same checksum: accepted as replay. - Duplicate chunk with different checksum: reject with `SF_IDEMPOTENCY_CONFLICT`. - Missing chunk sequence at commit time: reject with `SF_VERSION_CONFLICT`. - Chunk checksum algorithm is fixed (`CRC32C`) for interoperability; mismatched algorithm is rejected with `SF_BAD_REQUEST`. - Snapshot root hash algorithm mismatch at commit time: reject with `SF_BAD_REQUEST`. - Snapshot root hash mismatch (same algorithm, different digest) at commit time: reject with `SF_VERSION_CONFLICT`. - replay on same batch-idempotency key (`namespace_id`, `collection_id`, `source_partition`, `batch_id`) with different `batch_fingerprint`: reject with `SF_IDEMPOTENCY_CONFLICT`. - `StartSnapshotSync` replay with same `client_session_token` but different session parameters: reject with `SF_IDEMPOTENCY_CONFLICT`. - snapshot upload/commit on `aborted`/`expired` session: reject with `SF_VERSION_CONFLICT`. - upload stream first frame is not `SnapshotUploadInit`, or stream contains duplicated init frame after upload has started: reject with `SF_BAD_REQUEST`. - upload stream with duplicate `SnapshotUploadComplete`, post-complete chunk frames, or `total_chunks_sent` mismatch against acknowledged progress: reject with `SF_BAD_REQUEST` or `SF_VERSION_CONFLICT` by precedence. - `CommitSnapshotSync` before accepted upload EOF (`SnapshotUploadComplete` -> `SnapshotChunkAck.success.completed`) MUST fail closed (`SF_VERSION_CONFLICT`). - upload reconnect with an expired `resume_token` MUST fail closed as `SF_INVALID_QUERY`; caller MUST start a fresh `StartSnapshotSync` session because empty-token resume is admitted only on an unused session in this revision. - upload reconnect uses `resume_token` with principal/namespace mismatch: reject with `SF_FORBIDDEN`; if principal/namespace match but token-bound session/sequence payload mismatches, reject with `SF_IDEMPOTENCY_CONFLICT`. - `CommitSnapshotSync` / `AbortSnapshotSync` / `GetSnapshotSyncProgress` collection-binding mismatch (`sync_session_id` bound collection != request `CollectionRef`) MUST fail closed (`SF_FORBIDDEN` or `SF_BAD_REQUEST` by precedence). - Backpressure signal: - stream-level transient errors use retryable `ErrorStatus`. - caller retries from acknowledged sequence. - admission overload MUST use `SF_RATE_LIMITED` with retry hints in `ErrorStatus.diagnostics`. ### 4.4 Catalog-Authoritative Truth-Mode Validation Catalog-authoritative truth-mode rule: - query requests do not carry `collection_truth_mode`. - data-plane mutation/snapshot requests do not carry `collection_truth_mode`. - control-plane `PutCollectionSchema.definition.collection_truth_mode` is the authoritative public truth-mode declaration. Search Fabric MUST validate admitted requests against control-plane authoritative state before admission: - request path or caller kind incompatible with the resolved collection's authoritative `collection_truth_mode` -> reject (`SF_OWNERSHIP_CONFLICT`). - unknown collection/index descriptor at request time -> reject with: - `SF_RESOURCE_NOT_FOUND` when the target collection/index does not exist or is tombstoned / fully GC-released, - `SF_CATALOG_UNAVAILABLE` when the target collection exists but the authoritative descriptor state needed for admission is temporarily unavailable or still incomplete. - stale schema version in mutation/snapshot paths -> reject (`SF_VERSION_CONFLICT`) with retry guidance. ### 4.5 Stream Resume and Replay Determinism `StreamMutations` MUST support reconnect without ambiguous replay: - ACK cursor is `MutationAck.success.acked_cursor` (`commit_ts`, `source_partition`, `source_epoch`, `source_seq`), not only `commit_ts`. - server returns `success.acked_cursor.resume.token` in every success `MutationAck`. - every request batch MUST carry `MutationBatch.stream_identity`; reconnect batches MUST keep the same stream identity over one logical stream lifecycle. - client MAY reconnect using `MutationBatch.resume_cursor`; when token is present, server validates token scope (`namespace_id`, `collection_id`, `source_partition`, stream epoch). - stream scope (`source_partition`, `source_epoch`) MUST remain stable for one logical stream lifecycle; scope changes require establishing a new stream and resume chain. - token validation MUST include TTL and principal binding checks (Section `3.6`). - reconnect with non-zero `MutationBatch.resume_cursor` MUST carry non-empty `resume_cursor.resume.token`; cursor-only reconnect is malformed (`SF_BAD_REQUEST`). - when token is present, server MUST verify reconnect cursor tuple and `MutationBatch.stream_identity` exactly match token-embedded scope/cursor; mismatch is deterministic conflict (`SF_IDEMPOTENCY_CONFLICT`). Deterministic replay rules: - if reconnect cursor is older than retained dedup window, reject with retryable `SF_BACKEND_UNAVAILABLE` and include restart-checkpoint detail keys `restart_commit_ts`, `restart_source_partition`, `restart_source_epoch`, and `restart_source_seq`. - if provided token principal/namespace scope mismatches current stream/collection, reject with non-retryable `SF_FORBIDDEN`. - if provided token cursor/session payload mismatches current stream binding, reject with non-retryable `SF_IDEMPOTENCY_CONFLICT`. - if provided token is expired, reject with non-retryable `SF_INVALID_QUERY` and include the same restart-checkpoint detail keys `restart_commit_ts`, `restart_source_partition`, `restart_source_epoch`, and `restart_source_seq`. - same-producer retransmit MUST preserve `stream_identity`. - failover retransmit MUST preserve `source_partition`, `batch_id`, and `batch_fingerprint`, but increment `source_epoch`. ### 4.6 Flow Control, Quota, and Static Policy Contract Ingest plane admission MUST be bounded and discoverable: - service-static policy defines authoritative constraints: - limits (`max_batch_bytes`, `max_entries_per_batch`, `max_inflight_batches_per_stream`, `max_chunk_bytes`, `max_inflight_snapshot_chunks`) - governance (`default_mutation_resume_token_ttl_ms`, `default_snapshot_upload_resume_token_ttl_ms`, `mutation_idempotency_retention_ms`, `snapshot_resume_retention_ms`, `token_signature_algorithm`) - snapshot formats (`supported_payload_encodings`, `supported_compression_codecs`) - exceeding hard limits MUST fail deterministically (`SF_PAYLOAD_TOO_LARGE` or `SF_TOO_MANY_OPERATIONS`). - for operation-count rejection branches (`SF_TOO_MANY_OPERATIONS`, for example `max_entries_per_batch`), error details MUST include: - `limit_scope` (for example `max_entries_per_batch`, `max_page_depth`, or `max_predicate_ast_nodes`), - `configured_limit`, - `requested_value`. - update-constraint classification (to prevent `_update_by_query` / `patch_by_filter` storms): - Class A: explicit ID-list updates (`patch_rows`, `patch_columns`, ES bulk update-style explicit ID lists): - MUST reuse existing `max_entries_per_batch` limit (no new quota concept). - if explicit ID count exceeds the limit, gateway MUST fail closed at JSON parse/lowering stage with `400 SF_TOO_MANY_OPERATIONS`. - error details MUST use: `limit_scope=max_entries_per_batch`, `configured_limit`, `requested_value=`. - Class B: filter-based updates (`_update_by_query`, `patch_by_filter`): - MUST run deterministic two-phase execution with existing quota (no new config): - Phase 1 (probe): lower to canonical `ExecuteQuery`, set `probe_limit = min(caller_requested_limit, max_entries_per_batch, effective_max_result_window) + 1` (where `caller_requested_limit` defaults to `+inf` when source request does not provide an explicit selection size bound, and `effective_max_result_window` is the admitted resolved effective window bound after collection/default resolution), use `fetch_spec.limit = probe_limit`, and force exact hit counting (`track_total_hits.enabled=true`). - Phase 2 (check): if probe returns `< probe_limit` hits, gateway MAY continue to generate canonical `MutationBatch`. - Phase 3 (fail-closed): if probe returns `== probe_limit`, gateway MUST stop without fetching remaining hits and reject with `400 SF_TOO_MANY_OPERATIONS`. - when this clamp is hit, error details MUST use the dedicated selector-materialization clamp branch: - `limit_scope=selector_materialization_limit`, - `configured_limit=` (the actual admitted cutoff after applying `caller_requested_limit`, `max_entries_per_batch`, and `effective_max_result_window`), - `requested_value=`, - `effective_constraint_source` (`max_entries_per_batch` or `max_result_window` only; it identifies the smallest owner-managed internal bound participating in the clamp and MUST NOT mint a third value for `caller_requested_limit` even when the caller cap is smaller than both internal bounds). - transient ingest/snapshot pressure MUST fail with retryable `SF_RATE_LIMITED` and include: - `retry_after_ms` - `throttled_dimension` (`ingest_bytes`, `ingest_entries`, `snapshot_bytes`, `snapshot_chunks`) - `server_limit` - `MutationAck.success` and `SnapshotChunkAck.success` MUST return token-expiry metadata for deterministic reconnect behavior. - reconnect and replay planning MUST treat service-advertised retention windows as authoritative; callers SHOULD restart from full snapshot/build when retained windows are exceeded. ### 4.7 Deterministic Data-Plane Admission Order Before mutating state, Ingest Plane MUST evaluate checks in this order: 1. access scope (`namespace_id`/`collection_id` authorization). 2. request-level network dedupe (`RequestMeta.idempotency_key`) before downstream replay/ordering checks. 3. token/session validity (token signature/scope/TTL; snapshot session state transition validity). 4. hard limits and static policy compatibility. 5. catalog ownership/schema admissibility. 6. ordering/idempotency invariants (`source_partition/source_epoch/source_seq`, entry framing rules, batch-idempotency key + fingerprint). 7. apply and ACK. If multiple violations exist, this order defines evaluation flow only. Final emitted code MUST be selected strictly by `docs/specs/core/error-codes.md` Section `2.1` (canonical precedence arbitration). ### 4.8 Snapshot Chunk Encoding and Checksum Scope To avoid cross-implementation drift and maximize zero-copy deserialization performance, snapshot chunk framing is fixed: - payload encoding in this revision is `PAYLOAD_ENCODING_ROW_MUTATIONS_PROTO`. - Length-Delimited Framing Contract: when `payload_encoding` is `PAYLOAD_ENCODING_ROW_MUTATIONS_PROTO`, the uncompressed bytes within `SnapshotChunk.payload` MUST be encoded as a continuous stream of length-delimited `RowMutation` protobuf messages. - O(1) Pre-allocation Contract: `SnapshotChunk.payload_row_count` MUST exactly match the number of length-delimited `RowMutation` messages contained in the payload. This is mandatory so execution nodes can perform O(1) memory-arena pre-allocation before entering streaming deserialization. - allowed compression codecs are explicit enum values; default is `COMPRESSION_CODEC_NONE`. - `chunk_checksum` is always calculated over transmitted bytes (`payload`) after compression, using the declared `checksum_algorithm`. - root hash is calculated over canonical sequence of acknowledged chunk checksums (`chunk_seq` ascending), not over decoded row objects. - `SnapshotChunk.chunk_checksum` textual form MUST be lowercase hex without prefix; for `CHECKSUM_ALGORITHM_CRC32C`, width MUST be exactly 8 hex chars. - `CommitSnapshotSyncRequest.expected_root_hash` textual form MUST be lowercase hex without prefix; for `HASH_ALGORITHM_BLAKE3`, width MUST be exactly 64 hex chars. - invalid checksum/root-hash format or width MUST be rejected as malformed (`SF_BAD_REQUEST`). - changing encoding/compression/hash behavior requires coordinated service rollout before traffic cutover. ### 4.9 Snapshot Upload Determinism and Reconnect Rules - chunk sequencing is strictly monotonic within one `sync_session_id` (`chunk_seq` starts at `1` and increases by `1`). - server `SnapshotChunkAck.success.next_expected_seq` is authoritative for reconnect. - reconnect stream MUST begin with `SnapshotUploadInit` and provide either: - valid `resume_token` whose embedded sequence equals present `resume_from_chunk_seq`, or - empty token only when restarting from omitted/`1` `resume_from_chunk_seq` on an unused session. - if the presented `resume_token` is expired, server MUST reject with `SF_INVALID_QUERY`; caller MUST start a fresh `StartSnapshotSync` session because empty-token resume is admitted only on an unused session in this revision. - `SnapshotUploadComplete` is the only accepted EOF signal for upload stream termination; clients MUST send it exactly once before commit. - `SnapshotUploadComplete.total_chunks_sent` MUST equal `SnapshotChunkAck.success.next_expected_seq - 1` at completion point. - `CommitSnapshotSyncRequest.expected_chunks` MUST equal the accepted `SnapshotUploadComplete.total_chunks_sent`; mismatch is deterministic conflict (`SF_VERSION_CONFLICT`). - within one accepted stream attempt (`attempt_resume_floor` fixed by init), any `chunk_seq < attempt_resume_floor` MUST be rejected as malformed (`SF_BAD_REQUEST`). - for `attempt_resume_floor <= chunk_seq < next_expected_seq`, server MAY treat frame as replay only when checksum/payload equivalence holds; otherwise reject with `SF_IDEMPOTENCY_CONFLICT`. - if client sends `chunk_seq > next_expected_seq`, reject with `SF_VERSION_CONFLICT`. - once completion is acknowledged (`SnapshotChunkAck.success.completed`), further upload frames for that stream attempt MUST be rejected with state-conflict semantics. - callers MUST NOT issue parallel upload streams for one `sync_session_id`; concurrent stream admission is rejected with `SF_VERSION_CONFLICT`. ## 5. Query Plane API Design ### 5.1 Unified Query Request/Response `QueryRequest` is organized into four deterministic pillars: - routing/fencing (`meta`, `collection`, `expected_schema_generation_token`) - storage/consistency baseline (`consistency`) - retrieval/ranking core (`search_payload`) - fetch/output control (`fetch_spec`) Schema-lineage admission rule for query serving: - control-plane active schema pointer and query-serving schema generation are intentionally not identical during schema convergence. - `GetCollectionSchema` continues to expose the authoritative active generation for control-plane readback. - query execution MUST resolve against the latest `READY` serving generation, not merely the latest active generation when that generation is still `APPLYING`. - if the collection is active but no schema generation has been admitted yet, query execution MUST fail closed as `SF_CATALOG_UNAVAILABLE` with canonical details `catalog_unavailable_reason=schema_bootstrap` and `schema_bootstrap_state=no_active_schema_generation`. - if an active schema lineage exists but no `READY` serving generation exists yet, query execution MUST fail closed as `SF_CATALOG_UNAVAILABLE` with canonical details `catalog_unavailable_reason=schema_convergence` and `schema_convergence_state=no_ready_serving_generation`. - if `expected_schema_generation_token` is present, it MUST match the resolved serving generation token for the request; naming a newer active-but-not-`READY` generation is a deterministic `SF_VERSION_CONFLICT`. `ReadConsistencySpec` is intentionally read-only: - query requests MUST NOT carry write-topology hints (no `requested_write_mode`-style field in this revision). - consistency admissibility is determined only by read semantics (`read_ts`, `mode`, `max_staleness_ms`) and catalog policy. - whole-message omission of `QueryRequest.consistency` is malformed; callers MUST send an explicit canonical `ReadConsistencySpec`. - whole-message omission of `QueryRequest.fetch_spec` is malformed; callers MUST send an explicit canonical fetch/output contract rather than relying on a hidden first-page default. Upstream callers emitting canonical `QueryRequest` payloads include trusted relational query callers (via `RelationalQueryService`), Elasticsearch Query Gateway, and Turbopuffer Query Gateway. All must emit canonical requests with identical semantics for equivalent queries. `RelationalQueryService` is a wrapper-only transport for trusted relational routes: projected-query uses `ProjectedQueryRequest`, primary-query uses `PrimaryQueryRequest`, but the inner query still uses canonical `QueryRequest` semantics and remains the sole input to canonical `canonical_query_digest` derivation. Execution cooperation contract for canonical queries (`search_payload.global_filter` + optional `search_payload.root_retrieval`): - `search_payload.global_filter` is the global hard filter constraint and MUST apply to all candidate sets before final ranking output. - when `search_payload.global_filter` is absent/unset, Query Plane MUST treat it as `MatchAll` (no hard exclusions). - `search_payload.root_retrieval` is optional: - when present, it is the single canonical retrieval tree root. - when absent, request is filter-only / aggregation-only over `global_filter` (or `MatchAll` when both are absent) and score defaults to zero/omitted semantics. - canonical field-reference grammar (applies to `TextClause.fields[*]`, `TextMatchPredicate.fields[*]`, `VectorClause.field`, predicate fields, aggregation fields, and sort fields): - top-level field reference: `field_name`, - structured container child reference: `container_a.container_b.leaf_field` where intermediate segments MUST resolve through `object`/`nested` container `fields[]`, - multi-field subfield reference: append one final subfield segment to a resolved leaf field path (`leaf_field.subfield_name` or `container.leaf_field.subfield_name`), - `FieldSchema.name` and `FieldSubfieldSpec.name` MUST NOT contain `.`, so `.` is reserved as path separator for container traversal and optional final subfield selection, - deterministic resolution rule: engine MUST first resolve the longest valid field path through declared container topology, then resolve optional final subfield segment against the resolved leaf's `subfields[]`, - unknown/ambiguous field references are malformed and MUST fail closed (`SF_BAD_REQUEST`). - `TextClause` contract: - each `TextClause` MUST set non-empty `clause_id`, non-empty `fields[]`, and non-empty `query`. - `TextClause.fields[]` entries MUST be unique after gateway normalization; for source ecosystems where field order is not semantically meaningful (for example MySQL `MATCH(col_list)` and Elasticsearch `multi_match`), gateway MUST canonicalize to byte-ascending order before emission. - one `TextClause` is one logical cross-field retrieval unit: engines MUST evaluate token matching and term statistics over the declared `fields[]` set as one clause, not as an implicit post-fusion of unrelated field queries. - gateways MUST use `TextClause.fields[]` directly for cross-field source semantics (notably MySQL `MATCH(col_list)` and Elasticsearch `multi_match` branches whose scoring semantics are explicitly cross-field). - `TextMatchPredicate` contract: - `TextMatchPredicate` is the non-scoring text-filter carrier used by predicate trees (`search_payload.global_filter`, nested `child_filter`, `LogicalNot`, `LogicalAnd`, `LogicalOr`). - each `TextMatchPredicate` MUST set non-empty `fields[]` and non-empty `query`. - `TextMatchPredicate.fields[]` obey the same field-resolution and normalization rules as `TextClause.fields[]`. - `TextMatchPredicate` reuses the same `mode_override` branches as `TextClause`, but it has no `clause_id` and no `similarity_override` because predicate-tree evaluation does not participate in score fusion. - `TextClause.mode_override` is a oneof branch: - unset branch => follow schema `default_mode`, - `natural_language` branch => force natural-language mode with optional query-expansion/spell-check options, - `boolean_match` branch => force boolean mode, - `plain_match` branch => force plain-match mode and optionally set tokenizer-term combination operator through `PlainMatchOptions.operator` (`TEXT_MATCH_OPERATOR_OR` / `TEXT_MATCH_OPERATOR_AND`; unset defaults to `OR`). - `PlainMatchOptions.minimum_should_match` is optional string-typed threshold intent aligned with Elasticsearch grammar: - accepted forms include integer counts (`\"2\"`) and percentages (`\"75%\"`), - when present, gateway/query admission MUST validate syntax and lower deterministic token-threshold semantics into canonical query execution, - malformed expressions MUST fail closed (`SF_BAD_REQUEST`). - `WITH QUERY EXPANSION` lowering MUST use `TextClause.mode_override.natural_language.query_expansion=true`; query expansion is invalid outside natural-language branch. - `TextClause.similarity_override`, when present, overrides only this text clause scoring baseline (for example BM25 `k1`/`b`) and MUST NOT mutate stored schema defaults. - when `TextClause.similarity_override` message is present, `SimilarityModel.model` oneof MUST set exactly one branch; empty similarity payload is malformed (`SF_INVALID_ARGUMENT`). - each `VectorClause` branch may use clause-level `local_filter` and a typed `VectorClause.vector`; global hard filter remains authoritative at final materialization. - each `VectorClause.clause_id` MUST be non-empty and unique within the request. - `VectorClause.clause_topk` controls clause-local candidate generation cap before fusion: - when `clause_topk > 0`, engine MUST use the provided clause-local cap. - when `clause_topk = 0` or field is absent on wire, engine MUST deterministically fallback to `search_payload.global_candidate_topk`. - if both `clause_topk` and `search_payload.global_candidate_topk` resolve to `0`/absent, engine MUST deterministically fallback to `fetch_spec.limit`. - if the above terminal fallback still resolves to `0`, vector retrieval branch is bypassed (no ANN candidate generation for that clause). - `VectorClause.vector` MUST set exactly one `VectorValue.data` branch: - `f16_vector` => compact half-precision payload (`IEEE 754 binary16`, little-endian byte stream), - `float_vector` => exact-float payload, - `int8_vector` => quantized-int8 payload, - `binary_vector` => quantized-binary payload. - `VectorClause.vector` branch MUST be schema-compatible with target `FieldSchema.config.field_definition.vector.element_type`: - `f16_vector` requires `VECTOR_ELEMENT_TYPE_F16`, - `float_vector` requires `VECTOR_ELEMENT_TYPE_F32`, - `int8_vector` requires `VECTOR_ELEMENT_TYPE_INT8`, - `binary_vector` requires `VECTOR_ELEMENT_TYPE_BINARY`, - mismatch is malformed and MUST fail closed (`SF_SCHEMA_VIOLATION`). - query-vector dimension contract (MRL prefix-compatible retrieval): - let `Dschema` be schema-declared vector dimensions for target field, and `Dq` be query vector dimensions decoded from `VectorClause.vector`. - decoded `Dq` rules by `VectorValue.data` branch: - `float_vector`: `Dq = len(FloatVector.values)`. - `f16_vector`: `Dq = len(f16_vector_bytes) / 2`; odd byte-length is malformed and MUST fail closed (`SF_BAD_REQUEST`). - `int8_vector`: `Dq = len(int8_vector_bytes)`. - `binary_vector`: `Dq = len(binary_vector_bytes) * 8`. - admission MUST enforce `0 < Dq <= Dschema`. - `Dq < Dschema` is valid query truncation; engine MUST execute prefix-aligned retrieval by using stored-vector prefix of length `Dq` during distance computation and ANN traversal. - `Dq > Dschema` MUST fail closed as `SF_VECTOR_DIMENSION_MISMATCH`. - `Dq = 0` (empty vector) is malformed and MUST fail closed as `SF_BAD_REQUEST`. - empty `VectorValue.data` (no branch set) is a present-but-empty typed carrier and MUST fail closed (`SF_INVALID_ARGUMENT`). - `vector_clause.filter_strategy_hint` is an execution hint: - `VECTOR_FILTER_STRATEGY_AUTOMATIC` is the recommended default and lets the engine select strategy by statistics/cost. - `POST_FILTER`, `PRE_FILTER`, `ITERATIVE_FILTER`, and `FLAT_SEARCH` express explicit strategy preference, but MUST NOT weaken correctness constraints. - opaque per-query vector option bags are not part of this contract; vector retrieval tuning MUST use typed `VectorClause` fields and/or control-plane `FieldSchema.config.field_definition.vector.search_tuning`. - guardrail: `vector_clause.local_filter` is optimization-only for candidate pruning and MUST NOT relax global eligibility defined by `search_payload.global_filter`. - guardrail: before physical ANN traversal for top-level vector clauses, Query Broker planner MUST deterministically conjunct `search_payload.global_filter` with `vector_clause.local_filter` (or dispatch an equivalent joint-evaluation instruction to Reader nodes) so candidate generation honors the combined hard-filter semantics. - guardrail: before physical ANN traversal for vector clauses inside `NestedQueryPayload`, planner MUST deterministically conjunct nested context `child_filter` with `vector_clause.local_filter` for nested-candidate generation. - guardrail: engine MAY choose different physical plans for the conjuncted filter (masked ANN, bounded brute-force over PK set, or fallback), but final output MUST be equivalent to enforcing the conjunction as the authoritative eligibility constraint. - guardrail: before response emission and pagination token continuation, Query Plane MUST re-check `search_payload.global_filter` on materialized candidates after fusion/dedup. - `ScoringScalarClause` is a scalar predicate leaf inside the retrieval tree. - each `ScoringScalarClause` MUST set non-empty `clause_id` and non-empty `predicate`. - request-global `clause_id` uniqueness spans every leaf clause anywhere in the retrieval tree (`TextClause` / `VectorClause` / `ScoringScalarClause` / `NestedClause`), including nested query subtrees. - `RetrievalNode` contract: - each node MUST set exactly one `clause` branch (`text` / `vector` / `scalar` / `nested` / `group`). - `boost` defaults deterministically to `1.0` when omitted; present values MUST be finite and `>= 0`. - `MatchThreshold` contract: - when `MatchThreshold` message is present, exactly one threshold branch MUST be set: - `absolute_count` is an explicit minimum matched-should-clause count. - `percentage_ratio` is a ratio in `[0.0, 1.0]` evaluated over `ShouldMatchSpec.clauses[]`. - when `percentage_ratio` is present, engine MUST deterministically compute the required matched-clause count as `ceil(percentage_ratio * len(ShouldMatchSpec.clauses[]))`. - `ShouldMatchSpec` contract: - `clauses[]` MUST be non-empty and is machine-checked by the canonical protobuf contract. - `threshold` is optional: - when absent and the owning `RetrievalGroup.must_clauses[]` is empty, fallback is equivalent to `absolute_count=1`; - when absent and the owning `RetrievalGroup.must_clauses[]` is non-empty, fallback is equivalent to `absolute_count=0`. - `RetrievalGroup` contract: - at least one of `must_clauses[]` or `should_match` MUST be present; this cross-field cardinality rule is carried in the canonical protobuf message-level invariant inventory. - `must_clauses[]` are scoring-required retrieval branches and MAY be empty only when `should_match` is present; that conditional-emptiness rule is also carried in the canonical protobuf message-level invariant inventory. - `fusion_strategy=GROUP_FUSION_STRATEGY_UNSPECIFIED` deterministically falls back to `GROUP_FUSION_STRATEGY_WEIGHTED_SUM`. - `tie_breaker` is valid only when `fusion_strategy=GROUP_FUSION_STRATEGY_DIS_MAX`; when present it MUST be finite and in `[0.0, 1.0]`. - retrieval-tree boolean semantics: - `must_clauses[]`: every child MUST match and contributes score. - `should_match.clauses[]`: children are optional unless gated by `should_match.threshold` (or fallback threshold when absent); matched children contribute score. - hard filters (`filter` / `must_not`) are not represented in `RetrievalGroup`; they MUST live in predicate trees (`search_payload.global_filter`, nested `child_filter`) through scalar predicates, `TextMatchPredicate`, `NestedPredicate`, and `LogicalNot`. - retrieval-tree scoring semantics: - matched `must_clauses[]` and satisfied `should_match.clauses[]` scores within one `RetrievalGroup` are fused by `fusion_strategy`. - `GROUP_FUSION_STRATEGY_RRF`: Reciprocal Rank Fusion over matched scoring children. - `GROUP_FUSION_STRATEGY_WEIGHTED_SUM`: additive weighted score fusion. - `GROUP_FUSION_STRATEGY_DIS_MAX`: `max(child_score) + tie_breaker * sum(other_child_scores)`, with `tie_breaker` defaulting to `0.0` when omitted. - `ScoringScalarClause` is a constant-score scalar leaf: when its predicate matches, base clause score is `1.0` before node `boost`. - `NestedClause` is a retrieval-tree leaf rooted at one nested path. - each `NestedClause` MUST set non-empty `clause_id`, non-empty `path`, non-empty `query`, and explicit `score_mode` (`AVG` / `MAX` / `MIN` / `SUM` / `NONE`; `UNSPECIFIED` is malformed). - `NestedClause.query` is a local nested context (`NestedQueryPayload`) and MUST NOT carry global query controls (`routing`, top-level `aggregations`, or page cursor semantics). - `NestedQueryPayload.child_filter` is nested-local hard filter; `NestedQueryPayload.root_retrieval` is optional and, when absent, means filter-only nested evaluation. - `NestedClause.inner_hits` is optional extraction declaration; when set, `limit`/`offset`/`track_total_hits` apply per parent hit and MUST be admitted before execution. - `search_payload.dedup_by_doc_id` is the query-level dedupe intent after retrieval-tree evaluation; when `true`, final hit materialization MUST collapse duplicate top-level `doc_id` candidates before pagination and response emission. - `search_payload.routing` is optional physical-pruning input for partitioned collections: - `routing_keys[]` and `partition_filter` MAY be provided together or independently. - `routing_keys[]` MUST be unique and byte-ascending after gateway normalization. - `partition_filter`, when present, MUST reference only declared partition/routing fields; unsupported columns or non-partitioned collections MUST fail closed (`SF_UNSUPPORTED_OPTION`). - `target_tiers[]` is optional lifecycle-tier filter: - `[]` means no tier filter (query all tiers admitted by scope/readiness). - non-empty `target_tiers[]` MUST NOT include `DATA_TIER_UNSPECIFIED`. - gateway MUST dedupe and sort `target_tiers[]` by enum numeric value ascending before generating `canonical_query_digest`. - planner MUST apply routing/partition pruning before opening irrelevant S3 segment/object ranges. - `search_payload.aggregations[]` carries one or more recursive aggregation trees (`AggregationRequest`). - aggregation universe is query-shape dependent: - when `search_payload.root_retrieval` is absent, aggregations are evaluated over the admitted filter-only candidate universe (`search_payload.global_filter`, or `MatchAll` when absent). - when the admitted retrieval shape is metadata-only, aggregations are evaluated over the admitted same-snapshot metadata match set. - when the admitted retrieval shape is semantic-only or vector-dominant, aggregations are result-set facets rather than full-match-set analytics. - semantic/vector-dominant aggregation contract: - the aggregation universe is the broker-final candidate set after same-snapshot lane gathering, visibility filtering, LWW resolution, request-level fusion, query-level deduplication, and optional admitted rerank. - that universe is evaluated before final response pagination (`fetch_spec.first_page.offset`, `fetch_spec.limit`, continuation slicing), so page-window controls MUST NOT change aggregation results once the candidate set is fixed. - semantic/vector-dominant aggregation results are therefore budget-sensitive search-result facets and MUST NOT be interpreted as exact full-corpus or full-match-set analytics. - changing admitted candidate-generation or rerank semantics (for example `global_candidate_topk`, broker overfetch, semantic ANN recall policy, or rerank window) MAY change aggregation outputs because those controls change the admitted result-set universe. - in this revision, the supported semantic/vector-dominant result-set facet subset is limited to scalar metric nodes plus non-composite top-level bucket nodes (`terms` / `histogram` / `date_histogram` / `range`) over top-level scalar fields. - `composite`, `top_hits`, `nested`, and `reverse_nested` are unsupported whenever the admitted aggregation universe is semantic-only or vector-dominant result-set facets, and requests using any of those branches in that universe MUST fail closed as `SF_UNSUPPORTED_QUERY_FEATURE`. - internal broker observability contract for these reducer shapes: - metadata-reader partial aggregation merge and semantic/vector-dominant facet evaluation are both internal `QUERY_BROKER_PHASE_AGGREGATION_REDUCTION` work distinct from candidate-fusion `QUERY_BROKER_PHASE_FUSION`. - `QueryBrokerExecutionPlan.aggregation_subplans[]` remains metadata-reader partial reduction only, while `QueryBrokerExecutionPlan.facet_subplans[]` is reserved for semantic/vector-dominant broker-local facet evaluation. - `QueryBrokerExecutionReceipt` MUST report `fusion_input_candidates` / `fusion_output_candidates` separately from `metadata_aggregation_partial_inputs` / `metadata_aggregation_output_entries` and `facet_reduction_input_candidates` / `facet_reduction_output_entries`. - `AggregationRequest.name` MUST be sibling-unique at each tree level and MUST match regex `^[a-zA-Z0-9_.-]{1,128}$`. - aggregation node semantics: - `metric` nodes compute scalar metrics (`AGGREGATE_FUNCTION_COUNT` / `AGGREGATE_FUNCTION_SUM` / `AGGREGATE_FUNCTION_MIN` / `AGGREGATE_FUNCTION_MAX` / `AGGREGATE_FUNCTION_AVG`) or `cardinality`; metric nodes are leaf-only. - metric-branch exclusivity: `MetricAggregation.metric_type` is a oneof; exactly one branch MUST be set (`builtin_function` or `cardinality`). - `MetricAggregation.field` is the sole authoritative target field carrier for all metric branches. - `MetricAggregation` field-source rules: - `builtin_function` branch: - `COUNT` with empty `field` => document-count semantics. - `COUNT` with non-empty `field` => value-count semantics (count rows where target field is present and non-null). - `SUM` / `MIN` / `MAX` / `AVG` require non-empty `field`. - for `SUM` / `MIN` / `MAX` / `AVG`, target `MetricAggregation.field` MUST be schema-declared as numeric scalar type (`SCALAR_FIELD_TYPE_INT64`, `SCALAR_FIELD_TYPE_UINT64`, `SCALAR_FIELD_TYPE_FLOAT`, `SCALAR_FIELD_TYPE_DOUBLE`); type mismatch MUST fail closed as `SF_SCHEMA_VIOLATION`. - `cardinality` branch: `field` MUST be non-empty. - `MetricAggregation.missing_value_replacement`: when present and non-null, engines MUST substitute this atomic value for null/missing inputs before applying the metric; when absent or carrying `null_value`, null inputs MUST be ignored (not counted toward SUM/MIN/MAX/AVG value sets; COUNT semantics already exclude nulls by definition). - if `MetricAggregation.missing_value_replacement` is present and non-null, its atomic type MUST be legally castable to the schema-declared numeric type of the target field; type mismatch MUST fail closed during query admission (`SF_SCHEMA_VIOLATION`). - `MetricAggregation.builtin_function` MUST NOT be `AGGREGATE_FUNCTION_UNSPECIFIED`. - `bucket` nodes partition candidate sets (`terms` / `histogram` / `date_histogram` / `range` / `composite`) and carry recursive `sub_aggregations` directly on the `bucket` branch. - `nested.path` MUST be non-empty and MUST reference a schema-declared nested field path; `nested` carries recursive `sub_aggregations` directly on the `nested` branch. - `reverse_nested.path`, when present, MUST reference an ancestor nested path of the current nested context; unset `path` means root parent context, and `reverse_nested` carries recursive `sub_aggregations` directly on that branch. - `top_hits` nodes are leaf collectors. - `top_hits` result branch selection is context-dependent: nested contexts emit `TopHitsResult.nested_hits`; non-nested contexts emit `TopHitsResult.global_hits`. - `top_hits.limit`/`top_hits.offset` are per-bucket shallow-page controls; omitted `offset` defaults to `0`. - `TopHitsResult.{global_hits|nested_hits}` MUST be self-contained: each admitted branch MUST carry local `projected_field_names[]` and `highlight_field_names[]`, and branch-local hits MUST align against those local dictionaries (not `QuerySuccessPayload`-level headers). - `top_hits.attribute_projection` controls scalar projection for per-bucket hits and follows the same oneof intent semantics as top-level `fetch_spec.attribute_projection`. - if `TopHitsAggregation.attribute_projection` is absent, server MUST inherit top-level `fetch_spec.attribute_projection` intent for top-hits payload. - `top_hits.vector_projection_intent` controls per-bucket vector projection behavior with explicit oneof intent branches: - absent `TopHitsAggregation.vector_projection_intent` => deterministic fallback to `inherit`, - `inherit` => inherit top-level `fetch_spec.vector_projections`, - `explicit_map` => use only `explicit_map.vector_projections`, - `none` => return no vector fields for top-hits payload. - `explicit_map.vector_projections` MUST be non-empty; the canonical protobuf contract machine-checks that map cardinality and empty map is malformed (`SF_BAD_REQUEST`). - `top_hits.text_options` controls per-bucket highlight/snippet behavior and follows the same semantics as top-level `fetch_spec.text_options`. - when `TopHitsAggregation.text_options` is absent, server MUST inherit top-level `fetch_spec.text_options` intent for top-hits payload. - `top_hits.return_score=true` means emitted `TopHitsResult.{global_hits|nested_hits}.hits[].score` SHOULD be populated; `false` means `score` SHOULD be unset. - deterministic bucket guardrails: - `terms.size` MUST be > 0 when provided. - `terms.missing_value_replacement`: when present and non-null, rows where `terms.field` is null/missing MUST be bucketed using this replacement value; when absent or carrying `null_value`, null/missing rows are dropped from terms-bucket counting. - if `terms.missing_value_replacement` is present and non-null, its atomic type MUST be legally castable to the schema-declared atomic type of `terms.field`; mismatch MUST fail closed during query admission (`SF_SCHEMA_VIOLATION`). - `terms.order[]` uses `AggregationSortKey`; each entry MUST set exactly one target (`builtin` or `sub_aggregation_name`). - each `terms.order[].order` MUST be explicit; `SORT_ORDER_UNSPECIFIED` is malformed. - when `terms.order[].builtin` is used, value MUST be `BUILTIN_AGGREGATION_SORT_COUNT` or `BUILTIN_AGGREGATION_SORT_KEY`; `BUILTIN_AGGREGATION_SORT_UNSPECIFIED` is malformed. - when `terms.order[].sub_aggregation_name` is used, it MUST reference a direct child metric aggregation name under the same bucket node. - `composite` pagination contract: - `composite.sources[]` MUST be non-empty and declaration order is the comparator order. - each `CompositeSource.name` MUST be unique within one `CompositeAggregation` and MUST match regex `^[a-zA-Z0-9_.-]{1,128}$`. - each `CompositeSource` MUST set exactly one strategy branch (`terms` / `histogram` / `date_histogram`). - in this revision, each `CompositeSource` strategy target MUST resolve to a root-level scalar field; nested fields and sub-aggregation-derived sources are unsupported and MUST fail closed (`SF_UNSUPPORTED_QUERY_FEATURE`). - each `CompositeSource.order` MUST be explicit; `SORT_ORDER_UNSPECIFIED` is malformed. - `CompositeSource.missing_bucket=MISSING_VALUE_POLICY_UNSPECIFIED` resolves to deterministic default (`MISSING_VALUE_POLICY_LAST`). - `composite.limit` MUST be > 0. - if `composite.after_key` is present, its key set MUST exactly match `composite.sources[].name` and every value MUST be type-compatible with the matched source strategy. - comparator is lexicographic by `composite.sources[]`; each dimension comparison applies source `order` and `missing_bucket` policy. - Reader nodes MUST prune local buckets `<= composite.after_key` by this comparator before local top-N materialization. - Query Broker MUST k-way merge sorted reader outputs by the same comparator, emit global first `composite.limit` buckets, and set `BucketResult.composite.after_key` to the last emitted bucket `composite_key` when more buckets remain. - if request carries non-empty `composite.after_key`, it is compatibility-layer continuation state and MUST use `fetch_spec.composite_page` + carry non-empty `fetch_spec.composite_page.next_page_token`; any supplied `after_key` MUST exactly match the token-bound cursor or the request fails closed (`SF_INVALID_QUERY` by precedence). - `histogram.interval` MUST be > 0. - `date_histogram` MUST set exactly one of `calendar_interval` or `fixed_interval`. - `range.ranges[]` MUST be non-empty and the canonical protobuf contract machine-checks that cardinality; each range MUST preserve a non-empty interval under explicit boundary flags (`from_inclusive`/`to_inclusive`): - if both bounds are set and `from > to`, reject. - if both bounds are set and `from = to`, both `from_inclusive` and `to_inclusive` MUST be true; otherwise reject. - each `RangeAggregation.Range` MUST carry non-empty `key` before canonical query execution: - if source payload omits `key`, gateway MUST generate and populate a deterministic key string before canonical emission (for example canonical interval text with explicit inclusive/exclusive markers), - Search Fabric MUST use provided `RangeAggregation.Range.key` as the bucket identifier and MUST NOT auto-generate alternative formatter-specific keys. - `RangeAggregation.Range.key` values MUST be unique within one `RangeAggregation`; duplicate keys are malformed and MUST fail closed (`SF_INVALID_QUERY`). - final ordering is deterministic and MUST execute admitted normalized `fetch_spec.sort_keys`. - `fetch_spec.limit` semantics are mode-aware: - `limit > 0`: bounded hit retrieval. - `limit = 0`: - `ExecuteQuery`: valid only when `search_payload.aggregations` is non-empty OR `fetch_spec.track_total_hits` is explicitly configured (`enabled` or `threshold` branch); otherwise reject as `SF_BAD_REQUEST`. - `StreamQuery`: explicit unbounded export mode for hits (still bounded by service payload/backpressure/timeout policies). - nested `InnerHitsSpec.limit=0` follows a stricter contract than top-level `fetch_spec.limit=0`: it is valid only when nested `track_total_hits.enabled=true`, and in that mode engine returns parent-scoped nested counts without nested-hit materialization. - `fetch_spec.track_total_hits` controls total-hit accounting behavior: - if `fetch_spec.track_total_hits` is omitted: - `ExecuteQuery` and `StreamQuery` with `fetch_spec.limit > 0`: server MUST apply global default `TrackTotalHitsSpec{threshold=10000}`. - `StreamQuery` unbounded export (`fetch_spec.limit = 0`): server MUST implicitly use `TrackTotalHitsSpec{enabled=false}`. - if `fetch_spec.track_total_hits` message is present but `tracking` oneof is unset, request is malformed and MUST fail closed (`SF_INVALID_ARGUMENT`). - `enabled=true`: track exact total hits (`QuerySuccessPayload.total_hits.relation=TOTAL_HITS_RELATION_EQUAL_TO`). - `enabled=false`: disable tracking; server MAY omit `QuerySuccessPayload.total_hits`. - `threshold=N`: track up to N; return exact total when `actual_total <= N`, or lower bound (`value=N`, `relation=TOTAL_HITS_RELATION_GREATER_THAN_OR_EQUAL_TO`) when `actual_total > N`. - `fetch_spec.rescore` is an optional second-stage rerank contract: - stage-1 candidate generation still follows canonical retrieval + fusion + deterministic sort. - when present, `rescore.window_size` bounds rerank scope to top-N stage-1 candidates only. - when present, `RescoreSpec.strategy` oneof MUST set exactly one branch; empty rescore payload is malformed (`SF_INVALID_ARGUMENT`). - this revision reserves cross-encoder rerank strategy (`cross_encoder`) for Profile-3+ rollout; unsupported profile/mode MUST fail closed (`SF_UNSUPPORTED_OPTION`). - rerank execution MUST preserve deterministic tie-break (`doc_id asc`) after rerank-score ties. - sort-contract normalization is mandatory before execution: - if `fetch_spec.sort_keys` is empty, server MUST materialize canonical default sort (`score`, `order=DESC`) + (`doc_id`, `order=ASC`). - if caller-provided `fetch_spec.sort_keys` does not contain `doc_id`, server MUST append (`doc_id`, `order=ASC`) as final tie-break key. - each `SortKey` MUST set exactly one `target` branch: - `field`: physical field sort target (`FieldSortTarget.field` required, `mode`/`missing_value_policy` allowed, `nested_context` optional for nested-field sorting), - `score`: virtual score sort (`_score`) with no payload fields, - `doc_id`: virtual document-id sort (`_id`) with no payload fields. - if caller provides `SortKey` with `order=SORT_ORDER_UNSPECIFIED`, request is malformed and MUST fail closed with `SF_INVALID_QUERY`. - if `FieldSortTarget.mode` is present and equals `SORT_MODE_UNSPECIFIED`, request is malformed and MUST fail closed with `SF_INVALID_QUERY`. - if `FieldSortTarget.missing_value_policy` is present and equals `MISSING_VALUE_POLICY_UNSPECIFIED`, request is malformed and MUST fail closed with `SF_INVALID_QUERY`. - if `FieldSortTarget.missing_value_policy` is absent, engine MUST use deterministic default by `SortKey.order`: `SORT_ORDER_ASC -> MISSING_VALUE_POLICY_LAST`, `SORT_ORDER_DESC -> MISSING_VALUE_POLICY_FIRST`. - if `FieldSortTarget.mode` is absent, engine MUST use deterministic default by `SortKey.order` for multi-valued fields: `SORT_ORDER_ASC -> SORT_MODE_MIN`, `SORT_ORDER_DESC -> SORT_MODE_MAX`. - for `FieldSchema.config.capabilities.multi_valued=true`, explicit empty-array values (`[]`) MUST be treated as missing/null for sort extraction: - `MIN`/`MAX` mode MUST NOT synthesize a sortable scalar from an empty array. - effective branch MUST follow `FieldSortTarget.missing_value_policy` exactly (including defaulted policy when absent). - nested sort context guardrail: - if `FieldSortTarget.nested_context` is present, `nested_context.path` MUST be non-empty and MUST reference a schema-declared nested path. - when `FieldSortTarget.field` resolves to a nested field path, `FieldSortTarget.nested_context` MUST be present; omission is malformed (`SF_INVALID_QUERY`). - when `FieldSortTarget.field` is non-nested, `FieldSortTarget.nested_context` MUST be absent; presence is malformed (`SF_INVALID_QUERY`). - if `FieldSortTarget.nested_context.filter` is present, it is evaluated in nested-document scope before sort-value extraction (`mode`/`missing_value_policy` apply after filter narrowing). - sort target type-safety guardrail: - `FieldSortTarget.field` MUST resolve to a schema-declared scalar/keyword leaf field (root-level or nested leaf under valid `nested_context`). - sorting by vector fields, nested/object container fields, or other non-leaf structural fields is malformed and MUST fail closed (`SF_SCHEMA_VIOLATION` or `SF_BAD_REQUEST` by precedence). - when `SortKey.target` is `score` or `doc_id`, any field-only settings (`mode` / `missing_value_policy` / `nested_context`) are structurally impossible and MUST be rejected by gateway normalization if present in source wire payload. - `fetch_spec.attribute_projection` controls scalar-attribute return intent: - when `fetch_spec.attribute_projection` is absent, server MUST default to `return_all` with empty `exclude_fields` (all schema-declared scalar fields). - when present, `AttributeProjectionSpec.intent` oneof MUST set exactly one branch; unset intent is a present-but-empty typed carrier and MUST fail closed (`SF_INVALID_ARGUMENT`). - `return_all` means return all schema-declared scalar fields, then drop `exclude_fields`. - `return_explicit` means return only `include_fields`; if source payload carries both include and exclude semantics, gateway MUST resolve set subtraction before emitting canonical RPC payload. - `return_explicit.include_fields` MUST be non-empty (`return_none` is the explicit empty-scalar branch). - `return_none` means return no scalar attributes (Turbopuffer-equivalent `include_attributes=false` intent). - `include_fields` / `exclude_fields` entries MUST reference schema-declared scalar fields only; unknown or non-scalar field names are malformed (`SF_BAD_REQUEST`). - `InnerHitsSpec` contract: - `limit`/`offset` are shallow-page controls per parent hit; implementation MUST enforce service-static maxima. - `InnerHitsSpec.limit = 0` is valid ONLY when `track_total_hits.enabled=true`; in this mode, engine MUST skip nested hit materialization and return parent-scoped nested counts only. - `InnerHitsSpec.limit = 0` with `track_total_hits` omitted, `enabled=false`, or `threshold=N` is malformed and MUST fail closed (`SF_BAD_REQUEST`). - `track_total_hits` semantics are identical to top-level fetch contract but scoped per inner-hits result set. - `attribute_projection` controls scalar projection for nested hits and follows the same oneof intent semantics as top-level `fetch_spec.attribute_projection`. - if `InnerHitsSpec.attribute_projection` is absent, server MUST inherit top-level `fetch_spec.attribute_projection` intent for nested hits. - if `InnerHitsSpec.sort_keys` is empty, default sort is nested-score desc + `parent_array_offset` asc. - when `InnerHitsSpec.sort_keys` contains virtual `doc_id`, engine MUST map it to nested deterministic key `parent_array_offset` (not top-level `QueryHit.doc_id`). - if caller-provided `InnerHitsSpec.sort_keys` does not contain virtual `doc_id`, server MUST automatically append `parent_array_offset` with `SORT_ORDER_ASC` as the final tie-break key to guarantee shallow-paging determinism. - query hits MAY be emitted for both `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` collections and `COLLECTION_TRUTH_MODE_EXTERNAL` covering-read routes. - on `COLLECTION_TRUTH_MODE_EXTERNAL` covering-read routes, if covering-read cannot satisfy the full requested projection deterministically, response MUST NOT emit partial projected hits; behavior MUST follow Section `5.4` fallback matrix and Section `5.11` when the TiDB fast path is used. - `FieldData.list_value` is the canonical carrier for multi-valued fields and JSON-array typed payloads in projections and row values. - for schema-declared fields, `FieldData.list_value` is admissible only when `FieldSchema.config.capabilities.multi_valued=true`; otherwise list payload admission MUST fail closed (`SF_SCHEMA_VIOLATION`). - `ListValue.values[]` MUST contain only `AtomicValue` elements (no nested arrays and no vector elements). - `FieldData.object_array_value` is the canonical structured object-array carrier for container fields (`object`/`nested`); `ObjectArrayValue.documents[]` order MUST preserve source array order and map to deterministic `parent_array_offset` for nested extraction paths. - in merge write payloads (`PatchPayload.fields`), `FieldData.atomic_value.null_value = NULL_VALUE_NULL` is the canonical explicit-clear marker for any declared field type (`scalar` / `vector` / `list` / `object` / `nested`); callers MUST NOT encode clears via empty vector/list/object-array payload surrogates. - in full-replace payloads (`UpsertPayload.fields`), omitted schema fields are already treated as clear/remove intent; explicit `NULL_VALUE_NULL` remains valid but semantically redundant with omission. - predicate RHS type contract is atomic-only by protobuf shape: `ComparisonPredicate.value` and `RangePredicate` bounds use `AtomicValue`; `InListPredicate.values` is `ListValue`. - `PredicateNode` present with unset `node` oneof is a present-but-empty typed carrier and MUST fail closed (`SF_INVALID_ARGUMENT`). - logical predicate-node legality: - `LogicalAnd.children[]` and `LogicalOr.children[]` MUST each contain at least 2 child predicates. - every logical child MUST itself resolve to a non-empty `PredicateNode` after typed-carrier validation. - `LogicalNot.child` MUST be present and resolve to a non-empty `PredicateNode`. - empty/singleton logical forms after typed-carrier validation are malformed and MUST fail closed (`SF_INVALID_QUERY`). - `ComparisonPredicate` is equality-only (`COMPARE_OP_EQ` / `COMPARE_OP_NEQ`); ordering/range comparisons (`>`, `>=`, `<`, `<=`) MUST be expressed as `RangePredicate` (single- or double-bounded). - `RangePredicate` is the canonical ordering/range predicate: - at least one of `from` / `to` MUST be present. - when `from` or `to` is present, the bound MUST resolve to a non-empty `AtomicValue`; present-but-empty bound carriers are malformed (`SF_INVALID_ARGUMENT`). - if both bounds are present, their values MUST be mutually comparable and schema-type compatible. - if both bounds are present and `from > to`, request is malformed. - if both bounds are present and `from = to`, both `from_inclusive` and `to_inclusive` MUST be true; otherwise request is malformed. - malformed interval shapes or type-incompatible range bounds MUST fail closed (`SF_INVALID_QUERY`). - `InListPredicate.values.values[]` MUST be non-empty and homogeneous by atomic type (for example all `int64` or all `string`); empty or heterogeneous arrays are malformed and MUST be rejected (`SF_INVALID_QUERY` or `SF_SCHEMA_VIOLATION` by precedence). - `ExistsPredicate` is the canonical field-presence predicate. - `EXISTS(field)` MUST be represented as `PredicateNode.exists`. - `IsNullPredicate` is the explicit-null predicate (`AtomicValue.null_value`) for relational compatibility semantics and MUST be represented as `PredicateNode.is_null`. - when `AtomicValue.null_value` branch is selected, value MUST be `NULL_VALUE_NULL`; `NULL_VALUE_UNSPECIFIED` is malformed and MUST fail closed. - `NestedPredicate` is the canonical nested-scope boolean filter (`path` + recursive `child_filter`) and MUST be represented as `PredicateNode.nested`. - inside `NestedPredicate.child_filter`, field references MUST use absolute canonical paths (same grammar as top-level predicates, for example `comments.author`), not nested-relative short names. - `NOT EXISTS(field)` and `IS NULL(field)` are semantically distinct in this revision. - if `ComparisonPredicate` carries unsupported/non-equality `CompareOp` or type-incompatible operands for equality/inequality comparison, request MUST fail closed with `SF_INVALID_QUERY`. - vector materialization is controlled only by `fetch_spec.vector_projections` key set; projected schema vector fields are returned through `QueryHit.projected_values[i].vector_value` aligned with `QuerySuccessPayload.projected_field_names[i]`. - vector projection guardrails: - empty `fetch_spec.vector_projections` means no vector fields are returned in response projections. - caller MAY request per-field vector payload format through `fetch_spec.vector_projections[].format`. - `fetch_spec.vector_projections[].target_dimensions` is executable for MRL-style truncation in this revision. - if `target_dimensions` is unset or `0`, server MUST resolve it to schema vector dimensions (deterministic fallback). - if `target_dimensions` is non-zero, it MUST satisfy `target_dimensions <= schema dimensions`. - if `0 < target_dimensions < schema dimensions`, server MUST apply deterministic truncation before response encoding. - if `target_dimensions > schema dimensions`, server admission MUST fail closed with `SF_VECTOR_DIMENSION_MISMATCH`. - this hard-check is mandatory for mixed-version safety: a request carrying truncated dimensions (for example `256`) to a node serving a schema with different dimensions MUST be rejected deterministically. - if `vector_projections[].format` is absent, service uses collection/profile default; when no explicit default is configured, fallback is `VECTOR_PAYLOAD_FORMAT_EXACT_FLOAT`. - if `vector_projections[].format` is present and equals `VECTOR_PAYLOAD_FORMAT_UNSPECIFIED`, request is malformed and MUST fail closed (`SF_BAD_REQUEST`). - `vector_projections` keys MUST reference schema-declared vector fields; unknown or non-vector field keys are malformed (`SF_BAD_REQUEST`). - `vector_projections` keys are independent from `fetch_spec.attribute_projection` (scalar attributes only); vector projection requests MUST be evaluated solely against schema vector-field validity and policy gates. - `vector_projections` map MUST be canonicalized by byte-ascending field key before digest/signature/token binding. - allowed formats are policy-controlled at gateway/service by collection/profile. - quantized payload branches MUST remain deterministic for the same source vector and admitted schema generation. - when requested format is not allowed by policy, request MUST fail closed (`SF_UNSUPPORTED_OPTION`). - high-dimensional vector response guardrails: - for requests projecting high-dimensional vectors in `VECTOR_PAYLOAD_FORMAT_EXACT_FLOAT` (resolved from `vector_projections` + defaults), gateway/server MAY dynamically tighten effective `max_result_window` and/or payload budget. - if tightened payload budget is exceeded, request MUST fail with `413 SF_PAYLOAD_TOO_LARGE`. - error details MUST include `limit_scope` (`dynamic_payload_budget` or `dynamic_result_window`) and effective/ requested limit values. - text highlight output is controlled by `fetch_spec.text_options`: - `require_highlighting=false` (or omitted block) means highlighting is disabled. - `require_highlighting=true` is valid only when request includes at least one text clause and the admitted route/capability set supports highlight generation. - when `require_highlighting=true`, `explicit_fields` MUST be non-empty and each field MUST reference a schema-declared text-searchable field. - when `require_highlighting=false`, `explicit_fields` MUST be empty. - when highlighting is required and both tags are empty, defaults are `highlight_pre_tag=""` and `highlight_post_tag=""`. - when exactly one highlight tag is provided, request is malformed (`SF_BAD_REQUEST`). - Search Fabric MUST generate final highlight snippets using admitted tags and return them via `QueryHit.highlights`. - `QuerySuccessPayload.highlight_field_names[]` is the response-level highlight dictionary; `QueryHit.highlights[].field_index` MUST reference this dictionary. - Integration API receives only canonical clauses; source-specific query rewrites/parsing MUST be completed in Query Gateway before request reaches this API. - PIT-equivalent deep paging context is controlled by `fetch_spec.search_context`: - `search_context.keep_alive_ms` on first-page requests asks server to mint/refresh snapshot context. - when `search_context.keep_alive_ms` is present, value MUST be `> 0`; present `0` is malformed (`SF_BAD_REQUEST`). - first-page requests MUST NOT carry `search_context.token`; token-on-first-page semantics is malformed (`SF_INVALID_QUERY`). - `search_context.token` on continuation requests pins ordinary hit-pagination to an existing snapshot context; composite continuation no longer depends on this separate lineage carrier because `composite_page.token` already binds immutable snapshot lineage plus composite cursor. - for requests carrying non-empty compatibility-layer `CompositeAggregation.after_key`, request MUST use `fetch_spec.composite_page`, and `fetch_spec.composite_page.next_page_token` is mandatory. - when a response emits non-empty `BucketResult.composite.after_key`, it MUST also return non-empty `QuerySuccessPayload.pagination.composite_page.token` plus `composite_page.expires_at_unix_ms`; `search_context` is returned only when the request asked for keep-alive on the unary query path. `ExecuteQuery` transport contract: - `effective_read_ts`, `safe_ts` - `applied_schema_version`, `applied_schema_generation_token` - transport `OK` returns `ExecuteQuerySuccessResponse` - `ExecuteQuerySuccessResponse.success` is the unified success payload (`hits` + `aggregations`) with warning list - explicit aggregation truncation summary lives in `ExecuteQuerySuccessResponse.success.aggregation_truncation` for gateway depth-intercept/engine truncation visibility - pagination context lives in `QuerySuccessPayload.pagination` (`next_page.{token,expires_at_unix_ms}`, optional `search_context.{token,expires_at_unix_ms}`, optional `composite_page.{token,expires_at_unix_ms}`; composite-pagination responses with non-empty `BucketResult.composite.after_key` MUST populate the composite-page carrier, while `search_context` remains optional keep-alive state for ordinary unary pagination) - optional total-hit contract lives in `QuerySuccessPayload.total_hits` with Elasticsearch-aligned relation semantics (`TOTAL_HITS_RELATION_EQUAL_TO` / `TOTAL_HITS_RELATION_GREATER_THAN_OR_EQUAL_TO`) - optional vector field materialization appears under `ExecuteQuerySuccessResponse.success.hits[i].projected_values[j].vector_value` when `success.projected_field_names[j]` is a projected vector schema field. - response projection payload is columnar by contract: - `QuerySuccessPayload.projected_field_names[]` lists projected columns once per response in deterministic admitted order (scalar segment from `fetch_spec.attribute_projection`, vector segment from `fetch_spec.vector_projections`). - each `QueryHit.projected_values[]` MUST align index-by-index with `projected_field_names[]`. - `hits[].projected_values` length MUST equal `projected_field_names` length. - null/missing projected values MUST use `FieldData.atomic_value.null_value` placeholder at the same index. - top-hits aggregation payloads are independent from this top-level header and MUST use branch-local headers in `TopHitsResult.{global_hits|nested_hits}`. - response highlight payload is sparse by contract: - `QuerySuccessPayload.highlight_field_names[]` lists highlight field names once per response. - each `QueryHit.highlights[]` entry is field-local (`field_index`) and snippet-local (`snippets[]`). - each `field_index` MUST be within `highlight_field_names[]` bounds. - top-hits aggregation payloads are independent from this top-level dictionary and MUST use branch-local dictionaries in `TopHitsResult.{global_hits|nested_hits}`. - nested inner-hits payload is deterministic by named headers: - `QuerySuccessPayload.inner_hits_headers[]` defines per-name projection/highlight dictionaries. - each `QueryHit.inner_hits[]` carries `NamedInnerHitsResult` entries resolved by `name`. - `InnerQueryHit` identities are isolated (`nested_physical_id`) and include deterministic source-array metadata (`parent_array_offset`). - malformed/authz/catalog/runtime/query-execution failures use non-`OK` gRPC status plus canonical `ErrorStatus`. - `ErrorStatus.correlation_id` MUST be non-empty; when caller provided `RequestMeta.correlation_id`, error payload MUST echo that value, and otherwise the server MUST generate a canonical correlation ID. - the canonical proto models emitted `ErrorStatus.correlation_id` with machine-checkable non-empty validation; empty emission is malformed service output rather than a caller-visible optional branch. - `ErrorStatus.context` is the canonical typed machine-readable failure context layer. - `ErrorStatus.diagnostics` is the string diagnostic/projection layer and MAY mirror subsets of `ErrorStatus.context`. - on non-`OK` unary query failure, callers MUST read machine-readable failure context from `ErrorStatus.context`; callers MUST NOT depend on a public in-envelope unary query error carrier. - `ErrorStatus.context.query.execution` carries admitted-query / resolved-lineage failure context when known: - `canonical_query_digest` - `applied_schema_version` - `applied_schema_generation_token` - `ErrorStatus.context.query.visibility_timeout` carries canonical strict/bounded-staleness timeout context: - `required_read_ts` - `effective_read_ts` - `safe_ts` - bounded-staleness branch additionally carries `max_staleness_ms` and `actual_lag_ms` - `SF_VISIBILITY_TIMEOUT` is reserved for query/readiness visibility waiting only; write-side `VISIBILITY_POLICY_WAIT_FOR_VISIBLE` deadline expiry remains an ingest stream failure and MUST use retryable `SF_BACKEND_UNAVAILABLE` rather than `ErrorStatus.context.query.visibility_timeout`. - `ErrorStatus.context.query.schema_version_conflict` carries query-serving schema/token drift context when `ErrorStatus.code=SF_VERSION_CONFLICT`: - `expected_schema_generation_token` when the failed request or token chain fenced on a specific serving generation - `conflict_source` as the canonical enum-valued stale-fence source: - `QUERY_SCHEMA_CONFLICT_SOURCE_EXPECTED_SCHEMA_GENERATION_TOKEN` - `QUERY_SCHEMA_CONFLICT_SOURCE_CONTINUATION_PAGE_TOKEN` - `QUERY_SCHEMA_CONFLICT_SOURCE_COMPOSITE_PAGE_TOKEN` - `QUERY_SCHEMA_CONFLICT_SOURCE_SEARCH_CONTEXT_TOKEN` - deterministic precedence: - direct request fencing drift uses `QUERY_SCHEMA_CONFLICT_SOURCE_EXPECTED_SCHEMA_GENERATION_TOKEN` - continuation carrying `fetch_spec.continuation_page.next_page_token` uses `QUERY_SCHEMA_CONFLICT_SOURCE_CONTINUATION_PAGE_TOKEN` - composite continuation carrying `fetch_spec.composite_page.next_page_token` uses `QUERY_SCHEMA_CONFLICT_SOURCE_COMPOSITE_PAGE_TOKEN` - `QUERY_SCHEMA_CONFLICT_SOURCE_SEARCH_CONTEXT_TOKEN` is reserved for search-context-only continuation/PIT drift without a page token - therefore when continuation carries both page and search-context token families and both fence the same stale serving generation, the canonical projection MUST use the page-token branch for that request shape (`QUERY_SCHEMA_CONFLICT_SOURCE_CONTINUATION_PAGE_TOKEN` for any request carrying `fetch_spec.continuation_page.next_page_token`, including unary continuation and `StreamQuery` resume; `QUERY_SCHEMA_CONFLICT_SOURCE_COMPOSITE_PAGE_TOKEN` for composite continuation) - `ErrorStatus.diagnostics["expected_schema_generation_token"]` and `ErrorStatus.diagnostics["conflict_source"]` MAY mirror the same typed branch for compatibility/header/body projection, but machine-readable protobuf consumers MUST treat `context` as authoritative. - `ErrorStatus.context.query.catalog_unavailable.schema_bootstrap` carries query bootstrap-unavailable context when `ErrorStatus.code=SF_CATALOG_UNAVAILABLE`: - `schema_bootstrap_state` - `ErrorStatus.diagnostics["catalog_unavailable_reason"]=schema_bootstrap` and `ErrorStatus.diagnostics["schema_bootstrap_state"]` MAY mirror the same typed branch for compatibility/header/body projection. - `ErrorStatus.context.query.catalog_unavailable.schema_convergence` carries query serving-lineage convergence-unavailable context when `ErrorStatus.code=SF_CATALOG_UNAVAILABLE`: - `schema_convergence_state` - `ErrorStatus.diagnostics["catalog_unavailable_reason"]=schema_convergence` and `ErrorStatus.diagnostics["schema_convergence_state"]` MAY mirror the same typed branch for compatibility/header/body projection. - `ErrorStatus.context.query.catalog_unavailable.statistics_freshness` carries statistics snapshot freshness-miss context when `ErrorStatus.code=SF_CATALOG_UNAVAILABLE`: - `max_staleness_ms` - `latest_stats_commit_ts` - `latest_stats_collected_at_unix_ms` - `ErrorStatus.diagnostics["catalog_unavailable_reason"]=statistics_freshness` plus the same freshness values MAY mirror the typed branch for compatibility/header/body projection, but machine-readable protobuf consumers MUST treat `context` as authoritative. - `ErrorStatus.context.query.unsupported_feature` carries machine-readable query unsupported-feature context when `ErrorStatus.code=SF_UNSUPPORTED_QUERY_FEATURE` or `SF_UNSUPPORTED_OPTION`: - `unsupported_feature` - `normalized_path` - `fallback_hint` - strict aggregation-depth branches additionally carry `requested_aggregation_depth` and `max_aggregation_depth` - `ErrorStatus.context.query.limit_rejection` carries machine-readable query limit/budget rejection context when `ErrorStatus.code=SF_TOO_MANY_OPERATIONS` or `SF_PAYLOAD_TOO_LARGE`: - `limit_scope` - `configured_limit` - `requested_value` - `requested_offset` - `requested_limit` - `requested_window` - `effective_limit` - `requested_bytes` - `field` when the limit branch is field-local - `effective_constraint_source` for selector-materialization clamp branches; allowed values in this revision remain `max_entries_per_batch` and `max_result_window` only, while `configured_limit` carries the actual post-`caller_requested_limit` clamp cutoff - `ErrorStatus.context.descriptor_publishing_unavailable` carries metadata-parity descriptor-publication lag context when `ErrorStatus.code=SF_CATALOG_UNAVAILABLE` outside query-family RPCs: - `descriptor_state` - `missing_descriptor_fields` - `ErrorStatus.diagnostics["descriptor_state"]` / `ErrorStatus.diagnostics["missing_descriptor_fields"]` MAY mirror this typed branch for compatibility/string-only consumers. - `ErrorStatus.context.ingest_schema_boundary_invalidation` carries Flow-A DDL boundary rejection context when `ErrorStatus.code=SF_VERSION_CONFLICT`: - `invalidated_at_commit_ts` - `ErrorStatus.diagnostics["invalidated_at_commit_ts"]` MAY mirror this typed branch for compatibility/string-only consumers. - `ErrorStatus.context.allocator.limit_rejection` carries allocator machine-readable guardrail rejection context when `ErrorStatus.code=SF_TOO_MANY_OPERATIONS` on `AllocateIds`: - enum-valued `limit_scope` (`ALLOCATOR_LIMIT_SCOPE_PLAN_ITEMS`, `ALLOCATOR_LIMIT_SCOPE_FORWARD_RESERVATION_WINDOW`, or `ALLOCATOR_LIMIT_SCOPE_IDENTITY_SPACE_REMAINING`) - `configured_limit` - `requested_value` - `ErrorStatus.diagnostics["limit_scope"]`, `["configured_limit"]`, and `["requested_value"]` MAY mirror the same typed branch for compatibility/string-only consumers. - `ErrorStatus.context.allocator.catalog_unavailable.schema_bootstrap` carries allocator bootstrap-unavailable context when `ErrorStatus.code=SF_CATALOG_UNAVAILABLE` on `AllocateIds`: - `schema_bootstrap_state` - `ErrorStatus.diagnostics["catalog_unavailable_reason"]=schema_bootstrap` and `ErrorStatus.diagnostics["schema_bootstrap_state"]` MAY mirror the same typed branch for compatibility/string-only consumers. - `ErrorStatus.context.allocator.reservation_conflict` carries allocator dense-burn reservation-conflict context when `ErrorStatus.code=SF_VERSION_CONFLICT` on `AllocateIds`: - `doc_id` - enum-valued `conflict_reason` (`ALLOCATOR_RESERVATION_CONFLICT_REASON_BELOW_WATERMARK_BURNED_GAP`) - `ErrorStatus.diagnostics["doc_id"]` and `["allocator_conflict_reason"]` MAY mirror the same typed branch for compatibility/string-only consumers. - `ErrorStatus.context.allocator.receipt_validation` carries allocator-receipt validation context when allocator-backed ingest admission rejects the follow-on `MutationBatch` before row admission: - enum-valued `reason` (`ALLOCATOR_RECEIPT_VALIDATION_REASON_MISSING_REQUIRED_RECEIPT`, `ALLOCATOR_RECEIPT_VALIDATION_REASON_MALFORMED_RECEIPT`, `ALLOCATOR_RECEIPT_VALIDATION_REASON_SIGNATURE_INVALID`, `ALLOCATOR_RECEIPT_VALIDATION_REASON_EXPIRED_RECEIPT`, `ALLOCATOR_RECEIPT_VALIDATION_REASON_CALLER_MISMATCH`, `ALLOCATOR_RECEIPT_VALIDATION_REASON_SCOPE_MISMATCH`, `ALLOCATOR_RECEIPT_VALIDATION_REASON_REALIZED_DOC_IDS_MISMATCH`, `ALLOCATOR_RECEIPT_VALIDATION_REASON_ROUTE_INTENT_MISMATCH`, or `ALLOCATOR_RECEIPT_VALIDATION_REASON_SCHEMA_GENERATION_TOKEN_MISMATCH`) - `ErrorStatus.diagnostics["receipt_validation_reason"]` MAY mirror the same typed branch for compatibility/string-only consumers. - deterministic code split: - `reason=ALLOCATOR_RECEIPT_VALIDATION_REASON_CALLER_MISMATCH` -> `SF_FORBIDDEN` - every other allocator-receipt validation reason -> `SF_BAD_REQUEST` - top-level `ExecuteQuerySuccessResponse` observability fields are success-path observability only. - deterministic ordering contract: when scores are equal, tie-break by `doc_id` ascending. - score materialization contract: - if `fetch_spec.return_score=true`, `success.hits[].score` SHOULD be populated. - if `fetch_spec.return_score=false`, `success.hits[].score` MUST be unset (not `0.0` sentinel). ### 5.2 Compatibility Gateway Mapping All source-specific query/write syntax and semantics MUST be normalized in Ingest Gateway / Query Gateway. Integration API only accepts canonical typed payloads. | Input Ecosystem | Gateway Input | Canonical Mapping | |---|---|---| | MySQL FTS | `MATCH ... AGAINST` forms | one source `MATCH(col_list)` lowers to one canonical `TextClause{fields[]}`; boolean/natural-language branches lower through `TextClause.mode_override.boolean_match` / `TextClause.mode_override.natural_language`, and `WITH QUERY EXPANSION` lowers to `TextClause.mode_override.natural_language.query_expansion=true` | | MySQL FTS | default parser path (no `WITH PARSER` clause) / `WITH PARSER NGRAM` / `WITH PARSER MECAB` | parser intent is canonicalized at schema/control plane (`PutCollectionSchema` + `FieldSchema.config.field_definition.text.analysis.analyzer` / `FieldSchema.config.field_definition.text.analysis.options` / `FieldSchema.options`/`CollectionSettings.options`); Query API does not carry parser field, and explicit `WITH PARSER STANDARD` is outside active compatibility scope | | MySQL FTS | `CREATE/ALTER FULLTEXT INDEX`, parser/token-size constraints | gateway canonicalizes DDL semantics into `PutCollectionSchema` typed schema payload (`FieldSchema.config.field_definition.text.analysis` analyzer+options + `FieldSchema.config.field_definition.text.fulltext`); charset/collation are source-side supplemental metadata only | | MySQL FTS | forbidden `MATCH(...)` + `GROUP BY ... WITH ROLLUP` shape | reject at gateway pre-admission with `SF_UNSUPPORTED_SYNTAX`; MUST NOT emit canonical query request | | Elasticsearch | `query.bool`, `match`, `knn`, paging/sort subset | gateway lowers query structure to one canonical retrieval tree (`search_payload.root_retrieval`) composed from `RetrievalGroup` / `RetrievalNode`; plain `match` lowers to `TextClause.mode_override.plain_match` (including `PlainMatchOptions.operator` and `minimum_should_match` pushdown), `knn` lowers to `VectorClause`, bool `must`/`should` branches lower through `RetrievalGroup.must_clauses[]` / `ShouldMatchSpec.clauses[]`, bool `filter`/`must_not` branches lower through predicate trees (`search_payload.global_filter`, `NestedQueryPayload.child_filter`, `LogicalNot`, `NestedPredicate`, `TextMatchPredicate`), and `sort[].nested.path/filter` lowers to `FieldSortTarget.nested_context`; query-vector dimensions use the MRL prefix-compatible rule (`0 < Dq <= Dschema`); when one admitted Elasticsearch query activates both metadata-search and semantic vector retrieval branches, gateway MUST explicitly set `search_payload.dedup_by_doc_id=true` so final hits are deduplicated before pagination/response emission | | Elasticsearch | `query.multi_match` (Profile 2 subset) | gateway MUST choose retrieval-tree lowering by source `type`: `cross_fields` lowers to one `TextClause{fields[]}` cross-field leaf; `best_fields` (including default source type) lowers to a `RetrievalGroup{fusion_strategy=GROUP_FUSION_STRATEGY_DIS_MAX, tie_breaker=}` containing one single-field text leaf per field inside `should_match.clauses[]`; `most_fields` lowers to `RetrievalGroup{fusion_strategy=GROUP_FUSION_STRATEGY_WEIGHTED_SUM}` with one single-field text leaf per field inside `should_match.clauses[]`; unsupported source types fail closed | | Elasticsearch | scalar `bool.should` / `constant_score` boost branches | gateway lowers scalar/text/vector/nested soft-score branches directly into retrieval-tree leaves with node-local `boost`, placing them under `must_clauses[]` or `should_match.clauses[]` according to source bool structure; when source `bool` has `should` but no `must`/`filter` and omits `minimum_should_match`, gateway MUST normalize the owning `ShouldMatchSpec.threshold` to `absolute_count=1`; hard-filter eligibility constraints lower only through predicate trees (`search_payload.global_filter`, nested `child_filter`, `LogicalNot`) | | Elasticsearch | `nested` query with `inner_hits` subset | gateway lowers each nested branch to a `NestedClause` leaf inside the retrieval tree (`path`, `NestedQueryPayload`, `score_mode`, `InnerHitsSpec`), and maps nested `inner_hits.sort` `_id` target to canonical `parent_array_offset` sort semantics | | Elasticsearch | `refresh in {false,true,wait_for}` | maps to (`policy.durability=wal_chunk_durable`, `policy.visibility=async`) for `false`, and (`policy.durability=wal_chunk_durable`, `policy.visibility=wait_for_visible`) for `true` or `wait_for` | | Elasticsearch | `POST /{index}/_doc` auto-ID | realized `_id` MUST be decimal-string uint64 (lossless parseable by canonical `strconv.ParseUint(id, 10, 64)` equivalent); when the resolved collection uses allocator-backed create/insert identity, gateway obtains that realized ID through `IdAllocatorService.AllocateIds` before canonical mutation construction; Elasticsearch base64-like UUID auto-ID format is out of active profile | | Elasticsearch | `PUT/GET /{index}/_mapping` (Profile 1 subset) | gateway MUST resolve external `{index}` to canonical `CollectionNameBinding{namespace_id=, collection_name=}` before concrete collection lookup; when the name binding has no authoritative `CollectionRef` yet, gateway MUST first ensure collection identity through `PutCollectionMetadata.name_binding` and use the returned `CollectionDescriptor.collection` before first schema apply, and both state-mutating legs (`PutCollectionMetadata`, then `PutCollectionSchema`) MUST carry the same replay-stable non-empty outer `RequestMeta.idempotency_key` plus explicit gateway `RequestMeta.client_id` / `RequestMeta.client_kind`; gateway canonicalizes mapping payload/response through control-plane schema model (`DocIDMapping` + `FieldSchema[]` + optional `CollectionSettings`), applies via `PutCollectionSchema` on the resolved `CollectionRef`, and retrieves via `GetCollectionSchema` | | Elasticsearch | `search_after` subset (active profile) | token-only continuation: gateway canonicalizes admitted continuation requests to `fetch_spec.sort_keys` + `QueryRequest.fetch_spec.continuation_page.next_page_token`; raw source `search_after` values-array passthrough is out of active subset and MUST fail closed | | Elasticsearch | hit `_seq_no` / `_primary_term` response fields (OCC read loop) | gateway MUST project canonical `QueryHit.version_commit_ts` into Elasticsearch `_seq_no` on hit responses and MUST inject static `_primary_term=1` for client/SDK parse compatibility | | Elasticsearch | regular single-document `PUT/POST /{index}/_doc/{id}` index intent | source body lowers to one full-replace `RowMutation.operation.upsert`; gateway MUST reconstruct `result=created` + HTTP `201` vs `result=updated` + HTTP `200` from the authoritative post-apply `RowMutationApplyResultKind` returned in `MutationAck.success.item_outcomes[]` rather than from a prewrite existence guess; when Profile-2 `if_seq_no` is supplied, gateway lowers it to `RowMutation.expected_commit_ts`, otherwise `expected_commit_ts` is omitted for unconditional replace semantics | | Elasticsearch | create-only write intent (`POST /{index}/_create/{id}` or `op_type=create`) | gateway lowers to canonical `RowMutation.operation=upsert` + `RowMutation.expected_commit_ts=0`; canonical absent-doc logical version is `latest_commit_ts=0`, so already-existing docs deterministically reject with `SF_VERSION_CONFLICT` | | Elasticsearch | single-document `GET /{index}/_doc/{id}` read-by-id | gateway lowers to canonical point lookup (`ExecuteQuery` with exact `doc_id` filter, `fetch_spec.limit=1`, deterministic `_source` projection) and reconstructs Elasticsearch `found`, `_source`, `_seq_no`, and static `_primary_term=1`; absent canonical hit reconstructs `found=false` without synthetic placeholder source | | Elasticsearch | single-document `DELETE /{index}/_doc/{id}` delete intent | gateway lowers to one canonical `RowMutation.operation.delete`; response reconstruction MUST use authoritative post-apply `RowMutationApplyResultKind` from `MutationAck.success.item_outcomes[]` so `result=deleted` vs `result=not_found` is based on final apply state rather than a prewrite existence guess; Profile-2 `if_seq_no` lowers to `RowMutation.expected_commit_ts` when provided | | Elasticsearch | `POST /{index}/_update/{id}` admitted body subset | gateway MUST run deterministic point-read/evaluate/canonicalize before mutation emission: admitted body is `doc` plus optional one of `upsert` or `doc_as_upsert=true`, plus optional `detect_noop`; `script`, `scripted_upsert`, `_source`, and `retry_on_conflict` are out of subset; existing doc lowers to `RowMutation.operation.patch`, absent doc with admitted upsert branch lowers to `RowMutation.operation.upsert` + `expected_commit_ts=0`, and `detect_noop=true` with unchanged merged row becomes gateway-local `noop` with no canonical mutation | | Elasticsearch | write OCC params `if_seq_no` / `if_primary_term` (Profile 2 subset) | `if_seq_no` lowers to canonical `RowMutation.expected_commit_ts`; `if_primary_term=1` MUST be accepted as SDK-compat pass-through; `if_primary_term>1` MUST fail closed (`SF_VERSION_CONFLICT`) | | Elasticsearch | `POST/GET /{index}/_count` (Profile 2) | canonical count query (`search_payload.aggregations=[{name:\"count\", metric:{builtin_function:AGGREGATE_FUNCTION_COUNT}}]`, response uses `success.aggregations` tree with empty `hits`); pathless `GET/POST /_count` remains fail-closed | | Elasticsearch | `aggs.*.value_count` subset (Profile 2) | gateway lowers each `value_count(field=X)` metric to `AggregationRequest{metric{builtin_function:AGGREGATE_FUNCTION_COUNT, field:\"X\"}}`; missing/empty `field` is malformed (`SF_BAD_REQUEST`); execution semantics are value-count (count rows with non-null `X`) | | Elasticsearch | `aggs.*.terms.missing` subset (Profile 2) | gateway lowers to `TermsAggregation.missing_value_replacement`; omitted source `missing` keeps canonical drop-missing behavior, and type-incompatible `missing` payload is rejected by canonical query admission | | Elasticsearch | nested field sort subset | gateway lowers `sort[].nested.path` / `sort[].nested.filter` to `FieldSortTarget.nested_context`; nested leaf sorts require explicit nested context, root-field sorts MUST reject stray nested context, and `_id` inside nested sort continues to map to canonical deterministic nested tie-break semantics | | Elasticsearch | `nested` / `reverse_nested` aggregation subset | gateway lowers to `AggregationRequest.aggregation_type.nested` / `reverse_nested` context-switch nodes with recursive `sub_aggregations` | | Elasticsearch | `aggs.*.top_hits.highlight` subset (Profile 2) | gateway lowers per-top-hits highlight options to `TopHitsAggregation.text_options`; when source omits top-hits highlight branch, canonical execution inherits top-level `fetch_spec.text_options` | | Elasticsearch | `aggs.*.composite` + `after` subset | gateway lowers `sources[]` to `CompositeAggregation.sources[]` (order-preserving), lowers `after` object to typed `CompositeAggregation.after_key`, and reconstructs response `key`/`after_key` from `Bucket.composite_key.fields` + `BucketResult.composite.after_key.fields` | | Elasticsearch | `POST /{index}/_update_by_query` subset | gateway compiles subset query to canonical selector + deterministic ordered patch mutations (`doc_id asc`); selector materialization MUST also retain the selected current-row version marker for each chosen row so emitted mutations can carry deterministic compare-and-set fences; lowering MUST enforce two-phase probe cap (`fetch_spec.limit=min(caller_requested_limit,max_entries_per_batch,effective_max_result_window)+1`, `track_total_hits`, `caller_requested_limit=source size when provided else +inf`) and fail closed with `SF_TOO_MANY_OPERATIONS` when probe reaches the clamped cap; source `conflicts=abort` MUST map to fail-fast canonical execution, while `conflicts=proceed` MUST map to per-entry continuation with deterministic rejected-outcome accounting; ES response reconstruction MUST derive `updated` from admitted patch-mutation count, `version_conflicts` from rejected compare-and-set outcomes, and `failures[]` from non-conflict rejects | | Elasticsearch | `POST /{index}/_delete_by_query` subset | gateway compiles subset query to canonical selector + deterministic ordered delete mutations (`doc_id asc`); selector materialization MUST also retain the selected current-row version marker for each chosen row so emitted deletes can carry deterministic compare-and-set fences when the source conflict policy requires them; lowering MUST enforce two-phase probe cap (`fetch_spec.limit=min(caller_requested_limit,max_entries_per_batch,effective_max_result_window)+1`, `track_total_hits`, `caller_requested_limit=source size when provided else +inf`) and fail closed with `SF_TOO_MANY_OPERATIONS` when probe reaches the clamped cap; source `conflicts=abort` MUST map to fail-fast canonical execution, while `conflicts=proceed` MUST map to per-entry continuation with deterministic rejected-outcome accounting; ES response reconstruction MUST derive `deleted` from authoritative delete apply-result counts plus `version_conflicts` from rejected compare-and-set outcomes, and `failures[]` from non-conflict rejects | | Elasticsearch | `highlight` subset (Profile 2) | gateway normalizes accepted highlight options to `fetch_spec.text_options`; Search Fabric generates snippets via `pre_tag/post_tag` and returns `QueryHit.highlights`; gateway only performs protocol assembly/source-wire adaptation | | Elasticsearch | deep/complex aggregation request (`aggs` tree depth or shape exceeds gateway policy) | gateway enforces depth + defensive shape budgets (node/fanout/bucket); invalid trees are rejected before Query Broker forwarding with canonical `SF_UNSUPPORTED_QUERY_FEATURE` and ES-compatible `parsing_exception`/`search_phase_execution_exception` body projection; truncation branch emits `success.aggregation_truncation` + per-bucket `doc_count_error_upper_bound` | | Elasticsearch | pathless `POST /_msearch` header items | each header item MUST include explicit `index`; missing `index` fails closed (`SF_INDEX_TARGET_REQUIRED`) before canonical fan-out | | Internal SRE/Gateway Tooling | privileged digest probe request | internal debug gateway canonicalizes payload and calls `InternalDebugService.DebugDigestProbe`; success branch returns `gateway_normalized_digest` + `service_attested_digest` (and optional caller-vs-gateway `normalization_diffs[]`), mismatch returns `409 SF_DEBUG_DIGEST_MISMATCH` with digest fields in `ErrorStatus.diagnostics` | | Elasticsearch | non-atomic `/_bulk` item statuses, including `update` action subset | Ingest Gateway sets `policy.continue_on_error=true`; direct `index` / `create` / `delete` items each map to one `ROW_MUTATION`, while `update` items use the same deterministic point-read/evaluate/canonicalize subroutine as `POST /{index}/_update/{id}` and therefore emit at most one `ROW_MUTATION` or one gateway-local `noop` outcome; Search Fabric returns deterministic `MutationAck.success.item_outcomes[]`, including authoritative post-apply result kinds for successful row mutations; gateway MUST reconstruct Elasticsearch wire shape with top-level `errors` + per-item `status`/`error`; gateway MUST derive Flow-B stream identity from retry-stable idempotency identity (`Idempotency-Key` when provided, otherwise deterministic payload-hash-derived `batch_id`) rather than `RequestMeta.idempotency_key`; undeclared fields remain governed by the admitted strict/ignore policy, and gateways MUST NOT perform implicit schema creation inside bulk write handling | | Elasticsearch | external doc ID token | gateway MUST treat incoming ID as wire-level string token and validate lossless base-10 uint64 parse (Go-equivalent `strconv.ParseUint(id, 10, 64)`); parse failure/out-of-range MUST fail closed (`SF_INVALID_DOC_ID`) and MUST NOT normalize into canonical `DocumentID.id_value` | | MySQL/TiDB | source-row canonical doc-id normalization | when schema declares `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_UINT64`, source-field values MUST normalize losslessly into canonical `DocumentID.id_value.u64`; non-normalizable/out-of-range source values are ingest-time schema violations (`422 SF_SCHEMA_VIOLATION`), not wire-token parse failures | | MySQL/TiDB | federated-table `INSERT ... VALUES`, `REPLACE`, and point-target `UPDATE` / `DELETE` on collections whose authoritative descriptor declares `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` | TiDB direct-write gateway lowers one admitted SQL statement to one canonical `MutationBatch` on `IngestGatewayService.StreamMutations`; caller identity MUST use `RequestMeta.client_kind=CLIENT_KIND_TIDB_WRITE_GATEWAY` and `RequestMeta.client_id=tidb-write-gateway`; `MutationBatch.stream_identity` MUST be derived from a retry-stable logical SQL statement identity with `source_epoch=1` and deterministic row-order `source_seq`; caller MUST send the active `schema_generation_token`; every lowered row MUST send `RowMutation.commit_ts=0`; and the batch MUST use `policy.ingest_mode=streaming`, `policy.durability=wal_chunk_durable`, `policy.visibility=wait_for_visible`, `policy.atomicity=ATOMICITY_POLICY_ALL_OR_NOTHING`, and `policy.continue_on_error=false`, so any row-level rejection rejects the whole statement batch with no row-mutation visibility | | MySQL/TiDB | federated-table SQL features that require TiKV-backed local transaction/locking semantics | explicit multi-statement transactions that write federated tables, mixed local-TiKV + federated-table write transactions, locking reads (`FOR UPDATE` / pessimistic locking), savepoints/XA, `INSERT ... SELECT`, and non-point `UPDATE` / `DELETE` forms that TiDB cannot lower into a deterministic bounded `doc_id` mutation plan are outside the active subset and MUST fail closed as `SF_UNSUPPORTED_TABLE_OPERATION` | | Turbopuffer | external account + `{namespace}` token | external Turbopuffer account resolves to canonical `namespace_id` outside the current design scope (external control plane responsibility); within that resolved namespace, Turbopuffer `{namespace}` token maps to canonical `CollectionDescriptor.collection_name`, and gateway MUST resolve/ensure the concrete `CollectionRef` inside that namespace before schema/query/write/delete routes | | Turbopuffer | `rank_by`, `filters`, `top_k`, `consistency.level` | gateway lowers `rank_by` to exactly one admitted canonical primary-ranking branch (`VectorClause` for vector ranking or `TextClause.mode_override.plain_match` for lexical ranking), lowers `filters` to `search_payload.global_filter`, and lowers `top_k` to `search_payload.global_candidate_topk`; `strong` MUST lower to `CONSISTENCY_MODE_STRICT` with a gateway-acquired fresh `read_ts > 0`, `eventual` MUST lower to `CONSISTENCY_MODE_BEST_EFFORT` with `read_ts=0`; non-lowerable `rank_by` branches fail closed | | Turbopuffer | `POST/GET /v1/namespaces/{namespace}/schema` | gateway resolves the caller account to canonical `namespace_id` outside this design, resolves/creates the target collection through typed collection catalog operations (`ListCollections` / `GetCollectionMetadata` / `PutCollectionMetadata` as needed), canonicalizes the schema map into typed control-plane payload (`FieldSchema.config` + `FieldSchema.subfields`, optional `CollectionSettings` physical settings, synthesized `DocIDMapping{source_field=\"id\", source_field_type=SCALAR_FIELD_TYPE_UINT64, immutable=true, unique=true, generation_policy=DOC_ID_GENERATION_POLICY_CALLER_REQUIRED}` in the current profile), preserves scalar-vs-array cardinality through `FieldSchema.config.capabilities.multi_valued`, and uses `PutCollectionSchema` / `GetCollectionSchema` on the resolved `CollectionRef` | | Turbopuffer | query/write `distance_metric` | when request carries schema/update intent, gateway canonicalizes admitted metric into vector schema (`FieldSchema.config.field_definition.vector.distance_metric`); otherwise metric is validation-only against schema-declared vector metric and MUST be elided from canonical query/mutation payload after validation; mismatches fail closed | | Turbopuffer | `GET /v1/namespaces` | gateway lowers to canonical `ListCollections` inside the externally resolved account namespace, projects `NamespaceSummary.id` from `CollectionDescriptor.collection_name`, and keeps pagination/token semantics aligned with `ListCollections.success.next_page.token` | | Turbopuffer | `GET /v1/namespaces/{namespace}/metadata` | gateway resolves `{namespace}` to canonical collection lookup inside the externally resolved namespace/account, reads typed collection metadata from `GetCollectionMetadata`, reads schema from `GetCollectionSchema`, and MUST NOT synthesize `schema` / `approx_*` / `created_at` / `updated_at` / `encryption` / `index` from gateway-local guesses; the route requires `GetCollectionMetadata.success.descriptor.completeness.ready`, and `completeness.publishing` MUST fail closed with `SF_CATALOG_UNAVAILABLE` plus canonical completeness details | | Turbopuffer | `GET /v1/namespaces/{namespace}/hint_cache_warm` | gateway-local operational hint endpoint; it MAY enqueue cache-warm work but MUST NOT mutate canonical schema state or query digests | | Turbopuffer | `POST /v1/namespaces/{namespace}/_debug/recall` | gateway-local privileged debug workflow that fans out to canonical queries plus exhaustive comparison logic; no public canonical RPC is added solely for this wire path | | Turbopuffer | `upsert_condition` / `patch_condition` / `delete_condition` | gateway MUST normalize these wire-only condition DSLs with the same admitted predicate subset used for `filters`, then perform deterministic pre-read/evaluate gating over the current value of each explicitly targeted row before canonical mutation emission; rows failing the condition emit no canonical mutation and do not contribute to `rows_affected` / affected-id reconstruction | | Turbopuffer | `patch_rows` / `patch_columns` | gateway lowers to deterministic canonical patch mutation plan (`RowMutation.operation.patch`) on canonical `doc_id`; explicit source null-assignment intent MUST be lowered to `FieldData.atomic_value.null_value` | | Turbopuffer | `patch_by_filter` / `delete_by_filter` + `patch_by_filter_allow_partial` / `delete_by_filter_allow_partial` | gateway lowers filter selector to deterministic `doc_id asc` mutation/delete plan; lowering MUST enforce two-phase probe cap (`fetch_spec.limit=min(caller_requested_limit,max_entries_per_batch,effective_max_result_window)+1`, `track_total_hits`, `caller_requested_limit=source size when provided else +inf`); `*_allow_partial=true` MAY apply only the first deterministic prefix of that sorted match set and return `rows_remaining=true`, while `false` / omitted fail-close on over-cap branches | | Turbopuffer | `copy_from_namespace` | gateway runs snapshot-consistent source read and lowers to canonical snapshot + ingest flow | | Turbopuffer | `return_affected_ids` | gateway reconstructs affected-id lists from deterministic `MutationAck.success.item_outcomes[]` | | Turbopuffer | `DELETE /v2/namespaces/{namespace}` | gateway resolves collection by `collection_name` within the externally resolved account namespace and lowers lifecycle deletion to canonical `DropCollection`; absent collection follows route-level idempotent/not-found projection rules, and no namespace-level delete primitive is required in the canonical service | | Turbopuffer | write `encryption` | gateway treats write-request `encryption` as collection-descriptor intent, not row-mutation payload: it MUST be canonicalized through typed collection metadata ensure/apply (`PutCollectionMetadata.name_binding`) before mutation emission, MUST exactly match existing collection encryption when the collection already exists, and MUST NOT be forwarded as a dedicated data-plane proto field | | Turbopuffer | `disable_backpressure` | gateway lowers to `policy.ingest_mode` (`true -> INGEST_MODE_BATCH`, `false/omitted -> INGEST_MODE_STREAMING`); when lowering to `INGEST_MODE_BATCH`, gateway MUST also keep the current admitted batch-lane policy pair (`policy.visibility=wait_for_visible`, `policy.durability=ingest_ack`); when lowering to `INGEST_MODE_STREAMING` for `false` / omitted, gateway MUST keep the current admitted streaming policy tuple (`policy.visibility=wait_for_visible`, `policy.durability=wal_chunk_durable`); server hard limits remain authoritative and MAY still return `SF_RATE_LIMITED` | | Turbopuffer | `QueryConfig.vector_encoding` (`float`/`base64`) | gateway MUST apply as response-side vector serialization policy for `Row.vector`; canonical query input parsing is independent of this flag | | Turbopuffer | `include_attributes` / `exclude_attributes` | gateway normalizes only attribute projection (non-vector fields) to canonical `fetch_spec.attribute_projection` with deterministic include/exclude precedence | | Turbopuffer | `aggregate_by`, `group_by`, `limit` object form | gateway lowers source aggregation DSL to canonical `search_payload.aggregations`; `group_by` order is preserved through bucket grouping (`CompositeAggregation.sources[]` when multi-key grouping is used), `limit.total` lowers to outer result budget, and `limit.per.limit` lowers to per-group `TopHitsAggregation.limit`; mismatched `group_by` vs `limit.per.attributes` is invalid | | Turbopuffer | projected `Row.vector` return | active profile constrains row-return queries to collections whose only admissible vector field is the schema field named `vector`; gateway applies field-level `fetch_spec.vector_projections["vector"].format` (`VECTOR_PAYLOAD_FORMAT_EXACT_FLOAT` / `VECTOR_PAYLOAD_FORMAT_EXACT_F16` / `VECTOR_PAYLOAD_FORMAT_QUANTIZED_INT8` / `VECTOR_PAYLOAD_FORMAT_QUANTIZED_BINARY`) when policy allows; exact-float path may trigger dynamic payload/window limit tightening | | Turbopuffer | `Row.vector` output | gateway maps canonical columnar projection for the single admitted vector field `vector` to wire-level `Row.vector`; collections or query shapes that would require multiple vector fields in one row-return response are out of profile and MUST fail closed | | Turbopuffer | `aggregation_groups` response | gateway reconstructs grouped response objects deterministically from canonical bucket results (and per-group `TopHitsResult` payloads when present) | | Turbopuffer | `multiQuery` | ordered fan-out to independent canonical query executions; aggregate response preserves input query order, and outer `consistency.level=strong` MUST use one shared gateway-acquired strict snapshot (`read_ts`) across all child queries | | Turbopuffer | `explain_query.plan_text` | gateway-local textual rendering of the admitted canonical query/aggregation plan; this response artifact MUST NOT change canonical execution semantics or digest | | Turbopuffer | required top-level query billing/performance envelopes | gateway preserves required top-level envelopes and reconstructs required performance fields deterministically from canonical execution + gateway telemetry; nested billing object shapes remain opaque/non-contract to canonical query semantics | | Turbopuffer | external `Id` branches (`integer` / `string` / `uuid`) | active profile admits decimal-string IDs and non-negative integer IDs when losslessly parseable as canonical uint64; negative integers, floating-point numerics, `uuid`, and other non-lossless/non-decimal branches MUST fail closed (`SF_INVALID_DOC_ID`) | Elasticsearch PIT compatibility scope in active profile: - `POST /{index}/_pit` and `DELETE /_pit` remain out of active Elasticsearch Profile-2 scope and are deferred to Profile 3. - `search_after + PIT` token coupling is also deferred to Profile 3; active Profile-2 `search_after` uses canonical continuation token flow only. - active Profile-2 does not expose native stateless `search_after` sort-values arrays as canonical continuation input; continuation MUST use gateway-issued `continuation_page.next_page_token`. - until Profile 3 is active, ES PIT-route attempts MUST be rejected by profile mapping (`SF_UNSUPPORTED_ENDPOINT` / `SF_UNSUPPORTED_OPTION`) per `docs/specs/core/error-codes.md` precedence. Aggregation pagination support (composite `after_key`, active profile): - Elasticsearch composite aggregation continuation (`after`) is supported through canonical `CompositeAggregation.after_key`. - `after` object lowering MUST preserve deterministic source-name matching against `CompositeAggregation.sources[]`. - response-side `BucketResult.composite.after_key.fields` MUST be gateway-raised as Elasticsearch-compatible outer `after_key`. - any response that emits non-empty `BucketResult.composite.after_key` MUST also include canonical `pagination.composite_page.token` plus `composite_page.expires_at_unix_ms` so the next composite page can be requested deterministically. - requests carrying non-empty `CompositeAggregation.after_key` MUST carry the matching `fetch_spec.composite_page.next_page_token`; gateway/runtime may validate the request-side `after` object against the token-bound cursor, but the token remains authoritative continuation state. - this does not expose Elasticsearch PIT endpoints; gateway/runtime uses canonical composite-page token lifecycle internally for snapshot pinning and cursor continuity. - for `CompositeAggregation`, engine MUST skip non-composite uncertainty carriers entirely: the composite branch returns only `BucketResult.composite.{buckets,after_key}` and MUST NOT invent `sum_other_doc_count` / `doc_count_error_upper_bound` fields. Turbopuffer projection normalization rules: - `include_attributes=true` lowers to `fetch_spec.attribute_projection.return_all`; `exclude_attributes` lowers to `return_all.exclude_fields`. - `include_attributes=false` lowers to `fetch_spec.attribute_projection.return_none`; `exclude_attributes` then has no effect. - `include_attributes=[...]` lowers to `fetch_spec.attribute_projection.return_explicit`; when `exclude_attributes` is also present, gateway MUST perform set subtraction before emitting final `return_explicit.include_fields`. - these rules apply to attribute fields only; vector inclusion is controlled independently by vector-return policy. - normalized attribute projection field set MUST be deduplicated with byte-ascending deterministic ordering before canonical request emission. Turbopuffer schema/query/response lowering rules: - schema normalization MUST synthesize canonical `DocIDMapping{source_field=\"id\", source_field_type=SCALAR_FIELD_TYPE_UINT64, immutable=true, unique=true, generation_policy=DOC_ID_GENERATION_POLICY_CALLER_REQUIRED}` from the Turbopuffer top-level `id` carrier in the current profile; `id` is not a user-declared attribute entry in the schema map. - `rank_by` MUST lower to exactly one admitted canonical primary-ranking branch; source branches that imply multiple concurrent ranking strategies or non-deterministic gateway heuristics are outside active profile and MUST fail closed. - `top_k` lowers to `search_payload.global_candidate_topk`; `limit` continues to govern response/bucket budget and MUST NOT silently overwrite `top_k`. - `filters` MUST lower to canonical `search_payload.global_filter`; source filter branches that cannot be represented as canonical predicate ASTs are rejected before admission. - `aggregate_by` MUST lower to canonical `search_payload.aggregations`; non-lowerable DSL branches are rejected fail-closed before query admission. - `group_by` order MUST be preserved when lowering grouped requests; multi-key grouping lowers to `CompositeAggregation.sources[]` in request order. - `limit` integer form lowers to `fetch_spec.limit`. - `limit` object form lowers to canonical grouped-budget controls: - `total` lowers to outer row/group budget (`fetch_spec.limit` or bucket-strategy limit such as `CompositeAggregation.limit`, depending on the lowered query shape), - `per.attributes` defines group-key order for grouped row output and MUST match `group_by` when both are provided, - `per.limit` lowers to `TopHitsAggregation.limit`. - `distance_metric` rules are split by intent: - when request also carries schema/update intent, admitted metric MUST be canonicalized into `FieldSchema.config.field_definition.vector.distance_metric`, - when request is a pure query or mutation request against an existing schema, admitted metric is a validation-only assertion against the schema-declared vector metric and MUST be elided after validation, - validation-only `distance_metric` MUST NOT perturb `batch_fingerprint` or canonical query digest. - `QueryConfig.vector_encoding` is response-side vector serialization policy only and MUST NOT alter canonical query-input parsing or ranking semantics. - gateway response reconstruction MUST build `aggregation_groups[]` from canonical bucket results (and per-group `TopHitsResult` payloads when present) deterministically for the same admitted query. - `explain_query.plan_text` is gateway-local textual rendering over the admitted canonical plan and MUST NOT perturb canonical digest or execution semantics. Elasticsearch write canonical lowering rules: - single-document `POST /{index}/_update/{id}` admitted subset: - request body MUST contain `doc` object. - request body MAY also contain exactly one admitted upsert branch: - `upsert` object, or - `doc_as_upsert=true`. - request body MAY contain `detect_noop` boolean (`true` default semantics preserved by gateway evaluation). - `script`, `scripted_upsert`, `_source`, `retry_on_conflict`, and body shapes without `doc` are outside the active subset and MUST fail closed before canonical emission. - gateway MUST perform deterministic point-read/evaluate/canonicalize under the active schema-generation fence before deciding the canonical mutation plan. - point-read evaluation rules: - if current document is absent and no admitted upsert branch is present, gateway MUST return deterministic not-found (`404`) and emit no canonical mutation. - if current document is present, gateway MUST compute the merged post-update row using `doc` as patch intent. - if `detect_noop=true` and the merged row is byte-for-byte equivalent to the admitted current row after canonical field normalization, gateway MUST emit no canonical mutation and MUST reconstruct Elasticsearch `result=noop`. - if current document is present and mutation is not a noop, gateway MUST emit exactly one canonical `ROW_MUTATION` with `RowMutation.operation.patch`. - if current document is absent and admitted upsert branch is present, gateway MUST emit exactly one canonical `ROW_MUTATION` with `RowMutation.operation.upsert`; `doc_as_upsert=true` uses `doc` as the upsert payload, while explicit `upsert` uses the `upsert` object. - optimistic-write binding: - when source request carries Profile-2-admitted `if_seq_no`, gateway MUST validate it against the point-read version and lower the same value to `RowMutation.expected_commit_ts`. - when source request omits `if_seq_no`, gateway MUST still lower point-read version as `RowMutation.expected_commit_ts` for emitted `patch` mutations so the gateway-local read-modify-write decision remains compare-and-set stable. - absent-document upsert/create branches MUST lower `RowMutation.expected_commit_ts=0`. - response reconstruction: - emitted `patch` -> Elasticsearch `result=updated` - emitted `upsert` on absent document -> Elasticsearch `result=created` - noop branch -> Elasticsearch `result=noop` - gateway MUST set `MutationBatch.policy.continue_on_error=true` for ES bulk requests. - undeclared fields remain under the admitted strict/ignore policy for these routes; canonical write admission does not auto-add schema from write payloads. - incremental per-item schema mutation inside the same bulk request is forbidden in this profile because it can break client retry/idempotency assumptions. - bulk `update` item admitted subset: - metadata line MUST target exactly one concrete `_id` and resolved concrete `_index`. - source line uses the same admitted body subset and point-read/evaluate/canonicalize rules as single-document `POST /{index}/_update/{id}`. - profile-2 OCC metadata (`if_seq_no`, `if_primary_term=1`) follows the same lowering rules as the single-document route. - multiple `update` actions targeting the same concrete `(_index, _id)` within one admitted bulk request are outside this subset; later duplicates MUST be reconstructed as deterministic item-level rejects (`400 SF_UNSUPPORTED_OPTION`) rather than silently reinterpreted. - each admitted NDJSON action/source pair maps to at most one canonical `ROW_MUTATION` and keeps deterministic input order; item-local noop evaluation is gateway-local and emits no canonical mutation. - per-item response reconstruction MUST use `MutationAck.success.item_outcomes[]` (entry index + canonical result/error code). - for successful row-mutation items, gateway MUST use authoritative `apply_result` from `MutationAck.success.item_outcomes[]` rather than reconstructing create/delete existence-sensitive results from a gateway-local prewrite guess. - gateway HTTP response for accepted bulk requests MUST use Elasticsearch bulk wire shape: - top-level `errors` MUST be `true` iff at least one reconstructed item has failure status (`>=400`), otherwise `false`. - top-level `items[]` MUST preserve request action order. - each item MUST include Elasticsearch-compatible `status` and, on failure, Elasticsearch-compatible `error` object. - bulk `update` success items MUST reconstruct Elasticsearch `result` as `updated`, `created`, or `noop` from the same per-item canonicalization decision used before mutation emission. - deterministic item-status reconstruction baseline: - canonical request/contract/schema violations -> `400`, - not-found branches -> `404`, - conflict branches (`SF_VERSION_CONFLICT`, `SF_IDEMPOTENCY_CONFLICT`, `SF_OWNERSHIP_CONFLICT`, `SF_DEBUG_DIGEST_MISMATCH`) -> `409`, - throttling -> `429`, - backend/timeout branches -> `503`/`504` by canonical mapping. - partial-success bulk responses (`errors=true` with mixed item statuses) MUST remain HTTP `200` and MUST NOT be collapsed to whole-request failure. - top-level admission/capacity rejects (for example auth/ownership/rate-limit failures, payload-too-large, or max-operation overflow) MUST short-circuit as whole-request failures and MUST NOT be reconstructed as per-item statuses. - framing-level malformed batch errors (for example invalid NDJSON shape) remain whole-request failures and MUST use canonical error precedence. - if an item references undeclared fields while strict mode is active, the impacted item MUST be reconstructed as a deterministic schema failure rather than being deferred to later schema inference. - Elasticsearch by-query conflict-mode mapping: - `conflicts=abort` MUST lower to canonical fail-fast execution (`MutationBatch.policy.continue_on_error=false`) over the deterministic `doc_id asc` mutation plan; the first compare-and-set rejection stops further apply. - `conflicts=proceed` MUST lower to canonical per-entry continuation (`MutationBatch.policy.continue_on_error=true`) over the deterministic `doc_id asc` mutation plan so compare-and-set rejections are preserved in `MutationAck.success.item_outcomes[]` and counted into `version_conflicts`. - for both modes, selector materialization version markers remain the source of emitted `RowMutation.expected_commit_ts`. - no cross-batch atomicity is provided for gateway-lowered bulk/by-query execution: - client-observable intermediate states across multiple batches are allowed by design. - for stronger consistency requirements, caller MUST use single-batch write-then-read flows or rely on underlying storage-engine transactional snapshots. Turbopuffer write canonical lowering rules: - `upsert_condition` / `patch_condition` / `delete_condition`: - these are wire-only gateway evaluation inputs; current proto carries no dedicated row-condition field. - gateway MUST normalize them with the same admitted predicate subset and canonical field typing used for query `filters`. - gateway MUST resolve the explicit target row set first (`upsert_rows`, `upsert_columns`, `patch_rows`, `patch_columns`, `deletes`), then perform deterministic point-read evaluation against each target row's current value before canonical mutation emission. - a physically absent row is evaluated as missing/null field state. - for selector-driven branches, materialization output MUST retain `(doc_id, selected_version_commit_ts)` so later canonical mutations can produce deterministic compare-and-set conflicts instead of blind best-effort apply. - only rows whose current state satisfies the admitted condition may emit canonical mutations. - rows failing the condition MUST be skipped deterministically: they emit no canonical mutation, do not contribute to `rows_affected`, and do not appear in `upserted_ids` / `patched_ids` / `deleted_ids`. - condition parse failures, unsupported predicate branches, or schema-incompatible condition typing MUST fail closed before mutation emission. - `patch_rows` / `patch_columns`: - gateway MUST lower to deterministic canonical patch mutations (`RowMutation.operation.patch`) on canonical `doc_id`. - explicit ID list cardinality MUST be bounded by `max_entries_per_batch`; over-limit requests fail closed as `SF_TOO_MANY_OPERATIONS`. - patch merge semantics are preserved as patch intent; canonical ingest applies partial updates without requiring gateway-side read-modify-write loops. - if source patch payload requests explicit clear/remove for a field, gateway MUST emit `FieldData.atomic_value.null_value = NULL_VALUE_NULL` for that field (including vector/list/nested typed fields) instead of empty typed payload surrogates. - `delete_by_filter`: - gateway MUST compile filter to canonical selector query, materialize matching `doc_id`s, sort by `doc_id asc`, then emit deterministic delete mutations. - `delete_by_filter_allow_partial=false` requires fail-closed on first non-retryable delete branch. - `delete_by_filter_allow_partial=true` allows deterministic partial completion only over the ordered materialized prefix; if the probe hits `probe_limit`, gateway MAY apply the first `probe_limit-1` sorted `doc_id`s, MUST set `rows_remaining=true`, and MUST NOT advertise a resume token because current OpenAPI wire shape has no such field. - `patch_by_filter`: - gateway MUST compile to selector query + deterministic ordered patch mutation plan (`doc_id asc`) similar to `delete_by_filter`. - gateway MUST enforce the same two-phase probe contract used by `_update_by_query` (`fetch_spec.limit=min(caller_requested_limit,max_entries_per_batch,effective_max_result_window)+1`, `track_total_hits`, `caller_requested_limit=source size when provided else +inf`). - if selector materialization hits probe clamp (`min(caller_requested_limit,max_entries_per_batch,effective_max_result_window)+1`) and `patch_by_filter_allow_partial=false` (or omitted), request MUST fail closed as `400 SF_TOO_MANY_OPERATIONS` with selector-materialization clamp details (`limit_scope=selector_materialization_limit`, `configured_limit`, `requested_value`, `effective_constraint_source`). - if selector materialization hits probe clamp and `patch_by_filter_allow_partial=true`, gateway MAY apply only the first `probe_limit-1` sorted `doc_id`s, MUST set `rows_remaining=true`, and MUST keep `rows_patched` / `patched_ids` limited to that admitted prefix. - `schema`: - Turbopuffer write-route inline `schema` is semantically identical to `POST /v1/namespaces/{namespace}/schema` for the same external namespace token. - gateway MUST resolve the external account to canonical `namespace_id` outside this design, ensure or resolve the target collection through `PutCollectionMetadata.name_binding` / `GetCollectionMetadata.lookup.name_binding` using `CollectionNameBinding{namespace_id=, collection_name=}`, canonicalize inline `schema` through the same normalization rules as the dedicated schema endpoint, and issue a fenced `PutCollectionSchema.scope.collection=` before planning or emitting any row mutations. - inline schema apply uses the same retry-stable outer write identity as the surrounding write request; identical replay of the same inline-schema payload MUST be replay-safe, while replay under the same identity with a different canonicalized schema payload MUST fail closed (`SF_IDEMPOTENCY_CONFLICT` or `SF_VERSION_CONFLICT` by precedence). - if inline schema apply fails, gateway MUST emit no canonical row mutations for that request. - if inline schema apply succeeds and later row-level processing enters partial-completion behavior (`continue_on_error`, `patch_by_filter_allow_partial`, `delete_by_filter_allow_partial`, or condition-based skips), the applied schema remains committed; response reconstruction reflects only the emitted mutation outcomes and MUST NOT imply schema rollback. - `copy_from_namespace`: - gateway MUST run snapshot-consistent source read and lower to canonical snapshot + mutation ingest flow for the target collection resolved from the external account + `{namespace}` pair. - object-form source credentials/region are gateway-only and MUST NOT cross canonical gRPC boundary. - `encryption`: - write-request `encryption` is collection-descriptor intent, not row payload. - gateway MUST resolve or ensure the collection descriptor before mutation emission and canonicalize admitted encryption intent through `PutCollectionMetadata.name_binding=CollectionNameBinding{namespace_id=, collection_name=}`. - active admitted subset is omitted or a valid explicit encryption descriptor with exactly one mode selected; current wire-parity subset admits `{cmek:{key_name}}` only, and `key_name` MUST be non-empty. - when the collection already exists, provided encryption MUST exactly match the catalog descriptor; write-route encryption changes are outside the active subset and MUST fail closed. - dedicated encryption data MUST NOT be forwarded as an ingest-plane mutation field; downstream row mutations remain unchanged after descriptor ensure. - `return_affected_ids`: - gateway MUST reconstruct `upserted_ids`/`patched_ids`/`deleted_ids` from deterministic mutation outcomes (`MutationAck.success.item_outcomes[]`). - for explicit-ID source operations, emitted affected-ID arrays MUST preserve source-request ordering for the realized operation subset. - for selector-driven operations (`patch_by_filter`, `delete_by_filter`), emitted affected-ID arrays MUST preserve realized deterministic mutation-plan ordering (`doc_id asc`). - `disable_backpressure`: - gateway MUST lower `disable_backpressure=true` to `MutationBatch.policy.ingest_mode=INGEST_MODE_BATCH`. - when lowering `disable_backpressure=true`, gateway MUST also keep the current admitted batch-lane policy pair: `MutationBatch.policy.visibility=VISIBILITY_POLICY_WAIT_FOR_VISIBLE` and `MutationBatch.policy.durability=DURABILITY_POLICY_INGEST_ACK`. - gateway MUST lower `disable_backpressure=false` (or omitted) to `MutationBatch.policy.ingest_mode=INGEST_MODE_STREAMING`. - when lowering `disable_backpressure=false` (or omitted), gateway MUST keep the current admitted streaming policy tuple: `MutationBatch.policy.visibility=VISIBILITY_POLICY_WAIT_FOR_VISIBLE` and `MutationBatch.policy.durability=DURABILITY_POLICY_WAL_CHUNK_DURABLE`. - hard admission limits remain authoritative for both modes and server MAY still return `SF_RATE_LIMITED`. Turbopuffer vector return rules: - canonical API returns vectors via typed field projection (`FieldData.vector_value` as `VectorValue{data oneof}`); gateway encodes/decodes base64 at the edge. - `include_attributes`/`exclude_attributes` MUST NOT control vector inclusion. - vector inclusion/exclusion is a separate gateway decision path and MUST remain deterministic for the same admitted query shape. - active-profile row-return subset admits exactly one vector field and that field name MUST be `vector`; collections exposing any additional vector field are out of Turbopuffer row-return profile scope and MUST fail closed on requests that require `Row.vector` reconstruction. - gateway MUST source wire-level `Row.vector` only from canonical `fetch_spec.vector_projections["vector"]`; no implicit multi-field vector selection or best-effort merging is allowed. - if projected vector fields are requested but vector storage is unavailable, strict-mode requests MUST fail closed with `SF_UNSUPPORTED_QUERY_FEATURE`. - if projected vector payload is unavailable, strict-mode requests MUST fail closed with `SF_SCHEMA_VIOLATION`. - if projected vector payload dimension mismatches schema, strict-mode requests MUST fail closed with `SF_VECTOR_DIMENSION_MISMATCH`. - `VectorValue` payload typing is branch-discriminated only: `f16_vector`/`float_vector`/`int8_vector`/`binary_vector`; there is no secondary format discriminator inside payload. - `VectorPayloadFormat` is retained for projection intent (`VectorProjectionSpec.format`) and MUST NOT be treated as payload-branch discriminator at runtime. - projection choices are part of canonical digest through (`fetch_spec.attribute_projection` + canonicalized `fetch_spec.vector_projections`). Turbopuffer response reconstruction rules: - successful write routes MUST always emit `WriteResult.status=\"OK\"`; whole-request failures use canonical error projection instead of `WriteResult`. - `rows_upserted`, `rows_patched`, and `rows_deleted` MUST be reconstructed from the deterministic admitted source operation plan by operation kind, not from raw `MutationAck.success.metrics`, so replay-safe retries reproduce the same wire-visible counters. - `rows_affected` MUST equal the sum of the operation-kind counters emitted in that response. - `rows_remaining=true` is reserved for filter-based partial-completion flows where the gateway intentionally stops after an ordered admitted prefix; current OpenAPI wire shape does not expose a resume token, so `rows_remaining` is only a boolean continuation hint. Otherwise it MUST be omitted or `false`. - `upserted_ids`, `patched_ids`, and `deleted_ids` are emitted only when `return_affected_ids=true`. - For explicit-ID source operations, those arrays MUST preserve the original source-request order for the realized operation subset. - For selector-driven operations (`patch_by_filter`, `delete_by_filter`), those arrays MUST preserve the realized deterministic mutation-plan order (`doc_id asc`). - `message` is required by OpenAPI but is gateway-local explanatory text; callers MUST NOT treat it as a semantic contract input. - query performance envelope fields are deterministic gateway projections: - `server_total_ms`: server-side end-to-end time including queue/concurrency wait and response assembly, - `query_execution_ms`: canonical execution time excluding queue/concurrency wait; when unavailable, gateway MUST emit the same value as `server_total_ms`, - `cache_hit_ratio`: engine/gateway cache-hit ratio when available, otherwise `0.0`, - `cache_temperature`: derived from `cache_hit_ratio` (`hot` when `>= 0.95`, `warm` when `>= 0.50 && < 0.95`, otherwise `cold`), - `exhaustive_search_count`: exact-scan/brute-force candidate count when available, otherwise `0`, - `approx_namespace_size`: latest admitted collection row-count estimate from `GetCollectionMetadata.success.descriptor.approx_stats.approx_row_count` for the resolved Turbopuffer route namespace token; routes that need this field must observe `descriptor.completeness.ready`, and `completeness.publishing` MUST fail closed with `SF_CATALOG_UNAVAILABLE` rather than falling back to `0`. Cross-ecosystem ID parse guardrails: - wire-level external IDs are admitted as gateway-local ecosystem forms and normalized to canonical `DocumentID.id_value.u64` for internal execution. - admitted parse forms in active profile: - decimal string tokens losslessly parseable as base-10 uint64 (Go-equivalent `strconv.ParseUint(id, 10, 64)`), - non-negative integer numeric tokens (for typed-union APIs such as Turbopuffer `Id.integer`) that are losslessly representable as uint64. - parse failure or out-of-range input MUST emit `SF_INVALID_DOC_ID`. - canonical `DocumentID` remains the dedup/tie-break/pagination identity after parse succeeds. - Elasticsearch/Turbopuffer compatibility gateways MUST enforce strict external-ID normalization contract; non-normalizable/out-of-range IDs fail closed and MUST NOT be normalized. - source-field normalization behavior: - `SCALAR_FIELD_TYPE_UINT64`: value is directly mapped to `DocumentID.id_value.u64` (including `0`); source-field type/range violations at this stage are ingest-time schema violations and MUST fail closed as `422 SF_SCHEMA_VIOLATION`. - `SCALAR_FIELD_TYPE_STRING`: active profile does not admit string-source doc-id mapping; gateway/control plane MUST fail closed with `400 SF_DOC_ID_CONTRACT_VIOLATION`. - `SCALAR_FIELD_TYPE_BYTES`: active profile does not admit bytes-source doc-id mapping; gateway/control plane MUST fail closed with `400 SF_DOC_ID_CONTRACT_VIOLATION`. - uuid and other non-decimal/non-lossless string IDs MUST be rejected (`SF_INVALID_DOC_ID`); negative integers and floating-point numeric ID payloads MUST be rejected as well; lossy hashing/coercion is forbidden. - pagination/search-after continuation, fusion dedup, and tie-break ordering MUST run on canonical `DocumentID`; external ID rendering is a gateway-side response concern. Validation Rules: - `[Rule V-101] Type Support Check`: in current active profile (Phase 1), canonical `DocumentID` exposes only `DocumentID.id_value.u64`. - protobuf field numbers `2` and `3` remain reserved and MUST NOT be repurposed as active branches in this revision. - `[Rule V-102] Null/Zero Check`: `DocumentID` MUST be non-empty. - `DocumentID.id_value.u64 = 0` is valid. - if `id_value` oneof is unset, request is malformed and MUST fail closed with `400 SF_INVALID_ARGUMENT`. Parser canonicalization guardrails: - parser selection (`STANDARD`/`NGRAM`/`MECAB`) is schema/index metadata, not per-query runtime input. - gateway MUST normalize parser settings into control-plane schema payload (for example `FieldSchema.config.field_definition.text.analysis.analyzer`, `FieldSchema.config.field_definition.text.analysis.options`, `FieldSchema.config.field_definition.text.fulltext`, and/or supplemental `FieldSchema.options`/`CollectionSettings.options`) before `PutCollectionSchema`. - canonical parser-family mapping examples: - default parser path (no explicit `WITH PARSER` clause) -> `builtin_standard_v1` - `WITH PARSER NGRAM` -> `builtin_ngram_v1` - `WITH PARSER MECAB` -> `builtin_mecab_ja_v1` - tokenizer-option mapping example: - `WITH PARSER NGRAM` + `ngram_token_size=2` -> `text_analysis.analyzer=builtin_ngram_v1` + `text_analysis.options[\"tokenizer.ngram_token_size\"]=\"2\"` - explicit SQL `WITH PARSER STANDARD` is outside the active MySQL compatibility subset and MUST fail closed at gateway boundary (`SF_UNSUPPORTED_PARSER` or `SF_UNSUPPORTED_OPTION` by precedence); canonical `builtin_standard_v1` is reached only through the default parser path. - query-time parser override through this API is not supported and MUST fail closed at gateway boundary (`SF_UNSUPPORTED_PARSER` or `SF_UNSUPPORTED_OPTION` by precedence). - catalog snapshot used for query admission MUST remain authoritative for parser behavior in effect at the admitted schema generation. Schema canonicalization guardrails: - canonical control-plane schema is the only admissible cross-ecosystem schema truth (`DocIDMapping`, `FieldSchema[]`, optional `CollectionSettings` physical settings); source payloads MUST be normalized before admission. - schema adapters MUST preserve semantic equivalence across ecosystems: - Elasticsearch mapping/type/index flags and analysis settings, - Turbopuffer attribute schema (`filterable`, `regex`, `full_text_search`, `ann`, vector type syntax), - MySQL fulltext/index parser + tokenization/tuning constraints. - MySQL charset/collation directives are outside canonical search semantics; gateways MAY preserve them only as supplemental source metadata (`FieldSchema.options`/`CollectionSettings.options`) for round-trip needs. - `FieldSchema.options` / `CollectionSettings.options` use `map` and MAY carry supplementary ecosystem metadata (`AtomicValue`/`ListValue` only, no vector payload branch). - options metadata MUST NOT be the sole carrier for semantics that already have canonical typed fields. - gateway MUST canonicalize text-analysis semantics into deterministic `analyzer + options` payload (`FieldSchema.config.field_definition.text.analysis`) before schema admission. - analyzer IDs outside the built-in service-supported set MUST fail closed (`SF_UNSUPPORTED_OPTION`). - analyzer override key registry in this revision: - `stopwords` - `stemming` - `tokenizer.ngram_token_size` - override keys MUST be validated against service-supported keys; unsupported keys MUST fail closed with `SF_UNSUPPORTED_OPTION`. - override values in this revision MUST use `MetadataValue.atomic_value.string_value`; `list_value` and non-string atomic branches are malformed for analyzer overrides and MUST fail closed with `SF_SCHEMA_VIOLATION`. - override value grammar in this revision: - `stopwords` / `stemming`: `inherit|enabled|disabled` - `tokenizer.ngram_token_size`: decimal integer in `[1,10]` - value grammar violations or normalization conflicts MUST fail closed with `SF_SCHEMA_VIOLATION`. - canonical normalization rules for override map: - keys MUST be lowercase ASCII and trimmed; mixed-case aliases are invalid (no silent rewrite). - values for enum-like keys (`stopwords`,`stemming`) are read from `MetadataValue.atomic_value.string_value` and MUST be lowercase and trimmed. - integer-like values are read from `MetadataValue.atomic_value.string_value` and MUST be canonical decimal text without leading `+` or leading zeros (except `0` when grammar allows). - serialization for schema digest/equivalence checks MUST use byte-ascending key order. - if a source schema feature cannot be represented in canonical typed schema model, gateway MUST fail closed (`SF_UNSUPPORTED_OPTION`); silent dropping is forbidden. - source charset/collation directives MUST NOT affect canonical schema equivalence, retrieval semantics, or evolution-class decisions; they MAY be carried only as supplemental source metadata in `FieldSchema.options`/`CollectionSettings.options` when required by compatibility round-trip behavior. Determinism requirement: - same canonical request + same visible state => same ranking, dedup result, and continuation behavior. Gateway pre-admission subset enforcement rules: - gateway MUST apply source-profile subset validation before emitting canonical requests to this API. - Elasticsearch pathless `POST /_msearch` items missing `index` MUST fail closed as `SF_INDEX_TARGET_REQUIRED`. - Elasticsearch pathless `GET/POST /_count` MUST fail closed as `SF_INDEX_TARGET_REQUIRED`. - Elasticsearch active-profile `search_after` requests MUST use gateway-issued continuation token (`fetch_spec.continuation_page.next_page_token`); raw source sort-values-array continuation input is out of subset and MUST fail closed. - aggregation names violating `^[a-zA-Z0-9_.-]{1,128}$` MUST be rejected at gateway pre-admission (`SF_BAD_REQUEST`); invalid names MUST NOT reach compute layer. - Elasticsearch `update_by_query`/`delete_by_query` subset paths MUST reject unsupported script/task/throttle options before canonical emission. - Elasticsearch `update_by_query` lowering MUST enforce two-phase probe clamping (`fetch_spec.limit=min(caller_requested_limit,max_entries_per_batch,effective_max_result_window)+1` + `track_total_hits`; `caller_requested_limit=source size when provided else +inf`); probe hitting the clamped cap MUST fail closed as `SF_TOO_MANY_OPERATIONS` with selector-materialization clamp details (`limit_scope=selector_materialization_limit`, `configured_limit`, `requested_value`, `effective_constraint_source`). - Elasticsearch `delete_by_query` lowering MUST enforce the same two-phase probe clamping contract (`fetch_spec.limit=min(caller_requested_limit,max_entries_per_batch,effective_max_result_window)+1` + `track_total_hits`; `caller_requested_limit=source size when provided else +inf`); probe hitting the clamped cap MUST fail closed as `SF_TOO_MANY_OPERATIONS` with selector-materialization clamp details (`limit_scope=selector_materialization_limit`, `configured_limit`, `requested_value`, `effective_constraint_source`). - Elasticsearch highlight options outside declared subset MUST fail closed; accepted highlight settings MUST be normalized into `fetch_spec.text_options` and MUST NOT alter canonical ranking/filter execution semantics. - gateway MUST reject source sort payloads that try to attach field-only controls (`missing_value_policy`/`mode`/`nested_context`) to virtual targets (`_score`/`_id`) as malformed (`400 SF_BAD_REQUEST`). - gateway MUST enforce `fetch_spec.limit=0` guardrail before canonical emission: - unary `ExecuteQuery` path: reject only when both `search_payload.aggregations` is empty and `fetch_spec.track_total_hits` is not explicitly configured (`enabled` or `threshold`) (`SF_BAD_REQUEST`), - `StreamQuery` path: treat as explicit unbounded-export mode (do not rewrite to positive sentinel). - internal debug digest surface MUST run canonicalization and digest computation using the same serializer/version as production admission path; any debug-only serializer fork is forbidden. - gateway aggregation depth interceptor (`max_aggregation_depth`) MUST run before canonical emission and MUST deterministically choose reject-or-truncate branch per route policy. - gateway aggregation defensive validator (depth/node/fanout/bucket ceilings, including `max_composite_sources`) MUST run before canonical emission; violations fail closed (`SF_BAD_REQUEST` or `SF_UNSUPPORTED_QUERY_FEATURE` by precedence). - if aggregation tree is truncated at gateway, response MUST carry explicit truncation warning summary (`success.aggregation_truncation`) in addition to per-bucket `doc_count_error_upper_bound`. - when rejecting unsupported complex Elasticsearch query branches, gateway/server MUST include machine-readable details as defined by `docs/specs/core/error-codes.md` detail-key registry. - MySQL forbidden `MATCH + ROLLUP` shape MUST fail closed as `SF_UNSUPPORTED_SYNTAX`. - MySQL `WITH QUERY EXPANSION` MUST be normalized to `TextClause.mode_override.natural_language.query_expansion=true`; any non-natural-language branch attempting query expansion MUST fail closed (`SF_UNSUPPORTED_QUERY_FEATURE`). - Turbopuffer `vector_encoding` MUST be handled as response vector-serialization selection and MUST NOT alter canonical query-input parsing. - Turbopuffer `include_attributes`/`exclude_attributes` normalization MUST be deterministic and MUST complete before pagination digest generation. - Turbopuffer `patch_by_filter` lowering MUST enforce the same two-phase probe clamp (`fetch_spec.limit=min(caller_requested_limit,max_entries_per_batch,effective_max_result_window)+1` + `track_total_hits`; `caller_requested_limit=source size when provided else +inf`); clamp-hit selectors MUST fail closed as `SF_TOO_MANY_OPERATIONS` with selector-materialization clamp details (`limit_scope=selector_materialization_limit`, `configured_limit`, `requested_value`, `effective_constraint_source`). - Turbopuffer `Id` active-profile admission accepts decimal-string uint64 and non-negative integer uint64 forms only; negative integers, floating-point numerics, `uuid`, and non-decimal/non-lossless strings MUST fail closed (`SF_INVALID_DOC_ID`). - opaque per-query vector option bags are unsupported; gateway MUST NOT emit legacy `VectorClause.options`-style payloads. - projected vector fields under high-dimensional response path MUST apply deterministic field-level payload-format policy (`fetch_spec.vector_projections`) and dynamic payload/result-window budget checks before canonical execution. - unsupported source-profile options/features that cannot be canonically represented MUST fail closed (`SF_UNSUPPORTED_OPTION` or `SF_UNSUPPORTED_QUERY_FEATURE` by precedence), not silently dropped. - Turbopuffer billing/performance nested shapes MUST NOT be required by canonical planning/ranking logic; only top-level envelope requirements are contractual at this boundary. ### 5.3 Logical Query Intent vs Physical Plan QueryRequest carries only logical retrieval/ranking/fetch intent. Physical pushdown shape (for example partial aggregation, TopN cut, limit-offset pushdown) is internal CBO/execution behavior and MUST NOT be client-directed in this API. ### 5.4 Readiness and Fallback Matrix Strict mode formulas: - `effective_read_ts = min(read_ts, safe_ts)` Deterministic handling: - `STRICT` and `BOUNDED_STALENESS` use the remaining transport deadline budget as the upper bound for visibility waiting; when current readiness is not yet admissible, the server MUST keep rechecking fresh readiness/watermarks until the mode-specific admission condition is satisfied or the transport deadline expires. - strict admission condition: `safe_ts >= read_ts`. - if the transport deadline expires before strict admission succeeds: - projected query route: explicit fallback - primary query route and shared Search-Fabric-truth query routes: fail closed (`SF_VISIBILITY_TIMEOUT`) with detail keys `required_read_ts`, `effective_read_ts`, and `safe_ts`. - bounded-staleness admission condition: `tso_physical_ms(read_ts) - tso_physical_ms(effective_read_ts) <= max_staleness_ms`. - if the transport deadline expires before bounded-staleness admission succeeds: - projected query route: explicit fallback - primary query route and shared Search-Fabric-truth query routes: fail closed (`SF_VISIBILITY_TIMEOUT`) with detail keys `required_read_ts`, `effective_read_ts`, `safe_ts`, `max_staleness_ms`, and `actual_lag_ms` - projected query route + non-covering deterministic projection at the admitted schema snapshot => explicit fallback to caller-native plan. - best effort: - execute at `safe_ts`; degraded/non-exact success MUST use `success.partial` together with warning fields. - overlay-bearing `ProjectedQueryRequest` routes are the explicit projected-query exception: when `ExecuteProjectedQueryOrFallback` carries non-empty `snapshot_overlay.rows` under `BEST_EFFORT`, execution snapshot is carried by `snapshot_read_ts` rather than current `safe_ts`, and Query Gateway MUST materialize broker dispatch with `effective_read_ts = snapshot_read_ts`. - shared-surface rule: - `QueryGatewayService.ExecuteQuery` / `StreamQuery` remain canonical query surfaces and MUST NOT invent local fallback pseudo-statuses; `ExecuteQuery` uses direct unary success body + canonical non-`OK` failure, while `StreamQuery` remains an in-band frame protocol. - production relational query routes that need trusted relational snapshot-overlay execution context MUST use `RelationalQueryService`; the current TiDB deployment maps its projected-query route onto `RelationalQueryService.ExecuteProjectedQueryOrFallback`. - `GetReadiness` transport split: - transport `OK` returns `ReadinessSuccessResponse` directly. - malformed/not-found/catalog-unavailable/authz/runtime failures MUST use non-`OK` gRPC status plus canonical `ErrorStatus`; this RPC no longer exposes a public `success/error` envelope. - `ReadinessSuccessResponse` is the pre-query context carrier and MUST include: - `watermarks`, - `applied_schema_version`, - `applied_schema_generation_token`. - The canonical protobuf closes `ReadinessSuccessResponse.watermarks` as a machine-required field; transport-`OK` success without `watermarks` is invalid. - The canonical protobuf also closes `ReadinessSuccessResponse.applied_schema_version` as machine-required positive lineage state and `ReadinessSuccessResponse.applied_schema_generation_token` as a machine-required non-empty lineage token; transport-`OK` success without either field is invalid. - `GetReadiness` schema-lineage fields are query-serving fields, not active-control-plane-pointer fields: - `applied_schema_version` and `applied_schema_generation_token` identify the generation query execution would admit at that moment (`latest READY` generation). - `GetReadiness` MUST NOT return a newer active generation that is still `APPLYING`, `DRIFTED`, or `ERROR`. - requests targeting collections in internal lifecycle state `TOMBSTONED` MUST fail closed as `SF_RESOURCE_NOT_FOUND`. - requests targeting collections that never existed or whose prior tombstone reservation has already been fully GC-released MUST also fail closed as `SF_RESOURCE_NOT_FOUND`. - if the collection is active but no schema generation has been admitted yet, `GetReadiness` MUST fail closed as `SF_CATALOG_UNAVAILABLE` with `catalog_unavailable_reason=schema_bootstrap` and `schema_bootstrap_state=no_active_schema_generation`. - if the collection already has an admitted active schema lineage but no `READY` generation exists yet, `GetReadiness` MUST fail closed as `SF_CATALOG_UNAVAILABLE` with `catalog_unavailable_reason=schema_convergence` and `schema_convergence_state=no_ready_serving_generation`. Mode-coupled `ReadConsistencySpec` admission constraints: - `CONSISTENCY_MODE_STRICT`: - `consistency.read_ts` MUST be `> 0`. - `consistency.max_staleness_ms` MUST be `0`. - `CONSISTENCY_MODE_BOUNDED_STALENESS`: - `consistency.read_ts` MUST be `> 0`. - `consistency.max_staleness_ms` MUST be `> 0`. - `CONSISTENCY_MODE_BEST_EFFORT`: - `consistency.read_ts` MUST be `0` (caller does not pin snapshot TS in this mode). - `consistency.max_staleness_ms` MUST be `0`. ### 5.5 Limit, Ownership, and Feature Enforcement Before query execution, Query Plane MUST perform deterministic checks in this order: 1. access scope (`namespace_id`/`collection_id` authz from caller identity). 2. payload budget and AST complexity pre-admission (`max_query_payload_bytes`, `max_in_list_elements`, `max_predicate_ast_nodes`). 3. truth-mode/applicability (`collection_truth_mode`, collection policy, caller/route policy). 4. continuation token validity/coherence (`QueryRequest.fetch_spec.continuation_page.next_page_token`, optional same-lineage ordinary-continuation `fetch_spec.search_context.token`, and composite-pagination `fetch_spec.composite_page.next_page_token`, including signature/scope/TTL checks plus same-lineage parameter coherence whenever both token families are present on ordinary unary continuation paths). 5. feature compatibility (aggregate-function support and metric-field type safety, vector projection support, highlighting applicability). 6. hard limits (`QueryRequest.fetch_spec.limit`, `QueryRequest.fetch_spec.first_page.offset` when first-page mode is used, collection-level resolved `effective_max_result_window`, dynamic vector-response budget, `search_payload.global_candidate_topk`, aggregation-tree depth, bucket-size budgets, and composite-source dimension budgets). 7. consistency admissibility (`consistency.read_ts`, `safe_ts`, staleness bound). Query-plane defensive limits (service-static policy): - `max_query_payload_bytes` (`uint32`, bytes, default `16777216` / `16 MiB`): - enforced at Query Gateway HTTP/gRPC body-read layer before full parse/materialization. - requests exceeding this byte budget MUST be terminated fail-closed without continuing decode/allocation. - `max_in_list_elements` (`uint32`, count, default `65536`): - enforced during AST lowering for every `InListPredicate.values.values[]`. - limit is per `InListPredicate` instance (not aggregate across all predicates). - `max_predicate_ast_nodes` (`uint32`, count, default `4096`): - enforced during recursive predicate-tree parsing with one global node counter per request. - counter scope MUST include `search_payload.global_filter`, every predicate tree reachable from `search_payload.root_retrieval` (for example `VectorClause.local_filter`, `ScoringScalarClause.predicate`, `NestedClause.query.child_filter`, nested nested-query trees), `routing.partition_filter`, and nested-sort filter trees. Failure behavior: - access-scope mismatch -> fail closed (`SF_FORBIDDEN`). - unsupported endpoint/feature/option -> fail closed (`SF_UNSUPPORTED_ENDPOINT`, `SF_UNSUPPORTED_OPTION`, `SF_UNSUPPORTED_QUERY_FEATURE`). - query payload bytes exceeding `max_query_payload_bytes` MUST fail closed with `413 SF_PAYLOAD_TOO_LARGE`; `ErrorStatus.diagnostics` MUST include `limit_scope=dynamic_payload_budget`, `effective_limit`, and `requested_bytes`. - `InListPredicate.values.values[]` cardinality exceeding `max_in_list_elements` MUST fail closed with `400 SF_TOO_MANY_OPERATIONS`; `ErrorStatus.diagnostics` MUST include `limit_scope=max_in_list_elements`, `configured_limit`, `requested_value`, and `field`. - total predicate AST nodes exceeding `max_predicate_ast_nodes` MUST fail closed with `400 SF_TOO_MANY_OPERATIONS`; `ErrorStatus.diagnostics` MUST include `limit_scope=max_predicate_ast_nodes`, `configured_limit`, and `requested_value`. - unsupported complex ES query branches MUST include machine-readable detail keys as defined in `docs/specs/core/error-codes.md` for machine-driven downgrade/rewrite flows. - malformed `TextClause.mode_override` branch payload coherence (for example query expansion outside `natural_language`) -> fail closed (`SF_UNSUPPORTED_QUERY_FEATURE`). - `TextClause.similarity_override` present with unset `SimilarityModel.model` oneof -> fail closed (`SF_INVALID_ARGUMENT`). - malformed `PlainMatchOptions.minimum_should_match` expression syntax -> fail closed (`SF_BAD_REQUEST`). - `TextClause.fields[]` or `TextMatchPredicate.fields[]` empty, duplicate after normalization, or containing unknown/non-text-searchable fields -> fail closed (`SF_BAD_REQUEST` or `SF_SCHEMA_VIOLATION` by precedence). - `RetrievalNode` missing `clause` branch is a present-but-empty typed carrier and MUST fail closed (`SF_INVALID_ARGUMENT`). - `RetrievalNode` non-finite `boost` or negative `boost` -> fail closed (`SF_BAD_REQUEST`). - `RetrievalGroup` carrying neither `must_clauses[]` nor `should_match`, or carrying `should_match` with empty `clauses[]` -> fail closed (`SF_BAD_REQUEST`). - `RetrievalGroup.tie_breaker` present when `fusion_strategy != GROUP_FUSION_STRATEGY_DIS_MAX`, non-finite, or outside `[0.0, 1.0]` -> fail closed (`SF_BAD_REQUEST`). - `ShouldMatchSpec.threshold.absolute_count` exceeding `len(ShouldMatchSpec.clauses[])` -> fail closed (`SF_BAD_REQUEST`). - `ShouldMatchSpec.threshold.percentage_ratio` non-finite or outside `[0.0, 1.0]` -> fail closed (`SF_BAD_REQUEST`). - malformed query vector dimension (`Dq = 0`) -> fail closed (`SF_BAD_REQUEST`). - query vector dimension overflow (`Dq > Dschema`) -> fail closed (`SF_VECTOR_DIMENSION_MISMATCH`). - `PredicateNode` present with unset `node` oneof -> fail closed (`SF_INVALID_ARGUMENT`). - malformed logical predicate node (`LogicalAnd` / `LogicalOr` child count < 2 after typed-carrier validation, or `LogicalNot.child` absent after typed-carrier validation) -> fail closed (`SF_INVALID_QUERY`). - malformed `RangePredicate` (both bounds absent, `from > to`, equal bounds with an exclusive endpoint, or type-incompatible bounds after field resolution) -> fail closed (`SF_INVALID_QUERY`). - malformed `InListPredicate.values` (empty list or mixed atomic types such as `int64` + `string`) -> fail closed (`SF_INVALID_QUERY` or `SF_SCHEMA_VIOLATION` by precedence). - `ComparisonPredicate` carrying unsupported/non-equality `CompareOp` or type-incompatible operands for equality/inequality comparison -> fail closed (`SF_INVALID_QUERY`). - `SortKey.order=SORT_ORDER_UNSPECIFIED` -> fail closed (`SF_INVALID_QUERY`). - malformed sort target/value contract (for example source payload tries to attach `missing_value_policy`/`mode`/`nested_context` to `_score`/`_id`) -> fail closed (`SF_BAD_REQUEST`). - `fetch_spec.attribute_projection` present with unset `intent` oneof is a present-but-empty typed carrier and MUST fail closed (`SF_INVALID_ARGUMENT`). - malformed `fetch_spec.attribute_projection` contract (`return_explicit.include_fields` empty, or non-scalar/unknown attribute field names) -> fail closed (`SF_BAD_REQUEST`). - malformed aggregation tree shape (duplicate sibling names, invalid bucket strategy parameters, or illegal oneof/node shape) -> fail closed (`SF_BAD_REQUEST` or `SF_UNSUPPORTED_QUERY_FEATURE` by precedence). - metric type-safety violation: `MetricAggregation.builtin_function in {SUM, MIN, MAX, AVG}` targeting a non-numeric scalar schema field -> fail closed during query admission (`SF_SCHEMA_VIOLATION`). - if `MetricAggregation.missing_value_replacement` is present and non-null, its atomic type MUST be legally castable to the schema-declared numeric type of the target field; type mismatch MUST fail closed during query admission (`SF_SCHEMA_VIOLATION`). - if `TermsAggregation.missing_value_replacement` is present and non-null, its atomic type MUST be legally castable to the schema-declared atomic type of `TermsAggregation.field`; type mismatch MUST fail closed during query admission (`SF_SCHEMA_VIOLATION`). - malformed `CompositeAggregation` contract (`sources` empty, duplicate source names, source strategy unset, source-order unset, `after_key` key-set mismatch, or source-incompatible cursor value typing) -> fail closed (`SF_BAD_REQUEST`). - continuation request carrying non-empty `CompositeAggregation.after_key` without valid bound `fetch_spec.composite_page.next_page_token`, or with an `after_key` that does not exactly match the token-bound composite cursor -> fail closed (`SF_INVALID_QUERY`). - invalid `AggregationRequest.name` regex branch (`^[a-zA-Z0-9_.-]{1,128}$`) -> fail closed (`SF_BAD_REQUEST`). - if aggregation depth exceeds gateway `max_aggregation_depth`: - strict branch: fail closed (`SF_UNSUPPORTED_QUERY_FEATURE`) with machine-readable details defined in `docs/specs/core/error-codes.md` for the strict aggregation-depth branch, - truncation branch: continue with truncated tree and emit `success.aggregation_truncation` warning summary. - on Elasticsearch-compatibility routes, the strict aggregation-reject branch MUST project to ES-compatible `parsing_exception`; execution-time aggregation failures after admission MUST project to `search_phase_execution_exception`. - if `fetch_spec.text_options.require_highlighting=true` while request has no text clauses or the admitted route/capability set does not support highlight generation, fail closed (`SF_UNSUPPORTED_QUERY_FEATURE`). - if any `TopHitsAggregation.text_options.require_highlighting=true` while the corresponding query/aggregation scope has no text-clause retrieval context, fail closed (`SF_UNSUPPORTED_QUERY_FEATURE`). - continuation requests carrying empty `fetch_spec.continuation_page.next_page_token` are malformed and MUST fail closed (`SF_BAD_REQUEST`). - malformed continuation token envelope/signature/key-window failure -> fail closed (`SF_BAD_REQUEST`). - continuation token principal/scope mismatch -> fail closed (`SF_FORBIDDEN`). - continuation token expiry or token/request-parameter mismatch -> fail closed (`SF_INVALID_QUERY`). - `ExecuteQuery` with `fetch_spec.limit=0` and neither `search_payload.aggregations` nor explicit `fetch_spec.track_total_hits` intent (`enabled`/`threshold`) -> fail closed (`SF_BAD_REQUEST`). - `StreamQuery` with `fetch_spec.limit=0` is valid and means unbounded export; runtime flow-control and payload guards remain mandatory. - limit violation -> deterministic limit error (`SF_TOPK_TOO_LARGE` or `SF_TOO_MANY_OPERATIONS`). - malformed `search_payload.routing` (non-partition field refs, duplicate/unsorted routing keys after normalization, or unsupported collection/profile) -> fail closed (`SF_UNSUPPORTED_OPTION` or `SF_BAD_REQUEST` by precedence). - `fetch_spec.rescore` present with unset `RescoreSpec.strategy` oneof -> fail closed (`SF_INVALID_ARGUMENT`). - unsupported `fetch_spec.rescore` strategy/profile branch -> fail closed (`SF_UNSUPPORTED_OPTION`). - for first-page windows (`fetch_spec.limit > 0`), if `(fetch_spec.first_page.offset + fetch_spec.limit) > effective_max_result_window`, reject with `SF_TOO_MANY_OPERATIONS` and include machine-readable details defined in `docs/specs/core/error-codes.md`. - for high-dimensional projected vector requests, gateway/server MAY tighten effective payload/result-window limits; limit hit MUST fail with `SF_PAYLOAD_TOO_LARGE` (or `SF_TOO_MANY_OPERATIONS` for tightened window) and include machine-readable details defined in `docs/specs/core/error-codes.md`. - in non-strict modes, downgrade is allowed only when explicitly encoded as the `success.partial` branch together with warning fields. Vector projection admission rules: - `fetch_spec.vector_projections` is field-level override map (`key=vector field name`, `value=VectorProjectionSpec`). - for each projected vector field: - if override exists, apply overridden format. - `target_dimensions` semantics are deterministic: `unset` or `0` MUST fallback to schema vector dimensions. - when `0 < target_dimensions < schema dimensions`, server MUST apply deterministic truncation (MRL-style projection) before output encoding. - when `target_dimensions > schema dimensions`, request MUST be rejected as `SF_VECTOR_DIMENSION_MISMATCH` (fail-closed). - if override does not exist, apply collection/profile default; when default is unspecified, fallback is `VECTOR_PAYLOAD_FORMAT_EXACT_FLOAT`. - if `fetch_spec.vector_projections[].format` field is present and equals `VECTOR_PAYLOAD_FORMAT_UNSPECIFIED`, request is malformed and MUST fail closed (`SF_BAD_REQUEST`). - `VECTOR_PAYLOAD_FORMAT_EXACT_F16` requests explicit half-precision response encoding; unsupported field/policy branches MUST fail closed (`SF_UNSUPPORTED_OPTION`). - `VECTOR_PAYLOAD_FORMAT_QUANTIZED_INT8` requires collection/profile policy explicitly allows INT8 quantized projection. - `VECTOR_PAYLOAD_FORMAT_QUANTIZED_BINARY` requires collection/profile policy explicitly allows binary quantized projection. - unsupported projection format MUST fail closed (`SF_UNSUPPORTED_OPTION`). - unknown projection key or non-vector projection field branch MUST fail closed (`SF_BAD_REQUEST`). Aggregation truncation warning rules: - if gateway or engine truncates aggregation tree due depth/budget policy, `QuerySuccessPayload.aggregation_truncation` MUST be present. - top-level `aggregation_truncation.doc_count_error_upper_bound` MUST be >= every bucket-level `doc_count_error_upper_bound` included in the returned payload. - if truncation occurs and request asked for exact full-depth aggregation semantics, gateway MAY reject instead of truncating based on route policy. ### 5.6 Pagination and Continuation Determinism `pagination.next_page.token`, `pagination.search_context.token`, and `pagination.composite_page.token` MUST be cryptographically bound to: - canonical query digest, - `effective_read_ts`, - `applied_schema_generation_token`, - collection identity (`namespace_id`, `collection_id`), - committed manifest pin, - continuation lineage. - `pagination.next_page.token` MUST additionally bind: - page ordinal, - sort contract, - the last-emitted canonical sort cursor including the forced final `doc_id asc` tie-break. - `pagination.composite_page.token` MUST additionally bind: - page ordinal, - the canonical emitted composite cursor (`BucketResult.composite.after_key.fields`) for the next request, - exact-match coherence with any compatibility-layer request-side `CompositeAggregation.after_key` echo. - `pagination.search_context.token` MUST remain snapshot-lineage state only: - it binds the pinned snapshot lineage and expiry, - it MUST NOT encode page-local ordinal or last-emitted cursor state, - older unexpired tokens in the same lineage remain valid until natural expiry. `canonical_query_digest` contract: - digest algorithm is fixed to `HASH_ALGORITHM_BLAKE3` in this revision. - for cross-language digest stability, any float/double fields (including values carried by `AtomicValue` and `FloatVector`) MUST be canonicalized using strictly ordered IEEE 754 integer bit-patterns before hashing (for example Go `math.Float32bits` / `math.Float64bits`); `-0.0` MUST normalize to `+0.0`, and all NaN payloads MUST collapse to one canonical NaN bit-pattern per width. - digest input is canonical serialization of admitted query semantics (not raw wire bytes): - truth-mode and consistency semantics (resolved `collection_truth_mode`, admitted consistency mode, `consistency.read_ts`/staleness parameters), - canonicalized hard-filter semantics (`search_payload.global_filter`) including predicate-tree operator structure, `TextMatchPredicate` payloads, nested filter scopes, and deterministic child ordering where source order is non-semantic, - canonicalized retrieval-tree semantics (`search_payload.root_retrieval`) including every `RetrievalNode{boost}`, every `RetrievalGroup{must_clauses,should_match,fusion_strategy,tie_breaker}`, `ShouldMatchSpec.threshold` (including canonical fallback when absent), deterministic child ordering, `NestedClause.path`, `NestedClause.score_mode`, `NestedClause.query`, `TextClause.fields[]`, `TextClause.mode_override`, typed `VectorClause.vector`, `VectorClause.local_filter`, `VectorClause.filter_strategy_hint`, `TextClause.similarity_override`, and `ScoringScalarClause.predicate`, - query-level dedupe controls (`search_payload.dedup_by_doc_id`), - routing/partition-pruning semantics (`search_payload.routing.routing_keys`, `search_payload.routing.partition_filter`, canonicalized `search_payload.routing.target_tiers`), - projection and output semantics (`fetch_spec.attribute_projection`, `fetch_spec.return_score`, `fetch_spec.text_options`, canonicalized `fetch_spec.vector_projections`, `fetch_spec.rescore`, nested `InnerHitsSpec` payloads), - logical aggregation tree intent (`search_payload.aggregations`) with canonical recursive ordering of `sub_aggregations` (cursor payloads excluded), - normalized sort contract (including auto-appended `doc_id asc` tie-break), - retrieval candidate semantics (`search_payload.global_candidate_topk`) and deterministic semantic defaults that affect result-set definition. - transport-only fields that do not affect result semantics (for example idempotency/correlation metadata and tracing metadata) MUST NOT affect digest. - pagination/session control fields (`fetch_spec.limit`, `fetch_spec.page_window`, `fetch_spec.search_context`) are transport-state parameters and MUST NOT affect `canonical_query_digest`. - cursor payloads, including compatibility-layer `CompositeAggregation.after_key`, are continuation-transport parameters and MUST NOT affect `canonical_query_digest` to keep digest stable across a multi-page continuation chain. Rules: - if caller changes query/sort/consistency parameters while reusing token, reject with `SF_INVALID_QUERY`. - continuation snapshot override rule: - when request presents a valid `fetch_spec.continuation_page.next_page_token`, execution snapshot MUST use token-bound `effective_read_ts` as the authoritative read timestamp. - request payload `consistency` fields remain required for mode-coupled admission/coherence validation, but MUST NOT advance/recompute snapshot timestamp for that continuation execution. - this override is mandatory for `CONSISTENCY_MODE_BEST_EFFORT`: caller continues to send `read_ts=0`, while engine executes continuation at token-bound `effective_read_ts` (not latest `safe_ts`). - continuation requests MAY change `fetch_spec.limit` and `fetch_spec.search_context.keep_alive_ms`; these transport updates MUST NOT trigger digest mismatch by themselves. - continuation requests MAY change compatibility-layer `CompositeAggregation.after_key`; this cursor progression alone MUST NOT trigger digest mismatch. - each request MUST set exactly one `fetch_spec.page_window` branch (`first_page` or `continuation_page` or `composite_page`); unset page-window mode is malformed (`SF_BAD_REQUEST`). - `ExecuteQuery` limit contract: - `fetch_spec.limit > 0` => bounded hit retrieval. - `fetch_spec.limit = 0` is valid only when `search_payload.aggregations` is non-empty or `fetch_spec.track_total_hits` is explicitly configured (`enabled` or `threshold` branch); otherwise malformed (`SF_BAD_REQUEST`). - `StreamQuery` limit contract: - `fetch_spec.limit > 0` => bounded streamed hit count. - `fetch_spec.limit = 0` => unbounded streaming export mode. - continuation requests with `fetch_spec.continuation_page` MUST use token-bound query shape: - `fetch_spec.continuation_page.next_page_token` MUST be non-empty; empty token is malformed (`SF_BAD_REQUEST`). - `ExecuteQuery` continuation requests MUST use `fetch_spec.limit > 0` (unary `limit=0` paths are aggregation/total-hit-only and do not admit hit continuation). - `StreamQuery` continuation requests MAY use `fetch_spec.limit > 0` or `fetch_spec.limit = 0`; resume semantics are defined in Section `5.8`. - `fetch_spec.sort_keys` MUST be omitted or exactly match token-bound sort contract. - execution MUST resume strictly from the token-bound last-emitted canonical sort cursor; implementations MUST NOT rely on hidden server-side page/session state. - request MUST NOT include `fetch_spec.first_page`. - first-page requests MUST use `fetch_spec.first_page`; continuation tokens are invalid in first-page semantics. - first-page requests MUST NOT carry `fetch_spec.search_context.token`; token-on-first-page semantics is invalid (`SF_INVALID_QUERY`). - composite continuation requests with `fetch_spec.composite_page`: - MUST carry non-empty `fetch_spec.composite_page.next_page_token` bound to the same snapshot lineage and composite cursor. - request-side `CompositeAggregation.after_key` is optional compatibility-layer echo; when present, at least one `CompositeAggregation` MUST carry non-empty `after_key`, and that supplied cursor MUST exactly match the token-bound cursor or the request is malformed (`SF_INVALID_QUERY`). - MUST NOT rely on `fetch_spec.first_page.offset` semantics; offset-style paging and composite cursor paging are disjoint branches. - nested `InnerHitsSpec` never carries continuation cursor semantics in this revision; per-parent nested extraction is shallow-page only (`offset` + `limit`). - PIT-equivalent contract: - `ExecuteQuery` first-page requests MAY set `fetch_spec.search_context.keep_alive_ms` to request snapshot-context issuance. - when `fetch_spec.search_context.keep_alive_ms` is present, value MUST be `> 0`; present `0` is malformed (`SF_BAD_REQUEST`). - when server admits PIT-equivalent context, response MUST return `success.pagination.search_context.token` plus `success.pagination.search_context.expires_at_unix_ms`. - `ExecuteQuery` continuation requests MAY carry `fetch_spec.search_context.token`; `fetch_spec.composite_page` continuation requests no longer require it because the composite-page token already binds immutable snapshot lineage plus cursor. - `fetch_spec.search_context.keep_alive_ms` on continuation requests MAY extend context TTL under service policy. - when keep-alive is admitted, server MAY return a fresh `search_context.token` in the same lineage with a later expiry; clients SHOULD continue with the freshest returned token, while older unexpired tokens in the same lineage remain valid until natural expiry. - when a successful response emits non-empty `BucketResult.composite.after_key`, the response MUST return non-empty `success.pagination.composite_page.token` and `success.pagination.composite_page.expires_at_unix_ms`, even if the request did not set `keep_alive_ms`. - Elasticsearch Profile-2 compatibility does not expose this lifecycle (`/_pit` and `search_after + PIT` are Profile-3 deferred). - if token-bound `applied_schema_generation_token` no longer names a retained query-admissible serving lineage, reject with `SF_VERSION_CONFLICT`. - for this comparison, continuation/search-context validation MUST use the token-bound serving lineage itself rather than requiring equality with the current newest `READY` generation; later `READY` promotions alone do not invalidate already-issued query tokens. - if any continuation-page request shape carries both `fetch_spec.continuation_page.next_page_token` and `fetch_spec.search_context.token` and both fence the same stale serving generation, `ErrorStatus.context.query.schema_version_conflict.conflict_source` MUST be `QUERY_SCHEMA_CONFLICT_SOURCE_CONTINUATION_PAGE_TOKEN`; this includes ordinary unary continuation and any future request shape that reuses the same continuation-page token family, while `StreamQuery` resume uses the same token family without request-side `search_context.token`. - composite continuation fences schema retention through `fetch_spec.composite_page.next_page_token`; therefore stale composite continuation always projects `ErrorStatus.context.query.schema_version_conflict.conflict_source=QUERY_SCHEMA_CONFLICT_SOURCE_COMPOSITE_PAGE_TOKEN`. - `QUERY_SCHEMA_CONFLICT_SOURCE_SEARCH_CONTEXT_TOKEN` is used only when no page token participated in the rejected continuation. - if token snapshot is no longer admissible under strict mode, use the same readiness matrix as Section `5.4`. - token validation error-class mapping for continuation: - malformed token envelope/signature or key-id outside verify overlap -> `SF_BAD_REQUEST`, - principal/scope mismatch -> `SF_FORBIDDEN`, - expired token or semantic mismatch with bound query/page contract -> `SF_INVALID_QUERY`. - token signature/scope/principal/TTL checks MUST follow Section `3.6`. - default page-token TTL is service policy and MUST be treated as authoritative. - default search-context TTL is service policy and MUST be treated as authoritative. - maximum page depth is service policy and MUST be enforced deterministically. - page-token key rotation MUST preserve deterministic validation under overlap windows; key-id mismatches outside overlap are rejected with `SF_BAD_REQUEST`. - when one successful unary broker execution emits both `success.pagination.next_page` and `success.pagination.composite_page`, those page-token carriers remain semantically distinct but MUST share the same expiry timestamp because that execution uses one shared query-token issuance horizon. - when `ExecuteQuerySuccessResponse.success.pagination.next_page.token` is non-empty, `ExecuteQuerySuccessResponse.success.pagination.next_page.expires_at_unix_ms` MUST be non-zero; when `next_page.token` is empty, that field MUST be `0`. - when `ExecuteQuerySuccessResponse.success.pagination.search_context.token` is non-empty, `ExecuteQuerySuccessResponse.success.pagination.search_context.expires_at_unix_ms` MUST be non-zero; when `search_context.token` is empty, that field MUST be `0`. - when `ExecuteQuerySuccessResponse.success.pagination.composite_page.token` is non-empty, `ExecuteQuerySuccessResponse.success.pagination.composite_page.expires_at_unix_ms` MUST be non-zero; when `composite_page.token` is empty, that field MUST be `0`. - responses that emit non-empty `BucketResult.composite.after_key` therefore inherit this same `MUST` requirement for the `composite_page` carrier. - requests exceeding service-configured `max_page_depth` MUST fail with `SF_TOO_MANY_OPERATIONS`. - `max_page_depth` rejection details MUST include: - `limit_scope` (`max_page_depth`), - `configured_limit`, - `requested_value`. ### 5.7 Response Shape Determinism `ExecuteQuerySuccessResponse.success` uses a unified payload model: - The canonical protobuf closes `ExecuteQuerySuccessResponse.success` as a machine-required field; transport-`OK` unary query success without this payload is invalid. - `hits` and `aggregations` are independently optional and MAY coexist in one response. - `hits` are populated for Search-Fabric-truth requests and for external-truth covering-read requests. - `QueryHit.score` is optional: - `fetch_spec.return_score=true` -> `hits[].score` SHOULD be populated. - `fetch_spec.return_score=false` -> `hits[].score` MUST be unset. - `QueryHit.version_commit_ts` is the canonical row-version marker (`commit_ts`) for read-side optimistic concurrency loops. - when known, it SHOULD be populated for hit rows. - Elasticsearch Query Gateway MUST map this field to source `_seq_no` in response projection and MUST inject static `_primary_term=1` alongside `_seq_no` for client compatibility. - `QueryWarning.code` is enum-typed (`WarningCode`), not free-form string; emitted warning classifiers MUST use declared non-`WARNING_CODE_UNSPECIFIED` values. - `aggregations` carry user-facing recursive aggregate output trees (`AggregationResponse`), not shard-local partial states. - metric nullability is encoded only by `MetricResult.value.null_value`; no standalone `is_null` flag exists in this revision. - bucket result shapes are discriminated: - non-composite bucket results MUST use `BucketResult.non_composite`, - composite bucket results MUST use `BucketResult.composite`. - within `Bucket`, `key` and `composite_key` are mutually exclusive; composite-key contents live in `Bucket.composite_key.fields`. - for `RangeAggregation`, each emitted non-composite bucket key MUST use the request-side `RangeAggregation.Range.key` string verbatim (`Bucket.key.string_value`) so bucket identifiers remain gateway-deterministic. - `BucketResult.non_composite.buckets[]` order MUST follow normalized request strategy/order semantics, and each bucket MAY carry recursive `sub_aggregations`. - for non-composite bucket strategies, `BucketResult.non_composite.sum_other_doc_count` is best-effort; when exact omitted-bucket cardinality cannot be computed, uncertainty MUST be conveyed through `success.aggregation_truncation` and `BucketResult.non_composite.doc_count_error_upper_bound`. - for `CompositeAggregation`, `BucketResult.composite.buckets[]` order MUST follow normalized request strategy/order semantics, and `BucketResult.composite.after_key.fields` carries next-page cursor when additional buckets exist. - `success.aggregation_truncation` (when present) MUST surface prominent summary fields for clients: - `truncated=true`, - `requested_depth`, - `applied_depth_limit`, - `doc_count_error_upper_bound` (top-level bound summary), - optional `truncated_paths[]`. - if `success.aggregation_truncation.truncated=true`, success MUST use the `partial` branch. - total-hit contract: - if `fetch_spec.track_total_hits` is omitted, server MUST apply global default `TrackTotalHitsSpec{threshold=10000}`. - if `fetch_spec.track_total_hits` message is present but `tracking` oneof is unset, request is malformed and MUST fail closed (`SF_INVALID_ARGUMENT`). - `fetch_spec.track_total_hits.enabled=true` requests exact total hits (`success.total_hits.relation=TOTAL_HITS_RELATION_EQUAL_TO`). - `fetch_spec.track_total_hits.enabled=false` disables total-hit tracking; `success.total_hits` MAY be absent. - `fetch_spec.track_total_hits.threshold=N` requests threshold tracking: - if `actual_total <= N`, return exact total (`relation=TOTAL_HITS_RELATION_EQUAL_TO`, `value=actual_total`); - if `actual_total > N`, return lower bound (`relation=TOTAL_HITS_RELATION_GREATER_THAN_OR_EQUAL_TO`, `value=N`). - when `success.total_hits` is present, `relation` MUST be `TOTAL_HITS_RELATION_EQUAL_TO` or `TOTAL_HITS_RELATION_GREATER_THAN_OR_EQUAL_TO`; `TOTAL_HITS_RELATION_UNSPECIFIED` is invalid. - nested inner-hits response header contract: - `QuerySuccessPayload.inner_hits_headers[]` is a response-level dictionary keyed by `name`; names MUST be unique. - each header defines ordered `projected_field_names[]` and `highlight_field_names[]` for that inner-hits stream. - response-side inner-hits header resolution MUST be `O(1)` by name without map-based wire payloads. - nested inner-hits result contract: - `QueryHit.inner_hits[]` carries parent-scoped named result sets (`NamedInnerHitsResult.name`). - `InnerQueryHit.nested_physical_id` is nested-physical identity (not external business `DocumentID`) and MUST NOT alias top-level `QueryHit.doc_id`. - `InnerQueryHit.parent_array_offset` is source-array offset metadata and deterministic fallback key for nested sorting. - for each `NamedInnerHitsResult`, every `InnerQueryHit.projected_values[]`/`highlights[]` MUST align with the matching header by `name`. - `InnerQueryHit.inner_hits[]` allows recursive nested extraction under deeper nested paths. - projected vector fields MUST be controlled only by `fetch_spec.vector_projections`. - `QuerySuccessPayload.projected_field_names[]` MUST be response-level canonical projection header: - scalar attribute segment is resolved from admitted `fetch_spec.attribute_projection` intent, - vector segment is resolved from admitted `fetch_spec.vector_projections` keys, - final header ordering MUST be deterministic and unique for the same admitted request semantics. - each hit MUST carry `QueryHit.projected_values[]` in the same order as `projected_field_names[]`, and length MUST match exactly. - `QuerySuccessPayload.projected_field_names[]` scope is top-level `success.hits[]` only; aggregation `top_hits` payloads MUST use branch-local headers from `TopHitsResult.{global_hits|nested_hits}`. - when `success.hits` is empty (for example aggregation-only or total-hit-only responses), `projected_field_names` SHOULD be empty. - when projected schema vector fields are requested, corresponding `projected_values[i].vector_value` entries (where `projected_field_names[i]` is vector field name) MUST match effective projected dimensions resolved by `fetch_spec.vector_projections[].target_dimensions` (`unset/0 => schema dimensions`). - if a projected field is null or missing for a hit, `projected_values[i]` MUST carry `FieldData.atomic_value.null_value`. - if any requested projected vector field cannot provide a schema-valid vector payload, query execution MUST fail closed before response emission (`SF_SCHEMA_VIOLATION`); partial vector omission is forbidden. - highlight payload contract: - when `fetch_spec.text_options.require_highlighting=true`, highlight controls MUST be admitted as payload-shape semantics and MUST NOT perturb ranking/filter outcomes. - Search Fabric generates snippets using admitted `highlight_pre_tag`/`highlight_post_tag` and returns them through `QueryHit.highlights`. - gateway only performs protocol assembly/source-wire adaptation; it MUST NOT re-render snippets. - `QuerySuccessPayload.highlight_field_names[]` is the dictionary for `QueryHit.highlights[].field_index`. - `QuerySuccessPayload.highlight_field_names[]` scope is top-level `success.hits[]` only; aggregation `top_hits` payloads MUST use branch-local dictionaries from `TopHitsResult.{global_hits|nested_hits}`. - each `QueryHit.highlights[]` item is sparse and field-scoped; only fields with snippets are emitted. - when highlighting is disabled, `highlight_field_names` and `hits[].highlights` SHOULD both be empty. If response payload content violates these contracts, response is invalid and MUST be rejected by conformance tests. ### 5.8 StreamQuery Determinism Contract `StreamQuery` is for transport chunking of one canonical hit-query execution and supports resume-oriented continuation tokens for reconnect recovery. Rules: - `StreamQueryRequest` uses the same canonical hit-selection inputs as unary query for filtering, ranking, consistency evaluation, and projection intent (`meta`, `collection`, `expected_schema_generation_token`, `consistency`), but uses a dedicated hit-only `StreamSearchPayload` plus a dedicated `StreamResultFetchSpec`. - public `StreamQuery` callers MAY omit both `RequestMeta.idempotency_key` and `RequestMeta.correlation_id`; when supplied, those fields remain logical request/audit correlation metadata rather than the broker-side per-attempt execution key. - Query Gateway MUST mint a non-empty internal `StreamAttemptId` for every `QueryBrokerInternalService.StreamQuery` dispatch, including reconnect/resume attempts. - that dedicated `StreamAttemptId.value` carrier is machine-validated as required and non-empty on the internal stream request plus the terminal receipt event; empty `StreamAttemptId.value` values are malformed internal contract drift and MUST be rejected before any reconciliation starts. - Query Broker internal streaming uses `QueryBrokerStreamEvent`: - `public_frame` carries the exact public `StreamQueryFrame` contract on the internal hop, - `terminal_receipt` carries exactly one `QueryBrokerStreamExecutionReceipt` for the admitted broker stream attempt. - the terminal receipt is stage-aware: - attempts that already established truthful pinned execution telemetry MUST use the full `QueryBrokerExecutionReceipt` branch, - fatal attempts classified before `PinnedSnapshot` exists MUST use the explicit `early_failure` branch instead of fabricating a partial execution receipt, - `terminal_outcome` remains the authoritative final outcome for both shapes. - terminal-receipt coverage is terminal-attempt-complete for every broker-classified streamed attempt: - terminal success, - terminal in-band `StreamQueryFrame.outcome=error` after one or more frames, - broker-classified post-admission downstream/gateway termination through the internal-only `cancelled` receipt branch, - pre-first-frame fatal failure. - if the first internal event is a `terminal_receipt` with `terminal_outcome=error`, Query Gateway MUST translate that attempt back to the public pre-first-frame failure contract (`non-OK` gRPC status plus canonical `ErrorStatus`) rather than fabricating an in-band frame. - once any public frame has been emitted, terminal execution failure MUST still be reported publicly through `StreamQueryFrame.outcome=error`; the subsequent internal `terminal_receipt` is for reconciliation/telemetry only. - Query Gateway MUST reconcile by `stream_attempt_id.value`; if the internal broker stream ends or fails before the terminal receipt is observed for a locally visible terminal attempt, Gateway MUST execute deterministic fallback reconciliation (pessimistic lease release, mark estimator sample incomplete/non-authoritative, emit metric/alert). - whole-message omission of `StreamQueryRequest.consistency` or `StreamQueryRequest.fetch_spec` is malformed; stream callers MUST send both carriers explicitly. - `StreamSearchPayload` intentionally excludes `aggregations`; `StreamQuery` is hit-streaming only in this revision, so aggregation-bearing requests MUST use `ExecuteQuery` and any attempt to route source-side aggregations onto `StreamQuery` MUST fail closed as `SF_UNSUPPORTED_QUERY_FEATURE`. - `StreamQueryRequest.fetch_spec` shape is intentionally narrower than unary fetch: - page-window contract is only `first_page` or `continuation_page`; unset page-window mode is malformed (`SF_BAD_REQUEST`) - first-page mode MUST use `fetch_spec.first_page` with no offset field and MUST NOT carry `fetch_spec.continuation_page`, - resume mode MUST use `fetch_spec.continuation_page` with non-empty `next_page_token` and MUST NOT carry `fetch_spec.first_page`, - PIT/search-context controls and composite-page continuation are not part of the stream request shape in this revision. - `StreamQueryRequest.fetch_spec.limit` semantics: - `limit > 0`: bounded number of streamed hits. - `limit = 0`: unbounded hit export mode (full stream) and MUST NOT be treated as malformed. - `StreamQuery` unbounded-export total-hit guardrail: - when `fetch_spec.limit = 0`, explicit exact-hit tracking (`fetch_spec.track_total_hits.enabled=true`) MUST be rejected as `SF_BAD_REQUEST` to prevent blocking first-frame emission. - when `fetch_spec.limit = 0` and `fetch_spec.track_total_hits` is omitted, server MUST implicitly use `enabled=false`. - stream success payload is intentionally narrower than unary success payload: - stream frames MUST NOT return `aggregations`, - stream frames MUST NOT return `aggregation_truncation`, - stream pagination carries `next_page` only and MUST NOT return `search_context`. - if execution fails before the first frame is emitted, the RPC MUST fail with non-`OK` gRPC status plus canonical `ErrorStatus`. - after one or more frames have been emitted, terminal execution failure MUST be reported in-band through `StreamQueryFrame.outcome=error`. - StreamQuery framing/dictionary/resume invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-QUERY-001`. - To conserve bandwidth, `StreamQueryFrame.success` MUST populate projection and highlight dictionary arrays (`projected_field_names`, `highlight_field_names`, `inner_hits_headers`) ONLY in the first successful frame carrying one or more hits. - successful frames with zero hits MUST leave these header arrays empty and MUST NOT consume the dictionary-emission opportunity. - Subsequent hit-bearing successful frames MUST also leave these header arrays empty, and clients MUST reuse the headers established by the first hit-bearing success frame. - every `StreamQuery` RPC attempt, including resume attempts, MUST re-emit those header arrays in its first successful frame carrying one or more hits. - unbounded mode (`limit=0`) MUST still obey service flow-control, payload budget, timeout, and backpressure policies. - server-emitted stream frames MUST preserve the same deterministic ordering contract as unary query (execute admitted normalized `fetch_spec.sort_keys`, including default fallback and forced `doc_id asc` tie-break). - non-terminal success frames MUST use the `ok` branch, MUST carry one or more hits, and MUST leave `warnings[]` and `total_hits` empty. - only the terminal success frame MAY populate `warnings[]` and `total_hits`; when terminal execution ended with partial semantics, that frame MUST use the `partial` branch. - every `StreamQueryFrame` carrying `outcome=success` and one or more hits MUST include non-empty `success.pagination.next_page.token` representing the cursor position immediately after the last emitted hit in that frame. - terminal zero-hit success is legal only for the final no-more-hits completion case and MUST keep `success.pagination.next_page.token` empty. - client reconnect/resume MUST pass the most recently received non-empty `StreamQueryFrame.success.pagination.next_page.token` as `StreamQueryRequest.fetch_spec.continuation_page.next_page_token`. - terminal frame MUST carry final `outcome`; terminal success frames follow the same per-frame token rule above and do not admit further continuation after terminal completion. ### 5.9 PIT-Equivalent Search Context Lifecycle - Query Plane MAY issue PIT-equivalent `search_context.token` in `QuerySuccessPayload.pagination` when requested by `ExecuteQuery.fetch_spec.search_context.keep_alive_ms`. - context token is snapshot-scoped and MUST preserve deterministic paging consistency across refresh events for one admitted query shape. - query-plane realization in this revision is fully token-driven: - `search_context.token` is a signed self-contained snapshot token rather than a CCS-owned opaque handle, - token payload carries the immutable snapshot pin (`canonical_query_digest`, `effective_read_ts`, `applied_schema_generation_token`, committed manifest identity, collection identity) plus a search-context lineage identity and expiry, - ordinary continuation execution remains stateless with respect to snapshot contents because the broker can recover the pinned snapshot directly from the validated token payload. - no additional stateful query-plane owner is required: - CCS does not own PIT/search-context state and MUST NOT be queried on the hot path for `search_context.token` validation, - query execution and keep-alive remain fully reconstructable from the self-contained token itself, - there is no separate close-tombstone store or PIT registry in the active revision. - public PIT/search-context lifecycle in this revision is issuance + continuation + keep-alive only; there is no public early-close RPC. - query-page, composite-page, and search-context tokens remain valid until natural expiry unless the token-bound serving lineage stops being retained/query-admissible; there is no separate revocation store or close tombstone. - Elasticsearch gateway routes keep PIT lifecycle out of Profile-2 scope; public PIT endpoint parity is deferred to Profile 3. - keep-alive rules: - first-page or continuation requests MAY ask for keep-alive by setting `fetch_spec.search_context.keep_alive_ms`, - when admitted, server MAY mint a fresh token in the same lineage with extended expiry and return it in `QuerySuccessPayload.pagination.search_context`, - when the same response emits non-empty `BucketResult.composite.after_key`, server MUST return `QuerySuccessPayload.pagination.composite_page.token` together with `composite_page.expires_at_unix_ms`; `QuerySuccessPayload.pagination.search_context` remains optional and is returned only when keep-alive was explicitly admitted for the unary query path, - keep-alive MUST NOT change the pinned snapshot boundary or the admitted query digest, - clients SHOULD continue with the freshest returned token; older unexpired tokens in the same lineage remain valid until natural expiry. - malformed/scope-mismatched context token follows canonical token error mapping (`SF_BAD_REQUEST` / `SF_FORBIDDEN` by precedence); expiry on query execution continues to map to `SF_INVALID_QUERY`. ### 5.10 Collection Statistics Contract `GetCollectionStatistics` is a read-only `QueryGatewayService` RPC for optimizer/cardinality costing and planning-time heuristics. It returns one admitted statistics snapshot for one `CollectionRef`; it is not a catalog/schema substitute and it does not replace `GetReadiness`. The returned payload is a CCS-owned optimizer-statistics product and is distinct from descriptor-level `CollectionDescriptor.approx_stats`. Request contract: - request scope is `CollectionRef`. - unknown / tombstoned / fully GC-released collections MUST fail closed as `SF_RESOURCE_NOT_FOUND`. - active collections with no admitted schema lineage yet MUST fail closed as `SF_CATALOG_UNAVAILABLE` with the canonical schema-bootstrap details (`catalog_unavailable_reason=schema_bootstrap`, `schema_bootstrap_state=no_active_schema_generation`). - active collections with an admitted lineage but no query-serving-admissible `READY` generation yet MUST fail closed as `SF_CATALOG_UNAVAILABLE` with the canonical schema-convergence details (`catalog_unavailable_reason=schema_convergence`, `schema_convergence_state=no_ready_serving_generation`). - `target_fields[]` carries canonical field paths. - empty `target_fields[]` is the collection-only fast path: - `extended_requests[]` MUST also be empty, - server MUST return `collection_stats`, - server MUST NOT spend work serializing `field_stats[]` or `extended_stats[]`, - non-empty `extended_requests[]` with `target_fields=[]` is malformed and MUST fail closed (`SF_BAD_REQUEST`). - CCS MUST first resolve the current query-serving schema lineage (`latest READY` generation), then select one admitted optimizer-statistics snapshot from that lineage for the request, then validate `target_fields[]` and `extended_requests[]` against that selected snapshot's declared schema lineage and supported tuple family. - when `target_fields[]` is non-empty, every entry MUST resolve to a field path declared in the selected snapshot's schema lineage, and the list MUST be byte-unique after canonical field-path resolution within that lineage; unknown fields or duplicated target fields MUST fail closed (`SF_BAD_REQUEST`). - `extended_requests[]` is optional and requests joint statistics for multiple field paths: - every entry MUST carry `columns.len > 1`, - every entry MUST exactly match one ordered tuple declared in the selected/admitted statistics snapshot's supported extended-stat tuple family for that same selected schema lineage, - the list itself MUST be unique by exact ordered tuple identity, - the active revision does not allow ad hoc/on-demand computation for unsupported tuples, - unknown fields, singleton requests, malformed duplicated field paths inside one request, duplicated tuple requests across `extended_requests[]`, or tuple requests outside the declared support profile MUST fail closed (`SF_BAD_REQUEST`). - `max_staleness_ms` is a statistics-cache freshness budget, not a query visibility contract: - `0` => require the latest admitted statistics snapshot for the current query-serving schema lineage, - `> 0` => server MUST select the newest admitted statistics snapshot for that same lineage whose `collected_at_unix_ms` satisfies `now_unix_ms - collected_at_unix_ms <= max_staleness_ms`, - if no admitted snapshot for the current query-serving lineage satisfies that budget, the request MUST fail closed as `SF_CATALOG_UNAVAILABLE` with canonical details `catalog_unavailable_reason=statistics_freshness`, `max_staleness_ms`, `latest_stats_commit_ts`, and `latest_stats_collected_at_unix_ms`. - this RPC MUST NOT use `SF_VISIBILITY_TIMEOUT`; statistics snapshot fetch is not blocked on query-time visibility readiness. - the owning CCS statistics subsystem MUST publish immutable per-collection statistics snapshots keyed by composite durable identity (`applied_schema_generation_token`, `stats_commit_ts`) within collection scope, plus a rebuildable hot projection used to satisfy this RPC without mandatory S3 reads on the steady path. - `stats_commit_ts` remains the committed collection-state frontier covered by the snapshot, but it is not the durable snapshot key by itself because schema-only serving-lineage promotion may publish a new zero-row snapshot without advancing the user-data frontier. - every immutable statistics snapshot MUST also carry the schema lineage it was computed against: - `applied_schema_version`, - `applied_schema_generation_token`. - once any schema generation becomes query-serving-admissible for a collection, CCS MUST publish an initial zero-row optimizer-statistics snapshot for that lineage as part of the same materialization wave that makes that lineage the current `latest READY` serving generation, so the RPC has a canonical base snapshot independent of later data growth. - whenever a later schema generation becomes query-serving-admissible, CCS MUST likewise publish a zero-row optimizer-statistics snapshot for that new lineage before the lineage is allowed to become the current `latest READY` serving generation; that lineage transition MUST produce a distinct immutable snapshot durable identity even when no user-data commit frontier has advanced. - each immutable statistics snapshot MUST also declare its supported extended-stat tuple family: - tuple identity is the exact ordered canonical field-path tuple, - only tuples present in the selected/admitted snapshot's declared family may be requested through `extended_requests[]`, - the active revision does not permit on-demand computation for tuples outside that family. Response contract: - transport split: - transport `OK` returns `GetCollectionStatisticsSuccessResponse` directly. - malformed/authz/not-found/bootstrap/convergence/freshness/runtime failures MUST use non-`OK` gRPC status with canonical `ErrorStatus`; this RPC no longer exposes a public `success/error` response envelope. - `GetCollectionStatisticsSuccessResponse` MUST carry `stats_commit_ts`, the logical commit TS of the statistics snapshot returned. - `stats_commit_ts` identifies the committed collection-state frontier covered by the returned optimizer-statistics snapshot; together with `applied_schema_generation_token`, it identifies the immutable snapshot selected for this response. It is not a wall-clock publication timestamp. - `GetCollectionStatisticsSuccessResponse` MUST carry `collected_at_unix_ms`, the wall-clock collection timestamp of the returned optimizer-statistics snapshot. - `collected_at_unix_ms` is the authoritative freshness timestamp for successful `max_staleness_ms` evaluation; callers auditing freshness MUST use it rather than inferring freshness from `stats_commit_ts`. - optimizer-statistics lineage/freshness invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-QUERY-003`. - `GetCollectionStatisticsSuccessResponse` MUST also carry the schema lineage of the returned statistics snapshot: - `applied_schema_version`, - `applied_schema_generation_token`. - `field_stats[]`, `extended_stats[]`, and request-field legality are all interpreted against that returned query-serving snapshot lineage rather than against whatever newer active control-plane schema lineage may already exist. - `collection_stats` is always required on transport-`OK` success, including the collection-only fast path. - The canonical protobuf closes `GetCollectionStatisticsSuccessResponse.stats_commit_ts`, `GetCollectionStatisticsSuccessResponse.collected_at_unix_ms`, `GetCollectionStatisticsSuccessResponse.applied_schema_version`, `GetCollectionStatisticsSuccessResponse.applied_schema_generation_token`, and `GetCollectionStatisticsSuccessResponse.collection_stats` as machine-required success-path snapshot identity/state fields; transport-`OK` success without any of them is invalid. - `field_stats[]` MUST be returned in canonical `field_name` byte-ascending order. - `extended_stats[]` MUST be returned in canonical tuple-identity order: compare `columns[]` left-to-right using canonical field-path byte order, and when one tuple is a strict prefix of another, the shorter tuple sorts first. - `extended_stats[]` is keyed by the exact ordered tuple identity declared in the snapshot support profile; `ExtendedStatistics.columns` MUST echo that declared tuple order. - `FieldStatistics.stats_type` uses exactly-one branch typing. - `ScalarFieldStatistics.top_n[]` captures high-frequency skew; `histogram.buckets[]` represents the remaining long-tail distribution. `HistogramBucket.count` MUST exclude values already represented in `top_n[]`. - for `FieldCapabilitySpec.multi_valued=true`, explicit empty arrays are treated as null/missing for statistics purposes: they contribute to `null_count` and MUST NOT produce sortable extrema (`min_value` / `max_value`). - `VectorClusterCentroid.centroid` is a readback `VectorValue`; successful readback MUST set exactly one `VectorValue.data` branch, and that branch MUST be compatible with the target field's `VectorElementType`. - `VectorFieldStatistics.quantization_state` reflects the active stored/indexed vector quantization state, not request-side vector projection intent, and successful readback MUST NOT carry `VECTOR_QUANTIZATION_UNSPECIFIED`; admitted emitted values are `VECTOR_QUANTIZATION_NONE`, `VECTOR_QUANTIZATION_INT8`, and `VECTOR_QUANTIZATION_BINARY`. - TiDB-side CBO/optimizer SHOULD compare `stats_commit_ts` with its planning/read target and MAY apply uncertainty penalties when statistics are stale. Relevant error branches for this RPC: - canonical multi-violation precedence still follows `docs/specs/core/error-codes.md` Section `2.1`; this RPC does not define a local precedence override. - relevant branches include: - authorization/scope mismatch -> `SF_FORBIDDEN` - unknown collection -> `SF_RESOURCE_NOT_FOUND` - active collection with no admitted schema lineage -> `SF_CATALOG_UNAVAILABLE` - active collection with no query-serving-admissible `READY` generation -> `SF_CATALOG_UNAVAILABLE` - no admitted statistics snapshot satisfies the requested freshness budget -> `SF_CATALOG_UNAVAILABLE` - unknown field path, malformed extended-stat request, or unsupported extended-stat tuple -> `SF_BAD_REQUEST` - stats-budget or admission throttling -> `SF_RATE_LIMITED` ### 5.11 Relational Query Service Contract `RelationalQueryService` is the trusted query surface for relational engines that need route-scoped execution context on top of canonical query semantics. It exists so a relational engine can attach projected-route-only execution context without polluting shared `QueryRequest`, while still keeping projected-route fallback semantics out of the shared `QueryGatewayService` result contract. Admission is deployment-independent at the semantic level: authenticated callers are admitted through a trusted relational caller inventory that binds caller identities to route capabilities. The active revision uses two capabilities: - `projected_query`: caller may invoke `ExecuteProjectedQueryOrFallback` - `primary_query`: caller may invoke `ExecutePrimaryQuery` Callers using projected-query overlay semantics additionally need the projected-route snapshot-overlay capability: - caller may supply canonical `snapshot_overlay` - caller may supply monotonic comparable `snapshot_read_ts` In the current deployment, TiDB compute is the concrete caller mapping that holds both route capabilities, while only the projected-query route uses `snapshot_read_ts`. Request contract: - transport payload is route-specific: - `ExecuteProjectedQueryOrFallback` uses `ProjectedQueryRequest` - `ExecutePrimaryQuery` uses `PrimaryQueryRequest` - inner `query` field reuses canonical `QueryRequest` directly. - caller admission is capability-based rather than vendor-name-based: - `ExecuteProjectedQueryOrFallback` requires projected-query capability and snapshot-overlay capability - `ExecutePrimaryQuery` requires primary-query capability - the current deployment maps these capabilities onto `CLIENT_KIND_TIDB_COMPUTE`, but the public route contract does not reserve the service to TiDB by name - `snapshot_overlay` is relational-integration-private immediate-visibility execution context for trusted projected-query routes only; primary-query requests do not expose snapshot-overlay carriers at all in the active revision. - `snapshot_overlay.columns` defines the canonical overlay column dictionary when overlay is admitted. - when `snapshot_overlay.rows` is non-empty on `ExecuteProjectedQueryOrFallback`, the admitted query shape is intentionally narrower than the full canonical `QueryRequest` surface: - `query.search_payload.aggregations` MUST be empty, - retrieval/predicate trees MUST NOT use nested-query branches (`NestedClause`, `NestedPredicate`), - request MUST NOT ask for `top_hits` / `inner_hits`, - request MUST NOT require highlighting (`fetch_spec.text_options.require_highlighting=true`), - `query.fetch_spec.rescore` MUST be absent, - unsupported overlay-bearing query shapes MUST fail closed as `SF_UNSUPPORTED_QUERY_FEATURE` rather than silently dropping unsupported semantics. - `snapshot_overlay.rows` MUST be <= 1000. - `snapshot_overlay.columns[*].field_path` entries MUST be byte-unique after normalization and MUST be sorted byte-ascending. - every overlay row MUST carry `doc_id` plus `commit_ts > 0`; `is_deleted=true` requires empty `values`; non-tombstone rows MUST emit exactly one `values[i]` per `snapshot_overlay.columns[i]`, using explicit null placeholders for missing/unavailable fields; duplicate `(doc_id, commit_ts)` row identities in one request are malformed. - implementation-validated/conformance-backed overlay state split in the current revision: - no overlay => `snapshot_overlay.columns=[]`, `snapshot_overlay.rows=[]`, `snapshot_read_ts=0` - overlay present => `snapshot_overlay.columns` non-empty canonical dictionary, `snapshot_overlay.rows` non-empty, `snapshot_read_ts>0` - overlay reduction is deterministic: for one `doc_id`, the broker keeps only the highest visible `commit_ts` version before merging with persisted candidates; a visible tombstone wins over older snapshot/NRT versions. - canonical overlay digest contract: - hash input is normalized columnar overlay semantics rather than raw protobuf bytes, - columns are hashed in canonical `snapshot_overlay.columns[]` order, - rows are hashed after canonical sorting by (`doc_id`, `commit_ts`) ascending, - per-cell `FieldData` hashing follows the same float/NaN normalization rules used by `canonical_query_digest`, - continuation/token validation compares this canonical overlay digest, not caller-preserved incidental row order or protobuf map encoding. - STRICT/BOUNDED_STALENESS with overlay require `snapshot_read_ts = query.consistency.read_ts`. - BEST_EFFORT with overlay keeps `query.consistency.read_ts=0`, but `snapshot_read_ts` MUST be the authoritative execution snapshot for both persisted-reader evaluation and overlay visibility. - when overlay is present under `BEST_EFFORT`, `snapshot_read_ts` MUST equal the first-page/continuation execution snapshot materialized as `effective_read_ts`. - continuation requests carrying `snapshot_overlay` MUST resend the same canonical columnar overlay semantics and the same `snapshot_read_ts`. - token validation additionally binds a canonical overlay digest when present so page/search-context continuation stays deterministic without server-side query state. - overlay evaluation MUST reuse the same canonical analyzer selection, text mode semantics, and clause-scoring pipeline that the admitted query would use on persisted reader lanes; the overlay path is not an alternate exact-match text mode. - when overlay rows satisfy text or vector clauses, the emitted clause-level scores MUST be produced on the same lane-local scale as equivalent reader execution so later fusion, tie-break ordering, and pagination remain canonical for the admitted request. - `ExecutePrimaryQuery` intentionally carries no overlay fields, so primary-query callers cannot express partial overlay states such as `rows=[]` with non-empty `columns`. - `RelationalQueryService` is not a compatibility-edge contract for Elasticsearch or Turbopuffer and MUST fail closed for callers outside the trusted relational caller inventory. - `QueryGatewayService.ExecuteQuery` / `StreamQuery` are not the production contract when a relational engine needs trusted snapshot-overlay state. - the inner `query` still obeys all canonical `QueryRequest` validation, digest, token, and consistency rules before projected-route fallback selection, while `snapshot_overlay` / `snapshot_read_ts` add only relational-wrapper execution context. - projected/primary route split, overlay binding, and projected-route observability invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-QUERY-002`. - projected-route fallback ownership is closed before broker dispatch: - Query Gateway resolves every fallback-eligible projected-route branch (`stale_strict`, `staleness_budget_exceeded`, `non_covering_projection`) before issuing the internal broker RPC, - therefore the broker-side projected-route path is success/error only rather than fallback/error. RPC split: - `ExecuteProjectedQueryOrFallback(ProjectedQueryRequest) -> ProjectedQueryResponse` - the resolved collection descriptor MUST declare `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`. - transport `OK` returns `ProjectedQueryResponse`, which is a normal-result envelope rather than a success/error envelope. - malformed/authz/unsupported/runtime/backend failures MUST use non-`OK` gRPC status with canonical `ErrorStatus`. - `ProjectedQueryResponse` top-level observability fields (`effective_read_ts`, `safe_ts`, `canonical_query_digest`, `relational_execution_digest`, `applied_schema_version`, `applied_schema_generation_token`) carry the same meaning as on `ExecuteQuerySuccessResponse`, but they apply only to transport-`OK` normal results. - all top-level projected-route observability fields (`effective_read_ts`, `safe_ts`, `canonical_query_digest`, `relational_execution_digest`, `applied_schema_version`, `applied_schema_generation_token`) are mandatory on transport-`OK` normal results. - `canonical_query_digest` remains the shared semantic digest over the inner canonical `QueryRequest`. - `relational_execution_digest` is the exact projected-route execution digest over `canonical_query_digest` plus projected-route-only execution inputs (`snapshot_read_ts` and canonical overlay digest), so materially different overlay-bearing executions do not collapse onto one observability key. - both transport-`OK` projected-route normal-result branches MUST carry a non-empty `relational_execution_digest`: Query Broker computes it on the success path after projected-route overlay binding has been validated, and Query Gateway computes the same digest with the same canonical helper on the fallback path because no broker execution occurs. - both `success` and `fallback` therefore carry the full projected-route observability tuple rather than a best-effort subset. - exactly one `ProjectedQueryResponse.result` branch is valid on every emitted `OK` response: - `success` reuses canonical `QuerySuccessPayload` - `fallback` is a deterministic routing directive telling the caller to execute its native relational plan instead of retrying Search Fabric with the same admitted query inputs - `ExecutePrimaryQuery(PrimaryQueryRequest) -> ExecuteQuerySuccessResponse` - the resolved collection descriptor MUST declare `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`. - primary-route relational queries use success/error-only semantics with no fallback branch. - transport `OK` returns canonical `ExecuteQuerySuccessResponse`. - malformed/authz/unsupported/runtime/backend failures MUST use non-`OK` gRPC status with canonical `ErrorStatus`. Deterministic fallback branches: - fallback directive `reason` applies only to `ExecuteProjectedQueryOrFallback` and MUST be one of the three canonical branches below; `PROJECTED_QUERY_FALLBACK_REASON_UNSPECIFIED` is invalid. - strict stale snapshot => `reason=PROJECTED_QUERY_FALLBACK_REASON_STALE_STRICT` - bounded-staleness lag breach => `reason=PROJECTED_QUERY_FALLBACK_REASON_STALENESS_BUDGET_EXCEEDED` - non-covering projection for projected-route result materialization => `reason=PROJECTED_QUERY_FALLBACK_REASON_NON_COVERING_PROJECTION` - these are the only canonical non-error fallback branches in the active revision. Deterministic non-fallback error branches: - malformed request shape, ownership/applicability violations, unsupported query features, canonical schema-bootstrap / schema-convergence failures, quota/rate-limit rejection, and backend/runtime failures remain canonical errors projected through non-`OK` gRPC status plus `ErrorStatus`, rather than through `fallback`. Operational rule: - `GetReadiness` remains the general pre-query context helper and MAY still be used for diagnostics, planner warm-up, or non-hot-path admission decisions. - when TiDB uses `RelationalQueryService.ExecuteProjectedQueryOrFallback`, the gateway itself owns readiness evaluation and fallback selection inside that RPC; callers do not pay an extra external `GetReadiness` round trip on the projected-query hot path. - `ExecutePrimaryQuery` stays fail-closed success/error only; it does not admit relational snapshot-overlay carriers in the active primary-route model and does not reinterpret readiness misses or query-shape issues as fallback. ## 6. Control Plane API Design ### 6.1 Catalog Read and DDL Synchronization ### 6.1.1 Canonical Catalog Read RPCs Control-plane readback MUST use dedicated typed catalog RPCs. `GetSchemaSyncStatus` is a convergence/status RPC and MUST NOT be overloaded as a schema descriptor substitute. Boundary clarification: - `ListCollections`, `GetCollectionMetadata`, `GetCollectionSchema`, `ListNamespaces`, `GetNamespaceMetadata`, and `GetSchemaSyncStatus` are the canonical public catalog read surface. - These unary catalog-read RPCs now use direct success bodies on transport `OK` and non-`OK` gRPC status plus canonical `ErrorStatus` on failure; they no longer expose public `success/error` response envelopes. - Internal components that need admission-only CCS facts outside those public payloads (for example collection lifecycle state, descriptor revision/etag, or namespace-scoped collection-name-binding revision/etag used to invalidate cached `CollectionNameBinding` results) MUST use a private CCS admission-state read model or equivalent co-located internal interface. - when implementations cache `CollectionNameBinding -> CollectionRef` resolutions, both positive and negative cache entries MUST be bound to that namespace-scoped `collection_name_binding_revision`; a revision change invalidates the cached binding result regardless of whether the cached result was a hit or a miss. - That admission-state model is deployment-private, not part of the canonical public protobuf contract, and MUST NOT be treated as a second public catalog API. Canonical catalog-name contract (`namespace_name`, `collection_name`, `namespace_name_prefix`, `collection_name_prefix`): - admitted values are valid UTF-8 Unicode scalar sequences only. - callers MUST submit catalog names and catalog-name prefixes already normalized to Unicode NFC. - admitted catalog names and catalog-name prefixes MUST be at most 128 Unicode scalar values. - identity-bearing catalog names (`namespace_name`, `collection_name`) MUST be non-empty. - the canonical proto models `CollectionNameBinding.collection_name`, `PutNamespaceMetadataRequest.namespace_name`, and `NamespaceLookupKey.namespace_name` with machine-checkable non-empty validation, so empty-present identity binders are malformed before catalog mutation or lookup logic runs. - prefix filters are the only empty-string exception in this family: - empty `namespace_name_prefix` means no namespace prefix filter, - empty `collection_name_prefix` means no collection prefix filter. - the active revision rejects: - Unicode control characters, - Unicode bidirectional formatting/isolate control characters, - default-ignorable format characters. - non-canonical or otherwise rejected catalog-name input is malformed and MUST fail closed as `SF_BAD_REQUEST`; CCS and gateways MUST NOT silently normalize on write. - overlength catalog-name input is likewise malformed; CCS, gateways, replay indices, and catalog persistence MUST preserve the exact admitted string rather than truncating or hashing identity-bearing names. - equality, uniqueness, business replay identity, prefix filtering, catalog-page-token last-seen keys, and stable ascending ordering all operate on the NFC-normalized string. - canonical storage and comparison semantics are binary, not locale-aware: - implementations MUST use binary collation / bytewise comparison over canonical UTF-8, - case-insensitive, accent-insensitive, width-insensitive, or locale-specific ordering MUST NOT participate in canonical catalog identity semantics. - prefix matching is defined on the canonical NFC string: - empty prefix means no prefix filter, - non-empty prefix means canonical code-point prefix match on the NFC string, - because admitted catalog names are already canonical UTF-8 under binary collation, implementations MAY realize this as a byte-range scan. Turbopuffer mapping model in this revision: - external Turbopuffer account -> canonical `namespace_id` is explicitly outside the current design scope and belongs to external system control-plane/authz resolution. - this API consumes that already-resolved `namespace_id` as an input fact; it MUST NOT derive namespace identity from Turbopuffer path tokens. - external Turbopuffer `{namespace}` token maps to canonical `CollectionDescriptor.collection_name` inside the resolved `namespace_id`. `ListCollections` contract: - lists catalog-known collections within one canonical `namespace_id`. - transport `OK` returns `ListCollectionsSuccessResponse` directly. - only collections with internal lifecycle state `ACTIVE` are returned; `TOMBSTONED` collections MUST be omitted. - request supports deterministic pagination (`next_page_token`, `page_size`) and prefix filtering (`collection_name_prefix`). - stable ordering is `collection_name ASC` over the canonical NFC string under binary collation. - each `CollectionDescriptor` MUST carry: - collection identity (`collection`) - stable catalog `collection_name` - creation/update timestamps (`created_at_unix_ms`, `updated_at_unix_ms`) - typed aggregate stats (`approx_stats`) when available - typed collection encryption metadata (`encryption`) - typed collection index-progress metadata (`index`) - descriptor completeness state (`completeness`) - optional supplemental metadata (`options`) for source round-trip / operational keys only - `CollectionDescriptor.approx_stats` is the CCS-owned aggregate logical-bytes/row-count estimate for the active collection lineage and is advisory rather than a strict read-time snapshot. - `CollectionDescriptor.index` is the CCS-owned collection-level indexing/progress state for the active collection generation: - `CollectionDescriptor.index` MUST set exactly one branch (`up_to_date` or `updating`); empty `IndexDescriptor{}` carriers are semantically invalid. - `up_to_date` means CCS currently observes no outstanding indexing backlog for the active generation. - `updating.unindexed_bytes` is the remaining estimated backlog for that collection and generation. - collection-level `approx_stats` and `index` are allowed to lag the latest manifest-head advancement by one successful CCS projector/reconciler cycle; they MUST NOT be gateway-synthesized. - when `CollectionDescriptor.index` is still unpublished, successful descriptor reads MUST keep `completeness.publishing` active and `publishing.missing_fields[]` MUST include `index`; `ready` descriptors MUST NOT omit `index`. - when `next_page.token` is non-empty, `next_page.expires_at_unix_ms` MUST carry the unified-token expiry timestamp for that continuation token; when `next_page.token` is empty, `next_page.expires_at_unix_ms` MUST be `0`. - Turbopuffer `GET /v1/namespaces` MUST anchor on this RPC and project `NamespaceSummary.id` from `collection_name`. `GetCollectionMetadata` contract: - request lookup is typed: either direct `CollectionRef` or name-bound `CollectionNameBinding{namespace_id, collection_name}`. - transport `OK` returns `GetCollectionMetadataSuccessResponse` directly. - `lookup.target` MUST set exactly one branch; empty `CollectionLookupKey` carrier is malformed and MUST fail closed (`SF_INVALID_ARGUMENT`). - success payload returns exactly one authoritative `CollectionDescriptor`. - The canonical protobuf closes `GetCollectionMetadataSuccessResponse.descriptor` as a machine-required success payload. - requests targeting collections in internal lifecycle state `TOMBSTONED` MUST fail closed as `SF_RESOURCE_NOT_FOUND`. - requests targeting collections that never existed or whose prior tombstone reservation has already been fully GC-released MUST also fail closed as `SF_RESOURCE_NOT_FOUND`. - typed collection metadata (`created_at_unix_ms`, `updated_at_unix_ms`, `approx_stats`, `encryption`, `index`) MUST come from this RPC, not from gateway-local guesses or opaque `options` reconstruction. - `CollectionDescriptor.options` remains supplemental metadata only and MUST NOT be the sole carrier for semantics already modeled in typed collection metadata. - `CollectionDescriptor.completeness` MUST set exactly one branch: - `ready`: descriptor is complete for metadata-parity consumers; `approx_stats` and `index` MUST be populated. - `publishing`: lagging typed descriptor publication is not complete yet; `publishing.missing_fields[]` MUST be non-empty and enumerate the still-missing lagging typed metadata branches required for metadata-parity consumers. - `CollectionDescriptor.completeness` is CCS-owned publication state, not a gateway hint: - it is recomputed from authoritative CCS typed descriptor fields in materialized state rather than from separately authored mutable completeness state, - it MUST transition to `publishing` whenever a required lagging typed field is not yet published, - and it MUST NOT intentionally lag CCS materialized truth on successful reads. - descriptor completeness / missing-fields invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-CATALOG-001`. - `publishing.missing_fields[]` is a canonical public set projection for both collection and namespace descriptors in this revision: - allowed values are exactly `approx_stats` and `index` - values MUST be unique and returned in canonical byte-ascending order - `diagnostics["missing_descriptor_fields"]` MUST be formed by joining that same ordered list with commas when metadata-parity consumers fail closed on `publishing` - successful `CollectionDescriptor` readback MUST always carry `encryption`; implementations MUST NOT explain absent authoritative encryption through `completeness.publishing`. - The canonical protobuf closes `CollectionDescriptor.collection`, `CollectionDescriptor.collection_name`, `CollectionDescriptor.created_at_unix_ms`, `CollectionDescriptor.updated_at_unix_ms`, `CollectionDescriptor.encryption`, and `CollectionDescriptor.completeness` as machine-required success-path descriptor fields. - missing `approx_stats` is one such required-field-missing branch: - a descriptor with absent `approx_stats` MUST NOT be reported as `ready`, - and `publishing.missing_fields[]` MUST include `approx_stats`. - missing `index` is another such required-field-missing branch: - a descriptor with absent `index` MUST NOT be reported as `ready`, - and `publishing.missing_fields[]` MUST include `index`. - Turbopuffer `GET /v1/namespaces/{namespace}/metadata` and any other route that requires metadata-parity typed fields MUST anchor on this RPC plus `GetCollectionSchema`; gateway MUST NOT synthesize metadata defaults. - If `CollectionDescriptor.completeness.publishing` is the active branch, metadata-parity routes MUST fail closed with `SF_CATALOG_UNAVAILABLE` and surface `ErrorStatus.context.descriptor_publishing_unavailable`. - For that fail-closed branch, `ErrorStatus.context.descriptor_publishing_unavailable.descriptor_state` MUST be `publishing`, `ErrorStatus.context.descriptor_publishing_unavailable.missing_descriptor_fields[]` MUST equal the reported canonical missing-field set, and `diagnostics["descriptor_state"]` / `diagnostics["missing_descriptor_fields"]` MAY mirror the same values for compatibility/string-only consumers. `PutCollectionMetadata` contract: - request uses `CollectionNameBinding{namespace_id, collection_name}` as the name-based collection identity key. - transport `OK` returns `PutCollectionMetadataSuccessResponse` directly. - metadata-mutation success / lifecycle-conflict invariants for this section are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-CATALOG-002`. - this RPC is the typed create-or-update carrier for collection-descriptor state that exists independently of schema body (notably external collection name binding and typed encryption metadata). - `CollectionNameBinding` is a pre-identity-resolution key, not the long-term mutation owner: - first allocation for a never-before-seen active binding resolves on the collection-name lane and allocates a stable `collection_id`, - once an active binding resolves to an existing `CollectionRef`, the authoritative mutation apply MUST serialize on that collection's stable-id-owned lineage before commit/ACK. - `definition` is a required whole-message carrier on admitted requests: - whole-message omission is malformed (`SF_BAD_REQUEST`), - create/ensure branches still require explicit `definition.encryption`, - update branches preserve the current encryption descriptor only by omitting `definition.encryption` inside the present `definition` message. - success payload returns authoritative `CollectionDescriptor` with assigned/resolved `CollectionRef`. - successful public control-plane mutation ACK means the authoritative online read projection for the affected public catalog surface has already been updated, so subsequent successful public catalog reads observe the acknowledged descriptor state. - `created` means a new collection descriptor (and stable `collection_id`) was allocated. - `updated` means mutable typed/supplemental descriptor state changed. - `already_applied` means the authoritative post-apply descriptor state was already equal to the requested typed metadata state and the admitted command therefore produced no state change. - `already_applied` is state-based, not replay-key-based: - transport replay with the same dedupe key MUST return the original success payload verbatim, - a newly admitted metadata command with a different `idempotency_key` that finds the same already-converged target state MUST also return `already_applied`, - implementations MUST NOT force `updated` solely because the caller changed `idempotency_key`. - successful `PutCollectionMetadata` responses MUST emit exactly one of `created`, `updated`, or `already_applied`. - The canonical protobuf closes `PutCollectionMetadataSuccessResponse.descriptor` as a machine-required success payload. - non-canonical `collection_name` input is malformed (`SF_BAD_REQUEST`) by the canonical catalog-name contract. - create/ensure that allocates a new collection MUST carry explicit `definition.encryption`; omission on such requests is malformed (`SF_BAD_REQUEST`). - when `definition.encryption` is present, it MUST select exactly one valid mode; a present-but-empty carrier is malformed (`SF_INVALID_ARGUMENT`), and `cmek.key_name` MUST be non-empty. - update requests MAY omit `definition.encryption` only to preserve the existing collection encryption descriptor unchanged. - if the target namespace is in internal lifecycle state `QUIESCING` or `TOMBSTONED`, a create branch that would allocate a new collection in that namespace MUST fail closed as `SF_VERSION_CONFLICT` and include the canonical lifecycle-conflict detail keys from `docs/specs/core/error-codes.md`. - if `CollectionNameBinding{namespace_id, collection_name}` is still reserved by an internal-lifecycle-`TOMBSTONED` collection whose tombstone GC has not yet released the name, this RPC MUST NOT allocate a replacement collection; same-name recreation remains blocked until the reservation is released and the rejection MUST fail closed as `SF_VERSION_CONFLICT` with the canonical lifecycle-conflict detail keys. - when a request resolves an existing `CollectionRef`, the stable-id-owned apply phase MUST re-check the authoritative collection lifecycle/descriptor lineage before committing: - a concurrent transition that tombstones the collection, releases the binding, or otherwise invalidates the resolved active lineage MUST reject the metadata update deterministically as `SF_VERSION_CONFLICT` with the canonical lifecycle-conflict detail keys, - that post-resolution lifecycle-conflict detail shape is the same canonical family used for other lifecycle/name-reservation branches, with `reservation_scope=resolved_collection_lineage` and the tombstone reservation fields projected from the lineage that won the race, - implementations MUST NOT acknowledge a name-lane descriptor update after the collection-id-owned lineage has already transitioned to a conflicting terminal state. - Turbopuffer routes that create/ensure collection identity or enforce write-route `encryption` MUST anchor on this RPC before schema/mutation emission. `PutNamespaceMetadata` contract: - request uses stable catalog `namespace_name` as the namespace identity key for create/update/ensure. - transport `OK` returns `PutNamespaceMetadataSuccessResponse` directly. - this RPC is the typed create-or-update carrier for namespace-descriptor state that exists independently of collection/schema state (notably namespace identity allocation and typed namespace encryption metadata). - `namespace_name` is the pre-allocation routing key, not the long-term mutation owner: - first allocation for a never-before-seen active namespace resolves on the namespace-name lane and allocates a stable `namespace_id`, - once a namespace already has a stable `namespace_id`, subsequent namespace descriptor mutations MUST serialize on that namespace-owned lineage together with lifecycle transitions before commit/ACK. - `definition` is a required whole-message carrier on admitted requests: - whole-message omission is malformed (`SF_BAD_REQUEST`), - create/ensure branches still require explicit `definition.encryption`, - update branches preserve the current encryption descriptor only by omitting `definition.encryption` inside the present `definition` message. - success payload returns authoritative `NamespaceDescriptor` with assigned/resolved `namespace_id` and explicit public readable-state lifecycle projection. - The canonical protobuf closes `PutNamespaceMetadataSuccessResponse.descriptor` as a machine-required success payload. - successful public control-plane mutation ACK means the authoritative online read projection for the affected public catalog surface has already been updated, so subsequent successful public catalog reads observe the acknowledged descriptor state. - `created` means a new namespace descriptor (and stable `namespace_id`) was allocated. - `updated` means mutable typed/supplemental namespace state changed. - `already_applied` means the authoritative post-apply descriptor state was already equal to the requested typed metadata state and the admitted command therefore produced no state change. - `already_applied` is state-based, not replay-key-based: - transport replay with the same dedupe key MUST return the original success payload verbatim, - a newly admitted metadata command with a different `idempotency_key` that finds the same already-converged target state MUST also return `already_applied`, - implementations MUST NOT force `updated` solely because the caller changed `idempotency_key`. - successful `PutNamespaceMetadata` responses MUST emit exactly one of `created`, `updated`, or `already_applied`. - non-canonical `namespace_name` input is malformed (`SF_BAD_REQUEST`) by the canonical catalog-name contract. - create/ensure that allocates a new namespace MUST carry explicit `definition.encryption`; omission on such requests is malformed (`SF_BAD_REQUEST`). - when `definition.encryption` is present, it MUST select exactly one valid mode; a present-but-empty carrier is malformed (`SF_INVALID_ARGUMENT`), and `cmek.key_name` MUST be non-empty. - update requests MAY omit `definition.encryption` only to preserve the existing namespace encryption descriptor unchanged. - if `namespace_name` is still reserved by an internal-lifecycle-`TOMBSTONED` namespace whose lifecycle GC has not yet released the name, this RPC MUST NOT allocate a replacement namespace; same-name recreation remains blocked until the reservation is released and the rejection MUST fail closed as `SF_VERSION_CONFLICT` with the canonical lifecycle-conflict detail keys. - when a request resolves an existing namespace lineage, the stable-id-owned apply phase MUST re-check authoritative namespace revision/lifecycle before committing: - a concurrent transition that moves the namespace to a conflicting terminal lineage state MUST reject the metadata update deterministically as `SF_VERSION_CONFLICT` with the canonical lifecycle-conflict detail keys, - that post-resolution lifecycle-conflict detail shape is the same canonical family used for other lifecycle/name-reservation branches, with `reservation_scope=resolved_namespace_lineage` and `reservation_release_at_unix_ms` present only when the winning lineage state is `TOMBSTONED`, - implementations MUST NOT acknowledge a name-lane descriptor update after the namespace-owned lineage has already committed a conflicting lifecycle transition. - namespace provisioning workflows that need deterministic namespace allocation before any collection operation MUST anchor on this RPC. - namespace lifecycle transitions remain private CCS/admin workflows, but successful public namespace descriptors MUST project the authoritative readable-state lifecycle through `NamespaceDescriptor.lifecycle`: - `ACTIVE`: create/update/ensure is admitted normally, and successful `PutNamespaceMetadata` / `ListNamespaces` / `GetNamespaceMetadata` descriptors MUST set the `active` lifecycle branch. - `QUIESCING`: updates to the existing namespace descriptor remain admitted on the same namespace-owned lineage that serializes lifecycle transitions, but new collection allocation inside that namespace MUST be rejected by collection admission as `SF_VERSION_CONFLICT` with the canonical lifecycle-conflict detail keys; successful public descriptors MUST set the `quiescing` lifecycle branch. - `TOMBSTONED`: namespace readback fails closed as `SF_RESOURCE_NOT_FOUND`, `ListNamespaces` omits the namespace, same-name recreation remains blocked until internal lifecycle GC releases the reservation, and this state MUST NOT appear in successful public `NamespaceDescriptor.lifecycle` projection in this revision. - transition into internal lifecycle state `TOMBSTONED` is admitted only after every child collection in that namespace is already in a terminal hidden state (for example collection tombstone active and no longer publicly listable/readable); namespace tombstoning is therefore not allowed to leave publicly visible descendant collections behind. `GetCollectionSchema` contract: - request scope is `CollectionRef`. - transport `OK` returns `GetCollectionSchemaSuccessResponse` directly. - success payload is `CollectionSchemaDescriptor`, which MUST carry: - collection identity (`collection`) - applied schema lineage (`applied_schema_version`, `applied_commit_ts`, `applied_schema_generation_token`) - shared schema body (`definition`), where `definition` contains: - authoritative `collection_truth_mode` - schema payload (`doc_id_mapping`, `fields`, `settings`) - active evolution policy (`active_schema_update_policy`) - resolved query-window contract (`effective_max_result_window`) - requests targeting collections in internal lifecycle state `TOMBSTONED` MUST fail closed as `SF_RESOURCE_NOT_FOUND`. - requests targeting collections that never existed or whose prior tombstone reservation has already been fully GC-released MUST fail closed as `SF_RESOURCE_NOT_FOUND`. - active collections that have catalog identity but no admitted schema lineage yet MUST fail closed as `SF_CATALOG_UNAVAILABLE` rather than as `SF_RESOURCE_NOT_FOUND`. - when that schema-bootstrap branch is selected, `ErrorStatus.diagnostics` MUST include `catalog_unavailable_reason=schema_bootstrap` and `schema_bootstrap_state=no_active_schema_generation`. - this RPC remains control-plane readback of the authoritative active schema pointer; its success does not by itself mean the returned generation is already query-serving-admissible. - returned `effective_max_result_window` MUST be the authoritative effective value after control-plane default resolution; stateless gateways and execution nodes MUST NOT guess, synthesize, or locally default this field. - The canonical protobuf closes `GetCollectionSchemaSuccessResponse.schema` as a machine-required field and models `effective_max_result_window` as a non-optional scalar on both `GetCollectionSchema` and `SchemaSyncStatus` success readbacks. - Elasticsearch `GET /{index}/_mapping` and Turbopuffer `GET /v1/namespaces/{namespace}/schema` MUST anchor readback on this RPC after concrete collection resolution. `ListNamespaces` contract: - lists catalog-known namespaces using stable `NamespaceDescriptor` entries. - transport `OK` returns `ListNamespacesSuccessResponse` directly. - only namespaces with internal lifecycle state `ACTIVE` or `QUIESCING` are returned; `TOMBSTONED` namespaces MUST be omitted. - every returned descriptor MUST project readable-state lifecycle explicitly: `NamespaceDescriptor` MUST set exactly one public lifecycle branch (`active` or `quiescing`) according to authoritative CCS state. - request supports deterministic pagination (`next_page_token`, `page_size`) and prefix filtering (`namespace_name_prefix`). - stable ordering is `namespace_name ASC` over the canonical NFC string under binary collation. - one multi-page `ListNamespaces` traversal is not a snapshot session in this revision; pages are individually authoritative but may observe concurrent mutation between requests according to the catalog pagination contract below. - each `NamespaceDescriptor` MUST carry: - `namespace_id` - stable catalog `namespace_name` - creation/update timestamps (`created_at_unix_ms`, `updated_at_unix_ms`) - typed public lifecycle (`lifecycle`) - typed aggregate stats (`approx_stats`) when available - typed namespace encryption metadata (`encryption`) - typed namespace index-progress metadata (`index`) - descriptor completeness state (`completeness`) - optional supplemental metadata (`options`) for source round-trip / operational keys only - `NamespaceDescriptor.approx_stats` is the CCS-owned aggregate over active collections in that namespace and is advisory rather than a strict read-time snapshot. - namespaces with zero active child collections MUST publish the zero-child aggregate synchronously together with `index=up_to_date`, so that successful zero-child descriptors remain `completeness.ready` rather than lingering in `publishing`. - `NamespaceDescriptor.index` is the CCS-owned namespace aggregate indexing/progress state: - `NamespaceDescriptor.index` MUST set exactly one branch (`up_to_date` or `updating`); empty `IndexDescriptor{}` carriers are semantically invalid. - namespaces with zero active child collections MUST return `up_to_date`; - `up_to_date` means every active child collection is currently `up_to_date`, - `updating.unindexed_bytes` is the sum of the active child-collection backlog currently known to CCS. - `NamespaceDescriptor.completeness` MUST set exactly one branch: - `ready`: descriptor is complete for namespace metadata consumers; `approx_stats` and `index` MUST be populated. - `publishing`: lagging typed namespace descriptor publication is not complete yet; `publishing.missing_fields[]` MUST be non-empty and enumerate still-missing required lagging typed metadata branches. - `NamespaceDescriptor.completeness` is CCS-owned publication state, not a gateway hint: - it is recomputed from authoritative CCS typed namespace descriptor fields in materialized state rather than from separately authored mutable completeness state, - it MUST transition to `publishing` whenever a required lagging typed field is not yet published, - and it MUST NOT intentionally lag CCS materialized truth on successful reads. - successful `NamespaceDescriptor` readback MUST always carry `encryption`; implementations MUST NOT explain absent authoritative encryption through `completeness.publishing`. - missing `approx_stats` is one required-field-missing branch: - a descriptor with absent `approx_stats` MUST NOT be reported as `ready`, - and `publishing.missing_fields[]` MUST include `approx_stats`. - missing `index` is another required-field-missing branch: - a descriptor with absent `index` MUST NOT be reported as `ready`, - and `publishing.missing_fields[]` MUST include `index`. - namespace-level `approx_stats` may lag child collection projections by one successful namespace aggregation cycle and `index` may likewise lag while one or more active child collections still exist; they MUST NOT be gateway-synthesized. - transitions to zero active child collections are a special synchronous publication rule, not an eventually consistent aggregate: - when the active child-collection set becomes empty, CCS MUST recompute and publish `index=up_to_date` before acknowledging the control-plane transition that removed the last active child or before serving a subsequent successful namespace read from authoritative state, - implementations MUST NOT return a stale non-`up_to_date` namespace index once zero active child collections is the authoritative state. - when `next_page.token` is non-empty, `next_page.expires_at_unix_ms` MUST carry the unified-token expiry timestamp for that continuation token; when `next_page.token` is empty, `next_page.expires_at_unix_ms` MUST be `0`. - this RPC is namespace/account catalog readback; it is not the carrier for Turbopuffer external `namespace` listing in the active mapping model. `GetNamespaceMetadata` contract: - request lookup is typed: either direct `namespace_id` or stable `namespace_name`. - transport `OK` returns `GetNamespaceMetadataSuccessResponse` directly. - `lookup.target` MUST set exactly one branch; empty `NamespaceLookupKey` carrier is malformed and MUST fail closed (`SF_INVALID_ARGUMENT`). - success payload returns exactly one `NamespaceDescriptor`. - The canonical protobuf closes `GetNamespaceMetadataSuccessResponse.descriptor` as a machine-required success payload. - requests targeting namespaces in internal lifecycle state `TOMBSTONED` MUST fail closed as `SF_RESOURCE_NOT_FOUND`. - requests targeting namespaces that never existed or whose prior tombstone reservation has already been fully GC-released MUST also fail closed as `SF_RESOURCE_NOT_FOUND`. - successful readback MUST set exactly one `NamespaceDescriptor.lifecycle` branch (`active` or `quiescing`); `TOMBSTONED` is not a successful public-read outcome in this revision. - typed namespace metadata (`created_at_unix_ms`, `updated_at_unix_ms`, `lifecycle`, `approx_stats`, `encryption`, `index`) MUST come from this RPC, not from gateway-local guesses or from opaque `options` reconstruction. - `NamespaceDescriptor.options` remains supplemental metadata only and MUST NOT be the sole carrier for semantics already modeled in typed namespace metadata. - The canonical protobuf closes `NamespaceDescriptor.namespace_id`, `NamespaceDescriptor.namespace_name`, `NamespaceDescriptor.created_at_unix_ms`, `NamespaceDescriptor.updated_at_unix_ms`, `NamespaceDescriptor.encryption`, and `NamespaceDescriptor.completeness` as machine-required success-path descriptor fields. - `NamespaceDescriptor.completeness` follows the same fail-closed publication contract as `CollectionDescriptor.completeness`, including the canonical `publishing.missing_fields[]` vocabulary/order/detail-projection rules above: - `ready` means namespace typed metadata is complete for metadata-parity consumers, - `publishing` means `publishing.missing_fields[]` names the still-missing required typed fields, - metadata-parity consumers that require complete typed namespace metadata MUST fail closed with `SF_CATALOG_UNAVAILABLE` when `publishing` is active and MUST surface `ErrorStatus.context.descriptor_publishing_unavailable`. - for that fail-closed branch, `ErrorStatus.context.descriptor_publishing_unavailable.descriptor_state` MUST be `publishing`, `ErrorStatus.context.descriptor_publishing_unavailable.missing_descriptor_fields[]` MUST equal the reported canonical missing-field set, and `diagnostics["descriptor_state"]` / `diagnostics["missing_descriptor_fields"]` MAY mirror the same values for compatibility/string-only consumers. - this RPC is namespace/account metadata readback; it is not the carrier for Turbopuffer external collection metadata in the active mapping model. - `PutNamespaceMetadata.namespace_name` and `GetNamespaceMetadata.lookup.namespace_name` are the same canonical namespace-name identity surface: - callers that do not yet know `namespace_id` MAY read back the authoritative `NamespaceDescriptor` by `namespace_name` without list-scan fallback, - once a workflow has resolved `namespace_id`, later stable-id reads SHOULD prefer `lookup.namespace_id`. Control-plane catalog pagination token contract (`ListCollections`, `ListNamespaces`): - `next_page_token` is an opaque CCS-issued continuation token; empty means first page. - catalog page tokens are part of the unified token model from Section `3.6`; they MUST use the same signing format, `key_id`, TTL policy, and key-rotation overlap window as query page/search-context tokens. - callers MUST NOT treat the token as a plain last-seen key or construct it locally. - replaying a token with any bound request field changed MUST fail closed as `SF_BAD_REQUEST`. - structurally malformed, expired, signature-invalid, or semantic list-contract-mismatched catalog page tokens MUST fail closed as `SF_BAD_REQUEST`. - caller-identity or authorization mismatch against token-bound identity fields (`server-authenticated caller identity`, `client_id`, `client_kind`, and `principal_subject` when present) MUST fail closed as `SF_FORBIDDEN`. - `next_page.token` is empty iff no further page exists. - when `next_page.token` is non-empty, success payload MUST also return `next_page.expires_at_unix_ms` carrying the token expiry timestamp; when `next_page.token` is empty, that field MUST be `0`. - catalog list traversal semantics are keyset-based and per-page linearizable, not snapshot-pinned across the whole traversal: - each successful page reads from the latest authoritative online catalog projection at the time that page is served, - tokens do not bind a catalog snapshot revision or catalog-read timestamp in this revision, - concurrent create/drop/recreate activity between pages may therefore cause cross-page omissions or newly visible rows in one logical traversal, - callers that require whole-list snapshot semantics MUST use an external higher-level fence until a future revision introduces explicit list-snapshot binding. Read-only catalog RPCs (`ListCollections`, `GetCollectionMetadata`, `GetCollectionSchema`, `ListNamespaces`, `GetNamespaceMetadata`, `GetSchemaSyncStatus`) MAY omit `RequestMeta.idempotency_key`, but when present it remains trace/audit metadata. `PutCollectionMetadata` and `PutNamespaceMetadata` are state-mutating RPCs and therefore MUST carry non-empty `RequestMeta.idempotency_key`. #### Canonical Document-ID Allocation RPC `IdAllocatorService.AllocateIds` is the canonical pre-ingest control-plane surface for create/insert document-identity authority on collections whose schema chooses allocator-backed identity. It covers both allocator-issued IDs for omitted source-side identity and allocator reservation/validation of caller-supplied create/insert identity before canonical mutation construction. Contract rules: - allocator issuance is collection-scoped and admits only resolved stable `CollectionRef`; unresolved names are out of contract because same-name recreation must not share allocator state. - the RPC is replay-governed and therefore MUST carry non-empty `RequestMeta.idempotency_key`; replay identity is `(server-authenticated caller identity, client_id, client_kind, rpc_method, collection, idempotency_key)`. - request shape is explicitly route-scoped and source-ordered: - `route_intent` MUST be `ID_ALLOCATION_ROUTE_INTENT_CREATE_INSERT` in the current revision, - `plan.items[]` MUST be non-empty and preserve source-input order, - each `plan.items[]` entry MUST set exactly one branch: - `issue` for a source row whose create/insert identity is omitted, - `explicit_doc_id` for a caller-supplied create/insert identity that must be reserved/validated, - explicit `explicit_doc_id` values within one plan MUST be unique, - `plan.items[]` cardinality MUST NOT exceed the canonical allocator request-cardinality guardrail (`max_plan_items_per_request`); over-limit requests fail closed as `SF_TOO_MANY_OPERATIONS` with the allocator plan-size detail projection from `docs/specs/core/error-codes.md`. - same replay key plus identical canonical payload (`collection`, `route_intent`, and ordered `plan.items[]`) MUST return the original realized-ID result verbatim; same replay key plus different canonical payload MUST fail closed as `SF_IDEMPOTENCY_CONFLICT`, including cases where the conflicting retry would also violate allocator business rules on a fresh evaluation. - successful replay resolution is authoritative before fresh allocator admissibility re-evaluation: - later collection tombstone, policy removal, caller-profile change, or schema-generation rollover MUST NOT rewrite a previously successful replay into a fresh `SF_RESOURCE_NOT_FOUND`, `SF_OWNERSHIP_CONFLICT`, or other new-allocation outcome, - replay MUST return the originally recorded `realized_doc_ids[]` and the originally recorded `schema_generation_token`, - replay MUST also return an `allocator_receipt` whose validated claims are identical to the original successful allocator result, so the caller can still construct the follow-on `MutationBatch` without reopening allocator state, - current CCS admission state governs only fresh allocation attempts that do not hit a stored successful replay result. - fresh allocator admission executes `plan.items[]` sequentially in source order against one collection-local dense-burn reservation space and one returned `schema_generation_token`; the service MUST NOT split one admitted plan across multiple allocator apply units. - success returns `realized_doc_ids[]` in the same order and cardinality as `plan.items[]`: - `issue` items emit allocator-issued canonical `uint64` IDs with `allocator_issued=true`, - `explicit_doc_id` items echo the reserved caller-supplied IDs with `allocator_issued=false`, - later items in the same plan observe watermark changes caused by earlier explicit reservations in that same ordered execution, - the service MUST NOT partially succeed with a shorter result set. - success also returns the active `schema_generation_token` observed during allocator admission so the caller can immediately lower a follow-on `MutationBatch` against the same admitted schema fence; later ingest still re-validates that fence and may reject if the collection lineage changed after allocation. - success also returns `allocator_receipt`, a signed follow-on attestation that the caller MUST copy unchanged into `MutationBatch.allocator_receipt` whenever the batch realizes allocator-backed create/insert identity; ingest validates that receipt against caller identity, collection scope, route intent, realized-ID order/content, and `schema_generation_token`. - durable replay authority in the active revision is success-only: - replay-governed successful outcomes MUST be durably recorded and returned verbatim on identical replay, - failed attempts MAY be logged for audit/operations, but they do not define a second public replay payload family beyond the canonical non-`OK` gRPC + `ErrorStatus` failure surface unless a future revision specifies one explicitly. - the allocator is not an ingest-plane fallback: `StreamMutations` MUST still reject missing `RowMutation.doc_id`, and services MUST NOT perform hidden server-side ID allocation during ingest admission. - collection schema interaction: - `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL` collections MUST keep `DocIDMapping.generation_policy=DOC_ID_GENERATION_POLICY_CALLER_REQUIRED`, MUST keep `CollectionSettings.allocator_admission_policy` absent, and MUST reject `AllocateIds` on the write path. - `DOC_ID_GENERATION_POLICY_CALLER_REQUIRED` collections MUST reject `AllocateIds` on the write path. - `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL` collections MUST also carry `CollectionSettings.allocator_admission_policy`, whose `caller_profile` is the authoritative collection-level caller profile for allocator-backed create/insert routes and whose positive `max_plan_items_per_request` plus positive `max_forward_reservation_jump` fields are the authoritative discoverable allocator guardrail ceilings for that collection. - on those collections, both omitted-ID create/insert and explicit-ID create/insert resolve through `AllocateIds`; patch/delete/update-style routes remain explicit-identity-only and MUST NOT call allocator. - caller applicability in the active design: - Elasticsearch write gateway MAY use this RPC for auto-ID routes that omit source `_id`; if the same collection/caller path admits explicit create/insert `_id`, the gateway reserves that explicit `_id` through allocator before lowering the canonical mutation. - the current revision defines exactly one allocator caller profile: `ALLOCATOR_CALLER_PROFILE_ES_WRITE_GATEWAY_ONLY`. - that profile admits Elasticsearch write gateway create/insert routes only. - Turbopuffer write gateway and TiDB federated-table write gateway allocator use remain future-only design space and are out of contract for the current revision until a later canonical profile expansion lands. - allocator identity-space and watermark rules: - allocator-issued and allocator-reserved explicit create/insert IDs share one collection-local canonical identity space, - `next_doc_id` is a dense-burn watermark/exclusive upper bound: once allocator durable admission advances it to `N`, every lower `doc_id < N` becomes permanently unavailable for fresh allocation, including IDs skipped by earlier high explicit reservations or discarded lease tails, - fresh explicit create/insert reservation that names `doc_id < next_doc_id` MUST fail closed as `SF_VERSION_CONFLICT` with deterministic diagnostics `doc_id` and `allocator_conflict_reason=below_watermark_burned_gap`, - explicit create/insert reservation that would advance the watermark farther than the admitted forward-reservation window (`max_forward_reservation_jump`) MUST fail closed as `SF_TOO_MANY_OPERATIONS` with `limit_scope=allocator_forward_reservation_window`, - explicit create/insert reservation with `doc_id >= next_doc_id` and within the admitted forward-reservation window MUST advance allocator watermark to `doc_id + 1` as part of allocator durable admission, even if the later mutation never becomes visible, - allocator request execution MUST fail closed as `SF_TOO_MANY_OPERATIONS` with `limit_scope=allocator_identity_space_remaining` when the remaining `uint64` identity space cannot satisfy the rejected issuance/reservation step, - failed allocator admission MUST NOT advance watermark. - response-shape consequence for omitted-id routes: - the compatibility/native write route that allowed omission MUST return the realized document identity in source-input order, - route-owned response reconstruction MUST distinguish caller-supplied vs allocator-issued identity origin for each admitted source row when that distinction is externally observable, - this is a route-level/write-API obligation and does not require `MutationAck` itself to grow a second public identity carrier. - deterministic failure mapping: - unknown / tombstoned / fully GC-released collection lineage -> `SF_RESOURCE_NOT_FOUND`, - `route_intent=UNSPECIFIED`, empty `plan.items[]`, plan items with unset/invalid exact-one branch, duplicate explicit IDs in one plan, or other illegal create/insert allocator syntax -> `SF_INVALID_ARGUMENT` or `SF_BAD_REQUEST` by canonical malformed-vs-semantic precedence, - authenticated caller lacks collection access -> `SF_FORBIDDEN`, - collection identity exists but the authoritative allocator admission facts are temporarily unavailable or incomplete -> `SF_CATALOG_UNAVAILABLE`, - when that unavailable branch is selected because no active schema generation has been admitted yet, `ErrorStatus.context.allocator.catalog_unavailable.schema_bootstrap.schema_bootstrap_state` MUST be `no_active_schema_generation`, and diagnostics MUST include `catalog_unavailable_reason=schema_bootstrap` plus `schema_bootstrap_state=no_active_schema_generation`, - collection truth mode / `generation_policy` / `allocator_admission_policy` does not admit allocator authority for this caller path -> `SF_OWNERSHIP_CONFLICT`, - explicit-ID reservation collides with the dense-burn allocator watermark on the fresh-allocation path -> `SF_VERSION_CONFLICT` with `ErrorStatus.context.allocator.reservation_conflict.{doc_id, conflict_reason=ALLOCATOR_RESERVATION_CONFLICT_REASON_BELOW_WATERMARK_BURNED_GAP}`, - plan cardinality exceeds the admitted request-item ceiling -> `SF_TOO_MANY_OPERATIONS` with `ErrorStatus.context.allocator.limit_rejection.{limit_scope=ALLOCATOR_LIMIT_SCOPE_PLAN_ITEMS, configured_limit, requested_value}`, - explicit reservation exceeds the admitted forward-reservation window -> `SF_TOO_MANY_OPERATIONS` with `ErrorStatus.context.allocator.limit_rejection.{limit_scope=ALLOCATOR_LIMIT_SCOPE_FORWARD_RESERVATION_WINDOW, configured_limit, requested_value}`, - remaining allocator identity space is insufficient -> `SF_TOO_MANY_OPERATIONS` with `ErrorStatus.context.allocator.limit_rejection.{limit_scope=ALLOCATOR_LIMIT_SCOPE_IDENTITY_SPACE_REMAINING, configured_limit, requested_value}`, - durable allocator backend unavailable -> `SF_BACKEND_UNAVAILABLE`. - transport split: - transport `OK` returns `AllocateIdsSuccessResponse` directly, - malformed/not-found/policy/authz/runtime failures use non-`OK` gRPC status plus canonical `ErrorStatus`. ### 6.1.2 DDL Change Synchronization TiDB identity-bootstrap ownership contract: - `COLLECTION_TRUTH_MODE_EXTERNAL` collection identity bootstrap is TiDB-owned: - TiDB DDL/bootstrap logic, not TiDB query execution or CDC ingestion, owns the first-time namespace/collection identity ensure flow. - TiDB query/statistics/CDC callers MUST treat resolved `namespace_id` / `collection_id` as prerequisites and MUST NOT issue `PutNamespaceMetadata` or `PutCollectionMetadata` on the hot path. - aux-index namespace bootstrap MUST use `PutNamespaceMetadata` because bootstrap needs create-or-ensure semantics plus explicit namespace-encryption ownership: - `GetNamespaceMetadata.lookup.namespace_name` is available for read/verification only and MUST NOT be treated as an allocating namespace-ensure surface, - if name-based readback reports `SF_RESOURCE_NOT_FOUND`, TiDB bootstrap MUST still use `PutNamespaceMetadata` to allocate or ensure the missing namespace identity rather than inferring or synthesizing one locally, - caller identity MUST be `meta.client_id=tidb-ddl-adapter` and `meta.client_kind=CLIENT_KIND_TIDB_DDL_ADAPTER`, - `meta.idempotency_key` MUST be a non-empty replay-stable namespace-bootstrap transport identity derived from the owning TiDB DDL/bootstrap operation identity plus the target `namespace_name`; query/request-correlation IDs from read RPCs MUST NOT be reused for this mutation, - `definition.encryption` MUST come from the TiDB deployment's configured Search Fabric namespace-encryption policy for the owning TiDB cluster/tenant; CCS MUST NOT infer or default it on the caller's behalf. - aux-index collection bootstrap MUST use `PutCollectionMetadata`: - caller identity MUST be `meta.client_id=tidb-ddl-adapter` and `meta.client_kind=CLIENT_KIND_TIDB_DDL_ADAPTER`, - `meta.idempotency_key` MUST be a distinct non-empty replay-stable collection-bootstrap transport identity derived from the same owning TiDB DDL/bootstrap operation identity plus the target `CollectionNameBinding{namespace_id, collection_name}`, - `definition.encryption` MUST come from the effective TiDB-owned collection/index encryption policy for that table/index path; when TiDB has no collection-specific override, it MUST copy the namespace's effective encryption descriptor explicitly into the collection request rather than relying on server-side inheritance/defaulting. - if aux-index namespace bootstrap resolves an existing namespace lineage but the target collection binding does not yet exist, TiDB MUST continue with `PutCollectionMetadata` to allocate the missing collection; `namespace exists, collection missing` is an ordinary first-create branch, not an ownership error. - `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` / federated-table identity bootstrap is control-plane-provisioned: - namespace and collection identity MUST be created ahead of TiDB traffic by a direct control-plane provisioning workflow using `CLIENT_KIND_CONTROL_ADMIN`, and that provisioning workflow also owns the encryption policy for those identities. - the resulting canonical `namespace_id` and collection binding inventory MUST be distributed to TiDB deployment/bootstrap configuration before TiDB DDL/query traffic starts. - TiDB DDL/query/write callers MUST NOT create federated-table / Search-Fabric-truth namespace or collection identity through `PutNamespaceMetadata` / `PutCollectionMetadata`. - TiDB may resolve/verify those identities through `GetNamespaceMetadata.lookup.{namespace_id|namespace_name}` and `GetCollectionMetadata.lookup.{collection|name_binding}` using the pre-provisioned canonical identifiers/bindings. - if the required namespace or collection identity is absent on that read path, TiDB MUST fail closed with the canonical not-found branch (`SF_RESOURCE_NOT_FOUND`) and MUST NOT allocate a replacement identity. When TiDB executes search-related DDL: - It MUST call `PutCollectionSchema` with: - `scope.collection` - `scope.ddl_commit_ts` - `schema_version` - `fence.expected_prev_schema_version` - `fence.schema_generation_token` - `fence.enforce_monotonic_apply=true` - TiDB DDL/schema-sync caller MUST also capture replay-critical schema-admission context at DDL admission time and persist it into `CollectionSchemaDefinition.replay_context` for that schema apply. - in the current revision, `CollectionSchemaDefinition.replay_context` carries: - `sql_mode` - `time_zone` - parser/tokenizer semantics are not replay-context bag entries; they MUST already be canonicalized into typed schema fields (`FieldSchema.config.field_definition.text.analysis` / `fulltext`). - source-round-trip charset/collation and other non-authoritative ecosystem metadata MAY remain in `FieldSchema.options` / `CollectionSettings.options`, but they MUST NOT be treated as replay-critical execution state. `PutCollectionSchema.scope.ddl_commit_ts` caller contract: - TiDB DDL synchronization callers MUST populate the authoritative DDL completion TS. - Gateway-managed Search-Fabric-truth schema callers (for example Elasticsearch/Turbopuffer schema routes and inline schema ensure/apply) MUST set `scope.ddl_commit_ts=0`. - `scope.ddl_commit_ts=0` is a reserved non-DDL sentinel and MUST NOT be used by TiDB-authored DDL sync. When TiDB executes destructive search-related DDL that removes the Search Fabric collection lineage: - it MUST call `DropCollection` for the affected `CollectionRef`, - it MUST preserve the canonical drop replay/fencing semantics defined in Section `6.4`, - and downstream Flow-A workers for that lineage MUST not treat this branch as a schema-refresh-and-resume event. `SchemaFencePrecondition.schema_generation_token` caller contract: - admitted request-side values MUST be non-empty and at most 64 Unicode scalar values. - overlength fence tokens are malformed and MUST fail closed as `SF_BAD_REQUEST`; services MUST NOT silently truncate or hash them before fence comparison, replay-key derivation, or persistence. Fail policy: - Search-related DDL: fail closed on mismatch/reject/timeout. - Non-search DDL: fail open with async compensation allowed. `GetSchemaSyncStatus` success-state contract: - transport `OK` returns `SchemaSyncStatusSuccessResponse` directly. - `SchemaSyncStatusSuccessResponse` always carries `applied_schema_version`, `applied_commit_ts`, `applied_schema_generation_token`, and `active_schema_update_policy`. - `SchemaSyncStatusSuccessResponse.effective_max_result_window` MUST be populated with the effective query window bound after control-plane default resolution (`CollectionSettings.max_result_window` explicit value or resolved static default). - state is encoded by `oneof state` and MUST set exactly one branch: - `ready`: schema is fully converged for serving. - `applying`: schema transition is currently in-flight. - `drifted.reason`: drift detected; reason MUST be non-empty. - `error.reason`: terminal sync/apply failure; reason MUST be non-empty. - `GetCollectionSchema` and ordinary ownership/write admission resolve against the collection's authoritative active schema pointer. - `GetSchemaSyncStatus` reports the same active generation together with CCS convergence state for that generation. - query execution uses the latest `READY` serving generation: - if the active generation is `READY`, query-serving-admissible generation equals the active generation. - if the active generation is `APPLYING`, `DRIFTED`, or `ERROR`, query serving MUST continue to use the most recent `READY` generation if one exists. - if an active schema lineage exists but no `READY` generation exists yet, query execution MUST fail closed as `SF_CATALOG_UNAVAILABLE` with `catalog_unavailable_reason=schema_convergence` and `schema_convergence_state=no_ready_serving_generation`. - once a collection is tombstoned, this RPC MUST fail closed as `SF_RESOURCE_NOT_FOUND` rather than exposing stale pre-drop status. - requests targeting collections that never existed or whose prior tombstone reservation has already been fully GC-released MUST also fail closed as `SF_RESOURCE_NOT_FOUND`. - active collections that have catalog identity but no admitted schema lineage yet MUST fail closed as `SF_CATALOG_UNAVAILABLE` rather than as `SF_RESOURCE_NOT_FOUND`. - when that schema-bootstrap branch is selected, `ErrorStatus.diagnostics` MUST include `catalog_unavailable_reason=schema_bootstrap` and `schema_bootstrap_state=no_active_schema_generation`. Gateway-owned schema boundary: - Elasticsearch/Turbopuffer gateway-managed schemas are allowed only for `COLLECTION_TRUTH_MODE_SEARCH_FABRIC`. - Gateways MUST NOT issue schema changes for `COLLECTION_TRUTH_MODE_EXTERNAL` collections (TiDB-owned schema); such requests MUST be rejected with `SF_OWNERSHIP_CONFLICT`. - Ownership conflicts MUST be detected before any schema mutation is applied. Initial schema-creation zero-state contract: - `PutCollectionMetadata` may allocate/resolve a `CollectionRef` before any schema generation exists; this does not itself create schema lineage. - The first admitted `PutCollectionSchema` for such a collection MUST use the deterministic zero-state tuple: - `schema_version=1` - `fence.expected_prev_schema_version=0` - `fence.schema_generation_token="__initial__"` - `update_policy=SCHEMA_UPDATE_POLICY_INITIAL_CREATION` - `fence.enforce_monotonic_apply=true` - `__initial__` is a reserved request-side sentinel meaning "no previous generation exists". It participates in retry/idempotency matching for the initial apply, but MUST NOT become the persisted active `applied_schema_generation_token`; on acceptance, control plane assigns the first real generation token. ### 6.1.3 Strict Undeclared-Field Contract The canonical ingest contract does not support implicit schema creation from write payloads in this revision: - undeclared-field handling is limited to `REJECT` or `IGNORE`; - gateways MUST NOT infer, auto-add, quarantine, or replay undeclared-field rows as a hidden control-plane side effect; - callers that need new fields MUST issue explicit `PutCollectionSchema` first and retry the write after schema admission succeeds. Operational consequence: - this removes asynchronous schema-evolution state from the data-plane contract and keeps write success/failure fully determined by the currently admitted schema snapshot. ### 6.2 Structured Mapping Payload (No Opaque Schema Blob) Canonical schema body is modeled by shared message `CollectionSchemaDefinition`, reused by: - `PutCollectionSchemaRequest.definition` for desired-state command input - `CollectionSchemaDescriptor.definition` for applied-state catalog readback `CollectionSchemaDefinition` carries: - `CollectionTruthMode` - `DocIDMapping` - `FieldSchema[]` - optional `CollectionSettings` (collection-level physical settings; includes `IndexSortSpec` for sorted-index layout controls) This enables deterministic compatibility checking and avoids binary TiDB metadata leakage. Canonical schema payload requirements: - canonical model MUST be ecosystem-neutral and represent complete schema semantics needed by active MySQL/Elasticsearch/Turbopuffer compatibility contracts. - canonical schema governance is three-axis and orthogonal in this revision: - temporal axis: collection-level `CollectionSettings.lifecycle_policy` and manifest-published `DataTier` decide row/window placement and retention, - structural axis: field-level `FieldConfig.access_intent` expresses cache/latency expectation for that field family, - quality axis: vector-level `FieldSchema.config.field_definition.vector.retrieval_intent` expresses fidelity/compression/residual-preservation expectation for vector retrieval. - `DocIDMapping` is authoritative for document identity source and immutability/uniqueness constraints; active-profile gateway ingress MUST emit canonical IDs as `DocumentID.id_value.u64` after source normalization. - active-profile `DocIDMapping.source_field_type` contract: - `SCALAR_FIELD_TYPE_UINT64` is the only admitted source type. - `SCALAR_FIELD_TYPE_STRING` and `SCALAR_FIELD_TYPE_BYTES` violate the active canonical doc-id contract in this profile and MUST fail closed as `400 SF_DOC_ID_CONTRACT_VIOLATION`. - any other enum value is malformed schema declaration and MUST fail closed as `400 SF_INVALID_SCHEMA`. - active-profile `DocIDMapping.generation_policy` contract: - the active design admits exactly two generation strategies: - `DOC_ID_GENERATION_POLICY_CALLER_REQUIRED` - `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL` - `DOC_ID_GENERATION_POLICY_CALLER_REQUIRED` means every admitted write path MUST provide source-side identity before canonical mutation construction. - `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL` is legal only on `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` collections and means only create/insert-style routes admitted by `CollectionSettings.allocator_admission_policy.caller_profile` may use allocator authority; those routes MUST resolve collection scope first, then resolve create/insert identity through `IdAllocatorService.AllocateIds`, and then carry the returned `allocator_receipt` unchanged into the follow-on canonical `MutationBatch`. - `DOC_ID_GENERATION_POLICY_UNSPECIFIED` is malformed and MUST fail closed as `400 SF_INVALID_SCHEMA`. - neither strategy relaxes the canonical ingest rule that admitted `RowMutation.doc_id` is always explicit. - `patch`, `delete`, and update-style writes that target existing document identity MUST still carry explicit identity and MUST NOT call allocator even when the collection schema uses `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`. - `CollectionSettings.allocator_admission_policy` MUST be absent for `DOC_ID_GENERATION_POLICY_CALLER_REQUIRED` and present with an explicit `caller_profile` plus positive `max_plan_items_per_request` / positive `max_forward_reservation_jump` for `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`. - the current revision admits only `ALLOCATOR_CALLER_PROFILE_ES_WRITE_GATEWAY_ONLY`; schema admission MUST reject any other caller profile until a later canonical expansion explicitly enables it. - `FieldSchema` carries field identity/topology (`name`, `nullable`, `subfields`) and delegates physical definition to `FieldSchema.config`. - `FieldSchema.options` is a supplemental metadata map (`map`) for round-trip compatibility hints; values are restricted to atomic/list config payloads (no vectors). - `FieldSchema.config` is authoritative for: - capability flags (`config.capabilities`), - `config.capabilities.multi_valued=true` declares scalar-array semantics for this field (single atomic type, repeated values) and is required for compatibility round-trip branches that distinguish `type` vs `[]type` (for example Turbopuffer schema families). - `config.capabilities.multi_valued=false` (default) declares single-valued semantics; list payload admission for that field MUST fail closed. - typed field definition through `oneof field_definition`: - `config.field_definition.scalar_type` for simple scalar families (`BOOL`/`INT64`/`UINT64`/`DOUBLE`/`FLOAT`/`TIMESTAMP`/`JSON`/`BYTES`/`STRING`), - `config.field_definition.text` for text/full-text semantics, - `config.field_definition.vector` for vector semantics, - `config.field_definition.keyword` for keyword-only semantics (including overflow policy), - `config.field_definition.object` for non-nested structured object containers (`fields` + optional `dynamic_policy`), - `config.field_definition.nested` for nested-document field semantics (`fields` + optional `max_docs_per_parent` defensive threshold). - field-level structural intent through `config.access_intent`: - `ARCHIVAL` means the field is latency-insensitive and may remain S3-first / cache-cold under service policy, - `FREQUENT` means the field is expected to participate in normal steady-state cache heuristics, - `CRITICAL` means the field is a high-priority cache-residency / low-latency candidate under service policy, - `UNSPECIFIED` means caller did not request an explicit structural override; the effective field-access intent MUST still resolve deterministically for a fixed schema generation, - deterministic default resolution in this revision is `FREQUENT` whenever the message is absent or `intensity=FIELD_ACCESS_INTENSITY_UNSPECIFIED`, - `CRITICAL` is an admission-time intent, not a hard residency guarantee; service policy MAY still spill or deprioritize the field under failure or extreme pressure as long as logical query semantics remain unchanged, - explicit non-`UNSPECIFIED` structural intent is admitted only on materializable leaf fields and subfields; non-leaf `object` / `nested` container declarations MUST leave `access_intent` absent or `UNSPECIFIED`, - this intent is column/field-scoped and does not override the temporal placement decided by lifecycle `DataTier`. - exactly one `field_definition` branch MUST be set. - cross-validation is mandatory: selected `field_definition` branch and `config.capabilities` MUST be semantically coherent. - `TextAnalysisSpec` under `FieldSchema.config.field_definition.text.analysis` is a single analyzer identifier: - `analyzer` is a canonical string ID representing the full lexical pipeline (pre-filtering, tokenization, post-filtering). - `analyzer` MUST be selected from service-supported built-in analyzer IDs, unless a future extension explicitly enables custom registry sources. - unsupported analyzer IDs MUST fail closed (`SF_UNSUPPORTED_OPTION`). - analyzer-level overrides are optional typed metadata entries (`options: map`) and apply after analyzer baseline resolution. - active revision admits only `MetadataValue.atomic_value.string_value` in this map; `list_value` and non-string atomic branches are invalid for analyzer overrides and MUST fail closed (`SF_SCHEMA_VIOLATION`). - key registry in this revision: - `stopwords` - `stemming` - `tokenizer.ngram_token_size` - value grammar in this revision: - `stopwords` / `stemming`: `inherit|enabled|disabled` - `tokenizer.ngram_token_size`: decimal integer in `[1,10]` - override precedence is deterministic: `analyzer` baseline first, then override patch. - unsupported override keys or analyzer-incompatible keys MUST fail closed with `SF_UNSUPPORTED_OPTION`. - override value grammar violations MUST fail closed with `SF_SCHEMA_VIOLATION`. - built-in analyzer IDs in this revision: - `builtin_keyword_v1` - `builtin_standard_v1` - `builtin_standard_folded_v1` - `builtin_ngram_v1` - `builtin_mecab_ja_v1` - baseline defaults and fixed semantics in this revision (applies when an option key is absent, or when its value is `inherit`): - stopwords baseline: - enabled for `builtin_standard_v1`, `builtin_standard_folded_v1`, and `builtin_ngram_v1` - disabled for `builtin_keyword_v1` - stemming baseline: - disabled for all analyzers - `builtin_standard_v1` / `builtin_standard_folded_v1` token-length filter: - minimum token size: `3` - maximum token size: `84` - these bounds are fixed in this revision and are not configurable through `TextAnalysisSpec.options` - `builtin_ngram_v1` baseline `tokenizer.ngram_token_size`: - `2` (bigram) - default stopword list in this revision (lowercase ASCII set; used when stopword filtering is enabled): - `a`, `about`, `an`, `and`, `are`, `as`, `at`, `be`, `by`, `com`, `de`, `en`, `for`, `from`, `how`, `i`, `in`, `is`, `it`, `la`, `of`, `on`, `or`, `that`, `the`, `this`, `to`, `was`, `what`, `when`, `where`, `who`, `will`, `with`, `und`, `www` - custom stopword lists are not representable in canonical schema in this revision; callers may only enable/disable stopword filtering. - analyzer/option compatibility constraints in this revision: - `tokenizer.ngram_token_size` is compatible only with `builtin_ngram_v1`. - `stemming` is not compatible with `builtin_ngram_v1`. - Myanmar-specific analyzer/tokenizer IDs are not part of this revision; requests that require a distinct Myanmar parser surface are outside current canonical scope until a future analyzer-registry extension is introduced. - `FieldSchema.config.field_definition.text.fulltext` is the canonical field-level full-text contract with layered responsibilities: - `storage_tuning` is the physical storage/index-build contract: - `postings_options` centralizes Elasticsearch `index_options` semantic subset (`docs` / `freqs` / `positions` / `offsets`), - `postings_options=TEXT_POSTINGS_OPTIONS_UNSPECIFIED` means inherit service profile default postings behavior, - `precompute_phrases`, `precompute_prefixes`, and `precompute_document_terms` trade index/storage cost for faster phrase/prefix/term-aware retrieval, - `length_normalization` controls whether additional document-length payload is stored for score normalization. - `default_similarity` is the logical scoring baseline (currently `bm25` with `k1`/`b`; extensible via oneof). When `default_similarity` message is present, `SimilarityModel.model` oneof MUST set exactly one branch; empty branch fails closed (`SF_SCHEMA_VIOLATION`). - `default_search_behaviors` is the logical query-behavior baseline: - `default_mode` selects the default text match-mode branch when callers do not explicitly choose one. - `default_mode=TEXT_MODE_UNSPECIFIED` means inherit service-profile default mode; inherited mode resolution MUST be deterministic for a fixed profile + schema generation. - `natural_language` carries defaults scoped to natural-language mode (for example query expansion and spell-check toggles). - `boolean_match` carries defaults scoped to boolean mode. - `plain_match` carries defaults scoped to plain-match mode. - changing `storage_tuning` is a physical rebuild-class change; changing `default_similarity` or `default_search_behaviors` is a logical-baseline update and MAY take effect without full reindex under deterministic policy. - `FieldSchema.config.field_definition.vector.search_tuning` is the canonical vector retrieval/index acceleration tuning carrier: - `optimization_goal` expresses engine-agnostic intent (`UNSPECIFIED`, `BALANCED`, `EXACT_MATCH`, `READ_LATENCY`, `WRITE_FRESHNESS`, `MEMORY_COST`) and MUST be deterministically mapped by gateway/control-plane policy. - `quantization` expresses engine-agnostic compression strategy (`NONE`, `INT8`, `BINARY`) for memory/latency/recall trade-offs. - `optimization_goal=VECTOR_OPTIMIZATION_GOAL_UNSPECIFIED` means caller did not express an explicit optimization preference; control plane/runtime MUST deterministically fallback to `VECTOR_OPTIMIZATION_GOAL_BALANCED`. - `optimization_goal=VECTOR_OPTIMIZATION_GOAL_BALANCED` means caller explicitly requests a balanced overall experience; mapping MUST resolve deterministically for a given profile+schema generation. - typical balanced mapping: allocate SPfresh with sufficient memory cache, or allocate lightweight HNSW under policy control. - `quantization=VECTOR_QUANTIZATION_UNSPECIFIED` means no explicit quantization override; effective quantization policy MUST still be deterministic. - `FieldSchema.config.field_definition.vector.retrieval_intent` is the canonical quality-axis contract for vectors: - `ECONOMY` means prioritize compression density and tolerate larger accuracy loss when policy chooses the physical representation, - `STANDARD` means balanced production trade-off between memory, latency, and recall, - `HIGH_FIDELITY` means preserve the payload needed for high-accuracy rerank/audit-grade retrieval under service policy, including residual-preservation eligibility, - `UNSPECIFIED` means caller did not request an explicit quality override; the effective vector-retrieval intent MUST still resolve deterministically for a fixed schema generation, - deterministic default resolution in this revision is `STANDARD` whenever the message is absent or `accuracy_mode=VECTOR_ACCURACY_MODE_UNSPECIFIED`, - `HIGH_FIDELITY` constrains retained rerank/audit payload eligibility, not the exact first-stage ANN/index encoding chosen by the engine, - this intent governs fidelity class and residual-preservation eligibility; it does not by itself choose the row/window `DataTier`. - `FieldSchema.config.field_definition.keyword.overflow_protection` controls keyword overlong value handling on indexing hot path: - `OVERFLOW_BEHAVIOR_REJECT`: reject write admission with schema-validation precedence. - `OVERFLOW_BEHAVIOR_SKIP_INDEXING`: accept row but skip index entry generation for the overflowing field. - `OVERFLOW_BEHAVIOR_TRUNCATE_AND_INDEX`: truncate to `max_chars` and continue indexing deterministically. - validation constraints: `max_chars` MUST be `> 0`; `behavior` MUST NOT be `OVERFLOW_BEHAVIOR_UNSPECIFIED`. - for fields with `FieldSchema.config.capabilities.multi_valued=true`, `max_chars` applies per string element (per atomic value), not to aggregate array character length. - `FieldSchema.config.field_definition.object` is the canonical non-nested structured container contract: - `fields[]` declares child schema using the same `FieldSchema` carrier used at top level. - `dynamic_policy` controls undeclared child-field behavior for that object scope: - `DYNAMIC_MAPPING_POLICY_IGNORE`: ignore undeclared child fields for indexing (Elasticsearch compatibility maps this to `dynamic: false`), - `DYNAMIC_MAPPING_POLICY_STRICT`: fail closed on undeclared child fields, - `DYNAMIC_MAPPING_POLICY_UNSPECIFIED`: inherit collection/service default policy. - infer/add child-field creation is not admitted in this revision; object-container dynamic policy remains subordinate to the global strict undeclared-field contract. - object container branch preserves parent-child document scope (no nested query-scope split). - `FieldSchema.config.field_definition.nested` is the canonical nested structured container contract: - `fields[]` declares nested child schema using the same `FieldSchema` carrier. - `max_docs_per_parent` is an optional defensive threshold for nested fanout: - unset means service/profile default is used. - when present, value MUST be `> 0`. - write admission for one parent document exceeding this threshold MUST fail closed by schema/limit policy. - `FieldSchema.subfields[]` is the canonical field-level multi-field carrier (Elasticsearch `fields` equivalent): - this carrier is ONLY for same-value multi-index representations (for example one logical field indexed by multiple analyzers/strategies), - each subfield has its own `FieldConfig config`, - subfield config MUST NOT use container branches (`object`/`nested`) in this revision, - subfield `name` MUST be unique within parent field and MUST NOT recursively contain nested subfields in this revision, - `FieldSchema.name` and `FieldSubfieldSpec.name` MUST NOT contain `.` (dot is reserved for field-path traversal and optional final subfield addressing), - query-time subfield references MUST use `.` (for example `title.ngram` or `user.name.ngram`). - `CollectionSettings` is authoritative for: - optional collection-level physical index-sort layout declaration (`index_sort`), - optional collection-level query deep-pagination guardrail (`max_result_window`, Elasticsearch `index.max_result_window` equivalent for `from + size`), - when `max_result_window` is absent in create/update payloads, control plane MUST resolve a deterministic service-static default (for example `10000`) and treat it as the effective persisted value. - control-plane schema snapshot responses (especially `GetCollectionSchema.success.schema` and any descriptor exports consumed by stateless gateways) MUST return that effective value explicitly; `GetSchemaSyncStatus` MUST also echo the same resolved value for convergence/status polling. Gateways and execution nodes MUST NOT hardcode local fallback guesses. - omission of `effective_max_result_window` from an authoritative control-plane snapshot is contract drift and MUST surface validation/telemetry; runtime local defaulting is not part of the production contract. - CCS private admission-state snapshots keep the field machine-required as well: when `CollectionServingReadiness.state=READY`, the private `effective_max_result_window` fact MUST be `> 0` and equal the resolved serving-generation bound; when readiness is `SCHEMA_BOOTSTRAP` or `SCHEMA_CONVERGENCE`, that private field MUST be the explicit `0` sentinel because no READY serving generation is query-admissible yet. - optional collection-level lifecycle policy (`lifecycle_policy`) for partition-window sizing, tier transitions, and retention reclamation, - optional collection-level supplemental source metadata for round-trip needs (`options: map` with atomic/list values only), - for nested ecosystem settings payloads (for example Elasticsearch index settings JSON), gateway MUST flatten nested objects into dot-notation keys (for example `index.routing.allocation.require.size=large`) before assigning entries into flat `CollectionSettings.options`. - nested-object settings that are not flattened before `CollectionSettings.options` emission are malformed and MUST fail closed with `SF_SCHEMA_VIOLATION`. - `CollectionSettings.lifecycle_policy` remains the canonical temporal axis: - it governs row/window placement over time through manifest-published `DataTier`, - it is intentionally orthogonal to field-level `access_intent` and vector-level `retrieval_intent`, - engines MAY combine all three axes into physical execution states, but callers must not encode structural/quality semantics by overloading lifecycle transitions alone. ### 6.2.1 Canonical Schema Coverage Mapping (ES/Turbopuffer/MySQL) Control plane MUST support deterministic schema canonicalization across three ecosystems: | Ecosystem Semantic Source | Canonical Control-Plane Carrier | |---|---| | Elasticsearch `/{index}/_mapping` field types/index flags/analyzers/normalizers/object/nested config | `FieldSchema.config.field_definition.scalar_type`, `FieldSchema.config.capabilities`, `FieldSchema.config.field_definition.text.analysis`, `FieldSchema.config.field_definition.vector`, `FieldSchema.config.field_definition.object`, `FieldSchema.config.field_definition.nested` | | Elasticsearch `type: object` container + `dynamic` policy | `FieldSchema.config.field_definition.object.fields` + `FieldSchema.config.field_definition.object.dynamic_policy` | | Elasticsearch mapping multi-fields (`fields`) | `FieldSchema.subfields[]` | | Elasticsearch `index.sort.*` | `CollectionSettings.index_sort` | | Elasticsearch `index.max_result_window` | `CollectionSettings.max_result_window` | | Lifecycle tiering/retention controls (physical partition window + HOT/WARM/COLD transitions + retention horizon) | `CollectionSettings.lifecycle_policy` | | Field-level latency/cache intent | `FieldConfig.access_intent` | | Vector fidelity/compression/residual intent | `FieldSchema.config.field_definition.vector.retrieval_intent` | | Turbopuffer `AttributeSchemaConfig` (`type`, `filterable`, `regex`, `full_text_search`, `ann`) | `FieldSchema.config.field_definition.scalar_type`, `FieldSchema.config.capabilities` (`filterable`, `pattern_matchable`, `searchable`, `multi_valued` from scalar/array type family), `FieldSchema.config.field_definition.text.analysis`, `FieldSchema.config.field_definition.text.fulltext`, `FieldSchema.config.field_definition.vector`, `FieldSchema.options` | | MySQL FULLTEXT parser/token/tuning constraints (charset/collation out of canonical search semantics) | `FieldSchema.config.field_definition.text.analysis.analyzer`, `FieldSchema.config.field_definition.text.analysis.options`, `FieldSchema.config.field_definition.text.fulltext`, `FieldSchema.options`, `CollectionSettings.options` (supplemental source metadata / operational keys) | | Cross-ecosystem external ID contracts | `DocIDMapping` (`source_field`, `source_field_type=SCALAR_FIELD_TYPE_UINT64`, `immutable`, `unique`, `generation_policy`) + active-profile canonical output contract (`DocumentID.id_value.u64`) | Schema round-trip requirements: - gateway normalization (`source -> canonical`) and projection (`canonical -> source`) MUST be deterministic and versioned. - canonical projection to source wire shapes MUST preserve semantic equivalence for admitted features. - if a source asks for unsupported schema feature branch, rejection MUST happen before `PutCollectionSchema`. ### 6.3 Ownership and Contract Validation in Control Plane Control plane validates: - allowed `collection_truth_mode` values and route/caller policies derived from them - canonical `doc_id` contract - gateway ID parse contract is validated per compatibility profile: - Elasticsearch/MySQL/Turbopuffer compatibility paths MUST emit canonical `DocumentID.id_value.u64` on admitted requests. - `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_STRING` or `SCALAR_FIELD_TYPE_BYTES` MUST be rejected at schema admission as `400 SF_DOC_ID_CONTRACT_VIOLATION`. - source-field enum values outside `{SCALAR_FIELD_TYPE_UINT64, SCALAR_FIELD_TYPE_STRING, SCALAR_FIELD_TYPE_BYTES}` are malformed and MUST be rejected at schema admission (`400 SF_INVALID_SCHEMA`). - source payload values that cannot be normalized into canonical `DocumentID.id_value.u64` under `SCALAR_FIELD_TYPE_UINT64` are rejected at ingest (`422 SF_SCHEMA_VIOLATION`). - generation strategy contract is validated at schema and route admission: - only `DOC_ID_GENERATION_POLICY_CALLER_REQUIRED` and `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL` are admitted, - `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL` MUST force `DOC_ID_GENERATION_POLICY_CALLER_REQUIRED` and MUST keep `CollectionSettings.allocator_admission_policy` absent, - `CollectionSettings.allocator_admission_policy` MUST be absent for caller-required collections and present with an explicit `caller_profile` plus positive `max_plan_items_per_request` / positive `max_forward_reservation_jump` for allocator-optional collections, - the current revision admits only `ALLOCATOR_CALLER_PROFILE_ES_WRITE_GATEWAY_ONLY` in `CollectionSettings.allocator_admission_policy.caller_profile`, - allocator-backed create/insert identity is legal only on collection/caller combinations whose policy explicitly enables it, - explicit create/insert identity on allocator-optional collections MUST also resolve through allocator reservation/validation, - every allocator-backed follow-on `MutationBatch` MUST carry the matching `allocator_receipt` returned by `AllocateIds`, - patch/delete/update-style routes MUST still require explicit identity and MUST NOT call allocator even when the collection uses `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`. - fulltext parser/tokenizer compatibility contract encoded in typed schema payload (`FieldSchema.config.field_definition.text.analysis.analyzer` + `FieldSchema.config.field_definition.text.analysis.options` + `FieldSchema.config.field_definition.text.fulltext`) and supplemental options (including runtime prerequisite readiness for parser modes) - immutable doc-id-source policy for external-truth TiCDC projection modes - collection-settings contract validity (`CollectionSettings.index_sort` and settings-option constraints) - lifecycle-policy contract validity (`CollectionSettings.lifecycle_policy` time-field typing, partition-window floor, tier-transition monotonicity, and retention ownership guardrails) - typed field validity (`config.field_definition.scalar_type`, `config.capabilities`, `config.field_definition`, `config.field_definition.object.{fields,dynamic_policy}`, `config.field_definition.nested.{fields,max_docs_per_parent}`, `subfields[].config` constraints) - structural-intent validity (`config.access_intent` is admitted only on materializable leaf fields and subfields; absent or `UNSPECIFIED` resolves to deterministic effective `FREQUENT`) - cardinality validity (`config.capabilities.multi_valued` vs field payload shape): list payloads on `multi_valued=false` fields MUST be rejected, and `multi_valued=true` fields MUST preserve homogeneous atomic element typing in `ListValue`. - schema/action enum fail-closed rules: - `PutCollectionSchema.update_policy` MUST NOT be `SCHEMA_UPDATE_POLICY_UNSPECIFIED`. - first schema apply for a brand-new collection MUST use `SCHEMA_UPDATE_POLICY_INITIAL_CREATION`. - when catalog identity exists but no prior schema lineage exists, the first apply MUST use the full zero-state tuple (`schema_version=1`, `fence.expected_prev_schema_version=0`, `fence.schema_generation_token="__initial__"`, `update_policy=SCHEMA_UPDATE_POLICY_INITIAL_CREATION`, `fence.enforce_monotonic_apply=true`). - `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_UINT64` is admitted; `SCALAR_FIELD_TYPE_STRING`/`SCALAR_FIELD_TYPE_BYTES` MUST fail closed as `SF_DOC_ID_CONTRACT_VIOLATION`; any other enum value MUST fail closed as `SF_INVALID_SCHEMA`. - when `FieldConfig.field_definition.scalar_type` branch is selected, value MUST NOT be `SCALAR_FIELD_TYPE_UNSPECIFIED`. - vector tuning validity (`config.field_definition.vector.search_tuning.optimization_goal` / `config.field_definition.vector.search_tuning.quantization` deterministic admission and compatibility checks) - vector base-type validity (`config.field_definition.vector.dimensions > 0`, `config.field_definition.vector.element_type != VECTOR_ELEMENT_TYPE_UNSPECIFIED`, `config.field_definition.vector.distance_metric != DISTANCE_METRIC_UNSPECIFIED`) - vector quality-intent validity (`config.field_definition.vector.retrieval_intent` absent or `UNSPECIFIED` resolves to deterministic effective `STANDARD`; explicit `HIGH_FIDELITY` MUST be rejected if the active service/profile policy cannot preserve rerank/audit payload eligibility for the admitted lifecycle envelope) - deterministic index-overflow protection contract (`config.field_definition.keyword.overflow_protection.max_chars` + `behavior` action on overflow) - overflow-protection validity (`max_chars > 0` and non-`UNSPECIFIED` overflow behavior when overflow protection is configured) - three-axis cross-validation: - control plane MUST resolve and persist an effective structural intent for every materializable field and an effective quality intent for every vector field as part of the admitted schema generation, even when callers omit those overrides, - lifecycle policy alone MUST NOT be used to silently weaken a vector field from effective `HIGH_FIDELITY` to a residual-dropping storage class; any schema change that would make the previously admitted effective quality intent unrealizable MUST be rejected or classified as rebuild-class schema evolution, - `FieldAccessIntent` influences cache/cost policy only and MUST NOT be interpreted as direct row-placement authority; `CRITICAL` does not override lifecycle tier ownership. - cross-ecosystem schema equivalence constraints (adapter-provided schema fingerprint/coherence checks) Invalid tuple or contract mismatch is rejected before write/query admission. ### 6.4 Drop Fencing and ABA Protection `DropCollection` uses two orthogonal guards: - network-level retry deduplication via `RequestMeta.idempotency_key`. - optional monotonic CAS fencing (`fence`) to avoid ABA races. When CAS-safe drop is required, request MUST include monotonic fencing semantics: - `fence.expected_prev_schema_version`. - `fence.schema_generation_token`. - `fence.enforce_monotonic_apply`. Drop acceptance rule: - with `fence` present: drop request is accepted only when fence preconditions match active catalog state; if collection has already moved to a newer generation, old drop request is rejected deterministically. - with `fence` absent: drop is unconditional at schema-state layer, but retry replay safety MUST still be guaranteed by `RequestMeta.idempotency_key`. ### 6.5 Schema Update Policies and Query Safety Control plane MUST classify each schema update: - `SCHEMA_UPDATE_POLICY_INITIAL_CREATION`: day-zero schema creation for a new collection before any prior schema generation exists. - `SCHEMA_UPDATE_POLICY_ADDITIVE`: nullable field add or non-reinterpreting capability/settings additions; no rebuild required. - `SCHEMA_UPDATE_POLICY_BACKWARD_COMPATIBLE_REBUILD`: rebuild with old query compatibility preserved during transition. - `SCHEMA_UPDATE_POLICY_BREAKING_REBUILD`: shape change requiring fenced cutover. Safety rules: - query plane uses `applied_schema_version` from the serving-admissible catalog snapshot (`latest READY` generation); never mixes index generations in one page token chain. - query plane MUST bind continuation tokens to `applied_schema_generation_token` to prevent mixed-generation pagination. - if a newer active generation exists but is not yet `READY`, first-page execution MUST continue to use the serving-admissible `READY` generation rather than silently switching to the newer active pointer. - continuation/search-context validation MUST use the token-bound serving lineage as long as that lineage remains retained and query-admissible; it MUST NOT fail solely because a newer `READY` generation was promoted after the token was minted. - nullable transition policy is monotonic in additive path: - `nullable: false -> true` is a constraint relaxation and MAY be classified as `SCHEMA_UPDATE_POLICY_ADDITIVE`. - `nullable: true -> false` is a constraint tightening and MUST be classified as `SCHEMA_UPDATE_POLICY_BREAKING_REBUILD` (historical-data validation/rewrite required). - scalar-cardinality (`FieldCapabilitySpec.multi_valued`) transition policy is rebuild-class in both directions: - `multi_valued: false -> true` MUST be classified as `SCHEMA_UPDATE_POLICY_BREAKING_REBUILD` (single-valued to array-valued reinterpretation). - `multi_valued: true -> false` MUST be classified as `SCHEMA_UPDATE_POLICY_BREAKING_REBUILD` (array-to-single contraction with historical-data rewrite risk). - if `FieldSchema.config.field_definition.scalar_type`, `FieldSchema.config.field_definition.text.analysis`, `FieldSchema.config.field_definition.text.fulltext.storage_tuning`, `FieldSchema.config.field_definition.vector`, `FieldSchema.config.field_definition.object`, `FieldSchema.config.field_definition.nested`, or `FieldSchema.subfields[].config` is changed in a way that reinterprets existing indexed values, update policy MUST be rebuild-class. - changes limited to `FieldSchema.config.field_definition.text.fulltext.default_similarity` and/or `FieldSchema.config.field_definition.text.fulltext.default_search_behaviors` MAY be non-rebuild updates when deterministic compatibility and snapshot-versioning rules are satisfied. - changes that only raise or lower `FieldConfig.access_intent` MAY use additive/non-rebuild policy because they alter cache-priority intent rather than persisted logical field shape. - changes that alter the effective vector quality contract from `HIGH_FIDELITY` to `{STANDARD, ECONOMY}` or otherwise authorize residual dropping for previously protected vectors MUST be rebuild-class and MUST NOT be silently piggybacked on lifecycle-policy edits alone. - changes from `{STANDARD, ECONOMY}` to `HIGH_FIDELITY` are also rebuild-class whenever the currently published objects do not already retain the rerank/audit payload required by that stronger contract. - breaking rebuilds require strict fencing and readiness green before control-plane marks state `ready`. - if query hits collection generation drift, fail closed with `SF_VERSION_CONFLICT`. ### 6.6 Control Plane Static Policy Contract Control plane constraints are service-static in this revision: - supported `SchemaUpdatePolicy` set, drop-fencing rules, and request-metadata idempotency-dedupe policy, - maximum concurrent schema operations per collection/namespace, - built-in analyzer registry and override key/value grammar, - lifecycle static floors/limits (minimum `partition_window_ms`, admitted `DataTier` transition graph). Admission rules: - TiDB MUST not submit schema update policies outside the admitted service policy set under strict mode. - control-plane overload MUST fail with retryable `SF_RATE_LIMITED` and include the canonical detail keys defined in `docs/specs/core/error-codes.md`. - if request carries `FieldSchema.config.field_definition.text.analysis.analyzer` outside service-supported built-in analyzers, fail closed with `SF_UNSUPPORTED_OPTION`. - if request carries analyzer override keys outside service-supported registry, fail closed with `SF_UNSUPPORTED_OPTION`. - if request carries override values violating service grammar, fail closed with `SF_SCHEMA_VIOLATION`. ### 6.7 Deterministic Control-Plane Admission Order Before applying DDL state transitions, Control Plane MUST evaluate checks in this order: 1. access scope and caller authority. 2. request-level network dedupe (`RequestMeta.idempotency_key`) before durable replay/conflict checks. 3. schema-update-policy and static-policy support. 4. fencing preconditions (`SchemaFencePrecondition`). 5. ownership/doc-id/typed-schema contract validation. 6. concurrency/admission limits. 7. apply state transition. If a request violates multiple checks, this order defines evaluation flow only. Final emitted code MUST be selected strictly by `docs/specs/core/error-codes.md` Section `2.1` (canonical precedence arbitration). ### 6.8 Index Sort Contract (Elasticsearch `index.sort.*` Equivalent) Control plane schema contract supports collection-level physical sort declaration through `CollectionSettings.index_sort`. Normalization and mapping rules: - Ingest Gateway MUST normalize Elasticsearch index settings to canonical fields: - `index.sort.field` -> `IndexSortSpec.fields[].field` - `index.sort.order` -> `IndexSortFieldSpec.order` - `index.sort.mode` -> `IndexSortFieldSpec.mode` - `index.sort.missing` -> `IndexSortFieldSpec.missing_value_policy` - `index.max_result_window` -> `CollectionSettings.max_result_window` - Ingest Gateway MUST expand implicit Elasticsearch defaults into explicit canonical values before calling this API. Validation and safety rules: - if `CollectionSettings.index_sort` is present, `CollectionSettings.index_sort.fields` MUST be non-empty. - `CollectionSettings.index_sort` is admitted only for metadata-bearing collection layouts (for example metadata/full-text/hybrid schemas with sortable scalar metadata fields); pure vector-only collections MUST reject it fail-closed (`SF_UNSUPPORTED_OPTION` or `SF_SCHEMA_VIOLATION` by precedence). - if `CollectionSettings.max_result_window` is present, value MUST be > 0. - field order in `CollectionSettings.index_sort.fields[]` defines lexicographic key precedence for physical layout. - each `IndexSortFieldSpec.order` MUST be explicit and MUST NOT be `SORT_ORDER_UNSPECIFIED`; otherwise reject (`SF_SCHEMA_VIOLATION`). - each `IndexSortFieldSpec.mode` MUST be explicit and MUST NOT be `SORT_MODE_UNSPECIFIED`; otherwise reject (`SF_SCHEMA_VIOLATION`). - each `IndexSortFieldSpec.missing_value_policy` MUST be explicit and MUST NOT be `MISSING_VALUE_POLICY_UNSPECIFIED`; otherwise reject (`SF_SCHEMA_VIOLATION`). - unsupported field/type/mode combinations MUST fail closed (`SF_UNSUPPORTED_OPTION` or `SF_SCHEMA_VIOLATION`). - changing index-sort declaration between schema generations requires rebuild-class schema update policy (`SCHEMA_UPDATE_POLICY_BACKWARD_COMPATIBLE_REBUILD` or `SCHEMA_UPDATE_POLICY_BREAKING_REBUILD`), not additive-in-place mutation. - query output ordering still follows `QueryRequest.fetch_spec.sort_keys`; index-sort layout is a physical-plan optimization and MUST NOT weaken logical ordering/filter contracts. - changing `CollectionSettings.max_result_window` is non-rebuild control-plane update, but MUST be reflected immediately in query admission checks and deterministic error-detail echo semantics. ### 6.9 Control-Plane Idempotent Retry Contract `PutCollectionMetadata` retry rules: - `RequestMeta.idempotency_key` MUST be non-empty for retry-safe mutation semantics; empty idempotency key is malformed (`SF_BAD_REQUEST`). - network-level dedupe key is `(server-authenticated caller identity, meta.client_id, rpc_method, name_binding.namespace_id, name_binding.collection_name, meta.idempotency_key)`. - same key + semantically identical metadata payload (`definition`) MUST return the original success payload verbatim, including the originally admitted `status` and authoritative descriptor snapshot. - same key + different canonical metadata payload MUST fail closed (`SF_IDEMPOTENCY_CONFLICT`). - replay normalization does not admit a whole-message-absent `definition` branch in this revision because admitted requests must carry `definition` explicitly. - metadata mutations do not define a separate business replay identity in this revision; a different `idempotency_key` is an ordinary new metadata command against current state, not business replay. - when such a new metadata command reaches the authoritative apply phase, success classification is state-based: - if the post-apply descriptor state already matches the requested metadata exactly, the RPC MUST return `already_applied`, - if the request changes authoritative descriptor state, the RPC MUST return `created` or `updated` according to the resulting branch, - implementations MUST NOT treat a fresh `idempotency_key` as sufficient reason to upgrade a no-op command from `already_applied` to `updated`. `PutNamespaceMetadata` retry rules: - `RequestMeta.idempotency_key` MUST be non-empty for retry-safe mutation semantics; empty idempotency key is malformed (`SF_BAD_REQUEST`). - network-level dedupe key is `(server-authenticated caller identity, meta.client_id, rpc_method, namespace_name, meta.idempotency_key)`. - same key + semantically identical metadata payload (`definition`) MUST return the original success payload verbatim, including the originally admitted `status` and authoritative descriptor snapshot. - same key + different canonical metadata payload MUST fail closed (`SF_IDEMPOTENCY_CONFLICT`). - replay normalization does not admit a whole-message-absent `definition` branch in this revision because admitted requests must carry `definition` explicitly. - metadata mutations do not define a separate business replay identity in this revision; a different `idempotency_key` is an ordinary new metadata command against current state, not business replay. - when such a new metadata command reaches the authoritative apply phase, success classification is state-based: - if the post-apply descriptor state already matches the requested metadata exactly, the RPC MUST return `already_applied`, - if the request changes authoritative descriptor state, the RPC MUST return `created` or `updated` according to the resulting branch, - implementations MUST NOT treat a fresh `idempotency_key` as sufficient reason to upgrade a no-op command from `already_applied` to `updated`. `PutCollectionSchema` retry rules: - `RequestMeta.idempotency_key` MUST be non-empty for retry-safe mutation semantics; empty idempotency key is malformed (`SF_BAD_REQUEST`). - two replay identities apply and MUST both be enforced: - transport replay key: `(server-authenticated caller identity, meta.client_id, rpc_method, scope.collection.namespace_id, scope.collection.collection_id, meta.idempotency_key)` - business/schema replay key: `(scope.collection.namespace_id, scope.collection.collection_id, schema_version, fence.schema_generation_token)` - same transport replay key + semantically identical schema payload (`definition`, `update_policy`, `schema_version`, `fence`) MUST return the original success payload verbatim. - same transport replay key + different canonical schema payload MUST fail closed (`SF_IDEMPOTENCY_CONFLICT`). - same business/schema replay key + semantically identical schema payload under a different `idempotency_key` MUST return replay-safe success (`accepted` or `already_applied` according to the originally admitted outcome). - same business/schema replay key + different payload under a different `idempotency_key` MUST be rejected deterministically (`SF_IDEMPOTENCY_CONFLICT` or `SF_VERSION_CONFLICT` by precedence). - initial schema creation uses the same retry contract with the zero-state fence tuple (`schema_version=1`, `fence.expected_prev_schema_version=0`, `fence.schema_generation_token="__initial__"`). `DropCollection` retry rules: - `RequestMeta.idempotency_key` MUST be non-empty for retry-safe mutation semantics; empty idempotency key is malformed (`SF_BAD_REQUEST`). - two replay identities apply and MUST both be enforced: - transport replay key: `(server-authenticated caller identity, meta.client_id, rpc_method, collection.namespace_id, collection.collection_id, meta.idempotency_key)` - business/drop replay key: - with `fence` present: `(collection.namespace_id, collection.collection_id, fence.expected_prev_schema_version, fence.schema_generation_token, fence.enforce_monotonic_apply)` - with `fence` absent: `(collection.namespace_id, collection.collection_id, "__unconditional_drop__")` - same transport replay key after successful drop MUST return the `already_dropped` success branch. - same transport replay key with conflicting `fence` payload MUST fail closed (`SF_IDEMPOTENCY_CONFLICT` or `SF_VERSION_CONFLICT` by precedence). - same business/drop replay key under a different `idempotency_key` after a successful drop MUST return the `already_dropped` success branch. - a fresh request against a collection whose tombstone reservation is still active MUST return the `already_dropped` success branch even when the request is not a transport replay of the original call. - a request against a collection that never existed or whose tombstone reservation has already been fully GC-released MUST fail closed as `SF_RESOURCE_NOT_FOUND`. - a request whose `fence` does not match the current active collection state MUST fail closed as `SF_VERSION_CONFLICT` by canonical precedence, even if the same collection is later dropped by another command. ### 6.10 Canonical Schema Evolution and Equivalence Rules Canonical schema evolution MUST preserve both intra-ecosystem and cross-ecosystem semantics: - additive-compatible changes MAY add fields/subfields/capabilities that do not reinterpret existing persisted values. - semantic reinterpretation changes (for example text parser family switch, vector distance change, or canonical `doc_id` source-field contract change) MUST be rebuild-class changes. - `options` keys that materially affect retrieval semantics MUST either: - be modeled in typed fields in this API revision, or - be explicitly registered as canonical extension keys with deterministic compatibility checks. - `MetadataValue` in schema options is config-only and MUST NOT carry vector payloads; vector semantics belong to typed vector schema/query fields. Cross-ecosystem equivalence invariants: - ES mapping projection and Turbopuffer schema projection from the same canonical snapshot MUST produce semantically equivalent field/settings behavior for shared capabilities. - MySQL FULLTEXT canonical projection MUST preserve parser/tokenization/tuning contracts defined by admitted schema generation. - schema projection MUST be idempotent: `source -> canonical -> source` round-trip cannot drift for admitted feature branches. - equivalence violations discovered during apply/projection MUST fail closed (`SF_SCHEMA_VIOLATION` or `SF_UNSUPPORTED_OPTION` by precedence). ### 6.11 Lifecycle Policy Contract (Fail-Closed Guardrails) `CollectionSettings.lifecycle_policy` is optional and defines partition-window lifecycle behavior. Validation and ownership rules: - `lifecycle_policy.initial_tier` MUST be explicitly provided and MUST NOT be `DATA_TIER_UNSPECIFIED`. - `lifecycle_policy.time_partition_field` MUST reference a declared schema field whose type is `SCALAR_FIELD_TYPE_TIMESTAMP` or `SCALAR_FIELD_TYPE_INT64`; unknown field references MUST be rejected (`SF_SCHEMA_VIOLATION`). - online-ingest rows MUST provide non-null event-time payload for `time_partition_field`: - `RowMutation.operation=upsert`: `UpsertPayload.fields` MUST include non-null `time_partition_field`. - `RowMutation.operation=patch`: `PatchPayload.fields` MUST include non-null `time_partition_field` (explicit carry is required in lifecycle-enabled collections; no read-before-write dependency in this revision). - missing key or explicit-null value MUST be rejected (`SF_SCHEMA_VIOLATION`). - when `time_partition_field` type is `SCALAR_FIELD_TYPE_TIMESTAMP`, source values are stored as Unix microseconds; gateway/runtime MUST losslessly convert to milliseconds before lifecycle-time delta calculations. - `lifecycle_policy.partition_window_ms` MUST satisfy service static-policy minimum floor (for example `>= 3600000`) to prevent object explosion from over-fine partition windows. - late-arrival guard (`max_late_arrival_ms`) is evaluated only on online ingest flows: - `event_time_ms` MUST be extracted from the payload-carried `time_partition_field` value in the admitted mutation entry. - reject when `(tso_physical_ms - event_time_ms) > max_late_arrival_ms`. - future-arrival guard (`max_future_arrival_ms`) is evaluated only on online ingest flows: - reject when `(event_time_ms - tso_physical_ms) > max_future_arrival_ms`. - `lifecycle_policy.transitions[]` MUST be strictly ascending by `transition_after_ms` and MUST be monotonic downgrade-only by tier (`initial_tier -> ... -> COLD`). - each `TierTransition.target_tier` MUST NOT be `DATA_TIER_UNSPECIFIED`. - when `lifecycle_policy.retention_ms` is present, value MUST be strictly greater than `max(transitions.transition_after_ms)`. - lifecycle truth-mode guardrail: - lifecycle policy (especially physical reclamation via `retention_ms`) is allowed only when the authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`. - when the authoritative descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`, control plane MUST reject lifecycle-policy application with `SF_OWNERSHIP_CONFLICT`. Runtime realization rules: - admitted lifecycle policy MUST be materialized into manifest-visible window metadata and a rebuildable CCS hot projection: - committed manifests MUST publish the effective `DataTier` for every lifecycle-managed partition/window and the time-window bounds needed for deterministic query pruning, - CCS MAY maintain additional hot projection tables for lifecycle windows/transition scheduling, but those projections are derivative state rather than a second query-time source of truth. - query-time `search_payload.routing.target_tiers[]` pruning MUST read manifest-published tier metadata directly from the admitted committed snapshot; query hot paths MUST NOT perform per-request CCS lookups to decide tier membership. - lifecycle transition execution MUST preserve manifest-first correctness: - background workers may rewrite/move objects between HOT/WARM/COLD layouts, - a new tier becomes query-visible only after the committed manifest for the affected window/object set is advanced, - retention reclaim becomes physically deletable only after lifecycle policy says the window is beyond `retention_ms` and the committed manifest no longer references the old objects. - generic object-GC horizons MUST NOT reclaim lifecycle-managed data earlier than the collection's admitted `lifecycle_policy.retention_ms`. - recovery MUST rebuild lifecycle hot projections from the admitted lifecycle policy plus committed manifests; no separate handwritten lifecycle truth store is allowed. ## 7. Consistency and Latency Trade-offs ### 7.1 Query Consistency Modes | Mode | Formula | Latency | Consistency Behavior | |---|---|---|---| | `CONSISTENCY_MODE_STRICT` | `effective_read_ts=min(read_ts,safe_ts)` and strict policy enforcement | Highest | exact snapshot guarantee at API contract level | | `CONSISTENCY_MODE_BOUNDED_STALENESS` | same formula + lag bound check | Medium | bounded freshness/latency trade | | `CONSISTENCY_MODE_BEST_EFFORT` | default: `effective_read_ts=safe_ts`; projected-route overlay-bearing `ProjectedQueryRequest`: `effective_read_ts=snapshot_read_ts` | Lowest | highest availability, explicit staleness warnings; projected-route overlay requests are an explicit transport-scoped exception | ### 7.2 Write Durability and Visibility Policies Durability gate: | Policy | ACK Condition | Latency | Consistency Effect | |---|---|---|---| | `DURABILITY_POLICY_INGEST_ACK` | admitted and buffered | lowest | durability not guaranteed; visibility may lag | | `DURABILITY_POLICY_WAL_CHUNK_DURABLE` | durable WAL persisted and anchored on WAL-backed streaming lane | lower | visibility may lag; read-after-write not guaranteed without readiness | Visibility gate: | Policy | ACK Condition | Latency | Consistency Effect | |---|---|---|---| | `VISIBILITY_POLICY_ASYNC` | no visibility wait | lower | ACK may precede visibility | | `VISIBILITY_POLICY_WAIT_FOR_VISIBLE` | publish eligibility confirmed and committed-manifest CAS publish completed | higher | stronger read-after-write semantics | Current admitted combination note: - `INGEST_MODE_BATCH` is admitted only with `DURABILITY_POLICY_INGEST_ACK` plus `VISIBILITY_POLICY_WAIT_FOR_VISIBLE`, so the batch lane does not expose `DURABILITY_POLICY_WAL_CHUNK_DURABLE` as a separate public ACK surface in this revision. Atomicity gate: | Policy | Failure Behavior | Latency | Consistency Effect | |---|---|---|---| | `ATOMICITY_POLICY_FAIL_FAST_PARTIAL` | first rejecting row stops later rows; earlier accepted rows may remain applied | lower | optimized for bulk/degraded lanes, not statement atomicity | | `ATOMICITY_POLICY_ALL_OR_NOTHING` | any rejecting row rejects the whole admitted batch | higher | route-level statement/batch atomicity | ### 7.3 Design Decision Summary The API chooses: - durability-first and replay-safe writes, - explicit freshness signaling (`safe_ts/effective_read_ts`), - deterministic fallback/fail-closed behavior by route and truth mode. This keeps TiDB OLTP correctness independent from retrieval freshness lag. ### 7.4 Canonical Error Precedence Rule When multiple violations coexist in one request, code selection MUST follow `docs/specs/core/error-codes.md` Section `2.1` exactly. This document does not redefine tier order, mandatory detail keys, or tie-break rules. ## 8. Canonical Protobuf Contract The machine-readable canonical public/cross-language protobuf contract is defined in `api/proto/searchfabric/v1/search_fabric.proto`. Normative governance: - `[Rule P-201]` `api/proto/searchfabric/public-proto-set.txt` is the machine-readable manifest for the public/cross-language protobuf contract set in the current revision; it currently resolves to `searchfabric/v1/search_fabric.proto`. - `[Rule P-201]` `api/proto/searchfabric/system-proto-set.txt` is the machine-readable manifest for the full checked-out system owner set; every listed public/internal protobuf contract must remain mechanically valid and semantically aligned with the public contract. - `[Rule P-202]` the current revision keeps the public protobuf contract in `package searchfabric.v1` and the internal-only checked-in owner files in `package searchfabric.v1.internal`; package namespace, not path alone, is the authoritative public/internal protobuf boundary. - `api/proto/searchfabric/v1/search_fabric.proto` remains the canonical public/cross-language protobuf source for code generation and public contract consumption. - This document remains the semantic and compatibility specification for query/write/control behavior, fallback rules, and canonical error mapping. - Contract changes MUST update the owning `.proto` file(s) and the relevant normative sections in this document in the same change. - `gen/rust` and `gen/go` are reserved output locations for checked-in generated bindings; when generated artifacts are introduced beyond placeholders, they MUST be derived directly from `api/proto/searchfabric/v1/search_fabric.proto`, never from Markdown extraction or hand-maintained copies. Validation entry points: - `bash scripts/validate-api-contract.sh` (must validate the current public contract plus the checked-out internal proto owner set) - `while read rel_proto; do protoc --proto_path=api/proto --descriptor_set_out=/tmp/$(basename "${rel_proto%.proto}").pb "$rel_proto"; done < <(grep -vE '^[[:space:]]*(#|$)' api/proto/searchfabric/public-proto-set.txt)` - `while read rel_proto; do protoc --proto_path=api/proto --descriptor_set_out=/tmp/$(basename "${rel_proto%.proto}").pb "$rel_proto"; done < <(grep -vE '^[[:space:]]*(#|$)' api/proto/searchfabric/system-proto-set.txt)` ## 9. Feasibility and Rollout Notes ### 9.1 Feasibility Assessment This design is feasible with current architecture constraints because: - it reuses `read_ts/safe_ts` consistency contracts already modeled in design/test/emulator docs, - it keeps TiDB transactional semantics independent of search projection lag, - and it avoids introducing cross-system transactional coupling. Primary implementation risks: - Aggregation feature mismatch between gateway logical-query lowering assumptions and runtime support. - Recursive bucket aggregation fan-out can cause memory/latency spikes without bounded depth/size governance. - Stream backpressure behavior under large snapshot sessions. - Producer epoch management drift (`source_epoch` lifecycle) can cause false replay conflicts or duplicate acceptance. - Schema fencing timeout tuning during high DDL churn. - Session-leak pressure if snapshot lease/abort handling is not implemented with bounded resource accounting. - Ranking drift risk if gateway paths do not preserve deterministic normalized sort contracts (default `score desc, doc_id asc`; caller sort + forced `doc_id asc` tie-break). - Canonicalization drift risk if client/server implementations do not strictly follow `batch_fingerprint` and query-digest serialization rules. - Long pagination chains can fail nondeterministically if token-coherence validation and bounded retention windows are not implemented correctly. - Composite `after_key` pagination can trigger query-plan cache churn if plan-cache keys are coupled directly to page-varying cursor payloads. - External-ID parse-contract drift (string token + lossless `uint64`) can corrupt cross-ecosystem dedup/pagination behavior if gateways diverge in admission rules. - Highlight/projection assembly drift can break cross-page determinism if `highlight_field_names` dictionary alignment, sparse `QueryHit.highlights`, and projection-order contracts are not kept consistent with canonical ranking/filter semantics. ### 9.2 Rollout Sequence 1. Implement control-plane schema sync first (fencing + drift metrics). 2. Enable CDC ingest-plane stream with adapter-local shadow sampling and offline comparison. 3. Enable production-path canary analysis and result-diff checks from adapter-owned shadow telemetry. 4. Collection-scoped cutover for aux-index, keep deterministic fallback. 5. Controlled enablement of federated-table / `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` strict mode. ### 9.3 Validation Requirements - Stable invariant IDs for validator/test discoverability are tracked in `docs/testing/api-contract-conformance-matrix.md`; that matrix is coverage inventory only and always defers back to this document, `docs/specs/core/error-codes.md`, and `docs/specs/core/search-fabric-architecture.md` for canonical semantics. - Conformance suites for MySQL/ES/Turbopuffer gateways must assert canonical mapping. - Cross-client equivalence tests must verify TiDB federated-table queries against `COLLECTION_TRUTH_MODE_SEARCH_FABRIC`, Elasticsearch Query Gateway, and Turbopuffer Query Gateway emit identical canonical query semantics for equivalent logical queries, even when TiDB transports them through `RelationalQueryService`; TiDB projected-route fallback-vs-success behavior is verified separately by the dedicated fast-path tests. - Compatibility applicability tests must verify profile-to-`collection_truth_mode` boundaries are fail-closed under ownership violations. - Strict-mode stale snapshot behavior must match fallback matrix. - Relational query-service tests must verify `RelationalQueryService.ExecuteProjectedQueryOrFallback` accepts `ProjectedQueryRequest`, `RelationalQueryService.ExecutePrimaryQuery` accepts `PrimaryQueryRequest`, both reject raw `QueryRequest`, route admission is driven by trusted relational caller capability inventory rather than TiDB-branded service naming, `ProjectedQueryRequest` enforces the implementation-validated/conformance-backed overlay state split (`columns=[]`, `rows=[]`, `snapshot_read_ts=0` for no overlay; non-empty canonicalized `columns`, non-empty `rows`, `snapshot_read_ts>0` for overlay-present state), enforces `snapshot_overlay.columns[*].field_path` byte-uniqueness plus byte-ascending canonical order, enforces `snapshot_overlay.rows <= 1000`, rejects duplicate `(doc_id, commit_ts)` overlay row identities, requires tombstones to keep empty `values`, requires non-tombstone overlay rows to keep `values.len == columns.len`, requires `snapshot_read_ts = query.consistency.read_ts` for strict/bounded projected-route overlay requests, requires `query.consistency.read_ts=0` plus `snapshot_read_ts > 0` and `effective_read_ts = snapshot_read_ts` for best-effort projected-route overlay requests, requires continuation requests to resend canonical-overlay-equivalent rows with the same `snapshot_read_ts`, and verifies `PrimaryQueryRequest` exposes no snapshot-overlay carriers at all. - Projected-route fallback tests must additionally verify `ExecuteProjectedQueryOrFallback` is admitted only when the resolved collection descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`, that transport-`OK` `ProjectedQueryResponse.result=fallback` is used for stale-strict, bounded-staleness budget breach, and non-covering projection branches, that `PROJECTED_QUERY_FALLBACK_REASON_UNSPECIFIED` is never emitted, that all top-level projected-route observability fields (`effective_read_ts`, `safe_ts`, `canonical_query_digest`, `relational_execution_digest`, `applied_schema_version`, `applied_schema_generation_token`) are present on both transport-`OK` `success` and transport-`OK` `fallback`, that `ProjectedQueryResponse.relational_execution_digest` is always non-empty on transport-`OK` `success` and transport-`OK` `fallback`, differs when overlay-bearing execution inputs differ despite identical `canonical_query_digest`, and that malformed/authz/unsupported/runtime branches stay non-`OK` gRPC plus canonical `ErrorStatus` with no prior external `GetReadiness` call required to make fallback deterministic. - Internal broker projected-route success tests must verify successful `QueryBrokerInternalService.ExecuteQuery` responses for the `projected_query` branch always carry a non-empty `QueryBrokerExecuteResponse.relational_execution_digest`, that differing overlay-bearing execution inputs change that digest even when `canonical_query_digest` is unchanged, and that shared-query / primary-query success responses leave the field empty. - Query Broker internal stream-attempt / aggregation-reduction observability invariants for these tests are tracked by `docs/testing/api-contract-conformance-matrix.md` invariant `SF-CONF-QUERY-004`. - Primary-route query tests must verify `ExecutePrimaryQuery(PrimaryQueryRequest)` is admitted only when the resolved collection descriptor declares `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`, stays success/error only with no fallback branch, and exposes no projected-route overlay carriers. - TiDB federated-table remote-write tests must verify `StreamMutations` admits `CLIENT_KIND_TIDB_WRITE_GATEWAY` only for `COLLECTION_TRUTH_MODE_SEARCH_FABRIC`, that one SQL statement lowers to one canonical `MutationBatch` with `RowMutation.commit_ts=0`, that TiDB-facing defaults force `policy.durability=wal_chunk_durable` plus `policy.visibility=wait_for_visible`, and that this route also forces `policy.atomicity=ATOMICITY_POLICY_ALL_OR_NOTHING` plus `policy.continue_on_error=false`. - TiDB federated-table remote-write atomicity tests must verify any row-level rejection leaves the whole admitted statement batch with no row-mutation visibility. - Allocator replay tests must verify `AllocateIds` requires non-empty `RequestMeta.idempotency_key`, uses resolved `CollectionRef` as scope, keys replay on authenticated caller identity + `client_id` + `client_kind` + `rpc_method` + collection + `idempotency_key`, replays the original realized-ID result verbatim on identical retry for ordered mixed-mode plans, short-circuits fresh allocator admissibility re-evaluation on a stored successful replay hit (including after later tombstone, policy removal, caller-profile change, or schema-generation rollover), returns the originally recorded `schema_generation_token` together with the originally recorded `realized_doc_ids[]`, returns an `allocator_receipt` bound to that same replay-authoritative result tuple, rejects conflicting replay as `SF_IDEMPOTENCY_CONFLICT` even when the conflicting retry would also hit allocator watermark/conflict checks on a fresh evaluation, and never causes later `StreamMutations` admission to accept missing `RowMutation.doc_id`. - Allocator failure-audit tests must verify failed allocator attempts remain ordinary non-`OK` gRPC + canonical `ErrorStatus` outcomes, MAY be audited separately, and MUST NOT create a second success-shaped replay-truth family that can shadow a later successful replay or fresh admissibility decision. - Allocator catalog-unavailable tests must verify `AllocateIds` returns `SF_CATALOG_UNAVAILABLE` when collection identity exists but authoritative allocator admission facts are unavailable/incomplete, and that the no-active-schema branch emits `ErrorStatus.context.allocator.catalog_unavailable.schema_bootstrap.schema_bootstrap_state=no_active_schema_generation` plus `catalog_unavailable_reason=schema_bootstrap` / `schema_bootstrap_state=no_active_schema_generation` rather than `SF_RESOURCE_NOT_FOUND`. - Allocator typed-error tests must verify `SF_TOO_MANY_OPERATIONS` branches populate `ErrorStatus.context.allocator.limit_rejection`, dense-burn explicit-ID conflicts populate `ErrorStatus.context.allocator.reservation_conflict`, allocator bootstrap-unavailable failures populate `ErrorStatus.context.allocator.catalog_unavailable.schema_bootstrap`, and string diagnostics remain compatibility mirrors rather than the only machine-readable carrier. - Allocator receipt-validation tests must verify missing/forged/expired/scope-mismatched/realized-ID-mismatched/route-intent-mismatched/schema-token-mismatched follow-on receipts fail as `SF_BAD_REQUEST` with `ErrorStatus.context.allocator.receipt_validation.reason` plus `diagnostics["receipt_validation_reason"]`, while caller-mismatched valid receipts fail as `SF_FORBIDDEN` with that same typed branch. - Allocator policy-discoverability tests must verify allocator-optional collections expose `CollectionSettings.allocator_admission_policy.caller_profile`, `max_plan_items_per_request`, and `max_forward_reservation_jump` together, that schema admission rejects allocator-optional collections whose policy omits `caller_profile` or sets either guardrail ceiling to zero, and that the current revision rejects any `caller_profile` value other than `ALLOCATOR_CALLER_PROFILE_ES_WRITE_GATEWAY_ONLY`. - Allocator applicability tests must verify Elasticsearch auto-ID and explicit create/insert `_id` on allocator-optional collections both lower through `AllocateIds`, that the follow-on allocator-backed `MutationBatch` carries the matching `allocator_receipt`, that missing/forged/mismatched receipts are rejected before ingest row admission, that patch/delete/update-style routes still reject allocator use, and that Turbopuffer/TiDB allocator caller kinds remain schema-admission-invalid in the current revision. - Snapshot replay/idempotency tests must verify no duplicate visibility. - Batch replay tests must verify batch-idempotency keying uses `(namespace_id, collection_id, stream_identity.source_partition, batch_id)` and preserves deterministic `batch_fingerprint` conflict semantics. - Stream reconnect tests must verify deterministic resume by `(commit_ts, source_partition, source_epoch, source_seq, token)`. - Stream identity tests must verify stream-scope invariants through required `MutationBatch.stream_identity`, required `MutationBatch.schema_generation_token`, reconnect `MutationBatch.resume_cursor`, and `MutationAck.success.acked_cursor` on ACK with deterministic entry ordering by `(commit_ts, source_partition, source_epoch, source_seq)`; generation-fence tests MUST cover both Flow B active-generation strict match and Flow A historical-token + commit-ts-boundary admissibility. - Flow-A generation-boundary rejection tests must verify `max_entry_commit_ts` crossing token boundary emits `SF_VERSION_CONFLICT` with required `ErrorStatus.context.ingest_schema_boundary_invalidation.invalidated_at_commit_ts=`, and that `ErrorStatus.diagnostics.invalidated_at_commit_ts=` remains only an optional compatibility/string projection mirror. - Stateless gateway stream tests must verify Flow B deterministic identity rule: `source_partition` is retry-stable idempotency identity (client `Idempotency-Key` when provided, otherwise deterministic payload-hash-derived `batch_id`), `source_epoch=1`, 1-based `source_seq` array indexing, and dedup reliance on `(source_partition,batch_id) + batch_fingerprint`. - Flow-B fingerprint stability tests must verify `batch_fingerprint` excludes runtime `RowMutation.commit_ts` so hash remains identical before/after Commit Coordinator TSO injection. - Failover replay tests must verify same `(namespace_id, collection_id, source_partition, batch_id)` with higher `source_epoch` and unchanged business payload is accepted as replay-safe retransmit (no `SF_IDEMPOTENCY_CONFLICT`). - MutationEntry framing tests must verify deterministic acceptance/rejection for `ROW_MUTATION/PROGRESS_MARK/BARRIER_MARK` ordering rules. - Entry-level idempotency tests must verify same idempotency key + different payload is rejected (`SF_IDEMPOTENCY_CONFLICT`). - First-error retry tests must verify sliced-retry contract: when caller retries from `first_rejected_source_seq` with a payload suffix, retry MUST use a new `batch_id`; reusing original `batch_id` with changed payload MUST fail as `SF_IDEMPOTENCY_CONFLICT`. - Ingest operation-budget detail tests must verify `max_entries_per_batch` rejection emits `SF_TOO_MANY_OPERATIONS` with required details (`limit_scope=max_entries_per_batch`, `configured_limit`, `requested_value`). - Gateway-lowered selector-based mutation operations (`update_by_query`, `patch_by_filter`, `delete_by_filter`) tests must verify probe clamp enforcement (`min(caller_requested_limit,max_entries_per_batch,effective_max_result_window)+1`, with `caller_requested_limit=source size when provided else +inf`) and `SF_TOO_MANY_OPERATIONS` emission with selector-materialization clamp details (`limit_scope=selector_materialization_limit`, `configured_limit`, `requested_value`, `effective_constraint_source`). - Row-operation branch tests must verify `RowMutation.operation` is exactly-one (`upsert` / `patch` / `delete`), `delete` carries no field payload, and both `upsert` / `patch` carry `fields` + `schema_version`. - Row vector-write tests must verify `upsert`/`patch` vector payload dimensions are strict-write admission (`Dw == Dschema`) using decoded logical dimensions (`float=len(values)`, `f16=len(bytes)/2`, `int8=len(bytes)`, `binary=len(bytes)*8`), odd-length `f16` payloads reject as `SF_BAD_REQUEST`, and any `Dw != Dschema` rejects as `SF_VECTOR_DIMENSION_MISMATCH` (query-side MRL truncation is not admitted on writes). - Explicit-clear marker tests must verify `FieldData.atomic_value.null_value = NULL_VALUE_NULL` in patch paths deterministically clears previously stored scalar/vector/list/nested fields, and full-replace upsert omission semantics clear omitted schema fields. - DocID split-brain tests must verify strict strip+reject behavior: gateway MUST strip `DocIDMapping.source_field` before canonical emission, and Ingest Plane MUST reject any admitted `upsert`/`patch` payload that still carries this field as `SF_SCHEMA_VIOLATION`. - Mutation item-outcome shape tests must verify `MutationItemOutcome.outcome` oneof exclusivity, `source_seq` echo equality with `MutationBatch.entries[entry_index].source_seq`, and authoritative `apply_result` emission/preservation rules for successful `ROW_MUTATION` outcomes (`accepted` and `replayed`). - Mutation ACK item-outcome completeness tests must verify any admitted batch containing one or more `ROW_MUTATION` entries emits exactly one `item_outcomes[]` element per admitted entry even when `continue_on_error=false`, while progress-only / barrier-only batches may omit `item_outcomes[]`. - Batch-mode ACK reconstruction tests must verify the same completeness/order rule holds under `INGEST_MODE_BATCH`, including preservation of original `entry_index` / `source_seq` mapping rather than writer-local reordered outcome units. - Batch-mode atomicity tests must verify internal batch-lane chunking never weakens original `MutationBatch.policy.atomicity`: `ATOMICITY_POLICY_ALL_OR_NOTHING` admits no partial visibility before whole-batch success, and `ATOMICITY_POLICY_FAIL_FAST_PARTIAL` admits only the canonical accepted prefix. - Row optimistic-precondition tests must verify `RowMutation.expected_commit_ts` mismatch rejects with `SF_VERSION_CONFLICT`; `expected_commit_ts=0` enforces create-only semantics (`latest_commit_ts` for absent doc is `0`); conflict payloads MUST include canonical detail keys `doc_id`, `expected_commit_ts`, and `latest_commit_ts`; when `continue_on_error=true`, mismatched entries MUST emit `MutationItemOutcome.rejected.code=\"SF_VERSION_CONFLICT\"` and other entries continue deterministically; same-batch repeated mutations on one `doc_id` MUST evaluate preconditions against accumulated in-batch state (earlier accepted rows visible to later rows). - Continue-on-error framing tests must verify `policy.continue_on_error=true` rejects batches containing `PROGRESS_MARK` or `BARRIER_MARK` as `SF_BAD_REQUEST`. - Batch first-error tests must verify deterministic stop point, `MutationAck.success.rejection_context` cursor reporting, and typed canonical `ErrorCode` emission in `first_rejected_error_code`. - Batch-mode policy-combination tests must verify `INGEST_MODE_BATCH` rejects `VISIBILITY_POLICY_ASYNC` and `DURABILITY_POLICY_WAL_CHUNK_DURABLE` as `SF_BAD_REQUEST`, and that admitted batch-mode writes still emit one truthful `MutationAck` only after final visibility completion. - Ingest-metrics accounting tests must verify `accepted_entries + duplicate_entries + rejected_entries + skipped_entries` equals admitted entry count, and fail-fast tails under `continue_on_error=false` are counted via `skipped_entries`. - Watermark presence tests must verify `EngineWatermarks` returns exactly the enabled collection modalities: metadata-search and semantic-search frontiers are present only when those modalities exist on the target collection, `metadata_aux_ts` is present only when metadata-aux exists, untouched enabled modalities remain present with unchanged derived frontiers on ACK/readiness carriers, `ProgressMark.watermarks` omission means "no advance in this mark" rather than "modality absent from collection", and `safe_ts` is derived from the emitted enabled-modality fields only. - Undeclared-field policy tests must verify deterministic behavior for `REJECT` / `IGNORE` and `UNSPECIFIED -> REJECT` default. - Strict undeclared-field tests must verify same-bulk rows with undeclared fields are rejected or ignored according to the admitted policy, and gateways never synthesize hidden infer/apply/replay work on the caller's behalf. - Snapshot session tests must cover lease expiry and `AbortSnapshotSync` idempotency. - `StartSnapshotSync` idempotency tests must verify same `client_session_token` replay behavior, empty-token rejection (`SF_BAD_REQUEST`), `snapshot_ts > 0`, `schema_version > 0`, non-empty `schema_generation_token`, and parameter-mismatch conflicts. - `StartSnapshotSync` idempotency-key tests must verify non-empty `RequestMeta.idempotency_key` is required and replay dedupe semantics are deterministic. - Snapshot state-machine tests must verify invalid transitions are rejected deterministically. - Snapshot upload stream tests must verify mandatory `SnapshotUploadInit` first-frame contract, single-terminal `SnapshotUploadComplete` EOF frame, and rejection of duplicated init / duplicated complete / post-complete chunk frames. - Snapshot upload reconnect tests must verify `resume_token` scope/session/principal binding, deterministic `next_expected_seq` resume behavior, expired-token rejection as `SF_INVALID_QUERY` with fresh-session restart semantics, and EOF coherence (`SnapshotUploadComplete.total_chunks_sent == next_expected_seq - 1`). - Snapshot upload reconnect tests must verify per-attempt lower-bound rule: after init with `resume_from_chunk_seq=N`, any frame in that attempt carrying `chunk_seq < N` is rejected as `SF_BAD_REQUEST`; replaying older chunks requires a new stream attempt initialized with lower resume floor. - Snapshot upload init-presence tests must verify `resume_from_chunk_seq` optional semantics (`absent` allowed; present value MUST be `>= 1`; present `0` rejected as `SF_BAD_REQUEST`). - Snapshot session binding tests must verify `sync_session_id`-bound collection equals request `CollectionRef` for upload/commit/abort/progress paths; mismatch must fail closed (`SF_FORBIDDEN` or `SF_BAD_REQUEST` by precedence). - Mutation reconnect tests must verify cursor/token coherence (token-cursor mismatch rejection) and mandatory-token behavior for reconnect with non-zero cursor. - Snapshot chunk tests must verify payload encoding/compression static-policy enforcement and checksum-scope consistency. - Snapshot hash/checksum format tests must verify lowercase-hex width validation for `chunk_checksum` and `expected_root_hash`. - Snapshot commit tests must verify root-hash algorithm mismatch rejection, checksum-chain determinism, and fail-closed commit rejection when upload EOF completion was not successfully acknowledged. - Pagination token tests must verify query digest/snapshot binding and fail-closed behavior on token misuse. - Pagination token error-class tests must verify deterministic mapping on query-execution paths: malformed/signature/key-window failures -> `SF_BAD_REQUEST`, scope/principal mismatch -> `SF_FORBIDDEN`, expiry/semantic mismatch -> `SF_INVALID_QUERY`. - Pagination coherence tests must verify deterministic `QueryRequest.fetch_spec.page_window` mode exclusivity and continuation-page coherence with token-bound `fetch_spec.sort_keys`. - Pagination continuation-shape tests must verify `fetch_spec.continuation_page.next_page_token` is non-empty and empty-token branches are rejected as malformed (`SF_BAD_REQUEST`). - Pagination mode tests must verify requests with unset `fetch_spec.page_window` are rejected (`SF_BAD_REQUEST`) and exactly-one branch (`first_page` / `continuation_page` / `composite_page`) is enforced. - Pagination limit-contract tests must verify: - `ExecuteQuery`: `fetch_spec.limit=0` is accepted only when `search_payload.aggregations` is non-empty or `fetch_spec.track_total_hits` is explicitly configured (`enabled`/`threshold`); otherwise rejected (`SF_BAD_REQUEST`). - continuation requests remain bounded (`fetch_spec.limit > 0`). - `StreamQuery`: `fetch_spec.limit=0` is accepted as unbounded export mode. - Pagination digest-decoupling tests must verify cursor/transport controls (`CompositeAggregation.after_key`, `fetch_spec.limit`, `fetch_spec.page_window`, `fetch_spec.composite_page.next_page_token`, `fetch_spec.search_context`) do not alter `canonical_query_digest`, while semantic query-shape changes still do. - Pagination resource-governance tests must verify service-configured `max_page_depth` deterministic rejection (`SF_TOO_MANY_OPERATIONS`) with required detail keys (`limit_scope=max_page_depth`, `configured_limit`, `requested_value`). - Pagination window-limit tests must verify `(offset + limit) > effective_max_result_window` is rejected with `SF_TOO_MANY_OPERATIONS` and detail keys defined in `docs/specs/core/error-codes.md`. - Query payload-budget tests must verify requests exceeding `max_query_payload_bytes` are rejected pre-parse as `413 SF_PAYLOAD_TOO_LARGE` with required details (`limit_scope=dynamic_payload_budget`, `effective_limit`, `requested_bytes`). - Query `InListPredicate` budget tests must verify per-predicate element count above `max_in_list_elements` is rejected as `400 SF_TOO_MANY_OPERATIONS` with required details (`limit_scope=max_in_list_elements`, `configured_limit`, `requested_value`, `field`). - Query predicate-AST budget tests must verify total predicate nodes above `max_predicate_ast_nodes` are rejected as `400 SF_TOO_MANY_OPERATIONS` with required details (`limit_scope=max_predicate_ast_nodes`, `configured_limit`, `requested_value`). - Dynamic vector-budget tests must verify projected high-dimensional vector requests trigger deterministic tightened payload/result-window limits and emit detail keys defined in `docs/specs/core/error-codes.md` on `SF_PAYLOAD_TOO_LARGE` / `SF_TOO_MANY_OPERATIONS`. - PIT-equivalent tests must verify `fetch_spec.search_context.keep_alive_ms` optional presence semantics (present `> 0`, present `0` rejected as `SF_BAD_REQUEST`), issuance, continuation with `search_context.token`, fresh-token keep-alive in the same lineage, TTL extension, mandatory expiry return whenever `search_context.token` is returned, mandatory `composite_page` token/expiry return whenever a response emits non-empty `BucketResult.composite.after_key`, optional `search_context` return only when keep-alive is admitted, and natural-expiry-only invalidation semantics. - PIT first-page boundary tests must verify first-page requests always reject `fetch_spec.search_context.token` as `SF_INVALID_QUERY`; composite-pagination continuation must use `fetch_spec.composite_page` (not first-page branch). - PIT invalidation tests must verify older unexpired tokens in the same lineage remain usable until natural expiry, and that query failure uses canonical token error mapping (`SF_INVALID_QUERY`, `SF_BAD_REQUEST`, `SF_FORBIDDEN`) rather than a separate close-state surface. - Required-carrier tests must verify the canonical proto keeps `sf_required_field` on the mandatory public request-visible carrier inventory from Section `3.7`, and that missing annotated carriers fail closed before business execution. This conformance set MUST cover query/helper, snapshot/control/session-init, catalog, and ingest-stream carrier families rather than only the Query Gateway subset or unary RPCs. - `GetCollectionStatistics` fast-path tests must verify `target_fields=[]` requires empty `extended_requests[]`, returns `collection_stats` plus `stats_commit_ts`, `collected_at_unix_ms`, `applied_schema_version`, and `applied_schema_generation_token`, and keeps `field_stats[]` / `extended_stats[]` empty. - GetCollectionStatistics success-shape tests must verify `collection_stats` is always present on success, that `applied_schema_version` / `applied_schema_generation_token` are always returned, and that immutable snapshot identity is audited as the composite tuple (`applied_schema_generation_token`, `stats_commit_ts`). - `GetCollectionStatistics` transport-split tests must verify transport `OK` returns `GetCollectionStatisticsSuccessResponse` directly, while malformed/authz/not-found/bootstrap/convergence/freshness/runtime branches use non-`OK` gRPC status plus canonical `ErrorStatus` rather than a public `success/error` envelope. - GetCollectionStatistics ordering tests must verify `field_stats[]` returns canonical `field_name` byte-ascending order and `extended_stats[]` returns canonical tuple-identity order by comparing `columns[]` left-to-right using canonical field-path byte order, with the shorter tuple sorting first when one tuple is a strict prefix of another, and that the serialized response preserves that same deterministic order. - `GetCollectionStatistics` validation tests must verify unknown or duplicated `target_fields[]`, singleton `extended_requests[].columns`, duplicated field paths inside one extended request, and duplicated tuple requests across `extended_requests[]` fail closed as `SF_BAD_REQUEST`. - `GetCollectionStatistics` validation tests must verify non-empty `extended_requests[]` combined with `target_fields=[]` fails closed as `SF_BAD_REQUEST`. - `GetCollectionStatistics` validation tests must verify `extended_requests[].columns` outside the declared snapshot-supported extended-stat tuple family fail closed as `SF_BAD_REQUEST` rather than triggering ad hoc computation or silent omission. - `GetCollectionStatistics` precedence tests must verify canonical registry precedence still governs multi-violation cases for this RPC, including `SF_RESOURCE_NOT_FOUND` outranking `SF_RATE_LIMITED`. - `GetCollectionStatistics` shape tests must verify `FieldStatistics.stats_type` oneof exclusivity, `HistogramBucket.count` excludes `top_n[]`, and explicit empty arrays on `multi_valued=true` fields increase `ScalarFieldStatistics.null_count` without emitting `min_value` / `max_value`. - `GetCollectionStatistics` shape tests must verify `ExtendedStatistics.columns` echoes the exact ordered tuple identity declared in the snapshot support profile. - `GetCollectionStatistics` ownership/freshness tests must verify descriptor-level `approx_stats` is not reused as optimizer statistics, schema-bootstrap collections fail with canonical bootstrap details, admitted-but-not-`READY` generations fail with canonical schema-convergence details, first-serving generations expose an initial zero-row snapshot before becoming the current `latest READY` lineage, schema-only serving-lineage promotion still produces a distinct immutable snapshot durable identity even when no user-data frontier advanced, successful responses expose `collected_at_unix_ms` as the authoritative freshness timestamp, `max_staleness_ms` is evaluated against `collected_at_unix_ms` snapshot freshness rather than query visibility, bounded freshness chooses the newest eligible snapshot within the current query-serving lineage only, and a budget miss fails closed with canonical statistics-freshness details in `ErrorStatus.context.query.catalog_unavailable.statistics_freshness` (with `diagnostics` only as an optional mirror). - Elasticsearch mapping tests must verify `refresh` to (`policy.durability`, `policy.visibility`) normalization and active-profile `search_after` token-only continuation (`fetch_spec.sort_keys` + `QueryRequest.fetch_spec.continuation_page.next_page_token`), while raw sort-values-array continuation inputs are rejected as out-of-subset. - Elasticsearch mapping tests must verify omitted source `track_total_hits` deterministically lowers to canonical default `TrackTotalHitsSpec{threshold=10000}`. - Elasticsearch Profile-2 tests must verify PIT endpoints (`POST /{index}/_pit`, `DELETE /_pit`) and `search_after + PIT` coupling are rejected as deferred-to-Profile-3 behavior. - Elasticsearch mapping-endpoint tests must verify `PUT/GET /{index}/_mapping` first resolves external `{index}` to canonical `CollectionNameBinding{namespace_id=, collection_name=}`, ensures collection identity through `PutCollectionMetadata.name_binding` before the first schema apply when no authoritative `CollectionRef` exists yet, and then round-trips through canonical control-plane schema payload (`DocIDMapping`/`FieldSchema[]`/`CollectionSettings`) without semantic drift for admitted fields. - Elasticsearch mapping-endpoint tests must verify first-create `_mapping` two-step flows emit the same replay-stable non-empty outer `idempotency_key` plus explicit gateway `meta.client_id` / `meta.client_kind` on both `PutCollectionMetadata` and `PutCollectionSchema`. - Elasticsearch mapping-endpoint tests must verify gateway-managed schema writes set `PutCollectionSchema.scope.ddl_commit_ts=0`, and first schema creation uses the zero-state fence tuple (`schema_version=1`, `fence.expected_prev_schema_version=0`, `fence.schema_generation_token="__initial__"`, `update_policy=SCHEMA_UPDATE_POLICY_INITIAL_CREATION`). - Elasticsearch mapping-endpoint tests must verify unsupported mapping branches (outside canonical schema model) fail closed before control-plane apply (`SF_UNSUPPORTED_OPTION`). - Elasticsearch hybrid-lowering tests must verify requests that activate both metadata-search and semantic vector retrieval branches explicitly set `search_payload.dedup_by_doc_id=true`, use deterministic RRF at the compatibility layer, and emit no duplicate top-level `doc_id` hits after pagination. - Elasticsearch mapping tests must verify `fields` multi-field branches normalize to `FieldSchema.subfields[]` with deterministic subfield ordering, unique-name validation, stable `subfields[].config` projection, and strict non-container subfield guard (`subfields[].config.field_definition` MUST NOT be `object`/`nested`). - Elasticsearch mapping tests must verify `POST/GET /{index}/_count` uses canonical partial-aggregate count path and pathless `GET/POST /_count` fails closed (`SF_INDEX_TARGET_REQUIRED`). - Elasticsearch mapping tests must verify `size=0` lowering follows canonical `fetch_spec.limit=0` guardrail: accepted only when `search_payload.aggregations` is non-empty or `fetch_spec.track_total_hits` is explicitly configured (`enabled`/`threshold`); all other `size=0` requests fail closed (`SF_BAD_REQUEST`). - Elasticsearch mapping tests must verify `query.multi_match` deterministic lowering into retrieval-tree form: `cross_fields -> TextClause.fields[]`, `best_fields -> RetrievalGroup{DIS_MAX,tie_breaker}`, and `most_fields -> RetrievalGroup{WEIGHTED_SUM}`. - Elasticsearch mapping tests must verify scalar `bool.should` / `constant_score` boost branches are lowered into retrieval-tree leaves with node-local `boost`, are placed under `must_clauses[]` or `should_match.clauses[]` according to source bool structure, and when `bool.should` has no `must`/`filter` and omits `minimum_should_match`, gateway normalizes the owning `ShouldMatchSpec.threshold` to `absolute_count=1`. - Elasticsearch OCC loop tests must verify read responses expose `_seq_no` from canonical `QueryHit.version_commit_ts` and inject static `_primary_term=1`, subsequent writes with `if_seq_no` lower to `RowMutation.expected_commit_ts`, `if_primary_term=1` is admitted, and `if_primary_term>1` is rejected deterministically. - Elasticsearch mapping tests must verify `update_by_query`/`delete_by_query` subset lowering to deterministic ordered mutation plans and two-phase probe clamp rule (`min(caller_requested_limit,max_entries_per_batch,effective_max_result_window)+1`, `caller_requested_limit=source size when provided else +inf`). - Elasticsearch by-query conflict-mode tests must verify `conflicts=abort` lowers to fail-fast canonical execution (`continue_on_error=false`), `conflicts=proceed` lowers to per-entry continuation (`continue_on_error=true`), and `version_conflicts` is reconstructed from rejected compare-and-set outcomes rather than gateway-local guesses. - Elasticsearch aggregation-pagination tests must verify `aggs.*.composite` + `after` lower to `CompositeAggregation{sources,limit,after_key}` plus `fetch_spec.composite_page` continuation branch deterministically, and response-side `BucketResult.composite.after_key.fields` is raised as Elasticsearch outer `after_key`. - Elasticsearch aggregation-metric tests must verify `aggs.*.value_count(field=X)` lowers deterministically to `MetricAggregation{builtin_function=AGGREGATE_FUNCTION_COUNT, field=X}`, rejects missing/empty `field` as malformed (`SF_BAD_REQUEST`), and preserves value-count (non-null field) semantics. - Elasticsearch aggregation-pagination tests must verify continuation requests carrying non-empty `CompositeAggregation.after_key` require `fetch_spec.page_window=composite_page`, valid bound `fetch_spec.composite_page.next_page_token`, and exact cursor equality between the lowered `after_key` and the composite-page token payload; missing/mismatched cursor state is rejected (`SF_INVALID_QUERY`). - Elasticsearch aggregation guardrail tests must verify `CompositeAggregation.sources` over route policy ceiling (`max_composite_sources`) is rejected fail-closed (`SF_UNSUPPORTED_QUERY_FEATURE`) with canonical unsupported-feature detail keys. - Elasticsearch mapping tests must verify highlight subset behavior normalizes to `fetch_spec.text_options`, does not perturb ranking/filter execution semantics, and remains page-token/digest coherent. - Elasticsearch gateway tests must verify pathless `POST /_msearch` header items without `index` fail closed (`SF_INDEX_TARGET_REQUIRED`) before canonical query fan-out. - Elasticsearch bulk mapping tests must verify deterministic per-item status reconstruction over canonical write admission outcomes. - Elasticsearch bulk execution-mode tests must verify `policy.continue_on_error=true` yields deterministic `MutationAck.success.item_outcomes[]` for every action line, including authoritative post-apply `apply_result` on successful row mutations. - Elasticsearch bulk undeclared-field tests must verify undeclared fields inside one admitted bulk request are handled only by the admitted strict/ignore policy, never by implicit schema apply, and deterministic item-level schema failures surface as `SF_SCHEMA_VIOLATION`. - Elasticsearch bulk wire-shape tests must verify valid bulk requests always return HTTP `200`, set top-level `errors` correctly, and emit per-item `status` (`400`/`404`/`409`/`429`/`503`/`504`) with Elasticsearch-compatible item `error` objects for failed actions. - Elasticsearch aggregation defensive-error tests must verify admission-time aggregation rejects map to `parsing_exception` and execution-time aggregation failures map to `search_phase_execution_exception`, while canonical internal code remains `SF_UNSUPPORTED_QUERY_FEATURE`. - Bulk/By-Query visibility tests must verify gateway-lowered multi-batch execution has no cross-batch atomicity guarantee and may expose intermediate states; single-batch and storage-snapshot consistency branches must remain deterministic. - MySQL parser tests must verify parser selection is canonicalized via typed control-plane schema fields (`FieldSchema.config.field_definition.text.analysis.analyzer`, `FieldSchema.config.field_definition.text.analysis.options`, `FieldSchema.config.field_definition.text.fulltext`), query-side parser override is rejected, and runtime-prerequisite failures are deterministic. - MySQL tokenizer-option tests must verify `ngram_token_size` and related parser-driven tokenization settings are normalized to `FieldSchema.config.field_definition.text.analysis.options` with deterministic key/value canonicalization. - MySQL DDL/schema tests must verify FULLTEXT parser/tokenization/tuning semantics are normalized into canonical schema carriers (`FieldSchema.config.field_definition.text.analysis.analyzer`, `FieldSchema.config.field_definition.text.analysis.options`, `FieldSchema.config.field_definition.text.fulltext`) and charset/collation directives, when accepted for round-trip, are limited to supplemental `FieldSchema.options`/`CollectionSettings.options` metadata with no effect on canonical equivalence or evolution-class decisions. - Fulltext storage-tuning tests must verify `FieldSchema.config.field_definition.text.fulltext.storage_tuning` (`postings_options`, `precompute_phrases`, `precompute_prefixes`, `precompute_document_terms`, `length_normalization`) is admitted, round-trippable, and deterministic across gateways. - Fulltext logical-baseline tests must verify `FieldSchema.config.field_definition.text.fulltext.default_similarity` and `FieldSchema.config.field_definition.text.fulltext.default_search_behaviors` can update deterministically without physical-layout drift, and present-but-empty `SimilarityModel{}` (`model` oneof unset) is rejected as `SF_SCHEMA_VIOLATION`. - MySQL gateway tests must verify forbidden `MATCH + ROLLUP` shapes are rejected with deterministic `SF_UNSUPPORTED_SYNTAX` at pre-admission. - Turbopuffer mapping tests must verify `multiQuery` ordered fan-out stability, per-query error isolation, and outer `strong` consistency sharing one gateway-acquired strict `read_ts` across all child queries. - Turbopuffer mapping tests must verify deterministic `consistency.level` mapping (`strong`/`eventual`), including `strong -> CONSISTENCY_MODE_STRICT` with gateway-acquired `read_ts > 0` and `eventual -> CONSISTENCY_MODE_BEST_EFFORT` with `read_ts=0`, plus top-level billing/performance envelope preservation without nested-shape dependency in canonical logic. - Turbopuffer mapping tests must verify `vector_encoding` (`float`/`base64`) controls response-side vector serialization and does not mutate canonical query-input parsing. - Turbopuffer mapping tests must verify deterministic `include_attributes`/`exclude_attributes` normalization for attribute fields only, with stable ordering. - Turbopuffer mapping tests must verify schema lowering synthesizes canonical `DocIDMapping{source_field=\"id\", source_field_type=SCALAR_FIELD_TYPE_UINT64, immutable=true, unique=true, generation_policy=DOC_ID_GENERATION_POLICY_CALLER_REQUIRED}` in the current profile and round-trips without leaking `id` into user-attribute schema entries. - Turbopuffer mapping tests must verify `aggregate_by` / `group_by` / object-form `limit` lower deterministically to canonical aggregation tree controls (`AggregationRequest[]`, `CompositeAggregation.limit`, `TopHitsAggregation.limit`) and reject mismatched `group_by` vs `limit.per.attributes`. - Turbopuffer response tests must verify grouped queries reconstruct `aggregation_groups[]` deterministically from canonical bucket results and per-group top-hits payloads. - Turbopuffer explain tests must verify `plan_text` is a gateway-local artifact over the admitted canonical plan and does not affect canonical digest/execution. - Turbopuffer mapping tests must verify `Row.vector` return policy is independent from attribute projection, uses deterministic `fetch_spec.vector_projections["vector"]`, preserves query-digest binding, requires the sole admitted vector field to be named `vector`, and rejects row-return queries when the collection exposes any additional vector field. - Drop-collection contract tests must verify `DropCollectionRequest` is collection-scoped (no DDL commit timestamp in request shape), that route-level delete lowering uses `collection=`, and that fencing/idempotency still block ABA/conflicting replays deterministically. - Turbopuffer write mapping tests must verify deterministic lowering for `patch_by_filter`, `delete_by_filter`, and `copy_from_namespace`. - Turbopuffer write mapping tests must verify both `patch_by_filter` and `delete_by_filter` enforce the shared two-phase probe clamp and deterministic partial-prefix rule. - Turbopuffer write mapping tests must verify `return_affected_ids` reconstruction accuracy, including source-order preservation for explicit-ID operations and realized `doc_id asc` ordering for selector-driven operations, plus deterministic `disable_backpressure -> policy.ingest_mode` lowering (`true -> BATCH`, `false/omitted -> STREAMING`) and the required batch-lane policy pair (`BATCH -> visibility=wait_for_visible + durability=ingest_ack`) under throttling branches. - Turbopuffer schema tests must verify `POST/GET /v1/namespaces/{namespace}/schema` normalize/round-trip `AttributeSchemaConfig` (`filterable`, `regex`, `full_text_search`, `ann`, scalar/array type family, vector type) through typed canonical schema fields with no semantic drift (`filterable -> config.capabilities.filterable`, `regex -> config.capabilities.pattern_matchable`, `full_text_search=true|config -> config.capabilities.searchable=true`, `[]type -> config.capabilities.multi_valued=true`). - Turbopuffer schema tests must verify gateway-managed schema writes set `PutCollectionSchema.scope.ddl_commit_ts=0`, and first schema creation uses the zero-state fence tuple (`schema_version=1`, `fence.expected_prev_schema_version=0`, `fence.schema_generation_token="__initial__"`, `update_policy=SCHEMA_UPDATE_POLICY_INITIAL_CREATION`). - Turbopuffer inline-schema write tests must verify write-route `schema` is semantically identical to fenced `PutCollectionSchema`, executes before any mutation emission, replays idempotently under identical outer write identity, rejects conflicting schema replay under the same identity, and does not roll back already-applied schema when later row-level processing partially fails. - Turbopuffer namespace-mapping tests must verify external account resolution to canonical `namespace_id` stays outside this design, external `{namespace}` maps to `CollectionDescriptor.collection_name`, `GET /v1/namespaces` projects `NamespaceSummary.id=collection_name`, and schema/query/write/delete routes resolve concrete `CollectionRef` inside that namespace through typed collection catalog operations. - Turbopuffer metadata tests must verify `GET /v1/namespaces/{namespace}/metadata` reads typed collection metadata (`created_at_unix_ms`, `updated_at_unix_ms`, `approx_stats`, `encryption`, `index`) from `GetCollectionMetadata`, reads `schema` from `GetCollectionSchema` on the resolved collection, never synthesizes those fields from gateway-local defaults or opaque `options`, and requires `GetCollectionMetadata.success.descriptor.completeness.ready`; `completeness.publishing` MUST reject with `SF_CATALOG_UNAVAILABLE` plus canonical completeness details. - Turbopuffer query-performance tests must verify `approx_namespace_size` is sourced from `GetCollectionMetadata.success.descriptor.approx_stats.approx_row_count` for the resolved route namespace token (not canonical namespace catalog stats), and that routes requiring this field honor `descriptor.completeness.ready` rather than returning `0` while publication is still incomplete. - Vector schema tests must verify vector base-type invariants (`dimensions > 0`, non-`UNSPECIFIED` `element_type`/`distance_metric`, including `F16`/`F32`/`INT8`/`BINARY`) and `FieldSchema.config.field_definition.vector.search_tuning` (`optimization_goal`, `quantization` in `NONE`/`INT8`/`BINARY`) are admitted/round-trippable deterministically; incompatible combinations fail closed by precedence. - Vector schema tests must verify `optimization_goal=VECTOR_OPTIMIZATION_GOAL_UNSPECIFIED` deterministically falls back to balanced behavior, explicit `optimization_goal=VECTOR_OPTIMIZATION_GOAL_BALANCED` preserves the same balanced intent, and `quantization=VECTOR_QUANTIZATION_UNSPECIFIED` resolves deterministically. - Vector query tests must verify `VectorClause.vector` sets exactly one `VectorValue.data` branch (`f16_vector` / `float_vector` / `int8_vector` / `binary_vector`), branch-vs-schema `element_type` compatibility is enforced, query-vector dimensions use decoded logical rules (`float=len(values)`, `f16=len(bytes)/2`, `int8=len(bytes)`, `binary=len(bytes)*8`) and obey `0 < Dq <= Dschema` with deterministic `Dq < Dschema` prefix-aligned retrieval semantics (Reader computes ANN/distance on stored-vector prefix `Dq`), odd-length `f16` payload rejects as `SF_BAD_REQUEST`, `Dq > Dschema` rejects as `SF_VECTOR_DIMENSION_MISMATCH`, `Dq=0` rejects as `SF_BAD_REQUEST`, `clause_topk` fallback chain semantics (`0`/absent => `search_payload.global_candidate_topk`, then `fetch_spec.limit`, terminal `0` => bypass vector retrieval) are deterministic, and `local_filter` / `filter_strategy_hint` (`AUTOMATIC`, `POST_FILTER`, `PRE_FILTER`, `ITERATIVE_FILTER`, `FLAT_SEARCH`) admission/normalization behavior is deterministic and does not weaken global-filter correctness; planner MUST conjunct top-level `global_filter` + clause-local filter for top-level vector clauses and nested `child_filter` + clause-local filter for nested vector clauses before ANN traversal. - Query defaulting tests must verify absent `search_payload.global_filter` is interpreted as `MatchAll` and never narrows candidate eligibility. - Nested query tests must verify `NestedClause` admission (`path`, `score_mode`, nested-local query payload shape), nested recursion behavior, fail-closed rejection of `NestedQueryPayload` attempts to carry global-only controls, optional `NestedQueryPayload.root_retrieval` semantics (`absent -> filter-only nested evaluation`), and deterministic nested-local retrieval-tree execution. - Inner-hits fetch tests must verify `InnerHitsSpec.limit/offset/sort_keys/track_total_hits` semantics, `attribute_projection` oneof intent semantics (or inheritance from top-level `fetch_spec.attribute_projection` when absent), deterministic shallow paging per parent hit, `doc_id` sort-target mapping to `parent_array_offset`, and auto-append of `parent_array_offset ASC` when caller sort omits virtual `doc_id`. - Text-clause mode-override tests must verify branch-driven semantics: unset branch uses schema default mode; `natural_language` / `boolean_match` / `plain_match` branches override deterministically; query expansion is valid only in `natural_language`; `PlainMatchOptions.operator` unset defaults to `TEXT_MATCH_OPERATOR_OR`, explicit `OR`/`AND` both preserve deterministic token-combination semantics, and `PlainMatchOptions.minimum_should_match` expression parsing/lowering is deterministic. - Sort-key target tests must verify exactly-one `SortKey.target` branch (`field` / `score` / `doc_id`), deterministic auto-append of `doc_id` tie-break, and fail-closed rejection when `FieldSortTarget.field` resolves to vector/object/container (non-scalar/keyword-leaf) fields. - Sort-key virtual-target tests must verify source payloads attempting to attach `missing_value_policy`/`mode`/`nested_context` to `_score`/`_id` are rejected (`SF_BAD_REQUEST`) at gateway pre-admission. - Nested-sort tests must verify `FieldSortTarget.nested_context` contracts: required for nested field sorts, forbidden for non-nested fields, `path` validity, and optional `nested_context.filter` nested-scope behavior. - Multi-valued sort tests must verify explicit empty arrays (`multi_valued=true` fields) are treated as missing/null during `MIN/MAX` extraction and routed through `missing_value_policy` (including default policy branches when field policy is absent). - Aggregation tree tests must verify `AggregationRequest.name` sibling uniqueness, regex validity (`^[a-zA-Z0-9_.-]{1,128}$`), node-shape validity (`metric/top_hits` leaf constraints, `bucket` recursion constraints including `nested` / `reverse_nested` context-switch nodes), and deterministic recursive ordering. - Bucket aggregation tests must verify `terms`/`histogram`/`date_histogram`/`range`/`composite` parameter validation, `RangeAggregation.Range` inclusive-bound semantics (`from_inclusive`/`to_inclusive` non-empty interval rule), deterministic range-key generation/fill when source omits key, mandatory usage of provided range key in emitted bucket identifiers, duplicate-range-key rejection, `TermsAggregation.missing_value_replacement` null/missing-bucket semantics + type-castability checks, `TermsAggregation.order[]` (`AggregationSortKey`) target exclusivity, explicit non-`UNSPECIFIED` sort order, `builtin` enum validity (`BUILTIN_AGGREGATION_SORT_COUNT`/`BUILTIN_AGGREGATION_SORT_KEY`, reject `BUILTIN_AGGREGATION_SORT_UNSPECIFIED`), composite-source ordering/uniqueness/rule checks, root-level scalar-only composite-source target enforcement (nested sources rejected as `SF_UNSUPPORTED_QUERY_FEATURE`), and deterministic bucket ordering/count semantics. - Nested aggregation tests must verify `NestedAggregation.path`/`ReverseNestedAggregation.path` context legality and fail-closed behavior on invalid ancestor/root transitions. - Metric aggregation tests must verify builtin_function metrics (`AGGREGATE_FUNCTION_COUNT` / `AGGREGATE_FUNCTION_SUM` / `AGGREGATE_FUNCTION_MIN` / `AGGREGATE_FUNCTION_MAX` / `AGGREGATE_FUNCTION_AVG`) reject `AGGREGATE_FUNCTION_UNSPECIFIED`, enforce deterministic field-source rules (`MetricAggregation.field` is sole authority; `COUNT` MAY leave empty; non-`COUNT` builtin and `cardinality` require non-empty), enforce numeric type safety for `SUM`/`MIN`/`MAX`/`AVG` (`INT64`/`UINT64`/`FLOAT`/`DOUBLE` only; mismatch => `SF_SCHEMA_VIOLATION`), enforce `missing_value_replacement` castability to target numeric field type when present and non-null, and preserve deterministic null encoding through `MetricResult.value.null_value`. - Metric aggregation tests must verify `COUNT` branch dual semantics are deterministic: empty `field` means document count, non-empty `field` means value-count of non-null field values. - Top-hits aggregation tests must verify per-bucket `TopHitsResult.result_type` polymorphism (`global_hits` vs `nested_hits`), `TopHitsAggregation.limit/offset` shallow-paging semantics, `TopHitsAggregation.attribute_projection` oneof intent semantics (or inheritance from top-level `fetch_spec.attribute_projection` when absent), `TopHitsAggregation.vector_projection_intent` oneof semantics (`inherit` / `explicit_map` / `none`, with non-empty `explicit_map.vector_projections` validation), `TopHitsAggregation.text_options` semantics (or inheritance from top-level `fetch_spec.text_options` when absent), and deterministic ordering/projection/score/highlight behavior for the admitted branch. - Top-hits aggregation tests must verify each admitted branch (`global_hits` / `nested_hits`) is self-contained with local `projected_field_names`/`highlight_field_names`, and each branch-local hit aligns only to branch-local dictionaries. - Gateway aggregation-defense tests must verify pre-admission depth/node/fanout/bucket ceilings (including `max_aggregation_depth` and `max_composite_sources`) intercept malformed/adversarial trees before Query Broker forwarding, with deterministic reject/truncate branches and `success.aggregation_truncation` population where applicable. - Semantic result-set facet tests must verify semantic-only / vector-dominant aggregation universes admit only scalar metric nodes plus non-composite top-level bucket nodes (`terms` / `histogram` / `date_histogram` / `range`) over top-level scalar fields, and reject `composite`, `top_hits`, `nested`, and `reverse_nested` branches as `SF_UNSUPPORTED_QUERY_FEATURE`. - Query Broker aggregation-reduction observability tests must verify the internal proto exposes explicit `QUERY_BROKER_PHASE_AGGREGATION_REDUCTION` timing, keeps metadata partial reduction in `aggregation_subplans[]`, models semantic/vector-dominant broker-local facet evaluation through `facet_subplans[]`, and splits `fusion_input_candidates` / `fusion_output_candidates` from `metadata_aggregation_partial_inputs` / `metadata_aggregation_output_entries` plus `facet_reduction_input_candidates` / `facet_reduction_output_entries` in `QueryBrokerExecutionReceipt`. - Query feature-policy tests must verify projected vector-field requests are rejected when vector projection is disabled by service policy. - Vector projection tests must verify field-level `fetch_spec.vector_projections` admission (`VECTOR_PAYLOAD_FORMAT_EXACT_FLOAT` / `VECTOR_PAYLOAD_FORMAT_EXACT_F16` / `VECTOR_PAYLOAD_FORMAT_QUANTIZED_INT8` / `VECTOR_PAYLOAD_FORMAT_QUANTIZED_BINARY`), projection-key validity (schema vector field), default fallback behavior, policy gating, deterministic quantized output, MRL `target_dimensions` semantics (`unset/0 -> schema dimensions`, `0schema dimensions` -> `SF_VECTOR_DIMENSION_MISMATCH`), and digest coherence with `fetch_spec.attribute_projection`. - Attribute projection tests must verify `fetch_spec.attribute_projection` absence defaults to `return_all` with empty excludes, and when present `intent` obeys exactly-one branch semantics (`return_all` / `return_explicit` / `return_none`), `return_explicit.include_fields` non-empty rule, scalar-only field-name admission, and deterministic include/exclude normalization before canonical emission. - Vector projection tests must verify explicit override entries reject present `VectorProjectionSpec.format=VECTOR_PAYLOAD_FORMAT_UNSPECIFIED` as malformed (`SF_BAD_REQUEST`), while absent `format` deterministically resolves to collection/profile default fallback. - Total-hits tracking tests must verify `fetch_spec.track_total_hits` branches: omitted => default `threshold=10000` for unary and bounded stream (`limit>0`), while unbounded stream (`StreamQuery limit=0`) omission defaults to `enabled=false`; `enabled=true` exact count; `enabled=false` omitted/disabled `success.total_hits`; `threshold=N` producing `TOTAL_HITS_RELATION_EQUAL_TO` vs `TOTAL_HITS_RELATION_GREATER_THAN_OR_EQUAL_TO` relation with deterministic `value`; and any emitted `success.total_hits.relation` rejecting `TOTAL_HITS_RELATION_UNSPECIFIED`. - Stream unbounded-export tests must verify `StreamQuery limit=0` rejects explicit `track_total_hits.enabled=true` as `SF_BAD_REQUEST`. - Total-hits tracking tests must verify present-but-empty `TrackTotalHitsSpec{}` (no `tracking` branch) is rejected as malformed (`SF_INVALID_ARGUMENT`). - Query text-fetch tests must verify `fetch_spec.text_options.require_highlighting` admission rules, `explicit_fields` gating (`require_highlighting=true` => non-empty text-field list, `require_highlighting=false` => empty), default tag materialization (``,``), malformed one-tag-only rejection, engine-side snippet generation, and continuation digest coherence. - Control static-policy tests must verify typed schema-field admission gates are enforced deterministically. - Control static-policy tests must verify analyzer admission rejects IDs outside service-supported built-in analyzers with `SF_UNSUPPORTED_OPTION`. - Control static-policy tests must verify analyzer override key/value grammar enforcement and fail-closed behavior for unsupported keys or invalid values. - Control schema-validation tests must verify `CollectionSettings.index_sort` invariants (when present, fields must be non-empty; deterministic field-order semantics) are enforced deterministically. - Control schema-validation tests must verify `IndexSortFieldSpec.order/mode/missing_value_policy` reject `*_UNSPECIFIED` values deterministically (`SF_SCHEMA_VIOLATION`). - Control schema-validation tests must verify `CollectionSettings.max_result_window` presence semantics: unset is allowed, present value MUST be `> 0`, and present `0` is rejected (`SF_SCHEMA_VIOLATION`). - Control schema-validation tests must verify `PutCollectionSchema.scope.ddl_commit_ts` caller semantics: TiDB DDL sync uses authoritative non-zero DDL commit TS, while gateway-managed Search-Fabric-truth schema writes use the reserved `0` sentinel. - Control schema-fence tests must verify overlength `SchemaFencePrecondition.schema_generation_token` is rejected as `SF_BAD_REQUEST`, and that implementations never silently truncate/hash fence tokens before replay or CAS checks. - Control schema-validation tests must verify first schema creation on a collection with descriptor-but-no-lineage state uses the zero-state tuple (`schema_version=1`, `fence.expected_prev_schema_version=0`, `fence.schema_generation_token="__initial__"`, `update_policy=SCHEMA_UPDATE_POLICY_INITIAL_CREATION`) and that accepted applies replace the sentinel with a real active generation token. - Control schema-status tests must verify `SchemaSyncStatusSuccessResponse.effective_max_result_window` is always populated with resolved effective value (explicit schema value or control-plane default when schema omitted `max_result_window`), and missing echo is treated as contract drift rather than runtime-fallback-eligible behavior. - Control schema-validation tests must verify `CollectionSettings.lifecycle_policy` invariants: `time_partition_field` type contract (`TIMESTAMP`/`INT64` only), lifecycle-enabled `upsert` and `patch` payloads requiring explicit non-null `time_partition_field`, missing/null event-time fail-closed behavior, `partition_window_ms` static minimum floor, strict `transitions[].transition_after_ms` ascending order, monotonic downgrade-only `target_tier`, and `retention_ms > max(transitions.transition_after_ms)` when present. - Control ownership tests must verify lifecycle-policy applicability is restricted to `COLLECTION_TRUTH_MODE_SEARCH_FABRIC`; requests carrying lifecycle policy for `COLLECTION_TRUTH_MODE_EXTERNAL` collections MUST fail closed with `SF_OWNERSHIP_CONFLICT`. - Control schema-validation tests must verify `KeywordConfig.overflow_protection` contract (`max_chars > 0`, non-`UNSPECIFIED` `OverflowBehavior`) and deterministic overlong-value handling branches (`reject`, `skip_indexing`, `truncate_and_index`), including per-element `max_chars` semantics for `multi_valued=true` fields. - Control schema-validation tests must verify `ObjectConfig` / `NestedConfig` container contracts: `fields[]` child-schema validity, `ObjectConfig.dynamic_policy` enum admission/defaulting semantics under the active ignore-or-strict-only contract, and `NestedConfig.max_docs_per_parent` presence semantics (`unset` follows service default, present value MUST be `> 0`). - Control schema-validation tests must verify three-axis defaults and guardrails: absent/`UNSPECIFIED` `FieldConfig.access_intent` resolves to effective `FREQUENT`, absent/`UNSPECIFIED` `VectorTypeSpec.retrieval_intent` resolves to effective `STANDARD`, explicit non-`UNSPECIFIED` `access_intent` on `object`/`nested` containers is rejected, and schema admission fails closed if requested `HIGH_FIDELITY` cannot be realized under the active service/profile policy. - Control schema-validation tests must verify nullable transition monotonicity: `false -> true` may be additive, while `true -> false` is classified as `SCHEMA_UPDATE_POLICY_BREAKING_REBUILD`. - Control schema-validation tests must verify `FieldCapabilitySpec.multi_valued` evolution policy is rebuild-class in both directions (`false -> true` and `true -> false` both require `SCHEMA_UPDATE_POLICY_BREAKING_REBUILD`). - Control schema-validation tests must verify `FieldSchema.options` / `CollectionSettings.options` only admit `MetadataValue` (`atomic_value` / `list_value`) and reject any attempt to encode vector payload semantics through options metadata. - Control schema-validation tests must verify nested ecosystem settings are flattened to dot-notation keys before `CollectionSettings.options` emission; unflattened nested-shape payloads are rejected as `SF_SCHEMA_VIOLATION`. - Control schema-validation tests must verify `FieldSchema.name` and `FieldSubfieldSpec.name` reject `.`, and query-time field paths follow deterministic container traversal + optional final subfield suffix resolution rules. - Control schema-validation tests must verify `FieldCapabilitySpec.multi_valued` cardinality semantics are enforced deterministically (`false` => reject list payloads for that field; `true` => admit homogeneous list payloads with schema-castable atomic element typing). - Control catalog-read tests must verify `GetCollectionSchema` returns the authoritative applied schema snapshot (`CollectionSchemaDescriptor`) with schema lineage, shared schema body (`definition`), and resolved `effective_max_result_window`. - Control readiness tests must verify `GetReadiness.success.applied_schema_version` / `applied_schema_generation_token` report the resolved query-serving latest `READY` generation rather than a newer active-but-not-`READY` generation, that tombstoned / never-existent / fully GC-released collections fail closed as `SF_RESOURCE_NOT_FOUND`, that active collections with no admitted schema lineage fail closed with canonical schema-bootstrap details, and that active collections with an admitted lineage but no `READY` generation fail closed with canonical schema-convergence details. - Control collection-catalog tests must verify `ListCollections` deterministic pagination, prefix filtering, stable ordering over canonical NFC/binary ordering, `CollectionDescriptor` field presence/typing, and omission of internal-lifecycle-`TOMBSTONED` collections. - Control collection-catalog tests must verify `ListCollections.next_page_token` participates in the unified token model with the same signing, `key_id`, TTL, and rotation policy as query page/search-context tokens, remains opaque/bound to `rpc_method`, server-authenticated caller identity, admitted `client_id`, admitted `client_kind`, `principal_subject` when present, `namespace_id`, literal prefix after canonical NFC validation, effective page size, and last-emitted canonical sort key, fails closed as `SF_BAD_REQUEST` on malformed, list-contract-mismatched, signature-invalid, expired, or request-field-drifted pagination tokens, and fails closed as `SF_FORBIDDEN` on caller-identity (`server-authenticated caller identity`, `client_id`, `client_kind`, or `principal_subject` when present) or authorization mismatch. - Control collection-catalog tests must verify non-empty `ListCollections.success.next_page.token` always carries matching `next_page.expires_at_unix_ms`, and empty `next_page.token` forces `next_page.expires_at_unix_ms=0`. - Control collection-catalog tests must verify multi-page `ListCollections` traversal remains keyset-based rather than snapshot-pinned: each page is individually authoritative, and page tokens do not bind a catalog snapshot revision. - Control collection-metadata tests must verify `GetCollectionMetadata` returns one authoritative `CollectionDescriptor` for both `CollectionRef` and `CollectionNameBinding` lookup paths, fails closed as `SF_RESOURCE_NOT_FOUND` for internal-lifecycle-`TOMBSTONED` collections, that `CollectionDescriptor.completeness` sets exactly one branch (`ready` or non-empty `publishing.missing_fields[]`), that present `CollectionDescriptor.index` sets exactly one branch (`up_to_date` or `updating`), that successful descriptors always carry `encryption` independently of completeness state, that absent `approx_stats` forces `completeness.publishing` with `publishing.missing_fields[]` including `approx_stats`, that absent `index` forces `completeness.publishing` with `publishing.missing_fields[]` including `index`, and that `PutCollectionMetadata` deterministically returns exactly one of `created` / `updated` / `already_applied` with stable `CollectionRef` assignment. - Control collection-metadata validation tests must verify empty `CollectionLookupKey{}` rejects as `SF_INVALID_ARGUMENT` rather than drifting into `SF_BAD_REQUEST` or `SF_RESOURCE_NOT_FOUND`. - Control collection-metadata tests must verify metadata updates against an already resolved active binding serialize on the resolved `CollectionRef` lineage, and that concurrent tombstone/lifecycle conflicts are rejected deterministically as `SF_VERSION_CONFLICT` with the canonical lifecycle-conflict detail keys (`reservation_scope=resolved_collection_lineage`) rather than being acknowledged after the lineage changed. - Control collection-name-binding cache tests must verify namespace-scoped `collection_name_binding_revision` invalidates both negative misses and positive `CollectionNameBinding -> CollectionRef` cache entries across tombstone GC release and same-name recreation. - Control collection-metadata tests must verify `PutCollectionMetadata` rejects non-canonical `collection_name` input as `SF_BAD_REQUEST`, rejects create allocation into `QUIESCING` / `TOMBSTONED` namespaces as `SF_VERSION_CONFLICT` with canonical lifecycle-conflict details, and does not allocate a replacement collection while the same `(namespace_id, collection_name)` remains reserved by a tombstoned collection whose tombstone GC has not yet released the name. - Control collection-schema tests must verify `GetCollectionSchema` fails closed as `SF_RESOURCE_NOT_FOUND` for internal-lifecycle-`TOMBSTONED` collections rather than surfacing stale pre-drop schema snapshots. - Control namespace-mutation tests must verify `PutNamespaceMetadata` deterministic create/update/idempotent-replay behavior, success branches restricted to `created` / `updated` / `already_applied`, stable `namespace_id` allocation by `namespace_name`, conflicting-replay rejection under the same `(server-authenticated caller identity, client_id, rpc_method, namespace_name, idempotency_key)`, and `already_applied` on fresh-idempotency-key no-op commands whose requested state already matches authoritative namespace state. - Control namespace-mutation tests must verify `PutNamespaceMetadata` rejects non-canonical `namespace_name` input as `SF_BAD_REQUEST` and does not allocate a replacement namespace while the same `namespace_name` remains reserved by a tombstoned namespace whose lifecycle GC has not yet released the name, with the reservation conflict reported as `SF_VERSION_CONFLICT` plus canonical lifecycle-conflict details. - Control namespace-mutation tests must verify updates against an already allocated namespace serialize on the resolved `namespace_id` lineage together with lifecycle transitions, and that conflicting post-resolution lifecycle changes reject deterministically as `SF_VERSION_CONFLICT` with the canonical lifecycle-conflict detail keys (`reservation_scope=resolved_namespace_lineage`). - Control namespace-mutation tests must verify namespace provisioning/admin callers use explicit `RequestMeta.client_kind=CLIENT_KIND_CONTROL_ADMIN` rather than overloading gateway/TiDB caller kinds. - TiDB bootstrap-ownership tests must verify `COLLECTION_TRUTH_MODE_EXTERNAL` first-time identity bootstrap is TiDB-owned through `PutNamespaceMetadata` / `PutCollectionMetadata` with `meta.client_id=tidb-ddl-adapter`, `meta.client_kind=CLIENT_KIND_TIDB_DDL_ADAPTER`, replay-stable non-empty metadata `idempotency_key`s, and explicit create-path encryption descriptors, while `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` / federated-table TiDB callers fail closed on missing namespace/collection identity rather than implicitly creating it. - Control namespace-catalog tests must verify `ListNamespaces` deterministic pagination, prefix filtering, stable ordering over canonical NFC/binary ordering, and `NamespaceDescriptor` field presence/typing, including branch-presence `lifecycle` projection (`active` vs `quiescing`), `completeness` branch correctness, and synchronous zero-child publication (`approx_stats` zero aggregate plus `index=up_to_date`) both on initial create and on later transitions back to zero active child collections. - Control namespace-catalog tests must verify `GetNamespaceMetadata` accepts both `lookup.namespace_id` and `lookup.namespace_name`, returns the same authoritative `NamespaceDescriptor` on both lookup paths, rejects empty `NamespaceLookupKey{}` as `SF_INVALID_ARGUMENT`, and fails closed as `SF_RESOURCE_NOT_FOUND` for tombstoned / never-existent / fully GC-released namespace lineages. - Control namespace-catalog tests must verify `ListNamespaces.next_page_token` participates in the unified token model with the same signing, `key_id`, TTL, and rotation policy as query page/search-context tokens, remains opaque/bound to `rpc_method`, server-authenticated caller identity, admitted `client_id`, admitted `client_kind`, `principal_subject` when present, literal prefix after canonical NFC validation, effective page size, and last-emitted canonical sort key, fails closed as `SF_BAD_REQUEST` on malformed, list-contract-mismatched, signature-invalid, expired, or request-field-drifted pagination tokens, and fails closed as `SF_FORBIDDEN` on caller-identity (`server-authenticated caller identity`, `client_id`, `client_kind`, or `principal_subject` when present) or authorization mismatch. - Control namespace-catalog tests must verify non-empty `ListNamespaces.success.next_page.token` always carries matching `next_page.expires_at_unix_ms`, and empty `next_page.token` forces `next_page.expires_at_unix_ms=0`. - Control namespace-catalog tests must verify multi-page `ListNamespaces` traversal remains keyset-based rather than snapshot-pinned: each page is individually authoritative, and page tokens do not bind a catalog snapshot revision. - Control catalog-name tests must verify `namespace_name`, `collection_name`, and their prefix-filter counterparts admit only canonical NFC UTF-8 scalar sequences, enforce the 128-scalar-value maximum, reject control / bidi-control / default-ignorable format characters as `SF_BAD_REQUEST`, and never silently normalize on write. - Control namespace-metadata tests must verify `GetNamespaceMetadata` returns a single authoritative `NamespaceDescriptor`, that `NamespaceDescriptor.lifecycle` sets exactly one public lifecycle branch (`active` or `quiescing`) on success, that `NamespaceDescriptor.completeness` sets exactly one branch (`ready` or non-empty `publishing.missing_fields[]`), that present `NamespaceDescriptor.index` sets exactly one branch (`up_to_date` or `updating`), that successful descriptors always carry `encryption` independently of completeness state, and that absent `approx_stats` or absent `index` forces `completeness.publishing` with the corresponding missing-field entry, and that metadata-parity consumers fail closed with canonical completeness details when `publishing` is active. - Descriptor completeness canonicalization tests must verify `publishing.missing_fields[]` uses only `approx_stats` and `index`, never duplicates entries, returns them in canonical byte-ascending order, and that fail-closed `diagnostics["missing_descriptor_fields"]` is the comma-joined form of that same ordered list. - Descriptor completeness fail-closed tests must verify metadata-parity collection/namespace reads surface `ErrorStatus.context.descriptor_publishing_unavailable` first, with `descriptor_state="publishing"` and ordered `missing_descriptor_fields[]`, and treat `diagnostics["descriptor_state"]` / `diagnostics["missing_descriptor_fields"]` as compatibility-only mirrors. - Control schema-sync-status tests must verify `SchemaSyncStatusSuccessResponse.state` oneof exclusivity and branch semantics (`ready`, `applying`, non-empty `drifted.reason`, non-empty `error.reason`). - Cross-ecosystem schema equivalence tests must verify semantically equivalent ES mapping / Turbopuffer schema / MySQL FULLTEXT DDL payloads converge to equivalent canonical schema snapshots. - Turbopuffer applicability tests must verify direct Turbopuffer-native routes are admitted only for `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` and fail closed otherwise. - Elasticsearch external-ID tests must verify non-uint64/out-of-range wire tokens fail closed (`SF_INVALID_DOC_ID`) and are never normalized into canonical `DocumentID.id_value`. - MySQL/TiDB source doc-id normalization tests must verify non-`SCALAR_FIELD_TYPE_UINT64`-compatible or out-of-range source-field values fail closed as ingest-time `422 SF_SCHEMA_VIOLATION`, rather than wire-token `SF_INVALID_DOC_ID`. - DocID-mapping type tests must verify `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_STRING` or `SCALAR_FIELD_TYPE_BYTES` fails closed as `400 SF_DOC_ID_CONTRACT_VIOLATION`; unknown enum values still fail as `400 SF_INVALID_SCHEMA`. - DocID generation-policy tests must verify only `DOC_ID_GENERATION_POLICY_CALLER_REQUIRED` and `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL` are admitted, that `UNSPECIFIED` fails schema admission, that allocator-backed omission is rejected on caller-required collections, and that `COLLECTION_TRUTH_MODE_EXTERNAL` rejects `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`. - Omitted-id write-path tests must verify `DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL` permits allocator authority only on admitted create/insert-style routes for `COLLECTION_TRUTH_MODE_SEARCH_FABRIC`, lowers mixed explicit/omitted source batches into one ordered `AllocateIdsRequest.plan.items[]`, preserves explicit identity requirements for patch/delete/update-style routes, carries the returned `allocator_receipt` into the follow-on allocator-backed `MutationBatch`, and keeps `StreamMutations` identity-complete. - Allocator watermark tests must verify ordered mixed-mode plans execute sequentially against one shared watermark, that explicit create/insert reservation within the admitted forward-reservation window advances allocator watermark to `doc_id + 1` during allocator admission, that the allocator watermark is dense-burn so later fresh explicit create/insert requests for any skipped lower `doc_id < next_doc_id` fail as `SF_VERSION_CONFLICT` with `allocator_conflict_reason=below_watermark_burned_gap`, that over-window reservations fail as `SF_TOO_MANY_OPERATIONS` with `limit_scope=allocator_forward_reservation_window`, that identity-space exhaustion fails as `SF_TOO_MANY_OPERATIONS` with `limit_scope=allocator_identity_space_remaining`, and that failed allocator admissions do not advance watermark. - Realized-id response tests must verify routes that admit omitted identity return realized IDs in source-input order and preserve the distinction between caller-supplied and allocator-issued identity when the route exposes that detail. - Turbopuffer ID-parse tests must verify active-profile ID limitation: decimal-string and non-negative integer forms losslessly parseable as `DocumentID.id_value.u64` are accepted; negative integer, floating numeric, uuid, and non-decimal/non-lossless string branches are rejected (`SF_INVALID_DOC_ID`). - `[Rule V-101]` type-support tests must verify the current canonical contract exposes only `DocumentID.id_value.u64` and keeps protobuf field numbers `2` and `3` reserved/unreusable in this revision. - `[Rule V-102]` non-empty-ID tests must verify oneof-empty `DocumentID` is rejected as `400 SF_INVALID_ARGUMENT`, while `DocumentID.id_value.u64=0` is accepted. - Token security tests must verify signature, principal scope, TTL expiry, key-rotation overlap behavior, and deterministic canonical error selection for each token failure class. - Principal-subject tests must verify provided `RequestMeta.principal_subject` mismatches are rejected (`SF_FORBIDDEN`). - Client-identity tests must verify empty `RequestMeta.client_id` is rejected as malformed, `RequestMeta.client_id/client_kind` mismatches are rejected (`SF_FORBIDDEN`), `client_kind=CLIENT_KIND_UNSPECIFIED` is rejected for all callers, and admitted gateway/internal-service clients are always tagged with correct enum values. - Request-metadata validation tests must verify overlength `idempotency_key`, `correlation_id`, `client_id`, and `principal_subject` are rejected as `SF_BAD_REQUEST`, and that implementations never silently truncate those identity-bearing strings. - Ranking determinism tests must verify stable tie-break ordering for equal-score hits. - Score materialization tests must verify `QueryHit.score` optional semantics: `return_score=true` populates `score`; `return_score=false` leaves `score` unset (not `0.0` sentinel). - Hit-version tests must verify `QueryHit.version_commit_ts` is carried and stable for OCC-enabled read paths. - Warning typing tests must verify `QueryWarning.code` uses valid non-`WARNING_CODE_UNSPECIFIED` `WarningCode` values and rejects unknown enum branches by precedence. - Response-shape tests must verify unified success payload allows `hits`-only, `aggregations`-only, and `hits+aggregations` coexistence deterministically. - Response-shape tests must verify top-level columnar projection contract: `success.projected_field_names` is unique and ordered by admitted `fetch_spec.attribute_projection` + `fetch_spec.vector_projections` semantics, and every `success.hits[].projected_values` length/order exactly matches header. - Response-shape tests must verify highlight contract: `success.highlight_field_names` uniqueness/order stability, sparse `hits[].highlights`, in-range `field_index`, and snippet determinism under fixed tags. - Response-shape tests must verify top-hits self-contained header contract: each `aggregations[].top_hits.{global_hits|nested_hits}` branch carries local projection/highlight dictionaries and every branch-local hit aligns exactly to branch-local headers. - Response-shape tests must verify `success.total_hits` relation/value contract aligns with admitted `track_total_hits` mode. - Response-shape tests must verify named inner-hits dictionary contract: `success.inner_hits_headers` unique-by-name, and each `hits[].inner_hits[].hits[]` projection/highlight alignment with its matched header. - Response-shape tests must verify nested hit identity isolation (`InnerQueryHit.nested_physical_id` vs `QueryHit.doc_id`) and deterministic `parent_array_offset` semantics. - Unary query transport-split tests must verify transport `OK` returns `ExecuteQuerySuccessResponse` directly, while malformed/authz/catalog/runtime/query-execution failures use non-`OK` gRPC status plus canonical `ErrorStatus` rather than a public `QueryResponse` envelope. - Unary query error-context tests must verify canonical machine-readable failure context is carried in `ErrorStatus.context.query`, that visibility timeout branches use `ErrorStatus.context.query.visibility_timeout`, that query-serving schema/token drift uses `ErrorStatus.context.query.schema_version_conflict` with canonical enum-valued `conflict_source`, that statistics freshness misses use `ErrorStatus.context.query.catalog_unavailable.statistics_freshness`, and that callers are not required to inspect a public unary query error body. - Error envelope tests must verify `ErrorStatus.correlation_id` is always non-empty, echoes caller `RequestMeta.correlation_id` when provided, and otherwise is service-generated. - Error envelope tests must verify `ErrorStatus.code` uses the canonical protobuf `ErrorCode` enum value corresponding to the selected `SF_*` registry row, never emits `ERROR_CODE_UNSPECIFIED`, and treats any received public payload carrying `ERROR_CODE_UNSPECIFIED` as invalid contract drift. - aux-index covering-read tests must verify `success.hits` may be emitted when admitted `fetch_spec.attribute_projection` (and requested vector projections, if any) are fully coverable in Search Fabric and deterministic schema snapshot is preserved. - aux-index non-covering projection tests must verify fallback to TiDB path per Section `5.4` and forbid partial projected-hit emission. - Query canonicalization tests must verify deterministic retrieval-tree normalization (defaulted `boost=1.0`, canonical child ordering where source order is non-semantic, `RetrievalGroup.must_clauses[]` / `ShouldMatchSpec.clauses[]` placement, `GROUP_FUSION_STRATEGY_DIS_MAX` tie-breaker defaults/range validation, `ShouldMatchSpec.threshold` fallback semantics, and `percentage_ratio -> ceil(ratio * clause_count)` evaluation), deterministic `search_payload.global_filter` normalization, deterministic `VectorClause.local_filter` + `VectorClause.filter_strategy_hint` normalization, canonicalized `fetch_spec.vector_projections` ordering, and canonicalized `search_payload.routing.target_tiers[]` normalization (dedupe + enum-value ascending). - Query clause-identity tests must verify request-global leaf `clause_id` uniqueness across `TextClause` / `VectorClause` / `ScoringScalarClause` / `NestedClause` anywhere in the retrieval tree, and deterministic nested-tree isolation under `NestedQueryPayload.root_retrieval`. - Text scoring tests must verify `TextClause.similarity_override` changes per-clause ranking deterministically without mutating schema baseline defaults, and present-but-empty `similarity_override` (`SimilarityModel.model` oneof unset) is rejected as `SF_INVALID_ARGUMENT`. - Response-shape vector tests must verify projected vector-field requests reject any projected-hit path where expected vector columns are missing/invalid in aligned `projected_values[i].vector_value` entries (no partial vector omission). - Bucket-key typing tests must verify non-composite buckets use `Bucket.key` (AtomicValue) while composite buckets use `Bucket.composite_key.fields` only, and never return vector/list key carriers. - VectorValue payload tests must verify oneof-branch-only payload typing (`f16_vector` / `float_vector` / `int8_vector` / `binary_vector`), enforce branch-vs-schema compatibility (`F16` branch uses compact half payload bytes), reject empty payload branch, and keep deterministic projection encoding for `VectorProjectionSpec.format` intent. - ListValue payload tests must verify one-dimensional atomic array encoding/decoding stability (`ListValue.values[]` of `AtomicValue`), reject nested list/vector elements, enforce ingest gateway homogeneity checks (`mixed atomic types -> row rejected with SF_SCHEMA_VIOLATION`; `continue_on_error=true` keeps other rows and reports row failure through `MutationAck.success.item_outcomes[]`), enforce cardinality gate (`multi_valued=false` field rejects list payloads), and enforce undeclared-field behavior (`undeclared list field -> reject in strict mode, ignore in ignore mode`; schema-established list field accepts empty list and null-containing lists). - Object-array payload tests must verify structured object arrays use `FieldData.object_array_value` (not `ListValue`), preserve insertion order, reject non-object array elements by precedence, and reject depth `> 1` payloads (nested `object_array_value` inside `documents[].fields[*]`) with `SF_SCHEMA_VIOLATION`. - Predicate type-safety tests must verify predicate RHS carriers are atomic-only (`ComparisonPredicate.value`, `InListPredicate.values` as `ListValue`, `RangePredicate.from/to` as `AtomicValue`) and reject vector/list operands before execution; `PredicateNode` present with unset `node` oneof must reject as `SF_INVALID_ARGUMENT`; `LogicalAnd` / `LogicalOr` child-count `< 2` and missing `LogicalNot.child` must reject as `SF_INVALID_QUERY`; `RangePredicate` must require at least one bound, reject reversed/equal-exclusive intervals, and reject type-incompatible range bounds as `SF_INVALID_QUERY`; `InListPredicate.values.values[]` must be non-empty and homogeneous by atomic type; `NestedPredicate.child_filter` field references must use absolute canonical paths; `ComparisonPredicate` must accept only `COMPARE_OP_EQ`/`COMPARE_OP_NEQ` (ordering/range comparisons MUST use `RangePredicate`) and must fail closed on type-incompatible operands as `SF_INVALID_QUERY`; `ExistsPredicate`, `IsNullPredicate`, and `NestedPredicate` semantics must remain distinct. - Null-value typing tests must verify `AtomicValue.null_value` branch accepts only `NULL_VALUE_NULL`; `NULL_VALUE_UNSPECIFIED` in selected null branch is rejected as malformed. - Mutation ACK response-shape tests must verify `MutationAck.oneof outcome` never emits unset `outcome`, remains exclusive, and ensures `MutationAck.success` carries ACK cursor/watermarks/outcomes only on success branch. - Public response-shape tests must enforce the exact-one outcome/state/success-branch rules defined in Section `4.2` (`Outcome-presence and success-state rule for public response RPCs`) across snapshot/control/catalog, query/helper, and descriptor/status response families, rather than restating a second RPC inventory. - Snapshot/control/catalog success-state tests must verify `StartSnapshotSyncSuccessResponse` remains acceptance-only with no redundant status enum, that `SnapshotChunkAckSuccessPayload`, `CommitSnapshotSyncSuccessResponse`, `AbortSnapshotSyncSuccessResponse`, `PutCollectionSchemaSuccessResponse`, `DropCollectionSuccessResponse`, `PutCollectionMetadataSuccessResponse`, and `PutNamespaceMetadataSuccessResponse` each set exactly one success-state branch, that `SnapshotSyncProgressSuccessResponse.state` never emits `SNAPSHOT_SESSION_STATE_UNSPECIFIED`, and that `SchemaSyncStatusSuccessResponse.active_schema_update_policy` / `CollectionSchemaDescriptor.active_schema_update_policy` never emit `SCHEMA_UPDATE_POLICY_UNSPECIFIED`. - Typed exact-one carrier tests must verify the canonical proto keeps `sf_required_oneof` on request/schema/error-visible exact-one typed carriers whose semantics reject empty payloads, including identity/value carriers (`DocumentID.id_value`, `AtomicValue.value`, `VectorValue.data`, `FieldData.value`, `MetadataValue.value`), query/retrieval carriers (`PredicateNode.node`, `MatchThreshold.threshold_type`, `RetrievalNode.clause`, `SortKey.target`, `MetricAggregation.metric_type`, `AggregationSortKey.target`, `DateHistogramAggregation.interval`, `CompositeSource.strategy`, `TopHitsVectorProjectionIntent.intent`, `AggregationRequest.aggregation_type`, `BucketAggregation.strategy`), result/state carriers (`Bucket.key_type`, `BucketResult.result_shape`, `TopHitsResult.result_type`, `AggregationResponse.result`, `FieldStatistics.stats_type`), ingest/snapshot carriers (`RowMutation.operation`, `MutationEntry.entry`, `MutationItemOutcome.outcome`, `MutationAck.outcome`, `SnapshotUploadRequest.frame`, `SnapshotChunkAck.outcome`), and error/schema/helper carriers (`QueryVisibilityTimeoutContext.mode`, `QueryCatalogUnavailableContext.reason`, `FieldConfig.field_definition`, `AttributeProjectionSpec.intent`, `TrackTotalHitsSpec.tracking`, `RescoreSpec.strategy`, `SimilarityModel.model`, `EncryptionDescriptor.mode`, `CollectionLookupKey.target`, `NamespaceLookupKey.target`, and `AllocateIdPlanItem.item_kind`). - Versionless-contract tests must verify there are no runtime capability-handshake RPC dependencies for admission. - Ingest static-policy tests must verify retention-window configuration drives reconnect/replay behavior. - Ingest reconnect-policy tests must verify mandatory resume-token behavior and deterministic replay rejections. - Request-metadata symmetry tests must verify every public Query/Data/Control request family in Section `3.7` keeps mandatory `RequestMeta` machine-closed where proto3 presence can be expressed, including unary RPCs plus stream/session-init carriers such as `MutationBatch` and `SnapshotUploadInit`, rather than checking only readiness/progress/status subsets. - Request-meta idempotency tests must verify durable-replay RPCs (`PutCollectionMetadata`, `PutCollectionSchema`, `DropCollection`, `PutNamespaceMetadata`, `AllocateIds`, `StartSnapshotSync`, `StreamMutations`, `UploadSnapshotChunks`, `CommitSnapshotSync`, `AbortSnapshotSync`) enforce durable owner-defined `RequestMeta.idempotency_key` dedupe semantics and reject same dedupe key with conflicting canonical payload (`SF_IDEMPOTENCY_CONFLICT`). - Control namespace-lifecycle tests must verify a namespace cannot transition to internal lifecycle state `TOMBSTONED` until every child collection is already terminal-hidden and that namespace tombstoning never leaves publicly visible descendant collections behind. - Control namespace-lifecycle visibility tests must verify `QUIESCING` namespaces remain publicly listable/readable, that successful public `NamespaceDescriptor` payloads disclose lifecycle state through the `quiescing` branch, that corresponding readable active namespaces disclose lifecycle state through the `active` branch, and that `TOMBSTONED` namespaces remain omitted/not-found rather than surfacing a successful descriptor lifecycle. - Control namespace-stats tests must verify absent `NamespaceDescriptor.approx_stats` is treated only as temporary aggregate unavailability/publication lag, never as an intentional permanent omission contract, and always forces `NamespaceDescriptor.completeness.publishing` with `missing_fields[]` including `approx_stats`. - Drop-collection replay tests must verify transport replay and business replay both return the `already_dropped` success branch on successful duplicate drop intents, and that fresh requests against tombstoned vs never-existent/GC-released collections deterministically split into `already_dropped` vs `SF_RESOURCE_NOT_FOUND`. - Control collection-schema bootstrap tests must verify active collections with catalog identity but no admitted schema lineage cause both `GetCollectionSchema` and `GetSchemaSyncStatus` to fail closed as `SF_CATALOG_UNAVAILABLE`, not `SF_RESOURCE_NOT_FOUND`. - Control collection-schema bootstrap tests must verify that same bootstrap branch emits canonical details `catalog_unavailable_reason=schema_bootstrap` and `schema_bootstrap_state=no_active_schema_generation`. - Control serving-generation tests must verify `GetCollectionSchema` may expose an `APPLYING` active generation while query execution, `GetReadiness`, and `GetCollectionStatistics` remain pinned to the most recent `READY` generation, that query/readiness/statistics bootstrap without any admitted schema lineage fails closed as `SF_CATALOG_UNAVAILABLE` with `catalog_unavailable_reason=schema_bootstrap` and `schema_bootstrap_state=no_active_schema_generation`, that admitted-but-not-`READY` generations fail closed with `catalog_unavailable_reason=schema_convergence` and `schema_convergence_state=no_ready_serving_generation`, and that `expected_schema_generation_token` naming a newer active-but-not-`READY` generation fails closed as `SF_VERSION_CONFLICT`. - Control catalog-read miss tests must verify `GetCollectionMetadata`, `GetCollectionSchema`, `GetNamespaceMetadata`, and `GetSchemaSyncStatus` all fail closed as `SF_RESOURCE_NOT_FOUND` for never-existent or fully GC-released targets. - Idempotency-key presence tests must verify RPCs that require caller-supplied `RequestMeta.idempotency_key` (`PutCollectionMetadata`, `PutCollectionSchema`, `DropCollection`, `PutNamespaceMetadata`, `AllocateIds`, `StartSnapshotSync`, `StreamMutations`, `UploadSnapshotChunks`, `CommitSnapshotSync`, `AbortSnapshotSync`) reject empty values, and that helper/query/catalog-read RPCs (`ExecuteProjectedQueryOrFallback`, `ExecutePrimaryQuery`, `ExecuteQuery`, `StreamQuery`, `GetReadiness`, `GetCollectionStatistics`, `GetSnapshotSyncProgress`, `GetCollectionSchema`, `ListCollections`, `GetCollectionMetadata`, `ListNamespaces`, `GetNamespaceMetadata`, `GetSchemaSyncStatus`, `DebugDigestProbe`) allow omission without creating durable replay state. - Transport-deadline tests must verify RPC deadline propagation remains transport-level metadata/governance rather than canonical `RequestMeta` payload semantics. - Visibility-waiting tests must verify `STRICT` and `BOUNDED_STALENESS` use the remaining transport deadline budget as the upper bound for readiness waiting, keep rechecking fresh readiness/watermarks until the mode-specific admission condition succeeds or deadline expires, and then emit the canonical timeout/fallback branch with the required observability fields. - Enum-typing tests must verify rejection behavior for unknown/invalid `CompareOp`, `GroupFusionStrategy`, `SortOrder`, `WarningCode`, `BuiltinAggregationSort`, `NestedScoreMode`, `NullValue`, and `OverflowBehavior`, plus invalid predicate-tree oneof branch shapes; admitted branches carrying `COMPARE_OP_UNSPECIFIED`, `SORT_ORDER_UNSPECIFIED`, `BUILTIN_AGGREGATION_SORT_UNSPECIFIED`, `NESTED_SCORE_MODE_UNSPECIFIED`, `NULL_VALUE_UNSPECIFIED`, or `OVERFLOW_BEHAVIOR_UNSPECIFIED` must fail closed by canonical precedence. - Enum fail-closed matrix tests must send core request/action enums plus selected exact-one/schema enum elements with `*_UNSPECIFIED` and assert canonical rejection (`SF_BAD_REQUEST`, `SF_INVALID_QUERY`, or schema-validation precedence as applicable), including `CollectionTruthMode`, `ConsistencyMode`, `CompareOp`, `DurabilityPolicy`, `VisibilityPolicy`, `IngestMode`, `SchemaUpdatePolicy`, `ScalarFieldType`, `VectorElementType`, `DistanceMetric`, `AggregateFunction`, `BuiltinAggregationSort`, `SortOrder`, `NullValue`, `OverflowBehavior`, `ChecksumAlgorithm`, `HashAlgorithm`, `DigestOperationKind`, `PayloadEncoding`, and `CompressionCodec`; snapshot-path request enums (`SnapshotChunk.checksum_algorithm`, `SnapshotChunk.payload_encoding`, `SnapshotChunk.compression_codec`, `CommitSnapshotSyncRequest.expected_root_hash_algorithm`) MUST remain machine-closed in the canonical proto rather than validation-by-comment only; explicit deterministic-exception branches (`MutationBatch.policy.atomicity`, `MutationBatch.policy.undeclared_field`, `RetrievalGroup.fusion_strategy`, `CompositeSource.missing_bucket`) and config-enum inherit branches (`VectorOptimizationGoal`, `VectorQuantization`, `TextPostingsOptions`, `SearchBehaviorDefaults.default_mode`) must follow documented fallback semantics. - Routing-tier enum tests must verify `search_payload.routing.target_tiers[]` carrying `DATA_TIER_UNSPECIFIED` is rejected with `SF_INVALID_QUERY`. - Lifecycle-tier enum tests must verify `CollectionSettings.lifecycle_policy.initial_tier=DATA_TIER_UNSPECIFIED` and `CollectionSettings.lifecycle_policy.transitions[].target_tier=DATA_TIER_UNSPECIFIED` are both rejected with `SF_SCHEMA_VIOLATION`. - Read-consistency parameter tests must verify mode-coupled constraints are deterministic: STRICT requires `read_ts>0` and `max_staleness_ms=0`; BOUNDED_STALENESS requires `read_ts>0` and `max_staleness_ms>0`; BEST_EFFORT requires `read_ts=0` and `max_staleness_ms=0`. - Ingest/control admission-order tests must verify first-failure code determinism under multi-violation requests. - Digest/fingerprint conformance tests must verify cross-language stability for canonical serialization + `HASH_ALGORITHM_BLAKE3`. - Digest coverage tests must verify changes in projection/aggregation/candidate-topk/sort/nested-clause/inner-hits controls alter `canonical_query_digest` deterministically, while cursor/transport controls (`CompositeAggregation.after_key`, `fetch_spec.limit`, `fetch_spec.page_window`, `fetch_spec.composite_page.next_page_token`, `fetch_spec.search_context`) do not. - Query continuation-token tests must verify `next_page.token` is self-contained, binds the immutable snapshot tuple plus the last-emitted canonical sort cursor, and always carries matching `next_page.expires_at_unix_ms` when non-empty, so continuation remains deterministic when `fetch_spec.limit` changes and does not require hidden server-side session state. - StreamQuery conformance tests must verify `StreamQueryRequest` supports both first-page (offset-free `first_page`) and resume (`continuation_page.next_page_token` non-empty) modes, frame ordering, per-frame resume-token emission semantics, dual limit semantics (`limit>0` bounded, `limit=0` unbounded), first-hit-bearing-success-frame-only dictionary emission (`projected_field_names` / `highlight_field_names` / `inner_hits_headers`) with zero-hit frames preserving dictionary-emission eligibility, resume attempts re-emitting dictionaries on their own first hit-bearing frame, and client-side reuse for later hit-bearing frames in the same RPC attempt, non-terminal success frames always carrying one or more hits with the `ok` branch plus a non-empty token, terminal hit-bearing success frames also carrying a non-empty token, terminal zero-hit success being reserved for final no-more-hits completion with an empty token, reconnect using the most recently received non-empty token, terminal-only `warnings[]` / `total_hits`, pre-first-frame fatal failures using non-`OK` gRPC status plus canonical `ErrorStatus`, and post-frame terminal failures using in-band `StreamQueryFrame.outcome=error`. - StreamQuery receipt-reconciliation tests must verify the internal broker stream emits one final `QueryBrokerStreamEvent.terminal_receipt` / `QueryBrokerStreamExecutionReceipt` for every broker-classified terminal attempt (`ok`, `partial`, terminal in-band `error`, internal-only `cancelled`, and pre-first-frame fatal failure), that post-pinning attempts use the full `receipt` branch carrying `QueryBrokerExecutionReceipt` while fatal attempts classified before `PinnedSnapshot` exists use the explicit `early_failure` branch, that `stream_attempt_id.value` remains the sole idempotent join key, and that missing/late terminal receipts trigger deterministic fallback exactly once without reopening reconciliation state. - StreamQuery receipt-identity tests must verify the canonical/internal proto keeps machine-readable required-and-non-empty validation on `StreamAttemptId.value`, with both `QueryBrokerStreamRequest.stream_attempt_id` and `QueryBrokerStreamExecutionReceipt.stream_attempt_id` using that dedicated carrier, and that empty `StreamAttemptId.value` values are rejected before any receipt dedupe or timeout reconciliation logic runs. - StreamQuery pagination-mode tests must verify `StreamQueryRequest.fetch_spec.page_window` is mandatory, that unset mode is rejected as malformed (`SF_BAD_REQUEST`), and that exactly one branch (`first_page` or `continuation_page`) is accepted. - StreamQuery shape tests must verify the stream request/response surface is hit-streaming-only: `StreamSearchPayload` does not expose `aggregations`, `StreamQueryRequest.fetch_spec` does not expose unary-only PIT/search-context or composite-page controls, and stream pagination exposes `next_page` only with no response-side search-context carrier. - StreamQuery rejection tests must verify any attempt to route aggregation-bearing queries onto `StreamQuery` fails closed as `SF_UNSUPPORTED_QUERY_FEATURE`. - Routing-pushdown tests must verify `search_payload.routing` canonicalization (`routing_keys` uniqueness/order), partition-field-only predicate admission, and deterministic pruning behavior. - Rescore tests must verify `fetch_spec.rescore.window_size` bounds, cross-encoder top-k validation, present-but-empty `RescoreSpec{}` (`strategy` oneof unset) malformed-branch rejection (`SF_INVALID_ARGUMENT`), deterministic tie-break preservation, and profile-gated reject behavior (`SF_UNSUPPORTED_OPTION`) where rescore is not enabled. - Control-plane retry tests must verify `PutCollectionMetadata` / `PutNamespaceMetadata` / `PutCollectionSchema` / `DropCollection` idempotent replay and conflicting-replay rejection semantics, including `RequestMeta.idempotency_key` network dedupe behavior for metadata mutations and drop branches with/without `fence`. - Control-plane retry tests must verify `PutCollectionMetadata` and `PutNamespaceMetadata` are transport-replay-only: same transport replay key replays the original success payload, while a different `idempotency_key` on the same lineage is treated as an ordinary new metadata command rather than business replay. - Control-plane retry tests must verify metadata-mutation replay returns the original success payload verbatim, not merely the original status enum. - Control-plane retry tests must verify metadata mutations classify fresh-idempotency-key state-equivalent commands as `already_applied` rather than `updated`, so success-status choice depends on authoritative post-apply state instead of replay-key novelty. - Control-plane retry tests must verify `PutCollectionSchema` transport replay identity (`idempotency_key`) and business replay identity (`collection + schema_version + fence token`) compose deterministically under both same-request and different-request duplicate scenarios. - Aggregation uncertainty tests must verify non-composite `BucketResult.non_composite.sum_other_doc_count` remains best-effort with truncation uncertainty surfaced via `AggregationTruncationWarning`/`BucketResult.non_composite.doc_count_error_upper_bound`, while composite branches use `BucketResult.composite.after_key.fields` for continuation and do not expose non-composite uncertainty carriers. - Error code precedence must match `docs/specs/core/error-codes.md`. - Unsupported complex Elasticsearch-query tests must verify machine-readable detail keys follow `docs/specs/core/error-codes.md` detail-key registry for `SF_UNSUPPORTED_QUERY_FEATURE`/`SF_UNSUPPORTED_OPTION` branches. - Gateway REST-contract tests must verify HTTP status mapping, route-specific error-body contract (canonical envelope vs Elasticsearch error JSON vs Turbopuffer OpenAPI `ErrorResponse`), deterministic canonical-correlation header projection on compatibility routes (`X-SF-Code`, `X-SF-Correlation-Id`), and required 429 headers (`Retry-After`, `X-RateLimit-*`, `X-SF-Throttled-Dimension`). - Gateway REST-contract tests must verify `X-SF-Code` uses the exact `ErrorCode` symbolic identifier string rather than numeric enum projection or implementation-local formatting. - Internal debug-digest tests must verify `InternalDebugService.DebugDigestProbe` is high-priv/internal-only, returns `gateway_normalized_digest` + `service_attested_digest` with stable `HASH_ALGORITHM_BLAKE3` labeling, never emits `HASH_ALGORITHM_UNSPECIFIED` on success, and keeps deterministic mismatch handling (`409 SF_DEBUG_DIGEST_MISMATCH`). - Internal debug caller-payload tests must verify `caller_canonical_payload` branches are deterministic: omitted branch, structurally invalid branch (`SF_BAD_REQUEST`), valid-but-different branch with digest mismatch classification still based on gateway-vs-service digest pair (`SF_DEBUG_DIGEST_MISMATCH`), and success-branch optional caller-vs-gateway `normalization_diffs[]` reporting. Implementation-level executable test cases (ES Profile-2): 1. `ES_COUNT_BASIC`: Setup index `idx` with 3 docs, 2 matching `query.term`; Action: `POST /idx/_count`; Assert canonical lowering uses `search_payload.aggregations=[{name:\"count\", metric:{builtin_function:AGGREGATE_FUNCTION_COUNT}}]`, response emits `success.aggregations` tree with deterministic count result and keeps `hits` empty; Response returns `count=2` and stable shard summary; Consistency: repeating with same `read_ts` yields identical count and digest. 2. `ES_COUNT_PATHLESS_REJECT`: Action: `POST /_count`; Assert `400 SF_INDEX_TARGET_REQUIRED` and no canonical request emission. 3. `ES_SINGLE_UPDATE_PATCH_CREATE_NOOP`: Setup doc `42` with initial state and one absent doc `43`; Actions: (a) `POST /idx/_update/42` with `doc`, (b) `POST /idx/_update/43` with `doc` + `doc_as_upsert=true`, (c) repeat the same `doc` on `42` with `detect_noop=true`; Assert gateway performs deterministic point-read/evaluate/canonicalize, lowers patch branch to one `RowMutation.operation.patch` with compare-and-set `expected_commit_ts`, lowers absent-doc branch to `RowMutation.operation.upsert` with `expected_commit_ts=0`, and reconstructs `result=updated|created|noop` exactly. 4. `ES_BULK_UPDATE_SUBSET`: Setup docs `10` and `11`; Action: `POST /idx/_bulk` with one `update`/`doc` item, one `update`/`doc_as_upsert` item, one noop update, and one later duplicate-target `update` item; Assert bulk lowering uses `policy.continue_on_error=true`, admitted update items reuse the single-document update canonicalization rules, noop item emits no canonical mutation but reconstructs `result=noop`, and later duplicate-target update is emitted as deterministic item-level reject. 5. `ES_UPDATE_BY_QUERY_BASIC`: Setup docs with `status=old`; Action: `POST /idx/_update_by_query` using allowed assignment script (`ctx._source.status=params.status`); Assert gateway compiles deterministic `doc_id asc` mutation plan, response `updated=2`, `version_conflicts=0`; Consistency: subsequent `GET` at `effective_read_ts` shows `status=new` for updated docs only. 6. `ES_UPDATE_BY_QUERY_UNSUPPORTED_SCRIPT_REJECT`: Action: `POST /idx/_update_by_query` with unsupported script feature (for example non-assignment script or unsupported option); Assert `400 SF_UNSUPPORTED_QUERY_FEATURE`, and no canonical mutations executed. 7. `ES_DELETE_BY_QUERY_BASIC`: Setup docs with filter matching 2 rows; Action: `POST /idx/_delete_by_query` (allowed `conflicts=abort|proceed` subset); Assert deterministic `doc_id asc` delete plan, response `deleted=2`, `version_conflicts=0`; Consistency: `GET` returns `404` for deleted docs and `_count` reflects new total. 8. `ES_TERMS_BUCKET_WITH_SUB_METRIC`: Setup docs grouped by `brand`; Action: gateway lowers terms aggregation with nested metric (`avg_price`); Assert canonical request uses `AggregationRequest{name:\"brands\", bucket:{terms{field:\"brand\",size:N}, sub_aggregations:[{name:\"avg_price\",metric{builtin_function=AGGREGATE_FUNCTION_AVG,field:\"price\"}}]}}` and response bucket order/metric values are deterministic. 9. `ES_TERMS_BUCKET_WITH_TOP_HITS`: Setup docs grouped by `category`; Action: gateway lowers terms aggregation with nested top-hits (`size=3`, sort by score then `doc_id`); Assert each returned bucket includes deterministic `TopHitsResult.result_type` branch (`global_hits` or `nested_hits`) with aligned projection and score behavior without perturbing outer bucket counts. 10. `ES_COMPOSITE_AFTER_KEY_PAGINATION`: Setup docs grouped by multi-dimension keys (`date_histogram` + `terms`); Action: first request lowers to `CompositeAggregation.sources[]` with `limit=N`, follow-up request sends response `after_key`; Assert gateway lowers source `after` object to typed `CompositeAggregation.after_key`, forces `fetch_spec.page_window=composite_page`, Query Broker emits deterministic comparator order, any response with non-empty `BucketResult.composite.after_key` also returns `pagination.composite_page.token` + expiry, optionally returns `pagination.search_context.token` only when keep-alive was admitted, response raises `BucketResult.composite.after_key.fields`, and continuation requires the matching `composite_page` token whose bound cursor equals the request `after_key`. 11. `ES_NESTED_QUERY_WITH_INNER_HITS`: Setup docs with nested array field `comments[]`; Action: gateway lowers source nested query + `inner_hits` to a `NestedClause` leaf inside `search_payload.root_retrieval` with `InnerHitsSpec` (`limit`, `offset`, `track_total_hits`, sort); Assert response emits `success.inner_hits_headers[]` + parent `hits[].inner_hits[]` alignment, enforces per-parent shallow paging, and keeps nested identity isolation (`nested_physical_id`, `parent_array_offset`). 12. `ES_VALUE_COUNT_METRIC`: Setup docs where field `price` is present on a subset and null/missing on others; Action: `POST /idx/_search` with `aggs.price_count.value_count.field=\"price\"`; Assert gateway lowers to canonical `MetricAggregation{builtin_function=AGGREGATE_FUNCTION_COUNT, field:\"price\"}`, response count equals rows with non-null `price`, and missing/empty source `field` branch rejects as `400 SF_BAD_REQUEST`. ### 9.4 Quantified Compatibility Roadmap (Profile 2 -> Profile 3) Near-term objective (`Profile 2 Hardening`): - positioning: high-determinism subset compatibility, not full-source parity. - exit metrics: - deterministic replay/pagination/hash conformance pass rate: `100%` on release gate corpus. - unsupported-feature rejection precision (correct `SF_*` + machine-readable detail keys): `>= 99.9%`. - ES/Turbopuffer gateway canonical equivalence drift: `0` release-blocking mismatches. Mid-term objective (`Profile 3 Expansion`): - priority 1: Elasticsearch deep paging parity uplift: - PIT-equivalent search context token + lifecycle (`open`, `keep_alive`, `close`) fully conformance-tested. - `search_after + PIT-equivalent` continuity under refresh/rolling upgrade fault injection with `0` determinism breaks. - priority 2: Elasticsearch bulk parity uplift: - canonical `continue_on_error + item_outcomes` path becomes mandatory for `/_bulk`. - per-item status reconstruction accuracy: `100%` against deterministic replay corpus. - priority 3: Turbopuffer write-surface parity uplift: - `patch_by_filter`, `delete_by_filter`, `copy_from_namespace`, `return_affected_ids`, `disable_backpressure -> ingest_mode` all covered by gateway-to-canonical conformance tests. - response-shape parity for affected-ID and partial-completion branches: `100%` on golden corpus. - priority 4: two-stage rerank uplift: - promote `fetch_spec.rescore` from reserved contract to production-gated capability with bounded `window_size` execution and cross-encoder integration. - validate deterministic ordering stability (rerank-score ties still end with `doc_id asc`) under replay and mixed-gateway payloads. - priority 5: partition-pruning uplift: - make `search_payload.routing` mandatory for partition-aware workloads and enforce early object-range pruning before segment reads. - conformance corpus MUST include routing-key skew and partition-filter correctness under large time-window queries. - priority 6: TiCDC high-lag degradation uplift: - keep controlled offline-import support for importer/Lightning jobs behind the canonical snapshot-sync RPC lifecycle only (`StartSnapshotSync` -> `UploadSnapshotChunks` -> `CommitSnapshotSync`). - allow immutable artifact staging and fenced manifest CAS publish only as internal execution details owned by the admitted snapshot session; do not introduce a second public ingest protocol. - keep lane phase-gated with strict ownership checks, replay-safe job idempotency, and deterministic rollback back to online TiCDC path. Long-term objective (`Profile 3+ Continuous Parity`): - evolve compatibility testing from documentation-driven suites to spec-driven generation: - Turbopuffer: OpenAPI-driven request/response conformance case generation from `docs/turbopuffer/openapi.yml`. - Elasticsearch: API-spec/docs-driven endpoint/option matrix generation from `docs/elasticsearch/*` and compatibility-profile manifests. - target: automated spec-diff detection and conformance-case regeneration in CI, with fail-closed rollout when upstream spec drift is unaccounted for. ### 9.5 Conformance Automation Direction - CI pipeline SHOULD include: 1. source spec ingestion (`openapi.yml` / ES compatibility manifest), 2. generated canonical-lowering cases, 3. deterministic replay/hash/token validation runs, 4. profile-scoped parity scorecards. - release gate SHOULD reject promotion when generated conformance delta is non-empty and not explicitly approved in profile scope docs. ## 10. Summary This review-hardened API design preserves decoupling and improves implementation certainty by making: - stream and ACK semantics explicit, - source identity/epoch semantics explicit for failover-safe replay, - cross-partition mutation ordering and mixed-entry framing rules explicit, - reconnect and pagination determinism explicit, - snapshot start idempotency and session retry behavior explicit, - snapshot upload stream-init, resume-token loop, and explicit EOF completion frame (`SnapshotUploadComplete`) are explicit and reconnect-safe, - reconnect cursor/token coherence and deterministic mandatory-token reconnect semantics explicit, - response-shape and pagination-depth governance explicit, - query fetch-window contract explicit (`fetch_spec.limit` is global; exactly one page-window branch `first_page`/`continuation_page`/`composite_page` is required; unary `limit=0` is valid only for aggregation/total-hit-only intents and StreamQuery `limit=0` remains unbounded), - hot-path query option/weight carriers moved from maps to deterministic repeated entries, - vector clause local-filter semantics renamed (`local_filter`) and explicit filter-strategy hinting (`filter_strategy_hint`) added with deterministic correctness guardrails, - streaming/state carriers that still need in-envelope alternatives now use exact-one result carriers, while helper/query/control-read RPCs use the current direct-success transport split and `ProjectedQueryResponse` keeps exact-one `success|fallback` normal-result semantics, - aux-index covering-read path explicitly allowed to emit `success.hits` with admitted `fetch_spec.attribute_projection` (and requested vector projections) when projection is fully coverable in Search Fabric, - canonical Row.vector return strategy via columnar projection payload (`projected_field_names` + aligned `projected_values[i].vector_value`) and projection-field digest binding explicit, with Turbopuffer active-profile restriction to the single admitted vector field named `vector`, - vector payloads are now wrapped by `VectorValue` (typed `data` oneof only, including compact `f16_vector` branch) for exact/quantized deterministic transport, - write-path optimistic precondition is explicit via `RowMutation.expected_commit_ts` for compare-and-set conflict control, - query vector-projection policy gating and deterministic admission explicit, - query continuation token error-class mapping explicit (`SF_BAD_REQUEST` vs `SF_FORBIDDEN` vs `SF_INVALID_QUERY`), - multi-ecosystem scope explicit for TiDB, Elasticsearch Query Gateway, and Turbopuffer Query Gateway clients, - control-plane schema ownership boundaries explicit for gateway-managed Search-Fabric-truth collections, - request metadata includes explicit typed client identity (`client_id` + enum `client_kind`) for multi-gateway and internal service auditability, - compatibility-profile applicability boundaries explicit across MySQL/Elasticsearch/Turbopuffer, - dual-layer compatibility commitments explicit (`Wire Parity @ Gateway` vs `Canonical gRPC Contract @ Search Fabric`), - cross-ecosystem gateway normalization (refresh/search_after/_count/multi_match/update_by_query/delete_by_query/multiQuery + Turbopuffer vector/projection normalization + deterministic ID-token parse policy) explicit and auditable, - Elasticsearch composite-aggregation pagination (`after_key`) now has canonical typed contracts (`CompositeAggregation`, `Bucket.composite_key.fields`, `BucketResult.composite.after_key.fields`) with snapshot-bound continuation guarantees, - parser compatibility ownership moved to schema/control plane for cleaner Query API boundary, - gateway pre-admission subset enforcement explicit for msearch header-index and MySQL `MATCH + ROLLUP` restrictions, - Turbopuffer consistency-level mapping and billing-envelope opacity boundary explicit, - Elasticsearch/MySQL/Turbopuffer active-profile ID contract unified to lossless uint64 parse-only admission (Turbopuffer uuid/non-lossless branches explicitly deferred), - digest/fingerprint canonicalization rules explicit, - canonical query-digest field coverage explicit for all semantics-affecting query fields, - checksum/root-hash textual encoding rules explicit, - token security/TTL/key-rotation and deterministic fencing explicit, - schema mapping payloads structured, - canonical typed schema semantics now cover Elasticsearch mapping, Turbopuffer schema, and MySQL FULLTEXT schema/index contracts with deterministic round-trip rules, - supplemental schema metadata carriers (`FieldSchema.options`, `CollectionSettings.options`) are strongly typed via `MetadataValue` (`AtomicValue`/`ListValue`) and keep vectors out of config metadata path, - `TextAnalysisSpec` uses single `analyzer` baseline with inline typed `options` map (`map`) under explicit key registry, `MetadataValue.atomic_value.string_value` value grammar, canonical normalization, static-policy gating, and unified error-code branches, - `FieldSchema` now separates field identity/topology from physical definition via `FieldConfig` (`config.capabilities` + oneof `config.field_definition` branches: `scalar_type` / `text` / `vector` / `keyword` / `object` / `nested`) plus `subfields[].config`, - `FieldCapabilitySpec` now distinguishes `filterable` (exact boolean logic), `searchable` (retrieval/scoring), `pattern_matchable` semantics, and explicit field cardinality (`multi_valued`) for scalar-vs-array round-trip fidelity, - keyword overlong-value protection is now strongly isolated under `FieldSchema.config.field_definition.keyword.overflow_protection` (instead of global capabilities), - full-text schema now uses layered `FullTextSpec` (`storage_tuning`, `default_similarity`, `default_search_behaviors`) to separate physical rebuild contracts from logical scoring/query defaults, with mode-scoped behavior defaults (`default_mode`, `natural_language`, `boolean_match`, `plain_match`), - `VectorTypeSpec` now separates base vector semantics from retrieval tuning via `VectorSearchTuningSpec` (`optimization_goal`, `quantization`) instead of a single `ann_enabled` boolean, - `IndexSortSpec` is simplified to a fields-only declaration (no redundant `enabled` toggle), - ingest request/ack payloads are now domain-split (`IngestPolicy`, `StreamCursor`, `IngestMetrics`, `EngineWatermarks`, `RejectionContext`) to reduce flat-field coupling and improve reuse, - mutation ingest now supports deterministic undeclared-field handling hinting via `MutationBatch.policy.undeclared_field` (`REJECT`/`IGNORE`) with strict default and first-error cursor semantics, - query success pagination is now a cohesive `PageCursor` payload (`next_page.{token,expires_at_unix_ms}`) instead of scattered fields, - collection-level `max_result_window` is explicit in canonical schema and enforced with machine-readable query error details, - PIT-equivalent deep paging context (`search_context.token`) and natural-expiry lifecycle semantics are explicit for long pagination chains, - canonical ingest supports deterministic non-atomic bulk parity via `policy.continue_on_error` + `MutationAck.success.item_outcomes[]`, - Turbopuffer write-surface lowering rules (patch/filter-copy/affected-ids/`disable_backpressure -> ingest_mode`) are explicit at gateway boundary, - query fetch now supports typed highlight controls via `ResultFetchSpec.text_options` and engine-side snippet output (`QuerySuccessPayload.highlight_field_names` + sparse `QueryHit.highlights`) with deterministic tag defaults and digest coherence, - query fetch now supports Elasticsearch-aligned total-hit semantics via `ResultFetchSpec.track_total_hits` and `QuerySuccessPayload.total_hits` (`TOTAL_HITS_RELATION_EQUAL_TO` / `TOTAL_HITS_RELATION_GREATER_THAN_OR_EQUAL_TO`), - bool-sentinel oneof anti-patterns are removed (`MutationItemOutcome.accepted/replayed`, `SchemaSyncStatusSuccessResponse.ready/applying` use branch-presence semantics via empty messages), - proto3 presence is restored where needed (`SnapshotUploadInit.resume_from_chunk_seq`, `SearchContextSpec.keep_alive_ms` are `optional`), - `RequestMeta.idempotency_key` / `RequestMeta.correlation_id` split is explicit: durable replay RPCs use non-empty caller-supplied idempotency keys, read/helper RPCs may omit them, and correlation metadata is no longer overloaded as replay authority, - unsupported complex ES query rejections now require machine-readable details defined in `docs/specs/core/error-codes.md`, - gateway REST contract details are explicit (status mapping, route-specific error bodies, 429 headers, auth-mapping flow), - gateway diagnostics now includes internal-only/high-priv `InternalDebugService.DebugDigestProbe` parity surface with gateway-vs-service digest match validation, - query plane now exposes `RelationalQueryService` with route-specific request carriers (`ProjectedQueryRequest`, `PrimaryQueryRequest`) so projected-query overlay context is scoped only to the projected route, overlay state split is currently implementation-validated and conformance-backed rather than machine-closed by proto annotations, and `ExecuteProjectedQueryOrFallback` keeps explicit fallback semantics while `ExecutePrimaryQuery` stays success/error only, - query plane now exposes read-only collection statistics snapshot fetch (`GetCollectionStatistics`) with collection-fast-path, typed field/joint stats, explicit `collected_at_unix_ms` freshness observability, and explicit staleness budget for optimizer/CBO consumers on the current query-serving `latest READY` lineage, - projected vector return supports field-level typed payload-format policy (`fetch_spec.vector_projections`) with deterministic quantized-output, policy checks, and executable MRL truncation (`target_dimensions<=schema dimensions`), - vector query admission/execution supports MRL prefix truncation (`0 < Dq <= Dschema`), with deterministic prefix-aligned ANN/distance evaluation for `Dq < Dschema` and fail-closed overflow rejection (`Dq > Dschema`), - query payload now supports explicit partition-pruning hints (`search_payload.routing`) for routing-key / partition-filter pushdown, - query payload now supports one recursive retrieval tree (`search_payload.root_retrieval`) with array-based boolean grouping (`must_clauses[]` + `should_match`), local fusion, node-local boost, and recursive nested retrieval via `NestedClause` + `NestedQueryPayload`, - fetch payload now reserves two-stage rerank controls (`fetch_spec.rescore`) for Profile-3+ cross-encoder integration with bounded top-N window, - nested extraction now uses dedicated `InnerHitsSpec` (with per-parent `track_total_hits`) and response-side named headers/results (`inner_hits_headers` + `hits[].inner_hits`) for deterministic column alignment, - high-dimensional vector projection paths support deterministic dynamic payload/window tightening with explicit limit-scope error details, - gateway aggregation defenses (depth/node/fanout/bucket ceilings) and prominent truncation summary (`success.aggregation_truncation`) are explicit, - Query Broker/Reader composite execution contract is explicit (reader-side after-key pruning + broker-side k-way merge by lexicographic source comparator), - `ReadConsistencySpec` is read-topology only (`read_ts`/`mode`/`max_staleness_ms`) with no write-mode hint coupling, - `FieldData` now supports non-recursive `ListValue` (atomic-only elements) for one-dimensional multi-value fields/JSON arrays and dedicated `ObjectArrayValue` for structured object arrays, - `QueryHit.score` is optional, so `return_score=false` can be represented as field-unset instead of ambiguous `0.0`, - undeclared fields no longer trigger implicit schema evolution; callers must complete explicit control-plane schema mutation before retrying writes that introduce new fields, - snapshot encoding/compression/checksum scope explicit, - predicate/fusion/sort and recursive aggregation-tree contracts (`metric`/`bucket`/`nested`/`reverse_nested`/`top_hits`) typed and negotiable, with explicit range-bound inclusivity (`RangePredicate`, `RangeAggregation.Range`) and polymorphic top-hits result typing (`global_hits` vs `nested_hits`), - flow-control and static policy governance explicit across data/query/control planes, - reconnect/replay retention windows discoverable through service policy, - stream-query semantics and control-plane idempotent retry semantics explicit (`StreamQuery` is hit-streaming only with terminal-only totals/warnings, while control-plane retries use `RequestMeta.idempotency_key` for network dedupe and `fence` for business CAS on drop), - control-plane catalog read/write surface explicit (`ListCollections`, `GetCollectionMetadata`, `PutCollectionMetadata`, `GetCollectionSchema`, `PutNamespaceMetadata`, `ListNamespaces`, `GetNamespaceMetadata`) so compatibility readbacks, namespace provisioning, and Turbopuffer collection lifecycle no longer depend on vague catalog-snapshot projections, - projected vector-field completeness behavior (via aligned `projected_values[i].vector_value`) fail-closed and deterministic, - fallback/error behavior deterministic and auditable.