# Ingest Gateway (Flow B)

Date: 2026-03-13
Status: Frozen architecture note
Authors: Search Platform Architecture

## 1. Scope

This document defines the component-level architecture of `Ingest Gateway` for Flow B only: Search-Fabric-owned write traffic originating from Elasticsearch-compatible, Turbopuffer-compatible, and TiDB federated SQL remote-DML gateway surfaces.

The canonical public contract remains:
- `api/proto/searchfabric/v1/search_fabric.proto`
- `docs/specs/core/search-fabric-api.md`
- `docs/specs/core/error-codes.md`
- `docs/specs/core/search-fabric-architecture.md`

This document does not redefine public RPC semantics. It explains how Flow-B gateway instances should realize those semantics at production scale while remaining stateless and subordinate to catalog-driven ownership and schema validation.

Out of scope:
- Flow A (`TiCDC -> CDC Adapter -> Ingest Gateway`) normalization details
- public compatibility-route semantics already owned by `docs/compatibility/elasticsearch.md`, `docs/compatibility/turbopuffer.md`, and `docs/architecture/tidb-search-fabric-integration.md`
- downstream writer, commit, and visibility internals owned by `docs/specs/core/search-fabric-architecture.md`

## 2. Role and Boundary

Flow-B `Ingest Gateway` is the stateless ingress component between external native-search-compatible write APIs and the canonical ingest contract.

Its responsibilities are:
- terminate source wire protocols and consume authenticated caller context from `Unified Edge/Auth Ingress` for external traffic or trusted service identity for direct internal traffic
- resolve external route tokens into authoritative `CollectionRef`
- validate ownership, lifecycle, and active schema-generation fencing against `Catalog Control Service` (CCS)
- normalize admitted writes into canonical `MutationBatch`
- forward canonical batches into the shared ingest path and reconstruct source-specific success/error envelopes

Its non-responsibilities are:
- assigning canonical Flow-B `commit_ts`
- inventing ownership or schema state without CCS
- bypassing canonical idempotency or replay semantics with cache-local shortcuts

High-level Flow-B data path:

```text
External Native-Search Client
  -> Unified Edge/Auth Ingress
  -> ES / Turbopuffer Protocol Adapter inside Ingest Gateway
  -> authenticated context + tenant/account scope handoff
  -> CollectionNameBinding / CollectionRef resolution via CCS (+ local L1 admission cache)
  -> optional schema coordination / prewrite current-state and selector subroutines
  -> ownership + schema + payload admission
  -> canonical MutationBatch assembly
  -> Ingest Broker / StreamMutations
  -> ACK / error translation back to source wire shape
```

## 3. Design Rules

1. Canonical-contract first:
- all data-plane writes emitted by the gateway must conform to `MutationBatch`, `MutationEntry`, `RowMutation`, and `RequestMeta` as defined in `search_fabric.proto`
- lower-tier implementation details must not create parallel wire semantics

2. Zero-timestamp Flow B:
- every Flow-B `RowMutation.commit_ts` must be `0`
- only `Commit Coordinator` injects canonical TSO before persistence/apply ordering

3. Catalog-driven correctness:
- collection identity, authoritative `collection_truth_mode`, lifecycle state, and active schema generation are resolved from CCS
- cache entries are optimization hints only

4. Fail closed on metadata uncertainty:
- writes that cannot validate ownership or active schema against sufficiently fresh CCS state must fail as catalog/backend availability errors rather than speculatively proceeding

5. Retry-stable identity:
- gateway retries must preserve canonical replay identity (`RequestMeta.idempotency_key`, `batch_id`, `batch_fingerprint`, and Flow-B stream identity derivation)
- `RequestMeta.idempotency_key` is mandatory for `StreamMutations`, but it must not be reused as Flow-B data-plane `source_partition`

## 4. Request Lifecycle

### 4.1 Source Admission and Scope Resolution

For every Flow-B request:

1. For external traffic, `Unified Edge/Auth Ingress` authenticates the caller and resolves coarse tenant/account scope; for direct internal traffic, gateway accepts trusted service identity directly.
2. Gateway consumes that authenticated context, re-validates caller/scope coherence at the write-plane boundary, and maps the resolved external scope to the authoritative canonical `namespace_id`.
3. Source route tokens are lowered into typed catalog identity:
   - Elasticsearch `{index}` -> `CollectionNameBinding{namespace_id, collection_name}`
   - Turbopuffer route namespace token `{namespace}` -> `CollectionNameBinding{namespace_id, collection_name}`
4. Gateway resolves the authoritative `CollectionRef` through CCS:
   - `GetCollectionMetadata.lookup.name_binding` for existing collections
   - `PutCollectionMetadata.name_binding` only when the route semantics require ensuring collection identity before first schema or ingest admission

No downstream ingest or schema call may use a route token directly after this step; all subsequent calls must use the authoritative `CollectionRef`.

### 4.2 Optional Schema Coordination Side Path

Some Flow-B routes need control-plane coordination before mutation emission:
- first-write collection creation flows
- write-route collection-metadata ensure flows such as Turbopuffer `encryption`
- Elasticsearch `/{index}/_mapping`
- Turbopuffer schema writes

Design rules:
- Search-Fabric-truth collection metadata and schema mutation remain CCS control-plane operations; `Ingest Gateway` only orchestrates them
- first allocation through `PutCollectionMetadata.name_binding` MUST carry explicit `definition.encryption` with exactly one valid mode selected; if the selected mode is `cmek`, `key_name` must be non-empty
- when a route carries collection-level `encryption` and the collection already exists, the gateway must compare that value against the authoritative descriptor and fail closed on mismatch before any schema or row mutation is emitted
- before any gateway-managed `PutCollectionSchema`, the gateway must validate that the current authoritative truth mode is either unallocated (first schema) or already admitted for gateway-managed schema mutation (`collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`)
- gateway-managed schema apply must use `PutCollectionSchema.scope.ddl_commit_ts=0`
- first schema creation on an existing collection identity must use the canonical zero-state tuple:
  - `schema_version=1`
  - `fence.expected_prev_schema_version=0`
  - `fence.schema_generation_token="__initial__"`
  - `update_policy=SCHEMA_UPDATE_POLICY_INITIAL_CREATION`
  - `fence.enforce_monotonic_apply=true`
- when one outer request triggers metadata-ensure, schema-apply, and later row-mutation legs, all mutating control-plane RPCs must carry the same replay-stable outer `RequestMeta.idempotency_key`, `client_id`, and gateway `client_kind`
- undeclared fields do not trigger implicit schema creation in this revision; gateways must reject or ignore them according to the admitted `undeclared_field` policy and require explicit `PutCollectionSchema` before any retry that depends on new fields

This side path is not a second ingest protocol. It is a prerequisite control-plane step used only when the source route semantics require collection or schema state changes.

### 4.3 Internal Prewrite Read and Selector Lanes

Some admitted Flow-B write routes require internal read work before `MutationBatch` assembly:
- point-read routes:
  - Elasticsearch single-document `update`
  - Elasticsearch bulk `update` items
- selector/materialization routes:
  - Elasticsearch `_update_by_query`
  - Elasticsearch `_delete_by_query`
  - Turbopuffer `patch_by_filter`
  - Turbopuffer `delete_by_filter`

Design rules:
- this is an internal gateway subroutine, not a public query API reuse contract
- point-read/OCC routes must execute against a write-authoritative `Current-State Reader` rather than the stale-tolerant query plane, using the pinned internal carrier in `api/proto/searchfabric/v1/current_state_reader.proto`
- selector/materialization routes may execute against `Query Broker` snapshot semantics through the dedicated internal selector-materialization RPC using:
  - authoritative `CollectionRef`
  - active schema-generation fence
  - resolved `effective_max_result_window`
  - active schema/body fragments needed for type-aware route lowering
- point-read outputs must include deterministic current-row existence plus latest version marker used to derive `RowMutation.expected_commit_ts`
- selector outputs must include deterministic `(doc_id, selected_version_commit_ts)` ordered by `doc_id ASC`, plus the terminal `has_more_hits` / probe-limit signal needed for two-phase clamp enforcement
- existence-sensitive source response reconstruction for single-document `index` / `delete` must not depend on gateway-local prewrite classification; those routes use authoritative post-apply result kinds returned in ingest ACK item outcomes
- query-assisted write lanes must fail closed before mutation emission when:
  - point-read/OCC preconditions cannot be established
  - selector materialization hits the canonical probe clamp
  - route semantics require a lifecycle-policy carry field that is not present in the admitted patch payload
  - source `conflicts=abort|proceed` cannot be mapped onto the canonical batch execution mode

This subroutine owns read-before-write realism for canonical Flow-B routes; the later `MutationBatch` assembly stage only handles already-admitted mutation plans.

### 4.4 Canonical MutationBatch Assembly

Once the target collection and active schema generation are resolved, the gateway assembles one canonical `MutationBatch`.

Required fields:
- `meta`
- `batch_id`
- `collection`
- `policy`
- `batch_fingerprint`
- `entries[]`
- `stream_identity`
- `schema_generation_token`

Request metadata rules:
- `RequestMeta.idempotency_key` must be non-empty because `StreamMutations` is state-mutating
- `RequestMeta.client_id` must be non-empty
- `RequestMeta.client_id` must carry stable logical caller identity across retry/resume/pagination lifetimes; gateway instance identity belongs in tracing/telemetry rather than this field
- `RequestMeta.client_kind` must be explicit:
  - `CLIENT_KIND_ES_WRITE_GATEWAY` for Elasticsearch-compatible write routes
  - `CLIENT_KIND_TP_WRITE_GATEWAY` for Turbopuffer-compatible write routes
- `principal_subject`, when present, remains audit/authz context only

Flow-B stream identity rules:
- gateway must derive retry-stable `stream_identity.source_partition` from idempotency identity:
  - use the canonicalized client `Idempotency-Key` when the route/profile provides one
  - otherwise deterministically derive `batch_id` from the admitted payload hash and use that stable value, or a stable derivation of it, as `source_partition`
- `stream_identity.source_epoch` must be `1` for stateless Flow-B requests
- `MutationEntry.source_seq` must be the 1-based admitted item order in the batch
- gateway must not use `RequestMeta.idempotency_key` as `source_partition`

`batch_id` and `batch_fingerprint` rules:
- same full retry payload may reuse the same `batch_id`
- `batch_fingerprint` must follow the canonical BLAKE3 hashing contract over admitted mutation business content
- `RowMutation.commit_ts` and transport/envelope metadata remain excluded from fingerprint hashing
- if the client retries with a sliced payload after a first-error cursor, gateway must issue a new `batch_id`

### 4.5 Flow-B Normalization Rules

The gateway owns source-to-canonical lowering before `MutationBatch` emission.

Normalization requirements:
- external document IDs must be losslessly normalized to canonical unsigned `uint64`; non-lossless or out-of-range values fail closed as `SF_INVALID_DOC_ID`
- Elasticsearch/Turbopuffer route semantics determine whether the gateway emits `upsert`, `patch`, or `delete`
- gateway must strip the configured `DocIDMapping.source_field` from admitted upsert/patch payloads before emission
- `continue_on_error=true` is used only for source routes that require deterministic per-entry continuation semantics, such as Elasticsearch `/_bulk` and Elasticsearch by-query routes when source conflict policy is `conflicts=proceed`
- gateway-local `noop` outcomes may exist for source-wire reconstruction, but they must not emit synthetic canonical mutations

Payload hard checks already owned by the canonical API and therefore enforced here:
- `FieldData.list_value` must be homogeneous
- write-side vector dimensions must exactly match schema dimensions
- `object` / `nested` payload depth must not exceed one structural level
- for lifecycle-enabled collections, every emitted `patch` mutation must explicitly carry non-null `lifecycle_policy.time_partition_field`; the gateway must not backfill that field from point-read state
- malformed typed carriers still fail closed by canonical precedence (`SF_BAD_REQUEST`, `SF_SCHEMA_VIOLATION`, or `SF_VECTOR_DIMENSION_MISMATCH` as applicable)

## 5. Metadata Resolution and Cache Consistency

### 5.1 Descriptor Resolution Model

Gateway write admission requires these descriptor facts:
- authoritative `CollectionRef`
- authoritative `collection_truth_mode`
- authoritative `DocIDMapping.generation_policy`
- private collection lifecycle state
- active `schema_version`
- active `schema_generation_token`
- effective `max_result_window`
- allocator-admission policy when allocator-backed create/insert identity is in play
- lifecycle-policy carry requirements when lifecycle is enabled
- private admission-state freshness marker (`descriptor_revision` / etag) from the CCS-private owner `api/proto/searchfabric/v1/catalog_control_private.proto`

Source-of-truth split:
- public typed catalog reads (`GetCollectionMetadata`, `GetCollectionSchema`) remain the source for public descriptor payloads and route-level metadata/schema reconstruction
- private CCS admission-state reads carry internal-only facts needed for write admission and cache validation:
  - collection lifecycle state (at minimum `ACTIVE` vs `TOMBSTONED`; stricter admin-hold substates must collapse into the same fail-closed branch from the gateway's perspective)
  - descriptor revision / etag
  - name-binding negative-cache invalidation identity
  - authoritative `DocIDMapping.generation_policy` so allocator-backed and caller-required routes are distinguished without inference
  - `CollectionSettings.allocator_admission_policy` when allocator-backed create/insert identity is admitted for the collection
  - effective query-window and lifecycle carry requirements needed by query-assisted write lanes

The gateway may cache these facts locally, but the cache is never authoritative.

### 5.2 Recommended CCS Routing Strategy

For hyperscale collection cardinality, gateway instances should prefer sticky routing to CCS pods for name-resolution and descriptor fetches.

Recommended strategy:
- compute routing key from canonical scope:
  - collection-name lookup lane: hash of `(namespace_id, collection_name)`
  - collection-id lane: hash of `(namespace_id, collection_id)`
- map the key to a CCS StatefulSet ordinal with Jump Consistent Hash
- route to `ccs-{ordinal}.ccs-svc.sf.svc.cluster.local`

Why:
- it improves CCS-side L2 cache locality
- it avoids cluster-wide cache thrash under 1B-collection cardinality
- it keeps the routing rule deterministic without introducing a second control authority

This routing choice is an implementation optimization only. If a targeted pod is unavailable, the gateway must retry another healthy CCS endpoint or the cluster service VIP. Correctness still comes from authoritative CCS responses, not from routing affinity.

### 5.3 Recommended L1 Admission Cache

Gateway-local L1 admission cache should be short-lived and disposable.

Recommended Rust implementation:
- key space:
  - `(namespace_id, collection_name)`
  - `(namespace_id, collection_id)`
- value:
  - `CollectionRef`
  - authoritative `collection_truth_mode`
  - private lifecycle state
  - active schema pointer (`schema_version`, `schema_generation_token`)
  - effective `max_result_window`
  - lifecycle-policy carry requirements when lifecycle is enabled
  - descriptor revision / etag used for freshness checks
  - bounded-expiry metadata
- container:
  - sharded concurrent map (`DashMap` or equivalent)
  - `Arc`-shared immutable descriptor payloads
- admission/eviction:
  - W-TinyLFU or equivalent frequency-aware policy to protect against one-off scans
  - short negative-cache entries for deterministic not-found bursts

Cache rules:
- stale or unverifiable entries must trigger CCS re-fetch
- cache hits may avoid CCS only within the configured freshness budget
- positive and negative cache entries are validated against the private admission-state read model, not against gateway-synthesized heuristics
- gateway must not synthesize missing typed descriptor fields

### 5.4 Internal Invalidation Broadcast

To reduce stale-cache windows, gateways may maintain a private invalidation broadcast channel against CCS.

Important boundary:
- this watch channel is an internal deployment protocol, not part of the canonical public protobuf surface in `search_fabric.proto`
- the current public `CatalogControlService` contract is unchanged by this document
- if a machine-readable watch protocol is later introduced, it should remain an internal CCS deployment interface rather than being added casually to the public canonical service set

The watch channel improves freshness and reduces avoidable stale-cache hits, but it does not replace authoritative CCS admission checks.

#### 5.4.1 Logical Topology and Discovery

Recommended topology:
- CCS runs as a StatefulSet behind a Headless Service
- each gateway instance acts as a gRPC-stream client
- each CCS pod acts as a watch-stream server for the invalidation events it originates or relays

Discovery rules:
- on startup, the gateway resolves the Headless Service DNS name (for example `ccs-svc.sf.svc.cluster.local`) and collects the current Pod endpoints from the returned A records
- the gateway then establishes one long-lived watch stream per discovered CCS pod
- the gateway periodically re-resolves the Headless Service so scale-out, roll, and failover events update the watch set

Subscription-scope options:
- simplest first implementation: every gateway subscribes to every CCS pod
- optimized implementation: a gateway may subscribe only to the CCS routing lanes relevant to its deterministic Jump-Hash strategy, but this optimization must not weaken invalidation coverage during topology changes

#### 5.4.2 Private Wire Shape

Illustrative internal-only payload shape:

```proto
service CatalogControlWatchService {
  rpc WatchInvalidation(WatchInvalidationRequest) returns (stream InvalidationEvent);
}

message WatchInvalidationRequest {
  string gateway_id = 1;
  repeated uint32 subscribed_pod_ordinals = 2;
}

enum InvalidationEventType {
  INVALIDATION_EVENT_TYPE_UNSPECIFIED = 0;
  INVALIDATION_EVENT_TYPE_DESCRIPTOR_UPDATE = 1;
  INVALIDATION_EVENT_TYPE_SCHEMA_CHANGE = 2;
  INVALIDATION_EVENT_TYPE_DROP = 3;
  INVALIDATION_EVENT_TYPE_HEAD_ADVANCE = 4;
}

message InvalidationEvent {
  uint64 namespace_id = 1;
  uint64 collection_id = 2;
  string collection_name = 3;
  uint64 descriptor_revision = 4;
  optional uint64 manifest_head_version = 5;
  string applied_schema_generation_token = 6;
  InvalidationEventType event_type = 7;
}
```

Field intent:
- `namespace_id` + `collection_id` identify the collection-scoped cache entry to evict or mark stale
- `collection_name` allows the gateway to invalidate name-binding positive and negative cache entries
- `descriptor_revision` is the primary freshness marker for gateway admission caches
- `manifest_head_version` is optional diagnostic/context metadata and matters only when the same watch mechanism is reused by components that cache manifest-coupled descriptor fields
- `applied_schema_generation_token` is the active fencing token when the event affects schema lineage
- `event_type` helps the gateway choose whether to evict, hard-fail inflight work, or simply mark an entry stale

For Flow-B `Ingest Gateway`, `descriptor_revision` and `applied_schema_generation_token` are more important than manifest-head version. The gateway should not key invalidation correctness on `manifest_vN` alone.

#### 5.4.3 Trigger Chain

Recommended trigger chain:

1. A CCS-owned serving-state mutation commits:
- `PutCollectionMetadata`
- `PutCollectionSchema`
- `DropCollection`
- or a manifest-head advance is materialized in CCS after `Manifest Publisher` completes manifest CAS and sends `head_advance_intent`

2. The CCS pod that materializes the new authoritative state constructs an `InvalidationEvent`.

3. CCS fan-outs the event to all currently active watch streams whose subscription scope covers that collection.

4. The gateway receives the event and immediately removes or marks stale the matching L1 descriptor entry.

Event-source rules:
- schema changes and drops originate directly from CCS control-plane mutation handling
- manifest-head-derived invalidations originate only after CCS accepts the corresponding head-advance intent
- `Manifest Publisher` does not bypass CCS and push directly into gateway caches

#### 5.4.4 Gateway Receive Path

On event receipt, the gateway should execute this sequence:

1. locate the matching entry in the admission cache by `(namespace_id, collection_id)`
2. if `collection_name` is present, also evict any `(namespace_id, collection_name)` positive or negative cache entry
3. call `DashMap.remove(...)` or mark the entry stale with the new freshness marker
4. release the cache-resident object weight from the eviction policy accounting
5. leave any frequency sketch aging to the cache-policy implementation; exact decrement of TinyLFU-style admission counters is optional and often not worth synchronizing on the hot path
6. ensure the next request for that collection must re-resolve through CCS before mutation emission

Concurrency guidance:
- run stream receive, event decoding, and cache eviction in a dedicated Tokio task or dedicated worker queue
- do not rely on nonexistent Tokio strict task priorities; use isolation, bounded queues, and backpressure so invalidation handling is not blocked behind ordinary request work

#### 5.4.5 Reconnect and Resync

Because the watch stream is point-to-point and non-durable, reconnect logic must assume events may have been missed.

Required behavior:
- on stream break, the gateway starts exponential-backoff reconnect for that CCS pod
- on successful reconnect, the gateway marks all cache entries learned from or validated through that stream as suspect
- subsequent requests for suspect entries must revalidate through the private CCS admission-state read model before write admission

Cold-start or post-reconnect resync must use the right surface for the right cache:
- private admission-state cache repair uses the internal CCS admission-state read model
- public descriptor/schema refresh still uses:
  - `GetCollectionMetadata` for authoritative public collection descriptor refresh
  - `GetCollectionSchema` for active schema body and fencing token refresh
- `GetSchemaSyncStatus` only when the gateway also needs convergence state for a newly active generation

The gateway must not use `GetReadiness` for this purpose:
- `GetReadiness` belongs to `QueryGatewayService`
- it is a pre-query context carrier, not a CCS descriptor-repair primitive

Conservative fallback mode:
- if the gateway cannot cheaply determine the affected subset after a broken stream, it may invalidate the whole local admission cache or the whole shard-local subset and lazily repopulate through CCS

#### 5.4.6 Safety Net for Lost Events

If an invalidation event is missed in transit:
- the gateway may continue using stale schema, ownership, or lifecycle state
- downstream admission or execution may reject the request with one of:
  - `409 SF_VERSION_CONFLICT`
  - `409 SF_OWNERSHIP_CONFLICT`
  - `404 SF_RESOURCE_NOT_FOUND`
- on any of those stale-state branches, the gateway must force-invalidate the local admission-cache entry, bypass cache on the next lookup, and reload from CCS before any retry

This forced-invalidation path is the final correctness backstop. The watch stream only reduces the stale window; correctness still comes from catalog fencing and downstream version checks.

#### 5.4.7 Unavailable Watch Path

If the invalidation watch is unavailable:
- gateway falls back to TTL plus freshness-marker revalidation
- writes that cannot validate ownership/schema against sufficiently fresh CCS state fail closed as `SF_CATALOG_UNAVAILABLE`

## 6. Admission Guards

### 6.1 Ownership and Lifecycle

Flow-B admission is catalog-driven:
- `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL` must reject native-search-compatible writes with `409 SF_OWNERSHIP_CONFLICT`
- `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC` admits Search-Fabric-owned write gateways (Elasticsearch, Turbopuffer, and TiDB federated SQL remote DML)
- private collection lifecycle state `ACTIVE` is the only write-admissible branch for ordinary Flow-B mutation/schema work
- `TOMBSTONED` or any stricter private fail-closed admin-hold branch must reject write admission before mutation emission
- gateway-managed schema mutation must perform the same ownership validation before issuing `PutCollectionSchema`

### 6.2 Schema Fencing

`MutationBatch.schema_generation_token` is mandatory.

Flow-B rule:
- gateway must emit the current active generation token resolved from CCS
- if the caller route semantics carry stale schema assumptions or a cached descriptor is older than CCS active generation, gateway must reject with `SF_VERSION_CONFLICT` rather than emitting a mixed-generation batch

Gateway-side stale-cache recovery:
- if a downstream ingest component still rejects a Flow-B batch with `SF_VERSION_CONFLICT` because the gateway raced a schema change, the gateway must force-invalidate the local descriptor entry for that collection
- the next attempt must re-resolve active schema state from CCS before any new mutation emission

### 6.3 Payload and Budget Guards

Before forwarding a batch, the gateway must enforce:
- payload-size budgets (`SF_PAYLOAD_TOO_LARGE`)
- per-request operation-count budgets (`SF_TOO_MANY_OPERATIONS`)
- route/profile subset validation (`SF_UNSUPPORTED_*`)
- list homogeneity, vector dimension, and object-depth guards

Per-namespace throttling should use distributed token buckets with local prefetch/refill:
- primary unit is write-capacity cost, not only request count
- refill authority should come from the CCS-private quota-allocation owner `api/proto/searchfabric/v1/catalog_control_private.proto` rather than an undefined standalone quota service
- rejection code is `429 SF_RATE_LIMITED`
- response projection must preserve canonical retry metadata and source-wire headers/body shape

## 7. Downstream Forwarding, ACK Translation, and Retries

### 7.1 Forwarding

After admission, the gateway forwards the canonical batch into the shared ingest path:
- direct `StreamMutations` gRPC, or
- an internal `Ingest Broker` hop that preserves the canonical batch unchanged

The gateway must not mutate batch identity after admission. In particular, it must not rewrite:
- `RequestMeta.idempotency_key`
- `batch_id`
- `batch_fingerprint`
- `stream_identity`
- `schema_generation_token`

### 7.2 ACK and Error Translation

Gateway response reconstruction is source-profile-specific but must preserve canonical outcomes:
- single-route success/error projection follows the profile mapping rules in the compatibility docs
- Elasticsearch existence-sensitive write reconstruction must use authoritative `MutationAck.success.item_outcomes[]`, including post-apply result kind for successful row mutations
- Elasticsearch `/_bulk` reconstruction must use `MutationAck.success.item_outcomes[]`
- canonical `ErrorStatus.correlation_id` must remain visible through source-wire headers (`X-SF-Code`, `X-SF-Correlation-Id`) where required by compatibility contracts

### 7.3 Retry Behavior

Gateway-managed retries must preserve replay stability:
- same admitted payload retry:
  - reuse the same `RequestMeta.idempotency_key`
  - reuse the same `batch_id`
  - preserve `batch_fingerprint`
  - preserve Flow-B `source_partition`
- sliced retry after first-error cursor:
  - preserve outer client request lineage for tracing
  - generate a new `batch_id`
  - regenerate ordered `source_seq` values for the sliced batch

Failure-class handling:
- CCS unavailable during admission -> `503 SF_CATALOG_UNAVAILABLE`
- ingest/broker saturation or unavailable downstream buffers -> `503 SF_BACKEND_UNAVAILABLE`
- quota exhaustion -> `429 SF_RATE_LIMITED`
- deterministic ownership/schema conflicts -> canonical `409` classes

## 8. Suggested Runtime Decomposition

Recommended Rust-side module split:
- `protocol_adapters`
  - Elasticsearch and Turbopuffer wire parsing, request reconstruction, route-level subset validation
- `scope_resolver`
  - authenticated ingress context -> canonical `namespace_id`
  - route token -> `CollectionNameBinding`
- `descriptor_resolver`
  - public catalog reads plus private CCS admission-state reads, Jump Hash routing, and name-binding resolution
- `admission_cache`
  - L1 admission-cache implementation, negative cache, suspect/stale marking, invalidation application
- `schema_coordinator`
  - `PutCollectionMetadata` / `PutCollectionSchema` orchestration for first-create, encryption-ensure, and ownership-gated schema apply
- `prewrite_reader`
  - write-authoritative point-read/OCC execution for admitted read-modify-write routes
- `selector_planner`
  - Query-Broker-backed selector materialization via the dedicated internal selector RPC, deterministic `(doc_id, selected_version_commit_ts)` ordering, probe-cap enforcement, and response-shape inputs for by-query / by-filter write routes
- `mutation_compiler`
  - Flow-B stream identity derivation, canonical entry ordering, `MutationBatch` / `batch_fingerprint` construction
- `quota_gate`
  - local token bucket admission and overload shedding
- `ingest_client`
  - gRPC forwarding and ACK/error handling
- `response_translator`
  - source-wire success/error/body reconstruction

This split keeps protocol-specific behavior, metadata correctness, and canonical batch construction isolated from each other.

## 9. Performance and Concurrency Model

Recommended runtime model:
- Tokio async runtime
- request-scoped structured concurrency for:
  - authenticated-context intake / scope resolution
  - descriptor fetch / schema coordination
  - payload normalization
  - downstream forwarding
- `bytes::Bytes` and `Arc`-shared descriptors to reduce copy pressure

Hot-path guidance:
- keep descriptor-hit writes free of blocking remote control-plane work except when schema coordination is required
- parallelize item parsing/canonicalization where source semantics permit, but serialize final `MutationEntry.source_seq` assignment in admitted order
- apply load shedding before expensive canonicalization when quota or broker-saturation signals are already decisive

## 10. Observability

Minimum gateway metrics:
- Flow-B request rate, admitted rate, rejection rate by canonical `SF_*`
- descriptor-cache hit rate, miss rate, stale-hit rate, and negative-cache hit rate
- CCS resolution latency, error rate, and per-pod routing distribution
- invalidation-stream reconnect count and invalidation-to-eviction lag
- schema-coordination latency and schema-conflict rate
- batch size, entries per batch, and payload bytes
- rate-limit rejects by throttled dimension
- downstream broker/ingest latency, saturation, and retry count

Minimum tracing/logging keys:
- `idempotency_key`
- `correlation_id`
- transport trace metadata
- `client_id`
- `client_kind`
- `namespace_id`
- `collection_id`
- `batch_id`
- `source_partition`
- active `schema_generation_token`

These signals are required to distinguish source-wire incompatibility, stale catalog state, ownership violations, and downstream saturation.