# ID Allocator Service

Date: 2026-03-22
Status: Frozen architecture note
Authors: Search Platform Architecture

## 1. Scope

This document defines the component-level architecture of `IdAllocatorService`, the stateful control-plane authority for canonical `DocumentID.id_value.u64` realization on allocator-admitted create/insert routes.

The service exists to support one production-grade pre-ingest identity model across gateway-owned write paths without weakening the canonical ingest contract:
- Elasticsearch auto-ID on `POST /{index}/_doc`
- Elasticsearch-compatible explicit create/insert `_id` on collections that choose allocator-backed create/insert identity

Current canonical caller scope in this revision is intentionally narrower than the longer-term design space:
- admitted allocator-backed caller scope is production-closed to `ALLOCATOR_CALLER_PROFILE_ES_WRITE_GATEWAY_ONLY`
- Turbopuffer and TiDB allocator-backed create/insert routes remain future-only extension points and are not part of the current canonical caller-profile contract

The canonical public contract remains:
- `api/proto/searchfabric/v1/search_fabric.proto`
- `docs/specs/core/search-fabric-api.md`
- `docs/specs/core/error-codes.md`

This document does not redefine canonical ingest/query semantics. It explains how allocator-backed create/insert identity is realized at production scale while preserving the canonical rule that admitted `RowMutation.doc_id` is always explicit.

## 2. Boundary

### 2.1 Allocation Responsibility

`IdAllocatorService` is responsible for:
- authoritatively resolving create/insert document identity for collections whose schema chooses `DocIDMapping.generation_policy=DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`
- issuing canonical `uint64` IDs when source-side create/insert identity is omitted
- reserving or validating caller-supplied explicit create/insert IDs so explicit and implicit IDs share one collection-local reservation space
- guaranteeing replay-stable realized-ID results for the same admitted allocation request
- maintaining private per-collection allocator state (`next_doc_id`, state version, reservation lineage, replay truth)
- enforcing allocator-side admission checks that depend on authoritative collection facts
- returning the active write-admission `schema_generation_token` observed during allocation so the caller can construct the follow-on canonical mutation batch consistently
- providing high-throughput batch/range issuance without leaking allocator runtime state into the canonical public descriptor model

It is explicitly not responsible for:
- collection/name resolution
- namespace or collection descriptor mutation
- schema mutation
- hidden ingest-time allocation
- patch/delete/update-style identity resolution
- query-time point lookup or readback by generated ID
- caller-local SQL/REST response reconstruction

### 2.2 Relationship to Existing Components

The steady-state write pipeline is:

```text
Currently admitted write gateway (Elasticsearch write gateway)
    -> resolve/validate authoritative collection identity
    -> IdAllocatorService.AllocateIds
       - one ordered `plan.items[]` over source rows
       - `issue` item          (omitted create/insert identity)
       - `explicit_doc_id`     (caller-supplied create/insert identity)
    -> build canonical MutationBatch with explicit RowMutation.doc_id
       + allocator-returned schema_generation_token
       + allocator-returned allocator_receipt
    -> IngestGatewayService.StreamMutations
```

Key boundary rules:
- `IdAllocatorService` is a control-plane service, not an ingest-plane helper.
- `StreamMutations` keeps its canonical contract: `RowMutation.doc_id` remains required.
- allocator inputs MUST be stable `CollectionRef` values, not mutable name bindings such as `CollectionNameBinding`.
- allocator-issued or allocator-reserved IDs are valid only for the exact stable collection lineage named by `CollectionRef`.
- allocator runtime state is private allocator state and MUST NOT be projected into public `CollectionDescriptor` or `NamespaceDescriptor`.
- patch/delete/update-style routes remain explicit-identity-only and MUST NOT use allocator reservation as an ingest-time fallback.

### 2.3 Why the Service Is Separate from CCS

`IdAllocatorService` depends on CCS authoritative collection facts, but it is still a separate bounded context because:
- CCS traffic is metadata-heavy and relatively low-frequency, while ID allocation is hot-path control traffic with very different throughput and latency characteristics.
- allocator runtime state (`next_doc_id`, optional leased ranges, replay ledger, explicit-reservation burn history) is not descriptor state.
- allocator failure modes, scaling strategy, and hot-key mitigation are different from catalog/schema mutation.
- keeping allocator state out of CCS descriptor rows prevents catalog reads from becoming dependent on allocator hot-path churn.

This design therefore uses a single-direction dependency:

```text
IdAllocatorService -> CCS authoritative collection admission facts
CCS -X-> IdAllocatorService
```

CCS remains the source of truth for collection identity, lifecycle, `DocIDMapping.generation_policy`, and `CollectionSettings.allocator_admission_policy`. `IdAllocatorService` remains the source of truth for allocator runtime state and replay truth.
The allocator correctness path does not require a TiDB-backed online projection: correctness depends on one allocator authority model plus in-memory hot state derived from that authority, not on a second SQL-state commit surface.

## 3. Service Surface and Caller Ownership

### 3.1 Service Name

The preferred service name is `IdAllocatorService`, not the generic `AllocatorService`, because the component allocates canonical document identity rather than arbitrary resources.

### 3.2 Logical RPC Surface

This architecture assumes one canonical service method:
- `AllocateIds`

`AllocateIds` is authoritative for all allocator-admitted create/insert identity resolution and therefore admits one ordered mixed-mode allocation plan:
- `plan.items[].issue`: realize allocator-owned IDs for omitted create/insert identity
- `plan.items[].explicit_doc_id`: reserve or validate caller-supplied create/insert IDs inside allocator authority before canonical mutation construction

One request therefore covers mixed explicit/omitted source batches atomically under one replay-governed allocator apply unit and one returned `schema_generation_token`.

Range leasing remains an internal implementation optimization for allocator nodes. It MUST NOT appear as a second public client RPC in this revision, and it MUST NOT change the public replay contract, request shape, or caller-visible realized-ID results unless a future canonical proto/API revision intentionally promotes a separate public surface.

### 3.3 Caller-Ownership Matrix

| Scenario | Canonical caller | `AllocateIds` plan shape | Required `client_kind` | Notes |
| --- | --- | --- | --- | --- |
| Elasticsearch auto-ID | Elasticsearch write gateway | one `issue` item per omitted create/insert row | `CLIENT_KIND_ES_WRITE_GATEWAY` | Applies only to create/insert routes whose source `_id` is omitted. |
| Elasticsearch explicit create/insert `_id` on allocator-optional collection | Elasticsearch write gateway | one `explicit_doc_id` item per explicit create/insert row | `CLIENT_KIND_ES_WRITE_GATEWAY` | Explicit `_id` still must parse losslessly as canonical `uint64`, but allocator remains the admission authority for that create/insert identity. |

Current-revision contract note:
- only the Elasticsearch rows above are part of the admitted canonical caller profile represented by `ALLOCATOR_CALLER_PROFILE_ES_WRITE_GATEWAY_ONLY`
- schema admission MUST reject `CollectionSettings.allocator_admission_policy.caller_profile` values other than `ALLOCATOR_CALLER_PROFILE_ES_WRITE_GATEWAY_ONLY`

Caller-required collections bypass allocator entirely. Patch/delete/update-style routes also bypass allocator entirely and remain explicit-identity-only.

### 3.4 Future Extension Shapes (Not Admitted in the Current Revision)

The following shapes are intentionally not part of the current canonical caller-profile space. They remain design placeholders for a future contract expansion and MUST NOT be exposed through current schema admission or runtime policy state.

| Future-only scenario | Planned caller | Planned `AllocateIds` plan shape | Planned `client_kind` | Notes |
| --- | --- | --- | --- | --- |
| Turbopuffer omitted-ID create/insert | Turbopuffer write gateway | one `issue` item per omitted create/insert row | `CLIENT_KIND_TP_WRITE_GATEWAY` | Would require a later canonical expansion that explicitly enables allocator-backed create/insert identity for this caller kind. |
| Turbopuffer explicit create/insert `id` on allocator-optional collection | Turbopuffer write gateway | one `explicit_doc_id` item per explicit create/insert row | `CLIENT_KIND_TP_WRITE_GATEWAY` | Would share the same reservation space as omitted-ID issuance once explicitly admitted. |
| TiDB federated-table omitted-ID remote DML | TiDB write gateway | one `issue` item per omitted create/insert row | `CLIENT_KIND_TIDB_WRITE_GATEWAY` | Would still need to lower to deterministic explicit `RowMutation.doc_id` before canonical ingest. |
| TiDB federated-table explicit create/insert remote DML | TiDB write gateway | one `explicit_doc_id` item per explicit create/insert row | `CLIENT_KIND_TIDB_WRITE_GATEWAY` | Would keep explicit create/insert identity inside allocator authority once explicitly admitted. |

### 3.5 Admission Preconditions

Before admitting a fresh `AllocateIds` allocation, the service MUST validate:
- the referenced collection exists and is in an allocator-admissible lifecycle state
- the authoritative `collection_truth_mode` and write-policy inventory admit this caller path
- `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC`
- `DocIDMapping.generation_policy=DOC_ID_GENERATION_POLICY_ALLOCATOR_OPTIONAL`
- `CollectionSettings.allocator_admission_policy` is present and valid
- `CollectionSettings.allocator_admission_policy.caller_profile=ALLOCATOR_CALLER_PROFILE_ES_WRITE_GATEWAY_ONLY`
- the current caller kind is therefore `CLIENT_KIND_ES_WRITE_GATEWAY`
- `CollectionSettings.allocator_admission_policy.max_plan_items_per_request > 0`
- `CollectionSettings.allocator_admission_policy.max_forward_reservation_jump > 0`
- `route_intent=ID_ALLOCATION_ROUTE_INTENT_CREATE_INSERT`
- `plan.items[]` is non-empty and every item selects exactly one valid branch
- explicit `explicit_doc_id` values are unique within the plan
- plan cardinality does not exceed the admitted allocator request ceiling
- every explicit reservation stays within the admitted forward-reservation window, and the remaining collection-local `uint64` identity space can satisfy every issuance/reservation step in order

The allocator MUST fail closed when:
- the collection is unknown or tombstoned
- the caller is not authorized for allocator-backed create/insert identity on that collection
- the collection policy requires caller-supplied identity for every route (`DOC_ID_GENERATION_POLICY_CALLER_REQUIRED`)
- the request attempts to use allocator on patch/delete/update-like routes

Successful replay resolution is a separate earlier step. A stored successful replay result for the same replay key plus the same canonical allocation payload MUST short-circuit these fresh-allocation checks and return the original realized result verbatim.

The service does not accept unresolved names because same-name reuse after tombstone GC would otherwise create ABA risk.

## 4. Dependency on CCS

### 4.1 Authoritative Read Dependency

`IdAllocatorService` needs the CCS-private admission-state read model keyed by `CollectionRef` and owned by `api/proto/searchfabric/v1/catalog_control_private.proto`. That read model MUST provide at least:
- `namespace_id`
- `collection_id`
- lifecycle admission state
- authoritative `collection_truth_mode`
- active `DocIDMapping.generation_policy`
- active `CollectionSettings.allocator_admission_policy.caller_profile`
- active `CollectionSettings.allocator_admission_policy.max_plan_items_per_request`
- active `CollectionSettings.allocator_admission_policy.max_forward_reservation_jump`
- the active write-admission `schema_generation_token` needed by the caller to continue toward canonical mutation emission
- a descriptor validator / etag so allocator caches can be invalidated deterministically

This dependency is read-only and authoritative:
- allocator MUST NOT attempt to reconstruct collection lifecycle or ownership from stale local heuristics
- allocator MUST NOT perform a fresh allocation against a collection that CCS currently rejects
- allocator MUST treat CCS admission-state change as authoritative for future allocation even if allocator durable state still exists for historical replay/audit
- allocator MUST still return a previously recorded successful replay result verbatim when the replay key and canonical payload match, even if current CCS state would now reject a fresh allocation on that collection lineage

### 4.2 Cache Model

Allocator nodes MAY cache positive CCS admission-state snapshots keyed by `CollectionRef`, but only while the bound descriptor validator remains unchanged.

Allocator nodes MUST NOT:
- cache by `collection_name`
- treat name bindings as allocator scope keys
- continue issuing or reserving fresh IDs after CCS has reported lifecycle or ownership invalidation for that collection

### 4.3 Separation of State

CCS owns:
- collection identity
- lifecycle
- truth mode
- `DocIDMapping.generation_policy`
- `CollectionSettings.allocator_admission_policy` (explicit caller profile plus discoverable guardrail ceilings)
- active write-admission `schema_generation_token`

`IdAllocatorService` owns:
- `next_doc_id`
- allocation epoch / fence state
- replay ledger for allocation requests
- issued / reserved identity truth
- optional leased-range state
- watermark advancement that results from admitted explicit reservation on allocator-optional collections

This split keeps allocator runtime state out of catalog descriptor tables and avoids making CCS metadata writes contend with allocator hot-path issuance.

## 5. Allocation Model

### 5.1 Canonical Output

Allocator output is always canonical `DocumentID.id_value.u64` carried through `AllocateIdsSuccessResponse.realized_doc_ids[]`.

For compatibility paths that expose lexical IDs externally:
- Elasticsearch auto-ID response payloads continue to use decimal-string `_id`
- other gateways may project the same canonical `uint64` into their own wire-level ID field if their profile admits allocator-backed create/insert identity

The allocator therefore returns canonical numeric IDs first. Gateway-specific lexical rendering happens after allocation and outside the allocator contract.

Gateways that admit allocator-backed create/insert identity MUST:
- preserve source-input order in `realized_doc_ids[]`
- preserve whether each realized ID was allocator-issued or caller-supplied (`allocator_issued=true|false`) when their public route exposes that distinction
- carry the returned `schema_generation_token` into the follow-on canonical `MutationBatch`
- carry the returned `allocator_receipt` unchanged into the follow-on canonical `MutationBatch` whenever that batch realizes allocator-backed create/insert identity

### 5.2 Collection-Scoped Private State

Allocator state is private and scoped by stable `CollectionRef`.

Conceptually, each collection has:
- `next_doc_id`
- `allocation_epoch`
- `allocator_state_version`
- optional active lease upper bound / lease epoch
- last update timestamp

`next_doc_id` is the collection-local exclusive upper bound over the allocator-managed `uint64` identity space. Implementations MUST preserve the terminal exhausted sentinel losslessly even when that internal value equals `2^64` and therefore no longer fits in a `uint64` payload carrier.

The allocator key is the stable tuple `(namespace_id, collection_id)`.
It MUST NOT be `(namespace_id, collection_name)`.

### 5.3 Monotonicity, Burned Reservations, and Watermark Ownership

Allocator-backed IDs are opaque stable document identities.

Rules:
- uniqueness is required within one collection
- monotonic increase is preferred for operational predictability
- gaps are allowed
- `next_doc_id` is a dense-burn watermark: once allocator durable admission advances it to `N`, every lower `doc_id < N` becomes permanently unavailable for fresh allocation, including IDs skipped by earlier high explicit reservations or discarded lease tails
- IDs MUST NOT be reused once the watermark has advanced past them, even if the later write never becomes visible
- one ordered `plan.items[]` request executes sequentially in source order against one shared watermark
- `issue` items burn one fresh ID each from the current watermark
- `explicit_doc_id` items burn the named IDs in source-input order
- fresh explicit reservation naming `doc_id < next_doc_id` MUST fail as `SF_VERSION_CONFLICT` with deterministic diagnostics `doc_id` and `allocator_conflict_reason=below_watermark_burned_gap`
- explicit create/insert reservation with `doc_id >= next_doc_id` MUST advance allocator watermark to `doc_id + 1` as part of allocator durable admission itself when the jump stays within the admitted forward-reservation window
- explicit create/insert reservation whose jump exceeds that admitted forward-reservation window MUST fail as `SF_TOO_MANY_OPERATIONS`
- issuance/reservation that cannot be satisfied because the remaining allocator-managed `uint64` identity space is exhausted MUST fail as `SF_TOO_MANY_OPERATIONS`
- failed allocator admission MUST NOT advance watermark
- later ingest failure MUST NOT roll allocator state back or “return” previously issued/reserved IDs to the free pool
- if range leasing is enabled, any unused tail of a durably reserved lease MAY be discarded on process crash, drain, or ownership handoff; discarded IDs become burned gaps and MUST NOT be reintroduced into the free pool

Watermark ownership therefore lives entirely inside `IdAllocatorService`; the write path does not need a later post-commit `ObserveCommittedDocId` callback.

### 5.4 Route-Intent and Request Shape

Allocator authority is create/insert-only.

The request contract is machine-closed as:
- `route_intent=ID_ALLOCATION_ROUTE_INTENT_CREATE_INSERT`
- one required ordered allocation plan:
  - `plan.items[].issue`
  - `plan.items[].explicit_doc_id`
- deterministic guardrails:
  - `plan.items[]` MUST be non-empty and source-ordered
  - explicit IDs within one plan MUST be unique
  - total plan cardinality MUST stay within the admitted allocator request ceiling
  - explicit forward jumps MUST stay within the admitted forward-reservation window

This keeps route-intent enforcement, mixed-mode atomicity, and allocator guardrail enforcement authoritative at allocator admission time instead of relying on caller convention alone.

### 5.5 Admission Guardrails

The active design uses two deterministic allocator admission guardrails:
- `max_plan_items_per_request`: maximum admitted `plan.items[]` cardinality for one `AllocateIds` RPC
- `max_forward_reservation_jump`: maximum admitted forward jump from the current `next_doc_id` to a newly reserved explicit `doc_id`

Guardrail semantics:
- exceeding `max_plan_items_per_request` fails closed as `SF_TOO_MANY_OPERATIONS` with `limit_scope=allocator_plan_items`
- exceeding `max_forward_reservation_jump` fails closed as `SF_TOO_MANY_OPERATIONS` with `limit_scope=allocator_forward_reservation_window`
- when `remaining_identity_space = 2^64 - next_doc_id` is insufficient for a rejected issuance/reservation step, allocation fails closed as `SF_TOO_MANY_OPERATIONS` with `limit_scope=allocator_identity_space_remaining`

These guardrails are part of the canonical admission contract; in the active revision they are discoverable through the public `CollectionSettings.allocator_admission_policy` surface rather than only through out-of-band operator policy, even if a future implementation chooses to source the effective values from static service policy or a collection-bound policy surface.

## 6. Replay and Idempotency Model

### 6.1 Transport Replay Identity

Allocator requests require replay-stable request identity.

The transport replay key is derived from:
- server-authenticated caller identity
- `client_id`
- `client_kind`
- `rpc_method`
- `CollectionRef`
- `RequestMeta.idempotency_key`

Same transport replay key plus the same canonical allocation payload MUST return the original realized-ID result verbatim.

Same transport replay key plus a different canonical allocation payload MUST fail closed as an idempotency conflict.

Successful replay is resolved before fresh CCS admissibility checks. Once a successful allocation result has been durably recorded, later lifecycle/policy/schema changes do not retroactively convert that same replay into a fresh-allocation rejection; they only govern whether a new allocation may be admitted.

### 6.2 Canonical Allocation Payload Shape

The canonical allocation payload includes:
- target `CollectionRef`
- `route_intent`
- one ordered `plan.items[]` sequence whose items are each exactly one of:
  - `issue`
  - `explicit_doc_id`

`plan.items[]` MUST be non-empty, kept in source-input order, and explicit IDs must remain unique within the plan.

Patch/delete/update-style routes remain explicit-identity-only even on collections that admit allocator-backed create/insert identity.

### 6.3 Relationship to Later Mutation Replay

Allocator replay and mutation replay are separate layers:

1. allocation replay guarantees the same admitted allocation request returns the same realized IDs
2. mutation replay guarantees the same admitted mutation payload is not applied twice

This split is required because allocation happens before the gateway has fully constructed the final canonical `MutationBatch`.

When one admitted source batch mixes explicit and omitted IDs, the gateway lowers that batch into one ordered allocator plan and then binds `realized_doc_ids[]` back to source-input order before canonical mutation emission so later response reconstruction can return realized IDs deterministically in that same order.

The follow-on attestation boundary between those two replay layers is the signed `allocator_receipt`:
- allocator success returns the receipt together with `realized_doc_ids[]` and `schema_generation_token`
- the caller copies that receipt unchanged into `MutationBatch.allocator_receipt`
- ingest validates the receipt against caller identity, collection scope, route intent, realized-ID order/content, and `schema_generation_token` before trusting allocator-backed explicit `RowMutation.doc_id` values

Allocator replay MUST therefore survive writer retries and failover independently of later ingest replay.

### 6.4 Success Replay vs Failure Audit

The active canonical replay contract is success-oriented:

1. a successful allocation request is replay-authoritative and MUST be returned verbatim on identical replay
2. allocator failures remain canonical non-`OK` gRPC + `ErrorStatus` outcomes and do not define a second public replay payload family in this revision
3. implementations MAY persist failed attempts for audit/operations, but those records are not allowed to masquerade as success-shaped replay truth

This split avoids inventing undocumented sentinel `realized_doc_ids[]` or `schema_generation_token` values for failures that never reached a valid realized-ID outcome.

### 6.5 No Hidden Allocation Inside Ingest

`StreamMutations` MUST NOT silently allocate missing IDs.

Why:
- it would make canonical mutation shape depend on server-side hidden behavior
- it would break deterministic request hashing and replay reasoning
- it would blur the error boundary between allocation failure and mutation admission failure
- it would reopen the shared explicit/implicit ID-space race that allocator authority is intended to close

The allocator service keeps that concern explicit and observable.

## 7. Durable State and Recovery

### 7.1 Durable Object Families

Allocator durable truth lives in dedicated object families, separate from CCS catalog objects.
The minimum authoritative allocator persistence set in this design is:

```text
s3://bucket/id-allocator/state/{hash(namespace_id)[0:2]}/{namespace_id}/{collection_id}.json
s3://bucket/id-allocator/replay-success/{hash(transport_lookup_key)[0:2]}/{transport_lookup_key}.json
```

Authoritative object roles:
- `state/.../{collection_id}.json` is the durable allocator state for one collection
- `replay-success/.../{transport_lookup_key}.json` is the immutable success replay record for one admitted allocation request
- allocator-owned durable objects are authoritative for replay truth, watermark state, reservation history, and allocator recovery state
- fresh allocation admissibility remains CCS-owned authority whenever the request is not satisfied by a stored successful replay hit

Auxiliary non-authoritative artifacts MAY also exist:

```text
s3://bucket/id-allocator/failure-audit/{hash(allocation_attempt_id)[0:2]}/{allocation_attempt_id}.json
s3://bucket/id-allocator/recovery-segments/segment_{allocator_recovery_epoch}.json.zst
```

Auxiliary artifact roles:
- `failure-audit/.../{allocation_attempt_id}.json` is an optional immutable audit record for one failed attempt; it is not replay-authoritative in the active revision
- `recovery-segments/...` is an optional ordered audit/rebuild index that can assist forensic analysis or offline reconstruction; it is not part of the minimum authoritative replay/watermark state set

Normative authority rules:
- TiDB rows, caches, and other operational read models MUST be treated as optional materializations only
- loss or staleness of any optional materialization MUST NOT change replay, watermark, or reservation correctness

### 7.2 Durable State Payload

Each collection allocator-state object MUST persist at least:
- `namespace_id`
- `collection_id`
- `next_doc_id` (lossless exclusive-upper-bound encoding; MUST preserve the terminal exhausted sentinel even when it equals `2^64`)
- `allocation_epoch`
- `allocator_state_version`
- `updated_at_unix_ms`

If range leasing is enabled, the durable state MAY also persist:
- current active lease upper bound / last durably reserved range end
- lease epoch / owner fence for the currently recognized lease owner
- lease policy version

When leasing is enabled, the collection allocator-state object remains authoritative for:
- the collection-global dense-burn watermark
- the fact that a specific contiguous range has already been durably reserved/burned into allocator authority
- the active owner fence, if any, allowed to serve local `issue` requests from that already reserved range

It is not required to record every per-request lease-local issuance progress as a separate state rewrite. Per-request caller-visible truth inside an already durably reserved lease is carried by the corresponding immutable success replay object.

### 7.3 Replay Object Payload

Each successful allocation replay object MUST persist at least:
- `transport_lookup_key`
- caller-supplied `RequestMeta.idempotency_key`
- authenticated caller identity hash
- `client_id`
- `client_kind`
- `namespace_id`
- `collection_id`
- `route_intent`
- canonical ordered allocation plan payload
- canonical payload fingerprint
- realized `doc_id` result in source-input order with `allocator_issued` flags
- returned `schema_generation_token`
- signed allocator-receipt claims (or the exact receipt payload) for the follow-on mutation attestation
- creation/apply timestamps

If the implementation also persists failure-audit objects, those records SHOULD store canonical error/audit facts (for example selected `SF_*` code, contextual diagnostics, and timing), but they MUST remain distinct from success replay truth and MUST NOT require fabricated success-only fields such as `realized_doc_ids[]` or `schema_generation_token`.

### 7.4 Recovery Rules

Recovery rules:
- allocator durable state is rebuilt from `state/...` plus success replay records, not from CCS descriptor rows
- CCS remains the authoritative source for whether a collection is still eligible for future fresh allocation
- serving nodes MUST rebuild in-memory allocator state from authoritative allocator objects before resuming allocation traffic on a recovered shard/owner
- no TiDB row/table projection is required before allocation traffic resumes
- if a collection is tombstoned while allocator recovery is in progress, recovery restores historical success replay truth but future fresh allocation against that collection remains blocked by CCS admission

Partial-write rule:
- the allocator MUST define one internal apply epoch / fence for each successful fresh allocation or lease-reservation publication
- recovery MUST treat any request as incompletely committed if its required authoritative object set is missing:
  - non-leased fresh allocation or any request that durably advances allocator state/watermark directly -> updated `state/...` object plus matching `replay-success/...` object
  - lease-backed `issue` request served entirely inside an already durably reserved active lease -> pre-existing authoritative `state/...` object that already records the reserved lease boundary plus the matching per-request `replay-success/...` object
- recovery MUST replay/reconcile any such incomplete authority set before serving that collection shard again
- optional operational exports and auxiliary recovery-segment artifacts that lag or disappear do not participate in this reconciliation rule because they are not authority objects

## 8. Optional Operational Materialization

The allocator MAY publish optional operational materializations for debugging, inspection, or ad hoc analytics.
Those materializations are explicitly outside the correctness path.

Materialization rules:
- they MUST be rebuildable from authoritative allocator objects
- they MUST NOT be used as the source of truth for replay, watermark, or reservation conflict decisions
- allocator success ACK MUST NOT depend on synchronous refresh of an operational materialization
- stale, missing, or dropped materializations may degrade observability only; they MUST NOT change client-visible allocator semantics

If a TiDB-backed operational export is still desired, it SHOULD be documented and implemented as a read-only/debugging materialization populated asynchronously from authoritative allocator objects rather than as a serving-time CAS surface.

## 9. Apply Pattern

Every allocation request follows one logical pattern:

1. Resolve phase:
- validate caller identity, route intent, ordered plan shape, and allocator guardrails
- check replay index/cache for duplicate request handling
- if replay key + canonical payload fingerprint matches a stored successful result, return the stored `realized_doc_ids[]`, stored `schema_generation_token`, and an `allocator_receipt` bound to that same successful result tuple, then stop
- read authoritative collection admission state from CCS only when the request is not a successful replay hit
- if collection identity exists but CCS cannot supply the authoritative write-admission facts required by allocator, fail `SF_CATALOG_UNAVAILABLE`; when no active schema generation has been admitted yet, emit `ErrorStatus.context.allocator.catalog_unavailable.schema_bootstrap.schema_bootstrap_state=no_active_schema_generation` and reuse the canonical schema-bootstrap diagnostics (`catalog_unavailable_reason=schema_bootstrap`, `schema_bootstrap_state=no_active_schema_generation`)
- reject ineligible collections before touching allocator durable state for a fresh allocation

2. Durable-authority phase:
- execute the ordered allocation plan sequentially in authoritative allocator state under the collection-local allocator fence
- advance watermark as each admitted explicit reservation reaches or crosses `next_doc_id`
- fail closed before commit if any plan item would exceed the forward-reservation window or the remaining allocator-managed `uint64` identity space
- durably publish the authoritative success set required by the chosen execution path:
  - direct-state path (non-leased request, explicit reservation path, or any path that durably reserves/extends a lease boundary in allocator state):
    - updated collection allocator-state object
    - immutable successful allocation replay record with the realized result, returned `schema_generation_token`, and allocator-receipt claims
  - lease-local `issue` path satisfied entirely from an already durably reserved active lease:
    - immutable successful allocation replay record with the realized result, returned `schema_generation_token`, and allocator-receipt claims
    - no second per-request allocator-state rewrite is required because the authoritative state object already records the burned lease boundary and owner fence for that range
- optionally append an ordered recovery-segment entry for audit/offline rebuild assistance; this artifact is not part of the authoritative success set
- optionally write a separate failure-audit record when admission or durable apply fails; failure-audit records MUST NOT be used as success replay truth
- allocator success is committed only after the required authoritative object set for that execution path is durable; operational materializations are not part of the commit point

3. Hot-state / export phase:
- refresh in-memory allocator hot state from the newly committed authoritative result
- optionally enqueue or refresh any non-authoritative operational materialization
- return success without waiting for optional exports to catch up

Steady-state request flows are therefore:

```text
omitted create/insert ID:
  CCS admission -> plan.items=[issue] -> realized_doc_ids[allocator_issued=true] + allocator_receipt -> build RowMutation.doc_id + MutationBatch.allocator_receipt

caller-supplied explicit create/insert ID:
  CCS admission -> plan.items=[explicit_doc_id] -> realized_doc_ids[allocator_issued=false] + allocator_receipt -> build RowMutation.doc_id + MutationBatch.allocator_receipt

mixed explicit + omitted create/insert batch:
  CCS admission -> ordered mixed plan -> ordered realized_doc_ids[] + allocator_receipt -> build source-order RowMutation.doc_id values + MutationBatch.allocator_receipt
```

This mirrors CCS's durable-then-materialize discipline while keeping allocator state and replay truth separate from catalog mutation truth and while keeping optional operational exports out of the correctness boundary.

## 10. Collection Lifecycle Interaction

### 10.1 Drop and Tombstone

Allocator behavior for collection lifecycle:
- `ACTIVE` collections may allocate or reserve when collection policy admits it
- tombstoned collections MUST reject new allocation
- allocator state for tombstoned collections MAY remain durably stored for replay/audit until GC

### 10.2 Same-Name Recreation

Allocator correctness depends on stable collection identity:
- old `(namespace_id, collection_name)` lineage and new same-name lineage after recreation MUST NOT share allocator state
- allocator state is therefore keyed only by stable `collection_id`
- same-name recreation produces a new allocator state row/object because it also produces a new stable collection identity

### 10.3 State GC

Allocator state GC is separate from collection tombstone GC:
- replay records may need to outlive the collection's active serving life
- collection removal from CCS public read surface does not mean allocator replay truth can be deleted immediately
- allocator GC policy should therefore be defined in terms of audit/replay retention windows rather than public catalog visibility alone

## 11. Gateway Integration Patterns

### 11.1 Elasticsearch

For Elasticsearch `POST /{index}/_doc` without source `_id`:
- gateway resolves authoritative `CollectionRef`
- gateway calls `IdAllocatorService.AllocateIds(plan.items=[issue])`
- gateway renders the allocated canonical `uint64` as decimal-string `_id`
- gateway emits canonical mutation with explicit `RowMutation.doc_id`, the allocator-returned `schema_generation_token`, and the allocator-returned `allocator_receipt`

For Elasticsearch explicit create/insert `_id` on an allocator-optional collection:
- gateway resolves authoritative `CollectionRef`
- gateway parses `_id` losslessly to canonical `uint64`
- gateway calls `IdAllocatorService.AllocateIds(plan.items=[explicit_doc_id])`
- gateway emits canonical mutation with the reserved explicit `RowMutation.doc_id`, the allocator-returned `schema_generation_token`, and the allocator-returned `allocator_receipt`

For mixed explicit/omitted create batches on an allocator-optional collection:
- gateway resolves authoritative `CollectionRef`
- gateway lowers the source rows into one ordered mixed-mode allocator plan
- gateway calls `IdAllocatorService.AllocateIds(plan.items=[...])` once
- gateway binds the returned `realized_doc_ids[]` back to source-input order and propagates the returned `allocator_receipt` before canonical mutation emission

Patch/delete/update-style Elasticsearch routes remain explicit-identity-only and do not call allocator.

### 11.2 Future Extension Shapes (Non-Canonical in the Current Revision)

The current revision does not admit allocator-backed Turbopuffer or TiDB write paths. The following notes describe only how those routes would fit if a later canonical contract revision intentionally opens them.

Future Turbopuffer shape:
- gateway resolves authoritative `CollectionRef`
- a future schema contract would need to add a new allocator caller profile that admits `CLIENT_KIND_TP_WRITE_GATEWAY`
- omitted create/insert identity would lower to `issue`
- explicit create/insert identity would lower to `explicit_doc_id`
- canonical ingest truth would still remain explicit `uint64` `RowMutation.doc_id`

Future TiDB federated-table shape:
- TiDB write gateway resolves authoritative `CollectionRef`
- a future schema contract would need to add a new allocator caller profile that admits `CLIENT_KIND_TIDB_WRITE_GATEWAY`
- omitted create/insert identity would lower to `issue` items and caller-supplied create/insert identity to `explicit_doc_id` items inside one ordered allocator plan
- TiDB would still need to lower the statement into deterministic explicit `RowMutation.doc_id` values using the realized allocator result
- unsupported federated SQL shapes would remain unsupported; allocator admission would not relax the bounded-doc-id planning requirement for forms such as `INSERT ... SELECT` or non-point `UPDATE` / `DELETE`

## 12. Failure Model

Deterministic failure classes:
- unknown / tombstoned / fully GC-released collection lineage -> `SF_RESOURCE_NOT_FOUND`
- malformed request shape (`route_intent=UNSPECIFIED`, empty ordered plan, plan item with no valid branch, duplicate explicit IDs inside one plan, or other illegal create/insert allocator syntax) -> `SF_INVALID_ARGUMENT` or `SF_BAD_REQUEST` by canonical malformed-vs-semantic precedence
- authenticated caller lacks collection access -> `SF_FORBIDDEN`
- collection identity exists but authoritative allocator admission facts are unavailable or incomplete -> `SF_CATALOG_UNAVAILABLE`; when the branch is specifically “no admitted active schema generation yet”, `ErrorStatus.context.allocator.catalog_unavailable.schema_bootstrap.schema_bootstrap_state` MUST be `no_active_schema_generation`, and diagnostics MUST be `catalog_unavailable_reason=schema_bootstrap` plus `schema_bootstrap_state=no_active_schema_generation`
- collection truth mode / `DocIDMapping.generation_policy` / `CollectionSettings.allocator_admission_policy` does not admit allocator authority for this caller path -> `SF_OWNERSHIP_CONFLICT`
- fresh explicit reservation of any `doc_id < next_doc_id` collides with the dense-burn allocator watermark -> `SF_VERSION_CONFLICT` with `ErrorStatus.context.allocator.reservation_conflict.{doc_id, conflict_reason=ALLOCATOR_RESERVATION_CONFLICT_REASON_BELOW_WATERMARK_BURNED_GAP}` and the equivalent compatibility diagnostics `doc_id` / `allocator_conflict_reason=below_watermark_burned_gap`
- plan cardinality exceeds the admitted allocator request ceiling -> `SF_TOO_MANY_OPERATIONS` with `ErrorStatus.context.allocator.limit_rejection.{limit_scope=ALLOCATOR_LIMIT_SCOPE_PLAN_ITEMS, configured_limit, requested_value}` and compatibility diagnostics `limit_scope=allocator_plan_items`
- explicit reservation exceeds the admitted forward-reservation window -> `SF_TOO_MANY_OPERATIONS` with `ErrorStatus.context.allocator.limit_rejection.{limit_scope=ALLOCATOR_LIMIT_SCOPE_FORWARD_RESERVATION_WINDOW, configured_limit, requested_value}` and compatibility diagnostics `limit_scope=allocator_forward_reservation_window`
- remaining allocator-managed `uint64` identity space cannot satisfy the rejected issuance/reservation step -> `SF_TOO_MANY_OPERATIONS` with `ErrorStatus.context.allocator.limit_rejection.{limit_scope=ALLOCATOR_LIMIT_SCOPE_IDENTITY_SPACE_REMAINING, configured_limit, requested_value}` and compatibility diagnostics `limit_scope=allocator_identity_space_remaining`
- replay payload mismatch under the same transport replay key -> `SF_IDEMPOTENCY_CONFLICT`
- allocator durable state unavailable -> `SF_BACKEND_UNAVAILABLE`

The service MUST NOT partially succeed in the sense of returning fewer realized IDs than requested while marking the request successful. Partial realization would make caller retry behavior ambiguous.

## 13. Scaling Strategy

### 13.1 Baseline

The baseline service serves `AllocateIds` from in-memory per-collection allocator state owned by the active shard/leader and hydrated from authoritative allocator objects.

Baseline coordination rules:
- per-collection allocation remains fenced by allocator-owned epoch / ownership control, not by TiDB-row CAS
- replay lookup uses authoritative replay-success state plus in-memory cache
- authoritative state publication remains object-first under Section `9`; optional exports do not participate in admission or commit

### 13.2 Hot-Collection Optimization

For hot collections, allocator nodes MAY use leased contiguous ranges:
- one durable lease-reservation publication advances the collection-global watermark in authoritative allocator state and records the active lease owner fence for that reserved range
- one node serves many local `issue` plan items from that already durably reserved range without another allocator-state rewrite for each issued ID
- each successful lease-local `issue` request still durably writes its own immutable replay-success record, which remains the per-request caller-visible truth for replay
- explicit reservations still pass through the authoritative collection state and replay ledger because they must close the shared explicit/implicit ID space
- unconsumed leased IDs MAY be discarded during crash recovery, drain, or ownership handoff; discarded IDs remain permanently burned gaps rather than being reclaimed into a free pool

This optimization does not change the external replay contract.

### 13.3 Isolation

Allocator scaling and failure isolation should remain independent from CCS:
- separate rate limits
- separate concurrency control
- separate operational dashboards
- separate overload shedding

This is a primary reason the allocator remains a separate bounded context even though it depends on CCS facts.

## 14. Observability

Required metrics:
- `searchfabric_id_allocator_requests_total{client_kind,result,plan_shape}`
- `searchfabric_id_allocator_latency_ms{client_kind,plan_shape}`
- `searchfabric_id_allocator_plan_items_total{client_kind,item_kind}`
- `searchfabric_id_allocator_replay_total{classification}`
- `searchfabric_id_allocator_ccs_dependency_latency_ms`
- `searchfabric_id_allocator_state_conflict_retry_total`
- `searchfabric_id_allocator_range_lease_total{result}`

Telemetry label rules:
- `plan_shape=issue_only` when every `plan.items[]` entry is `issue`
- `plan_shape=explicit_only` when every `plan.items[]` entry is `explicit_doc_id`
- `plan_shape=mixed` when one request contains both `issue` and `explicit_doc_id` items
- `item_kind=issue` counts omitted-ID plan items
- `item_kind=explicit_doc_id` counts caller-supplied explicit-reservation plan items

Required structured logs:
- authenticated caller identity hash
- `client_id`
- `client_kind`
- `namespace_id`
- `collection_id`
- `RequestMeta.idempotency_key`
- `route_intent`
- `plan_shape`
- `issue_item_count`
- `explicit_item_count`
- replay classification
- CCS descriptor validator / etag

Always-on required logs are summary-level only:
- `ordered allocation plan summary` MUST be shape/count-oriented rather than a full per-item ID dump
- implementations MUST NOT require full realized-ID / explicit-reservation lists on every successful hot-path request

Debug / audit-only logs MAY additionally include:
- full realized doc IDs
- explicit reservation lists
- per-item allocation-plan details

## 15. Implementation Rollout Notes (Non-Normative)

Steady-state semantics are already fixed by the canonical contract:
- allocator-admitted create/insert routes always resolve identity through `IdAllocatorService`
- admitted mutations always carry explicit `RowMutation.doc_id`
- patch/delete/update-style routes remain explicit-identity-only

Implementation sequencing MAY still stage callers incrementally, but temporary deployment sequencing MUST NOT be mistaken for the steady-state contract described above.

## 16. Summary

`IdAllocatorService` is the control-plane authority for canonical create/insert document identity before canonical mutation emission.

Its defining properties are:
- separate bounded context from CCS
- single-direction dependency on CCS authoritative collection facts
- per-collection private allocator state keyed by stable `CollectionRef`
- replay-stable realized-ID results
- shared authority over omitted-ID issuance and explicit create/insert reservation
- watermark ownership inside allocator admission, with burned reservations and no post-commit callback path
- no hidden allocation inside `StreamMutations`

This preserves the canonical ingest contract while giving the current Elasticsearch write path one coherent production-ready allocator model and leaving Turbopuffer / TiDB allocator expansion as an explicit future contract change rather than an accidentally admitted policy state.