# TiDB Enhancement Profile

Date: 2026-02-28
Status: Draft (Normative)
Owner: TiDB Integration / Metadata Search / Query Gateway

## 1. Scope

This document defines phased TiDB enhancement delivery for MySQL full-text search and SQL indexing capabilities.

Normative interpretation:
- This document governs enhancement-scope behavior together with the canonical lowering boundary defined by `docs/specs/core/search-fabric-api.md`.
- Enhancement items that cannot be lowered deterministically to canonical Search Fabric APIs are out of scope until the canonical API is extended; they must not be implemented as gateway-local semantic forks.

In scope:
- Enhancement profile definitions and roadmap phase binding
- MySQL full-text search enhancement scope
- SQL enhancement scope for flexible multi-field filtering index
- Ownership applicability boundary (`collection_truth_mode`) for TiDB-led enhancement path
- Phase-bound delivery commitments (private preview, GA Readiness, development delivery)
- Independent release/readiness track relative to MySQL compatibility profiles
- Functional / Reliability / Economics gate definitions per phase-bound profile

## 2. Profiles and Phase Binding

Terminology convention:
- `Profile` defines enhancement-scope boundary and conformance scope.
- `Milestone` and schedule commitments are defined by `docs/governance/roadmap.md` phases, not by separate numbering in this document.

### 2.1 Profile 1 (`2026-08-31` TiDB Private-Preview Checkpoint): Flexible Filtering Index Private Preview Profile

- Phase window: TiDB private-preview checkpoint (`2026-08-01` to `2026-08-31`).
- Scope: deliver private preview for:
  - Index with flexible filtering on any combination of multi-fields.
- Delivery status target: private preview only.

### 2.2 Profile 2 (Roadmap Phase 3): Flexible Filtering Index GA Readiness Profile

- Phase window: `Phase 3` (`2026-09-01` to `2026-10-31`).
- Scope: strict superset of Profile 1 for the same SQL enhancement capability.
- GA Readiness target:
  - Index with flexible filtering on any combination of multi-fields reaches GA Readiness.

### 2.3 Profile 3 (Roadmap Phase 4A / 4B Split): MySQL Fulltext Search Enhancement Delivery Profile

- Phase window: broader `Phase 4` window (`2026-11-01` to `2027-01-31`) with one Phase-4A MySQL-blocking subset lane and one same-date Phase-4B multilingual-extension tail lane.
- Scope: deliver the remaining implementation work that backs `docs/compatibility/mysql-fulltext.md` Profile 3 external semantics:
  - table-level stopword/tokenizer configurability in canonical applicability mode
  - operational hardening for CJK tokenization on the existing MySQL-compatible parser/tokenizer surface
  - Myanmar analyzer/tokenizer coverage as a TiDB-enhancement-only extension layered on top of MySQL compatibility, delivered in the final Profile-3 stage together with the last CJK hardening tranche
  - partitioned-table plumbing and lifecycle support required by the MySQL-compatible FTS path
- Clarification:
  - Japanese parser coverage is already represented by the active `MECAB` parser surface in `docs/compatibility/mysql-fulltext.md`; this delivery profile does not re-open that baseline capability.
  - Myanmar support is not part of the baseline MySQL compatibility registry today; when delivered, it is tracked only by this TiDB enhancement profile and MUST land through an explicit analyzer-registry extension plus the final Profile-3 hardening stage.
- Delivery status target: development delivery in Phase 4, split between:
  - a MySQL-blocking subset whose certification-critical foundation freezes on the `2026-12-15` Phase-4A foundation checkpoint and whose final handoff closes on the `2027-01-31` internal engineering qualification checkpoint;
  - a TiDB-only multilingual extension tail (for example Myanmar support), if admitted in the same phase, that closes on the same-date post-GA expansion checkpoint and is not a prerequisite for MySQL Profile 3.

### 2.4 Phase-1 Baseline Clarification for Commitment-A Decomposition

- Roadmap Phase-1 Commitment-A rollout may include baseline ad-hoc multi-field predicate planning via non-profile fallback execution.
- Roadmap Phase 1 also includes TiDB federated-table write/read integration and `COLLECTION_TRUTH_MODE_SEARCH_FABRIC` ownership enablement as part of the integrated governance plan in `docs/governance/roadmap.md`; that workstream is outside this document's Flexible Filtering Index profile line.
- That Phase-1 baseline is not counted as Flexible Filtering Index profile delivery in this document.
- This document's first profile-scoped indexed-acceleration delivery starts at Profile 1 (the `2026-08-31` TiDB private-preview checkpoint), then Profile 2 (`Phase 3`, GA Readiness).
- Auditable ownership clarification: Phase-1 baseline fallback capability is validated only as part of the integrated Phase-1 roadmap acceptance package in `docs/governance/roadmap.md` Section `6` and remains outside this document's profile conformance gates.

## 3. SQL Enhancement Scope

### 3.1 Flexible Multi-Field Filtering Index Capability

Capability statement:
- The system supports index planning and execution for flexible filtering on any combination of multi-fields.

Phase binding:
- `Phase 1`: baseline non-profile fallback planning may exist, but it is outside this document's profile conformance scope.
- `2026-08-31` TiDB private-preview checkpoint: private preview rollout.
- `Phase 3`: GA Readiness rollout.

## 4. MySQL Fulltext Search Enhancement Delivery Scope (Phase 4A / 4B Split)

Profile 3 covers the following delivery items that back the MySQL-facing compatibility contract:

1. Table-level stopword and tokenization configuration
- Implement table-level stopword/tokenizer configuration for the external semantics defined in `docs/compatibility/mysql-fulltext.md`.

2. Multi-lingual tokenizer coverage
- Implement operationally complete CJK tokenizer coverage on the existing MySQL-compatible parser/tokenizer surface.
- Japanese `MECAB` runtime support remains part of the already-declared baseline parser surface.
- Myanmar-specific parser/analyzer support is deferred to the final Profile-3 tranche, where it is delivered together with the last CJK hardening work as a TiDB-enhancement overlay rather than baseline MySQL compatibility.

3. Partitioned-table support
- Implement partitioned-table support that satisfies the architecture contract in `docs/specs/core/search-fabric-architecture.md` Section `11.6` and the MySQL-facing compatibility contract in `docs/compatibility/mysql-fulltext.md` Profile 3.

Delivery of these Profile-3 items remains TiDB-authoritative at DDL apply time:
- search-relevant DDL MUST synchronize through canonical `PutCollectionSchema`, carrying `scope.collection`, `scope.ddl_commit_ts`, authoritative monotonic `schema_version` sourced from TiDB DDL/catalog lineage, and `SchemaFencePrecondition` (`expected_prev_schema_version`, `schema_generation_token`, `enforce_monotonic_apply=true`).
- `PutCollectionSchema.update_policy` MUST be explicit and MUST NOT be `SCHEMA_UPDATE_POLICY_UNSPECIFIED`; the first apply for a collection that already has catalog identity but no prior schema lineage MUST use `SCHEMA_UPDATE_POLICY_INITIAL_CREATION`, and later TiDB-authored DDL MUST classify into additive vs rebuild policy according to the canonical schema-diff rules.
- when TiDB emits the first schema generation for a collection that already has catalog identity but no prior schema lineage, the request MUST use the full canonical zero-state tuple (`schema_version=1`, `fence.expected_prev_schema_version=0`, `fence.schema_generation_token="__initial__"`, `update_policy=SCHEMA_UPDATE_POLICY_INITIAL_CREATION`, `fence.enforce_monotonic_apply=true`).
- the same schema-sync RPC MUST carry replay-stable non-empty `PutCollectionSchema.meta.idempotency_key` plus explicit adapter `PutCollectionSchema.meta.client_id` / `PutCollectionSchema.meta.client_kind=CLIENT_KIND_TIDB_DDL_ADAPTER`.
- DDL admission MUST capture replay-critical schema-admission context and persist it into `CollectionSchemaDefinition.replay_context` for the same schema generation, so retry/replay/rollback paths preserve the original `sql_mode` / `time_zone` execution context while tokenizer semantics remain in typed schema fields and charset/collation data remains supplemental metadata.
- Compatibility/readback remains canonical: `GetCollectionSchema` is the authoritative schema descriptor readback surface, while `GetSchemaSyncStatus` is convergence/status observation for the active admitted generation; neither is reconstructed from source DDL text.

## 5. Release Independence and Coexistence with MySQL Compatibility

This document defines a TiDB enhancement track that is independent from MySQL-compatible API profile release readiness.

Independence rules:
- TiDB enhancement capabilities outside MySQL full-text API compatibility follow their own rollout rhythm and readiness gates.
- It is valid for the `2026-07-31` public-preview milestone to close for MySQL/Elasticsearch while TiDB enhancement Profile 1 remains on the later `2026-08-31` TiDB private-preview checkpoint.
- Such coexistence is not a conflict: they are different capability lines with different validation surfaces and risk envelopes.
- MySQL-compatible API behavior remains governed by `docs/compatibility/mysql-fulltext.md`; TiDB enhancement behavior remains governed by this document.
- For Phase-4 Profile-3 items, this document governs delivery/readiness only; the externally visible SQL/parser/partition semantics remain governed by `docs/compatibility/mysql-fulltext.md`.

## 6. Roadmap Binding

This document binds to `docs/governance/roadmap.md` phase semantics as an independent enhancement track:
- `Phase 1`: TiDB federated-table integration and Search-Fabric-truth ownership enablement land in roadmap governance, but remain outside this document's profile-scoped Flexible Filtering Index commitments.
- `2026-08-31` TiDB private-preview checkpoint: private preview target for flexible multi-field filtering index (TiDB enhancement track).
- `Phase 3`: GA Readiness target for flexible multi-field filtering index (TiDB enhancement track).
- `Phase 4A` / `2026-12-15` foundation checkpoint + `2027-01-31` internal engineering qualification checkpoint: the MySQL-blocking Profile-3 subset (table-level stopword/tokenizer config, CJK hardening, partitioned-table support) first freezes its certification-critical foundation and then closes final cross-document handoff to MySQL Profile 3.
- `Phase 4B` / post-GA expansion checkpoint: any TiDB-only multilingual extension tail admitted in Phase 4 closes separately and is not a prerequisite for MySQL Profile 3.

## 6.1 Ownership Applicability Boundary

Canonical applicability mode for this enhancement track:
- `collection_truth_mode=COLLECTION_TRUTH_MODE_EXTERNAL`

Applicability rules:
- Profile 1/2 flexible-filtering-index commitments are normative for the canonical applicability mode only.
- Profile 3 MySQL fulltext enhancement delivery commitments are also normative for the canonical applicability mode.
- In canonical mode, authoritative mutations are TiDB-led (`TiDB -> TiCDC`); direct native-search-compatible writes are out of this track's conformance scope and handled as ownership-policy violations.
- `collection_truth_mode=COLLECTION_TRUTH_MODE_SEARCH_FABRIC` may exist in architecture (`docs/specs/core/search-fabric-architecture.md`) but is outside this document's conformance commitments unless a phase-bound addendum explicitly includes it per `docs/governance/roadmap.md` Section `3.1`.

## 6.2 Document ID Contract and Active Lowering Limitations (Inherited)

For TiDB enhancement conformance in canonical applicability mode, the `doc_id` schema contract is inherited and mandatory:
- Canonical document identifier MUST lower to `DocumentID.id_value.u64`.
- Control-plane schema MUST declare `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_UINT64`.
- `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_STRING` and `SCALAR_FIELD_TYPE_BYTES` are intentionally unsupported in this enhancement track until canonical API scope expands.
- Source table MUST expose canonical `doc_id` source as:
  - unsigned `int64` `PRIMARY KEY`, or
  - unsigned `int64` `UNIQUE NOT NULL` column, or
  - unsigned `int64` `AUTO_INCREMENT` column (`NOT NULL` and uniqueness-preserving).
- Canonical `doc_id` source is immutable after insert; TiCDC events attempting to mutate canonical `doc_id` source MUST be rejected as contract violations.
- Non-conforming tables are out of enhancement-conformance scope and MUST be rejected during onboarding/configuration validation.

## 7. Gate Criteria (Functional / Reliability / Economics)

### 7.1 Profile 1 Gate (`2026-08-31` TiDB Private-Preview Checkpoint, Private Preview)

Functional gate:
- Flexible multi-field filtering index supports planner/executor path for arbitrary field combinations in declared preview scope.
- Preview conformance suite for declared SQL enhancement subset passes.

Reliability gate:
- Replay/idempotency and failover behavior pass preview fault-injection checklist with no release-blocking defects.
- No cross-tenant isolation violations in preview cohorts.

Economics gate:
- TiDB Enhancement Profile-1 metrics stay within the profile-scoped subset of the milestone-closing TiDB private-preview workload bundle and the corresponding envelope declared by `docs/governance/roadmap.md` Sections `3.2` and `3.3`.

### 7.2 Profile 2 Gate (`Phase 3`, GA Readiness)

Functional gate:
- Profile 1 scope is complete and promoted to GA Readiness with production-ready behavior.
- GA Readiness acceptance suite for flexible multi-field filtering index passes.

Reliability gate:
- Production SLO/error-budget validation passes under sustained multi-tenant traffic patterns.
- On-call runbooks and rollback drills are validated.

Economics gate:
- Throughput/latency/cost stay within the profile-scoped subset of the milestone-closing Phase-3 multi-tenant workload bundle and the Phase-3 GA-Readiness envelope declared by `docs/governance/roadmap.md` Sections `3.2` and `3.3`.

### 7.3 Phase-4A Gate (`2026-12-15` Foundation Checkpoint + `2027-01-31` Internal Engineering Qualification Checkpoint, MySQL-Blocking Subset)

Functional gate:
- The MySQL-blocking subset (table-level stopword/tokenizer configuration, partitioned-table support, and the declared CJK hardening tranche) freezes its certification-critical foundation on `2026-12-15` and is fully implemented/integration-tested by the `2027-01-31` internal engineering qualification checkpoint.
- TiDB-owned search DDL paths use the canonical DDL envelope (`ddl_commit_ts`, `SchemaFencePrecondition`, monotonic apply, and persisted replay-critical context in `CollectionSchemaDefinition.replay_context`) consistently across normal apply, replay, and rollback flows.
- Cross-document compatibility handoff to `docs/compatibility/mysql-fulltext.md` Profile 3 / Phase-4A internal-engineering-qualification requirements is complete.

Reliability gate:
- New tokenizer and partitioned-table paths pass regression and failover safety checks with no release-blocking defects.
- Upgrade/migration safety checks for newly enabled capabilities are validated.

Economics gate:
- The MySQL-blocking subset stays within the accepted `2026-12-15` Phase-4A foundation package plus the accepted `2027-01-31` internal engineering qualification package for the certification-critical checkpoint.

### 7.4 Phase-4B Gate (`2027-01-31` Post-GA Expansion Checkpoint, TiDB-Only Multilingual Extension Tail)

Functional gate:
- Any TiDB-only multilingual extension tail (for example Myanmar support) closes only after the required analyzer-registry extension is admitted and compatibility/test documents are updated in lockstep.
- This tail is explicitly non-blocking to `docs/compatibility/mysql-fulltext.md` Profile 3 and does not reopen either the `2026-12-15` Phase-4A foundation checkpoint or the final Phase-4A MySQL-blocking subset gate.

Reliability and search-quality gate:
- Any retrieval-affecting tokenizer/analyzer change admitted in this tail MUST publish a fresh checkpoint-closing benchmark bundle and satisfy the continuation rules in `docs/testing/search-quality-evaluation.md`.
- New multilingual tokenizer/analyzer paths pass regression and failover safety checks with no release-blocking defects.

Economics gate:
- The TiDB-only multilingual extension tail stays within the declared Phase-4B post-GA expansion closeout package in `docs/testing/roadmap-workload-bundles.md`.

## 8. Deterministic Error Mapping

Determinism rule:
- For unsupported, out-of-scope, or contract-violating TiDB enhancement behavior, the system MUST fail closed with deterministic error code mapping; no silent fallback is allowed.
- Canonical `SF_*` code definitions (status class, retryability, semantic contract) are centralized in `docs/specs/core/error-codes.md` as the source of truth.
- This section only defines TiDB enhancement scenario-to-code mapping and MUST reuse canonical codes from `docs/specs/core/error-codes.md`.
- At the internal Search Fabric API boundary reached after TiDB-side lowering, canonical/native error projection uses the canonical envelope fields from `docs/specs/core/error-codes.md` Section `2`: `code`, `message`, `retryable`, `correlation_id`, optional typed `context`, and optional string `diagnostics`; this sentence does not redefine TiDB SQL/client-visible wire diagnostics.
- If multiple violations apply to the same request, code selection MUST follow deterministic precedence defined in `docs/specs/core/error-codes.md` Section `2.1`.

### 8.1 Profile and Capability Boundary Errors

- Enhancement feature is outside active profile scope: `400 SF_UNSUPPORTED_ENHANCEMENT_FEATURE`
- Partitioned-table enhancement requested before Phase-4/Profile-3 window: `400 SF_PARTITIONED_TABLE_UNSUPPORTED`
- Unsupported table-level enhancement operation for active profile: `400 SF_UNSUPPORTED_TABLE_OPERATION`

### 8.2 Contract and Ownership Errors

- Source-row `doc_id` value not losslessly normalizable as canonical `SCALAR_FIELD_TYPE_UINT64` (including non-`unsigned int64` or out-of-range values): `422 SF_SCHEMA_VIOLATION`
- Canonical doc-id schema branch requesting `DocIDMapping.source_field_type=SCALAR_FIELD_TYPE_STRING` or `SCALAR_FIELD_TYPE_BYTES`: `400 SF_DOC_ID_CONTRACT_VIOLATION`
- Canonical doc-id schema branch requesting malformed source-field enum outside the admitted set: `400 SF_INVALID_SCHEMA`
- Missing canonical `SCALAR_FIELD_TYPE_UINT64` `unsigned int64 NOT NULL` unique key source for `doc_id`: `400 SF_DOC_ID_CONTRACT_VIOLATION`
- Attempted mutation of canonical `doc_id` source after insert (including TiCDC `doc_id` source update): `400 SF_DOC_ID_CONTRACT_VIOLATION`
- Write path violates the authoritative `collection_truth_mode` policy: `409 SF_OWNERSHIP_CONFLICT`
- Catalog/descriptor service unavailable for ownership/schema validation: `503 SF_CATALOG_UNAVAILABLE`

### 8.3 Runtime Timeout Errors

- Query execution timeout: `504 SF_QUERY_TIMEOUT`
- Wait-for-visible timeout (only when caller explicitly requests retrieval-side wait-for-visible acknowledgment, i.e., `visibility_policy=wait_for_visible`): `504 SF_VISIBILITY_TIMEOUT`

### 8.4 Common Access, Admission, and Backend Errors

- Authentication failure or missing credentials: `401 SF_UNAUTHORIZED`
- Authenticated caller lacks required namespace/collection/API permission: `403 SF_FORBIDDEN`
- Target table/index does not exist in active enhancement scope: `404 SF_RESOURCE_NOT_FOUND`
- Document version/create-only conflict, or enhancement write/configuration state conflict surfaced by TiDB or control plane: `409 SF_VERSION_CONFLICT`
- Tenant/API throttling or admission quota rejection: `429 SF_RATE_LIMITED`
- Payload/schema/type violation: `422 SF_SCHEMA_VIOLATION`
- Vector payload dimension mismatch on hybrid payload path: `422 SF_VECTOR_DIMENSION_MISMATCH`
- Required backend dependency temporarily unavailable: `503 SF_BACKEND_UNAVAILABLE`

### 8.5 Generic and Internal Fallback Errors

- Malformed request envelope/body that does not match any narrower deterministic mapping: `400 SF_BAD_REQUEST`
- Query structure invalid for active enhancement profile capability scope: `400 SF_INVALID_QUERY`
- Unclassified internal server failure after deterministic mapping rules are exhausted: `500 SF_INTERNAL`