# Emulator-Based TiDB Integration Test and Automation Plan

## 1. Objective

Design a production-grade, deterministic integration test framework for the **TiDB Query Plane -> Emulator** read path, using Emulator as the Search Fabric-side test double and removing dependency on the heavy asynchronous CDC write chain during query-focused integration testing.

The target is to validate:

1. SQL predicate translation and RPC request correctness.
2. Snapshot visibility correctness under MVCC (`read_ts`, `commit_ts`, tombstone).
3. Query pagination / PIT-equivalent search-context lifecycle correctness (`next_page_token`, `search_context.token`, natural expiry / keep-alive refresh).
4. Failure handling contracts (`429`, `503`, timeout) and resource cleanup.
5. Stable, non-flaky execution in local and CI environments.

## 2. Scope and Non-Goals

### 2.1 In Scope

- Query-plane integration for auxiliary index acceleration and snapshot read correctness against Emulator.
- Data setup by short-circuit write injection through the test facade (`inject_writes` / `inject_changefeed_batch`).
- Deterministic `safe_ts` advancement and visibility checks without physical sleep.
- Contract testing at the TiDB <-> Emulator RPC boundary.

### 2.2 Out of Scope

- End-to-end CDC correctness (`TiDB -> TiCDC -> CDC Adapter -> Ingest Gateway`) for these cases.
- Full TiDB <-> Search Fabric system integration testing with production ingest/control/stateful components.
- Full durability/throughput benchmarking of ingest/batch pipelines.
- Search Fabric internal algorithm quality benchmarking (ANN/BM25 quality).

## 3. Current Baseline and Pain Points

Current TiCI integrationtest2 path is full-chain and sleep-based:

- `tests/integrationtest2/run-tests.sh:1069-1092` starts `minio + tici + tiflash + ticdc + changefeed`.
- `tests/integrationtest2/t/tici/tici_integration.test:49` uses `DO SLEEP(10)` before assertions.
- `tests/integrationtest2/run-tests.sh:512,533,554,832,846` uses fixed startup sleeps.

These patterns increase runtime, increase flaky probability, and blur query-plane failures with ingest-chain availability.

## 4. Design Principles

1. **CQRS-aligned test slicing**: keep query-plane tests independent from CDC flow.
2. **Deterministic timestamps**: explicit `commit_ts` and explicit `read_ts`.
3. **No timing assumptions**: no `time.Sleep` for correctness waiting.
4. **State-driven readiness**: visibility is gated by `safe_ts` assertions.
5. **Boundary observability first**: validate request payloads and fallback behavior at RPC boundary.
6. **Minimal environment**: prefer the smallest test-double runtime that preserves canonical behavior; production bundle decomposition is not a requirement.

Reference architecture constraints:

- `docs/specs/core/search-fabric-architecture.md:26-27,139,439-465` (Flow A + Flow B parallel ingress, NRT delta visibility).
- `docs/specs/core/search-fabric-architecture.md:331-334` (private profile bundles).
- `docs/specs/core/search-fabric-architecture.md:427,430` (`safe_ts` bound and strict read semantics).

## 5. Short-Circuited Data Setup

## 5.1 Data Setup Channel

For query-plane tests, bypass TiCDC and inject data through a dedicated **test facade** that is intentionally decoupled from production topology:

- Direct-write facade (`inject_writes`) for low-cost synthetic MVCC setup.
- Canonical batch-replay facade (`inject_changefeed_batch`) for CDC/order/idempotency cases that need canonical stream identity.
- Both surfaces are test-only. They preserve runtime semantics needed by tests, but they are not production Flow-B endpoints.

Required fields for the direct-write facade:

- `collection_ref`
- `writes[]`
  - each write carries `doc_id`, `commit_ts`, `op_type` (`upsert` or `delete` tombstone), and `payload`
- deterministic replay/idempotency cases SHOULD also provide explicit `batch_id` and `stream_identity`; when those cases also validate historical replay or DDL-boundary behavior, explicit `schema_generation_token` is mandatory.

Recommended setup envelope:

```json
{
  "collection_ref": {"namespace_id": 1001, "collection_id": 42},
  "batch_id": "case-basic-1",
  "stream_identity": {"source_partition": "__test__/direct/case-basic-1", "source_epoch": 1},
  "writes": [
    {"doc_id": 7, "commit_ts": 1010, "op_type": "upsert", "payload": {"title": "alpha"}},
    {"doc_id": 7, "commit_ts": 1030, "op_type": "delete"}
  ],
  "policy": {"durability": "wal_chunk_durable", "visibility": "wait_for_visible"}
}
```

## 5.2 Timestamp and Visibility Control

### Commit timestamp assignment

Use explicit timestamp plans per case on the test-only direct-write facade:

- For single-version cases: monotonic sequence (`T1 < T2 < ...`).
- For MVCC cases: scripted timeline (`create`, `update`, `delete(tombstone)`).

When test code needs a real TSO anchor, use TiDB oracle:

- `tests/realtikvtest/flashbacktest/flashback_test.go:79-83` shows `GetOracle().GetTimestamp(...)`.

### Safe timestamp advancement (no sleep)

Do not wait by wall-clock delay. Instead:

1. Inject writes with target `commit_ts`.
2. Poll readiness until `safe_ts >= target_commit_ts`:
   - canonical query/readiness contract assertions MUST use canonical `GetReadiness` with `RequestMeta` and concrete `CollectionRef`;
   - test-only `GET /__test__/readiness` is allowed only as a fixture-control shortcut for local harness plumbing and MUST NOT replace canonical `GetReadiness` in contract assertions.
3. Only then execute TiDB query with target `read_ts`.

State gate:

```text
STRICT / BOUNDED_STALENESS:
  effective_read_ts = min(read_ts, safe_ts)
  visible_for_snapshot(R_eff) iff commit_ts <= R_eff

BEST_EFFORT:
  request read_ts = 0
  effective_read_ts = safe_ts
  visible_for_snapshot(safe_ts) iff commit_ts <= safe_ts

TiDB SQL strict assertion requires:
  requested_read_ts is satisfied exactly
If strict mode computes R_eff < requested_read_ts:
  aux-index path must fallback to TiDB native plan,
  search-native path must fail-closed with `SF_VISIBILITY_TIMEOUT`.
```

### Deterministic polling helper

Use bounded polling (deadline + short tick), not fixed sleep:

```text
wait_until(deadline, tick) { query_state(); assert(predicate); }
```

## 5.3 Test Data Lifecycle

Use per-test isolated identifiers:

- `namespace_id` unique per test run.
- `collection_id` unique per test case.
- `doc_id` ranges reserved by test case.

This avoids cross-test state bleed and simplifies cleanup.

## 6. Query Plane Integration Scenarios

## 6.1 Auxiliary Index Acceleration

### Purpose

Verify TiDB planner/executor translates SQL predicates into expected Search query payload, and returned `doc_id` set is correct.

### SQL-to-request assertion points

- Primary assertion boundary (target architecture):
  - Planner emits `SearchPlanFragment` with generated canonical carriers already populated (`QueryRequest.collection`, `integration_type=INTEGRATION_TYPE_AUX_INDEX`, `consistency`, `expected_schema_generation_token`, `search_payload`, `fetch_spec`).
  - Executor/ACL only finalizes `RequestMeta` / timeout wiring and calls `QueryGatewayService.ExecuteQuery` with `integration_type=INTEGRATION_TYPE_AUX_INDEX`.
  - Test harness captures outbound Search Fabric RPC request and validates payload.
- Transitional compatibility anchor (while old TiCI path still exists):
  - `pkg/planner/core/operator/logicalop/logical_datasource.go:654-855`
  - `pkg/expression/fts_helper.go:159-232`
  - `pkg/executor/distsql.go:351-373`
  - `pkg/distsql/distsql.go:64-66`

### Assertions

1. Plan-level:
   - SQL uses auxiliary acceleration path (`integration_type=INTEGRATION_TYPE_AUX_INDEX`) as expected.
2. Request-level:
   - canonical query request carriers are correct: unary `QueryRequest` and streaming `StreamQueryRequest` both preserve `collection`, `consistency`, `integration_type`, and fetch-limit semantics appropriate to their dedicated fetch shapes.
   - executor-finalized `QueryRequest.meta` is correct for the canonical query contract and the stricter TiDB integration policy: the `RequestMeta` envelope is present, `client_kind=CLIENT_KIND_TIDB_COMPUTE`, `client_id` is non-empty, `correlation_id` is always non-empty for TiDB-issued `ExecuteQuery` calls, and `principal_subject` is preserved when available to the executor.
   - top-level hard filters are emitted in `search_payload.global_filter`, optional retrieval logic is emitted in `search_payload.root_retrieval`, and nested branches use `NestedQueryPayload.child_filter` / `NestedQueryPayload.root_retrieval` without collapsing those roles back into one mixed AST.
   - `search_payload.global_candidate_topk` is clamped by planner/session policy before RPC dispatch and remains independent from final `fetch_spec.limit`.
   - when paging/PIT is requested, `fetch_spec.page_window` and `fetch_spec.search_context` carry the canonical branch/field combination for first-page, continuation, or composite continuation flow.
   - when SQL/session semantics require total-hit tracking, planner/executor lowering sets `fetch_spec.track_total_hits` to the expected canonical branch (`enabled` or `threshold`) rather than relying on implicit defaults.
3. Consistency-level:
   - strict correctness is determined by readiness admission plus canonical outcome semantics, not by unconditional echo of observability fields.
   - canonical `GetReadiness` checks used by this suite carry `RequestMeta` plus concrete `CollectionRef`; test-only `/__test__/readiness` is not a substitute for query-plane contract assertions.
   - when response carries `effective_read_ts`, it must be coherent with the chosen fallback/error decision.
4. Result-level:
   - returned `doc_id` set equals expected set (exact match).
   - when total-hit tracking is enabled or thresholded, `success.total_hits.value` and `success.total_hits.relation` match the canonical contract; when total-hit tracking is explicitly disabled, `success.total_hits` is absent.
5. Lifecycle-level:
  - first-page requests that ask for PIT-equivalent retention assert canonical admitted behavior: when the emulator admits PIT-equivalent context, the response returns `pagination.search_context.token` and `pagination.search_context.expires_at_unix_ms`.
  - malformed page-window/search-context branches reject fail-closed: token-on-first-page, `keep_alive_ms=0`, invalid composite continuation, wrong-lineage or wrong-page-ordinal continuation-token reuse, unset/invalid `page_window` combinations, and any attempt to smuggle unary-only PIT/search-context controls into the dedicated `StreamQueryRequest` shape.
  - continuation requests validate `continuation_page.next_page_token` deterministically and, when `keep_alive_ms` is used to extend PIT-equivalent retention, the returned `search_context.expires_at_unix_ms` must refresh coherently with service policy.
  - composite continuation requires valid `composite_page.next_page_token`, and any carried `after_key` must exactly match the token-bound composite cursor.
  - the current revision has no public early-close RPC for search-context tokens; PIT-equivalent retention ends only by natural expiry, and `keep_alive_ms` refresh remains bounded by service policy.
  - query continuation with an expired `search_context.token` rejects fail-closed.

## 6.2 MVCC Snapshot Isolation with Tombstone

### Purpose

Verify version visibility for different `read_ts` values across update/delete history.

### Version path anchors

- Version extraction and propagation:
  - `pkg/executor/distsql.go:1531-1552`
  - `pkg/executor/builder.go:5180-5192`
  - `pkg/distsql/request_builder.go:172-210`
- Versioned cop command switch:
  - `pkg/kv/kv.go:582-585`
  - `pkg/store/copr/coprocessor.go:567-568,1717-1737`

### Example timeline

| doc_id | commit_ts | op       | expected visibility |
|---|---:|---|---|
| 10 | 1000 | upsert v1 | visible at `read_ts >= 1000` |
| 10 | 1020 | upsert v2 | replaces v1 at `read_ts >= 1020` |
| 10 | 1050 | tombstone | hidden at `read_ts >= 1050` |

Read checks:

- `read_ts=1010` -> v1 visible.
- `read_ts=1030` -> v2 visible.
- `read_ts=1060` -> not visible.

## 7. Local and CI Bundling Strategy

Use Emulator as the dedicated minimal **test-double runtime** for fast startup.

- It may embed query execution, readiness tracking, and test-only injection helpers in one process.
- It is not required to preserve production bundle boundaries such as `query_bundle` / `ingest_bundle` / `control_bundle`.
- Its only purpose is to provide the smallest runtime that still obeys canonical query, visibility, and replay semantics.

### 7.1 CDC-Free Minimal Environment Blueprint (Key Design)

This suite must run in a **CDC-free test-double profile** (`query_plane_min`) to remove heavy dependencies while preserving canonical read-path semantics.

#### Components kept

- TiDB test process (SQL planner/executor under test).
- Emulator runtime implementing canonical query/readiness behavior.
- Test-only facade surfaces on Emulator side:
  - direct-write injection endpoint,
  - canonical batch-replay endpoint,
  - visibility control/readiness endpoint,
  - fault-profile endpoint for deterministic contract-failure injection.

#### Components removed

- TiCDC.
- CDC Adapter.
- MinIO/S3 sink chain for test setup.
- Production bundle decomposition requirements for ingest/control/background.

#### Why correctness is still valid

- Query-plane correctness depends on canonical `commit_ts/read_ts/safe_ts` and ownership/error semantics, not on production service decomposition.
- The test facade replays canonical write-envelope and readiness rules needed by the assertions.
- Visibility remains state-gated by `safe_ts`, so snapshot assertions stay equivalent for read-plane validation even though the runtime is a test double.

CDC-free Emulator topology:

```text
                       (removed in this suite)
TiDB -> TiKV -> TiCDC -> CDC Adapter -> Ingest Gateway

                       (used in this suite)
Test Runner -> Emulator Test Facade (`inject_writes` / `inject_changefeed_batch`) -> Emulator runtime
      |                                                                          |
      +-------------------------------- SQL via TiDB ----------------------------+
```

### 7.2 Test-Only Setup API Contract (Concrete)

Expose test-only endpoints guarded by `intest`/test profile flags:

1. `POST /__test__/inject_writes`
   - Input: `collection_ref` + `writes[]`, with optional `idempotency_key`, `batch_id`, `stream_identity`, `schema_generation_token`, `policy`, `resume_cursor`.
   - Behavior: synthetic direct-write facade that preserves caller `commit_ts`, normalizes one canonical `MutationBatch`, and admits it through the same ingest machinery as canonical writes while remaining a non-canonical setup mode.
   - Deterministic defaults when omitted:
     - `batch_id = blake3("batch:" + collection_ref + normalized writes)`
     - `idempotency_key = blake3("request:" + collection_ref + normalized writes)`
     - `stream_identity = {"source_partition":"__test__/direct/<batch_id>","source_epoch":1}`
     - `writes[i].source_seq = i+1`
     - omitted `policy` resolves to emulator static ingest defaults (`durability=WAL_CHUNK_DURABLE`, `visibility=ASYNC`, `undeclared_field=REJECT`, `ingest_mode=STREAMING`)
   - Caller MAY omit `schema_generation_token` only for a synthetic single-generation convenience fixture with no replay/idempotency/DDL-boundary correctness assertions.
   - In that shortcut, emulator MUST deterministically resolve the current active schema-generation token for `collection_ref` from the collection catalog snapshot at setup admission time and stamp it onto the normalized canonical batch.
   - If no active schema-generation token exists, or if the case is validating replay/idempotency/historical/DDL-boundary behavior, the shortcut is invalid and the test setup must provide the token explicitly.
   - This endpoint is not production Flow B; caller-supplied `commit_ts` is legal only here.
2. `POST /__test__/inject_changefeed_batch`
   - Input: test-only replay envelope carrying canonical `entries[]` plus `collection_ref`; this is not raw `MutationBatch` JSON.
   - Replay/idempotency-focused cases MUST provide explicit `batch_id`, `stream_identity`, and `schema_generation_token`.
   - Caller MAY omit `schema_generation_token` only for a synthetic single-generation shortcut; emulator then deterministically resolves the active collection token from the catalog snapshot at setup admission time. This omission is not valid for historical replay, DDL-boundary validation, or replay-correctness cases. Caller SHOULD provide explicit `source_table{keyspace_id?,table_id}` when table-path identity matters.
   - Behavior: admit through the same changefeed-ingest path as canonical replay, using canonical batch/fingerprint rules with no compatibility replay schema.
3. `GET /__test__/readiness?namespace_id=&collection_id=`
   - Output: `safe_ts`, modality applied ts values.
   - This is a test-only fixture-control surface. It is not the canonical query-plane `GetReadiness` RPC and MUST NOT be used as the assertion authority for request metadata or concrete `CollectionRef` binding.
4. `POST /__test__/inject_progress`
   - Input: modality progress updates (for example resolved-ts advancement or explicit `applied_ts_*` refresh for a synthetic case).
   - Behavior: updates modality `applied_ts` and recomputes `safe_ts = min(enabled modality applied ts, including metadata_aux only when that optional modality is present)` inside Search Fabric logic.
   - Guardrail: direct `safe_ts` overwrite is forbidden on this endpoint.
5. Optional `POST /__test__/fault_profile`
   - Input: deterministic fault/override payload scoped by `(namespace_id, collection_id)`.
   - Included branches:
     - error injection (`SF_RATE_LIMITED`, `SF_BACKEND_UNAVAILABLE`, `SF_QUERY_TIMEOUT`)
     - visibility override through per-modality applied-ts overrides (`force_applied_ts_metadata_search`, `force_applied_ts_semantic_search`, optional `force_applied_ts_metadata_aux`)
   - Guardrail: visibility overrides are test-only and MUST be explicitly clearable (`clear_forced_visibility=true`).

### 7.2.1 How CDC Changefeed Data Is Replaced

Without a live TiCDC process, the suite does **not** fetch runtime changefeed from CDC.
Instead, it feeds a **canonical replay stream** into the same normalization boundary used by ingest:

1. `ROW_MUTATION` entries (upsert/delete with canonical `doc_id` + `commit_ts`).
2. `PROGRESS_MARK` entries (resolved-ts plus modality watermark instruction).
3. Optional `BARRIER_MARK` entries when a case needs an explicit ordering fence.

This preserves read-plane correctness checks because query visibility depends on `commit_ts/read_ts/safe_ts`, not on CDC process existence.

#### A. CDC-derived fixture source

Use prebuilt fixture files that mirror TiCDC sink semantics used by current integration flow:

- Current full-chain test sink is configured with canal-json + TiDB extension + row key:
  - `tests/integrationtest2/run-tests.sh:252`
  - `protocol=canal-json&enable-tidb-extension=true&output-row-key=true`

Fixture generation policy:

- `golden` fixtures are captured from one controlled TiCDC run (outside the minimal suite), translated into canonical `entries[]`, and committed/versioned.
- query-plane minimal suite replays these canonical fixtures directly via test APIs.

#### B. CDC-to-canonical translator in test harness

Add a translator layer in setup utilities:

- Input: `canal-json-with-tidb-extension` fixture records (or table-driven synthetic records).
- Output: canonical replay batches containing one collection/stream envelope plus canonical `MutationEntry entries[]` (test-only facade JSON may wrap `MutationBatch.collection`, `MutationBatch.stream_identity`, and `entries[]`, but the canonical replay shape remains batch-scoped collection/stream metadata plus entry-scoped payloads).

Minimal entry set required for this suite:

- `ROW_MUTATION` (upsert/delete; entry payload carries canonical `doc_id`, `commit_ts`, and mutation payload, while `namespace_id` / `collection_id` live on the owning batch envelope and `source_partition` / `source_epoch` live on the owning stream identity).
- `PROGRESS_MARK` (advance apply/progress watermark deterministically).

`PROGRESS_MARK` handling rule:
- must update modality `applied_ts` first, then derive `safe_ts = min(enabled modality applied ts, including metadata_aux only when that optional modality is present)`.
- test harness is not allowed to set `safe_ts` directly through normal progress commands; only explicit `POST /__test__/fault_profile` per-modality applied-ts overrides may be used for deterministic fault injection.

Optional command set (only when needed by a specific case):

- `BARRIER_MARK` (explicit ordering barrier for multi-step assertions).

#### C. Test API for changefeed-equivalent replay

Add a dedicated replay endpoint:

4. `POST /__test__/inject_changefeed_batch`
   - Input: ordered test-only envelope carrying canonical `entries[]`; the outer envelope is a facade contract, not raw canonical `MutationBatch` JSON.
   - Behavior: execute through the same admission path as canonical ingest replay, but without CDC runtime dependency.
   - Historical replay, idempotency, and DDL-boundary cases MUST include explicit `batch_id`, `stream_identity`, and `schema_generation_token`; omission of `schema_generation_token` is reserved only for synthetic active-generation fixtures, where emulator deterministically resolves the token from the current collection catalog snapshot before normalization.

Example:

```json
{
  "collection_ref": {"namespace_id": 1001, "collection_id": 42},
  "batch_id": "case-cdc-1",
  "stream_identity": {"source_partition": "table:9001", "source_epoch": 1},
  "schema_generation_token": "g_17",
  "entries": [
    {"source_seq": 1, "row_mutation": {"doc_id": {"u64": 10}, "commit_ts": 1000, "upsert": {"schema_version": 17, "fields": {"title": {"atomic_value": {"string_value": "v1"}}}}}},
    {"source_seq": 2, "row_mutation": {"doc_id": {"u64": 10}, "commit_ts": 1050, "delete": {}}},
    {"source_seq": 3, "progress_mark": {"resolved_ts": 1050, "watermarks": {"metadata_search_ts": 1050, "semantic_search_ts": 1050}}}
  ]
}
```

#### D. What this solves and what it does not

Solved for this suite:

- deterministic replay of CDC-equivalent data + progress instructions.
- full query-plane verification without TiCDC process startup.

Not solved (kept in separate suite):

- TiCDC runtime behavior itself (changefeed scheduling, sink retries, network partitions).
- end-to-end CDC transport guarantees.

### 7.2.2 Fast Fixture Production for Frequently Changing Cases

When test cases evolve frequently, fixture production must be incremental and automated instead of manual full re-record.

#### A. CaseSpec as the single source of capture intent

Define one `CaseSpec` per test case (YAML/JSON), including:

- target collections and filter rules,
- DDL + DML scenario script,
- whether canonical `PROGRESS_MARK` / `BARRIER_MARK` entries are required,
- expected capture window and completion criteria.

Adding a new case should primarily mean adding/editing one `CaseSpec`, not editing recorder internals.

#### B. Capture-once, replay-many baseline

Keep one real TiCDC capture lane for fixture generation only:

1. start from `start_ts`,
2. run scenario script,
3. wait until `checkpoint-ts/resolved-ts` passes scenario max `commit_ts`,
4. export raw sink output,
5. convert to canonical replay fixture (`entries[]`).

Daily query-plane tests remain CDC-free and only consume replay fixtures.

#### C. Persistent benchmark environment for speed

Use a long-lived fixture-generation cluster (PD/TiKV/TiDB/TiCDC) instead of per-case full boot:

- create changefeed per case,
- execute case SQL,
- wait drain condition,
- export and remove changefeed.

This removes repeated cold-start cost and makes fixture generation fast enough for frequent case updates.

#### D. Incremental regeneration (changed-cases-only)

Compute a stable hash from `CaseSpec` + scenario SQL + conversion options:

- if hash unchanged: reuse existing fixture,
- if hash changed: regenerate only that case.

Do not re-record all fixtures when only a subset of cases changed.

#### E. Deterministic normalization and verification

Normalize converted fixtures before commit:

- preserve canonical batch framing first (`ROW_MUTATION` groups before trailing `PROGRESS_MARK` / `BARRIER_MARK` entries within one admitted batch),
- never globally sort across batch boundaries or move `PROGRESS_MARK` / `BARRIER_MARK` outside their owning admitted batch,
- within one admitted batch's row-entry section only, preserve one canonical row-entry order and then assign `source_seq` from that order; batch-level `namespace_id` / `collection_id` and `stream_identity.{source_partition,source_epoch}` remain fixed envelope fields rather than per-entry sort keys,
- remove run-specific noise fields,
- keep semantic fields required for replay.

Verify determinism:

- record/convert same case twice,
- normalized outputs must be byte-identical.

Non-deterministic fixture output must fail fixture generation pipeline.

#### F. Two-lane execution model

- PR lane:
  - generate/verify changed cases only (fast feedback).
- Nightly lane:
  - broader or full fixture re-record verification against baseline (drift detection).

This keeps day-to-day iteration fast while preserving long-term fixture quality.

#### G. Recommended command contract

```bash
make cdc-fixture-record CASE=<case_name>
make cdc-fixture-convert CASE=<case_name>
make cdc-fixture-verify CASE=<case_name>
make cdc-fixture-update CHANGED_ONLY=1
```

The exact implementation may vary, but the contract should preserve:

- per-case capture,
- deterministic conversion,
- incremental regeneration.

Test-only API guardrails:

- Disabled by default in production build/runtime.
- Enabled only under test profile flag (for example `--enable-test-apis`).
- Namespace/collection access restricted to test-owned IDs.

### 7.3 Deterministic Startup Sequence (No CDC)

1. Start Emulator with `--profile=query_plane_min --enable-test-apis`.
2. Wait on health/readiness endpoint (bounded polling).
3. Create or load test collection descriptors (fixture or setup RPC).
4. Replay canonical CDC-derived fixtures via `inject_changefeed_batch` (or use `inject_writes` for synthetic cases).
5. Wait until readiness reports `safe_ts >= max(commit_ts)` after canonical `PROGRESS_MARK` application (or equivalent `inject_progress` update for a synthetic case).
6. Run TiDB SQL assertions with explicit `read_ts`.

### 7.4 CI Pipeline Shape for the CDC-Free Suite

1. `setup-min-env`: boot Emulator only.
2. `contract-mode`: run with fault profiles (`429/503/timeout`) enabled.
3. `normal-mode`: run deterministic snapshot and logical-query-planning tests.
4. `teardown`: assert zero leaked workers/connections, stop the test runtime.

This keeps PR feedback fast while preserving a separate nightly full-chain CDC suite.

### 7.5 Failure Isolation Rules in Minimal Environment

- If setup API fails, fail test at setup phase; do not execute SQL assertions.
- If readiness cannot reach target `safe_ts` within deadline, fail with state dump (`safe_ts`, per-modality applied ts).
- Each test case uses isolated `(namespace_id, collection_id)` to avoid shared-state interference.

### Local startup contract

```text
TiDB test process
  -> Emulator runtime
  -> Emulator test facade endpoints (same process or embedded sidecar)
```

### CI stages

1. Start Emulator.
2. Health check test-runtime endpoints.
3. Run short-circuited query-plane integration suite.
4. Publish request/response traces and failure artifacts.

### Environment goals

- Startup in seconds, not minutes.
- No TiCDC, no MinIO dependency for query-plane-only suite.
- Deterministic collection/namespace isolation per job shard.

## 8. Contract Testing and Boundary Fault Injection

## 8.1 Emulator Contract Boundary

Use Emulator as the single contract substrate for this suite; do not introduce a second query-RPC stub with divergent semantics.

Fault-path coverage is injected through Emulator test surfaces:

- deterministic response/failure scripts via `POST /__test__/fault_profile`
- configurable failure injection:
  - `429 SF_RATE_LIMITED`
  - `503 SF_BACKEND_UNAVAILABLE`
  - deadline-exceeded / timeout
- deterministic pagination/search-context lifecycle checks:
  - malformed or empty `continuation_page.next_page_token`
  - wrong-lineage or wrong-page-ordinal token reuse rejected fail-closed
  - PIT issuance via `fetch_spec.search_context.keep_alive_ms`, with present `keep_alive_ms=0` rejected fail-closed
  - admitted PIT issuance returns `search_context.expires_at_unix_ms`
  - continuation-time keep-alive extension refreshes `search_context.expires_at_unix_ms` coherently with emulator search-context TTL policy
  - first-page requests carrying `search_context.token` rejected fail-closed
  - unset/invalid `page_window` branch combinations rejected fail-closed
  - composite continuation requiring valid `after_key` plus matching `composite_page.next_page_token`
  - `StreamQueryRequest` shape excludes search-context fields and aggregation carriers entirely; emulator conformance rejects any non-canonical attempt to introduce unary-only PIT controls or aggregation-bearing stream payloads on the stream path
  - expired `search_context.token` on query continuation rejects fail-closed
  - there is no public early-close branch to test; PIT-equivalent search-context retention ends only by natural expiry

- deterministic query pre-admission checks:
  - oversized query payload rejected as `413 SF_PAYLOAD_TOO_LARGE` with canonical payload-budget details
  - `InListPredicate.values.values[]` over `max_in_list_elements` rejected fail-closed with canonical operation-budget details
  - predicate-tree node count over `max_predicate_ast_nodes` rejected fail-closed with canonical operation-budget details
- every error response MUST carry non-empty `ErrorStatus.correlation_id`; when request metadata supplied `RequestMeta.correlation_id`, the error envelope must echo that value, otherwise the service must emit its own non-empty canonical correlation identifier
- readiness/visibility overrides remain collection-scoped emulator state, not stub-local behavior

## 8.2 TiDB behavior expectations

For **auxiliary acceleration** path:

- fail-open to TiDB native plan on Search RPC failure.
- query still succeeds if TiDB can execute fallback path.
- in strict mode, fallback is mandatory whenever Search Fabric cannot admit the exact requested snapshot; if the response carries `effective_read_ts`, any value below requested `read_ts` must be coherent with that fallback decision.

For **search-native collection** path:

- fail-closed with explicit error propagation.
- in strict mode, inability to admit the exact requested snapshot must return `SF_VISIBILITY_TIMEOUT`; if the response carries `effective_read_ts`, any value below requested `read_ts` must be coherent with that error outcome.

## 8.3 Resource-release assertions

After injected failures, verify request pipeline cleanup:

- cop iterator close must cancel outstanding RPCs and wait workers:
  - `pkg/store/copr/coprocessor.go:2746-2753`
- batch cop iterator close must do the same:
  - `pkg/store/copr/batch_coprocessor.go:1385-1389`

Additional observability hooks:

- primary: Search Fabric client request/response hooks at `QueryGatewayService.ExecuteQuery` boundary.
- compatibility (pre-cutover only): request inspect hook `pkg/distsql/distsql.go:64-66`.
- compatibility (pre-cutover only): cop send hook via failpoint pattern:
  - `pkg/executor/distsql_test.go:623-637`

## 9. Deflake Engineering Standard

## 9.1 Forbidden in this suite

- Any correctness gating via `time.Sleep(...)`.
- Hard-coded fixed long waits for ingest/readiness.
- Shared mutable global fixtures across test cases.
- Assertions that depend on log timing order without state confirmation.

## 9.2 Required patterns

1. State polling with explicit deadline and tick.
2. Event/callback hooks for request inspection.
3. Explicit timestamp timeline (`commit_ts`, `read_ts`) in each case.
4. Per-test isolated identifiers and deterministic cleanup.
5. Structured failure assertions (error code + fallback path + resource release).

## 9.3 Generic test hooks

Minimal hook interfaces:

```go
type SearchQueryRequestHook func(req *searchfabricv1.QueryRequest)
type SearchQueryResponseHook func(resp *searchfabricv1.QueryResponse, err error)
type VisibilityHook func(namespaceID, collectionID uint64, readiness ReadinessSnapshot)
type FaultProfile struct {
    NamespaceID           uint64
    CollectionID          uint64
    Code                  string // SF_RATE_LIMITED, SF_BACKEND_UNAVAILABLE, SF_QUERY_TIMEOUT
    RetryAfter            time.Duration
    ForceAppliedTSMetadataSearch *uint64 // optional
    ForceAppliedTSSemanticSearch *uint64 // optional
    ForceAppliedTSMetadataAux    *uint64 // optional; nil keeps metadata-aux derived/absent state
    ClearForcedVisibility        bool
}
```

## 10. Reference Test Flow (Go Pseudocode)

```go
func TestQueryPlane_ShortCircuit_MVCCSnapshot(t *testing.T) {
    store := realtikvtest.CreateMockStoreAndSetup(t)
    tk := testkit.NewTestKit(t, store)
    tk.MustExec("use test")

    // 1) Allocate isolated identifiers for this test case.
    nsID := uint64(1001001)
    collectionID := uint64(42001)

    // 2) Build deterministic commit timeline.
    // doc=10: v1@1000 -> v2@1020 -> tombstone@1050
    writes := []WriteEnvelope{
        {NamespaceID: nsID, CollectionID: collectionID, DocID: 10, CommitTS: 1000, Op: Upsert, Metadata: map[string]any{"title": "v1"}},
        {NamespaceID: nsID, CollectionID: collectionID, DocID: 10, CommitTS: 1020, Op: Upsert, Metadata: map[string]any{"title": "v2"}},
        {NamespaceID: nsID, CollectionID: collectionID, DocID: 10, CommitTS: 1050, Op: Delete},
    }

    // 3) Short-circuit setup: inject directly into Emulator's canonical ingest facade.
    require.NoError(t, sfTestClient.InjectWrites(context.Background(), WriteBatch{
        CollectionRef: CollectionRef{NamespaceID: nsID, CollectionID: collectionID},
        BatchID:       "case-mvcc-basic-1",
        StreamIdentity: StreamIdentity{
            SourcePartition: "__test__/direct/case-mvcc-basic-1",
            SourceEpoch:     1,
        },
        Writes: writes,
    }))
    require.NoError(t, sfTestClient.InjectProgress(context.Background(), ProgressBatch{
        NamespaceID:              nsID,
        CollectionID:             collectionID,
        AppliedTSMetadataSearch:  1050,
        AppliedTSSemanticSearch:  1050,
    }))

    // 4) Deterministically wait for readiness state derived from applied_ts.
    require.Eventually(t, func() bool {
        rd, err := sfTestClient.GetReadiness(context.Background(), ReadinessRequest{
            Meta: RequestMeta{
                RequestID: "test-readiness-case-1",
                ClientID:  "tidb-itest",
                ClientKind: CLIENT_KIND_TIDB_COMPUTE,
            },
            Collection: CollectionRef{NamespaceID: nsID, CollectionID: collectionID},
        })
        if err != nil {
            return false
        }
        if rd.GetSuccess() == nil {
            return false
        }
        watermarks := rd.GetSuccess().GetWatermarks()
        return watermarks.GetMetadataSearchTs() >= 1050 &&
            watermarks.GetSemanticSearchTs() >= 1050 &&
            watermarks.GetSafeTs() >= 1050
    }, 5*time.Second, 20*time.Millisecond)

    // 5) Assert SQL->Search RPC translation at the seam contract boundary.
    var capturedReq *searchfabricv1.QueryRequest
    sfTestClient.SetExecuteQueryRequestHook(func(req *searchfabricv1.QueryRequest) {
        capturedReq = proto.Clone(req).(*searchfabricv1.QueryRequest)
    })

    // 6) Snapshot checks.
    // read_ts=1010 -> v1 visible
    setReadTS(tk, 1010)
    tk.MustQuery("SELECT doc_id FROM sf_aux_idx_t WHERE ...").Check(testkit.Rows("10"))
    require.NotNil(t, capturedReq)
    require.EqualValues(t, nsID, capturedReq.Collection.NamespaceId)
    require.EqualValues(t, collectionID, capturedReq.Collection.CollectionId)
    require.EqualValues(t, searchfabricv1.IntegrationType_INTEGRATION_TYPE_AUX_INDEX, capturedReq.IntegrationType)
    require.EqualValues(t, 1010, capturedReq.Consistency.ReadTs)
    require.NotNil(t, capturedReq.Meta)
    require.NotEmpty(t, capturedReq.Meta.RequestId)
    require.NotEmpty(t, capturedReq.Meta.ClientId)
    require.Equal(t, searchfabricv1.ClientKind_CLIENT_KIND_TIDB_COMPUTE, capturedReq.Meta.ClientKind)

    // read_ts=1030 -> v2 visible
    setReadTS(tk, 1030)
    tk.MustQuery("SELECT doc_id FROM sf_aux_idx_t WHERE title='v2'").Check(testkit.Rows("10"))

    // read_ts=1060 -> tombstoned, invisible
    setReadTS(tk, 1060)
    tk.MustQuery("SELECT doc_id FROM sf_aux_idx_t WHERE doc_id=10").Check(testkit.Rows())

    // strict-mode stale snapshot handling:
    // if strict-read readiness cannot advance safe_ts to the requested snapshot, aux path must fallback.
    forcedAppliedTS := uint64(1000)
    require.NoError(t, sfTestClient.SetFaultProfile(context.Background(), FaultProfile{
        NamespaceID:                   nsID,
        CollectionID:                  collectionID,
        ForceAppliedTSMetadataSearch:  &forcedAppliedTS,
        ForceAppliedTSSemanticSearch:  &forcedAppliedTS,
    }))
    setReadTS(tk, 2000)
    tk.MustQuery("SELECT doc_id FROM sf_aux_idx_t WHERE title='v2'").Check(testkit.Rows("10")) // result from fallback plan
    require.Greater(t, getFallbackCounter(t, "strict_snapshot_not_met"), int64(0))
}
```

## 11. Rollout Plan for the New Suite

1. Add new query-plane-only suite first; keep existing full-chain CDC suite as separate coverage layer.
2. Move current sleep-based query assertions to short-circuited mode.
3. Keep one minimal CDC smoke case to validate wiring, not functional breadth.
4. Gate CI on short-circuited suite for fast feedback; run full-chain suite as nightly/regression.

## 12. Expected Outcome

- Significant runtime reduction for query-plane integration tests.
- Near-elimination of sleep-induced flakes.
- Better fault localization (query translation/visibility/pagination/search-context/fallback issues are isolated).
- Production-aligned confidence on read-path correctness under MVCC, pagination/search-context lifecycle, and failure scenarios.