# PR Gate E2E Fast And Stable Roadmap

## Goal

Optimize `CI -> E2E API Stable Scenes` for pull requests.

Target:

- runtime stays within `6~8` minutes
- materially fewer hangs / fake hangs / flaky reds
- keep most existing stable-scene coverage, but allow high-volatility scenes to move out of the core gate

Non-goals for this phase:

- full per-run isolated TiDB branch/database provisioning
- broad test-product redesign outside PR gate E2E
- eliminating every flaky case before improving observability and guardrails

## Current Problems

Observed issues on the shared `ci-test` path:

- shared remote TiDB state causes cross-run and cross-case interference
- `apply-metadata` / `apply-tenant` add avoidable latency and write amplification
- the runner can sit in a long-running step without quickly surfacing which case is stuck
- high-volatility scenes such as `order`, `quote`, and `lead_status_timestamp` can dominate gate instability
- when the job hangs, current logs are too weak to quickly answer:
  - which case is active
  - whether the backend is unhealthy or only slow
  - whether the DB is blocked, dirty, or drifting

## Design Principles

1. Stabilize before broadening
2. Fail fast instead of hanging
3. Keep shared `ci-test` single-tenant at execution time
4. Skip expensive apply steps when baseline artifacts are already present
5. Separate core gate coverage from high-volatility extended coverage

## Recommended Approach

Use a staged path:

1. add fail-fast + observability to the E2E runner
2. serialize PR-gate use of shared `ci-test`
3. keep `verify-first, fallback-apply` for platform + tenant setup
4. split scenes into `gate-core` and `gate-extended`
5. fix high-volatility scenes one cluster at a time
6. only consider per-run isolated DB branches if the above still cannot meet the target

This gives the best speed/risk balance for the current repository and infrastructure.

## Architecture Changes

### 1. Runner Guardrails

Add explicit execution guardrails to `tests/e2e-api/runner.py` and the launcher path:

- per-case timeout, default `60~90s`
- per-scene timeout
- active-case heartbeat logging under parallel execution
- timeout diagnostics including:
  - case id
  - recent request
  - recent `query_after`
  - recent teardown step
  - backend health result

Expected effect:

- hanging jobs convert into bounded, diagnosable failures
- GitHub Actions logs reveal the stuck case quickly

### 2. Shared DB Single-Tenant Execution

Protect the shared `ci-test` DB with workflow `concurrency` so PR gate E2E jobs do not overlap.

Recommended policy:

- one concurrency group for every job that mutates or depends on shared `ci-test`
- `cancel-in-progress: false`

Expected effect:

- fewer cross-run collisions
- fewer shared-state false failures

### 3. Verify-First Environment Prep

Keep the environment fast-path:

- `ci-test-db-refresh.yml`
  - verify platform baseline
  - only run `apply-metadata` on verify failure
- `ci.yml`
  - verify tenant baseline
  - only run `apply-tenant` on verify failure

Expected effect:

- less setup latency
- less write amplification against shared `ci-test`
- less drift introduced by repeated unnecessary apply passes

### 4. Gate Layering

Split PR-gate scenes into two buckets.

Suggested initial cut:

- `gate-core`
  - `demo`
  - `validation`
  - `contract`
  - `quote_line`
- `gate-extended`
  - `quote`
  - `order`
  - `order_line`
  - `lead_status_timestamp`

Rule:

- PR required gate uses `gate-core`
- `gate-extended` remains visible but isolated from the core gate while stabilization is ongoing

Expected effect:

- faster and more stable merge gate
- preserve broader signal without making it block every PR immediately

### 5. High-Volatility Scene Cleanup

Prioritize targeted cleanup for:

- `order`
- `quote`
- `lead_status_timestamp`

Likely work includes:

- remove fixed-id collisions
- tighten setup/teardown symmetry
- reduce cross-case coupling
- verify async / flow-dependent timing assumptions
- isolate cases that still need serial execution

## Execution Plan

### Milestone 1: Stop Hangs

Scope:

- add per-case timeout
- add active-case heartbeat logging
- add timeout diagnostics

Success criteria:

- no PR-gate E2E run hangs indefinitely
- a stuck run identifies the active case from logs

### Milestone 2: Serialize Shared `ci-test`

Scope:

- add workflow concurrency around PR-gate `ci-test` usage

Success criteria:

- no overlapping PR-gate jobs use shared `ci-test`
- reduction in cross-run state collisions

### Milestone 3: Keep Fast Setup Path

Scope:

- preserve and validate `verify-first, fallback-apply`
- expose pre-verify/apply decisions in job summaries

Success criteria:

- healthy runs skip unnecessary apply steps
- failures still fall back safely

### Milestone 4: Split Gate-Core And Gate-Extended

Scope:

- move high-volatility scenes out of the required PR gate

Success criteria:

- required PR gate trends toward `6~8` minutes
- gate red rate drops materially

### Milestone 5: Stabilize Extended Scenes

Scope:

- repair high-volatility cases until they can be reconsidered for promotion back into the core gate

Success criteria:

- `quote`, `order`, `lead_status_timestamp` each have explicit root-cause fixes or documented containment

## Rollout Order

Recommended order:

1. Milestone 1
2. Milestone 2
3. validate fast-path setup behavior
4. Milestone 4
5. Milestone 5

## Risks

### Risk 1: Verify Too Narrow

If baseline verify misses required artifacts, setup may skip apply and push failures downstream.

Mitigation:

- keep final verify in place
- keep fallback apply path
- expand checks only when a missing dependency is observed in practice

### Risk 2: Core Gate Too Small

If too many scenes move out, PR protection may lose useful coverage.

Mitigation:

- keep `gate-extended` visible
- review regressions after each scene move

### Risk 3: Timeout Too Aggressive

Low timeouts can convert slow-but-valid runs into false reds.

Mitigation:

- start with moderate defaults
- tune from real timing data

## Metrics

Track:

- total PR-gate E2E wall time
- setup time before test execution
- number of runs exceeding `10` minutes
- flaky failure rate by scene
- top slowest cases
- percentage of runs skipping `apply-metadata`
- percentage of runs skipping `apply-tenant`

## Decision Log

Chosen strategy:

- optimize PR gate first
- target a balanced `fast + stable` outcome
- keep most coverage, but split out high-volatility scenes
- do not start with per-run isolated DB branches

## Next Step

Implementation should start with:

1. runner fail-fast and observability
2. shared `ci-test` concurrency gate
3. then gate-core / gate-extended split