# QA Regression Scripts

Reproducible regression tests for jieni's 8-story QA scope (#732 / #730 / #1331-#1336). Each AC is one shell script with **machine-checkable verdict** (single-line JSON output). Run between staging builds; diff results to detect real changes.

## Why this exists

Prior rounds (R2-R6) ran ad-hoc bash that wasn't version-controlled. Across-round comparison was unreliable because:

1. Fixtures weren't persistent (each round built fresh, with subtle differences)
2. Test scripts weren't in git (couldn't reproduce R4 from R6)
3. Verdicts were subjective (PASS/PARTIAL/FAIL judged on the fly)
4. Staging drifts daily (couldn't attribute behavior changes to specific PRs)

This framework addresses (1)-(3). Staging drift (4) is monitored via the snapshot in `common/lib.sh::record_env_snapshot`. See [issue #1439 errata](https://github.com/pingcap-inc/nexuscrm-pingcap-pack/issues/1439#issuecomment-4465145398) for the full methodology audit.

## Layout

```
qa/regression/
├── README.md                      this file
├── common/
│   ├── lib.sh                     emit_verdict, env_snapshot, fixture helpers
│   ├── auth.sh                    get admin/sales tokens
│   ├── setup_fixture.sh           idempotent fixture build (sales user, min profile, test accounts)
│   └── teardown_fixture.sh        cleanup fixture (only if owner=this run)
├── r6/                            R6 baseline (29 ACs, 8 stories ex-#780)
│   ├── 732_AC1_soql_rls_filter.sh
│   ├── 732_AC4_button_hide.sh
│   ├── 730_AC2_perm_cascade.sh
│   ├── 1333_AC1_sr_no_criteria.sh
│   ├── 1335_AC3_rt_post.sh
│   └── ...                        (29 total — initial set covers ~5; expand over rounds)
├── run_all.sh                     runs all scripts, aggregates JSON to results.jsonl
└── results/                       per-run outputs (gitignored except sample)
    └── <build>_<timestamp>/
        ├── env_snapshot.json
        ├── results.jsonl
        └── diff_vs_<prior_run>.md
```

## How to run

```bash
# Single AC
./r6/732_AC1_soql_rls_filter.sh

# All ACs, output to timestamped dir
./run_all.sh

# Diff against prior run
./diff_runs.sh results/v17787_20260516/ results/v17787_20260517/
```

## Script contract

Every test script MUST:

1. **Header comment block** with:
   - Story + AC number + canonical AC text (copy-paste from issue body)
   - PASS criteria (machine-checkable boolean expression)
   - FAIL conditions
   - BLOCKED conditions (when test can't run at all)

2. **Source `common/lib.sh`** for `emit_verdict()` and fixture helpers

3. **Idempotent**: re-running gives the same result (or detects fixture drift and BLOCKEDs out)

4. **No leftover state**: cleanup or use shared persistent fixtures from `common/setup_fixture.sh`

5. **Output exactly one JSON line at end** via `emit_verdict`:
   ```json
   {"story":"#732","ac":"AC1","verdict":"PASS","build":"v17787...","ts":"2026-05-16T01:00:00Z","evidence":"sales sees 1 (own), 0 (hidden)"}
   ```

   Verdict values: `PASS` / `FAIL` / `BLOCKED` / `INCONCLUSIVE`. **NO `PARTIAL`** — if you can't decide, the criteria isn't tight enough; tighten or split the AC.

## Environment

Scripts read `HOST`, `ADMIN_USER`, `ADMIN_PW`, `CLIENT_ID`, `CLIENT_SECRET` from env. Defaults match staging `d2ivgwliouoia2.cloudfront.net` + `nexuscrm-ui` client. Override for different envs:

```bash
HOST=https://my-other-env.example.com ./run_all.sh
```

## Relationship to `staging-inspection/`

This directory is **complementary, not competitive** with the canonical Staging Inspection framework:

- **`staging-inspection/`** — CI-driven, runs against prod-mirror environments, narrow coverage of high-confidence assertions, fail-loud
- **`qa/regression/`** (this dir) — manually triggered, broad coverage of jieni's 8-story scope, includes flaky/aspirational ACs, fail-soft (BLOCKED/INCONCLUSIVE allowed)

**Promotion path**: as a script here stabilizes (3+ runs identical verdict against known builds), port it to `staging-inspection/` so it runs in CI. This `qa/regression/` dir is the staging area; Staging Inspection is the production lane.

See [pack#1356](https://github.com/pingcap-inc/nexuscrm-pingcap-pack/issues/1356) for the broader fixture pipeline work this complements.

## Round provenance

- **R6** (2026-05-16) — initial framework + first 5 example scripts. Reference: [pack#1439](https://github.com/pingcap-inc/nexuscrm-pingcap-pack/issues/1439).