# NexusCRM E2E API Tests

## Overview

This directory contains end-to-end API tests for NexusCRM. The tests validate the backend API against a shared TiDB Cloud database.

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    E2E API Test Runner                      │
│                   (tests/e2e-api/)                         │
│                                                              │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐ │
│  │  runner.py   │───▶│   api.py     │───▶│  Backend     │ │
│  │  (Test Exec) │    │  (HTTP Client)│    │  (localhost) │ │
│  └──────────────┘    └──────────────┘    └──────────────┘ │
│         │                   │                    │           │
│         │                   │                    │           │
│         ▼                   ▼                    ▼           │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐  │
│  │    db.py     │    │  TiDB Cloud │◀───│  Port 43001  │  │
│  │ (DB Access)  │───▶│  (nexuscrm) │    │              │  │
│  └──────────────┘    └──────────────┘    └──────────────┘  │
└─────────────────────────────────────────────────────────────┘
```

## Database Configuration

The test suite uses **TiDB Cloud** as the database backend. Database connection is configured via environment variables.

### Environment Variables

Both `DB_*` and `TIDB_*` prefixes are supported:

```bash
# Using TIDB_* prefix (recommended)
export TIDB_HOST=gateway01.us-east-1.prod.aws.tidbcloud.com
export TIDB_PORT=4000
export TIDB_USER=4L9XRdFL5mC7PSr.root
export TIDB_PASSWORD=Admin123!
export TIDB_DATABASE=nexuscrm

# Using DB_* prefix (equivalent)
export DB_HOST=<tidb-host>
export DB_PORT=4000
export DB_USERNAME=<tidb-user>
export DB_PASSWORD=<tidb-password>
export DB_DATABASE=nexuscrm

# Authentication
export NEXUSCRM_PASSWORD=<admin-password>
```

### Local .env File

For local development, create a `.env` file in the project root:

```bash
TIDB_HOST=gateway01.us-east-1.prod.aws.tidbcloud.com
TIDB_PORT=4000
TIDB_USER=4L9XRdFL5mC7PSr.root
TIDB_PASSWORD=Admin123!
TIDB_DATABASE=nexuscrm
NEXUSCRM_PASSWORD=Admin123!
```

## Running Tests Locally

### Using Make

```bash
make test-e2e-api
```

By default this runs the current `gate-core` scenes:

```bash
demo,validation,contract,quote_line
```

### With Custom Environment

```bash
TIDB_HOST=<host> \
TIDB_USER=<user> \
TIDB_PASSWORD=<password> \
TIDB_DATABASE=nexuscrm \
NEXUSCRM_PASSWORD=<password> \
make test-e2e-api
```

### With Parallel Execution

```bash
E2E_API_PARALLEL=8 make test-e2e-api
```

Available parallel modes:
- `E2E_API_PARALLEL=1` - Sequential execution (default)
- `E2E_API_PARALLEL=8` - 8 concurrent workers

### Run Specific Scenes

```bash
cd tests/e2e-api
python3 runner.py --scene demo,quote,validation
```

To run the current `gate-extended` scenes explicitly:

```bash
E2E_API_SCENES=quote,order,order_line,lead_status_timestamp make test-e2e-api
```

### Run Single Case

```bash
cd tests/e2e-api
python3 runner.py --case ORD-003
```

### Runner Options

```bash
python3 runner.py --help

Options:
  --scene SCENE       Run only cases in this scene or comma-separated scene list
  --case CASE_ID      Run only this specific case ID
  --skip-setup        Skip seed data setup
  --parallel N        Number of cases to run in parallel (default: 1)
  --fail-on-failures  Exit with code 1 when any case fails or errors
  --format FORMAT     Output format: json, junit, or both (default: json)
```

## CI/CD Integration

### GitHub Actions Workflow

The E2E API tests run in CI via `.github/workflows/ci.yml` in the `api-tests` job.

#### Configuration

1. **Database**: Uses GitHub Environment `ci-test` with secrets/vars:
   - `TIDB_HOST`
   - `TIDB_USER`
   - `TIDB_PASSWORD`
   - `TIDB_DATABASE`

2. **Authentication**:
   - `NEXUSCRM_PASSWORD` or `NEXUS_ADMIN_PASSWORD`

#### Manual Trigger (workflow_dispatch)

The CI supports manual triggering with selectable test modes:

1. Go to **Actions** → **CI** → **Run workflow**
2. Select `test_mode`:
   - `all` - Run all tests (default)
   - `api-only` - Only E2E API tests
   - `backend-only` - Backend unit tests + API tests
   - `frontend-only` - Frontend tests only
   - `sentry-only` - Sentry tests only

#### Local vs CI Comparison

| Aspect | Local | CI |
|--------|-------|-----|
| Database | `.env` file | GitHub Environment `ci-test` |
| Backend | Auto-started on port 43001 | Auto-started on port 43001 |
| Parallel | `E2E_API_PARALLEL=N` | `E2E_API_PARALLEL=4` |
| Test Cases | Same `cases/` directory | Same `cases/` directory |
| Database Host | Same TiDB Cloud instance | Same TiDB Cloud instance |

**Important**: Both local and CI connect to the **same TiDB Cloud database** (`nexuscrm`). Test data may be shared.

Current PR-gate split:

- `gate-core` required: `demo,validation,contract,quote_line`
- `gate-extended` non-blocking: `quote,order,order_line,lead_status_timestamp`

## Test Cases

### Directory Structure

```
cases/
├── automation/     # Flow/automation regression tests
├── demo/           # Basic smoke tests
├── quote/          # Quote-related scenarios
├── order/          # Order creation and validation
├── order_line/     # Order line item operations
├── contract/        # Contract management
├── quote_line/      # Quote line items
├── approval/        # Approval flow tests
├── validation/      # Input validation tests
├── lead_convert/    # Lead conversion
├── hubspot/         # HubSpot integration
└── sdr/            # SDR-specific scenarios
```

### Case Format

Each test case is a JSON file:

```json
{
  "id": "ORD-003",
  "name": "Order Status should have default value",
  "scene": "order",
  "object": "Order",
  "method": "POST",
  "endpoint": "/services/data/v63.0/sobjects/Order",
  "setup": [
    {
      "type": "sql",
      "sql": "INSERT INTO Account ...",
      "extract": { "account_id": "Id" }
    }
  ],
  "payload": {
    "AccountId": "{account_id}",
    "Status": "Draft"
  },
  "expects": { "status_code": 201 },
  "query_after": {
    "object": "Order",
    "retry": 15,
    "delay_ms": 2000,
    "assertions": [
      { "field": "Status", "operator": "is_not_null" }
    ]
  },
  "teardown": [
    { "type": "sql", "sql": "DELETE FROM Order WHERE ..." }
  ],
  "skip": false,
  "priority": "P0",
  "bug_ref": "#1234"
}
```

### Assertion Operators

| Operator | Description |
|----------|-------------|
| `eq` / `equals` | Exact match |
| `ne` | Not equal |
| `gt` | Greater than |
| `contains` | String contains substring |
| `not_contains` | String does not contain |
| `is_null` | Field is null |
| `is_not_null` | Field is not null |

### Variable Substitution

Cases support variable substitution:

- `{seed_Account_0}` - From seed_data.json
- `{account_id}` - Extracted from setup step
- `{order_id}` - Extracted from API response

### Async AfterSave Assertions

AfterSave flows and other outbox-driven side effects are asynchronous. For
assertions that depend on the worker, always use `query_after.retry` with
`delay_ms`; do not add fixed `time.sleep(...)` calls before checking the final
record state. A fixed sleep samples one timing point and can fail before the
worker has picked up the outbox event, creating a false regression.

Use this pattern for AfterSave / outbox-driven assertions:

```json
"query_after": {
  "object": "Invoice_Collection__c",
  "retry": 30,
  "delay_ms": 2000,
  "assertions": [
    { "field": "Marketplace__c", "operator": "eq", "value": "r2224-marketplace" }
  ]
}
```

## Troubleshooting

### Tests Hang / Timeout

If tests hang indefinitely, check:
- Database connection is accessible
- Backend service is running on port 43001
- Network connectivity to TiDB Cloud

HTTP requests have a **30-second timeout** to prevent indefinite hanging.

### 401 Unauthorized

- Backend running but session expired
- Runner auto re-authenticates
- If persists: check credentials in environment

### 429 Rate Limit

- Too many requests to backend
- API client automatically retries up to 3 times
- Each retry waits based on `retry_after` field (default 2 seconds)
- If still failing after 3 retries, returns error

To reduce rate limiting:
- Reduce parallel workers: `E2E_API_PARALLEL=1`

### 404 on query_after

- Seed data not written to DB
- Check setup step output for errors
- Run with `E2E_API_SKIP_SETUP=0` to reset

### Pass Rate Suddenly Low

Database may have residual data. Either:
1. Run with `E2E_API_SKIP_SETUP=0` to reset
2. Manually truncate tables:

```bash
mysql -h <host> -P 4000 -u <user> -p'<password>' nexuscrm \
  -e "TRUNCATE TABLE Lead; TRUNCATE TABLE Account; TRUNCATE TABLE Opportunity;"
```

## Test Data Mechanism

Each run starts with `_setup_seed_data()`:

1. **Truncate** core tables (Account, Lead, Opportunity, Quote, Order, etc.)
2. **Insert** records from `seed_data.json` in dependency order
3. **Substitute** `{seed_*}` variables with real database IDs

### Seed Data

`seed_data.json` contains records for:
- Account (10 records)
- Lead (10 records)
- Opportunity (10 records)
- Quote (5 records)
- Contract (5 records)
- Pricebook2 / Product2 (price book entries)

Uses Salesforce-style IDs (e.g., `00Q5g...`). On insert, backend auto-generates UUIDs.

## Directory Structure

```
tests/e2e-api/
├── runner.py              # Main test runner
├── api.py                 # HTTP client with auth
├── db.py                 # Database helper
├── config.py              # Environment config
├── seed_data.json        # Seed test data
├── generate_report.py     # Report parser
├── cases/                 # Test case JSON files
│   ├── demo/
│   ├── automation/
│   ├── quote/
│   ├── order/
│   ├── order_line/
│   ├── contract/
│   ├── quote_line/
│   ├── approval/
│   ├── validation/
│   └── ...
└── reports/              # Generated test reports
```

## Credentials

| Purpose | Value |
|---------|-------|
| Backend URL | `http://127.0.0.1:43001` (local) |
| OAuth client_id | `test-gosf-client` |
| OAuth client_secret | `test-gosf-secret-123` |
| Test User | `admin@test.com` |
| Test Password | Set via `NEXUSCRM_PASSWORD` |

## Case Repair Playbook

For guidance on fixing failing tests, see [case-repair-playbook.md](./case-repair-playbook.md).

## Frontend Probes

For frontend-aligned API probes (mirrors real frontend requests), see `frontend_case_probe.py`.