# dataflow-service e2e logic

## Available Cases (Dedicated)

Cases are registered in `test/e2e/internal/case_driver/cases_registry_dedicated.go`.

### Changefeed (CDC) Cases

| Case Name | Cloud Provider | Description |
|-----------|---------------|-------------|
| kafka-changefeed-smoke | AWS/GCP/Azure | Kafka Changefeed smoke test |
| kafka-changefeed-kernel-configuration | AWS/GCP/Azure | Kafka kernel configuration test |
| kafka-private-link | AWS/GCP/Azure | Kafka private link test |
| kafka-test-connection | AWS/GCP/Azure | Kafka connection test |
| kafka-validate-changefeed-params | AWS/GCP/Azure | Kafka parameter validation |
| kafka-avro-changefeed | Azure | Kafka Avro format test |
| scale-changefeed | AWS/GCP/Azure | Changefeed scaling test |
| mysql-changefeed-smoke | AWS/GCP/Azure | MySQL source changefeed smoke test |
| s3-changefeed-smoke | AWS | S3 sink smoke test |
| serverless-changefeed-smoke | AWS | Serverless cluster changefeed test |
| gcs-changefeed-smoke | GCP | GCS sink smoke test |
| azure-blob-storage-changefeed-smoke | Azure | Azure Blob Storage sink test |
| pulsar-changefeed-smoke | AWS/GCP/Azure | Pulsar sink smoke test |
| validate-all-machine-type | AWS/GCP/Azure | Validate all machine types |
| cdc-recreate | AWS/GCP/Azure | CDC recreate test |
| changefeed-config-operations | AWS/GCP/Azure | Changefeed config operations |
| ticdc-cluster-config-operations | AWS/GCP/Azure | TiCDC cluster config operations |
| changefeed-open-api | AWS/GCP/Azure | Open API operations |

### Backup Cases

| Case Name | Cloud Provider | Description |
|-----------|---------------|-------------|
| backup-setting-smoke | AWS/GCP/Azure | Backup setting smoke test |
| snapshot-backup-smoke | AWS/GCP/Azure | Snapshot backup smoke test |
| snapshot-restore-smoke | AWS/GCP/Azure | Snapshot restore smoke test |
| pitr-backup-smoke | AWS/GCP/Azure | Point-in-time recovery backup test |
| pitr-restore-smoke | AWS/GCP/Azure | Point-in-time recovery restore test |
| dual-region-backup-smoke | AWS/GCP/Azure | Dual region backup test |
| backup-export-smoke | AWS/GCP/Azure | Backup export test |

## Available Case Groups

| Group Name | Description |
|------------|-------------|
| all | All cases (auto-assembled) |
| awsAll | All AWS cases (auto-assembled) |
| gcpAll | All GCP cases (auto-assembled) |
| azureAll | All Azure cases (auto-assembled) |
| smoke | Smoke test cases (awsSmoke + gcpSmoke + azureSmoke) |
| weeklyRegression | Weekly regression (awsWeeklyRegression + gcpWeeklyRegression + azureWeeklyRegression) |
| awsSmoke | AWS smoke: Kafka + MySQL + S3 changefeed, PiTR restore |
| gcpSmoke | GCP smoke: Kafka + MySQL + GCS changefeed, PiTR restore |
| azureSmoke | Azure smoke: Kafka + MySQL changefeed, PiTR restore |
| awsWeeklyRegression | Full AWS weekly regression suite |
| gcpWeeklyRegression | Full GCP weekly regression suite |
| azureWeeklyRegression | Full Azure weekly regression suite |
| awsNewRegion | AWS new region validation |
| gcpNewRegion | GCP new region validation |
| azureNewRegion | Azure new region validation |

## How to Use Cases

### Using case groups
```bash
make run-e2e CASE_GROUP=smoke
make run-e2e CASE_GROUP=weeklyRegression
```

### Using specific cases (comma-separated)
```bash
make run-e2e CASES=kafka-changefeed-smoke,mysql-changefeed-smoke
```

### Filtering by domain
- `DOMAIN=cdc` - Changefeed cases only
- `DOMAIN=br` - Backup cases only

## What `make run-e2e` really does

Root `Makefile` target `run-e2e` is a thin wrapper around:

```bash
gh workflow run e2e-test.yaml
```

Defaults from the root Makefile:

- `BRANCH=master`
- `DOMAIN=cdc`
- `TEST_ENV=dev`
- `CASE_GROUP=smoke`
- `CASES=""`
- `ORG_ID=""`
- `PROJECT_ID=""`
- `CLUSTER_IDS=""`
- region defaults:
  - AWS: `us-west-2,us-west-1`
  - GCP: `us-west1,us-west2`
  - Azure: `eastus2,eastus`

It requires `AUTH_TOKEN` in the environment and passes it to the workflow as `auth_token`.

## What the workflow does

Workflow: `.github/workflows/e2e-test.yaml`

- Trigger: `workflow_dispatch`
- Runner: `self-hosted`
- Checks out the repo
- Configures git for private modules with `secrets.GIT_ACTION_BOT`
- Sets up Go from `test/e2e/go.mod`
- Sets up Node 24 and installs `js-yaml` + `fs`
- Auths to GCP via OIDC
- Exports workflow inputs into env vars
- Picks OpenAPI keys based on `inputs.env`
- Runs:

```bash
cd test/e2e
DOMAIN=${{ inputs.domain }} make dedicated-e2e-test-${{ inputs.env }}
```

- Publishes JUnit report from `./test/e2e/e2e-report.xml`
- Explicitly fails if `<failure` exists in the XML

## Local path vs CI path

### CI / remote dispatch path
Use this when you want the standard org-managed e2e run on the self-hosted runner.

Entry points:
- root `make run-e2e`
- `gh workflow run e2e-test.yaml`

### Local execution path
Use this when you want to run the Go e2e harness on the current machine.

Entry points in `test/e2e/Makefile`:
- `make dedicated-e2e-test-local`
- `make dedicated-e2e-test-dev`
- `make dedicated-e2e-test-staging`
- `make dedicated-e2e-test-prod`

These ultimately run `gotestsum` over:
- `run_dedicated_test.go`
- `run_test.go`

with `-args -env=<env>` and optional `-domain=<domain>`.

## Config behavior

`test/e2e/internal/config/config.go`:
- loads `config.common.yaml`
- overlays `config.<env>.yaml`
- defaults `domain` to `cdc`
- validates domain in `{cdc, br}`
- if `tidbCloudToken` is empty, attempts browser-style login using configured console user/password
- if `tidbCloudToken` is present, skips login and uses the token directly

## Practical implications for the skill

1. Prefer workflow dispatch for routine execution.
2. Always use the `tidbcloud-github-access` skill entry script for `tidbcloud/*` repo access.
3. Require the caller to provide TiDB Cloud `AUTH_TOKEN` unless they explicitly want the local login/config path.
4. Treat `cases` as overriding `case_group` semantics from the workflow.
5. Mention that the workflow only exposes region inputs for AWS/GCP/Azure in the root Makefile wrapper, even though the workflow file also defines AliCloud.