# Date Integration Multi-Cloud Test Environment

This directory contains reproducible Terraform roots for disposable Dataflow
Service E2E dependencies. It is test infrastructure, not a production
deployment.

The current resource inventory, verification status, endpoints, and credential
retrieval commands are documented in
[`docs/ready-resources.md`](docs/ready-resources.md).

## Layout and committed deployment parameters

| Terraform root | Default target | Service ingress | Object storage auth |
|---|---|---|---|
| `clouds/aws` | AWS `118688039167`, `us-west-2` | Public high ports and one shared MySQL/Kafka PrivateLink service | IAM role and scoped AK/SK |
| `clouds/gcp` | `data-integration-team-int-8f71`, `us-west1` | One shared MySQL/Kafka PSC attachment; public IP retained with no public service rule | VM service-account IAM |
| `clouds/azure` | `data-integration-team-internal-azure`, `japaneast` | One shared MySQL/Kafka Private Link Service; public IP retained with no public service rule | HTTPS-only account SAS (incl. an import container for dedicated import) |
| `clouds/alicloud` | `data-integration-team-internal-alicloud` (`5083883504403318`), `ap-southeast-1` | One shared MySQL/Kafka NLB PrivateLink service; EIP retained with no public service rule | ECS RAM role plus a scoped OSS AK/SK for premium import |

All non-sensitive execution parameters are committed in `deployment.mk` and
all operations use the top-level `Makefile`. Use `CLOUD=aws`, `CLOUD=gcp`,
`CLOUD=azure`, or `CLOUD=alicloud`; AWS is the default.

The roots share the same Compose and service-install bootstrap. They deploy
MySQL 8.4, Kafka 3.9 KRaft, four Kafka Proxy listeners, and optionally Pulsar
Standalone/Proxy and Confluent-compatible Schema Registry. Pulsar is enabled
for the current deployments. Schema Registry is supported by every root
but enabled and exposed only on AWS. Its disposable endpoint requires HTTP
Basic authentication using the generated E2E username and password. Images
use immutable tag-and-digest references and include both amd64 and arm64 where
required.

AWS `c8g.large`, GCP `e2-custom-2-4096`, Azure `Standard_D2pls_v6`, and
Alibaba Cloud `ecs.c9i.large` provide two vCPUs and 4 GiB memory. Container
limits and JVM heaps are tuned for this lightweight E2E profile, with a 2-GiB
guest swap file. The bootstrap log emits a memory/container snapshot for
capacity verification.

Every deployment receives a stable random deployment identity. Supported
resources carry tracking/audit metadata including
`usedby=<deployment-name>`. S3, GCS, Azure Blob, and OSS lifecycle policies
delete test objects after one day.

## Remote state

All four roots use the versioned, encrypted S3 backend bucket
`date-integration-tfstate-118688039167-us-west-2` in account `118688039167`.
The bucket is created and hardened idempotently by AWS CLI before Terraform
initialization. State keys are:

- `date-integration-test-env/aws/terraform.tfstate`
- `date-integration-test-env/gcp/terraform.tfstate`
- `date-integration-test-env/azure/terraform.tfstate`
- `date-integration-test-env/alicloud/terraform.tfstate`

The backend contains test credentials and must remain private. No Secrets
Manager, Parameter Store, or cloud secret-store resource is created.

`make init CLOUD=<cloud>` remains the explicit force-reconfiguration command.
Read-only output targets (`outputs`, `e2e-config`, and `sync-e2e-config`) first
compare the local backend metadata with the configured S3 bucket, state key,
region, profile, and lockfile setting. They reuse matching metadata and run the
full bucket bootstrap plus `terraform init -reconfigure` only when metadata is
missing or stale.

## Usage

Prerequisites are Terraform 1.11+, Docker, Python 3, `jq`, AWS profile
`118688039167-cp9`, a logged-in `gcloud` user, an Azure CLI login for
subscription `7f070020-6caa-4321-a999-2d9cce2a6c17`, and Alibaba Cloud CLI
profile `5083883504403318-poweruser` authenticated to that account. The Makefile derives
a short-lived Google provider token from `gcloud auth print-access-token`.

```bash
make validate-all
make plan CLOUD=aws
make up CLOUD=aws APPLY_FLAGS=-auto-approve
make smoke CLOUD=aws
make smoke-storage CLOUD=aws
make audit-tags CLOUD=aws
make outputs CLOUD=aws
make seed-import CLOUD=alicloud APPLY_FLAGS=-auto-approve
make seed-import CLOUD=azure
```

For Alibaba Cloud, `smoke-storage` executes an OSS write/read/delete round trip
inside ECS with RAM-role credentials, and `audit-tags` verifies the live
Terraform state carries the generated `usedby` identity on every taggable
resource. `seed-import` uploads both import sample formats onto the clouds that
own a self-managed import source: a TiDB-importable CSV under `import/csv/` and
a dumpling SQL dump under `import/sql/bikeshare/`. The AliCloud OSS import user
and Azure Blob SAS can read both prefixes. The E2E source URI remains
provider-specific (`import/csv/` for premium OSS and `import/sql/` for
dedicated Azure), while the other format is available for cross-provider
fixture validation. Both scripts upload from the tracked
`scripts/seeddata/import/{csv,sql}/` subdirectories.

Both cloud seeders now carry the complete CSV bundle: `test-schema-create.sql`,
`test.a-schema.sql`, and the unchanged `test.a.0001.csv`. Default mode also
includes exactly the existing dumpling table schema and data files. Each seeder
validates all five tracked fixtures before reading Terraform outputs or making
any cloud call, then uploads and reads back every object. The upload order is
the three CSV bundle objects followed by the two existing SQL objects.
The schema files describe the existing CSV columns (`id`, `name`, and
`amount`). The seeders only upload these files; the import service applies the
schema files when it runs a CSV import. The seed commands perform no SQL
execution.

Synchronize generated test-server sections after all four states are live:

```bash
make sync-e2e-config
cd ../../e2e
go run ./script/sync_test_server_config.go -verify-unrelated-against-git
```

All Terraform-generated test dependencies and their disposable fixture
credentials are written once to `config.common.yaml`: Kafka, MySQL, Pulsar,
S3, GCS, Azure Blob Storage, backup export, Essential export, and Schema
Registry. The OSS and Azure import provider blocks are synchronized in place;
their generated source and credential fields are managed, while external
provider references and unknown import fields remain user-owned. The OSS
import block also carries the generated `importedDBTable` and
`customCSVFormat` contract. AWS Essential export assumes the fixture-managed S3 access role and
uses only the fixture's `cluster-export/` prefix. Tracked
dev/staging/prod configuration and ignored mode-`0600` `config.local*.yaml`
files contain only environment-specific settings. This convention applies only
to these disposable test services, not TiDB Cloud API or production
credentials.

## Connectivity and security decisions

AWS exposes MySQL, anonymous Kafka, SASL/PLAIN Kafka, optional Pulsar, and
Schema Registry on stable random high ports. MySQL and both private Kafka
listeners share one NLB and one Endpoint Service. The NLB spans every
available AZ in the target region and has cross-zone load balancing enabled,
so a consumer endpoint can use any regional AZ while the disposable backend
remains a single EC2 instance. Cross-AZ data-transfer charges apply when an
NLB node forwards traffic to the instance in another AZ. Because the AWS
provider exposes Endpoint Service zones as a computed-only field, a
Terraform-managed idempotent provisioner reconciles the Service zone list
after the NLB update and whenever deep-sleep recovery creates a new Service.

GCP, Azure, and Alibaba Cloud retain fixed public IP resources for
management/future use but have no public service ingress rule. Their E2E
fragments therefore contain only private MySQL/Kafka dependencies. Each cloud
uses one load balancer frontend and one producer service for all three
MySQL/Kafka ports.

Private producer access is restricted to the committed consumer allowlists.
AWS permits the producer account and the approved DBAAS, QA, and PingCAP
accounts. GCP PSC uses manual acceptance with connection reconciliation for
the five approved consumer projects. Azure limits visibility and automatic
approval to the internal Date Integration and Dedicated non-production
subscriptions. Override the corresponding cloud variable only when the
approved consumer inventory changes. The deploy-time values are explicitly
passed from `deployment.mk`; matching Terraform defaults provide safe behavior
for direct module use.

Alibaba Cloud uses one internal, two-zone, cross-zone NLB and one NLB-backed
PrivateLink endpoint service. The producer account is automatically allowed;
the committed whitelist also permits the staging dataplane and DBAAS dev
accounts. The ECS EIP is used for outbound bootstrap and operations only. OSS
is private, AES-256 encrypted, accessed by a scoped ECS RAM role for
bootstrap/smoke, and retains separate CDC, backup-export, and import prefixes.
Premium import from OSS authenticates with a dedicated long-lived AK/SK RAM
user scoped to the `import/csv/` and `import/sql/` prefixes (a RAM role is not
an available auth mode for premium OSS import), so the fixture exposes OSS AK/SK
credentials and the import URI/endpoint as outputs. The one-day lifecycle
rule covers all objects except `import/`, including future disposable
prefixes.

AWS and GCP storage also carry extensible external-consumer grant maps. AWS
backup export matches approved cross-account role-name patterns in the bucket
policy, while CDC consumers assume the fixture's shared S3 role through
account- and external-ID-specific trust statements. Generated CDC dependency
configuration uses that role instead of static AK/SK credentials. The shared
S3 role also reads the `import/` prefix for import E2E (read-only object
access; the one-day lifecycle rule covers only the `cdc/`, `backup-export/`,
and `cluster-export/` disposable prefixes). GCP uses
`backup-export/` and `cdc/` managed folders plus the predefined
`roles/storage.objectUser` role so object listing remains prefix-scoped instead
of bucket-wide. External consumers receive `roles/storage.bucketViewer` on the
test bucket for bucket metadata access.
Environment-specific AWS accounts, role patterns, external IDs, and GCP
service accounts are declared only in `deployment.mk`; the Makefile passes
them to Terraform, whose direct-use defaults contain no external consumers.

Kafka Proxy PrivateLink/PSC traffic is plaintext. TLS is not required by AWS
PrivateLink, GCP PSC, or Azure Private Link themselves; TLS-specific Kafka E2E
cases remain unsupported by this stack. SASL/PLAIN also has no TLS and is
appropriate only for disposable test data.

Azure Storage enables Shared Key access solely to mint a committed-policy,
absolute-expiry account SAS. The SAS is HTTPS-only, Blob-only, limited to
container/object access, and grants read/write/create/add/list/delete without
account-service administration. The account and containers stay non-public.
The unused backup-export container is retained for future Azure support, and an
`import` container is provided for dedicated import. Imported test data under
`import/sql/` is excluded from the one-day lifecycle rule so dedicated import
sources are not deleted out from under a running import. Azure lifecycle
filters are positive-only, so the policy explicitly lists the disposable
`cdc/` and `backup-export/` containers.

## Sleep and wake

```bash
make sleep CLOUD=aws
make wake CLOUD=aws APPLY_FLAGS=-auto-approve
make sleep-deep CLOUD=aws APPLY_FLAGS=-auto-approve
```

Normal sleep stops/deallocates the VM and retains the private service name.
Deep sleep first checks consumer connections, then removes charged load
balancer/private-service resources. Waking from deep sleep recreates producer
identifiers, so consumers must recreate endpoints and DNS. Persistent disks,
fixed public IPs, ports, credentials, networks, and remote state are retained.
On AWS, wake waits through SSM for the live `date-integration.service` unit to
finish its service readiness and seed checks before the smoke tests begin.