# Ready Date Integration Test Resources

## Conclusion

**The AWS, GCP, Azure, and Alibaba Cloud test deployments now exist in their
target accounts, and each intended private-connectivity producer service has
been created.** AWS public service endpoints are reachable. GCP, Azure, and
Alibaba Cloud retain public IPs for bootstrap and operations but do not expose
service ports publicly.

This snapshot was refreshed on **2026-08-07 (Asia/Shanghai)**. It contains no
passwords, tokens, access keys, or SAS values.

| Cloud | Runtime status | Private service status | Directly usable dependencies |
|---|---|---|---|
| AWS | EC2 `running`; public service ports reachable | Endpoint Service `Available` | Public MySQL, Kafka, Pulsar, Schema Registry; PrivateLink MySQL/Kafka; S3 |
| GCP | VM `RUNNING` | PSC service attachment present | PSC MySQL/Kafka; GCS |
| Azure | VM `VM running` | Private Link Service `Succeeded` | Private Link MySQL/Kafka; Azure Blob |
| Alibaba Cloud | ECS `running`; bootstrap and seed passed | Endpoint Service `Active`; all NLB targets `Available` | PrivateLink MySQL/Kafka; OSS through ECS RAM role |

## AWS

### Resource identity

- Account: `118688039167`
- Region: `us-west-2`
- Deployment / `usedby`: `date-integration-test-6c9801be53fe`
- Instance: `i-0a825e4fa7ca4878c`, `c8g.large` (2 vCPU, 4 GiB)
- Instance status: `running`
- Public IP: `34.218.43.86`
- Shared MySQL/Kafka Endpoint Service:
  `com.amazonaws.vpce.us-west-2.vpce-svc-0079492dbba5cce76`
- Supported physical AZ IDs: `usw2-az1`, `usw2-az2`, `usw2-az3`,
  `usw2-az4`
- Shared NLB: one internal NLB across all four AZs, with cross-zone load
  balancing enabled and all three target groups healthy

AWS account identity, EC2 state, Endpoint Service state, and S3 lifecycle were
refreshed through profile `118688039167-cp9` during this snapshot. Previous
instances are terminated and are not part of the ready environment.

### Service endpoints

| Service | Endpoint or port | Access | Verification |
|---|---|---|---|
| MySQL | `34.218.43.86:49609` | Public | TCP reachable |
| Kafka anonymous | `34.218.43.86:49626` | Public, PLAINTEXT | TCP reachable |
| Kafka SASL/PLAIN | `34.218.43.86:49650` | Public, no TLS | TCP reachable |
| MySQL | port `49609` | Shared PrivateLink service | Producer service Available |
| Kafka anonymous | port `49682` | Shared PrivateLink service | Producer service Available |
| Kafka SASL/PLAIN | port `49718` | Shared PrivateLink service | Producer service Available |
| Pulsar binary | `pulsar://34.218.43.86:49760` | Public, token auth | TCP reachable |
| Pulsar Admin HTTP | `http://34.218.43.86:49806` | Public, token auth | TCP reachable |
| Schema Registry | `http://34.218.43.86:49848` | Public, HTTP Basic auth | Correct credentials returned HTTP 200; anonymous and wrong credentials returned HTTP 401 |

Kafka PrivateLink consumers must create DNS records under
`date-integration.kafka.test`; the currently advertised AZ record is
`usw2-az2.date-integration.kafka.test`.

### Object storage

- Bucket: `date-integration-test-6c9801be53fe-118688039167-us-west-2`
- CDC prefix: `s3://date-integration-test-6c9801be53fe-118688039167-us-west-2/cdc/`
- Backup export prefix:
  `s3://date-integration-test-6c9801be53fe-118688039167-us-west-2/backup-export/`
- Essential export prefix:
  `s3://date-integration-test-6c9801be53fe-118688039167-us-west-2/cluster-export/`
- IAM role name: `date-integration-test-6c9801be53fe-s3-access`
- IAM role ARN:
  `arn:aws:iam::118688039167:role/date-integration-test-6c9801be53fe-s3-access`
- Access: the scoped IAM role above and scoped AK/SK credentials.
- Lifecycle: test objects expire after one day.

## GCP

### Resource identity

- Project: `data-integration-team-int-8f71`
- Region / zone: `us-west1` / `us-west1-a`
- Deployment / `usedby`: `date-integration-test-c875e9d88019`
- VM: `date-integration-test-c875e9d88019-services`
- Machine type: `e2-custom-2-4096` (2 vCPU, 4 GiB)
- VM status: `RUNNING`
- Retained public IP: `34.82.215.233`
- PSC service attachment:
  `projects/data-integration-team-int-8f71/regions/us-west1/serviceAttachments/date-integration-test-c875e9d88019-services`
- PSC connection preference: automatic acceptance.

The public IP is retained for lifecycle stability, but no public service
firewall rule exists.

### Private service ports

| Service | Port | Access |
|---|---:|---|
| MySQL | `48165` | Shared PSC attachment |
| Kafka anonymous | `48238` | Shared PSC attachment, PLAINTEXT |
| Kafka SASL/PLAIN | `48274` | Shared PSC attachment, no TLS |

Kafka consumers must create the advertised DNS record
`us-west1.date-integration.gcp.kafka.test` after creating their PSC endpoint.
The generated Premium dependency maps that record to endpoint availability
zone `us-west1-a`.
Pulsar is deployed on the VM but is not exposed by PSC. Schema Registry is
disabled in this deployment.

### Object storage

- Bucket: `dit-c875e9d88019-data-integration-team-int-8f71`
- CDC prefix:
  `gs://dit-c875e9d88019-data-integration-team-int-8f71/cdc/`
- Backup export prefix:
  `gs://dit-c875e9d88019-data-integration-team-int-8f71/backup-export/`
- Service account ID: `dit-c875e9d88019`
- Service account email:
  `dit-c875e9d88019@data-integration-team-int-8f71.iam.gserviceaccount.com`
- Bucket IAM roles: `roles/storage.objectAdmin` and
  `roles/storage.legacyBucketReader`.
- Access: VM service-account IAM; no AK/SK credential is created.
- Lifecycle: live configuration verified; objects expire after one day.

## Azure

### Resource identity

- Subscription: `data-integration-team-internal-azure`
  (`7f070020-6caa-4321-a999-2d9cce2a6c17`)
- Region: `japaneast`
- Deployment / `usedby`: `date-integration-test-0af0ecda743c`
- Resource group: `date-integration-test-0af0ecda743c-rg`
- VM: `date-integration-test-0af0ecda743c-services`
- Machine type: `Standard_D2pls_v6` (2 vCPU, 4 GiB)
- VM status: `VM running`
- Retained public IP: `52.253.108.41`
- Private Link Service alias:
  `date-integration-test-0af0ecda743c-services.6a4dce80-0afe-44b4-a730-04e6c4a7dec2.japaneast.azure.privatelinkservice`
- Private Link Service provisioning state: `Succeeded`.

The public IP is retained, but no public service security rule exists.

### Private service ports

| Service | Port | Access |
|---|---:|---|
| MySQL | `40272` | Shared Private Link Service |
| Kafka anonymous | `40345` | Shared Private Link Service, PLAINTEXT |
| Kafka SASL/PLAIN | `40381` | Shared Private Link Service, no TLS |

Kafka consumers must create the advertised DNS record
`japaneast.date-integration.azure.kafka.test` after creating their private
endpoint. The generated Premium dependency maps that record to Azure logical
availability zone `1`. Pulsar is deployed on the VM but is not exposed by the
Private Link Service. Schema Registry is disabled in this deployment.

### Object storage

- Storage account: `dit0af0ecda743c`
- CDC container: `https://dit0af0ecda743c.blob.core.windows.net/cdc/`
- Reserved backup-export container:
  `https://dit0af0ecda743c.blob.core.windows.net/backup-export/`
- Import prefix (dedicated import):
  `https://dit0af0ecda743c.blob.core.windows.net/import/sql/`
- Account status: `Succeeded`; HTTPS-only and Shared Key access are enabled.
- Containers are private even though storage public-network access is enabled.
- Access: long-lived, HTTPS-only account SAS. Managed Identity access is not
  part of this fixture.
- SAS absolute expiry: `2036-01-01T00:00:00Z`.
- Lifecycle: live configuration verified; blobs in `cdc/` and `backup-export/`
  expire after one day. The `import/` container is deliberately excluded from
  the one-day rule so dedicated import sources are not deleted out from under a
  running import. Azure lifecycle policies use positive-only prefix filters,
  so disposable containers are listed explicitly.
- Backup-export infrastructure is reserved, but Azure backup-export E2E
  consumption is not implemented yet.

## Alibaba Cloud

### Resource identity

- Account: `data-integration-team-internal-alicloud`
  (`5083883504403318`)
- Region: `ap-southeast-1`
- Deployment / `usedby`: `date-integration-test-e67767108c71`
- ECS: `i-t4n7shx86aisjgf62bce`, `ecs.c9i.large` (2 vCPU, 4 GiB)
- Retained EIP: `47.237.95.89`; no public service security-group ingress
- Retained encrypted data disk: `d-t4n2ghqtodhhiet6a7m7`, 50-GiB ESSD PL0
- VPC: `vpc-t4nvann1ggjnmndtnlu0f`
- vSwitches: `vsw-t4npo0rgwfm7h6mjt2giv` in `ap-southeast-1c` and
  `vsw-t4n3u4do47qw5pt3tx7p8` in `ap-southeast-1a`
- Security group: `sg-t4nd27genx3ezzckzbig`
- Shared internal NLB: `nlb-cm8o06lcyl5b0nur0z`, two zones with cross-zone
  forwarding enabled
- Shared PrivateLink Endpoint Service:
  `com.aliyuncs.privatelink.ap-southeast-1.epsrv-t4nrjz9f15k9zrcwjju8`
- Supported zones: `ap-southeast-1c`, `ap-southeast-1a`
- Audit tags include `usedby=date-integration-test-e67767108c71`; the live-state
  audit passed for all 18 taggable resources.

`ecs.c7.large` was unavailable for purchase in the NLB-compatible Singapore
zones. The deployment therefore uses the same-size `ecs.c9i.large`, selected
in an intersection of current ECS stock and NLB-supported zones.

### Private service ports

| Service | Port | Access |
|---|---:|---|
| MySQL | `45499` | Shared PrivateLink Endpoint Service |
| Kafka anonymous | `45572` | Shared PrivateLink Endpoint Service, PLAINTEXT |
| Kafka SASL/PLAIN | `45608` | Shared PrivateLink Endpoint Service, no TLS |

Kafka consumers must create the advertised DNS record
`b1.ap-southeast-1.date-integration.alicloud.kafka.test` after creating their
VPC endpoint. The generated Premium dependency maps the corresponding zone
record to service availability zone `ap-southeast-1a`. Pulsar is deployed on
the ECS instance but is not exposed by the Endpoint Service. Schema Registry
is disabled in this deployment.

### Object storage

- Bucket: `dit-e67767108c71`
- CDC prefix: `oss://dit-e67767108c71/cdc/`
- Reserved backup-export prefix: `oss://dit-e67767108c71/backup-export/`
- Import prefix: `oss://dit-e67767108c71/import/csv/` (premium import from OSS)
- Import endpoint: `oss-ap-southeast-1.aliyuncs.com`
- Access: scoped ECS RAM role `dateintegrationteste67767108c71ecs` for
  bootstrap/smoke, plus a dedicated AK/SK RAM user
  (`dateintegrationteste67767108c71import`) for premium OSS import; access keys
  are scoped to both `import/csv/` and `import/sql/`, and are stored in the
  remote state and the generated E2E config.
- Security: private ACL and AES-256 server-side encryption.
- Lifecycle: all objects and incomplete multipart uploads expire after one day
  except the `import/` prefix; the OSS negative prefix filter also covers new
  disposable prefixes automatically.
- Validation: ECS RAM-role write/read/content/delete smoke passed on
  2026-08-07; the `import/csv/` and `import/sql/` prefixes are seeded with
  TiDB-importable CSV and SQL samples via `make seed-import CLOUD=alicloud`.

## Credential Retrieval

Credential values are intentionally omitted from this document. The canonical
values live in the encrypted remote Terraform state and the generated E2E
configuration.

- Remote state bucket:
  `date-integration-tfstate-118688039167-us-west-2`
- Tracked targets: `test/e2e/config.common.yaml`, `test/e2e/config.dev.yaml`,
  `test/e2e/config.staging.yaml`, and `test/e2e/config.prod.yaml`
- Local targets: `test/e2e/config.local*.yaml` (mode `0600`)
- Shared test dependencies, including their disposable credentials, live in
  `test/e2e/config.common.yaml`; environment files retain only
  environment-specific settings.

Run credential commands only in a private terminal. Terraform marks the output
as sensitive in normal output, but `-json` and `-raw` deliberately reveal the
values. Do not paste their output into tickets, chat, CI logs, or documentation.

### Initialize access

Run from `test/infrastructure/date-integration-test-env`. Initialization reads
the remote state from the shared AWS state bucket:

```bash
aws sso login --profile 118688039167-cp9

make init CLOUD=aws
make init CLOUD=gcp
make init CLOUD=azure
make init CLOUD=alicloud
```

All four backend keys are in AWS, so a valid AWS SSO session is required even
when retrieving non-AWS outputs. The Alibaba Cloud backend key has no resource
outputs until its first successful apply. A provider login is not required for
an existing `terraform output`; it is required for provider operations such as
`plan`, `apply`, `smoke`, or direct cloud-resource inspection.

### Retrieve one service credential

Use the appropriate cloud root and select only the needed field:

```bash
# MySQL test user and password.
terraform -chdir=clouds/aws output -json service_credentials | jq -r '.mysql.user'
terraform -chdir=clouds/aws output -json service_credentials | jq -r '.mysql.password'

# Kafka SASL/PLAIN username and password.
terraform -chdir=clouds/aws output -json service_credentials | jq -r '.kafka.username'
terraform -chdir=clouds/aws output -json service_credentials | jq -r '.kafka.password'

# Pulsar client token.
terraform -chdir=clouds/aws output -json service_credentials | jq -r '.pulsar.client_token'

# AWS Schema Registry test credentials.
terraform -chdir=clouds/aws output -json service_credentials | jq -r '.schema_registry.username'
terraform -chdir=clouds/aws output -json service_credentials | jq -r '.schema_registry.password'

# AWS S3 AK/SK credentials.
terraform -chdir=clouds/aws output -json service_credentials | jq -r '.s3.access_key_id'
terraform -chdir=clouds/aws output -json service_credentials | jq -r '.s3.secret_access_key'

# Azure Blob SAS and its absolute expiry.
terraform -chdir=clouds/azure output -json service_credentials | jq -r '.azure_blob.sas_token'
terraform -chdir=clouds/azure output -json service_credentials | jq -r '.azure_blob.expiry'
```

Replace `clouds/aws` with `clouds/gcp`, `clouds/azure`, or `clouds/alicloud`
for the corresponding MySQL, Kafka, or Pulsar credentials after that root is
deployed. GCP storage uses VM service-account IAM, and Alibaba Cloud storage
uses an ECS RAM role for bootstrap/smoke but now exposes an OSS AK/SK for
premium import. Azure storage uses SAS rather than Managed Identity or AK/SK.

Additional available fields are:

| Cloud | Credential fields |
|---|---|
| AWS | `mysql.root_password`, `mysql.user`, `mysql.password`, `kafka.username`, `kafka.password`, `schema_registry.username`, `schema_registry.password`, `pulsar.token_secret`, `pulsar.proxy_token`, `pulsar.admin_token`, `pulsar.client_token`, `s3.access_key_id`, `s3.secret_access_key`, `s3.role_external_id` |
| GCP | `mysql.root_password`, `mysql.user`, `mysql.password`, `kafka.username`, `kafka.password`, `pulsar.token_secret`, `pulsar.proxy_token`, `pulsar.admin_token`, `pulsar.client_token` |
| Azure | `mysql.root_password`, `mysql.user`, `mysql.password`, `kafka.username`, `kafka.password`, `pulsar.token_secret`, `pulsar.proxy_token`, `pulsar.admin_token`, `pulsar.client_token`, `azure_blob.sas_token`, `azure_blob.expiry` |
| Alibaba Cloud | `mysql.root_password`, `mysql.user`, `mysql.password`, `kafka.username`, `kafka.password`, `pulsar.token_secret`, `pulsar.proxy_token`, `pulsar.admin_token`, `pulsar.client_token`, `oss.access_key_id`, `oss.access_key_secret` |

Retrieve the Alibaba Cloud OSS premium-import credentials (the RAM-user access
key) and import prefix from the `service_credentials` and `object_storage`
outputs:

```bash
# Alibaba Cloud OSS premium-import AK/SK (RAM user, scoped to import/csv/ and import/sql/).
terraform -chdir=clouds/alicloud output -json service_credentials | jq -r '.oss.access_key_id'
terraform -chdir=clouds/alicloud output -json service_credentials | jq -r '.oss.secret_access_key'

# Alibaba Cloud OSS import prefix and public endpoint.
terraform -chdir=clouds/alicloud output -json object_storage | jq -r '.import_uri'
terraform -chdir=clouds/alicloud output -json object_storage | jq -r '.public_endpoint'
```

This replaces the previously hardcoded shared `import-samples` AK/SK: inject
the fixture-generated credentials into your premium config (or run
`make sync-e2e-config`), then set `importSampleData.oss.accessKeyID` /
`accessKeySecret` from the values above.

### Retrieve or synchronize complete E2E configuration

Print one cloud's complete sensitive E2E fragment to the current terminal:

```bash
make e2e-config CLOUD=aws
make e2e-config CLOUD=gcp
make e2e-config CLOUD=azure
make e2e-config CLOUD=alicloud
```

Synchronize all endpoints and credentials into the tracked staging/prod files
and mode-`0600` local files:

```bash
make sync-e2e-config
cd ../../e2e
go run ./script/sync_test_server_config.go -verify-unrelated-against-git
```

The sync command is the preferred retrieval path for E2E execution because it
avoids manual copy/paste and keeps the endpoint and credential from the same
Terraform state revision. Run it after each cloud apply so endpoints, MySQL/
Kafka/Pulsar credentials, and the Alibaba Cloud OSS premium-import
sourceURL/endpoint stay in sync with the same state revision. The Alibaba Cloud
fragment carries the OSS `importSampleData.oss` source (sourceURL, endpoint,
AK/SK); the synchronizer merges it into `config.common.yaml` and redacts the
AK/SK to placeholders, preserving the external AWS/GCP/Azure reference sources.
Run it immediately after the first successful Alibaba Cloud apply; do not
invent or commit placeholder endpoints or credentials.

## Operations

Run all commands from `test/infrastructure/date-integration-test-env`:

```bash
# Inspect one cloud without printing credentials in shared logs.
make outputs CLOUD=aws

# Re-run protocol and storage smoke checks.
make smoke CLOUD=aws
make smoke CLOUD=gcp
make smoke CLOUD=azure
make smoke CLOUD=alicloud

# Stop only the VM and retain the private producer service name.
make sleep CLOUD=aws

# Restore a stopped deployment.
make wake CLOUD=aws APPLY_FLAGS=-auto-approve

# Remove charged LB/private-service resources after connection prechecks.
make sleep-deep CLOUD=aws APPLY_FLAGS=-auto-approve
```

AWS wake waits for the live systemd service through SSM after the EC2 state
change. The subsequent smoke test therefore starts only after the containers
and seed checks are ready.

Deep sleep recreates the producer service identifier on wake. Consumers must
then recreate their Interface Endpoint, PSC endpoint, Azure Private Endpoint,
or Alibaba Cloud VPC endpoint and update DNS. Persistent disks, fixed public
IPs, ports, credentials, networks, and remote state remain stable.

## Known Limitations

- Kafka proxy traffic is plaintext. SASL/PLAIN does not provide transport
  encryption in this fixture.
- Consumer-side private endpoints and DNS records are intentionally not
  created by the producer Terraform roots.
- GCP, Azure, and Alibaba Cloud Pulsar containers are running but have no
  external ingress.
- Alibaba Cloud sleep/wake and deep-sleep lifecycle exercises are not included
  in this running-environment handoff; plan-time lifecycle coverage remains in
  the Terraform tests.
- Schema Registry is enabled only on AWS and uses HTTP Basic authentication
  without TLS.
- This environment contains disposable test data only and is not suitable for
  production workloads.