# Date Integration Test Environment Cost and Performance Baseline

## Executive conclusion

**The verified initial three-cloud baseline remains approximately USD 252.95
per 730-hour month while the AWS, GCP, and Azure VMs and private load balancers
are running.** Normal sleep is approximately USD 96.09 per month, and deep
sleep is approximately USD 43.16 per month. Alibaba Cloud is not included in
those three-cloud totals even though its deployment is now complete.
Its account quote records `ecs.c9i.large` at **$46.85/month**; ESSD and EIP
planning estimates add **$8.31/month**, and its NLB instance adds
**$14.60/month**. The Alibaba Cloud producer's fixed-cost estimate is therefore
**$69.76/month**, before variable LCU charges. The
four-cloud fixed-cost estimate is **$322.71/month**. These are
retail-price engineering estimates
before discounts, taxes, traffic, object storage, API requests, snapshots, and
consumer-side private-endpoint charges.

**The verified 2-vCPU/4-GiB profile is sufficient for bootstrap, smoke tests,
and low-concurrency E2E use on all four clouds. It is not a throughput
benchmark.** All four deployments passed service bootstrap checks, and the
deployment verification recorded at least 1.2 GiB of guest-available memory
after bootstrap. Alibaba Cloud reported 1,274 MiB available with zero swap in
use. The short post-deployment monitoring window shows low average CPU use on
the original three clouds, but includes initialization activity and does not
establish a supported
producer rate, database TPS, latency SLO, or concurrent-test limit.

This document is the comparison point for later Terraform, image, service,
machine-type, storage, or networking changes.

## Baseline identity and scope

| Field | Value |
|---|---|
| Baseline ID | `2026-08-06-initial` |
| Observation cutoff | `2026-08-06T02:59:44Z` |
| Pricing basis | Public on-demand prices in USD; 730 hours/month for AWS/GCP/Azure, plus the Alibaba Cloud account quote recorded on 2026-08-07 |
| AWS deployment | `date-integration-test-6c9801be53fe`, account `118688039167`, `us-west-2` |
| GCP deployment | `date-integration-test-c875e9d88019`, project `data-integration-team-int-8f71`, `us-west1-a` |
| Azure deployment | `date-integration-test-0af0ecda743c`, subscription `7f070020-6caa-4321-a999-2d9cce2a6c17`, `japaneast` |
| Alibaba Cloud deployment | `date-integration-test-e67767108c71`, account `5083883504403318`, `ap-southeast-1`; `ecs.c9i.large` |
| Live state at cutoff | All four VMs running; each cloud's producer private-connectivity service exists |
| Services included | MySQL, Kafka KRaft, four Kafka Proxy listeners, Pulsar Standalone and Pulsar Proxy on all four clouds; Schema Registry on AWS only |
| Storage included | 30-GiB root disk, 50-GiB data disk, and one lifecycle-managed object bucket/account on every cloud |

The environment is disposable and single-node by design. The baseline covers
the producer infrastructure only. Consumer-side private endpoints, test-cluster
costs, internet or cross-zone transfer, and Dataflow Service itself are outside
scope.

## Cost baseline

### Monthly fixed-cost estimate

| Cloud | VM | Block storage | Fixed public IPv4 | Private LB | Estimated running total |
|---|---:|---:|---:|---:|---:|
| AWS | $58.22 | $6.40 | $3.65 | $16.43 | **$84.70** |
| GCP | $40.38 | $8.00 | $3.65 budget | $18.25 | **$70.28** |
| Azure | $58.25 | $17.81 | $3.65 | $18.25 | **$97.97** |
| Alibaba Cloud | $46.85 account quote | $6.12 PL0 estimate | $2.19 retention estimate | $14.60 instance fee + LCU | **$69.76 + LCU** |
| **Verified AWS/GCP/Azure combined** | **$156.85** | **$32.21** | **$10.95** | **$52.93** | **$252.95** |

Alibaba Cloud adds the following known fixed-cost estimate:

- VM: `$46.85/month` from the account quote.
- ESSD: `(30 GiB + 50 GiB) / 100 GiB × $7.65 = $6.12/month`, using the
  supplied PL0 subscription baseline as a planning proxy. The deployed disks
  are Pay-As-You-Go, so a Singapore PL0 hourly rate is still needed for an
  exact bill estimate. The supplied PL1 Pay-As-You-Go reference of
  `$0.0320/100 GiB-hour` is not applied to PL0 disks.
- EIP retention: `$0.003/hour × 730 = $2.19/month`. Pay-by-traffic outbound
  transfer remains variable and is excluded.
- Internal NLB instance fee: `$0.02/hour × 730 = $14.60/month`. The official
  reference price is billed hourly per NLB instance. The deployment has one
  shared NLB across two zones, not one separately billed NLB per zone.

The resulting four-cloud fixed-cost estimate is
`$252.95 + $46.85 + $6.12 + $2.19 + $14.60 = $322.71/month`, plus Alibaba
Cloud NLB LCU and other usage-dependent charges.

Rounding is applied per displayed component, so displayed columns can differ
from totals by one cent.

The component calculations are:

- AWS `c8g.large`: `$0.07976/hour`; two gp3 volumes totaling 80 GiB at
  `$0.08/GiB-month`; one public IPv4 at `$0.005/hour`; one NLB at
  `$0.0225/hour`.
- GCP `e2-custom-2-4096`: two E2 vCPUs at `$0.02181159/vCPU-hour` plus 4 GiB
  at `$0.00292353/GiB-hour`; 80 GiB of zonal balanced PD at
  `$0.10/GiB-month`; one internal forwarding-rule minimum at `$0.025/hour`.
  The public IPv4 catalog has a zero-price first tier and `$0.005/hour` after
  744 aggregate hours. Because eligibility and other project usage can consume
  that tier, this baseline budgets the full `$3.65/month`.
- Azure `Standard_D2pls_v6`: `$0.0798/hour`; the 30-GiB and 50-GiB Premium
  SSDs bill as P4 and P6 at `$6.0724/month` and `$11.7399/month`; one Standard
  static IPv4 at `$0.005/hour`; one Standard Load Balancer included-rule meter
  at `$0.025/hour`.
- Alibaba Cloud `ecs.c9i.large`: the account quote provided on 2026-08-07 lists
  `$0.0838/hour` and `$46.85/month`. The fixed-cost table uses the quoted
  monthly amount. A straight 730-hour multiplication of the hourly figure is
  `$61.17`, so the two values are retained as distinct quote fields rather
  than treated as equivalent calculations.

AWS prices were queried from the AWS Price List API on the baseline date. GCP
prices were queried from the Cloud Billing Catalog services `Compute Engine`
and `Networking`. Azure prices were queried from the Azure Retail Prices API.
The Alibaba Cloud VM price was supplied from the target-account quote; its
ESSD and EIP values use the supplied planning baselines. The NLB instance
and LCU reference prices come from the official NLB billable-items page. The
page identifies them as reference prices and directs customers to the buy page
for the actual account price.

### Sleep-mode estimate

| Mode | AWS | GCP | Azure | Alibaba Cloud known amount | Four-cloud known subtotal | What remains charged |
|---|---:|---:|---:|---:|---:|---|
| Running | $84.70 | $70.28 | $97.97 | **$69.76 + LCU** | **$322.71 + LCU** | VM, disks, public IP, LB |
| Normal sleep | $26.48 | $29.90 | $39.71 | **$22.91 + LCU** | **$119.00 + LCU** | Disks, public IP, LB/private producer service |
| Deep sleep | $10.05 | $11.65 | $21.46 | **About $8.31** | **About $51.47** | Disks and public IP; LB/private producer resources removed |

These figures assume each mode is held for the full month. Switching modes is
prorated by each provider's billing granularity. Deep sleep changes private
service identifiers when the environment is recreated. Alibaba Cloud running
and normal-sleep estimates include the NLB instance fee but exclude variable
LCU consumption. The deep-sleep estimate uses the PL0 subscription proxy and
EIP retention rate above; object-storage requests and bytes remain variable.

### Variable and excluded costs

The following must be added to the fixed baseline when material:

- AWS NLB usage at `$0.006/NLCU-hour`, GCP internal passthrough load-balancer
  data processing at `$0.008/GiB` in Oregon, and Azure Standard Load Balancer
  data processing at `$0.005/GB`.
- Internet egress, cross-zone/cross-region traffic, and consumer-side private
  endpoint hourly/data-processing charges.
- Alibaba Cloud NLB LCU consumption costs `$0.005/LCU/hour`. For each listener,
  hourly LCUs are the maximum of the new-connection, concurrent-connection,
  and data-transfer dimensions; the NLB total is the sum across its three TCP
  listeners. This fixture uses an internal NLB, so no NLB-attached internet EIP
  transfer fee applies.
- Alibaba Cloud PrivateLink endpoint fees are consumer-side and are outside
  this producer baseline because Terraform creates the Endpoint Service but no
  VPC endpoint. At the supplied `$0.01/endpoint-zone-hour`, a consumer endpoint
  spanning two zones would add about `$14.60/month`, before bidirectional data
  processing charges.
- S3, GCS, Azure Blob, and OSS stored bytes, requests, and
  retrieval/operation charges. Test objects expire after one day, but a high
  daily ingest volume can still be expensive.
- The versioned AWS remote-state bucket. Its normal footprint is negligible,
  but obsolete object versions should be monitored.
- Snapshots, support plans, taxes, negotiated discounts, credits, and committed
  use discounts.

## Capacity and configured performance envelope

### Compute, disk, and service allocation

| Cloud | VM architecture | vCPU / memory | Persistent disks | Enabled container limit total | Private ingress |
|---|---|---:|---|---:|---|
| AWS | Graviton4 Arm64, `c8g.large` | 2 / 4 GiB | 30 + 50 GiB gp3 | 3,712 MiB | One NLB and Endpoint Service for MySQL and Kafka |
| GCP | E2 x86, `e2-custom-2-4096` | 2 / 4 GiB | 30 + 50 GiB `pd-balanced` | 3,200 MiB | One internal passthrough LB and PSC service attachment |
| Azure | Cobalt 100 Arm64, `Standard_D2pls_v6` | 2 / 4 GiB | 30-GiB P4 + 50-GiB P6 Premium SSD | 3,200 MiB | One Standard LB and Private Link Service |
| Alibaba Cloud | x86, `ecs.c9i.large` | 2 / 4 GiB | 30 + 50 GiB ESSD PL0 | 3,200 MiB | One internal cross-zone NLB and PrivateLink Endpoint Service |

Every guest also has a 2-GiB swap file. Container limits are ceilings rather
than reservations, so their sum can approach physical memory without requiring
that amount at idle. AWS is 512 MiB higher because Schema Registry is enabled.

The main configured memory controls are:

- MySQL: 768-MiB container limit and 256-MiB InnoDB buffer pool.
- Kafka: 768-MiB container limit and 384-MiB JVM heap.
- Four Kafka Proxy processes: 64 MiB each.
- Schema Registry on AWS: 512-MiB container limit and 384-MiB JVM heap.
- Pulsar Standalone: 1-GiB container limit, 384-MiB heap, and 384-MiB direct
  memory ceiling.
- Pulsar Proxy: 384-MiB container limit, 192-MiB heap, and 128-MiB direct
  memory ceiling.

### Provider limits relevant to this deployment

These are provider specifications, not measured application throughput:

- AWS `c8g.large` has 0.937-Gbps baseline and 12.5-Gbps peak network bandwidth.
  Its EBS path has 630-Mbps/78.75-MB/s/3,600-IOPS baseline and
  10-Gbps/1,250-MB/s/40,000-IOPS maximum. Each gp3 volume supplies 3,000 IOPS
  and 125 MiB/s without additional provisioning, so the instance baseline can
  be the aggregate steady-state limiter.
- A 2-vCPU E2 VM has an up-to-4-Gbps egress class. The two zonal balanced PDs
  share the provider's 3,000-IOPS baseline before machine-type and workload
  limits. E2 can be placed on Intel or AMD hosts, so CPU model is not stable
  across recreations.
- Azure `Standard_D2pls_v6` supports 3,750 uncached Premium SSD IOPS and
  106 MB/s steady throughput at the VM level. Its P4 disk supplies 120 IOPS and
  25 MB/s and its P6 disk supplies 240 IOPS and 50 MB/s, with provider-defined
  bursting. The small disk tiers, rather than the VM limit, are the expected
  steady-state disk constraint.

### Application-level constraints

- Kafka is one KRaft broker/controller with replication factor 1. The smoke
  topic has three partitions. There is no broker HA, and throughput cannot be
  scaled horizontally without changing the topology.
- MySQL is one server with ROW binlog, FULL row image, GTID, a 256-MiB buffer
  pool, and at most 200 configured connections. It is tuned for CDC correctness
  and E2E concurrency rather than benchmark TPS.
- Pulsar is Standalone plus one Proxy. It is not representative of a clustered
  Pulsar deployment.
- Kafka Proxy adds an extra user-space hop. Private Kafka and SASL/PLAIN paths
  are intentionally plaintext for disposable tests.
- GCP and Azure retain public IPs but expose no public service ingress.
  Alibaba Cloud is configured for the same EIP-with-private-only-ingress model.
  AWS exposes the configured high ports publicly.

## Observed post-deployment baseline

### Cloud VM metrics

| Cloud | Approx. sampled duration | CPU average | CPU maximum | Network received | Network sent | Disk read | Disk written |
|---|---:|---:|---:|---:|---:|---:|---:|
| AWS | 12.6 h | 2.27% | 3.97% | 26.69 MiB | 19.65 MiB | 93.20 MiB | 3.10 GiB |
| GCP | 11.4 h | 14.44% | 63.16% | 1,005.26 MiB | 9.80 MiB | 658.79 MiB | 8.12 GiB |
| Azure | 11.7 h | 2.57% | 64.19% | 1,048.78 MiB | 65.60 MiB | 370.20 MiB | 6.45 GiB |
| Alibaba Cloud | Bootstrap window; metrics not yet baselined | Pending | Pending | Pending | Pending | Pending | Pending |

The windows begin at initial deployment and therefore include image download,
package installation, container startup, database initialization, and smoke
seeding. GCP and Azure CPU maxima are initialization peaks. The figures are
provider metrics aggregated at five-minute resolution; disk and network totals
can combine multiple disks or interfaces.

Azure reported 1.66 GiB average available memory over its sampled window. The
deployment smoke evidence recorded at least 1.2 GiB available memory after
bootstrap on every cloud. Standard AWS and GCP VM metrics do not include guest
memory, so no comparable time-series minimum is claimed for those clouds.

### Validated behavior

- All four live service bootstraps and cloud smoke checks passed.
- AWS public MySQL, anonymous Kafka, SASL Kafka, Pulsar, and Schema Registry
  checks passed; NLB targets and the Endpoint Service were healthy/available.
- GCP and Azure guest bootstrap checks and shared private producer
  infrastructure checks passed.
- AWS S3 role and AK/SK access passed. Azure SAS write/list/delete passed.
- The final Terraform plans were no-op at deployment handoff.
- Alibaba Cloud formatting, validation, shared-entry-point regression, shell
  checks, and all three Terraform tests pass. The PowerUser apply created the
  full ECS, EIP, ESSD, NLB, PrivateLink, RAM, and OSS topology. Service bootstrap,
  all eight expected running containers, NLB target health, Endpoint Service
  status, external public-port blocking, OSS ECS RAM-role write/read/delete,
  E2E synchronization, and the final no-op plan all passed. Performance
  monitoring and variable-usage price observation remain outside this
  deployment validation.

### What this baseline does not prove

This baseline does not establish maximum or sustained MySQL TPS, Kafka/Pulsar
messages per second, proxy latency, PrivateLink/PSC/Private Link throughput,
storage tail latency, failure recovery time, or safe parallel E2E case count.
No dedicated load test was run because it would add workload and alter the
environment beyond documenting its current state.

## Rules for comparing future changes

A future change should update this document or add a dated comparison entry
with all of the following:

1. Terraform commit, deployment `usedby`, cloud regions, machine types, disk
   types/sizes, enabled services, and sleep mode.
2. Current provider retail prices and the same 730-hour monthly formulas.
3. A monitoring window of at least 12 hours that includes bootstrap, plus a
   separate steady-state window when available.
4. Average and maximum CPU, available memory where the guest metric exists,
   network bytes, disk bytes, bootstrap duration, and smoke result.
5. An explanation for a fixed running-cost increase above 10%, idle average CPU
   above 20%, a post-bootstrap available-memory observation below 1 GiB, any
   swap activity during steady state, or a failed smoke test.

For throughput-sensitive changes, first define and preserve a repeatable load
profile: MySQL dataset and client concurrency, Kafka/Pulsar message size,
partition/topic count, producer acknowledgements, test duration, warm-up, and
p50/p95/p99 latency. Compare only runs using the same profile and private/public
network path.

## Reproduction references

Configuration sources in this directory are `deployment.mk`, the four cloud
Terraform roots, and `templates/compose.yaml.tftpl`. Runtime inventory and
metrics for the initial baseline were collected through `aws`, `gcloud`, and
`az` read-only APIs. Alibaba Cloud inventory is now recorded; a comparable
post-bootstrap monitoring window and account-scoped price snapshot remain to
be added.

Provider references:

- [AWS EC2 compute-optimized instance specifications](https://docs.aws.amazon.com/ec2/latest/instancetypes/co.html)
- [AWS gp3 performance](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html)
- [AWS VPC public IPv4 pricing](https://aws.amazon.com/vpc/pricing/)
- [AWS Elastic Load Balancing pricing](https://aws.amazon.com/elasticloadbalancing/pricing/)
- [Google E2 machine specifications](https://cloud.google.com/compute/docs/general-purpose-machines#e2_machine_types)
- [Google Persistent Disk performance](https://cloud.google.com/compute/docs/disks/performance)
- [Google disk pricing](https://cloud.google.com/compute/disks-image-pricing)
- [Google Cloud Load Balancing pricing](https://cloud.google.com/load-balancing/pricing)
- [Azure Dplsv6 specifications](https://learn.microsoft.com/azure/virtual-machines/sizes/general-purpose/dplsv6-series)
- [Azure managed disk types and performance](https://learn.microsoft.com/azure/virtual-machines/disks-types)
- [Azure Retail Prices API](https://learn.microsoft.com/rest/api/cost-management/retail-prices/azure-retail-prices)
- [Alibaba Cloud PrivateLink service publication](https://help.aliyun.com/en/privatelink/share-your-service/)
- [Alibaba Cloud NLB billable items](https://www.alibabacloud.com/help/en/slb/network-load-balancer/product-overview/nlb-billable-items)