# TiDB Cloud VPC Architecture This document describes TiDB Cloud deployment architecture organized by VPC, including component distribution and network connectivity. --- ## VPC Overview | VPC Type | Deployment Repo | Scope | Responsibility | |----------|-----------------|-------|----------------| | **Global Control Plane VPC** | `infra-cd` | Global | Global control plane services | | **Regional Control Plane VPC** | `infra-cd` | Region-level | Regional control plane services | | **Data Plane VPC** | `aws-shared-cd` | Region-level | User cluster runtime environment (EKS/ACK/GKE) | | **O11y/Observability VPC** | `observability-cd`, `mimir-cd`, `clinic-cd` | Region-level | Observability infrastructure | | **Security VPC** | `InfraSec` | Global/Region | Security, audit, authentication | | **Customer VPC (BYOC)** | `byoc-account-setup` | Customer | Customer's own cloud account VPC | --- ## VPC Topology ``` ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ Global Control Plane VPC │ │ (Global control plane - unified management entry) │ ├─────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────┐ ┌──────────────────────┐ ┌──────────────────────────────┐ │ │ │ api-gateway │ │ tidb-mgmt-service │ │ dedicated-global-service │ │ │ │ (API Gateway)│ │ (Global TiDB Mgmt) │ │ (Dedicated Global Svc) │ │ │ └──────────────┘ └──────────────────────┘ └──────────────────────────────┘ │ │ │ │ ┌──────────────────────────┐ ┌──────────────────┐ ┌──────────────────────────┐ │ │ │ serverless-global-service│ │ nextgen-global │ │ config-center │ │ │ │ (Serverless Global Svc) │ │ -global-service │ │ (Config Center) │ │ │ └──────────────────────────┘ └──────────────────┘ └──────────────────────────┘ │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │ │ │ account │ │ billing │ │ auth-server │ │ message-service │ │ │ │ (Account) │ │ (Billing) │ │ (Auth) │ │ (Message Svc) │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────────────┘ │ │ VPC Peering / Transit Gateway │ (Global to Regional Control Plane) ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ Regional Control Plane VPC │ │ (Regional control plane - per Region deployment) │ ├─────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────────┐ │ │ │ infra-api │ │ infra-provider │ │ cluster-service-ng │ │ │ │ (Infra API) │ │ (Infra Provider) │ │ (Cluster Service NG) │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────────┘ │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────────┐ │ │ │ core-svc │ │ dataservices │ │ dbaas │ │ │ │ (Core Svc) │ │ (Data Svc) │ │ (DBaaS Control) │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────────┘ │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │serverless-service│ │ tcidm (regional)│ │ │ │ (Serverless) │ │ (Regional IDM) │ │ │ └──────────────────┘ └──────────────────┘ │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────────┐ │ │ │ metering │ │ devtier-store │ │ eventrouter │ │ │ │ (Metering) │ │ -service │ │ (Event Router) │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────────────┘ │ │ VPC Peering / PrivateLink │ (Control Plane to Data Plane) ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ Data Plane VPC (EKS/ACK/GKE) │ │ (Data plane - user cluster runtime) │ ├─────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────┐ │ │ │ Kubernetes Infrastructure Layer │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ │ │ aws-shared-cd │ │ eks-freetier │ │ ack-freetier │ │ │ │ │ │ (EKS Cluster) │ │ (EKS FreeTier) │ │ (ACK FreeTier) │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────┐ │ │ │ Operator Management Layer │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ │ │ tidb-operator │ │ tiflow-operator │ │ tikv-operator │ │ │ │ │ │ (TiDB Operator) │ │ (TiFlow Op) │ │ (TiKV Operator) │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────┐ │ │ │ Data Plane Shared Services (aws-shared-provider) │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ │ │ tidb-gateway │ │ tidb-http-gateway│ │ gateway-controller │ │ │ │ │ │ (TiDB Gateway) │ │ (HTTP Gateway) │ │ (Gateway Controller) │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────┘ │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ │ │ audit-log │ │ meta-server │ │ cluster-agent │ │ │ │ │ │ (Audit Log) │ │ (Meta Svr) │ │ (Cluster Agent) │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────┐ │ │ │ User TiDB Cluster Layer (per-user Namespace) │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ │ │ TiDB Cluster │ │ TiKV Cluster │ │ PD Cluster │ │ │ │ │ │ (SQL Compute) │ │ (Storage) │ │ (Coordination) │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────┘ │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ │ │ TiFlash Cluster │ │ CDC/TiFlow │ │ │ │ │ │ (Analytics) │ │ (Replication) │ │ │ │ │ └──────────────────┘ └──────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────────────┘ ▲ │ VPC Peering / PrivateLink / Cloud Enterprise │ (Data Plane to Customer VPC) ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ Customer VPC (BYOC) │ │ (Customer's own cloud account VPC) │ │ ┌─────────────────────────────────────────────────────────────────────────────────┐ │ │ │ byoc-account-setup │ │ │ │ Connect to Data Plane VPC via VPC Peering or PrivateLink │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ O11y/Observability VPC │ │ (Observability infrastructure VPC) │ ├─────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────┐ │ │ │ Metrics System │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ │ │ VictoriaMetrics │ │ metrics-server │ │ mimir-cd │ │ │ │ │ │ (vminsert/ │ │ (central_metrics/│ │ (Mimir Deploy) │ │ │ │ │ │ vmselect/ │ │ scrape_gateway) │ │ │ │ │ │ │ │ vmstorage) │ │ │ │ │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────┐ │ │ │ Logging System │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ │ │ loki │ │ promtail │ │ │ │ │ │ │ │ (Log Storage) │ │ (Log Collector) │ │ │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────┐ │ │ │ Observability Platform Services │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ │ │ agent-control │ │ data-collector │ │ diagnosis-query │ │ │ │ │ │ -server │ │ (collector/ │ │ (Diagnosis Query) │ │ │ │ │ │ (Agent Ctrl) │ │ processor) │ │ │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────┘ │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ │ │diagnosis-compute │ │ globalserver │ │ regionalserver │ │ │ │ │ │ (Diagnosis) │ │ (Global Svr) │ │ (Regional Svr) │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────┘ │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ │ │ vmbackupmanager │ │ ng-monitoring │ │ │ │ │ │ (VM Backup) │ │ (TiDB Monitor) │ │ │ │ │ └──────────────────┘ └──────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────┐ │ │ │ Monitoring Components │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────────┐ │ │ │ │ │ monitoring │ │ reload │ │ datadog-on-cloud │ │ │ │ │ │ (Monitoring) │ │ (Reload) │ │ (DataDog Integration) │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────────────────┐ │ Security VPC │ │ (Security infrastructure VPC) │ ├─────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────┐ │ │ │ auditservice │ │ InfraSec │ │ auth-server │ │ kube-oidc │ │ │ │ (Audit Svc) │ │ (Security CD) │ │ (Auth) │ │ (K8s OIDC) │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────┘ └──────────────┘ │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ │ │ chaos-svc │ │ network-policy │ │ │ │ │ │ (Chaos) │ │ -controller │ │ │ │ │ └──────────────────┘ └──────────────────┘ └──────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────────────┘ ``` --- ## Detailed Component List by VPC ### 1. Global Control Plane VPC Global control plane providing unified management entry across all regions. | Component | Repository | Type | Responsibility | Interfaces | |-----------|------------|------|----------------|------------| | **api-gateway** | api-gateway | Service | Unified API gateway | HTTP API | | **tidb-mgmt-service** | tidb-management-service | Service | Global TiDB management | gRPC | | **dedicated-global-service** | tidb-management-service | Service | Dedicated cluster global service | gRPC (VpcService, PrivateLinkConnectionService) | | **serverless-global-service** | tidb-management-service | Service | Serverless global service | gRPC (ServerlessPrivateLinkConnectionService) | | **nextgen-global-service** | tidb-management-service | Service | Next-gen cluster service | gRPC (PrivateLinkConnectionService) | | **config-center** | config-center | Service | Global configuration center | HTTP + gRPC | | **account** | account | Multi-service | Account service | HTTP + gRPC (central/openapi/worker/provider/acl) | | **billing** | billing | Multi-service | Billing service | HTTP (server/worker/stress) | | **auth-server** | auth-server | Service | Authentication service | HTTP + gRPC | | **message-service** | message-service | Multi-service | Message service | HTTP + gRPC (api/central) | | **ecosystem-service** | ecosystem-service | Multi-service | Ecosystem service | HTTP (server/worker) | **Deployment Repository**: `infra-cd` --- ### 2. Regional Control Plane VPC Regional control plane deployed independently per region, responsible for regional resource management. | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **infra-api** | infra-api | Multi-service | Regional infrastructure API (infra-provider/config-manager/debug) | | **infra-provider** | infra-provider | Multi-service | Infrastructure provisioning (29 components) | | **cluster-service-ng** | cluster-service-ng | Multi-service | Cluster service NG (cluster_ng_server/executor/autoscale_stub) | | **core-svc** | core-svc | Multi-service | Core services (portal-server/svc-server/health-checker) | | **dataservices** | dataservices | Multi-service | Data services (dataapi/adminapi/gateway/regioncentral/central) | | **dbaas** | dbaas | Multi-service | DBaaS control services (core-server/billing2-server/worker) | | **serverless-service** | serverless-service | Service | Serverless service | | **tcidm** | tcidm | Service | Regional identity service (regional_tcidm_service) | | **metering** | metering | Multi-tool | Metering service (networkmetering/instancemetering/nextgenmetering) | | **devtier-store-service** | devtier-store-service | Multi-service | DevTier storage service | | **eventrouter** | eventrouter | Service | Event router | | **dataflow-service** | dataflow-service | Service | Dataflow service | **Deployment Repository**: `infra-cd` --- ### 3. Data Plane VPC (EKS/ACK/GKE) Data plane where user clusters actually run. #### 3.1 Kubernetes Infrastructure Layer | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **aws-shared-cd** | aws-shared-cd | Deployment | EKS cluster deployment (Pulumi) | | **eks-freetier** | aws-shared-cd | Deployment | EKS Free Tier deployment | | **ack-freetier** | aws-shared-cd | Deployment | Alibaba Cloud ACK Free Tier deployment | | **gke-provider** | gke-provider | Library | GCP GKE deployment support | **Deployment Repositories**: - `aws-shared-cd` (Primary for EKS/ACK) - `infra-cd` (Primary for GKE, uses `gke-provider` as library) #### 3.2 Operator Management Layer | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **tidb-operator** | tidb-operator | Operator | TiDB cluster lifecycle management (9 controllers) | | **tiflow-operator** | tiflow-operator | Operator | Data replication service management | | **tikv-operator** | tikv-operator | Operator | TiKV cluster management | | **devtier-tidb-operator** | devtier-tidb-operator | Operator | DevTier TiDB management | #### 3.3 Data Plane Shared Services (aws-shared-provider) > AWS-specific shared services for data plane operations. | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **tidb-gateway** | aws-shared-provider | Service | TiDB connection gateway | | **tidb-http-gateway** | aws-shared-provider | Service | HTTP connection gateway | | **gateway-controller** | aws-shared-provider | Service | Gateway controller | | **audit-log** | aws-shared-provider | Service | Audit log service | | **meta-server** | aws-shared-provider | Service | Metadata service | | **vpa-agent** | aws-shared-provider | Service | Vertical Pod Autoscaler agent | | **csi-node-taint** | aws-shared-provider | Service | CSI node taint management | | **tidb-worker** | aws-shared-provider | CLI | TiDB worker tool | | **ops-service** | aws-shared-provider | Service | Operations service | | **manager** | aws-shared-provider | CLI | Management CLI | | **cluster-agent** | infra-provider | Service | Cluster agent for control plane communication (gRPC: ClusterAgentService) | #### 3.4 Data Plane Shared Services (shared-provider) > Cloud-agnostic shared services. Note: `csi-node-taint` appears in both providers but may have cloud-specific implementations. | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **dispatcher** | shared-provider | Service | Request dispatcher | | **throttler** | shared-provider | Service | Rate limiter | | **config-service-mock** | shared-provider | Service | Config service mock | | **csi-node-taint** | shared-provider | Service | CSI node taint | #### 3.5 User TiDB Cluster Layer (per-user) | Component | Description | |-----------|-------------| | **TiDB Cluster** | SQL compute nodes | | **TiKV Cluster** | Distributed storage nodes | | **PD Cluster** | Cluster coordination and metadata | | **TiFlash Cluster** | Columnar storage analytics nodes | | **CDC/TiFlow** | Change data capture and replication | --- ### 4. O11y/Observability VPC Observability infrastructure for monitoring, logging, and tracing. #### 4.1 Metrics System | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **VictoriaMetrics** | VictoriaMetrics | Multi-service | Time-series database (vminsert/vmselect/vmstorage/vmagent/vmalert) | | **metrics-server** | metrics-server | Multi-service | Metrics service (central_metrics/scrape_gateway/push_pipeline) | | **mimir-cd** | mimir-cd | Deployment | Grafana Mimir deployment | | **vmbackupmanager** | observability | Service | VM backup manager | #### 4.2 Logging System | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **loki** | loki | Multi-service | Log storage and query | | **promtail** | loki | Service | Log collection agent | #### 4.3 Observability Platform Services | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **agent-control-server** | observability | Service | Agent control service | | **data-collector** | observability | Multi-service | Data collection (collector/processor) | | **diagnosis-query** | observability | Service | Diagnosis query service | | **diagnosis-compute** | observability | Service | Diagnosis compute service | | **globalserver** | observability | CLI | Global server | | **regionalserver** | observability | CLI | Regional server | | **opsserver** | observability | CLI | Operations server | | **taskcollector** | observability | CLI | Task collector | | **dataserver** | observability | CLI | Data server | | **mockmetricsserver** | observability | Service | Mock metrics server | #### 4.4 Monitoring Components | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **monitoring** | monitoring | Service | Monitoring system | | **reload** | monitoring | Service | Config hot reload | | **ng-monitoring** | ng-monitoring | CLI | TiDB monitoring component | | **datadog-on-cloud** | datadog-on-cloud | Library | DataDog cloud integration | **Deployment Repositories**: - `observability-cd` (Primary for Observability VPC) - `mimir-cd` (Mimir metrics deployment) - `clinic-cd` (Clinic diagnosis platform deployment) --- ### 5. Security VPC Security infrastructure VPC. | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **auditservice** | auditservice | Multi-service | Audit service (api/subscriber/worker) | | **InfraSec** | InfraSec | Deployment | Security deployment configuration | | **auth-server** | auth-server | Service | Authentication service | | **kube-oidc** | kube-oidc | Multi-service | K8s OIDC authentication (kube-oidc-proxy/login) | | **chaos-svc** | chaos-svc | Multi-service | Chaos engineering service | | **network-policy-controller** | infra-provider | Service | Network policy controller | > **Note:** `chaos-mesh-cd` was previously listed here but has been removed as it's marked as "nearly-empty" in the filtered repos list. --- ### 6. Customer VPC (BYOC) Customer's own cloud account VPC. | Component | Repository | Type | Responsibility | |-----------|------------|------|----------------| | **byoc-account-setup** | byoc-account-setup | Deployment | BYOC account initialization | **Connectivity**: VPC Peering / PrivateLink / Cloud Enterprise Network (Alibaba Cloud) --- ## Network Connectivity ### VPC Peering / Connectivity Matrix | Source VPC | Target VPC | Connection Method | Purpose | |------------|------------|-------------------|---------| | Global Control Plane | Regional Control Plane | VPC Peering / TGW | Global to regional control | | Regional Control Plane | Data Plane | VPC Peering / PrivateLink | Control plane to data plane | | Data Plane | Customer VPC (BYOC) | VPC Peering / PrivateLink | Data plane to customer | | Observability VPC | Data Plane | VPC Peering | Monitoring data collection | | Security VPC | All VPCs | VPC Peering | Security audit | ### Network Policies | Policy | Description | Enforcement | |--------|-------------|-------------| | **Control Plane → Data Plane** | Unidirectional access, control plane manages data plane resources | AWS Security Groups, K8s NetworkPolicy | | **Data Plane → Customer VPC** | Via PrivateLink/VPC Peering, customer access to TiDB clusters | VPC Peering rules, PrivateLink | | **Observability → Data Plane** | Read-only access for metrics and logs collection | K8s NetworkPolicy, IAM roles | | **Security → All VPCs** | Audit log collection and policy enforcement | VPC Peering, Security Groups | --- ## Deployment Responsibility Matrix | VPC Type | Primary Deployment Repo | Team | Deployment Frequency | |----------|------------------------|------|----------------------| | Global Control Plane | `infra-cd` | Platform Team | On-demand | | Regional Control Plane | `infra-cd` | Platform Team | On-demand | | Data Plane | `aws-shared-cd` (EKS/ACK), `infra-cd` (GKE) | Platform Team | New region / scale-out | | Observability | `observability-cd`, `mimir-cd`, `clinic-cd` | O11y Team | On-demand | | Security | `InfraSec` | Security Team | On-demand | | Customer VPC | `byoc-account-setup` | Customer + Support | New customer onboarding | --- ## Appendix: CD Deployment Repositories ### Core Deployment Repositories (5) > Primary deployment repositories for TiDB Cloud infrastructure. | CD Repository | Deployment Target | Tech Stack | |---------------|-------------------|------------| | **infra-cd** | Global/Regional Control Plane | Pulumi (TypeScript) | | **aws-shared-cd** | Data Plane (EKS/ACK) | Pulumi (TypeScript) | | **app-delivery** | Application service delivery | Shell | | **auto-deploy** | Auto deployment | Shell | | **oneconfig** | Configuration management | Shell | ### Extended Infrastructure Repositories > Specialized deployment repositories for specific infrastructure domains. | CD Repository | Deployment Target | Tech Stack | |---------------|-------------------|------------| | **observability-cd** | Observability VPC | Pulumi / Helm | | **mimir-cd** | Mimir monitoring | Pulumi (TypeScript) | | **clinic-cd** | Clinic diagnosis platform | Pulumi (TypeScript) | | **InfraSec** | Security infrastructure | TypeScript | | **byoc-account-setup** | Customer VPC | Terraform/CloudFormation | --- *Document Generated: 2026-03-26* *Based on knowledge base repository scan results*