# deployments/ — CD Manifests, Helm Charts & Region Lifecycle > K8s manifests, Helm charts, env-specific values, and scripts for deploying > the O11y platform across AWS/GCP/Azure/Alicloud regions. > Tooling: kubectl, Helm, GitHub Actions, ops-server API. --- ## Structure ``` deployments/ ├── basic/ # One-shot infra primitives (buckets, IAM, WAF, certs, ingress, PrivateLink) ├── charts/ # Helm charts — vendored upstream + in-repo custom (24 charts) ├── cloud-addon/ # Cloud-provider add-ons (ingress-nginx, aws-lb-controller, external-dns, cluster-autoscaler) ├── dataplane/ # Regional dataplane: VMCluster, Loki, Alertmanager, SLI reporter, ingress ├── drhouse/ # Regional services: auth-server, metering, diagnosis, deltalake, task-collector ├── ft/ # FT job manifests, cronjobs, datalink configs (test deployments, not test code) ├── globalserver/ # Global server manifests, Kong, Karma, Promxy, metrics pipelines ├── infra-v2/ # Cluster infra: kube-prometheus-stack, vector, velero, cert-manager, vm-operator, volume-autoscaler ├── lib/ # Shared shell helpers (diff-lib.sh, lib.sh) ├── regional-server/ # Templated per-region deploys for regional-server + data-server (envs/variants) ├── tests/ # Webconsole E2E config files (dev/staging/prod) ├── deploy-groups.txt # Canonical JSON: env → vendor → regions → clusters (source of truth) ├── diff-components.json # CI diff workflow component registry ├── register-new-region.sh # Add region to deploy-groups.txt └── unregister-region.sh # Orchestrate destroy across all components + remove from deploy-groups.txt ``` --- ## Key Concepts ### Manifest Types (from README.md) | Type | Lifecycle | Examples | |------|-----------|---------| | **One-shot** | Deploy once | VPC, EKS/GKE, NAT, DNS verification, GCP PSC | | **Static** | Rarely updated | Namespaces, CRDs, StorageClass, ServiceAccount, ingress-nginx, WAF | | **Dynamic** | Frequently updated | ConfigMaps, Deployments (GS/RS), ClusterRoles | --- ## Environment & Values Pattern Most components follow this structure: ``` component/ ├── common/ # Shared base values (optional) ├── envs/ │ ├── dev/ # regional-{vendor}-{logical_region}.yaml │ ├── staging/ # regional-{vendor}-{logical_region}.yaml │ └── prod/ # regional-{vendor}-{logical_region}.yaml (CI-diffed) ├── variants/ # {env}-{vendor}-values-tmpl.yaml (templates for new regions) ├── deploy-new-region.sh ├── destroy-region.sh └── diff.sh # (if registered in diff-components.json) ``` **Naming convention**: `{env}-{vendor}-{logical_region}.yaml` or `regional-{vendor}-{logical_region}.yaml` --- ## Region Lifecycle ### Register New Region ```bash # 1. Add to deploy-groups.txt bash deployments/register-new-region.sh [isByoc] [managedByOpsServer] # 2. Deploy components (order matters) bash deployments/infra-v2/deploy-new-region.sh bash deployments/cloud-addon/deploy-new-region.sh ... bash deployments/dataplane/deploy-new-region.sh ... bash deployments/regional-server/deploy-new-region.sh ... bash deployments/drhouse/deploy-new-region.sh ... bash deployments/globalserver/register-new-region.sh ... ``` - Set `managedByOpsServer` to `true` for regions that must deploy infra-v2 through `deploy-with-ops.sh`. Managed deploys always use the environment’s Global Server hostname and bootstrap automatically when the `ops-server` HelmRelease is absent. ### Unregister Region ```bash # Orchestrates destroy across all components + removes from deploy-groups.txt bash deployments/unregister-region.sh # Internally calls: infra-v2, regional-server, cloud-addon, drhouse, dataplane destroy-region.sh # Then: globalserver/unregister-region.sh # Finally: removes entry from deploy-groups.txt ``` ### Deploy Updates (Existing Regions) AWS BYOC regions do not deploy the O11Y `external-dns` release. Their read ingress uses a hostless rule so API Gateway can address the native load balancer DNS name. For an existing AWS BYOC region, deploy the hostless manifest before removing the `external-dns` release and legacy Route 53 records. ```bash # Component-level deploy (direct helm/kubectl) bash deployments/infra-v2/deploy.sh # Managed regions route through the ops-server API when managedByOpsServer=true in deploy-groups.txt MANAGED_BY_OPS_SERVER=true bash deployments/infra-v2/deploy.sh ... # Internally delegates to deploy-with-ops.sh, uses the environment's Global Server URL, and authenticates with Auth0 client credentials ``` --- ## CI Diff Workflow PR changes to prod env files trigger automated diffs posted as PR comments. ### How It Works 1. CI reads `diff-components.json` → builds matrix of (component, vendor) jobs 2. Each job runs `component/diff.sh [output_file]` 3. diff.sh sources `lib/diff-lib.sh` and calls `kubectl_diff` / `helm_diff` 4. Results aggregated into single PR comment ### Adding a New Component to Diff 1. Create `deployments//diff.sh` (source `lib/diff-lib.sh`, use `parse_diff_args`, `kubectl_diff`, `helm_diff`, `write_diff_summary`) 2. Register in `diff-components.json`: ```json {"name": "my-component", "path": "deployments/my-component", "description": "...", "chart_paths": ["deployments/charts/my-chart"]} ``` 3. See `ADDING_COMPONENT_DIFF.md` for full guide ### Run Diff Locally ```bash # Requires kubeconfig for target cluster bash deployments/dataplane/diff.sh aws prod us-east-1 us-east-1 /tmp/diff.md ``` --- ## Shared Libraries (`lib/`) | File | Functions | Purpose | |------|-----------|---------| | `lib.sh` | `helmrelease_exists()`, `create_or_update_helmrelease()`, `sed_inplace()` | Ops-server HelmRelease API helpers | | `diff-lib.sh` | `parse_diff_args()`, `kubectl_diff()`, `helm_diff()`, `filter_diff_noise()`, `has_meaningful_changes()`, `write_diff_summary()` | CI diff pipeline helpers | --- ## Charts (`charts/`) 24 Helm charts. Mix of vendored upstream and in-repo custom. ### In-Repo Custom Charts | Chart | Purpose | |-------|---------| | `regional-server` | Regional server + data server + resourceworker (main app chart) | | `ops-server` | Ops server | | `auth-server` | Authentication server | | `metering-server` | Metering server | | `diagnosis-query` | Diagnosis query service | | `deltalake-ops-server` | Deltalake operations server | | `o11ydiagnosis-deltalake` | O11Y diagnosis deltalake | | `o11ydiagnosis-logstats` | O11Y diagnosis log stats | | `o11y-mixin` | Prometheus recording rules and alerts mixin | ### Vendored Upstream Charts | Chart | Upstream | Notable | |-------|----------|---------| | `kube-prometheus-stack` | prometheus-community | Subcharts: grafana, kube-state-metrics, node-exporter | | `kong` | Kong/charts | Subchart: postgresql (Bitnami) | | `ingress-nginx` | kubernetes/ingress-nginx | Current version | | `ingress-nginx-v411` | kubernetes/ingress-nginx | Pinned v4.11 | | `cert-manager` | jetstack | | | `velero` | vmware-tanzu | Backup/restore | | `vector` | vectordotdev | Log pipeline | | `external-dns` | kubernetes-sigs | | | `aws-load-balancer-controller` | eks-charts | | | `alertmanager` | prometheus-community | | | `karma` | karma | Alert dashboard | | `promxy` | promxy | Prometheus federation | | `eventrouter` | heptio | K8s event routing | | `volume-autoscaler` | DevOps Nirvana | PVC autoscaler | | `loki-ha` | grafana | HA Loki (log storage) | ### Common Values Patterns ```yaml # Seen across most charts: global: environment: "" # dev/staging/prod cloudProvider: "" # aws/gcp/azure/alicloud region: "" logicalRegion: "" image: repository: "" tag: "" pullPolicy: IfNotPresent resources: requests: { cpu: "", memory: "" } limits: { cpu: "", memory: "" } service: type: ClusterIP ports: [] ingress: enabled: false className: "" serviceAccount: create: true name: "" crds: enabled: true upgrade: false # regional-server chart skipExisting: false ``` --- ## Component Directories — Quick Reference | Directory | Scope | Key Subcomponents | Has diff.sh? | |-----------|-------|-------------------|--------------| | `basic/` | One-shot infra | bucket, IAM, WAF, certs, ingress, PrivateLink | No | | `cloud-addon/` | Cloud add-ons | ingress-nginx (read/write), aws-lb-controller, external-dns, cluster-autoscaler | Yes | | `dataplane/` | Regional data | VMCluster, Loki, Alertmanager, dataplane-ingress, SLI reporter | Yes | | `drhouse/` | Regional services | auth-server, metering, diagnosis-query, deltalake, task-collector | No | | `globalserver/` | Global control plane | Kong, Karma, Promxy, metrics pipelines | No | | `infra-v2/` | Cluster infra | kube-prometheus-stack, vector, velero, cert-manager, vm-operator, volume-autoscaler, eventrouter | No | | `regional-server/` | App deploys | regional-server, data-server | Yes | --- ## deploy-groups.txt Canonical JSON mapping all environments/regions. Used by: - `register-new-region.sh` / `unregister-region.sh` - CI deploy workflows (matrix generation) Structure: `{ "": { "": [{ "region": "...", "logical_region": "...", "clusters": ["..."], "isByoc": "false" }] } }` Environments: `dev`, `staging`, `prod` **Vendors**: aws, gcp, azure, alicloud --- ## Rules - **Env file naming**: `{mode}-{vendor}-{logical_region}.yaml` (e.g., `regional-aws-us-east-1.yaml`) - **New region setup**: Always run `register-new-region.sh` first, then component deploy scripts in order - **New components**: Follow `ADDING_COMPONENT_DIFF.md` to integrate with CI diff - **deploy-with-ops.sh**: Used when `managedByOpsServer=true` — deploys via the Global Server `/ops/api/v1` proxy instead of direct helm - **BYOC clusters**: Handled via `isByoc` flag in deploy-groups.txt; skipped by CI diff - **No Chinese characters** in scripts or comments - **Bump Chart Version**: Always bump the chart version when changes manifests in the chart.