We are moving dev & stg applications from prod ArgoCD to dev/stg ArgoCD, this may cause PRs blocked from merging. See https://pingcap.feishu.cn/wiki/AupmwcEcuiI3UckTbo8ctYe4nZe for more detail. # app-delivery TiDB Cloud services YAML manifests repo. We use [ArgoCD](https://github.com/argoproj/argo-cd/) to synchronize application configuration manifests to all our environments. The ArgoCD Web UI is on (Feilian is required) * prod: https://tidbcloud-cd.pingcap.net * dev: https://dev-tidbcloud-cd.pingcap.net * staging: https://staging-tidbcloud-cd.pingcap.net ## Workflow ### Add new service deployment 1. Create a new service directory in the root directory: `mkdir my-awesome-service` 2. Create a `base` directory in the service directory: `mkdir my-awesome-service/base` 3. Put the service manifests in the `my-awesome-service/base` directory. For tokens and credentials, we should save them in K8s `Secret` and reference the `Secret` in `volumeMounts` or `env`. But please **DO NOT** store the Secret directly in git, instead use `ExternalSecret` described in [Secret Management](#Secret-Management) section. 4. Create a `kustomization.yaml` file in `my-awesome-service/base` directory: `kustomize create --autodetect` 5. Create a `envs` directory in the service directory: `mkdir my-awesome-service/envs`. 6. Create an overlay directory, usually the overlay directory is named as `{env}-{region}`. For example: `dev-us-west-2`, `stg-us-central1`. Run `kustomize build my-awesome-service/envs/{env}-{region}` to check if the generated manifests match your expectation. 7. Register your service to the target cluster by creating an applicationset file in `app-registry` directory. You can take existing applicationset as an example. You'd better deploy in one `{env}-{region}` for the first time you add a new service, and then add more envs and regions in later PRs. 7. Send a PR to deploy your awesome service to your desired K8s cluster. Once the PR is merged, you can see the application deployment status in https://tidbcloud-cd.pingcap.net. If the deployment fails, the ArgoCD Web UI should have enough info (the k8s event, live yaml and pod logs) for you to debug. 8. Once the service is deployed successfully, you might want to deploy it in more envs and regions. Just create other overlay directories and adding it in `app-registry/my-awesome-service.yaml` ### Update existing application With kustomize, we use different folders for different environments, this allows us to update applications gradually by region, by env. Common mistakes and solution: * Adding or updating a common label for application. Since kustomize can add common labels to all manifests including `Deployment` and `StatefulSet`, updating common labels to already deployed `Deployment` and `StatefulSet` would also add it to `spec.selector.matchLabels`. However `spec.selector.matchLabels` field is immutable. So the updating would fail. Solution: TODO * Modify `base` directory manifests directly. When modify the kustomize `base` directory manifests directly, all existing deployed applications would be triggered to update including prod. This is of course very dangerous. Solution: **DO NOT** edit kustomize `base` directly. Instead update the specific env configurations one by one or in batch, when every environment has been rollout, move the common changes to kustomize `base` configuration. # Secret-Management We don't store K8s Secret directly in the git repository, neither plaintext nor base64 encoded values are secure in private git repository. However we do still use K8s Secret for applications. The raw K8s Secret is replaced with External-Secrets-Operator's (ESO) `ExternalSecret` CR. And the real token and credential values are stored in Cloud Vendor's Secret Manager Service. The ESO will reconcile the `ExternalSecret` to generate a K8s Secret. Using Cloud Vendor's Secret Manager, we can setup an automated rotation mechanism without the application's awareness in the future. This can greatly enhance our security. Generally, there are two usage scenarios for Secret: mount the whole secret as file in container, reference the specific secret key-value in container env. ## Mount the whole Secret as File If we want to store sensitive data in a file and reference the whole data as a K8s Secret value. For example, we want to use raw plaintext file of infra-api kubeconfig in a pod. The K8s Secret looks like this ``` yaml apiVersion: v1 kind: Secret metadata: name: dev-infra-api-kubeconfig type: Opaque stringData: config: | ``` And we mount it on `/.kube/config` in pod as below ``` yaml spec: containers: - name: eks-provider volumeMounts: - mountPath: /.kube name: kubeconfig ... volumes: - name: kubeconfig secret: secretName: dev-infra-api-kubeconfig ``` To do this in gitops style, we need to create a secret in AWS/GCP Secret Manager. Suppose we name it as `dev-infra-apiserver-kubeconfig` on AWS Secret Manager, and paste the content of dev infra-api kubeconfig into it. Then we reference it in `ExernalSecret` as below and no need to change application Deployment. ``` yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: dev-infra-api-cluster-secret spec: refreshInterval: 1h # rate ESO pulls cloud vendor's secret manager secretStoreRef: kind: ClusterSecretStore name: default target: name: dev-infra-api-kubeconfig # name of the k8s Secret to be created creationPolicy: Owner data: - secretKey: config # the k8s secret key remoteRef: key: dev-infra-apiserver-kubeconfig # name of AWS Secret Manager Secret ``` ## Reference Secret Key-Value Pair as Env Sometimes we want to use several related secret values, such as auth0 client id and token. We want to reference the specific key-value pair in pod spec, such as container env. The K8s Secret is in this form: ``` yaml apiVersion: v1 kind: Secret metadata: name: proxy-auth0-client type: Opaque stringData: proxyAuth0ClientID: proxyAuth0ClientSecret: ``` And reference the two key-value pair value in pod spec: ``` yaml spec: containers: - name: eks-provider env: - name: PROXY_AUTH0_CLIENT_ID valueFrom: secretKeyRef: key: proxyAuth0ClientID name: proxy-auth0-client - name: PROXY_AUTH0_CLIENT_SECRET valueFrom: secretKeyRef: key: proxyAuth0ClientSecret name: proxy-auth0-client ``` *Security Tips: for secret as env, always use k8s secret and reference the secret as above. DO NOT put the env value directly in pod spec, otherwise it would be easily leaked.* To do this in gitops style, we create an AWS Secret Manager secret. Suppose we name it as `dev-proxy-auth0-client`, store the two key-value pairs as json format: `{"proxyAuth0ClientID": "", "proxyAuth0ClientSecret": ""}`. Then we reference it in `ExernalSecret` as below and no need to change application Deployment. ``` yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: dev-infra-api-cluster-secret spec: refreshInterval: 1h # rate ESO pulls cloud vendor's secret manager secretStoreRef: kind: ClusterSecretStore name: default target: name: proxy-auth0-client # name of the k8s Secret to be created creationPolicy: Owner dataFrom: - extract: key: dev-proxy-auth0-client # name of the AWS Secrt Manager secret ``` # Advanced The above workflow is focusing on [Kustomize](https://kustomize.io/). However ArgoCD also supports Helm Chart, Jsonnet and plain YAML/JSON manifests. Read the ArgoCD official document if you want to use these format. It is suggested to use Kustomize for our own services and Helm Chart for third-party applications if the upstream provides Helm Chart format. The Kustomize overlay configuration can be natually mapped to our different environments and regions. # Develop ## CRD & Webhook To sync CRD and Webhook from [infra-provider](https://github.com/tidbcloud/infra-provider) ``` shell make branch=main stage=dev sync-infra-api-crd make branch=main stage=dev sync-infra-api-webhook # prod, for example make branch=new-arch/release-0.94 stage=prod sync-infra-api-crd make branch=new-arch/release-0.94 stage=prod sync-infra-api-webhook ```