# ArgoCD self maintenance This directory contains the yaml manifests for ArgoCD self maintenance. ## Bootstrap 1. Setup a GKE cluster to host ArgoCD. Both the cluster and nodepool need to have workload identity enabled. The nodepool has `dedicated=argocd` label and following taint ```yaml - effect: NoSchedule key: dedicated value: argocd ``` 2. Deploy ArgoCD a. Create ArgoCD namespace: `kubectl create namespace argocd` b. Install ArgoCD in GKE: `kustomize build argocd/ | kubectl apply -n argocd -f -` c. Change default admin password https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli d. Retrieve the ingress IP address: `kubectl get ingress -n argocd` e. Request IT to configure DNS (`tidbcloud-cd.pingcap.net`) for the ingress IP address f. Request IT to provide SSL certificate for the DNS, and create a K8s secret for it: `kubectl -n argocd create secret tls tls-argocd-secret --cert=/path/to/cert/tidbcloud-cd.pingcap.net.crt --key=/path/to/key/tidbcloud-cd.pingcap.net.key` g. Configure [Google OAuth](https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/google/#openid-connect-using-dex). h. Configure [IAP](https://cloud.google.com/iap/docs/enabling-kubernetes-howto) 3. Configure GitHub token for `tidbcloud/app-delivery` repo, currently we are use tidbcloud-bot's token: `kubectl -n argocd create -f github-repo-secret.yaml` ``` yaml # github-repo-secret.yaml apiVersion: v1 kind: Secret metadata: name: app-delivery labels: argocd.argoproj.io/secret-type: repository stringData: url: https://github.com/tidbcloud/app-delivery.git username: bot password: ghp_xxxx_bot_token_xxxx # tidbcloud-bot github access token ``` 4. Register external-secrets-operator `ApplicationSet` to deploy external-secrets-operator (ESO) to `in-cluster` by `kubectl -n argocd apply -f ../app-registry/external-secrets-operator.yaml`. ESO is used to manage target cluster registration 5. Register `target-cluster-config` `ApplicationSet` to configure the target clusters that are to be managed: `kubectl -n argocd apply -f ../app-registry/argocd-target-clusters-config.yaml` ## Target cluster registration To allow ArgoCD to manage applications in an external K8s cluster, we need to do the following steps: 1. Create a Cloud identity (AWS IAM Role or GCP Service Account) for ArgoCD 2. Grant the ArgoCD identity to have admin access to the target k8s cluster. Remember to whitelist the NAT IP of ArgoCD GKE cluster. 3. Create a K8s `Secret` in `argocd` namespace with label `argocd.argoproj.io/secret-type=cluster` ### EKS Cluster 0. Create an OIDC Provider (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) Since we are running ArgoCD on GKE cluster and we don't want to use static IAM User credentials, ArgoCD pod can not use AWS IAM Role directly. We can use GKE as an external Identity Provider and let the AWS identity to trust the external identity. ```shell aws iam create-open-id-connect-provider \ --url "https://container.googleapis.com/v1/projects/pingcap-dbaas-build/locations/us-central1/clusters/cicd-cluster-us-central1" \ --client-id-list https://container.googleapis.com/v1/projects/pingcap-dbaas-build/locations/us-central1/clusters/cicd-cluster-us-central1 \ --thumbprint-list 08745487e891c19e3078c1f2a07e452950ef36f6 ``` The `thumbprint-list` is retrieved from the gke cluster provider. There are two options to get the thumbprint: * Retrieve from AWS console If we try to create an OIDC Provider in AWS console and input the `https://container.googleapis.com/v1/projects/pingcap-dbaas-build/locations/us-central1/clusters/cicd-cluster-us-central1` for *Provider URL*, and click *Get thumbprint*, the AWS web console would help us to retrieve this thumbprint. * Retrieve from CLI 1. Get jwks_uri FQDN by `curl https://container.googleapis.com/v1/projects/pingcap-dbaas-build/locations/us-central1/clusters/cicd-cluster-us-central1/.well-known/openid-configuration | jq .jwks_uri`. The FQDN is `container.googleapis.com` 2. Get the top intermediate CA in the certificate authority chain from FQDN `openssl s_client -servername container.googleapis.com -showcerts -connect container.googleapis.com:443`. Copy and save the last certificate in the command output as `certificate.crt` 3. Get the certificate thumbprint by `openssl x509 -in certificate.crt -fingerprint -noout | awk -F'=' '{print $2}'`. Remove the colon characters and we'll get the thumbprint as with the one from AWS console except it's uppercase. See the AWS official document for more explanation https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html 1. Create an IAM Role for ArgoCD. Currently this is done manually, since we only have to do this twice one for dev&stg AWS Account and one for prod AWS controlplane Account. We are using these two IAM Roles: `arn:aws:iam::385595570414:role/argocd-oidc-role` and `arn:aws:iam::380838443567:role/argocd-oidc-role`. We need to allow the external identity (GKE identity provider) to assume this IAM Role. This is done by adding the following trust policy to this IAM Role. Remember to change th AWS account id in this policy according to the IAM Role AWS account. `aws iam create-role --role-name argocd-oidc-role --assume-role-policy-document file://trust-policy.json` ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::385595570414:oidc-provider/container.googleapis.com/v1/projects/pingcap-dbaas-build/locations/us-central1/clusters/cicd-cluster-us-central1" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "container.googleapis.com/v1/projects/pingcap-dbaas-build/locations/us-central1/clusters/cicd-cluster-us-central1:aud": "https://container.googleapis.com/v1/projects/pingcap-dbaas-build/locations/us-central1/clusters/cicd-cluster-us-central1" } } } ] } ``` 2. Grant the IAM Role to have admin access to the EKS clusters we are going to manage. This is done by adding the role to the target EKS cluster's `kube-system` namespace `aws-auth` ConfigMap (ref https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html). Since we have lots of EKS clusters, this step is automated by Pulumi in https://github.com/tidbcloud/infra/blob/3bdebc573c19f59e39e1b73760dff997a903b5a8/deploy/eks/Pulumi.dev-base-us-west-2.yaml#L7 ```yaml data: mapRoles: | ... - rolearn: 'arn:aws:iam::385595570414:role/argocd-oidc-role' username: argocd-oidc-role groups: - 'system:masters' ``` 3. Create a K8s `Secret` in ArgoCD's GKE `argocd` namespace with label `argocd.argoproj.io/secret-type=cluster` The K8s `Secret` tells ArgoCD how to access the target EKS cluster. We can treat it as kubeconfig, though a bit different format. Following is an example ```yaml apiVersion: v1 kind: Secret metadata: name: staging-us-west-2 namespace: argocd labels: argocd.argoproj.io/secret-type: cluster type: Opaque stringData: name: staging-seed-us-west-2 # an easy to remember name, doesn't have to be the same as k8s cluster name from cloud provider server: https://08A...22C.gr7.us-west-2.eks.amazonaws.com # eks apiserver url config: | { "execProviderConfig": { "command": "argocd-k8s-auth", "args": [ "aws", "--cluster-name", "staging-seed-us-west-2" ], "env": { "AWS_ROLE_ARN": "arn:aws:iam::385595570414:role/argocd-oidc-role" }, "apiVersion": "client.authentication.k8s.io/v1beta1" }, "tlsClientConfig": { "insecure": false, "caData": "" } } ``` We need to patch argocd-server and argocd-application-controller Deployment to have following env: ```yaml - name: AWS_WEB_IDENTITY_TOKEN_FILE value: /var/run/secrets/kubernetes.io/serviceaccount/token ``` This is done by the argocd bootstrap kustomize `argocd/env-aws.yaml`. We manage this secret in ArgoCD itself with external-secrets-operator. The `cluster-secret/xxx-cluster-secret.yaml` files are `ExternalSecret` resource for the clusters that are managed by ArgoCD. The `ExternalSecret` will be reconciled by external-secret-operator deployed in the ArgoCD GKE cluster to convert to a K8s `Secret`. The real secret data is stored in JSON format in `pingcap-dbaas-build` GCP project Secret Manager https://console.cloud.google.com/security/secret-manager?authuser=0&project=pingcap-dbaas-build&supportedpurview=project. Eg: `gcloud secrets create staging-us-west-2-cluster-config --data-file staging-us-west-2-cluster-config.json --project pingcap-dbaas-build` ```json { "name": "staging-seed-us-west-2", "server": "https://08A...22C.gr7.us-west-2.eks.amazonaws.com", "config": { "execProviderConfig": { "command": "argocd-k8s-auth", "args": [ "aws", "--cluster-name", "staging-seed-us-west-2" ], "env": { "AWS_ROLE_ARN": "arn:aws:iam::385595570414:role/argocd-oidc-role" }, "apiVersion": "client.authentication.k8s.io/v1beta1" }, "tlsClientConfig": { "insecure": false, "caData": "" } } } ``` ### GKE Cluster 1. Create a GCP identity (Service Account) for ArgoCD We created the GCP Service Account `tidbcloud-cd-argocd@pingcap-dbaas-build.iam.gserviceaccount.com` for ArgoCD. ArgoCD is deployed in a GKE cluster, to allow ArgoCD pod use this Service Account, we enabled workload identity for the GKE cluster and nodepool. So we need two more steps: * Allow `argocd-application-controller` and `argocd-server` k8s service account to impersonate GCP Service Account `tidbcloud-cd-argocd@pingcap-dbaas-build.iam.gserviceaccount.com` ```shell gcloud iam service-accounts add-iam-policy-binding tidbcloud-cd-argocd@pingcap-dbaas-build.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:pingcap-dbaas-build.svc.id.goog[argocd/argocd-application-controller]" --project pingcap-dbaas-build gcloud iam service-accounts add-iam-policy-binding tidbcloud-cd-argocd@pingcap-dbaas-build.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:pingcap-dbaas-build.svc.id.goog[argocd/argocd-server]" --project pingcap-dbaas-build ``` * Patch the `argocd-application-controller` and `argocd-server` k8s service account with annotation `iam.gke.io/gcp-service-account=tidbcloud-cd-argocd@pingcap-dbaas-build.iam.gserviceaccount.com`. This is already done in the argocd bootstrap kustomize `argocd/sa-annotation.yaml` 2. Grant the ArgoCD identity to have admin access to the target k8s cluster Grant the Service Account `tidbcloud-cd-argocd@pingcap-dbaas-build.iam.gserviceaccount.com` "Kubernetes Engine Admin" Role in `pingcap-gardener` and `dbaas-prod` GCP Project. 3. Create a K8s `Secret` in ArgoCD's GKE cluster `argocd` namespace with label `argocd.argoproj.io/secret-type=cluster` The K8s `Secret` tells ArgoCD how to access the target EKS cluster. We can treat it as kubeconfig, though a bit different format. Following is an example ```yaml apiVersion: v1 kind: Secret metadata: name: staging-us-central1 namespace: argocd labels: argocd.argoproj.io/secret-type: cluster type: Opaque stringData: name: staging-seed-us-central1 # an easy to remember name, doesn't have to be the same as k8s cluster name from cloud provider server: https://35.xx.xx.203 # gke apiserver url config: | { "execProviderConfig": { "command": "argocd-k8s-auth", "args": [ "gcp" ], "apiVersion": "client.authentication.k8s.io/v1beta1" }, "tlsClientConfig": { "insecure": false, "caData": "" } } ``` We manage this secret in ArgoCD itself with external-secrets-operator. The `cluster-secret/xxx-cluster-secret.yaml` files are `ExternalSecret` resource for the clusters that are managed by ArgoCD. The `ExternalSecret` will be reconciled by external-secret-operator deployed in the ArgoCD GKE cluster to convert to a K8s `Secret`. The real secret data is stored in JSON format in `pingcap-dbaas-build` GCP project Secret Manager https://console.cloud.google.com/security/secret-manager?authuser=0&project=pingcap-dbaas-build&supportedpurview=project. Eg: `gcloud secrets create prod-us-central1-cluster-config --data-file prod-us-central1-cluster-config.json --project pingcap-dbaas-build` ```json { "name": "prod-seed-us-central1", "server": "https://35.xx.xx.203", "config": { "execProviderConfig": { "command": "argocd-k8s-auth", "args": [ "gcp" ], "apiVersion": "client.authentication.k8s.io/v1beta1" }, "tlsClientConfig": { "insecure": false, "caData": "" } } } ``` ### Infra APIServer 1. Create a Cloud identity for ArgoCD The infra apiserver is a self-managed etcd + kube-apiserver. Currently, we are using static kubeconfig for infra-apiserver. So there is no Cloud identity involved. But in the future we might create separate K8s serviceaccounts for different applications and generate a kubeconfig using the serviceaccount (https://veducate.co.uk/creating-a-kubeconfig-file-for-a-kubernetes-service-account/). 2. Grant the ArgoCD identity to have admin access to the infra apiserver Currently we are using the same admin kubeconfig of infra-apiserver. So nothing to do in this step. 3. Create a K8s `Secret` in ArgoCD's GKE `argocd` namespace with label `argocd.argoproj.io/secret-type=cluster` ```yaml apiVersion: v1 kind: Secret metadata: name: dev-infra-api namespace: argocd labels: argocd.argoproj.io/secret-type: cluster type: Opaque stringData: name: dev-infra-api server: https://us-west-2.xxx.com:443 config: | { "tlsClientConfig": { "insecure": false, "caData": "", "certData:" "", "keyData": "" } } ``` The `cluster/secret/xx-infra-api-cluster-secret.yaml` files are `ExternalSecret` resource for the infra-apiservers that are managed by ArgoCD. The `ExternalSecret` will be reconciled by external-secret-operator deployed in the ArgoCD GKE cluster to convert to a K8s `Secret`. The real secret data is stored in JSON format in `pingcap-dbaas-build` GCP project Secret Manager https://console.cloud.google.com/security/secret-manager?authuser=0&project=pingcap-dbaas-build&supportedpurview=project. Eg: `gcloud secrets create dev-infra-api-cluster-config --data-file dev-infra-api-cluster-config.json --project pingcap-dbaas-build` ```json { "name": "dev-infra-api", "server": "https://us-west-2.xxx.com:443", "config": { "tlsClientConfig": { "insecure": false, "caData": "", "certData:" "", "keyData": "" } } } ```