# DBaaS Operation 101 This document will describe how to access and explore the DBaaS infrastructure. ## Environments Generally we have different environments that hosts DBaaS backend instances for different purposes. Each environment may host multiple DBaaS instances: - [dev](../../deploy/garden/dev-acre.sh): hosts all **local development** DBaaS backend instances and the nightly instance, which is the default backend of [dev.tidbcloud.com](https://dev.tidbcloud.com/console). For developers, you are most likely to be interested in this environment. - [stability](../../deploy/garden/stability-acre.sh): hosts the instance for stability test and the instance for e2e test. - [staging](../../deploy/garden/stability-acre.sh): hosts the **staging** instance, which is the backend of [staging.tidbcloud.com](https://staging.tidbcloud.com/). All releases will be firstly tested in staging before released to production. - [prod](../../deploy/garden/prod-acre.sh): hosts the **prod** instance, which is the backend of the production - [tidbcloud.com](https://tidbcloud.com). ## Prerequisites These tools are necessary to access the DBaaS infrastructure: - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) 1.16.73+ - [aws-iam-authenticator](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html) - [gcloud](https://cloud.google.com/sdk/gcloud) - [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) 1.12+ - [Kubectx](https://github.com/ahmetb/kubectx#kubectl-plugins-macos-and-linux) (You can use [Krew] to install) - [konfig](https://github.com/corneliusweig/konfig#installation) (You can use [Krew] to install) ## Setup Accounts All environments are deployed in the cloud. You need an AWS and GCP account to access. With a dev account, you can access `dev`, `stability` and `staging` environments. With a prod account, you can access the `prod` environment. Generally, developers only need `dev` accounts. If you don't have any cloud account, you can apply it by open **Feishu - Workspace - Approval** to submit a new request. Select the `Operation and maintenance account` in the IT section and choose **Cloud Resource Request** as application type and **Cloud IAM User** as resource type, complete all the fields then submit it and wait for approval. PingCAP IT Service will contact you with your _AWS Login Credential_, _AWS Access Key_ and _AWS Access Secret_. - _AWS Login Credential_: You can use this credential to log into the AWS web console at https://pingcap.signin.aws.amazon.com/console - _AWS Access Key_ and _AWS Access Secret_: Used for accessing the AWS API from command-line interface later. Don't lose it. ## Setup Credentials ### AWS You can use [aws-vault](https://github.com/99designs/aws-vault) to access AWS safely from command-line: ```shell $ aws-vault add dev Enter Access Key Id: Enter Secret Key: ``` If you have prod accounts, you can add it to the aws-vault as well: ```shell $ aws-vault add prod Enter Access Key Id: Enter Secret Key: ``` Remove the plaintext credentials in `$HOME/.aws/credentials` if they exist. Then, you can prefix your `aws` commands with the `aws-vault exec ${env} --` prefix to specify the account to use. For example, to get current IAM user using your `dev` account ```shell aws-vault exec dev -- aws sts get-caller-identity ``` To retrieve the EKS cluster list in region `us-west-2` using your `dev` account: ```shell aws-vault exec dev -- aws eks list-clusters --region us-west-2 ``` **NOTE**: without special instructions, all `aws` commands are supposed to be executed under `aws-vault`, for example, `aws-vault exec ${env} -- aws sts get-caller-identity` instead of `aws sts get-caller-identity`. ### GCP ```shell gcloud auth application-default login ``` **NOTE**: without special instructions, all `gcloud` commands are supposed to be executed after you login with the proper google account. ## How to Access Environments ### 1. Grant Permissions By default your dev or prod account does not have permission to access the corresponding environment. - Request permission for dev cluster: add the login username of your _AWS Login Credential_ in the `deploy/eks/Pulumi.dev-base-us-west-2.yaml` and submit the PR, like [this](https://github.com/tidbcloud/infra/pull/541/files). - Request permission for staging cluster: similarly, add your name in the `deploy/eks/Pulumi.staging-base-us-west-2.yaml`. After the PR is merged, you need to cherry pick it into staging branch with `/cherry-pick staging` like [this](https://github.com/tidbcloud/infra/pull/1361). You will have permission to access the corresponding cluster after the PR merge and CD complete. ### 2. Retrieve Kubeconfig To use `kubectl` to access K8s clusters in these environments, you need Kubeconfig files. We have some handy scripts for you to set them up, see below. > Kubeconfig will be stored in the default location `$HOME/.kube/config`. **2.1 Retrieve and store Kubeconfig for Base and Seed cluster in AWS EKS cluster:** ```shell aws-vault exec dev -- hack/aws-kubeconfig.sh ``` > You need to change `dev` into `prod` if you need to access prod environment. **2.2 Retrieve and store Kubeconfig for the Gardener cluster:** ```shell BASE_ENV=dev aws-vault exec dev -- hack/gardener.sh # If you want to retrieve the staging Gardener kubeconfig: BASE_ENV=staging aws-vault exec dev -- hack/gardener.sh ``` > You need to change `dev` into `prod` if you need to access prod environment. **2.3 Retrieve and store Kubeconfig for Base and Seed cluster in GCP GKE cluster:** ```shell # for dev $ gcloud config set project pingcap-gardener $ ./hack/gcp-kubeconfig.sh # for prod $ gcloud config set project dbaas-prod $ ./hack/gcp-kubeconfig.sh ``` > See details in [GCP Configuration](https://www.jhanley.com/google-cloud-understanding-gcloud-configurations/). ### 3. Verify Access To see all clusters that you can access now: ```shell kubectl config get-contexts --output=name ``` Sample output: ```text arn:aws:eks:us-east-1:xxxx:cluster/dev-seed-us-east-1 arn:aws:eks:us-west-2:xxxx:cluster/dev-base-us-west-2 arn:aws:eks:us-west-2:xxxx:cluster/dev-seed-us-west-2 arn:aws:eks:us-west-2:xxxx:cluster/stability-base-us-west-2 ...... ``` Then you can try to connect to one of the cluster and list objects using `kubectl`: ```shell # Switch to use the `dev-base` cluster. # You can also execute `kubectl ctx` to select a cluster (context) interactively. kubectl ctx `kubectl config get-contexts --output=name | grep dev-base` # List all pods in the nightly DBaaS backend in dev-base cluster. kubectl get po -n nightly ``` Sample output: ```text NAME READY STATUS RESTARTS AGE backup-tidb-backup-x... 0/1 Completed 0 9h billing-dtu69ys2-6b575cf6fb-65mgh 1/1 Running 0 87m central-lzqm1kgd-54c5558d58-q4twl 1/1 Running 0 87m db-discovery-5777d57db7-47ns4 1/1 Running 0 112d db-pd-0 1/1 Running 0 127d db-tidb-0 2/2 Running 0 119d db-tikv-0 1/1 Running 1 127d monitor-monitor-77985496d-hc9fj 3/3 Running 0 113d ``` ## How to Access User Cluster You can use Kubectl to access a user cluster deployed in a shoot. - A shoot is deployed for each `(tenant, cloudRegion)`. - Each user cluster is deployed in a shoot. ### 1. Get the Shoot ID of the Cluster from Thanos For dev environment, visit https://thanos-query.oauth.dev.shared.aws.tidbcloud.com/ and query `dbaas_tidb_cluster_info{provider="aws",namespace="$PROJECT"}` where PROJECT is usually your username. From the query result you can see the shoot id that each cluster is deployed in via the `shoot` label. > Note: In dev environment all DBaaS backend shares the same Thanos service. You don't need to change the URL for your own DBaaS backend instance. ### 2. Import Shoot Kubeconfig ```shell kubectl --context garden get secret -n garden-nightly .kubeconfig --template={{'.data.kubeconfig'}} | base64 -d | tee shoot_config konfig import --save shoot_config ``` **Example: Get the Kubeconfig for the `c138a5fe` shoot in the nightly DBaaS backend:** ```shell kubectl --context garden get secret -n garden-nightly c138a5fe.kubeconfig --template={{'.data.kubeconfig'}} | base64 -d | tee shoot_config kubectl konfig import --save shoot_config ``` ### 3. Connect to the Shoot After Kubeconfig is imported, you can switch Kubectl to access this K8s shoot cluster by using: ```shell kubectl ctx shoot---- ``` **Example: Explore the `c138a5fe` shoot in the nightly DBaaS backend:** ```shell # Switch to the shoot kubectl ctx shoot--nightly--c138a5fe # You may also select interactively: kubectl ctx # > Switched to context "shoot--nightly--c138a5fe". # ======= # List clusters in the shoot kubectl get ns # > NAME STATUS AGE # > cert-manager Active 186d # > crossplane-system Active 136d # > default Active 186d # > kube-node-lease Active 186d # > kube-public Active 186d # > kube-system Active 186d # > logging Active 186d # > tidb-admin Active 186d # > tidb1288802679551692800 Active 186d # > tidb1291018570389524480 Active 180d # > tidb1297729982109323264 Active 162d # > tidb1297773330761256960 Active 162d # > tidb1297787037612511232 Active 162d # > ... # ======= # Explore a specific cluster by cluster ID. # Note: You can learn the cluster ID from the Thanos query result as well. kubectl ns tidb1356426856680656896 # > Context "shoot--......." modified. # > Active namespace is "tidb1356426856680656896". kubectl get po # > NAME READY STATUS RESTARTS AGE # > db-discovery-6858... 1/1 Running 0 5h3m # > db-monitor-77d89c... 3/3 Running 0 5h3m # > db-pd-0 1/1 Running 0 5h3m # > db-tidb-0 2/2 Running 0 5h2m # > db-tidb-extra-acc... 2/2 Running 0 5h3m # > db-tidb-initializ... 0/1 Completed 0 5h1m # > db-tikv-0 1/1 Running 0 5h3m # > db-tikv-1 1/1 Running 0 5h3m # > db-tikv-2 1/1 Running 0 5h3m ``` ### Advanced: Retrieve Kubeconfigs Manually
Click to expand The following diagram shows how these kubeconfigs are associated: ```text +-----------+ |ENV-acre.sh| ++----------+ | +---------------------------------------------+ +--------------+ +>+ eks_kubeconfig us-west-2 ENV-base-us-west-2 +-->+ Base Cluster | | ++--------------------------------------------+ +--------------+ | | +----------------------------------------------------------+ +--------------+ | | | kubectl get secret -n garden garden-kubeconfig-for-admin +-->+ Gardener API | | +->| --template={{'.data.kubeconfig'}} | base64 -d | +--------------+ | +----------------------------------------------------------+ | | +-------------------------------------------------------+ +---------------+ | | | kubectl get secret -n garden-PROJECT SHOOT.kubeconfig +-->+ Shoot Cluster | | +->| --template={{'.data.kubeconfig'}} | base64 -d | +---------------+ | +-----------------------------------+-------------------+ | +---------------------------------------+ +--------------+ +>+ eks_kubeconfig REGION ENV-seed-REGION +-------->+ Seed Cluster | | +---------------------------------------+ +--------------+ | +---------------------------------------------+ +--------------+ +>+ gke_kubeconfig GCP_PROJECT REGION ENV-NAME +-->+ Seed Cluster | +---------------------------------------------+ +--------------+ ``` The `eks_kubeconfig` function refers to `aws eks --region update-kubeconfig --name --kubeconfig `, and the `gke_kubeconfig` function refers to `KUBECONFIG= gcloud container clusters get-credentials --region --project `. **Examples:** Take `dev-acre.sh` as an example, you can explore the landscape using the following commands: ```shell # get kubeconfig of base cluster $ aws eks --region us-west-2 update-kubeconfig --name dev-base-us-west-2 --kubeconfig dev-base-us-west-2 # get kubeconfig of Gardener API $ KUBECONFIG=dev-base-us-west-2 kubectl get secret -n garden garden-kubeconfig-for-admin --template={{'.data.kubeconfig'}} | base64 -d > gardener-api # get kubeconfig of a shoot $ KUBECONFIG=gardener-api kubectl get secret -n garden-nightly 25bafc01.kubeconfig --template={{'.data.kubeconfig'}} | base64 -d > shoot-25bafc01.kubeconfig # get kubeconfig of aws seed us-west-2 $ aws eks --region us-west-2 update-kubeconfig --name dev-seed-us-west-2 --kubeconfig dev-seed-us-west-2 # Explore BASE $ export KUBECONFIG=dev-base-us-west-2 # dbaas components $ kubectl get po -n nightly NAME READY STATUS RESTARTS AGE billing-dtu69ys2-69bbbdffd7-gw2qr 1/1 Running 0 3h42m central-lzqm1kgd-c974cc547-clwzp 1/1 Running 0 3h42m # Gardener ApiServer components $ kubectl get deployment -n garden garden-kube-apiserver 3/3 3 3 81d gardener-apiserver 1/1 1 1 54d gardener-controller-manager 1/1 1 1 54d gardener-resource-manager 1/1 1 1 80d gardener-scheduler 1/1 1 1 54d # Explore Gardener API $ export KUBECONFIG=gardener-api # Shoot list $ kubectl get shoot -n garden-nightly NAME CLOUDPROFILE VERSION SEED DOMAIN HIBERNATION OPERATION PROGRESS APISERVER CONTROL NODES SYSTEM AGE 1ba126c3 aws 1.16.9 aws-us-east-1 1ba126c3.nightly.shoot.dev.shared.aws.tidbcloud.com Awake Succeeded 100 True True True True 18d 23645c95 gcp 1.16.9 gcp-us-west2 23645c95.nightly.shoot.dev.shared.aws.tidbcloud.com Awake Succeeded 100 True True True True 77d # Seed list $ kubectl get seed NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION aws-us-west-2 Ready aws us-west-2 80d v1.4.1-dev.13 v1.16.13-eks-2ba888 gcp-us-west1 Ready gcp us-west1 68d v1.4.1-dev.13 v1.16.11-gke.5 # Explore Shoot $ export KUBECONFIG=shoot-25bafc01.kubeconfig # tidb-operator and dbaas local controllers of shoot $ kubectl get deployment -n tidb-admin NAME READY UP-TO-DATE AVAILABLE AGE dbaas-proxy 2/2 2 2 78d dbaas-reporter 2/2 2 2 78d tidb-controller-manager 1/1 1 1 78d tidb-scheduler 1/1 1 1 78d # user's TiDB cluster $ kubectl get statefulsets.apps.pingcap.com -n tidb1297778711088599040 NAME DESIRED CURRENT AGE db-pd 3 3 6h23m db-tidb 2 2 6h19m db-tiflash 3 3 6h19m db-tikv 3 3 6h20m # Explore Seed $ export KUBECONFIG=dev-seed-us-west-2 # Gardenlet $ kubectl get po -n garden NAME READY STATUS RESTARTS AGE gardenlet-6c79b57447-hzwwr 1/1 Running 0 5h15m # Extension provider $ kubectl get po -n extension-provider-aws-h6tzp NAME READY STATUS RESTARTS AGE gardener-extension-provider-aws-66bfb8899c-bwgn7 1/1 Running 0 5h16m # ControlPlane of Shoot $ kubectl get deployment -n shoot--nightly--25bafc01 NAME READY UP-TO-DATE AVAILABLE AGE cloud-controller-manager 1/1 1 1 78d kube-apiserver 1/1 1 1 78d kube-controller-manager 1/1 1 1 78d kube-scheduler 1/1 1 1 78d machine-controller-manager 1/1 1 1 78d # Machinedeployment and machine of Shoot $ kubectl get machinedeployment -n shoot--nightly--25bafc01 NAME READY DESIRED UP-TO-DATE AVAILABLE AGE shoot--nightly--25bafc01-99d6609-z1 1 1 1 1 10d shoot--nightly--25bafc01-99d660a-z1 1 1 1 1 10d shoot--nightly--25bafc01-99d660b-z1 1 1 1 1 10d ... $ kubectl get machine -n shoot--nightly--25bafc01 NAME STATUS AGE shoot--nightly--25bafc01-99d6609-z1-5d65557c8c-cn75m Running 7d6h shoot--nightly--25bafc01-99d660a-z1-6d9dc8f469-tgssw Running 7d6h shoot--nightly--25bafc01-99d660b-z1-7f5df95ff7-fgf67 Running 7d6h ... # Infra status of Shoot $ kubectl get infra -n shoot--nightly--25bafc01 NAME TYPE REGION STATUS AGE 25bafc01 aws us-west-2 Succeeded 78d ```
## Kubernetes Map After you can access all the Kubernetes clusters, it is important to know what components a cluster host and which cluster should be diagnosed first when an emergency happened. The following table shows what you can find in different kinds of Kubernetes clusters: | Cluster | Namespace | Component(s) | Example | | ------------ | ------------------------ | ------------------------ | ------------------------------------------------- | | Base | ${Project} | DBaaS Backend | `kubectl get po -n nightly` | | Base | garden | Gardener ApiServer | `kubectl get po -n garden` | | Base | loki | Loki Cluster | `kubectl get po -n loki` | | Base | monitoring | Thanos Cluster | `kubectl get po -n monitoring` | | Gardener API | garden | Seed Metadata | `kubectl get secret -n garden` | | Gardener API | - | Extensions Metadata | `kubectl get controllerinstallation` | | Gardener API | garden-${Project} | Shoot Spec | `kubectl get shoot -n garden-nightly` | | Gardener API | garden-${Project} | Shoot kubeconfig | `kubectl get secret -n garden-nightly` | | Shoot | tidb-admin | TiDB Operator | `kubectl get po -n tidb-admin` | | Shoot | tidb-admin | Proxy + Billing Reporter | `kubectl get secret -n tidb-admin` | | Shoot | tidb${ClusterID} | **User's Cluster** | `kubectl get po -n tidb1295267775312302080` | | Seed | extension-${comp} | Gardener Extension | `kubectl get po -n extension-provider-aws-h6tzp` | | Seed | garden | Gardenlet | `kubectl get po -n garden` | | Seed | shoot-${Project}-${Name} | Shoot ControlPlane | `kubectl get po -n shoot--nightly--25bafc01` | | Seed | shoot-${Project}-${Name} | Shoot Machine Meta | `kubectl get machine -n shoot--nightly--25bafc01` | ## Logging and Monitoring DBaaS has a global view of logging, metrics and alerts, which are protects by Google OAuth: - Grafana (logging and metrics): [dev](https://grafana.oauth.dev.shared.aws.tidbcloud.com/), [prod](https://grafana.oauth.prod.aws.tidbcloud.com/) - Alertmanager (alerts, as a complement of pagerduty): [dev](https://alertmanager.oauth.dev.shared.aws.tidbcloud.com/), [prod](https://alertmanager.oauth.prod.aws.tidbcloud.com/#/alerts) Most of the dashboards have a `cluster` parameter so you can dive into the monitoring of each cluster: ![grafana](./images/grafana-cluster.png) ## Bastion Machine For security reasons, we have applied firewall rules to prod environment. To make the firewall rules work, we need to access Kubernetes API server with a stable IP. We can use cloud-managed VPN such as AWS Client VPN in theory, but in China the VPN is easily be interrupted. So we use AWS SSM instead. We create a separate VPC to run bastion machines, the bastion machine does not have any inbound port opened, we need to access it via AWS SSM with AWS credentials. After the `bastion` pulumi project is deployed in dev or prod, the VPC NAT IP is shown in the output. This IP needs to be added to the EKS/GKE/Shoot API server whitelist IP. To use AWS SSM, we need to install [session-manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html), for macOS you can install by `brew cask install session-manager-plugin`. We can use the following command to connect to the bastion machine to do operations: 1. Find the instance id of your bastion ```sh iam_user=dengshuan aws-vault exec ${env} -- aws ec2 describe-instances --filters "Name=tag:Name,Values=bastion-${iam_user}" ``` If it's not there, then add it to the `bastion` pulumi stack and trigger a new deployment. 2. Start a session ```sh aws-vault exec ${env} -- aws ssm start-session --target ${instance_id} ``` If you don't have a bastion yet, request one in [this project]() and follow the [change process]() to apply your change to production.