# DBaaS deployment

* Garden cluster

  Garden cluster requires a K8s cluster (Base cluster), currently we create an EKS cluster in us-west-2 as Base cluster. And Garden cluster is deployed in Base cluster as normal pods. Garden cluster is globally deployed.

* Gardener Seed cluster

  Gardener Seed cluster also requires a K8s cluster, we deploy a K8s cluster per cloud region by using cloud vendor managed K8s cluster such as EKS and GKE. And register it into the Garden cluster.

* DBaaS control plane

  DBaaS control plane is our application layer, it currently includes central-db, central-server, billing-server.

* Customer TiDB clusters

  Customers' TiDB clusters requires a K8s cluster (Shoot cluster), the Shoot cluster's control plane is deployed in the Seed cluster as normal Pods. And the TiDB clusters are deployed in the Shoot cluster. A customer's TiDB clusters in the same region live in the same Shoot cluster. These are managed by DBaaS control plane.

The K8s cluster required by Garden cluster and Seed cluster are deployed using [Pulumi](https://www.pulumi.com). And Gardener staffs are deployed using scripts provided by [Gardener project](https://github.com/gardener/garden-setup). The DBaaS control plane is also deployed using Pulumi.

## Deploy Garden cluster

### Deploy Base K8s cluster

```sh
cd eks
npm install
export ENV=dev
export region=us-west-2
# NOTE: due to name collision, the state bucket of prod is "prod-dbaas-pulumi-state"
aws-vault exec ${env} -- pulumi login --cloud-url s3://${ENV}-pulumi-state
pulumi stack init ${ENV}-base-${region}
pulumi config set eks:env ${ENV}
pulumi config set eks:cluster base
pulumi config set eks:k8s-version 1.16
pulumi config set eks:worker-type m5.xlarge
pulumi config set eks:worker-count 3
pulumi config set eks:isBase true
pulumi config set eks:region ${region}
pulumi config set aws:region ${region}
pulumi config set eks:pagerduty_key ${key}
pulumi config set eks:slack_url ${url}
aws-vault exec ${env} -- pulumi up
pulumi stack output kubeconfig > garden/kubeconfig
```

### Deploy Gardener control plane

```sh
cd garden
```

Copy the `garden/acre.yaml.example` to `acre.yaml` and update `acre.yaml` as you need.

1. Name your environment:

    ```sh
    export ENV=dev
    ```

2. Write a script `${ENV}-acre.sh` to generate `acre.yaml`, note that:
    * **DO NOT INCLUDE THE REAL CREDENTIALS IN THIS SCRIPT, USE PLACEHOLDER INSTEAD**
    * EKS kubeconfig can be inlined, because the authentication is performed by iam-authenticator using you local credentials
    * GKE kubeconfig can be inlined, but you **MUST REMOVE THE ACCESS_TOKEN AND SET A DUMMY EXPIRE TIME**, so that garden-setup will request an access token on the fly using your local credentials

    ```sh
    cat <<EOF
    landscape:
      name: <name>                            # general Gardener landscape identifier, for example, `my-gardener`
      domain: <name>.shared.aws.tidbcloud.com # unique basis domain for DNS entries
      cluster:                      # information about your base cluster
        kubeconfig: ./kubeconfig    # path to your `kubeconfig` file, rel. to directory `landscape` (defaults to `./kubeconfig`)
        networks:                   # CIDR IP ranges of base cluster, The nodes CIDR and pods CIDR must not be a subset of each other
          nodes: <Nodes CIDR>
          pods: <Pods CIDR>
          services: <Service CIDR>
      iaas:
        - name: <type>                              # name of the seed, <gcp|aws>
          type: <type>                              # iaas provider, <gcp|aws>
          region: <region>                          # region for initial seed
          zones:
            - <major region>-<minor region>-<zone>   # example: us-west2-a
            - <major region>-<minor region>-<zone>   # example: us-west2-b
            - <major region>-<minor region>-<zone>   # example: us-west2-c
          credentials:
            accessKeyID: ${AWS_ACCESS_KEY_ID}
            secretAccessKey: ${AWS_SECRET_ACCESS_KEY}
      dns:
        type: aws-route53
        credentials:
          accessKeyID: ${AWS_ACCESS_KEY_ID}
          secretAccessKey: ${AWS_SECRET_ACCESS_KEY}
      identity:
        users:
          - email: <email>
            username: <username>
            password: <password>
    EOF
    ```

    For critical environment, register the base cluster as `soil` instead of `seed`, which does not host any shoot:

    ```yaml
    cat <<EOF
    landscape:
      cluster:
        kubeconfig: ./kubeconfig  # path to the kubeconfig of base cluster
      iaas:
        - name: base
          type: <aws|gcp>
          mode: soil              # do not run shoots in base cluster
          kubeconfig:
            {}                    # this kubeconfig should be the kubeconfig of base cluster
        - name: aws-us-west-2     # actual seed
          type: aws
          mode: seed              # run in seed mode
          # other fields of seed cluster...
    EOF
    ```

   > Note: New seed MUST be added after the original seeds.

3. Deploy:

    ```sh
    export AWS_ACCESS_KEY_ID=<SA_AWS_ACCESS_ID>
    export AWS_SECRET_ACCESS_KEY=<SA_AWS_SECRET_ACCESS_KEY>
    export GOOGLE_APPLICATION_CREDENTIALS=<PATH_TO_SA_CREDENTIALS>
    export DASHBORAD_PASSWORD=<DASHBORAD_PASSWORD>
    # Create a bucket to store the state of garden-setup
    aws-vault exec ${env} -- aws s3api create-bucket --region=${region} --create-bucket-configuration LocationConstraint=${region} --bucket ${ENV}-garden-state
    aws-vault exec ${env} --no-session -- ./garden-setup.sh order -A
    aws-vault exec ${env} --no-session -- ./garden-setup.sh deploy -A
    ```

Now starts to install Gardener in your base EKS cluster. The installation can take about 30 minutes. The script prints out status messages to the terminal window so that you can check the status of the installation.

During the installation, you can open a second terminal window which current directory is your `garden` directory. Set the `KUBECONFIG` environment variable and watch the progress of the Gardener installation:

```sh
export KUBECONFIG=./${ENV}-kubeconfig
watch -d kubectl -n garden get pods,ingress,sts,svc
```

The other terminal window will show the newly created Kubernetes resources after a while and if their deployment was successful. Wait until the last component is deployed and all created Kubernetes resources are in status `Running`.

When the installation is done, get the Gardener dashboard UI:

```sh
./garden-setup.sh url
```

You can access the dashboard with the email and password configured in the `acre.yaml`.

## Deploy Seed cluster

### Deploy Seed k8s cluster

* AWS

    ```sh
    cd eks
    export ENV=dev
    export region=ap-northeast-1
    pulumi login --cloud-url s3://${ENV}-pulumi-state
    pulumi stack init ${ENV}-seed-${region}
    pulumi config set eks:env ${ENV}
    pulumi config set eks:cluster seed
    pulumi config set eks:k8s-version 1.16
    pulumi config set eks:worker-type m5.xlarge
    pulumi config set eks:worker-count 2
    pulumi config set eks:region ${region}
    pulumi config set aws:region ${region}
    aws-vault exec ${ENV} -- pulumi up
    pulumi stack output kubeconfig > garden/seed-${region}-kubeconfig
    ```

* GCP

    ```sh
    cd gcp
    npm install
    export ENV=dev
    export region=us-west2
    pulumi login --cloud-url s3://${ENV}-pulumi-state
    pulumi stack init ${ENV}-seed-${region}
    pulumi config set gcp:project ${GCP_PROJECT}
    pulumi config set gcp:region us-west2
    pulumi config set gke:project ${GCP_PROJECT}
    pulumi config set gke:env ${ENV}
    pulumi config set gke:cluster seed
    pulumi config set gke:region us-west2
    pulumi config set gke:worker-count 2
    pulumi config set gke:gardenerServiceAccountName ${ENV}-${region}
    pulumi up
    pulumi stack output kubeconfig > garden/seed-${region}-kubeconfig
    ```

### Add seeds in current landscape

1. Edit the `${ENV}-acre.sh`, add new seed in the acre template.

2. Run `garden-setup.sh order -A`

3. Run `garden-setup.sh deploy -An`