# DBaaS Architecture

DBaaS provisions Kubernetes clusters through [Gardener](#gardener) and runs TiDB clusters on these Kubernetes clusters by leveraging [tidb-operator](https://github.com/pingcap/tidb-operator). This simple principle derives the following architecture:

Architecture is [described here](https://docs.google.com/document/d/1mLtet6wiIJysi9mUbFAIpTVjaW4_Fq5YH7opxWbeyqU/edit?n=TiDBCloud_Architecture_Introduction).

We are now attempting to move to our next-generation architecture that does not require Gardener.
However, this will take some time and be inspired by Gardener.

## Gardener

[Gardener](https://gardener.cloud/) is a kubernetes-as-a-service software. It allows us to manage Kubernetes clusters (so called "Shoot" in Gardener, distinguished from the infra clusters where the Gardener runs on) through Custom Resources. A Shoot consists of two parts:

- Shoot control-plane: the kubernetes master of a shoot cluster, including kube-apiserver, etcd, cloud-controller-manager, etc.
- Shoot nodes: the kubernetes nodes that actually run the workload.

The infrastructure of Gardener consists of different kinds of Kubernetes clusters:

- Base: a Kubernetes cluster where the Gardener ApiServer runs on.
- Seed: a set of Kubernetes clusters where a Shoot control-plane runs on. Also, each Seed will run a component called "Gardenlet", which is responsible for managing the Shoot control plane in Seed and the Shoot nodes in the Cloud provider.

The infrastructure of Gardener has multiple responsibilities, in the following chapters, we will see that the infra is also used to run the DBaaS server and the monitoring system.

### Multi-provider Support

Gardenlet manages Shoot control-plane and Shoot nodes, but there are provider specific implementation details in the management. For example, the nodes on AWS should be managed through the AWS API. Gardenlet adopts a [plugin based architecture](https://github.com/gardener/gardener/blob/master/docs/proposals/01-extensibility.md) to provider multi-provider support. In essence, instead of talking to the Cloud API directly, Gardenlet creates custom resources and waits for the resources to be processed by the plugins of specific Cloud. This is why there are various extension Pods run in Seed.

The key custom resources including:

- `ControlPlane`: describes the Shoot control-plane, processed by the cloud-provider extension, e.g. `extension-provider-aws`, `extension-provider-gcp`.
- `Infrastructure`: describes the infrastructure (VPC, security group, etc.) of Shoot nodes, processed by the cloud-provider extension too.
- `MachineDeployment`, `MachineSet`, `Machine`: the relation between these resources is just like `Deployment`, `ReplicaSet` and `Pod`, and they are processed by the `machine-controller-manager`(MCM). MCM is part of Shoot control-plane so it is created by the cloud-provider extension, therefore, it has the knowledge of different cloud-providers injected.

**Hands on**: try to run `kubectl get controlplane -A`, `kubectl get infra -A` and `kubectl get machinedeployment,machine -A` in Seed to get a solid understanding of these concepts.

### Contract with Gardener

We define the following contracts so that the up-front applications can safely use builtin-in Gardener resources without explicitly configuration or query:

- The `Seed` names must be `${Provider}-${Region}`, for example, `aws-us-west-2`, `gcp-us-west2`
- The `CloudProfile` names must be `${Provider}`, for example, `aws`, `gcp`
- The `SecretBinding` names must be `${Provider}`, for example, `aws`, `gcp`

## Central

Central is the main backend service of DBaaS. It responds to some API requests directly but also acts as the API gateway and dispatches request to other services like Billing.

In central, APIs that operate external resources are asynchronous, e.g. create a cluster, take a backup. The request is first authenticated and authorized, then validated and transformed according to our business logic and stored to the database finally. A group of workers will then discover and apply the pending requests in database, and check-point the progress back to the database.

For example, when you create a cluster in DBaaS, the workers will provision a Shoot cluster in Gardener, prepare some control components in Shoot and deploy the TiDB cluster to Shoot. Every steps will be check-pointed to the database, so if error happens and the process is retried, the workers can carry on from the previous checkpoint without leaking any resources, which provides fault-tolerance.

## Shoot

As mentioned in the [Gardener](#gardener) part, Shoot is a Kubernetes cluster that Gardener manages for us. An important feature of Shoot is that is has separated control-plane and nodes, so that:
 
1. we can manage the control-plane in another kubernetes cluster, which greatly reduce the maintenance burden
2. the nodes run in a dedicated VPC, which provides isolation between Shoots

In order to manage the workloads of customers in Shoot, the shoot local controllers are needed, including:

- `tidb-operator`, which provides the management interface to TiDB cluster, backup, restore and more on a Kubernetes cluster
- `dbaas-proxy`, proxy the PD, prometheus and other services in Shoot, so that they can be safely accessed from the Central service
- `billing-reporter`, collect billing metrics in Shoot
- `cert-manager`, which signs tls certs for TiDB endpoint

To reduce the cost, Shoots are created on demand, and if a Shoot is not used anymore (the tenant no longer runs TiDB cluster on this Shoot), it could be reclaimed to release the resources.

### Shoot and multi-tenancy

There are two requirements that derives the topology of Shoots as shown in the architecture graph:

- Multi-tenant isolation: to archive isolation, we provision dedicated Shoots for each tenant.
- Low latency: the closer to our customer, the better. So for a single tenant, we will provision Shoot on each region and each cloud provider where the tenant requests for TiDB clusters.

For example, if tenant A request TiDB clusters in AWS us-west-2 and GCP us-west2, tenant B request TiDB clusters in AWS us-east-1 and AWS us-east-2, DBaaS will provision two Shoots for tenant A and two Shoots for tenant B.

It is important to note that the topology is derived from the production requirement. For example, if our customers request cheaper clusters with compromising on isolation, it would make sense to share Shoot between tenants for specific tiers of clusters.

## Billing

As mentioned above, there is a component called `billing-reporter` in each Shoot, which is responsible for collecting the usage metrics of the customer's cluster and push the metrics to a global message queue. The Billing service deployed in the Base cluster will consume these metrics and calculate the usage fee. Besides, the service will also query the Cloud API to get the cost that cannot be easily collected from our side, e.g. the cost of network traffic.

## Monitoring

The DBaaS runs a cross-region, cross-cloud and cross-k8s infrastructure, so a global monitoring is of vital importance for us to understand and operate the system. DBaaS adopt thanos as the global metrics storage and loki as the global logging storage. Every kubernetes cluster has a prometheus installed and it will remote write the metrics to thanos. Similarly, every node runs fluent-bit that ships logs to the global loki cluster. The overall architecture could be illustrated like the following:

![monitoring](../static/monitoring.png)

## Put it all together

Here is an example to put it all together (some steps are simplified):
 
1. You send a cluster creation request to DBaaS
2. You request is processed by the Central service
3. The Central authenticates, validates, and transforms you request and then stores it to the Database
4. A worker goroutine of Central discovers the request from database and starts processing
5. It locates which Shoot the TiDB should run on by the metadata of the TiDB
6. If the Shoot does not exist yet, the worker will create the Shoot first and wait it becomes ready:
    1. The worker constructs a Shoot spec, including which Seed the Shoot runs on. The Seed that has the same region and cloud with the Shoot will be chosen. Because the region and cloud is only available to customer after the corresponding Seed is created, so the Seed is definitely valid
    2. The worker create the Shoot spec in the Gardener ApiServer
    3. Gardenlet of the specified Seed will take the spec and start reconciling the Shoot
    4. Firstly the gardenlet creates a namespace in Seed, named `shoot--${project}--${name}` 
    5. Gardenlet creates a custom resource `Etcd` and wait it becomes ready. `Etcd` is processed by a controller called `etcd-druid`, deployed in the `garden` namespace
    6. Gardenlet creates a custom resource `ControlPlane` and wait it becomes ready. `ControlPlane` represents a Kubernetes control-plane and is processed by the extension controller of the target cloud
    7. Gardenlet creates a custom resource `Infra` and wait it becomes ready. `Infra` represents the infrastructure of Shoot nodes and is processed by the extension controller of the target cloud
    8. Gardenlet creates a custom resource `Worker` and wait it becomes ready
        1. The extension controller of the target cloud reconciles the `Worker` resource and creates one or more `MachineDeployment`(s) in the Shoot namespace
        2. `MachineDeployment` is reconciled by the machine-controller-manager in Shoot control-plane, actual VMs get created in the Cloud
    9. Gardenlet test the availability of Shoot and wait the whole Shoot becomes functional
    10. Controller in central creates local controllers in Shoot, including `tidb-operator`, `dbaas-proxy` and `billing-reporter`
    11. After the Shoot is ready and all the controllers are ready, the Central will mark the Shoot as **ready**
8. After the Shoot is ready, the worker update the Shoot spec to add more workers in Shoot to host the new TiDB
9. The worker creates IAM role for TiDB Pods
11. The worker creates KMS key for TiDB, which is used for data encryption
12. Worker creates custom resources in Shoot, including `TidbCluster` and `TidbMonitor`, which are processed by `tidb-operator`. The reconciliation process of `tidb-operator` is omitted here, you can refer [tidb-operator docs](https://docs.pingcap.com/tidb-in-kubernetes/v1.0/tidb-operator-overview) for detail.
13. After the `TidbCluster` is ready, the worker create a DNS record for its endpoint
14. The worker store the state to database, you observe that your cluster is ready (of the worker is continuously saving checkpoint to database of the whole process in favor of retry on failure)

Other processes like Backup, Restore may have different steps, but the same principle applies.