## Overview [![Build Status](http://107.150.125.75:8080/job/build_tidb_operator_master/badge/icon)](http://107.150.125.75:8080/job/build_tidb_operator_master/) The TiDB operator manages TiDB clusters deployed to [Kubernetes](https://kubernetes.io) and automates tasks related to operating a TiDB cluster. Document List: - [Requirements](#requirements) - [Manage TiDB volumes](#manage-tidb-volumes) - [Deploy the TiDB operator](#deploy-the-tidb-operator) - [Create a TiDB cluster](#create-a-tidb-cluster) - [Resize and apply a rolling update to the cluster](#resize-and-apply-a-rolling-update-to-the-cluster) - [Design](#design) ## Requirements Before deploying the TiDB operator, make sure the following requirements are satisfied: * Kubernetes >= v1.7 * [RBAC](https://kubernetes.io/docs/admin/authorization/rbac) is enabled * The [`TaintBasedEvictions`](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#taint-based-evictions) feature gates are enabled * [Helm](https://helm.sh) is installed. For installation procedure, see the [Helm document](charts/README.md) * Your user has permission to deploy Your helm install needs proper permissions, e.g. kubectl create serviceaccount tiller --namespace kube-system kubectl create -f charts/lib/tiller-clusterrolebinding.yaml helm init --service-account tiller --upgrade ## Manage TiDB volumes With Kubernetes, you can use [Requests and Limits](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md) to achieve CPU and memory isolation, but it is very hard to achieve storage isolation in Kubernetes natively. So `tidb-volume-manager` is added to Kubernetes as a [`HostPath`](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes)-based [`PersistentVolume` provisioner](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#provisioner). Before deploying the TiDB operator, add the `HostPath` to the `volumes` section of `charts/tidb-operator/values.yaml `. For example: ```yaml volumes: - name: kube-node-1 directories: - path: /tidb/tikv-dir-1 size: 500Gi - path: /tidb/pd-dir-1 size: 20Gi - name: kube-node-2 directories: - path: /tidb/pd-dir-1 size: 20Gi - path: /tidb/tikv-dir-1 size: 300Gi - name: kube-node-3 directories: - path: /tidb/pd-dir-1 size: 20Gi - path: /tidb/tikv-dir-2 size: 500Gi ``` * `name`: Kube node name * `directories`: Each node can have multiple directories used for local PV * `path`: It must have prefix `volumeManager.pvDir` in `charts/tidb-operator/values.yaml` because `tidb-volume-manager` can only manage directories under `volumeManager.pvDir`, for example: `/tidb` * `size`: the size of the path Each directory here can only be mounted by one `TiKV` instance at a time, and **for the best I/O performance and the best isolation effect, you must mount one device to one directory only**. For example, you can mount `/dev/sdb` to `/tidb/tikv-dir-1` and mount `/dev/sdc` to `/tidb/tikv-dir-2`. On GKE, you can extend a script like this to help with your setup. ./charts/lib/gke-operator-values.sh ## Deploy the TiDB operator The TiDB operator is composed of `tidb-controller-manager`, `tidb-scheduler` and `tidb-volume-manager`. 1. Use the following command to deploy the TiDB operator all-in-one: ``` $ helm install charts/tidb-operator -n tidb-operator --namespace=pingcap ``` 2. The TiDB operator automatically creates the following components: ``` $ kubectl get daemonset -n pingcap -l "app=tidb-operator" NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE tidb-volume-manager 3 3 0 3 0 1m $ kubectl get deployment -n pingcap -l "app=tidb-operator" NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE tidb-controller-manager 1 1 1 0 1m tidb-scheduler 1 1 1 0 1m $ kubectl get configmap -n pingcap -l "app=tidb-operator" NAME DATA AGE tidb-scheduler-policy 1 1m tidb-volume 1 1m $ kubectl get storageclasses NAME TYPE pingcap-volume-provisioner pingcap.com/volume-provisioner $ kubectl get customresourcedefinitions NAME KIND tidbclusterbackups.pingcap.com CustomResourceDefinition.v1beta1.apiextensions.k8s.io tidbclusters.pingcap.com CustomResourceDefinition.v1beta1.apiextensions.k8s.io tidbreviews.pingcap.com CustomResourceDefinition.v1beta1.apiextensions.k8s.io tidbsets.pingcap.com CustomResourceDefinition.v1beta1.apiextensions.k8s.io $ kubectl get pod -n pingcap NAME READY STATUS RESTARTS AGE tidb-controller-manager-3999554014-1h171 1/1 Running 0 1m tidb-scheduler-3675482103-rrzjj 2/2 Running 0 1m tidb-volume-manager-06cb9 1/1 Running 0 1m tidb-volume-manager-3zqpw 1/1 Running 0 1m tidb-volume-manager-9xph6 1/1 Running 0 1m ``` ## Create a TiDB cluster After deploying the TiDB operator, follow the following steps to create a TiDB cluster: 1. Use the following command to create a `TidbCluster` object which is a [CustomResourceDefinition](https://kubernetes.io/docs/concepts/api-extension/custom-resources/#customresourcedefinitions) (CRD) available as of Kubernetes 1.7. ``` $ helm install charts/tidb-cluster -n tidb-demo --namespace=tidb-demo ``` 2. (Optional) You can adjust some attributes in the `charts/tidb-cluster/values.yaml` file to meet your requirements. 3. Wait a few minutes to get all TiDB components running: ``` $ kubectl get pod -n tidb-demo NAME READY STATUS RESTARTS AGE demo-cluster-monitor-1511209211-g75zb 2/2 Running 0 1m demo-cluster-pd-g6g9d 1/1 Running 0 1m demo-cluster-pd-qjtwd 1/1 Running 0 1m demo-cluster-pd-sx4nf 1/1 Running 0 1m demo-cluster-tidb-gdz39 1/1 Running 0 1m demo-cluster-tidb-s215h 1/1 Running 0 1m demo-cluster-tikv-75qv9 2/2 Running 0 1m demo-cluster-tikv-mxtbd 2/2 Running 0 1m demo-cluster-tikv-zg8nh 2/2 Running 0 1m $ kubectl get service -n tidb-demo NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE demo-cluster-grafana 10.100.56.31 3000:30649/TCP 1m demo-cluster-pd 10.101.198.226 2379/TCP 1m demo-cluster-pd-peer None 2379/TCP,2380/TCP 1m demo-cluster-prometheus 10.104.67.215 9090:30681/TCP 1m demo-cluster-tidb 10.102.7.54 4000:32751/TCP 1m demo-cluster-tikv-peer None 20160/TCP 1m ``` ## Resize and apply a rolling update to the cluster Follow the following steps to resize the cluster: 1. Edit the `charts/tidb-cluster/values.yaml` file to change the number of `pd` replicas to 3, and change the number of `tikv` replicas to 5. 2. Use the following command to update the object. ``` $ helm upgrade tidb-demo charts/tidb-cluster --namespace=tidb-demo ``` 3. Wait a few minutes and then the number of `pd` and `tikv` instances are updated: ``` $ kubectl get pod -n tidb-demo NAME READY STATUS RESTARTS AGE demo-cluster-monitor-1511209211-g75zb 2/2 Running 0 2m demo-cluster-pd-g6g9d 1/1 Running 0 2m demo-cluster-pd-qjtwd 1/1 Running 0 2m demo-cluster-pd-sx4nf 1/1 Running 0 2m demo-cluster-tidb-gdz39 1/1 Running 0 2m demo-cluster-tidb-s215h 1/1 Running 0 2m demo-cluster-tikv-75qv9 2/2 Running 0 2m demo-cluster-tikv-mxtbd 2/2 Running 0 2m demo-cluster-tikv-n8b9r 2/2 Running 0 2m demo-cluster-tikv-sjn6w 2/2 Running 0 1m demo-cluster-tikv-zg8nh 2/2 Running 0 1m ``` To apply a rolling update to the cluster, follow the above procedure. ## Design ![TiDB Operator Architecture on Kubernetes](media/kubernetes.png) The TiDB operator consists of the following components: ### tidb-controller-manager `tidb-controller-manager` is the main component of the TiDB operator. Its job is to watch all the `tidbclusters` resources created by the user, and then create/delete/update TidbSet of all TiDB components according to the specifications in the `.yaml` of `TidbCluster` object. ### tidb-scheduler `tidb-scheduler` is a [Kubernetes scheduler extender](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduler_extender.md) to schedule `TiKV`. ### tidb-volume-manager `tidb-volume-manager` is a HostPath-based PersistentVolume provisioner, responsible for privisioning and deleting the [PersistentVolumes(PVs)](https://kubernetes.io/docs/concepts/storage/persistent-volumes) released by the TiDB operator. ### TiDB operator workflow The TiDB operator workflow is as below: 1. Create a [PersistentVolumeClaims(PVC)](https://kubernetes.io/docs/concepts/storage/persistent-volumes). 2. Create a `TiKV` pod. 3. `kube-scheduler` selects some nodes suitable for this pod and sends these nodes to `tidb-scheduler`. 4. `tidb-scheduler` tries to find a node with available `HostPath`-based `PV`s, which are supplied by `tidb-volume-manager`. 5. `tidb-volume-manager` creates a `PV` related to `PVC` for `tidb-scheduler`. 6. The `TiKV` pod is scheduled to a suitable node and next Kubernetes inspects the `PVC` to find the bound `PV` and mounts that volume for the `TiKV` pod. ### Contributing See the [contributing guide](./docs/CONTRIBUTING.md).