# Deploy a TiDB Cluster > - **Objective:** Learn to deploy a TiDB cluster on AWS (with Kubernetes) > - **Prerequisites:** <<<<<<< Updated upstream > - background knowledge of TiDB components ======= > - background knowledge of TiDB components(PD, TiDB, TiKV, TiFlash) >>>>>>> Stashed changes > - AWS account > - **Optionality:** Required > - **Estimated time:** 40min - 60min ## Install TiDB Cluster > **Optionality:** Required ### Prepare yaml file #### Command Following command copies the pre-defined `db.yaml.example` file and replace `CLUSTER_NAME` with new cluster name `my-cluster`. > **Note:** If you have changed the TiDB cluster name (`default_cluster_name` in `variables.tf`), remember to replace `my-cluster` in your configuration. ```{.bash .copyable} cluster_name=my-cluster sed "s/CLUSTER_NAME/$cluster_name/" manifests/db.yaml.example > poc.yaml <<<<<<< Updated upstream namespace=poc KUBECONFIG=$PWD/credentials/kubeconfig export KUBECONFIG kubectl create namespace "$namespace" ``` #### Configure ======= ```{.bash .copyable} helm repo add pingcap https://charts.pingcap.org/ ``` ```{.output} "pingcap" has been added to your repositories ``` >>>>>>> Stashed changes The default configuration in your `poc.yaml` file will create a fully-functional cluster without any modifications. <<<<<<< Updated upstream However, there are a lot of possible configuration options for the TidbCluster resource. You can read about those at [Configure a TiDB Cluster in Kubernetes](https://docs.pingcap.com/tidb-in-kubernetes/stable/configure-a-tidb-cluster) and the full [TidbCluster API reference](https://github.com/pingcap/tidb-operator/blob/master/docs/api-references/docs.md#tidbcluster). ======= ```{.bash .copyable} kubectl create namespace tidb-admin ``` ```{.output} namespace/tidb-admin created ``` 3. Install TiDB Operator ```{.bash .copyable} helm install --namespace tidb-admin tidb-operator pingcap/tidb-operator --version v1.1.12 ``` ```{.output} NAME: tidb-operator LAST DEPLOYED: Mon Jun 1 12:31:43 2020 NAMESPACE: tidb-admin STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Make sure tidb-operator components are running: kubectl get pods --namespace tidb-admin -l app.kubernetes.io/instance=tidb-operator ``` 4. Check TiDB Operator compoments are running: ```{.bash .copyable} kubectl get pods --namespace tidb-admin -l app.kubernetes.io/instance=tidb-operator ``` ```{.output} NAME READY STATUS RESTARTS AGE tidb-controller-manager-6d8d5c6d64-b8lv4 1/1 Running 0 2m22s tidb-scheduler-644d59b46f-4f6sb 2/2 Running 0 2m22s ``` #### Deploy a TiDB cluster and the monitoring component This section describes how to deploy a TiDB cluster and its monitoring component in AWS EKS. 1. Create namespace To create a namespace to deploy the TiDB cluster, run the following command: ```{.bash .copyable} kubectl create namespace tidb-cluster ``` ```{.output} namespace/tidb-cluster created ``` 2. Download the sample TidbCluster and TidbMonitor configuration files: ```{.bash .copyable} curl -O https://raw.githubusercontent.com/pingcap/tidb-operator/master/examples/aws/tidb-cluster.yaml && \ curl -O https://raw.githubusercontent.com/pingcap/tidb-operator/master/examples/aws/tidb-monitor.yaml ``` There are many possible configuration options for the TidbCluster resource. You can read about those at [Configure a TiDB Cluster in Kubernetes](https://docs.pingcap.com/tidb-in-kubernetes/stable/configure-a-tidb-cluster). An example is shown below that can be used to bring up the cluster without any changes to the configuration. >>>>>>> Stashed changes In many cases, the configuration of TidbCluster can be changed even after it's deployed. For example, you can choose to add TiFlash, TiDB Binlog, TiCDC, and other features after the cluster is already created. <<<<<<< Updated upstream ##### Security ======= In this guide we'll use the following customized tidb-cluster.yaml that you downloaded to your working durectory. > Note: Make a backup of the original tidb-cluster.yaml. Copy the config below, and paste into your file(tidb-cluster.yaml) and save. ```{.bash .copyable} apiVersion: pingcap.com/v1alpha1 kind: TidbCluster metadata: name: tidb-test-cluster spec: version: v4.0.10 timezone: UTC configUpdateStrategy: RollingUpdate pvReclaimPolicy: Retain schedulerName: tidb-scheduler enableDynamicConfiguration: true tiflash: baseImage: pingcap/tiflash maxFailoverCount: 3 replicas: 1 storageClaims: - resources: requests: storage: 100Gi storageClassName: gp2 pd: baseImage: pingcap/pd replicas: 3 requests: storage: "10Gi" config: log: level: info replication: location-labels: - zone max-replicas: 3 nodeSelector: dedicated: pd tolerations: - effect: NoSchedule key: dedicated operator: Equal value: pd tikv: baseImage: pingcap/tikv replicas: 3 requests: storage: "100Gi" config: {} nodeSelector: dedicated: tikv tolerations: - effect: NoSchedule key: dedicated operator: Equal value: tikv tidb: baseImage: pingcap/tidb replicas: 2 service: annotations: service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true' service.beta.kubernetes.io/aws-load-balancer-internal: '0.0.0.0/0' service.beta.kubernetes.io/aws-load-balancer-type: nlb exposeStatus: true externalTrafficPolicy: Local type: LoadBalancer config: log: level: info performance: max-procs: 0 tcp-keep-alive: true annotations: tidb.pingcap.com/sysctl-init: "true" podSecurityContext: sysctls: - name: net.ipv4.tcp_keepalive_time value: "300" - name: net.ipv4.tcp_keepalive_intvl value: "75" - name: net.core.somaxconn value: "32768" separateSlowLog: true nodeSelector: dedicated: tidb tolerations: - effect: NoSchedule key: dedicated operator: Equal value: tidb ``` To deploy the TidbCluster and TidbMonitor CR in the EKS cluster, run the following command: >>>>>>> Stashed changes It's best to decide at the time you create your cluster if you want to enable TLS. In TiDB, you can choose to [Enable TLS for the MySQL Client](https://docs.pingcap.com/tidb-in-kubernetes/stable/enable-tls-for-mysql-client) and/or [Enable TLS between TiDB Components](https://docs.pingcap.com/tidb-in-kubernetes/stable/enable-tls-between-components). <<<<<<< Updated upstream In both cases, you need to enable TLS in the CRD and add Kubernetes Secrets to give the cluster resources acceess to TLS certificates. If you want to enable TLS in your cluster, you should first follow the [Create TLS Certificates Using CFSSL](/hc/articles/360050038113) guide. After creating Secrets with your TLS certificates, make these modifications to the `poc.yaml` file: - If you want to enable TLS between MySQL-compatible clients and TiDB server, set `spec.tidb.tlsClient.enabled` to `true` - If you want to enable TLS between the components of TiDB cluster, set `spec.tlsCluster.enabled` to `true` ======= ```{.output} NAME READY STATUS RESTARTS AGE tidb-test-cluster-discovery-66b79b8c7c-lw4bh 1/1 Running 0 14d tidb-test-cluster-pd-0 1/1 Running 0 8d tidb-test-cluster-pd-1 1/1 Running 0 8d tidb-test-cluster-pd-2 1/1 Running 0 8d tidb-test-cluster-tidb-0 2/2 Running 0 13d tidb-test-cluster-tidb-1 2/2 Running 0 13d tidb-test-cluster-tiflash-0 4/4 Running 0 13d tidb-test-cluster-tiflash-1 4/4 Running 0 13d tidb-test-cluster-tikv-0 1/1 Running 0 13d tidb-test-cluster-tikv-1 1/1 Running 0 13d tidb-test-cluster-tikv-2 1/1 Running 0 13d ``` #### Connect to TiDB Because TiDB supports the MySQL protocol and most of its syntax, you can connect to TiDB using the MySQL client. 1. Install the MySQL client To connect to TiDB, you need a MySQL-compatible client installed on the host where kubectl is installed. This can be the mysql executable from an installation of MySQL Server, MariaDB Server, Percona Server, or a standalone client executable from your operating system's package repository. >>>>>>> Stashed changes ### Create TiDB pods ```{.bash .copyable} <<<<<<< Updated upstream kubectl create -f poc.yaml -n "$namespace" ``` A number of different pods will be created in groups. First, the PD pods, then the TiKV pods, and later the TiDB pods. ======= kubectl get svc -n tidb-cluster ``` ```{.output} NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE basic-grafana LoadBalancer 10.100.229.87 a3015c015de9c4261a11a22c6d4e1eae-1790807270.us-east-2.elb.amazonaws.com 3000:31836/TCP 13d basic-monitor-reloader NodePort 10.100.133.14 9089:31910/TCP 13d basic-prometheus NodePort 10.100.187.169 9090:30823/TCP 13d tidb-test-cluster-discovery ClusterIP 10.100.171.69 10261/TCP,10262/TCP 14d tidb-test-cluster-pd ClusterIP 10.100.41.246 2379/TCP 14d tidb-test-cluster-pd-peer ClusterIP None 2380/TCP 14d tidb-test-cluster-tidb LoadBalancer 10.100.161.61 aad71644d20b242438aec031d7205e5d-220c7832fccfabd0.elb.us-east-2.amazonaws.com 4000:31582/TCP,10080:31361/TCP 14d tidb-test-cluster-tidb-peer ClusterIP None 10080/TCP 14d tidb-test-cluster-tiflash-peer ClusterIP None 3930/TCP,20170/TCP 14d tidb-test-cluster-tikv-peer ClusterIP None 20160/TCP 14d ``` In this case, the TiDB service is called basic-tidb. Run the following command to forward this port from the local host to the cluster: ```{.bash .copyable} kubectl port-forward -n tidb-cluster svc/tidb-test-cluster-tidb 4000 > pf4000.out & ``` Connect to the TiDB service ```{.bash .copyable} mysql -h 127.0.0.1 -P 4000 -u root ``` ```{.output} Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 241517 Server version: 5.7.25-TiDB-v5.0.0 TiDB Server (Apache License 2.0) Community Edition, MySQL 5.7 compatible Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> ``` 2. Load some sample data and run some queries >>>>>>> Stashed changes #### Verify You can use `kubectl get pod` to see the progress of the deployment. ```{.bash .copyable} watch kubectl -n "$namespace" get pod ``` ```{.output} <<<<<<< Updated upstream NAME READY STATUS RESTARTS AGE my-cluster-discovery-d9854558f-gxpsb 1/1 Running 0 7m6s my-cluster-pd-0 1/1 Running 0 7m6s my-cluster-pd-1 1/1 Running 2 7m6s my-cluster-pd-2 1/1 Running 0 7m6s my-cluster-tidb-0 2/2 Running 0 3m19s my-cluster-tidb-1 2/2 Running 0 3m19s my-cluster-tikv-0 1/1 Running 0 3m38s my-cluster-tikv-1 1/1 Running 0 3m38s my-cluster-tikv-2 1/1 Running 0 3m38s ``` As soon as you see pods for all 3 pod types and all pods show "Running", you can move on to [connecting to your cluster](/hc/en-us/articles/360049691613). ======= mysql> create table hello_world (id int unsigned not null auto_increment primary key, v varchar(32)); Query OK, 0 rows affected (0.17 sec) ``` ```{.bash .copyable} select \* from information_schema.tikv_region_status where db_name=database() and table_name='hello_world'\G ``` ```{.output} mysql> select \* from information_schema.tikv_region_status where db_name=database() and table_name='hello_world'\G \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** 1. row \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** REGION_ID: 2 START_KEY: 7480000000000000FF3700000000000000F8 END_KEY: TABLE_ID: 55 DB_NAME: test TABLE_NAME: hello_world IS_INDEX: 0 INDEX_ID: NULL INDEX_NAME: NULL EPOCH_CONF_VER: 5 EPOCH_VERSION: 23 WRITTEN_BYTES: 0 READ_BYTES: 0 APPROXIMATE_SIZE: 1 APPROXIMATE_KEYS: 0 1 row in set (0.03 sec) Query the TiDB version: mysql> select tidb_version()\G \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** 1. row \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** tidb_version(): Release Version: v4.0.0 Edition: Community Git Commit Hash: 689a6b6439ae7835947fcaccf329a3fc303986cb Git Branch: heads/refs/tags/v4.0.0 UTC Build Time: 2020-05-28 01:37:40 GoVersion: go1.13 Race Enabled: false TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306 Check Table Before Drop: false 1 row in set (0.00 sec) Query the TiKV store status: mysql> select \* from information_schema.tikv_store_status\G \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** 1. row \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** STORE_ID: 4 ADDRESS: basic-tikv-0.basic-tikv-peer.tidb-cluster.svc:20160 STORE_STATE: 0 STORE_STATE_NAME: Up LABEL: null VERSION: 4.0.0 CAPACITY: 58.42GiB AVAILABLE: 36.18GiB LEADER_COUNT: 3 LEADER_WEIGHT: 1 LEADER_SCORE: 3 LEADER_SIZE: 3 REGION_COUNT: 21 REGION_WEIGHT: 1 REGION_SCORE: 21 REGION_SIZE: 21 START_TS: 2020-05-28 22:48:21 LAST_HEARTBEAT_TS: 2020-05-28 22:52:01 UPTIME: 3m40.598302151s 1 rows in set (0.01 sec) ``` Query the TiDB cluster information: ```{.bash .copyable} select \* from information_schema.cluster_info\G ``` ```{.output} mysql> select \* from information_schema.cluster_info\G \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** 1. row \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** TYPE: tidb INSTANCE: basic-tidb-0.basic-tidb-peer.tidb-cluster.svc:4000 STATUS_ADDRESS: basic-tidb-0.basic-tidb-peer.tidb-cluster.svc:10080 VERSION: 5.7.25-TiDB-v4.0.0 GIT_HASH: 689a6b6439ae7835947fcaccf329a3fc303986cb START_TIME: 2020-05-28T22:50:11Z UPTIME: 3m21.459090928s \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** 2. row \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** TYPE: pd INSTANCE: basic-pd:2379 STATUS_ADDRESS: basic-pd:2379 VERSION: 4.0.0 GIT_HASH: 56d4c3d2237f5bf6fb11a794731ed1d95c8020c2 START_TIME: 2020-05-28T22:45:04Z UPTIME: 8m28.459091915s \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** 3. row \***\*\*\*\*\*\*\***\*\*\*\***\*\*\*\*\*\*\*** TYPE: tikv INSTANCE: basic-tikv-0.basic-tikv-peer.tidb-cluster.svc:20160 STATUS_ADDRESS: 0.0.0.0:20180 VERSION: 4.0.0 GIT_HASH: 198a2cea01734ce8f46d55a29708f123f9133944 START_TIME: 2020-05-28T22:48:21Z UPTIME: 5m11.459102648s 3 rows in set (0.01 sec) ``` 8. Access the TiDB Dashboard Forward the port for TiDB Dashboard so that you can access it locally: ```{.bash .copyable} kubectl port-forward -n tidb-cluster svc/tidb-test-cluster01-pd 2379:2379 > pf2379.out & ``` 9. Access Grafana dashboard You can forward the port for Grafana so that you can access Grafana dashboard locally: ```{.bash .copyable} kubectl port-forward -n tidb-cluster svc/basic-grafana 3000:3000 > pf3000.out & ``` You can access Grafana dashboard at [http://localhost:3000] on the host where you run kubectl. Note that if you are not running kubectl on the same host (for example, in a Docker container or on a remote host), you cannot access Grafana dashboard at [http://localhost:3000] from your browser. The default username and password in Grafana are both admin. > **Note:** From time to time, the port forwarding process will terminate. Simply re-issue the port forwarding commands. If you get any messages stating that the port is already in use, make sure there are no other running processes holding the port open, or forward a different local port to the remote port. >>>>>>> Stashed changes