# Deploy an EKS Cluster > - **Objective:** Learn to deploy a Kubernetes cluster on AWS (with EKS) > - **Prerequisites:** > - AWS account > - **Optionality:** Required > - **Estimated time:** 40min ## Prepare tools > - **Optionality:** Required <<<<<<< Updated upstream To deploy a Kubernetes Cluster on AWS EKS, some tools are required to interact with AWS and Kubernetes. The tools can be installed on any host that can access the AWS server, and subsequent operations will be performed on this host. > **Note** > > You can continue to manually download and install tools, or [use the all-in-one prepared Docker image](#use-all-in-one-prepared-docker-image) directly. > If you do choose to use the Docker image, note that port forwarding for web-based services is more complex, because ports are forwarded from the Docker container rather than your local machine. ======= To deploy a Kubernetes Cluster on AWS EKS, some tools are required to interact with AWS and Kubernetes. The tools can be installed on any computer that can access the AWS APIs, and all subsequent operations will be performed on that computer. > **Note** > > Installation of these tools in this guide were completed on a Linux Client machine. These tools are also supported on Windows and macOS well. References to tool versions were at the time of writing. >>>>>>> Stashed changes ### Manually download tools Below is the prerequisites list. Please follow the instruction links to install them. | Tool | Use | Version | Install instructions | | :-- | :-- | :-- | :-- | | awscli | Interact with AWS services in command line | >=1.16.73 (including 2.x) | [Installing the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) | | Terraform | Build, change, and destroy AWS infrastructure | >=0.12 | [Installing Terraform](https://learn.hashicorp.com/terraform/getting-started/install.html) | |kubectl | Run commands against Kubernetes clusters | >=1.12 | [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) | | Helm | The package manager for Kubernetes | >=2.11.0, !=2.16.4, <3.0.0 | [Installing Helm](https://helm.sh/docs/intro/install/) | | jq | Command-line JSON processor | - | [Download jq](https://stedolan.github.io/jq/download/) | | aws-iam-authenticator | Provide authentication to the Kubernetes cluster | - | [Installing aws-iam-authenticator](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html) | #### Verify tools > **Note:** Please pay special attention to helm's version; helm 3.0 is not supported. ```{.output} $ aws --version aws-cli/2.0.7 Python/3.7.3 Linux/4.15.0-66-generic botocore/2.0.0dev11 $ terraform --version Terraform v0.12.24 $ kubectl version --client Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} $ helm version --client Client: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"} $ jq --version jq-1.5-1-a5b5cbe $ aws-iam-authenticator version {"Version":"v0.5.0","Commit":"1cfe2a90f68381eacd7b6dcfa2bf689e76eb8b4b"} ``` #### Troubleshooting ##### bash: /usr/local/bin/helm: cannot execute binary file Please check whether the correct platform was selected when downloading tools. ### Use all-in-one prepared Docker image > If you have already downloaded and installed the tools, skip ahead to [Create Kubernetes Cluster](#create-kubernetes-cluster). Run the following command to enter the container environment that contains all required tools: ```{.bash .copyable} docker run -it disksing/tidb-eks-tools:latest ``` ## Create Kubernetes Cluster - **Optionality:** Required ### Configure AWS credentials Use `aws configure` to setup access keys. > **Note:** The access key must have at least these permissions: > > - Create VPC > - Create EBS volumes > - Create EC2 instance > - Create roles #### Commands > **Note:** the following keys are examples, please replace with your own keys. ```{.bash .copyable} aws configure ``` ```{.output} AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY Default region name [None]: us-west-2 Default output format [None]: json ``` #### Verify Keys ```{.bash .copyable} aws configure list ``` ```{.output} Name Value Type Location ---- ----- ---- -------- profile None None access_key ******************** shared-credentials-file secret_key ******************** shared-credentials-file region us-west-2 config-file ~/.aws/config ``` <<<<<<< Updated upstream ### Clone the `tidb-operator` repo The `tidb-operator` project provides utilities and configurations for creating an EKS cluster that is suitable for deployment of TiDB components. #### Commands ```{.bash .copyable} git clone --branch release-1.1 --depth=1 https://github.com/pingcap/tidb-operator.git cd tidb-operator/deploy/aws && ls ``` ======= ### 1. Create a EKS cluster and a node pool According to AWS Official Blog recommendation and EKS Best Practice Document, since most of the TiDB cluster components use EBS volumes as storage, it is recommended to create a node pool in each availability zone (at least 3 in total) for each component when creating an EKS. Save the following configuration as the cluster.yaml file. ```{.bash .copyable} apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: tidb-eks-cluster region: us-east-2 nodeGroups: - name: admin desiredCapacity: 1 privateNetworking: true labels: dedicated: admin - name: tidb-2a desiredCapacity: 1 privateNetworking: true availabilityZones: ["us-east-2a"] labels: dedicated: tidb taints: dedicated: tidb:NoSchedule - name: tidb-2b desiredCapacity: 1 privateNetworking: true availabilityZones: ["us-east-2b"] labels: dedicated: tidb taints: dedicated: tidb:NoSchedule - name: tidb-2c desiredCapacity: 0 privateNetworking: true availabilityZones: ["us-east-2c"] labels: dedicated: tidb taints: dedicated: tidb:NoSchedule - name: pd-2a desiredCapacity: 1 privateNetworking: true availabilityZones: ["us-east-2a"] labels: dedicated: pd taints: dedicated: pd:NoSchedule - name: pd-2b desiredCapacity: 1 privateNetworking: true availabilityZones: ["us-east-2b"] labels: dedicated: pd taints: dedicated: pd:NoSchedule - name: pd-2c desiredCapacity: 1 privateNetworking: true availabilityZones: ["us-east-2c"] labels: dedicated: pd taints: dedicated: pd:NoSchedule - name: tikv-2a desiredCapacity: 1 privateNetworking: true availabilityZones: ["us-east-2a"] labels: dedicated: tikv taints: dedicated: tikv:NoSchedule - name: tikv-2b desiredCapacity: 1 privateNetworking: true availabilityZones: ["us-east-2b"] labels: dedicated: tikv taints: dedicated: tikv:NoSchedule - name: tikv-2c desiredCapacity: 1 privateNetworking: true availabilityZones: ["us-east-2c"] labels: dedicated: tikv taints: dedicated: tikv:NoSchedule ``` Only two TiDB nodes are required, so you can set the desiredCapacity of the tidb-1c node group to 0. You can scale out this node group any time if necessary. Execute the following command to create the cluster: >>>>>>> Stashed changes #### Verify clone ```{.output} README.md aws-tutorial.tfvars clusters.tf default-cluster.yaml demo.tfvars main.tf manifests outputs.tf prod.tfvars variables.tf versions.tf ``` ### Install Terraform Dependencies Terraform needs to install some necessary components to work. #### Commands ```{.bash .copyable} terraform init ``` ```{.output} <<<<<<< Updated upstream ... Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. ``` ### Configure EKS Cluster Some Terraform configuration files are pre-defined in the `aws` directory. You can modify the configuration file to customize deploy region, cluster size, instance type, etc. Since applying for resources is a time-consuming operation (about 20+ minutes), we recommend that you first create a cluster using the default configuration we provide, and then during the creation process you can take your time to learn how to customize the configurations. > **Note:** you can [create EKS cluster](#create-eks-cluster) with default configuration files directly. #### Configure AWS region You can customize AWS region by editing the `variables.tf` file. All avaliable values are listed in comments. The default value is `us-west-2`. ======= 2021-07-07 15:07:39 [ℹ] eksctl version 0.52.0 2021-07-07 15:07:39 [ℹ] using region us-east-2 NAME REGION EKSCTL CREATED tidb-eks-cluster us-east-2 True >>>>>>> Stashed changes ``` variable "region" { description = "AWS region" # supported regions: # US: us-east-1, us-east-2, us-west-2 # Asia Pacific: ap-south-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-northeast-1 # Europe: eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1 default = "us-west-2" } ``` <<<<<<< Updated upstream #### Configure cluster name You can customize EKS cluster name or TiDB cluster name by editing the `variables.tf` file. > **Note:** Be aware, if you change the cluster name, in the following steps, some commands may need to be adjusted accordingly. ``` variable "eks_name" { description = "Name of the EKS cluster. Also used as a prefix in names of related resources." default = "my-cluster" } variable "default_cluster_name" { default = "my-cluster" } ``` #### Configure cluster size You can customize cluster size by editing the `variables.tf` file. By default, the cluster contains 3 PDs, 3 TiKVs and 2 TiDBs. > **Note:** if you change the cluster size, in the following steps, the output of some commands and cluster status may be quite different from what is shown. ``` variable "default_cluster_pd_count" { default = 3 } variable "default_cluster_tikv_count" { default = 3 } variable "default_cluster_tidb_count" { default = 2 } ``` #### Configure instance type You can customize instance type by editing the `variables.tf` file. The default confirations are listed below. The default configuration is suitable for basic functional evaluation, not suitable for performance tests. [AWS EC2 instance pricing](https://aws.amazon.com/ec2/pricing/on-demand/) contains more detailed information about instance types and their prices. ``` variable "default_cluster_pd_instance_type" { default = "c5d.large" } variable "default_cluster_tikv_instance_type" { default = "c5d.large" } variable "default_cluster_tidb_instance_type" { default = "c4.large" } variable "default_cluster_monitor_instance_type" { default = "c5.large" } ``` ### Create EKS Cluster We will use Terraform to create AWS resources and create EKS clusters based on the information provided by the configuration file. #### Commands ```{.bash .copyable} terraform apply ``` Terraform does not directly perform operations, but first outputs the operation plan in the form of a diff. At the end of the command line output, there will be a line summarizing the operations to be performed and prompting you to confirm: ```{.output} Plan: 66 to add, 0 to change, 0 to destroy. ... Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: ``` To proceed, you need to type `yes` and hit Enter. The running process will last more than 20 minutes, and a status message will be output every 10 seconds during the running process, which includes the total time currently used: ```{.output} module.tidb-operator.null_resource.setup-env: Still creating... [1m10s elapsed] module.tidb-operator.null_resource.setup-env: Still creating... [1m20s elapsed] module.tidb-operator.null_resource.setup-env: Still creating... [1m30s elapsed] module.tidb-operator.null_resource.setup-env: Still creating... [1m40s elapsed] module.tidb-operator.null_resource.setup-env: Still creating... [1m50s elapsed] module.tidb-operator.null_resource.setup-env: Still creating... [2m0s elapsed] ``` > **Note:** If you are creating using the default configration, you can take this time to learn about how to [Configure EKS Cluster](#configure-eks-cluster). When it is finally completed, you will see command line output like this: ```{.output} Apply complete! Resources: 66 added, 0 changed, 0 destroyed. Outputs: bastion_ip = [ "52.25.125.150", ] default-cluster_monitor-dns = not_created default-cluster_tidb-dns = not_created eks_endpoint = https://F989422954FCA9A08505C4BBE98FA194.sk1.us-west-2.eks.amazonaws.com eks_version = 1.15 kubeconfig_filename = credentials/kubeconfig_my-cluster region = us-west-2 ``` The output contains the information needed to connect the EKS cluster. The `bastion_ip` is the IP address of an EC2 server that can access the TiDB server. You can use `terraform output` to display the output later any time you want to reference it. ======= ```{.bash .copyable} kubectl get pods -n kube-system ```` ```{.output} NAME READY STATUS RESTARTS AGE aws-node-2l7nj 1/1 Running 0 11m aws-node-2wvds 1/1 Running 0 9m22s aws-node-552vg 1/1 Running 0 5m30s aws-node-9nk7b 1/1 Running 0 12m aws-node-g7czc 1/1 Running 0 8m30s aws-node-lbw28 1/1 Running 0 7m14s aws-node-qj9tp 1/1 Running 0 10m aws-node-tfq5x 1/1 Running 0 4m28s aws-node-wvhzq 1/1 Running 0 6m23s coredns-56b458df85-85xhz 1/1 Running 0 19m coredns-56b458df85-8ptr9 1/1 Running 0 19m kube-proxy-7b97x 1/1 Running 0 10m kube-proxy-7j6jz 1/1 Running 0 6m23s kube-proxy-7wwdf 1/1 Running 0 8m30s kube-proxy-84n97 1/1 Running 0 12m kube-proxy-9v5mw 1/1 Running 0 4m28s kube-proxy-mqzjb 1/1 Running 0 7m14s kube-proxy-mx7cn 1/1 Running 0 11m kube-proxy-qk7rr 1/1 Running 0 9m22s kube-proxy-wkw2c 1/1 Running 0 5m30s ```` >>>>>>> Stashed changes #### Troubleshooting ##### Error: No valid credential sources found for AWS Provider You need to [configure AWS credentials](#configure-aws-credentials) before creating the cluster. ##### Error: Provider produced inconsistent final plan This is a known issue with Terraform. Please execute `terraform apply` again to resolve the situation. ##### Error: deployments.apps "tiller-deploy" not found The installed Helm version is too new. Currently the Terraform scripts for TiDB Operator do not support helm 3. Please refer to [Manually download tools](#Manually-download-tools) to install the correct version. ##### `terraform apply` command fails In some cases, a failure of `terraform apply` can be resolved by simply running it again.