# TiDB Course 303 Exercises Environment Management Guide (for Trainer) ***CAUTION: Always keep the cost in mind. This guide will spin up multiple on-demand EC2 instances.*** ## Lab Environment Diagram ## Notes 1. This lab setup prepares 10 EC2 nodes (C5.xlarge) per student. 2. **DO NOT** deploy the PingCAP Education classroom CloudFormation stacks in any PingCAP production environment. 3. In order to execute all the classroom environment setup steps listed in this guide, it requires you install **Python3**, **boto3** and **awscli** on your local `macOS` machine in advance. 4. You need an active AWS account in order to run these setups. And, your IAM user or role should have the permissions to create various AWS resources, it should be an admin IAM user/role or a power user/role. The identity you use must have the permissions to manage following services or resources (not limited to): VPC, EC2, IAM roles, Auto Scaling Group, and CloudFormation stacks. 5. **MAKE SURE** you follow the instructions to tear down the classroom environment on the end of training class everyday, otherwise AWS will keep charging our team. ## One-off environment setup 1. Clone the [TiDB course 303 repo](https://github.com/pingcap-inc/tidb-course-303) from Github. 2. EC2 key pair and S3 bucket management: + For PingCAP Education Cost Center + Place the private key `pe-class-key-oregon.pem` under `~/.ssh` on your macOS, with permission `400`. You will also share it with your students during the training. + Add the private key identity to the OpenSSH authentication agent by executing `ssh-add ~/.ssh/pe-class-key-oregon.pem`. In order to avoid executing this command each time you need to initialize the classroom, you can add it to your shell initialization script, such as `~/.bashrc` or `~/.bash_profile`. + For PingCAP Education APAC Cost Center + Create an EC2 key pair named `pe-class-key-singapore.pem` in region `ap-southeast-1`. + Place the private key `pe-class-key-singapore.pem` under `~/.ssh` on your macOS, with permission `400`. You will also share it with your students during the training. + Add the private key identity to the OpenSSH authentication agent by executing `ssh-add ~/.ssh/pe-class-key-singapore.pem`. In order to avoid executing this command each time you need to initialize the classroom, you can add it to your shell initialization script, such as `~/.bashrc` or `~/.bash_profile`. + For PingCAP Education Japan Cost Center + Place the private key `pe-class-key-japan.pem` under `~/.ssh` on your macOS, with permission `400`. You will also share it with your students during the training. + Add the private key identity to the OpenSSH authentication agent by executing `ssh-add ~/.ssh/pe-class-key-japan.pem`. In order to avoid executing this command each time you need to initialize the classroom, you can add it to your shell initialization script, such as `~/.bashrc` or `~/.bash_profile`. + *Contact Todd (guanglei.bao@pingcap.com) for the the private keys if you don't already have them.* + *There are several private keys available, they are used to connect to different AWS account EC2 instances for different cost centers, contact Todd (guanglei.bao@pingcap.com) for details.* ## Routines for each class ### Before the class, create a CloudFormation stack to allocate the required resources 1. Set AWS credentials environment variables for command line and programmatic access from Global Education Team AWS portal - https://pingcap.awsapps.com/start#/. 2. And, open the AWS console, stay at EC2 page, customize the instances table to display three additional tag columns, they are `trainer`, `role`, `student`. You will find it's handy. Your EC2 console instances page should look like the following screenshot: 3. Setup AWS credentials for yourself. a. If you use permanent IAM power user, follow the [instructions](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html) to set up the **default** awscli profile. b. If you use temporary IAM power user or IAM role, set your credentials to following environment variables in your local terminal: ``` $ export AWS_ACCESS_KEY_ID="" $ export AWS_SECRET_ACCESS_KEY="" $ export AWS_SESSION_TOKEN="" ``` 4. Create the class CloudFormation stack. Provide your name and your email as input parameters (for resource tagging purpose, otherwise the demo steps will fail): ``` $ cd tidb-course-303/lab-guide/env_v3/setup__cost_center/ $ ./01_deploy_classroom_by_trainer_name_and_email.sh ``` ``` $ ./01_deploy_classroom_by_trainer_name_and_email.sh guanglei guanglei.bao@pingcap.com { "StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXX:stack/guanglei-class/1aa6baa0-ee42-11ed-a1b9-06ee049f6e4d" } ``` 5. Wait the Cloud Formation stack run to completion. Run following script to check the status, until you get the `COMPLETION` status on `StackStatus` attribute in the output. It will create 1 VPC, 4 subnets, 5 auto scaling groups: ``` $ ./show_classroom_by_trainer_name.sh ``` ``` $ ./show_classroom_by_trainer_name.sh guanglei { "Stacks": [ { "StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXX:stack/guanglei-class/63795c40-efcf-11ed-907c-02430e44aec7", "StackName": "guanglei-class", "Parameters": [ { "ParameterKey": "TrainerEmail", "ParameterValue": "guanglei.bao@pingcap.com" }, { "ParameterKey": "TrainerName", "ParameterValue": "guanglei" } ], "CreationTime": "2023-05-11T07:42:32.581Z", "RollbackConfiguration": {}, "StackStatus": "CREATE_COMPLETE", "DisableRollback": false, "NotificationARNs": [], "Capabilities": [ "CAPABILITY_IAM" ], "Tags": [], "EnableTerminationProtection": false, "DriftInformation": { "StackDriftStatus": "NOT_CHECKED" } } ] } ``` ### IMPORTANT - At the class end (on the last day, when the class is over), tear down the class CloudFormation to clean up everything related to the class 1. On your local machine, under `setup__cost_center/` directory, run `remove-classroom-stack-by-trainer-name.sh`: ``` $ cd setup__cost_center/ $ ./remove_classroom_by_trainer_name.sh guanglei $ ./show_classroom_by_trainer_name.sh guanglei { "Stacks": [ { ... "StackStatus": "DELETE_IN_PROGRESS", ... } ] } ``` 2. Verify that the Cloud Formation stack had been dropped successfully. Run `show-classroom-stack-by-trainer-name.sh` until you get a `Stack with id -class does not exist` error message: ``` $ ./show_classroom_by_trainer_name.sh guanglei An error occurred (ValidationError) when calling the DescribeStacks operation: Stack with id guanglei-class does not exist ``` ## Routines for each training day 1. Before the class begin, start EC2 instances for your class according to the number of students in the class. ``` $ ./02_start_classroom_by_students_count_and_trainer_name.sh ``` For example, for trainer guanglei with 1 students (note that the total students count should be 2, one for guanglei himself, one for the student): ``` $ ./02_start_classroom_by_students_count_and_trainer_name.sh 2 guanglei ################################################## # Waiting for 150 seconds for nodes starting up. # ################################################## 2 students with 2 monitor node(s). 2 students with 6 pd node(s). 2 students with 6 kv node(s). 2 students with 4 db node(s). 2 students with 2 tiflash node(s). ... ``` 2. We use `monitor1` node as the Control Machine during the training, that is, the public IP address of the node which `Role` is `monitor1`. Run following script `03_prepare_classroom_by_students_count_and_trainer_name.sh` to prepare `monitor1` node for all students automatically: Note: Make sure that you have completed the instructions for private key portion outlined in the previous step titled [One-off environment setup] before proceeding with this step. ``` $ ./03_prepare_classroom_by_students_count_and_trainer_name.sh ``` + For example, for trainer Guanglei with 1 students (note that the total students count should be 2, 1 for guanglei himself, 1 for the student): ``` $ ./03_prepare_classroom_by_students_count_and_trainer_name.sh 2 guanglei # Preparation for all the student nodes is being carried out in the background simultaneously. # Usually, it takes approximately 5 minutes to complete no matter what the students count. # Please return to this terminal and execute "./check_monitor_by_students_count_and_trainer_name.sh 2 guanglei-bao" for verification later. # Let's wait for 5 minutes and enjoy. ``` 3. Check nodes status. ``` $ ./check_nodes_by_trainer_name.sh ``` + For example, for Guanglei with 1 students, **user1** is reserved for Guanglei himself, **user2** is for the student: ``` $ ./check_nodes_by_trainer_name.sh guanglei # Running Nodes for All Trainers. | Instance ID | Public IP | Private IP | Name | Instructor | Student | Role | Up Time (mins) | | :------------------ | :------------- | :---------- | :------ | :--------- | :------ | :------- | :------------- | | i-085ccd67a6b70d590 | 34.220.51.131 | 10.90.1.184 | db | guanglei | user1 | db1 | 4 | | i-06ab53bf78266b181 | 34.219.27.41 | 10.90.2.93 | db | guanglei | user1 | db2 | 4 | | i-04263540802494fb1 | 35.92.177.38 | 10.90.3.149 | kv | guanglei | user1 | kv1 | 3 | | i-0b981f98b9ac0040c | 34.219.103.230 | 10.90.3.154 | kv | guanglei | user1 | kv2 | 3 | | i-01f0925b19a8a1027 | 34.220.120.216 | 10.90.1.152 | kv | guanglei | user1 | kv3 | 3 | | i-04f908dbd4977fbfb | 35.85.246.165 | 10.90.4.153 | monitor | guanglei | user1 | monitor1 | 3 | | i-07620ecbcd6661eda | 35.92.126.179 | 10.90.3.145 | pd | guanglei | user1 | pd1 | 3 | | i-0cfc7c045b63e475d | 34.208.208.115 | 10.90.3.113 | pd | guanglei | user1 | pd2 | 3 | | i-0fcb5d25ba041fdb7 | 34.220.104.87 | 10.90.1.212 | pd | guanglei | user1 | pd3 | 3 | | i-0dcbd10a0a3d510ae | 34.222.201.136 | 10.90.4.49 | tiflash | guanglei | user1 | tiflash1 | 3 | | Instance ID | Public IP | Private IP | Name | Instructor | Student | Role | Up Time (mins) | | :------------------ | :------------- | :---------- | :------ | :--------- | :------ | :------- | :------------- | | i-072eca255791e9c33 | 34.208.216.126 | 10.90.2.200 | db | guanglei | user2 | db1 | 4 | | i-033dfda0a8cfe53c4 | 35.164.65.116 | 10.90.3.166 | db | guanglei | user2 | db2 | 4 | | i-08ba4121dca378e3b | 54.203.34.252 | 10.90.1.27 | kv | guanglei | user2 | kv1 | 3 | | i-0c88f1b75c466b46f | 34.209.242.166 | 10.90.2.20 | kv | guanglei | user2 | kv2 | 3 | | i-0e7dcd704c12b6b1a | 34.218.242.197 | 10.90.2.181 | kv | guanglei | user2 | kv3 | 3 | | i-00fd9f04321460988 | 44.242.172.68 | 10.90.4.231 | monitor | guanglei | user2 | monitor1 | 3 | | i-056e4ef89170edd5f | 54.202.185.133 | 10.90.1.85 | pd | guanglei | user2 | pd1 | 3 | | i-0283444b8698fd57c | 35.89.140.92 | 10.90.2.61 | pd | guanglei | user2 | pd2 | 3 | | i-0c1acd4f6daa0ac59 | 35.88.155.44 | 10.90.2.148 | pd | guanglei | user2 | pd3 | 3 | | i-0bc066cb151ea7b41 | 35.86.1.159 | 10.90.4.22 | tiflash | guanglei | user2 | tiflash1 | 3 | # Monitor Nodes. | Instance ID | Public IP | Private IP | Name | Instructor | Student | Role | Up Time (mins) | | :------------------ | :------------ | :---------- | :------ | :--------- | :------ | :------- | :------------- | | i-04f908dbd4977fbfb | 35.85.246.165 | 10.90.4.153 | monitor | guanglei | user1 | monitor1 | 27 | | Instance ID | Public IP | Private IP | Name | Instructor | Student | Role | Up Time (mins) | | :------------------ | :----------- | :--------- | :------ | :--------- | :------ | :------- | :------------- | | i-0cfbe7277ab0dc1ff | 35.86.221.19 | 10.90.4.76 | monitor | guanglei | user2 | monitor1 | 15 | # Class Total Cost: 0.20000000000000004 (USD) # Reporting End - 2023-05-11 16:20:42.016529 with 20 nodes. ``` 4. Provide the private key and the above table (with relevant information only, for example, user2 information to user2 only) to all students. + For example, distribute following information belongs to user2 to the student who you map to user2: ``` | Instance ID | Public IP | Private IP | Name | Instructor | Student | Role | Up Time (mins) | | :------------------ | :------------ | :---------- | :------ | :--------- | :------ | :------- | :------------- | | i-00fd9f04321460988 | 44.242.172.68 | 10.90.4.231 | monitor | guanglei | user2 | monitor1 | 3 | ``` 5. If you would like to `SSH` to a student's machine, you can use the following command for example (there are several private keys available, they are used to connect to different AWS account EC2 instances from different cost centers, contact Todd for details.): Note: Make sure that you have completed the instructions for private key portion outlined in the previous step titled [One-off environment setup] before proceeding with this step. `SSH` login in to node `monitor1`. The option `-A` is crucial as it enables the forwarding of connections from an authentication agent. As an instructor, it is important to inform your students about the significance of using `-A` as it is necessary to support hands-on steps without OS password throughout the exercise workflows. ``` $ ssh -A ec2-user@ ``` 6. Open the AWS console, stay at EC2 page, customize the instances table to display three additional tag columns, they are `trainer`, `role`, `student`. You will find it's handy. Your EC2 console instances page should look like the following screenshot: 7. (Optional) If student nodes need replacement during the hands-on practice, those nodes need to be re-tagged. Follow the steps below to replace all EC2 nodes belonging to a student: Select EC2 instances filter by `trainer` and `student` tags and **terminate** them, for example you want to replace all nodes for `user2`: Wait for 5 minutes, check on AWS console to make sure that there are 10 new nodes exists before executing the following script to re-tag the EC2 instances for the student: ``` $ ./retag_and_prepare_by_student_name_and_trainer_name.sh ``` + For example, if trainer Guanglei need to re-tag nodes for student user2: ``` $ ./retag_and_prepare_by_student_name_and_trainer_name.sh user2 guanglei Tagging i-061e5379eb67c331e with student:user2 Tagging i-0d289c27be7b272b5 with student:user2 Tagging i-0710e44f89fa9a36e with student:user2 Tagging pd nodes completed. Tagging i-07d8b48949fa3a2e4 with student:user2 Tagging i-09dd31ca5443545be with student:user2 Tagging i-0e8fde59dee75e0b0 with student:user2 Tagging kv nodes completed. Tagging i-009fa6d2749e0bdbf with student:user2 Tagging db nodes completed. Tagging i-03c6a8f6681333266 with student:user2 Tagging tiflash nodes completed. ``` + You might encounter following exception, it normally means auto-scaling group need more time to spin up new instances. ``` $ ./retag_and_prepare_for_student.sh user2 guanglei # Tagging nodes for student user2 # Insufficient nodes count for node type pd. Needs 3, but only 1 available. Normally, you should wait for a few minutes, and try again. ``` + The solution is wait for no more than 5 minutes and try again: ``` $ ./retag_and_prepare_for_student.sh user2 # Tagging nodes for student user2 # Tagging i-0bf6f1d61d23cde99 with student:user2 Tagging i-0e04f733dbca9f3de with student:user2 Tagging i-0176ae6b8dfe2b006 with student:user2 Tagging pd nodes completed. Tagging i-0b0032e02e862c102 with student:user2 Tagging i-0478a80bf6c3cf034 with student:user2 Tagging i-062295616f951eba7 with student:user2 Tagging kv nodes completed. Tagging i-04f2d2ae17ec5a45a with student:user2 Tagging db nodes completed. Tagging i-081482221ec426383 with student:user2 Tagging tiflash nodes completed. ``` 8. **IMPORTANT:** When the class is over for the day. Run following script as soon as possible to terminate all EC2 instances: ``` $ ./stop_classroom_by_trainer_name.sh ``` For example: ``` $ ./stop_classroom_by_trainer_name.sh guanglei # Terminating all nodes for trainer guanglei. ``` Wait for a few minutes, run check again, make sure it returns an empty list under your name: ``` $ ./check_nodes_by_trainer_name.sh guanglei # Running Instances | Instance ID | Public IP | Private IP | Name | Instructor | Student | Type | Up Time (mins) | Cost (USD) | | :---------- | :-------- | :--------- | :--- | :--------- | :------ | :--- | :------------- | :--------- | # Class Total Cost: 0.0 (USD) ```