1 00:00:00,000 --> 00:00:14,040 [Music] When we talk about TiDB, why do we call it a distributed database? Before we deep 2 00:00:14,040 --> 00:00:19,380 dive into the architectures, I want to give you a quick answer to another straightforward question. 3 00:00:19,980 --> 00:00:27,300 Why should we use TiDB? When you start to use TiDB you will soon realize the following benefits. 4 00:00:27,300 --> 00:00:34,440 Firstly, TiDB has seamless scalability. In TiDB all key components are decoupled with each 5 00:00:34,440 --> 00:00:40,980 other and support horizontal scalability. Secondly, TiDB provides high availability. 6 00:00:41,820 --> 00:00:46,740 For those stateful components consistent data replications are supported. 7 00:00:47,520 --> 00:00:52,500 You can achieve high availability easily by deploying multiple such components or 8 00:00:52,500 --> 00:00:57,780 guarantee a smooth Disaster Recovery by deploying them across geographic locations. 9 00:00:58,920 --> 00:01:05,100 Thirdly, TiDB is actually a horizontal scalable Key-Value database with SQL and 10 00:01:05,100 --> 00:01:12,720 transaction compatibility. By using TiDB you get benefits of NoSQL and SQL at the same time. 11 00:01:15,240 --> 00:01:21,660 And, your database can grow into massive magnitude volume that MySQL database cannot even imagine. 12 00:01:23,760 --> 00:01:29,340 The auto sharding in TiDB is completely transparent to applications, meaning zero 13 00:01:29,340 --> 00:01:35,580 application changes needed when your data grows. Lastly, you can talk to TiDB using 14 00:01:35,580 --> 00:01:41,760 standard SQL, meaning you already have the ticket to the Big Data party. No ETL 15 00:01:41,760 --> 00:01:48,240 or extraction transformation and load required. TiDB can store and retrieve any amount of data. 16 00:01:49,200 --> 00:01:55,740 And, it provides both row based store and columnar storage suitable for transaction processing and 17 00:01:55,740 --> 00:02:02,760 analytical processing at the same time. I'm going to cover it in more detail in this course later. 18 00:02:07,080 --> 00:02:13,560 Here goes an example architecture without TiDB. We placed six database clients instance here, 19 00:02:14,220 --> 00:02:20,880 in the practice, of course, there could be much more. The architecture is already Cloud friendly. 20 00:02:20,880 --> 00:02:27,900 The availability zone here stands for the data centers or AZ provided by cloud service providers. 21 00:02:29,040 --> 00:02:35,880 And, we have MySQL for online transaction processing. The standby database here is a 22 00:02:35,880 --> 00:02:41,700 full replica of the leader database for high availability and disaster recovery purposes. 23 00:02:42,480 --> 00:02:48,840 With the leader for writes and read replica for reads you can handle a pretty serious workload. 24 00:02:48,840 --> 00:02:55,500 And, there is a cache for transit objects or hotspot objects to improve the read performance. 25 00:02:56,400 --> 00:03:03,600 NoSQL database here is a horizontal scalable persistent store and its nodes can span across availability zones 26 00:03:03,600 --> 00:03:10,740 to achieve the requirements of high availability, disaster recovery, performance and data growth at 27 00:03:10,740 --> 00:03:19,800 scale. The data layer architecture is complicated to some extent. Let's zoom in MySQL master. As the 28 00:03:19,800 --> 00:03:25,560 business grows the data will eventually outgrows the single machine. One way of handling the 29 00:03:25,560 --> 00:03:31,500 increasing writes is to split the data over multiple databases using what's called sharding. 30 00:03:32,880 --> 00:03:38,220 For example to begin with you might store all information for users with the user ID 31 00:03:38,220 --> 00:03:45,840 starting with the letter A through Z in a single database. To increase the write capacity you 32 00:03:45,840 --> 00:03:52,800 split the database into two. Well, some users are handling the first database and some in the second. 33 00:03:53,520 --> 00:03:58,620 Here we use the first letter of the user ID as an example, but in practice people 34 00:03:58,620 --> 00:04:04,260 sometimes use consistent hash to determine the shard used for a specific piece of data. 35 00:04:05,580 --> 00:04:10,680 When you need to increase capacity further you do the same thing and split data set again. 36 00:04:11,700 --> 00:04:15,540 Adding shards that makes generating reports gets more and more difficult. 37 00:04:17,280 --> 00:04:22,800 Those are only for write operations, you also need replicas in local availability zone for 38 00:04:22,800 --> 00:04:28,260 read performance and standby database per shard in the peering AZ for high availability 39 00:04:28,260 --> 00:04:34,680 and disaster recovery. This is no fun at all for the developers and Ops team. 40 00:04:36,120 --> 00:04:41,520 By using TiDB the data layer architecture is straightforward and the exposed complexity 41 00:04:41,520 --> 00:04:50,040 is dramatically mitigated like NoSQL, TiDB is distributed and is capable of a horizontal scaling. 42 00:04:51,660 --> 00:04:55,980 TiDB handles sharding automatically and it's transparent to your application. 43 00:04:57,300 --> 00:05:04,080 Unlike NoSQL, TiDB supports transaction and SQL in native and is compatible with MySQL. 44 00:05:04,740 --> 00:05:11,100 Within TiDB, the replication happens automatically, the sharding happens automatically, the row based 45 00:05:11,100 --> 00:05:17,460 and columnar store transformation happens automatically. It acts in a position as 46 00:05:17,460 --> 00:05:22,740 a multiple purpose database to replace MySQL and NoSQL database simultaneously. 47 00:05:24,120 --> 00:05:27,900 TiDB is versatile, so you can focus on your application. 48 00:05:28,740 --> 00:05:36,780 Users acquire data agility without worrying about the future of growing pain. But I have to mention 49 00:05:36,780 --> 00:05:42,960 that even TiDB is versatile it cannot eliminate the hot spot bottleneck caused by your application. 50 00:05:43,800 --> 00:05:49,200 So, cache components like Redis are still required for the application hotspot cases. 51 00:05:50,280 --> 00:05:55,260 We will discuss how to mitigate hotspot in detail in the TiDB Performance Tuning course 52 00:05:55,260 --> 00:06:01,800 in the database operation learning path and in TiDB specific feature course in the developer learning path. 53 00:06:02,820 --> 00:06:09,660 The core components of TiDB database, in short TiDB, is composed of three server clusters. 54 00:06:10,320 --> 00:06:14,700 All of which communicate with each other to form a database cluster. 55 00:06:16,500 --> 00:06:24,000 As the components include TiDB server cluster TiKV server cluster and the placement driver or PD 56 00:06:24,000 --> 00:06:32,100 cluster. All three sub-clusters are decoupled with each other. The number of TiDB servers does not 57 00:06:32,100 --> 00:06:37,920 have to be equal to the number of TiKV servers and does not have to be equal to the number 58 00:06:37,920 --> 00:06:46,920 of PD servers. For example, you can have two TiDB servers, three PD servers, and six TiKV servers. 59 00:06:46,920 --> 00:06:54,000 The first server cluster I would like to discuss about is TiDB server cluster. Do not be confused 60 00:06:54,000 --> 00:07:01,080 between TiDB and TiDB server cluster. The TiDB server cluster is a component, a sub-cluster of TiDB. 61 00:07:02,400 --> 00:07:08,520 The TiDB sub-cluster which represents SQL layer exposes the connection endpoint of MySQL protocol. 62 00:07:09,480 --> 00:07:14,280 TiDB server instance was in the cluster that are responsible for accepting connections from the 63 00:07:14,280 --> 00:07:20,340 database clients. The TiDB server cluster is stateless, and in practice multiple instance 64 00:07:20,340 --> 00:07:26,220 of TiDB server can be launched. And an unified DNS name can be provided through a load 65 00:07:26,220 --> 00:07:35,040 balancing component such as LBS, HAProxy or F5 BIG-IP. If you are in the public cloud, the 66 00:07:35,040 --> 00:07:41,880 Elastic Load Balancing service provided by the cloud service providers also works. Thus, 67 00:07:41,880 --> 00:07:47,820 client connections can be evenly distributed across multiple TiDB server instances without explicit 68 00:07:47,820 --> 00:07:55,200 efforts in your application. The TiDB servers do not actually store the table data, it only 69 00:07:55,200 --> 00:08:00,960 parses SQL statements and forward a data request to the storage layer which is TiKV cluster. 70 00:08:02,340 --> 00:08:08,280 The second sub-cluster I would like to discuss about is PD or placement driver server cluster. 71 00:08:09,180 --> 00:08:14,280 It is the metadata management module for the entire TiDB which stores the real-time data 72 00:08:14,280 --> 00:08:19,980 distribution of each TiKV node and the overall topology configuration of TiDB. 73 00:08:20,880 --> 00:08:27,060 It generates the transaction ID for distributed transaction which is based on the TSO or timestamp 74 00:08:27,060 --> 00:08:33,420 Oracle. TSO is a timestamp to guarantee that two changes happen within a database in a 75 00:08:33,420 --> 00:08:38,100 temporal sequential order even the database is a distributed database. 76 00:08:39,540 --> 00:08:45,240 PD not only stores metadata, it also issues standard scheduling commands to specific 77 00:08:45,240 --> 00:08:51,780 TiKV nodes based on data distribution that is reported in real-time by TiKV nodes. 78 00:08:53,160 --> 00:08:59,880 PD is considered to be the brain of the entire cluster. As the cluster backed by Raft algorithm, 79 00:08:59,880 --> 00:09:06,480 PD needs to be deployed to an odd number nodes. It is usually recommended to deploy three nodes at 80 00:09:06,480 --> 00:09:13,500 least. As long as the majority nodes are alive your cluster is open for reads and write operations. 81 00:09:16,740 --> 00:09:21,840 The last sub-cluster, the TiKV cluster is responsible for storing table data. 82 00:09:22,440 --> 00:09:27,720 From an external view, TiKV is a distributed transactional key-value NoSQL database. 83 00:09:29,220 --> 00:09:36,180 TiKV also uses Raft algorithm for replication to ensure data consistency in disaster recovery. 84 00:09:37,860 --> 00:09:42,840 The load balancing of the data placement among different TiKV nodes are scheduled 85 00:09:42,840 --> 00:09:49,500 by PD, the brain of the cluster. I'm going to show you a single example similar to the 86 00:09:49,500 --> 00:09:55,560 steps shown on the slides so that you can have a basic understanding of how TiKV stores data. 87 00:09:56,340 --> 00:10:01,500 I'm going to use TiUP command line tool to launch a Playground sandbox on my local machine. 88 00:10:02,760 --> 00:10:09,780 Once the TiDB is up and running, I will use Raw KV API to do simple put/get operations to show you 89 00:10:09,780 --> 00:10:17,400 that TiKV is a key-value store. The script itself is pretty intuitive, it calls TiUP to spin up 90 00:10:17,400 --> 00:10:23,940 a whole TiDB system tagged with the name classroom. We deployed three TiDB instances, three 91 00:10:23,940 --> 00:10:31,140 PD instances and three TiKV instances and one TiFlash instance. Let's set aside TiFlash 92 00:10:31,140 --> 00:10:36,960 instance for now. We'll cover TiFlash later in this course. And, no worries if you don't have 93 00:10:36,960 --> 00:10:42,660 TiUP right now. You will have TiUP installed after lab 1 at the end of this unit. So, let's do it. 94 00:10:48,960 --> 00:10:55,380 You can see that PD is up, then TiKV and TiDB are getting up. In a real 95 00:10:55,380 --> 00:10:59,580 world production environment, you can spin up this servers on different nodes. 96 00:11:00,420 --> 00:11:06,660 Please note that the Playground is just a sandbox for testing purpose only. Okay, it's all ready by now. 97 00:11:11,640 --> 00:11:15,600 You can see that we use a Raw client to build a connection. 98 00:11:18,240 --> 00:11:20,460 The endpoints we are using are from PD. 99 00:11:22,920 --> 00:11:26,580 We simply put and get a new key-value pair. 100 00:11:27,720 --> 00:11:33,720 And then, let's try to get a value from a non-existing key. Let's see what happens. 101 00:11:40,380 --> 00:11:46,800 That is what we expected, right? We can actually put key-value pair into TiKV. 102 00:11:46,800 --> 00:11:52,560 And, we can get value from key 1, but not from non-existing key, key 0. 103 00:11:53,400 --> 00:12:00,720 So, this is the first time and the last time we use Raw KV interface for this course. After that, 104 00:12:00,720 --> 00:12:06,300 throughout this course the only language you will use to communicate was TiDB will be SQL.