1 00:00:00,000 --> 00:00:14,040 [Music] When we talk about TiDB, why do we call it  a distributed database? Before we deep   2 00:00:14,040 --> 00:00:19,380 dive into the architectures, I want to give you a  quick answer to another straightforward question.   3 00:00:19,980 --> 00:00:27,300 Why should we use TiDB? When you start to use TiDB  you will soon realize the following benefits.   4 00:00:27,300 --> 00:00:34,440 Firstly, TiDB has seamless scalability. In TiDB  all key components are decoupled with each   5 00:00:34,440 --> 00:00:40,980 other and support horizontal scalability.  Secondly, TiDB provides high availability.   6 00:00:41,820 --> 00:00:46,740 For those stateful components consistent  data replications are supported.   7 00:00:47,520 --> 00:00:52,500 You can achieve high availability easily  by deploying multiple such components or   8 00:00:52,500 --> 00:00:57,780 guarantee a smooth Disaster Recovery by  deploying them across geographic locations.   9 00:00:58,920 --> 00:01:05,100 Thirdly, TiDB is actually a horizontal  scalable NoSQL database with SQL and   10 00:01:05,100 --> 00:01:12,720 transaction compatibility. By using TiDB you  get benefits of NoSQL and SQL at the same time. 11 00:01:15,240 --> 00:01:21,660 And, your database can grow into massive magnitude  volume that MySQL database cannot even imagine. 12 00:01:23,760 --> 00:01:29,340 The auto sharding in TiDB is completely  transparent to applications, meaning zero   13 00:01:29,340 --> 00:01:35,580 application changes needed when your data  grows. Lastly, you can talk to TiDB using   14 00:01:35,580 --> 00:01:41,760 standard SQL, meaning you already have  the ticket to the Big Data party. No ETL   15 00:01:41,760 --> 00:01:48,240 or extraction transformation and load required.  TiDB can store and retrieve any amount of data.  16 00:01:49,200 --> 00:01:55,740 And, it provides both row based store and columnar  storage suitable for transaction processing and   17 00:01:55,740 --> 00:02:02,760 analytical processing at the same time. I'm going  to cover it in more detail in this course later. 18 00:02:07,080 --> 00:02:13,560 Here goes an example architecture of TiDB.  We placed six database clients instance here, 19 00:02:14,220 --> 00:02:20,880 in the practice, of course, there could be much  more. The architecture is already Cloud friendly.   20 00:02:20,880 --> 00:02:27,900 The availability zone here stands for the data  centers or AZ provided by cloud service providers.   21 00:02:29,040 --> 00:02:35,880 And, we have MySQL for online transaction  processing. The standby database here is a   22 00:02:35,880 --> 00:02:41,700 full replica of the leader database for high  availability and disaster recovery purposes.   23 00:02:42,480 --> 00:02:48,840 With the leader for writes and read replica for  reads you can handle a pretty serious workload.   24 00:02:48,840 --> 00:02:55,500 And, there is a cache for transit objects or  hotspot objects to improve the read performance.  25 00:02:56,400 --> 00:03:03,600 NoSQL database here is a horizontal scalable persistent store  and its nodes can span across availability zones   26 00:03:03,600 --> 00:03:10,740 to achieve the requirements of high availability,  disaster recovery, performance and data growth at   27 00:03:10,740 --> 00:03:19,800 scale. The data layer architecture is complicated  to some extent. Let's zoom in MySQL master. As the   28 00:03:19,800 --> 00:03:25,560 business grows the data will eventually outgrows  the single machine. One way of handling the   29 00:03:25,560 --> 00:03:31,500 increasing writes is to split the data over  multiple databases using what's called sharding.   30 00:03:32,880 --> 00:03:38,220 For example to begin with you might store  all information for users with the user ID   31 00:03:38,220 --> 00:03:45,840 starting with the letter A through Z in a single  database. To increase the write capacity you   32 00:03:45,840 --> 00:03:52,800 split the database into two. Well, some users are  handling the first database and some in the second.   33 00:03:53,520 --> 00:03:58,620 Here we use the first letter of the user  ID as an example, but in practice people   34 00:03:58,620 --> 00:04:04,260 sometimes use consistent hash to determine the shard used for a specific piece of data.   35 00:04:05,580 --> 00:04:10,680 When you need to increase capacity further you  do the same thing and split data set again.   36 00:04:11,700 --> 00:04:15,540 Adding shards that makes generating reports  gets more and more difficult. 37 00:04:17,280 --> 00:04:22,800 Those are only for write operations, you also  need replicas in local availability zone for   38 00:04:22,800 --> 00:04:28,260 read performance and standby database per  shard in the peering AZ for high availability   39 00:04:28,260 --> 00:04:34,680 and disaster recovery. This is no fun  at all for the developers and Ops team.   40 00:04:36,120 --> 00:04:41,520 By using TiDB the data layer architecture is  straightforward and the exposed complexity   41 00:04:41,520 --> 00:04:50,040 is dramatically mitigated like NoSQL, TiDB is  distributed and is capable of a horizontal scaling.   42 00:04:51,660 --> 00:04:55,980 TiDB handles sharding automatically and  it's transparent to your application.   43 00:04:57,300 --> 00:05:04,080 Unlike NoSQL, TiDB supports transaction and  SQL in native and is compatible with MySQL.  44 00:05:04,740 --> 00:05:11,100 Within TiDB, the replication happens automatically,  the sharding happens automatically, the role based  45 00:05:11,100 --> 00:05:17,460 and columnar store transformation happens  automatically. It acts in a position as   46 00:05:17,460 --> 00:05:22,740 a multiple purpose database to replace MySQL and NoSQL database simultaneously.   47 00:05:24,120 --> 00:05:27,900 TiDB is versatile, so you  can focus on your application.   48 00:05:28,740 --> 00:05:36,780 Users acquire data agility without worrying about  the future of growing pain. But I have to mention   49 00:05:36,780 --> 00:05:42,960 that even TiDB is versatile it cannot eliminate  the hot spot bottleneck caused by your application.   50 00:05:43,800 --> 00:05:49,200 So, cache components like Redis are still  required for the application hotspot cases.   51 00:05:50,280 --> 00:05:55,260 We will discuss how to mitigate hotspot in  detail in the TiDB Performance Tuning course   52 00:05:55,260 --> 00:06:01,800 in the database operation learning path and in TiDB  specific feature course in the developer learning path.   53 00:06:02,820 --> 00:06:09,660 The core components of TiDB database, in short TiDB, is composed of three server clusters.   54 00:06:10,320 --> 00:06:14,700 All of which communicate with each  other to form a database cluster.   55 00:06:16,500 --> 00:06:24,000 As the components include TiDB server cluster  TiKV server cluster and the placement driver or PD   56 00:06:24,000 --> 00:06:32,100 cluster. All three sub-clusters are decoupled with  each other. The number of TiDB servers does not   57 00:06:32,100 --> 00:06:37,920 have to be equal to the number of TiKV servers  and does not have to be equal to the number   58 00:06:37,920 --> 00:06:46,920 of PD servers. For example, you can have two TiDB  servers, three PD servers, and six TiKV servers.   59 00:06:46,920 --> 00:06:54,000 The first server cluster I would like to discuss  about is TiDB server cluster. Do not be confused   60 00:06:54,000 --> 00:07:01,080 between TiDB and TiDB server cluster. The TiDB  server cluster is a component, a sub-cluster of TiDB.   61 00:07:02,400 --> 00:07:08,520 The TiDB sub-cluster which represents SQL layer  exposes the connection endpoint of MySQL protocol.   62 00:07:09,480 --> 00:07:14,280 TiDB server instance was in the cluster that are  responsible for accepting connections from the   63 00:07:14,280 --> 00:07:20,340 database clients. The TiDB server cluster is  stateless, and in practice multiple instance   64 00:07:20,340 --> 00:07:26,220 of TiDB server can be launched. And an unified  DNS name can be provided through a load   65 00:07:26,220 --> 00:07:35,040 balancing component such as LVS, HAProxy or  F5 BIG-IP. If you are in the public cloud, the   66 00:07:35,040 --> 00:07:41,880 Elastic Load Balancing service provided by  the cloud service providers also works. Thus,   67 00:07:41,880 --> 00:07:47,820 client connections can be evenly distributed across  multiple TiDB server instances without explicit   68 00:07:47,820 --> 00:07:55,200 efforts in your application. The TiDB servers  do not actually store the table data, it only   69 00:07:55,200 --> 00:08:00,960 parses SQL statements and forward a data request  to the storage layer which is TiKV cluster.   70 00:08:02,340 --> 00:08:08,280 The second sub-cluster I would like to discuss  about is PD or placement driver server cluster.   71 00:08:09,180 --> 00:08:14,280 It is the metadata management module for the  entire TiDB which stores the real-time data   72 00:08:14,280 --> 00:08:19,980 distribution of each TiKV node and the  overall topology configuration of TiDB.   73 00:08:20,880 --> 00:08:27,060 It generates the transaction ID for distributed  transaction which is based on the TSO or timestamp   74 00:08:27,060 --> 00:08:33,420 Oracle. TSO is a timestamp to guarantee that  two changes happen within a database in a   75 00:08:33,420 --> 00:08:38,100 temporal sequential order even the  database is a distributed database. 76 00:08:39,540 --> 00:08:45,240 PD not only stores metadata, it also issues  standard scheduling commands to specific   77 00:08:45,240 --> 00:08:51,780 TiKV nodes based on data distribution  that is reported in real-time by TiKV nodes.   78 00:08:53,160 --> 00:08:59,880 PD is considered to be the brain of the entire  cluster. As the cluster backed by Raft algorithm,   79 00:08:59,880 --> 00:09:06,480 PD needs to be deployed to an odd number nodes.  It is usually recommended to deploy three nodes at   80 00:09:06,480 --> 00:09:13,500 least. As long as the majority nodes are alive your  cluster is open for reads and write operations. 81 00:09:16,740 --> 00:09:21,840 The last sub-cluster, the TiKV cluster  is responsible for storing table data.   82 00:09:22,440 --> 00:09:27,720 From an external view, TiKV is a distributed  transactional key-value NoSQL database.   83 00:09:29,220 --> 00:09:36,180 TiKV also uses Raft algorithm for replication  to ensure data consistency in disaster recovery.   84 00:09:37,860 --> 00:09:42,840 The load balancing of the data placement  among different TiKV nodes are scheduled   85 00:09:42,840 --> 00:09:49,500 by PD, the brain of the cluster. I'm going  to show you a single example similar to the   86 00:09:49,500 --> 00:09:55,560 steps shown on the slides so that you can have  a basic understanding of how TiKV stores data.   87 00:09:56,340 --> 00:10:01,500 I'm going to use TiUP command line tool to  launch a Playground sandbox on my local machine.   88 00:10:02,760 --> 00:10:09,780 Once the TiDB is up and running, I will use Raw KV  API to do simple put/get operations to show you   89 00:10:09,780 --> 00:10:17,400 that TiKV is a key-value store. The script itself  is pretty intuitive, it calls TiUP to spin up   90 00:10:17,400 --> 00:10:23,940 a whole TiDB system tagged with the name  classroom. We deployed three TiDB instances, three    91 00:10:23,940 --> 00:10:31,140 PD instances and three TiKV instances and  one TiFlash instance. Let's set aside TiFlash   92 00:10:31,140 --> 00:10:36,960 instance for now. We'll cover TiFlash later in  this course. And, no worries if you don't have   93 00:10:36,960 --> 00:10:42,660 TiUP right now. You will have TiUP installed after  lab 1 at the end of this unit. So, let's do it. 94 00:10:48,960 --> 00:10:55,380 You can see that PD is up, then TiKV  and TiDB are getting up. In a real   95 00:10:55,380 --> 00:10:59,580 world production environment, you can  spin up this servers on different nodes.   96 00:11:00,420 --> 00:11:06,660 Please note that the Playground is just a sandbox for  testing purpose only. Okay, it's all ready by now. 97 00:11:11,640 --> 00:11:15,600 You can see that we use a Raw  client to build a connection. 98 00:11:18,240 --> 00:11:20,460 The endpoints we are using are from PD. 99 00:11:22,920 --> 00:11:26,580 We simply put and get a new key-value pair.   100 00:11:27,720 --> 00:11:33,720 And then, let's try to get a value from a  non-existing key. Let's see what happens. 101 00:11:40,380 --> 00:11:46,800 That is what we expected, right? We can  actually put key-value pair into TiKV.   102 00:11:46,800 --> 00:11:52,560 And, we can get value from key 1, but  not from non-existing key, key 0.   103 00:11:53,400 --> 00:12:00,720 So, this is the first time and the last time we  use Raw KK interface for this course. After that,   104 00:12:00,720 --> 00:12:06,300 throughout this course the only language you  will use to communicate was TiDB will be SQL.