EN: |+| Welcome to the 101 TiDB Technical Essentials course, this is module 1 |+| TiDB database architecture overview. EN: My name is Jeff and I am your instructor. EN: For this module |+| after completing this module you will understand the overall architecture of TiDB database and |+| describe the main features of TiDB server, TiKV and the PD. EN: Before we get started |+| there's something you need to know when we say TiDB we can mean one of two things. EN: The first is the |+| database itself it's called TiDB, second there is a TiDB component within the TiDB database. EN: |+| Once you understand this it shouldn't cause a problem. EN: Let's move on to TiDB architecture. EN: |+| TiDB is a distributed relational database that supports both online transaction processing and |+| online analytical processing. EN: Databases that support both OLTP and OLAP use cases |+| are also known as Hybrid Transactional Analytical databases or HTAP databases. EN: |+| TiDB has horizontal scale out and scale in, financial grade high availability, |+| real-time HTAP and Cloud native distributed support. EN: In addition, TiDB is compatible with |+| the MySQL 5.7 client protocol and it is compatible with the MySQL ecosystem. EN: The three components of |+| the TiDB database are PD cluster, TiDB server, and TiKV. EN: The PD cluster is the control center of |+| the entire database, think of it as the brains of TiDB database. EN: It is responsible for generating |+| Global Timestamp Oracle timestamps, scheduling the distribution of regions in TiKV nodes, |+| collecting the health and distribution status of TiKV nodes and assigning transaction IDs to |+| distributed transactions. EN: PD itself is also composed of at least three nodes and has |+| high availability capabilities. EN: We recommend that you deploy an odd number of PD nodes. EN: |+| The TiDB server, this is the component I mentioned previously it is responsible for handling client |+| Connections in other words when you connect to a TiDB database your client will connect to TiDB |+| server component. EN: The TiDB server is responsible for client connections, parsing SQL statements, |+| compiling SQL statements, generating execution plans, and returning results to the client |+| application. EN: Execution plans are the steps the database will take in order to satisfy the query. EN: |+| You may be asking yourself where does TiDB store its data? EN: That's a great question, the OLTP data |+| is stored in the TiKV cluster in the form of key value pairs. EN: The TiKV cluster is responsible for |+| data persistence concurrency control, filtering data at the storage level so that queries run |+| fast, High Availability and ensuring that the data is consistent. EN: We will discuss how this is |+| done in the TiKV module. EN: The TiDB server layer is stateless, in practice we will run multiple TiDB |+| server instances. EN: Each of these instances will have its own IP address that users can use to |+| connect to but this isn't how we do it in practice we want to provide a single IP address to the |+| outside world through a load balancer such as LVS, HA proxy or F5. EN: Using a load balancer ensures that |+| the client connections can be evenly distributed across multiple TiDB server instances. EN: TiFlash is |+| a columnar storage engine that provides the OLAP capabilities to TiDB. EN: TiKV is the primary target |+| for the data. EN: TiKV will synchronize its data with TiFlash using the Raft consensus algorithm, |+| this ensures load balancing and strong consistent reads. EN: TiFlash also supports real-time updates. EN: |+| The combination of TiKV and TiFlash is what makes TiDB a HTAP database. EN: Next, we will discuss the |+| main functions implementation principles and features of TIDB, TiKV, TiFlash and PD. EN: |+| Let's take a detailed look at the main functions and implementation principles of TiDB server. EN: |+| TiDB server itself is only responsible for the SQL Computing engine and does not store data, |+| so TiDB server is stateless. EN: When our system faces a large number of concurrent connections, we can |+| load balance by adding TiDB server instances to improve high concurrency processing capabilities. EN: |+| The first thing we need to look at is one handling client connections. EN: Users |+| are authenticated by username and password and the connected users are empowered, that means |+| that they can query the database these tasks are mainly done by the protocol layer module. EN: |+| Number two is parse and compile SQL statements. EN: After listening to the SQL statement. EN: |+| TiDB server parses and compiles it during the compilation process it optimizes and |+| generates an execution plan, these tasks are done by the parse and compile modules. EN: |+| Relational and KV data transformation. EN: Since data is stored on TiKV in the form of key value pairs, |+| there's a big difference between relational database tables which have tables, |+| indexes and rows and a key value pair structure. EN: If you want to execute SQL, you must be able to convert |+| between relational data and KV storage. EN: Next, SQL statement execution. EN: According to the compiled |+| execution plan, it is executed on TiKV in batches. EN: The data is read by the executor module and the |+| DistSQL module. EN: And the transaction related operations are completed by the transaction |+| module and the KV module. EN: Execution of online DDL. EN: The TiDB database is designed so that data |+| definition language operations do not lock tables. EN: This is done by the schema load module the worker |+| module and the start job module. EN: And finally, garbage collection. EN: The garbage collection |+| back-end module is responsible for recovering data from old versions that have expired and |+| this part will be explained in the TiKV course. EN: Next, let's take a look at the TiKV module. EN: |+| We've mentioned TiKV a number of times, it handles storing our relational data that is OLTP data. EN: |+| Persistent data. EN: TiKV provides the TiDB database with persistent relational storage |+| to make this happen PingCAP chose the RocksDB storage engine. EN: RocksDB is based |+| on the LSM tree data structure. EN: RocksDB is the persistent storage engine for TiKV. EN: |+| The RocksDB engine uses mechanisms such as write-ahead logs to ensure that data is not lost. EN: |+| Next bullet, distributed transaction support, TiDB uses a two-phase commit approach that is based on |+| the Percolator transaction model published by Google. EN: The Percolator transaction model |+| ensures that storage lock space is allocated separately on each KV node, this uses RocksDB |+| column family locks, this is also called a column cluster. EN: The Timestamp Oracle and the PD also |+| play a part in this process. EN: The TiDB database supports two locking modes, optimistic locking |+| and pessimistic locking. EN: Next bullet, strong consistency and high availability of replicas. EN: |+| In the Raft algorithm data is stored in regions, each region has three replicas, |+| by default a replica cannot be larger than 96 megabytes, if a replica exceeds this size it is |+| split into multiple regions, this helps ensure that TiDB can scale dynamically. EN: |+| Next is MVCC or multi-version concurrency control. EN: MVCC is essential for highly concurrent reads and |+| writes. EN: It is the basis for functions such as concurrency control, snapshot isolation level, |+| distributed transactions and fast data recovery. EN: TiKV's MVCC implementation is done by adding a |+| version number after the key and after the historical version data reaches a certain |+| period of time, it is cleaned up by TiDB server garbage collection. EN: Next up, coprocessor. EN: This |+| is really operator push down. EN: The co-processor function enables TiKV to perform data filtering, |+| partial aggregations, etc on the local node, so that the SQL calculation can be pushed down to the |+| distributed storage nodes to complete it. EN: This mechanism not only reduces Network interaction |+| costs, but also makes full use of the Computing resources, CPU, cache and memory of the TiDB |+| server and also makes full use of the distributed storage nodes parallel processing power. EN: |+| Next up we're going to talk about the placement driver, do you remember the placement driver? EN: |+| We mentioned it earlier it's most commonly referred to as the PD. EN: It is the brains of |+| the TiDB database. EN: The PD is responsible for metadata storage for the entire TiKV cluster. EN: |+| The PD is responsible for maintaining where data lives on the TiDB cluster, |+| it is responsible for maintaining information used by Raft as well. EN: |+| TiDB servers obtain routing information from the PD and then use that information |+| to access data in the TiKV cluster. EN: In addition, topological label information is stored in the PD, |+| we will cover this in an upcoming module. EN: I previously made a recommendation regarding the PD, |+| do you remember what it was? EN: I'm going to make an uncomfortable pause here. EN: |+| That was uncomfortable. EN: The recommendation was there should be an odd number of PD nodes |+| in the PD cluster. EN: We'll cover why that is in an upcoming module. EN: Next, the PD is responsible for |+| assigning a global ID and a transaction ID. EN: The global ID we use such as store ID, |+| region ID, peer ID, table ID index ID, etc and the transaction ID of the distributed |+| transactions are also assigned by the PD. EN: The PD is responsible for generating a global timestamp |+| using the Timestamp Oracle. EN: Timestamp Oracle is a global timestamp used to guarantee the order |+| of transactions in the database. EN: It's very important for the database to know the exact |+| order of transactions in case we have to roll things back we will introduce the details later. EN: |+| The PD is responsible for collecting cluster information for scheduling. EN: The TiKV node |+| periodically reports the status of the cluster to PD. EN: The PD is responsible for scheduling the |+| region among TiKV nodes according to relevant status. EN: This has to do with what we talked about |+| splitting replicas at 96 megabytes, etc, in the previous section. EN: We will introduce the details |+| of this later. EN: And finally, the PD is responsible for providing TiDB dashboard service. EN: The PD hosts |+| a small web server that runs the TiDB dashboard. EN: The dashboard provides a place where you can |+| easily find information about your TiDB database. EN: Once again we will discuss this later as well. EN: |+| The second component in our storage story is TiFlash, recall that TiFlash implements the |+| analytics side of HTAP. EN: This is a good time to mention that TiKV as the OLTP side of storage is |+| required. EN: PinpCAP has customers whose use cases don't require analytics, these customers only |+| use TiKV for their storage. EN: If really when they decide they need enhanced analytics capabilities |+| they can easily install TiFlash thereby adding OLAP capabilities to their TiDB environment. EN: |+| Here are some capabilities which make TiFlash special. EN: |+| Asynchronous replication. EN: TiKV is the primary storage component in fact TiFlash cannot work |+| without it. EN: Another way to say this I can have a TiDB database with only TiKV but I can't |+| have one with only TiFlash. EN: When you insert update or load data into TiDB it is written |+| to TiKV first then replicated to TiFlash. EN: This operation has performed asynchronous. EN: There's |+| even a special Raft role that performs this task, the Raft Learner handles this. EN: |+| If a TiFlash node goes down or if the network has high latency the data can be read from TiKV, |+| This redundancy ensures normal operations. EN: This replication mechanism also inherits the automatic |+| load balancing and high availability of the TiKV system, it does not rely on additional |+| replication pipelines, but directly receives TiKV data transmissions in a one-to-many |+| manner as long as the data in TiKV is not lost a replica of TiFlash can be created at any time. EN: |+| Next, consistency TiFlash provides the same snapshot isolation level support as TiKV, |+| this ensures that the read data is up to date and that is ensuring that previously written |+| data can be read this consistency is achieved by verifying the replication progress of the data. EN: |+| Each time our read request is received, the region replica and TiFlash initiates a progress |+| check a very light procedure call request to the leader node and only responds to reads if |+| the process ensures that it contains at least the data covered by the read request timestamp. EN: |+| Next, number three or the third one. EN: Column oriented store improves analytics query |+| efficiency. EN: Yeah we've said this before but it's worth repeating, TiFlash is a columnar data store. EN: |+| Columnar data stores were designed for analytics workflows, row based storage |+| is best used for transactional processing, so row based storage is ideal for use cases |+| involving indexes and small result sets. EN: Columnar storage is ideal for use cases |+| involving table scans and analytics techniques that require reading the data multiple times. EN: |+| The introduction of MPP are massively parallel processing functional architecture distributes |+| computing tasks to multiple server nodes in parallel. EN: After the calculation is completed |+| on each node, the results of each part are summarized together to obtain the final answer. EN: |+| This processing model makes the analysis query service on TiFlash more efficient. EN: |+| Next, business isolation. EN: This is a good one, as mentioned earlier the column store is very |+| friendly to statistical analysis queries or OLAP and the row store is friendly to transaction OLTP, |+| so TiKV and TiFlash can be used for two distinct business scenarios or use cases, OLTP and OLAP. EN: |+| Using one environment for both of these use cases saves admin and processing cost. EN: Plus |+| and this is huge it means that you don't need to create and manage OLTP to OLAP |+| ETL processes, Thus saving more time and resources. EN: |+| And finally this one's my favorite, whether it's TiKV or TiFlash their SQL entry is the |+| TiDB server. EN: So what does that mean? EN: This means that your queries for both sides go through TiDB |+| server and the optimizer in TiDB server can choose between TiKV and TiFlash based on the query. EN: So, |+| this is based on the SQL statement. EN: So, as a DBA or developer, you write the query and TiDB chooses |+| the best place to go for the data. EN: You don't have to worry about it. EN: Totally transparent to you. EN: |+| This marks the end of the first module. EN: We've taken a brief look at TiDB and its architecture. EN: |+| So you can now describe the overall architecture of the TiDB database and you understand ... EN: well, |+| understand isn't really a goal, let's say you can describe the main functions of TiDB server, TiKV |+| and the placement driver. EN: Can you describe the difference between TiDB database and TiDB server? EN: |+| Database refers to the entire database environment. EN: TiDB server on the other hand |+| refers to the SQL processing component of the TiDB database. EN: |+| So TiDB server is a component of TiDB database. EN: |+| Thank you for taking the time to view this module we hope you enjoyed it and learned a great deal.