# TiDB Cluster Load Test Performance Analysis Guide DRAFT: TiDB Cluster Load Test Performance Analysis Guide (draft translation)     ## TiDB performance map [TiDB Performance Map](https://github.com/pingcap/tidb-map/blob/master/maps/performance-map.png) **TiDB performance map**         **1 Performance medical examination form**         1.1 Before compression testing   1.1.1 System medical examination form   **1.2 Under pressure test**         1.2.1 Piezometric end physical checklist   1.2.2 TiDB Body Checklist   1.2.3 TiKV Medical Checklist   1.2.4 PD Medical Examination Form   12 **2 Flowchart of data reading and writing**         **14** 2.1 TiKV Data Write Flowchart   14 **2.2 TiKV Data Reading Flowchart**         **14** **2.3 TiDB Data Write Flowchart**         **15** 2.4 TiDB Data Read Flowchart   15 **3 Operating Instructions**         **16** **3.1 CPU**         **16** 3.1.1 tidb-server   16 3.1.2 tikv-server   17 **3.2 Disks**         **17** 3.2.1 Using fio to pressure test the disks used by TiKV   17 **3.3 Network**         **18** 3.3.1 Network latency   18 3.3.2 Network bandwidth   18 **4 Performance POC FAQ**         **20** * * * ## 1 Performance Health Check --------------- The checklist provides the items that need to be checked during the performance test, as well as the meaning and reference values of the items. The purpose is to enable new hires to analyze the bottlenecks in the load test by following the list. ### 1.1 Before Test --------------- #### 1.1.1 System checklist

Category

Items

Results

Reference value / meaning

Deployment-related

Is the test tool on the same server room as the TiDB cluster -

 

If the test tool and TiDB cluster are in different server rooms, be aware of the network bandwidth of the server room and the network latency, See section 3.3.2.

 

 

 

System-related

TiKV machine fio load test

 

Reference 3.2.1
  1. Network bandwidth
  2. Network latency

 

In particular, pay attention to the network bandwidth of the machine where the test tool is located and the machine where haproxy (if any) is located.

NUMA

 

Reference 3.1.1

Business-related

Is there a self-incrementing id

 

Self-incrementing ids or indexes can lead to hotspot issues

Whether there is a self-incrementing index

 

Is there a read hotspot

 

For example, frequent scans of small tables

Are there write hotspots

 

 
### 1.2 During Test --------------- #### 1.2.1 Test tool checklist

Category

Items

result

Reference value/meaning

Test Tool

The machine where the test tool is located

  1. CPU usage
  2. CPU load
  3. Inbound/outbound bandwidth
  4. NIC send/receive packet soft interrupt hit full single core

 

Reference values.

  1. CPU usage should be below 90%.
  2. CPU load should be lower than the number of cores.
  3. Note if the bandwidth is full.

 

The command mpstat -P ALL 2 lists at all cores. The column %soft is the soft interrupt. You can also use top -H to see if one of the ksoftirqd threads is full.

haproxy machine (if present)

  1. CPU usage
  2. CPU load
  3. Inbound/outbound bandwidth

 

Reference values.

  1. CPU usage should be below 90%.
  2. CPU load should be lower than the number of cores.
  3. Check if the bandwidth is hit full.
#### 1.2.2 TiDB Checklist

Category

Items

 

Reference value/meaning

monitoring

QPS

 

Overall system QPS.

 

TIP: Edit this panel to show the third equation (Equation C) to get the theoretical QPS upper limit. The theoretical QPS should be 10% higher than the actual QPS. If the value is too high, it means the pressure is not in the database.

Duration

 

Reference value: .999 should be less than 500ms under OLTP load

Internal SQL QPS

 

QPS of internal SQL executed by TiDB, not user SQL. e.g. reading system variables when creating a connection, automatic update of statistics, GC owner updating the safe point, etc.

 

Reference value: within 20

Connection Count

 

The number of connections from the client to the tidb.

 

Reference value: within 500, with emphasis on balance.

Heap Memory Usage

 

TiDB memory usage.

 

Reference: OLTP load should be under 3 GB with TiDB local latch on and under 1 GB with latch off.

Transaction Statement Num

 

The number of statements in a single transaction.

 

Reference value: within 500

Transaction Retry Num

 

The number of transaction retries.

 

Reference: 3 and above is high, 6 and above is considered a very serious conflict.

Lock Resolve OPS

 

Frequency of lock resolving due to read-write/write-write conflicts.

 

Reference: expired and not_expired should both be within 500.

KV Backoff OPS

 

The number of waits caused by a transaction waiting for a lock or region route update.

 

Reference value: txnLockFast and txnLock should be within 500.

regionMiss should be within 5000 when importing data.

KV Cmd Duration

 

The duration between sending a request to TiKV and receiving a reply.

 

Reference value: .99 should be within 500ms except for data import and batch insert scenarios.

PD TSO RPC Duration

 

The latency of TiDB getting ts from PD. This is mainly related to the latency from TiDB to PD. If the load on the tidb-server machine is particularly high, it may also cause high tso duration.

 

Reference value: within 30ms if TiDB and PD are in the same server room.

Get Token Duration

 

The delay of requesting a token. The number of SQL executed at the same time cannot exceed the setting of the parameter token-limit in the configuration file. Increase token-limit If the value is high.

 

Reference value: within 1 ms

AutoID QPS

 

When the cached Auto-ID is exhausted, the QPS for getting more cache from  TiKV

 

Reference value: within 0.5

AutoID Duration

 

Time to fetch the Auto-ID cache. Multiple TiDBs fetching the Auto-ID cache for the same table at the same time will increase this value. For example, multiple threads concurrently writing to a table

 

Reference value: within 100 ms

configuration file

token-limit

 

The default is 1000, which means that up to 1000 requests will be processed simultaneously, and the rest will be queued. If the number of connections to a single tidb-server exceeds 1000, the parameter needs to be increased.

Reference value: token-limit should be greater than the number of connections.

txn-local-latches

 

Enabled by default with capacity 2048000, used to detect write conflicts in the same TiDB and reduce the cost of transaction retries. Can be turned off if the scenario is confirmed to be conflict-free.

 

Reference value: Default parameter

prepared-plan-cache

 

Default is off. When on, cache execution plan for reuse. Sysbench uses the prepare/execute form for all statements except the insert statements, so it is recommended to turn it on. In addition to the TiDB side, client code also needs to confirm whether to use Prepare semantics. For example, JDBC clients need to configure the setting  useServerPrepStmts

 

Reference value: enabled = true

system variable

tidb_constraint_check_in_place

 

Off by default, on only for compatibility. Check uniqueness for every row inserted. Recommended turning off for load test.

Reference value: off

 

tidb_max_chunk_size

 

Default 1024, the maximum number of rows in a chunk during the execution process

Reference values.

OLTP Applications = 32

OLAP application = 1024 or more

Resource consumption

CPU usage

 

Reference 3.1.1

CPU load

 

Should be less than the number of cores, refer to 3.1.1

common problems

AutoID conflict issues

 

TiDB will request a range of IDs from TiKV if the table has AutoIncrement columns, or has implicit IDs, and then distribute the ID in memory. When the IDs in memory are exhausted, another range will be requested. This operation “read + update” key "TID:table_id ", which could lead to write/write conflicts. Consider increasing the allocation step of AutoID.

 

In addition, specifying a value for AutoIncrement may also trigger an update of this key, which can also introduce write-write conflicts if concurrency is high. It is not recommended to specify the value of the AutoIncrement column frequently at insert time

NUMA issues

 

Reference 3.1.1
* #### 1.2.3 TiKV Medical Checklist

Category

Items

result

Reference value/meaning

System-related

Node-exporter

  1. CPU usage
  2. CPU load
  3. CPU IO wait
  4. Network bandwidth

 

Reference values.

  1. CPU usage should be below 85%. Please refer to 3.1.2
  2. CPU load should be lower than the number of cores. Please refer to 3.1.2
  3. CPU IO wait should ideally be below 5%, otherwise, it means that disk IO is busy and you need to check if the disk load is reaching a bottleneck.
  4. Both inbound/outbound bandwidths should be below the bandwidth limit.

Disk-performance

  1. IO latency
  2. IO utilization
  3. Disk load

 

 

Reference values.

  1. The io latency for write should ideally be less than 4ms, and the io latency for read can be slightly higher than for write since most reads are likely to go to cache.
  2. IO util should ideally be below 90%.
  3. Disk load should preferably be below 5.

TiKV Core Module CPU Usage

TiKV raftstore CPU (write related) latch wait duration

 

Is the wait balance between TiKVs? Unbalanced means there is a write hotspot problem.

 

Reference value: preferably less than raftstore.store-pool-size \* 85%

TiKV apply CPU (write related)

 

Is there a balance between TiKVs? Unbalanced means there is a write hotspot problem.

 

Reference value: preferably less than raftstore.apply-pool-size \* 90%

TiKV scheduler worker CPU (write-related)

 

Is there a balance between TiKVs? Unbalanced means there is a write hotspot problem.

 

Reference value: preferably less than storage.scheduler-worker-pool-size * 90%

TiKV storage readpool CPU (read related)

 
  • Is there a balance between TiKVs? Unbalanced means there is a read hotspot problem.

 

Reference value: preferably below readpool.storage.[high|normal|low]-concurrency \* 90%

TiKV Coprocessor CPU (read related)

 

Is there a balance between TiKVs? Unbalanced means there is a read hotspot problem.

 

Reference value: preferably below readpool.coprocessor.[high|normal|low]-concurrency * 90%

 

TiKV Delay (write related)

gRPC duration

 

The total time spent on the TiKV side of the request.

 

Potential network issues can be found by comparing the TiKV gRPC duration with the TiDB KV duration. For example, if the gRPC duration is very short in TiKV but the KV duration of TiDB is very long, it means that the network latency between TiDB and TiKV may be high or the bandwidth of the network card between TiDB and TiKV is full.

Scheduler duration

 

Consists of the following four parts.

  1. latch-wait duration
  2. async-snapshot duration
  3. The processing time of the command in the scheduler
  4. async-write duration

Scheduler latch wait duration

 

Reasons for high latch wait latency.

  1. Multiple transactions writing to the same key at the same time cause a conflict, which requires waiting for a lock.
  2. The number of keys written to TiKV at the same time is very high (e.g., a large concurrent import event) and exceeds the default value of latch 2048000. In such cases, adjust the latch count to alleviate the wait. The corresponding parameter is [storage] scheduler-concurrency.

 

Reference value: .99 Preferably less than 20ms.

Storage async-snapshot duration

 

Delay in raft snapshot request (need to fetch a snapshot before reading).

 

Reference value: .99 should be less than 5ms.

Storage async-write duration

 

raft write latency, consisting mainly of the following components.

  1. Propose wait duration
  2. Commit wait duration (no metric for now)
  3. Append log duration
  4. Apply wait duration
  5. Apply log duration

Propose wait duration

 

The delay between sending a request to the raftstore and when the raftstore starts processing the request. If this delay is high, the raftstore is busy or the append log is time-consuming, which means that the raftstore cannot process the request in time. (Note that a high proposed wait in version 2.1 may be due to a high number of individual TiKV leaders, e.g. more than 10k, resulting in high latency for periodic tasks. ---- Check the .99 duration of raftstore events under the raft process to confirm. In this case, you need to enable region merge to reduce the number of regions).

 

Reference value: preferably below 50-100ms

Append log duration

 

The time it takes to write raft log/ Writing raft logs to raftdb.

 

Reference value: preferably less than 50ms

Apply wait duration

 

The delay between when committed raft logs are sent to the apply thread and when the apply thread applies those logs. If this latency is high, the apply thread is busy. You need to check if this is due to high apply CPU or due to high disk load.

 

Reference value: preferably less than 100ms

Apply log duration

 

Time consumed in execution of raft log. Applying log means writing user data to kvdb.

 

Reference value: preferably less than 100ms

RocksDB kvdb write duration

 

The time it takes to write kv RocksDB.

 

Reference value: max Preferably less than 50ms

RocksDB raftdb write duration

 

Time consumed to write into raft RocksDB.

 

Reference value: Preferably less than 50ms max

TiKV Delay (Read Related)

gRPC duration

 

Same as the gRPC duration in TiKV write-related latency.

 

Reference value: None

Coprocessor wait duration

 

Coprocessor request’s wait time.

 

Reference value: less than 50ms, ideally. If you find that the wait duration is high during the stress test, e.g., higher than 50ms, it means that the Coprocessor is queuing and you need to check the CPU usage of the Coprocessor.

Storage async-snapshot duration

 

Same as async-snapshot duration in TiKV write-related latency.

Coprocessor

Scan keys

 

How many keys were scanned by a single request.

 

The metric can assist in identifying potential problems with full table sweeps of query results that should be indexed.

Table scan - details

 

The number of next, prev, seek, etc. operations caused by table scan requests per unit time.

 

Reference value: None

Table scan - perf statistics

 

Reference values:

  1. If the delete_skipped value is high, it means that a large number of tombstones exist inside RocksDB.
  2. If the many differences of next/prev are observed between key_skipped and details, it means that there are many internal versions of RocksDB.

 

Index scan - details

 

The number of next, prev, seek etc. operations by index scan requests per unit time.

 

Reference value: None

Index scan - perf statistics

 

Reference values.

  1. If the delete_skipped value is high, it means that a large number of tombstones exist inside RocksDB.
  2. If the many differences of next/prev are observed between key_skipped and details, it means that there are many internal versions of RocksDB.

Peer migration related

Snapshot: rate snapshot message

 

Speed of peer scheduling.

 

Reference value: there should not be too much scheduling during the stress test, below 10 is recommended.

.9999 snapshot size

 

region The size of the Peer.

 

Reference: should be less than 200MB. A large snapshot may cause variation in stress tests.

Scheduler

pending commands

 

The number of requests being processed. This is usually a write request.

 

Reference value: Generally less than 5k. Also check if there is a single TiKV that is significantly higher than other TiKVs, if so state.

  1. This TiKV may be a write hotspot
  2. This TiKV may be slower than other TiKV writes

RocksDB kvdb

write flow

 

Write-in traffic conditions.

 

You can see how much traffic there is between TiKV's by adding by instance to the metric expression. Ideally, all TiKV's will have similar write traffic, and if one TiKV is significantly higher than the others, there is a write hotspot issue.

 

Reference value: None

read flow

 

Read the traffic situation.

 

You can see how much traffic there is between TiKV's by adding by instance to the metric expression. Ideally, all TiKV's will have similar read traffic, if one TiKV is significantly higher than the others, there is a read hotspot issue.

keys flow

 

write flow and read flow are in byte units. On the other hand keys flow is a flow calculated based on the number of keys. Ideally, all TiKVs have the same keys flow. A TiKV with a high keys flow with the same write flow will typically obtain a higher CPU usage and is a potential bottleneck.

 

Reference value: None
TiKV Notes: 1. Most of the latency-related metrics shown on grafana is an average value of all TiKVs, and the individual TiKV values can be viewed by modifying grafana's expression and adding **by instance** at the end of the expression. This way you can spot potential problems/cases where a single TiKV is much slower than other TiKVs. 2. Some latency-related metrics also need to **be looked at for .999**, such as **append log duration**, **apply log duration**. The reason is that TiKV uses a pipeline model for processing requests, and if a jam occurs in the middle, it will cause a latency increase for the next batch tasks. 3. If you're having trouble looking at the metric above, you can refer to the TiDB performance map above. #### 1.2.4 PD Examination Form

coverage

Inspection items

result

Reference value/meaning

Cluster

Store Status

 

Check for TiKVs with abnormal states, such as disconnect.

 

Reference value: 0

Region Health

 

Since there are a small number of extra_peer / learner_peer / pending_peer in the scheduling, regions that have too many of these three or have other states need to be analyzed.

 

Reference value: value for the three should be less than 20 in the default configuration, the others are 0

Operator

Schedule operator create

 

Generate scheduling records, and if the region is balanced with a large number of generated schedules for a long period of time then you need to pay attention to the cause.

 

Reference: balance-leader > 100, balance-region > 10, hot-write < 10, lasts more than 1h Indicates a problem.

Schedule operator timeout

 

Normally there are no scheduling timeouts or a small number of timeouts, if there are persistent scheduling timeouts, you need to pay attention to the cause.

 

Reference value: below 1 OPM

Statistics -Balance

Store available

 

If the node configuration is the same, the available space should not vary too much from one TiKV node to another.

 

Reference value: Difference not exceeding 50% for space occupancy below 60%, a difference not exceeding 20%; for space occupancy between 60 and 80%; difference not exceeding 5% for space occupancy above 80%

Store leader/region score

 

The scores of TiKV nodes should be similar, and the curves should converge when the scores change, if they don't converge and there are no down or offline node operations, there is a scheduling problem

 

Reference value: None

Statistics - hotspot

Hot write region's leader/peer distribution

 

If there are write hotspots, the leader/peer distribution of the hotspots should be relatively even.

 

Reference value: None

etcd

disk WAL fsync duration

 

disk flashing too slowly can cause the PD leader to cut off service.

 

Reference value: preferably less than 100ms

TiDB

Handle requests duration

 

The main focus is tso response time, which should be kept at the millisecond range

 

Reference value: preferably less than 10ms

Heartbeat

99% Region heartbeat latency

 

If latency is increasing, indicating the PD has reached the bottleneck of processing region heartbeat

 

Reference value: preferably less than 50ms
## 2 Data reading and writing flowchart ------------------------------- ### 2.1 TiKV Data Writing Flowchart ------------------------------- ![TiKV Data Writing Flowchart](/hc/article_attachments/4410271544855/image-2.png) ### 2.2 TiKV Data Reading Flowchart ------------------------------- ![TiKV Data Reading Flowchart](/hc/article_attachments/4410281641623/image-3.png) * * * ## 3 Operational instructions ------------------------------- ``` CPU, disk, and network are the core resources inside a distributed system, and we need to check whether a particular resource of a node becomes the bottleneck of the whole system during the stress test. ``` ### 3.1 CPU ------------------------------- #### 3.1.1 tidb-server The tidb-server is compute-oriented, so theoretically the CPU can be maxed out during the stress test. However, TiDB often allocates large memory blocks, and the current Go 1.13 implementation is flawed in this area, resulting in very serious lock contention, which results in a cap limit, perhaps 80%. If this happens, you can choose to try compiling the binary package with Go 1.14 or the soon-to-be-released Go 1.15. Note that all versions prior to v4.0.1, v3.0.15, and v3.1.2 **cannot be compiled** directly with Go 1.15; the DDL module in those versions incorrectly relies on a bug fixed in Go 1.15, which causes problems with all DDL executions. Or you can avoid this problem by using multiple TiDB instances with NUMA-tied cores. If you don't tie cores in NUMA architecture, the CPU utilization rate may reach 80% and can't go up, the Page Migration of NUMA may be higher, and the corresponding Minor Page Fault will also increase, which will seriously affect the Query Duration, you may see the following situation on the monitor. ![NUMA Statistics](/hc/article_attachments/4410281642391/image-4.png) The above NUMA statistics can be obtained by several Promethean queries as follows. ```bash rate(node\_vmstat\_numa\_hit{instance="$host"}\[1m\]) rate(node\_vmstat\_numa\_local{instance="$host"}\[1m\]) rate(node\_vmstat\_numa\_miss{ instance="$host"}\[1m\]) rate(node\_vmstat\_numa\_foreign{instance="$host"}\[1m\]) rate(node\_vmstat\_numa\_pages\_migrated{instance="$host"}\[ 1m\]) ``` Generally speaking, it is normal for NUMA Page Migration to be around one or two hundred. To see the numa status of the machine via `numactl --hardware`. ```bash $ numactl --hardware _available: 2 nodes (0-1)_ _node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38_ _node 0 size: 65442 MB_ _node 0 free: 7463 MB_ _node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39_ _node 1 size: 65536 MB_ _node 1 free: 4255 MB_ _node distances:_ _node 0 1_   _0: 10 21_   _1: 21 10_ ``` Bind the two tidb-servers to separate nodes via taskset. ```bash # taskset -cp 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39 tidb-server1-pid # taskset -cp 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38 tidb-server2-pid ``` After tying cores via taskset, the performance.max-procs parameter of tidb-server needs to be set to the number of cores in a single node, e.g. it should be set to 20 in the example above. #### 3.1.2 TiKV\-server CPU load should not be higher than the number of CPU cores. TiKV-server includes not only the computation part but also operations such as reading and writing to disk. ### 3.2 Disks ------------- #### 3.2.1 Using fio to pressure test the disks used by TiKV bs=4k mainly depends on iops, because the amount of data per IO is relatively small and iops tends to bottleneck first ```bash \# fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=write -size=10G -filename=test -name="PingCAP max throughput" -iodepth=4 -runtime=60  _write:_ _IOPS=57.7k__, BW=225MiB/s (236MB/s) (10.0GiB/45465msec)_ ``` ```bash bs=64k mainly depends on band-width, the amount of data per IO is relatively large, bandwidth tends to bottleneck first \# fio -ioengine=libaio -bs=64k -direct=1 -thread -rw=write -size=10G -filename=test -name="PingCAP max throughput" -iodepth=4 -runtime=60  _write: IOPS=12.3k,_ _BW=772MiB/s_ _(809MB/s) (10.0GiB/13269msec)_ ``` bs=512k mainly depends on the band-width, the amount of data per IO is relatively large, the bandwidth tends to reach the bottleneck first. Compared with the 64k result, if the bandwidth is higher than 64k, it means 64k does not reach the bandwidth limit, and the bandwidth is larger. ```bash \# fio -ioengine=libaio -bs=512k -direct=1 -thread -rw=write -size=10G -filename=test -name="PingCAP max throughput" -iodepth=4 -runtime=60 ``` ### 3.3 Network --------------- TiDB is a distributed system, and the components are connected to each other through the network. When performing performance pressure testing, you need to pay attention not only to the performance of each component itself but also to the network between components. The main concern of the network is latency and bandwidth. #### 3.3.1 Network latency In addition, the client sends the request to the tidb-sever, the tidb-sever goes PD to ts, the tidb-server sends the request to the TiKV-server, and the TiKV-server sends it back to the tidb-server. You can see that a request needs to go through many transfers between components. So if the network latency is high, the latency of the request will increase exponentially. This is especially important when deploying across server rooms. Get the latency between the components with the ping command. ```bash \# ping 18.237.168.166 _PING 18.237.168.166 (18.237.168.166) 56(84) bytes of data._ _64 bytes from 18.237.168.166: icmp\_seq=1 ttl=63 time=0.686 ms_ _64 bytes from 18.237.168.166: icmp\_seq=2 ttl=63 time=0.321 ms_ _64 bytes from 18.237.168.166: icmp\_seq=3 ttl=63 time=0.354 ms_ _64 bytes from 18.237.168.166: icmp\_seq=4 ttl=63 time=0.331 ms_ _64 bytes from 18.237.168.166: icmp\_seq=5 ttl=63 time=0.339 ms_ ``` The **two-site, three-center deployment** scenario requires special attention. TiKV-server and PD-sever off-site should preferably not have a leader. the pressure test program (client) and tidb-server should preferably be deployed in the local server room as well. #### 3.3.2 Network bandwidth 1\. The machine where the pressure test program (client) is located The piezo program is the point of origin of all requests and the final point of reception of all results. So the network bandwidth of the machine it is on is very critical. When writing the pressure test, generally the egress traffic of the client's machine is higher, and when reading the pressure test, the ingress traffic of the client's machine will be higher. If the NIC bandwidth of the client's machine has already hit its limit, you need to add more press tests to the machine. 2\. haproxy machine If haproxy is used for load balancing during the pressure test, you need to pay special attention to whether the bandwidth of the machine where haproxy is located is full, as the haproxy node is the transit node for all traffic. 3\. tidb-server machine You also need to pay attention to the network bandwidth usage of the machine where the tidb-server is located during the compression test. Especially if there are only a few or even only one tidb-server. 4\. TiKV-server machine The network bandwidth usage of the TiKV-server machine is definitely something to look at as well. For example, are there any write hotspots or read hotspots that are causing a single TiKV-server to have much more traffic than other TiKV-servers, thus causing the network bandwidth of this TiKV-server machine to hit full capacity. Network traffic can be viewed with the nload or iptraf commands. To view network traffic using nload. ```bash \# nload _Device ens5 \[172.31.22.58\] (1/2):_ _\======================================================================_ _Incoming:_                                                                                                                          _Curr: 1.02 kBit/s_                                                                                                                          _Avg: 1.15 kBit/s_                                                                                                                          _Min: 520.00 Bit/s_                                                                                                                          _Max: 4.11 kBit/s_                                                                                                                          _Ttl: 33.62 GByte_ _Outgoing:_                                                                                                                        _Curr: 9.55 kBit/s_                                                                                                                        _Avg: 10.39 kBit/s_                                                                                                                        _Min: 4.05 kBit/s_                                                                                                                        _Max: 14.34 kBit/s_                                                                                                                        _Ttl: 584.38 GByte_ ``` * * * ## 4 Performance POC FAQ ------------------------ ### References This is a draft translation of [TiDB 集群压测性能瓶颈分析指南](https://docs.google.com/document/d/1PNtOB8-SOelLIjHsxjXuWOo8RjPB7LHkEBbOd1fX7sc/edit#heading=h.iw14kt6g182h) [tidb-map.svg](https://drive.google.com/file/d/1YR6n8DpWcw_SQSVa24S6T8WIMMOLg74K/view)