# The Principle of Sync-diff-inspector

__*Sync-diff-inspector principle*__
__*sync-diff-inspector architecture*__

![right](./student-guide/diagram/SyncDiffInspectorPI_aa_001_20220701.png)
-------------------------------------------------------------
Step 1: First, verify the table schema, and keep only the indexes that exist upstream and downstream. According to the statistics of the table, the index is used as the scope, and the table is divided into chunks (chunks) in parallel by multiple threads (workers) in the order of the indexes. Some indexes contain columns that are not unique, and there may be a large number of rows with the same index value, so chunks will be divided unevenly. When selecting indexes, the Sync-Diff-Inspector selects the primary key or unique index first, followed by the index with the lowest repetition rate.
(For example, the default chunkSize of each table is 50,000 rows, and each table divides up to 10,000 chunks)

Step 2: Read the chunks at both ends through the chunk channel to the worker responsible for comparing and generating the modify SQL to prepare for parallel comparison.

Step 3: In the process of data synchronization, there is generally only a small amount of data inconsistency. Based on this assumption, when the checksum of a chunk is found to be inconsistent in the upstream and downstream checksums during the verification process, the original chunk can be divided into two sub-chunks of similar size by dichotomy, and the sub-chunk can be performed on the sub-chunk The checksum comparison further narrows the possible range of inconsistent rows. The advantage of this optimization is that the time and memory resources consumed by checksum comparison are far less than the cost of comparing data row by row. By continuously narrowing the possible range of inconsistent rows through checksum comparison, the data rows that need to be compared line-by-line can be reduced, the comparison speed is accelerated, and the memory loss is reduced. Also, since each checksum calculation is equivalent to traversing a sub-chunk after two points once, theoretically, multiple additional costs are not considered, and the cost of a binary check is equivalent to only doing checksums twice for the original chunk.

Step 4. Start a line-by-line check for chunks with inconsistent checksums upstream and downstream.

Step 5: Generate fix SQL for different rows.

Step 6. The Diff process records breakpoint information every ten seconds. When the validator abnormally exits at some point, running Sync-Diff-Inspector again will continue the verification from the most recently saved breakpoint. If the configuration file of the Sync-Diff-Inspector changes the next time it is runtime, the Sync-Diff-Inspector discards the breakpoint information and checks again.