# Wormhole

## Wormhole introduction

Wormhole is a data synchronization tool with which users can implement full and incremental data synchronization via the Web control panel.

## Wormhole features

Wormhole has the following features:

- A SQL-based real-time pipeline framework
- Support data migration between homogeneous and disparate data sources, for example, MySQL -> TiDB, MongoDB -> TiDB
- Support full and incremental data synchronization
- Provide nice and accessible Web management interface
- High availability and auto failover
- A monitor alert mechanism
- Support sharding aggregation
- Deploy Kubernetes using a single line of command

## Wormhole architecture

As shown in the following diagram, Wormhole has three components: 

- Portal
    Synchronize the upstream data, parse the data, and write the data into Kafka
- Stargate
    Obtain the data from Kafka, construct SQL statements, and write these statements into TiDB
- Ether
    Manage the task scheduler of Portal and Stargate, provide the management interface, and use Kafka for persistent storage of intermediate data

![Wormhole Architecture](media/wormhole-architecture.jpg)

## Wormhole synchronization process

![Wormhole Synchronization Process](media/wormhole-synchronization-process.jpg)

## Wormhole usage limits

For details, see [the Wormhole document](doc/wormhole-user-guide.md#wormhole-usage-limits).

## Pre-check

Wormhole provides the pre-check feature. It checks whether the configuration of MySQL, TiDB and Kafka meets the requirements and sends the check result to the user. 

For details, see [the Wormhole document](doc/wormhole-user-guide.md#pre-check).

## Deploy

### Deploy using Docker Compose

You can deploy Wormhole using Docker Compose only in a test environment.

For details, see [the Wormhole document](doc/wormhole-user-guide.md#deploy-using-docker-compose).

### Deploy Wormhole on multiple machines using Docker 

For details, see [the Wormhole document](doc/wormhole-user-guide.md#deploy-wormhole-on-multiple-machines-using-docker).

### Deploy using Binary

For details, see [the Wormhole document](doc/wormhole-user-guide.md#deploy-using-binary).
    
## Synchronize data using Wormhole

1. Enter IP + port + /ether/web/ (for example, http://192.168.0.0:2379/ether/web/) to get into the Wormhole homepage:

    ![Wormhole Homepage](media/wormhole-homepage.png)

2. Create the data source.

    1. Click `CREATE` in the homepage to enter the following page. Currently, Wormhole supports synchronizing single data source and the sharding data.

    2. Click `CREATE` to add the data source information. The fields marked in red are required fields. `Type` indicates the database type for synchronization (it supports MySQL currently); `Name` indicates the data source name:

    ![Create Data Source](media/create-data-source.png)

    You can choose `Single Data Source` to synchronize the single data source; you can choose `Multiple Data Sources` to synchronize the sharding data; you can fill in the connection configuration of multiple databases.

3. After committing in the data source creation page, data source pre-check starts. The check result is as below:

    ![Pre-check Result](media/precheck-result.png)

    If the pre-check fails, you should modify the data source configuration to pass the pre-check.

4. Choose tables for synchronization.
 
    Click the blue arrow in the upper right corner to move on. 

    - If you chose `Single Data Source` in the last step, you can see the result shown in the following picture. And then select the databases and tables to synchronize.

        ![Select Tables](media/select-tables.png)

    - If you chose sharding in the last step, you need to specify the prefix of the source database and table and specify the pattern of the target schema and table, as shown in the following picture:

        ![Configure Data Source](media/configure-data-source.png)

    The database and table list will automatically filter the databases and tables which meet the requirements. You can also select the tables you need to synchronize.

> **Note:** The sharding table schemas should be the same, including the column name, column type and index. Otherwise, the synchronization fails.

5. Create the target database (TiDB).

    Specify the target database information. 
    
    - `Pipeline Name` is a specific name of a pipeline.
    - `Type` is the target source type (only supports `TiDB` currently). 
    - `Name` is the target name. 

    ![Create Target](media/create-target.png)

6. Pipeline pre-check

    After creating a complete pipeline for data synchronization task, this pipeline will be pre-checked. The check result is as shown in the following picture:

    ![Pipeline Pre-check Result](media/pipeline-precheck-result.png)

7. Synchronize

    If the pipeline pre-check is successful, you can start the synchronization process. You can click the eye icon in the upper right corner in the interface to show the detailed synchronization information.

## Wormhole FAQ

For details, see [the Wormhole document](doc/wormhole-user-guide.md#wormhole-faq).