# Data Migration Overview
## Full Data Migration

* **From Amazon Aurora Snapshots / CSV / SQL Dumps**
  Use [TiDB Lightning](https://docs.pingcap.com/tidb/stable/tidb-lightning-overview) to import full datasets into TiDB.

* **From MySQL or MariaDB**
  Use [Dumpling](https://docs.pingcap.com/tidb/stable/dumpling-overview) to export data as CSV or SQL dumps for use with TiDB Lightning or direct import.

* **Small Datasets (e.g., < 1 TiB)**
  [TiDB Data Migration (DM)](https://docs.pingcap.com/tidb/stable/dm-overview) supports both full and incremental replication for smaller workloads.

---

## Incremental Replication

* **From MySQL, MariaDB, or Aurora**
  Use TiDB DM to replicate binlog data into TiDB for near-zero downtime cutover.

---

## Migration Scenarios

You can select an appropriate strategy based on your use case:

* [Migrate from Amazon Aurora to TiDB](#Migrate-from-Amazon-Aurora-to-TiDB)
* [Migrate from MySQL to TiDB](#Migrate-from-MySQL-to-TiDB)
* [Migrate and Merge MySQL Shards](#Migrate-and-Merge-MySQL-Shards)
* [Migrate from Vitess](#Migrate-from-Vitess)
* [Migrate from Files (CSV,SQL,Parquet)](#Migrate-from-Files-(CSV,SQL,Parquet))
* [Replicate Incremental Changes between TiDB Clusters](#Replicate-Incremental-Changes-between-TiDB-Clusters)
* [Continuous replication from sources using `gh-ost` or `pt-osc`](#Continuous-replication-from-sources-using-`gh-ost`-or-`pt-osc`)
* [Handling schema differences (e.g., downstream has extra columns)](#Handling-schema-differences-(e.g.,-downstream has-extra-columns))
* [Binlog filtering by event or SQL expression](#Binlog-filtering-by-event-or-SQL-expression)

# Prerequisites

Before you begin any migration, complete the following global checks and configuration steps on your MySQL-compatible source. Then, follow the environment-specific subtasks for your particular deployment.

---

### 1. Global MySQL Prerequisites

1. **Check MySQL Version**

   ```sql
   SELECT VERSION();
   ```

   Ensure you’re running a supported version (MySQL 5.6 or 5.7; limited 8.0 support). ([dev.mysql.com][1])

2. **Enable Binary Logging**

   ```sql
   SHOW VARIABLES LIKE 'log_bin';
   ```

   If `log_bin` is `OFF`, enable it in your configuration (`my.cnf` or parameter group). ([dev.mysql.com][2])

3. **Check & Set Binlog Format**

   ```sql
   SHOW VARIABLES LIKE 'binlog_format';
   ```

   Must be `ROW`. If not:

   ```sql
   SET GLOBAL binlog_format = 'ROW';
   ```

   ([dev.mysql.com][3])

4. **Check & Set Binlog Row Image**

   ```sql
   SHOW VARIABLES LIKE 'binlog_row_image';
   ```

   Must be `FULL`. If not:

   ```sql
   SET GLOBAL binlog_row_image = 'FULL';
   ```

5. **Create Replication User for DM**

   ```sql
   CREATE USER 'dm_user'@'%' IDENTIFIED BY 'your_password';
   GRANT REPLICATION SLAVE, REPLICATION CLIENT, SELECT ON *.* TO 'dm_user'@'%';
   FLUSH PRIVILEGES;
   ```

   ([dev.mysql.com][4])

---

### 2. Environment-Specific Prerequisites

#### 2.1 AWS RDS for MySQL

```bash
aws rds modify-db-parameter-group \
  --db-parameter-group-name your-param-group \
  --parameters \
    "ParameterName=binlog_format,ParameterValue=ROW,ApplyMethod=pending-reboot" \
    "ParameterName=binlog_row_image,ParameterValue=FULL,ApplyMethod=pending-reboot" \
    "ParameterName=log_bin,ParameterValue=1,ApplyMethod=pending-reboot"
# Then reboot your RDS instance
```

#### 2.2 AWS Aurora MySQL

```sql
SHOW GLOBAL VARIABLES LIKE 'aurora_binlog_replication_enabled';
```

```bash
aws rds modify-db-cluster-parameter-group \
  --db-cluster-parameter-group-name your-cluster-param-group \
  --parameters \
    "ParameterName=aurora_binlog_replication_enabled,ParameterValue=1,ApplyMethod=immediate"
```

#### 2.3 GCP Cloud SQL MySQL

```bash
gcloud sql instances patch your-instance \
  --database-flags log_bin=on,binlog_format=ROW,binlog_row_image=FULL
```

#### 2.4 Azure Database for MySQL

```bash
az mysql server configuration set \
  --resource-group your-resource-group \
  --server your-server-name \
  --name binlog_format \
  --value ROW
```

#### 2.5 Sharded MySQL

1. Repeat **Global Prereqs** on each shard.
2. Verify that **schemas are identical** across all shards.

#### 2.6 Self-Hosted MySQL

1. **Edit** your MySQL config (e.g., `/etc/my.cnf`):

   ```ini
   [mysqld]
   log-bin
   binlog_format=ROW
   binlog_row_image=FULL
   server-id=1
   ```
2. **Restart MySQL**:

   ```bash
   systemctl restart mysqld
   ```

---

# Migrate from Amazon Aurora to TiDB

# Migrate from MySQL to TiDB

# Migrate and Merge MySQL Shards

# Migrate from Vitess

# Migrate from Files (CSV,SQL,Parquet)

# Replicate Incremental Changes between TiDB Clusters

# Continuous replication from sources using `gh-ost` or `pt-osc`

# Handling schema differences (e.g., downstream has extra columns)

# Binlog filtering by event or SQL expression