# TiDB Migration Runbook ## 1. TiDB Data Migration (DM) TiDB Data Migration (DM) is an integrated tool designed for MySQL-compatible sources. It supports both full data dumps and real-time incremental replication from MySQL to TiDB. (DM is available for self-managed TiDB clusters and is also integrated in TiDB Cloud for MySQL data migration tasks.) ### MySQL-compatible (MySQL, MariaDB, Aurora, Percona) * **Full Migration:** DM can perform a full dump and load of data from MySQL-compatible databases into TiDB. It dumps the source data and imports it into TiDB over SQL. This method is suitable for small-to-moderate datasets (PingCAP recommends DM for datasets < 1 TiB). For very large datasets, consider using Dumpling + Lightning for faster import (see Section 2). DM works with both TiDB Cloud (Dedicated clusters via the Cloud console) and self-managed TiDB as targets. * **Incremental Replication:** DM supports continuous incremental replication by reading the MySQL binlog and applying DML/DDL changes to TiDB in real time. It ensures TiDB stays in sync with the source after the full load, minimizing downtime during cutover. This is applicable to any MySQL/MariaDB/Aurora source. (TiDB Cloud’s DM service or a self-deployed DM cluster can be used for this purpose.) * **Full + Incremental:** DM can run an initial full migration followed by ongoing binlog replication in one unified task. The typical migration runbook is: use DM to do an initial bulk data sync, then let DM continuously replicate new changes until you are ready to switch over with near-zero downtime. This approach applies only to MySQL-compatible sources (since DM is MySQL-protocol specific) and can be used for both TiDB Cloud and on-premise TiDB deployments. ## 🚀 MySQL to TiDB Migration Using TiDB DM (Beginner-Friendly, Step-by-Step) **1. Prepare the VM** ``` sudo apt update && sudo apt install -y curl git curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh source /home/azureuser/.bashrc tiup install dm dmctl ``` **2. Generate & Customize Topology** ``` tiup dm template > topology.yaml ``` Edit topology.yaml to match your Azure VM: ``` global: user: "azureuser" ssh_port: 22 deploy_dir: "/dm-deploy" data_dir: "/dm-data" server_configs: master: log-level: info worker: log-level: info master_servers: - host: 10.0.0.6 name: master1 ssh_port: 22 port: 8261 config: log-level: info worker_servers: - host: 10.0.0.6 ssh_port: 22 port: 8262 config: log-level: info ``` **3. Deploy & Start the DM Cluster** ``` tiup dm deploy azure2tidb 8.5.1 ./topology.yaml \ -u azureuser -i ~/.ssh/azuressh.pem tiup dm list tiup dm start azure2tidb tiup dm display azure2tidb ``` **4. Configure the MySQL Data Source** Create source-mysql-01.yaml: ``` source-id: "mysql-01" from: host: "azure-mysql.mysql.database.azure.com" port: 3306 user: "stephen" password: "********" security: ssl-ca: "/home/azureuser/.ssh/DigiCertGlobalRootCA.crt.pem" ``` Register it: ``` tiup dmctl --master-addr 10.0.0.6:8261 operate-source create \ ./source-mysql-01.yaml ``` **5. Create & Start the Replication Task** Create task.yaml: ``` name: "mysql-to-tidb" task-mode: "all" meta-schema: "dm_meta" target-database: host: "tidb-cloud-private-endpoint-a0fb1653.nxdcdb1qbrz2.clusters.eastus2.prod.azure.tidb-cloud.com" port: 4000 user: "" password: "" session: foreign_key_checks: ON block-allow-list: rule-1: ignore-dbs: ["mysql","information_schema","performance_schema","sys"] ignore-tables: - db-name: "mysql" tbl-name: "user" - db-name: "mysql" tbl-name: "db" mysql-instances: - source-id: "mysql-01" block-allow-list: "rule-1" ``` Validate & launch: ``` tiup dmctl --master-addr 10.0.0.6:8261 check-task ./task.yaml tiup dmctl --master-addr 10.0.0.6:8261 start-task ./task.yaml ``` **6. Stop & Restart the Task (Optional)** ``` tiup dmctl --master-addr 10.0.0.6:8261 stop-task mysql-to-tidb ``` Restart, removing old checkpoints ``` tiup dmctl --master-addr 10.0.0.6:8261 start-task --remove-meta ./task.yaml ``` --- ## 2. Dumpling + TiDB Lightning **Dumpling** is a data export tool for MySQL/TiDB, and **TiDB Lightning** is a high-speed import tool for TiDB. Together, they provide an efficient path for full data migration, especially for large volumes. The typical process is: use Dumpling to dump the source data into SQL or CSV files, then use TiDB Lightning to quickly ingest those files into TiDB. This method is often faster than a direct SQL import because Lightning can create SST files and ingest data at lower level. It’s applicable to self-managed TiDB and to TiDB Cloud (Dedicated). (TiDB Cloud provides a native way to import Dumpling exports, CSV, or Parquet files via the web console, internally leveraging Lightning.) Below we detail the approach per source: ### MySQL-compatible (MySQL, MariaDB, Aurora, Percona) * **Full Migration:** Use **Dumpling** to export the entire MySQL-compatible database into a set of SQL dump files or CSV files. Then, apply those to TiDB using **TiDB Lightning** for a fast bulk import. Lightning can run in local (physical) import mode for best performance, especially for large datasets, but note that during import the TiDB cluster is unavailable for service. This Dumpling+Lightning workflow is ideal for large MySQL/Aurora databases. For example, Amazon Aurora MySQL can be migrated by exporting an Aurora snapshot to S3 and then importing it with TiDB Lightning. (Lightning has built-in support for Aurora snapshot exports and Parquet files as well.) This method is supported on TiDB Cloud Dedicated (you can upload dumps or connect Lightning to your cloud instance) and on self-managed TiDB. For TiDB Cloud Serverless, you would similarly dump and use the import feature (though serverless has size limits, so for large data the dedicated tier is recommended). * **Incremental Replication:** Dumpling + Lightning by themselves do **not** handle ongoing replication — they are for one-time bulk load. If you need to keep data in sync after the full import, you will incorporate a CDC tool. Typically, after using Lightning for the full load, you would use TiDB DM or another binlog replicator to capture incremental changes from the MySQL source. For instance, you might take a dump of MySQL on day 1, import it to TiDB, and then use DM to continuously apply binlog events so that TiDB catches up with any changes that happened during and after the dump. Lightning does not apply binlogs, so plan to use DM, Debezium, or AWS DMS for the incremental phase (see “Full + Incremental” below). * **Full + Incremental:** A common migration strategy is to combine Lightning for the initial load and DM (or a similar CDC tool) for change replication. First, perform the full migration using Dumpling and TiDB Lightning – this loads all historical data fast. Then, start replicating binlog changes from the source MySQL into TiDB using DM. The DM task can start from the last binlog position/time of the dump to ensure continuity. This approach minimizes downtime: you only need a brief outage to stop the source, dump any last changes, and switch over once TiDB is synced. This pattern is supported for both TiDB Cloud (Dedicated) and self-managed TiDB. (In TiDB Cloud, you might do the Lightning import via the built-in Import tool, then use the Data Migration service for binlog sync.) ### PostgreSQL * **Full Migration:** Dumpling is a MySQL-specific tool, so it cannot export PostgreSQL. Instead, use PostgreSQL’s native tools to perform a full export. For instance, use `pg_dump` to dump the PostgreSQL database data. You have two main options for format: (a) **SQL scripts** – you could dump as plain SQL and then attempt to run that on TiDB, but direct SQL may need manual adjustments due to syntax differences; or (b) **CSV** – export tables to CSV files (PostgreSQL’s `COPY TO` or `pg_dump --inserts` can help). Once you have the data in CSV or a relational format, use **TiDB Lightning** to import it into TiDB. Lightning can read standard CSV files given a schema for the table. You’ll need to create the table schemas in TiDB (possibly converting PostgreSQL types to TiDB-compatible types) before or during the Lightning import. This method can be used for TiDB Cloud as well: for example, export PG data to CSV, upload to an S3 or GCS bucket, and use TiDB Cloud’s import feature to load it. * **Incremental Replication:** Since neither Dumpling nor Lightning handle incremental replication, you will rely on external CDC tools for PostgreSQL changes. **AWS DMS** is a primary option here (it can read Postgres WAL via logical replication slots and continuously update a TiDB target). Another approach is Debezium’s PostgreSQL connector, which streams changes from Postgres into Kafka; you would then need a consumer to apply those changes to TiDB (potentially using a Kafka Connect JDBC sink or a custom application writing to TiDB). **Airbyte** and **Fivetran** also support Postgres -> MySQL-compatible replication, typically by pulling data at intervals or via WAL for low-latency updates. These third-party tools treat TiDB as a MySQL target, so they work for both TiDB Cloud and self-managed TiDB (ensure network access is configured). * **Full + Incremental:** A full + incremental migration from PostgreSQL will use a combination of the above. For example, do a one-time full export of all data from PostgreSQL (using dumps/files and Lightning to import). Once the bulk load is done, initiate an incremental replication pipeline using AWS DMS or a similar CDC tool to capture new transactions from Postgres and apply them to TiDB. You would run the CDC until TiDB has caught up to the latest changes, then switch over your application to TiDB. In TiDB Cloud, AWS DMS is recommended for this scenario (you’d create a DMS task for full + CDC). In a self-managed environment, you could use DMS or other open-source solutions. Remember to disable or carefully plan for foreign keys, triggers, or other PostgreSQL-specific features, as TiDB (being MySQL-like) will not have those; they might need to be re-implemented or dropped during migration. ### Oracle * **Full Migration:** Dumpling cannot be used on Oracle databases. To migrate Oracle data, first extract the data from Oracle using Oracle’s tools. One approach is using **Oracle’s expdp/impdp (Data Pump)** to unload tables to a flat file (commonly as CSV or a proprietary dump file). Alternatively, use a custom script or tool to pull data (for example, SQL\*Plus spool to CSV). Once you have the data in a format like CSV or a SQL INSERT script (with Oracle syntax), you should **convert it to TiDB-compatible format**. This may involve mapping Oracle data types (NUMBER, DATE, CLOB, etc.) to MySQL types and converting Oracle-specific SQL syntax (e.g., NVARCHAR2 to VARCHAR, or removing Oracle hints). After conversion, you can use **TiDB Lightning** to import the data. If using CSV, Lightning can directly ingest it after you create the table schema in TiDB. If using SQL, you might instead run the SQL through TiDB after adjustments (Lightning can import MySQL-format SQL dumps, but an Oracle SQL dump would need heavy editing). PingCAP’s docs often recommend using AWS DMS for Oracle migrations to TiDB, as DMS can handle both schema and data (with some limitations) – DMS would do the conversion to MySQL-compatible schema on the fly for you in many cases. * **Incremental Replication:** Neither Dumpling nor Lightning handle Oracle redo log replication. For ongoing changes, consider **Oracle GoldenGate** or **AWS DMS CDC**. GoldenGate is a robust (but licensed) solution that can replicate Oracle changes into a MySQL target (TiDB can appear as a MySQL target). AWS DMS, when configured for Oracle, will read Oracle’s redo logs (requires supplemental logging enabled on Oracle) and continuously apply changes to TiDB. **Debezium** also has an Oracle CDC connector (which uses XStream or LogMiner under the hood) to capture changes into Kafka – but applying those to TiDB would require a sink mechanism. **Fivetran** and other cloud ETL services can also replicate Oracle data into TiDB (they often use log-based capture similar to GoldenGate under the hood). These incremental solutions apply to TiDB Cloud and self-managed TiDB alike – the key is to treat TiDB as a MySQL-compatible target. Ensure the TiDB schema matches the Oracle source (after conversion) so that DMS or any tool can map changes correctly. * **Full + Incremental:** For an Oracle migration with minimal downtime, use a combo of full export + incremental sync. For example, do a full export of Oracle data (via files or an initial DMS full load) and import that into TiDB. Once the bulk load is done, start an **AWS DMS** task in CDC-only mode from the Oracle redo log position that corresponds to the export completion. This will apply all changes that occurred during and after the bulk load, bringing TiDB up-to-date. Finally, switch over the application to TiDB when the lag is zero. This approach (full via Lightning or DMS, plus CDC via DMS/GoldenGate) is recommended for complex migrations. TiDB Cloud Dedicated supports this via AWS DMS (with Oracle as source), and for on-prem TiDB you can similarly use DMS or GoldenGate. Note that you must handle Oracle-specific features (PL/SQL, sequences, etc.) in application or in TiDB (TiDB doesn’t support sequences; you may use auto-increment or other workarounds). ### SQL Server * **Full Migration:** Since Dumpling cannot export SQL Server, you should perform a full data export using Microsoft tools. Options include **BCP (Bulk Copy Program)** to export tables to CSV/TSV, or **SQL Server Integration Services (SSIS)** to transfer data, or even generating scripts with inserts. The simplest is often BCP out each table to a CSV file. Once you have the data in CSV format, use **TiDB Lightning** to load the data into TiDB. Create the tables in TiDB with equivalent schema (taking care to map data types: e.g. `VARCHAR(MAX)` to `TEXT`, `NUMERIC`/`DECIMAL` as-is, etc., and handle SQL Server’s BIT, DATETIMEOFFSET, etc. with compatible MySQL types). Lightning will quickly ingest the CSVs into TiDB. If the dataset is small, an alternative is to use the **Migration Assistant** in Azure Data Studio or write a custom ETL to move data via ODBC. However, for large volumes, Lightning is faster. This approach works for TiDB Cloud (you can upload the CSVs to cloud storage and import) or self-managed TiDB. * **Incremental Replication:** To capture SQL Server changes, you’ll rely on external CDC. **AWS DMS** can be used – it supports SQL Server as a source for CDC (requires enabling the SQL Server CDC feature or using transaction log backups). DMS will then continuously apply changes to the TiDB target. Another option is Debezium’s SQL Server connector, which reads the SQL Server transaction log (via CDC) and streams changes to Kafka. You would then need a sink to apply those to TiDB (e.g., a Kafka Connect JDBC sink or a custom app). **Airbyte** and **Fivetran** also offer SQL Server to MySQL replication connectors – these likely use the CDC approach under the hood to get low-latency changes. All these can keep TiDB in sync after the initial load. Make sure to resolve any differences like identity columns vs auto-increment in TiDB, since incremental tools will replicate inserts without explicitly preserving identities unless configured. * **Full + Incremental:** For minimal downtime migration, use a full + CDC approach. First, do the full data transfer (using files/Lightning or an initial AWS DMS full load). Once done, set up **AWS DMS CDC** from SQL Server to TiDB starting from the end of that full load. The DMS task can actually do both in one go (full load then switch to CDC) if configured. That will keep applying changes (inserts, updates, deletes) from SQL Server into TiDB continuously. When you’re ready, quiesce the SQL Server, let the last changes apply to TiDB, and then cut over to TiDB. This method is supported in TiDB Cloud (the DMS service can be used to migrate SQL Server RDS or on-prem to TiDB Cloud) and for self-managed TiDB. If not using DMS, you could script something similar with Debezium or other tools, but DMS is an all-in-one solution for this scenario. Remember to verify that all constraints and triggers are either migrated or reimplemented as needed, since TiDB will not automatically enforce some SQL Server-specific logic (TiDB has no native foreign key enforcement as of current versions). ### MongoDB * **Full Migration:** Dumpling/Lightning are designed for relational data, but you can still leverage Lightning for MongoDB after transforming the data. The key challenge is converting MongoDB’s JSON documents into a tabular format suitable for TiDB. One strategy is: for each MongoDB collection, decide on a target table schema in TiDB (define columns for the fields you need). Then use **mongoexport** to export the collection to JSON or CSV. If exporting to JSON, you might need an extra transformation step to convert JSON into CSV rows (for example, using a Python script or an ETL tool, especially if the JSON has nested structures). If you can directly export to CSV (flattening the document), do so. Once you have CSV files, you can use **TiDB Lightning** to load them into TiDB. Lightning will treat each file as a table import (you’ll need to configure the CSV mapping and have the table created). This full migration is largely a manual ETL process – there is no one-step tool. It applies to both TiDB Cloud and self-managed TiDB (for Cloud, you’d import the CSVs via the web console or using Lightning connecting to the cloud cluster). * **Incremental Replication:** MongoDB’s changes are recorded in the oplog (if using a replica set). To continuously replicate those to TiDB, use a tool that can read the oplog and convert operations into SQL. **AWS DMS** supports MongoDB change capture: it will read the MongoDB oplog and can insert/update rows in the MySQL target for you. However, DMS has limitations in how it maps JSON to a flat table (it may require a predefined mapping). **Debezium’s MongoDB connector** can also capture MongoDB changes (it produces change events with JSON fields). To apply these to TiDB, you likely need a custom consumer that interprets the event and does an `INSERT` or `UPDATE` on TiDB. **Airbyte** and other ETL platforms can sometimes replicate from Mongo by periodically querying or using the oplog if supported (Airbyte’s Mongo connector might be in beta for CDC). In summary, incremental replication from MongoDB is possible but requires a mapping strategy from NoSQL to SQL data. This is a specialized scenario – ensure the source MongoDB data model is compatible with a structured schema on TiDB. * **Full + Incremental:** A migration from MongoDB to TiDB with minimal downtime would involve: (1) an initial bulk export/import of all data, and (2) tailing the oplog to capture new writes during and after that bulk load. For example, you perform the full dump of all collections (as described above) and load them into TiDB (which could take some hours for large data). Meanwhile, you have a CDC process (like Debezium or DMS) reading the MongoDB oplog from the time you started the dump. That CDC process will keep track of inserts/updates/deletes that happened during the dump and after. Once the TiDB import is done, the CDC tool will apply those missed changes to TiDB. Finally, when ready to switch, you stop writes to Mongo, let the last oplog events apply to TiDB, and then switch the application to TiDB. This approach is complex due to the schema mapping, but it can be done. It’s applicable to TiDB Cloud (using DMS for example, with Mongo as source) or self-managed TiDB with custom setups. Dumpling/Lightning help with the bulk part, but the incremental part relies on third-party tooling. ### Others (CSV/TSV, Parquet, Avro, BigQuery, SAP) * **Full Migration:** Dumpling is not used for non-MySQL sources, but TiDB Lightning is very useful if you already have data in **file formats**. If your source data is a collection of **CSV or TSV files**, you can import them directly into TiDB using TiDB Lightning (or the TiDB Cloud import service). Prepare a corresponding table schema in TiDB for each CSV, then Lightning will CSV-bulk load the data. For **Parquet files** (common if exporting from data lakes or big data systems), TiDB Lightning can natively ingest Parquet as well. For example, if migrating a data warehouse or Hadoop dataset, you might export tables as Parquet and use Lightning to load them (there is a documented process for migrating Parquet from Hive to TiDB). **Avro** files are not directly supported by Lightning, so you should convert Avro to another format (you could use a Spark job or other utility to read Avro and output CSV/Parquet). For **BigQuery**, since it’s a cloud data warehouse, you’d typically export BigQuery data to Google Cloud Storage in CSV or Parquet format and then use TiDB Lightning to import that into TiDB. BigQuery’s native exports to GCS (in CSV/Avro/Parquet) combined with TiDB Cloud’s ability to import from GCS make this straightforward. For **SAP** systems: if the source is an **SAP HANA** or **SAP ASE** database, you can use their export tools or AWS DMS to extract data. A one-time full extract (either to files or via a direct pipeline) would be performed. If to files (CSV), use Lightning to load into TiDB. If using a pipeline (like DMS), it might directly load to TiDB without needing Lightning. Note that **schema conversion** might be needed (for example, SAP HANA’s types to MySQL types) – consider using tools like AWS Schema Conversion Tool (SCT) to assist with heterogeneous schema conversion if available. * **Incremental Replication:** With files or data lakes, traditional incremental replication doesn’t apply, except by repeating batch imports. For example, if you have an **ongoing feed of CSV/Parquet files** (like daily logs), you could set up a scheduled job to import new files into TiDB continuously (Lightning can be run in iterative fashion for new batches, or use SQL `LOAD DATA` for smaller files). For **BigQuery**, you might not do incremental replication; instead you might use an ETL service (like Fivetran) that periodically pulls new rows from BigQuery and upserts them into TiDB – though this is uncommon since BigQuery is usually a sink, not a source, but technically you could query for changes if there’s a timestamp. For **SAP HANA/ASE**, if you need ongoing replication, AWS DMS supports CDC from these databases (for example, reading HANA’s logs) and can continuously apply to TiDB. In general, if the original source is a database (like HANA), treat it as such and use CDC; if the source is just files, you’ll be doing periodic reloads rather than true replication. (Lightning itself is meant for batch operations, not continuous sync.) * **Full + Incremental:** Combining full and incremental for these “other” sources depends on the nature of the source. If the source is a **data warehouse (BigQuery)**, often you do a one-time full load and that’s it (any new data would perhaps continue to go into BigQuery, and you might not incrementally sync it unless there’s a plan to decommission BigQuery). If the source is an **operational system like SAP**, then a full export plus an incremental CDC via a tool like DMS is a valid strategy. For example, migrate all current SAP data via an export or initial DMS load, then let AWS DMS capture new transactions from SAP into TiDB until cutover. This is effectively the same pattern used for Oracle or SQL Server. Since Dumpling isn’t used here, TiDB Lightning handles the bulk import if working with files. Ensure the cutover plan accounts for any business logic that might not directly transfer (for SAP, consider how to handle things like application-level incremental extracts or ID ranges). This approach can be used for TiDB Cloud (especially using cloud services like DMS or Fivetran to pull from sources into TiDB Cloud) and for self-managed TiDB with on-prem connectors. Always test the end-to-end process on subsets of data before the final migration to iron out any data mapping issues. --- ## 3. Third-Party Migration Tools (AWS DMS, Airbyte, Debezium, Fivetran, etc.) Several third-party tools and services can facilitate migrating data from various sources to TiDB. These tools often support a wide range of source databases and can handle both full data loads and ongoing replication. TiDB’s MySQL compatibility means that many tools will treat TiDB as if it were a MySQL target, which makes it broadly supported. Below, we outline migration approaches using third-party tools for each source type. (All these methods apply to **TiDB Cloud (Dedicated)** and **TiDB Self-Managed** deployments. TiDB Cloud Serverless can also be a target, but ensure the service is accessible – e.g., use the public endpoint or set up networking as needed. In cloud scenarios, it’s recommended to run migration tools in the same cloud region for performance.) ### MySQL-compatible (MySQL, MariaDB, Aurora, Percona) * **Full Migration:** You can use **AWS Database Migration Service (DMS)** to do a one-time full load from a MySQL-compatible source into TiDB. DMS will connect to the MySQL source, read all tables, and bulk-copy them to TiDB (which it sees as a MySQL database). This is useful if you prefer a managed solution or if you’re migrating from cloud sources like Amazon RDS/Aurora directly to TiDB Cloud. In DMS, you would configure a migration task in **Full Load** mode (optionally with CDC). Other third-party ETL solutions like **Airbyte** can also perform full extracts: for instance, Airbyte’s MySQL source connector can dump all data and its TiDB (or MySQL) destination connector will write to TiDB. **Fivetran** (a cloud ETL service) similarly can copy all rows from MySQL into TiDB as an initial sync. The advantage of these tools is they handle schema creation and data typing automatically in many cases. Note that if using DMS, you may need to adjust some settings (e.g., exclude MySQL system databases, ensure row-format binlog if CDC is on). All these full migration methods work with TiDB Cloud Dedicated (PingCAP docs specifically highlight DMS for migrating MySQL/Aurora to TiDB Cloud) and also with on-prem TiDB (for DMS, your TiDB must be reachable from the AWS DMS replication instance). * **Incremental Replication:** Third-party tools shine for CDC replication. **AWS DMS** can be run in ongoing replication mode (either after an initial load or from a specific binlog position) to continuously apply source changes into TiDB. For MySQL sources, DMS uses the MySQL binlog to capture changes (so make sure the source’s binlog\_format=ROW and retention is set appropriately). DMS will translate each binlog event into the equivalent operation on TiDB. **Debezium**, an open-source CDC platform, provides a MySQL connector that streams binlog events out. While Debezium itself doesn’t apply changes to the target, you can pair it with a **Kafka Connect MySQL sink** or custom consumer to apply the changes to TiDB. **Airbyte** has a CDC mode for MySQL as well (it internally can use Debezium for MySQL sources to get incremental updates). Airbyte would then push those changes to TiDB via its destination connector. **Fivetran** offers log-based replication for MySQL sources too – it will read the binlog and continuously update the target (Fivetran manages the infrastructure, so it appears seamless). All these allow near-real-time replication, keeping TiDB in sync. This is particularly useful for a migration with minimal downtime. These apply equally to TiDB Cloud (just ensure network connectivity: e.g., for AWS DMS you might use VPC peering or allow DMS’s IP; for Airbyte/Fivetran, you may need to whitelist their egress IPs or use a VPN). * **Full + Incremental:** Third-party tools often support doing both phases in one framework. For example, **AWS DMS** can be configured to do an initial full copy of all data and then switch to CDC mode to replicate ongoing changes. This single DMS task can thus handle the entire migration (you’d run it until you’re ready to cut over). The PingCAP documentation explicitly recommends AWS DMS for migrating heterogeneous databases, but it works for MySQL as well if you prefer not to use DM. Similarly, **Airbyte** allows you to do a full sync initially, and if you enable CDC for the source, subsequent syncs will pull incremental changes. Airbyte can thus continuously top-up TiDB after the first load (though Airbyte’s scheduling might introduce some lag depending on how it’s set). **Debezium** could be combined with a manual full dump: e.g., you do a full migration with Lightning, then start Debezium to consume binlog from that point. This requires careful coordination of the binlog position. **Fivetran** by default will do an initial sync of all data, then automatically switch to incremental updates (for MySQL it uses binlog). The “full + incremental” approach with these tools will ensure TiDB has all data and stays current. This approach is applicable to TiDB Cloud Dedicated (with DMS or Fivetran, etc., connecting to the TiDB Cloud endpoint – many companies use it to migrate RDS MySQL to TiDB Cloud) and to self-managed TiDB instances alike. One thing to note: if you have a sharded MySQL setup merging into one TiDB, DM could handle shard merging; third-party tools would treat each source separately. In that case, you might run multiple DMS tasks or use an intermediate step to merge. ### PostgreSQL * **Full Migration:** For a Postgres source, third-party tools are often the recommended route since TiDB has no native equivalent to DM for PG. **AWS DMS** is commonly used: it can connect to PostgreSQL (on-prem or AWS RDS) and migrate data to TiDB. DMS will do an initial table-by-table copy. It also can convert the schema reasonably well, but some manual tweaking might be needed (for example, PostgreSQL `TEXT` becomes MySQL `LONGTEXT`, `BOOL` might become TINYINT(1), etc.). The TiDB documentation suggests using AWS DMS for heterogeneous migrations like PostgreSQL. If you use DMS for the full load, you simply configure the source as Postgres and target as MySQL (TiDB), select the tables, and run the task. Other options include **Airbyte**, which has a Postgres source connector that can do a full extract of data (either via SQL queries or copying) and send it to TiDB using the TiDB destination connector. Airbyte can handle large volumes but may require chunking tuning for very big tables. **Fivetran** also supports PostgreSQL as a source – it will pull all rows (either via an initial snapshot query or by reading the WAL for an initial snapshot). Fivetran then loads the data into TiDB. All these methods will create the tables in TiDB and fill them with Postgres data. Keep in mind any PostgreSQL-specific features (serial columns, arrays, JSONB) – these might get converted to MySQL equivalents or need custom handling (e.g., JSONB -> JSON in TiDB). * **Incremental Replication:** To keep TiDB updated with ongoing Postgres changes, use a CDC capable tool. **AWS DMS** supports Postgres CDC by reading the WAL (it requires a replication slot and the wal\_level set to logical). DMS will then apply changes (INSERT/UPDATE/DELETE) to TiDB continuously. This can achieve sub-second to second-level delays depending on throughput. **Debezium** has a well-known Postgres connector that does the same (reads WAL via a slot). Debezium will output changes to Kafka in a Debezium JSON format. While Debezium doesn’t apply to TiDB directly, you could consume these events and execute them on TiDB. One could use the Kafka Connect JDBC sink to write changes to TiDB (though careful with schema mapping). **Airbyte** can do CDC for Postgres as well: you’d configure the Postgres source with CDC (Airbyte uses Debezium under the hood for PG), and Airbyte will propagate changes to TiDB. The Airbyte Postgres source supports both XMIN (polling) and WAL (log-based) incremental modes – for high volume, log-based (Debezium) is preferable. **Fivetran** offers a seamless Postgres to MySQL replication; it sets up logical replication and continuously syncs data (with some small delay, typically minutes at most). These tools ensure that after the initial sync, any new transactions on Postgres are reflected in TiDB. This is crucial for a low-downtime migration where Postgres might still be in use while you migrate. * **Full + Incremental:** A combined approach with third-party tools is straightforward for Postgres. **AWS DMS** is a one-stop solution: you create one task that does Full Load + CDC. It will copy all the data, then keep streaming changes until you decide to cut over. During the CDC phase, you can monitor lag and consistency. Once you’re ready, you stop writes to Postgres, let DMS apply the remaining changes, and then finalize the cutover to TiDB. If using **Airbyte**, you might first do a full refresh sync (which copies all data), then switch the connection to incremental + CDC mode. Airbyte will then only push new changes (via Debezium) for subsequent sync runs. However, Airbyte’s incremental sync might not be continuous; it could be on a schedule (say every 5 minutes it fetches new changes). Adjust the schedule to your needs (possibly as frequent as possible for near-real-time). With **Debezium**, you’d do a manual combination: e.g., use `pg_dump` or `AWS SCT/DMS` for full load, then start Debezium from a consistent snapshot to stream changes. This requires ensuring the Debezium slot starts at the LSN just after the snapshot. **Fivetran** essentially does this automatically – initial sync then change sync. All these approaches are valid for TiDB Cloud (Dedicated). In fact, the official recommendation for Postgres to TiDB Cloud is to use AWS DMS. For a self-managed TiDB target, the same tools can be used (AWS DMS can connect to an on-prem TiDB if network allows). Be mindful of data type nuances (e.g. Postgres NUMERIC high precision could become DECIMAL in TiDB, etc.) and test the migration on a subset beforehand. ### Oracle * **Full Migration:** Third-party tools or services are commonly used to handle Oracle’s migration because they can automate some of the complex conversions. **AWS DMS** supports Oracle as a source and MySQL as a target; it will extract data from Oracle and load into TiDB, handling basic schema conversion on the fly. For example, DMS can take an Oracle NUMBER and map it to a MySQL DECIMAL, Oracle DATE to MySQL DATETIME, and so forth. However, not everything converts cleanly (Oracle’s CLOB, BLOB, etc., might need attention). DMS also doesn’t convert PL/SQL objects – those you’d have to rewrite in TiDB (or in your application). To use DMS, ensure Oracle’s supplemental logging is on (for later CDC) and that the user has the necessary permissions. **Oracle SQL Developer Migration Workbench** is another utility that can extract Oracle data and generate MySQL-compatible scripts, but it’s more involved. In modern workflows, companies often use **Fivetran** or **Hevo** etc., which have Oracle connectors: these will read all data from Oracle and load into a target database. They manage a lot under the hood (including initial extract via Oracle’s APIs). **Airbyte** has an Oracle connector (community-supported) which can do full extracts, though it may not be as battle-tested. Regardless of tool, the initial full load of Oracle data into TiDB should include schema translation. You might use AWS’s **Schema Conversion Tool (SCT)** prior to running DMS to get an accurate TiDB schema. SCT can produce a MySQL-compatible DDL from an Oracle schema, highlighting any incompatible elements. Once the TiDB schema is ready, the data load can proceed via DMS or similar. This process is applicable to TiDB Cloud (AWS DMS can migrate from Oracle RDS or on-prem into TiDB Cloud) and self-hosted TiDB. * **Incremental Replication:** Oracle to TiDB CDC can be handled by enterprise or open-source tools. **AWS DMS** can do ongoing replication from Oracle: it uses Oracle’s redo logs via either LogMiner or binary reader API (depending on Oracle version and DMS settings) to capture changes and apply them to TiDB. This requires that the Oracle source has supplemental logging enabled and a redo retention that can accommodate the migration duration. The DMS task will continually translate Oracle DML into MySQL DML for TiDB. **Oracle GoldenGate (OGG)** is a high-end solution that many use for Oracle to MySQL replication – it’s very efficient and supports a wide range of data type mappings. If the organization already uses GoldenGate, they could set up a replication channel from Oracle to TiDB (TiDB can use the MySQL ODBC driver for GoldenGate target). **Debezium’s Oracle connector** (which is now part of Red Hat Integration) can stream Oracle changes using XStream (requires Oracle Enterprise with GoldenGate license) or using LogMiner for Oracle Standard. Debezium would output events for each change; again, a consumer would need to apply them to TiDB. Some users build custom solutions where Debezium writes to Kafka and then a lightweight app reads from Kafka and issues SQL `UPSERTS/DELETES` to TiDB. **Fivetran** also supports Oracle CDC (they install an agent or use LogMiner to get changes) – and it will automatically update the target TiDB. These options vary in cost and complexity. AWS DMS is a convenient choice for TiDB Cloud migrations (since everything can be in AWS), whereas GoldenGate might be in play for on-prem enterprise environments. * **Full + Incremental:** Using third-party tools, the full+incremental workflow for Oracle often involves an initial DMS load followed by DMS CDC. For instance, you create one AWS DMS task that includes full load and change replication. DMS will copy all tables (maybe during an initial load window), then automatically switch to reading redo logs and keep TiDB in sync. You monitor the replication lag and once it’s small, you can plan the cutover. If using **GoldenGate**, you’d do something similar: e.g., use Oracle’s export for initial load or even GoldenGate’s initial load feature, then have GoldenGate continue to apply changes until caught up. GoldenGate allows very minimal downtime cutovers as it can keep up with changes in real-time. With **Debezium**, a bit more manual coordination is needed: perhaps use an initial full dump (via another tool or Debezium’s snapshot feature) and then let Debezium stream changes; you’d stop the snapshot when done and have Debezium continue from there. **Fivetran** essentially abstracts this – it does snapshot then CDC behind the scenes. In all cases, carefully verify that things like Oracle’s SEQUENCE-generated keys are handled (TiDB has auto-increment but no sequences; you may need to initialize auto-increment values to avoid collisions). If there are identity columns or triggers in Oracle, the new system might need equivalent logic. TiDB Cloud users migrating off Oracle typically rely on AWS DMS (with maybe SCT for schema) as per PingCAP’s guidance, while on-prem may use a variety of CDC products. Testing a small subset (or a single table) migration end-to-end with these tools is highly recommended to ensure the configuration is correct before doing the whole database. ### SQL Server * **Full Migration:** Third-party tools can simplify SQL Server to TiDB migration. **AWS DMS** supports SQL Server as a source and will migrate schema and data into TiDB (MySQL target). DMS can pull from an on-prem SQL Server or Azure SQL and create tables in TiDB, then bulk load data. It will handle data type conversion for common types (e.g., NVARCHAR -> VARCHAR, UNIQUEIDENTIFIER -> CHAR(36) or similar, etc.), but you should review the converted schema. If you use DMS, ensure the SQL Server user has the necessary permissions (db\_owner or datareader on all tables) and that the source is configured (if using CDC, the database must have SQL Server CDC enabled on the tables). Apart from DMS, **Airbyte** has a SQL Server source that can do a full extract (either by SELECT \* or using incremental if configured) and send to TiDB. **Fivetran** can also migrate all data from SQL Server – it will pull each table, likely in chunks, until all data is loaded. One can also use Microsoft’s tools: for example, generate BCP files and then load via MySQL client or Lightning. But the question focuses on third-party, so leveraging DMS or similar is ideal. TiDB Cloud documentation leans on AWS DMS for heterogeneous migrations including SQL Server, so that is a proven path. * **Incremental Replication:** For ongoing changes from SQL Server, **AWS DMS** in CDC mode is a primary method. It requires either enabling SQL Server’s CDC feature on the relevant databases/tables or using the transaction log (DMS can read the transaction log backups if CDC isn’t enabled, but CDC is simpler). DMS will capture changes (in a manner similar to replication) and apply them to TiDB in near real-time. Another method is using Debezium’s SQL Server CDC connector, which attaches to SQL Server’s CDC (so you must enable CDC on SQL Server for each table). Debezium will then stream every insert/update/delete into Kafka as an event. A sink can apply those to TiDB. **Airbyte** can also utilize CDC for SQL Server (it uses Debezium under the hood when available). If not, Airbyte might fall back to incremental polling (i.e., using an updated\_at timestamp to pull new rows, which is less real-time and requires such columns). **Fivetran** supports SQL Server CDC by enabling the SQL Server CDC feature and reading from the CDC tables; it will then apply the changes to TiDB. All these tools essentially allow TiDB to stay in sync with SQL Server after the full load. Ensure that if your SQL Server uses any unsupported constructs (like computed columns, or certain column types like SQL\_VARIANT), the tool either ignores or handles them, or you manually adjust those (for example, computed columns can be recomputed in TiDB via expressions if needed, but often it’s easier to treat them as regular columns). * **Full + Incremental:** Using third-party tools, a minimal-downtime SQL Server migration would look like: run an AWS DMS task that does full load + CDC. DMS will copy all data from SQL Server to TiDB, then automatically keep applying changes as they come. You monitor until ready to cut over. This is straightforward and centralizes the process in one service. If using Airbyte, you might do an initial sync then turn on CDC for ongoing sync. With **Debezium**, you would do maybe a one-time bulk load (outside Debezium) and then use Debezium to stream changes. However, Debezium’s SQL Server connector can also do an initial snapshot of the tables before it starts CDC, which serves as the full load. That snapshot though can be heavy on the SQL Server if not managed (DMS might be more controlled in that sense). **Fivetran** will handle both initial and incremental automatically (initial sync then it reads CDC). One concern is identity columns: if your SQL Server tables have `IDENTITY` PKs, TiDB will use AUTO\_INCREMENT for those. During CDC, inserts coming from SQL Server will specify the exact identity values (since the tool will include the ID), so it should preserve them; just ensure TiDB’s auto\_increment counter is set beyond the max to avoid collisions if you ever insert new without specifying IDs. Also, transactions in SQL Server are captured and applied serially in TiDB; if the load is high, ensure the DMS instance is sized appropriately to keep up. In TiDB Cloud migrations, AWS DMS is the go-to for SQL Server sources, and on self-hosted TiDB you can use DMS as well or an alternative like Striim or SymmetricDS if already in use (those are less common, so we stick to DMS/Debezium here). ### MongoDB * **Full Migration:** Third-party data integration tools can simplify the MongoDB -> TiDB migration by handling document conversion. **AWS DMS** does have support for MongoDB as a source in full copy mode: it can read all documents in each MongoDB collection and create corresponding tables in TiDB. DMS will attempt to infer a schema for the collection (since MongoDB is schemaless). It typically maps each field to a column and chooses a SQL type (e.g., strings, numbers). However, if your documents have varying structures, this can be challenging. DMS might skip documents that don’t fit the inferred schema or put JSON into a single column. It’s important to have a clear schema model for your Mongo data before using DMS. If DMS is used, it will create tables and load data accordingly. **Airbyte** has a MongoDB source connector (in beta) that can pull all documents and a MySQL destination that can write to TiDB – but similar issues of mapping apply. **Fivetran** offers a MongoDB connector which flattens nested data: it may create multiple tables for nested arrays or sub-documents (e.g., one table for the main document, additional tables for array elements). This is actually useful to preserve relationships. Fivetran’s approach is to systematically map JSON to a relational schema. This could be a good option if supported. For a custom approach, you might use an ETL pipeline (like writing a Python script using PyMongo to fetch all documents, transform them into rows, then insert into TiDB via MySQL connector). Regardless of method, the full migration of MongoDB data requires deciding how to represent JSON structures in TiDB tables. Simple collections (flat fields) are easiest. Complex ones may need multiple tables or JSON columns (TiDB does support a JSON type if you choose to store raw JSON, but then querying that is not as flexible as structured columns). * **Incremental Replication:** To replicate ongoing MongoDB changes, the primary mechanism is reading the **MongoDB oplog** (Mongo’s equivalent of a binlog, available when running as a replica set). **AWS DMS** supports this: after an initial load, DMS can tail the oplog and apply each operation to TiDB. It will translate MongoDB operations (insert/update/delete) to SQL operations. For inserts, it will insert a row; for updates, it will update the corresponding row by primary key (\_id would serve as primary key in TiDB if using DMS’s mapping); deletes remove the row. Keep in mind, if an update in Mongo adds a new field, the target schema might not have a column for it (DMS might ignore that field or you must have predefined all possible columns). **Debezium’s MongoDB connector** also reads the oplog and produces change events. Each event contains the JSON of the change (fields before and after). You would need to consume those and perform the equivalent SQL action on TiDB. **Airbyte** doesn’t yet have a mature CDC for Mongo (as of now), but they may in the future. **Fivetran** does support ongoing Mongo sync by tailing the oplog as well – they effectively do similar to DMS, continuously updating the target. All of these require the MongoDB source to be running with replication (even a single-node replica set works for an oplog). If the source is Atlas or another service, ensure it allows external connections and offers an oplog. For an on-prem Mongo, ensure the replica set is set up. This incremental replication keeps TiDB up-to-date after the bulk load. It’s applicable to TiDB Cloud or local TiDB as long as connectivity is sorted (DMS, if used, would ideally run in AWS close to TiDB Cloud for performance). * **Full + Incremental:** The combination for MongoDB migration using third-party tools typically goes: use AWS DMS to do both full load and change replication. DMS will copy all current documents to TiDB (creating the schema on the fly), then start tailing the oplog to capture new writes and updates. You would let this run until a cutover point. During this time, you should not alter the MongoDB collection schema (e.g., introduce completely new fields) without updating the target schema in TiDB, or else those new fields won’t be replicated. When cutting over, you stop application writes to Mongo, let DMS apply any last ops, then switch. If using Fivetran, it would automatically handle initial sync and then use oplog for continued sync. The timing of cutover can be orchestrated by monitoring Fivetran’s reported lag. A custom Debezium + custom loader solution would need more work: you might do an initial export via script (or Debezium’s snapshot feature) then rely on Debezium for changes. Because MongoDB to relational is complex, verify the data consistency carefully (for instance, count of documents vs count of rows, spot-check documents). In TiDB Cloud context, if using AWS DMS, you’d set up the Mongo source (maybe an EC2 instance running DMS connector for Mongo) and target TiDB Cloud. This is doable but not as turnkey as homogeneous migrations. For on-prem TiDB, you could run DMS locally or in AWS as well. In summary, third-party tools can automate much of the Mongo->TiDB replication, but the schema mapping is the hardest part and often requires custom consideration. ### Others (CSV/TSV, Parquet, Avro, BigQuery, SAP) * **Full Migration:** When dealing with sources that are not live databases but rather data files or specialty systems, third-party tools can assist in extracting and loading data, though often the simpler route is to use TiDB’s native Lightning for the load. For **flat files (CSV/TSV)** that you already have, you typically don’t need a third-party service; just use Lightning to import. However, if the question of third-party arises, you might consider something like **AWS DMS CSV support**: DMS can migrate from a folder of CSV files as a source in some configurations (usually it targets Redshift or S3, not TiDB, so this is less relevant here). If you have **Apache Parquet or Avro** files in a data lake, you could use an ETL tool like **Apache NiFi or Spark** to read those and write to TiDB. But again, an easier way is to convert them and use Lightning. **BigQuery** as a source doesn’t have a direct CDC tool (BigQuery is a destination in most pipelines). The typical method to migrate BigQuery is to export the tables to Google Cloud Storage as CSV or Avro (BigQuery has an export feature), then load into TiDB. If you prefer using a service, you could use **Fivetran** or **Airbyte** with BigQuery: Airbyte has a BigQuery source connector (it treats a query or table export as source data) and can pipe data to TiDB, but it might internally just run export queries anyway. For **SAP systems (e.g., SAP ERP or CRM data)**, often the source is actually a database (SAP HANA or Oracle or SQL Server under the hood for SAP). In those cases, you’d treat it as Oracle/SQL Server above. If it’s the SAP application layer, you might use **SAP Data Services** or **SAP SLT** replication to pull data out. AWS DMS supports SAP ASE and SAP HANA as sources, as noted, so you could use DMS for those as well (for HANA, DMS treats it similarly to other relational sources). In short, for full migration from these “other” sources: either export to files and load (using Lightning, no third-party needed beyond storage), or use a data integration service that has a connector for that source and for TiDB. * **Incremental Replication:** For file-based sources, incremental updates usually mean new files arriving. Tools like **Airbyte** or **NiFi** can be set up to watch a directory or ingest new files periodically and load them to TiDB. There isn’t a transaction log to read, so it’s more about scheduling repeated loads. For example, if an external system drops a daily CSV of new records, you could have a process that picks it up and inserts into TiDB daily. If the “other” source is an analytic DB like **BigQuery**, incremental could be emulated by running periodic queries for new data (e.g., data where date > last sync date) and merging that into TiDB. Some ETL tools can do that with scheduling. **Fivetran** might not pull incrementally from BigQuery because it’s not a typical source, but it does from sources like Salesforce, etc., which are more SaaS. If the source is **SAP HANA/ERP**, there are specialized SAP replication tools (SAP SLT can replicate data changes from SAP HANA or ECC to another DB in real-time). Alternatively, treat HANA like any DB: AWS DMS can do HANA CDC using its transaction logs. That way TiDB gets updates as they happen on HANA. Summarizing: incremental replication from non-traditional sources either doesn’t apply (for static files) or relies on treating the source’s DB (if any) similarly to the earlier sections (e.g., HANA like Oracle). * **Full + Incremental:** In scenarios like migrating from an **SAP business system** or a legacy system that isn’t directly supported, you often do a one-time data dump (full migration) and then, if needed, switch to a periodic or real-time feed of new data. For instance, an SAP system on Oracle could be migrated using Oracle techniques (full export + CDC). An old system producing monthly CSV reports could be “migrated” by loading historical CSVs, then every month new CSVs are ingested into TiDB to keep it up-to-date. The runbook for such a case would involve setting up an automated pipeline for those new files. Using cloud integration services like **AWS Glue** or **Google DataFlow** in combination with TiDB’s APIs could also be a way to process incremental file data into TiDB. The key is to clearly delineate the one-time load and the ongoing process. In TiDB Cloud, the initial load might be done via the built-in import feature for files, and ongoing updates might rely on an external service to push data in (TiDB Cloud has a public endpoint to which services can connect). For self-managed TiDB, you have full control and could even script directly on the server. In conclusion, third-party tools provide flexibility especially for sources that are not straightforward, but they often require careful setup. Always verify data consistency after using any third-party service by comparing row counts and checksums between source and TiDB, and use TiDB’s **sync-diff-inspector** or similar if available to double-check. With a solid plan per the above, you can migrate from virtually any data source to TiDB, whether in the cloud or on-premises. **Sources:** PingCAP TiDB documentation and migration guides, as well as best practices for heterogeneous database replication.