# Using TiCDC - Eventual Consistent Replication in Disaster Scenario

__*TiCDC eventual consistency replication*__

![left](./student-guide/diagram/TiCDCDisasterRecovery_aa_001_20220622.png)

-------------------------------------------------------------
Starting with v5.3.0, TiCDC supports backing up incremental data from upstream TiDB to S3 storage or NFS file systems of downstream clusters. When the upstream cluster is completely unusable due to a disaster, TiCDC can restore the downstream cluster to the nearest consistent state, which provides the eventual consistency replication capability for disaster recover scenario, ensures that applications can quickly switch to the downstream cluster, avoids database unavailability for a long time, and improves business continuity.

Currently, TiCDC supports the replication of incremental data from TiDB clusters to TiDB or MySQL-compatible database systems (including Aurora, MySQL, and MariaDB). In the event of an upstream disaster, if TiCDC is operating normally and the upstream TiDB cluster does not experience a significant increase in data replication latency, the downstream cluster can recover the cluster within 5 minutes and lose up to 10 seconds of data before the problem occurred, i.e., RTO <= 5 mins, P95 RPO <= 10s.

When the upstream TiDB cluster has the following conditions, it will cause TiDc latency to rise, which in turn affects the RPO:

TPS has risen sharply in a short period of time
There is a big transaction or a long transaction upstream
Reload or upgrade the upstream TiKV cluster or TiCDC cluster
Execute a DDL statement that takes a long time, for example: add index
Using an overly aggressive PD scheduling strategy, resulting in frequent region leader migrations or region merge/splits
Prerequisites of use

Prepare a highly available S3 storage or NFS system to store TiCDC's real-time incremental data backup files that can be accessed in the event of an upstream disaster.
TiCDC enables this feature for changefeeds that require eventual consistency in disaster scenario. The following configuration is added to the changefeed configuration file:

[consistent]
# Consistency Level, Options Are:
# None: the Default, Non-disaster Scenario, Which Guarantees Eventual Consistency Only If the Task Specifies Finished-ts.
# Eventual: Use Redo Logs to Provide Eventual Consistency in the Event of an Upstream Disaster.
level = "eventual"

# The Size of a Single Redo Log File in MiB. the Default Value is 64. the Recommended Value is No More Than 128.
max-log-size = 64

# The Interval Between Refreshing or Uploading Redo Logs to S3, in Milliseconds, Default 1000, Recommended Range 500-2000.
flush-interval = 1000

# Store Redo Logs in the Form of Nfs (NFS Directory), S3 (upload to S3)
storage = "s3://logbucket/test-changefeed? endpoint= http://$S3_ENDPOINT/"

Disaster recover

When a disaster occurs upstream, you need to manually recover downstream using the cdc redo command. The recover process is as follows:

Ensure that the TiCDC process has abort to prevent the upstream service from resuming during data recover, and that TiCDC restarts synchronizing data.
Use cdc binary for data recover. The specific command is as follows:
cdc redo apply —tmp-dir=" /tmp/cdc/redo/apply"\
    —storage=" s3://logbucket/test-changefeed? endpoint= http://10.0.10.25:24927/ "\
    —sink-uri= "mysql: //normal: 123456 @10 .0.10. 55:3306/"
In the above command:

tmp-dir: Specify the temporary directory to download the TiCDC incremental data backup file.
storage: Specify the address to store the TiCDC incremental data backup file, which is an S3 or NFS directory.
sink-uri: The downstream address to which the data is recover. Scheme only supports mysql.