# Change Data Capture (CDC) > - **Objective:** Learn to deploy TiCDC in a TiDB cluster on AWS (with Kubernetes) > - **Prerequisites:** > - Background knowledge of TiDB components > - Background knowledge of Kubernetes and TiDB Operator > - Background knowledge of [TiCDC](https://pingcap.com/docs/stable/ticdc/ticdc-overview/) > - **Optionality:** Optional > - **Estimated time:** 30 mins ## Deploy Downstream TiDB Cluster > - **Optionality:** Optional Change Data Capture requires a second (downstream) cluster for the primary (upstream) cluster to write to. Please consult the instructions in [Create Downstram TiDB Cluster](/hc/en-us/articles/360050538833#create-downstream-tidb-cluster). ## Provision TiCDC Nodes ```{.copyable} variable "create_cdc_node_pool" { description = "whether creating node pool for cdc" default = true } variable "cluster_cdc_count" { default = 3 } variable "cluster_cdc_instance_type" { default = "c5.2xlarge" } ``` To apply the changes, you can run: ```{.bash .copyable} terraform apply ``` It might take 10 minutes or more to finish the process. ## Deploy TiCDC To deploy TiCDC, you can edit `TidbCluster` CR: ```{.bash .copyable} kubectl edit tc ${upstream} -n ${upstream_namespace} ``` In the editor, add the TiCDC specification: ```{.copyable} ticdc: baseImage: pingcap/ticdc replicas: 3 ``` Once you have save the changes, TiDB operator starts to deploy TiCDC. You can use the following command to check the status of TiCDC pods: ```{.bash .copyable} kubectl get pod -n ${upstream_namespace} ``` ```{.output} NAME READY STATUS RESTARTS AGE basic-discovery-6bb656bfd-sps8z 1/1 Running 0 4h7m basic-pd-0 1/1 Running 0 4h7m basic-pd-1 1/1 Running 0 4h7m basic-pd-2 1/1 Running 2 4h7m basic-ticdc-0 1/1 Running 0 3h15m basic-ticdc-1 1/1 Running 0 3h15m basic-ticdc-2 1/1 Running 0 3h15m basic-tidb-0 2/2 Running 0 4h6m basic-tidb-1 2/2 Running 0 4h6m basic-tikv-0 1/1 Running 0 4h7m basic-tikv-1 1/1 Running 0 4h7m basic-tikv-2 1/1 Running 0 4h7m ``` You can use the following command to check the status of TiCDC service: ```{.bash .copyable} kubectl get svc -n ${upstream_namespace} ``` ```{.copyable} NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE basic-discovery ClusterIP 10.108.13.111 10261/TCP 3h37m basic-pd ClusterIP 10.103.226.105 2379/TCP 3h37m basic-pd-peer ClusterIP None 2380/TCP 3h37m basic-ticdc-peer ClusterIP None 8301/TCP 165m basic-tidb ClusterIP 10.108.186.92 4000/TCP,10080/TCP 3h35m basic-tidb-peer ClusterIP None 10080/TCP 3h35m basic-tikv-peer ClusterIP None 20160/TCP 3h36m ``` You should take notes of the `ClusterIP` of `basic-pd` and `basic-tidb`, which will be used by TiCDC to create changefeed. ## Create Changefeed To create a change feed, you first login to one of the TiCDC pod: ```{.bash .copyable} kubectl exec -it basic-ticdc-0 -n ${upstream_namespace} sh ``` Inside the pod, you can first inspect the TiCDC cluster: ```{.bash .copyable} /cdc cli capture list --pd="http://{pd_CLUSTER-IP}:2379" ``` ```{.output} [ { "id": "391d4695-a4fb-456a-b800-5a07fb1bc9d6", "is-owner": false, "address": "basic-ticdc-0.basic-ticdc-peer.demo.svc:8301" }, { "id": "659b88a5-0656-47bf-997f-f47956ae9e1e", "is-owner": true, "address": "basic-ticdc-2.basic-ticdc-peer.demo.svc:8301" }, { "id": "c83b6c55-8293-4613-9f49-73c6142abc75", "is-owner": false, "address": "basic-ticdc-1.basic-ticdc-peer.demo.svc:8301" } ] ``` To create a changefeed, you can execute the following command: ```{.bash .copyable} /cdc cli changefeed create --sink-uri="mysql://root:@{tidb_CLUSTER-IP}:4000/" --pd="http://${pd_CLUSTER-IP}:2379" ``` ```{.output} Create changefeed successfully! ID: 145ee6dd-1220-43f2-8d0b-423ab175944f Info: {"sink-uri":"mysql://root:@10.104.118.45:4000/","opts":{},"create-time":"2020-05-30T19:34:11.4398499Z","start-ts":417036304749166593,"target-ts":0,"admin-job-type":0,"sort-engine":"memory","sort-dir":".","config":{"case-sensitive":true,"filter":{"ignore-txn-start-ts":null,"ddl-white-list":null},"mounter":{"worker-num":16},"sink":{"dispatch-rules":null},"cyclic-replication":{"enable":false,"replica-id":0,"filter-replica-ids":null,"id-buckets":0,"sync-ddl":false}}} ``` You can check the current in progress processes: ```{.bash .copyable} cdc cli processor list --pd="http://${pd_CLUSTER-IP}:2379" ``` ```{.output} [ { "changefeed-id": "145ee6dd-1220-43f2-8d0b-423ab175944f", "capture-id": "e2692613-9aaf-408e-8718-3d710fd2117e" } ] ``` ## Run Sysbench It is recommended to explore TiCDC with an empty database. ``` mysql-host=${upstream_tidb_EXTERNAL-IP} mysql-port=4000 mysql-user=root mysql-db=cdc time=1200 threads=8 report-interval=10 db-driver=mysql ``` To prepare data, you can run the following command: ```{.bash .copyable} sysbench --config-file=config oltp_point_select --tables=1 --table-size=1000 prepare ``` ## Verify Data ### Verify Data is Synced You can get the checksum of the `cdc.sbtest1` table in both the upstream and downstream TiDB clusters: ```{.bash .copyable} mysql -h ${upstream_tidb_EXTERNAL-IP} -P 4000 -u root ``` ```{.sql .copyable} admin checksum table cdc.sbtest1; ``` ```{.bash .copyable} mysql -h ${downstream_tidb_EXTERNAL-IP} -P 4000 -u root ``` ```{.sql .copyable} admin checksum table cdc.sbtest1; ``` The value of the checksum should match. You can run SQL queries for further data verifications,. ## Cleanup ### Remove Changefeed ```{.bash .copyable} kubectl exec -it basic-ticdc-0 -n ${upstream_namespace} sh ``` ```{.bash .copyable} /cdc cli changefeed remove --changefeed-id=145ee6dd-1220-43f2-8d0b-423ab175944f --pd="http://10.103.226.105:2379" ``` Check the remove is successful: ```{.bash .copyable} cdc cli processor list --pd="http://${pd_CLUSTER-IP}:2379" ``` ```{.output} [] ``` ### Remove TiCDC in TidbCluster CR You can remove TiCDC frin `TidbCluster` CR： ```{.bash .copyable} kubectl edit tc ${upstream} -n ${upstream_namespace} ``` ### Delete TiCDC StatefulSet After that, you can delete the TiCDC StatefulSet: ```{.bash .copyable} kubectl get sts -n ${upstream_namespace} ``` ```{.output} NAME READY AGE basic-pd 0/3 2d12h basic-ticdc 0/3 2d11h basic-tidb 0/2 2d12h basic-tikv 0/3 2d12h ``` ```{.bash .copyable} kubectl delete sts basic-ticdc -n ${upstream_namespace} ``` ```{.output} statefulset.apps "basic-ticdc" deleted ``` You can verify that the StatefulSet is successfully deleted: ```{.bash .copyable} kubectl get pod -n ${upstream_namespace} ``` #### Troubleshooting In case that the TiCDC pods are stuck in the `Terminating` state, you can force delete them by: ```{.bash .copyable} kubectl delete pod basic-ticdc-2 -n ${upstream_namespace} --force --grace-period=0 ```