# [KB]关于 TiCDC 下游同步到 Kafka 删除 Changefeed 后 Memory Usage 持续增加的问题

## Description

查看 TiCDC 监控 Changefeed checkpoint 在 1:35 分创建 Changefeed，2:08 分删除 Changefeed

![know-issue-ticdc-kafka-message-tool-large1.png](/images/TiCDC/know-issue-ticdc-kafka-message-tool-large1.png)

查看 TiCDC 监控 Memory usage，1:35 分开始内存增加，2:08 删除 Changefeed 后内存仍然持续增长

![know-issue-ticdc-kafka-message-tool-large2](/images/TiCDC/know-issue-ticdc-kafka-message-tool-large2.png)

## Root Cause

TiCDC BUG，AsyncClient 发生错误关闭时可能会产生死锁，导致后续内存堆积。

Github Issue：[https://github.com/pingcap/tiflow/issues/2978](https://github.com/pingcap/tiflow/issues/2978)

PR：[https://github.com/pingcap/tiflow/pull/3003](https://github.com/pingcap/tiflow/pull/3003)

## Diagnostic Steps

1. 查看 TiCDC 监控 Memory usage，找到内存占用多的进程

    ![know-issue-ticdc-kafka-message-tool-large3](/images/TiCDC/know-issue-ticdc-kafka-message-tool-large3.png)

2. 使用如下命令获取对应 TiCDC 进程的 goroutine 信息

    ```shell
    curl "http://172.xx.xx.xx:8300/debug/pprof/goroutine?debug=2" > goroutine 
    ```

3. 查看 goroutine 文件，包含类似如下错误栈，同时存在关键字 `kafka.(\*kafkaSaramaProducer).stop 和 sync.(\*Mutex).lockSlow`

```golang
goroutine 384 [semacquire, 8647 minutes]:
sync.runtime_SemacquireMutex(0xc3c6f225a4, 0x0, 0x1)
    runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc3c6f225a0)
    sync/mutex.go:138 +0xfc
sync.(*Mutex).Lock(...)
    sync/mutex.go:81
sync.(*RWMutex).Lock(0xc3c6f225a0)
    sync/rwmutex.go:98 +0x97
github.com/pingcap/ticdc/cdc/sink/producer/kafka.(*kafkaSaramaProducer).stop(0xc3c6f225a0)
    github.com/pingcap/ticdc/cdc/sink/producer/kafka/kafka.go:199 +0x3e
github.com/pingcap/ticdc/cdc/sink/producer/kafka.(*kafkaSaramaProducer).Close(0xc3c6f225a0, 0x0, 0x0)
    github.com/pingcap/ticdc/cdc/sink/producer/kafka/kafka.go:211 +0x5e
github.com/pingcap/ticdc/cdc/sink.(*mqSink).Close(0xc3c6f22a20, 0xc2f9052de0, 0x2c35b18)
    github.com/pingcap/ticdc/cdc/sink/mq.go:269 +0x33
github.com/pingcap/ticdc/cdc/sink.(*Manager).Close(...)
    github.com/pingcap/ticdc/cdc/sink/manager.go:78
github.com/pingcap/ticdc/cdc.(*oldProcessor).stop(0xc002822000, 0x31cbea0, 0xc000b1c200, 0x17, 0xc0072abcf8)
    github.com/pingcap/ticdc/cdc/processor.go:1190 +0x5a9
github.com/pingcap/ticdc/cdc.(*Capture).handleTaskEvent(0xc000da6960, 0x31cbea0, 0xc000b1c200, 0xc423b5e450, 0x1, 0x0)
    github.com/pingcap/ticdc/cdc/capture.go:283 +0xd7
github.com/pingcap/ticdc/cdc.(*Capture).Run(0xc000da6960, 0x31cbea0, 0xc000b1c200, 0x0, 0x0)
    github.com/pingcap/ticdc/cdc/capture.go:211 +0x48a
github.com/pingcap/ticdc/cdc.(*Server).run.func4(0x0, 0x0)
    github.com/pingcap/ticdc/cdc/server.go:267 +0x3f
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000c60690, 0xc000c64260)
    golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
    golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:54 +0x66
```

## Resolution

升级到 v4.0.16，v5.0.6，v5.1.4，v5.2.2，v5.3.0 及以上版本修复

## Workaround

kill 内存持续增加的进程

## Affect Version(optional)

v4.0.0,v4.0.1,v4.0.2,v4.0.3,v4.0.4,v4.0.5,v4.0.6,v4.0.7,v4.0.8,v4.0.9,v4.0.10,v4.0.11,v4.0.12,v4.0.13,v5.0.0,v5.0.1,v5.0.2,v5.0.3,v5.1.0,v4.0.14,v5.1.1,v4.0.15,v5.2.0,v5.2.1,v5.0.4,v5.1.2,v5.0.5,v5.1.3