# TiKV Status Server and Profiling Known Cases

Source note: adapted from the Feishu `Storage oncall handbook` <https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb>.

## Client TLS Mismatch Causes Status-Port Access Failure

- Phenomenon:
  - Accessing the TiKV status port fails when the cluster expects TLS but the client does not provide a valid TLS setup.
- Trigger / Preconditions:
  - TLS is enabled on the cluster.
  - The client uses plain HTTP or an incompatible TLS configuration to access the status port.
- Key logs:

```text
["Status server error: TLS handshake error"]
```

- Root cause:
  - The TiKV status server rejects the client's TLS handshake.
  - TiKV usually logs only the handshake failure and does not include enough source information to identify the caller.
- Workaround:
  - Configure the client with the correct TLS settings before accessing the status endpoint.
- Related oncall:
  - v7.5.0 TiKV logged `Status server error: TLS handshake error` `NAID-10945`

## TiKV Pprof Collection Deadlocks Raftstore

- Phenomenon:
  - Profile collection through `pprof` can deadlock and block raftstore threads.
  - Monitoring and heartbeat symptoms may follow because the TiKV instance becomes functionally stuck.
- Trigger / Preconditions:
  - `pprof` collection is requested against an affected TiKV build.
- Workaround:
  - Restart TiKV.
- Fixed version:
  - `v8.5.3` / `v8.1.3` / `v7.5.7` / `v7.1.7` / `v6.5.13`
- Related oncall:
  - v7.5.6 TiKV had no monitoring and no heartbeat `TCOC-3585`
  - `TCOC-3962`

## TiKV Profile Collection Fails on User-Built Binaries

- Phenomenon:
  - Dashboard or manual profile collection fails on self-built TiKV binaries.
- Trigger / Preconditions:
  - The binary was built without the required profiling-related features.
- Root cause:
  - Profile collection depends on build-time support such as `jemalloc` and `mem-profiling`.
- Workaround:
  - Enable `jemalloc` and `mem-profiling` when building `tikv-server`.
  - Run TiKV with:

```bash
export MALLOC_CONF="prof:true,prof_active:false"
```

- Related oncall:
  - v8.1.1 user-built TiKV failed to collect memory profile `GTOC-7462`

## TiKV Profile Interface Leaks File Descriptors

- Phenomenon:
  - After continuous profiling is enabled, the TiKV file descriptor count keeps increasing.
  - Many descriptors appear as `pipe:[xxx]`.
- Trigger / Preconditions:
  - Continuous profiling is enabled or profile collection is repeated frequently.
- Key commands:

```bash
lsof -p <tikv> | grep -c "sst"
lsof -p <tikv> | grep -v "sst" > out
```

- Workaround:
  - Disable continuous profiling.
- Fixed version:
  - `v6.1.7` / `v6.5.3` / `v7.1.0`
- Related oncall:
  - v6.1.6 TiKV file descriptor count kept increasing `TICKET-5072`
  - v6.5.0 TiKV restarted because of FD exhaustion `TICKET-1986`

## Collecting GDB Backtrace in Cloud Takes Too Long

- Phenomenon:
  - Backtrace collection in cloud or containerized environments takes far longer than expected.
  - A workflow that normally finishes in a few minutes may take tens of minutes.
- Trigger / Preconditions:
  - The pod lacks `SYS_CAP_PTRACE`, so `gdb` must run on the host instead of inside the container.
  - Host-to-artifact extraction may also require S3 relay and extra permissions.
- Root cause:
  - The delay can come from host-container process interactions, environment permissions, or platform specifics such as ARM.
- Workaround:
  - No universal workaround is documented in the source material; plan for host-level collection and artifact export when `SYS_CAP_PTRACE` is unavailable.
- Related oncall:
  - v7.5.6 collecting profile information took more than 50 minutes `TCOC-3596`