---
name: tikv-status-server
description: |
  Reference handbook for TiKV status-server and profiling issues.
  Use when investigating status-port access failures, TLS handshake errors,
  pprof deadlocks, profile collection failures, file-descriptor leaks, or
  slow backtrace collection through TiKV status interfaces.
---

# TiKV Status Server Handbook

Known TiKV status-port, pprof, and profile-collection patterns for on-call diagnosis.

Source note: adapted from the Feishu `Storage oncall handbook` <https://pingcap.feishu.cn/wiki/RNzIwCexFiA1uXkRCvzcsIocnVb>.

## When To Use This Skill

Use this skill when:

- TiKV status port cannot be accessed correctly
- dashboard or manual `pprof` collection fails
- profile collection itself causes stalls or deadlocks
- continuous profiling leaks file descriptors

## First Checks

- Confirm whether the failure is access-related, collection-related, or caused by profiling itself.
- Check whether TLS is enabled on the cluster and whether the client matches it.
- Record whether the issue comes from Dashboard, manual `curl`, `pprof`, or `gdb`.
- For suspected profiling side effects, collect the first abnormal log before retrying.

## Quick Commands

TiKV status server provides HTTP endpoints for runtime state and profiling. Common examples:

```bash
curl -o tikv_cpu.proto -H 'Content-Type: application/protobuf' 'http://127.0.0.1:20182/debug/pprof/profile?seconds=30'
go tool pprof --http=0.0.0.0:1234 tikv_cpu.proto
```

## Known Cases

### Client TLS mismatch causes status-port access failure

- Detailed reference: [`references/status-server-known-cases.md`](references/status-server-known-cases.md)
- Phenomenon: when the cluster is configured with TLS, clients that access the TiKV status port without valid TLS trigger:

```text
["Status server error: TLS handshake error"]
```

- Note: TiKV usually logs only this error line and does not include enough source information for the offending client.
- Workaround: configure proper TLS on the client side.
- Related oncall:
  - v7.5.0 TiKV logged `Status server error: TLS handshake error` `NAID-10945`

### TiKV pprof collection deadlocks raftstore ([tikv#18474](https://github.com/tikv/tikv/issues/18474))

- Detailed reference: [`references/status-server-known-cases.md`](references/status-server-known-cases.md)
- Phenomenon: pprof collection can deadlock and block raftstore threads.
- Workaround: restart TiKV.
- Fixed version: `v8.5.3` / `v8.1.3` / `v7.5.7` / `v7.1.7` / `v6.5.13`
- Related oncall:
  - v7.5.6 TiKV had no monitoring and no heartbeat `TCOC-3585`
  - `TCOC-3962`

### TiKV profile collection fails on user-built binaries

- Detailed reference: [`references/status-server-known-cases.md`](references/status-server-known-cases.md)
- Phenomenon: if a user-built TiKV binary misses the necessary build options, dashboard cannot fetch profile data correctly.
- Workaround:
  - enable `jemalloc` and `mem-profiling` when building `tikv-server`
  - set:

    ```bash
    export MALLOC_CONF="prof:true,prof_active:false"
    ```

- Related oncall:
  - v8.1.1 user-built TiKV failed to collect memory profile `GTOC-7462`

### TiKV profile interface leaks file descriptors ([tikv#14224](https://github.com/tikv/tikv/issues/14224))

- Detailed reference: [`references/status-server-known-cases.md`](references/status-server-known-cases.md)
- Phenomenon: after enabling continuous profiling, TiKV file descriptor count keeps increasing and many descriptors appear as `pipe:[xxx]`.
- Diagnostic commands:

```bash
lsof -p <tikv> | grep -c "sst"
lsof -p <tikv> | grep -v "sst" > out
```

- Workaround: disable continuous profiling.
- Fixed version: `v6.1.7` / `v6.5.3` / `v7.1.0`
- Related oncall:
  - v6.1.6 TiKV file descriptor count kept increasing `TICKET-5072`
  - v6.5.0 TiKV restarted because of FD exhaustion `TICKET-1986`

### Collecting GDB backtrace in cloud takes too long

- Detailed reference: [`references/status-server-known-cases.md`](references/status-server-known-cases.md)
- Phenomenon: container environments without `SYS_CAP_PTRACE` require collecting backtraces on the host, which can take far longer than expected.
- Notes:
  - `gdb` runtime may exceed 20 minutes
  - extracting files from the host may require S3 relay and extra permissions
  - the delay may relate to ARM or host-container process interactions
- Related oncall:
  - v7.5.6 collecting profile information took more than 50 minutes `TCOC-3596`

## Related Skills

- For panic and crash-first incidents, read `skills/diagnosis/tikv-panic/SKILL.md`.
- For runtime and performance investigation, read `skills/diagnosis/tikv-performance/SKILL.md`.