# TiDB (Go) GC Tail Latency Tuning (serverless-kernel + Docker)

This document targets TiDB running in the **serverless-kernel** flavor inside containers (Docker / Kubernetes), where TiDB memory limit is controlled by env `MIN_MEMORY_LIMIT`/`MAX_MEMORY_LIMIT` and the runtime uses `runtime/debug.SetMemoryLimit` + a GOGC tuner.

Goal: reduce low-load GC jitter that hurts p99/p999 latency, mainly by avoiding **memory scale-down ("shrink")** that pushes the Go heap toward a small soft limit and triggers frequent GC/assist.

## Configuration Summary (Env)

* `MIN_MEMORY_LIMIT` / `MAX_MEMORY_LIMIT` (required): memory limit bounds (bytes, integer).
* `DISABLE_MEMORY_SHRINK` (optional): `true/false`. When `true`, keep scale-up but disable scale-down ("shrink").
* `MEMORY_SHRINK_COOLDOWN` (optional): Go duration (e.g. `5m`, `30s`). Non-positive (e.g. `0s`) disables shrink.
* `GOGC` (optional): Go GC percent baseline (e.g. `200`).

## 0. What Is (and Is Not) Related

* Go GC tuning here is about the **TiDB process** runtime (GOGC, MemoryLimit, heap growth).
* `tidb_gc_*` variables are **TiKV MVCC GC** related and usually do not fix TiDB process GC-induced tail latency.

## 1. Quick Diagnosis: Is It "shrink-driven GC"?

### 1.1 SQL: observe instance memory and MemoryLimit-triggered GC

Run every ~5s for 2-3 minutes (low-load window):

```sql
SELECT
  MEMORY_TOTAL, MEMORY_LIMIT, MEMORY_CURRENT, CURRENT_OPS, GC_LAST, GC_TOTAL
FROM performance_schema.memory_usage;
```

Interpretation:
* `CURRENT_OPS = 'shrink'` frequently: memory scaler or memory limit logic is actively shrinking and is a strong signal for jitter.
* `GC_TOTAL` increases: Go GC is being triggered under MemoryLimit logic (see `GC_LAST/GC_TOTAL`).

### 1.2 SQL: check GOGC tuner knobs (global scope)

```sql
SELECT @@GLOBAL.tidb_enable_gogc_tuner,
       @@GLOBAL.tidb_gogc_tuner_threshold,
       @@GLOBAL.tidb_gogc_tuner_min_value,
       @@GLOBAL.tidb_gogc_tuner_max_value;
```

Notes:
* Prefer `SELECT @@GLOBAL...` over `SHOW VARIABLES` when you see empty/odd values.
* In this serverless-kernel codebase, the memory limit tuner percentage is effectively fixed to **1.1** (110%) at runtime in non-test builds.

### 1.3 Container env: confirm serverless memory scaler is configured

Inside the TiDB container:

```bash
env | egrep 'MIN_MEMORY_LIMIT|MAX_MEMORY_LIMIT|NAMESPACE|POD_NAME|NODE_IP|GOGC|GOMEMLIMIT'
```

You should normally see `MIN_MEMORY_LIMIT` and `MAX_MEMORY_LIMIT` set. If `NAMESPACE/POD_NAME/NODE_IP` are present, a memory scaler loop may be running.

### 1.4 Metrics (optional but recommended)

```bash
curl -s http://127.0.0.1:10080/metrics | egrep \
  'tidb_server_gogc|go_memstats_gc_cpu_fraction|go_memstats_heap_inuse_bytes|go_memstats_next_gc_bytes|go_gc_duration_seconds_(count|sum)|process_resident_memory_bytes'
```

Things to watch:
* `go_memstats_gc_cpu_fraction` high or spiky at low load
* `go_gc_duration_seconds_count` increasing fast even when QPS is low
* `tidb_server_gogc` staying close to 100/200 most of the time

## 2. Primary Fix (Recommended): Raise and Pin MIN/MAX Memory Limit

If you are okay trading memory for more stable tail latency, the most effective action is to **avoid scaling down** by pinning the memory limit high enough.

Because this serverless-kernel flavor effectively uses:
* `runtime/debug.SetMemoryLimit( ServerMemoryLimit * 1.1 )`

you should keep `ServerMemoryLimit * 1.1` comfortably below the container memory limit (24GB here), leaving headroom for:
* non-heap memory
* Go runtime overhead
* jemalloc / libc allocations (if any)
* stacks, mmap, etc.

### 2.1 Suggested starting point (24GB container): pin to 18GiB

Set both `MIN_MEMORY_LIMIT` and `MAX_MEMORY_LIMIT` to **18GiB**:

* `MIN_MEMORY_LIMIT = 19327352832` (18 * 1024^3)
* `MAX_MEMORY_LIMIT = 19327352832`

This makes the Go soft limit target roughly `19.8GiB` (18GiB * 1.1), which is typically a safer fit under 24GB.

If you still see frequent GC and have headroom, consider 20GiB pinning, but it is riskier on a 24GB container. Prefer increasing container limit to >= 28GB first.

### 2.1.1 Optional: keep scale-up but disable scale-down ("shrink")

If you want to keep `MAX_MEMORY_LIMIT` for bursts but avoid low-load shrink jitter, set:

* `DISABLE_MEMORY_SHRINK=true`

Or set a non-positive cooldown (also disables shrink):

* `MEMORY_SHRINK_COOLDOWN=0s`

This requires a TiDB restart (env-based).

### 2.2 How to apply (examples)

*Docker run*
```bash
docker run ... \
  -e MIN_MEMORY_LIMIT=19327352832 \
  -e MAX_MEMORY_LIMIT=19327352832 \
  ...
```

*docker-compose*
```yaml
services:
  tidb:
    environment:
      MIN_MEMORY_LIMIT: "19327352832"
      MAX_MEMORY_LIMIT: "19327352832"
```

*Kubernetes (Deployment)*
```yaml
env:
  - name: MIN_MEMORY_LIMIT
    value: "19327352832"
  - name: MAX_MEMORY_LIMIT
    value: "19327352832"
```

Then restart the TiDB pod/container.

### 2.3 Rollback

Revert `MIN_MEMORY_LIMIT/MAX_MEMORY_LIMIT` to the previous values and restart.

## 3. Secondary Fix (Online): Make GOGC Tuner Less Aggressive

This is safe to try online and often provides quick tail-latency relief if GC is too frequent.

Recommended starting values:

```sql
SET GLOBAL tidb_enable_gogc_tuner = ON;

-- Pull threshold closer to memory limit, allowing higher GOGC at low load.
SET GLOBAL tidb_gogc_tuner_threshold = 0.85;

-- Allow higher GOGC.
SET GLOBAL tidb_gogc_tuner_max_value = 800;

-- Raise the floor so it doesn't fall back to 100 too often.
SET GLOBAL tidb_gogc_tuner_min_value = 200;
```

Rollback:
```sql
SET GLOBAL tidb_gogc_tuner_threshold = 0.6;
SET GLOBAL tidb_gogc_tuner_max_value = 500;
SET GLOBAL tidb_gogc_tuner_min_value = 100;
```

## 4. Optional (Restart): Set a baseline GOGC via env

If you want a conservative baseline even before the tuner adapts:

* `GOGC=200` (more conservative) or `GOGC=300` (more tail-latency oriented)

Example:
```bash
docker run ... -e GOGC=200 ...
```

Notes:
* The tuner may override dynamically, but this can help as a baseline/floor together with `tidb_gogc_tuner_min_value`.

## 5. Validation Checklist

After applying (2) and/or (3), validate over a low-load window (30-60 minutes recommended):

1. `performance_schema.memory_usage`:
   * `CURRENT_OPS` is no longer frequently `shrink`
   * `GC_TOTAL` growth rate decreases
2. `/metrics`:
   * `go_memstats_gc_cpu_fraction` decreases
   * `go_gc_duration_seconds_count` increments slower
3. Latency:
   * p99/p999 improves and jitter reduces

## 6. Safety Notes

* Pinning memory higher reduces GC frequency but can increase RSS. Watch `process_resident_memory_bytes` and container OOM events.
* If you run multiple TiDB components in the same container/pod, ensure the pinned budget still leaves enough headroom.