# Design: Uniform Per-Tenant CPU Cap for Essential

> **Status**: Canonical design draft for task #1. Owner: @Codex.
> **Clarified requirements** (from @debin-jia, 2026-05-26):
> 1. Use only the existing CPU throttle **token bucket capability**; burst, fair allocation, and dynamic adjustment are all disabled by default.
> 2. Do **not** derive each tenant's CPU resource limit from PD service_limit.
> 3. Treat all tenants as **equal**: use one configured threshold (for example, 40%) as the uniform hard cap, and allow free competition within that cap.
> 4. The global bucket limits node-level total CPU usage, while the per-tenant hard cap prevents any single tenant from overloading the node.

## 1. Background

The `release-nextgen-202603` branch serves as the shared codebase for both **Essential** and **Premium** deployment modes. The existing keyspace-level CPU throttle provides a rich set of capabilities — global token bucket, per-keyspace token bucket, burst, fair allocation, dynamic refill rate adjustment, PD service_limit integration — designed primarily for Premium where the number of tenants is small and per-tenant configuration is feasible.

**Essential is fundamentally different:**

- **Many more tenants** — dozens to hundreds per TiKV node
- **All tenants are equal** — no per-tenant weighting, no VIP tenants, no fair-share quotas
- **No per-tenant configuration** — configuring `keyspace_cpu_quotas` or relying on PD service_limit for every tenant is impractical
- **Highest priority**: prevent any single tenant from bringing down the TiKV node

In the current implementation (`cpu_throttle.rs`), keyspaces without configured quotas take the `skip_throttle = true` path — **completely bypassing tenant-level CPU control**. Also, the existing global bucket is only a hard node-level limit when `enforce_global_limit = true`; otherwise it is best-effort tracking after keyspace allocation. Essential needs to close both gaps.

## 2. Problem Statement

**Goal for this v1 design**: Limit any single tenant's **coprocessor/read-path CPU** usage to a configured maximum (e.g., 40% of total CPU), using only the token bucket mechanism from the existing CPU throttle.

**Constraints**:
- All tenants are equal — same hard cap for everyone
- No per-tenant PD service_limit dependency
- Burst, fair allocation, dynamic adjustment all **off by default** for Essential
- Global bucket must be a hard node-level cap (`enforce_global_limit = true`)

### 2.1 Scope Clarification: Not Read + Write Aggregate Yet

`tenant_max_cpu_ratio` in this document is a **per-keyspace coprocessor/read CPU hard cap on the current TiKV node**. It is not yet a hard cap for the tenant's combined read + write CPU usage.

The reason is the implementation boundary: this v1 reuses the existing `ReadLimiter` / `CpuThrottleManager` path, and the node-level bucket is still configured by `max-read-cpu-ratio`. The only request path that currently has precise token accounting and actual CPU measurement is the coprocessor/read path.

Read-path CPU accounting works as follows:

1. Before execution, `try_allocate(keyspace_id, estimated_cpu_us)` deducts estimated CPU tokens from both the per-tenant bucket and the global bucket.
2. During execution, `CpuObserveFuture` measures the request future's actual thread CPU time across polls. If `enable_runtime_token_management` is enabled, `CpuTokenCheckFuture` periodically tops up from the same tenant/global buckets.
3. After execution, the final actual CPU usage is recorded on `CpuTokenHandle`, metrics are emitted per `keyspace_id`, and the resource event is published as `CpuType::UnifiedReadPool`.

If the required product semantics are **tenant read + write total CPU <= tenant_max_cpu_ratio**, this v1 scope is insufficient. The follow-up design must make the tenant bucket a shared per-keyspace CPU ledger and charge both request paths into it:

- Read/coprocessor requests continue to charge the existing read path.
- Write requests need to enter the same bucket through a `WriteLimiter` in `Scheduler::process_write`.
- Heavy/batch write commands can be measured with `CpuObserveFuture`; light write commands likely need sampling plus adaptive estimation to avoid excessive overhead.
- RaftWorker, RaftIo, and Apply CPU are not covered by wrapping `process_write`; exact end-to-end write CPU attribution needs a separate design because those workers batch and interleave commands from many keyspaces.

Therefore, if we want the config name `tenant-max-cpu-ratio` to mean read + write aggregate, the implementation must be expanded before enabling it as an Essential hard cap. Otherwise the narrower v1 behavior should be treated as a read/coprocessor cap, even if the config name stays generic for compatibility.

## 3. Design: Two-Bucket Simplicity

### 3.1 Architecture

The Essential CPU control model is deliberately minimal:

```
                    ┌─────────────────────────────┐
                    │     Global Token Bucket      │
                    │  capacity = cores * 0.6 *    │
                    │            refill_interval   │
                    │  (max-read-cpu-ratio = 0.6)  │
                    └─────────────┬───────────────┘
                                  │
                                  │ try_allocate
                                  ▼
                    ┌─────────────────────────────┐
                    │  Per-Tenant Token Bucket     │
                    │  capacity = cores * 0.4 *    │
                    │            refill_interval   │
                    │  (tenant-max-cpu-ratio=0.4)  │
                    │                              │
                    │  Same capacity for ALL       │
                    │  tenants. Auto-created on    │
                    │  first request.              │
                    └─────────────────────────────┘
```

That's it. Two mandatory token buckets:
1. **Global bucket** — prevents total coprocessor CPU from exceeding `max_read_cpu_ratio` of node CPU (node protection)
2. **Per-tenant bucket** — prevents any single keyspace's coprocessor/read CPU from exceeding `tenant_max_cpu_ratio` of node CPU (noisy-neighbor protection)

`tenant_max_cpu_ratio` is calculated against node logical CPU cores, not against
`max_read_cpu_ratio` again. For example, on a 16-core node with `max-read-cpu-ratio = 0.6`
and `tenant-max-cpu-ratio = 0.4`, the global read cap is 9.6 cores and a single tenant cap
is 6.4 cores. This avoids double discounting the tenant cap; the global bucket still bounds
aggregate coprocessor/read usage.

All tenants get the **same bucket capacity**. Within the cap, tenants compete freely via the token bucket (first-come, first-served within refill windows). No fair allocation weighting, no burst, no dynamic refill adjustment, and no PD service_limit-derived quota.

### 3.2 Allocation Flow

```
Request → try_allocate(keyspace_id, estimated_cpu_us)
  │
  ├─ 1. Refill global bucket and this tenant's bucket
  │
  ├─ 2. If tenant cap mode is enabled, ensure this tenant has a uniform-cap bucket
  │
  ├─ 3. Allocate from per-tenant bucket
  │      └─ fail → return TenantCpuExhausted / KeyspaceCpuExhaustedNoBurst
  │
  ├─ 4. Allocate from global bucket (required in Essential)
  │      └─ fail → rollback tenant allocation and return GlobalCpuExhausted
  │
  └─ 5. Return CpuTokenHandle (tracks both global + keyspace allocations)
```

Key points:
- **Both buckets must pass** — the implementation may allocate tenant first and global second with rollback to reuse the current path, but Essential semantics require both tenant and global tokens.
- **Auto-created tenant buckets** — first request from a keyspace triggers bucket creation; no manual per-tenant config needed
- **No burst** — tenant bucket exhausted = reject. No fallback to global-only allocation.
- **No fair allocation** — all tenants equal, same bucket size
- **No dynamic adjustment** — refill rate is fixed at the configured cap
- **Runtime token top-up is still allowed** — it only deducts additional tokens from the same buckets during execution; it is not burst, fair allocation, or dynamic refill adjustment.

### 3.3 The Critical Change: Eliminating `skip_throttle`

Current code at `cpu_throttle.rs:1892`:
```rust
if !self.keyspace_quotas.contains_key(&keyspace_id) {
    return Ok((CpuTokenHandle { skip_throttle: true, ... }, false));
    //                                   ^^^^^^^^^^^^^^^^
    //                          THIS IS THE PROBLEM FOR ESSENTIAL
}
```

New behavior when `tenant_max_cpu_ratio > 0`:
```rust
if !self.keyspace_quotas.contains_key(&keyspace_id) {
    // Auto-create bucket with uniform cap
    let cap_quota_us = self.compute_tenant_cap_quota_us();
    self.ensure_tenant_cap_bucket(keyspace_id, cap_quota_us);
    // Fall through to normal allocation. Both keyspace and global buckets
    // must succeed because enforce_global_limit is required in Essential.
}
```

No more `skip_throttle`. Every keyspace gets a bucket. Every keyspace is bounded.

For Essential, this should not depend on `keyspace_cpu_quotas`: the uniform tenant cap is the source of truth. The deployment profile should keep `keyspace-cpu-quotas = ""` and `enable-pd-service-limit = false`; config validation should reject conflicting settings while `tenant-max-cpu-ratio > 0`.

### 3.4 What We DON'T Use (Essential Defaults)

| Feature | Essential default | Why off |
|---------|------------------|---------|
| **Burst** | `off` | Hard cap must not be bypassed. Tenants are equal, no reason to let one tenant exceed cap. |
| **Fair allocation** | `off` | All tenants equal, no weighting needed. Free competition within the cap is the desired behavior. |
| **Dynamic adjustment** | `off` | Fixed cap is simpler and more predictable. No need to adjust refill rates based on watermarks. |
| **PD service_limit** | `off` | Essential tenants don't have per-tenant service limits. The uniform cap replaces PD-derived quotas. |
| **keyspace_cpu_quotas config** | Empty / rejected | No per-tenant quota config. The single `tenant_max_cpu_ratio` applies to all. |
| **low-priority DAG global bucket** | `off` | Not needed for the Essential tenant cap. If enabled later, it must be an extra bucket before the same cap-aware tenant/global allocation path. |

`enable_runtime_token_management` can remain enabled because it charges the same global and tenant buckets as actual CPU is consumed. It does not change the tenant cap or create fairness weights. With the current implementation, enabling it caps the initial per-request estimate to `min(global_bucket, tenant_bucket) / 10`, then runtime top-up deducts more tokens from the same buckets.

### 3.5 Premium Compatibility

Premium continues to use the full feature set (burst, fair allocation, dynamic adjustment, PD service_limit). The Essential path is activated by setting `tenant_max_cpu_ratio > 0`. When `tenant_max_cpu_ratio = 0` (default), behavior is unchanged — `skip_throttle` still applies for unknown keyspaces.

This means the two modes coexist in the same codebase via configuration:

| Mode | `tenant-max-cpu-ratio` | Burst | Fair | Dynamic | PD service_limit |
|------|----------------------|-------|------|---------|-----------------|
| **Essential** | `0.4` | off | off | off | off |
| **Premium** | `0.0` (default) | on | on | on | on |

Essential must also set `enforce-global-limit = true`; otherwise the existing global bucket is not a strict node-level cap.

## 4. Configuration

### 4.1 Essential Profile TOML

```toml
[resource-control]
enabled = true
max-read-cpu-ratio = 0.6          # Node-level: coprocessor uses at most 60% CPU

[resource-control.cpu-throttle]
enabled = true
enforce-global-limit = true           # Global bucket is a hard node-level cap
tenant-max-cpu-ratio = 0.4        # Per-tenant: each keyspace capped at 40% CPU
tenant-cap-idle-ttl = "10m"       # Remove auto-created cap buckets after this idle time
keyspace-cpu-quotas = ""          # Essential does not configure per-tenant quotas

# Essential defaults (all off)
enable-burst = false
enable-fair-allocation = false
enable-dynamic-adjustment = false
enable-pd-service-limit = false
low-priority-dag-global-enabled = false
enable-runtime-token-management = true
```

### 4.2 New Config Field

**`CpuThrottleConfig` (cpu_config.rs):**
```rust
/// Maximum CPU ratio any single tenant can consume.
/// 0.0 = disabled (Premium mode: skip_throttle for unknown keyspaces).
/// > 0.0 = Essential mode: auto-create per-tenant buckets with this cap,
///         burst/fair/dynamic should be off.
/// Example: 0.4 = each tenant capped at 40% of total node CPU.
pub tenant_max_cpu_ratio: f64,  // default: 0.0
pub tenant_cap_idle_ttl_ms: u64, // default: 10 minutes, hot-updatable
```

**`CpuThrottleSettings` (config.rs):**
```rust
#[online_config]
pub tenant_max_cpu_ratio: f64,  // default: 0.0, hot-updatable
pub tenant_cap_idle_ttl: ReadableDuration, // default: 10min, hot-updatable
```

### 4.3 Validation

- `tenant_max_cpu_ratio` must be in `[0.0, 1.0]`
- `tenant_cap_idle_ttl` must be at least `1ms`
- When `tenant_max_cpu_ratio > 0`, Essential profile validation should require:
  - `enforce_global_limit = true`
  - `enable_burst = false`
  - `enable_fair_allocation = false`
  - `enable_dynamic_adjustment = false`
  - `enable_pd_service_limit = false`
  - `low_priority_dag_global_enabled = false` unless its path is explicitly made cap-aware
  - `keyspace_cpu_quotas` is empty
- Startup validation and online config hot update must use the same cross-field checks. If an online change tries to enable `tenant_max_cpu_ratio > 0` with a conflicting option, reject the entire config update.
- `tenant_max_cpu_ratio` can be larger than `max_read_cpu_ratio` — the global bucket is still the effective bottleneck

## 5. Implementation Plan

### 5.1 New Config Fields

Add `tenant_max_cpu_ratio: f64` (default `0.0`) and `tenant_cap_idle_ttl` (default `10m`) to both `CpuThrottleConfig` and `CpuThrottleSettings`.

### 5.2 `CpuThrottleManager` Changes (cpu_throttle.rs)

**New fields:**
```rust
tenant_max_cpu_ratio_bits: AtomicU64,  // f64 bits for lock-free reads
tenant_cap_keyspaces: DashSet<u32>,    // keyspaces auto-created by uniform cap mode
tenant_cap_idle_ttl_ms: AtomicU64,     // hot-updatable idle eviction TTL
```

**New method — compute cap quota:**
```rust
fn compute_tenant_cap_quota_us(&self) -> u64 {
    let ratio = f64::from_bits(
        self.tenant_max_cpu_ratio_bits.load(Ordering::Acquire)
    );
    if ratio <= 0.0 {
        return u64::MAX;  // disabled
    }
    let cpu_cores = SysQuota::cpu_cores_quota().max(1.0);
    let cap_millicores = (cpu_cores * ratio * 1000.0) as u64;
    CpuThrottleConfig::millicores_to_us_per_interval(
        cap_millicores,
        self.refill_interval_ms.max(1)
    )
}
```

**Modified — `try_allocate_with_burst()` (current `skip_no_quota` path):**

Replace the `skip_throttle` block:
```rust
if !self.keyspace_quotas.contains_key(&keyspace_id) {
    let cap_quota_us = self.compute_tenant_cap_quota_us();
    if cap_quota_us != u64::MAX {
        // Essential mode: auto-create bucket with uniform cap
        self.ensure_tenant_cap_bucket(keyspace_id, cap_quota_us);
        // Fall through to normal allocation path below
        // Both keyspace and global allocations must succeed because
        // enforce_global_limit is required in Essential.
    } else {
        // Premium mode: legacy skip_throttle
        return Ok((CpuTokenHandle { skip_throttle: true, ... }, false));
    }
}
```

`ensure_tenant_cap_bucket()` should create/update `keyspace_buckets`, insert the keyspace into `tenant_cap_keyspaces`, and set metrics. It should not write a per-tenant quota derived from PD service_limit.

When `tenant_max_cpu_ratio > 0`, configured or PD-derived keyspace quota updates should not change the effective Essential cap. The simplest first implementation is to reject the conflicting config combination, keep PD service_limit disabled, and treat all buckets created in this mode as uniform-cap buckets.

`try_allocate_low_priority_dag_with_burst()` needs an explicit guard in cap mode. Because the existing code uses a global-only fallback when no keyspace quota exists, either keep `low_priority_dag_global_enabled = false` in Essential validation, or make that fallback call the same cap-aware tenant/global allocation path after the low-priority DAG bucket succeeds.

The runtime `allocate_more` path must also be verified in cap mode. With burst and dynamic adjustment disabled, runtime top-up should only try the same tenant bucket and global bucket; it must not enter burst or global-only fallback.

Auto-created tenant cap buckets should be evicted after `tenant_cap_idle_ttl` with no request and no in-flight tokens, so long-running Essential nodes do not accumulate buckets for deleted or inactive keyspaces. `tenant_cap_idle_ttl` is hot-updatable; lowering it should make the next cleanup scan use the new TTL.

**Hot-update method:**
```rust
pub fn update_tenant_max_cpu_ratio(&self, ratio: f64) {
    self.tenant_max_cpu_ratio_bits.store(
        ratio.to_bits(), Ordering::Release
    );
    // Recompute auto-created tenant-cap bucket capacities.
    if ratio > 0.0 {
        let cap_quota_us = self.compute_tenant_cap_quota_us();
        let keyspaces: Vec<u32> = self.tenant_cap_keyspaces
            .iter()
            .map(|entry| *entry.key())
            .collect();
        for keyspace_id in keyspaces {
            self.ensure_tenant_cap_bucket(keyspace_id, cap_quota_us);
        }
    } else {
        // Disable Essential cap mode. Remove only auto-created cap buckets;
        // configured Premium buckets, if any, are managed by keyspace_cpu_quotas.
        let keyspaces: Vec<u32> = self.tenant_cap_keyspaces
            .iter()
            .map(|entry| *entry.key())
            .collect();
        for keyspace_id in keyspaces {
            self.remove_keyspace(keyspace_id);
        }
        self.tenant_cap_keyspaces.clear();
    }
}
```

### 5.3 Hot-Update Wiring (subscriber.rs)

```rust
if old_cpu.tenant_max_cpu_ratio != new_cpu.tenant_max_cpu_ratio {
    manager.update_tenant_max_cpu_ratio(new_cpu.tenant_max_cpu_ratio);
}
if old_cpu.tenant_cap_idle_ttl != new_cpu.tenant_cap_idle_ttl {
    manager.update_tenant_cap_idle_ttl(new_cpu.tenant_cap_idle_ttl.as_millis());
}
```

### 5.4 Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `cpu_throttle_tenant_cap_ratio` | Gauge | Current `tenant_max_cpu_ratio` |
| `cpu_throttle_tenant_cap_quota_us` | Gauge | Computed cap quota in CPU microseconds |
| `cpu_throttle_allocations{reason="tenant_cap_exhausted"}` | Counter | Requests rejected because tenant at hard cap |
| `cpu_throttle_allocations{reason="global_exhausted"}` | Counter | Existing global bucket exhaustion; required to validate node-level hard cap |

### 5.5 Error Semantics

Prefer reusing the existing keyspace exhaustion surface for the first implementation:
```rust
ThrottleError::KeyspaceCpuExhaustedNoBurst  // tenant cap exhausted, no burst fallback
```

The coprocessor endpoint's retry loop already treats it as a retry-with-backoff throttle and returns `CpuThrottled` on timeout. A dedicated `TenantCpuExhausted` variant can be added later only if logs or metrics need a clearer reason; it is not required for the core behavior.

## 6. Code Location Summary

| File | Change |
|------|--------|
| `components/resource_control/src/cpu_config.rs` | Add `tenant_max_cpu_ratio: f64` and `tenant_cap_idle_ttl_ms`, validation |
| `components/resource_control/src/config.rs` | Add to `CpuThrottleSettings`, wire `to_cpu_throttle_config()` |
| `components/resource_control/src/cpu_throttle.rs` | `compute_tenant_cap_quota_us()`, `ensure_tenant_cap_bucket()`, idle cleanup, modify `skip_throttle` path, hot-update methods |
| `components/resource_control/src/subscriber.rs` | Wire hot-update in `update_config()` |
| `components/resource_control/src/metrics.rs` | New metrics |

**Estimated code change**: small and localized, mostly in `cpu_throttle.rs` plus config / metrics / tests.

## 7. Verification Plan

1. **Single tenant at cap**: One tenant sends heavy coprocessor/read requests → read-path CPU capped at `tenant_max_cpu_ratio * cores`. Other tenants unaffected.
2. **Multiple tenants at cap**: Each tenant independently capped at the same ratio. Global bucket prevents total overshoot.
3. **Tenant cap + global cap**: With `enforce_global_limit = true`, when `tenant_max_cpu_ratio > max_read_cpu_ratio` (e.g., 0.7 cap with 0.6 global), global bucket is the effective bottleneck.
4. **No skip_throttle**: Verify no keyspace gets `skip_throttle = true` when cap is active.
5. **No cap bypass**: With cap active, keyspace exhaustion does not enter burst mode; `enable_burst = false` is enforced by config.
6. **No PD/service-limit dependency**: With `enable_pd_service_limit = false`, tenant buckets are still auto-created and capped uniformly; no PD service_limit fetch is needed.
7. **Conflicting config rejected**: Cap mode rejects non-empty `keyspace_cpu_quotas`, `enable_pd_service_limit = true`, `enable_fair_allocation = true`, `enable_dynamic_adjustment = true`, `enable_burst = true`, or `enforce_global_limit = false`.
8. **Hot-update validation**: Online config changes with conflicting cap-mode settings are rejected as a whole.
9. **Runtime top-up**: `CpuTokenCheckFuture` / `allocate_more` only deducts from the same tenant and global buckets in cap mode; no burst/global-only fallback.
10. **Low-priority DAG disabled or cap-aware**: Essential defaults `low_priority_dag_global_enabled = false`; if enabled in a test profile, unknown keyspaces still pass tenant cap after the low-priority DAG bucket.
11. **Premium unchanged**: `tenant_max_cpu_ratio = 0.0` → legacy behavior (skip_throttle for unknown keyspaces).
12. **Hot update**: Changing `tenant_max_cpu_ratio` online updates all auto-created tenant bucket capacities.
13. **Cap disable**: Setting ratio to 0.0 removes auto-created cap buckets and reverts unknown keyspaces to legacy skip_throttle.
14. **Idle TTL hot update**: Changing `tenant_cap_idle_ttl` online changes when auto-created idle buckets are evicted; lowering the TTL should take effect on the next cleanup scan.

## 8. Rollout

1. **Code change** (this design) — localized changes in `resource_control`
2. **Unit tests** — bucket auto-creation, cap enforcement, hot-update
3. **Scope acknowledgement** — before staging or production enablement, product and operations must treat `tenant_max_cpu_ratio` as a coprocessor/read-path cap only. Write-heavy tenants can still consume write, apply, raftstore, or raft-io CPU until the follow-up shared read+write ledger exists.
4. **Staging Essential** — enable with `tenant_max_cpu_ratio = 0.4`, observe metrics
5. **Production Essential** — start conservative (e.g., 0.5), tighten based on data