# Background

The current resource control mechanisms of Cloud Storage Engine are as follows:

- **IoRateLimiter**: Limit the IO (WriteOnly/ReadOnly/All_IO) rate and perform traffic control based on priority (High/Medium/Low)
    
- **WriteRateLimiter**: Limit the write rate to prevent the storage layer from being overloaded
    
- **WorkerLimiter**: Limit the concurrency of worker tasks and support two-level concurrency control at both the global and keyspace levels
    
- **MemoryLimiter**: Limit the memory usage
    
- **FlowController**: Write traffic control at scheduler layer, with dynamic write speed adjustment based on L0 file and memtable counts
    
- **RaftEngine WAL Throttle**: Throttle write operations when WAL compaction lags behind
    

  

For TiDB Cloud Essential, there is a resource control and isolation [solution](https://github.com/tidbcloud/cloud-storage-engine/issues/2054), and an [improvement](https://pingcap.feishu.cn/wiki/KeeJw61fKiJS9LkAYedcH1FTnJh) design. Current capabilities are:

- Isolate heavy and light read requests in the read pool.
    
- High-load regions or keyspaces don't consume more than half the resources of the node.
    
- When the water level remains high, all cop requests of the detected top keyspace will be offloaded to remote execution.
    

However, this resource control mechanism remains coarse-grained and cannot completely isolate the impacts among multiple keyspaces.

  

When a hot region receives multiple read requests, or when heavy queries execute, these read operations can consume most CPU resources, potentially overloading a CSE node.

We have collected some real-world cases from our customers related to resource control. Plaid, Gusto, CTW, and HangZhou Bank have all encountered cases where Resource Control fails to effectively limit resource usage, resulting in certain CSE nodes being overloaded and subsequently affecting the performance of the entire cluster ([Preventing TiKV Single-Node Overload with Resource Control](https://pingcap.feishu.cn/wiki/DvTgwShWbipVLoke64ScTZF3nLd)).

  

Overall, current mechanisms have critical limitations:

1. **Node Overloaded Risk**: A CSE node may be overloaded
    
2. **Resource Monopolization**: A single keyspace may monopolize a CSE node's resources
    
3. **Lack of CPU-based Resource Control**:
    
    1. **No CPU Usage Monitoring**: No periodic monitoring of actual CPU usage for resource control
        
    2. **No CPU-Based Throttling**: No mechanism to throttle based on real CPU usage at keyspace or request level
        
    3. **No Keyspace-Level CPU Isolation**: Unlike remote coprocessor (which uses WorkerLimiter), local read operations lack keyspace-level CPU protection
        
4. **Quota Proportionality Violation**: When a CSE node runs under high load (eg. 70%), actual resource allocation diverges from configured quotas.
    

# Design Goals

1. **Prevent Node Overload**: Protect a CSE node from being overloaded by read requests
    
2. **CPU-based Resource Control**:
    
    1. **CPU Usage Monitoring**: Periodically measure real CPU usage (both global and per-keyspace)
        
    2. **CPU-Based Throttling**: Throttle read requests when:
        
        1. **Global threshold**: Total node CPU usage is high (e.g., > 80%)
            
        2. **Keyspace** **threshold**: Single keyspace's CPU usage reaches a high level (eg. 60-70%) of total CPU resources
            
    3. **Keyspace-Level CPU Isolation**: Prevent cross-keyspace impacts
        
3. **Fairness**: When node runs on high pressure, allocate resources proportionally to keyspace configured quotas
    
4. **Low Overhead**: Minimal performance impact on normal operations
    

  

# Architecture Overview  

## Key Component 1: CPU Usage Monitor

CPU usage monitoring is built on the existing `resource_control` sliding window and is updated
every `stats-interval` by `ReadSubscriber` (no extra polling loop is added).

**Data sources**

- Global CPU: cumulative thread CPU counters collected by `ResourceCollector`.
- Per-keyspace CPU: per-request CPU samples reported via resource metering tags.

**Windowing and ratios**

`InstantUsages` keeps a window of samples within `window-size`. The ratios are computed as:

```Plain
global_cpu_time = delta(latest_global, oldest_global)   // cumulative counters
keyspace_cpu_time = sum(per_request_samples_in_window)  // per-request samples
max_cpu_time = cpu_cores * max_read_cpu_ratio * window_size
global_ratio = global_cpu_time / max_cpu_time
keyspace_ratio = keyspace_cpu_time / max_cpu_time
```

These ratios are pushed into `CpuThrottleManager` and exported as:

- `tikv_cpu_usage_monitor_global_ratio`
- `tikv_cpu_usage_monitor_keyspace_ratio{keyspace_id}`

  

## **Key Component 2:** **Keyspace Quota Source (Config + PD service_limit)**

**Config quotas**

- Per-keyspace quotas are configured via
  `keyspace-cpu-quotas = "keyspace_id:millicores,..."` in
  `[resource-control.cpu-throttle]`.
- Quotas are converted to **microseconds per refill interval** based on
  `refill-interval`.
- Global bucket capacity is derived from `cpu_cores * max_read_cpu_ratio *
  refill_interval_ms * 1000`, so the global budget is tied to the read pool CPU budget.

**PD service_limit (implemented)**

- The server periodically fetches each keyspace's `service_limit` (RU/s) from PD.
- `service_limit = 0` means unlimited: it does **not** participate in proportional allocation.
  The per-keyspace quota is removed unless `pd-override-config = false` and the
  keyspace has a static config quota.
- `service_limit > 0` participates in proportional allocation:
  `quota_us = global_capacity_us * (service_limit_i / sum(service_limit))`.
  This uses **ratios only** (no RU-to-CPU calibration is needed).
- Updates are applied only when relative change exceeds
  `pd-service-limit-update-threshold`.
- PD unreachable: keep last successful quotas.

**Keyspace discovery and lifecycle**

- Known keyspaces = config quotas + runtime discovered keyspaces from coprocessor DAG requests.
- Keyspace name is resolved via `to_keyspace_name()` from the local cache; if missing, skip this
  round and let the background PD fetch populate the cache.
- If the cached keyspace state is `Archived` or `Tombstone`, the keyspace is removed from tracking
  and its quota is cleared.
- PD `service_limit` 404s are treated as deletions only after
  `pd-service-limit-consecutive-404` or `pd-service-limit-ttl`.

## Key Component 3: Proportional Fair Allocation Algorithm

**Principle**: When node reaches high utilization, distribute remaining capacity proportionally to keyspace quotas.

**Algorithm**:

```Plain
When Node CPU Utilization > 70%:
  available_cpu = total_node_cpu - current_usage

  For each keyspace:
    if keyspace_usage < keyspace_quota:
      # Keyspace has unused quota
      active_keyspaces_quota = sum(keyspace_quotas for keyspaces within quota)
      fair_share = available_cpu * (keyspace_quota / active_keyspaces_quota)
      keyspace_can_use = min(keyspace_quota - keyspace_usage, fair_share)
    else:
      # Keyspace exceeded quota
      keyspace_can_use = 0  # Hard throttle
```

**Example**:

```Plain
Node: 8 cores total, currently 6.0 cores used (75%), 2.0 cores available

Keyspace quotas:   A=4 cores (50%), B=2 cores (25%), C=2 cores (25%)
Current usage:     A=2.5,          B=1.5,          C=2.0
Remaining quota:   A=1.5,          B=0.5,          C=0.0 (at limit)

Scenario: Three keyspaces have pending workload
  - A has requests needing 1.5 cores
  - B has requests needing 1.0 cores
  - C has requests needing 0.5 cores
  Total demand: 3.0 cores, but only 2.0 available at node level

Without proportional fair scheduling (FCFS):
  A=1.5, B=0.5, C=0 → unfair to B/C

With proportional fair scheduling:
  Step 1: Check each keyspace's remaining quota
    A can take up to 1.5 (has quota)
    B can take up to 0.5 (has quota)
    C can take 0.0 (already at quota limit)

  Step 2: Distribute 2.0 available cores by quota ratio among eligible keyspaces
    Eligible: A (quota 4) + B (quota 2) = 6 cores total eligible quota
    A gets: 2.0 * (4/6) = 1.33 → capped at 1.33
    B gets: 2.0 * (2/6) = 0.67 → capped at 0.5 (remaining quota)
    C gets: 0 (already at quota limit)

Final allocation:
  A: 2.5 + 1.33 = 3.83 cores ✓
  B: 1.5 + 0.5  = 2.0 cores (reaches quota limit) ✓
  C: 2.0 cores (already at limit, no more allocation) ✓

Result: Resources allocated proportionally (4:2 ratio), respecting individual quotas
```

## Key Component 4: Keyspace Cpu Throttler (Token Bucket-Based)

Throttles requests using token bucket algorithm for proactive CPU control.

**Token Bucket Advantages**:

- **Proactive**: Tokens checked BEFORE request execution based on estimated cpu consumption
    
- **Burst Handling**: Allows short bursts while limiting average rate
    
- **Smooth Throttling**: Natural rate limiting without sudden stops
    
- **Adaptive**: CPU monitoring adjusts refill rates dynamically
    
- **Low Overhead**: Token check is fast (atomic operations)
    

**How Token Bucket Works with CPU Monitoring?**

1. **Token Bucket (Proactive)**: Controls rate BEFORE requests run
    
2. **CPU Monitoring (Adaptive)**: Adjusts token refill rates based on actual CPU usage
    

  

Token Bucket Parameters:

- **Refill Rate**: How fast tokens are added (CPU microseconds per second)
    
- **Refill Interval**: How often tokens are added (e.g., every 100ms)
    
- **Bucket Capacity**: Maximum tokens that can accumulate
    

  

This component provides:

- **Proactive protection**: Tokens prevent overload before it happens
    
- **Adaptive control**: CPU monitoring adjusts rates based on actual usage
    
- **Best of both worlds**: Fast token checks + accurate CPU-based adjustments
    

  

**Structure**:

```Rust
pub struct KeyspaceCpuThrottler {
    cpu_monitor: Arc<CpuUsageMonitor>,

    // Global token bucket: limits total CPU consumption rate
    global_cpu_token_bucket: Arc<tikv_util::time::Limiter>,

    // Per-keyspace token buckets: limits CPU consumption rate per keyspace
    keyspace_token_buckets: Arc<DashMap<u32, Arc<KeyspaceCpuTokenBucket>>>,

    config: Arc<CpuThrottlerConfig>,
}

pub struct KeyspaceCpuTokenBucket {
    // Token bucket for CPU time (tokens = CPU microseconds)
    cpu_token_bucket: tikv_util::time::Limiter,

    keyspace_id: u32,
    // Current refill rate (CPU microseconds per second)
    refill_rate: AtomicU64,
}

pub struct CpuThrottlerConfig {
    // Token bucket refill interval
    pub refill_interval: Duration,  // e.g., 100ms
    // Estimated CPU time per request (for token consumption)
    pub estimated_cpu_per_request_us: u64,  // e.g., 1000 (1ms)
    // Dynamic adjustment based on watermarks
    pub enable_dynamic_adjustment: bool,
    pub high_watermark: f64,  // e.g., 0.8
    pub low_watermark: f64,   // e.g., 0.5
    // Fair allocation and burst
    pub enable_fair_allocation: bool,
    pub fair_allocation_threshold: f64,  // e.g., 0.7
    pub enable_burst: bool,
    pub burst_threshold: f64,  // e.g., 0.9
    // Runtime token management
    pub enable_runtime_token_management: bool,
    pub runtime_check_interval_us: u64,
    pub additional_allocation_threshold: f64,
    pub per_allocation_us: u64,
}
```

**Key Methods**:

```Rust
impl KeyspaceCpuThrottler {
    /// Acquire tokens before request execution (PROACTIVE throttling)
    /// Implements token bursting for zero-wait when system has capacity
    pub async fn acquire_tokens(
        &self,
        keyspace_id: u32,
        estimated_cpu_us: Option<u64>,
    ) -> Result<CpuTokenPermit, ThrottleError> {
        let cpu_us = estimated_cpu_us.unwrap_or(self.config.estimated_cpu_per_request_us);

        // Step 1: Check global CPU usage (from monitor) - fast atomic read
        let global_cpu_ratio = self.cpu_monitor.global_cpu_usage.load(Ordering::Relaxed);
        let keyspace_cpu_ratio = self.cpu_monitor.keyspace_cpu_usage
            .get(&keyspace_id)
            .map(|v| v.load(Ordering::Relaxed))
            .unwrap_or(0.0);

        // Step 2: Try to acquire tokens from global bucket
        let global_wait = self.global_cpu_token_bucket
            .consume_duration(cpu_us as usize);

        // Step 3: Try to acquire tokens from keyspace bucket
        let keyspace_bucket = self.get_or_create_keyspace_bucket(keyspace_id);
        let keyspace_wait = keyspace_bucket.cpu_token_bucket
            .consume_duration(cpu_us as usize);

        // Step 4: Token Bursting Strategy (ZERO-WAIT for OLTP)
        let (final_wait, burst_mode) =
            if global_wait == Duration::ZERO && self.can_burst_tokens(keyspace_id) {
                // CASE 1: Token Bursting (Preferred for OLTP)
                // System has capacity, use global tokens immediately
                (Duration::ZERO, true)
            } else if global_wait == Duration::ZERO && keyspace_wait == Duration::ZERO {
                // CASE 2: Both buckets have tokens (normal operation)
                (Duration::ZERO, false)
            } else if keyspace_cpu_ratio > keyspace_threshold {
                // CASE 3: Keyspace using too much CPU (strict throttling)
                let max_wait = global_wait.max(keyspace_wait);
                (max_wait, false)
            } else {
                // CASE 4: Normal operation, need to wait
                let max_wait = global_wait.max(keyspace_wait);
                (max_wait, false)
            };

        // Step 5: Wait only if necessary
        if !final_wait.is_zero() {
            tokio::time::sleep(final_wait).await;
        }

        Ok(CpuTokenPermit {
            keyspace_id,
            tokens_consumed: cpu_us,
            burst_mode,
        })
    }

    /// Check if keyspace can burst tokens from global bucket
    /// For latency-sensitive OLTP: zero-wait when system has capacity
    fn can_burst_tokens(&self, keyspace_id: u32) -> bool {
        if !self.config.enable_burst {
            return false;
        }

        let global_cpu = self.cpu_monitor.global_cpu_usage.load(Ordering::Relaxed);
        let keyspace_cpu = self.cpu_monitor.keyspace_cpu_usage
            .get(&keyspace_id)
            .map(|v| v.load(Ordering::Relaxed))
            .unwrap_or(0.0);

        // Bursting conditions (all must be true):
        // 1. Global CPU is low (system has capacity)
        // 2. Keyspace CPU is not excessive
        global_cpu < self.config.burst_threshold &&
        keyspace_cpu < keyspace_threshold  // derived from quota/usage policy
    }

    /// Update token bucket refill rates based on CPU usage (ADAPTIVE)
    pub fn update_refill_rates(&self, cpu_usage: &KeyspaceCpuUsage) {
        if !self.config.enable_dynamic_adjustment {
            return;
        }

        // Update global bucket based on high/low watermarks.
        if cpu_usage.global_cpu_ratio > self.config.high_watermark {
            // Reduce global refill rate (throttling).
        } else if cpu_usage.global_cpu_ratio < self.config.low_watermark {
            // Restore global refill rate.
        }

        // Update per-keyspace buckets based on usage vs quota.
        for (keyspace_id, cpu_ratio) in &cpu_usage.keyspace_cpu_ratios {
            if let Some(bucket) = self.keyspace_token_buckets.get(keyspace_id) {
                if cpu_ratio > self.keyspace_quota_ratio(keyspace_id) * 0.9 {
                    // Reduce refill rate.
                    bucket.update_refill_rate(/* new_rate */);
                } else {
                    // Restore base rate.
                    bucket.restore_base_rate();
                }
            }
        }
    }
}
```

> **Note:** The Rust code above is *illustrative pseudocode* for the keyspace-level
> CPU throttling design. It is not a verbatim representation of the current
> implementation or exact API names (for example, the real types are
> `CpuThrottleConfig`/`CpuThrottleSettings`). Refer to the source code for the
> authoritative definitions. 

### Runtime token management and per-future CPU accounting

- Initial token allocation uses a short wait-retry loop until the request deadline.
  Wait duration and retry count are exported as metrics.
- Request execution is wrapped by `CpuObserveFuture`, which measures CPU time per
  future poll (not per thread) to avoid over-counting on shared executors.
- When `enable-runtime-token-management` is enabled,
  `CpuTokenCheckFuture` periodically checks consumed CPU
  (`runtime-check-interval-us`). If consumption exceeds
  `additional-allocation-threshold` of the current allocation, it
  allocates `per-allocation-us` more tokens. Failure before the
  deadline returns a throttled error.

### Quota shrink and debt handling

- Each keyspace tracks **inflight allocated tokens** (estimated CPU time not yet released).
- When a quota is reduced at runtime, if `inflight > new_quota`, the bucket enters
  a **debt** state (`available = 0`) until inflight drops below the new quota.
- This avoids interrupting in-flight requests while preventing new ones from
  overshooting the updated quota.

### Estimated CPU per request (static + adaptive)

- Default estimate: `estimated-cpu-per-request-us`.
- Per-keyspace overrides: `keyspace-estimated-cpu-per-request-us`.
- If `enable-adaptive-estimated-cpu-per-request-us` is enabled, each
  keyspace estimate is updated once per `stats-interval` using an EMA over
  observed per-request CPU time.

# Configuration

```TOML
[resource-control]
enabled = true
stats-interval = "1s"
window-size = "1m"
max-read-cpu-ratio = 0.6
enable-qps-bytes-limiter = true

[resource-control.cpu-throttle]
enabled = true
keyspace-cpu-quotas = "1:1000,2:2000"
enforce-global-limit = false

# PD service_limit integration
enable-pd-service-limit = true
pd-service-limit-fetch-interval = "30s"
pd-service-limit-request-timeout = "5s"
pd-service-limit-ttl = "10m"
pd-service-limit-consecutive-404 = 3
pd-service-limit-update-threshold = 0.01
pd-override-config = true

estimated-cpu-per-request-us = 1000
keyspace-estimated-cpu-per-request-us = "1:1500,2:5000"
enable-adaptive-estimated-cpu-per-request-us = true
refill-interval = "10ms"
burst-window = "100ms"
enable-dynamic-adjustment = true
high-watermark = 0.8
low-watermark = 0.5
# Legacy admission burst-share guard; separate from read-pool keyspace fairness.
enable-fair-allocation = true
fair-allocation-threshold = 0.7
enable-burst = true
burst-threshold = 0.9
keyspace-burst-enabled = "1:true,2:false"

# Runtime token management
enable-runtime-token-management = true
runtime-check-interval-us = 2000
additional-allocation-threshold = 0.5
per-allocation-us = 4000
```

`resource-control.max-read-cpu-ratio` can be updated online while CPU throttle
is disabled, which updates the legacy read-limiter budget. When CPU throttle is
enabled, the value determines the startup global token-bucket capacity and an
online change is rejected; update the static configuration and restart instead.

Global bucket capacity is derived from the CPU rate and `burst-window`, while
each refill adds CPU rate times `refill-interval`. Per-keyspace quotas are
configured in millicores and converted to microseconds per refill interval.
Burst allocations from the global bucket remain capped by the per-interval
refill budget, so a larger burst capacity does not increase the long-term CPU
rate.

The `100ms` burst-window default is a compatibility choice rather than a claim
that it is optimal for every workload. Before capacity and refill cadence were
decoupled, the default `100ms` refill budget was also the bucket capacity, so
retaining a `100ms` capacity window avoids silently increasing burst allowance.
For the same CPU rate, `300ms` or `500ms` would cache three or five times as many
tokens and permit correspondingly larger short-term bursts. Such values remain
configurable, but should be selected with workload-specific CPU-cap and tail
latency validation rather than used as a larger fleet-wide default.

Upgrade compatibility notes:

- `refill-interval` and `burst-window` are restart-only. A deployment that does
  not explicitly configure `refill-interval` adopts the new `10ms` default at
  its next restart, including when keyspace fairness is disabled. Legacy
  deployments that need the previous retry cadence should explicitly set
  `refill-interval = "100ms"`; `50ms` is an intermediate option.
- Legacy admission and runtime allocation wait for the exhausted bucket's next
  refill boundary. Token refunds do not proactively wake an already-armed
  timer. With an explicit `100ms` refill interval, a waiter refunded just after
  it starts sleeping can therefore wait until that boundary. The implementation
  intentionally avoids a fixed 10ms poll, which would increase retries for all
  throttled requests.

# Metrics

Key CPU throttle metrics:

- `tikv_cpu_usage_monitor_global_ratio`
- `tikv_cpu_usage_monitor_keyspace_ratio{keyspace_id}`
- `tikv_cpu_throttle_allocations_total{result}`
- `tikv_cpu_throttle_global_bucket_available_us`
- `tikv_cpu_throttle_global_bucket_capacity_us`
- `tikv_cpu_throttle_global_refill_rate_us_per_interval`
- `tikv_cpu_throttle_keyspace_bucket_available_us{keyspace_id}`
- `tikv_cpu_throttle_keyspace_bucket_capacity_us{keyspace_id}`
- `tikv_cpu_throttle_keyspace_refill_rate_us_per_interval{keyspace_id}`
- `tikv_cpu_throttle_keyspace_inflight_us{keyspace_id}`
- `tikv_cpu_throttle_keyspace_debt_us{keyspace_id}`
- `tikv_cpu_throttle_fair_allocations_total{keyspace_id}`
- `tikv_cpu_throttle_burst_total{keyspace_id}`
- `tikv_cpu_throttle_burst_tokens_used_total{keyspace_id}`
- `tikv_cpu_token_wait_duration_seconds{keyspace_id,result}`
- `tikv_cpu_token_allocation_retries_total{keyspace_id}`
- `tikv_cpu_token_runtime_wait_duration_seconds{keyspace_id,result}`
- `tikv_cpu_token_runtime_allocations_total{result}`
- `tikv_cpu_throttle_request_cpu_time_seconds{keyspace_id}`
- `tikv_cpu_throttle_request_actual_to_estimated_ratio{keyspace_id}`
- `tikv_service_limit_fetch_total{result}`
- `tikv_service_limit_fetch_duration_seconds`
- `tikv_service_limit_last_success_timestamp`
- `tikv_service_limit_keyspace_value{keyspace_id}`
- `tikv_service_limit_keyspace_quota_us{keyspace_id}`

# Behavior

## Normal Operation

- **Token Buckets Active**: Global and per-keyspace token buckets refill at base rates
    
- **Proactive Throttling**: Requests acquire tokens BEFORE execution
    
- **CPU Monitoring**: CPU usage measured periodically (every 2 seconds)
    
- **No Delays**: When tokens available, requests proceed immediately
    

  

## High CPU Usage - Global Threshold

When total node CPU usage exceeds the global threshold (e.g., > 80%, configurable):

- **Global token refill rate reduced**: Fewer tokens available globally
    
- **All keyspaces affected**: All requests wait longer for global tokens
    
- **Adaptive adjustment**: Refill rate decreases proportionally to CPU usage
    
- **Protects the node**: Prevents total system overload
    
      
    

## High CPU Usage - Keyspace Threshold

When a single keyspace's CPU usage exceeds threshold (e.g., > 80% of total CPU, configurable):

- **Keyspace** **token refill rate reduced**: Fewer tokens for that keyspace
    
- **Only that** **keyspace** **throttled**: Other keyspaces continue normally
    
- **Requests wait for tokens**: Requests from that keyspace wait longer
    
- **Prevents monopolization**: Single keyspace cannot consume all CPU
    
- **Current scope**: Only local coprocessor DAG requests are throttled
- **Future**: Extend to other read requests (get/batch-get)
    
      
    

## Token Bucket Behavior

- **Token Consumption**: Each request consumes tokens based on estimated CPU time
    
- **Token Refill**: Tokens refill continuously at configured rate
    
- **Smooth Throttling**: Natural rate limiting without sudden stops
    
- **Wait Time**: If tokens unavailable, requests wait (with timeout)
    

## Token Bursting for OLTP (Zero-Wait Mechanism)

### Overview

Token bursting is a zero-wait mechanism designed for latency-sensitive OLTP workloads. When the node has CPU capacity available, keyspaces can immediately use global tokens (with limited amount) instead of waiting for their own token bucket to refill.

### Problem Statement

**Scenario**:

- Global CPU: 30% (system has 70% capacity)
    
- Global token bucket: Has many tokens
    
- Keyspace A token bucket: Empty
    
- Keyspace A CPU: 25% (not excessive)
    

**Without Bursting**: Request waits for keyspace bucket refill → latency jitter

**With Bursting**: Request uses global tokens immediately → zero wait

### Design Principles

1. **Zero-Wait Priority**: Immediate execution when capacity available
    
2. **Fast Path**: Atomic reads only (no locks, ~10-50ns overhead)
    
3. **No Jitter**: Consistent low latency for OLTP
    
4. **Capacity-Based**: Conservative threshold (40% CPU) ensures safety
    
5. **Mission-Critical**: Designed for latency-sensitive workloads
    

### Bursting Conditions

All must be true:

```Rust
global_bucket_has_tokens &&           // Global tokens available
global_cpu < burst_threshold &&    // System has capacity (< 40%)
keyspace_cpu < keyspace_threshold     // Keyspace not excessive (< 65%)
```

### Performance Characteristics

- Bursting Path: ~10-50 nanoseconds (atomic reads + token check)
    
- Normal Path: ~10-50 nanoseconds (atomic reads + token check)
    
- Throttling Path: Wait time depends on token refill rate
    
- No Jitter: Bursting ensures zero-wait when capacity available
    

# Trade-offs

## Advantages

1. **Proactive Throttling**: Tokens checked BEFORE execution, prevents overload
    
2. **Fairness**: Token buckets ensure no single keyspace monopolizes CPU
    
3. **Burst Handling**: Naturally handles bursts while limiting average rate
    
4. **Smooth Throttling**: Natural rate limiting without sudden stops
    
5. **Adaptive**: CPU monitoring adjusts token refill rates dynamically
    
6. **Low Overhead**: Token check is fast (~10-50ns)
    
7. **Predictable**: Token-based throttling more predictable than reactive monitoring
    
8. **Multi-Level Defense**: Global + keyspace token buckets provide comprehensive protection
    
9. **Zero-Wait Bursting**: Ensures zero latency jitter for OLTP when capacity available
    
10. **Capacity Utilization**: Bursting mechanism ensures system capacity used efficiently
    

## Considerations

1. **CPU Estimation**: Need to estimate CPU time per read request
    
2. **False Positives**: May throttle legitimate high-load keyspaces (need adjust threshold)
    
3. **Estimation Accuracy**: Over/under-estimation affects fairness
    

# Testing Strategy

### Unit Tests

- Test token acquisition/release
    
- Test adaptive refill rate calculation
    
- Test timeout handling
    

### Integration Tests

- Test with multiple keyspaces
    
- Test hot region scenario
    
- Test recovery after load decreases
    
- Test get/batch-get throttling (future)
    
- Test mixed workload
    
- Test keyspace isolation
    

### Benchmark Tests

- Measure overhead of token acquisition
    
- Compare with/without throttler
    
- Test under various load patterns
    

  

# Future Enhancements

1. **More accurate resource needs estimation**: Estimate CPU consumption by the request details (based on key / block count)
    
2. **Write Request Coverage**: Control CPU time on write requests
    
3. **Keyspace-Aware Thread Pool Management**: Distribute threads to keyspaces based on resource quota
    
4. **Cross-Node Coordination**: Coordinate limits across CSE nodes
    
5. **Query Classification**: Different CPU quotas for different query types (OLTP vs OLAP)
    
6. **Preemption**: Ability to pause/resume queries based on priority
    

  

# References

- [Amazon DynamoDB Multi-Tenant Resource Control (USENIX ATC 2022)](https://www.usenix.org/system/files/atc22-elhemali.pdf)
    
- Linux CFS (Completely Fair Scheduler) - inspiration for proportional scheduling
    
- [Resource control and isolation](https://github.com/tidbcloud/cloud-storage-engine/issues/2054) in TiDB Cloud Serverless
    
- [Resource Control](https://pingcap.feishu.cn/wiki/KeeJw61fKiJS9LkAYedcH1FTnJh)
    
- [Coupang Serverless POC Issues](https://pingcap.feishu.cn/wiki/LPjRwGmMLiAbkxkzNkLcjTbjn9d)
    
- [Preventing TiKV Single-Node Overload with Resource Control](https://pingcap.feishu.cn/wiki/DvTgwShWbipVLoke64ScTZF3nLd)
    
- [Resource Control on TiKV Enhancement](https://pingcap.feishu.cn/wiki/JT1iwo1IQimXJQkVYJNcRVaInph)