# Worker Scaler

The Worker Scaler feature dynamically spawn load_data_worker pods on receive load_data task, and clean up the 
load_data_worker pods when the load_data task is finished.

## Configuration

Default configuration values are set as follows:

```toml
[worker-scaler]
run = false
namespace = "tidb-serverless"
template-sts-name = "tikv-api"
name = "load-data-worker"
worker-port = 19500
max-size = "1TB"
spawn-data-size = "2GB"
spawn-running-tasks = 8
worker-min-storage-gb = 20
worker-max-cores = 4
expire-seconds = 600
worker-count-limit = 1024
shared-pool-enabled = false
shared-pool-name = "ldw-shared"
shared-min-replicas = 0
shared-max-replicas = 64
shared-scale-up-tasks-per-pod = 16
shared-max-tasks-per-pod = 24
shared-idle-pod-expire-sec = 600

[worker-scaler.shared-sts-config]
# Optional dedicated sts config for ldw-shared.
# If not set (or invalid), worker-scaler uses built-in shared defaults.
# request-cpu = 4
# limit-cpu = 4
# worker-num = 1
# storage-size-gb = 64
# request-memory = 15
# limit-memory = 15
# node-group = ""
# node-selector-key = ""
# node-selector-value = ""
```

- The `run` option runs the Worker-Scaler within a `tikv-worker` process.
- The `template-sts-name` option specifies the name of the stateful-set used as a template for spawning `load-data-worker` pods.
- The `name` option is used to name the spawned worker stateful-set.
- The `max-size` option limits the maximum data size to load, if the data size is larger than this value, we return error to the client.
- The `spawn-data-size` controls whether a load-data task should spawn a dedicated worker pod for large tasks.
- The `spawn-running-tasks` is used only when shared pool is disabled. In that mode, when the locally running tasks exceeds `spawn-running-tasks`, we spawn a new `load-data-worker`, otherwise the task is executed locally on the `tikv-api` pod.
- The `worker-max-cores` specifies the maximum cores for the `load-data-worker` pod.
- The `expire-seconds` specifies the expiration time for the `load-data-worker` pod, if the pod is not used for `expire-seconds`, we will delete the pod.
- The `worker-count-limit` limits the max running `load-data-worker` pods.
- The `shared-pool-enabled` toggles the shared load-data worker pool for small tasks.
- The `shared-pool-name` sets the StatefulSet and Service name for the shared pool.
- The `shared-min-replicas` and `shared-max-replicas` bound the shared pool size. The
  default `shared-min-replicas = 0` allows the shared pool to scale to zero when it is idle.
- The `shared-scale-up-tasks-per-pod` controls when a pod triggers shared pool scale-up.
- The `shared-max-tasks-per-pod` is the soft cap for concurrently running tasks per shared pod.
- The `shared-idle-pod-expire-sec` controls how long an idle shared pod must stay idle before it can be scaled down.
- The `shared-sts-config` provides a dedicated StatefulSet spec for `ldw-shared`; if omitted (or invalid), the scaler falls back to built-in shared defaults (`request-cpu=4`, `limit-cpu=4`, `request-memory=15Gi`, `limit-memory=15Gi`, `storage-size-gb=64`, `worker-num=1`).

## Implementation

When the `tikv-api` process received a `load_data` initial request, we check if the task should be run locally, delegated to a shared pool,
or spawn a dedicated `load-data-worker` pod.
This determination is based on the `data_size` parameter and shared pool configuration.

To spawn a new `load-data-worker` pod, we create a new stateful-set based on the `template-sts-name` option, and change the
new stateful-set name based on the unique task id, so each task creates a unique stateful-set.
The pod's cpu, memory and storage is set based on the `data_size` of the task.

For small tasks, the worker scaler schedules the task onto a shared pool of StatefulSet pods (`shared-pool-name`).
The pool keeps `shared-min-replicas` ready and scales based on total running tasks and `shared-max-tasks-per-pod`.
With the default `shared-min-replicas = 0`, the pool can stay at zero replicas until a small task arrives.

When the load-data-worker pod is running, we return the pod's address to the client and the client will send the load_data
request to the new pod.

The worker-scaler periodically checks the per-task `load-data-worker` pods, if the pod is not used for `expire-seconds`, we will
delete the stateful-set and the pvc. The shared pool is scaled up and down independently based on its aggregated load.

### Shared pool scaling behavior

The shared pool is a StatefulSet (`shared-pool-name`) with a headless Service. Each task is pinned to a
specific shared pod so task state stays consistent across requests.

Scale-up:
- The worker scaler periodically refreshes shared pod status and aggregates running tasks across the pool.
- Desired replicas = `ceil(total_running_tasks / shared-scale-up-tasks-per-pod)`, clamped to
  `[shared-min-replicas, shared-max-replicas]`.
- When a new small task arrives and all running pods are at capacity, the scaler updates the StatefulSet
  replicas to the desired value and waits for a running pod before assigning the task.
- If the pool is at zero replicas, a new small task scales the StatefulSet to at least one replica
  before assignment.

Scale-down:
- If the desired replicas is smaller than the current replica count, the scaler evaluates pods from the
  highest ordinal downward.
- A pod is eligible for scale-down only when it is running and healthy, and it has zero running tasks,
  zero idle tasks, and has stayed idle for at least `shared-idle-pod-expire-sec`.
- Shared StatefulSet sets `persistentVolumeClaimRetentionPolicy.whenScaled=Delete`, so PVCs for scaled-down replicas are cleaned up by Kubernetes.
- The pool is never scaled below `shared-min-replicas`. When `shared-min-replicas = 0`, this allows
  scale-to-zero after all shared pods are idle for `shared-idle-pod-expire-sec`.