# YATP Simulator `yatp-sim` is a command-line tool that generates controlled workloads on a real YATP read pool. Its primary use is to run the same workload with keyspace fair scheduling enabled and disabled. The tool expands configured keyspace groups, submits CPU-consuming task futures through `ReadPoolHandle`, and runs every task in cooperatively scheduled slices. It supports: - the real keyspace-fair and regular multilevel YATP queues; - open-loop and closed-loop arrivals; - independent concurrency for every keyspace and task class; - CPU work and synchronous blocking inside each task slice; - deterministic jitter for task, slice, and blocking durations; - draining or cooperative cancellation when generation stops; - a configurable idle cooldown before the read pool shuts down; - production `KeyspaceMetrics` enabled for observation-cost fidelity and the keyspace queue-length report; - separate simulator-owned per-task-runtime metrics collected from YATP poll events for all other report measurements; - periodic terminal statistics plus standalone HTML and JSON reports; and - optional per-task slow thresholds with per-poll scheduling and execution timelines in file reports. ## Quick start Run the example workload with keyspace fairness enabled: ```bash cargo run -p yatp-sim --release -- \ --config configs/five-tenants.toml ``` Run the same workload with keyspace fairness disabled: ```bash cargo run -p yatp-sim --release -- \ --config configs/five-tenants.toml \ --set read-pool.keyspace-fairness.enabled=false \ --set read-pool.fair-global-cpu-admission.enabled=false \ --set report-path=target/yatp-sim/five-tenants-fairness-off.html \ --set summary-path=target/yatp-sim/five-tenants-fairness-off.json ``` The second command uses CLI overrides so both runs share exactly the same workload definition while changing only the scheduling/admission mode and output file names: ```toml [read-pool.keyspace-fairness] enabled = true # false selects the regular multilevel queue ``` Use release mode for scheduling experiments. The simulator performs real CPU work, and debug-build overhead can distort workloads with short slices. To create another workload, copy the example, change the keyspace groups and task fields, and use overrides as above to keep all workload fields identical when comparing the two queue modes. Each run prints one interval summary per `stats-print-interval`, followed by an aggregate run summary after the workload finishes, and writes: - a standalone HTML report to `report-path`; and - a machine-readable JSON report to `summary-path`. Task rows in each terminal summary start with `[keyspace/task]`, are column-aligned, and include whole-task latency average (`task_lat_avg`) and P99 (`task_lat_p99`) for tasks completed during that interval. Latency uses `us`, `ms`, or `s` according to its magnitude and is `N/A` when the interval has no completed latency samples. A task whose configured start time has not reached the interval is omitted instead of being printed as an all-zero row. The interval header contains the rounded capture time and total CPU measured from the native YATP worker thread clocks, such as `[15s] total_cpu_cores=4`. This includes scheduler, queue, and other worker thread CPU outside task polls; warmup intervals additionally include the `(WARMUP)` marker. The final run summary divides the configured duration at every task start and end boundary. The resulting `Phase1`, `Phase2`, and so on therefore represent different active workload combinations rather than warmup and measurement. The phase header lists its active `{keyspace}/{task}` workloads. Each workload then has one QPS/CPU line and one whole-task latency line containing average and P50/P80/P95/P99. The phase ends with total CPU measured from the native YATP worker thread clocks and its utilization relative to the configured worker capacity. Numeric values retain up to three decimal places without insignificant trailing zeros. Drain and cooldown are excluded. The HTML report renders the same phase summary in place of the old measurement keyspace and task tables. When slow-task tracing is enabled, the final terminal summary prints only the number of completed tasks that exceeded their configured threshold. Individual slow-task and poll details are written to the JSON and HTML reports instead of being printed to stdout. The commands above produce separate output files under `target/yatp-sim/`, so running one does not overwrite the other. The example enables detailed tracing only for the `other/normal` workload through its `slow-threshold` setting. ## Configuration Pass a TOML file through `--config`. Durations use strings such as `100ms`, `5s`, or `2m`. CPU costs, synchronous blocking times, jitter, and virtual-time precharge use integer microseconds. An absolute config path is used unchanged. For a relative path, the simulator first checks the process working directory and then falls back to the `yatp-sim` crate root. Consequently, `--config configs/five-tenants.toml` works both from the repository root with `cargo run -p yatp-sim` and after changing into `tools/yatp-sim`. Relative `report-path` and `summary-path` values still use the process working directory. ### Command-line overrides The TOML file remains the base configuration. Repeat the following options to override individual fields after the file is loaded: ```text --set FIELD=VALUE --set-keyspace GROUP FIELD=VALUE --set-task GROUP TASK FIELD=VALUE ``` For example: ```bash cargo run -p yatp-sim --release -- \ --config configs/five-tenants.toml \ --set duration=30s \ --set read-pool.workers=8 \ --set read-pool.keyspace-fairness.enabled=false \ --set read-pool.fair-global-cpu-admission.enabled=false \ --set-keyspace B count=2 \ --set-task B normal qps=500 \ --set-task B burst enabled=false ``` `--set` accepts the simulator fields and full `read-pool.*` paths listed below. `--set-keyspace` selects an existing keyspace group by `name` and currently accepts `count`. `--set-task` selects an existing task by its group name and task name and accepts every task field except `name`. Selectors are exact and case-sensitive. Overrides cannot add, delete, or rename groups and tasks. The selector and assignment are separate command-line values. Quote a selector by itself when its name contains spaces: ```bash --set-keyspace "noisy tenant" count=3 # valid --set-task "noisy tenant" "normal read" qps=50 # valid --set-keyspace "noisy count=3" # invalid: only one value ``` Assignment values use their normal command-line spelling, such as `30s`, `false`, `open-loop`, or `high`, without TOML string quotes. If a path contains spaces, quote the complete assignment, for example `--set 'report-path=target/my report.html'`. Only the first `=` separates the field from its value, so later `=` characters are preserved in paths. All overrides are applied before the effective configuration is validated. This allows related fields to change together. For example, changing an open-loop task to closed-loop also requires explicitly clearing its optional `qps`: ```bash --set-task noisy normal arrival-mode=closed-loop \ --set-task noisy normal qps=null ``` If the same target field is overridden more than once, the last value wins. Unknown fields or selectors, invalid values, and invalid final combinations fail before the read pool starts. The effective configuration, including overrides, is stored in the JSON and HTML reports. A keyspace group override changes the group template. If `count = 3`, a task override for that group applies to all three expanded keyspaces. To configure only one instance differently, define it as a separate group in the TOML file. Unknown fields are rejected. Invalid combinations such as open-loop without `qps`, closed-loop with `qps`, zero concurrency, or an excessive fairness precharge fail before the read pool starts. ### Complete example This example keeps two keyspaces backlogged. The noisy keyspace has more concurrent tasks, while both task classes use the same CPU cost and slice size. ```toml duration = "60s" warmup-duration = "5s" stats-print-interval = "1s" random-seed = 1 drain-on-stop = true cooldown-duration = "5s" report-path = "target/yatp-sim/report.html" summary-path = "target/yatp-sim/summary.json" [read-pool] workers = 4 [read-pool.keyspace-fairness] enabled = true use-wall-time = false slice-precharge-us = 1000 [read-pool.fair-global-cpu-admission] enabled = false max-cpu-cores = 4.0 refill-interval = "100ms" admit-interval-us = 1000 request-timeout = "1h" [[keyspace-groups]] name = "noisy" count = 1 [[keyspace-groups.tasks]] name = "cpu-heavy" arrival-mode = "closed-loop" concurrency = 64 task-cpu-us = 100000 task-cpu-jitter-us = 0 slice-cpu-us = 1000 slice-cpu-jitter-us = 0 blocking-wall-us = 0 blocking-wall-jitter-us = 0 priority = "normal" start-time = "0s" duration = "60s" slow-threshold = "250ms" [[keyspace-groups]] name = "victim" count = 1 [[keyspace-groups.tasks]] name = "latency-sensitive" arrival-mode = "closed-loop" concurrency = 4 task-cpu-us = 100000 slice-cpu-us = 1000 ``` Fields that are omitted from the second task use the defaults below. ### Simulator fields | Field | Default | Description | | --- | --- | --- | | `duration` | `1m` | Total workload-generation time, including warmup. Drain time is not included. | | `warmup-duration` | `0s` | Initial generation time reported separately and excluded from the measurement summary. Must be shorter than `duration`. | | `stats-print-interval` | `1s` | Target interval for terminal summaries and report time-series windows. | | `random-seed` | `1` | Seed for task, slice, and blocking-time jitter. It does not control YATP or OS scheduling. | | `drain-on-stop` | `true` | Whether submitted tasks finish normally after generation stops. See [Stopping](#stopping). | | `cooldown-duration` | `5s` | Idle delay after draining or cancellation completes and report data is frozen. The cooldown is excluded from all report metrics; `0s` disables it. | | `report-path` | `target/yatp-sim/report.html` | Standalone HTML report path. Parent directories are created automatically. | | `summary-path` | `target/yatp-sim/summary.json` | Machine-readable JSON report path. It must differ from `report-path`. | ### Read-pool fields | Field | Default | Description | | --- | --- | --- | | `workers` | `4` | Fixed YATP worker count. The simulator does not use worker auto-scaling. | | `keyspace-fairness.enabled` | `false` | Uses the keyspace-fair queue when true and the regular multilevel queue when false. | | `keyspace-fairness.use-wall-time` | `true` | Charges time spent inside each `Future::poll()` when true; otherwise charges thread CPU time. | | `keyspace-fairness.slice-precharge-us` | `0` | Virtual time charged at poll start and reconciled after the poll. Maximum: `1_000_000`. | | `fair-global-cpu-admission.enabled` | `false` | Enables the production fair-queue global CPU admission controller. Requires `keyspace-fairness.enabled = true`. | | `fair-global-cpu-admission.max-cpu-cores` | `4.0` | Maximum aggregate task CPU. The token capacity is this value multiplied by `refill-interval`. | | `fair-global-cpu-admission.refill-interval` | `100ms` | Global CPU token-bucket refill interval. Minimum: `1ms`. | | `fair-global-cpu-admission.admit-interval-us` | `1000` | Target maximum actual task CPU between fair CPU checkpoints while a poll slice runs. Checkpoint and clock-read overhead can cause a small overshoot. Must be greater than zero. | | `fair-global-cpu-admission.request-timeout` | `1h` | Time after task submission when admission changes to `Bypassed` for that task. Must be greater than zero. | `use-wall-time` only includes synchronous time inside a poll. It does not include time while a future is pending and not being polled. When fair global CPU admission is enabled, the simulator constructs the real `CpuThrottleManager`, `ReadLimiter`, and `FairGlobalCpuAdmissionController`. Before the fair queue pops a task, the controller waits until the global bucket has positive tokens. A popped task receives a permit unless its `request-timeout` has expired. While a poll slice runs, the simulator calls the production `fair_cpu_checkpoint()` function after each `admit-interval-us` deadline and after the final remainder. For example, a `120us` slice with a `50us` interval has checkpoint deadlines at approximately `50us`, `100us`, and `120us` of actual task thread CPU. `Continue` keeps running the current poll slice. If `YieldToScheduler` ends the poll, the next dispatch restarts the full sampled slice CPU target, capped by the task's remaining total CPU, instead of running only the old slice tail. The task-wide CPU target is charged by the actual thread CPU consumed by the poll, including simulator control and checkpoint code. A bypassed task has no permit, so its checkpoints continue without throttling. Synchronous `blocking-wall-us` runs once before the slice's CPU work and is not charged to the global CPU bucket. The simulator creates unregistered Prometheus collectors for both production `KeyspaceMetrics` and its task-runtime report metrics. `KeyspaceMetrics` is installed on the pool by default to retain the production observation cost, and the report reads its fair-queue length histogram at keyspace scope. All other report measurements use the simulator-owned task-runtime collectors. Nothing is registered in the process-global Prometheus registry. ### Keyspace groups A group creates one or more keyspaces with the same task configuration: ```toml [[keyspace-groups]] name = "oltp" count = 10 ``` | Field | Default | Description | | --- | --- | --- | | `name` | required | Unique group label used to identify the configuration. It is not the scheduling key. | | `count` | `1` | Number of keyspaces created from the group. | Every generated keyspace receives a distinct `u32` keyspace ID. All task classes inside one expanded group instance share that keyspace ID. Fair scheduling uses this ID; group and task names do not affect scheduling. A group may have no task entries, which creates idle keyspaces for idle-capacity experiments. The whole configuration must contain at least one enabled task class. The simulator currently implements equal-weight fairness and does not accept a `weight` field. ### Task fields Each task configuration is instantiated independently for every keyspace in its group. `concurrency` is therefore per keyspace and per task class; it is not shared by all keyspaces in a group. | Field | Default | Description | | --- | --- | --- | | `name` | required | Task-class label, unique inside its group. | | `enabled` | `true` | Whether this task class is expanded into runnable task runtimes. | | `arrival-mode` | required | `open-loop` or `closed-loop`. | | `qps` | required for open-loop | Offered arrivals per second for each keyspace. It is rejected in closed-loop mode. | | `concurrency` | required | Maximum outstanding tasks of this class for each keyspace. | | `task-cpu-us` | required | Total actual task thread CPU target. The caller caps each poll's slice target by the task CPU still remaining. | | `task-cpu-jitter-us` | `0` | Maximum random deviation applied once to a task's total CPU target. | | `slice-cpu-us` | required | Actual task thread CPU target for one logical slice. Reaching it reschedules the task. | | `slice-cpu-jitter-us` | `0` | Maximum random deviation sampled independently for every logical slice. | | `blocking-wall-us` | `0` | Synchronous non-CPU work assigned to every logical slice. | | `blocking-wall-jitter-us` | `0` | Maximum random deviation sampled for each logical slice's blocking time. | | `priority` | `normal` | `high`, `normal`, or `low`; it controls ordering inside the selected keyspace. | | `start-time` | `0s` | Time after simulator start when this task class begins generating work. | | `duration` | `0s` | Generation duration. Zero means until the global `duration` ends. | | `slow-threshold` | `0s` | Whole-task latency threshold for retaining a detailed per-poll trace. `0s` disables tracing for this task class and is equivalent to an infinite threshold. | A disabled task remains in the effective configuration but is not expanded into a runtime for any keyspace in its group, so it produces no workload or task report series. Its other fields are still validated, which makes enabling it later through `--set-task GROUP TASK enabled=true` safe. At least one task class in the complete configuration must remain enabled. The generation window is the half-open interval: ```text [start-time, min(global duration, start-time + task duration)) ``` Tasks already submitted at the end of this window continue running. The window only controls new arrivals and closed-loop replacement tasks. #### Slow-task tracing `slow-threshold` is evaluated against whole-task latency, from immediately before successful submission until the final poll completes. A successfully completed task is slow only when its latency is strictly greater than the configured threshold. A task whose latency exactly equals the threshold is not slow. Cooperatively cancelled tasks are not retained as slow tasks. For example, this retains details for completed tasks whose whole-task latency exceeds 20 ms: ```toml [[keyspace-groups.tasks]] name = "latency-sensitive" arrival-mode = "open-loop" qps = 200 concurrency = 10 task-cpu-us = 5000 slice-cpu-us = 500 slow-threshold = "20ms" ``` Determining whether a task is slow requires waiting for it to complete. When a non-zero threshold is configured, the simulator therefore temporarily retains that task's poll trace while it is outstanding. Traces for tasks at or below the threshold are discarded on completion; traces for slow tasks are kept in the final report. With the default `0s`, no per-poll trace buffer is allocated. Enabling a threshold can consume significant memory when there are many outstanding tasks, many polls per task, or many slow tasks. #### Open-loop arrivals Open-loop mode produces arrivals at absolute deadlines derived from `qps`: ```text deadline(n) = simulation start + start-time + n / qps ``` `n` begins at zero. For example, `qps = 2` in `[0s, 1s)` produces arrivals at `0ms` and `500ms`; the boundary at `1s` is excluded. If the task class already has `concurrency` outstanding tasks, that arrival is suppressed. It is not delayed and submitted later. The report records every arrival as `offered`, every accepted arrival as `submitted`, and the rest as `suppressed`. Thus, for an open-loop runtime: ```text offered = submitted + suppressed ``` If the controller reaches the generation boundary before it has processed all earlier deadlines, those remaining arrivals are also recorded as suppressed rather than silently dropped. #### Closed-loop arrivals Closed-loop mode fills the task class to `concurrency` at `start-time`. Every completion immediately submits one replacement while its generation window is still active. `qps` is not used and must not be configured. For example, a group with `count = 10` and a closed-loop task with `concurrency = 4` maintains up to four outstanding tasks for each keyspace, or 40 tasks in total. #### Task slices A task samples its total `task-cpu-us` once, then consumes that actual thread CPU target in logical slices. Each slice samples `slice-cpu-us` and `blocking-wall-us` once. The synchronous `run_slice` path performs the wall sleep first, exactly once, then runs CPU work without awaiting. It only returns whether the async wrapper should yield to the scheduler for global admission or reschedule after normal slice completion. The thread CPU clock starts before per-poll task and slice setup. Consequently, sampling, cancellation checks, CPU-clock reads, checkpoint calls, and other simulator control code inside the task all consume the configured task and slice targets. The busy loop only fills the difference up to the next target; it is not itself the complete definition of simulated CPU work. With global CPU admission enabled, checkpoints use actual elapsed task CPU deadlines. A `120us` slice and `50us` interval therefore behaves like `50 + 50 + 20`, apart from small clock/checkpoint overshoot. When every checkpoint returns `Continue`, reaching the sampled slice target returns `Reschedule`. If a checkpoint returns `YieldToScheduler`, the logical slice is retained and the next poll starts its CPU accounting again from the full sampled slice target, capped by the task's remaining total CPU. It does not inherit a small tail from the interrupted poll, and the slice's wall sleep is not repeated. Before every `run_slice` call, `run_task` caps the slice CPU target by the task's remaining CPU. After `run_slice` returns, it reads elapsed thread CPU again and adds that poll's actual consumption to the task total. If the total has reached `task-cpu-us`, the task returns ready directly; otherwise it follows the `run_slice` result and awaits either `yield_to_scheduler()` or `reschedule()`. CPU work is measured with thread CPU time. Synchronous blocking intentionally occupies a YATP worker but consumes little thread CPU, which makes it useful for comparing CPU-time and wall-time fairness accounting. Fair scheduling can choose another keyspace only between polls. A running CPU chunk or synchronous wall sleep cannot be preempted by automatic scheduling, so smaller slices and admission intervals normally produce smoother short-window scheduling. ### Jitter and `random-seed` The three jitter fields sample uniformly from an inclusive range: ```text sampled = uniform(max(minimum, base - jitter), base + jitter) ``` The minimum is `1us` for task and slice CPU and `0us` for blocking wall time. A jitter value of zero uses the base value exactly. - `task-cpu-jitter-us` is sampled once when an offered task is submitted. - `slice-cpu-jitter-us` is sampled independently for every logical slice. - `blocking-wall-jitter-us` is sampled independently for every logical slice. Samples are derived from `random-seed`, the task-runtime identity, its offered sequence number, the logical-slice index, and the sample kind. A global-admission yield does not resample the current slice target or advance this index; a normal slice reschedule does. Worker execution order therefore does not change the sampled targets even though actual dispatch order and the amount of busy-loop filler may differ. `random-seed` does not make an entire run deterministic. Worker races, CPU frequency, host load, and OS scheduling remain real-time effects. ### Stopping At the global `duration`, the simulator stops producing open-loop arrivals and closed-loop replacements. - `drain-on-stop = true` waits for every submitted task to finish all remaining slices before destroying the pool. - `drain-on-stop = false` sets a shared cancellation flag. Submitted tasks exit at their next cooperative check between CPU chunks or slice work, and the simulator waits for those cooperative exits before destroying the pool. Cancellation cannot interrupt a CPU burn or synchronous sleep already running inside a poll. Consequently, `false` shortens shutdown but is not an instantaneous hard abort. After draining or cooperative cancellation completes, the simulator freezes the report data and keeps the read pool alive without submitting work for `cooldown-duration`. It then shuts down the pool, writes the reports, and exits. This idle cooldown defaults to `5s`, is excluded from periodic windows and all summary phases, and can be disabled with `cooldown-duration = "0s"`. ## Metrics and reports The collector and machine-readable JSON separate measurements into four phases: - `warmup`: `[0, warmup-duration)`, omitted when `warmup-duration = "0s"`; - `measurement`: `[warmup-duration, duration)`, used by the main report tables; - `drain`: after generation stops until all submitted tasks finish or cooperatively cancel, omitted when no drain interval is observed; and - `total`: the complete measured run, including warmup and drain but excluding the final cooldown. Periodic `windows` use the same phase labels and target `stats-print-interval`. These are real-time snapshots, so an interval boundary can be slightly later than its target if the workload controller is busy. ### Metric sources Most report metrics are owned by the simulator and keyed by one expanded `(keyspace, task class)` runtime. Queue length is the exception: it is read from the production `KeyspaceMetrics` fair-queue histogram and reported once per keyspace. Scheduler fidelity comes from the real YATP queues and fairness-accounting path. | Metric | Source | Scope | Fairness off | | --- | --- | --- | --- | | task-poll CPU time | simulator YATP poll observer | keyspace and task class | available | | task-poll wall time | simulator YATP poll observer | keyspace and task class | available | | task-slice queue wait duration | simulator YATP dispatch observer | keyspace and task class | available | | fair-queue task-slice waits | simulator YATP dispatch observer | keyspace and task class | unavailable | | queued task slices before enqueue | production `KeyspaceMetrics` | keyspace | unavailable | | offered, submitted, suppressed, completed, cancelled | simulator controller | keyspace and task class | available | | whole-task latency | simulator completion path | keyspace and task class | available | | slow-task poll timeline | YATP poll tracker observer | individual completed task and poll | available | Fair-queue wait counts are empty and displayed as `N/A` when fairness is disabled. They are not reported as zero, because the regular multilevel queue does not provide that dispatch metadata. Queue-wait duration remains available for both queue modes. The histogram meanings are specific: - keyspace queue length is the number of slices from all task classes in the same keyspace already queued immediately before a new slice is enqueued; - task-slice waits is the number of successful pops from the whole fair queue while that slice was waiting, not only pops from the same keyspace; and - task-slice wait duration is the dispatch queue wait reported by YATP, from a slice entering the selected queue until a worker starts its dispatch. Task-poll CPU and wall time cover time inside `Future::poll()` only. They do not include time waiting in a queue. The whole-task latency starts immediately before a successful `ReadPoolHandle::spawn_with_options()` and ends when the task future exits, so it includes initial queueing, later requeueing, CPU work, synchronous blocking, and yield cycles. Cooperatively cancelled tasks increment `cancelled` but are excluded from the completed-task latency histogram and the slow-task list. The YATP poll tracker records CPU, wall, queue wait, and fair-queue waited-slice count into task-runtime collectors for every poll. It retains the complete per-poll timeline only for task classes with a non-zero slow threshold. Separately, the simulator registers every YATP worker's native thread handle and reads the worker thread CPU clocks at each snapshot. `total_cpu_seconds` and `total_cpu_cores` come from the delta of those thread clocks, rather than from summing task-runtime CPU. They therefore include scheduler, queue, and other worker-thread CPU outside the observed task polls. Native worker CPU sampling is supported on Linux and macOS; the simulator rejects startup on other platforms. Keyspace rows in JSON are derived by merging these task-runtime collectors, except for `fair_queue_length`, which is a window delta of the production keyspace histogram. It is `null` when keyspace fairness is disabled. Histogram percentiles are bucket upper-bound approximations, not exact sample percentiles. JSON distributions include the raw cumulative buckets and an `overflow_count` so another tool can reinterpret them. If a percentile rank falls above the largest finite bucket, that percentile is `null`/`N/A` instead of being understated. The production keyspace queue-length histogram uses `1, 2, 4, ... 2048` finite buckets. The simulator-owned waited-slice histogram uses `0, 1, 2, 4, ... 2048`, while queue-wait duration uses `0s, 10us, 40us, ... 41.94304s`. Larger values contribute to `overflow_count` through the implicit `+Inf` bucket. A task that spans a window boundary is counted in the window where its completion is observed; therefore a window can have more completions than submissions because of tasks carried in from the previous window. Snapshots are not a stop-the-world operation: workers continue updating the production keyspace queue histogram and task-runtime collectors while the controller reads them. The final poll metrics are recorded before its completion notification, so under a completion backlog poll data may appear one window earlier than the corresponding lifecycle completion, but not later. ### Report files Both output files contain the same `ReportData` model: - `config`: the effective configuration, including defaults; - `keyspaces` and `task_runtimes`: the expanded workload identities; - `windows`: periodic warmup, measurement, and drain samples; and - `summary`: aggregate `warmup`, `measurement`, optional `drain`, and `total` windows, plus numbered `workload_phases` split at task start and end boundaries; and - `slow_tasks`: completed tasks whose whole-task latency strictly exceeded their non-zero threshold, including their ordered poll timelines. The top-level `schema_version` is `1`. Prefer the JSON file for scripts, AI analysis, and comparisons between fairness-on and fairness-off runs. The HTML file is standalone and focuses on warmup and measurement; it does not render the drain summary or drain interval points. The embedded JSON keeps drain data so scripts can still account for tasks that finish after generation stops. The HTML includes the workload-phase summary and full-width time-series charts, followed by an always-visible slow-task section. Slow tasks are grouped by `{keyspace}/{task}` workload. The section itself is not collapsible: it immediately shows the explanation, legend, and one collapsed summary for each workload group. Tasks within a group are ordered by submission time. Opening a workload group creates its collapsed task summaries, and each poll timeline is created only when its task is first opened, so large reports do not eagerly create every task timeline. A group header shows its workload and slow-task count. Each task summary is a single bold line showing only the workload, simulator-relative start, whole-task duration, and poll count; task IDs, runtime IDs, offer sequence, threshold, and the redundant interval are omitted. Opening a task first shows a task overview with its simulator-relative start, whole-task duration, summed wait, queue-wait, other-wait, poll-wall, and poll-CPU times, and total fair-queue waited slices. An immediate repoll with no slice count contributes zero to that total. If any queued dispatch has no slice-count observation, the total is `N/A`. Overview durations use approximately four significant digits. Each task timeline is one contiguous, full-width bar with one selectable item per poll. Every item shows its whole block duration and divides the item into the total pre-poll wait followed by the actual `Future::poll()` wall time while remaining a single click target. The static timeline example gives these two parts equal width as a color key; real poll items use their measured durations. Clicking a poll highlights that item and opens a second, full-width bar. Its wait and poll-wall phases are distributed by measured duration, with their displayed ratio capped at `6:1` in either direction. Their labels, tooltips, and details retain the original measurements. The second bar renders the wait composition with other pre-poll time on the left and scheduler queue wait on the right, then splits poll wall time into CPU and non-CPU wall shares. Parts within each phase have no minimum display width and keep their measured proportions. When the two non-zero parts have a ratio strictly greater than `10:1`, the smaller part keeps its proportional color, tooltip, and accessibility label but hides its visible text. The queue-wait segment also reports how many fair-queue task slices were dispatched while it waited. These inner widths are duration compositions rather than execution-order timelines: other wait is an aggregate remainder, and CPU time may be interleaved with non-CPU time. If measured CPU exceeds poll wall time, the bar caps only its layout width at poll wall while retaining the raw CPU duration and labeling the segment `layout capped`. The selected-poll details show only its poll and dispatch numbers, wait and poll durations, CPU/wall ratio, and number of fair-queue task slices dispatched ahead of it. Task-wide timings remain in the task header and overview instead of being repeated for every poll. The other pre-poll time includes wake waiting and wrapper time that cannot be attributed to the scheduler queue. The wait count is `N/A` when fair-queue information is unavailable. Durations and ratios use approximately four significant digits. Very short top-level poll items receive a minimum display width, while the selected poll's inner parts remain strictly proportional to their measured durations. The time-series charts appear in this order: 1. `Task latency avg`; 2. `Task latency p99`; 3. `Task-slice wait p99`; 4. `Task QPS`; 5. `CPU consume`; 6. `Fair-queue task-slice waits p99`; and 7. `Keyspace queue length p99`. The first six charts use `{keyspace}-{task}` series from `window.tasks`. `CPU consume` additionally shows a `total` series read directly from the YATP worker thread CPU clocks. The last chart uses one series per keyspace from `window.keyspaces` and reads the production queue-length metric. Histogram p99 values remain `N/A` when they fall above the largest finite bucket. Time-valued axes, tick labels, and tooltips automatically select `us`, `ms`, or `s` from the displayed range. Axis ticks keep only useful significant digits, while tooltips retain more precision for exact inspection. Hovering over a chart shows the current values for its visible series. Clicking a legend label focuses the chart on that series; clicking the only visible label again restores all series. The configuration and interval-detail table are collapsed by default. The report does not load a CDN or other network resource. Its markup, CSS, and JavaScript are maintained separately in [`templates/report.html`](templates/report.html); Rust only serializes and safely embeds the report data. ## How the simulator works The simulator uses the real CSE read-pool and YATP queue paths. It does not implement its own scheduler. ```mermaid flowchart LR G["Open/closed-loop generator"] --> S["ReadPoolHandle::spawn_with_options"] S --> Q{"Fairness enabled?"} Q -->|yes| F["Keyspace-fair queue"] Q -->|no| M["Regular multilevel queue"] F --> P["Worker polls the task future"] M --> P P --> D{"Task complete or cancelled?"} D -->|yes| C["Notify workload controller"] D -->|no| Y["yield_to_scheduler"] Y --> Q ``` With fairness enabled, `YatpKeyspaceFairQueue` groups tasks by keyspace. The outer queue selects a non-empty keyspace by virtual time; that keyspace's multilevel inner queue selects a task. `ReadPoolHandle` wraps every task. In fair mode, that wrapper accounts each `Future::poll()` using thread CPU or wall time. Optional precharge is applied and reconciled by the same production path: ```text poll start: virtual_time += slice_precharge_us poll finish: virtual_time += measured_us - slice_precharge_us ``` When measured time is below the precharge, the production downward-correction path is used. With fairness disabled, the same generated futures run through the regular multilevel queue, so queue selection is the intended difference. The simulator submits tasks with `spawn_with_options`. A YATP poll tracker receives dispatch metadata before each poll and CPU/wall measurements afterward; it maintains task-runtime CPU, wall, queue-wait, and fair-queue wait-count metrics and emits completion after the final poll. When `slow-threshold` is non-zero, the same tracker retains the poll sequence until the task finishes, then discards it for a normal task or moves it into `slow_tasks` for a completed slow task. Fair-global CPU admission metadata and the poll tracker are attached directly through `ReadPoolTaskOptions`. In fair mode, the production keyspace inner queue records its current length in `KeyspaceMetrics` immediately before every task-slice push. Initial submissions and cooperative requeues from all task classes in the same keyspace therefore contribute to one shared queue-length histogram. The workload controller uses one runtime for each `(keyspace, task class)`. It owns arrival deadlines, outstanding counts, and offered sequence numbers. The same controller records task lifecycle counts and whole-task latency. A completion notification is delivered only after the simulator observer has recorded that task's final poll. At a capture deadline, the controller reads the cumulative metrics immediately rather than delaying the snapshot to empty a busy completion queue. Sampling continues during drain, and the final summary is captured before the read pool is shut down. Terminal interval output is emitted once per keyspace and task class rather than as one run-wide aggregate. ## Usage notes - This is a real-time workload runner, not a deterministic discrete-event simulator. - Keyspace fairness is equal-weight and work-conserving; it is not a CPU hard cap. - Scheduling is cooperative, and a running poll cannot be preempted by queue fairness. - Priority affects ordering within a selected keyspace. It does not bypass cross-keyspace virtual-time ordering in fair mode. - Optional fair-global CPU admission is independent of keyspace queue fairness; it can be enabled or disabled through the read-pool configuration. ## Relevant implementation Before changing scheduling behavior, check: - [`src/read_pool.rs`](../../src/read_pool.rs), which wraps read-pool tasks and accounts each poll; - [`components/tikv_util/src/yatp_pool/mod.rs`](../../components/tikv_util/src/yatp_pool/mod.rs), which constructs YATP pools and implements the keyspace-fair queue; - [`components/tikv_util/src/fairness.rs`](../../components/tikv_util/src/fairness.rs), which provides shared virtual-time primitives; and - [`components/tikv_util/benches/yatp_fair_pool/mod.rs`](../../components/tikv_util/benches/yatp_fair_pool/mod.rs), which is the existing real-YATP benchmark and pool-construction reference. The resource-control simulation introduced by [`e8d3100ffd6f8fb59934db456c3479da5bd3ee4a`](https://github.com/tidbcloud/cloud-storage-engine/commit/e8d3100ffd6f8fb59934db456c3479da5bd3ee4a) is a useful workload-configuration reference, but its token admission and retry loop do not model YATP queue fairness.