# Big Query Interference Workload SOP

## 1. Purpose

This SOP runs securities foreground traffic beside one or more large,
report-shaped read queries. It is intended to verify that a large query cannot
consume unbounded RU when the foreground workload is already close to the
configured Max RCU/RU budget.

TiPoC records query execution evidence, but RU containment must be confirmed
with control-plane RU metrics from the same time window.

## 2. Preconditions

Before starting a run, confirm:

1. The target TiDB cluster has the securities/TPC-E schema and prepared data.
2. The control plane Max RCU is configured, for example `5000`.
3. The foreground target is configured below the ceiling, for example about
   `4000 RU`.
4. Pressure nodes, SQL endpoints, credentials, and the object store are
   reachable.
5. The run has enough duration to observe foreground stabilization, query
   startup, throttling/cancellation, and recovery after the large query.

Recommended minimum timing:

```yaml
pressure:
  duration_seconds: 300
  warmup_seconds: 60
  cooldown_seconds: 60
```

With the default big-query delay of 60 seconds, report queries start 60
seconds after the foreground enters its ready/stable window.

## 3. Configuration

Declare Big Query as a canonical operation in the source spec:

```yaml
operations:
  - id: big_query
    enabled: true
    runtime:
      concurrency: 3
      query_ids: [report_01, report_06, report_09]
      start_after_foreground_ready_seconds: 60
      duration_seconds: 180
      query_timeout_seconds: 60
```

| Runtime field | Meaning | Default/constraint |
| --- | --- | --- |
| `enabled` | Enables or disables the operation at init | Must be enabled at init to run |
| `start_after_foreground_ready_seconds` | Delay after `foreground_ready` | `60`, non-negative |
| `concurrency` | Total in-flight large-query workers | `1` to `10` |
| `query_ids` | Built-in report IDs | All `report_01` to `report_10` |
| `duration_seconds` | Big Query operation window | Foreground duration when omitted |
| `query_timeout_seconds` | Per-query timeout | Positive |

Run-time changes use the generic canonical-ID keyed override document:

```json
{
  "securities/tpce/big_query": {
    "concurrency": 3,
    "query_ids": ["report_01", "report_06", "report_09"],
    "start_after_foreground_ready_seconds": 60
  }
}
```

```bash
tipoc run \
  --manifest runs/big-query-c03/attempt-001/run-profile-manifest.json \
  --operation-runtime-overrides runs/big-query-c03/runtime-overrides.json
```

The override never rewrites the init manifest. The resolved typed runtime is
recorded in `run/operation-runtime-resolution.json` and referenced by the
generic operation artifact index.

## 4. Query and node allocation

There are ten built-in read-only report queries: `report_01` through
`report_10`. The complete SQL text is defined in
`projections/tidb/operations/securities/big_query/statements.go`.

`concurrency` controls the number of simultaneous query workers. It is
independent from the number of selected query IDs.

For eight pressure nodes, `concurrency=3`, and
`query_ids=report_01,report_06,report_09`, the deterministic allocation is:

| Pressure-node order | Big query |
| --- | --- |
| First node with a pressure slot | `report_01` |
| Second node with a pressure slot | `report_06` |
| Third node with a pressure slot | `report_09` |
| Remaining five nodes | No big-query worker |

Node order is the sorted pressure-node order from the manifest, not SSH launch
completion order.

When the number of selected IDs differs from the concurrency, workers follow
the query-list dispatch rules: the first batch follows list order, completed
workers take the next ID, and IDs are reused cyclically as needed. For a
multi-node run, follow-up IDs are partitioned by global worker ownership into
non-overlapping node-local sequences. The next completed worker on each node
takes the next ID in that node's sequence; the coordinator is not involved in
per-query dispatch. A single-node run therefore retains the original global
completion queue. When the number of nodes is smaller than the global
concurrency, the worker count is split across nodes so the total remains equal
to the configured concurrency.

## 5. Standard execution procedure

1. Set the control-plane Max RCU/RU limit and record its value.
2. Start the foreground securities workload with the selected mix, usually
   `exchange_matching`, and target approximately 80% of the ceiling.
3. Allow the foreground workload to reach `foreground_ready`.
4. Let TiPoC wait for the configured ready delay, normally 60 seconds.
5. TiPoC starts the selected report queries through an independent SQL pool.
6. Observe the control-plane RU rate, foreground latency, errors, and query
   cancellation/throttling during the interference window.
7. Wait for the interference window and foreground cooldown to finish.
8. Preserve the run artifacts before cleanup.

The public stage sequence remains:

```text
init -> prepare -> run -> cleanup
```

Big-query interference is part of `run`; it is not a separate public stage.

## 6. Ten-case concurrency matrix

Run the same foreground and cluster configuration for each case, changing only
the big-query concurrency:

| Case | Concurrency |
| --- | ---: |
| `securities-big-query-c01` | 1 |
| `securities-big-query-c02` | 2 |
| `securities-big-query-c03` | 3 |
| `securities-big-query-c04` | 4 |
| `securities-big-query-c05` | 5 |
| `securities-big-query-c06` | 6 |
| `securities-big-query-c07` | 7 |
| `securities-big-query-c08` | 8 |
| `securities-big-query-c09` | 9 |
| `securities-big-query-c10` | 10 |

Use the testcase contract in
`testcase/securities-big-query-interference-10case.json` as the base matrix.
For focused validation of the node-to-query mapping, run c03 with
`report_01,report_06,report_09` explicitly selected.

## 7. Required evidence

Collect all evidence using a common UTC time window and the same run ID:

### TiPoC artifacts

- `run/operation-runtime-resolution.json`
- `run/operation-artifacts.json`
- `run/operations/securities/tpce/big_query/big-query-receipt.json`
- worker and node receipts referenced by the operation artifact index
- `run/live-runtime-execution.json`
- `run/run-receipt.json`
- `run/summary.json`
- `run/run-progress.json` and `run/run-progress.jsonl`, when available

`run-receipt.json` and `live-runtime-execution.json` contain only
`operation_artifact_index_ref`; Big Query details live in the
operation-owned receipt. That receipt includes:

- resolved concurrency and query IDs;
- `started_at` and `finished_at`;
- started, completed, failed, and canceled query counts;
- per-query elapsed-time and error summaries.

### Control-plane and cluster metrics

Record, at minimum:

- configured Max RCU/RU;
- total RU rate and peak RU rate;
- foreground RU rate;
- foreground p95/p99 latency and error rate;
- query-level throttling, cancellation, timeout, or rejection signals;
- TiDB/TiKV CPU, memory, connection, and coprocessor indicators when
  available.

## 8. Pass/fail criteria

Pass only when all applicable conditions are satisfied:

1. The combined workload stays within the configured control-plane envelope,
   or the control plane explicitly throttles/rejects/cancels excess work.
2. A large report query cannot silently consume several times the configured
   Max RCU and leave the budget depleted for later requests.
3. Foreground small reads and matching operations remain within the agreed
   latency and error-rate thresholds.
4. The receipt timestamps and control-plane metrics overlap and can be
   correlated by run ID/time window.
5. The actual query IDs and concurrency match the requested case.

Fail or mark the result inconclusive when the control-plane metric window is
missing, the foreground was not near the target load, or query execution
evidence cannot be correlated with the run.

## 9. Safety and abort procedure

- The built-in statements are read-only `SELECT` statements.
- Each query has a deadline and the interference window is bounded.
- Do not add ad-hoc SQL to this workload without a new reviewed query ID.
- Stop the run through the normal process/context cancellation path if the
  foreground error rate or cluster health becomes unsafe.
- Preserve the run directory and control-plane metric export before invoking
  cleanup.

If a run is interrupted, record the interruption time and reason. A partial
receipt is evidence of an interrupted run, not a successful containment result.

## 10. Troubleshooting

| Symptom | Check |
| --- | --- |
| No report queries started | `enabled`, query IDs, SQL endpoint, and `foreground_ready_at` |
| Queries start too early | warmup, ready delay, and receipt timestamps |
| Fewer workers than requested | pressure slots, node allocation, and distributed runner logs |
| Query failures | per-query `last_error`, schema/data readiness, and query timeout |
| RU result cannot be judged | export control-plane RU metrics for the exact receipt window |
| Foreground degradation | compare c01–c10 against the no-interference baseline |