# TiPOC Deployment POC SOP

This is the main runbook. Keep the public product flow exactly:

```text
init -> prepare -> run -> cleanup
```

`rerun` is an operator wrapper around the same public `run` stage. It creates a
new try directory under the same attempt. It is not a fifth public product
stage.

Use the SOP files in this order:

| Step | File | Use it for |
| ---: | --- | --- |
| 1 | `poc-intake-form.md` | Freeze the POC inputs before any command runs. |
| 2 | `node-and-time-planning.md` | Pick generation nodes, pressure nodes, and starting duration/concurrency values. |
| 3 | `environment-setup.md` | Create EC2, S3, SQL, and observability bindings. |
| 4 | `deployment-poc-sop.md` | Run `init`, `prepare`, `run`, optional same-stage rerun tries, and `cleanup`. |

Two execution modes are supported:

| Mode | Rule |
| --- | --- |
| Manual | A human fills the form and runs the commands. |
| AI-assisted | AI prepares non-secret commands and checks evidence; a human owns secrets, approvals, and external actions. |

Do not paste passwords, private keys, access keys, SQL DSNs, Clinic tokens, or
Grafana cookies into these documents. Use references such as `env:TIDB_DSN`,
`env:AWS_PROFILE`, instance roles, secret-file paths, or secret-manager ARNs.

## 0. Quick Rules

1. Run all commands from the `tipoc-tv` repository root.
2. `tipoc run` accepts only `--manifest`.
3. Changes to `spec.yaml` after `init` do not affect the already compiled
   manifest used by `run`.
4. Use `scripts/rerun/run-again.sh` when you want another pressure pass after a
   completed run.
5. Same-attempt rerun is continuation evidence. It becomes clean comparison
   evidence only after an approved prepare-end BR restore or a fresh prepare
   into a clean target.
6. Create a new attempt when you change scale, source generation, prepare
   policy, restore policy, or any other prepare-bound setting that should start
   from a clean prepare boundary.
7. Clinic and Loki are evidence inputs only. They are not product inputs to
   `tipoc init`, `tipoc prepare`, or `tipoc run`.
8. For Cloud target-managed Global Sort, set
   `TIPOC_GLOBAL_SORT_URI_REF=target-managed`. Do not let Cloud POC examples
   render a user-side Global Sort workspace from the source object prefix.

## 1. Prepare The Environment

Complete the environment page before `init`:

1. Choose one SQL path: local TiUP, public endpoint, or PrivateLink.
2. Create or confirm EC2 nodes in the same region and VPC path as the TiDB
   endpoint and S3 bucket.
3. Create the S3 bucket and attempt prefix.
4. Confirm TiDB can read the source prefix for import. EC2 S3 read/write proof
   is not enough.
5. Record any observability bindings you want for evidence.

### 1.1 SQL Path

| Deployment | SQL path | Minimum check |
| --- | --- | --- |
| TiUP | local `127.0.0.1:4000` | Start TiUP and verify SQL. |
| Essential | public endpoint or PrivateLink | Verify EC2-to-TiDB SQL. |
| Premium | public endpoint or PrivateLink | Verify EC2-to-TiDB SQL. |

Cloud SQL-only default:

```bash
TIPOC_PD_ENDPOINT=''
TIPOC_GLOBAL_SORT_URI_REF=target-managed
TIPOC_BR_ENABLED=false
```

Use that default unless the environment owner provides an approved PD endpoint,
managed backup path, or self-managed Global Sort workspace.

Public endpoint check from each EC2 node:

```bash
SQL_HOST='<public-sql-host>'
SQL_PORT='4000'

curl -fsS https://checkip.amazonaws.com
getent hosts "${SQL_HOST}" || nslookup "${SQL_HOST}"
nc -vz "${SQL_HOST}" "${SQL_PORT}"
mysql --ssl-mode=REQUIRED \
  -h "${SQL_HOST}" \
  -P "${SQL_PORT}" \
  -u "${TIDB_USER}" \
  -p \
  -e 'select tidb_version();'
```

PrivateLink check from each EC2 node:

```bash
SQL_HOST='<privatelink-sql-host>'
SQL_PORT='4000'

getent hosts "${SQL_HOST}" || nslookup "${SQL_HOST}"
nc -vz "${SQL_HOST}" "${SQL_PORT}"
mysql --ssl-mode=REQUIRED \
  -h "${SQL_HOST}" \
  -P "${SQL_PORT}" \
  -u "${TIDB_USER}" \
  -p \
  -e 'select tidb_version();'
```

If the PrivateLink service requires acceptance, first create the AWS Interface
VPC Endpoint without private DNS, register the generated `vpce-...` endpoint id
in TiDB Cloud, wait for `available`, then enable private DNS or use one of the
AWS endpoint DNS names returned by `describe-vpc-endpoints`. Do not build
`TIPOC_SQL_ENDPOINT` from the endpoint id itself.

Wrapper binding:

```bash
TIPOC_SQL_ENDPOINT='<host>:4000'
TIPOC_TIDB_CREDENTIALS_REF='env:TIDB_DSN'
```

### 1.2 EC2 Nodes

Each EC2 node needs:

| Item | Requirement |
| --- | --- |
| OS | Amazon Linux 2023 or Ubuntu 22.04 |
| CPU architecture | Match the `tipoc` binary |
| Disk | Enough for logs, receipts, binary, and temporary files |
| Network | Outbound SQL `4000`, S3, SSH, and observability endpoints |
| IAM | Prefer an instance role scoped to the attempt S3 prefix |

Before long-running `prepare` or `run`, verify the deployed binary on every
generation and pressure node:

```bash
TIPOC_BIN='/data/tipoc/tipoc'
EXPECTED_ARCH='aarch64'       # or x86_64
EXPECTED_SHA256='<sha256>'

uname -m
file "${TIPOC_BIN}"
sha256sum "${TIPOC_BIN}"
test "$(uname -m)" = "${EXPECTED_ARCH}"
test "$(sha256sum "${TIPOC_BIN}" | awk '{print $1}')" = "${EXPECTED_SHA256}"
```

Do not start the stage when any node has the wrong architecture, a missing
binary, a non-executable binary, or a different SHA-256 digest.

For external pressure fleets, use the standard deployment helper instead of an
ad hoc SSH loop when replacing the remote binary:

```bash
scripts/rerun/deploy-tipoc-binary.sh \
  --local-binary ./tipoc \
  --remote-binary /data/tipoc/tipoc \
  --node-file <local-node-inventory> \
  --ssh-key <local-private-key> \
  --ssh-user ec2-user \
  --expected-arch aarch64 \
  --bastion <bastion-ssh-target-if-needed>
```

The helper uses the same node inventory shape as external run launchers:

```text
node_id ssh_host ssh_mode private_ip expected_slots expected_sql_pool
```

It first checks every node for the expected architecture and for no active
`tipoc run` process. Only after all nodes pass does it upload the binary,
atomically replace the remote file, set executable mode, and verify the remote
SHA-256 matches the local binary.

Create nodes with AWS CLI if needed:

```bash
AWS_REGION='<region>'
AMI_ID='<ami-id>'
INSTANCE_TYPE='<instance-type>'
KEY_NAME='<ec2-keypair-name>'
SUBNET_ID='<subnet-id>'
SECURITY_GROUP_ID='<security-group-id>'
INSTANCE_PROFILE='<instance-profile-name>'
POC_NAME='<poc-name>'

aws ec2 run-instances \
  --region "${AWS_REGION}" \
  --image-id "${AMI_ID}" \
  --instance-type "${INSTANCE_TYPE}" \
  --key-name "${KEY_NAME}" \
  --subnet-id "${SUBNET_ID}" \
  --security-group-ids "${SECURITY_GROUP_ID}" \
  --iam-instance-profile Name="${INSTANCE_PROFILE}" \
  --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":200,"VolumeType":"gp3","DeleteOnTermination":true}}]' \
  --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=${POC_NAME}-node-a},{Key=Project,Value=tipoc-poc}]"
```

Example node lists:

```bash
TIPOC_GENERATION_NODES='node-a=10.0.0.10:15,node-b=10.0.0.11:15'
TIPOC_PRESSURE_NODES='node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512'
TIPOC_PRESSURE_NODES='node-a=10.0.0.10:1:512:256,node-b=10.0.0.11:1:512:256'
```

### 1.3 S3 Bucket And Prefix

Create the bucket:

```bash
AWS_REGION='<region>'
BUCKET='<globally-unique-bucket-name>'

aws s3api create-bucket \
  --bucket "${BUCKET}" \
  --region "${AWS_REGION}" \
  --create-bucket-configuration LocationConstraint="${AWS_REGION}"
```

For `us-east-1`:

```bash
aws s3api create-bucket \
  --bucket "${BUCKET}" \
  --region us-east-1
```

Block public access and enable default encryption:

```bash
aws s3api put-public-access-block \
  --bucket "${BUCKET}" \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

aws s3api put-bucket-encryption \
  --bucket "${BUCKET}" \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
```

Create attempt prefixes:

```bash
DEPLOYMENT='<essential-or-premium>'
RUN_ID='<run-id>'
ATTEMPT_ID='attempt-001'
BASE="tipoc/${DEPLOYMENT}/${RUN_ID}/${ATTEMPT_ID}"

for prefix in source br scratch evidence; do
  aws s3api put-object \
    --bucket "${BUCKET}" \
    --key "${BASE}/${prefix}/.keep" \
    --body /dev/null
done
```

Wrapper binding:

```bash
TIPOC_OBJECT_STORE_KIND='s3'
TIPOC_OBJECT_STORE_URI="s3://${BUCKET}/${BASE}"
TIPOC_OBJECT_STORE_CREDENTIAL_REF='env:AWS_PROFILE'
AWS_PROFILE='<profile-name>'
AWS_REGION='<region>'
```

Validate EC2 S3 access:

```bash
aws s3 ls "s3://${BUCKET}/${BASE}/"
echo probe > /tmp/tipoc-s3-probe.txt
aws s3 cp /tmp/tipoc-s3-probe.txt "s3://${BUCKET}/${BASE}/source/probe.txt"
aws s3 cp "s3://${BUCKET}/${BASE}/source/probe.txt" -
aws s3 rm "s3://${BUCKET}/${BASE}/source/probe.txt"
```

The TiDB import path must also read the same prefix during `prepare`. If TiDB
cannot read the prefix, stop and fix that before `init`.

### 1.4 Observability

Clinic and Loki are evidence collection inputs. They are not product inputs.

Record them only if you want them for the final report:

| Binding | Example |
| --- | --- |
| Clinic cluster id | `<cluster-id>` |
| Clinic credential reference | `env:CLINIC_API_KEY` |
| Clinic environment | `prod` or approved target |
| Loki/Grafana access | Explore URL, redacted cURL, or exported JSON |

## 2. Run `init`

Pick one wrapper and one template:

| Deployment | Wrapper | Template |
| --- | --- | --- |
| TiUP | `scripts/env/tiup/init-attempt.sh` | `scripts/env/templates/tiup.spec.yaml` |
| Essential | `scripts/env/essential/init-attempt.sh` | `scripts/env/templates/essential.spec.yaml` |
| Premium | `scripts/env/premium/init-attempt.sh` | `scripts/env/templates/premium.spec.yaml` |

### 2.0 Workload Selection

The workload registry has four operator-facing user labels. Select the label in
the intake form, then use the matching `tipoc init --template` value or render
an equivalent spec that resolves to the same registry entry.

| user label | init template | industry | model | workload mix | init | prepare | run |
| --- | --- | --- | --- | --- | --- | --- | --- |
| `tpce` | `securities-tpce` | `securities` | `tpce` | `tpce` | `supported_now` | `supported_now` | `supported_now` |
| `exchange` | `securities-exchange-matching` | `securities` | `tpce` | `exchange_matching` | `supported_now` | `supported_now` | `supported_now` |
| `tpce-read-only` | `securities-tpce-read-only` | `securities` | `tpce` | `tpce_read_only` | `supported_now` | `supported_now` | `supported_now` |

`tpce`, `exchange`, and `tpce-read-only` are supported now for `init`,
`prepare`, and `run`. Follow [`../tpce-read-only-sop.md`](../tpce-read-only-sop.md)
for its traffic-class and mutation-guard acceptance checks.

Direct template examples:

```bash
tipoc init --template securities-tpce --out runs/<run-id>/attempt-001
tipoc init --template securities-exchange-matching --out runs/<run-id>/attempt-001
tipoc init --template securities-tpce-read-only --out runs/<run-id>/attempt-001
```

For wrapper-rendered specs, set the workload fields consistently. Do not mix a
label from one row with source profile, runtime profile, or workload mix from
another row.

Wrapper defaults:

| Deployment | Default import thread | Default scale |
| --- | ---: | --- |
| TiUP | `16` | `50G` |
| Essential | `24` | `1T` |
| Premium | `32` | `1T` |

All wrappers write `runs/<run-id>/<attempt-id>/spec.yaml` and then run
`tipoc init` unless `--render-only` is set.

Wrapper options:

| Option | Meaning |
| --- | --- |
| `--run-id VALUE` | Set the run id. |
| `--attempt-id VALUE` | Set the attempt id, usually `attempt-001`. |
| `--out PATH` | Attempt output root. |
| `--scale VALUE` | Target data size, for example `50G` or `1T`. |
| `--tipoc PATH` | `tipoc` binary path. Default is `tipoc`. |
| `--remote-bootstrap plan` | Render a remote bootstrap plan. This is the safe default. |
| `--remote-bootstrap execute` | Execute approved remote bootstrap actions. |
| `--render-only` | Render `spec.yaml` and stop before `tipoc init`. |
| `--help` | Print wrapper help. |

Common wrapper environment bindings:

| Variable | Use it for |
| --- | --- |
| `TIPOC_SQL_ENDPOINT` | TiDB SQL endpoint in `host:port` form. |
| `TIPOC_TIDB_CREDENTIALS_REF` | SQL credential reference such as `env:TIDB_DSN`. |
| `TIPOC_PD_ENDPOINT` | Keep empty for Cloud SQL-only POC unless an approved PD path exists. |
| `TIPOC_GLOBAL_SORT_URI_REF` | Use `target-managed` for Cloud target-managed Global Sort; use a base workspace URI only for self-managed paths. |
| `TIPOC_BR_ENABLED` and `TIPOC_BR_STORAGE_URI` | Turn prepare-end BR baseline on or off. |
| `TIPOC_OBJECT_STORE_KIND`, `TIPOC_OBJECT_STORE_URI`, and `TIPOC_OBJECT_STORE_CREDENTIAL_REF` | Source-object destination and credentials. |
| `TIPOC_GENERATION_NODES` | Generation-node list for CSV source-object creation. |
| `TIPOC_PRESSURE_NODES` | Physical pressure-node topology and per-node SQL pool. |
| `TIPOC_PRESSURE_CONCURRENCY` | Total requested concurrency across pressure nodes. |
| `TIPOC_PRESSURE_DURATION_SECONDS`, `TIPOC_PRESSURE_WARMUP_SECONDS`, `TIPOC_PRESSURE_COOLDOWN_SECONDS`, and `TIPOC_PRESSURE_PROGRESS_INTERVAL_SECONDS` | Runtime window controls. |
| `TIPOC_PREPARE_OPS` | Prepare op set such as `generate,import,add-index`. |
| `TIPOC_WORKLOAD_MIX` and `TIPOC_WORKLOAD_PROFILE` | Runtime mix and workload profile. |
| `TIPOC_SSH_USER`, `TIPOC_SSH_PRIVATE_KEY_REF`, and `TIPOC_REMOTE_ATTEMPT_ROOT` | Remote bootstrap identity and remote attempt path. |

#### Securities Hot Range Workload Profile

Use a hot range profile when the test goal is a business hotspot rather than a
canonical TPC-E reference run. The hotspot is applied before SQL generation: the
runtime selects hot transaction lineage first, and `insert`, `update`, and read
statements consume that same lineage. Do not model this as ad hoc SQL
predicates or read-only query filtering.

The current environment wrappers do not have first-class flags for the new hot
range fields. Render the spec first, edit `pressure.workload_profile` in
`runs/<run-id>/<attempt-id>/spec.yaml`, then run `tipoc init`. Do not add hot
range fields to a profile named `canonical`; canonical profiles must stay free
of hotspot parameters.

Example: concentrate a 10% symbol range at the default 10x density target.

```yaml
pressure:
  workload_mix: tpce
  workload_profile:
    name: hot-range
    hot_dimensions:
      - symbol
    hot_range_percent: 10
    hot_range_placement: middle
    hot_density_ratio_target: 10
```

With `hot_range_percent: 10` and `hot_density_ratio_target: 10`, TiPOC derives
about `52.63%` of transaction pressure into the selected 10% range. Supported
`hot_range_placement` values are `front`, `middle`, `back`, and `segments`.
`hot_range_percent` must be greater than 0 and no more than 50.

For a segmented hotspot, the segment intervals are half-open percentage ranges:
`[start_percent, start_percent + width_percent)`. Segments must not overlap and
their widths must sum to `hot_range_percent`.

```yaml
pressure:
  workload_mix: tpce
  workload_profile:
    name: hot-range
    hot_dimensions:
      - symbol
    hot_range_percent: 10
    hot_range_placement: segments
    hot_range_segments:
      - start_percent: 5
        width_percent: 4
      - start_percent: 60
        width_percent: 6
    hot_density_ratio_target: 10
```

Acceptance evidence must come from the run receipt and closeout:

1. `workload_profile` records the requested range, placement, density target,
   and derived hot pressure percent.
2. `workload_distribution.density_proofs` records the resolved ordinal ranges,
   hot and comparison sample counts, normalized transaction densities, density
   ratio, and proof status.
3. A hotspot run passes the P0 business proof only when the run status is
   passed, blocked statements are zero, and the density proof ratio is at least
   the declared target.

### 2.1 Render First

Use render-only first so you can review `spec.yaml` before `init`.

**TiUP only — start TiDB first.** A local TiDB instance must be running before
you run any of the commands below. Start TiUP playground first:

```bash
tiup playground v7.5.0
```

Verify that `127.0.0.1:4000` is reachable before continuing. Essential and
Premium deployments do not require this step.

TiUP example:

```bash
TIPOC_OBJECT_STORE_URI=none \
TIPOC_GLOBAL_SORT_URI_REF=none \
TIPOC_BR_ENABLED=false \
TIPOC_PREPARE_OPS=generate,import,add-index \
scripts/env/tiup/init-attempt.sh \
  --run-id run-tiup-50g-001 \
  --attempt-id attempt-001 \
  --scale 50G \
  --out runs/run-tiup-50g-001/attempt-001 \
  --render-only
```

Essential example:

```bash
TIPOC_SQL_ENDPOINT='<essential-sql-host>:4000' \
TIPOC_TIDB_CREDENTIALS_REF='env:TIDB_DSN' \
TIPOC_PD_ENDPOINT='' \
TIPOC_GLOBAL_SORT_URI_REF=target-managed \
TIPOC_BR_ENABLED=false \
TIPOC_OBJECT_STORE_KIND='s3' \
TIPOC_OBJECT_STORE_URI='s3://<bucket>/tipoc/essential/run-essential-1t-001/attempt-001' \
TIPOC_OBJECT_STORE_CREDENTIAL_REF='env:AWS_PROFILE' \
TIPOC_GENERATION_NODES='node-a=10.0.1.10:12,node-b=10.0.1.11:12' \
TIPOC_PRESSURE_NODES='node-a=10.0.1.10:1:384,node-b=10.0.1.11:1:384' \
TIPOC_PRESSURE_CONCURRENCY=768 \
TIPOC_PRESSURE_DURATION_SECONDS=1800 \
TIPOC_SSH_USER='ec2-user' \
TIPOC_SSH_PRIVATE_KEY_REF='env:TIPOC_SSH_KEY' \
TIPOC_REMOTE_ATTEMPT_ROOT='/data/tipoc/run-essential-1t-001/attempt-001' \
scripts/env/essential/init-attempt.sh \
  --run-id run-essential-1t-001 \
  --attempt-id attempt-001 \
  --scale 1T \
  --out runs/run-essential-1t-001/attempt-001 \
  --remote-bootstrap plan \
  --render-only
```

Premium example:

```bash
TIPOC_SQL_ENDPOINT='<premium-sql-host>:4000' \
TIPOC_TIDB_CREDENTIALS_REF='env:TIDB_DSN' \
TIPOC_PD_ENDPOINT='' \
TIPOC_GLOBAL_SORT_URI_REF=target-managed \
TIPOC_BR_ENABLED=false \
TIPOC_OBJECT_STORE_KIND='s3' \
TIPOC_OBJECT_STORE_URI='s3://<bucket>/tipoc/premium/run-premium-1t-001/attempt-001' \
TIPOC_OBJECT_STORE_CREDENTIAL_REF='env:AWS_PROFILE' \
TIPOC_GENERATION_NODES='node-a=10.0.0.10:15,node-b=10.0.0.11:15,node-c=10.0.0.12:15,node-d=10.0.0.13:15' \
TIPOC_PRESSURE_NODES='node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512,node-c=10.0.0.12:1:512,node-d=10.0.0.13:1:512' \
TIPOC_PRESSURE_CONCURRENCY=2048 \
TIPOC_PRESSURE_DURATION_SECONDS=1800 \
TIPOC_SSH_USER='ec2-user' \
TIPOC_SSH_PRIVATE_KEY_REF='env:TIPOC_SSH_KEY' \
TIPOC_REMOTE_ATTEMPT_ROOT='/data/tipoc/run-premium-1t-001/attempt-001' \
scripts/env/premium/init-attempt.sh \
  --run-id run-premium-1t-001 \
  --attempt-id attempt-001 \
  --scale 1T \
  --out runs/run-premium-1t-001/attempt-001 \
  --remote-bootstrap plan \
  --render-only
```

**Premium one-command template.** Copy the block below, replace every `<...>`
placeholder with the value from your intake form, and paste into your terminal.

```bash
# Fill in these values from your intake form:
#   SQL endpoint    → <premium-sql-host>  (from §2)
#   Bucket name     → <bucket>            (from §3)
#   Region          → <region>            (from §3)
#   EC2 private IPs → <ip1> <ip2> <ip3> <ip4>  (from §4)
#   SSH key path    → <ssh-key-path>      (from §4)

TIPOC_SQL_ENDPOINT='<premium-sql-host>:4000' \
TIPOC_TIDB_CREDENTIALS_REF='env:TIDB_DSN' \
TIPOC_PD_ENDPOINT='' \
TIPOC_GLOBAL_SORT_URI_REF=target-managed \
TIPOC_BR_ENABLED=false \
TIPOC_OBJECT_STORE_KIND='s3' \
TIPOC_OBJECT_STORE_URI='s3://<bucket>/tipoc/premium/<run-id>/attempt-001' \
TIPOC_OBJECT_STORE_CREDENTIAL_REF='env:AWS_PROFILE' \
TIPOC_GENERATION_NODES='node-a=<ip1>:15,node-b=<ip2>:15,node-c=<ip3>:15,node-d=<ip4>:15' \
TIPOC_PRESSURE_NODES='node-a=<ip1>:1:512,node-b=<ip2>:1:512,node-c=<ip3>:1:512,node-d=<ip4>:1:512' \
TIPOC_PRESSURE_CONCURRENCY=2048 \
TIPOC_PRESSURE_DURATION_SECONDS=1800 \
TIPOC_SSH_USER='ec2-user' \
TIPOC_SSH_PRIVATE_KEY_REF='<ssh-key-path>' \
TIPOC_REMOTE_ATTEMPT_ROOT='/data/tipoc/<run-id>/attempt-001' \
scripts/env/premium/init-attempt.sh \
  --run-id <run-id> \
  --attempt-id attempt-001 \
  --scale 1T \
  --out runs/<run-id>/attempt-001 \
  --remote-bootstrap plan \
  --render-only
```

Direct `tipoc init` after review:

```bash
tipoc init \
  --spec runs/<run-id>/attempt-001/spec.yaml \
  --out runs/<run-id>/attempt-001 \
  --remote-bootstrap plan
```

If remote bootstrap execution is approved:

```bash
tipoc init \
  --spec runs/<run-id>/attempt-001/spec.yaml \
  --out runs/<run-id>/attempt-001 \
  --remote-bootstrap execute
```

Expected `init` outputs:

| Output | Path |
| --- | --- |
| Rendered spec | `runs/<run-id>/attempt-001/spec.yaml` |
| Compiled manifest | `runs/<run-id>/attempt-001/run-profile-manifest.json` |
| Prepare command | `runs/<run-id>/attempt-001/commands/prepare.sh` |
| Run command | `runs/<run-id>/attempt-001/commands/run.sh` |
| Cleanup command | `runs/<run-id>/attempt-001/commands/cleanup.sh` |
| Runbook | `runs/<run-id>/attempt-001/runbook.md` |

### 2.2 Review `spec.yaml`

Check these fields after render:

| Field | Expected |
| --- | --- |
| `identity.run_id` | matches the intake form |
| `identity.attempt_id` | usually `attempt-001` |
| `scenario.scale.target_size` | target data size |
| `target_cluster.sql_endpoint` | proven SQL endpoint |
| `target_cluster.pd_endpoint` | empty for SQL-only Cloud POC |
| `object_store.uri` | attempt source prefix |
| `generation.nodes` | selected generation nodes |
| `prepare.csv_object_target_bytes` | `536870912` by default |
| `pressure.duration_seconds` | intended duration |
| `pressure.concurrency` | intended total concurrency |
| `pressure.node_specs` | selected pressure nodes |
| `workload.init_support.status` | `supported_now` for all current templates |
| `workload.prepare_support.status` | `supported_now` for `tpce` and `exchange` |
| `workload.run_support.status` | `supported_now` for `tpce` and `exchange` |
| `pressure.workload_mix` | selected workload mix from the workload selection table |
| `pressure.workload_profile.name` | usually `canonical` |
| `cleanup.policy` | expected cleanup behavior |

If any field is wrong, rerender the spec. Do not hand-edit it unless the change
is reviewed and recorded.

For Cloud target-managed Global Sort, confirm `prepare.global_sort_uri_ref` is
empty in the compiled manifest after `init`. The wrapper value
`TIPOC_GLOBAL_SORT_URI_REF=target-managed` renders that empty manifest field.

## 3. Run `prepare`

The generated command bundle writes:

```text
commands/prepare.sh
commands/run.sh
commands/cleanup.sh
```

Prepare preflight for Cloud targets:

| Gate | Required evidence |
| --- | --- |
| SQL host | Public or PrivateLink SQL host resolves from every EC2 node and TCP `4000` passes. |
| SQL credential | `TIPOC_TIDB_CREDENTIALS_REF` resolves on the coordinator without printing the DSN or password. |
| EC2 S3 access | Generation nodes can list, write, read, delete, and multipart-upload under the attempt prefix. |
| TiDB import S3 access | Prefer TiDB Cloud AWS Role ARN based external storage access for long imports, with the external ID handled by the TiDB Cloud UI workflow and AWS role trust relationship. TiDB Cloud import workers can list and read the exact parent `source/` prefix and its table child prefixes, or an approved credential-bearing import URI is valid for the full import/index window plus safety margin. |
| Global Sort mode | `prepare.global_sort_uri_ref` is empty for target-managed Global Sort, or TiDB has write/read/delete scratch-prefix access for self-managed Global Sort. |

Do not start `prepare --ops import` or `prepare --ops import,add-index` until
these gates are recorded. A completed source-generation stage proves only that
EC2 nodes wrote source objects; it does not prove that TiDB import workers can
read those objects.

For TiPOC prepare, `source/` is the attempt-level parent prefix. Generated
source objects are stored under table-level child prefixes such as
`source/<table>/...`, and each table can have many objects. Configure and prove
access at the parent `source/` prefix, then let TiPOC submit each table import
against that table's child prefix or object pattern.

For TiDB Cloud AWS-backed imports, record the chosen import access mode before
launch:

1. Role ARN path, preferred for large imports: either complete the TiDB Cloud
   Import page permission workflow manually, or have an AWS admin/operator use
   the approved automation path to create or update the IAM role. In both
   cases, record only redacted evidence of the TiDB Cloud AWS account ID,
   external ID, role validation result, bucket, parent source prefix, region, and
   the import URI parameter shape accepted by the target cluster. In security
   enhanced mode, do not assume explicit `external-id` in the SQL URI is legal;
   keep the external ID in the TiDB Cloud UI/trust configuration unless a probe
   proves the target cluster accepts the explicit parameter. Treat the Import
   page-generated Role ARN as attempt-scoped unless the environment owner says
   it is durable. For each new import attempt, regenerate or reconfirm the Role
   ARN from the Import page before writing source URIs or rerunning prepare.
2. Query-credential path: record the credential reference and expiration without
   printing `access-key`, `secret-access-key`, or `session-token` values. The
   expiration must cover the expected full `import,add-index` duration plus the
   approved safety margin. Unknown or insufficient TTL blocks import.

Bucket policy or EC2 instance-role access alone does not satisfy TiDB security
enhanced mode when the manifest or import source URI is a naked `s3://...` path.
The generated import URI or manifest must carry approved credential fields or
Role ARN fields. Probe the target cluster before a full import and record the
accepted redacted shape:

```text
# SEM-compatible candidate after TiDB Cloud UI/trust setup:
s3://<bucket>/<source-prefix>/<table>/<part-or-pattern>.csv?region=<aws-region>&role-arn=<ROLE_ARN>

# Non-SEM or product-accepted explicit external ID shape:
s3://<bucket>/<source-prefix>/<table>/<part-or-pattern>.csv?region=<aws-region>&role-arn=<ROLE_ARN>&external-id=<EXTERNAL_ID>
```

For remote Cloud runs, materialize `env:TIDB_DSN` on the coordinator through a
secret file or approved secret manager before running SQL or import probes. See
[environment-setup.md §4.1](environment-setup.md#41-sql-credential-materialization-for-remote-runs).
The operator or AI must check only `present`/`missing` and probe pass/fail
status. Never record the raw DSN.

Run the generated prepare command:

```bash
runs/<run-id>/attempt-001/commands/prepare.sh
```

Or call `tipoc prepare` directly:

```bash
tipoc prepare \
  --manifest runs/<run-id>/attempt-001/run-profile-manifest.json \
  --ops generate,import,add-index
```

Default prepare ops:

```text
generate,import,add-index,backup
```

Use the default only when the approved environment supports a BR baseline. For
Cloud SQL-only POC with no PD/BR path, render:

```bash
TIPOC_PREPARE_OPS=generate,import,add-index \
TIPOC_BR_ENABLED=false
```

Supported prepare options:

| Option | Use it for |
| --- | --- |
| `--ops generate` | Generate source objects only. No TiDB connection required — CSV objects are written directly to S3. |
| `--ops generate,import` | Generate and import without prepare-end indexes. |
| `--ops generate,import,add-index` | Cloud SQL-only full prepare without BR. |
| `--ops generate,import,add-index,backup` | Full prepare with BR baseline. |
| `--ops import` | Resume from existing source artifacts in the attempt root. |
| `--ops import,add-index` | Resume import and build indexes. |
| `--ops import,add-index,backup` | Resume import/index and take BR baseline. |
| `--ops restore` | Restore a prior prepare-end BR baseline. |
| `--finish` | Validate externally completed import and indexes, then mark prepare complete. |

If a previous command ran only `--ops generate`, resume the remaining Cloud
prepare stage from the coordinator only:

```bash
TIPOC_SSH_USER='ec2-user'
TIPOC_SSH_KEY_PATH='/path/to/key.pem'
COORDINATOR_HOST='<coordinator-ssh-host>'
REMOTE_ROOT='/data/tipoc/<run-id>/attempt-001'
REMOTE_BIN='/data/tipoc/tipoc'
REMOTE_MANIFEST="${REMOTE_ROOT}/run-profile-manifest.json"

ssh -i "${TIPOC_SSH_KEY_PATH}" "${TIPOC_SSH_USER}@${COORDINATOR_HOST}" "
  set -euo pipefail
  test -x '${REMOTE_BIN}'
  test -s '${REMOTE_MANIFEST}'
  test -s '${REMOTE_ROOT}/secrets/tidb-dsn'
  jq -e --arg root '${REMOTE_ROOT}' '.evidence_root == \$root' \
    '${REMOTE_MANIFEST}' >/dev/null
  mkdir -p '${REMOTE_ROOT}/logs/coordinator'
  ts=\$(date -u +%Y%m%dT%H%M%SZ)
  (
    set +x
    export TIDB_DSN=\$(tr -d '\r\n' < '${REMOTE_ROOT}/secrets/tidb-dsn')
    exec '${REMOTE_BIN}' prepare \
      --manifest '${REMOTE_MANIFEST}' \
      --ops import,add-index
  ) > '${REMOTE_ROOT}/logs/coordinator/prepare-import-index-'\${ts}'.log' 2>&1 &
  echo \$! > '${REMOTE_ROOT}/logs/coordinator/prepare-import-index-'\${ts}'.pid'
  echo prepare_import_index_started
"
```

This resume path does not start generation workers. It runs the TiDB import and
post-import index operations from the coordinator against the generated S3
source objects and the target TiDB cluster. Stop if the binary, manifest, or DSN
secret file is missing. Stop if `manifest.evidence_root` is absent or does not
equal the real remote attempt root; otherwise TiPOC may read stale evidence from
a default path such as `~/runs/.../prepare`. Stop if the TiDB import-worker S3
proof has not passed or has expired. If the previous attempt failed with
`ExpiredToken`, replace the credential mode first, preferably with Role ARN based
access whose URI shape has passed a target-cluster probe. Do not clean failed
tables, resume import, or start pressure until the corrected import access mode
is recorded.

Restore is destructive to the target prepare boundary. Use it only when the
owner approves target cleanup, and include the required confirmation token:

```bash
tipoc prepare \
  --manifest runs/<run-id>/attempt-001/run-profile-manifest.json \
  --ops restore \
  --restore-backup-receipt /path/to/br-checkpoint-receipt.json \
  --restore-confirm clean-target-and-restore
```

If you have only a backup URI:

```bash
tipoc prepare \
  --manifest runs/<run-id>/attempt-001/run-profile-manifest.json \
  --ops restore \
  --restore-backup-uri s3://<bucket>/<prefix>/br/<run-profile-id> \
  --restore-confirm clean-target-and-restore
```

Rules:

1. Run `prepare` only for workloads whose manifest shows
   `workload.prepare_support.status=supported_now`.
2. Formal prepare for `tpce` and `exchange` uses generated TPC-E CSV source
   objects.
3. Formal S3 prepare writes source objects directly to S3-compatible storage.
4. Do not stage complete local CSV files and upload them afterward.
5. Do not replace prepare with `INSERT`, `LOAD DATA`, manual fill, or query-only shortcuts.
6. The default CSV source-object target is `536870912` bytes, about `512 MiB`.
7. The product derives the internal Global Sort workspace path. Do not provide
   per-table internal Global Sort URIs.
8. `trade_request` is the explicit empty prepare table for TPC-E source
   prepare.
9. A full TPC-E source prepare reports `33` catalog tables and `32` imported
   non-empty tables.

### 3.1 Distributed EC2 generation prepare

Large Cloud prepares that use multiple EC2 generation nodes are not started by
running `tipoc prepare` on the coordinator only. The current prepare generation
path requires one prepare process per generation node. The coordinator owns the
ledger and HTTP endpoint; workers claim generation work from that endpoint.

Preferred launch path:

```bash
scripts/prepare/start-generation.sh \
  --manifest runs/<run-id>/attempt-001/run-profile-manifest.json \
  --dry-run

scripts/prepare/start-generation.sh \
  --manifest runs/<run-id>/attempt-001/run-profile-manifest.json
```

In the normal case, `--manifest` is the only required launcher input. The
manifest already contains the logical launch contract; do not provide the node
list again. The script reads these values from the manifest:

| Manifest-derived value | Use |
| --- | --- |
| `generation.nodes` or `scenario.generation.nodes` | Generation node ids and default node addresses |
| `init.coordinator_node_id` or `scenario.init.coordinator_node_id` | Coordinator node; falls back to the first generation node |
| `init.remote_attempt_root` or `scenario.init.remote_attempt_root` | Remote root for the default manifest, binary, logs, pid files, and ledger |
| `init.ssh_user` or `scenario.init.ssh_user` | SSH user |
| `init.ssh_private_key_ref` or `scenario.init.ssh_private_key_ref` | SSH key reference; `env:NAME` is resolved when `NAME` contains a key-file path |
| `object_store.uri` or `scenario.object_store.uri` | S3/object-store prefix context recorded in the sanitized receipt with query strings and fragments stripped |

All other flags are optional runtime overrides for execution-environment
differences:

| Override | When to use it |
| --- | --- |
| `--node-hosts 'node-a=host-a,node-b=host-b'` | The manifest node addresses are worker-private/VPC addresses used for worker-to-coordinator traffic, but the launcher runs outside the VPC and must SSH through public IPs, DNS names, or approved bastion-reachable hostnames. |
| `--ssh-key PATH` | The manifest `env:NAME` key ref cannot be resolved in the current shell, or an approved operator key override is needed. |
| `--ssh-user USER` | The SSH login user differs from the manifest user. |
| `--coordinator-endpoint-host HOST` | Workers must reach the coordinator through a private IP or DNS name different from the coordinator generation-node address in the manifest. |
| `--remote-root PATH`, `--remote-manifest PATH`, `--remote-bin PATH` | The remote deployment paths differ from the manifest defaults. |
| `--receipt PATH` | The sanitized launch receipt should be written somewhere other than `<local-attempt-root>/prepare/generation-launch-receipt.json`. |
| `--port N` | The coordinator listen and worker endpoint port should differ from `18080`. |
| `--coordinator-wait-seconds N` | The coordinator needs a readiness timeout other than `60` seconds before worker launch. |

Do not pass S3 inputs or credentials to the launcher. The object-store URI and
prefix come from the manifest, and the launch receipt strips any `?query` or
`#fragment` before recording prefix context. Verify S3 prefix hygiene separately
before launch: confirm the source prefix is the intended attempt prefix, no
stale generation processes are writing to it, and any cleanup of failed partial
objects or multipart uploads has been approved.

Common outside-VPC example, where manifest node addresses are private but SSH
uses public hosts:

```bash
TIPOC_EC2_KEY=/path/to/key.pem \
scripts/prepare/start-generation.sh \
  --manifest runs/<run-id>/attempt-001/run-profile-manifest.json \
  --node-hosts 'node-a=<node-a-public-ip>,node-b=<node-b-public-ip>,node-c=<node-c-public-ip>' \
  --coordinator-endpoint-host '<node-a-private-ip>' \
  --ssh-key "${TIPOC_EC2_KEY}" \
  --dry-run
```

The dry-run writes and prints a sanitized planned receipt with node id, SSH host,
role, log path, pid path, planned status, and coordinator wait setting. A live
launch preflights each remote node for the executable binary and manifest,
starts the coordinator, waits until `127.0.0.1:<port>` accepts TCP on the
coordinator host, then starts workers and writes launched pids without recording
secret values or full command strings.

Before launch, distribute the same artifacts to every generation node:

| Artifact | Same path requirement |
| --- | --- |
| `tipoc` binary | Same remote binary path on every node |
| `run-profile-manifest.json` | Same remote manifest path on every node |
| secret refs | Same environment names or same secret-file paths, never raw secret values in docs or receipts |

Distribution and checksum template:

```bash
TIPOC_SSH_USER='ec2-user'
TIPOC_SSH_KEY_PATH='/path/to/key.pem'
REMOTE_ROOT='/data/tipoc/<run-id>/attempt-001'
TIPOC_BIN='./tipoc'
MANIFEST='runs/<run-id>/attempt-001/run-profile-manifest.json'
EXPECTED_SHA256='<sha256-of-linux-binary>'

for host in \
  <node-a-public-ip> \
  <node-b-public-ip> \
  <node-c-public-ip> \
  <node-d-public-ip> \
  <node-e-public-ip> \
  <node-f-public-ip> \
  <node-g-public-ip> \
  <node-h-public-ip>; do
  ssh -i "${TIPOC_SSH_KEY_PATH}" "${TIPOC_SSH_USER}@${host}" \
    "mkdir -p ${REMOTE_ROOT}/bin ${REMOTE_ROOT}/logs/prepare"
  scp -i "${TIPOC_SSH_KEY_PATH}" "${TIPOC_BIN}" \
    "${TIPOC_SSH_USER}@${host}:${REMOTE_ROOT}/bin/tipoc"
  scp -i "${TIPOC_SSH_KEY_PATH}" "${MANIFEST}" \
    "${TIPOC_SSH_USER}@${host}:${REMOTE_ROOT}/run-profile-manifest.json"
  ssh -i "${TIPOC_SSH_KEY_PATH}" "${TIPOC_SSH_USER}@${host}" \
    "chmod +x ${REMOTE_ROOT}/bin/tipoc && echo '${EXPECTED_SHA256}  ${REMOTE_ROOT}/bin/tipoc' | sha256sum -c -"
done
```

Every node must report the expected checksum. If any node differs, stop and
redistribute before prepare starts.

Start the coordinator first. Use the coordinator private IP in the worker
endpoint, not a public IP.

```bash
REMOTE_ROOT='/data/tipoc/<run-id>/attempt-001'
MANIFEST_PATH="${REMOTE_ROOT}/run-profile-manifest.json"
COORDINATOR_PRIVATE_IP='<coordinator-private-ip>'

TIPOC_PREPARE_GENERATION_NODE_ID='node-a' \
TIPOC_PREPARE_GENERATION_COORDINATOR_LISTEN='0.0.0.0:18080' \
TIPOC_PREPARE_GENERATION_LEDGER_PATH="${REMOTE_ROOT}/prepare/generation-ledger.json" \
"${REMOTE_ROOT}/bin/tipoc" prepare \
  --manifest "${MANIFEST_PATH}" \
  --ops generate
```

After the coordinator is listening, start one worker process on each remaining
generation node:

```bash
REMOTE_ROOT='/data/tipoc/<run-id>/attempt-001'
MANIFEST_PATH="${REMOTE_ROOT}/run-profile-manifest.json"
COORDINATOR_PRIVATE_IP='<coordinator-private-ip>'

TIPOC_PREPARE_GENERATION_NODE_ID='<node-b>' \
TIPOC_PREPARE_GENERATION_COORDINATOR_ENDPOINT="http://${COORDINATOR_PRIVATE_IP}:18080" \
"${REMOTE_ROOT}/bin/tipoc" prepare \
  --manifest "${MANIFEST_PATH}" \
  --ops generate
```

8-node worker launch template:

```bash
TIPOC_SSH_USER='ec2-user'
TIPOC_SSH_KEY_PATH='/path/to/key.pem'
REMOTE_ROOT='/data/tipoc/<run-id>/attempt-001'
COORDINATOR_PRIVATE_IP='<coordinator-private-ip>'

for node in \
  node-b=<node-b-public-ip> \
  node-c=<node-c-public-ip> \
  node-d=<node-d-public-ip> \
  node-e=<node-e-public-ip> \
  node-f=<node-f-public-ip> \
  node-g=<node-g-public-ip> \
  node-h=<node-h-public-ip>; do
  node_id="${node%%=*}"
  host="${node#*=}"
  ssh -i "${TIPOC_SSH_KEY_PATH}" "${TIPOC_SSH_USER}@${host}" \
    "cd ${REMOTE_ROOT} && nohup env \
      TIPOC_PREPARE_GENERATION_NODE_ID=${node_id} \
      TIPOC_PREPARE_GENERATION_COORDINATOR_ENDPOINT=http://${COORDINATOR_PRIVATE_IP}:18080 \
      ${REMOTE_ROOT}/bin/tipoc prepare \
        --manifest ${REMOTE_ROOT}/run-profile-manifest.json \
        --ops generate \
      > ${REMOTE_ROOT}/logs/prepare/${node_id}.log 2>&1 &"
done
```

Minimum live checks:

```bash
# From every worker after the coordinator process is running:
nc -vz "${COORDINATOR_PRIVATE_IP}" 18080

# On the coordinator:
tail -f "${REMOTE_ROOT}/logs/prepare/coordinator.log"

# S3 source prefix progress:
aws s3 ls "s3://<bucket>/tipoc/<deployment>/<run-id>/attempt-001/source/" --recursive --summarize
```

Known limitation: `runner_launch.mode=coordinator_ssh` applies to the `run`
stage pressure runners. It does not launch prepare generation workers. Use
`scripts/prepare/start-generation.sh` for prepare generation launch, or
explicitly start all generation-node prepare processes with an approved external
wrapper that does the same artifact distribution, checksum validation, startup
order, and secret redaction.

Prepare evidence to keep:

| Evidence | Path pattern |
| --- | --- |
| Prepare receipt | `prepare/prepare-receipt.json` |
| Dataset manifest | `prepare/dataset-manifest.json` |
| Source publication receipt | `prepare/source-publication-receipt.json` |
| Import plan | `prepare/import-plan.json` |
| Import SQL | `prepare/import-sql.json` |
| Import checkpoint | `prepare/current-attempt-import-checkpoint.json` |
| Statistics validation | `prepare/statistics-validation-receipt.json` |
| ADD INDEX evidence | `prepare/final-index-records.json` or equivalent receipt section |
| BR checkpoint, if enabled | `prepare/br-checkpoint-receipt.json` |

### 3.2 Clean prepare rerun and S3 prefix hygiene

Use a clean prepare rerun when the source prefix or generation ledger may
contain partial output from a failed or aborted generation pass. Do not mix
objects from two generation attempts under one source prefix.

Clean rerun order:

1. Stop every active prepare process on coordinator and workers.
2. Preserve logs, stage logs, progress JSON, and the current manifest under the
   local attempt evidence root.
3. Confirm the target SQL state is either untouched, explicitly disposable, or
   restored to the approved prepare boundary.
4. Delete only the approved attempt source prefix and stale multipart uploads.
5. Redistribute the manifest and binary, verify checksums, then relaunch
   coordinator and workers.

S3 cleanup template:

```bash
BUCKET='<bucket>'
BASE='tipoc/<deployment>/<run-id>/attempt-001'

aws s3 rm "s3://${BUCKET}/${BASE}/source/" --recursive

aws s3api list-multipart-uploads \
  --bucket "${BUCKET}" \
  --prefix "${BASE}/source/" \
  --query 'Uploads[].{Key:Key,UploadId:UploadId}' \
  --output json
```

Abort stale multipart uploads only after checking that no prepare process is
still writing to the same prefix:

```bash
aws s3api abort-multipart-upload \
  --bucket "${BUCKET}" \
  --key '<object-key>' \
  --upload-id '<upload-id>'
```

### 3.3 When prepare fails

`prepare` contains multiple sub-steps (generate → import → add-index → backup).
Any of them can fail. Here are the most common failures and how to diagnose them:

| Symptom | Likely cause | Diagnostic command |
| --- | --- | --- |
| SQL connection refused | TiDB not running or wrong endpoint | `mysql -h <host> -P 4000 -u root -e 'select 1'` |
| S3 read/write failure | Missing IAM permissions or network | `aws s3 ls s3://<bucket>/<prefix>/` |
| Import timeout | Large dataset or insufficient TiDB resources | Check TiDB dashboard CPU/memory |
| BR backup failure | BR endpoint unreachable or no permission | Check `TIPOC_BR_STORAGE_URI` and PD endpoint |
| Worker exits with manifest not found | `run-profile-manifest.json` was not copied to the same remote path on that node | `ssh <node> 'ls -l /data/tipoc/<run-id>/attempt-001/run-profile-manifest.json'` |
| Worker cannot reach coordinator | Security group lacks same-SG inbound TCP `18080`, wrong coordinator private IP, or coordinator not listening | `nc -vz <coordinator-private-ip> 18080` from each worker |
| Worker heartbeat or record requests timeout around 30 seconds | Coordinator ledger persistence or HTTP client timeout bottleneck under concurrent generation | Check coordinator logs for `RecordObject`, `Heartbeat`, ledger persist latency, and client timeout values |

After fixing the root cause, resume from the failed step with `--ops` instead
of starting over:

```bash
# Skip generate, continue from import
tipoc prepare --manifest ... --ops import,add-index

# Add indexes only
tipoc prepare --manifest ... --ops add-index
```

If the problem persists, record the symptoms and diagnosis in
[`sop-doc-issues.md`](sop-doc-issues.md).

For 8-node or larger generation, treat repeated `RecordObject`/`Heartbeat`
timeouts as a product/runtime bottleneck, not an operator retry issue. Capture
the binary checksum, node count, source prefix, coordinator logs, worker logs,
and ledger size. Known product fixes include avoiding full ledger persistence
on every heartbeat/object record, compacting ledger JSON writes, using a longer
coordinator HTTP client timeout, and using larger S3 multipart parts with
bounded upload concurrency for large source objects.

## 4. Run `run`

The generated command bundle writes:

```bash
runs/<run-id>/attempt-001/commands/run.sh
```

Or call `tipoc run` directly:

```bash
tipoc run --manifest runs/<run-id>/attempt-001/run-profile-manifest.json
```

`run` has no stage flags beyond `--manifest`. Duration, concurrency, SQL pool
size, pressure nodes, workload mix, and workload profile come from the
compiled manifest.

`run` and `prepare` differ in distributed launch behavior. A manifest may use
`runtime.runner_launch.mode=coordinator_ssh` so the run coordinator starts
remote pressure runners over SSH and writes
`run/pressure-runner-launch-plan.json` plus
`run/pressure-runner-launch-receipt.json`. That support is for pressure
runners only. It does not start prepare generation workers and must not be used
as evidence that prepare has automatic worker launch.

Review these values before the first run:

| Field | Common starting value |
| --- | --- |
| `pressure.duration_seconds` | `1800` for about `30 mins` |
| `pressure.warmup_seconds` | `300` |
| `pressure.cooldown_seconds` | `60` |
| `pressure.progress_interval_secs` | `30` |
| `pressure.concurrency` | choose from the sizing table |
| `pressure.node_specs` | one or more physical pressure nodes |
| `workload.run_support.status` | `supported_now` before pressure starts |
| `pressure.workload_mix` | selected supported workload mix |
| `pressure.workload_profile.name` | `canonical` |

Example run shape:

```bash
TIPOC_PRESSURE_DURATION_SECONDS=1800 \
TIPOC_PRESSURE_WARMUP_SECONDS=300 \
TIPOC_PRESSURE_COOLDOWN_SECONDS=60 \
TIPOC_PRESSURE_PROGRESS_INTERVAL_SECONDS=30 \
TIPOC_PRESSURE_CONCURRENCY=2048 \
TIPOC_PRESSURE_NODES='node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512,node-c=10.0.0.12:1:512,node-d=10.0.0.13:1:512' \
scripts/env/premium/init-attempt.sh \
  --run-id run-premium-1t-001 \
  --attempt-id attempt-001 \
  --scale 1T \
  --out runs/run-premium-1t-001/attempt-001 \
  --render-only
```

Run evidence to keep:

| Evidence | Path pattern |
| --- | --- |
| Run receipt | `run/run-receipt.json` |
| Global closeout | `run/global-run-closeout.json` |
| Markdown report | `run/run-report.md` |
| Semantic proof | `run/semantic-proof.json` |
| Query evidence | `run/query-evidence.json` |
| Node reports | `run/*node*report*` or generated report paths |

The report should state whether the run is the initial pass, same-attempt
continuation, or clean comparison, plus pressure nodes, total concurrency, SQL
pool size, duration, QPS, blocked statements, latency, workload mix/profile,
and the run result.

For an external or try-local pressure run, also keep a short live-monitor
window that proves traffic actually started:

| Signal | Required evidence |
| --- | --- |
| Fleet liveness | every configured pressure node has a live `tipoc run` child process after startup |
| Worker freshness | fresh worker progress count equals the expected concurrency |
| Traffic | QPS and statement counts come from the selected run directory, such as `run/try-03/run-progress.jsonl` |
| Errors | startup and progress samples show zero unexpected errors, or list the exact nonzero error count |
| Clinic window | absolute UTC start and end time, cluster id, environment, and raw metric artifacts |

For workload read/write ratio, report a precise time-window DML ratio:

```text
write_share = (Insert + Update + Delete + Replace) /
              (Select + Insert + Update + Delete + Replace)
```

Use Clinic or Prometheus `sum by (sql_type)
(rate(tidb_server_handle_query_duration_seconds_count[1m]))` for the same UTC
window as the pressure sample. Report `Begin`, `Commit`, `internal`, and
`other` separately; do not put them in the DML read/write denominator unless
the report explicitly says it is an all-SQL ratio.

`INFORMATION_SCHEMA.CLUSTER_STATEMENTS_SUMMARY_HISTORY` is useful for top SQL
and total latency, but its summary rows can cover coarse buckets such as
`09:30:00Z` to `10:00:00Z`. Do not use a bucket-overlap result as the precise
read/write ratio for a shorter pressure window unless the bucket boundaries
match the run window or the report clearly labels it as bucket-level evidence.

When Clinic or Prometheus returns empty TiKV process series, record that as
missing metric coverage. Do not interpret an empty TiKV series as zero TiKV
load unless another first-hand source proves that.

### 4.1 Rerun After One Completed Run

Use one of these paths after a completed run:

| Path | Use it for | Result label |
| --- | --- | --- |
| Try-local rerun wrapper | Another pass with the same or adjusted runtime pressure shape, including warmup, cooldown, or progress interval changes. | `continuation` unless a clean target was restored or reprepared first |
| Same-manifest continuation | Manual recovery path for timing-only edits when an operator intentionally preserves the whole prior `run/` directory first. | `continuation` |

#### 4.1.1 Same-manifest continuation for timing-only recovery

Ordinary reruns should use `scripts/rerun/run-again.sh` so each pass writes a
fresh `run/try-0x` report. Use this same-manifest path only when all four are
true:

1. `stage.current_state=run_closeout_published`
2. `run/run-continuation.json` exists with `status=ready`
3. you want to change only warmup, cooldown, or progress interval
4. you have preserved the entire existing `run/` evidence directory

Back up the compiled manifest, continuation file, and complete run evidence
directory before editing. A same-manifest rerun writes to the same `run/`
directory and can replace `run-receipt.json`, reports, and closeout artifacts.

```bash
cp -a runs/<run-id>/attempt-001/run \
  runs/<run-id>/attempt-001/run.before-continuation-$(date -u +%Y%m%dT%H%M%SZ)
cp runs/<run-id>/attempt-001/run-profile-manifest.json \
  runs/<run-id>/attempt-001/run-profile-manifest.before-continuation.json
cp runs/<run-id>/attempt-001/run/run-continuation.json \
  runs/<run-id>/attempt-001/run/run-continuation.before-continuation.json
```

Edit only these compiled manifest fields:

1. `runtime.warmup_seconds` and `scenario.pressure.warmup_seconds`
2. `runtime.cooldown_seconds` and `scenario.pressure.cooldown_seconds`
3. `runtime.progress_interval_secs` and `scenario.pressure.progress_interval_secs`

Then rerun the same manifest:

```bash
tipoc run --manifest runs/<run-id>/attempt-001/run-profile-manifest.json
```

Do not use this timing-only path for ordinary reruns, planned duration,
concurrency, SQL pool, pressure-node, CE-worker-cap, workload-mix, or
workload-profile changes.

#### 4.1.2 Try-local rerun wrapper for changed runtime pressure

Use `scripts/rerun/run-again.sh` when you want another pressure pass under the
same attempt root with a new try-local manifest. The wrapper creates the next
`run/try-0x` directory, writes `run/try-0x/run-profile-manifest.json`, writes
`rerun-receipt.json`, and then calls the same public `run` stage:

```text
tipoc run --manifest <attempt-root>/run/try-0x/run-profile-manifest.json
```

The rerun wrapper requires:

1. `stage.current_state=run_closeout_published`
2. a ready `run/run-continuation.json` or `run/try-0x/run-continuation.json`
3. the same attempt root

Typical runtime-shape continuation:

```bash
scripts/rerun/run-again.sh \
  --from-attempt runs/<run-id>/attempt-001 \
  --try auto \
  --warmup-seconds 300 \
  --cooldown-seconds 60 \
  --progress-interval-seconds 30 \
  --concurrency 2048 \
  --sql-pool-size 512 \
  --pressure-nodes 'node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512'
```

This creates the next unused `run/try-0x` directory. If previous run output
exists but no ready continuation exists, the wrapper stops before pressure
starts.

For `runtime.runner_launch.mode=external`, use a two-step handoff:

1. Generate the try-local manifest and sync it to all pressure nodes:

   ```bash
   scripts/rerun/run-again.sh \
     --from-attempt runs/<run-id>/attempt-001 \
     --local-attempt-root <local-attempt-root> \
     --remote-attempt-root <remote-attempt-root> \
     --try auto \
     --duration-seconds 3600 \
     --concurrency 8192 \
     --sql-pool-size 1024 \
     --pressure-nodes 'node-a=10.0.0.10:1:1024,node-b=10.0.0.11:1:1024' \
     --sync-try-to-pressure-nodes \
     --dry-run
   ```

2. Start the selected try with the approved external launcher. The launcher
   must preflight every node, submit all non-coordinator workers first, start
   the coordinator last, and write per-node pid, status, exit, and log files.

Do not run a non-dry `run-again.sh` from an operator laptop for an external
fleet unless that machine is intentionally the runner for every pressure node.
For external fleets, the standard path is generate or dry-run, sync the selected
try directory, then launch the fleet with the external launcher.

Concrete external launch files are run-specific operator artifacts. Generate or
instantiate them inside the active project or attempt workspace, but do not
commit node addresses, launch ids, private-key paths, bastion hosts, DSNs, or
attempt-local wrapper copies to the TiPOC repository. The committed contract is
the generic rerun/stop script interface plus the SOP fields below.

Supported rerun inputs:

| Flag | Meaning |
| --- | --- |
| `--from-attempt` | Start from an attempt root. This is the normal entry point. |
| `--from-manifest` | Start from an attempt manifest or a prior try-local manifest. |
| `--local-attempt-root` | Local mirror of the attempt root. Use it when the source manifest keeps remote `evidence_root` paths. |
| `--remote-attempt-root` | Remote attempt root used to write remote-compatible manifest and log paths. Required for ambiguous external local-mirror reruns. |
| `--try` | Use `auto` or an explicit `try-01` style directory name. |
| `--duration-seconds` | Total run duration. Supported, but prefer a new attempt for planned comparison windows. |
| `--warmup-seconds` | Warmup excluded from the measured window. |
| `--cooldown-seconds` | Cooldown period. |
| `--progress-interval-seconds` | Progress write interval. |
| `--concurrency` | Global requested worker concurrency. This is continuation-only unless a clean target was restored or reprepared first. |
| `--concurrency-steps` | Sequential static concurrency values, for staged pressure. |
| `--sql-pool-size` | Per-node SQL pool size override. This is continuation-only unless a clean target was restored or reprepared first. |
| `--pressure-nodes` | Pressure node list and optional CE worker cap. It must still include the current coordinator node. |
| `--workload-mix` | Workload mix, usually `tpce`. This is continuation-only unless a clean target was restored or reprepared first. |
| `--workload-profile` | Workload profile, usually `canonical`. This is continuation-only unless a clean target was restored or reprepared first. |
| `--exchange-matching-write-percent` | Exchange-matching CE selector target. This is not by itself the observed SQL DML write-share target. |
| `--exchange-matching-sql-write-percent` | Exchange-matching SQL DML write-share steering target. Pass it explicitly when changing `--exchange-matching-write-percent`; pass `0` only to intentionally disable SQL write-share shaping. |
| `--query-cost-mode` | Query Cost mode for the new try: `off`, `latency_only`, or `latency_and_statement_ru`. For paired measurements, pass it together with `--query-cost-attempt-ru-source`. |
| `--query-cost-attempt-ru-source` | Per-attempt RU source: `off` or `tidb_last_query_info`. The latter requires `latency_and_statement_ru`. |
| `--query-cost-attempt-ru-timeout-milliseconds` | Same-session RU probe timeout, from 1 through 30000 milliseconds. |
| `--query-cost-attempt-ru-failure-budget` | Bounded probe-failure budget, from 1 through 1000000. |
| `--query-cost-poll-interval-seconds` | Query-cost Statement Summary side-poll interval for the new try. Minimum 5 seconds; requires query cost enabled in the source manifest. |
| `--query-cost-poll-timeout-seconds` | Per-poll timeout for the query-cost Statement Summary side path. Must be positive and lower than the final poll interval. |
| `--query-cost-isolation-status` | Freeze `verified_test_environment`, `declared`, or `unverified` run-exclusive RU attribution evidence in both manifest contracts. Must be paired with a reason. |
| `--query-cost-isolation-reason` | Bounded operator evidence for the isolation status. This is attribution evidence, not an automatic Resource Group or Statement Summary inference. |
| `--tipoc` | Alternate `tipoc` binary path. |
| `--dry-run` | Write the try-local manifest and receipt without starting pressure. |
| `--sync-try-to-pressure-nodes` | Sync the selected try directory to non-coordinator pressure nodes for `external` or `coordinator_ssh` runner launch modes. |

For an existing hot range attempt, omit `--workload-profile` unless you are
intentionally changing the profile name; the rerun manifest preserves the
source `runtime.workload_profile` fields, including `hot_range_percent`,
placement, segments, and density target. Do not try to create a complete hot
range profile by passing only `--workload-profile hot-range`; the rerun wrapper
does not add the required range fields. For a new hotspot shape, render and
initialize a new attempt, or restore/reprepare the target before treating the
next run as clean comparison evidence.

Example staged pressure ramp:

```bash
scripts/rerun/run-again.sh \
  --from-attempt runs/<run-id>/attempt-001 \
  --try auto \
  --duration-seconds 1800 \
  --warmup-seconds 300 \
  --cooldown-seconds 60 \
  --concurrency-steps 1024,2048,4096 \
  --sql-pool-size 512 \
  --pressure-nodes 'node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512,node-c=10.0.0.12:1:512,node-d=10.0.0.13:1:512'
```

There is no in-window dynamic ramp today. Gradual pressure increase uses
sequential tries, not one `run` process that changes pressure while it is still
running.

Operational rules:

1. If you change duration, concurrency, SQL pool, pressure nodes, CE worker
   cap, workload mix, or workload profile inside the same attempt, report the
   result as continuation evidence unless a clean target was restored or
   reprepared first.
2. For clean comparison after runtime writes, first restore an approved
   prepare-end BR baseline or rerun prepare into a clean target, then start the
   next run.
3. If the coordinator node or attempt root changes, start a new attempt.
4. If previous run output exists but no ready continuation exists, stop and fix
   the attempt boundary before pressure starts.
5. For external local-mirror reruns, keep concrete launch files, node
   inventories, launch ids, private-key paths, and bastion targets in the
   active project or attempt workspace. Do not commit those instance-specific
   files into the TiPOC repository.

Do not edit `run-continuation.json` by hand to force a rerun. If a prior
continuation is blocked, treat that as a product or evidence-boundary issue and
start from a clean prepared target or repair the product behavior before
running another formal pressure pass.

#### 4.1.3 Recovering an interrupted same-attempt run

Use this path when the TiPOC process was killed, the host restarted, or the
runner disappeared before normal run closeout. It is a **continuation**, not a
clean comparison: retain the target data and do not re-import or reset it.

The recovery invariant is per trade-ID lease. The next run must start after
the highest durable/live safe cursor for every lease; a global throughput
result or a secondary operation result must never discard a usable foreground
continuation.

1. Identify the newest terminal predecessor. It must have all three files:
   `run-profile-manifest.json` at `run_closeout_published`, a `ready`
   `run-continuation.json`, and a passed `global-run-closeout.json`. A normal
   run may use `run/`; a rerun normally uses the newest `run/try-0x/`.
2. Revalidate the retained prepare source and predecessor without importing:

   ```bash
   tipoc prepare \
     --manifest runs/<run-id>/attempt-001/run-profile-manifest.json \
     --finish \
     --continuation-from runs/<run-id>/attempt-001/run/try-02/run-profile-manifest.json
   ```

   This checks immutable prepare evidence, the terminal boundary, current
   runtime-owned tables, and prepare-end indexes. It must not create a new
   import checkpoint or replay import.
3. Start the normal public stage, not a private recovery command:

   ```bash
   tipoc run --manifest runs/<run-id>/attempt-001/run-profile-manifest.json
   ```

   `run` compares any attempt-local recovery boundary with the prepare-bound
   terminal boundary per lease. It selects a boundary only when it is at least
   as advanced for every lease; incomparable or regressing boundaries fail
   closed. `run-again.sh` ultimately calls this same `run` path, so direct run
   and try-local rerun share the recovery rule.
4. Before calling the next round runnable, record: selected predecessor
   manifest path and SHA-256, selected `next_runtime_start_trade_id`, every
   lease's `next_start_trade_id`, runtime-table snapshot fingerprint, and
   schema/index evidence. After the round, retain its terminal
   `run-continuation.json` even if optional Operation evaluation is not passed.

Only a true immutable-prepare mismatch, runtime-table corruption, incompatible
lease geometry, or exhausted trade-ID budget blocks this recovery. Do not mark
the foreground data state unusable merely because an optional action or an
operation-side threshold failed.

For the exact contract, evidence fields, and failure matrix, see
[Issue #65 continuation recovery](../issue-65-continuation-recovery.md).

Use a new attempt instead of a rerun when you change any prepare-bound setting
that should start from a clean prepare boundary, such as scale, source object
shape, import policy, BR policy, restore policy, or the target prepare
baseline.

#### 4.1.4 Standard stop for external pressure runs

For `runtime.runner_launch.mode=external`, stop a running or underperforming
pressure fleet with `scripts/rerun/stop-external-run.sh`. This is the standard
stop primitive for generated external launchers and replaces ad hoc per-node
`kill` commands.

```bash
scripts/rerun/stop-external-run.sh \
  --launch-id <launch-id> \
  --remote-root <remote-attempt-root> \
  --node-file <local-node-inventory> \
  --ssh-key <local-private-key> \
  --manifest <remote-try-manifest>
```

The same values may be supplied through matching `TIPOC_RUN_*` environment
variables. The required node inventory shape is the same as the external start
launcher:

```text
node_id ssh_host ssh_mode private_ip expected_slots expected_sql_pool
```

Use `ssh_mode=direct` for directly reachable nodes and `ssh_mode=proxy` for
nodes reached through a bastion. For proxy records, also provide
`--bastion <ssh-target>` or `TIPOC_RUN_BASTION=<ssh-target>`.

The stop script reads wrapper pid files from
`<remote-attempt-root>/logs/<launch-id>/<node-id>.pid`, stops the wrapper
process tree, and, when `--manifest` is provided, also stops stray
`tipoc run` processes whose command line still references that manifest. It is
idempotent for already stopped nodes, fails closed for missing required local
inputs or missing remote launch-log directories, and redacts DSN/password/token
style values from SSH output.

Record stop evidence beside the run evidence:

| Evidence | Required content |
| --- | --- |
| stop command | non-secret command shape, launch id, remote root, manifest path, and node inventory path |
| stop output | per-node `cleanup=stopped` or `cleanup=not_running` result |
| post-stop liveness | every configured pressure node has no live `tipoc run` child for the stopped manifest |
| rerun decision | whether the next run is continuation evidence or a clean comparison from a restored or reprepared target |

### 4.2 Multi-round Pressure Simulation

Use this section when the goal is not one run, but a sequence of pressure
rounds. The sequence can be operated manually or by an AI agent preparing the
non-secret commands.

Before the first round, write the planned matrix from `poc-intake-form.md` into
the attempt evidence root:

```bash
mkdir -p runs/<run-id>/attempt-001/run
cat > runs/<run-id>/attempt-001/run/round-plan.md <<'EOF'
| Round | Workload mix | Duration seconds | Concurrency | SQL pool | Boundary | Evidence path |
| --- | --- | ---: | ---: | ---: | --- | --- |
| initial | tpce | 1800 | 1024 | 512 | prepared target | run/ |
| try-01 | tpce | 1800 | 2048 | 512 | continuation | run/try-01/ |
| try-02 | tpce | 3600 | 2048 | 512 | continuation | run/try-02/ |
| try-03 | exchange_matching | 1800 | 2048 | 512 | continuation unless restored | run/try-03/ |
EOF
```

#### 4.2.1 Initial TPC-E control run

Render the first attempt with canonical TPC-E runtime settings:

```bash
TIPOC_WORKLOAD_MIX=tpce \
TIPOC_WORKLOAD_PROFILE=canonical \
TIPOC_PRESSURE_DURATION_SECONDS=1800 \
TIPOC_PRESSURE_WARMUP_SECONDS=300 \
TIPOC_PRESSURE_COOLDOWN_SECONDS=60 \
TIPOC_PRESSURE_PROGRESS_INTERVAL_SECONDS=30 \
TIPOC_PRESSURE_CONCURRENCY=1024 \
TIPOC_PRESSURE_NODES='node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512' \
scripts/env/premium/init-attempt.sh \
  --run-id <run-id> \
  --attempt-id attempt-001 \
  --scale <scale> \
  --out runs/<run-id>/attempt-001 \
  --remote-bootstrap plan \
  --render-only
```

Review `spec.yaml`, run `init`, run `prepare`, then run the generated
`commands/run.sh` or direct `tipoc run --manifest ...` command. Keep
`run/run-receipt.json`, `run/global-run-closeout.json`, and `run/run-report.md`.

#### 4.2.2 Change concurrency or duration

Use the rerun wrapper for ordinary follow-up pressure tries:

```bash
scripts/rerun/run-again.sh \
  --from-attempt runs/<run-id>/attempt-001 \
  --try auto \
  --duration-seconds 1800 \
  --warmup-seconds 300 \
  --cooldown-seconds 60 \
  --progress-interval-seconds 30 \
  --concurrency 2048 \
  --sql-pool-size 512 \
  --pressure-nodes 'node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512'
```

Use sequential static steps when the goal is a simple ceiling search:

```bash
scripts/rerun/run-again.sh \
  --from-attempt runs/<run-id>/attempt-001 \
  --try auto \
  --duration-seconds 1800 \
  --warmup-seconds 300 \
  --cooldown-seconds 60 \
  --concurrency-steps 1024,2048,4096 \
  --sql-pool-size 512 \
  --pressure-nodes 'node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512'
```

Each value in `--concurrency-steps` becomes one try directory. There is no
in-window dynamic ramp.

#### 4.2.3 Switch from TPC-E to exchange matching

Use `--workload-mix exchange_matching` when the next round should use the
exchange matching workload mix:

```bash
scripts/rerun/run-again.sh \
  --from-attempt runs/<run-id>/attempt-001 \
  --try auto \
  --duration-seconds 1800 \
  --warmup-seconds 300 \
  --cooldown-seconds 60 \
  --concurrency 2048 \
  --sql-pool-size 512 \
  --pressure-nodes 'node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512' \
  --workload-mix exchange_matching \
  --workload-profile canonical
```

This is a runtime-shape continuation when it uses the same attempt target. If
the result must be compared as a clean TPC-E versus exchange matching A/B test,
restore the approved prepare-end baseline or run a fresh prepare before the
exchange matching round.

#### 4.2.4 Change exchange matching read/write ratio

Preferred clean path: render a new attempt or restore a clean prepare-end
baseline, then set the ratio fields during render.

```bash
TIPOC_WORKLOAD_MIX=exchange_matching \
TIPOC_WORKLOAD_PROFILE=canonical \
TIPOC_EXCHANGE_MATCHING_WRITE_PERCENT=90 \
TIPOC_EXCHANGE_MATCHING_SQL_WRITE_PERCENT=70 \
TIPOC_PRESSURE_DURATION_SECONDS=1800 \
TIPOC_PRESSURE_WARMUP_SECONDS=300 \
TIPOC_PRESSURE_COOLDOWN_SECONDS=60 \
TIPOC_PRESSURE_PROGRESS_INTERVAL_SECONDS=30 \
TIPOC_PRESSURE_CONCURRENCY=2048 \
TIPOC_PRESSURE_NODES='node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512' \
scripts/env/premium/init-attempt.sh \
  --run-id <run-id>-exchange-ratio-90-70 \
  --attempt-id attempt-001 \
  --scale <scale> \
  --out runs/<run-id>-exchange-ratio-90-70/attempt-001 \
  --remote-bootstrap plan \
  --render-only
```

After `init`, confirm the compiled manifest contains the intended fields:

```bash
jq '{
  workload_mix: .runtime.workload_mix,
  profile: .runtime.workload_profile.name,
  exchange_matching_write_percent: .runtime.workload_profile.exchange_matching_write_percent,
  exchange_matching_sql_write_percent: .runtime.workload_profile.exchange_matching_sql_write_percent
}' runs/<run-id>-exchange-ratio-90-70/attempt-001/run-profile-manifest.json
```

Supported values are:

| Field | Valid values |
| --- | --- |
| `exchange_matching_write_percent` | integers from `20` through `90` |
| `exchange_matching_sql_write_percent` | `0`, or integers from `10` through `90` |

For a same-attempt continuation rerun, use the first-class ratio flags instead
of hand-editing the try-local manifest:

```bash
scripts/rerun/run-again.sh \
  --from-attempt runs/<run-id>/attempt-001 \
  --try auto \
  --workload-mix exchange_matching \
  --workload-profile canonical \
  --exchange-matching-write-percent 90 \
  --exchange-matching-sql-write-percent 70 \
  --concurrency 2048 \
  --dry-run
```

The wrapper writes both ratio fields into `runtime.workload_profile` and
`scenario.pressure.workload_profile`, and records the override values in
`run/<try-id>/rerun-receipt.json`. Same-attempt reruns remain continuation
evidence unless the target was restored or reprepared before the comparison.

#### 4.2.5 Manual and AI handoff for every round

Before each round:

1. Confirm the round row exists in `run/round-plan.md`.
2. Confirm the previous run has a ready `run-continuation.json` when using
   `run-again.sh`.
3. Confirm whether the evidence label is `continuation` or `clean comparison`.
4. Confirm the manifest fields with `jq` before pressure starts.
5. Stop if a required secret, approval, restore, cleanup, remote command, SQL
   mutation, S3 mutation, or environment change has not been approved by the
   human operator.

After each round, append one row to the final report draft:

| Round | Result | Workload mix | Duration | Concurrency | SQL pool | Statement QPS | Blocked statements | Evidence path |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | --- |
| `initial` |  |  |  |  |  |  |  |  |

If a step is unclear or requires product/script behavior that the SOP does not
cover, add an item to `docs/poc-sop/sop-doc-issues.md` before continuing.

## 5. Run `cleanup`

The generated cleanup command is:

```bash
runs/<run-id>/attempt-001/commands/cleanup.sh
```

Or call `tipoc cleanup` directly:

```bash
tipoc cleanup --manifest runs/<run-id>/attempt-001/run-profile-manifest.json
```

Cleanup rules:

1. Preserve receipts, reports, and raw logs unless the cleanup policy says
   otherwise.
2. Delete only approved attempt-scoped scratch data.
3. Never delete a shared bucket.
4. Stop or terminate EC2 nodes only according to the written cleanup policy.
5. `cleanup` is not the reporting stage.
6. Accepted validation evidence is preserved by policy; do not clean its
   evidence root as part of ordinary SOP cleanup.
7. `/tmp` evidence from four-workload master verification is transient unless
   it is copied into an archived report or durable evidence directory.

Keep these cleanup outputs when present:

| Evidence | Path |
| --- | --- |
| Cleanup receipt | generated cleanup receipt path |
| Retained evidence list | final report or cleanup evidence file |
| Deleted prefixes list | final report or cleanup evidence file |

## 6. Final Handoff

The final report should be short and direct:

```markdown
# <POC name> Report

### Summary
- Result: pass / fail / blocked
- Run type: initial / continuation / clean comparison
- Data size:
- SQL path:
- Observed QPS:

### Environment
- TiDB endpoint reference:
- EC2 nodes:
- S3 source prefix:

### init
- Spec path:
- Manifest path:

### prepare
- Source object count:
- Source bytes:
- Imported tables:
- Empty table:
- Statistics validation:
- BR status:

### run
- Pressure nodes:
- Concurrency:
- SQL pool:
- Duration:
- Workload mix:
- Workload profile:
- Exchange matching write percent:
- Exchange matching SQL write percent:
- Statement QPS:
- Blocked statements:
- Evidence paths:

### pressure rounds
| Round | Result | Evidence label | Workload mix | Duration | Concurrency | SQL pool | Statement QPS | Evidence path |
| --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- |
| initial |  |  |  |  |  |  |  |  |

### cleanup
- Preserved:
- Deleted:

### Next Action
- Owner:
- Action:
- Evidence needed:
```

Failure is acceptable. Do not rewrite the outcome to make the POC look
successful.

## 7. Release And Verification Gates

The completed workload registry delivery used this gate order:

1. Open or reuse a GitHub issue that states the objective and acceptance
   criteria.
2. Create a clean local worktree on a non-`master` branch for the bounded
   change.
3. Pass the required pre-merge validation from that worktree.
4. Open a PR linked to the issue.
5. Start code review and wait for at least one approving review with required
   conversations resolved.
6. Merge the reviewed PR into `master`. Use direct `master` pushes only for
   explicitly approved emergency repair.
7. Verify the four workload selection paths from clean committed `master`
   source.

Record accepted validation evidence in the project evidence store:

```text
<evidence-root>/reports/<validation-report>.md
```

Record the product commit, preserved evidence root, and any observability access
notes with environment-neutral host names:

```bash
ssh -L 3000:127.0.0.1:3000 <validation-host>
```

Record the master merge report in the project evidence store:

```text
<evidence-root>/reports/<master-merge-report>.md
```

Record the four-workload master verification report in the project evidence
store:

```text
<evidence-root>/reports/<four-workload-verification-report>.md
```

Three-workload selection verification command pattern:

```bash
ROOT=/tmp/tipoc-master-workload-selection-20260615-bbe67f2
go run ./cmd/tipoc init --template securities-tpce --out "${ROOT}/securities-tpce"
go run ./cmd/tipoc init --template securities-exchange-matching --out "${ROOT}/securities-exchange-matching"
go test ./core/contracts ./core/scenario ./cmd/tipoc ./core/execution -count=1
```

Evidence retention policy:

| Evidence class | Retention rule |
| --- | --- |
| Accepted validation evidence | Preserve the run evidence and runtime logs; do not run ordinary cleanup against that validation root. |
| Master merge report | Keep the report in the coordinator evidence repo because it records the ff-only merge command, source commit, and post-merge state. |
| Four-workload `/tmp` verification root | Treat `/tmp/tipoc-master-four-workloads-20260615-bbe67f2` as transient unless copied into a durable report or archived evidence directory. |

---

**Next**: Record any issues or missing steps in [sop-doc-issues.md](sop-doc-issues.md).