# TiPOC Rerun Operator Scripts ## Build A Deployable TiPOC Binary Formal Internet builds must use the fail-closed builder, including builds whose source is a linked Git worktree: ```bash scripts/rerun/build-tipoc-binary.sh \ --source "$TIPOC_SOURCE" \ --output "$TIPOC_BINARY" ``` The builder rejects dirty source, clones exact HEAD into a temporary full Git checkout, and activates the output only after `go version -m` reports `vcs=git`, the exact source revision, and `vcs.modified=false`. A direct `go build` is not deployable evidence. The deploy helper repeats the metadata check against the clean source HEAD before any upload. ## Run Again From A Local Mirror Use `scripts/rerun/run-again.sh` when creating a new `run/try-0x` rerun manifest from an existing attempt. For a copied local mirror of a remote attempt, keep the remote paths in the source manifest and provide the local and remote attempt roots explicitly: ```bash scripts/rerun/run-again.sh \ --from-manifest "$LOCAL_ATTEMPT/run/try-06/run-profile-manifest.json" \ --local-attempt-root "$LOCAL_ATTEMPT" \ --remote-attempt-root "$REMOTE_ATTEMPT" \ --try auto \ --duration-seconds 300 \ --exchange-matching-write-percent 70 \ --exchange-matching-sql-write-percent 70 \ --dry-run ``` `--local-attempt-root` is the source of prepare evidence, prior try evidence, and the new local try files. `--remote-attempt-root` is used to set the generated try manifest's `runtime.runner_launch.remote_manifest_path` and remote log path. If those remote fields are already present in the source manifest, the script can derive the remote attempt root; pass the flag when there is any ambiguity. Direct local attempts still work with only `--from-attempt` or `--from-manifest`. The script fails before writing a run try when the local prepare evidence directory is missing. For an attempt with query cost enabled, the periodic Statement Summary RU collector can be tuned per try without hand-editing the compiled manifest: ```bash scripts/rerun/run-again.sh \ --from-attempt "$ATTEMPT_ROOT" \ --query-cost-poll-interval-seconds 5 \ --query-cost-poll-timeout-seconds 2 \ --dry-run ``` The interval must be at least 5 seconds and the timeout must be positive and strictly lower than the interval. The wrapper first verifies that the source scenario and frozen query-cost contract agree, then updates both copies and records the overrides in `rerun-receipt.json`. It preserves the frozen, SHA-pinned target deployment attestation. The same formal path rotates the three Query Cost modes required for an attempt-RU overhead campaign. Pass mode and source together so the intended delta is explicit in `rerun-receipt.json`: ```bash # Exact Query Cost bypass. scripts/rerun/run-again.sh --from-attempt "$ATTEMPT_ROOT" \ --query-cost-mode off \ --query-cost-attempt-ru-source off \ --dry-run # Business-statement latency only. scripts/rerun/run-again.sh --from-attempt "$ATTEMPT_ROOT" \ --query-cost-mode latency_only \ --query-cost-attempt-ru-source off \ --query-cost-deployment-attestation-path "$ATTESTATION_PATH" \ --query-cost-deployment-attestation-sha256 "$ATTESTATION_SHA256" \ --dry-run # Latency, Statement Summary, and same-session last-query-info RU. scripts/rerun/run-again.sh --from-attempt "$ATTEMPT_ROOT" \ --query-cost-mode latency_and_statement_ru \ --query-cost-attempt-ru-source tidb_last_query_info \ --query-cost-attempt-ru-timeout-milliseconds 500 \ --query-cost-attempt-ru-failure-budget 100 \ --query-cost-deployment-attestation-path "$ATTESTATION_PATH" \ --query-cost-deployment-attestation-sha256 "$ATTESTATION_SHA256" \ --dry-run ``` The Go materializer validates the final mode/source matrix, timeout and failure budget, rejects source scenario/frozen-contract drift before claiming a try directory, and updates both contracts atomically. These flags reuse the prepared attempt and public `tipoc run` stage; they do not create a sidecar stage or authorize hand-edited manifests. When an `off` source attempt is enabled for a later try, pass the attestation path and SHA-256 together. The materializer verifies the target-bound document, freezes only its verified evidence into the try manifest, and never retains the mutable path. To change hotspot shape for one new try, pass the same strict JSON or YAML override accepted by `tipoc run`: ```bash scripts/rerun/run-again.sh \ --from-attempt "$ATTEMPT_ROOT" \ --workload-profile-override ./hotspot-round.json \ --dry-run ``` The helper passes the file to the hidden Go materializer; it does not parse hotspot fields in Bash. The new try stores the complete resolved profile in `run-profile-manifest.json`, writes `workload-profile-resolution.json`, embeds that resolution in `rerun-receipt.json`, and publishes `workload-profile-materialization.json` last as the commit marker. The Go materializer exclusively claims a previously absent try directory; any existing or partially written try is rejected. Standard run and preflight verify the marker and profile digest before execution. Non-dry runs then invoke the standard `tipoc run --manifest ...` path. For external or coordinator-SSH launches, add `--sync-try-to-pressure-nodes` after confirming the manifest has SSH user, key reference, coordinator node, pressure node addresses, and any required bastion. The sync path copies the new try files to the pressure nodes and writes remote-compatible manifest output paths; local non-dry-run execution is blocked for that mode so remote paths are not accidentally written from the operator host. ## Deploy TiPOC Binary Use `scripts/rerun/deploy-tipoc-binary.sh` before a rerun when replacing the remote `tipoc` binary across an external pressure-node inventory: ```bash scripts/rerun/deploy-tipoc-binary.sh \ --source "$TIPOC_SOURCE" \ --local-binary ./tipoc \ --remote-binary /home/ec2-user/tipoc/bin/tipoc \ --node-file "$TIPOC_RUN_NODE_FILE" \ --ssh-key "$TIPOC_RUN_SSH_KEY" \ --ssh-user ec2-user \ --expected-arch aarch64 ``` For `ssh_mode=proxy`, also set `TIPOC_DEPLOY_BASTION` or pass `--bastion`. The deploy script uses the same node inventory shape as the external start launcher: ```text node_id ssh_host ssh_mode private_ip expected_slots expected_sql_pool ``` The helper first checks every node for the expected architecture and for an empty `tipoc run` process set. Only after every node passes that preflight does it upload the local binary, recheck the node for active `tipoc run`, atomically replace the remote binary, set executable mode, and verify the remote SHA-256 matches the local SHA-256. Output is intended to be copied into operator evidence and redacts DSN, password, token, and object-store credential shapes from SSH output. ## Stop External Run Use `scripts/rerun/stop-external-run.sh` as the reusable stop primitive for generated external pressure-run launchers. It accepts flags or matching `TIPOC_RUN_*` environment variables: ```bash scripts/rerun/stop-external-run.sh \ --launch-id "$TIPOC_RUN_LAUNCH_ID" \ --remote-root "$TIPOC_RUN_REMOTE_ROOT" \ --node-file "$TIPOC_RUN_NODE_FILE" \ --ssh-key "$TIPOC_RUN_SSH_KEY" \ --manifest "$TIPOC_RUN_MANIFEST" ``` The node file uses the same shape as `start-run-template.sh`: ```text node_id ssh_host ssh_mode private_ip expected_slots expected_sql_pool ``` For `ssh_mode=proxy`, also set `TIPOC_RUN_BASTION` or pass `--bastion`.