# tipoc-tv

`tipoc-tv` drives the four public stages for a TPC-E-derived securities
workload. The public product flow is exactly:

1. `init`
2. `prepare`
3. `run`
4. `cleanup`

There is no public `validate`, `report`, `archive`, or `closeout` stage.
Validation, semantic proof, and report assembly are `run` closeout
responsibilities. `cleanup` only applies the configured cleanup policy.

## Deployment POC SOP

If you are running a TiPOC deployment POC, start with the zero-knowledge SOP
instead of this command reference:

<https://github.com/tidbcloud/TiPoC/tree/master/docs/poc-sop#readme>

The SOP walks through intake, node sizing, environment setup, `init`,
distributed `prepare`, `run`, `cleanup`, and evidence collection. It is the
operator-facing entry point for humans or AI agents that need to execute a POC
from scratch.

The sections below are the product command reference; they are not a
replacement for the SOP when running a full POC.

## Command Summary

The direct command shape is:

```bash
tipoc init --spec <spec.yaml> --out <attempt-root>
tipoc prepare --manifest <attempt-root>/run-profile-manifest.json
tipoc run --manifest <attempt-root>/run-profile-manifest.json
tipoc cleanup --manifest <attempt-root>/run-profile-manifest.json
```

After `init`, the attempt also contains wrapper scripts under `commands/` that
call the same public front door with the compiled manifest.

## Initialize An Attempt

`init` compiles a user-facing `spec.yaml` into a run-profile manifest and writes
the attempt-local command bundle, runbook, bootstrap plan, and node facts.

`--out` is required. Exactly one input source is required:

1. `--template <name>` starts from a built-in template. Current built-in names
   are `premium` and `tiup-op`.
2. `--spec <path>` initializes from a user-edited spec.
3. `--from-attempt <attempt-root>` reuses the previous attempt's `spec.yaml`.

Start from a template when creating a new environment-specific spec:

```bash
tipoc init --template premium --out runs/run-001/attempt-001
```

Edit the generated `spec.yaml`, then initialize the attempt:

```bash
tipoc init --spec runs/run-001/attempt-001/spec.yaml --out runs/run-001/attempt-001
```

To reuse a previous attempt's spec as the next attempt baseline:

```bash
tipoc init --from-attempt runs/run-001/attempt-001 --attempt-id attempt-002 --out runs/run-001/attempt-002
```

`--attempt-id` is optional and is useful when deriving a new attempt from an old
spec. `init` overwrites the attempt root in the copied spec with the new `--out`
path.

`init` writes the canonical attempt files:

```text
spec.yaml
run-profile-manifest.json
init/init-receipt.json
init/remote-bootstrap-plan.json
init/bootstrap-remotes.sh
init/nodes/<node-id>/node-bootstrap.json
commands/prepare.sh
commands/run.sh
commands/cleanup.sh
runbook.md
```

By default, remote bootstrap is planned but not executed. Use
`--remote-bootstrap=execute` only when the declared SSH user, password or key
reference, node addresses, and target attempt paths are ready. `init` does not
start `prepare`, `run`, `cleanup`, cloud resources, TiUP clusters, or object
store cleanup.

## Coordinator Contract

The machine where `init` is run is the entry node. The first declared generation
node is the default coordinator; if no generation node exists, the first
pressure node becomes coordinator. If the entry node is not the coordinator,
`init` plans a sync of the complete attempt/control state to the coordinator.

Every bootstrapped node receives its own `node-bootstrap.json` with generation
worker ownership, pressure worker ownership, stage routing, and coordinator log
polling information. Commands run on the coordinator start coordinator behavior;
commands run on other initialized nodes delegate to the coordinator endpoint and
poll coordinator status for user-visible progress.

Distributed Cloud source generation is the important exception to avoid
misreading this contract. Running `tipoc prepare` only on the coordinator does
not automatically start prepare workers on other EC2 generation nodes. For
multi-node Cloud generation, follow the SOP and start one prepare process per
generation node with the same manifest and binary, worker-to-coordinator TCP
`18080`, and the proper `TIPOC_PREPARE_GENERATION_*` environment variables.

## Prepare Data

`prepare` materializes the generated TPC-E securities source data and prepares
the TiDB target according to the manifest:

```bash
runs/run-001/attempt-001/commands/prepare.sh
# or
tipoc prepare --manifest runs/run-001/attempt-001/run-profile-manifest.json
```

The default full prepare operation set is:

```text
generate,import,add-index,backup
```

Use `--ops` only when intentionally running a bounded prepare sub-path:

```bash
tipoc prepare --manifest runs/run-001/attempt-001/run-profile-manifest.json --ops generate
tipoc prepare --manifest runs/run-001/attempt-001/run-profile-manifest.json --ops generate,import,add-index
```

The accepted operation names are `generate`, `import`, `add-index`, `backup`,
and `restore`. `restore` is mutually exclusive with the other operations.
`add-index` requires `import` in the same operation set, and `backup` requires
both `import` and `add-index`.

Use `--finish` only after import and required indexes were completed outside
the current `prepare` invocation and the live TiDB target can be checked:

```bash
tipoc prepare --manifest runs/run-001/attempt-001/run-profile-manifest.json --finish
```

To bind a new attempt to the retained target state after a completed run, pass
the prior terminal run or try manifest explicitly:

```bash
tipoc prepare \
  --manifest runs/run-002/attempt-002/run-profile-manifest.json \
  --finish \
  --continuation-from runs/run-001/attempt-001/run/try-02/run-profile-manifest.json
```

The prior run closeout captures all runtime-owned table counts in one TiDB
snapshot after foreground and Operation traffic have stopped. The new attempt
validates that ready continuation boundary, the passed global closeout,
target/workload/projection identity, complete runtime ownership and leases,
the exact 33-table import plan/checkpoint, prepare-owned table counts, current
runtime-owned counts, and prepare-end indexes before Operation prepare hooks
run. `prepare/post-run-continuation-baseline.json` references the prior
manifest, continuation, closeout, dataset, sizing, trade-ID budget, source
publication, import plan, and checkpoint by absolute path and SHA256 digest.
Those immutable artifacts are not copied and no new import checkpoint is
created. Run reporting keeps original imported rows separate from current
continuation rows. An active run, incompatible identity, missing or modified
evidence, table/index-set drift, or later row-count drift fails closed.

Restore is destructive and requires an explicit backup binding and confirmation
token:

```bash
tipoc prepare \
  --manifest runs/run-001/attempt-001/run-profile-manifest.json \
  --ops restore \
  --restore-backup-uri s3://bucket/tipoc/backup/attempt-001 \
  --restore-backup-receipt runs/run-001/attempt-001/prepare/br-checkpoint-receipt.json \
  --restore-confirm clean-target-and-restore
```

Formal prepare uses generated TPC-E CSV source objects and TiDB
`IMPORT INTO ... FROM FILE FORMAT 'CSV'`. When an object-store URI is
configured, source objects are streamed directly to that object store. Do not
prepare data through `INSERT`, `LOAD DATA`, stage-table merges, or
`IMPORT INTO ... FROM SELECT`.

`prepare.csv_object_target_bytes` defaults to `536870912` bytes. It is an
object target size, not a padding rule. Small tables may produce one smaller
object, and larger tables may produce multiple objects.

Prepare progress is written under the attempt evidence root as
`prepare-progress.json` and `prepare-progress.jsonl` when prepare execution
starts.

## Run Pressure

`run` executes the duration-based securities workload against the prepared
TiDB target:

```bash
runs/run-001/attempt-001/commands/run.sh
# or
tipoc run --manifest runs/run-001/attempt-001/run-profile-manifest.json
```

`run` requires `prepare_complete` or a compatible prior
`run_closeout_published` state. It does not accept prepare-only flags such as
`--ops` or `--finish`.

The runtime preserves one TPC-E securities world. Horizontal pressure nodes own
their assigned worker slots, CE deck ranges, trade-id leases, and actor
bindings; they must not become independent query-only worlds.

`run` publishes runtime evidence under the attempt evidence root, including:

```text
run/run-progress.json
run/run-progress.jsonl
run/runtime-plan.json
run/runtime-execution-records.json
run/timeline-validation.json
run/query-evidence.json
run/semantic-proof.json
run/run-continuation.json
run/run-receipt.json
run/global-run-closeout.json
run/run-report.md
run/run-report.html
```

The measured pass/fail throughput is the stable window after warmup and before
cooldown or pressure drain. The full window remains in evidence, but it is not
the throughput gate.

## Cleanup

`cleanup` applies the cleanup policy recorded in the manifest:

```bash
runs/run-001/attempt-001/commands/cleanup.sh
# or
tipoc cleanup --manifest runs/run-001/attempt-001/run-profile-manifest.json
```

It writes `cleanup/cleanup-receipt.json` and advances the manifest to
`cleanup_done`. It does not create the main validation report, and it does not
implicitly delete AWS, S3, TiUP, remote machines, or external cluster resources
unless a future implementation explicitly adds and evidences such behavior.

## Environment Script Packs

Reusable environment script packs live under `scripts/env/`. They render a
user-editable `spec.yaml` and then call the same public `tipoc init` front door.
They do not start `prepare`, `run`, `cleanup`, SSH bootstrap, cloud resources,
or object-store cleanup by default.

Local `10G` and `50G` TiUP tests use one script family and differ only by
scale:

```bash
scripts/env/tiup/init-attempt.sh --scale 10G --out runs/local-10g/attempt-001 --render-only
scripts/env/tiup/init-attempt.sh --scale 50G --out runs/local-50g/attempt-001 --render-only
```

Premium and future cloud shapes have dedicated script packs so users can swap
machine, cluster, and object-store values without editing product code:

```bash
TIPOC_SQL_ENDPOINT='tidb-host:4000' \
TIPOC_PD_ENDPOINT='pd-host:2379' \
TIPOC_OBJECT_STORE_URI='s3://bucket/tipoc/1t/attempt-001' \
TIPOC_GENERATION_NODES='node-a=10.0.0.10:15,node-b=10.0.0.11:15' \
TIPOC_PRESSURE_NODES='node-a=10.0.0.10:1:512,node-b=10.0.0.11:1:512' \
scripts/env/premium/init-attempt.sh --scale 1T --out runs/premium-1t/attempt-001 --render-only
```

After reviewing the rendered spec, initialize the attempt:

```bash
tipoc init --spec runs/premium-1t/attempt-001/spec.yaml --out runs/premium-1t/attempt-001 --remote-bootstrap plan
```

Use `--remote-bootstrap execute` only when the SSH user, credential reference,
node addresses, and target attempt paths have been checked. See
`scripts/env/README.md` for the full variable list and the `tiup`, `premium`,
`essential`, and `dedicated` script families.

The same four command stages are used for local TiUP, premium, future
essential, and future dedicated environments. Environment-specific values belong
in `spec.yaml` or the script-pack variables, not in product Go package names or
chat-only setup state. Future `essential` and `dedicated` packs are templates
until their live environment bindings are provided.