# Issue #65: Continue a Prepared Target After an Interrupted or Completed Run

## Problem

A large prepared TiDB target is expensive to import again. A completed run,
an operation/action failure after foreground traffic, or an ungraceful process
stop must not turn otherwise valid data into an unusable target. The next
pressure pass must continue from a safe trade-ID cursor rather than restart
from an older cursor or require a new import.

## Contract

The continuation boundary is a per-lease safety contract, not a single QPS or
stage-status bit. A usable boundary contains:

- the next trade-ID start for every runtime lease and the aggregate next start;
- runtime-owned table snapshot and fingerprint;
- prepared-domain, runtime-ownership, target, workload, projection, and
  schema/index binding evidence;
- the source terminal manifest, ready `run-continuation.json`, and passed
  global closeout, all bound by path and SHA-256.

Every normal run closeout writes a new boundary. A foreground run that reaches
closeout remains data-continuable even when an optional operation action or
operation-side acceptance threshold fails. Only real data/evidence corruption
or trade-ID exhaustion makes the target non-continuable.

## Recovery procedure

### Completed predecessor

Use the newest terminal `run/` or `run/try-0x/` manifest as
`--continuation-from`, then run the standard public `run` stage:

```bash
tipoc prepare --manifest <attempt>/run-profile-manifest.json --finish \
  --continuation-from <attempt>/run/try-02/run-profile-manifest.json
tipoc run --manifest <attempt>/run-profile-manifest.json
```

`prepare --finish` reuses immutable prepare evidence and validates live target
state; it does not import. The resulting
`prepare/post-run-continuation-baseline.json` points at the selected terminal
evidence. If a newer valid terminal boundary advances every lease, it replaces
the current prepare binding. Equal, regressing, or incomparable lease cursors
are rejected.

Use the ordinary `run` command for the first recovered pass. Once that pass
has written its own normal closeout, use `scripts/rerun/run-again.sh` for a
try-local pass with changed runtime pressure. The wrapper creates a new
`run/try-N` manifest, but it must hand execution to the same `run` recovery
path; it is not an alternative cursor-selection implementation.

### Process killed before closeout

Do not edit evidence JSON or guess a cursor. The killed run is not a terminal
predecessor, so select the newest *earlier* completed predecessor and keep the
target untouched. Revalidate it with `prepare --finish --continuation-from`,
then start the normal `run` command. Before workers begin, `run` obtains the
live runtime trade-ID high-water and publishes an attempt-local recovery
boundary when needed. It compares that recovery with the prepare-bound
predecessor per lease and refuses any ambiguous regression.

`scripts/rerun/run-again.sh` is a convenience wrapper that creates a new
`run/try-N` manifest and then invokes the same public `run` path. It does not
have a separate safety model.

### Operator decision table

| Previous state | Safe next action | Must not do |
| --- | --- | --- |
| Latest run has terminal manifest, ready boundary, and passed closeout | Bind it with `prepare --finish --continuation-from`, then start the next `run` (or a try-local `run-again` after normal recovery). | Import, reset data, or reuse an older cursor. |
| Latest process was forcibly killed before closeout | Bind the newest earlier terminal predecessor; let `run` recover from live high-water. | Treat the killed run as terminal or hand-edit a boundary. |
| Optional operation/action acceptance failed after foreground closeout | Reuse the foreground terminal boundary if all continuation evidence is valid. | Mark the target or foreground workload failed solely for that secondary result. |
| Immutable prepare binding, runtime snapshot, lease geometry, or trade budget is invalid | Stop and repair/reprepare as required. | Attempt continuation by overriding validation. |

## Acceptance evidence

Before the next pressure round, retain:

1. passed `prepare-completion-validation.json` and
   `post-run-continuation-baseline.json`;
2. provenance paths/SHA-256 values for the selected predecessor;
3. selected aggregate and per-lease next trade-ID values;
4. runtime-owned table snapshot fingerprint and prepare-end index evidence.

After the round, retain its `global-run-closeout.json`,
`run-continuation.json`, worker summaries, and operation artifacts. This makes
the following round recoverable without importing the prepared data again.

## Non-goals

- This is not a clean-baseline restore. A continued result is labelled
  `continuation` unless the target was restored or reprepared.
- It does not permit changing prepare-bound scale, source, import policy, or
  schema without a new prepared attempt.
- It never permits manual editing of continuation files to bypass validation.