# tipoc-tv Design Capsule This file is the repo-contained design contract for the third-version `tipoc` TPC-E securities benchmark implementation. The full design books live in the outer project `dev/` directory and may not ship with this product repository. When those external documents are absent, this capsule and `docs/CRITICAL-PATH-MAP.md` are the minimum design truth that future implementation work must load before editing critical logic. ## Product Objective `tipoc-tv` implements a TPC-E-derived securities brokerage simulation in Go. The accepted completion target is a real local `50G` four-stage execution on `mac-mini-2`, followed by staged pressure increases until the target machine's processing ceiling is observed. The first accepted `50G` pass must prove: 1. one numeric `50G` run profile through `init -> prepare -> run -> cleanup` 2. generated TPC-E CSV source objects published directly to S3-compatible object storage when an object-store URI is configured 3. TiDB `IMPORT INTO ... WITH GLOBAL SORT` prepare loading 4. 33 catalog tables represented in prepare-load evidence 5. `trade_request` as the single explicit empty prepare table 6. post-import statistics validation and BR restore-boundary evidence 7. zero blocked runtime statements 8. measured `throughput.statement_qps >= 10000` 9. project guard pass with `tools/tiforge_execution_guard.py --attempt-root` ## Public Workflow The product public stage model has exactly four stages: 1. `init` 2. `prepare` 3. `run` 4. `cleanup` Do not add public product stages such as `validate`, `report`, `archive`, or `closeout`. Validation, reporting, archive packaging, and closeout are internal run evidence concerns. `cleanup` only applies the declared cleanup policy. ## TPC-E Securities Model The securities model uses the TPC-E-style brokerage world rather than a generic SQL workload. The catalog has 33 tables. Empty tables still count as catalog tables and must be created and reported. `trade_request` is expected to be empty in prepare for the current model and must appear as the explicit empty prepare table in run evidence. The Issue #95 order-lifecycle extension freezes 126 logical runtime statement identities: the existing 124 identities plus Trade Result's conditional insufficient-position cancellation and its terminal history record. The two extensions preserve the immutable sell side and quantity, execute only after authoritative holding locks prove the position insufficient, and must not be executed unconditionally by a compatibility fallback. The runtime actors are: 1. Customer Emulator, or `CE`, which consumes global deck tokens. 2. BrokerageHouse/SUT, which receives CE requests and executes TiDB SQL. 3. Market Exchange Emulator, or `MEE`, which owns Trade-Result and Market-Feed byproducts for Trade-Order. 4. DataMaintenance, or `DM`, which advances the global data-maintenance table rotation. 5. TradeCleanup, which runs at run closeout after a complete DM rotation boundary. Horizontal pressure nodes share one logical securities world. They may own different CE deck token ranges, trade id leases, MEE shards, DM leases, and TradeCleanup ranges. They must not create independent worlds or query-only substitutes. ## CE Transaction Mix The CE deck uses EGen-style weighted transaction selection. A CE token maps to exactly one CE transaction family: | family | weight | | --- | ---: | | Trade-Status | 190 | | Market-Watch | 180 | | Security-Detail | 140 | | Customer-Position | 130 | | Trade-Order | 101 | | Trade-Lookup | 80 | | Trade-Update | 20 | | Broker-Volume | 49 | | total | 890 | `Trade-Result` and `Market-Feed` are not CE deck families. They are MEE byproducts of a selected `Trade-Order`. `DataMaintenance` is DM-driven rather than CE-deck-driven. `TradeCleanup` is closeout-only. ## Prepare Data Generation Generated data uses deterministic LCG64 streams and stable ownership ranges so generation can scale horizontally. The generator divides load units and table row ranges into work units owned by generation workers. Each worker derives row content from deterministic stream coordinates instead of sharing mutable random state. This lets additional nodes or single-node worker slots increase prepare throughput without changing generated row identity. `prepare.csv_object_target_bytes` defaults to `536870912` bytes, or 512 MiB. It is configurable. The value is a target for rolling CSV source objects, not a hard semantic limit. A table larger than the target naturally emits multiple objects, and a small table may emit one smaller object. Implementations must not pad objects with meaningless rows and must not generate a larger complete file and then split it across CSV row boundaries. For `100T` and larger datasets, users may raise the target to values such as `1G` or `2G` to reduce object count. The planner still has to report whether the remaining object count is sufficient to keep TiDB `IMPORT INTO` parallel. ## Source Publication And Import The current supported formal source combination is: 1. `prepare.source_kind=generated_tpce` 2. `prepare.source_format=csv` 3. `prepare.import_executor=tidb_import_into_file` 4. `prepare.import_mode=import` Future formats or sources may be declared but must fail closed before side effects until implemented. `IMPORT INTO ... FROM SELECT` is not a valid source kind for this product because it reads existing TiDB query results instead of generating TPC-E data or migrating an upstream MySQL source. When `object_store.uri` is configured, formal prepare writes generated CSV source objects directly to that object store. It must not create complete local CSV source files and upload them afterward. Large S3-compatible source objects must use bounded-memory streaming multipart publication with part receipts, rolling checksums, complete/abort behavior, and `s3://...` source URIs emitted from generation time. TiDB target loading uses table-level `IMPORT INTO` with Global Sort. The prepare/import checkpoint is table-level and attempt-bound so prepare is reentrant only when run id, attempt id, profile id, table plan digest, and terminal imported state match the current attempt. The scenario provides a base Global Sort workspace URI, not per-table internal workspace paths. The import planner derives a table-scoped Global Sort URI for each table and uses that URI in the table's `CLOUD_STORAGE_URI` and plan digest. This keeps TiDB permission precheck and Global Sort scratch paths isolated between table imports while keeping user-facing environment bindings small. ## Index, Statistics, And BR Secondary index placement is frozen. Runtime-critical inline indexes stay in `CREATE TABLE` and are built as part of `IMPORT INTO`. Prepare-end supplemental `ADD INDEX` is limited to: 1. `i_dm_s_symb_date` 2. `i_t_s_symb_dts` 3. `i_holding_ca_symb_dts` Do not move all indexes back to post-import DDL. Prepare must run post-import `ANALYZE TABLE` and statistics validation before the BR boundary. BR evidence must preserve the restore boundary as a clean prepare-end checkpoint. The BR plan and receipts also preserve the selected TiUP component. An empty component uses `br`, while a target with a known TiDB/TiKV release should bind a matching component such as `br:v9.0.0-beta.2.pre-nightly` so backup and restore use a compatible BR binary. ## Runtime And Closeout Runtime executes a duration-based pressure window with concurrent pressure workers. Each worker consumes its allocated CE token and trade id range. The runtime must follow state-machine causality: 1. CE token consumption precedes BrokerageHouse/SUT handling. 2. Trade-Order commit precedes SendToMarket. 3. SendToMarket precedes MEE Trade-Result. 4. MEE ticker entry precedes Market-Feed. 5. DM advances one global table step at a time. 6. TradeCleanup runs only after a complete DM rotation boundary at closeout. The run receipt and global closeout must expose elapsed seconds, executed statement count, measured statement QPS, required minimum QPS, worker summaries, temporal closeout, query evidence, topology, and prepare-load summary. ## Pressure Ceiling Protocol After the first accepted `50G` pass, discrete pressure-ceiling steps must start from the same prepare-end BR baseline by default. Each step restores the clean baseline, runs a warmup window that is excluded from measured QPS, then runs the configured measurement window, defaulting to 3 minutes. This keeps concurrency steps comparable and prevents one step's runtime writes from contaminating the next step's initial securities world. Continuous ramp-up without restore is still valid, but it is a separate endurance or evolving-state test. Its evidence must not be mixed with the restore-based pressure-ceiling curve. ## Workload Profiles And Hotspots The default workload profile is `canonical`: it preserves the TPC-E CE transaction mix and derives customer, account, security, symbol, broker, and trade lineage from the owned deck token, trade id lease, prepared securities world, and deterministic random streams. Hotspot simulation is a required industry-standard capability, but it must be a declared workload profile rather than an accidental SQL implementation detail. Profiles such as `hot-symbol`, `hot-account`, `hot-broker`, `hot-trade-status`, `market-burst`, or `zipf-skew` may bias the entity selector toward a declared hot set or skew distribution. They must still consume CE deck tokens, preserve the runtime state machine, and route Trade-Result, Market-Feed, DataMaintenance, and TradeCleanup through their normal owners. Do not write fixed hotspot literals inside SQL builders. SQL builders consume lineage inputs only. The manifest, run receipt, and report must expose the selected workload profile, hotspot dimensions, hot-set size, skew parameters, lineage seed, and observed distribution summary so canonical and hotspot runs remain explainable and reproducible. The implementation boundary is `ScenarioSpec.pressure.workload_profile -> RuntimePlan.workload_profile -> runtime lineage selector -> LineageInput -> SQL context`. The selector may set explicit lineage values for customer, account, security ordinal, symbol, broker, or recent trade. The SQL context may render those values, but it must not decide hotspot policy. The observed distribution summary must be built from selected lineage values, not by parsing SQL text after the fact. ## Forbidden Shortcuts Do not reintroduce any of these shortcuts: 1. readiness, smoke, fixed small counts, or customer-count overrides as `50G` substitutes 2. TiForge workflow, task, gate, or planning vocabulary in product runtime concepts 3. local complete CSV staging for formal S3 prepare 4. complete-object in-memory single PUT for large formal S3 source objects 5. all-secondary-indexes post-import DDL 6. missing table-level import checkpoint, statistics validation, BR boundary, or 33-table prepare-load evidence 7. `SELECT COUNT(*)`, review SQL, placeholder SQL, no-op SQL, or query-only fallback for missing securities transactions 8. fixed all-family runtime bundles where every iteration executes every CE family 9. hotspot behavior hidden as hardcoded SQL literal values instead of declared workload profile lineage 10. weakening semantic, timeline, blocked-statement, throughput, or attempt-root guards to turn a failed run into a pass 11. treating old attempts produced by previous binaries as current pass evidence ## Critical Paths Critical paths and local instructions are defined in: 1. `docs/CRITICAL-PATH-MAP.md` 2. `core/execution/AGENTS.md` 3. `industries/securities/AGENTS.md` 4. `industries/securities/bootstrap/AGENTS.md` 5. `industries/securities/runtime/AGENTS.md` 6. `projections/tidb/AGENTS.md` Before editing any critical path, load this capsule, the critical path map, the repo root `AGENTS.md`, and the path-scoped `AGENTS.md`. ## Required Guards From the product repository, run: ```bash python3 tools/critical_path_guard.py python3 tools/go_lint_guard.py go test ./... ``` From the outer project repository, run: ```bash python3 tools/tiforge_execution_guard.py python3 tools/tiforge_execution_guard.py --attempt-root ``` The attempt-root form is required before claiming the real `50G` D8 pass.