# TiPOC SOP Documentation Issues Use this file when the SOP is unclear, when an operator or AI agent needs more research before executing, or when the current scripts do not expose a command surface that the SOP needs. Do not put secrets in this file. ## Open Issues ### SOP-ISSUE-001: Add first-class rerun flags for exchange matching ratios Status: closed Owner surface: `scripts/rerun/run-again.sh`, `docs/poc-sop/deployment-poc-sop.md` Problem: the init wrapper can render `pressure.workload_profile.exchange_matching_write_percent` through `TIPOC_EXCHANGE_MATCHING_WRITE_PERCENT` and can render `pressure.workload_profile.exchange_matching_sql_write_percent` through `TIPOC_EXCHANGE_MATCHING_SQL_WRITE_PERCENT`. Same-attempt rerun also needs first-class flags so operators do not hand-edit try-local manifests. Impact: closed. A human or AI can run multi-round pressure with duration, concurrency, SQL pool, pressure-node, workload-mix, workload-profile, exchange matching deck write ratio, and observed SQL write ratio overrides. Closure evidence: `scripts/rerun/run-again.sh --help` exposes `--exchange-matching-write-percent` and `--exchange-matching-sql-write-percent`. The wrapper writes both fields into `runtime.workload_profile` and `scenario.pressure.workload_profile`, records both values in `rerun-receipt.json`, and keeps the evidence label rule unchanged: same-attempt rerun is continuation evidence unless the target was restored or reprepared. Close condition: satisfied by focused dry-run evidence proving both fields land in the try-local manifest and receipt. ### SOP-ISSUE-002: Capture the first real multi-round run as a filled example Status: open Owner surface: `docs/poc-sop/deployment-poc-sop.md`, project evidence store Problem: this SOP pass validates the documented command surfaces and parameter boundaries, but it does not execute a real TiDB, S3, EC2, Clinic, or pressure environment. The runbook is therefore ready as an operator procedure, not as a proven environment-specific transcript. Impact: a human or AI can follow the SOP, but the first execution in a new environment should preserve the filled intake form, rendered spec, manifests, receipts, reports, and any blocked steps as a reusable filled example. Workaround: after the first approved real run, copy a redacted command transcript and evidence index into the project evidence store and link it from this issue. Close condition: a real multi-round attempt has been run with at least one `tpce` round and one `exchange_matching` round, evidence paths are recorded without secrets, and the SOP has been checked against the transcript. ### SOP-ISSUE-003: Add a zero-knowledge starting point before the intake form Status: closed Owner surface: `docs/poc-sop/README.md`, `docs/poc-sop/poc-intake-form.md` Problem: the SOP is correct when the operator already knows what TiPOC is, what TPC-E measures, what deployment type to pick, and what environment resources are available. A human or AI agent who has never seen TiPOC before opens the intake form and sees a dense table of empty fields with no explanation of what any option means or how to choose. Impact: a zero-knowledge user abandons the SOP before filling the first field. The current entry assumes domain knowledge that the intake form itself does not provide. Workaround: a human can ask a colleague or an AI assistant to explain each field, but the SOP itself should not require external interpretation to complete step one. Close condition: 1. `docs/poc-sop/README.md` opens with a "First time here?" section that asks four routing questions (deployment, scale, workload, available resources) and links to the relevant paths. 2. `docs/poc-sop/poc-intake-form.md` includes a short explanation per field group before the table, or the README explains each column before the user reaches the form. 3. A zero-knowledge tester can answer the routing questions and know which deployment wrapper, template, scale, and workload to pick without reading any document outside this `docs/poc-sop/` directory. ### SOP-ISSUE-004: Prepare generation lacks coordinator-owned SSH launcher Status: mitigated Owner surface: `scripts/prepare/start-generation.sh`, product prepare generation launcher, `docs/poc-sop/deployment-poc-sop.md` Problem: the run stage has a coordinator-owned SSH launch path for pressure runners when the manifest requests `runtime.runner_launch.mode=coordinator_ssh`, but prepare generation still requires an explicit prepare process on every generation node. Running `tipoc prepare` only on the coordinator does not start the other EC2 generation nodes. Impact: mitigated for SOP-driven operations. Large prepares can still appear stuck or underutilized if operators bypass the prepare launcher and expect run-stage launch behavior to apply to prepare. Failure is easy to misdiagnose because the coordinator process is healthy while remote workers were never started. Mitigation: use `scripts/prepare/start-generation.sh` from the distributed EC2 generation prepare section in `deployment-poc-sop.md`. In the normal case, `--manifest` is the only required input: the script derives generation nodes, default addresses, coordinator node, SSH user/key refs, remote paths, and object-store URI context from the manifest. Optional overrides cover only execution-environment differences such as outside-VPC SSH hosts, unresolved key refs, alternate coordinator endpoint host, remote path differences, port, coordinator wait timeout, and receipt location. The script preflights remote binary and manifest presence, starts the coordinator, waits for its local TCP port before launching workers, and writes a sanitized launch receipt with object-store query strings and fragments stripped. Manual fallback: distribute the same binary and manifest to every generation node, verify binary checksums, open worker-to-coordinator TCP `18080`, start the coordinator with `TIPOC_PREPARE_GENERATION_COORDINATOR_LISTEN` and a ledger path, then start each worker with `TIPOC_PREPARE_GENERATION_NODE_ID` and `TIPOC_PREPARE_GENERATION_COORDINATOR_ENDPOINT`. Close condition: fully closed only when prepare has a product-owned coordinator launcher or generated wrapper that starts all configured generation workers from the coordinator, writes sanitized launch receipts, validates remote manifest and binary path consistency, verifies coordinator readiness, and preserves the public `init -> prepare -> run -> cleanup` flow. The current operator script closes the SOP hand-written-command gap but is not product-owned automatic prepare launch behavior. ### SOP-ISSUE-005: Large distributed generation needs explicit ledger and HTTP timeout diagnostics Status: open Owner surface: product prepare coordinator implementation, generated runbooks, `docs/poc-sop/deployment-poc-sop.md` Problem: 8-node EC2 generation can expose coordinator bottlenecks when worker `RecordObject` or `Heartbeat` calls synchronously persist a large ledger or use a short HTTP client timeout. The operator-facing symptom is repeated worker timeout around the coordinator endpoint even after the security group and manifest distribution are correct. Impact: operators may keep restarting workers or clearing S3 when the real issue is coordinator-side persistence and timeout behavior. This can waste a large source-generation attempt and leave partial S3 objects under the attempt prefix. Workaround: capture coordinator logs, worker logs, ledger size, binary checksum, node count, source prefix, S3 write throughput, and worker CPU utilization. Treat repeated ledger request timeouts as product/runtime evidence. Use a binary that contains the narrow coordinator fixes before rerunning at the same scale. Close condition: product logs and generated runbooks expose a clear diagnostic for coordinator ledger persistence latency and client timeout settings, and the prepare coordinator remains stable under the documented multi-node generation shape. ### SOP-ISSUE-006: PrivateLink endpoint acceptance and DNS readiness were underspecified Status: mitigated Owner surface: `docs/poc-sop/environment-setup.md`, `docs/poc-sop/deployment-poc-sop.md`, `docs/poc-sop/poc-intake-form.md` Problem: the SOP said to create or identify a VPC endpoint, enable private DNS, and validate SQL, but a real TiDB Cloud PrivateLink service can require owner acceptance before private DNS can be enabled. Operators may also confuse the endpoint id or service name with a connectable SQL hostname. Impact: a zero-knowledge operator can see `Active` or `available` state and still fail to connect because the actual `DnsEntries` host was not copied, private DNS has not propagated, or the endpoint was created outside the EC2 generation/pressure VPC path. Mitigation: the environment setup now documents the acceptance-first flow: create the Interface VPC Endpoint without private DNS when required, register or approve the generated `vpce-...` id in TiDB Cloud, wait for `available`, enable private DNS only after acceptance, allow EC2-to-endpoint TCP `4000`, and validate DNS/TCP/SQL from every EC2 node. The deployment SOP also states that `TIPOC_SQL_ENDPOINT` must not be built from the endpoint id itself. Close condition: fully closed after a real filled transcript proves the documented PrivateLink path from endpoint creation through SQL login without any operator-inferred steps. ### SOP-ISSUE-007: TiDB import-worker access to generated S3 source was too abstract Status: mitigated Owner surface: `docs/poc-sop/environment-setup.md`, `docs/poc-sop/deployment-poc-sop.md`, `docs/poc-sop/poc-intake-form.md` Problem: the SOP correctly said that EC2 S3 access is not enough, but it only listed proof categories for TiDB import access. It did not give concrete permission boundaries, credential lifetime requirements, Global Sort scratch handling, Role ARN URI-shape probing, or a minimal import probe pattern. Impact: source generation can complete successfully while `prepare --ops import,add-index` still fails because TiDB Cloud import workers cannot list or read the attempt-level `source/` parent prefix and its table child prefixes, temporary query credentials expire during a multi-hour import/index stage, a security enhanced mode cluster rejects a naked `s3://...` URI or an explicit `external-id` URI parameter, or a self-managed Global Sort scratch prefix lacks write/read/delete access. Mitigation: the environment setup now separates EC2 generation-node access, TiDB import-worker source read access, and self-managed Global Sort scratch access. It documents TiDB Cloud external storage access through an approved Role ARN or AWS access key, minimum parent source-prefix permissions that cover table child prefixes, scratch-prefix permissions when `CLOUD_STORAGE_URI` is used, credential TTL handling for query-string credentials, and a small `IMPORT INTO` probe template with redacted evidence requirements. It now treats external ID as TiDB Cloud UI and AWS trust-relationship configuration unless the target-cluster probe proves an explicit URI parameter is accepted. The deployment SOP now blocks Cloud prepare import until these gates are recorded. Close condition: fully closed after a real filled transcript proves the TiDB Cloud import-worker S3 access path and shows the documented probe or equivalent owner proof, including the accepted redacted Role ARN URI shape, before import starts. ### SOP-ISSUE-008: SQL credential materialization is still too implicit Status: mitigated Owner surface: `docs/poc-sop/environment-setup.md`, `docs/poc-sop/deployment-poc-sop.md` Problem: the SOP used `env:TIDB_DSN` as a secret reference and correctly warned operators not to paste secrets, but a zero-knowledge operator still needed to infer how to safely place the DSN on the remote coordinator, verify it without printing it, and avoid shell-history or receipt leakage. Impact: `prepare --ops import,add-index` can remain blocked even after PrivateLink DNS/TCP is ready because no live SQL credential is materialized for the process that starts `tipoc`. Mitigation: the environment setup now documents a non-printing remote secret-file materialization procedure, a `present`/`missing` check, SQL probe expectations, and the rule that the DSN must never be passed as a command-line argument or recorded in manifests, receipts, issues, or chat. Close condition: fully closed after a real filled transcript proves DSN materialization, SQL probe success, and cleanup/redaction behavior without any secret exposure. ### SOP-ISSUE-009: Generation-only resume to import/add-index needed a remote launcher recipe Status: mitigated Owner surface: `docs/poc-sop/deployment-poc-sop.md` Problem: the SOP listed `--ops import,add-index`, but after a generation-only distributed prepare a zero-knowledge operator still needed to infer whether to start all generation nodes again or run only the coordinator-side TiDB import and index stage. Impact: operators can either fail to start import/index or accidentally restart generation workers when the correct next step is a coordinator-owned resume against existing source objects. Mitigation: the deployment SOP now includes a fail-closed coordinator resume recipe that checks the remote binary, manifest, and DSN secret file, exports the DSN without printing it, starts `tipoc prepare --ops import,add-index`, and writes timestamped log and pid files. Close condition: fully closed after a real filled transcript proves the resume recipe starts import/index and records terminal prepare evidence.