---
name: tidb-benchmark-narrative
description: >-
  Guides PMM and DevRel in presenting TiDB performance data credibly — from
  internal test results to published benchmark posts. Covers what makes a
  benchmark publishable, how to write about HTAP and AI workload results without
  overclaiming, the pipeline from eval result to distribution, and an explicit
  list of benchmark traps that destroy technical credibility. Use before publishing
  any performance claim, benchmark comparison, or test result externally.
---

# TiDB benchmark narrative

**Role:** Eval Strategist. Performance data is TiDB's most powerful trust signal with engineering audiences — and the most dangerous to misuse. A credible benchmark earns developer trust for months. A benchmark caught cherry-picking or omitting methodology earns a HN thread that damages credibility for years. This skill is about designing benchmark moments intentionally and publishing them honestly.

---

## 1. TiDB benchmark landscape

TiDB performance content falls into three distinct categories, each requiring different rigor:

| Category | Examples | Rigor level | Primary audience |
|---------|---------|------------|----------------|
| **Industry-standard benchmarks** | TPC-C, TPC-H, YCSB, sysbench | Highest — full reproducibility required | Engineers evaluating TiDB vs competitors |
| **AI/vector workload benchmarks** | ANN (approximate nearest neighbor) accuracy/latency, RAG pipeline latency, agent workflow throughput | Medium-high — methodology newer; community scrutiny growing | AI developers evaluating unified DB vs separate vector + relational |
| **Customer proof points** | "Plaid migrated 96 services"; "Manus handles 1M+ tenants"; "Pinterest 10x P99 latency" | Customer-approved claims only; not raw benchmark numbers | Buyers seeking social proof; enterprise evaluation |

**Critical distinction:** Customer proof points are testimonials, not benchmarks. Don't present them as benchmark data. Don't present benchmark data as if it's universally applicable.

---

## 2. Publishability criteria

Publish a benchmark only when all three are true:

| Criterion | What it means | TiDB-specific note |
|-----------|--------------|-------------------|
| **Methodology is reproducible** | Test setup, hardware/config, data set, and scoring are fully described and can be independently run | TiDB benchmarks must specify: node count, TiKV/TiFlash instance types, TiDB version, workload generator, and dataset size |
| **The result is novel** | Shows something the audience does not yet know | "TiDB is fast" is not novel. "TiDB's HTAP performance on a mixed 70/30 read/write workload at 50M rows, compared to MySQL + ClickHouse two-system setup with ETL lag" is novel. |
| **ICP-relevant** | The measured capability matters to the engineers in your target ICP | TPC-C matters to DBA audiences; ANN latency matters to AI developers; ClickHouse-comparable analytical latency matters to HTAP buyers |

A benchmark with weak methodology that shows strong results is worse than no benchmark — it invites replication that undermines trust.

---

## 3. TiDB-specific benchmark context requirements

TiDB's HTAP architecture means benchmarks are inherently more complex than single-workload systems. Always specify:

**For OLTP benchmarks (TPC-C, sysbench):**
```
- TiDB version: [v8.x]
- Cluster: [N TiKV nodes + N TiDB nodes] on [instance type]
- Workload: [warehouses / threads / transaction mix]
- Comparison baseline: [MySQL version / Aurora config / CockroachDB version] on [equivalent hardware]
- What was NOT tested: [e.g., did not test with TiFlash enabled; did not test cross-region replication]
```

**For HTAP benchmarks (TPC-H + TPC-C mixed):**
```
- OLTP workload: [TPC-C config]
- OLAP workload: [TPC-H queries, which ones]
- TiFlash enabled: [Yes / No] — required for HTAP claims
- Data freshness: [TiFlash replication lag measured at time of OLAP query]
- What was NOT tested: [e.g., TiFlash not tested at extreme scale; specific queries may vary]
```

**For AI / vector workload benchmarks:**
```
- Vector dimensions: [e.g., 1536-dim for OpenAI embeddings]
- Dataset: [ANN benchmark standard dataset or describe custom]
- Index type: [HNSW params]
- Query type: [pure ANN / hybrid ANN + SQL filter]
- Latency percentiles: P50, P95, P99 — report all, not just P50
- Comparison: [pgvector / Pinecone / Weaviate config and version]
- What was NOT tested: [scale beyond X million vectors; concurrent OLTP during ANN queries]
```

---

## 4. Writing benchmark content: rules

### Do

- State the **exact task**, **exact setup**, and **exact metric** in the first paragraph — methodology before conclusion
- Name the **comparison baseline** explicitly: "compared to [product version] on [date] using [method]"
- State **what the product does NOT do well** in the same post — this is the single most credibility-building move
- Show **raw numbers** in a table that readers can export or query
- Link to **eval code** where possible; open eval = 10× more credible
- For HTAP claims: always show **both OLTP and OLAP results** — not just whichever is stronger

### Do not

- Lead with the conclusion before the methodology ("TiDB is 5× faster" as the headline without setup)
- Use **relative improvement without absolute numbers** ("50% improvement" means nothing without baseline)
- Suppress failure cases or weaker benchmark queries from the data
- Claim "MySQL compatible" without linking to the compatibility docs
- Run OLAP benchmark without TiFlash enabled and call it "TiDB HTAP performance"
- Publish vector benchmark without specifying HNSW parameters and dataset

---

## 5. Benchmark distribution pipeline

```
INTERNAL EVAL RESULT
  ↓
Methodology documentation (write BEFORE distribution planning)
  ↓
"What this is NOT" limitations check (from tidb-product-narrative)
  ↓
Asset preparation: technical blog post + data table + reproducibility package (GitHub repo)
  ↓
48-hour seed window: 5-10 engineers who will find this genuinely interesting
  ↓
Primary publication: TiDB blog or PingCAP engineering blog
  ↓
HN "Show HN" or "Tell HN" depending on type
  ↓
Twitter/X thread → AI/DB newsletters → community Discords
  ↓
Engage with technical criticism within 4 hours — unanswered challenges read as evasion
```

**Timing:** Tuesday–Thursday, late morning Pacific or China morning (for simultaneous EN/CN push). Avoid major conference weeks (KubeCon, PgConf, NeurIPS) unless you're attending and can ride the attention wave.

---

## 6. Writing the HN/Twitter post

**HN format:**
- Title: `Show HN: [What you measured] — [single most interesting finding, factual]`
- Example: `Show HN: HTAP benchmark — TiDB vs MySQL+ClickHouse two-system setup, 50M rows mixed workload`
- First comment (required, write before posting): methodology, what was tested, what was NOT tested, known limitations, link to raw data and reproducibility package

**Twitter/X thread format:**
- Tweet 1: the finding, precisely stated, with the key number
- Tweets 2–4: one finding per tweet with data visualization or table screenshot
- Final tweet: link to full post + link to reproducibility package on GitHub
- Reply to every substantive technical challenge within 4 hours

---

## 7. When NOT to publish

| Situation | Why to wait |
|-----------|------------|
| Methodology is not yet reproducible | Will be challenged and found lacking; damages credibility more than silence |
| Result is strong only on benchmarks the product was optimized for | Community will discover this |
| Cannot respond to technical questions for 72 hours | HN/X discussions peak within 48 hours; silence reads as evasion |
| HTAP benchmark without TiFlash enabled | Publishing OLTP-only results as "HTAP" performance is misleading |
| Vector benchmark without hybrid query testing | Pure ANN latency ignores TiDB's differentiation (unified queries) |

---

## Pairs with

- `tidb-messaging-platform` — benchmark results are evidence for messaging; align framing before publishing
- `tidb-competitive-map` — competitive benchmark comparisons require approved competitor framing
- `tidb-messaging-review` — all benchmark content goes through messaging review before publishing
- `tidb-content-review` — final check before any benchmark content ships
- `tidb-tech-story` — benchmark posts follow practitioner voice rules: methodology before conclusion, limitations disclosed
- `@tidb-gtm-brain/playbooks/benchmark-standards.md` — approved methodology templates and standards