# TiDB Benchmark Standards

> Methodology requirements and approved test results for TiDB performance claims.  
> Load before writing any performance content. All benchmark claims must trace to an entry here.

Last updated: 2025

---

## Publication criteria (must meet all three)

1. **Methodology is reproducible:** Test setup, hardware, config, and scoring are fully described and runnable independently
2. **The result is novel:** Shows something the audience doesn't already know
3. **ICP-relevant:** The measured capability matters to the target ICP

---

## Required disclosure fields by benchmark type

### OLTP benchmarks (TPC-C, sysbench, YCSB)

```
TiDB version: [v8.x or specific]
Cluster config: [N TiDB nodes + N TiKV nodes on {instance type}]
TiFlash enabled: [Yes / No]
Workload: [TPC-C warehouses / sysbench threads / YCSB config]
Dataset: [size in GB, number of rows]
Duration: [test window]
Comparison baseline: [{product} {version} on {equivalent hardware}]
Benchmark date: [YYYY-MM-DD]
Reproducibility: [link to test scripts / methodology doc]
What was NOT tested: [specific exclusions]
```

### HTAP benchmarks (TPC-H + TPC-C mixed)

```
[All OLTP fields above, plus:]
TiFlash replication lag at query time: [measured latency from TiKV write to TiFlash read]
OLAP workload: [TPC-H queries tested — list specific query numbers]
Query routing: [cost-based optimizer / manual hint]
Mixed workload ratio: [OLTP TPS during OLAP execution]
What was NOT tested: [e.g., "TPC-H Q{N} excluded due to {reason}"]
```

### Vector search benchmarks (ANN / hybrid query)

```
TiDB version: [v8.x or specific]
Vector dimensions: [e.g., 1536 for OpenAI, 768 for other]
Dataset: [ANN benchmark standard / custom — describe]
Index type: HNSW
HNSW params: [M, ef_construction, ef_search]
Query type: [pure ANN / hybrid ANN+SQL filter / describe]
Metrics: [recall@K, P50/P95/P99 latency, QPS]
Concurrency: [concurrent clients]
Comparison: [{vector DB} {version} with {index config}]
What was NOT tested: [e.g., "concurrent OLTP writes during ANN queries not tested"]
```

---

## Benchmark writing rules

### Do
- **Always qualify latency with a percentile** (p50, p95, or p99) — bare "latency" numbers are meaningless without one. Throughput in QPS or rows/sec.
- State exact task, setup, and metric in the first paragraph — methodology before conclusion
- Name the comparison baseline with product name + version + date
- State what the product does NOT do well in the same post
- Show raw numbers in an exportable table
- Link to reproducibility package (GitHub preferred)
- For HTAP: always show both OLTP and OLAP results together

### Do not
- Lead with the conclusion before the methodology
- Use relative improvement without absolute numbers ("50% better" means nothing without baseline)
- Suppress failure cases or weaker query results
- Claim "MySQL compatible" without the caveats
- Run OLAP benchmark without TiFlash enabled and call it "HTAP performance"
- Publish vector benchmark without specifying HNSW params and dataset

---

## Approved benchmark results (can reference externally)

*[This section to be populated as benchmarks are run and approved by PMM + Engineering]*

| Benchmark | Config | Result | Date | Methodology doc | Status |
|-----------|-------|--------|------|----------------|--------|
| [TBD] | [TBD] | [TBD] | [TBD] | [TBD] | Pending |

---

## Distribution pipeline

```
Internal eval result
  ↓ Methodology documentation (before distribution planning)
  ↓ "What this is NOT" limitations check
  ↓ PMM + Engineering review
  ↓ Asset prep: blog + data table + GitHub reproducibility package
  ↓ 48-hour seed to 5-10 target engineers
  ↓ Primary: TiDB blog + HN Show HN (same morning, T–W–Th)
  ↓ Secondary: Twitter/X thread → AI/DB newsletters → community channels
  ↓ Engage with technical criticism within 4 hours
```

**Timing:** Tuesday–Thursday, late morning Pacific or China morning for simultaneous EN/CN push.  
**Avoid:** Major conference weeks (KubeCon, NeurIPS, PGConf) unless attending.

---

## When NOT to publish

| Situation | Reason |
|-----------|--------|
| Methodology not yet reproducible | Will be challenged; damages credibility |
| Result only strong on optimized benchmarks | Community will discover this |
| Cannot respond to technical questions for 72h | HN/X peaks in 48h; silence = evasion |
| HTAP benchmark without TiFlash enabled | Misleading to call it HTAP |
| Vector benchmark without hybrid query testing | Misses TiDB's differentiation |

---

## Competitive benchmark rules

- Never benchmark against a competitor without PMM + Legal review
- Use the competitor's recommended configuration from their own documentation
- Run the comparison on equivalent hardware or document the hardware differences
- Do not suppress competitor results that outperform TiDB in specific queries
- Include a "when to use [competitor] instead" section in the post — this is the single most credibility-building move