---
title: Distributed Tracing for TiDB
feature_id: distributed-tracing
owner: Ben Meadowcroft
status: draft
target_release: TBD
created: 2026-03-17
last_updated: 2026-03-23
---

# Distributed Tracing for TiDB

## 1. Executive Summary

This feature adds native distributed tracing support for TiDB using OpenTelemetry-compatible trace export and trace-context propagation from application requests into TiDB spans.

It is intended primarily for [PER-002 Application Developer](../../shared/personas.md), [PER-006 Database Reliability Engineer](../../shared/personas.md), and [PER-003 SRE / Platform Operator](../../shared/personas.md), with secondary relevance to [PER-005 Cloud Service Team / Internal Service Operator](../../shared/personas.md).

The feature closes the current observability gap where application traces stop at the database client boundary and where slow database activity in shared clusters is difficult to attribute to the originating service, tenant, or request.

The expected outcome is faster end-to-end troubleshooting, better collaboration between application and database teams, and a modern tracing foundation that replaces legacy OpenTracing-based behavior.

---

## 2. Press Release

### PINGCAP ANNOUNCES NATIVE DISTRIBUTED TRACING SUPPORT IN TIDB, ENABLING END-TO-END DATABASE TROUBLESHOOTING WITH OPENTELEMETRY

*PingCAP's native distributed tracing support enables developers and operators to diagnose database-backed application performance issues with a unified trace view across application and TiDB layers.*

SUNNYVALE, Calif., [Date] - PingCAP today announced native distributed tracing support for TiDB, giving application developers, database reliability engineers, and platform operators a new way to troubleshoot performance issues across complex distributed systems. The new capability allows TiDB to emit database traces using OpenTelemetry-compatible export and correlate those traces with upstream application requests, reducing the time and friction required to understand where latency is introduced.

"To help customers operate modern distributed applications, we need database observability that fits naturally into the same workflows they already use for application troubleshooting," said Ed Huang, CTO of PingCAP. "Distributed tracing for TiDB gives teams a shared view across application and database boundaries so they can identify performance bottlenecks faster and collaborate on fixes with less guesswork."

### About Distributed Tracing for TiDB

TiDB's distributed tracing support enables TiDB to export traces to a wide range of OpenTelemetry-compatible observability backends while preserving trace context propagated from the application tier into database spans.

Key capabilities include:

- end-to-end traceability from application request to TiDB execution
- OpenTelemetry-based trace export to existing customer observability tooling
- configurable sampling controls, including probabilistic sampling, parent-sampled capture, and slow-query capture
- rate limiting and detail controls to keep tracing operationally safe
- improved attribution of slow database work back to the originating request or service

These capabilities support two high-value troubleshooting workflows:

- application-down debugging, where developers follow a single trace from the API or service layer into TiDB internals to identify where time is spent
- bottom-up database attribution, where database specialists start from a slow query, trace, or performance symptom and identify the originating service or request path

### Availability

Distributed tracing support is planned for TiDB Self-Managed and supported TiDB Cloud tiers, with OpenTelemetry-compatible export and documented connectivity options including private connectivity in supported TiDB Cloud environments.

---

## 3. FAQ

### What problem does this solve?

Today, TiDB users can inspect slow queries, statement summaries, and database-local telemetry, but they cannot reliably connect that information to the originating application trace or request path.

This creates two recurring problems:

- application developers see the database tier as a black box once the request leaves the client library
- database specialists cannot easily identify which service, tenant, or request caused an observed performance issue in shared clusters

### Why now?

Several factors make this urgent:

- strategic customers have identified end-to-end tracing and tenant-level attribution as adoption blockers
- OpenTracing is legacy technology and no longer the right foundation for new product investment
- OpenTelemetry has become the industry standard with broad backend support across open-source, SaaS, and cloud observability platforms
- TiDB Cloud and self-managed customers increasingly expect observability features to fit existing tracing workflows instead of requiring TiDB-specific tooling

### Why this approach?

This proposal adopts OpenTelemetry for trace export and uses propagated trace context, primarily through SQL comment-based instrumentation, to link application traces with TiDB spans.

This approach is preferable because:

- it aligns with widely adopted open standards
- it minimizes application-side friction compared with session-variable approaches
- it allows dynamic sampling and rate limiting so operators can bound cost and overhead
- it supports both application-down debugging and bottom-up database attribution workflows

### What is out of scope?

- replacing TiDB's existing metrics and logging stack with OpenTelemetry metrics or logs
- automatic instrumentation of every supported application framework or ORM
- vendor-specific analysis UX inside TiDB for each tracing backend
- initial tracing support for all adjacent PingCAP products such as TiCDC, DM, BR, or TiProxy

---

## 4. Jobs to Be Done

#### JTBD-001  
When I am debugging an application request with end-user latency,  
I want the request trace to continue into TiDB internals,  
So I can identify whether planning, execution, or storage work is responsible.

#### JTBD-002  
When I am investigating a slow query or noisy workload from the database side,  
I want to trace the work back to the originating service, tenant, or request,  
So I can coordinate remediation with the right application owner quickly.

#### JTBD-003  
When I operate a production TiDB deployment,  
I want to enable and tune tracing without downtime and with bounded overhead,  
So I can improve visibility without destabilizing the system or overwhelming my tracing backend.

#### JTBD-004  
When my organization already standardizes on an observability platform,  
I want TiDB to export traces using supported OpenTelemetry-compatible connectivity patterns,  
So I can adopt database tracing without adding a separate analysis stack.

---

## 5. Problem and Context

TiDB's current tracing and performance workflow is fragmented for customers running modern distributed applications.

Today, customers typically combine:

- application-side tracing that ends at the SQL client boundary
- TiDB-local tools such as slow query logs, statement summaries, and dashboard views
- manual correlation between database events and application context

This creates material friction in several common situations:

- application teams cannot see where time is spent once work enters TiDB
- database reliability teams cannot attribute slow database work to the originating request path or service
- multi-tenant or shared-cluster operators struggle to identify workload ownership when several services share one TiDB cluster
- legacy OpenTracing support does not align with current customer tooling or industry direction

The problem is especially acute for strategic accounts operating many applications on a shared data platform, where high-value troubleshooting workflows require a common view across application and database boundaries.

OpenTelemetry provides the right ecosystem fit for this feature because it is broadly supported by tools such as Jaeger, Datadog, CloudWatch, Grafana, Honeycomb, and cloud-vendor observability stacks.

For context propagation, SQL comment-based approaches are attractive because they can often be adopted through existing ORM and client-library integrations and avoid connection-state ambiguity introduced by session-scoped alternatives.

---

## 6. Goals and Non-Goals

### Goals

#### G-001  
Enable TiDB to export distributed traces using OpenTelemetry-compatible endpoints.

#### G-002  
Propagate application trace context into TiDB spans so application and database work can be correlated.

#### G-003  
Support safe and configurable trace capture policies, including probabilistic sampling, slow-query capture, parent-sampled capture, and rate limiting.

#### G-004  
Provide enough TiDB span detail to support both application-down debugging and bottom-up database attribution workflows.

#### G-005  
Support both self-managed and TiDB Cloud deployments with appropriate connectivity and operational controls.

### Non-Goals

#### NG-001
Replace TiDB's existing metrics and logging systems with OpenTelemetry metrics or logs.

#### NG-002
Build full application instrumentation libraries for every language, ORM, or framework.

#### NG-003  
Create backend-specific analysis experiences inside TiDB for Jaeger, Datadog, Grafana, Honeycomb, or similar tools.

#### NG-004  
Deliver first-release tracing support for all other PingCAP data movement or backup products.

---

## 7. System Invariants

This feature should explicitly preserve relevant shared invariants from [shared/system-invariants.md](../../shared/system-invariants.md), especially [`INV-GLOBAL-004`](../../shared/system-invariants.md#inv-global-004-silent-corruption-is-unacceptable), [`INV-GLOBAL-005`](../../shared/system-invariants.md#inv-global-005-unhealthy-states-must-be-observable), [`INV-GLOBAL-008`](../../shared/system-invariants.md#inv-global-008-mixed-version-operation-must-be-bounded-and-defined), and [`INV-GLOBAL-010`](../../shared/system-invariants.md#inv-global-010-operator-actions-must-be-traceable).

#### INV-001 Tracing Must Not Affect Query Semantics
Enabling distributed tracing must not change query results, transaction semantics, or commit behavior.

#### INV-002 Export Failures Must Not Block Query Execution
Trace export failures, collector unavailability, or sampling configuration changes must not block normal query execution.

#### INV-003 Valid Trace Context Must Preserve Correct Correlation
When valid upstream trace context is present, TiDB must preserve correct trace correlation rather than attaching spans to the wrong request lineage.

#### INV-004 Unsampled Requests Must Not Emit Full Trace Payloads
Unsampled requests must not emit full trace payloads unless another configured capture rule applies.

#### INV-005 Misconfiguration and Exporter Failures Must Be Observable
Material misconfiguration or exporter health failures must be observable through supported logs, metrics, or status surfaces.

#### INV-006 Legacy OpenTracing Upgrade Behavior Must Be Bounded
Upgrade behavior from legacy OpenTracing configuration must remain explicitly documented and operationally bounded during migration.

---

## 8. User / Operator Scenarios

Scenario narratives are maintained in [Scenarios.md](./Scenarios.md).

The scenario set covers:

- cluster-level enablement and OpenTelemetry export
- application-down debugging with propagated trace context
- bottom-up attribution from slow database activity to application requests
- live sampling-policy changes, rate limiting, and session-scoped diagnostics
- exporter failure handling, private connectivity, and upgrade behavior

---

## 9. Functional Requirements

### FR-001 OpenTelemetry Trace Export

Priority: P0

Supports: `JTBD-003`, `JTBD-004`, `SCN-001`, `SCN-007`, `SCN-008`

TiDB supports export of eligible database traces to a configured OpenTelemetry-compatible endpoint. The capability is disabled by default.

#### Acceptance Criteria

Given tracing is disabled,  
When TiDB processes queries,  
Then TiDB does not export distributed traces.

Given tracing is enabled and a valid OpenTelemetry endpoint is configured,  
When TiDB processes requests that satisfy the active capture policy,  
Then TiDB exports TiDB spans to the configured backend.

### FR-002 Dynamic Sampling Policy

Priority: P0

Supports: `JTBD-003`, `SCN-004`, `SCN-005`, `SCN-006`

TiDB allows operators to configure and update cluster-level capture policy without planned downtime, including probabilistic sampling, slow-query capture, parent-sampled capture, and a global trace rate limit.

#### Acceptance Criteria

Given tracing is enabled on a running cluster,  
When an operator changes sampling policy values,  
Then the new policy takes effect without requiring a TiDB restart.

Given a rate limit is configured,  
When eligible trace volume exceeds the configured limit,  
Then TiDB enforces the limit consistently rather than exporting unbounded trace traffic.

### FR-003 Session-Scoped Diagnostic Capture

Priority: P1

Supports: `JTBD-001`, `JTBD-003`, `SCN-010`

TiDB supports an optional session-scoped tracing mode so a user can capture all trace activity for a current diagnostic session without changing the cluster-wide baseline.

#### Acceptance Criteria

Given cluster tracing is enabled,  
When a user enables session-scoped capture for the current session,  
Then TiDB captures trace activity for that session according to the documented session rule.

Given the session-scoped mode is disabled or the session ends,  
When subsequent work runs outside that session,  
Then session-specific capture no longer applies.

### FR-004 Trace Context Propagation

Priority: P0

Supports: `JTBD-001`, `JTBD-002`, `SCN-002`, `SCN-003`, `SCN-005`, `SCN-011`

TiDB parses supported propagated trace context from SQL statements, including `traceparent` and `tracestate`, and uses that context for correlation and sampling decisions.

#### Acceptance Criteria

Given a SQL statement contains valid propagated trace context in a supported format,  
When TiDB processes the statement,  
Then emitted TiDB spans are correlated to the upstream trace lineage.

Given the propagated context is missing or malformed,  
When TiDB processes the statement,  
Then query execution continues normally and TiDB follows documented local tracing behavior without misattributing the request.

### FR-005 TiDB Span Coverage and Detail Levels

Priority: P0

Supports: `JTBD-001`, `JTBD-002`, `SCN-002`, `SCN-006`, `SCN-010`

TiDB emits spans for high-level database processing stages and supports configurable inclusion of deeper storage-layer detail where applicable.

#### Acceptance Criteria

Given a traced request is captured,  
When a user inspects the TiDB trace,  
Then the trace contains enough stage-level detail to distinguish major database processing work.

Given storage-layer detail capture is disabled,  
When TiDB exports a trace,  
Then TiDB still emits the documented SQL-layer trace detail without the optional deeper layer spans.

### FR-006 Database-to-Application Attribution Metadata

Priority: P0

Supports: `JTBD-002`, `SCN-003`

TiDB spans include enough normalized metadata, such as identifiers and query-related enrichments, to support attribution from database-observed work back to the originating application request.

#### Acceptance Criteria

Given a traced slow or otherwise interesting query execution,  
When an operator starts from the TiDB span or slow-query context,  
Then they can identify the related trace lineage and the originating application request path using emitted metadata and trace identifiers.

### FR-007 Private Connectivity for Trace Export

Priority: P0

Supports: `JTBD-004`, `SCN-008`

TiDB Cloud supports trace export over a customer-approved private connectivity path where that deployment model and service tier support it.

#### Acceptance Criteria

Given a TiDB Cloud deployment supports private connectivity for tracing export,  
When an operator configures a supported private endpoint,  
Then TiDB exports trace traffic through that path instead of requiring public-internet egress for the tracing backend.

### FR-008 TiDB Cloud Private Endpoint Provisioning

Priority: P1

Supports: `JTBD-004`, `SCN-008`

TiDB Cloud provides a workflow to create or attach the private endpoint needed for supported private trace export scenarios.

#### Acceptance Criteria

Given a customer wants private trace export from TiDB Cloud,  
When they provide the required partner endpoint information through the supported workflow,  
Then TiDB Cloud creates or attaches the endpoint and makes it selectable for tracing export configuration.

### FR-009 Safe Live Reconfiguration and Exporter Health Signaling

Priority: P0

Supports: `JTBD-003`, `SCN-001`, `SCN-004`, `SCN-007`

Tracing configuration changes are logged, and exporter connectivity problems and recovery are surfaced through supported operational signals.

#### Acceptance Criteria

Given an operator changes tracing configuration,  
When the change is applied,  
Then the change is recorded in operator-visible logs or equivalent audit surfaces.

Given the configured exporter becomes unreachable after successful setup,  
When TiDB detects the condition,  
Then the failure is surfaced as an operational error and recovery is also surfaced when connectivity returns.

### FR-010 Legacy OpenTracing Upgrade Handling

Priority: P0

Supports: `JTBD-003`, `SCN-009`

TiDB defines explicit upgrade behavior for deployments that previously relied on legacy OpenTracing configuration.

#### Acceptance Criteria

Given a deployment upgrades from a version where legacy OpenTracing was enabled,  
When the upgrade completes,  
Then TiDB follows the documented compatibility behavior and surfaces a deprecation warning rather than silently changing tracing behavior.

Given a deployment upgrades from a version where legacy OpenTracing was not enabled,  
When the upgrade completes,  
Then TiDB does not begin exporting traces unless the new distributed tracing capability is explicitly configured.

---

## 10. Non-Functional Requirements

The feature should inherit applicable shared requirements from [security-requirements.md](../../shared/security-requirements.md), [observability-requirements.md](../../shared/observability-requirements.md), and [compatibility-guidelines.md](../../shared/compatibility-guidelines.md). Feature-specific exceptions should be documented explicitly during design review.

### NFR-001 Sampled Trace Performance Overhead

Category: performance

Priority: P0

For sampled traces, added P95 query-latency overhead should remain under 5 percent relative to an equivalent uninstrumented deployment.

Notes: Benchmark against a comparable deployment where distributed tracing instrumentation is not enabled.

### NFR-002 Non-Sampled Trace Performance Overhead

Category: performance

Priority: P0

For non-sampled traces with tracing enabled, added P95 query-latency overhead should remain under 1 percent relative to an equivalent uninstrumented deployment.

Notes: This applies when the feature is enabled but a request is not captured by the active sampling policy.

### NFR-003 Sensitive Query Data Handling

Category: security

Priority: P0

Trace payloads must not include raw SQL text by default; normalized query text and metadata should be used instead.

Shared References: [`SEC-005`](../../shared/security-requirements.md#sec-005-secrets-and-key-management), [`SEC-006`](../../shared/security-requirements.md#sec-006-transport-and-cryptographic-protection), [`SEC-007`](../../shared/security-requirements.md#sec-007-secure-defaults-and-surface-minimization), [`SEC-011`](../../shared/security-requirements.md#sec-011-sensitive-data-minimization-and-masking), [`SEC-012`](../../shared/security-requirements.md#sec-012-safe-error-handling-and-information-disclosure-control), [`SEC-016`](../../shared/security-requirements.md#sec-016-dependency-and-extension-safety)

### NFR-004 Operational Telemetry Expectations

Category: observability

Priority: P0

The feature must satisfy shared observability expectations for health, error visibility, alertability, operator-debuggable state, and convention consistency.

Shared References: [`OBS-001`](../../shared/observability-requirements.md#obs-001-health-and-progress-metrics), [`OBS-002`](../../shared/observability-requirements.md#obs-002-error-visibility), [`OBS-004`](../../shared/observability-requirements.md#obs-004-alertable-failure-signals), [`OBS-008`](../../shared/observability-requirements.md#obs-008-operator-debuggable-state), [`OBS-009`](../../shared/observability-requirements.md#obs-009-convention-consistency)

### NFR-005 Compatibility and Upgrade Boundaries

Category: compatibility

Priority: P0

Supported compatibility and upgrade behavior must follow the shared compatibility guidance for backward-compatible behavior, rolling upgrade support, mixed-version semantics, protocol compatibility, metadata compatibility, rollback boundaries, and unsupported combinations.

Shared References: [`COMP-001`](../../shared/compatibility-guidelines.md#comp-001-preserve-backward-compatible-behavior-by-default), [`COMP-003`](../../shared/compatibility-guidelines.md#comp-003-support-rolling-upgrade-where-product-policy-allows), [`COMP-004`](../../shared/compatibility-guidelines.md#comp-004-mixed-version-semantics-must-be-explicit), [`COMP-006`](../../shared/compatibility-guidelines.md#comp-006-protect-protocol-compatibility), [`COMP-007`](../../shared/compatibility-guidelines.md#comp-007-metadata-format-changes-need-a-compatibility-story), [`COMP-008`](../../shared/compatibility-guidelines.md#comp-008-downgrade-and-rollback-boundaries-must-be-clear), [`COMP-010`](../../shared/compatibility-guidelines.md#comp-010-document-unsupported-combinations)

### NFR-006 Supported Deployment Targets

Category: compatibility

Priority: P0

Initial deployment targets are TiDB Self-Managed, TiDB Cloud Essentials, and TiDB Cloud Premium. TiDB Cloud Dedicated is a follow-on target, and TiDB Cloud Starter is out of scope.

### NFR-007 Trace Export Networking Modes

Category: networking

Priority: P0

TiDB Cloud supports public-internet egress for trace export by default. Private connectivity is supported only in documented deployment tiers and networking modes.

### NFR-008 Initial Backend Validation Set

Category: interoperability

Priority: P0

The initial backend validation set should include Jaeger, Datadog, and AWS CloudWatch. Grafana and Honeycomb validation are follow-on targets.

### NFR-009 TiDB Cloud Usability Consistency

Category: usability

Priority: P0

TiDB Cloud tracing configuration should follow existing settings and networking patterns so users can enable the feature with minimal new workflow complexity.

### NFR-010 Exporter Status Visibility

Category: operability

Priority: P0

Exporter misconfiguration, failed connection attempts, and connection recovery events must be surfaced in logs or equivalent operator-visible status.

---

## 11. Success Metrics

| Metric | Target |
|---|---|
| End-to-end trace correlation | Validated in at least two representative customer or design-partner workflows covering application-to-TiDB trace linkage |
| Bottom-up attribution | Operators can identify the originating service or request from a slow-query investigation in validation scenarios |
| Performance overhead | Meets `NFR-001` and `NFR-002` targets during benchmark validation |
| Operational safety | Sampling and exporter configuration changes apply without planned downtime in validation environments |
| Ecosystem fit | Initial release validated against the planned OpenTelemetry backend compatibility set |