--- title: Distributed Tracing for TiDB Scenarios feature_id: distributed-tracing owner: Ben Meadowcroft status: draft target_release: TBD created: 2026-03-17 last_updated: 2026-03-17 --- # Scenarios This document describes operator and system workflows for the distributed tracing feature. --- ## SCN-001: Enable OpenTelemetry Trace Export Validates: `FR-001`, `FR-009`, `INV-002`, `INV-005` Actor: SRE / Platform Operator Steps 1. The operator enables distributed tracing for a TiDB cluster and configures a valid OpenTelemetry-compatible endpoint. 2. The operator runs representative application traffic that satisfies the active capture policy. 3. The operator inspects the configured tracing backend for TiDB spans. Expected Result Eligible TiDB traces arrive at the configured backend, and normal query execution continues without interruption. --- ## SCN-002: App-Down Debugging with Propagated Trace Context Validates: `FR-004`, `FR-005`, `INV-001`, `INV-003` Actor: Application Developer Steps 1. The application emits a sampled request trace and includes propagated trace context in SQL statements using a supported instrumentation approach. 2. TiDB processes the traced request and captures the request according to the active tracing policy. 3. The developer opens the trace in their observability backend and drills from the application span into TiDB spans. Expected Result The developer can follow one correlated trace from the application request into TiDB processing and identify which database stages contributed to latency. --- ## SCN-003: Bottom-Up Attribution from Slow Database Work Validates: `FR-004`, `FR-006`, `INV-003`, `INV-005` Actor: Database Reliability Engineer Steps 1. The engineer starts from a slow-query record, trace span, or other TiDB-observed performance signal. 2. The engineer uses emitted trace identifiers and normalized metadata to locate the related TiDB trace. 3. The engineer follows the trace lineage to identify the originating application request or service. Expected Result The engineer can attribute the database issue to the correct upstream request path or service without relying only on manual log correlation. --- ## SCN-004: Adjust Sampling Policy on a Running Cluster Validates: `FR-002`, `FR-009`, `INV-002`, `INV-005` Actor: SRE / Platform Operator Steps 1. The operator enables tracing on a running cluster with an initial sampling policy. 2. The operator changes probabilistic sampling, slow-query capture, parent-sampled capture, and the global trace rate limit. 3. Application traffic continues while the updated policy is applied. 4. The operator verifies that new trace behavior reflects the updated policy. Expected Result Sampling policy changes take effect without a restart, and the operator can observe the new behavior through emitted traces and operational signals. --- ## SCN-005: Parent-Sampled Request Forces TiDB Capture Validates: `FR-002`, `FR-004`, `INV-003` Actor: Application Developer Steps 1. The operator sets cluster probabilistic sampling to zero while leaving parent-sampled capture enabled. 2. The application sends a request whose upstream trace is already marked as sampled and propagates that context to TiDB. 3. TiDB processes the request. Expected Result TiDB captures and emits spans for the request because the upstream trace context indicates a sampled parent lineage. --- ## SCN-006: Slow Query Capture with Minimal Baseline Sampling Validates: `FR-002`, `FR-005`, `INV-004` Actor: Database Reliability Engineer Steps 1. The operator configures probabilistic sampling at zero and leaves slow-query capture enabled. 2. A representative workload issues both normal-latency and slow queries. 3. The engineer inspects the tracing backend for captured TiDB spans. Expected Result Slow queries are traced according to policy while ordinary requests without another matching rule are not exported as full traces. --- ## SCN-007: Export Endpoint Becomes Unavailable Validates: `FR-001`, `FR-009`, `INV-002`, `INV-005` Actor: SRE / Platform Operator Steps 1. The cluster is actively exporting traces to a working tracing backend. 2. The configured endpoint becomes unreachable or unavailable. 3. Application traffic continues while TiDB detects the exporter failure. 4. The endpoint later becomes reachable again. Expected Result Query execution continues, exporter failure is surfaced through supported operational signals, and recovery is also surfaced when connectivity returns. Mitigation (optional) - Reduce trace volume or disable tracing temporarily if the failure creates avoidable operational noise. - Correct endpoint or network configuration before re-enabling the prior policy. --- ## SCN-008: TiDB Cloud Trace Export over Private Connectivity Validates: `FR-007`, `FR-008`, `FR-009`, `INV-005` Actor: Cloud Service Team / Internal Service Operator Steps 1. A TiDB Cloud user selects a supported private connectivity workflow for the tracing backend. 2. The user provides the required partner endpoint information and attaches or creates the endpoint through the supported flow. 3. The user enables distributed tracing with the private connectivity option. 4. The user generates representative traced traffic and verifies backend receipt. Expected Result TiDB Cloud exports traces through the configured private path in supported environments, and endpoint selection and health remain operator-visible. --- ## SCN-009: Upgrade from Legacy OpenTracing Configuration Validates: `FR-010`, `INV-006` Actor: SRE / Platform Operator Steps 1. The operator starts with a deployment where legacy OpenTracing behavior is configured. 2. The operator upgrades the deployment to a version that supports the new distributed tracing feature. 3. The operator inspects tracing behavior and upgrade-time warnings after the upgrade. Expected Result Tracing behavior follows the documented compatibility boundary, and TiDB surfaces a deprecation warning instead of silently changing legacy tracing behavior. --- ## SCN-010: Session-Scoped Diagnostic Trace Capture Validates: `FR-003`, `FR-005`, `INV-002` Actor: Application Developer Steps 1. The cluster has distributed tracing enabled with a conservative baseline policy. 2. The developer enables session-scoped trace capture for a targeted debugging session. 3. The developer runs the problematic application workflow through that session. 4. The developer disables the session-scoped mode after collecting the needed traces. Expected Result The targeted session is captured for debugging without requiring a broad cluster-wide increase in trace volume. --- ## SCN-011: Missing or Malformed Trace Context Validates: `FR-004`, `FR-001`, `INV-001`, `INV-003` Actor: Application Developer Steps 1. The application sends SQL statements without propagated trace context or with malformed context fields. 2. TiDB processes the statements under the configured local tracing policy. 3. The developer inspects query behavior and any resulting TiDB traces. Expected Result Query execution continues normally, and TiDB either traces the request using local policy or skips export according to policy, without incorrectly linking the work to another trace. --- ## Optional Checklist - [x] Include at least one success-path scenario. - [x] Include configuration/operational scenarios. - [x] Include failure/rollback cases.