# TiDB-X Metrics Compatibility Review and Launch Risk ## Goal This note re-evaluates the current TiDB-X metrics implementation for: - Premium - Essential - Starter The main review target is whether the exported metrics and query semantics match TiDB-X product expectations. Dedicated is treated as a compatibility baseline only. This revision supersedes the earlier "mostly low-risk cleanup" conclusion. ## Inputs Reviewed - `04-metric-output-and-coverage.md` - `05-validation-and-rollout.md` - TiDB-X static-check output under `check/tidb-x-check/static-check/` - `check/tidb-x-check/static-check/10-current-code-metrics-status.md` - Dedicated regression static-check output under `check/dedicated-regression-check/static-check/` - current query/build/transform implementation in `pipeline/internal/datasource/central_thanos/` - current Prometheus and OTel output transformation logic ## Overall Conclusion TiDB-X metrics support is still not launch-ready for Premium / Essential / Starter. Current status after this refresh: - DB-performance metrics now follow instance-level grouping for all TiDB-X plans - storage metrics now follow the frontend quantile-based query shape for both row-based and columnar storage - Prometheus `tidbcloud_db_queries_using_plan_cache_ops` now exports `type`, matching Dedicated - RU metrics are confirmed to be emitted for TiDB-X on all three export paths - Prometheus still exports one more CDC metric than OTel: checkpoint timestamp - static-check documents have been refreshed to reflect the current code - real Premium / Essential / Starter rollout evidence is still not present in this branch The remaining launch concern is no longer DB aggregation or frontend storage alignment. The remaining launch concern is the unresolved scale risk of per-region storage aggregation plus missing E2E/sign-off evidence. There is also still an export-path parity gap for CDC checkpoint timestamp on Datadog / New Relic. ## Executive Summary ### Ready / Mostly OK - TiDB-X exported labels are transformed to `instance_id` / `instance_name` - TiDB-X output removes `project_*`, `instance`, and `component` from user-facing labels - TiDB-X RU metrics are present in current Prometheus and OTel code paths - storage metric descriptions are already using `row-based` and `columnar` - Dedicated Prometheus and New Relic output remain aligned with the `release/20260320` baseline ### Not Ready - `db_row_storage_used_bytes` and `db_columnar_storage_used_bytes` still use per-region aggregation and have not been optimized to a lower-cardinality per-keyspace source - Datadog / New Relic still do not emit the CDC checkpoint timestamp metric that Prometheus emits - there is no recorded real-cluster TiDB-X E2E or sign-off evidence in this branch ## Review of the Reported Comments ### 1. For metrics that should stay the same as Dedicated on 3rd-party platforms, the metric names can remain the same #### Current status - Mostly addressed #### Assessment For DB and CDC metrics, TiDB-X generally keeps the historical external metric names instead of inventing a separate TiDB-X naming family. This matches the compatibility expectation. The remaining naming oddities are mostly inherited Prometheus CDC legacy names: - `tidbcloud_ticdc_owner_resolved_ts_lag` - `tidbcloud_changefeed_Replica_rows` These are consistency issues, but they are not the primary launch blockers. #### Risk - Low for launch ### 2. `tidb_cloud.db_database_time` should not keep `component` / `instance` #### Current status - Addressed #### Assessment This item has now been fixed in the current branch. All TiDB-X plans now use the same instance-level DB grouping semantics: - Premium - Essential - Starter That means the DB-performance set no longer keeps `component` / `instance` in PromQL grouping for Premium. The fix applies to the shared DB aggregate queries, including: - `db_total_connection` - `db_active_connections` - `db_disconnections` - `db_database_time` - `db_query_per_second` - `db_failed_queries` - `db_command_per_second` - `db_queries_using_plan_cache_ops` - `db_average_query_duration` - `db_transaction_per_second` This is the intended behavior for TiDB-X product semantics. The refreshed static-check output now matches the code. #### Risk - Low ### 3. `db_columnar_storage_used_bytes` and `db_row_storage_used_bytes` descriptions should use `columnar storage` and `row-based storage` #### Current status - Addressed #### Assessment The descriptions in current metadata and builder comments already use: - row-based storage - columnar storage This item is no longer a launch concern. #### Risk - Low ### 4. `db_row_storage_used_bytes` needs per-keyspace aggregation to control scale #### Current status - Not addressed #### Assessment The current implementation still uses per-region aggregation inside a frontend-aligned smoothing query: - `quantile_over_time(0.5, sum(avg by(keyspace_id, region_id) (tikv_store_size_bytes{..., type="used"}))[5m]) or vector(0)` This is still the per-region approach. It removes replica duplication, but it does not solve the main scale problem mentioned in the review comment: - when region count is high, the metrics query still has to process per-region data - the query cardinality and returned intermediate data can still become too large There is no new per-keyspace aggregation path in the current implementation, and no sign that the old per-TiKV path has been replaced by a per-keyspace path. This repository can only query the series that already exist; it cannot eliminate the underlying per-region scan cost by PromQL rewrite alone. So this optimization likely requires an upstream lower-cardinality source such as: - a native per-keyspace metric - a backend recording rule - another pre-aggregated storage usage series This is exactly the unresolved risk called out in the previous review. #### Risk - High - Launch blocker for storage metrics #### Required fix direction - add or consume a per-keyspace aggregation path for TiDB-X storage usage - keep the current frontend-aligned query semantics, but switch to a lower-cardinality source when available - validate the new query shape against large-region tenants ### 5. `db_columnar_storage_used_bytes` should align with Web console query semantics #### Current status - Addressed #### Assessment The current implementation now follows the same high-level query shape used by the frontend storage charts: - `quantile_over_time(0.5, sum(avg by(keyspace_id, region_id) (... type="tiflash_used"))[5m]) or vector(0)` The same frontend-aligned smoothing logic is also applied to: - `db_row_storage_used_bytes` This change aligns metrics-integration with the frontend storage chart semantics without changing the existing matcher logic or the `or vector(0)` fallback behavior. What remains unresolved is not the frontend alignment itself, but the scale risk from continuing to aggregate through per-region series. #### Risk - Low for semantics alignment - separate High risk remains for per-keyspace scale optimization ## Additional Findings From the Current Review ### 6. Prometheus `tidbcloud_db_queries_using_plan_cache_ops` label parity with Dedicated #### Current status - Addressed #### Assessment The current branch now exports `type` in the Prometheus path for `tidbcloud_db_queries_using_plan_cache_ops`. This brings TiDB-X Prometheus output back in line with: - Dedicated semantics - Datadog output - New Relic output So this item should no longer be treated as an open TiDB-X-only inconsistency. #### Risk - Low ### 7. Datadog / New Relic still miss TiDB-X CDC checkpoint timestamp #### Current status - Not addressed #### Assessment The current repository has all the pieces required to describe this metric, but the OTel query flow is still incomplete: - `generated_metrics.go` defines `tidb_cloud.cdc_checkpoint_ts` - `metric_builder_wrapper.go` defines `RecordQueryCdcCheckpointTs` - `thanos.go` does not currently call `QueryTiCDCCheckpointTS` So the current TiDB-X exporter behavior is: - Prometheus: emits checkpoint timestamp - Datadog: does not emit checkpoint timestamp - New Relic: does not emit checkpoint timestamp This is no longer a documentation-only issue. It is a real code-path inconsistency between Prometheus and OTel exports. #### Risk - Medium - not the main launch blocker, but it is still a user-visible parity gap ### 8. TiDB-X exported labels are correct, but query semantics are not uniformly correct #### Current status - Partially addressed #### Assessment The label transformation layer is working as intended: - `cluster_id` -> `instance_id` - `cluster_name` -> `instance_name` - `project_*` removed - `instance` and `component` removed This is true for both: - Prometheus encoding path - OTel transform path However, that output cleanup should not be confused with semantic correctness. For DB-performance metrics, the grouping issue that previously affected Premium has now been fixed. The remaining semantic gaps are concentrated in the storage metrics rather than the DB aggregate metrics. #### Risk - Medium - now mainly driven by storage query semantics ### 9. No real TiDB-X rollout evidence is present in this branch #### Current status - Not addressed #### Assessment The validation doc requires: - real cluster verification for target plans and regions - rollout evidence in an E2E or sign-off artifact But this branch only contains: - static-check documents - an E2E report template There is no completed Premium / Essential / Starter E2E or sign-off report in the reviewed docs. Given that there is currently no online TiDB-X pipeline instance, this is a separate rollout blocker even if the remaining code issues were fixed. #### Risk - High - Launch blocker ## Dedicated Compatibility Status Dedicated is not the main review target here, but the baseline is still important. Current status: - Prometheus Dedicated output: aligned with `release/20260320` - New Relic Dedicated output: aligned with `release/20260320` - Datadog Dedicated output: one remaining known diff for `tidb_cloud.node_cpu_seconds_total_rate` So Dedicated compatibility is not the reason TiDB-X launch is blocked. The blockers are on the TiDB-X side. ## Recommended Classification ### Launch blockers - storage metrics still lack a lower-cardinality per-keyspace aggregation source - no recorded real-cluster Premium / Essential / Starter E2E or sign-off evidence ### Should fix before launch - confirm the upstream plan for lower-cardinality storage aggregation and how metrics-integration will consume it - restore CDC checkpoint timestamp parity on Datadog / New Relic ### Low-priority cleanup - legacy Prometheus CDC metric naming consistency - internal typo `RecordQueryCdcChangefeedRelicaRows` - `NewRelic` vs `New Relic` style unification ## Recommended Priority ### P0 - per-keyspace storage aggregation - TiDB-X real-cluster validation and sign-off evidence ### P1 - upstream storage aggregation plan and integration follow-up - OTel CDC checkpoint timestamp parity ### P2 - CDC Prometheus legacy naming cleanup - internal typo cleanup - document naming-style cleanup ## Suggested Implementation Order 1. Keep the refreshed static-check output as the current code baseline. 2. Add or consume a lower-cardinality per-keyspace storage aggregation source and validate it on large-region tenants. 3. Produce real Premium / Essential / Starter validation evidence before rollout. ## Final Summary The previous conclusion that these were "mostly low-risk cleanup items" is no longer accurate. The current implementation shows: - Dedicated compatibility is broadly under control - DB-performance metrics across Premium / Essential / Starter are now close to the intended TiDB-X model - RU metrics are present, but the old static-check docs had been understating the current inventory - Prometheus and OTel still differ on CDC checkpoint timestamp - storage metrics are now aligned with frontend query semantics, but still have an unresolved scale risk - rollout evidence for TiDB-X is still missing Therefore the current TiDB-X metrics implementation for Premium / Essential / Starter should be considered: - functionally advanced - partially aligned - not yet ready for production launch