# TiProxy And Unknown Component Pipeline Fix ## Summary This fix hardens the component CPU propagation path from `global` to `pipeline`. The original issue was triggered when the management service started returning newer or unexpected dedicated-cluster components. `global` could write those values into pipeline config, and older `pipeline` binaries were not guaranteed to handle them safely. After this fix: - `global` only writes `TiDB`, `TiKV`, and `TiFlash` component CPU information to downstream config - `PD`, `TiProxy`, `TiCDC`, `UNSPECIFIED`, and future unknown components are filtered out before they are written - `pipeline` normalizes and filters component config on read, so historical configs with stale values do not crash the process - `node_cpu_capacity_cores` only uses `TiDB`, `TiKV`, and `TiFlash` This means a `global` rollout alone is enough to stop new unsupported component values from being pushed into an older `pipeline`. ## Problem Dedicated cluster CPU information is propagated through multiple lifecycle paths: - Datadog / New Relic integration create - Datadog / New Relic integration update - Prometheus metrics-key create - Prometheus metrics-key cluster scale - Dedicated cluster CPU refresh Before this fix: - `global` converted management-service component types into pipeline config component types - unsupported or newly introduced values could still enter downstream config - some `pipeline` code paths assumed configured component types were known - CPU capacity metric logic had duplicated component-to-CPU mapping code - user-visible CPU capacity handling was not explicitly limited to the components users actually care about The practical risk was that a newer control plane or management service could write component values that an older `pipeline` image did not expect. ## Root Cause There were three separate gaps. 1. Component filtering was not consistently enforced at the `global` write boundary. 2. `pipeline` config ingestion and metrics-key ingestion did not share one normalization path. 3. CPU capacity metric code repeated its own component mapping logic in multiple places. There was also an expectation mismatch around `TiProxy` and `PD`: - management service may surface them in dedicated cluster topology - but user-facing CPU capacity metrics do not need them - so writing them into downstream pipeline config only increases compatibility risk ## Final Design ### Global Responsibilities `global` is the write boundary. It now treats only these dedicated-cluster components as downstream-relevant: - `COMPONENT_TYPE_TIDB` - `COMPONENT_TYPE_TIKV` - `COMPONENT_TYPE_TIFLASH` Everything else is dropped before downstream write: - `COMPONENT_TYPE_PD` - `COMPONENT_TYPE_TIPROXY` - `COMPONENT_TYPE_TICDC` - `COMPONENT_TYPE_UNSPECIFIED` - any future unknown component This filtering is applied in all config generation paths that matter: - target cluster fetch for integration create / update - Prometheus target-cluster YAML generation - dedicated CPU refresh payload generation - runtime config patching So `global` no longer writes `PD` or `TiProxy` into pipeline config at all. ### Regional Responsibilities `regional` is intentionally left unchanged. It is a transport / apply layer in this chain: - it applies helm config received from `global` - it stores `cluster_config_map` for Prometheus metrics keys - it patches runtime config through shared helpers from `global/pkg/helm/pipeline` It does not define component support policy itself. ### Pipeline Responsibilities `pipeline` is the read boundary and compatibility layer. It now: - normalizes `servicePlan` and `targetClusters.components` on config load - uses the same normalization for both main config and Prometheus `cluster_config_map` - silently drops unsupported component values on read - avoids panicking on unsupported component type mapping - computes CPU capacity from a shared helper that only returns `TiDB`, `TiKV`, and `TiFlash` This keeps older or stale config entries from breaking startup even if they already exist in stored config. ## Code Changes ### Global Main files: - `global/pkg/helm/pipeline/types.go` - `global/pkg/helm/pipeline/runtime_config.go` - `global/internal/domain/workflow/action/pipeline/updatetargetcluster/updatetargetcluster.go` Key changes: - `GetTargetClusterByClusterID` now filters dedicated cluster components to `TiDB/TiKV/TiFlash` - `TIFLASH_LEARNER` is normalized to `TIFLASH` - dedicated CPU refresh request generation filters unsupported components - runtime config patching removes unsupported existing entries as well as unsupported incoming entries - update-target-cluster refresh short-circuits when filtered component list is empty ### Pipeline Main files: - `pipeline/central/config/config.go` - `pipeline/central/config/config_init.go` - `pipeline/internal/service/prometheus/manager.go` - `pipeline/internal/datasource/central_thanos/prometheus_builder.go` - `pipeline/internal/otel_collector/receiver/metadb/datafetcher.go` - `pipeline/internal/otel_collector/receiver/metadb/scrape.go` - `pipeline/central/model/util/core_svc.go` Key changes: - `NormalizeTargetCluster` now normalizes both service plan and component list - unsupported component types such as `TiCDC`, `TiProxy`, `UNSPECIFIED`, and future values are dropped during config load - Prometheus metrics-key config now goes through the same normalization path as main config - CPU capacity logic uses shared `NodeCPUCapacityComponentMap()` - `node_cpu_capacity_cores` only emits capacity for `TiDB/TiKV/TiFlash` - unsupported component mapping returns `(value, ok)` instead of panicking ## Behavior After Fix ### Create / Update Integration For Datadog and New Relic pipeline create / update: - `global` fetches cluster topology - `global` filters component list to `TiDB/TiKV/TiFlash` - filtered config is sent to dataplane - `pipeline` still filters again during config load for compatibility ### Prometheus Metrics Key Create / Scale For Prometheus metrics-key create / scale: - `global` builds target-cluster YAML from the filtered component set - `regional` stores it as `cluster_config_map` - `pipeline` normalizes that config again when reading it ### Dedicated CPU Refresh For dedicated cluster CPU refresh: - `global` filters the incoming component list - runtime config patching updates only supported component entries - stale unsupported entries already present in config are removed ## Operational Outcome The important operational result is: - if `global` is upgraded first, it stops writing unsupported components into downstream config - an older `pipeline` image is therefore much less likely to fail because of a newly introduced component from management service This is the main reason `PD` and `TiProxy` were removed from downstream propagation: they are not needed for user-facing CPU capacity metrics, so keeping them only adds compatibility risk. ## Validation Validated with targeted tests: ```bash cd global && mkdir -p /tmp/go-build /tmp/go-tmp && GOCACHE=/tmp/go-build GOTMPDIR=/tmp/go-tmp go test ./pkg/helm/pipeline ./internal/infrastructure/dataplane ./internal/domain/workflow/action/pipeline/updatetargetcluster cd pipeline && mkdir -p /tmp/go-build /tmp/go-tmp && GOCACHE=/tmp/go-build GOTMPDIR=/tmp/go-tmp go test ./central/config ./central/model/util ./internal/service/prometheus ./internal/datasource/central_thanos ./internal/otel_collector/receiver/metadb ``` ## Guidance For Future Components If management service introduces another component in the future: - do not automatically propagate it into downstream pipeline config - first decide whether the component is actually needed for user-facing metrics - if it is not needed, keep filtering it at the `global` boundary - if it is needed, add support end-to-end in `global` write logic and `pipeline` read / metrics logic together The default safe behavior should remain: - `global` filters unknown or unneeded components before write - `pipeline` filters unknown components again during read