# Dedicated Gray Rollout Q&A

The following analysis uses `release/20260320` as the old-version baseline and assumes the following by default:

- `Global` is always upgraded first
- Existing users only have `Dedicated`
- Existing instances are not expected to restart or be proactively recreated
- `memory capacity / disk capacity` no longer depends on the old config fields

## Q1: If `Global` is upgraded first but some `Regional` instances are still on the old version, what is the expected impact?

**Key question**

Will only `CPU Capacity Cores` fail to update, while everything else continues to behave normally?

**Conclusion**

Yes.

Based on the current code path and the assumptions you already confirmed, the expected impact should be limited to:

- Only `CPU Capacity Cores` cannot be refreshed after `Dedicated DD/NR scale`
- Existing instances continue running without impact
- Other normal behaviors such as `create/delete` are expected to remain normal
- This scenario should not be overstated as "the entire pipeline is malfunctioning"

The reason is that the new `Global` has already switched `Dedicated` scale operations to the new dedicated CPU refresh path, while the old `Regional` does not have the corresponding `/pipeline/dedicated-cpu` API.  
Therefore, the impact should be limited to CPU core configuration refresh itself.

**Risk assessment**

- Risk level: medium
- Scope: `Dedicated DD/NR` scale operations in regions still running the old `Regional`
- Result: `CPU Capacity Cores` cannot be refreshed into pipeline runtime
- Does it affect continuous operation of existing instances: no
- Are other behaviors expected to remain normal: yes
- Recommended for gray rollout: yes, as long as this limitation is accepted

## Q2: If newly created `Datadog/Newrelic` pipelines in `Global` still default to the old image, what is the expectation?

**Key question**

If newly created `Datadog/Newrelic pipelines` still use the old image by default, can they continue running?

**Conclusion**

They are expected to keep running.

`Global` writes the pipeline image tag into the `helm_config` sent down to `Regional`.  
Therefore, as long as `Global` still uses the old image, newly created `Datadog/Newrelic pipelines` will also continue pulling the old image by default.  
Under the current assumption that "existing users only have Dedicated," this scenario is still expected to run normally and should not be considered a production compatibility issue.

**Risk assessment**

- Risk level: low
- Scope: newly created `Dedicated Datadog/Newrelic pipelines`
- Result: they still run the old image by default, but are expected to continue working
- Impact on existing instances: none
- Impact on release capability: new features may not actually land with the old image
- Recommended for gray rollout: yes

## Q3: If `Regional` is upgraded but the `Datadog/Newrelic pipeline` still uses the old image, what is the expected change?

**Key question**

If `Regional` has been upgraded but the `Datadog/Newrelic pipeline` is still running the old image, what is the most visible expected change?

**Conclusion**

The main expected change should be limited to this: `Datadog` will emit additional CPU Usage metrics calculated from rate.

The reason is that after `Regional` is upgraded, the Helm rendering path will use the new vector template behavior. Even if the pipeline workload image is still old, this creates a combination of "old binary + new template."  
Given the current template differences, the clearest output change is that `Datadog` will no longer drop `tidb_cloud.node_cpu_seconds_total_rate`, so it will emit additional CPU Usage metrics derived from rate.  
This is more of a metric output difference than an instance availability issue.

**Risk assessment**

- Risk level: low
- Scope: newly created or recreated `Datadog` instances in upgraded regions
- Result: `Datadog` emits additional CPU Usage metrics calculated from rate
- Does this mean direct unavailability: no
- Recommended for gray rollout: yes, as long as this metric output change is acceptable

## Q4: Is there any risk for existing `Dedicated` tenants in this gray rollout?

**Key question**

For existing `Dedicated` legacy tenants that are already running, does this old/new mixed deployment create production risk?

**Conclusion**

There is expected to be no obvious risk, and functionality should continue to work normally.

Under the current assumptions, the impact of this gray rollout on existing `Dedicated` tenants should remain low risk.  
Existing instances are not expected to restart or be proactively recreated, and normal functionality should remain intact.  
The only specific limitation that still needs attention is the one described in Q1: after `DD/NR Dedicated scale` in regions still running the old `Regional`, `CPU Capacity Cores` cannot be refreshed.

**Risk assessment**

- Risk level: low
- Scope: existing `Dedicated` legacy tenants
- Result: no obvious risk is expected, and functionality should continue normally
- Does it affect continued production usage: no
- Recommended for gray rollout: yes

## Q5: If both `Global` and `Regional` are upgraded but the `Datadog/Newrelic pipeline` still uses the old image, will startup fail because of missing config?

**Key question**

Under the combination of `new Global + new Regional + old Datadog/Newrelic pipeline image`, can the old pipeline fail to start because of config field changes?

**Conclusion**

Based on the current code and current prod config, startup failure due to missing config is not expected.

In the config sent by the new `Global` to the pipeline, fields such as `servicePlan` and `bizTypeConfigs` have been added, and the `components` content has been simplified. However, the old pipeline config loading logic is tolerant of additional fields and will not fail just because extra fields are present.  
At the same time, the key field that the old pipeline actually depends on at startup, `o11yGlobalServer`, especially the old `bizType`, is still retained in the current `Global` prod config.  
Therefore, this combination is more likely to result in runtime behavior or metric output differences, rather than startup failure caused by missing config.

**Risk assessment**

- Risk level: low
- Scope: `new Global + new Regional + old Datadog/Newrelic pipeline image`
- Will it fail to start because of missing config: not expected
- Main risk type: metric output differences, not startup failure
- Boundary to watch: if `Global` later removes the old `o11yGlobalServer.bizType`, the old image may then fail config validation
- Recommended for gray rollout: yes

## One-Page Conclusion

- In this mixed-deployment scenario, the only clearly identified hard risk is that `CPU Capacity Cores` cannot be refreshed when `Regional` is not upgraded; other behaviors are expected to remain normal, including node scale up/down
- Existing legacy `Dedicated` instances can continue running, and the overall risk is low
- If `DD/NR pipeline` in `Global` continues using the old image, it will not affect existing instances, but it will affect which binary is actually used by newly created or recreated instances later
- For existing `Dedicated` tenants, the overall risk is low and functionality is expected to remain normal
- `new Global + new Regional + old DD/NR pipeline image` is not expected to fail startup because of missing config

## Code Basis

- `Dedicated scale` has already switched to the new runtime refresh flow: [cluster_scaling.go](global/internal/domain/event_bus/consumer/integration/cluster_scaling.go#L40)
- The new `Regional` adds the `/pipeline/dedicated-cpu` API: [services.proto](regional/api/integration/v1/services.proto#L44)
- `Prometheus` still uses the old `metrics-key/*` remote interface: [remote_config.go](global/internal/infrastructure/promconfig/remote_config.go#L14)
- Whether `Regional` is bypassed depends on `globalManagedPipeline`: [config.go](global/internal/infrastructure/config/config.go#L219)
- `Global` writes the pipeline image tag directly into the remotely applied `helm_config`: [remote_pacakge.go](global/internal/infrastructure/helm/remote_pacakge.go#L14)
- The current `Datadog/Newrelic` vector template behavior already differs from the old version: [configmap.yaml](helm-charts/pipeline/charts/vector/templates/configmap.yaml#L22)