# Datadog and New Relic Integration Nodepool Management

## Scope

This document records the maintenance model for cloud-specific nodepools used by Datadog and New Relic integration workloads.

The expected scheduling contract is:

- dedicated nodepool for the integration workload
- Kubernetes label for workload selection, for example `component=metrics-integration`
- Kubernetes taint for isolation, for example `app=metrics-integration:NoSchedule`
- autoscaling enabled where the cloud provider supports managed autoscaling
- minimum size can be `0` when scale-from-zero is supported and correctly configured

## AWS EKS

AWS EKS managed nodegroups are backed by EC2 Auto Scaling Groups. Cluster Autoscaler discovers and manages those Auto Scaling Groups based on tags. The Amazon EKS Cluster Autoscaler best practices guide documents that Cluster Autoscaler can scale node groups to and from zero, but when no node exists it must infer labels, taints, and some special resources from Auto Scaling Group tags.

Reference: https://docs.aws.amazon.com/eks/latest/best-practices/cas.html

### Required Autoscaler Discovery Tags

The nodegroup Auto Scaling Group must be discoverable by the cluster's Cluster Autoscaler. Existing managed nodegroups usually already have these tags, but they should be verified:

```text
k8s.io/cluster-autoscaler/enabled=true
k8s.io/cluster-autoscaler/<cluster-name>=owned
```

Example:

```text
k8s.io/cluster-autoscaler/enabled=true
k8s.io/cluster-autoscaler/o11y-prod-regional-us-west-2=owned
```

### Required Scale-From-Zero Template Tags

Managed nodegroup labels and taints are applied to real Kubernetes Nodes after nodes are created. They are not sufficient for scale-from-zero because Cluster Autoscaler has no Node object to inspect when the nodegroup size is `0`.

For integration nodepools, add these Auto Scaling Group tags:

```text
k8s.io/cluster-autoscaler/node-template/label/component=metrics-integration
k8s.io/cluster-autoscaler/node-template/taint/app=metrics-integration:NoSchedule
```

These tags let Cluster Autoscaler simulate a future node with the required label and taint, so a pending integration Pod with a matching `nodeSelector` and `tolerations` can trigger scale-out from `0`.

### Example Command

The following command is an example for the `us-west-2` regional EKS cluster. This is a production write operation and must be reviewed before execution.

```bash
aws autoscaling create-or-update-tags \
  --profile clinic-prod \
  --region us-west-2 \
  --tags \
    "ResourceId=eks-metrics-integration-becc53e4-ba5a-f3ae-a881-33b0d9d58685,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/node-template/label/component,Value=metrics-integration,PropagateAtLaunch=true" \
    "ResourceId=eks-metrics-integration-becc53e4-ba5a-f3ae-a881-33b0d9d58685,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/node-template/taint/app,Value=metrics-integration:NoSchedule,PropagateAtLaunch=true"
```

### Verification

After tagging the Auto Scaling Group, verify the tags with a read-only command:

```bash
aws autoscaling describe-tags \
  --profile clinic-prod \
  --region <region> \
  --filters Name=auto-scaling-group,Values=<asg-name> \
  --query 'Tags[?starts_with(Key, `k8s.io/cluster-autoscaler/`)].{Key:Key,Value:Value,PropagateAtLaunch:PropagateAtLaunch}' \
  --output table \
  --no-cli-pager
```

The workload Pod must also match the nodepool contract:

```yaml
nodeSelector:
  component: metrics-integration

tolerations:
  - key: app
    operator: Equal
    value: metrics-integration
    effect: NoSchedule
```

## GCP GKE

GCP nodepools do not need extra post-creation tags for this use case.

The required maintenance model is:

- create a dedicated nodepool for the integration workload
- configure labels and taints on the nodepool at creation time
- enable GKE cluster autoscaling or nodepool autoscaling at creation time
- set the minimum node count to `0` only when the workload can tolerate cold start latency

Once autoscaling is enabled, GKE manages node scale-out and scale-in for the nodepool.

## Azure AKS

Azure nodepools do not need extra post-creation tags for this use case.

The required maintenance model is:

- create a dedicated nodepool for the integration workload
- configure labels and taints on the nodepool at creation time
- enable the AKS cluster autoscaler for the nodepool at creation time
- set the minimum node count to `0` only when the workload can tolerate cold start latency

Once autoscaling is enabled, AKS manages node scale-out and scale-in for the nodepool.

## Alibaba Cloud ACK

Open questions:

- whether ACK nodepools support reliable scale-from-zero for tainted integration workloads
- whether labels and taints configured on the nodepool are visible to the autoscaler while the nodepool size is `0`
- whether additional provider-specific tags, annotations, or scaling group metadata are required

Until this is validated, do not assume ACK behaves the same as GKE or AKS for scale-from-zero.

## Operational Notes

- Treat nodepool creation and Auto Scaling Group tagging as production write operations.
- Prefer read-only discovery commands before generating any write command.
- Keep nodepool scheduling properties consistent: labels, taints, instance shape, and capacity type should be uniform inside one nodepool.
- For AWS, always add or verify `node-template` tags when a dedicated integration nodegroup can scale to `0`.