# Alibaba Cloud Ingress, ALB, and DNS Deployment Notes

## Goal

Document the Alibaba Cloud deployment model used by TiDB-X Metrics Integration and record the bootstrap issues we hit with ALB, Ingress, DNS, and cert-manager.

This note is meant to prevent the same initialization failure from reappearing in future environment rollouts.

## Scope

This applies to the Alibaba Cloud entrypoints under:

- `pipeline/deployment/prom-app-delivery/*/prom-regional-service/...`
- `regional/deployment/kustomize/*/metrics-integration-regional-loadbalancer/...`

The current manifests use:

- `AlbConfig` to define the ALB instance and listeners
- `IngressClass` to bind Kubernetes Ingress objects to that `AlbConfig`
- `Ingress` to define host-based routing rules
- `cert-manager` with ACME HTTP01 to manage TLS secrets
- public DNS records to point the hostname to the ALB entrypoint

## Resource Model

### 1. `AlbConfig` owns the ALB instance

`AlbConfig` is the source of truth for the ALB instance itself, including:

- Internet-facing vs internal exposure
- vSwitch mappings
- listener definitions such as `HTTP:80` and `HTTPS:443`

If a listener is missing from `AlbConfig`, the ALB controller cannot expose that port.

### 2. `Ingress` must explicitly bind to ALB listeners

On Alibaba Cloud, defining listeners in `AlbConfig` is not enough.
The `Ingress` must also declare:

```yaml
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
```

Without this annotation, the ALB may exist but the Ingress rules are not associated with both listeners.
In practice, this looks like:

- ALB instance created
- no usable `443` listener for the service
- TLS traffic not forwarding even though `spec.tls` exists on the Ingress

### 3. `cert-manager` manages the TLS secret

The `Ingress` references a TLS secret, for example:

```yaml
spec:
  tls:
    - hosts:
        - <host>
      secretName: prom-integration-tls
```

`cert-manager` creates and maintains that secret based on the Ingress annotations and the referenced `ClusterIssuer`.

### 4. DNS is the last external dependency

After Kubernetes resources are applied, the remaining external action is to add the public DNS record for the configured host so that ACME HTTP01 validation can reach the ALB.

Once DNS resolves to the ALB entrypoint and port `80` is reachable, certificate issuance can complete automatically.

## Bootstrap Failure We Hit

The original Alibaba Cloud manifests had two coupled problems:

1. The Ingress referenced TLS and expected the certificate secret to appear before HTTPS could become usable.
2. cert-manager was configured to create a separate HTTP01 solver Ingress for the same ALB ingress class.

That created a bad dependency chain:

1. The business Ingress wanted HTTPS and a valid secret.
2. The certificate secret depended on ACME HTTP01 validation.
3. HTTP01 validation depended on ALB listener and routing readiness.
4. ALB reconciliation could be blocked by a misconfigured Ingress on the same ALB.

The visible symptom was:

- ALB instance existed
- listener state was incomplete or ineffective
- the ACME challenge stayed `pending`
- the TLS secret never became ready
- the service never became reachable over the intended hostname

## Chosen Bootstrap Pattern

We standardized on the following pattern for Alibaba Cloud:

### 1. Keep both listeners in `AlbConfig`

`AlbConfig` must define:

```yaml
listeners:
  - port: 80
    protocol: HTTP
  - port: 443
    protocol: HTTPS
```

### 2. Bind the Ingress to both listeners

Every Alibaba Cloud service Ingress must declare:

```yaml
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
```

This is required so ALB listener configuration and Ingress routing are actually associated.

### 3. Use in-place HTTP01 solving

The Ingress must declare:

```yaml
acme.cert-manager.io/http01-edit-in-place: "true"
```

instead of creating a separate solver Ingress by annotation override.

Why this matters:

- cert-manager edits the existing business Ingress in place for `/.well-known/acme-challenge/*`
- cert-manager automatically marks the generated `Certificate` for temporary certificate issuance
- the referenced TLS secret is populated with a temporary self-signed certificate until the real ACME certificate is issued

This removes the most fragile part of the previous design: a second solver Ingress competing on the same ALB reconciliation path.

### 4. Keep the `ClusterIssuer` on HTTP01

The `ClusterIssuer` can still use:

```yaml
solvers:
  - http01:
      ingress:
        ingressClassName: <alb-ingress-class>
```

The key change is not the issuer itself.
The key change is that the application Ingress uses `http01-edit-in-place` so cert-manager reuses the same Ingress instead of generating another one.

## Resulting Deployment Flow

After this change, the deployment flow becomes:

1. Apply Kubernetes manifests.
2. The ALB controller creates or reconciles the ALB from `AlbConfig`.
3. The Ingress binds to `HTTP:80` and `HTTPS:443` through `listen-ports`.
4. cert-manager creates a `Certificate` from the Ingress annotations.
5. cert-manager writes a temporary certificate into the referenced TLS secret.
6. Add the public DNS record for the hostname so it resolves to the ALB entrypoint.
7. Let’s Encrypt reaches the ALB over HTTP01 on port `80`.
8. cert-manager replaces the temporary certificate with the signed certificate.
9. HTTPS traffic becomes fully valid without another manifest change.

Operationally, this means:

- after deployment, adding DNS is enough to unblock certificate issuance
- the service no longer depends on a second solver Ingress becoming healthy first
- the initial ALB bring-up is much less likely to deadlock

## Current Manifest Conventions

The Alibaba Cloud manifests should keep the following conventions:

### Pipeline Prometheus entrypoint

- `AlbConfig` defines `80/443`
- `Ingress` declares `alb.ingress.kubernetes.io/listen-ports`
- `Ingress` declares `acme.cert-manager.io/http01-edit-in-place: "true"`
- `Ingress` declares `cert-manager.io/cluster-issuer`

### Regional service entrypoint

Use the same pattern as pipeline.
Do not leave regional on the older solver-Ingress pattern while pipeline uses in-place solving.

## Deployment Checklist

Before rollout:

1. Confirm the target hostname in the environment kustomization is correct.
2. Confirm `AlbConfig` includes both `HTTP:80` and `HTTPS:443`.
3. Confirm the Ingress includes `alb.ingress.kubernetes.io/listen-ports`.
4. Confirm the Ingress includes `acme.cert-manager.io/http01-edit-in-place: "true"`.
5. Confirm the Ingress references the correct `ClusterIssuer`.
6. Confirm the ALB ingress class name matches the issuer solver configuration.

During rollout:

1. Apply the manifests.
2. Wait for the ALB and listeners to reconcile.
3. Add the DNS record for the configured host.
4. Verify the ACME challenge no longer stays pending.
5. Verify the TLS secret is created.
6. Verify HTTPS works on the final hostname.

After rollout:

1. Check that the `Certificate` is `Ready=True`.
2. Check that the Ingress has the expected hostname and address.
3. Check that the ALB listener rules forward to the correct service backend.

## Troubleshooting

### ALB exists but port `443` is not working

Check both sides:

- `AlbConfig` includes an `HTTPS:443` listener
- the `Ingress` includes `alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'`

If only `AlbConfig` is configured, the Ingress may still not attach rules to `443`.

### Certificate stays pending

Check:

- DNS resolves the public hostname to the ALB entrypoint
- port `80` is reachable from the public internet
- cert-manager challenge resources do not show HTTP01 routing failures
- the Ingress uses `acme.cert-manager.io/http01-edit-in-place: "true"`

### ALB changes do not take effect even though the new Ingress looks correct

Alibaba Cloud ALB reconciliation is sensitive to misconfigured Ingresses on the same ALB.
If multiple Ingresses share one ALB instance, one broken Ingress can block the others from taking effect.

Inspect:

- all Ingresses using the same `IngressClass`
- ALB controller events
- `kubectl describe` output for the failing Ingress and `AlbConfig`

### Manual console changes disappear

Do not treat the Alibaba Cloud console as the source of truth for ALB behavior.
The controller continuously reconciles the ALB from Kubernetes resources.
Make changes in the manifests, not in the console.

## Useful Commands

```bash
kubectl get ingress -A
kubectl describe ingress <ingress-name> -n <namespace>
kubectl get albconfig -A
kubectl describe albconfig <albconfig-name> -n <namespace>
kubectl get certificate -A
kubectl describe certificate <certificate-name> -n <namespace>
kubectl get challenge -A
kubectl describe challenge <challenge-name> -n <namespace>
kubectl get secret <tls-secret-name> -n <namespace>
```

## References

- Alibaba Cloud ALB Ingress FAQ:
  https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/alb-ingress-faq
- Alibaba Cloud ALB Ingress configuration reference:
  https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/alb-ingress-configuration-dictionary
- cert-manager annotations reference:
  https://cert-manager.io/docs/reference/annotations/
- cert-manager certificate behavior:
  https://cert-manager.io/docs/usage/certificate/