# Monitoring Gate for Gray Release

Use Grafana Prometheus datasource per region:

- Name pattern: `Prometheus-regional-{logical_region}`

This naming pattern is used in:

- `deployments/globalserver/register-new-region.sh`

## Check sequence (run in order)

Before running any query, derive the namespace filter from
`references/components.md`.

- Single namespace example: `namespace="observability"`
- Multi-namespace example:
  `namespace=~"cluster-backup|kube-system|observability|ops-system|self-monitoring"`

In the queries below, replace `__NS_FILTER__` with that selector.

For namespace-scoped checks (for example, infra or any workflow spanning more
than one deployment), omit the `deployment=` / `container=` label or use
`sum(...)`. For component-scoped checks, append the deployment/container filter
after `__NS_FILTER__`, for example:
`{__NS_FILTER__, deployment="{component}"}`.

---

### Step 0 — Wait for rollout convergence

A rolling update naturally has unavailable replicas mid-flight. Checking before
convergence creates false alarms. Always confirm the rollout is complete first.

```promql
kube_deployment_status_observed_generation{__NS_FILTER__}
!=
kube_deployment_metadata_generation{__NS_FILTER__}
```

Pass: empty result (all deployments converged).

If non-empty: wait 60 s and re-check, up to **5 minutes**. If still mismatched
after 5 minutes, the rollout is stalled — treat as a failure.

---

### Step 1 — Ready replicas

```promql
kube_deployment_status_replicas_ready{__NS_FILTER__} < 1
```

Pass: empty result (every deployment has ≥ 1 ready replica).

---

### Step 2 — Unavailable replicas (with one retry)

```promql
kube_deployment_status_replicas_unavailable{__NS_FILTER__} > 0
```

Pass: empty result.

**If this is the only failure** (Steps 1, 3, 4 all pass): wait **2 minutes**
and re-query once. A single unavailable replica immediately after convergence is
a normal Kubernetes scheduling artifact — a pod being evicted or rescheduled
transiently. If still non-zero after the retry, treat as a real failure.

Do **not** apply the retry if restarts or CrashLoopBackOff are also present —
those signals confirm a real problem.

---

### Step 3 — CrashLoopBackOff (required, no grace period)

```promql
max_over_time(kube_pod_container_status_waiting_reason{__NS_FILTER__, reason="CrashLoopBackOff"}[5m])
```

Pass: empty result.

CrashLoopBackOff is unambiguous — the container cannot start and Kubernetes has
begun backing off. No retry. Pause the rollout immediately and ask the user
what to do next.

---

### Step 4 — Restart spike (5 m window)

```promql
increase(kube_pod_container_status_restarts_total{__NS_FILTER__}[5m])
```

Pass: `0` or empty.

Use **5 minutes** (not 10) so the window starts after the deploy wave completes,
avoiding pre-existing restarts from before the rollout. If this fires, always
cross-check with Step 3: a restart without CrashLoopBackOff may be a one-off
OOM or pre-existing churn rather than a deploy regression. If the signal
remains abnormal after the full gate logic, pause and ask the user what to do
next instead of auto-continuing.

---

## Interpreting results — signal vs. noise

| Pattern | Likely cause | Action |
|---------|--------------|--------|
| unavailable > 0, no restarts, no CrashLoop | Rolling-update scheduling blip | Re-check in 2 min; if cleared, proceed |
| unavailable > 0 **+** CrashLoop | Real pod startup failure | Pause rollout; ask user whether to rollback or investigate |
| restarts > 0, no CrashLoop | One-off restart (OOM, pre-existing) | Pause rollout; ask user whether to re-check or investigate |
| restarts > 0 **+** CrashLoop | Persistent startup failure | Pause rollout; recommend rollback and ask user to confirm |
| ready < 1 | Deployment not progressing at all | Pause rollout; recommend rollback and ask user to confirm |
| generation mismatch > 5 min | Rollout stalled (resource constraint, image pull failure) | Pause rollout; ask user whether to wait, investigate, or rollback |

## Gate decision

- All checks pass (or the sole unavailable blip cleared on retry) → proceed to next wave.
- Any unresolved failure or confirmed service/pod abnormality → pause rollout,
  report the failure pattern from the table above, and ask the user what to do
  next before taking further action.