# SES Email Setup

How NexusCRM sends email via Amazon SES. Covers architecture, initial setup, per-environment configuration, DNS records, the staging recipient allowlist, smoke testing, and operational troubleshooting.

## Architecture

```
Go backend (EmailSender interface)
   │
   └── SMTPSender (net/smtp + STARTTLS)
          │
          └── email-smtp.<region>.amazonaws.com:587
                 │
                 └── AWS SES (account 604175033442, us-east-1)
```

The backend uses a pluggable `ports.EmailSender` interface. The SMTP implementation (`backend/internal/infrastructure/email/smtp_sender.go`) is selected automatically by `NewEmailSenderFromEnv` when `SMTP_HOST` is set. SES is used purely via its SMTP endpoint, so the backend has zero AWS SDK dependency.

Callers of `ports.EmailSender.Send()`:
- `backend/internal/features/automation/action_service.go:230` — Flow Email Alert action (also used by approval-process notifications via `approval_service.go:1080`)
- `backend/internal/interfaces/rest/oauth_handler.go:808` — password reset emails (admin users only)

## Environments

| Env | AWS account | Region | Sending domain | From address | Recipient restriction |
|-----|------|--------|----------------|--------------|-----------------------|
| staging | 604175033442 | us-east-1 | `mail.nexuscrm-staging.pingcap.com` | `noreply@mail.nexuscrm-staging.pingcap.com` | **12 internal addresses** (IAM allowlist) |
| prod | 604175033442 | us-east-1 | `mail.nexuscrm.pingcap.com` | `noreply@mail.nexuscrm.pingcap.com` | None |

Each environment has its own IAM user (`nexuscrm-<env>-ses-smtp`) and its own SES domain identity, so credentials, IAM policies, and sending reputation are isolated per env — even though they share one AWS account.

SES sandbox is **lifted** (production access granted). Account quota: **50,000 emails/day**, **14 emails/sec**.

## Terraform surface

All SES resources live in `infra/ses.tf`:

| Resource | Purpose |
|---|---|
| `aws_ses_domain_identity.email` | Domain registration; gated on `var.smtp_ses_domain != ""` |
| `aws_ses_domain_dkim.email` | Easy DKIM → 3 CNAME tokens |
| `aws_iam_user.ses_smtp` | Per-env SMTP user, name `nexuscrm-<env>-ses-smtp` |
| `aws_iam_user_policy.ses_smtp` | `ses:SendRawEmail` + optional recipient allowlist Condition |
| `aws_iam_access_key.ses_smtp` | SES SMTP creds derived via `ses_smtp_password_v4` |

The SSM parameters in `infra/ssm.tf` (`SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASSWORD`, `SMTP_FROM`, `SMTP_TLS`) are injected into the ECS task definition from `infra/ecs.tf`, so the running backend always has fresh credentials after a deploy.

DNS records are exposed via outputs (`infra/outputs.tf`):
- `ses_dkim_records` — 3 DKIM CNAMEs
- `ses_verification_token` — domain verification TXT

The `deploy.yml` workflow prints both to the GitHub Actions step summary after every successful apply — copy DNS records directly from the Actions run page.

### Terraform variables

| Variable | Default | Notes |
|---|---|---|
| `smtp_ses_domain` | `""` | Empty disables all SES resources for the workspace |
| `smtp_from_address` | `noreply@nexuscrm.pingcap.com` | SMTP_FROM env var value |
| `smtp_recipient_allowlist` | `[]` | Exact-match recipient allowlist. Non-empty → IAM policy adds `ForAllValues:StringEquals ses:Recipients` condition |

All three are injected from GitHub environment-scoped secrets by `deploy.yml`:
```yaml
TF_VAR_smtp_ses_domain: ${{ secrets.TF_VAR_SMTP_SES_DOMAIN }}
TF_VAR_smtp_from_address: ${{ secrets.TF_VAR_SMTP_FROM_ADDRESS }}
TF_VAR_smtp_recipient_allowlist: ${{ secrets.TF_VAR_SMTP_RECIPIENT_ALLOWLIST }}
```

For list variables like `TF_VAR_SMTP_RECIPIENT_ALLOWLIST`, the secret value must be a **JSON array string**, e.g. `["a@pingcap.com","b@pingcap.com"]`.

## First-time setup per environment

### 1. Set GitHub environment secrets

GitHub repo → **Settings → Environments → `<env>`**:

```
TF_VAR_SMTP_SES_DOMAIN     = mail.<env-specific>.pingcap.com
TF_VAR_SMTP_FROM_ADDRESS   = noreply@mail.<env-specific>.pingcap.com
```

Optional (staging-style restriction):
```
TF_VAR_SMTP_RECIPIENT_ALLOWLIST = ["alice@pingcap.com","bob@pingcap.com"]
```

### 2. Trigger deploy

```bash
gh workflow run Deploy --repo pingcap-inc/MetaDrivenPlatform4AI \
  --ref feature/new-aws-account -f environment=<env>
```

Or GitHub → Actions → **Deploy** workflow → Run workflow → pick environment.

Terraform will:
- Create or replace `aws_ses_domain_identity.email`
- Rotate DKIM tokens if the domain changed
- Create/update the SMTP IAM user, access key, and SSM parameters
- Restart the ECS service with the new env vars (takes ~3–5 min to stabilize)

The identity starts in `PENDING` state — the ECS service is healthy but real sends fail until DNS records resolve.

### 3. Add DNS records

From the workflow step summary, copy the records under **SES DNS Records**:

```
# Domain verification (TXT)
_amazonses.mail.<domain>  TXT  "<verification-token>"

# DKIM (CNAME x3)
<token1>._domainkey.mail.<domain>  CNAME  <token1>.dkim.amazonses.com
<token2>._domainkey.mail.<domain>  CNAME  <token2>.dkim.amazonses.com
<token3>._domainkey.mail.<domain>  CNAME  <token3>.dkim.amazonses.com

# SPF (recommended — add manually)
mail.<domain>  TXT  "v=spf1 include:amazonses.com ~all"

# DMARC (recommended — add manually)
_dmarc.mail.<domain>  TXT  "v=DMARC1; p=none; rua=mailto:dmarc@pingcap.com"
```

Add them to the authoritative DNS for `pingcap.com`. Trailing-dot and quoting rules depend on the DNS provider's UI.

**If your local resolver can't reach pingcap.com authoritative NS**, verify propagation via Google Public DNS over HTTPS:

```bash
curl -s "https://dns.google/resolve?name=_amazonses.mail.<domain>&type=TXT" | jq
```

### 4. Verify

```bash
AWS_PROFILE=nexuscrm aws sesv2 get-email-identity \
  --email-identity mail.<env-domain>.pingcap.com \
  --region us-east-1 \
  --query '{Verified:VerifiedForSendingStatus,DKIM:DkimAttributes.Status}'
```

Expect `Verified: true` and `DKIM: SUCCESS`. The `_amazonses` TXT record alone gets you `Verified: true` quickly; `DKIM: SUCCESS` takes an extra minute or two as SES re-polls the DKIM CNAMEs.

## Recipient allowlist (staging only)

To prevent staging from sending email to real customers during testing, staging's SES IAM user has a `ForAllValues:StringEquals ses:Recipients` condition on `ses:SendRawEmail`. SES evaluates this against every envelope recipient (To/Cc/Bcc), so a single unauthorized address rejects the entire send with:

```
554 Access denied: User `arn:aws:iam::604175033442:user/nexuscrm-staging-ses-smtp'
is not authorized to perform `ses:SendRawEmail' on resource
`arn:aws:ses:us-east-1:604175033442:identity/mail.nexuscrm-staging.pingcap.com'
```

The enforcement is **at the AWS layer** — even if application code bypasses the `SMTPSender` abstraction or an attacker gets the SMTP credentials, AWS still rejects any recipient outside the list.

Prod has **no allowlist** — `TF_VAR_SMTP_RECIPIENT_ALLOWLIST` is unset in the prod environment secret, so the Terraform variable defaults to `[]`, which skips injection of the Condition block entirely.

### Current staging allowlist

Managed via the `TF_VAR_SMTP_RECIPIENT_ALLOWLIST` GitHub secret in the **staging** environment. Read the live IAM policy if you need the current list:

```bash
AWS_PROFILE=nexuscrm aws iam get-user-policy \
  --user-name nexuscrm-staging-ses-smtp \
  --policy-name ses-send-email \
  --query 'PolicyDocument.Statement[0].Condition."ForAllValues:StringEquals"."ses:Recipients"'
```

### Adding or removing addresses

1. GitHub repo → **Settings → Environments → staging** → edit `TF_VAR_SMTP_RECIPIENT_ALLOWLIST`.
   Value must remain a JSON array: `["a@pingcap.com","b@pingcap.com",...]`
2. Trigger a staging Deploy: `gh workflow run Deploy --ref feature/new-aws-account -f environment=staging`
3. Wait ~5 min for the Terraform apply + ECS rollout (rollout only happens because SSM is unchanged — IAM policy change alone doesn't restart ECS; the new policy is effective immediately for new SMTP connections).
4. Verify:
   ```bash
   AWS_PROFILE=nexuscrm aws iam get-user-policy \
     --user-name nexuscrm-staging-ses-smtp \
     --policy-name ses-send-email
   ```

### Why IAM condition, not application-layer filter

- **Unbypassable**: a future bug, a compromised SMTP credential, or a direct `aws ses send-email` call with the staging IAM user all fail the same way
- **Zero backend code change**: `SMTPSender` stays generic, no new flag to test
- **Per-env isolation**: staging and prod use different IAM users, so the policy is naturally scoped
- **Already Terraform-managed**: lives with the rest of `infra/ses.tf`

## Smoke testing

Use this standalone Python script to validate SMTP end-to-end without touching backend code. Runs directly against the env's SMTP credentials from SSM:

```bash
export AWS_PROFILE=nexuscrm
ENV=staging  # or prod

export SMTP_USER=$(aws ssm get-parameter --name /nexuscrm-$ENV/SMTP_USER \
  --region us-east-1 --with-decryption --query 'Parameter.Value' --output text)
export SMTP_PASS=$(aws ssm get-parameter --name /nexuscrm-$ENV/SMTP_PASSWORD \
  --region us-east-1 --with-decryption --query 'Parameter.Value' --output text)
export SMTP_HOST=$(aws ssm get-parameter --name /nexuscrm-$ENV/SMTP_HOST \
  --region us-east-1 --query 'Parameter.Value' --output text)
export SMTP_FROM=$(aws ssm get-parameter --name /nexuscrm-$ENV/SMTP_FROM \
  --region us-east-1 --query 'Parameter.Value' --output text)

python3 <<'PYEOF'
import smtplib, ssl, os
from email.mime.text import MIMEText
msg = MIMEText("SES smoke test", "plain", "utf-8")
msg["Subject"] = "SES smoke test"
msg["From"] = os.environ["SMTP_FROM"]
msg["To"] = "xiaopeng.zhu@pingcap.com"   # must be in staging allowlist
ctx = ssl.create_default_context()
with smtplib.SMTP(os.environ["SMTP_HOST"], 587, timeout=30) as s:
    s.ehlo(); s.starttls(context=ctx); s.ehlo()
    s.login(os.environ["SMTP_USER"], os.environ["SMTP_PASS"])
    resp = s.sendmail(os.environ["SMTP_FROM"], ["xiaopeng.zhu@pingcap.com"], msg.as_string())
    print(f"250 OK (resp={resp})")
PYEOF
```

Expected outcomes:

| Env | Recipient | Expected |
|---|---|---|
| staging | In the 12-address allowlist | **250 OK**, delivered, DKIM-signed |
| staging | Not in allowlist (e.g. any other `@pingcap.com`, any `@gmail.com`, any `@example.com`) | **554 Access denied** from SES at SMTP DATA stage |
| staging | Mix of allowed + not allowed in one envelope | **554 Access denied** (`ForAllValues` — one bad recipient rejects the whole send) |
| prod | Any recipient | **250 OK** |

On receipt, verify:
- Email lands in inbox (not spam)
- Raw headers show `dkim=pass` with the sending domain (e.g. `mail.nexuscrm-staging.pingcap.com`)
- `From` header matches `$SMTP_FROM`

## Operational troubleshooting

### `554 Access denied` on staging for a legitimate recipient

The recipient is not in the staging allowlist. Either:
- Add them to `TF_VAR_SMTP_RECIPIENT_ALLOWLIST` secret and redeploy staging (see above), or
- Use the prod environment for that send

### `550 Email address is not verified`

SES is still in sandbox for the account, or the from-address domain is not verified. Check:
```bash
AWS_PROFILE=nexuscrm aws sesv2 get-account --region us-east-1 \
  --query '{Production:ProductionAccessEnabled,Sending:SendingEnabled}'
```
If `ProductionAccessEnabled: false`, file a sandbox-lift request (see "Lift SES sandbox" below).

### `454 Throttling`

Account is hitting `MaxSendRate` (14/s) or `Max24HourSend` (50,000/day). Check:
```bash
AWS_PROFILE=nexuscrm aws sesv2 get-account --region us-east-1 --query 'SendQuota'
```

The current `SMTPSender` has no retry logic — a 454 bubbles up to the caller (Flow/OAuth path). If this happens with any regularity, consider adding a retry-with-backoff wrapper inside `infrastructure/email/`, or request a rate-limit increase from AWS Support.

### Emails silently dropping (no SMTP error, no inbox)

- Check if the recipient hard-bounced previously — SES account-level suppression list blocks those. List:
  ```bash
  AWS_PROFILE=nexuscrm aws sesv2 list-suppressed-destinations --region us-east-1
  ```
- Remove an entry after confirming the address is valid again:
  ```bash
  AWS_PROFILE=nexuscrm aws sesv2 delete-suppressed-destination \
    --email-address user@example.com --region us-east-1
  ```

### Rotating SMTP credentials

```bash
cd infra
AWS_PROFILE=nexuscrm terraform workspace select <env>
AWS_PROFILE=nexuscrm terraform taint 'aws_iam_access_key.ses_smtp[0]'
```

Then trigger a Deploy workflow run. Terraform generates new credentials, updates SSM, ECS picks them up on next task rollout.

### DNS not propagating

If `aws sesv2 get-email-identity` shows `DKIM: FAILED` with `ErrorType: HOST_NOT_FOUND`, the DKIM CNAMEs haven't reached SES's resolvers yet. Verify via DoH:
```bash
curl -s "https://dns.google/resolve?name=<token>._domainkey.mail.<domain>&type=CNAME" | jq
```
If DoH shows the record but `aws sesv2` still says FAILED, wait a few minutes and re-query — SES retries with exponential backoff.

## Lift SES sandbox (already done, documented for future accounts)

New AWS accounts start in SES sandbox: 200 emails/day, 1/sec, only verified recipients. Production access is requested once per account:

AWS console → **SES → Account dashboard → Request production access**.

Approved on 2026-04-14 for account `604175033442`. Use-case text kept for future accounts:

> NexusCRM is an internal CRM platform built on a Salesforce-compatible API. It sends
> transactional emails only: password reset, email verification, approval-workflow
> notifications, and Flow-automation email alerts triggered by user actions. Recipients
> are employees and customers who have explicitly signed up for the product. Expected
> volume: under 1,000 emails per day initially, scaling with customer count. All bounces
> and complaints are handled by SES's account-level suppression list; we will wire up
> SNS bounce/complaint notifications when volume justifies the operational overhead.
> We comply with the AWS Acceptable Use Policy and will maintain bounce rate under 5%
> and complaint rate under 0.1%.

Mail type: **Transactional**. Approval typically takes under 24 hours.

## Known gotchas

**Terraform ternary with object literals.** `ses.tf` builds the IAM policy statement dynamically based on whether the allowlist is empty. The first attempt used:
```hcl
statement = length(allowlist) > 0 ? { Effect=..., Action=..., Condition=... } : { Effect=..., Action=... }
```
Terraform rejects this at plan time with `Inconsistent conditional result types` because the two branches have different keys. Use `merge()` instead — it is untyped and happily merges objects with different shapes. The pattern currently in `ses.tf`:
```hcl
condition_block = length(allowlist) > 0 ? { Condition = {...} } : {}
statement = merge({ Effect=..., Action=..., Resource="*" }, condition_block)
```

**`ses:Recipients` catches Bcc.** You cannot split a disallowed recipient into Bcc to bypass the allowlist — SES checks the full envelope recipient set.

**`ForAllValues` semantics.** This IAM condition operator means *every* listed value must match *at least one* value in the allowlist. A single unauthorized recipient rejects the whole send. That is the safe default for a testing environment (never accidentally leak a real customer email).

**Cross-env account sharing.** staging and prod live in the same AWS account (`604175033442`). Per-env IAM users and per-env SMTP credentials make this fine — each env gets its own blast radius for reputation and credentials. Sandbox lift is account-wide, so lifting it for prod also unlocks staging.