# Region Awareness — Hardening & Production Readiness

**Date:** 2026-04-09
**Status:** Draft hardening checklist
**Scope:** Error handling, fallbacks, observability, rollback, feature flags, and migration safety for each phase

---

## 1. Error Handling Strategies by Phase

### Phase 1: Schema + Core Engine

| Failure Mode | Impact | Handling Strategy |
|---|---|---|
| **`regionBestWindows` JSON is malformed** | Silent scheduling failures — posts get no slots or wrong slots | Validate JSON shape at write time (TrackConfig update API) AND at read time (`findNextSlot`). Use a Zod schema: `z.record(z.nativeEnum(Region), z.array(z.object({ day: z.string(), hour: z.number().min(0).max(23) })))`. If parse fails at read time, log `WARN region.bestWindows.parse_failed { trackId, raw }` and fall back to `bestWindows` (existing field). Never silently swallow. |
| **Timezone conversion fails (invalid IANA identifier)** | Post scheduled at wrong UTC hour | The region→timezone map is a hardcoded constant (`lib/region-config.ts`), not user input. Validate at startup with `Intl.DateTimeFormat(tz)` — if it throws, fail fast with `FATAL: invalid timezone for region`. This should never happen at runtime; it's a code error. |
| **DST transition edge case — slot shifts ±1hr** | Post publishes at 10am instead of 9am (or vice versa) twice per year | Use `date-fns-tz` `zonedTimeToUtc()` which handles DST automatically via IANA identifiers. Add an integration test: schedule a post for 2:30 AM on the spring-forward date in `America/Los_Angeles` — the library should skip to 3:00 AM, not produce an invalid time. Log `INFO region.dst_adjustment { region, originalHour, adjustedHour }` when DST causes a shift. |
| **`Campaign.region` is null — consumer forgets GLOBAL fallback** | Region-dependent logic skips the post entirely or throws | Create a helper `resolveRegion(campaign: Campaign): Region` that returns `campaign.region ?? 'GLOBAL'`. Use this everywhere instead of raw field access. Lint rule: flag direct `campaign.region` access outside the helper. |
| **`reassignTrackSlots()` changes existing GLOBAL post times** | Users see scheduled posts shift unexpectedly after region feature deploys | When `reassignTrackSlots()` runs for the first time after deploy, compare new slot times to old ones. If any existing post moves by > 0 minutes, log `WARN region.reassign.post_shifted { postId, oldTime, newTime }`. Consider: only reassign posts created *after* the feature ships; leave existing posts at their current times. |
| **Seeding `regionBestWindows` with wrong defaults** | Posts for APAC scheduled at AMERICAS hours | Don't auto-seed. Ship with `regionBestWindows = "{}"` (empty) for all existing tracks. This triggers the fallback to `bestWindows`, preserving current behavior. Let the team configure region windows explicitly via admin UI or API after validating the hours. |

### Phase 2: HubSpot Integration

| Failure Mode | Impact | Handling Strategy |
|---|---|---|
| **Region resolution chain returns unexpected value** | Post assigned wrong region, scheduled at wrong time, geo-targeted incorrectly | Each resolution level should log which level resolved the region: `INFO region.resolved { level: "utm_suffix", value: "APAC", campaignId, payload }`. If no level resolves, default to GLOBAL and log `WARN region.resolved.default { campaignId }`. This makes debugging trivial. |
| **HubSpot `social_region` property doesn't exist** | Resolution level 2 fails silently | Check for the property's existence on startup or first webhook. If missing, skip that level and log `WARN hubspot.social_region.missing — skipping campaign property resolution`. Do not throw. |
| **Multi-region split (`"APAC,EMEA"`) creates duplicate posts** | Double post volume, track cap consumed faster than expected | Before creating posts, check current track slot usage. If splitting would exceed `weeklySlotCap`, log `WARN region.multiregion.cap_exceeded { campaignId, requestedRegions, availableSlots }` and create posts only for regions that fit, in order. Return a partial-success response to HubSpot with a note about which regions were dropped. |
| **Race condition: concurrent webhooks write different regions to same Campaign** | Campaign.region flip-flops unpredictably | Use an upsert with a `WHERE updatedAt < now()` guard, or use Prisma's `update` with optimistic concurrency (version field). If the write conflicts, the second webhook should read the current Campaign.region and skip the write if it's already set. Log `WARN region.campaign.race { campaignId, attempted, existing }`. |
| **Translation layer fails mid-processing** | Post created without region, or region set but post not created | Wrap the translation + post creation in a transaction. If any step fails, roll back the entire webhook processing. Return 500 to HubSpot so it retries. Log `ERROR hubspot.translation.failed { campaignId, step, error }`. |

### Phase 3: Platform Targeting

| Failure Mode | Impact | Handling Strategy |
|---|---|---|
| **LinkedIn API rejects targeting params** | Post fails to publish entirely | Catch the targeting-specific error. Retry once without targeting params (publish as GLOBAL). Log `WARN linkedin.targeting.rejected { postId, region, error, fallback: "global" }`. Set `post.metadata.targetingFailed = true` for analytics visibility. Mark the post as published (not failed) since the content still went out. |
| **LinkedIn API rate limit hit during targeting** | Publish queue backs up | Use existing rate-limit retry logic. If rate-limited on a targeting call specifically, the retry should include targeting (don't drop it on retry). Log `WARN linkedin.targeting.rate_limited { postId, retryAfter }`. |
| **Region→country mapping has invalid URNs** | LinkedIn API 400 error | Validate URN format at build/startup time: each entry must match `urn:li:country:[a-z]{2}`. Add a unit test that checks all URNs in the mapping config are well-formed. At runtime, if LinkedIn returns 400 on a targeting call, fall back to no-targeting publish. |
| **Facebook targeting used before publisher is built** | Runtime error on a code path that shouldn't exist yet | Guard with `if (publisher.supportsTargeting)` check. Facebook publisher stub should return `supportsTargeting = false`. No targeting code runs for unsupported publishers. |

### Phase 4 & 5: UI

| Failure Mode | Impact | Handling Strategy |
|---|---|---|
| **`regionBestWindows` API returns malformed data to frontend** | TimeSlotPicker shows wrong or no slots | Frontend should validate the response shape. If invalid, show existing `bestWindows` slots with a toast: "Region-specific windows unavailable, showing default schedule." |
| **Analytics `?region=` filter returns empty for a region with posts** | User thinks region has no data | Check if region is being correctly passed as enum value (not display name). API should validate `region` param and return 400 for invalid values, not empty results. |
| **Calendar shows too many colors (all 4 regions active)** | Visual noise, hard to read | Default to "All Regions" with subtle left-border color. Provide a toggle to highlight a single region (dims others). Don't show color legend unless > 1 region is visible. |

---

## 2. Fallback Behaviors — Graceful Degradation Paths

Every region-aware feature must degrade to GLOBAL behavior when region data is missing or invalid.

```
Degradation Chain:
  regionBestWindows[region] → bestWindows → hardcoded default (MON/WED/FRI 9am UTC)
  campaign.region           → GLOBAL
  post.region               → campaign.region → GLOBAL
  targeting params          → no targeting (publish to all)
  region filter (UI)        → "All Regions" (no filter)
  analytics by region       → aggregate (no breakdown)
```

### Specific degradation rules:

1. **Scheduling fallback:** If `regionBestWindows` is empty/malformed for a given region, use the track's existing `bestWindows`. If that's also empty, use a hardcoded default: Monday, Wednesday, Friday at 09:00 UTC. Log `WARN region.schedule.fallback { trackId, region, level: "bestWindows"|"hardcoded" }`.

2. **Targeting fallback:** If geo-targeting fails for any reason (API error, invalid URNs, rate limit after retries exhausted), publish without targeting. The post reaches everyone instead of a region — acceptable degradation. Never fail a publish because targeting failed.

3. **Resolution fallback:** If the entire 5-level region resolution chain fails (all levels throw), default to GLOBAL. Log `ERROR region.resolution.all_failed { campaignId }`. This should be extremely rare — the final level is a hardcoded default.

4. **UI fallback:** If the region config API (`/api/v1/region-config`) is unreachable, the UI should hide region-specific features (selectors, filters, badges) rather than showing broken UI. Show a subtle "Region features unavailable" indicator in the compose form.

---

## 3. Monitoring & Observability

### 3a. Metrics to Track

| Metric | Type | Labels | Alert Threshold |
|---|---|---|---|
| `region.posts.scheduled` | Counter | `region`, `track` | None (informational) |
| `region.posts.published` | Counter | `region`, `network`, `targeted` | None (informational) |
| `region.targeting.failed` | Counter | `region`, `network`, `error_type` | > 5 in 1 hour |
| `region.targeting.fallback` | Counter | `region`, `network` | > 10 in 1 hour |
| `region.schedule.fallback` | Counter | `region`, `fallback_level` | > 3 in 1 hour (may indicate config issue) |
| `region.resolution.level` | Histogram | `level` (1-6) | p95 > level 4 (too many posts falling through to defaults) |
| `region.bestwindows.parse_error` | Counter | `trackId` | > 0 (immediate alert — config is broken) |
| `region.dst_adjustment` | Counter | `region` | None (expected twice/year, but useful to confirm) |
| `region.campaign.race` | Counter | `campaignId` | > 0 (investigate concurrency) |

### 3b. Structured Log Events

All region-related logs should include a `region.*` prefix for easy filtering. Key events:

```
INFO  region.resolved              { campaignId, postId, level, value }
INFO  region.schedule.slot         { postId, region, localTime, utcTime, timezone }
INFO  region.targeting.applied     { postId, region, network, countries[] }
INFO  region.dst_adjustment        { region, date, originalHour, adjustedHour }
WARN  region.bestwindows.fallback  { trackId, region, fallbackTo }
WARN  region.targeting.rejected    { postId, region, network, error, action: "published_without_targeting" }
WARN  region.campaign.race         { campaignId, attempted, existing }
WARN  region.multiregion.cap       { campaignId, regions, availableSlots, droppedRegions }
ERROR region.bestwindows.parse     { trackId, rawJson, error }
ERROR region.resolution.failed     { campaignId, step, error }
ERROR region.translation.failed    { campaignId, step, error }
```

### 3c. Alerts to Configure

| Alert | Condition | Severity | Action |
|---|---|---|---|
| Region best windows parse failure | `region.bestwindows.parse_error > 0` | P1 | Check TrackConfig data; fix malformed JSON immediately |
| Targeting failures spike | `region.targeting.failed > 5/hr` | P2 | Check LinkedIn API status; review targeting params |
| All posts falling to GLOBAL | `region.resolution.level p95 > 5` for 1hr | P3 | Check HubSpot `social_region` property; review webhook payloads |
| Campaign race condition | `region.campaign.race > 0` | P3 | Review concurrent webhook handling; may need stricter locking |
| DST adjustment outside expected window | `region.dst_adjustment` outside Mar/Nov | P3 | Investigate — may indicate timezone config error |

### 3d. Dashboard Additions

Add a "Region Health" section to the existing admin dashboard:
- Posts by region (last 7 days) — bar chart
- Targeting success rate by network — percentage
- Resolution level distribution — shows how posts are getting their region assigned
- Fallback rate — percentage of posts using fallback scheduling

---

## 4. Rollback Plans by Phase

### Phase 1: Schema + Core Engine

**Rollback trigger:** `findNextSlot()` returns incorrect times, or existing GLOBAL posts shift unexpectedly.

**Steps:**
1. Feature flag `REGION_SCHEDULING_ENABLED` → `false` (see §5). This makes `findNextSlot()` ignore the `region` param and use existing `bestWindows` logic.
2. The `Campaign.region` and `regionBestWindows` columns stay in the database (nullable, no harm). No reverse migration needed.
3. Verify: all posts schedule using pre-region logic. Existing behavior restored.

**Data safety:** No data is lost. The new columns are additive. Posts created with region values keep them, but the scheduling engine ignores them when the flag is off.

### Phase 2: HubSpot Integration

**Rollback trigger:** Webhook translation creates posts with wrong regions, or `campaign.region` writes cause errors.

**Steps:**
1. Feature flag `REGION_HUBSPOT_RESOLUTION` → `false`. Translation layer skips all region resolution; posts get `region = null` (treated as GLOBAL).
2. If `campaign.region` writes caused data issues, run a cleanup query: `UPDATE Campaign SET region = NULL WHERE region IS NOT NULL AND updatedAt > [deploy_timestamp]`.
3. Verify: webhook creates posts as before, all treated as GLOBAL.

**Risk:** Posts created during the broken window may have incorrect regions. The cleanup query above resets campaigns. Posts keep their region values but behave as GLOBAL when Phase 1 flag is also off.

### Phase 3: Platform Targeting

**Rollback trigger:** LinkedIn publish failures increase, or posts are geo-targeted incorrectly.

**Steps:**
1. Feature flag `REGION_GEO_TARGETING` → `false`. `publishPost()` skips all targeting params.
2. No data cleanup needed — targeting is a runtime behavior, not stored data.
3. Verify: LinkedIn publishes succeed without targeting params. All posts reach global audience.

**This is the safest phase to roll back** — targeting is purely additive and the publish flow worked without it before.

### Phase 4 & 5: UI Changes

**Rollback trigger:** UI bugs, broken filters, or performance issues with region queries.

**Steps:**
1. Feature flag `REGION_UI_ENABLED` → `false`. Hides all region selectors, filters, badges, and analytics breakdowns.
2. No backend changes needed — the API endpoints still accept `?region=` but the UI won't send it.
3. Verify: all admin views render as before without region elements.

**Partial rollback:** Individual UI components can be toggled independently:
- `REGION_UI_COMPOSE` — compose form region selector
- `REGION_UI_CALENDAR` — calendar region filter + colors
- `REGION_UI_ANALYTICS` — analytics region breakdown
- `REGION_UI_QUEUE` — queue page region filter

---

## 5. Feature Flags

### Flag Hierarchy

```
REGION_AWARENESS (master kill switch)
├── REGION_SCHEDULING_ENABLED     (Phase 1: timezone-aware slot generation)
├── REGION_HUBSPOT_RESOLUTION     (Phase 2: webhook region resolution chain)
├── REGION_GEO_TARGETING          (Phase 3: LinkedIn/Facebook targeting params)
└── REGION_UI_ENABLED             (Phase 4-5: all UI features)
    ├── REGION_UI_COMPOSE
    ├── REGION_UI_CALENDAR
    ├── REGION_UI_ANALYTICS
    └── REGION_UI_QUEUE
```

**Master flag `REGION_AWARENESS`:** When `false`, the entire feature is disabled. All sub-flags are ignored. The system behaves as if region awareness was never built.

### Implementation

Store flags in environment variables (simplest) or in the existing app config:

```typescript
// lib/feature-flags.ts
export const regionFlags = {
  enabled:            process.env.REGION_AWARENESS === 'true',
  scheduling:         process.env.REGION_SCHEDULING_ENABLED === 'true',
  hubspotResolution:  process.env.REGION_HUBSPOT_RESOLUTION === 'true',
  geoTargeting:       process.env.REGION_GEO_TARGETING === 'true',
  ui:                 process.env.REGION_UI_ENABLED === 'true',
  uiCompose:          process.env.REGION_UI_COMPOSE === 'true',
  uiCalendar:         process.env.REGION_UI_CALENDAR === 'true',
  uiAnalytics:        process.env.REGION_UI_ANALYTICS === 'true',
  uiQueue:            process.env.REGION_UI_QUEUE === 'true',
} as const;

export function isRegionFeatureEnabled(flag: keyof typeof regionFlags): boolean {
  if (!regionFlags.enabled) return false;
  return regionFlags[flag];
}
```

### Rollout Plan

| Week | Flags Enabled | Audience | Success Criteria |
|---|---|---|---|
| Week 1 | `REGION_AWARENESS`, `REGION_SCHEDULING_ENABLED` | Internal team only | Posts schedule at correct regional times; no existing post times shift |
| Week 2 | + `REGION_HUBSPOT_RESOLUTION` | Internal + HubSpot test campaigns | Webhook creates posts with correct regions; resolution chain logs show correct levels |
| Week 3 | + `REGION_GEO_TARGETING` | Internal + 1 real LinkedIn campaign | LinkedIn post targets correct countries; no publish failures |
| Week 4 | + `REGION_UI_ENABLED` (all sub-flags) | All users | UI shows region selectors, filters, badges; no visual regressions |

---

## 6. Data Migration Safety

### 6a. Campaign.region Column

**Migration SQL:**
```sql
ALTER TABLE "Campaign" ADD COLUMN "region" TEXT NULL;
```

**Safety analysis:**
- `NULL`able column with no default → **zero-downtime migration**. No table rewrite needed (Postgres adds nullable columns instantly). SQLite (dev) also handles this without locking.
- No existing queries break — they don't reference the column.
- The column uses `TEXT` (not an enum type at the database level) because Prisma maps `Region?` to a text column with application-level validation. No need for a database-level enum migration later.

**Verification steps:**
1. Run migration on a database copy first: `npx prisma migrate deploy --preview`
2. Confirm no table lock: check `pg_locks` during migration (should be brief `ACCESS EXCLUSIVE` for `ALTER TABLE`, not a full table rewrite)
3. Run the existing test suite — all tests should pass since no existing code references `Campaign.region`

### 6b. TrackConfig.regionBestWindows Column

**Migration SQL:**
```sql
ALTER TABLE "TrackConfig" ADD COLUMN "regionBestWindows" TEXT NOT NULL DEFAULT '{}';
```

**Safety analysis:**
- `NOT NULL` with `DEFAULT '{}'` → Postgres uses the default for existing rows without rewriting the table (Postgres 11+). For older Postgres, this triggers a table rewrite — check version first.
- The default `'{}'` means all existing tracks have no region-specific windows and will fall back to `bestWindows` — **preserving current behavior exactly**.
- No existing queries break.

**Do NOT seed data during migration.** Seeding region-specific windows requires product decisions (which hours for which regions). Ship with empty defaults and let the team configure via the admin API after the feature is live.

**Verification steps:**
1. After migration, verify fallback: `findNextSlot("DEVELOPER")` (no region) should return the same time as before the migration.
2. After migration, verify new path: `findNextSlot("DEVELOPER", "APAC")` with empty `regionBestWindows` should also fall back to `bestWindows` and return the same time.

### 6c. Migration Order

```
1. Run Phase 1 migrations (Campaign.region + TrackConfig.regionBestWindows)
2. Deploy code with all feature flags OFF
3. Verify: application starts, all existing functionality works, tests pass
4. Enable REGION_SCHEDULING_ENABLED flag
5. Monitor for 24 hours
6. Proceed to Phase 2
```

**Never deploy code that references new columns before the migration runs.** The CI pipeline should run migrations before deploying the new application version. If using blue/green deploys, the old version must not break when the new columns exist (it won't — they're additive).

### 6d. Reverse Migration (Emergency)

If the columns must be removed (unlikely — they're harmless when unused):

```sql
ALTER TABLE "Campaign" DROP COLUMN IF EXISTS "region";
ALTER TABLE "TrackConfig" DROP COLUMN IF EXISTS "regionBestWindows";
```

**Only run this if:** the columns are actively causing issues (e.g., Prisma client errors). Normally, just disable feature flags and leave the columns in place.

---

## 7. Risk Mitigation Matrix

Cross-references risks from REGION_AWARENESS_ESTIMATION.md with specific mitigations:

| Risk (from Estimation) | Mitigation | Validation |
|---|---|---|
| **DST-correct timezone conversion in `findNextSlot()`** (Phase 1, High complexity) | Use `date-fns-tz` with IANA identifiers; never use fixed UTC offsets. Integration test for spring-forward and fall-back dates. | Test: schedule post for 2:30 AM on Mar 9 2026 in `America/Los_Angeles` → should adjust to 3:00 AM. |
| **`regionBestWindows` malformed JSON = silent failures** (Phase 1) | Zod validation at write AND read time. Parse failure falls back to `bestWindows` + emits `ERROR` log + increments `region.bestwindows.parse_error` metric. | Test: write `{"APAC": "invalid"}` to `regionBestWindows` → `findNextSlot` uses `bestWindows`, metric fires. |
| **`reassignTrackSlots()` shifts existing posts** (Phase 1) | Skip reassignment for posts created before feature flag was enabled. Only reassign posts with `region != null`. | Test: enable flag → run reassign → verify pre-existing GLOBAL posts haven't moved. |
| **5-level resolution chain needs full test coverage** (Phase 2) | One unit test per level (5 tests) + one integration test for the full chain + one test for all-levels-failing → GLOBAL default. | Test plan E-01 through E-05 in REGION_AWARENESS_TEST_PLAN.md. |
| **Campaign.region write is a new write path** (Phase 2) | Transaction wraps post creation + campaign update. If campaign update fails, roll back post creation too. | Test: simulate campaign write failure → verify no orphan posts created. |
| **Race condition on Campaign.region** (Phase 2, E-16) | Optimistic concurrency via `updatedAt` check. Second write logs a warning and no-ops if region already set. | Test: fire 2 concurrent webhooks with different regions → only first write persists, second logs `WARN`. |
| **LinkedIn API rejects targeting** (Phase 3) | Catch and retry without targeting. Publish succeeds (without geo filter) + log `WARN` + set `metadata.targetingFailed`. | Test: mock LinkedIn returning 400 on targeting → post publishes globally, flag set. |
| **Region→country mapping is a product decision** (Phase 3) | Ship with a reasonable default mapping but mark it as "configurable" in the admin API. Require marketing sign-off before enabling `REGION_GEO_TARGETING` flag. | Review: marketing approves country list before Week 3 rollout. |
| **Frontend needs region→timezone mapping** (Phase 4) | Expose via `/api/v1/region-config` endpoint. Don't duplicate the config in frontend code. Single source of truth in `lib/region-config.ts`, served to frontend via API. | Test: change timezone mapping in config → frontend picks up new value without rebuild. |
| **Bulk approve by region — partial failures** (Phase 5) | Return per-post results: `{ approved: [...], failed: [...] }`. Show toast with count of successes and failures. Don't roll back successful approvals if some fail. | Test: approve 5 posts in batch, 1 fails → 4 approved, toast shows "4 approved, 1 failed". |

---

## 8. Pre-Deploy Checklist

Before enabling each phase's feature flag in production:

### All Phases
- [ ] Existing test suite passes on `main` (no pre-existing failures)
- [ ] Migration has been run on staging and verified
- [ ] Feature flag is confirmed `false` in production env
- [ ] Monitoring dashboards are set up for the phase's metrics
- [ ] Rollback steps are documented in the runbook and tested on staging

### Phase 1 Specific
- [ ] `findNextSlot()` unit tests pass for all 4 regions + null
- [ ] DST edge case integration test passes
- [ ] `regionBestWindows` Zod schema validation is in place
- [ ] `resolveRegion()` helper is used everywhere (no raw `campaign.region` access)
- [ ] Existing GLOBAL posts are verified to not shift times after deploy

### Phase 2 Specific
- [ ] e2e tests for `campaign.region` writes are in place (3-4 new tests)
- [ ] Full resolution chain integration test passes
- [ ] Race condition test (E-16) passes
- [ ] Marketing team has created `social_region` HubSpot property (or flag is set to skip it)

### Phase 3 Specific
- [ ] LinkedIn targeting URNs validated against LinkedIn API docs
- [ ] Country mapping reviewed and approved by marketing
- [ ] Fallback-to-global behavior tested with mock API failures
- [ ] Test publish with targeting on LinkedIn test org succeeds

### Phase 4-5 Specific
- [ ] RegionSelector component renders correctly with all region values
- [ ] Calendar performance tested with 100+ posts across 4 regions
- [ ] Analytics query has index on `post.region` (check with `EXPLAIN ANALYZE`)
- [ ] All region UI features hidden when `REGION_UI_ENABLED = false`