# APAC tenant signal pipeline

This project monitors APAC TiDB Cloud tenants, enriches and scores them, and
sends daily and weekly summaries to Feishu. The six Python 3.12 Lambda
functions and their supporting AWS resources are managed as one SAM stack.

## Tenant scope

The regular active-tenant flow selects DevOps tenants that meet all of these
conditions:

- `status = 'active'`;
- `kind = 'external'`;
- `sales_force_region = 'APAC'`;
- `country NOT IN ('JP')`, so CN is included and JP is excluded;
- `plan NOT IN ('free_trial', 'new_user', 'closing')`.

The new-signup flow selects tenants created during the previous CST natural
day with `country_region = 'APAC'` and excludes JP. It intentionally does not
filter plans, because new signups commonly use `new_user` or `free_trial`.

## Data flow

```text
Daily EventBridge rule
        |
        v
   Dispatcher --sync--> HubSpot snapshot tables
        |
        +--sync--> Effective Owner Sync --> BigQuery
                         |               \-> Owner snapshot + existing Tenant rows
        |
        v
   Monitor SQS --> Worker --> Active or Sales-owned table
                         |
                         v
                    Scoring SQS --> Scoring --> score snapshot
                                           \-> HubSpot Contact priority

Daily Summary rule --------------------> APAC DB --> Feishu
Weekly Metrics rule -------------------> APAC DB --> Feishu
```

Dispatcher refreshes the local HubSpot target-list snapshot, selects the exact
Tenant batch, synchronously completes Effective Owner Sync, then emits one
message per tenant with that `sync_id`. A failed HubSpot refresh is non-blocking
and leaves the latest successful snapshot in use; a failed Owner sync blocks
fan-out. Worker enriches, applies HubSpot-before-sales-owner routing, upserts
exactly one tenant table, and requests a score recalculation.
Summary and Weekly Metrics read only `apac_active_tenants`.

## Components

| Function | Trigger | Responsibility | Runtime configuration |
| --- | --- | --- | --- |
| [Effective Sales Owner Sync](APAC-Effective-Sales-Owner-Sync-Task/README.md) | Synchronous Dispatcher invocation or manual dry-run | Read Nexus CRM through BigQuery, publish a versioned decision snapshot, and update existing Tenant rows | 720 s, 512 MB |
| [Dispatcher](APAC-Tenants-Monitor-Fanout-Dispatcher-Task/README.md) | Daily EventBridge or manual invocation | Refresh HubSpot, select tenants/signups, complete Owner sync, and fan out messages | 900 s, 256 MB |
| [Worker](APAC-Tenants-Monitor-Fanout-Worker-Task/README.md) | Monitor SQS, batch size 1 | Enrich, route to Active or Sales-owned, and publish a bucket-aware scoring message | 900 s, 512 MB, reserved concurrency 2 |
| [Scoring](APAC-Tenants-Scoring-Task/README.md) | Scoring SQS, batch size 1, maximum concurrency 2 | Score the selected bucket, retain normal score while forcing HubSpot matches to P0, then write Active priority to the existing Contact matched by tenant email | 180 s, 256 MB |
| [Weekly Metrics](APAC-Weekly-Tenant-Metrics-Task/README.md) | Weekly EventBridge or manual invocation | Aggregate APAC stock and previous-week signup metrics and send a Feishu card | 120 s, 256 MB |
| [Summary](APAC-Tenants-Summary-Task/README.md) | Daily EventBridge or manual invocation | Send APAC totals and up to 10 detail rows per section to Feishu | 120 s, 512 MB |

All six functions use active X-Ray tracing. Their configuration is supplied
through Lambda environment variables managed by CloudFormation. Sensitive
values are `NoEcho` stack parameters and must not be committed to source,
template defaults, or `samconfig.toml`.

Important processing behavior:

- Three known large tenants use Worker's forced Sales-owned fast path. They
  retain P0, move directly to `apac_sales_owned_tenants`, and skip all
  enrichment and scoring calls. Their IDs are documented in the Worker README.
- Worker always retrieves every DevOps cluster page so service-plan counts are
  accurate. At `cluster_count >= 10`, it skips per-cluster Clinic QPS requests
  and stores QPS as unknown.
- A Worker scoring-message failure uses the existing SQS retry behavior, so a
  retry runs the complete tenant operation again. Database writes are upserts.
- HubSpot target matches take precedence over APAC sales ownership. They remain
  Active, receive a normal numeric score, and have final priority `P0`.
- Other tenants whose Effective Owner matches a configured APAC Sales email move to
  `apac_sales_owned_tenants`, are scored normally, and are excluded from both
  reporting functions.
- Summary shows accurate totals while limiting each P0, P1, and recent-import
  detail section to 10 rows.

## Database

The target database is `regional_support`. [`schema.sql`](schema.sql) defines
the APAC tables without copying EMEA data. Existing environments apply
[`migration_hubspot_sales_routing.sql`](migration_hubspot_sales_routing.sql)
before the Lambda deployment.

| Table | Purpose |
| --- | --- |
| `apac_active_tenants` | Current unified tenant record, enrichment signals, score, priority, tracking state, and Effective Owner email |
| `apac_sales_owned_tenants` | Enriched and scored tenants already owned by APAC Sales, including Effective Owner email; excluded from reporting |
| `apac_effective_sales_owner_snapshot` | Versioned CRM Owner candidates and Effective Owner decisions, including only the selected Owner email |
| `apac_effective_sales_owner_sync_state` | Pointer and aggregate status for the latest complete Owner snapshot |
| `apac_hubspot_target_accounts` | Latest complete successful mirror of HubSpot target list membership and match fields |
| `apac_hubspot_target_sync_state` | Last successful snapshot and last refresh attempt metadata |
| `apac_tenant_events` | Import/event records used by the recent-import summary section |
| `apac_tenant_score_snapshots` | One score and priority snapshot per tenant per CST day |
| `domain_intelligence` | Cached email-domain classification and business information |

There is no separate `apac_new_signup` table. New signups are stored in
`apac_active_tenants` with `source_type = 'new_signup'`. None of these five
Lambda functions currently writes `apac_tenant_events`; its data must come
from an external event producer.

## AWS infrastructure

[`template.yaml`](template.yaml) is the source of truth for:

- six `AWS::Serverless::Function` resources;
- monitor and scoring queues, each with a dead-letter queue;
- Worker and Scoring SQS event-source mappings;
- three EventBridge schedule rules;
- least-privilege Lambda IAM roles;
- retained CloudWatch log groups with configurable retention;
- Lambda error and DLQ-visible-message alarms.

Deployment target:

| Setting | Value |
| --- | --- |
| AWS account | `552185537300` |
| Region | `us-west-2` |
| CloudFormation stack | `apac-tenant-monitoring` |
| AWS CLI profile | `552185537300_DBaaS-DevUser-Role` |

Schedule and Scoring runtime state was last re-verified on 2026-07-21. This is
a point-in-time record; CloudFormation and AWS runtime state remain
authoritative.

## Schedules

The template uses separate state parameters for the three schedules and its
defaults match the production rules verified on 2026-07-21:

| Task | Default state | EventBridge expression | UTC | Asia/Shanghai |
| --- | --- | --- | --- | --- |
| Dispatcher | `ENABLED` | `cron(30 0 * * ? *)` | Daily 00:30 | Daily 08:30 |
| Summary | `ENABLED` | `cron(0 4 * * ? *)` | Daily 04:00 | Daily 12:00 |
| Weekly Metrics | `DISABLED` | `cron(0 7 ? * MON *)` | Monday 07:00 | Monday 15:00 |

The production stack parameters were aligned with these values on 2026-07-21.
Sensitive parameters such as `APACSalesEmails` must still be supplied securely
or explicitly preserved during later deployments.

## Validate and build

Run these commands from this directory:

```bash
python3 -m unittest discover -s tests -p 'test_*.py' -v
sam validate --lint --template-file template.yaml
sam build --template-file template.yaml
```

Deploy the built template through the approved secret-injection workflow:

```bash
sam deploy \
  --template-file .aws-sam/build/template.yaml \
  --stack-name apac-tenant-monitoring \
  --region us-west-2 \
  --profile 552185537300_DBaaS-DevUser-Role \
  --capabilities CAPABILITY_IAM \
  --resolve-s3
```

Required stack parameter values must be supplied securely at deploy time. Do
not add them to this command, shell history, repository files, or documentation
as plain text.

## Rollout checklist

1. Preserve the production schedule state: Dispatcher and Summary enabled,
   Weekly disabled.
2. Apply the routing migration, then deploy HubSpot/APAC Sales parameters
   through the approved secret-injection workflow.
3. Invoke Dispatcher with a small `limit`; verify the HubSpot mirror count and
   that both SQS queues drain.
4. Confirm that the monitor and scoring DLQs remain empty.
5. Validate both tenant destinations, P0 override, and score snapshots.
6. Manually invoke Summary and Weekly Metrics and confirm Sales-owned tenants
   are absent.
7. Enable the schedules with a SAM stack update and monitor Lambda errors and
   DLQ alarms during the first complete run.