# Design Documents ## Table of Contents - [Design Documents](#design-documents) - [Table of Contents](#table-of-contents) - [Introduction](#introduction) - [Motivation or Background](#motivation-or-background) - [Detailed Design](#detailed-design) - [Test Design](#test-design) - [Functional Tests](#functional-tests) - [Scenario Tests](#scenario-tests) - [Compatibility Tests](#compatibility-tests) - [Impacts \& Risks](#impacts--risks) - [Unresolved Questions](#unresolved-questions) ## Introduction This document describes the EMEA tenant event capture path in `pkg/service/userbehavior`. The feature copies selected upstream tenant-related events into an independent queue, filters them by tenant attributes equivalent to the EMEA paid external tenant SQL predicate, and persists matching events into the `emea_tenant_events` table in a configured target database. ## Motivation or Background The user behavior service already consumes tenant-related eventbus topics for Lark notifications, but those flows are optimized for subscription dispatch and apply notification-specific filtering. Regional support needs a durable event table for active external EMEA paid tenants, with enough raw event context to support later analysis and business expansion. The motivating tenant predicate is equivalent to: ```sql SELECT * FROM tenants WHERE status = 'active' AND kind = 'external' AND sales_force_region = 'EMEA' AND plan NOT IN ('free_trial', 'new_user') AND country IN (...EMEA allow list plus IL and IN...); ``` The implementation must keep these dimensions configurable, so future regional or plan-based expansion does not require code changes. It must also avoid writing database credentials into source code and avoid coupling the new persistence flow to existing Lark notification subscriptions. Non-goals: - This feature does not replace existing user behavior Lark subscriptions. - This feature does not backfill historical events. - This feature does not guarantee delivery if `EventJob` processing returns an error, because the existing event job framework logs errors but does not requeue failed events. - This feature does not create a general-purpose event warehouse schema. It stores enough normalized fields and the raw JSON payload for regional support use cases. ## Detailed Design ### Event Sources `UserBehaviorService` already subscribes to four external eventbus streams: - `DBaasTenantPlanEvent`, topic `tenant_plan` - `DBaasTenantEvent`, topic `OrgEvent` - `DBaaSClusterEvent`, topic `ClusterEvent` - `DBaaSDisplayEvent`, topic `display_event` Each existing handler now also calls `enqueueTenantEventCapture` near the handler entry: ```go s.enqueueTenantEventCapture(newTenantPlanCaptureEvent(uuid, ev)) s.enqueueTenantEventCapture(newOrgCaptureEvent(uuid, ev)) s.enqueueTenantEventCapture(newClusterCaptureEvent(uuid, ev)) s.enqueueTenantEventCapture(newDisplayCaptureEvent(uuid, ev)) ``` This keeps the capture path independent from the original notification path: ```text eventbus topic -> HandleXXXEvent -> s.ubCh -> UBEventDispatch -> existing subscription/Lark flow -> s.tenantEventCaptureCh -> TenantEventCapture -> tenant filter -> emea_tenant_events ``` The capture path applies configurable event-level filters before enqueueing. The default event filter is intentionally narrow: ```yaml eventFilters: display_event: excludedStatuses: ["started"] ``` Only event sources that appear in `eventFilters` are captured. For each source: - `eventTypes`: optional allow list for that source only - `excludedStatuses`: optional deny list for that source only With these defaults, `cluster_event`, `tenant_plan`, and `org_event` are not persisted by capture unless explicitly enabled in configuration. This avoids duplicate records where a single cluster operation produces both a `display_event` and a `cluster_event`, and it also skips `display_event` rows that only mark operation start. ### Unified Capture Event Different upstream payloads are normalized into `tenantEventCaptureEvent`: ```go type tenantEventCaptureEvent struct { EventUUID string EventSource string EventType string EventStatus string TenantID uint64 ClusterID *uint64 EventTime *time.Time RawEvent any } ``` The event source values are: - `tenant_plan` - `org_event` - `cluster_event` - `display_event` `RawEvent` stores the original upstream event object and is serialized to JSON at persistence time. This preserves fields that are not normalized yet and avoids repeated schema changes for every upstream event variation. ### Queue and Worker `UserBehaviorService` owns a dedicated channel: ```go tenantEventCaptureCh chan *tenantEventCaptureEvent ``` The service registers an event job: ```go service.NewEventJob("TenantEventCapture", s.tenantEventCaptureCh, s.DispatchTenantEventCapture) ``` `NewEventJob` starts with `concurrency=1`, so the new path processes events sequentially by default. This avoids unnecessary pressure on `tenantSvc.GetTenant` and the target TiDB/MySQL database. If throughput becomes insufficient, concurrency can be added later as a configuration field, but that should be paired with downstream capacity checks. ### Early Exclusion The capture path supports `excludedTenantIDs`. `enqueueTenantEventCapture` checks the tenant ID before putting the event into the channel: ```go if slices.Contains(s.conf.TenantEventCapture.ExcludedTenantIDs, ev.TenantID) { return } ``` This avoids queue growth and tenant-service lookups for known noisy tenants. The initial excluded IDs are expected to be configured as: ```yaml excludedTenantIDs: - 1372813089209300444 - 1372813089209260919 - 1372813089209245725 - 1372813089209302377 - 1372813089209301246 ``` ### Tenant Filter `DispatchTenantEventCapture` resolves the current tenant snapshot using: ```go tenant, err := s.tenantSvc.GetTenant(ctx, ev.TenantID) ``` Then `matchTenantEventCaptureTenant` always requires: - `tenant.status == "active"` - `tenant.kind == "external"` It also applies configurable filters: - `regions` - `excludedPlans` - `countries` `regions` uses `tenant.sales_force_region` first. If `sales_force_region` is empty, it falls back to the tenant's country-derived region field. `excludedPlans` is opt-in: if it is not configured, plan is not checked. `countries` uses a default EMEA country allow list when it is not configured: ```text GB, DE, FR, IT, NL, BE, CH, AT, IE, LU, MC, AD, SE, DK, FI, NO, IS, AX, FO, ES, PT, GR, MT, CY, GI, VA, SM, PL, CZ, HU, RO, SK, SI, HR, BG, EE, LV, LT, RS, BA, ME, MK, AL, TR, IL, IN ``` The comparison is case-insensitive for string lists and country values. ### Configuration The feature is off by default: ```go type TenantEventCaptureConfig struct { Enabled bool `yaml:"enabled" default:"false"` AutoMigrate bool `yaml:"autoMigrate" default:"false"` TargetDB *DatabaseConfig `yaml:"targetDB"` Regions []string `yaml:"regions"` ExcludedPlans []string `yaml:"excludedPlans"` ExcludedTenantIDs []uint64 `yaml:"excludedTenantIDs"` Countries []string `yaml:"countries"` EventFilters map[string]TenantEventCaptureEventFilter `yaml:"eventFilters"` } type TenantEventCaptureEventFilter struct { EventTypes []string `yaml:"eventTypes"` ExcludedStatuses []string `yaml:"excludedStatuses"` } ``` Example deployment configuration: ```yaml services: userbehavior: tenantEventCapture: enabled: true autoMigrate: false targetDB: host: gateway01.us-west-2.prod.aws.tidbcloud.com port: 4000 user: root password: ${REGIONAL_SUPPORT_DB_PASSWORD} database: regional_support tlsConfigKey: "true" regions: ["EMEA"] excludedPlans: ["free_trial", "new_user"] countries: ["DE", "GB", "AE", "IR", "AD", "AF", "AL", "NL"] eventFilters: display_event: excludedStatuses: ["started"] excludedTenantIDs: - 1372813089209300444 - 1372813089209260919 - 1372813089209245725 - 1372813089209302377 - 1372813089209301246 ``` Credentials must be supplied through the existing config and secret mechanism. Source code and design docs must not contain database passwords or full DSNs. TiDB Cloud target databases require secure transport. If `targetDB.tlsConfigKey` is omitted, the service defaults it to `"true"` before generating the MySQL DSN. This matches the Lambda implementation that connects with a default SSL context. To add `cluster_event` later, add a source-specific filter: ```yaml services: userbehavior: tenantEventCapture: eventFilters: display_event: excludedStatuses: ["started"] cluster_event: eventTypes: ["ClusterCreated"] ``` ### Target Database and Table When `tenantEventCapture.enabled` is true, `NewService` creates a separate GORM connection using the configured `targetDB`. If `targetDB.tlsConfigKey` is empty, the service sets it to `"true"` before calling `gormutil.Connect`. When `autoMigrate` is true, service startup runs: ```go tenantEventCaptureDB.AutoMigrate(&userbehavior.TenantEventCapture{}) ``` The model table name is `emea_tenant_events`: ```go func (TenantEventCapture) TableName() string { return "emea_tenant_events" } ``` The table stores both normalized fields and raw event JSON: | Column | Purpose | | --- | --- | | `event_uuid` | Upstream event UUID | | `event_source` | `tenant_plan`, `org_event`, `cluster_event`, or `display_event` | | `event_type` | Source-specific event type | | `event_status` | Source-specific status when available | | `tenant_id` | Tenant ID from the event and resolved tenant | | `tenant_name` | Tenant snapshot name | | `tenant_status` | Tenant snapshot status | | `tenant_kind` | Tenant snapshot kind | | `tenant_plan` | Tenant snapshot plan | | `sales_force_region` | Tenant snapshot Salesforce region | | `country` | Tenant snapshot country | | `cluster_id` | Cluster ID when available | | `event_time` | Event timestamp when available | | `raw_event` | JSON payload of the original event | `event_source + event_uuid` has a unique index. Writes use `OnConflict DoNothing`, making duplicate event delivery idempotent. ### Error Handling If the target database connection cannot be created while the feature is enabled, service initialization fails. This makes misconfiguration visible at startup. If event processing fails later, the existing event job framework records metrics and logs the error. It does not requeue the event. This is consistent with existing event jobs but is a known limitation for durable capture use cases. ### Cluster Event Noise The default capture filter excludes `cluster_event`, so cluster events do not increase capture queue depth unless explicitly enabled. If `cluster_event` is enabled later, note that the capture hook runs before the original `ServicePlan == "Starter"` notification filter. A high volume of unrelated starter cluster events can then increase capture queue depth and tenant lookup load. Known mitigations: - Configure noisy tenant IDs in `excludedTenantIDs`, which skips before enqueue. - Add a future `skipStarterClusterEvents` config if starter events are confirmed to have no business value for this capture table. - Add configurable event job concurrency only after validating tenant service and target database capacity. ## Test Design ### Functional Tests Unit tests cover: - Status and kind are fixed filters: only active external tenants pass. - Region and excluded plan filters are skipped when the corresponding config list is not set. - The default country allow list is applied when `countries` is not set. - Configured `free_trial` and `new_user` plans are excluded. - Configured regions use `sales_force_region` first and fall back to country region when Salesforce region is empty. - Countries outside a configured allow list are excluded. - Matching tenants persist rows into `emea_tenant_events`. - Non-matching tenants do not persist rows. - `excludedTenantIDs` are skipped before enqueue. - Default event filters keep only non-started `display_event` rows. - Capture source and event type filters can be configured to enable additional sources such as `cluster_event`. The implemented package-level test command is: ```bash GOCACHE=/private/tmp/devops-work-go-build-cache go test ./pkg/service/userbehavior ``` Configuration loader compatibility is covered by: ```bash GOCACHE=/private/tmp/devops-work-go-build-cache go test ./pkg/infra/config/... ``` ### Scenario Tests Recommended scenario checks before enabling in production: - Enable `tenantEventCapture` with a staging or temporary target database and `autoMigrate=true`; verify the `emea_tenant_events` table is created. - Publish or replay sample `display_event` messages for a matching tenant; verify non-started rows appear and `started` rows do not. - If additional sources are configured, publish or replay sample `tenant_plan`, `OrgEvent`, and `ClusterEvent` messages; verify only configured sources and event types appear. - Replay a duplicate event UUID for the same event source; verify no duplicate row is inserted. - Send events for excluded tenant IDs; verify no rows are inserted and tenant service lookup volume does not increase. - Send events for non-EMEA or free trial tenants; verify no rows are inserted. - Run with `autoMigrate=false` after table creation; verify service startup and writes still succeed. ### Compatibility Tests The new flow is additive and disabled by default. Existing user behavior subscriptions and Lark notification dispatch continue to use `s.ubCh` and `DispatchUBEvents`. Compatibility checklist: - Eventbus compatibility: existing subscriptions are unchanged; handlers only add a second enqueue side effect. - Database compatibility: the target table is in a separately configured database, not the main devops DB. - Upgrade compatibility: with `enabled=false`, the new code path is inert. With `enabled=true`, `targetDB` must be configured. - Downgrade compatibility: disabling the config stops new writes; existing table data remains in the target database. - Security compatibility: target DB credentials must be injected through the existing config/secret path and must not be committed. ## Impacts & Risks Impacts: - Provides a durable EMEA tenant event table for regional support analysis. - Adds one extra in-memory queue and one event job in `UserBehaviorService`. - Adds one tenant service lookup for each captured event that is not skipped by event filters or `excludedTenantIDs`. - Adds writes to the configured target database for matching tenants. Risks: - High-volume irrelevant events may slow the capture queue if event source/type/status filters are widened too far, especially if `cluster_event` is enabled. - Downstream tenant service or target database latency directly affects capture throughput. - The existing `EventJob` framework does not retry failed events, so transient target DB errors can lose capture records. - `AutoMigrate` can modify schema at service startup. It is useful for initial table creation, but production should prefer explicit schema management or use `autoMigrate=true` only during controlled rollout. - The filter uses the current tenant snapshot at processing time, not the tenant state at event creation time. If tenant attributes change between event emission and processing, the capture decision follows the latest tenant state. ## Unresolved Questions - Should starter cluster events be skipped through a dedicated `skipStarterClusterEvents` config? - Should `TenantEventCapture` event job concurrency be configurable after measuring queue depth and downstream QPS capacity? - Is at-least-once durable capture required? If yes, failed event processing needs retry or dead-letter handling beyond the current `EventJob` framework. - Should there be a retention policy or partitioning strategy for `emea_tenant_events` as event volume grows? - Should the target table be managed by an explicit migration script instead of GORM `AutoMigrate` in production?