# User Behavior Service - Design Document ## Overview The User Behavior Service is a notification and event monitoring system designed to track and notify stakeholders about significant user activities within the TiDB Cloud platform. The service listens to various platform events (tenant changes, cluster operations, data imports, backups, etc.), filters them based on configurable criteria, and sends formatted notifications to Lark chat groups. ### Key Objectives - **Real-time Monitoring**: Track critical user behaviors and platform activities in real-time - **Intelligent Filtering**: Apply multi-level filtering to reduce noise and focus on relevant events - **Flexible Notifications**: Send customizable notifications to internal and external Lark chats - **Business Intelligence**: Identify potential customers and track product adoption patterns - **Scalability**: Handle high-volume event streams with minimal latency --- ## Architecture Overview ### High-Level Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ Event Sources │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Tenant │ │ Cluster │ │ Display │ │ Cron │ │ │ │ Events │ │ Events │ │ Events │ │ Jobs │ │ │ └─────┬────┘ └─────┬────┘ └─────┬────┘ └─────┬────┘ │ │ │ │ │ │ │ └────────┼─────────────┼──────────────┼──────────────┼────────────┘ │ │ │ │ v v v v ┌─────────────────────────────────────────────────────────────────┐ │ Event Bus Handlers │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ HandleDBaasTenantPlanEvent │ │ │ │ HandleDBaasAITenantEvent │ │ │ │ HandleDBaaSClusterEvent │ │ │ │ HandleDBaaSDisplayEvent │ │ │ └────────────────────┬─────────────────────────────────┘ │ │ │ (creates UBEvent) │ └───────────────────────┼──────────────────────────────────────────┘ │ v ┌──────────────────┐ │ UB Event │ │ Channel │ │ (buffered) │ └────────┬─────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ Event Dispatcher │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ 1. Preprocess Event (enrich metadata) │ │ │ │ 2. Load Subscriptions from DB │ │ │ │ 3. Apply Filters: │ │ │ │ - Tenant type (paid/free) │ │ │ │ - Internal/external tenant │ │ │ │ - Cluster type (dedicated/shared) │ │ │ │ - Region/Tenant selectors │ │ │ │ - Custom text filters (expr language) │ │ │ │ 4. Render Template │ │ │ │ 5. Build Lark Card │ │ │ └────────────────────┬─────────────────────────────────┘ │ └───────────────────────┼──────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ Notification Delivery │ │ ┌─────────────────┐ ┌─────────────────┐ │ │ │ Internal Lark │ │ External Lark │ │ │ │ (Lark SDK) │ │ (Webhook) │ │ │ └─────────────────┘ └─────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ ``` --- ## Core Components ### 1. Service Layer (`user_behavior_service.go`) **Main Service Structure (`svc`)**: ```go type svc struct { *service.JobService conf config.UserBehaviorServiceConfig ubCh chan UBEvent // Event channel (buffer: 100) larkCliSet *larkcli.LarkClient // Internal Lark client repo repo.UserBehaviorRepo // Database repository logger logr.Logger userSvc user.UserServiceAPI tenantSvc tenant.TenantServiceAPI accountCli accountcli.Client billingCli billingcli.Client coresvcCli *coresvccli.CoreSvcCli dataflowCli *dataflowcli.DataflowCli eventBus *eventbus.EventBus externalLarkCli externallarkcli.ExternalLarkCli llm *llm_cli.LLMCli handledTenant map[uint64]struct{} // Deduplication cache } ``` **Key Responsibilities**: - Event subscription and dispatching - Event preprocessing and enrichment - Integration with external services - Subscription management - Cron job scheduling ### 2. Repository Layer (`repo/`) **Repository Interface**: ```go type UserBehaviorRepo interface { // Lark Chat Management LoadChat(ctx context.Context, id uint64) (*entity.LarkChat, error) QueryChats(ctx context.Context, opts ...repoopt.Option) ([]*entity.LarkChat, error) SaveChat(ctx context.Context, chat *entity.LarkChat) error UpsertChatByChatId(ctx context.Context, chat *entity.LarkChat) (*entity.LarkChat, error) // Subscription Management LoadSubscription(ctx context.Context, id uint64) (*ubent.Subscription, error) QuerySubscriptions(ctx context.Context, opts ...repoopt.Option) ([]*ubent.Subscription, error) SaveSubscription(ctx context.Context, subscription *ubent.Subscription) error PatchSubscription(ctx context.Context, subscription *ubent.Subscription) error DeleteSubscription(ctx context.Context, id uint64) error } ``` **Components**: - `driver_chat.go`: Lark chat persistence operations - `driver_sub.go`: Subscription persistence operations - `cvt/`: Entity-to-model conversion layer ### 3. Event System #### Event Interface (`UBEvent`) All events implement the `UBEvent` interface: ```go type UBEvent interface { HandleUBEvent(ctx context.Context) error PreProcess(ctx context.Context) error PostProcess(ctx context.Context) error ProcessTemplate(ctx context.Context, template string) (string, error) DefaultText(ctx context.Context) string GetAvailableParams(ctx context.Context) []string GetUBEventType() ubent.UBEventTp CustomFilter(ctx context.Context, textFilter string) (bool, error) } ``` #### Event Types 1. **Tenant Plan Events** (`ubevent_tenantplan.go`) - Tracks changes to tenant plans (free → POC → on-demand) - Identifies when tenants add credit cards - Monitors POC starts 2. **Startup Events** (`ubevent_startup.go`) - **New Tenant Activation**: Tracks non-individual tenant signups - **First Cluster Creation**: Monitors first Essential Serverless cluster creation - **First Import**: Tracks first successful data import - Includes LLM-powered company triage for filtering educational organizations 3. **Cluster Events** (`ubevent_cluster.go`) - Creation, deletion, pause, resume - Scaling operations with before/after topology - Restoration events - Volume snapshot restoration 4. **Backup Events** (`ubevent_backup.go`) - Manual backups (auto backups are filtered out) - Backup success/failure notifications 5. **Restore Events** (`ubevent_restore.go`) - Cluster restoration from backups 6. **Import Events** (`ubevent_import.go`) - Data import operations - Success/failure tracking 7. **CDC Events** (`ubevent_cdc.go`) - Changefeed operations (create, pause, resume, edit, scale, delete, failed) 8. **DM Job Events** (`ubevent_dmjob.go`) - Data Migration job monitoring - Scanned periodically for specific tenants 9. **Display Events** (`ubevent_display_ev.go`) - UI-level events from display event bus - Maps to other event types 10. **Queue Events** (`ubevent_queue.go`) - Queue-related operations ### 4. Event Processing Flow #### Event Preprocessing (`PreprocessUBEvent`) The preprocessing step enriches events with contextual data: **Steps**: 1. **Fetch tenant information** from TenantService or AccountService 2. **Determine business region** (SalesForce region or country region) 3. **Classify tenant**: - Internal vs. External - Paid vs. Free (POC, on-demand, or free tier) 4. **Fetch cluster details** (for cluster-related events) 5. **Apply event-specific filters**: - Skip non-essential clusters for startup events - Filter personal email domains - Detect educational organizations using LLM - Skip sample data imports - Count clusters for context 6. **Generate extra info**: - Topology changes (before/after) - Cluster count - Company information (via LLM web search) **Special Logic**: - **Email Filtering**: Personal email domains are filtered for free-tier startup events - **LLM Triage**: For new tenant signups, the service uses LLM to: - Search company information based on email domain - Visit company webpages to verify the email domain matches (filters out companies with similar but different domains) - Check webpage content (contact pages, footers) to confirm the exact domain is used - Extract company name, headquarters, vertical, business model, revenue, funding - Filter out educational organizations (schools, universities) - **Deduplication**: Uses `handledTenant` map to prevent duplicate tenant activation events #### Event Dispatching (`DispatchUBEvents`) **Flow**: ``` 1. Preprocess event (enrich metadata) ↓ 2. Query matching subscriptions from database ↓ 3. For each subscription: ↓ 3.1 Apply filters: - Paid tenant filter - External tenant filter - Dedicated cluster filter - Region selectors - Tenant selectors - Custom text filters (expr language) ↓ 3.2 Process template or use default text ↓ 3.3 Build Lark card ↓ 3.4 For each chat in subscription: - Internal: Send via Lark SDK - External: Send via webhook (async goroutine) ``` ### 5. Subscription System #### Subscription Entity ```go type Subscription struct { ID uint64 Name string SubscribeType SubscribeTp EventType UBEventTp // Event type filter Template string // Custom message template Selector SelectorTp // Region/tenant selectors LarkChats []*LarkChatSimplified // Target chats OnlyPaidTenant bool // Filter: paid only OnlyExternalTenant bool // Filter: external only OnlyDedicatedCluster bool // Filter: dedicated only TextFilter string // Custom expr filter Creator string // Creator email DetailLink *string // Link in card } ``` #### Subscription Filtering Subscriptions support multi-level filtering: 1. **Event Type Filter**: Match specific event types (tenant, cluster, backup, etc.) 2. **Tenant Type Filters**: - `OnlyPaidTenant`: Only notify for POC/on-demand tenants - `OnlyExternalTenant`: Skip internal tenants 3. **Cluster Type Filter**: - `OnlyDedicatedCluster`: Skip serverless clusters 4. **Selector Filters**: - `RegionSelectors`: Match specific business regions - `TenantSelectors`: Match specific tenant IDs 5. **Custom Text Filter**: - Use `expr` language to write custom boolean expressions - Access to original event fields - Example: `EventType == "ClusterCreated" && IsDevTier == false` ### 6. Notification System #### Internal Lark Integration Uses official Lark SDK for internal notifications: - Chat management via Lark API - Syncs chats containing "UB" in name - Sends interactive cards - Message format: JSON card with markdown content #### External Lark Integration Custom webhook-based integration for external Lark workspaces: - HTTP POST requests with custom bot - Supports both card and text messages - No authentication required (webhook URL is secret) - Implemented in `externallarkcli/` #### Message Card Format All notifications are sent as Lark interactive cards: - **Header**: Turquoise colored "User Behavior Event" - **Body**: Markdown-formatted message - **Card Link**: Optional detail URL - **Features**: Forward enabled, wide screen mode --- ## Event Flow Diagram ``` ┌─────────────────┐ │ Platform Events │ │ (EventBus) │ └────────┬────────┘ │ v ┌────────────────────────────────────────┐ │ Event Bus Handlers │ │ - HandleDBaasTenantPlanEvent │ │ - HandleDBaasAITenantEvent │ │ - HandleDBaaSClusterEvent │ │ - HandleDBaaSDisplayEvent │ └────────┬───────────────────────────────┘ │ │ (Convert to UBEvent) v ┌────────────────────┐ │ UB Event Channel │ │ (buffered: 100) │ └────────┬───────────┘ │ v ┌───────────────────────────────────────────────┐ │ PreprocessUBEvent │ │ │ │ ┌─────────────────────────────────────────┐ │ │ │ Fetch tenant info (name, alias, region) │ │ │ └─────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────┐ │ │ │ Classify tenant (internal/paid) │ │ │ └─────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────┐ │ │ │ Fetch cluster details │ │ │ └─────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────┐ │ │ │ Apply filtering logic │ │ │ │ - Personal emails │ │ │ │ - Educational orgs (LLM) │ │ │ │ - Sample data imports │ │ │ │ - First cluster check │ │ │ └─────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────┐ │ │ │ Generate extra info │ │ │ │ - Topology changes │ │ │ │ - Cluster counts │ │ │ │ - Company info (LLM) │ │ │ └─────────────────────────────────────────┘ │ └───────────────┬───────────────────────────────┘ │ │ (eventInfo) v ┌───────────────────────────────────────────────┐ │ DispatchUBEvents │ │ │ │ ┌─────────────────────────────────────────┐ │ │ │ Query subscriptions by event type │ │ │ └─────────────────────────────────────────┘ │ │ │ │ │ v │ │ ┌─────────────────────────────────────────┐ │ │ │ For each subscription: │ │ │ │ │ │ │ │ ┌────────────────────────────────────┐│ │ │ │ │ Apply filters ││ │ │ │ │ - Paid tenant filter ││ │ │ │ │ - External tenant filter ││ │ │ │ │ - Dedicated cluster filter ││ │ │ │ │ - Region/tenant selectors ││ │ │ │ │ - Custom text filter (expr) ││ │ │ │ └────────────────────────────────────┘│ │ │ │ │ │ │ │ │ v │ │ │ │ ┌────────────────────────────────────┐│ │ │ │ │ Process template or default text ││ │ │ │ └────────────────────────────────────┘│ │ │ │ │ │ │ │ │ v │ │ │ │ ┌────────────────────────────────────┐│ │ │ │ │ Build Lark card ││ │ │ │ └────────────────────────────────────┘│ │ │ │ │ │ │ │ │ v │ │ │ │ ┌────────────────────────────────────┐│ │ │ │ │ Send to each chat ││ │ │ │ │ - Internal: Lark SDK ││ │ │ │ │ - External: Webhook (async) ││ │ │ │ └────────────────────────────────────┘│ │ │ └─────────────────────────────────────────┘ │ └───────────────────────────────────────────────┘ ``` --- ## Scheduled Jobs (Cron) ### 1. SyncLarkChats **Schedule**: Configurable via `SyncLarkChatsInterval` **Purpose**: Sync internal Lark chats from the global Lark workspace **Logic**: - List all chats from Lark API - Filter chats with "UB" in the name - Upsert to database (by ChatID) - Store: name, ChatID, avatar URL, type (global), external flag ### 2. ScanDMJobEvents **Schedule**: Same as `SyncLarkChatsInterval` **Purpose**: Poll for Data Migration job events **Logic**: - Query subscriptions for DM job events - For each subscription with tenant selectors: - List clusters for the tenant - List DM jobs for each cluster - Generate events for jobs created in last 15 minutes - Send to UB event channel **Rationale**: DM jobs don't emit real-time events, so periodic scanning is required --- ## External Dependencies ### Service Clients 1. **UserService**: User management operations 2. **TenantService**: Tenant metadata and classification 3. **AccountService**: Organization/tenant details from account service 4. **BillingService**: Billing and plan information 5. **CoreSvcCli**: Cluster management operations (list, get, count) 6. **DataflowCli**: Data migration and changefeed operations 7. **LarkClient**: Internal Lark workspace integration 8. **LLMClient**: AI-powered company triage and web search ### Event Bus Subscriptions 1. **DBaasTenantPlanEvent**: Tenant plan changes 2. **DBaasTenantEvent**: Tenant lifecycle events (activation) 3. **DBaaSClusterEvent**: Cluster lifecycle and operations 4. **DBaaSDisplayEvent**: UI-level display events --- ## Data Model ### Entities #### LarkChat ```go type LarkChat struct { ID uint64 Name *string LarkTp LarkType // Global, US, etc. IsExternal bool // Internal vs External Lark ChatID *string // For internal Lark ChatURL *string // For external webhooks AvatarURL *string } ``` #### Subscription ```go type Subscription struct { ID uint64 Name string SubscribeType SubscribeTp EventType UBEventTp Template string TextFilter string Selector SelectorTp LarkChats []*LarkChatSimplified OnlyPaidTenant bool OnlyExternalTenant bool OnlyDedicatedCluster bool Creator string DetailLink *string } ``` #### SelectorTp ```go type SelectorTp struct { RegionSelectors []RegionSelectorTp // e.g., ["us-east-1", "eu-west-1"] TenantSelectors []TenantSelectorTp // e.g., ["123", "456"] } ``` --- ## Key Features ### 1. Intelligent Event Filtering **Multi-Level Filtering**: - Pre-processing filters (before DB query) - Subscription-level filters (during dispatch) - Custom expression filters (expr language) **Filter Examples**: ```go // Only first Essential Serverless cluster if resp.GetCount() != 1 { ret.ForceSkip = true } // Skip personal email domains if isPersonalEmail(emailDomain) { ret.ForceSkip = true } // Skip educational organizations (LLM-based) if isEducationalTriage(llmResult) { ret.ForceSkip = true } // Skip sample data imports if strings.Contains(detail, "s3://tidbcloud-sample") { ret.ForceSkip = true } ``` ### 2. LLM-Powered Triage For new tenant signups, the service uses LLM to: - Extract email domain - Search the web for company information - **Visit company webpages to verify email domain**: When multiple companies have similar domain names, the LLM checks webpage content (contact pages, about pages, footers) to confirm which company actually uses the exact email domain - Discard results where webpage content shows a different email domain - Return structured fields: - Company Name - Headquarters - Vertical - Business Model - Annual Revenue (approximate) - Funding Series - Filter out educational organizations based on keywords **Educational Keyword Detection**: - "school district", "public schools", "university", "college" - "k-12", "high school", "middle school", "elementary school" - Checks company name, vertical, and full triage text ### 3. Topology Change Visualization For cluster scaling events, the service generates before/after topology text: ``` TiDB(DefaultGroup): 2 x (8C16G) -> 4 x (8C16G) TiKV: 3 x (8C32G,500GiB,EBS_GP3) -> 3 x (16C64G,1000GiB,EBS_GP3) Extra Group(analytics): 1 x TiDB Nodes, 0 x TiProxy Nodes -> 2 x TiDB Nodes, 1 x TiProxy Nodes ``` ### 4. Deduplication - Prevents duplicate tenant activation events using in-memory map - Note: Deduplication state is lost on service restart ### 5. Flexible Templating - Subscriptions can define custom message templates - Falls back to default text if template is empty - Template processing is event-type specific ### 6. Mock Event Testing - `SendMockEventToSubscription` API allows testing subscriptions - Sends a mock message to all chats in the subscription - Useful for validating chat configuration --- ## API Surface ### Public API (`UserBehaviorServiceAPI`) ```go type UserBehaviorServiceAPI interface { // Subscription Management ListSubscriptions(ctx, ListSubscriptionParam) ([]*Subscription, int64, error) GetSubscription(ctx, id uint64) (*Subscription, error) CreateSubscription(ctx, CreateSubscriptionParam) (*uint64, error) DeleteSubscription(ctx, id uint64) error EditSubscription(ctx, id uint64, EditSubscriptionParam) error // Service Control ChangeSubscriptionAvailability(ctx, available bool) error // Testing SendMockEventToSubscription(ctx, id uint64) error // Chat Management GetLarkChatOptions(ctx) ([]*LarkChatSimplified, error) AddExternalLarkChat(ctx, AddExternalLarkChatParam) error } ``` --- ## Configuration ```go type UserBehaviorServiceConfig struct { SyncLarkChatsInterval *time.Duration // Interval for chat sync and DM job scan // Other service-specific configs } ``` --- ## Performance Considerations ### Async Notification Delivery - Notifications are sent in separate goroutines - Prevents blocking the main dispatch loop - Each chat receives notifications independently ### Event Channel Buffering - UB event channel has buffer size of 100 - Prevents event loss during processing spikes - May need tuning based on event volume ### Database Query Optimization - Subscriptions are queried by event type - Reduces unnecessary subscription evaluation - Indexed lookups for chat loading ### LLM Rate Limiting - LLM triage is called only for new tenant signups - Filtered by personal email first to reduce calls - Consider implementing caching for repeated domains --- ## Error Handling ### Event Processing Errors - Errors in preprocessing return early, preventing dispatch - Errors logged but don't crash the service - Individual notification failures don't affect other chats ### External Service Failures - Graceful handling of service client errors - Fallback between TenantService and AccountService - Logged errors include context (tenant ID, cluster ID, etc.) ### Database Errors - Repository errors bubble up to callers - Subscription query failures prevent event dispatch - Chat loading failures skip that specific chat --- ## Security Considerations ### Email Domain Filtering - Large personal email domain list (~3700+ domains) - Prevents spam from individual/test accounts - O(1) lookup using pre-computed set ### Internal vs External Filtering - `OnlyExternalTenant` filter prevents internal notifications - Tenant kind classification from multiple sources - SalesForce kind takes precedence ### Webhook Security - External Lark webhooks are secrets - No authentication required (webhook URL is the secret) - HTTPS transport enforced --- ## Future Improvements ### 1. Enhanced Deduplication **Current Issue**: In-memory deduplication state is lost on restart **Proposed Solution**: - Persist deduplication state to Redis or database - Use time-based expiry (e.g., 24 hours) - Consider event UUID-based deduplication ### 2. Template Language **Current Limitation**: Template processing is minimal **Proposed Enhancement**: - Support for variable substitution (e.g., `{{tenant_name}}`) - Conditional rendering (e.g., `{{if .IsPaid}}POC tenant{{end}}`) - Use Go templates or similar templating engine ### 3. Metrics and Monitoring **Current Gap**: No metrics exposed **Proposed Metrics**: - Event processing rate by type - Notification success/failure rate - Subscription evaluation count - LLM triage latency - Event channel depth ### 4. Retry Mechanism **Current Behavior**: Failed notifications are not retried **Proposed Enhancement**: - Dead letter queue for failed notifications - Exponential backoff retry - Max retry limit ### 5. Advanced Filtering **Proposed Enhancements**: - Time-based filters (e.g., business hours only) - Rate limiting per tenant (prevent notification spam) - Aggregation (batch multiple events into one notification) - Severity levels for different event types ### 6. LLM Caching **Current Issue**: Each new tenant signup triggers LLM call **Proposed Solution**: - Cache LLM triage results by email domain - TTL-based cache expiry (e.g., 30 days) - Shared cache across service instances ### 7. Subscription Versioning **Current Limitation**: Editing subscriptions affects historical behavior **Proposed Enhancement**: - Version subscriptions on edit - Keep historical subscription configurations - Audit trail for subscription changes ### 8. Event Replay **Current Gap**: No way to replay historical events **Proposed Feature**: - Store processed events in database - API to replay events to subscriptions - Useful for testing new subscriptions ### 9. Notification History **Current Gap**: No record of sent notifications **Proposed Feature**: - Store notification delivery records - Track delivery status (sent, failed, retried) - API to query notification history ### 10. Multi-Channel Support **Current Limitation**: Only Lark notifications supported **Proposed Enhancement**: - Email notifications - Slack integration - Webhook to arbitrary endpoints - SMS notifications for critical events --- ## Testing Strategy ### Unit Testing **Coverage Areas**: - Event preprocessing logic - Filter evaluation - Template rendering - Topology text generation - Email domain filtering - Educational organization detection ### Integration Testing **Coverage Areas**: - Database operations (repository layer) - External service integration (mocked) - Event bus subscription - Lark API integration ### End-to-End Testing **Test Scenarios**: - New tenant signup → LLM triage → notification - Cluster creation → subscription match → Lark card delivery - Subscription filtering (various filter combinations) - Mock event testing API --- ## Operational Runbook ### Common Operations #### Adding a New Event Type 1. Create event struct implementing `UBEvent` interface 2. Add event type to `ubent.UBEventTp` enum 3. Implement event generator function (e.g., `newXxxUBEvent`) 4. Add event handler to relevant event bus subscription 5. Implement preprocessing logic in `PreprocessUBEvent` 6. Update documentation #### Adding a New Lark Chat **Internal Lark**: - Ensure chat name contains "UB" - Wait for next `SyncLarkChats` cron run - Or trigger manual sync **External Lark**: - Use `AddExternalLarkChat` API - Provide webhook URL and chat name #### Debugging Missing Notifications 1. Check event preprocessing (logs at INFO level) 2. Verify subscription filters match event 3. Check subscription event type matches 4. Verify chat is properly configured 5. Check Lark API rate limits #### Debugging Duplicate Notifications 1. Check `handledTenant` map (in-memory) 2. Verify event deduplication logic 3. Check for multiple matching subscriptions 4. Review event source (upstream may emit duplicates) --- ## Monitoring and Alerting ### Key Metrics (Proposed) - `ub_events_received_total{event_type}`: Total events received - `ub_events_processed_total{event_type}`: Total events processed - `ub_events_filtered_total{event_type, reason}`: Events filtered out - `ub_subscriptions_matched_total{event_type}`: Subscriptions matched - `ub_notifications_sent_total{chat_type, status}`: Notifications sent - `ub_llm_calls_total{status}`: LLM triage calls - `ub_event_channel_depth`: Current event channel buffer depth ### Alerts (Proposed) - Event channel near capacity (>80 events) - High notification failure rate (>5%) - LLM triage latency spike (>30s) - No events processed in last hour (service health check) --- ## Dependencies Graph ``` User Behavior Service ├── Infrastructure │ ├── EventBus (subscriptions to 4 event types) │ ├── Database (via db.TransProvider) │ └── Logger (logr.Logger) ├── Internal Services │ ├── UserService (user management) │ ├── TenantService (tenant metadata) │ └── OperationService (background jobs) ├── External Service Clients │ ├── AccountService (org/tenant details) │ ├── BillingService (plan information) │ ├── CoreService (cluster operations) │ ├── DataflowService (DM jobs) │ └── LLM Service (company triage) └── Notification Channels ├── Internal Lark (official SDK) └── External Lark (webhook) ``` --- ## Glossary - **UB Event**: User Behavior Event - an internal event abstraction - **Subscription**: A rule that maps events to notification channels - **LarkChat**: A Lark chat group (internal or external webhook) - **Selector**: Region or tenant-based filter in subscription - **Event Preprocessing**: Enriching events with contextual metadata - **Triage**: LLM-based company information extraction - **Display Event**: UI-level event from eventbus-datastruct - **Essential Serverless**: Free-tier serverless cluster type - **POC**: Proof of Concept (paid trial plan) - **On-Demand**: Pay-as-you-go plan --- ## References ### Related Code Locations - Event entities: `entity/userbehavior/` - Event bus: `pkg/infra/eventbus/` - Lark client: `pkg/infra/clients/larkcli/` - Account client: `pkg/infra/clients/accountcli/` - Core service client: `pkg/infra/clients/coresvccli/` - Display events: `github.com/tidbcloud/eventbus-datastruct/event` - Cluster events: `github.com/tidbcloud/tidb-management-service/utilities/clusterevent` ### External Documentation - Lark Open API: https://open.larksuite.com/ - Expr Language: https://expr-lang.org/ --- ## Changelog | Version | Date | Author | Changes | |---------|------|--------|---------| | 1.0 | 2026-03-04 | Claude Code | Initial design document | | 1.1 | 2026-03-06 | Claude Code | Enhanced LLM triage with webpage content verification for accurate email domain matching | --- ## Appendix A: Event Type Matrix | Event Type | Source Event | Trigger | Key Filters | Typical Use Case | |------------|--------------|---------|-------------|------------------| | TenantPlan | TenantPlanEvent | Plan change | Paid tenant | Monitor POC starts, credit card additions | | Startup | TenantEvent | New activation | Personal email, educational org | Identify potential customers | | Startup | DisplayEvent (CreateCluster) | First essential cluster | Internal tenant, cluster count | Track first cluster creation | | Startup | DisplayEvent (CreateImport) | First import | Personal email + free tier, sample data | Identify serious users | | Cluster | ClusterEvent | Lifecycle ops | Internal tenant, shared cluster | Monitor cluster operations | | Backup | DisplayEvent | Backup ops | Auto backups | Track manual backups | | Restore | DisplayEvent | Restore ops | - | Monitor restore operations | | Import | DisplayEvent | Import ops | - | Track data imports | | CDC | DisplayEvent | Changefeed ops | - | Monitor CDC operations | | DMJob | Cron scan | Job creation | Tenant-specific | Track DM jobs | --- ## Appendix B: Filter Evaluation Order 1. **Event Preprocessing** (before DB query): - Event type acceptance (e.g., only specific cluster events) - Deduplication check (for tenant activation) - First cluster check - Personal email filter - Educational organization filter (LLM) - Sample data filter - ForceSkip flag evaluation 2. **Subscription Matching** (DB query): - Event type exact match 3. **Subscription Filtering** (during dispatch): - Paid tenant filter (`OnlyPaidTenant`) - External tenant filter (`OnlyExternalTenant`) - Dedicated cluster filter (`OnlyDedicatedCluster`) - Region selector match - Tenant selector match - Custom text filter (expr) 4. **Template Processing**: - Custom template OR default text 5. **Notification Delivery**: - Per-chat delivery (async) --- ## Appendix C: Sample Notification Messages ### Tenant Plan Change (POC Start) ``` [** us-east-1 **] Tenant **Acme Corp**(12345, alias: acme) starts the POC. ``` ### New Tenant Activation ``` [** eu-west-1 **] New non-individual tenant [**TechStartup**](https://ops.tidbcloud.com/orgs/67890)(id: 67890). Email domain is techstartup.io - Company Name: TechStartup Inc - Headquarter: San Francisco, CA - Vertical: SaaS - Business Model: B2B - Approximate Annual Revenue: $5M-$10M - Funding Series: Series A (Please notice that these info are search from the internet and validate as needed. AI confidence level: 85%) ``` ### First Cluster Creation ``` [** us-west-2 **] First essential instance of tenant [**DataCo**](https://ops.tidbcloud.com/orgs/11111) (id: 11111, email domain: dataco.com) is being created. Cluster ID is [**22222**](https://ops.tidbcloud.com/orgs/11111/clusters/22222). ``` ### Cluster Scaling ``` [** ap-southeast-1 **] Tenant **BigCorp**(33333, alias: bigcorp) scaled a Dedicated cluster **prod-db**(44444). TiDB(DefaultGroup): 2 x (8C16G) -> 4 x (8C16G) TiKV: 3 x (8C32G,500GiB,EBS_GP3) -> 3 x (16C64G,1000GiB,EBS_GP3) The tenant now has 5 active dedicated cluster(s). ``` --- *End of Design Document*