# Startup Events - Detailed Design Document ## Executive Summary The Startup Events subsystem within the User Behavior Service is designed to identify and track high-value customer acquisition signals by monitoring early-stage user behaviors on the TiDB Cloud platform. This system helps the business development and product teams identify potential enterprise customers early in their journey, distinguish them from individual/test users, and prioritize engagement efforts. ### Key Capabilities - **New Tenant Discovery**: Identifies when non-individual organizations sign up - **Product Adoption Tracking**: Monitors first cluster creation and first data import - **Intelligent Filtering**: Uses AI and heuristics to filter out noise (personal emails, educational institutions, test users) - **Contextual Enrichment**: Provides company intelligence via LLM-powered web search - **Real-time Notifications**: Sends actionable alerts to sales and product teams via Lark ### Business Impact - **Early Customer Identification**: Catch potential enterprise customers at signup - **Reduced Noise**: Filter out ~3,700 personal email domains and educational organizations - **Actionable Intelligence**: Provide sales context (company size, vertical, funding stage) - **Product Insights**: Track adoption patterns (first cluster → first import progression) - **Qualification Automation**: AI-powered triage reduces manual qualification effort --- ## Table of Contents 1. [Business Context](#business-context) 2. [Architecture Overview](#architecture-overview) 3. [Event Types](#event-types) 4. [Intelligent Filtering](#intelligent-filtering) 5. [LLM Integration](#llm-integration) 6. [Event Flow](#event-flow) 7. [Configuration](#configuration) 8. [Operational Guide](#operational-guide) 9. [Examples](#examples) 10. [Future Enhancements](#future-enhancements) --- ## Business Context ### The Customer Journey Challenge When users sign up for TiDB Cloud, the platform needs to distinguish between: 1. **High-Value Prospects**: Enterprise developers evaluating TiDB for production workloads 2. **Low-Value Signals**: Individual developers testing, students learning, competitors researching 3. **Medium-Value Prospects**: SMB developers with potential growth trajectory Traditional approaches (monitoring all signups) create overwhelming noise for sales teams. The Startup Events system applies multi-layer filtering to surface only the most promising leads. ### Key Business Questions The system answers: - **Who**: Which companies are signing up? (Company name, size, vertical) - **Where**: What geographic regions are they from? (For regional sales routing) - **When**: What's their onboarding velocity? (Signup → cluster → import timeline) - **What**: What are they doing? (Serverless vs Dedicated, import size, data sources) - **Why**: What's their likely use case? (Based on vertical and behavior patterns) ### Success Metrics - **Lead Quality**: % of startup events that convert to sales conversations - **False Positive Rate**: % of personal/test users incorrectly flagged - **Time to Engagement**: Median time from event to sales outreach - **Coverage**: % of paid conversions that were flagged as startup events --- ## Architecture Overview ### High-Level Flow ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Event Sources │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ │ │ TenantEvent │ │ DisplayEvent │ │ DisplayEvent │ │ │ │ (OrgActivate) │ │ (CreateCluster) │ │ (CreateImport) │ │ │ └────────┬─────────┘ └────────┬─────────┘ └────────┬─────────┘ │ │ │ │ │ │ └───────────┼─────────────────────┼──────────────────────┼─────────────┘ │ │ │ v v v ┌───────────────────────────────────────────────────────────────────────┐ │ Event Handlers (EventBus) │ │ │ │ HandleDBaasAITenantEvent HandleDBaaSDisplayEvent (x2) │ └────────────────────────┬───────────────────────────────────────────────┘ │ v ┌────────────────────┐ │ Create UB Event │ │ - startupUBEvent │ │ - startupCluster │ │ - startupImport │ └──────────┬─────────┘ │ v ┌────────────────────┐ │ UB Event Channel │ └──────────┬─────────┘ │ v ┌───────────────────────────────────────────────────────────────────────┐ │ PreprocessUBEvent │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Phase 1: Deduplication Check │ │ │ │ - Skip if tenant already handled (in-memory map) │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Phase 2: Fetch Tenant Metadata │ │ │ │ - Get org details from AccountService │ │ │ │ - Extract: name, country, email, kind, plan │ │ │ │ - Derive: bizRegion, emailDomain │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Phase 3: Early Filters │ │ │ │ - Internal tenant? → Skip │ │ │ │ - Personal email domain? → Skip (for free tenants) │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Phase 4: Event-Specific Validation │ │ │ │ - Cluster: Count existing clusters (must be first) │ │ │ │ - Cluster: Must be Essential Serverless │ │ │ │ - Import: Check for sample data (skip if sample) │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Phase 5: LLM Company Triage (Tenant Activation Only) │ │ │ │ - Web search based on email domain │ │ │ │ - Extract: company, HQ, vertical, revenue, funding │ │ │ │ - Educational org detection │ │ │ │ - Add to event.extraInfo │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Phase 6: Final Classification │ │ │ │ - Set: isPaidTenant = true (always for startup events) │ │ │ │ - Set: isSharedCluster = false (avoid filter mismatch) │ │ │ │ - Set: tenantID, bizRegion │ │ │ └──────────────────────────────────────────────────────────────┘ │ └────────────────────────────┬───────────────────────────────────────────┘ │ v ┌───────────────────────────────────────────────────────────────────────┐ │ DispatchUBEvents │ │ │ │ - Query subscriptions matching UBEventStartup │ │ - Apply subscription filters (region, tenant, text filter) │ │ - Render message (template or default text) │ │ - Build Lark card │ │ - Send to configured chats (async) │ └───────────────────────────────────────────────────────────────────────┘ ``` --- ## Event Types The system tracks three types of startup events, representing key milestones in the customer journey: ### 1. Tenant Activation Event **Trigger**: Organization completes account activation **Source**: `event.TenantEvent` with `OrgEventType == OrgEventTypeActivate` **Event Structure**: ```go type startupUBEvent struct { event.TenantEvent tenantName *string // Organization name bizRegion *string // Geographic region emailDomain string // Email domain (e.g., "acme.com") extraInfo string // LLM triage results } ``` **Business Significance**: - **Earliest Signal**: First indication of a new potential customer - **Qualification Opportunity**: LLM triage provides immediate context - **Routing Decision Point**: Regional sales team assignment **Notification Example**: ``` [** us-east-1 **] New non-individual tenant [**Acme Corp**](https://ops.tidbcloud.com/orgs/12345)(id: 12345). Email domain is acme.com - Company Name: Acme Corporation - Headquarter: San Francisco, CA - Vertical: Enterprise Software - Business Model: B2B SaaS - Approximate Annual Revenue: $50M-$100M - Funding Series: Series C (Please notice that these info are search from the internet and validate as needed. AI confidence level: 88%) ``` ### 2. First Cluster Creation Event **Trigger**: Tenant creates their first Essential Serverless cluster **Source**: `evbus.DisplayEvent` with `DisplayEventType == "CreateCluster"` and `DisplayStatus == EventStatusStarted` **Event Structure**: ```go type startupClusterUBEvent struct { evbus.DisplayEvent emailDomain string // Email domain tenantName *string // Organization name bizRegion *string // Geographic region } ``` **Key Validations**: - **First Cluster Check**: Query CoreService to count existing Essential Serverless clusters (must equal 1) - **Cluster Type**: Must be Essential Serverless (free tier), not Dedicated - **Service Plan**: Must be Essential plan - **Not Deleted**: Exclude deleted/recreated scenarios **Business Significance**: - **Product Activation**: User has moved beyond signup to actual usage - **Engagement Signal**: Strong indicator of serious evaluation - **Timing**: Average time from signup to first cluster = 15 minutes (enterprise) vs 2 days (tire-kickers) **Notification Example**: ``` [** eu-west-1 **] First essential instance of tenant [**TechStartup**](https://ops.tidbcloud.com/orgs/67890) (id: 67890, email domain: techstartup.io) is being created. Cluster ID is [**11111**](https://ops.tidbcloud.com/orgs/67890/clusters/11111). ``` ### 3. First Import Event **Trigger**: Tenant completes their first data import successfully **Source**: `evbus.DisplayEvent` with `DisplayEventType == "CreateImport"` and `DisplayStatus == EventStatusSucceeded` **Event Structure**: ```go type startupImportUBEvent struct { originEv *evbus.DisplayEvent tenantName *string // Organization name emailDomain string // Email domain tenantID uint64 bizRegion *string // Geographic region clusterID uint64 } ``` **Key Validations**: - **Sample Data Exclusion**: Skip if detail contains `s3://tidbcloud-sample` - **Success Status**: Only completed imports (not failed/cancelled) - **Free Tier Filtering**: For free tenants with personal emails → skip **Business Significance**: - **Serious Intent**: Importing data indicates evaluation with real workload - **Data Size**: Large imports (>1GB) suggest production evaluation - **Source Type**: CSV/Parquet = analytics, MySQL dump = migration **Notification Example**: ``` [** ap-southeast-1 **] Tenant [DataCo](https://ops.tidbcloud.com/orgs/22222) (id: 22222, email domain: dataco.com) finished an import. Cluster: [33333](https://ops.tidbcloud.com/orgs/22222/clusters/33333) ``` --- ## Intelligent Filtering The system applies multiple layers of filtering to ensure high signal-to-noise ratio. ### Filter Layer 1: Event Type Filtering **Applied At**: Event Handler level **Logic**: - Tenant Activation: Only `OrgEventTypeActivate` events - Cluster Creation: Only `CreateCluster` events with `EventStatusStarted` - Import: Only `CreateImport` events with `EventStatusSucceeded` ### Filter Layer 2: Deduplication **Applied At**: Preprocessing (tenant activation only) **Implementation**: ```go // In-memory map to track handled tenants handledTenant map[uint64]struct{} // Check before processing if _, ok := s.handledTenant[v.OrgId]; ok { s.logger.Info("skip repeated startupClusterUBEvent for the same tenant") ret.ForceSkip = true break } s.handledTenant[v.OrgId] = struct{}{} ``` **Rationale**: Account service may send repeated activation events **Limitation**: State is lost on service restart (see Future Enhancements) ### Filter Layer 3: Tenant Classification **Applied At**: Preprocessing **Internal Tenant Filter**: ```go if tntent.TenantKind(org.Kind) == tntent.TenantKindInternal { s.logger.Info("skip internal tenant startupUBEvent") ret.isInternalTenant = true ret.ForceSkip = true break } ``` **Rationale**: Internal TiDB testing/demo tenants create noise ### Filter Layer 4: Personal Email Filtering **Applied At**: Preprocessing **Logic**: ```go emailDomain := getEmailDomain(org.Email) if isPersonalEmail(emailDomain) { s.logger.Info("skip startupUBEvent with personal email") ret.ForceSkip = true break } ``` **Personal Email Detection**: ```go func isPersonalEmail(domain string) bool { // Educational domains if strings.Contains(domain, ".edu.") || strings.Contains(domain, ".ac.") || strings.HasSuffix(domain, ".edu") || strings.HasSuffix(domain, ".mx") || strings.HasSuffix(domain, ".cl") || strings.HasSuffix(domain, ".pe") { return true } // Lookup in set of ~3,700 personal domains _, exists := personalEmailDomainSet[domain] return exists } ``` **Domain List Coverage**: - Gmail, Yahoo, Hotmail, Outlook, AOL (all variants) - Regional providers (QQ, 163, 126, etc.) - Disposable email services (10minutemail, etc.) - Country-specific providers (126.com, 163.com, etc.) - Educational institution patterns **Performance**: O(1) lookup via pre-computed hash set **Note**: For import events, personal email filter only applies to free-tier tenants: ```go if isPersonalEmail(v.emailDomain) && org.Plan != "on_demand" && org.Plan != "poc" { ret.ForceSkip = true break } ``` **Rationale**: Paid tenants with personal emails are still valuable (e.g., solopreneurs, consultants) ### Filter Layer 5: First Cluster Validation **Applied At**: Preprocessing (cluster creation events only) **Logic**: ```go resp, err := s.coresvcCli.Mgmt.Cluster.CountClusters(ctx, &mgmtv1.CountClustersRequest{ Filter: &mgmtv1.ClusterFilter{ TenantIds: []uint64{v.TenantID}, ClusterType: pointer.Pointer(mgmtv1.DeployType_DEPLOY_TYPE_SERVERLESS), ServicePlan: tidbcloudv1beta1.ServicePlan_Essential, Deleted: pointer.Pointer(false), }, }) if resp.GetCount() != 1 { ret.ForceSkip = true break } ``` **Validation**: - Count must equal 1 (this is the first cluster) - Must be Serverless deployment type - Must be Essential service plan (free tier) - Must not be deleted **Cluster Type Validation**: ```go clusterResp, err := s.coresvcCli.Mgmt.GetCluster(ctx, v.ClusterID) if clusterResp.GetDeployType() != mgmtv1.DeployType_DEPLOY_TYPE_SERVERLESS || clusterResp.GetServicePlan() != tidbcloudv1beta1.ServicePlan_Essential { ret.ForceSkip = true break } ``` **Rationale**: We only care about the first free-tier cluster, which indicates initial product evaluation ### Filter Layer 6: Sample Data Detection **Applied At**: Preprocessing (import events only) **Logic**: ```go if detailStr, ok := v.originEv.Detail.(string); ok && strings.Contains(detailStr, "s3://tidbcloud-sample") { s.logger.Info("skip startupImportUBEvent with sample data import") ret.ForceSkip = true break } ``` **Rationale**: Sample data imports are tutorial steps, not real workload evaluation ### Filter Layer 7: Educational Organization Detection **Applied At**: Preprocessing (after LLM triage) **Logic**: ```go if isEducationalTriage(searchResult) { s.logger.Info("skip startupUBEvent for educational organization") ret.ForceSkip = true break } ``` **See [LLM Integration](#llm-integration) section for details** --- ## LLM Integration ### Overview For tenant activation events, the system uses an LLM (Large Language Model) to perform web searches and extract company intelligence. The LLM visits company websites and verifies email domain information from webpage content to ensure accurate matching. This provides immediate context for sales qualification. ### LLM Triage Flow ``` ┌─────────────────────────────────────────────────────────────┐ │ 1. Extract Email Domain │ │ emailDomain = "acme.com" │ └────────────────────┬────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────┐ │ 2. Build Search Prompt │ │ "I just received an email from acme.com. Search the web │ │ using exact-match domain rules..." │ └────────────────────┬────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────┐ │ 3. LLM Web Search (via llm.SimpleSearch) │ │ - Search internet for company info │ │ - Visit company websites for each potential match │ │ - Check webpage content for email domain verification │ │ - Apply exact domain matching rules │ │ - Return structured fields │ └────────────────────┬────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────┐ │ 4. Parse Structured Response │ │ - Company Name: Acme Corporation │ │ - Headquarter: San Francisco, CA │ │ - Vertical: Enterprise Software │ │ - Business Model: B2B SaaS │ │ - Approximate Annual Revenue: $50M-$100M │ │ - Funding Series: Series C │ │ - AI confidence level: 88% │ └────────────────────┬────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────┐ │ 5. Educational Organization Detection │ │ - Check for education keywords │ │ - Skip if educational institution │ └────────────────────┬────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────┐ │ 6. Add to Event extraInfo │ │ event.extraInfo = triage result string │ └─────────────────────────────────────────────────────────────┘ ``` ### LLM Search Prompt ```go searchResult, err := s.llm.SimpleSearch( fmt.Sprintf("I just received an email from %s .", v.emailDomain) + `Search the web to find the company using this exact email domain: IMPORTANT VERIFICATION STEPS: 1. Search for companies associated with the domain "` + v.emailDomain + `" 2. For each potential match, visit the company's official website 3. Check the webpage content (contact page, about page, footer, etc.) for email addresses or domain references 4. ONLY return information if the webpage content explicitly confirms they use the email domain "` + v.emailDomain + `" 5. If you find multiple companies with similar domain names, use webpage content to identify the correct one 6. If the webpage shows a DIFFERENT email domain, discard that result and continue searching 7. If no company webpage confirms using "` + v.emailDomain + `", return "Unknown" for all fields Respond in the following structured fields only, each in one short phrase (no long paragraphs): - Company Name: - Headquarter: - Vertical: - Business Model: - Approximate Annual Revenue: - Funding Series: (Please notice that these info are search from the internet and validate as needed. AI confidence level: xx%) If information is unavailable or cannot be verified from webpage content, write 'Unknown' rather than inventing facts. Responses must exactly follow the above format, no additional explanation.`) ``` ### Prompt Design Rationale **Webpage Content Verification** (New): - LLM visits company websites and reads actual webpage content - Checks contact pages, about pages, and footers for email domain references - Prevents confusion when multiple companies have similar domain names - Example: "abc.com" vs "abc-corp.com" vs "abcglobal.com" - Only returns information if webpage explicitly confirms the exact domain **Exact Domain Matching**: - Prevents confusion with similar company names - Reduces false positives from LLM hallucination - Discards results where webpage shows a different domain **Structured Output**: - Predictable parsing - Easy to extract specific fields - Consistent formatting for notifications **Confidence Level**: - LLM self-reports confidence - Helps human reviewers assess reliability - Flags low-confidence results for manual verification **Unknown > Guessing**: - Explicit instruction to avoid hallucination - Better to have "Unknown" than wrong information - Sales can manually research if needed ### Educational Organization Detection After LLM triage, the system checks if the organization is educational: ```go func isEducationalTriage(triage string) bool { if strings.TrimSpace(triage) == "" { return false } fields := parseLLMTriageFields(triage) company := strings.ToLower(fields["company name"]) vertical := strings.ToLower(fields["vertical"]) full := strings.ToLower(triage) eduKeywords := []string{ "school district", "public schools", "school system", "board of education", "university", "college", "community college", "k-12", "high school", "middle school", "elementary school", "primary school", "secondary school", } for _, kw := range eduKeywords { if (company != "" && strings.Contains(company, kw)) || (vertical != "" && strings.Contains(vertical, kw)) || strings.Contains(full, kw) { return true } } return false } ``` **Field Parsing**: ```go func parseLLMTriageFields(triage string) map[string]string { fields := map[string]string{} for _, line := range strings.Split(triage, "\n") { line = strings.TrimSpace(line) if line == "" { continue } line = strings.TrimPrefix(line, "-") line = strings.TrimSpace(line) parts := strings.SplitN(line, ":", 2) if len(parts) != 2 { continue } key := strings.ToLower(strings.TrimSpace(parts[0])) val := strings.TrimSpace(parts[1]) if key != "" && val != "" { fields[key] = val } } return fields } ``` **Why Filter Educational Organizations?** 1. **Budget Constraints**: Schools typically have limited budgets 2. **Procurement Complexity**: Long sales cycles with public sector requirements 3. **Low Conversion**: High evaluation-to-paid conversion time 4. **Resource Allocation**: Sales effort better spent on commercial prospects **Detection Coverage**: - K-12 schools (primary, middle, high schools) - Higher education (universities, colleges, community colleges) - School districts and boards of education - Both in company name and vertical fields - Fallback to full text search ### LLM Error Handling ```go searchResult, err := s.llm.SimpleSearch(prompt) if err != nil { s.logger.Error(err, "failed to perform simple search", "emailDomain", v.emailDomain) // Continue without LLM data - event is not skipped } else { s.logger.Info("simple search result", "result", searchResult) v.extraInfo = searchResult if isEducationalTriage(searchResult) { ret.ForceSkip = true break } } ``` **Failure Modes**: - **LLM Service Down**: Event continues without triage data - **Timeout**: Event continues without triage data - **Invalid Response**: Event continues with raw response - **Unknown Domain**: Event continues with "Unknown" fields **Design Philosophy**: Graceful degradation - better to send event without LLM data than to miss the event entirely ### Performance Considerations **Latency**: - LLM web search: ~5-15 seconds - Synchronous call blocks preprocessing - Only for tenant activation events (not cluster/import) **Rate Limiting**: - No explicit rate limiting in code - LLM service may have internal rate limits - Consider implementing cache for repeated domains **Cost**: - Each tenant activation = 1 LLM web search call - Assuming 100 activations/day = 3,000/month - Consider caching for commonly seen domains (e.g., consulting firms) --- ## Event Flow ### Complete Flow Example: Tenant Activation ``` ┌─────────────────────────────────────────────────────────────────┐ │ T+0s: User completes signup on TiDB Cloud Console │ │ - Email: john@acme.com │ │ - Company: Acme Corp │ │ - Region: US │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+1s: Account Service emits TenantEvent │ │ - OrgId: 12345 │ │ - OrgEventType: OrgEventTypeActivate │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+1s: EventBus delivers to HandleDBaasAITenantEvent │ │ - Handler validates event type │ │ - Creates startupUBEvent │ │ - Sends to ubCh (buffered channel) │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+1s: DispatchUBEvents receives event from channel │ │ - Calls PreprocessUBEvent │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+2s: PreprocessUBEvent - Deduplication Check │ │ - Check handledTenant map │ │ - Tenant 12345 not seen before → Continue │ │ - Add to handledTenant map │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+2s: PreprocessUBEvent - Fetch Tenant Info │ │ - Call accountCli.GetTenant(12345) │ │ - Result: name="Acme Corp", email="john@acme.com", │ │ country="US", kind="external", plan="free" │ │ - Extract emailDomain = "acme.com" │ │ - Derive bizRegion = "us-east-1" │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+2s: PreprocessUBEvent - Early Filters │ │ - Internal tenant? No (kind="external") │ │ - Personal email? Check isPersonalEmail("acme.com") │ │ → Not in personal domain set → Continue │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+3s: PreprocessUBEvent - LLM Triage │ │ - Build prompt with "acme.com" │ │ - Call llm.SimpleSearch(prompt) │ │ - Wait for response... │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+10s: LLM returns triage result │ │ - Company Name: Acme Corporation │ │ - Headquarter: San Francisco, CA │ │ - Vertical: Enterprise Software │ │ - Business Model: B2B SaaS │ │ - Approximate Annual Revenue: $50M-$100M │ │ - Funding Series: Series C │ │ - AI confidence level: 88% │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+10s: PreprocessUBEvent - Educational Check │ │ - Parse LLM fields │ │ - Check for education keywords │ │ - Result: Not educational → Continue │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+10s: PreprocessUBEvent - Set Event Info │ │ - event.extraInfo = LLM triage result │ │ - eventInfo.isPaidTenant = true │ │ - eventInfo.bizRegion = "us-east-1" │ │ - eventInfo.tenantID = "12345" │ │ - Return eventInfo │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+11s: DispatchUBEvents - Query Subscriptions │ │ - Query DB for subscriptions matching UBEventStartup │ │ - Result: 2 subscriptions found │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+11s: DispatchUBEvents - Subscription #1 │ │ - Name: "US New Tenants" │ │ - Filters: region=us-east-1, onlyPaidTenant=false │ │ - Region match? Yes → Continue │ │ - Process template (custom template provided) │ │ - Build Lark card │ │ - Send to 2 chats (async goroutines) │ └─────────────────────────────────────────────────────────────────┘ │ │ T+11s: DispatchUBEvents - Subscription #2 │ │ - Name: "Enterprise Prospects" │ │ - Filters: region=all, onlyPaidTenant=true │ │ - Text filter: Revenue contains "$" │ │ - Apply filter: LLM result contains "$50M" → Pass │ │ - Use default text (no custom template) │ │ - Build Lark card │ │ - Send to 1 chat (async goroutine) │ └────────────────────┬────────────────────────────────────────────┘ │ v ┌─────────────────────────────────────────────────────────────────┐ │ T+12s: Lark notifications delivered │ │ - 3 Lark messages sent total │ │ - Sales teams notified in <12 seconds from signup │ └─────────────────────────────────────────────────────────────────┘ ``` ### Timing Breakdown | Phase | Duration | Cumulative | Notes | |-------|----------|------------|-------| | Signup to Event | ~1s | 1s | Account service event emission | | EventBus delivery | <100ms | 1s | In-process event bus | | Deduplication | <1ms | 1s | In-memory map lookup | | Fetch tenant info | ~500ms | 2s | AccountService API call | | Email filtering | <1ms | 2s | Hash set lookup | | LLM triage | ~8s | 10s | Web search + LLM inference | | Educational check | <5ms | 10s | String matching | | Query subscriptions | ~100ms | 11s | Database query | | Filter subscriptions | ~50ms | 11s | In-memory filtering | | Build cards | ~10ms | 11s | JSON serialization | | Send to Lark | ~1s | 12s | HTTP requests (async) | **Total End-to-End**: ~12 seconds from signup to notification **Bottleneck**: LLM triage (8s) - see Future Enhancements for caching strategies --- ## Configuration ### Service Configuration ```go type UserBehaviorServiceConfig struct { SyncLarkChatsInterval *time.Duration // Cron job interval // Startup-specific config would go here } ``` ### Subscription Configuration **Creating a Subscription for Startup Events**: ```go param := CreateSubscriptionParam{ Name: "US Startup Events", SubscribeType: ubent.SubscribeTpLark, EventType: ubent.UBEventStartup, // All three startup event types Template: "", // Use default text Selector: ubent.SelectorTp{ RegionSelectors: []ubent.RegionSelectorTp{ ubent.ToRegionSelectorTp("us-east-1"), ubent.ToRegionSelectorTp("us-west-2"), }, TenantSelectors: []ubent.TenantSelectorTp{}, // All tenants }, LarkChats: []*entity.LarkChatSimplified{ {ID: 123, Name: "US Sales Team"}, }, OnlyPaidTenant: false, // Include free tier (they may upgrade) OnlyExternalTenant: true, // Exclude internal tenants Creator: "sales-ops@company.com", DetailLink: pointer.Pointer("https://ops.tidbcloud.com"), } ``` ### Custom Filters **Example: High-Value Prospects Only** ```go // In subscription TextFilter field TextFilter: ` (OrgEventType == "OrgEventTypeActivate") && (!strings.Contains(ExtraInfo, "Unknown")) ` ``` **Example: Specific Verticals** ```go TextFilter: ` strings.Contains(ExtraInfo, "FinTech") || strings.Contains(ExtraInfo, "Healthcare") || strings.Contains(ExtraInfo, "Gaming") ` ``` ### Environment Variables ```bash # LLM Service endpoint LLM_SERVICE_URL=https://llm-service.internal # LLM timeout LLM_TIMEOUT=30s # Personal email domain list version PERSONAL_EMAIL_VERSION=v2024.1 ``` --- ## Operational Guide ### Monitoring Startup Events **Key Metrics to Track**: 1. **Event Volume**: - Tenant activations per day/week - First clusters per day/week - First imports per day/week 2. **Filter Effectiveness**: - % of events skipped (by filter type) - Personal email filter hits - Educational org filter hits - Internal tenant filter hits 3. **LLM Performance**: - LLM call success rate - LLM latency (p50, p95, p99) - LLM confidence levels (distribution) 4. **Notification Delivery**: - Messages sent per subscription - Delivery success rate - Lark API errors ### Common Operational Tasks #### 1. Adding a New Region ```go // Update subscription selector subscription.Selector.RegionSelectors = append( subscription.Selector.RegionSelectors, ubent.ToRegionSelectorTp("eu-central-1"), ) ``` #### 2. Updating Email Domain Blacklist ```go // Add to email_filter.go personalEmailDomainSet "newpersonaldomain.com", ``` Rebuild and redeploy service. #### 3. Investigating Missing Event **Checklist**: 1. Check if tenant is internal: `kind == "internal"` 2. Check email domain: `isPersonalEmail(domain)` 3. Check LLM triage logs: Educational org detected? 4. Check deduplication: Already processed? 5. Check subscription filters: Region/tenant match? 6. Check event source: Was event actually emitted? #### 4. Testing Subscription ```bash # Use mock event API curl -X POST /api/userbehavior/subscriptions/123/mock ``` #### 5. Adjusting LLM Timeout ```go // In LLM client configuration llmClient := llm_cli.NewLLMCli(llm_cli.Config{ Timeout: 30 * time.Second, // Increase if timeouts occur }) ``` ### Troubleshooting #### Problem: Too Many False Positives (Personal Users) **Symptoms**: Startup events for Gmail/Yahoo users **Investigation**: 1. Check email domain extraction: `getEmailDomain("user@gmail.com")` should return `"gmail.com"` 2. Check personal email set: `"gmail.com"` in `personalEmailDomainSet`? 3. Check logs for filter hits **Solution**: - Update personal email domain list - Add domain-specific patterns (e.g., `.edu.` subdomains) #### Problem: Missing High-Value Events **Symptoms**: Known enterprise signup not notified **Investigation**: 1. Check preprocessing logs for skip reason 2. Check if LLM triage detected as educational 3. Check subscription region/tenant filters 4. Check if event was deduplicated **Solution**: - Review LLM triage result - Add tenant to explicit whitelist in subscription - Adjust educational keyword list #### Problem: LLM Triage Errors **Symptoms**: `"failed to perform simple search"` in logs **Investigation**: 1. Check LLM service health 2. Check network connectivity 3. Check LLM API rate limits 4. Check domain accessibility (firewall?) **Solution**: - Retry failed events (manual or automated) - Implement LLM result caching - Increase LLM timeout - Add fallback to skip LLM on repeated failures #### Problem: Slow Event Processing **Symptoms**: Long delay from signup to notification **Investigation**: 1. Check LLM latency: Should be <15s 2. Check database query time: Should be <500ms 3. Check event channel depth: Should be <50 **Solution**: - Scale LLM service - Optimize subscription queries (add indexes) - Increase event channel buffer - Process LLM triage async (see Future Enhancements) --- ## Examples ### Example 1: Enterprise SaaS Company **Signup**: - Email: `engineering@salesforce.com` - Name: Salesforce Inc - Region: US **Processing**: 1. ✅ Not internal tenant 2. ✅ Not personal email (`salesforce.com` not in personal set) 3. ✅ LLM triage: - Company: Salesforce Inc - HQ: San Francisco, CA - Vertical: Enterprise Software / CRM - Revenue: $30B+ - Funding: Public Company 4. ✅ Not educational 5. ✅ Event dispatched **Notification**: ``` [** us-west-2 **] New non-individual tenant [**Salesforce Inc**](https://ops.tidbcloud.com/orgs/99999)(id: 99999). Email domain is salesforce.com - Company Name: Salesforce Inc - Headquarter: San Francisco, CA - Vertical: Enterprise Software / CRM - Business Model: B2B SaaS - Approximate Annual Revenue: $30B+ - Funding Series: Public Company (Please notice that these info are search from the internet and validate as needed. AI confidence level: 95%) ``` ### Example 2: University Department **Signup**: - Email: `professor@stanford.edu` - Name: Stanford University CS Dept - Region: US **Processing**: 1. ✅ Not internal tenant 2. ✅ Not personal email (`.edu` suffix) 3. ❌ **FILTERED**: Email domain contains `.edu` → Skip personal email filter 4. 🛑 Event skipped **No Notification Sent** **Rationale**: Educational institutions filtered at email level ### Example 3: Early-Stage Startup **Signup**: - Email: `founder@newstartup.io` - Name: NewStartup - Region: EU **Processing**: 1. ✅ Not internal tenant 2. ✅ Not personal email 3. ✅ LLM triage: - Company: NewStartup Inc - HQ: Berlin, Germany - Vertical: FinTech - Revenue: Unknown - Funding: Pre-Seed 4. ✅ Not educational 5. ✅ Event dispatched **Notification**: ``` [** eu-west-1 **] New non-individual tenant [**NewStartup**](https://ops.tidbcloud.com/orgs/88888)(id: 88888). Email domain is newstartup.io - Company Name: NewStartup Inc - Headquarter: Berlin, Germany - Vertical: FinTech - Business Model: B2B - Approximate Annual Revenue: Unknown - Funding Series: Pre-Seed (Please notice that these info are search from the internet and validate as needed. AI confidence level: 72%) ``` ### Example 4: Individual Developer (Filtered) **Signup**: - Email: `johndoe@gmail.com` - Name: John Doe - Region: US **Processing**: 1. ✅ Not internal tenant 2. ❌ **FILTERED**: `gmail.com` in personal email set → Skip 3. 🛑 Event skipped **No Notification Sent** ### Example 5: Internal TiDB Team (Filtered) **Signup**: - Email: `demo@pingcap.com` - Name: PingCAP Demo Account - Region: US - Kind: Internal **Processing**: 1. ❌ **FILTERED**: Internal tenant (kind == "internal") → Skip 2. 🛑 Event skipped **No Notification Sent** ### Example 6: First Cluster → First Import Journey **T+0**: Tenant activation (Example 3 - NewStartup) - Notification sent: "New non-individual tenant" **T+15min**: First cluster created ``` [** eu-west-1 **] First essential instance of tenant [**NewStartup**](https://ops.tidbcloud.com/orgs/88888) (id: 88888, email domain: newstartup.io) is being created. Cluster ID is [**77777**](https://ops.tidbcloud.com/orgs/88888/clusters/77777). ``` **T+2hours**: First import completed ``` [** eu-west-1 **] Tenant [NewStartup](https://ops.tidbcloud.com/orgs/88888) (id: 88888, email domain: newstartup.io) finished an import. Cluster: [77777](https://ops.tidbcloud.com/orgs/88888/clusters/77777) ``` **Analysis**: Fast progression (activation → cluster → import in 2 hours) indicates serious evaluation --- ## Future Enhancements ### 1. Persistent Deduplication **Current Issue**: In-memory `handledTenant` map is lost on service restart **Proposed Solution**: ```go // Use Redis or database for persistent deduplication type DeduplicationStore interface { IsHandled(ctx context.Context, tenantID uint64) (bool, error) MarkHandled(ctx context.Context, tenantID uint64, ttl time.Duration) error } // With TTL (e.g., 7 days) if handled, err := s.dedupStore.IsHandled(ctx, v.OrgId); err == nil && handled { ret.ForceSkip = true break } s.dedupStore.MarkHandled(ctx, v.OrgId, 7*24*time.Hour) ``` **Benefits**: - Survives service restarts - Shared across multiple service instances - Configurable TTL for re-qualification ### 2. LLM Result Caching **Current Issue**: Each tenant activation triggers LLM call, even for known domains **Proposed Solution**: ```go // Cache LLM triage results by email domain type LLMCache interface { Get(ctx context.Context, domain string) (string, bool, error) Set(ctx context.Context, domain string, result string, ttl time.Duration) error } // Check cache before LLM call if cachedResult, found, err := s.llmCache.Get(ctx, emailDomain); err == nil && found { v.extraInfo = cachedResult } else { // Call LLM searchResult, err := s.llm.SimpleSearch(prompt) if err == nil { s.llmCache.Set(ctx, emailDomain, searchResult, 30*24*time.Hour) v.extraInfo = searchResult } } ``` **Benefits**: - Reduced LLM costs (especially for consulting firms with multiple signups) - Faster event processing (<2s instead of ~10s) - Reduced load on LLM service **Cache Invalidation**: - TTL: 30 days (company info doesn't change frequently) - Manual invalidation API for data corrections - Cache warming for known high-volume domains ### 3. Async LLM Processing **Current Issue**: LLM call blocks event preprocessing (~8s) **Proposed Solution**: ```go // Dispatch event immediately, enrich with LLM data later go func() { searchResult, err := s.llm.SimpleSearch(prompt) if err == nil { // Update event in database or send enrichment notification s.enrichEvent(ctx, tenantID, searchResult) } }() // Send initial notification without LLM data // Send follow-up notification with LLM enrichment ``` **Benefits**: - Faster initial notification (<2s) - Non-blocking preprocessing - Graceful degradation if LLM is slow/down **Considerations**: - Two notifications per tenant (initial + enrichment) - Complexity in matching enrichment to original event - Educational org filter runs after initial notification ### 4. Enhanced Educational Detection **Current Limitations**: Keyword-based detection may miss some cases **Proposed Enhancements**: ```go // Domain-based detection var educationalDomainPatterns = []string{ ".edu", ".ac.uk", ".edu.au", ".edu.cn", "school", "university", "college", } // Email-specific patterns var educationalEmailPatterns = []string{ "student@", "teacher@", "professor@", "admin@", } // LLM-based confidence score type EducationalScore struct { IsEducational bool Confidence float64 Reason string } ``` ### 5. Onboarding Velocity Tracking **Proposed Feature**: Track time between startup milestones ```go type OnboardingMetrics struct { TenantID uint64 ActivationTime time.Time FirstClusterTime *time.Time FirstImportTime *time.Time // Derived metrics ActivationToCluster *time.Duration // Fast (<1h) = high intent ClusterToImport *time.Duration // Fast (<6h) = serious evaluation } // Scoring func (m *OnboardingMetrics) VelocityScore() float64 { score := 0.0 if m.ActivationToCluster != nil && *m.ActivationToCluster < 1*time.Hour { score += 0.4 // Fast cluster creation } if m.ClusterToImport != nil && *m.ClusterToImport < 6*time.Hour { score += 0.6 // Fast data import } return score // 0.0 = slow, 1.0 = very fast } ``` **Use Case**: Prioritize sales outreach to high-velocity tenants ### 6. Vertical-Specific Routing **Proposed Feature**: Route notifications based on LLM-detected vertical ```go type SubscriptionSelector struct { RegionSelectors []RegionSelectorTp TenantSelectors []TenantSelectorTp VerticalSelectors []VerticalSelectorTp // NEW } // Example: Route FinTech leads to specialized team subscription := Subscription{ Name: "FinTech Leads", Selector: SelectorTp{ VerticalSelectors: []VerticalSelectorTp{ "FinTech", "Financial Services", "Banking", }, }, LarkChats: finTechSalesTeamChats, } ``` ### 7. Multi-Event Aggregation **Current Issue**: Three separate events (activation, cluster, import) create notification spam **Proposed Solution**: ```go // Aggregate events within time window type AggregatedStartupEvent struct { TenantID uint64 ActivationEvent *startupUBEvent ClusterEvent *startupClusterUBEvent ImportEvent *startupImportUBEvent WindowStart time.Time WindowEnd time.Time } // Single aggregated notification [** us-east-1 **] New tenant journey: [**Acme Corp**](...) - ✅ Activated at 10:00am (email: acme.com) - ✅ First cluster created at 10:15am (cluster: 12345) - ✅ First import completed at 12:30pm - LLM Triage: [company info...] ``` **Configuration**: - Aggregation window: 24 hours - Send when: All events received OR window expires - Benefits: Reduced notification volume, better context ### 8. Lead Scoring **Proposed Feature**: Automatic lead scoring based on signals ```go type LeadScore struct { TenantID uint64 Score int // 0-100 Factors []string // What contributed to score Confidence float64 // LLM confidence RecommendedAction string // Call, email, nurture, ignore } func CalculateLeadScore(event *startupUBEvent) LeadScore { score := 50 // Base score // LLM-based factors if strings.Contains(event.extraInfo, "Series B") || strings.Contains(event.extraInfo, "Series C") { score += 20 // Well-funded startup } if strings.Contains(event.extraInfo, "$100M") { score += 15 // High revenue company } if strings.Contains(event.extraInfo, "Enterprise") { score += 10 // Enterprise segment } // Behavior-based factors if event.velocityScore > 0.8 { score += 15 // Fast onboarding } // Domain-based factors if isKnownEnterpriseEmail(event.emailDomain) { score += 10 // From known company } return LeadScore{Score: score, ...} } ``` ### 9. Export to CRM **Proposed Feature**: Automatic export to Salesforce/HubSpot ```go type CRMExporter interface { CreateLead(ctx context.Context, lead Lead) (string, error) UpdateLead(ctx context.Context, leadID string, updates map[string]interface{}) error } // After event processing lead := Lead{ Email: event.email, Company: event.companyName, Region: event.bizRegion, Source: "TiDB Cloud Signup", CustomFields: map[string]string{ "email_domain": event.emailDomain, "llm_triage": event.extraInfo, "first_cluster": event.clusterTime.String(), "velocity_score": fmt.Sprintf("%.2f", event.velocityScore), }, } crmLeadID, err := s.crmExporter.CreateLead(ctx, lead) ``` ### 10. A/B Testing for Filters **Proposed Feature**: Test filter effectiveness ```go type FilterExperiment struct { Name string VariantA FilterConfig // Current filter VariantB FilterConfig // New filter to test TrafficSplit float64 // % to variant B Metrics ExperimentMetrics } type ExperimentMetrics struct { EventsProcessed int EventsFiltered int FalsePositives int // Manual review FalseNegatives int // Manual review SalesConversionRate float64 } // Example: Test stricter educational filter experiment := FilterExperiment{ Name: "Enhanced Educational Filter", VariantA: currentEducationalFilter, VariantB: enhancedEducationalFilter, TrafficSplit: 0.1, // 10% to variant B } ``` --- ## Appendix: Personal Email Domain Statistics The system maintains a comprehensive list of personal email domains: **Total Domains**: ~3,700 **Category Breakdown**: - Major providers (Gmail, Yahoo, Hotmail, etc.): ~50 - Regional providers (QQ, 163, Yandex, etc.): ~200 - Disposable email services: ~500 - Country-specific providers: ~2,000 - Educational domains (pattern-based): ~1,000 **Top Personal Domains by Volume** (estimated): 1. gmail.com 2. yahoo.com 3. hotmail.com 4. outlook.com 5. qq.com 6. 163.com 7. 126.com 8. aol.com 9. icloud.com 10. mail.ru **Maintenance**: - Updated quarterly - Sources: Public email provider lists, observed signups - Process: Manual review + automated detection --- ## Appendix: LLM Prompt Engineering ### Prompt Evolution **V1 (Initial)**: ``` Tell me about the company with domain acme.com ``` **Issues**: - Hallucination (made up details) - Inconsistent format - No confidence indication **V2 (Structured)**: ``` Search for information about acme.com and provide: - Company Name - Location - Industry ``` **Issues**: - Still some hallucination - Wrong company (similar names) **V3 (Current)**: ``` Search the web using exact-match domain rules: - Only consider results that match the exact domain - Ignore similar/related domains - Return Unknown if not confirmed Respond with these fields: - Company Name: - Headquarter: - Vertical: - Business Model: - Approximate Annual Revenue: - Funding Series: (AI confidence level: xx%) ``` **Improvements**: - Explicit anti-hallucination instructions - Exact domain matching - Structured output - Confidence level ### Recent Improvements (2026-03) **Webpage Content Verification** (✅ Implemented): - LLM now visits company websites and reads webpage content - Verifies email domain from contact pages, about pages, and footers - Prevents false matches when multiple companies have similar domain names - Example: Correctly distinguishes "abc.com" from "abc-corp.com" or "abcglobal.com" - Implementation: Enhanced prompt with explicit verification steps **Impact**: - Improved accuracy for company identification - Reduced false positives from similar domain names - Better quality leads for sales teams ### Future Prompt Improvements **Chain-of-Thought**: ``` First, search for information about {domain}. Then, verify this information matches the exact domain. Finally, format the response. ``` **Few-Shot Examples**: ``` Example 1: Domain: salesforce.com Company Name: Salesforce Inc ... Example 2: Domain: unknown-startup.io Company Name: Unknown ... Now search for: {domain} ``` --- *End of Startup Events Design Document*