# LLM Email Domain Verification - Testing Guide

## Overview

This guide covers how to test the enhanced LLM prompt that verifies email domains using webpage content.

---

## Test Scenarios

### Scenario 1: Exact Domain Match (Should Return Company Info)

**Test Cases**:
- `salesforce.com` - Well-known company with clear email domain
- `stripe.com` - Tech company with public contact info
- `mongodb.com` - Database company
- `pingcap.com` - Your own company (should be easy to verify)

**Expected Result**: LLM returns accurate company information

**Validation**:
- Company Name should be correct
- Headquarters should be accurate
- All fields populated (not "Unknown")
- AI confidence level > 80%

---

### Scenario 2: Similar Domain Names (Should Distinguish Correctly)

**Test Cases**:
- `abc.com` - Should find ABC Technologies (hypothetical)
- `abc-corp.com` - Should find ABC Corporation (different company)
- `data.com` - Should find specific Data.com company
- `datacorp.com` - Should find different company

**Expected Result**: LLM visits webpages and returns info ONLY for the exact domain

**Validation**:
- Result matches the EXACT input domain
- LLM doesn't confuse with similar domains
- If unsure, returns "Unknown" instead of guessing

---

### Scenario 3: Domain Mismatch (Should Return Unknown)

**Test Cases**:
- `nonexistent-startup-xyz-2026.com` - Fake domain
- `test123456.com` - Test domain without real company
- `example.com` - Generic example domain

**Expected Result**: LLM returns "Unknown" for all fields

**Validation**:
- Company Name: Unknown
- Headquarter: Unknown
- All fields show "Unknown"
- No hallucinated information

---

### Scenario 4: Multiple Companies, Wrong Domain (Should Return Unknown)

**Test Case**:
- Search for `xyz.com` but only `xyz-global.com` exists on webpages

**Expected Result**:
- LLM finds "XYZ Global" in search results
- Visits their website and sees `@xyz-global.com`
- Recognizes domain mismatch
- Returns "Unknown" for all fields

---

## Testing Methods

### Method 1: Direct Script Testing (Fastest)

**Use the provided test script**:

```bash
cd test/llm_prompt_test

# Test a well-known company
go run main.go salesforce.com

# Test a similar domain scenario
go run main.go stripe.com

# Test a non-existent domain
go run main.go fake-company-xyz-2026.com
```

**Advantages**:
- Quick feedback (5-15 seconds per test)
- No need to trigger actual events
- Easy to iterate on different domains

**What to Check**:
1. Does the response follow the structured format?
2. Are the facts accurate (verify manually)?
3. Does it return "Unknown" for non-existent domains?
4. Check the confidence level percentage

---

### Method 2: Unit Test in Service

Create a test file to mock and verify the logic:

```go
// pkg/service/userbehavior/user_behavior_service_test.go

func TestLLMEmailDomainVerification(t *testing.T) {
	tests := []struct {
		name          string
		emailDomain   string
		wantCompany   string
		shouldHaveInfo bool
	}{
		{
			name:           "Well-known company",
			emailDomain:    "salesforce.com",
			wantCompany:    "Salesforce",
			shouldHaveInfo: true,
		},
		{
			name:           "Non-existent domain",
			emailDomain:    "fake-company-xyz-2026.com",
			wantCompany:    "Unknown",
			shouldHaveInfo: false,
		},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			// Test implementation here
		})
	}
}
```

---

### Method 3: Integration Test with Mock Event

Create a test that sends a mock tenant activation event:

```bash
# Use the service API to send a mock event
curl -X POST http://localhost:8080/api/v1/userbehavior/subscriptions/{id}/mock
```

**Setup**:
1. Create a test subscription for startup events
2. Configure it to send to a test Lark chat
3. Trigger the mock event
4. Check the Lark notification

**What to Verify**:
- Event is processed correctly
- LLM triage appears in notification
- Information is accurate
- No educational organizations leaked through

---

### Method 4: End-to-End Test (Production-like)

**Setup**:
1. Create a test tenant with a specific email domain
2. Activate the tenant
3. Monitor logs for LLM call
4. Check Lark notification

**Caution**: This creates real data in the system

---

## Specific Test Domains

### ✅ Recommended Test Domains (Real Companies)

```bash
# Tech companies with clear email domains
go run test_llm_prompt.go stripe.com
go run test_llm_prompt.go shopify.com
go run test_llm_prompt.go atlassian.com
go run test_llm_prompt.go databricks.com
go run test_llm_prompt.go snowflake.com

# Your own company
go run test_llm_prompt.go pingcap.com
```

### ⚠️ Edge Cases to Test

```bash
# Very similar domain names (manual verification needed)
go run test_llm_prompt.go data.com
go run test_llm_prompt.go datadog.com

# Domains that might have multiple companies
go run test_llm_prompt.go abc.com
go run test_llm_prompt.go general.com

# Non-existent domains
go run test_llm_prompt.go fake-startup-2026.com
go run test_llm_prompt.go test123xyz.com
```

### 🚫 Domains to Avoid

- Personal email domains (will be filtered before LLM)
- Internal test domains
- Competitor domains (for privacy)

---

## Validation Checklist

After running tests, verify:

### ✅ Accuracy
- [ ] Real companies return correct information
- [ ] Company name matches the domain
- [ ] Headquarters location is accurate
- [ ] Vertical/industry makes sense

### ✅ Domain Verification
- [ ] Similar domains are distinguished correctly
- [ ] No confusion between `abc.com` and `abc-corp.com`
- [ ] LLM checks webpage content, not just search results

### ✅ Unknown Handling
- [ ] Fake domains return "Unknown"
- [ ] No hallucinated company names
- [ ] Confidence level is appropriate

### ✅ Format Compliance
- [ ] Response follows structured format
- [ ] All required fields present
- [ ] No extra explanations outside format

### ✅ Performance
- [ ] Response time < 30 seconds
- [ ] No timeouts
- [ ] Graceful error handling

---

## Debugging Tips

### If LLM Returns Wrong Company

**Check**:
1. Does the webpage actually show the email domain?
2. Is the domain redirecting to another domain?
3. Are there multiple companies with similar names?

**Action**:
- Manually visit the company website
- Check the contact page for email domain
- Compare with LLM reasoning (if available)

### If LLM Returns "Unknown" for Real Company

**Possible Causes**:
1. Company website doesn't show email addresses
2. Website is behind authentication
3. Domain is very new (not indexed)
4. Website uses a different domain for contact

**Action**:
- Manually verify the website content
- Check if contact info is visible
- Consider if this is acceptable behavior

### If LLM Hallucinates Information

**Symptoms**:
- Made-up company names
- Incorrect headquarters
- Fake revenue/funding info

**Action**:
- Review prompt instructions
- Check if domain actually exists
- Verify confidence level is low
- Consider this a bug if confidence is high

---

## Monitoring in Production

### Logs to Watch

```bash
# Successful LLM calls
grep "simple search result" logs/service.log

# Failed LLM calls
grep "failed to perform simple search" logs/service.log

# Educational org filtering
grep "skip startupUBEvent for educational organization" logs/service.log
```

### Metrics to Track

- LLM call success rate
- Average response time
- Rate of "Unknown" results
- Educational org filter rate

---

## Expected LLM Response Format

### Good Response Example

```
- Company Name: Salesforce Inc
- Headquarter: San Francisco, CA
- Vertical: Enterprise Software
- Business Model: B2B SaaS
- Approximate Annual Revenue: $30B+
- Funding Series: Public (NYSE: CRM)
(Please notice that these info are search from the internet and validate as needed. AI confidence level: 95%)
```

### Unknown Response Example

```
- Company Name: Unknown
- Headquarter: Unknown
- Vertical: Unknown
- Business Model: Unknown
- Approximate Annual Revenue: Unknown
- Funding Series: Unknown
(Please notice that these info are search from the internet and validate as needed. AI confidence level: 0%)
```

---

## Troubleshooting

### Error: "failed to perform simple search"

**Causes**:
- Gemini API key missing/invalid
- Network connectivity issues
- API rate limit exceeded

**Solution**:
```bash
# Check config
cat config/config.yaml | grep GeminiKey

# Test API key
curl -H "Authorization: Bearer $GEMINI_KEY" https://generativelanguage.googleapis.com/v1/models
```

### Error: "Error loading config"

**Solution**:
```bash
# Run from correct directory
cd /Users/siyuchen/Code/devops

# Or set config path
export CONFIG_PATH=/path/to/config.yaml
```

### LLM Returns Empty Result

**Possible Causes**:
- Response parsing failed
- LLM returned unexpected format
- Timeout occurred

**Debug**:
- Check raw LLM response in logs
- Verify prompt format
- Test with simpler domain first

---

## Next Steps After Testing

### If Tests Pass ✅

1. Monitor first few production events closely
2. Compare LLM results with manual research
3. Collect feedback from sales team
4. Track false positive/negative rates

### If Tests Fail ❌

1. Identify which scenarios fail
2. Adjust prompt wording if needed
3. Consider adding more verification steps
4. Document edge cases for future reference

---

## Continuous Improvement

### Collect Examples

As you test, collect examples of:
- Perfect matches (for validation)
- Edge cases (for future prompt improvement)
- Failures (for debugging)

### Feedback Loop

1. Test with diverse domains weekly
2. Compare LLM results with sales feedback
3. Adjust prompt based on findings
4. Update test scenarios

---

*Last Updated: 2026-03-06*