# Alert/Ticket Source Preparation Pipeline

This document explains how Investibook prepares alert and ticket source data for workspace initialization.

## Overview

Investibook follows a new data pipeline flow that prioritizes getting cluster information from Jira first, then enriching it with cluster metadata and optional analysis insights.

Supported Jira issue types:
- **O11Y-***: Observability alerts - Alerts with raw alert JSON in customfield_10160
- **TCOC-***: TiDBCloud Oncall alerts - Alerts with raw alert JSON in customfield_10160
- **TCID-***: TiDBCloud Ticket - Tickets with direct custom fields for cluster and org metadata

**Note:** Only the prefixes listed above are supported. Issues with other prefixes (e.g., ST-, DBAAS-) will fail with an unsupported issue type error.

## Pipeline Steps

### 1. Fetch Issue Details from Jira (Required)

**Purpose:** Get the cluster_id, tenant_id, and other metadata from the Jira issue.

The pipeline automatically detects the issue type based on the prefix and uses the appropriate extraction method.

**API:** Jira REST API `/rest/api/2/issue/{alert_id}`

**Authentication:** Basic auth using Jira username + API token

**Extracted Fields:**
- `cluster_id` - **Required for next steps**
  - If value starts with "tidb" prefix, it will be trimmed (e.g., "tidb123456" → "123456")
- `tenant_id`
- Full issue data (for building alert_meta.json)

**Configuration:**
```yaml
jira_base_url: "https://pingcap.atlassian.net"
jira_username: "your-email@pingcap.com"
jira_api_token: "ATATT3xFfGF0..."
```

**Example Request:**
```http
GET /rest/api/2/issue/O11Y-1956041
Authorization: Basic base64(username:api_token)
```

**Field Extraction Priority (Fast Path → Fallback):**

The function uses a **two-tier extraction strategy**:

**1. Fast Path (Direct Custom Fields):**
- Check `customfield_10149` for `cluster_id`
- Check `customfield_10157` for `tenant_id`
- If both are present → **Use them immediately** (skip raw alert JSON parsing)

**2. Fallback Path (Parse Raw Alert JSON):**

If custom fields are not present, parse `customfield_10160` (raw alert JSON string):

- Parse JSON and extract from `labels` object:
  - **cluster_id:** `labels.tidb_cluster_id` → `labels.cluster_id` → `labels.name`
  - **tenant_id:** `labels.o11y_tenant_id` → `labels.tenant_id`

**Example Responses:**

*Scenario 1: Fast path with direct custom fields*
```json
{
  "key": "O11Y-1956041",
  "fields": {
    "customfield_10149": "123456",
    "customfield_10157": "tenant_abc123"
  }
}
```
→ Extracted immediately: `cluster_id="123456"`, `tenant_id="tenant_abc123"`

*Scenario 2: Fallback with tidb_cluster_id and prefix*
```json
{
  "key": "O11Y-1956042",
  "fields": {
    "customfield_10160": "{\"labels\": {\"tidb_cluster_id\": \"tidb789012\", \"o11y_tenant_id\": \"tenant_def456\", \"alertname\": \"HighCPUUsage\"}}"
  }
}
```
→ Extracted: `cluster_id="789012"` (prefix trimmed), `tenant_id="tenant_def456"`

*Scenario 3: Multiple fallbacks*
```json
{
  "key": "O11Y-1956043",
  "fields": {
    "customfield_10160": "{\"labels\": {\"name\": \"cluster_999\", \"tenant_id\": \"tenant_ghi789\"}}"
  }
}
```
→ Extracted: `cluster_id="cluster_999"`, `tenant_id="tenant_ghi789"`

**Cluster ID Extraction Logic:**
1. Try `customfield_10149` (direct)
2. If empty, try `labels.tidb_cluster_id` from raw alert JSON
3. If empty, try `labels.cluster_id` from raw alert JSON
4. If empty, try `labels.name` from raw alert JSON
5. If value starts with "tidb" prefix, trim it:
   - `"tidb123456"` → `"123456"`
   - `"123456"` → `"123456"` (no prefix, unchanged)

**Tenant ID Extraction Logic:**
1. Try `customfield_10157` (direct)
2. If empty, try `labels.o11y_tenant_id` from raw alert JSON
3. If empty, try `labels.tenant_id` from raw alert JSON

**Additional Data Extracted:**
From the full Jira issue, the function also builds `alert_meta.json` containing:
- `jira_issue_id`
- `summary` (issue title)
- `description` (issue description)
- `alert_name` (from `labels.alertname`)
- `component` (from `labels.component`)
- `status` (from `fields.status.name`)
- `priority` (from `fields.priority.name`)
- `created` and `updated` timestamps

---

#### TiDBCloud Ticket Extraction (TCID-*)

For TiDBCloud Tickets (TCID-*), a simpler extraction method is used:

**Direct Custom Fields:**
- `customfield_10088` → `cluster_id` (FIELD_TIDBCLOUD_CLUSTER_ID)
- `customfield_10086` → `tenant_id` (FIELD_TIDBCLOUD_ORG_ID)

**Example Request:**
```http
GET /rest/api/2/issue/TCID-12345
Authorization: Basic base64(username:api_token)
```

**Example Response:**
```json
{
  "key": "TCID-12345",
  "fields": {
    "customfield_10088": "987654",
    "customfield_10086": "tenant_xyz789",
    "summary": "Database connection timeout",
    "status": {"name": "In Progress"}
  }
}
```
→ Extracted: `cluster_id="987654"`, `tenant_id="tenant_xyz789"`

**Note on Asset Fields:**
JSM Asset fields (customfield_10110 for org asset, customfield_10111 for cluster asset) are not currently supported as they require JSM Asset API access which is not yet implemented in investibook. If you need to support tickets that only have asset fields, please file an issue.

---

**Failure Handling:**
- If issue prefix is unsupported: **Pipeline fails immediately**
- If Jira is not configured: Log warning, return (None, None, None)
- If Jira API fails: Log warning, return (None, None, None)
- If raw alert JSON parsing fails: Log warning, continue with extracted values
- If cluster_id is missing: **Pipeline fails** (cluster_id is required)

### 2. Fetch Cluster Metadata from Clinic (Required)

**Purpose:** Get detailed cluster information like version, org_id, etc.

**API:** Clinic HTTP API `/dashboard/clusters2`

**Authentication:**
- API key (Bearer token) OR
- Browser cookie + CSRF token

**Input:** `cluster_id` from Step 1

**Output:** Comprehensive cluster metadata object:
```json
{
  "tenant_id": "tenant_abc",
  "clinic_org_id": "org_123",
  "cluster_id": "cluster_xyz789",
  "cluster_name": "my-production-cluster",
  "tidb_version": "v7.5.0",
  "cluster_type": "DEDICATED",
  "cloud_provider": "AWS",
  "region": "us-west-2",
  "status": "AVAILABLE",
  "create_timestamp": "2024-01-15T10:30:00Z"
}
```

**Field Mappings from Clinic API:**
- `clusterProviderName` → `cloud_provider`
- `clusterDeployType` → `cluster_type`
- `clusterRegionName` → `region`
- `clusterCreatedAt` → `create_timestamp`

**Configuration:**
```yaml
# Option 1: API Key
clinic_endpoint: "https://clinic.pingcap.com"
clinic_api_key: "your-api-key"

# Option 2: Cookie (local debugging)
clinic_endpoint: "https://clinic.pingcap.com"
clinic_cookie: "session=eyJ..."
clinic_csrf_token: "abc123..."
```

**Failure Handling:**
- If Clinic is not configured: Log warning, use minimal cluster_meta
- If Clinic API fails: Log warning, use minimal cluster_meta with data from Jira
- Minimal cluster_meta (with empty/unknown values):
  ```json
  {
    "tenant_id": "from_jira",
    "clinic_org_id": "",
    "cluster_id": "from_jira",
    "cluster_name": "",
    "tidb_version": "unknown",
    "cluster_type": "",
    "cloud_provider": "",
    "region": "",
    "status": "",
    "create_timestamp": ""
  }
  ```

### 3. Fetch Jira Comments and Generate AI Summary (Optional)

**Purpose:** Fetch Jira comments for context and generate an AI-powered summary.

**APIs:**
- Jira REST API: `/rest/api/2/issue/{alert_id}/comment`
- OpenAI API: `https://api.openai.com/v1/chat/completions`

**Authentication:**
- Jira: Basic auth (same as step 1)
- OpenAI: Bearer token (API key)

**Input:** `alert_id` from Jira

**Output:** Two files written to workspace:
- `comments.json` - Full Jira comments payload
- `comments_summary_ai.md` - AI-generated summary or fallback summary

**Configuration:**
```yaml
# Jira (already configured in step 1)
jira_base_url: "https://pingcap.atlassian.net"
jira_username: "your-email@pingcap.com"
jira_api_token: "ATATT3xFfGF0..."

# OpenAI for AI summary (optional)
openai_api_key: "sk-your-openai-key"
openai_model: "gpt-4o"  # defaults to gpt-4o if not specified
```

**AI Summary Generation:**

1. **Fetch comments** from Jira API
2. **Build prompt** with all comments (author, timestamp, body)
   - Each comment truncated to 600 chars to avoid overly long prompts
3. **Call OpenAI gpt-4o** with system prompt:
   ```
   You are an AIOps expert, please generate a concise summary of the Jira comments.
   Requirements:
   1. The summary should be concise and professional.
   2. The summary must include:
       - Current status
       - Root cause
       - Final conclusion
   3. If there are clear next steps or owners, summarize them in one sentence.
   ```
4. **Write summary** to `comments_summary_ai.md`

**Fallback Strategy:**

If OpenAI API call fails or API key is not configured:
- Generate a **simple deterministic summary** using the last comment:
  - Extract last comment author, timestamp, and body excerpt (first 200 chars)
  - Write to `comments_summary_ai.md` in format:
    ```
    Last comment from {author} at {timestamp}:

    {excerpt}
    ```

**Failure Handling:**
- If Jira is not configured: Skip (log warning)
- If comments API call fails: Skip (log warning)
- If no comments exist: Skip
- If OpenAI API fails: Use fallback summary
- Comment fetching is **optional** - pipeline continues without it

### 4. Fetch Analysis Insights from Guardian + S3 (Optional)

**Purpose:** Download analysis reports if available.

**APIs:**
- Guardian API: `/api/v1/aiops/guardian/analyses`
- S3: Download insight.json and diagnostic_package.json

**Authentication:**
- Guardian: No auth
- S3: AWS credentials (from environment or ~/.aws/credentials)

**Input:**
- `alert_id` (to query Guardian)
- `tenant_id`, `cluster_id`, `analysis_uuid`, `created_at` (for S3 path)

**Output:** Two files written to workspace:
- `insight.json` (analysis insights)
- `diagnostic_package.json` (diagnostic data)

**Configuration:**
```yaml
aiops_api_url: "http://localhost:9090"
s3_bucket: "tidbcloud-aiops-assets-dev"
aws_region: "us-west-2"
```

**S3 Path Format:**
```
guardian_service/analysis/{tenant_id}/{cluster_id}/{YYYY_MM}/{analysis_uuid}/insight.en.json
guardian_service/analysis/{tenant_id}/{cluster_id}/{YYYY_MM}/{analysis_uuid}/diagnostic_package.json
```

**Failure Handling:**
- If Guardian API not configured: Skip (log info)
- If no analyses found: Skip (log info)
- If S3 download fails: Skip (log warning)
- Analysis fetch is **optional** - pipeline continues without it

### 5. Write Metadata Files

**Purpose:** Finalize workspace setup with all collected data.

**Actions:**
1. Write `cluster_meta.json` to workspace (from Clinic API or fallback)
2. Write `alert_meta.json` to workspace (from Jira issue data)
3. Build file list for frontend (all files in alerts/{alert_id}/ directory)
4. Update task status to "completed" with file list and cluster metadata

**Files Written:**
```
workspace/
└── alerts/
    └── {alert_id}/
        ├── cluster_meta.json          # Cluster information from Clinic
        ├── alert_meta.json            # Alert metadata from Jira
        ├── comments.json              # Jira comments (if available)
        ├── comments_summary_ai.md     # AI summary of comments (if available)
        ├── insight.json               # Guardian analysis (if available)
        └── diagnostic_package.json    # Guardian diagnostics (if available)
```

**Note on .opencode/ directory:**
The `.opencode/opencode.json` configuration file is **NOT** copied from templates during source preparation. Instead, it is written at **session-creation time** by `main.py` using the `_write_workspace_opencode_json()` function, which:
- Resolves absolute paths for MCP tool servers
- Injects dynamic credentials from in-memory cache
- Ensures each session has the correct MCP server configuration

**Output:** Task completion with:
- `status`: "completed"
- `files`: Array of file objects with path, type, and size
- `cluster_meta`: Cluster metadata dict
- `alert_dir`: Relative path to alert directory (e.g., "alerts/O11Y-1956041")

## Decision Flow Diagram

```
Start
  ↓
1. Check Issue Prefix
  ↓
  O11Y-* or TCOC-*?
    YES → Fetch via _fetch_alert_from_jira
          (Try customfield_10149/10157 first, fallback to parsing customfield_10160)
  ↓
  TCID-*?
    YES → Fetch via _fetch_ticket_from_jira
          (Extract customfield_10088 and customfield_10086)
  ↓
  Other prefix?
    → FAIL (Unsupported issue type)
  ↓
Got cluster_id?
  NO → FAIL (cluster_id required)
  YES ↓
2. Fetch Cluster Meta from Clinic
  ↓
Success?
  NO → Use minimal cluster_meta from Jira
  YES → Use full cluster_meta from Clinic
  ↓
3. Fetch Jira Comments
  ↓
Got comments?
  NO → Skip
  YES ↓
    Call OpenAI for AI summary
    ↓
    Success?
      NO → Use fallback summary (last comment)
      YES → Write AI summary
  ↓
4. Got tenant_id + cluster_id?
  NO → Skip Guardian analysis
  YES ↓
    Fetch Analysis from Guardian + S3
    ↓
    Success?
      NO → Continue (analysis optional)
      YES → Continue
  ↓
5. Write cluster_meta.json and alert_meta.json
  ↓
Build file list (all files in alerts/{alert_id}/)
  ↓
Update task status to "completed"
  ↓
Task Complete

Note: .opencode/opencode.json is written at session-creation time by main.py
```

## Configuration Priority

### Required Services

1. **Jira** - Must be configured
   - Provides cluster_id (required for all other steps)
   - Provides tenant_id, org_id

2. **Clinic** - Recommended but not required
   - Enriches cluster metadata
   - Falls back to minimal metadata if unavailable

### Optional Services

3. **Guardian + S3** - Optional
   - Provides analysis insights
   - Pipeline works without it

## Example Configurations

### Minimal Configuration (Jira + Clinic cookie)

```yaml
# investibook.yaml
jira_base_url: "https://pingcap.atlassian.net"
jira_username: "your-email@pingcap.com"
jira_api_token: "ATATT3xFfGF0..."

clinic_endpoint: "https://clinic.pingcap.com"
clinic_cookie: "session=eyJ..."
clinic_csrf_token: "abc123..."
```

**Result:** Gets cluster metadata, no analysis insights

### Full Configuration (All services)

```yaml
# investibook.yaml
# Jira (required)
jira_base_url: "https://pingcap.atlassian.net"
jira_username: "your-email@pingcap.com"
jira_api_token: "ATATT3xFfGF0..."

# Clinic (recommended)
clinic_endpoint: "https://clinic.pingcap.com"
clinic_api_key: "your-api-key"

# LLM for comment summary (direct OpenAI call, no ACP agent)
openai_api_key: "sk-your-openai-key"
openai_model: "gpt-4o"

# Guardian + S3 (optional)
aiops_api_url: "http://localhost:9090"
s3_bucket: "tidbcloud-aiops-assets-dev"
aws_region: "us-west-2"
```

**Result:** Gets cluster metadata + analysis insights

## Error Handling Philosophy

1. **Jira failure → Pipeline fails**
   - cluster_id is essential, can't proceed without it

2. **Clinic failure → Use fallback**
   - Use minimal cluster_meta with data from Jira
   - Pipeline continues

3. **Guardian/S3 failure → Skip analysis**
   - Analysis is optional enhancement
   - Pipeline continues

## Logging

Each step logs its progress:

```
INFO: Fetching alert details from Jira for O11Y-1956041
INFO: Extracted from Jira alert O11Y-1956041: cluster_id=123456, tenant_id=tenant_abc
INFO: Fetching cluster metadata from Clinic API for 123456
INFO: Using Clinic API key authentication
INFO: Fetched cluster_meta: {'tenant_id': 'tenant_abc', 'cluster_id': '123456', ...}
INFO: Fetching comments from Jira for O11Y-1956041
INFO: Fetched 5 comments for Jira alert O11Y-1956041
INFO: Wrote comments.json
INFO: Wrote comments_summary_ai.md via OpenAI gpt-4o
INFO: Fetching analysis for alert O11Y-1956041 from Guardian API
INFO: Selected analysis abc-def-123 (status=COMPLETED)
INFO: S3 keys: insight=guardian_service/analysis/tenant_abc/123456/2024_01/abc-def-123/insight.en.json, diagnostic=...
INFO: Downloading files from S3 bucket tidbcloud-aiops-assets-dev
INFO: Downloaded insight.json (45678 bytes)
INFO: Downloaded diagnostic_package.json (123456 bytes)
INFO: Wrote cluster_meta.json
INFO: Wrote alert_meta.json
INFO: Prepared 6 files for workspace ws_123 (alert dir: alerts/O11Y-1956041)
```

## Testing

To test the pipeline flow:

```bash
cd backend

# Test Jira connection
uv run python -c "
from services.config import settings
print('Jira configured:', bool(settings.jira_username and settings.jira_api_token))
"

# Test Clinic connection
uv run python -c "
from services.config import settings
print('Clinic configured:', bool(settings.clinic_api_key or (settings.clinic_cookie and settings.clinic_csrf_token)))
"

# Test Guardian connection
uv run python -c "
from services.config import settings
print('Guardian configured:', bool(settings.aiops_api_url))
"
```

## See Also

- [CONFIG.md](./CONFIG.md) - Configuration guide
- [CLINIC_AUTH.md](./CLINIC_AUTH.md) - Clinic authentication details
- [backend/services/source_preparer.py](../backend/services/source_preparer.py) - Implementation