# Sentry — Ops Monitoring for NexusCRM

> Status: **Deployed** | Date: 2026-04-12 (spec) / 2026-04-15 (deployed) | Owner: amber
>
> **Live deployment** (nexuscrm AWS account 604175033442, `prod` workspace):
> - Long-running `sentry-service` on `nexuscrm-prod-cluster` (Fargate, 10-min cycle, multi-target prod + staging)
> - Scheduled `sentry-daily-report` via EventBridge `cron(0 1 * * ? *)` = 09:00 UTC+8 daily
> - Managed by `infra/sentry.tf` in `prod` workspace only
> - Ops runbook: [`.agent/skills/sentry-ops/SKILL.md`](../../.agent/skills/sentry-ops/SKILL.md)
>
> **Design deviations from v1 spec** (all resolved — see §10):
> 1. Multi-target service (originally single-target)
> 2. `data.write_stall` rule removed (too noisy on bursty traffic)
> 3. CRIT-only Lark notifications with `[env]` prefix
> 4. Dedicated read-only TiDB user (`ro_nexuscrm` on new prod cluster since 2026-04-16)
>
> **Deployment note:** sentry is deployed **manually from local** (build → ECR push → ECS force-redeploy). CI/CD does NOT cover sentry — see runbook §6.

## 1. Problem Statement

AWS-native observability (CloudWatch, ECS metrics, ALB metrics) already covers infrastructure health — CPU, memory, disk, HTTP 5xx at the load balancer, container restarts. Two classes of failure are **not** covered today:

| Class | Examples | Why AWS misses it |
|-------|----------|-------------------|
| **Abnormal CRM data volume** | A code bug writes 500k rows to `Contact` in an hour; a faulty cleanup job deletes 20% of `Opportunity`; an integration replays webhooks and floods `_System_OutboxEvent` with failed rows | Lives inside TiDB, invisible to ECS/ALB metrics |
| **Go application internal state** | goroutine leak, DB connection pool exhaustion, panic surge caught by recovery middleware, integration queue stall | Process-internal; CloudWatch sees the container is alive but not *what* it is doing |

We need a small, separately-deployed ops service — **sentry** — that continuously watches these two signal classes and alerts on anomalies.

### 1.1 Goals

- Detect abnormal row-count spikes/drops across **all** CRM tables within ~10 minutes
- Detect Go runtime and app-level health regressions (goroutine leak, pool exhaustion, panic surge, integration stall)
- Alert to a dedicated Lark ops channel with **human-readable** descriptions, not raw numbers
- Run fully outside the CRM backend process so it survives CRM outages
- Low operational overhead, predictable cost (< $20/month all-in)

### 1.1.1 Target Environment

**Primary target for v1: `staging`.** Production deployment is a future follow-up once sentry has proven stable on staging.

Staging has a critical operational quirk that the design must accommodate: the **TiDB branch backing staging is reset twice daily from the `main` cluster at 09:00 and 18:00 UTC+8** (i.e., **01:00 and 10:00 UTC**). After each reset, staging's database state snaps to whatever `main` has — every table's row count, `MAX(Id)`, and `MAX(LastModifiedDate)` change discontinuously, with no relationship to the previous cycle.

This breaks a naive delta-based alerting design. See **§3.7 TiDB Branch Reset Handling** for how sentry detects and absorbs these resets without false positives.

### 1.2 Non-Goals

- **Root-cause analysis.** Sentry describes *what* is wrong; a human investigates *why*. LLM output is bounded to fact restatement, never causal speculation.
- **Auto-remediation.** Sentry never writes to TiDB, never calls CRM write APIs, never runs shell. Read-only throughout.
- **Infrastructure metrics.** CPU, memory, disk, network — AWS already covers these.
- **Log aggregation or full-text search.** No CloudWatch Logs scraping, no log file parsing.
- **Historical dashboards.** Out of scope for v1; S3 layout leaves room for Athena + QuickSight later.
- **Daily digest reports.** Alert-only. Quiet on healthy days.

## 2. Architecture

```
┌───────────────────────────────────────┐      ┌──────────────────────────┐
│  CRM Backend (Go) — existing          │      │  sentry (Go) — new        │
│                                       │      │  ECS long-running task    │
│  NEW endpoint (internal + token):     │◄─────│  internal cron: 10 min    │
│  GET /internal/sentry-metrics         │ HTTP │                           │
│       runtime / db_pool / http /      │      │  Each cycle:              │
│       integrations                    │      │  1. Snapshot all tables   │
└───────────────────────────────────────┘      │     (TiDB read-only)      │
                                               │  2. Scrape CRM endpoint   │
┌───────────────────────────────────────┐      │  3. Diff vs in-memory     │
│  TiDB (existing)                      │◄─────│     LRU (last 24h)        │
│  Read-only user for sentry            │  SQL │  4. Evaluate rules        │
└───────────────────────────────────────┘      │  5. If breach:            │
                                               │     → call Qwen           │
┌───────────────────────────────────────┐      │     → post to Lark        │
│  S3: nexus-sentry-{env}/              │◄─────│  6. PUT new snapshot      │
│    snapshots/YYYY/MM/DD/HHMM.json     │      │                           │
│    alerts/YYYY/MM/DD/*.json           │      │                           │
└───────────────────────────────────────┘      └────────┬─────────────────┘
                                                        │
┌───────────────────────────────────────┐               │
│  DashScope (Qwen) — qwen-max          │◄──────────────┤
│  OpenAI-compatible endpoint           │               │
│  API key in AWS Secrets Manager       │               │
└───────────────────────────────────────┘               │
                                                        │
┌───────────────────────────────────────┐               │
│  Lark ops alert channel (webhook bot) │◄──────────────┘
└───────────────────────────────────────┘
```

### 2.1 Why a Separate Service

| Reason | Explanation |
|--------|-------------|
| Survive CRM outages | A monitor embedded inside the thing it monitors goes down with it. Alerting must keep working when CRM is sick. |
| Blast radius isolation | Sentry bugs (memory leak, runaway loop, bad SQL) must not affect production CRM. |
| Independent iteration | Alert rules and thresholds change frequently. We do not want every threshold tweak to require a CRM PR review and full CRM redeploy. |

The only coupling point is the `/internal/sentry-metrics` endpoint on CRM, which is additive and read-only.

### 2.2 Repo Layout

```
MetaDrivenPlatform4AI/
├── backend/          # existing CRM
│   └── internal/ops/sentry_metrics/   # NEW: read-only metrics endpoint
├── sentry/           # NEW: sibling to backend/
│   ├── cmd/sentry/main.go
│   ├── internal/
│   │   ├── collector/  # TiDB + CRM scrapers
│   │   ├── store/      # S3 snapshot store + in-mem LRU
│   │   ├── rules/      # threshold evaluation
│   │   ├── llm/        # Qwen client (OpenAI-compatible)
│   │   ├── notify/     # Lark webhook client
│   │   └── config/     # env + secret loading
│   ├── Dockerfile
│   └── go.mod
└── shared/           # shared constants (existing)
```

Sentry is a standalone Go module with its own `go.mod` and `Dockerfile`. It imports `shared/pkg/constants` for authoritative table/field name lists but does not import `backend/`.

## 3. Component Design

### 3.1 Data Volume Collector

**Scope**: all base tables in the CRM database. Tables are discovered dynamically each cycle — new tables are picked up automatically without a sentry redeploy.

**Discovery query**:
```sql
SELECT TABLE_NAME
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = ?
  AND TABLE_TYPE = 'BASE TABLE'
  AND TABLE_NAME NOT LIKE 'sentry\_%' ESCAPE '\';
```

**Per-table snapshot query** (default path):
```sql
SELECT
  COUNT(*)             AS row_count,
  MAX(`Id`)            AS max_id,
  MAX(`LastModifiedDate`) AS max_updated_at,
  MAX(`CreatedDate`)   AS max_created_at
FROM `{table}`;
```

**Column name convention**: CRM tables follow a Salesforce-style schema with PascalCase columns: `Id` (PK, VARCHAR), `CreatedDate`, `LastModifiedDate`, `SystemModstamp`, `IsDeleted`. Sentry queries use backticks to preserve case. Tables missing `Id` (rare edge case) fall back to `COUNT(*)` only; `max_id`, `max_updated_at`, `max_created_at` are emitted as `NULL` in the snapshot for those tables and delta rules that rely on them are skipped.

**Concurrency**: worker pool of 8 goroutines, per-table query timeout 5 seconds.

**Large table degradation**:
- If a `COUNT(*)` exceeds the 5-second timeout, the table is added to a `slow_tables` set persisted in S3.
- Subsequent cycles use `information_schema.TABLES.TABLE_ROWS` for slow tables instead (TiDB stats-based estimate, single SQL for all slow tables, O(ms)).
- A daily cron at 03:00 UTC retries `COUNT(*)` on slow tables to see if they can be promoted back.
- Slow-table snapshots are tagged `"source": "info_schema_estimate"` so rule evaluation can apply looser thresholds.

**Stats freshness suppression**: when a slow table's estimate jumps because TiDB just ran `ANALYZE`, the first post-refresh delta produces a false spike. Sentry queries `SHOW STATS_HEALTHY WHERE table_name = ?` before evaluation; if `healthy` flipped in the last cycle, the delta alert for that table is suppressed for one cycle and logged to S3 for audit.

**Connection**: dedicated read-only TiDB user `sentry_ro` with `SELECT` only on the CRM database. Credentials in AWS Secrets Manager.

### 3.2 Application Health Collector — CRM Endpoint

**New endpoint** on CRM backend: `GET /internal/sentry-metrics`

**Authentication**:
- Bound to an internal-only listener (separate from the public `0.0.0.0:8080`), on a private port reachable only inside the VPC.
- Bearer token header `X-Sentry-Token`, value in AWS Secrets Manager, shared between CRM and sentry ECS tasks.
- Rate limited to 1 req/sec; sentry calls it every 10 minutes, so this is pure defense against misconfiguration.

**Response schema**:
```json
{
  "collected_at": "2026-04-12T09:20:00Z",
  "version": "abc123de",
  "uptime_sec": 86400,
  "runtime": {
    "goroutines": 2341,
    "heap_alloc_mb": 512,
    "heap_inuse_mb": 480,
    "gc_pause_p99_ms": 12,
    "num_gc": 1234
  },
  "db_pool": {
    "open_conns": 18,
    "in_use": 15,
    "idle": 3,
    "wait_count": 421,
    "wait_duration_ms": 850
  },
  "http": {
    "requests_total": 12345,
    "panic_count": 3,
    "error_5xx_count": 42
  },
  "outbox": {
    "pending": 12,
    "processed": 4321,
    "failed": 3
  }
}
```

`panic_count` and `error_5xx_count` are **monotonic counters** maintained by the CRM process since start. Sentry computes deltas across snapshots. A process restart resets them; sentry detects the reset via decreasing counter and treats the next value as a fresh baseline (no spurious alert).

**Implementation effort on CRM side**: small.
- **Runtime stats** — `runtime.MemStats` + `runtime.NumGoroutine()` directly.
- **DB pool stats** — already available via `TiDBConnection.DB().Stats()` at `backend/internal/infrastructure/database/tidb.go`; the endpoint reads it verbatim.
- **HTTP 5xx** — Prometheus counter `nexuscrm_http_requests_total` (status label) already exists in `backend/internal/interfaces/middleware/metrics.go`. Sentry endpoint exposes the current `5xx` total as an int64 by summing the relevant label values.
- **Panic counter** — **new**. Add `panicRecoveredTotal` counter to `metrics.go` and increment it in `recovery.go` (existing middleware already catches panics; only the counter call is missing).
- **Outbox pending/processed/failed** — query `SELECT Status, COUNT(*) FROM _System_OutboxEvent GROUP BY Status`. Constants already exist in `shared/pkg/constants/enums.go` (`OutboxStatusPending`, `OutboxStatusProcessed`, `OutboxStatusFailed`). This query runs inside the CRM process and uses its existing `*sql.DB`, so no extra connection is needed.

### 3.3 Snapshot Store (S3)

**Layout**:
```
s3://nexus-sentry-{env}/
├── snapshots/YYYY/MM/DD/HHMM.json     # one per 10-min cycle, ~50KB
├── slow_tables.json                    # set of table names, rewritten as needed
└── alerts/YYYY/MM/DD/HHMM-{rule}.json  # one per fired alert, for audit
```

**Snapshot format**:
```json
{
  "snapshot_ts": "2026-04-12T09:20:00Z",
  "sentry_version": "v0.1.0",
  "tables": [
    {
      "name": "contacts",
      "row_count": 523411,
      "max_id": 891234,
      "max_updated_at": "2026-04-12T09:19:58Z",
      "max_created_at": "2026-04-12T09:19:55Z",
      "source": "count_star"
    }
  ],
  "app": { /* full /internal/sentry-metrics payload */ }
}
```

**Cold start**: on boot, sentry lists `snapshots/` keys for the last 48 hours, loads them into an in-memory LRU (cap 300 entries), and uses them for delta computation. Cold start fetches ~288 objects × 50KB ≈ 14MB — completes in seconds.

**Steady state**: 1 PUT per cycle. No GETs (LRU served in-process). S3 request cost is negligible.

**Retention**: S3 lifecycle policy
- 0–90 days: `STANDARD`
- 90–365 days: `GLACIER_DEEP_ARCHIVE`
- 365+ days: delete

Alerts (`alerts/`) use the same lifecycle.

**Cost estimate**: 144 snapshots/day × 50KB = 7.2MB/day = 2.6GB/year. Well under $0.10/month.

### 3.4 Rule Engine

Rules evaluate a snapshot against historical LRU entries (10-minute-ago, 1-hour-ago, 24-hour-ago). Each rule returns either `OK` or an alert with level (`WARN` | `CRIT`) and a structured context payload.

Rules are **data-driven** (a Go slice of `Rule` structs), not hardcoded SQL. Adding a rule = appending to the slice + a unit test.

#### 3.4.1 Data volume rules

| ID | Rule | Threshold | Level |
|----|------|-----------|-------|
| `data.row_spike_warn` | row_count vs 1h ago | `> +50%` **and** delta `> 5000` | WARN |
| `data.row_spike_crit` | row_count vs 1h ago | `> +100%` **or** delta `> 200000` | CRIT |
| `data.row_drop_warn` | row_count vs 1h ago | `< -20%` | WARN |
| `data.row_drop_crit` | row_count vs 1h ago | `< -50%` | CRIT |
| `data.outbox_backlog_warn` | `_System_OutboxEvent` where `Status='pending'` | count `> 50000` | WARN |
| `data.outbox_backlog_crit` | `_System_OutboxEvent` where `Status='pending'` | count `> 200000` | CRIT |
| `data.outbox_failed_surge` | `_System_OutboxEvent` where `Status='failed'` | delta over 1h `> 500` | WARN |

**`data.write_stall` removed (2026-04-13)**: the original spec included a `MaxUpdatedAt > 2h` rule for a fixed set of core tables. The prod smoke test showed that production write volume is bursty — off-hours gaps on `_System_OutboxEvent` and `_System_AuditLog` regularly exceed 2h without representing any real problem. The rule produced too many false positives to keep on. Real outbox / audit problems are still caught by `data.outbox_backlog_*`, `data.outbox_failed_surge`, `data.row_spike_*`, and `data.row_drop_*`.

**Estimated-source tolerance**: when a table's `source = info_schema_estimate`, all `%` thresholds are doubled (e.g., `+50%` becomes `+100%`) to account for TiDB stats noise.

#### 3.4.2 App health rules

| ID | Rule | Threshold | Level |
|----|------|-----------|-------|
| `app.goroutine_leak` | goroutines trend | `+1000` over 1h and no downward dip | WARN |
| `app.heap_growth` | heap_inuse_mb trend | `+100%` over 2h | WARN |
| `app.db_pool_wait` | `wait_count` delta | `> 500` in 10 min | CRIT |
| `app.panic_surge` | `panic_count` delta | `> 20` in 10 min | CRIT |
| `app.http_5xx_warn` | 5xx rate (10-min window) | `> 3%` | WARN |
| `app.http_5xx_crit` | 5xx rate (10-min window) | `> 10%` | CRIT |
| `app.outbox_stall` | `outbox.pending` > 0 and non-decreasing | over 30 min (3 consecutive same-epoch cycles) | CRIT |

#### 3.4.3 Alert suppression

- **Dedup window**: after a rule fires, the same `{rule_id, entity}` pair is suppressed for 60 minutes unless it escalates from WARN → CRIT.
- **Cold-start quiet period**: sentry does not fire alerts in its first 2 cycles after boot (needs history to compare against).
- **Stats-refresh suppression**: see §3.1.
- **TiDB branch reset suppression**: see §3.7. All row-count delta rules are muted from the announced reset time until 20 minutes after it.

### 3.5 LLM Alert Translator (Qwen)

**Purpose**: turn a structured rule breach into a clear Lark message. The LLM **restates facts**, does not speculate causes.

**Model**: `qwen-max` via DashScope OpenAI-compatible endpoint.

**Endpoint**: `https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions`

**Auth**: `Authorization: Bearer ${DASHSCOPE_API_KEY}`, key loaded from AWS Secrets Manager on boot (not env var, not image).

**Prompt shape** (system + user):
```
SYSTEM:
You are an ops alert formatter for a CRM system.
Your job is to restate a structured alert as a clear, concise Lark
message in English. Rules:
1. State the facts ONLY. Do NOT speculate about causes.
2. Do NOT suggest actions beyond "human investigation recommended".
3. Use at most 5 lines. No emoji except the leading severity icon
   (🟡 for WARN, 🔴 for CRIT) which is already provided by the caller.
4. Include the metric name, current value, baseline, and delta.

USER:
{JSON payload of the rule breach}
```

**Call parameters**:
- `temperature: 0.2` (low variance)
- `max_tokens: 200`
- request timeout: 10 seconds
- retries: 2 with exponential backoff (0.5s, 2s)

**Failure degradation**: if all retries fail, sentry posts a **template-based** alert to Lark instead (pre-written format strings per rule ID). Alerting is never blocked by LLM unavailability.

**Cost estimate**: at < 20 alerts/day (typical quiet ops), ~500 input tokens + ~200 output tokens per call. `qwen-max` pricing ~¥0.04/1K input + ~¥0.12/1K output → ~¥0.044/call → ~¥26/month (~$3.60). Well under the $20/month budget.

### 3.6 Lark Notifier

**Transport**: Lark custom-robot webhook (incoming webhook) posted to a dedicated ops alert channel. URL stored in AWS Secrets Manager.

**Message format**: Lark interactive card with:
- Title: `🟡 WARN` or `🔴 CRIT` + rule id
- Body: LLM-translated text (or fallback template)
- Fields: metric name, current, baseline, delta
- Footer: snapshot S3 URL (for audit), sentry version, env

**Signing**: Lark webhook HMAC signing enabled, secret in AWS Secrets Manager alongside the URL.

**Delivery failure**: if Lark POST fails (2 retries), sentry writes the alert to a local queue file on the ECS task's ephemeral volume. A background goroutine retries every minute. If sentry restarts before draining, those alerts are lost — acceptable because the full alert is also archived to `alerts/` in S3, so nothing is silently dropped.

### 3.7 TiDB Branch Reset Handling

The staging TiDB branch is reset from `main` twice daily at **09:00 and 18:00 UTC+8** (**01:00 and 10:00 UTC**). Each reset replaces the entire database state at once. Without explicit handling, every single row-count delta rule would fire simultaneously across every table in the cycle after a reset — catastrophic false-positive storm.

#### 3.7.1 Epoch Model

Sentry treats each inter-reset window as a distinct **epoch**. An epoch is an integer counter incremented at every known reset boundary. Every snapshot is tagged with the epoch it belongs to, and **delta rules only compare snapshots within the same epoch**. Cross-epoch comparisons are definitionally disallowed — there is no meaningful "1 hour ago" across a reset.

```go
type Snapshot struct {
    Ts        time.Time
    EpochID   int64      // increments at each reset boundary
    Tables    []TableRow
    App       AppHealth
}
```

At boot and at every cycle tick, sentry computes the current epoch by counting reset boundaries elapsed since a fixed reference date (e.g., `2026-01-01T00:00:00Z`). Two resets per day → epoch increments deterministically without any external signal.

#### 3.7.2 Reset Suppression Window

For **20 minutes starting at each reset time** (i.e., the first 2 cycles of each new epoch), all row-count delta rules are globally suppressed. Rationale:

1. The reset itself is instantaneous from TiDB's perspective, but CRM backend workers (HubSpot inbound, activity processors) may take several minutes to re-observe the new state and stabilize.
2. The first post-reset snapshot has no same-epoch predecessor to compare against, so delta rules are meaningless anyway.
3. The second snapshot (10 min post-reset) has one predecessor but with only 10 minutes of history — insufficient signal.
4. By the third cycle (20 min post-reset), sentry has two same-epoch snapshots and can compute meaningful 10-minute deltas. Hourly and 24-hourly deltas simply return `no-data` until enough same-epoch history accumulates (60 min, 24h respectively). A `no-data` verdict is **not** an alert — it is a silent skip.

#### 3.7.3 App-Health Rules Are Unaffected

The CRM backend **process** does not restart with the branch reset — only the database state changes. Go runtime metrics (goroutines, heap, GC, DB pool, panic counters, HTTP 5xx counters) are continuous across a reset and are evaluated normally.

The one cross-over concern is outbox state: `_System_OutboxEvent` pending/failed counts are DB-backed, so they change discontinuously at reset. They are covered by the row-count suppression window — no extra handling needed. The `app.outbox_stall` rule (which watches pending non-decreasing over 3 cycles) is also suppressed during the 20-minute window, since its "non-decreasing" check would otherwise spuriously trip on the first post-reset cycle when pending might legitimately jump.

#### 3.7.4 Reset Schedule as Configuration

The reset schedule is declared in sentry config, not hardcoded:

```
SENTRY_RESET_SCHEDULE=01:00,10:00   # UTC, comma-separated HH:MM entries
SENTRY_RESET_WINDOW_MIN=20          # suppression duration
SENTRY_RESET_EPOCH_REFERENCE=2026-01-01T00:00:00Z
```

For production deployment (no TiDB branch reset), `SENTRY_RESET_SCHEDULE` is set to empty. Sentry then runs with a single permanent epoch and no reset suppression. Same code path, environment-driven behavior.

#### 3.7.5 Observability of the Reset Itself

Each reset is logged by sentry with `level=INFO, event=epoch_boundary, old_epoch=N, new_epoch=N+1, suppressed_until=<ts>`. If a reset does **not** actually occur at the scheduled time (TiDB branch ops job fails), sentry has no direct way to detect it — the epoch still advances but snapshots look consistent across the fake boundary. This is an acceptable false-negative: if the reset is missing, staging continues operating on yesterday's state, which is a TiDB-branch-level issue outside sentry's remit.

## 4. Deployment

### 4.1 ECS Task Definition

| Field | Value |
|-------|-------|
| Launch type | Fargate |
| CPU | 256 (.25 vCPU) |
| Memory | 512 MB |
| Task count | 1 (singleton — no concurrency support needed) |
| Network mode | `awsvpc`, private subnet, same VPC as CRM |
| Logging | CloudWatch Logs (`/ecs/sentry`) |
| Platform version | `LATEST` |

Sentry is a long-running process with an internal ticker; it does not rely on EventBridge cron. If the task dies, ECS restarts it and cold-start re-hydrates state from S3.

**Why singleton**: the work is cheap (minutes of CPU per hour) and concurrent writers would clobber the S3 snapshot timeline. An active/standby HA setup is unnecessary for v1 — if sentry is down for a cycle, we miss one 10-min window and recover on next restart. Monitor the monitor via a separate CloudWatch alarm on the ECS task's running count (out of scope for this spec).

### 4.2 IAM

**Task role** (`SentryTaskRole`):
```
s3:GetObject           arn:aws:s3:::nexus-sentry-{env}/*
s3:PutObject           arn:aws:s3:::nexus-sentry-{env}/snapshots/*
s3:PutObject           arn:aws:s3:::nexus-sentry-{env}/alerts/*
s3:PutObject           arn:aws:s3:::nexus-sentry-{env}/slow_tables.json
s3:ListBucket          arn:aws:s3:::nexus-sentry-{env}
secretsmanager:GetSecretValue  arn:aws:secretsmanager:{region}:{acct}:secret:sentry/*
logs:CreateLogStream   arn:aws:logs:{region}:{acct}:log-group:/ecs/sentry:*
logs:PutLogEvents      arn:aws:logs:{region}:{acct}:log-group:/ecs/sentry:*
```

Explicitly **not granted**:
- Any `s3:DeleteObject` — retention handled by lifecycle policy, sentry cannot delete.
- Any RDS/TiDB write permission — sentry uses username/password auth to a read-only TiDB user.
- Any `bedrock:*` — no Bedrock usage.
- Any `ecs:*`, `ec2:*`, `iam:*` — sentry has no need for infrastructure control.

### 4.3 Secrets Manager Entries

| Secret name | Content | Consumers |
|-------------|---------|-----------|
| `sentry/tidb-ro` | `{host, port, user, password, database}` read-only TiDB creds | sentry |
| `sentry/crm-endpoint-token` | Bearer token for `/internal/sentry-metrics` | sentry + CRM |
| `sentry/dashscope-api-key` | Qwen API key | sentry |
| `sentry/lark-webhook` | `{url, signing_secret}` | sentry |

All loaded once at sentry boot, cached in memory, refreshed on 401/403 responses.

### 4.4 Configuration

Environment variables (set on ECS task, non-secret):
```
SENTRY_ENV=staging                       # v1 target; prod deferred
SENTRY_REGION=<same as nexuscrm staging> # confirmed in Phase 1
SENTRY_S3_BUCKET=nexus-sentry-staging
SENTRY_CRM_INTERNAL_URL=http://crm-staging.internal:8081/internal/sentry-metrics
SENTRY_CYCLE_INTERVAL=10m
SENTRY_LRU_CAP=300
SENTRY_DB_WORKER_POOL=8
SENTRY_QUERY_TIMEOUT=5s

# TiDB branch reset handling (see §3.7)
SENTRY_RESET_SCHEDULE=01:00,10:00        # UTC — staging branch resets twice daily
SENTRY_RESET_WINDOW_MIN=20
SENTRY_RESET_EPOCH_REFERENCE=2026-01-01T00:00:00Z
```

For a future production deployment: `SENTRY_ENV=prod`, `SENTRY_S3_BUCKET=nexus-sentry-prod`, and `SENTRY_RESET_SCHEDULE=` (empty) to disable reset suppression.

Secret ARNs (pointers, not values):
```
SENTRY_SECRET_TIDB=arn:aws:secretsmanager:...:secret:sentry/tidb-ro-XXXXX
SENTRY_SECRET_CRM_TOKEN=arn:...:secret:sentry/crm-endpoint-token-XXXXX
SENTRY_SECRET_DASHSCOPE=arn:...:secret:sentry/dashscope-api-key-XXXXX
SENTRY_SECRET_LARK=arn:...:secret:sentry/lark-webhook-XXXXX
```

## 5. Failure Modes and Degradation

| Failure | Impact | Behavior |
|---------|--------|----------|
| TiDB unreachable | No new snapshot | Log error; skip cycle; emit a self-health alert via Lark if > 3 cycles in a row fail |
| CRM `/internal/sentry-metrics` unreachable | No app-health metrics | Continue with data-volume metrics only; emit self-health alert after 3 failures |
| S3 unreachable | Cannot persist or hydrate | Keep in-memory LRU; retry S3 PUT on next cycle; emit self-health alert |
| Qwen API down or slow | LLM translation unavailable | Fall back to template-based alert; alerting is never blocked |
| Lark webhook down | Alert cannot be delivered | Retry 2x; archive alert to S3 regardless; emit CloudWatch metric for ops to notice |
| Secrets Manager down | Cannot boot | Fail loud on startup; ECS restarts; eventually consistent |
| Sentry crash/restart | Missed 1 cycle | Cold-start rehydrates from S3 in seconds; next cycle resumes |
| Slow table | Over 5s COUNT(*) timeout | Degrade to `information_schema` estimate (§3.1) |
| TiDB stats refresh | False spike on estimated tables | Suppressed via `SHOW STATS_HEALTHY` check (§3.1) |
| Monotonic counter reset (CRM restart) | Apparent negative delta | Detected and treated as fresh baseline (§3.2) |
| **TiDB branch reset (staging, 01:00/10:00 UTC)** | Every table's row-count discontinuously changes | Epoch model isolates each inter-reset window; 20-min suppression window mutes all row-count delta rules (§3.7) |
| Reset did not actually fire at scheduled time | Sentry still advances epoch | Silent no-op; staging branch ops is outside sentry's remit |
| Qwen generates speculative output | Misleading alert | Mitigated by low-temperature + strict system prompt; audited via S3 alert archive; not fully preventable |

## 6. Rollout Plan

**Single milestone — deliver v1 complete.** Mode A (daily digest) is out of scope; only alert-mode (Mode B) is built. Work split into review-sized chunks but shipped together once verified.

### Phase 1: CRM endpoint (1–2 days)
- Add `backend/internal/ops/sentry_metrics/` handler
- Bearer-token middleware, internal listener on a second port
- Unit tests for response schema
- Deploy behind a feature flag; verify sentry can scrape a staging instance

### Phase 2: Sentry service skeleton (2–3 days)
- `sentry/` module with config loader, secret fetching, S3 client, TiDB client
- Collector: table discovery, COUNT(*) queries, slow-table degradation
- Snapshot PUT to S3, cold-start LRU hydration
- No rules yet, no alerting yet — verify snapshots accumulate correctly in S3 for 24h

### Phase 3: Rule engine + Lark alerts (2 days)
- Implement rules from §3.4 as data-driven `Rule` slice with unit tests
- Template-based alert formatter (the fallback path, built first because it must work without LLM)
- Lark webhook client with signing
- Alerts routed to a **test channel**, not production, for 48h observation

### Phase 4: Qwen integration (1 day)
- OpenAI-compatible client to DashScope
- Prompt template + cost-per-call telemetry
- Graceful fallback to template when Qwen fails
- Still pointing at test channel

### Phase 5: Cutover (0.5 day)
- Observe test-channel alert volume for one week
- Tune noisy rules (raise thresholds if false-positive rate > 20%)
- Switch Lark webhook URL secret to production ops channel
- Done

**Total estimate**: ~7–10 working days for a single developer. A code-review gate (see project `/review` skill) is required before the Phase 5 cutover.

## 7. Testing Strategy

| Layer | Tests |
|-------|-------|
| Rule engine | Table-driven unit tests covering every rule ID × (below / at / above threshold) × (estimated / exact source). Target: 100% line coverage in `rules/`. |
| Collector | Integration test against a local TiDB with seeded rows: snapshot accuracy, slow-table path, timeout handling. |
| S3 store | LocalStack-backed unit tests for PUT, LIST, lifecycle metadata, LRU rehydration. |
| CRM endpoint | Unit test on handler: token auth, JSON schema validation, `runtime.MemStats` wiring. |
| Qwen client | Mocked HTTP server returning canned and error responses; verify fallback path. |
| Lark client | Mocked webhook endpoint; verify signing, retry, archival-on-failure. |
| End-to-end | Staging: seed a synthetic row-count spike, confirm alert lands in test Lark channel within 10 min. |

Minimum coverage gate: 80% on `sentry/internal/` per project `CLAUDE.md` testing rule.

## 8. Security Considerations

- **Read-only at every layer.** TiDB user is `SELECT`-only. IAM has no delete/write beyond the sentry bucket and only within its own prefixes. CRM endpoint is read-only. Sentry has no remediation tools.
- **Secret hygiene.** No secrets in env vars, image, or code. Secrets Manager ARN references only; values fetched at boot and never logged.
- **Network isolation.** CRM's `/internal/sentry-metrics` binds to a private listener, not the public ALB. Security group allows inbound from sentry's task security group only.
- **PII exposure in LLM calls.** Qwen receives metric names, counts, and table names — **no row content, no user data**. The rule engine never extracts sample rows. Enforced by keeping the `Collector` layer and `LLM` layer strictly separated: collectors produce only aggregates.
- **Audit trail.** Every alert is archived to `s3://.../alerts/` as the canonical record. Lark messages are advisory; S3 is the source of truth.
- **Token rotation.** CRM-sentry bearer token and Qwen API key rotatable via Secrets Manager version updates; sentry picks up the new value within one cycle on auth failure.

## 9. Open Questions

| # | Question | Owner | Status |
|---|----------|-------|--------|
| OQ1 | Exact AWS region for sentry ECS task and S3 bucket | amber | **Resolved**: same region as `nexuscrm-staging` ECS deployment. Confirm exact region name (e.g., `ap-northeast-1`) in Phase 1. |
| OQ2 | ~~Final list of "core tables" for the write-stall rule~~ | amber | **Retired (2026-04-13)**: `data.write_stall` rule was removed entirely after prod smoke-testing showed bursty traffic produced too many off-hours false positives. Core-tables list is no longer needed. |
| OQ3 | Sentry self-alarm via CloudWatch | amber | **Resolved**: **no self-alarm**. Sentry is a single ECS task; if it dies, the next alert is simply missed. Acceptable for v1 staging-only scope. |
| OQ4 | Lark ops channel creation + webhook bot provisioning | amber | **Resolved (2026-04-13)**: webhook `94403f3b-...` then rotated to `392fed29-...` on 2026-04-13. HMAC signing secret in `sentry/lark-webhook`. |

All open questions resolved. The only remaining v1 follow-up is deploying the CRM-side `/internal/sentry-metrics` endpoint to prod so sentry-service's app-health scraper has a real target (currently using `unreachableScraperURL` placeholder via cycle soft-fail).

## 10. Deployed State (as of 2026-04-15)

### 10.1 Target environment

| | Value |
|---|---|
| AWS account | **nexuscrm** (604175033442) |
| Region | `us-east-1` |
| Terraform workspace | `prod` |
| TF state bucket | `nexuscrm-terraform-state-604175033442` |
| ECS cluster | `nexuscrm-prod-cluster` (shared with `nexuscrm-prod-backend`, **resource-isolated**: Fargate tasks run on independent Firecracker micro-VMs) |
| VPC / subnets | private subnets `subnet-0ed2cef671fb8cd4b`, `subnet-09f9da006ced0db58` |
| Security group | `sg-09da7217085de086e` (egress-all) |

### 10.2 Live workloads

| Workload | Type | Binary | Trigger |
|---|---|---|---|
| `sentry-service` | ECS Service (desired=1, Fargate) | `/usr/local/bin/sentry` | Internal 10-min ticker |
| `sentry-daily-report` | ECS RunTask via EventBridge | `/usr/local/bin/sentry-probe` | `cron(0 1 * * ? *)` = 09:00 UTC+8 |

Both use the same image `604175033442.dkr.ecr.us-east-1.amazonaws.com/nexus-sentry:latest` (multi-binary Alpine Docker image, non-root).

**Resource footprint per Fargate task**: 0.25 vCPU × 512 MB. Total monthly cost estimate: ~$10 (service 24/7 + daily-report 30s × 30). Fargate tasks run on **independent Firecracker micro-VMs** — cluster-sharing with CRM backend produces zero resource contention.

### 10.3 Multi-target mode (v2, 2026-04-14)

`sentry-service` reads `sentry/service-targets` (JSON array) and runs **one fully-isolated cycle per target per tick**:

```json
[
  {
    "name": "prod",
    "tidb": {"host": "gateway01.us-east-1.prod.aws.tidbcloud.com", "port": 4000,
             "user": "7h9bWQC1var8dpv.sentry_ro", "password": "...", "database": "nexuscrm"}
  },
  {
    "name": "staging",
    "tidb": {"host": "gateway01.us-east-1.prod.aws.tidbcloud.com", "port": 4000,
             "user": "3inMWnPuRvxvwRf.root", "password": "...", "database": "nexuscrm"}
  }
]
```

Per-target isolation:
- Own `*sql.DB` connection pool with per-target TLS config key (`tidbsentry-<name>`)
- Own `*store.LRU` (history never crosses target boundaries)
- Own `*rules.Suppressor` (dedup state is per-target)
- Own `*store.S3Store` with prefix `snapshots/<name>/` so prod + staging never overwrite each other
- Shared: epoch tracker, rule engine, Qwen client, Lark client, log group

Cycles run **sequentially** per tick (~16s total for 2 targets), well under the 10-min budget. Adding a 3rd target just appends another `serviceTarget` entry to the JSON and force-deploys the service — no code change.

### 10.4 Alert routing (v2 behavior)

- **CRIT only** → Lark. `NotifyOnlyCRIT: true` in the `Cycle` struct skips WARN results from reaching the Notifier.
- WARN results are still evaluated, logged to CloudWatch (`cycle: skip notify (level=WARN, ...)`), and auditable via `aws logs tail`.
- Alert messages are prefixed with `[prod]` / `[staging]` via `Cycle.EnvLabel` so readers of the shared ops channel can tell which cluster fired.

Example Lark message on CRIT:

```
[prod] 🔴 **CRIT** data.row_spike_crit
Row-count spike on `Contact`: now 890,000 rows
(baseline 1h ago: 300,000, +196.7%).
Human investigation recommended.
```

### 10.5 Secrets (resource-only, values managed out-of-band)

| Secret name | Consumer | Format |
|---|---|---|
| `sentry/service-targets` | sentry-service | JSON array `[{name, tidb, crm?}]` |
| `sentry/probe-targets-json` | sentry-daily-report | JSON array `[{name, host, port, user, password, database}]` |
| `sentry/dashscope` | both | Plain Qwen API key string |
| `sentry/lark-webhook` | both | JSON `{url, signing_secret}` |

All four are declared as `aws_secretsmanager_secret` in `infra/sentry.tf` (shell only). Values are bootstrapped via `aws secretsmanager put-secret-value` and **never enter Terraform state or git**. Rotation does not require a `terraform apply`.

### 10.6 Known gaps

1. **CRM `/internal/sentry-metrics` endpoint not yet deployed to prod.** sentry-service scrapes `unreachableScraperURL` (`http://127.0.0.1:9/...`) which fails fast with ECONNREFUSED; the cycle's soft-fail path logs once per target per cycle and continues. `data.*` rules still evaluate normally. Once the endpoint ships, update the target JSON's `crm` field and `update-service --force-new-deployment`.
2. **Staging TiDB still uses admin creds** (`3inMWnPuRvxvwRf.root`). Prod uses dedicated `sentry_ro`. Staging is ephemeral so risk is low, but worth rotating when convenient.
3. **`SHOW STATS_HEALTHY` post-`ANALYZE` suppression** (spec §3.1) not implemented. Doubled tolerance on estimated-source rules absorbs most of the noise in practice.
4. **`slow_tables.json` S3 persistence** for sentry-service is in-memory only — re-learned on each task restart.