# Sentry Ops Monitoring — Implementation Plan

> **Status: ✅ SHIPPED (2026-04-15)** — PR #837 on branch `feat/sentry-ops-monitoring`.
> All 5 phases complete. Live in nexuscrm account (604175033442) `prod` workspace.
> For ops procedures (log inspection, secret rotation, manual task runs, troubleshooting),
> see [`.agent/skills/sentry-ops/SKILL.md`](../../.agent/skills/sentry-ops/SKILL.md).
>
> **Scope evolution during implementation**:
> - v1 spec said single-target, deployed to staging workspace
> - **v2 shipped** multi-target (prod + staging), deployed to prod workspace in the new nexuscrm AWS account
> - `data.write_stall` rule removed after prod smoke-test false positives
> - CRIT-only Lark with `[env]` prefix (noise reduction)
> - Dedicated `sentry_ro` TiDB read-only user

---

**Goal:** Build and deploy a standalone Go service (`sentry`) that snapshots CRM data-volume and app-health every 10 minutes, evaluates threshold rules, and alerts to Lark via Qwen-translated natural language — targeting the nexuscrm `prod` workspace (= the de-facto production deployment in the nexuscrm AWS account).

**Architecture:** Standalone Go module at `sentry/` (sibling to `backend/`, `shared/`), deployed as a singleton Fargate task in the same VPC as CRM. Snapshots persisted to S3, rules evaluated in-process against an LRU cache, alerts translated via DashScope (Qwen) with template fallback. Reads metrics from a new `GET /internal/sentry-metrics` endpoint added to CRM backend. Staging TiDB branch resets twice daily — absorbed by an epoch model with a 20-minute suppression window.

**Tech Stack:** Go 1.24.3, Gin (CRM side only — sentry uses `net/http` directly), `go-sql-driver/mysql`, `aws-sdk-go-v2` (new to repo), DashScope OpenAI-compatible API, Lark custom-robot webhook, AWS S3 + Secrets Manager + ECS Fargate.

**Spec:** [`docs/design/sentry-ops-monitoring.md`](../design/sentry-ops-monitoring.md) — read this first.

---

## File Structure

### New files in `backend/` (Phase 1 — CRM side)

| Path | Purpose |
|------|---------|
| `backend/internal/interfaces/rest/sentry_metrics_handler.go` | Gin handler for `GET /internal/sentry-metrics`. Reads runtime/MemStats, DB pool Stats, Prometheus 5xx counter, outbox counts. |
| `backend/internal/interfaces/middleware/internal_token_auth.go` | Static-bearer-token middleware used only by `/internal/*` routes. |
| `backend/internal/interfaces/rest/sentry_metrics_handler_test.go` | Unit test for handler — token auth, JSON schema, counter reset handling. |
| `backend/internal/interfaces/middleware/internal_token_auth_test.go` | Unit test for token middleware. |

### Modified files in `backend/`

| Path | Change |
|------|--------|
| `backend/internal/interfaces/middleware/metrics.go` | Add `panicRecoveredTotal` Prometheus counter and exported `RecordPanicRecovered()` + `GetHTTP5xxCount()` helpers. |
| `backend/internal/interfaces/middleware/recovery.go` | Call `middleware.RecordPanicRecovered()` inside the existing `defer ... recover()`. |
| `backend/internal/infrastructure/database/tidb.go` | Add `Stats() sql.DBStats` method that returns `tc.db.Stats()`. |
| `backend/cmd/server/routes.go` | Register `/internal/sentry-metrics` under the internal-token middleware group. |

### New files in `sentry/` (Phases 2–5 — new service)

```
sentry/
├── go.mod
├── README.md
├── cmd/
│   └── sentry/
│       └── main.go                       # entry point, wires deps, starts cycle loop
├── internal/
│   ├── config/
│   │   ├── config.go                     # env + secret loading
│   │   └── config_test.go
│   ├── secrets/
│   │   ├── aws_secrets.go                # AWS Secrets Manager client
│   │   └── aws_secrets_test.go
│   ├── collector/
│   │   ├── tidb.go                       # TiDB snapshot collector
│   │   ├── tidb_test.go
│   │   ├── crm.go                        # CRM /internal/sentry-metrics scraper
│   │   └── crm_test.go
│   ├── store/
│   │   ├── s3.go                         # S3 PUT + LIST + hydration
│   │   ├── s3_test.go
│   │   ├── lru.go                        # in-memory snapshot LRU
│   │   └── lru_test.go
│   ├── epoch/
│   │   ├── epoch.go                      # TiDB branch reset epoch model
│   │   └── epoch_test.go
│   ├── rules/
│   │   ├── rules.go                      # rule types + registry
│   │   ├── data.go                       # data.* rules
│   │   ├── app.go                        # app.* rules
│   │   ├── suppress.go                   # dedup + cold-start + epoch-window suppression
│   │   └── rules_test.go                 # table-driven tests per rule
│   ├── llm/
│   │   ├── qwen.go                       # DashScope OpenAI-compatible client
│   │   ├── qwen_test.go
│   │   ├── prompt.go                     # system + user prompt templates
│   │   └── template.go                   # non-LLM fallback formatter
│   ├── notify/
│   │   ├── lark.go                       # Lark webhook client with HMAC signing
│   │   └── lark_test.go
│   ├── types/
│   │   └── types.go                      # Snapshot, TableRow, AppHealth, RuleResult
│   └── cycle/
│       ├── cycle.go                      # one full cycle (collect → diff → alert → persist)
│       └── cycle_test.go
```

### Infrastructure files

| Path | Purpose |
|------|---------|
| `infra/sentry.Dockerfile` | Multi-stage Alpine Go build, mirrors `infra/backend.Dockerfile` |
| `infra/terraform/sentry/*.tf` | ECS task definition, IAM role, S3 bucket + lifecycle, Secrets Manager entries — defer actual TF authoring to Phase 5; this plan only creates placeholders |
| `.github/workflows/ci.yml` | Add `sentry-test` job |
| `.github/workflows/deploy.yml` | Add `sentry-build-and-deploy` job (Phase 5, not in initial PR) |

### Module wiring

`sentry/go.mod`:
```go
module github.com/nexuscrm/sentry

go 1.24.3

require (
    github.com/go-sql-driver/mysql v1.9.0
    github.com/aws/aws-sdk-go-v2 v1.32.0
    github.com/aws/aws-sdk-go-v2/config v1.28.0
    github.com/aws/aws-sdk-go-v2/service/s3 v1.66.0
    github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.34.0
    github.com/nexuscrm/shared v0.0.0-00010101000000-000000000000
)

replace github.com/nexuscrm/shared => ../shared
```

Sentry does **not** import `github.com/nexuscrm/backend` — the only coupling is the HTTP response schema of `/internal/sentry-metrics`, which sentry re-declares as its own struct in `sentry/internal/types/types.go`.

---

## Phase 1: CRM Metrics Endpoint

### Task 1.1: Add DB pool Stats accessor

**Files:**
- Modify: `backend/internal/infrastructure/database/tidb.go`

- [ ] **Step 1: Read current tidb.go to locate the TiDBConnection struct**

Run: `grep -n "type TiDBConnection" backend/internal/infrastructure/database/tidb.go`
Expected: one hit around line 22-24.

- [ ] **Step 2: Add Stats() method after the existing DB() accessor**

```go
// Stats returns a snapshot of database pool statistics.
// Callers use it to expose pool health to monitoring.
func (tc *TiDBConnection) Stats() sql.DBStats {
    return tc.db.Stats()
}
```

- [ ] **Step 3: Build backend to verify**

Run: `cd backend && go build ./internal/infrastructure/database/...`
Expected: no errors.

- [ ] **Step 4: Commit**

```bash
git add backend/internal/infrastructure/database/tidb.go
git commit -m "feat(backend): expose TiDBConnection.Stats() for ops monitoring"
```

---

### Task 1.2: Add panic-recovered Prometheus counter

**Files:**
- Modify: `backend/internal/interfaces/middleware/metrics.go`
- Modify: `backend/internal/interfaces/middleware/recovery.go`

- [ ] **Step 1: Read current metrics.go to find the counter definitions**

Run: `grep -n "prometheus\.New\|promauto\.NewCounter" backend/internal/interfaces/middleware/metrics.go`
Expected: list of existing counters. Note the package and registration pattern.

- [ ] **Step 2: Add panic counter**

In `metrics.go`, add beside the existing counter declarations:

```go
var panicRecoveredTotal = promauto.NewCounter(
    prometheus.CounterOpts{
        Name: "nexuscrm_panic_recovered_total",
        Help: "Total number of panics caught by the recovery middleware.",
    },
)

// RecordPanicRecovered increments the panic counter. Called by recovery middleware.
func RecordPanicRecovered() {
    panicRecoveredTotal.Inc()
}
```

- [ ] **Step 3: Add helper to read current 5xx count**

Still in `metrics.go`:

```go
// GetHTTP5xxCount returns the current total of HTTP responses with status >= 500.
// Implementation sums the `status` label values 500..599 from httpRequestsTotal.
func GetHTTP5xxCount() int64 {
    var total int64
    // Sum the existing Prometheus counter's 5xx children. Iterate 500..599.
    mfs, err := prometheus.DefaultGatherer.Gather()
    if err != nil {
        return -1
    }
    for _, mf := range mfs {
        if mf.GetName() != "nexuscrm_http_requests_total" {
            continue
        }
        for _, m := range mf.Metric {
            for _, lbl := range m.Label {
                if lbl.GetName() == "status" {
                    if code, err := strconv.Atoi(lbl.GetValue()); err == nil && code >= 500 && code < 600 {
                        total += int64(m.Counter.GetValue())
                    }
                }
            }
        }
    }
    return total
}

// GetPanicRecoveredCount returns the current total of recovered panics.
func GetPanicRecoveredCount() int64 {
    mfs, err := prometheus.DefaultGatherer.Gather()
    if err != nil {
        return -1
    }
    for _, mf := range mfs {
        if mf.GetName() == "nexuscrm_panic_recovered_total" {
            for _, m := range mf.Metric {
                return int64(m.Counter.GetValue())
            }
        }
    }
    return 0
}
```

Add `"strconv"` to imports if not present.

- [ ] **Step 4: Wire into recovery middleware**

In `recovery.go`, find the `defer func() { if err := recover(); err != nil { ... } }()` block. Add `RecordPanicRecovered()` before the existing logging:

```go
defer func() {
    if err := recover(); err != nil {
        RecordPanicRecovered()      // ← NEW
        // ... existing log + response code ...
    }
}()
```

- [ ] **Step 5: Build + vet**

Run: `cd backend && go build ./... && go vet ./...`
Expected: clean.

- [ ] **Step 6: Write unit test for counters**

Create `backend/internal/interfaces/middleware/metrics_counters_test.go`:

```go
package middleware

import "testing"

func TestRecordPanicRecovered(t *testing.T) {
    start := GetPanicRecoveredCount()
    RecordPanicRecovered()
    got := GetPanicRecoveredCount()
    if got != start+1 {
        t.Fatalf("want %d got %d", start+1, got)
    }
}

func TestGetHTTP5xxCount_NoneYet(t *testing.T) {
    n := GetHTTP5xxCount()
    if n < 0 {
        t.Fatalf("unexpected error value %d", n)
    }
}
```

Run: `cd backend && go test ./internal/interfaces/middleware/...`
Expected: PASS.

- [ ] **Step 7: Commit**

```bash
git add backend/internal/interfaces/middleware/metrics.go \
        backend/internal/interfaces/middleware/recovery.go \
        backend/internal/interfaces/middleware/metrics_counters_test.go
git commit -m "feat(backend): add panic-recovered Prometheus counter and 5xx/panic read helpers"
```

---

### Task 1.3: Add internal-token auth middleware

**Files:**
- Create: `backend/internal/interfaces/middleware/internal_token_auth.go`
- Create: `backend/internal/interfaces/middleware/internal_token_auth_test.go`

- [ ] **Step 1: Write the failing test**

Create `internal_token_auth_test.go`:

```go
package middleware

import (
    "net/http"
    "net/http/httptest"
    "testing"

    "github.com/gin-gonic/gin"
)

func TestInternalTokenAuth(t *testing.T) {
    gin.SetMode(gin.TestMode)
    cases := []struct {
        name       string
        authHeader string
        wantStatus int
    }{
        {"missing header", "", http.StatusUnauthorized},
        {"wrong scheme", "Basic xyz", http.StatusUnauthorized},
        {"wrong token", "Bearer bad", http.StatusUnauthorized},
        {"correct token", "Bearer right-token", http.StatusOK},
    }
    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            r := gin.New()
            r.Use(InternalTokenAuth("right-token"))
            r.GET("/internal/x", func(c *gin.Context) { c.String(http.StatusOK, "ok") })
            req := httptest.NewRequest(http.MethodGet, "/internal/x", nil)
            if tc.authHeader != "" {
                req.Header.Set("Authorization", tc.authHeader)
            }
            rec := httptest.NewRecorder()
            r.ServeHTTP(rec, req)
            if rec.Code != tc.wantStatus {
                t.Errorf("status = %d, want %d", rec.Code, tc.wantStatus)
            }
        })
    }
}
```

- [ ] **Step 2: Run test to see it fails to compile**

Run: `cd backend && go test ./internal/interfaces/middleware/... -run TestInternalTokenAuth`
Expected: FAIL, "undefined: InternalTokenAuth".

- [ ] **Step 3: Implement the middleware**

Create `internal_token_auth.go`:

```go
package middleware

import (
    "crypto/subtle"
    "net/http"
    "strings"

    "github.com/gin-gonic/gin"
)

// InternalTokenAuth returns a Gin middleware that rejects requests whose
// Authorization header does not carry "Bearer <expected>". Used only for
// /internal/* routes consumed by trusted ops services (e.g. sentry).
//
// The comparison is constant-time to prevent timing oracles, even though
// the token is static — no harm, no cost.
func InternalTokenAuth(expected string) gin.HandlerFunc {
    expectedBytes := []byte(expected)
    return func(c *gin.Context) {
        auth := c.GetHeader("Authorization")
        const prefix = "Bearer "
        if !strings.HasPrefix(auth, prefix) {
            c.AbortWithStatus(http.StatusUnauthorized)
            return
        }
        got := []byte(strings.TrimPrefix(auth, prefix))
        if subtle.ConstantTimeCompare(got, expectedBytes) != 1 {
            c.AbortWithStatus(http.StatusUnauthorized)
            return
        }
        c.Next()
    }
}
```

- [ ] **Step 4: Run test again**

Run: `cd backend && go test ./internal/interfaces/middleware/... -run TestInternalTokenAuth -v`
Expected: all 4 subtests PASS.

- [ ] **Step 5: Commit**

```bash
git add backend/internal/interfaces/middleware/internal_token_auth.go \
        backend/internal/interfaces/middleware/internal_token_auth_test.go
git commit -m "feat(backend): add constant-time internal bearer-token middleware"
```

---

### Task 1.4: Build the sentry-metrics handler

**Files:**
- Create: `backend/internal/interfaces/rest/sentry_metrics_handler.go`
- Create: `backend/internal/interfaces/rest/sentry_metrics_handler_test.go`

- [ ] **Step 1: Write failing test first**

Create `sentry_metrics_handler_test.go`:

```go
package rest

import (
    "encoding/json"
    "net/http"
    "net/http/httptest"
    "testing"

    "github.com/gin-gonic/gin"
)

func TestSentryMetricsHandler_Schema(t *testing.T) {
    gin.SetMode(gin.TestMode)
    h := NewSentryMetricsHandler(nil /* nilDB */, "v-test")
    r := gin.New()
    r.GET("/internal/sentry-metrics", h.Get)

    rec := httptest.NewRecorder()
    req := httptest.NewRequest(http.MethodGet, "/internal/sentry-metrics", nil)
    r.ServeHTTP(rec, req)

    if rec.Code != http.StatusOK {
        t.Fatalf("status = %d, want 200", rec.Code)
    }
    var body map[string]any
    if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
        t.Fatalf("bad json: %v", err)
    }
    for _, k := range []string{"collected_at", "version", "uptime_sec", "runtime", "db_pool", "http", "outbox"} {
        if _, ok := body[k]; !ok {
            t.Errorf("missing field %q", k)
        }
    }
    if body["version"].(string) != "v-test" {
        t.Errorf("version = %v", body["version"])
    }
}
```

- [ ] **Step 2: Run test — expect compile failure**

Run: `cd backend && go test ./internal/interfaces/rest/... -run TestSentryMetricsHandler_Schema`
Expected: FAIL, "undefined: NewSentryMetricsHandler".

- [ ] **Step 3: Implement the handler**

Create `sentry_metrics_handler.go`:

```go
package rest

import (
    "context"
    "database/sql"
    "net/http"
    "runtime"
    "time"

    "github.com/gin-gonic/gin"

    "github.com/nexuscrm/backend/internal/interfaces/middleware"
)

type SentryMetricsHandler struct {
    db        *sql.DB
    version   string
    startedAt time.Time
}

func NewSentryMetricsHandler(db *sql.DB, version string) *SentryMetricsHandler {
    return &SentryMetricsHandler{
        db:        db,
        version:   version,
        startedAt: time.Now(),
    }
}

// Get handles GET /internal/sentry-metrics.
//
// Returns a JSON snapshot of process-internal state: Go runtime,
// DB pool stats, HTTP error counters, and outbox queue counts.
// Sentry polls this every 10 minutes. See
// docs/design/sentry-ops-monitoring.md §3.2.
func (h *SentryMetricsHandler) Get(c *gin.Context) {
    var mem runtime.MemStats
    runtime.ReadMemStats(&mem)

    var pool sql.DBStats
    if h.db != nil {
        pool = h.db.Stats()
    }

    outbox := h.queryOutboxCounts(c.Request.Context())

    resp := gin.H{
        "collected_at": time.Now().UTC().Format(time.RFC3339),
        "version":      h.version,
        "uptime_sec":   int64(time.Since(h.startedAt).Seconds()),
        "runtime": gin.H{
            "goroutines":      runtime.NumGoroutine(),
            "heap_alloc_mb":   mem.HeapAlloc / 1024 / 1024,
            "heap_inuse_mb":   mem.HeapInuse / 1024 / 1024,
            "gc_pause_p99_ms": lastGCPauseMS(&mem),
            "num_gc":          mem.NumGC,
        },
        "db_pool": gin.H{
            "open_conns":       pool.OpenConnections,
            "in_use":           pool.InUse,
            "idle":             pool.Idle,
            "wait_count":       pool.WaitCount,
            "wait_duration_ms": pool.WaitDuration.Milliseconds(),
        },
        "http": gin.H{
            "requests_total":   middleware.GetHTTPRequestsTotalCount(),
            "error_5xx_count":  middleware.GetHTTP5xxCount(),
            "panic_count":      middleware.GetPanicRecoveredCount(),
        },
        "outbox": outbox,
    }
    c.JSON(http.StatusOK, resp)
}

// lastGCPauseMS returns the most recent GC pause in milliseconds.
func lastGCPauseMS(m *runtime.MemStats) uint64 {
    if m.NumGC == 0 {
        return 0
    }
    idx := (m.NumGC + 255) % 256
    return m.PauseNs[idx] / 1_000_000
}

// queryOutboxCounts returns pending/processed/failed counts grouped by Status.
// Uses a 2-second query timeout; missing rows are treated as 0.
// On any error returns a map with all-zero counts — the endpoint must never fail
// because of an outbox query error.
func (h *SentryMetricsHandler) queryOutboxCounts(ctx context.Context) gin.H {
    result := gin.H{"pending": 0, "processed": 0, "failed": 0}
    if h.db == nil {
        return result
    }
    ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
    defer cancel()
    rows, err := h.db.QueryContext(ctx, "SELECT `Status`, COUNT(*) FROM `_System_OutboxEvent` GROUP BY `Status`")
    if err != nil {
        return result
    }
    defer rows.Close()
    for rows.Next() {
        var status string
        var count int64
        if err := rows.Scan(&status, &count); err != nil {
            continue
        }
        switch status {
        case "pending":
            result["pending"] = count
        case "processed":
            result["processed"] = count
        case "failed":
            result["failed"] = count
        }
    }
    return result
}
```

Also add `GetHTTPRequestsTotalCount()` helper to `metrics.go` (same pattern as `GetHTTP5xxCount`).

- [ ] **Step 4: Run test**

Run: `cd backend && go test ./internal/interfaces/rest/... -run TestSentryMetricsHandler_Schema -v`
Expected: PASS (skips outbox query because `db == nil`).

- [ ] **Step 5: Commit**

```bash
git add backend/internal/interfaces/rest/sentry_metrics_handler.go \
        backend/internal/interfaces/rest/sentry_metrics_handler_test.go \
        backend/internal/interfaces/middleware/metrics.go
git commit -m "feat(backend): add /internal/sentry-metrics handler"
```

---

### Task 1.5: Register the route behind token middleware

**Files:**
- Modify: `backend/cmd/server/routes.go`
- Modify: `backend/cmd/server/main.go` (only if env-var loading needed there)

- [ ] **Step 1: Locate route registration area**

Run: `grep -n "registerRoutes\|/internal\|/health\|/metrics" backend/cmd/server/routes.go`

- [ ] **Step 2: Add the route group**

At the end of `registerRoutes`, add:

```go
if token := os.Getenv("SENTRY_INTERNAL_TOKEN"); token != "" {
    sentryMetricsHandler := rest.NewSentryMetricsHandler(db, version)
    internal := router.Group("/internal", middleware.InternalTokenAuth(token))
    internal.GET("/sentry-metrics", sentryMetricsHandler.Get)
}
```

If `SENTRY_INTERNAL_TOKEN` is unset (e.g., in tests), the route is simply not registered — fail-closed.

- [ ] **Step 3: Build**

Run: `cd backend && go build ./cmd/server/...`
Expected: clean.

- [ ] **Step 4: Smoke test locally**

```bash
cd backend
export SENTRY_INTERNAL_TOKEN=test-token
go run ./cmd/server/ &  # wait for it to come up
curl -sS -H "Authorization: Bearer test-token" http://localhost:3001/internal/sentry-metrics | head -c 500
curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:3001/internal/sentry-metrics  # expect 401
kill %1
```
Expected: first curl returns JSON with all 7 top-level keys; second returns `401`.

- [ ] **Step 5: Commit**

```bash
git add backend/cmd/server/routes.go
git commit -m "feat(backend): register /internal/sentry-metrics route"
```

---

## Phase 2: Sentry Service Skeleton

### Task 2.1: Initialize sentry module

**Files:**
- Create: `sentry/go.mod`
- Create: `sentry/cmd/sentry/main.go`
- Create: `sentry/README.md`

- [ ] **Step 1: Create module directory and go.mod**

```bash
mkdir -p sentry/cmd/sentry sentry/internal/{config,secrets,collector,store,epoch,rules,llm,notify,types,cycle}
cd sentry
cat > go.mod <<'EOF'
module github.com/nexuscrm/sentry

go 1.24.3

require (
    github.com/go-sql-driver/mysql v1.9.0
    github.com/aws/aws-sdk-go-v2 v1.32.0
    github.com/aws/aws-sdk-go-v2/config v1.28.0
    github.com/aws/aws-sdk-go-v2/service/s3 v1.66.0
    github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.34.0
    github.com/nexuscrm/shared v0.0.0-00010101000000-000000000000
)

replace github.com/nexuscrm/shared => ../shared
EOF
```

- [ ] **Step 2: Stub main.go**

Create `sentry/cmd/sentry/main.go`:

```go
package main

import (
    "context"
    "log"
    "os"
    "os/signal"
    "syscall"
)

func main() {
    ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
    defer cancel()

    log.Printf("sentry starting: env=%s region=%s", os.Getenv("SENTRY_ENV"), os.Getenv("SENTRY_REGION"))

    <-ctx.Done()
    log.Printf("sentry shutting down")
}
```

- [ ] **Step 3: Download deps and build**

```bash
cd sentry && go mod tidy && go build ./...
```
Expected: `sentry` binary builds; no errors.

- [ ] **Step 4: Commit**

```bash
git add sentry/
git commit -m "feat(sentry): scaffold module skeleton"
```

---

### Task 2.2: Types package

**Files:**
- Create: `sentry/internal/types/types.go`
- Create: `sentry/internal/types/types_test.go`

- [ ] **Step 1: Write failing test**

`types_test.go`:

```go
package types

import (
    "encoding/json"
    "testing"
    "time"
)

func TestSnapshotRoundTrip(t *testing.T) {
    s := Snapshot{
        SnapshotTs: time.Date(2026, 4, 12, 9, 20, 0, 0, time.UTC),
        EpochID:    42,
        Version:    "v0.1.0",
        Tables: []TableRow{
            {Name: "Contact", RowCount: 100, MaxID: ptrStr("abc"), Source: SourceCountStar},
        },
        App: AppHealth{Runtime: RuntimeStats{Goroutines: 10}},
    }
    b, err := json.Marshal(s)
    if err != nil { t.Fatal(err) }
    var out Snapshot
    if err := json.Unmarshal(b, &out); err != nil { t.Fatal(err) }
    if out.EpochID != 42 || len(out.Tables) != 1 || out.Tables[0].Name != "Contact" {
        t.Fatalf("roundtrip mismatch: %+v", out)
    }
}

func ptrStr(s string) *string { return &s }
```

- [ ] **Step 2: Implement types.go**

```go
package types

import "time"

type Source string

const (
    SourceCountStar       Source = "count_star"
    SourceInfoSchemaEst   Source = "info_schema_estimate"
)

type Snapshot struct {
    SnapshotTs time.Time  `json:"snapshot_ts"`
    EpochID    int64      `json:"epoch_id"`
    Version    string     `json:"sentry_version"`
    Tables     []TableRow `json:"tables"`
    App        AppHealth  `json:"app"`
}

type TableRow struct {
    Name          string     `json:"name"`
    RowCount      int64      `json:"row_count"`
    MaxID         *string    `json:"max_id,omitempty"`
    MaxUpdatedAt  *time.Time `json:"max_updated_at,omitempty"`
    MaxCreatedAt  *time.Time `json:"max_created_at,omitempty"`
    Source        Source     `json:"source"`
}

type AppHealth struct {
    CollectedAt time.Time    `json:"collected_at"`
    Version     string       `json:"version"`
    UptimeSec   int64        `json:"uptime_sec"`
    Runtime     RuntimeStats `json:"runtime"`
    DBPool      DBPoolStats  `json:"db_pool"`
    HTTP        HTTPStats    `json:"http"`
    Outbox      OutboxStats  `json:"outbox"`
}

type RuntimeStats struct {
    Goroutines    int    `json:"goroutines"`
    HeapAllocMB   uint64 `json:"heap_alloc_mb"`
    HeapInuseMB   uint64 `json:"heap_inuse_mb"`
    GCPauseP99MS  uint64 `json:"gc_pause_p99_ms"`
    NumGC         uint32 `json:"num_gc"`
}

type DBPoolStats struct {
    OpenConns       int   `json:"open_conns"`
    InUse           int   `json:"in_use"`
    Idle            int   `json:"idle"`
    WaitCount       int64 `json:"wait_count"`
    WaitDurationMS  int64 `json:"wait_duration_ms"`
}

type HTTPStats struct {
    RequestsTotal int64 `json:"requests_total"`
    Error5xxCount int64 `json:"error_5xx_count"`
    PanicCount    int64 `json:"panic_count"`
}

type OutboxStats struct {
    Pending   int64 `json:"pending"`
    Processed int64 `json:"processed"`
    Failed    int64 `json:"failed"`
}

// RuleResult is the output of evaluating a single rule against a snapshot.
type RuleResult struct {
    RuleID   string         `json:"rule_id"`
    Entity   string         `json:"entity"`    // table name, integration name, or ""
    Level    string         `json:"level"`     // "WARN" or "CRIT"
    Breached bool           `json:"breached"`
    Context  map[string]any `json:"context"`   // for LLM + Lark rendering
}
```

- [ ] **Step 3: Run test**

Run: `cd sentry && go test ./internal/types/...`
Expected: PASS.

- [ ] **Step 4: Commit**

```bash
git add sentry/internal/types/
git commit -m "feat(sentry): add core types"
```

---

### Task 2.3: Config loader

**Files:**
- Create: `sentry/internal/config/config.go`
- Create: `sentry/internal/config/config_test.go`

- [ ] **Step 1: Failing test**

```go
package config

import (
    "os"
    "testing"
    "time"
)

func TestLoadFromEnv(t *testing.T) {
    t.Setenv("SENTRY_ENV", "staging")
    t.Setenv("SENTRY_REGION", "ap-northeast-1")
    t.Setenv("SENTRY_S3_BUCKET", "nexus-sentry-staging")
    t.Setenv("SENTRY_CRM_INTERNAL_URL", "http://crm.internal:8081/internal/sentry-metrics")
    t.Setenv("SENTRY_CYCLE_INTERVAL", "10m")
    t.Setenv("SENTRY_RESET_SCHEDULE", "01:00,10:00")
    t.Setenv("SENTRY_RESET_WINDOW_MIN", "20")
    t.Setenv("SENTRY_RESET_EPOCH_REFERENCE", "2026-01-01T00:00:00Z")

    cfg, err := Load()
    if err != nil { t.Fatal(err) }
    if cfg.Env != "staging" { t.Errorf("env=%s", cfg.Env) }
    if cfg.CycleInterval != 10*time.Minute { t.Errorf("interval=%v", cfg.CycleInterval) }
    if len(cfg.ResetSchedule) != 2 { t.Errorf("schedule len=%d", len(cfg.ResetSchedule)) }
    _ = os.Unsetenv
}

func TestLoadMissingRequired(t *testing.T) {
    t.Setenv("SENTRY_ENV", "")
    _, err := Load()
    if err == nil { t.Fatal("want error for missing env") }
}
```

- [ ] **Step 2: Implement**

```go
package config

import (
    "errors"
    "fmt"
    "os"
    "strconv"
    "strings"
    "time"
)

type Config struct {
    Env                 string
    Region              string
    S3Bucket            string
    CRMInternalURL      string
    CycleInterval       time.Duration
    LRUCap              int
    DBWorkerPool        int
    QueryTimeout        time.Duration
    ResetSchedule       []ResetTime
    ResetWindow         time.Duration
    ResetEpochReference time.Time

    // Secret ARNs (values fetched separately from Secrets Manager)
    SecretARNTiDB       string
    SecretARNCRMToken   string
    SecretARNDashScope  string
    SecretARNLark       string
}

type ResetTime struct {
    Hour   int
    Minute int
}

func Load() (*Config, error) {
    env := os.Getenv("SENTRY_ENV")
    if env == "" {
        return nil, errors.New("SENTRY_ENV is required")
    }
    cycleIntervalStr := getenvDefault("SENTRY_CYCLE_INTERVAL", "10m")
    cycleInterval, err := time.ParseDuration(cycleIntervalStr)
    if err != nil {
        return nil, fmt.Errorf("SENTRY_CYCLE_INTERVAL: %w", err)
    }
    windowMin, err := strconv.Atoi(getenvDefault("SENTRY_RESET_WINDOW_MIN", "20"))
    if err != nil {
        return nil, fmt.Errorf("SENTRY_RESET_WINDOW_MIN: %w", err)
    }
    refStr := getenvDefault("SENTRY_RESET_EPOCH_REFERENCE", "2026-01-01T00:00:00Z")
    ref, err := time.Parse(time.RFC3339, refStr)
    if err != nil {
        return nil, fmt.Errorf("SENTRY_RESET_EPOCH_REFERENCE: %w", err)
    }
    schedule, err := parseResetSchedule(os.Getenv("SENTRY_RESET_SCHEDULE"))
    if err != nil {
        return nil, err
    }

    return &Config{
        Env:                 env,
        Region:              os.Getenv("SENTRY_REGION"),
        S3Bucket:            os.Getenv("SENTRY_S3_BUCKET"),
        CRMInternalURL:      os.Getenv("SENTRY_CRM_INTERNAL_URL"),
        CycleInterval:       cycleInterval,
        LRUCap:              getIntDefault("SENTRY_LRU_CAP", 300),
        DBWorkerPool:        getIntDefault("SENTRY_DB_WORKER_POOL", 8),
        QueryTimeout:        getDurDefault("SENTRY_QUERY_TIMEOUT", 5*time.Second),
        ResetSchedule:       schedule,
        ResetWindow:         time.Duration(windowMin) * time.Minute,
        ResetEpochReference: ref,
        SecretARNTiDB:       os.Getenv("SENTRY_SECRET_TIDB"),
        SecretARNCRMToken:   os.Getenv("SENTRY_SECRET_CRM_TOKEN"),
        SecretARNDashScope:  os.Getenv("SENTRY_SECRET_DASHSCOPE"),
        SecretARNLark:       os.Getenv("SENTRY_SECRET_LARK"),
    }, nil
}

func parseResetSchedule(s string) ([]ResetTime, error) {
    s = strings.TrimSpace(s)
    if s == "" {
        return nil, nil // production: no reset
    }
    var out []ResetTime
    for _, part := range strings.Split(s, ",") {
        part = strings.TrimSpace(part)
        hm := strings.SplitN(part, ":", 2)
        if len(hm) != 2 {
            return nil, fmt.Errorf("invalid reset entry %q", part)
        }
        h, err := strconv.Atoi(hm[0])
        if err != nil || h < 0 || h > 23 { return nil, fmt.Errorf("bad hour %q", part) }
        m, err := strconv.Atoi(hm[1])
        if err != nil || m < 0 || m > 59 { return nil, fmt.Errorf("bad minute %q", part) }
        out = append(out, ResetTime{Hour: h, Minute: m})
    }
    return out, nil
}

func getenvDefault(k, d string) string {
    if v := os.Getenv(k); v != "" { return v }
    return d
}

func getIntDefault(k string, d int) int {
    if v := os.Getenv(k); v != "" {
        if n, err := strconv.Atoi(v); err == nil { return n }
    }
    return d
}

func getDurDefault(k string, d time.Duration) time.Duration {
    if v := os.Getenv(k); v != "" {
        if n, err := time.ParseDuration(v); err == nil { return n }
    }
    return d
}
```

- [ ] **Step 3: Run tests**

Run: `cd sentry && go test ./internal/config/... -v`
Expected: PASS both.

- [ ] **Step 4: Commit**

```bash
git add sentry/internal/config/
git commit -m "feat(sentry): config loader with env + reset schedule parsing"
```

---

### Task 2.4: Epoch model

**Files:**
- Create: `sentry/internal/epoch/epoch.go`
- Create: `sentry/internal/epoch/epoch_test.go`

- [ ] **Step 1: Failing test — cover boundary math, no-reset mode, suppression window**

```go
package epoch

import (
    "testing"
    "time"

    "github.com/nexuscrm/sentry/internal/config"
)

func TestComputeEpoch(t *testing.T) {
    ref := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
    sched := []config.ResetTime{{Hour: 1, Minute: 0}, {Hour: 10, Minute: 0}}
    tracker := New(sched, ref, 20*time.Minute)

    cases := []struct {
        at     time.Time
        wantID int64
    }{
        {time.Date(2026, 1, 1, 0, 30, 0, 0, time.UTC), 0},   // before first reset
        {time.Date(2026, 1, 1, 1, 0, 0, 0, time.UTC), 1},    // at 01:00
        {time.Date(2026, 1, 1, 5, 0, 0, 0, time.UTC), 1},    // between resets
        {time.Date(2026, 1, 1, 10, 0, 0, 0, time.UTC), 2},   // at 10:00
        {time.Date(2026, 1, 2, 0, 30, 0, 0, time.UTC), 2},   // day 2 before 01:00
        {time.Date(2026, 1, 2, 1, 0, 0, 0, time.UTC), 3},    // day 2 01:00
    }
    for _, tc := range cases {
        if got := tracker.EpochAt(tc.at); got != tc.wantID {
            t.Errorf("at %v: got %d want %d", tc.at, got, tc.wantID)
        }
    }
}

func TestInSuppressionWindow(t *testing.T) {
    ref := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
    sched := []config.ResetTime{{Hour: 1, Minute: 0}}
    tracker := New(sched, ref, 20*time.Minute)

    cases := []struct {
        at    time.Time
        want  bool
    }{
        {time.Date(2026, 1, 1, 0, 59, 0, 0, time.UTC), false}, // before reset
        {time.Date(2026, 1, 1, 1, 0, 0, 0, time.UTC),  true},  // at reset
        {time.Date(2026, 1, 1, 1, 10, 0, 0, time.UTC), true},  // inside window
        {time.Date(2026, 1, 1, 1, 20, 0, 0, time.UTC), false}, // exactly at window end
        {time.Date(2026, 1, 1, 1, 30, 0, 0, time.UTC), false}, // after window
    }
    for _, tc := range cases {
        if got := tracker.InSuppressionWindow(tc.at); got != tc.want {
            t.Errorf("at %v: got %v want %v", tc.at, got, tc.want)
        }
    }
}

func TestNoReset_SingleEpoch(t *testing.T) {
    ref := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
    tracker := New(nil, ref, 20*time.Minute)
    if tracker.EpochAt(time.Now()) != 0 {
        t.Error("no-reset mode should stay in epoch 0 forever")
    }
    if tracker.InSuppressionWindow(time.Now()) {
        t.Error("no-reset mode should never be in a suppression window")
    }
}
```

- [ ] **Step 2: Implement**

```go
package epoch

import (
    "sort"
    "time"

    "github.com/nexuscrm/sentry/internal/config"
)

type Tracker struct {
    schedule  []config.ResetTime // sorted by hour*60+minute
    reference time.Time
    window    time.Duration
}

func New(schedule []config.ResetTime, reference time.Time, window time.Duration) *Tracker {
    sorted := append([]config.ResetTime(nil), schedule...)
    sort.Slice(sorted, func(i, j int) bool {
        return sorted[i].Hour*60+sorted[i].Minute < sorted[j].Hour*60+sorted[j].Minute
    })
    return &Tracker{schedule: sorted, reference: reference, window: window}
}

// EpochAt returns the epoch ID for the given time.
// Epoch 0 is the window before the first reset after reference.
func (t *Tracker) EpochAt(at time.Time) int64 {
    if len(t.schedule) == 0 {
        return 0
    }
    if at.Before(t.reference) {
        return 0
    }
    // Count how many scheduled reset timestamps have occurred on or before `at`.
    // Each reset bumps the epoch by 1.
    day := t.reference.Truncate(24 * time.Hour)
    count := int64(0)
    for !day.After(at) {
        for _, rt := range t.schedule {
            boundary := time.Date(day.Year(), day.Month(), day.Day(), rt.Hour, rt.Minute, 0, 0, time.UTC)
            if boundary.Before(t.reference) {
                continue
            }
            if !boundary.After(at) {
                count++
            }
        }
        day = day.Add(24 * time.Hour)
    }
    return count
}

// InSuppressionWindow reports whether `at` falls within [boundary, boundary+window)
// of any reset in the schedule. Returns false if the schedule is empty.
func (t *Tracker) InSuppressionWindow(at time.Time) bool {
    if len(t.schedule) == 0 {
        return false
    }
    day := at.Truncate(24 * time.Hour)
    for _, rt := range t.schedule {
        boundary := time.Date(day.Year(), day.Month(), day.Day(), rt.Hour, rt.Minute, 0, 0, time.UTC)
        if !at.Before(boundary) && at.Before(boundary.Add(t.window)) {
            return true
        }
    }
    return false
}
```

- [ ] **Step 3: Run tests**

Run: `cd sentry && go test ./internal/epoch/... -v`
Expected: all 3 tests PASS.

- [ ] **Step 4: Commit**

```bash
git add sentry/internal/epoch/
git commit -m "feat(sentry): epoch tracker for TiDB branch reset handling"
```

---

### Task 2.5: LRU snapshot store (in-memory)

**Files:**
- Create: `sentry/internal/store/lru.go`
- Create: `sentry/internal/store/lru_test.go`

- [ ] **Step 1: Failing test — bounded size, retrieval by ts, same-epoch filtering**

```go
package store

import (
    "testing"
    "time"

    "github.com/nexuscrm/sentry/internal/types"
)

func TestLRU_AddAndBound(t *testing.T) {
    lru := NewLRU(3)
    for i := 0; i < 5; i++ {
        lru.Add(types.Snapshot{
            SnapshotTs: time.Unix(int64(i), 0),
            EpochID:    0,
        })
    }
    if lru.Len() != 3 {
        t.Errorf("len=%d want 3", lru.Len())
    }
    // Oldest two should be evicted
    if _, ok := lru.At(time.Unix(0, 0)); ok {
        t.Error("ts=0 should be evicted")
    }
    if _, ok := lru.At(time.Unix(4, 0)); !ok {
        t.Error("ts=4 should remain")
    }
}

func TestLRU_FindWithin(t *testing.T) {
    lru := NewLRU(10)
    base := time.Date(2026, 4, 12, 9, 0, 0, 0, time.UTC)
    for i := 0; i < 10; i++ {
        lru.Add(types.Snapshot{
            SnapshotTs: base.Add(time.Duration(i) * 10 * time.Minute),
            EpochID:    1,
        })
    }
    // Find snapshot closest to 1 hour before the latest, same epoch
    target := base.Add(90 * time.Minute).Add(-1 * time.Hour) // = base + 30m
    snap, ok := lru.FindClosest(target, 1)
    if !ok { t.Fatal("not found") }
    if snap.SnapshotTs != base.Add(30*time.Minute) {
        t.Errorf("got %v want %v", snap.SnapshotTs, base.Add(30*time.Minute))
    }
}

func TestLRU_FindClosest_WrongEpoch(t *testing.T) {
    lru := NewLRU(10)
    base := time.Date(2026, 4, 12, 9, 0, 0, 0, time.UTC)
    lru.Add(types.Snapshot{SnapshotTs: base, EpochID: 5})
    if _, ok := lru.FindClosest(base, 6); ok {
        t.Error("different epoch must not match")
    }
}
```

- [ ] **Step 2: Implement**

```go
package store

import (
    "sort"
    "sync"
    "time"

    "github.com/nexuscrm/sentry/internal/types"
)

type LRU struct {
    mu   sync.RWMutex
    cap  int
    data []types.Snapshot // time-ordered ascending
}

func NewLRU(cap int) *LRU { return &LRU{cap: cap} }

func (l *LRU) Add(s types.Snapshot) {
    l.mu.Lock()
    defer l.mu.Unlock()
    l.data = append(l.data, s)
    sort.SliceStable(l.data, func(i, j int) bool {
        return l.data[i].SnapshotTs.Before(l.data[j].SnapshotTs)
    })
    if len(l.data) > l.cap {
        l.data = l.data[len(l.data)-l.cap:]
    }
}

func (l *LRU) Len() int {
    l.mu.RLock()
    defer l.mu.RUnlock()
    return len(l.data)
}

func (l *LRU) At(ts time.Time) (types.Snapshot, bool) {
    l.mu.RLock()
    defer l.mu.RUnlock()
    for _, s := range l.data {
        if s.SnapshotTs.Equal(ts) {
            return s, true
        }
    }
    return types.Snapshot{}, false
}

// FindClosest returns the snapshot in the given epoch whose SnapshotTs is nearest to target.
// Only same-epoch snapshots are considered. Returns ok=false if no same-epoch snapshot exists.
func (l *LRU) FindClosest(target time.Time, epoch int64) (types.Snapshot, bool) {
    l.mu.RLock()
    defer l.mu.RUnlock()
    var best types.Snapshot
    var bestDelta time.Duration = 1<<62
    found := false
    for _, s := range l.data {
        if s.EpochID != epoch {
            continue
        }
        d := s.SnapshotTs.Sub(target)
        if d < 0 { d = -d }
        if d < bestDelta {
            bestDelta = d
            best = s
            found = true
        }
    }
    return best, found
}
```

- [ ] **Step 3: Run tests**

Run: `cd sentry && go test ./internal/store/... -run TestLRU -v`
Expected: all PASS.

- [ ] **Step 4: Commit**

```bash
git add sentry/internal/store/lru.go sentry/internal/store/lru_test.go
git commit -m "feat(sentry): in-memory snapshot LRU with epoch-aware lookup"
```

---

### Task 2.6: TiDB collector

**Files:**
- Create: `sentry/internal/collector/tidb.go`
- Create: `sentry/internal/collector/tidb_test.go` (integration-tagged)

- [ ] **Step 1: Write integration test that seeds a table and snapshots it**

```go
//go:build integration

package collector

import (
    "context"
    "database/sql"
    "os"
    "testing"
    "time"

    _ "github.com/go-sql-driver/mysql"
)

func TestTiDBCollector_SnapshotSeededTable(t *testing.T) {
    dsn := os.Getenv("SENTRY_TEST_DSN")
    if dsn == "" {
        t.Skip("SENTRY_TEST_DSN unset")
    }
    db, err := sql.Open("mysql", dsn)
    if err != nil { t.Fatal(err) }
    defer db.Close()

    _, _ = db.Exec("DROP TABLE IF EXISTS `sentry_test`")
    _, err = db.Exec("CREATE TABLE `sentry_test` (`Id` VARCHAR(32) PRIMARY KEY, `CreatedDate` DATETIME, `LastModifiedDate` DATETIME)")
    if err != nil { t.Fatal(err) }
    _, err = db.Exec("INSERT INTO `sentry_test` VALUES ('a','2026-04-12 09:00:00','2026-04-12 09:05:00'), ('b','2026-04-12 09:01:00','2026-04-12 09:06:00')")
    if err != nil { t.Fatal(err) }

    c := NewTiDBCollector(db, 8, 5*time.Second)
    rows, err := c.Snapshot(context.Background(), []string{"sentry_test"}, nil)
    if err != nil { t.Fatal(err) }
    if len(rows) != 1 { t.Fatalf("rows=%d", len(rows)) }
    if rows[0].RowCount != 2 { t.Errorf("count=%d", rows[0].RowCount) }
    if rows[0].MaxID == nil || *rows[0].MaxID != "b" { t.Errorf("max_id=%v", rows[0].MaxID) }

    _, _ = db.Exec("DROP TABLE `sentry_test`")
}
```

- [ ] **Step 2: Implement collector**

```go
package collector

import (
    "context"
    "database/sql"
    "fmt"
    "sync"
    "time"

    "github.com/nexuscrm/sentry/internal/types"
)

type TiDBCollector struct {
    db           *sql.DB
    workerPool   int
    queryTimeout time.Duration
}

func NewTiDBCollector(db *sql.DB, workerPool int, queryTimeout time.Duration) *TiDBCollector {
    return &TiDBCollector{db: db, workerPool: workerPool, queryTimeout: queryTimeout}
}

// DiscoverTables returns all base table names in the current database,
// excluding sentry's own artifacts (if any were ever created).
func (c *TiDBCollector) DiscoverTables(ctx context.Context) ([]string, error) {
    const q = `
        SELECT TABLE_NAME
        FROM information_schema.TABLES
        WHERE TABLE_SCHEMA = DATABASE()
          AND TABLE_TYPE = 'BASE TABLE'
          AND TABLE_NAME NOT LIKE 'sentry\_%' ESCAPE '\'
    `
    rows, err := c.db.QueryContext(ctx, q)
    if err != nil { return nil, err }
    defer rows.Close()
    var out []string
    for rows.Next() {
        var t string
        if err := rows.Scan(&t); err != nil { return nil, err }
        out = append(out, t)
    }
    return out, rows.Err()
}

// Snapshot runs COUNT(*) + MAX(Id) + MAX(CreatedDate) + MAX(LastModifiedDate)
// over every table in `tables` in parallel.
// Tables listed in `slowTables` skip COUNT(*) and use information_schema.TABLES.TABLE_ROWS instead.
func (c *TiDBCollector) Snapshot(ctx context.Context, tables []string, slowTables map[string]bool) ([]types.TableRow, error) {
    // Bulk pre-fetch for slow tables
    estimates := map[string]int64{}
    if len(slowTables) > 0 {
        rows, err := c.db.QueryContext(ctx,
            `SELECT TABLE_NAME, TABLE_ROWS FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE()`)
        if err == nil {
            defer rows.Close()
            for rows.Next() {
                var name string
                var n int64
                if err := rows.Scan(&name, &n); err == nil {
                    estimates[name] = n
                }
            }
        }
    }

    var (
        wg  sync.WaitGroup
        mu  sync.Mutex
        out = make([]types.TableRow, 0, len(tables))
        sem = make(chan struct{}, c.workerPool)
    )
    for _, table := range tables {
        table := table
        sem <- struct{}{}
        wg.Add(1)
        go func() {
            defer wg.Done()
            defer func() { <-sem }()
            row := c.snapshotOne(ctx, table, slowTables, estimates)
            mu.Lock()
            out = append(out, row)
            mu.Unlock()
        }()
    }
    wg.Wait()
    return out, nil
}

func (c *TiDBCollector) snapshotOne(
    ctx context.Context,
    table string,
    slowTables map[string]bool,
    estimates map[string]int64,
) types.TableRow {
    row := types.TableRow{Name: table, Source: types.SourceCountStar}
    if slowTables[table] {
        row.RowCount = estimates[table]
        row.Source = types.SourceInfoSchemaEst
        return row
    }

    qCtx, cancel := context.WithTimeout(ctx, c.queryTimeout)
    defer cancel()

    q := fmt.Sprintf(
        "SELECT COUNT(*), MAX(`Id`), MAX(`LastModifiedDate`), MAX(`CreatedDate`) FROM `%s`",
        table, // NOTE: table name came from information_schema, not user input — safe to interpolate
    )
    var (
        count      int64
        maxID      sql.NullString
        maxUpdated sql.NullTime
        maxCreated sql.NullTime
    )
    err := c.db.QueryRowContext(qCtx, q).Scan(&count, &maxID, &maxUpdated, &maxCreated)
    if err != nil {
        // Degrade: record as slow, fall back to estimate
        row.RowCount = estimates[table]
        row.Source = types.SourceInfoSchemaEst
        return row
    }
    row.RowCount = count
    if maxID.Valid { row.MaxID = &maxID.String }
    if maxUpdated.Valid { t := maxUpdated.Time; row.MaxUpdatedAt = &t }
    if maxCreated.Valid { t := maxCreated.Time; row.MaxCreatedAt = &t }
    return row
}
```

- [ ] **Step 3: Run integration test against local TiDB**

```bash
cd sentry
export SENTRY_TEST_DSN="root:@tcp(127.0.0.1:4000)/nexuscrm"
go test -tags=integration ./internal/collector/... -run TestTiDBCollector_SnapshotSeededTable -v
```
Expected: PASS. If local TiDB is not running, test is skipped.

- [ ] **Step 4: Write a unit test (no DB) for Snapshot() with a mocked database.sql driver**

Skip if `sqlmock` is not easily available; rely on the integration test. Cite the skip in the plan review.

- [ ] **Step 5: Commit**

```bash
git add sentry/internal/collector/tidb.go sentry/internal/collector/tidb_test.go
git commit -m "feat(sentry): TiDB collector with parallel COUNT(*) + MAX() and slow-table degradation"
```

---

### Task 2.7: CRM endpoint scraper

**Files:**
- Create: `sentry/internal/collector/crm.go`
- Create: `sentry/internal/collector/crm_test.go`

- [ ] **Step 1: Failing test with `httptest.Server`**

```go
package collector

import (
    "context"
    "encoding/json"
    "net/http"
    "net/http/httptest"
    "testing"
    "time"
)

func TestCRMScraper_HappyPath(t *testing.T) {
    srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if r.Header.Get("Authorization") != "Bearer tkn" {
            http.Error(w, "unauthorized", http.StatusUnauthorized)
            return
        }
        _ = json.NewEncoder(w).Encode(map[string]any{
            "collected_at": time.Now().UTC().Format(time.RFC3339),
            "version": "v-test",
            "uptime_sec": 3600,
            "runtime": map[string]any{"goroutines": 42, "heap_alloc_mb": 100, "heap_inuse_mb": 90, "gc_pause_p99_ms": 5, "num_gc": 10},
            "db_pool": map[string]any{"open_conns": 5, "in_use": 2, "idle": 3, "wait_count": 0, "wait_duration_ms": 0},
            "http":    map[string]any{"requests_total": 1234, "error_5xx_count": 3, "panic_count": 0},
            "outbox":  map[string]any{"pending": 12, "processed": 99, "failed": 1},
        })
    }))
    defer srv.Close()

    s := NewCRMScraper(srv.URL, "tkn", 5*time.Second)
    got, err := s.Scrape(context.Background())
    if err != nil { t.Fatal(err) }
    if got.Runtime.Goroutines != 42 { t.Errorf("goroutines=%d", got.Runtime.Goroutines) }
    if got.Outbox.Pending != 12 { t.Errorf("pending=%d", got.Outbox.Pending) }
}

func TestCRMScraper_Unauthorized(t *testing.T) {
    srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        http.Error(w, "nope", http.StatusUnauthorized)
    }))
    defer srv.Close()
    s := NewCRMScraper(srv.URL, "wrong", time.Second)
    if _, err := s.Scrape(context.Background()); err == nil {
        t.Error("want error on 401")
    }
}
```

- [ ] **Step 2: Implement**

```go
package collector

import (
    "context"
    "encoding/json"
    "fmt"
    "net/http"
    "time"

    "github.com/nexuscrm/sentry/internal/types"
)

type CRMScraper struct {
    url    string
    token  string
    client *http.Client
}

func NewCRMScraper(url, token string, timeout time.Duration) *CRMScraper {
    return &CRMScraper{
        url:   url,
        token: token,
        client: &http.Client{Timeout: timeout},
    }
}

func (s *CRMScraper) Scrape(ctx context.Context) (*types.AppHealth, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, s.url, nil)
    if err != nil { return nil, err }
    req.Header.Set("Authorization", "Bearer "+s.token)
    resp, err := s.client.Do(req)
    if err != nil { return nil, err }
    defer resp.Body.Close()
    if resp.StatusCode != http.StatusOK {
        return nil, fmt.Errorf("crm scraper: status %d", resp.StatusCode)
    }
    var out types.AppHealth
    if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
        return nil, fmt.Errorf("crm scraper: decode: %w", err)
    }
    return &out, nil
}
```

- [ ] **Step 3: Run tests**

Run: `cd sentry && go test ./internal/collector/... -run TestCRMScraper -v`
Expected: both PASS.

- [ ] **Step 4: Commit**

```bash
git add sentry/internal/collector/crm.go sentry/internal/collector/crm_test.go
git commit -m "feat(sentry): CRM /internal/sentry-metrics scraper"
```

---

### Task 2.8: AWS Secrets Manager client + S3 store

**Files:**
- Create: `sentry/internal/secrets/aws_secrets.go`
- Create: `sentry/internal/store/s3.go`
- Create: `sentry/internal/store/s3_test.go`

- [ ] **Step 1: Secrets helper**

```go
package secrets

import (
    "context"
    "encoding/json"
    "fmt"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/service/secretsmanager"
)

type Manager struct { client *secretsmanager.Client }

func New(client *secretsmanager.Client) *Manager { return &Manager{client: client} }

// GetString returns the secret string (plain value).
func (m *Manager) GetString(ctx context.Context, arn string) (string, error) {
    out, err := m.client.GetSecretValue(ctx, &secretsmanager.GetSecretValueInput{
        SecretId: aws.String(arn),
    })
    if err != nil { return "", err }
    if out.SecretString == nil {
        return "", fmt.Errorf("secret %s has no string value", arn)
    }
    return *out.SecretString, nil
}

// GetJSON unmarshals the secret into the provided target.
func (m *Manager) GetJSON(ctx context.Context, arn string, target any) error {
    s, err := m.GetString(ctx, arn)
    if err != nil { return err }
    return json.Unmarshal([]byte(s), target)
}
```

- [ ] **Step 2: S3 store**

```go
package store

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "sort"
    "time"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/service/s3"

    "github.com/nexuscrm/sentry/internal/types"
)

type S3Store struct {
    client *s3.Client
    bucket string
}

func NewS3Store(client *s3.Client, bucket string) *S3Store {
    return &S3Store{client: client, bucket: bucket}
}

func snapshotKey(ts time.Time) string {
    return fmt.Sprintf("snapshots/%04d/%02d/%02d/%02d%02d.json",
        ts.Year(), ts.Month(), ts.Day(), ts.Hour(), ts.Minute())
}

func (s *S3Store) PutSnapshot(ctx context.Context, snap types.Snapshot) error {
    b, err := json.Marshal(snap)
    if err != nil { return err }
    _, err = s.client.PutObject(ctx, &s3.PutObjectInput{
        Bucket:      aws.String(s.bucket),
        Key:         aws.String(snapshotKey(snap.SnapshotTs.UTC())),
        Body:        bytes.NewReader(b),
        ContentType: aws.String("application/json"),
    })
    return err
}

// HydrateRecent lists the last `hours` hours of snapshots and returns them ordered by time.
func (s *S3Store) HydrateRecent(ctx context.Context, hours int) ([]types.Snapshot, error) {
    cutoff := time.Now().UTC().Add(-time.Duration(hours) * time.Hour)
    var out []types.Snapshot
    paginator := s3.NewListObjectsV2Paginator(s.client, &s3.ListObjectsV2Input{
        Bucket: aws.String(s.bucket),
        Prefix: aws.String("snapshots/"),
    })
    for paginator.HasMorePages() {
        page, err := paginator.NextPage(ctx)
        if err != nil { return nil, err }
        for _, obj := range page.Contents {
            if obj.LastModified == nil || obj.LastModified.Before(cutoff) {
                continue
            }
            snap, err := s.getSnapshot(ctx, aws.ToString(obj.Key))
            if err == nil {
                out = append(out, snap)
            }
        }
    }
    sort.SliceStable(out, func(i, j int) bool { return out[i].SnapshotTs.Before(out[j].SnapshotTs) })
    return out, nil
}

func (s *S3Store) getSnapshot(ctx context.Context, key string) (types.Snapshot, error) {
    out, err := s.client.GetObject(ctx, &s3.GetObjectInput{
        Bucket: aws.String(s.bucket),
        Key:    aws.String(key),
    })
    if err != nil { return types.Snapshot{}, err }
    defer out.Body.Close()
    body, err := io.ReadAll(out.Body)
    if err != nil { return types.Snapshot{}, err }
    var snap types.Snapshot
    if err := json.Unmarshal(body, &snap); err != nil {
        return types.Snapshot{}, err
    }
    return snap, nil
}
```

- [ ] **Step 3: S3 test**

Use a fake interface: create a `type s3API interface { PutObject(...); ListObjectsV2(...); GetObject(...) }` wrapper, assert the PUT input has the right key format. Keep the test compile-level only if fakes are cumbersome — real end-to-end via Phase 5 deploy is the canonical check.

```go
package store

import (
    "testing"
    "time"
)

func TestSnapshotKey(t *testing.T) {
    ts := time.Date(2026, 4, 12, 9, 20, 0, 0, time.UTC)
    got := snapshotKey(ts)
    want := "snapshots/2026/04/12/0920.json"
    if got != want {
        t.Errorf("got %q want %q", got, want)
    }
}
```

- [ ] **Step 4: Build + test**

Run: `cd sentry && go mod tidy && go build ./... && go test ./internal/store/... -v`
Expected: PASS.

- [ ] **Step 5: Commit**

```bash
git add sentry/internal/secrets/ sentry/internal/store/s3.go sentry/internal/store/s3_test.go sentry/go.mod sentry/go.sum
git commit -m "feat(sentry): S3 snapshot store and AWS Secrets Manager client"
```

---

## Phase 3: Rules + Alerting

### Task 3.1: Rule types and registry

**Files:**
- Create: `sentry/internal/rules/rules.go`
- Create: `sentry/internal/rules/rules_test.go`

- [ ] **Step 1: Write failing test**

```go
package rules

import (
    "testing"
    "time"

    "github.com/nexuscrm/sentry/internal/types"
)

func TestEvaluateAll_NoRules(t *testing.T) {
    eng := NewEngine(nil)
    snap := types.Snapshot{SnapshotTs: time.Now(), EpochID: 1}
    if got := eng.Evaluate(snap, nil); len(got) != 0 {
        t.Errorf("expected 0 results, got %d", len(got))
    }
}
```

- [ ] **Step 2: Implement engine + rule interface**

```go
package rules

import (
    "github.com/nexuscrm/sentry/internal/types"
)

// HistoryFinder is satisfied by store.LRU.
type HistoryFinder interface {
    FindClosest(target time.Time, epoch int64) (types.Snapshot, bool)
}

// Rule evaluates a current snapshot against historical snapshots and
// returns zero or more results.
type Rule interface {
    ID() string
    Evaluate(current types.Snapshot, history HistoryFinder) []types.RuleResult
}

type Engine struct {
    rules []Rule
}

func NewEngine(rules []Rule) *Engine { return &Engine{rules: rules} }

func (e *Engine) Evaluate(current types.Snapshot, history HistoryFinder) []types.RuleResult {
    var results []types.RuleResult
    for _, r := range e.rules {
        results = append(results, r.Evaluate(current, history)...)
    }
    return results
}
```

Add `import "time"` at top.

- [ ] **Step 3: Run tests**

Run: `cd sentry && go test ./internal/rules/...`

- [ ] **Step 4: Commit**

```bash
git add sentry/internal/rules/rules.go sentry/internal/rules/rules_test.go
git commit -m "feat(sentry): rule engine skeleton"
```

---

### Task 3.2: Data volume rules (row spike / drop / write stall / outbox)

**Files:**
- Create: `sentry/internal/rules/data.go`
- Create: `sentry/internal/rules/data_test.go`

- [ ] **Step 1: Write table-driven failing tests for each rule**

Cover:
- `RowSpikeRule` at +49% (WARN no), +51% (WARN yes), +99% delta < 200k (WARN only), +101% (CRIT yes)
- `RowDropRule` at -19% (no), -21% (WARN), -49% (WARN), -51% (CRIT)
- `WriteStallRule`: core table `LastModifiedDate` > 2h old → WARN
- `OutboxBacklogRule`: pending 49999 (no), 50001 (WARN), 200001 (CRIT)
- `OutboxFailedSurgeRule`: delta > 500 in 1h → WARN
- Estimated-source tolerance: thresholds double for `SourceInfoSchemaEst`

(Full test code omitted here — use sub-tests with named cases, check rule IDs and levels explicitly.)

- [ ] **Step 2: Implement each rule as a struct with `ID()` and `Evaluate()`**

Define per-rule constants at top:

```go
const (
    RowSpikeWarnPct  = 0.50
    RowSpikeCritPct  = 1.00
    RowSpikeAbsMin   = 5000
    RowSpikeAbsCrit  = 200000
    RowDropWarnPct   = -0.20
    RowDropCritPct   = -0.50
    WriteStallMaxAge = 2 * time.Hour
    OutboxWarn       = 50000
    OutboxCrit       = 200000
    OutboxFailedSurgeWarn = 500
)
```

Each rule compares current snapshot against a `history.FindClosest(currentTs - 1h, currentEpoch)` result. If not found (fresh epoch, cold start), skip.

- [ ] **Step 3: Run tests**

Run: `cd sentry && go test ./internal/rules/... -v`
Expected: all table-driven cases PASS.

- [ ] **Step 4: Commit**

```bash
git add sentry/internal/rules/data.go sentry/internal/rules/data_test.go
git commit -m "feat(sentry): data volume rules"
```

---

### Task 3.3: App health rules

Similar structure to 3.2. Rules:

- `GoroutineLeakRule` (+1000 over 1h)
- `HeapGrowthRule` (+100% over 2h)
- `DBPoolWaitRule` (wait_count delta > 500 in 10 min)
- `PanicSurgeRule` (panic_count delta > 20 in 10 min)
- `HTTP5xxRule` (5xx rate > 3% WARN, > 10% CRIT)
- `OutboxStallRule` (pending > 0 non-decreasing over 3 consecutive same-epoch cycles)

**Files:** `sentry/internal/rules/app.go`, `sentry/internal/rules/app_test.go`

- [ ] **Step 1: Write table-driven tests for each rule**
- [ ] **Step 2: Implement**
- [ ] **Step 3: Run tests**
- [ ] **Step 4: Commit**

---

### Task 3.4: Suppression layer

**Files:**
- Create: `sentry/internal/rules/suppress.go`
- Create: `sentry/internal/rules/suppress_test.go`

Implements:
- **Dedup window**: per `{rule_id, entity}` 60-min cooldown (in-memory map with expiry)
- **Cold-start quiet period**: first 2 cycles after boot
- **Epoch suppression window**: delegate to `epoch.Tracker.InSuppressionWindow(ts)`; when true, drop all `data.*` rules and `app.outbox_stall` only

Tests cover:
- First call after cooldown fires; second within window does not
- WARN → CRIT escalation bypasses cooldown
- Cold-start skips both cycles 0 and 1
- Epoch suppression drops data rules but keeps `app.goroutine_leak`

Commit: `feat(sentry): rule suppression layer`

---

### Task 3.5: Template-based Lark formatter (fallback path — build first)

**Files:**
- Create: `sentry/internal/llm/template.go`
- Create: `sentry/internal/llm/template_test.go`

- [ ] **Step 1: Write failing test**

```go
func TestTemplateFormat_RowSpike(t *testing.T) {
    r := types.RuleResult{
        RuleID:  "data.row_spike_warn",
        Entity:  "Contact",
        Level:   "WARN",
        Context: map[string]any{
            "current": int64(523411),
            "baseline_1h": int64(300000),
            "delta_pct": 74.5,
        },
    }
    msg := FormatTemplate(r)
    if !strings.Contains(msg, "Contact") || !strings.Contains(msg, "74.5") {
        t.Errorf("unexpected message: %s", msg)
    }
}
```

- [ ] **Step 2: Implement per-rule format strings, one function**

Commit: `feat(sentry): template-based alert formatter (LLM fallback)`

---

### Task 3.6: Qwen (DashScope) client

**Files:**
- Create: `sentry/internal/llm/qwen.go`
- Create: `sentry/internal/llm/qwen_test.go`
- Create: `sentry/internal/llm/prompt.go`

- [ ] **Step 1: Mock DashScope with httptest.Server that asserts the request body matches OpenAI chat-completions schema**
- [ ] **Step 2: Implement client with system + user prompt, temperature 0.2, max_tokens 200, 10s timeout, 2 retries**
- [ ] **Step 3: Fallback on any failure — caller should call `template.Format` if `Translate()` returns error**
- [ ] **Step 4: Assert prompt template contains the anti-speculation guardrails from spec §3.5**
- [ ] **Step 5: Commit**

---

### Task 3.7: Lark webhook notifier with HMAC signing

**Files:**
- Create: `sentry/internal/notify/lark.go`
- Create: `sentry/internal/notify/lark_test.go`

- [ ] **Step 1: httptest.Server that verifies the HMAC signature + timestamp**
- [ ] **Step 2: Implement client — POST interactive card, `sign` field per Lark docs**
- [ ] **Step 3: Retry 2x on non-2xx; on final failure return error (caller persists to S3 alerts/ regardless)**
- [ ] **Step 4: Commit**

Reference Lark custom robot signing format: `sign = base64(hmac_sha256("{timestamp}\n{secret}", ""))`.

---

### Task 3.8: Cycle orchestrator

**Files:**
- Create: `sentry/internal/cycle/cycle.go`
- Create: `sentry/internal/cycle/cycle_test.go`

- [ ] **Step 1: Write a test that wires fakes for collector/scraper/store/llm/lark and runs one cycle**

Fakes return canned data; assert:
- Snapshot is added to LRU
- Snapshot is PUT to S3 (mock store)
- When a rule breaches, LLM is called, then Lark is called
- When LLM returns error, template fallback is used and Lark is still called
- When Lark fails, alert is archived to S3 alerts/

- [ ] **Step 2: Implement `Cycle.Run(ctx)` that executes the 7-step pipeline from the spec §2 diagram**
- [ ] **Step 3: Commit**

---

### Task 3.9: Wire main.go — full cycle loop

**Files:**
- Modify: `sentry/cmd/sentry/main.go`

- [ ] **Step 1: Load config, resolve secrets, build AWS clients, connect TiDB, construct cycle**
- [ ] **Step 2: Start a `time.NewTicker(cfg.CycleInterval)` loop that calls `cycle.Run(ctx)` each tick, stops on ctx.Done**
- [ ] **Step 3: Log cycle duration, rule-fire counts, cost counters**
- [ ] **Step 4: Run locally against staging-like env (or mock everything) — verify one full cycle executes without panic**
- [ ] **Step 5: Commit**

---

## Phase 4: Infra + CI

### Task 4.1: Sentry Dockerfile

**Files:**
- Create: `infra/sentry.Dockerfile`

- [ ] **Step 1: Copy `infra/backend.Dockerfile` and adapt paths**

Change: copy `sentry/go.mod`, `sentry/go.sum`, `shared/`, `sentry/`. Build `./cmd/sentry/`. Expose no port (sentry has no HTTP surface).

- [ ] **Step 2: Build the image locally**

Run: `docker build -f infra/sentry.Dockerfile -t sentry:dev .`
Expected: successful build.

- [ ] **Step 3: Commit**

```bash
git add infra/sentry.Dockerfile
git commit -m "feat(sentry): multi-stage Dockerfile for ECS"
```

---

### Task 4.2: CI job for sentry unit tests

**Files:**
- Modify: `.github/workflows/ci.yml`

- [ ] **Step 1: Add a new job `sentry-test` that runs `cd sentry && go test ./...` on `ubuntu-latest` with Go 1.24.3**
- [ ] **Step 2: Push branch, open PR to verify the job runs green (skip until after full PR submission — just locally assert yaml lints with `actionlint` if available)**
- [ ] **Step 3: Commit**

---

## Phase 5: Deploy (deferred — documented only)

Terraform + deploy workflow changes are captured as placeholder tasks in `docs/design/sentry-ops-monitoring.md` §6. Do not author `.tf` files in this PR — that is a follow-up PR requiring AWS account access and staging bucket provisioning.

- [ ] **Step 1**: Create stub directory `infra/terraform/sentry/README.md` explaining the follow-up scope.
- [ ] **Step 2**: Commit and stop.

---

## Final Integration Check

- [ ] **Step A: Run all tests in sentry**

```bash
cd sentry && go test ./...
```
Expected: all PASS. Integration-tagged tests skip if `SENTRY_TEST_DSN` unset.

- [ ] **Step B: Run backend tests for touched files**

```bash
cd backend && go test ./internal/interfaces/...
```
Expected: all PASS.

- [ ] **Step C: Static checks**

```bash
cd backend && go vet ./...
cd ../sentry && go vet ./...
```
Expected: clean.

- [ ] **Step D: Run `/review` skill per project CLAUDE.md** — required dual-model pre-merge gate before opening PR.

- [ ] **Step E: Open PR from `feat/sentry-ops-monitoring` → `main`** with a description referencing both `docs/design/sentry-ops-monitoring.md` and this plan.

---

## Out of Scope (YAGNI Reminders)

- ❌ Athena / QuickSight dashboards — spec §3.3 leaves hooks but defers
- ❌ Multi-region failover — singleton Fargate, accept a missed cycle
- ❌ Self-alarm / sentry-watches-sentry — §9 OQ3 resolved no
- ❌ Auto-remediation — spec §1.2 non-goal
- ❌ Daily digest — spec §1.2 non-goal
- ❌ CloudWatch Logs scraping — no log source
- ❌ HubSpot-specific metrics — deferred until integration_inbox/outbox refactor lands

---

## Open Questions Parked from Spec

- **OQ2** (final core-tables list) — confirmed during Task 3.2 against staging schema dump
- **OQ4** (Lark channel) — can remain unresolved; sentry posts to a developer-private test channel until the final ops channel is provisioned