# Sentry — AWS Deployment Runbook

> **Status: DEPLOYED** (2026-04-15) — live in the nexuscrm AWS account
> (`604175033442`) `prod` Terraform workspace. Managed by
> [`infra/sentry.tf`](../sentry.tf). For day-to-day operations (tail logs,
> rotate secrets, manual runs, troubleshoot), see the
> [`sentry-ops` skill](../../.agent/skills/sentry-ops/SKILL.md).
>
> **This runbook is kept for historical reference and for greenfield
> deployment to a new AWS account.** Most day-to-day work does NOT
> require reading this document — the ops skill is the right entry point.

Two workloads, one Docker image, one ECS cluster.

| Workload | Binary | Trigger | ECS construct |
|---|---|---|---|
| **sentry-service** | `sentry` | Internal 10-min ticker | Long-running ECS Service (desired count = 1) |
| **sentry-daily-report** | `sentry-probe` | EventBridge cron `0 1 * * ? *` (= 09:00 UTC+8) | Scheduled one-shot ECS RunTask |

Both share the same ECR image (`infra/sentry.Dockerfile` builds all three binaries: `sentry`, `sentry-probe`, `sentry-lark-send`). The container `command:` field selects which one runs.

---

## Architecture

```
                ┌─────────────────────────────────────────┐
                │ AWS account (same region as nexuscrm)   │
                │                                         │
  GitHub Actions│  ┌─ ECR ──────────────┐                 │
  push image ──▶│  │ nexus-sentry:latest│                 │
                │  └────────┬───────────┘                 │
                │           │ pulled by                   │
                │           ▼                             │
                │  ┌─ ECS Cluster: nexus-sentry ────┐    │
                │  │                                 │    │
                │  │  Service: sentry-service        │    │
                │  │  ├─ Task def: nexus-sentry-svc  │    │
                │  │  ├─ Container cmd: ["sentry"]   │    │
                │  │  ├─ Desired count: 1            │    │
                │  │  └─ Always running, 10-min loop │    │
                │  │                                 │    │
                │  │  Scheduled task: daily-report   │    │
                │  │  ├─ Task def: sentry-probe      │    │
                │  │  ├─ Container cmd:              │    │
                │  │  │    ["sentry-probe"]          │    │
                │  │  └─ Triggered by EventBridge    │    │
                │  └────────┬────────────────────────┘    │
                │           │                              │
                │           ▼ (env from)                   │
                │  ┌─ Secrets Manager ──────────────┐     │
                │  │  sentry/tidb-prod              │     │
                │  │  sentry/tidb-staging           │     │
                │  │  sentry/dashscope              │     │
                │  │  sentry/lark-webhook           │     │
                │  │  sentry/probe-targets-json     │     │
                │  └─────────────────────────────────┘    │
                │                                         │
                │  ┌─ EventBridge ──────────────────┐     │
                │  │  Rule: cron(0 1 * * ? *)       │     │
                │  │  Target: ECS RunTask           │     │
                │  │  Task def: sentry-probe        │     │
                │  └─────────────────────────────────┘    │
                │                                         │
                │  ┌─ S3 ───────────────────────────┐     │
                │  │  nexus-sentry-snapshots        │     │
                │  │  (used by sentry-service only) │     │
                │  └─────────────────────────────────┘    │
                │                                         │
                │  ┌─ CloudWatch Logs ──────────────┐     │
                │  │  /ecs/nexus-sentry             │     │
                │  └─────────────────────────────────┘    │
                └─────────────────────────────────────────┘
                                │
                                ▼ outbound TLS
                  ┌──────────────────────────────┐
                  │ TiDB Cloud serverless        │
                  │ gateway01.us-east-1.prod.... │
                  └──────────────────────────────┘
                                │
                                ▼ outbound HTTPS
                  ┌──────────────────────────────┐
                  │ DashScope (Qwen)             │
                  │ Lark webhook (open.feishu)   │
                  └──────────────────────────────┘
```

---

## Prerequisites

- AWS account with admin (or scoped IAM) for first-time setup
- AWS CLI configured (`aws sts get-caller-identity` works)
- Docker Desktop running (or buildx)
- The exact region you'll deploy in. **Use the same region as your existing CRM/staging deployment** to avoid cross-region traffic charges. Recommended: `us-east-1` (since TiDB Cloud cluster is in `us-east-1`).
- A VPC with at least one private subnet that has either NAT egress or VPC endpoints for ECR / Secrets Manager / Logs / S3. If you don't have one, the simplest first-time setup uses the **default VPC** with a public subnet and `assignPublicIp = ENABLED` on the task — fine for staging but not ideal for prod-grade isolation.

Set these shell variables — used throughout the runbook:

```bash
export AWS_REGION=us-east-1
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export ECR_REPO=nexus-sentry
export ECS_CLUSTER=nexus-sentry
export S3_BUCKET=nexus-sentry-snapshots-${AWS_ACCOUNT_ID}
export LOG_GROUP=/ecs/nexus-sentry
export SUBNET_ID=subnet-xxxxxxxxxxxxxxxxx     # private subnet with egress
export SECURITY_GROUP_ID=sg-xxxxxxxxxxxxxxxxx # egress-only SG
```

---

## Step 1 — One-time AWS resources

### 1.1 ECR repo

```bash
aws ecr create-repository \
  --repository-name $ECR_REPO \
  --region $AWS_REGION \
  --image-scanning-configuration scanOnPush=true
```

### 1.2 CloudWatch log group

```bash
aws logs create-log-group --log-group-name $LOG_GROUP --region $AWS_REGION
aws logs put-retention-policy --log-group-name $LOG_GROUP --retention-in-days 30 --region $AWS_REGION
```

### 1.3 S3 bucket (for sentry-service snapshots)

```bash
aws s3api create-bucket --bucket $S3_BUCKET --region $AWS_REGION
aws s3api put-bucket-versioning --bucket $S3_BUCKET \
  --versioning-configuration Status=Enabled
aws s3api put-bucket-encryption --bucket $S3_BUCKET \
  --server-side-encryption-configuration '{
    "Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]
  }'
aws s3api put-public-access-block --bucket $S3_BUCKET \
  --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
```

Lifecycle policy — auto-archive after 90 days, delete after 1 year:

```bash
cat > /tmp/lifecycle.json <<'EOF'
{
  "Rules": [{
    "ID": "snapshots-aging",
    "Status": "Enabled",
    "Filter": {"Prefix": "snapshots/"},
    "Transitions": [{"Days": 90, "StorageClass": "DEEP_ARCHIVE"}],
    "Expiration": {"Days": 365}
  }]
}
EOF
aws s3api put-bucket-lifecycle-configuration \
  --bucket $S3_BUCKET --lifecycle-configuration file:///tmp/lifecycle.json
```

### 1.4 Secrets Manager entries

**sentry/tidb-prod** — read-only TiDB creds for prod (used by sentry-service):

```bash
aws secretsmanager create-secret \
  --name sentry/tidb-prod \
  --secret-string '{
    "host": "gateway01.us-east-1.prod.aws.tidbcloud.com",
    "port": 4000,
    "user": "<readonly-user>.root",
    "password": "<readonly-password>",
    "database": "nexuscrm"
  }'
```

> ⚠️ Create a **dedicated read-only user** in TiDB before this step. The probe and service should never use the admin user. SQL: `CREATE USER 'sentry_ro'@'%' IDENTIFIED BY '<random>'; GRANT SELECT ON nexuscrm.* TO 'sentry_ro'@'%'; GRANT SELECT ON information_schema.* TO 'sentry_ro'@'%';`

**sentry/probe-targets-json** — multi-target list for the daily-report probe (prod + staging):

```bash
aws secretsmanager create-secret \
  --name sentry/probe-targets-json \
  --secret-string '[
    {
      "name": "prod",
      "host": "gateway01.us-east-1.prod.aws.tidbcloud.com",
      "port": 4000,
      "user": "<readonly-user>.root",
      "password": "<readonly-password>",
      "database": "nexuscrm"
    },
    {
      "name": "staging",
      "host": "gateway01.us-east-1.prod.aws.tidbcloud.com",
      "port": 4000,
      "user": "<staging-readonly-user>.root",
      "password": "<staging-readonly-password>",
      "database": "nexuscrm"
    }
  ]'
```

**sentry/dashscope** — Qwen API key:

```bash
aws secretsmanager create-secret \
  --name sentry/dashscope \
  --secret-string 'sk-5d3ceef5452f4ee4bae1e51f43ce5718'
```

**sentry/lark-webhook** — Lark URL + signing secret:

```bash
aws secretsmanager create-secret \
  --name sentry/lark-webhook \
  --secret-string '{
    "url": "https://open.feishu.cn/open-apis/bot/v2/hook/392fed29-5d0b-4511-b7cb-c83b86921ddb",
    "signing_secret": "dhFdCoD6XCLtPNmHkdAYUg"
  }'
```

### 1.5 IAM roles

**Execution role** — ECS pulls image, writes logs, fetches secrets:

```bash
cat > /tmp/exec-trust.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "ecs-tasks.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}
EOF

aws iam create-role \
  --role-name SentryExecutionRole \
  --assume-role-policy-document file:///tmp/exec-trust.json

aws iam attach-role-policy \
  --role-name SentryExecutionRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

# Inline policy for Secrets Manager read of sentry/* secrets
cat > /tmp/exec-secrets.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["secretsmanager:GetSecretValue"],
    "Resource": "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:secret:sentry/*"
  }]
}
EOF
aws iam put-role-policy \
  --role-name SentryExecutionRole \
  --policy-name SentrySecretsRead \
  --policy-document file:///tmp/exec-secrets.json
```

**Task role** — what the running container can call:

```bash
aws iam create-role \
  --role-name SentryTaskRole \
  --assume-role-policy-document file:///tmp/exec-trust.json

cat > /tmp/task-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::${S3_BUCKET}",
        "arn:aws:s3:::${S3_BUCKET}/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": [
        "arn:aws:s3:::${S3_BUCKET}/snapshots/*",
        "arn:aws:s3:::${S3_BUCKET}/alerts/*",
        "arn:aws:s3:::${S3_BUCKET}/slow_tables.json"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:secret:sentry/*"
    }
  ]
}
EOF

aws iam put-role-policy \
  --role-name SentryTaskRole \
  --policy-name SentryTaskPermissions \
  --policy-document file:///tmp/task-policy.json
```

> Note: explicitly **no** `s3:DeleteObject`, **no** RDS/EC2/IAM permissions. Sentry is read-only by design.

### 1.6 ECS cluster

```bash
aws ecs create-cluster --cluster-name $ECS_CLUSTER --region $AWS_REGION
```

---

## Step 2 — Build and push the image

```bash
# Authenticate Docker to ECR
aws ecr get-login-password --region $AWS_REGION | \
  docker login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com

# Build for linux/amd64 (Fargate is amd64 unless you opt into Graviton)
cd /path/to/repo
docker build --platform linux/amd64 -f infra/sentry.Dockerfile \
  -t ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPO}:latest .

docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPO}:latest
```

---

## Step 3 — Register task definitions

### 3.1 sentry-service (long-running, 10-min ticker)

```bash
cat > /tmp/td-sentry-service.json <<EOF
{
  "family": "nexus-sentry-service",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "executionRoleArn": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/SentryExecutionRole",
  "taskRoleArn": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/SentryTaskRole",
  "containerDefinitions": [{
    "name": "sentry",
    "image": "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPO}:latest",
    "essential": true,
    "command": ["sentry"],
    "environment": [
      {"name": "SENTRY_ENV",                  "value": "prod"},
      {"name": "SENTRY_REGION",               "value": "${AWS_REGION}"},
      {"name": "SENTRY_S3_BUCKET",            "value": "${S3_BUCKET}"},
      {"name": "SENTRY_CYCLE_INTERVAL",       "value": "10m"},
      {"name": "SENTRY_QUERY_TIMEOUT",        "value": "10s"},
      {"name": "SENTRY_DB_WORKER_POOL",       "value": "8"},
      {"name": "SENTRY_RESET_SCHEDULE",       "value": ""},
      {"name": "SENTRY_RESET_EPOCH_REFERENCE","value": "2026-01-01T00:00:00Z"},
      {"name": "SENTRY_SECRET_TIDB",          "value": "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:secret:sentry/tidb-prod"},
      {"name": "SENTRY_SECRET_DASHSCOPE",     "value": "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:secret:sentry/dashscope"},
      {"name": "SENTRY_SECRET_LARK",          "value": "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:secret:sentry/lark-webhook"}
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "${LOG_GROUP}",
        "awslogs-region": "${AWS_REGION}",
        "awslogs-stream-prefix": "sentry-service"
      }
    }
  }]
}
EOF

aws ecs register-task-definition --cli-input-json file:///tmp/td-sentry-service.json
```

### 3.2 sentry-daily-report (one-shot, EventBridge-driven)

The daily-report task uses a slightly different env shape since `sentry-probe` reads creds directly from env vars (not from Secrets Manager ARNs). We use the ECS `secrets:` field to inject the secret values into env at task launch.

```bash
cat > /tmp/td-sentry-daily-report.json <<EOF
{
  "family": "nexus-sentry-daily-report",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "executionRoleArn": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/SentryExecutionRole",
  "taskRoleArn": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/SentryTaskRole",
  "containerDefinitions": [{
    "name": "sentry-probe",
    "image": "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPO}:latest",
    "essential": true,
    "command": ["sentry-probe"],
    "environment": [
      {"name": "PROBE_TOP_N",   "value": "10"},
      {"name": "PROBE_TIMEOUT", "value": "10s"},
      {"name": "QWEN_API_URL",  "value": "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"},
      {"name": "QWEN_MODEL",    "value": "qwen3.6-plus-2026-04-02"},
      {"name": "QWEN_TIMEOUT",  "value": "120s"}
    ],
    "secrets": [
      {"name": "PROBE_TARGETS_JSON",  "valueFrom": "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:secret:sentry/probe-targets-json"},
      {"name": "QWEN_API_KEY",        "valueFrom": "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:secret:sentry/dashscope"},
      {"name": "LARK_WEBHOOK_URL",    "valueFrom": "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:secret:sentry/lark-webhook:url::"},
      {"name": "LARK_SIGNING_SECRET", "valueFrom": "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:secret:sentry/lark-webhook:signing_secret::"}
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "${LOG_GROUP}",
        "awslogs-region": "${AWS_REGION}",
        "awslogs-stream-prefix": "sentry-daily-report"
      }
    }
  }]
}
EOF

aws ecs register-task-definition --cli-input-json file:///tmp/td-sentry-daily-report.json
```

> The `:url::` and `:signing_secret::` suffixes on the secret ARN extract a JSON field from the secret so the env var receives only the string value, not the whole JSON envelope. This requires the secret to be stored as JSON (which we did in §1.4).

---

## Step 4 — Launch the long-running service

```bash
aws ecs create-service \
  --cluster $ECS_CLUSTER \
  --service-name sentry-service \
  --task-definition nexus-sentry-service \
  --desired-count 1 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_ID],securityGroups=[$SECURITY_GROUP_ID],assignPublicIp=ENABLED}"
```

> If your subnet has NAT egress, set `assignPublicIp=DISABLED` instead.

Verify it's running:

```bash
aws ecs describe-services --cluster $ECS_CLUSTER --services sentry-service \
  --query 'services[0].{status:status,running:runningCount,desired:desiredCount}'
```

Tail logs:

```bash
aws logs tail $LOG_GROUP --follow --filter-pattern "sentry-service"
```

---

## Step 5 — Schedule the daily report (EventBridge → ECS RunTask)

EventBridge needs its own IAM role to call ECS RunTask:

```bash
cat > /tmp/eb-trust.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "events.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}
EOF

aws iam create-role \
  --role-name SentryEventBridgeInvokeRole \
  --assume-role-policy-document file:///tmp/eb-trust.json

cat > /tmp/eb-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["ecs:RunTask"],
    "Resource": "arn:aws:ecs:${AWS_REGION}:${AWS_ACCOUNT_ID}:task-definition/nexus-sentry-daily-report:*"
  }, {
    "Effect": "Allow",
    "Action": ["iam:PassRole"],
    "Resource": [
      "arn:aws:iam::${AWS_ACCOUNT_ID}:role/SentryExecutionRole",
      "arn:aws:iam::${AWS_ACCOUNT_ID}:role/SentryTaskRole"
    ]
  }]
}
EOF
aws iam put-role-policy \
  --role-name SentryEventBridgeInvokeRole \
  --policy-name SentryRunTask \
  --policy-document file:///tmp/eb-policy.json
```

Create the schedule. **09:00 UTC+8 = 01:00 UTC**. EventBridge cron uses 6 fields (`min hr dom mon dow yr`):

```bash
aws events put-rule \
  --name sentry-daily-report \
  --schedule-expression "cron(0 1 * * ? *)" \
  --description "Sentry daily ops report — fires 01:00 UTC = 09:00 UTC+8"
```

Wire the rule to the ECS task:

```bash
cat > /tmp/eb-target.json <<EOF
[{
  "Id": "1",
  "Arn": "arn:aws:ecs:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${ECS_CLUSTER}",
  "RoleArn": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/SentryEventBridgeInvokeRole",
  "EcsParameters": {
    "TaskDefinitionArn": "arn:aws:ecs:${AWS_REGION}:${AWS_ACCOUNT_ID}:task-definition/nexus-sentry-daily-report",
    "TaskCount": 1,
    "LaunchType": "FARGATE",
    "PlatformVersion": "LATEST",
    "NetworkConfiguration": {
      "awsvpcConfiguration": {
        "Subnets": ["${SUBNET_ID}"],
        "SecurityGroups": ["${SECURITY_GROUP_ID}"],
        "AssignPublicIp": "ENABLED"
      }
    }
  }
}]
EOF

aws events put-targets --rule sentry-daily-report --targets file:///tmp/eb-target.json
```

---

## Step 6 — Verify

### 6.1 Force the daily-report task immediately

```bash
aws ecs run-task \
  --cluster $ECS_CLUSTER \
  --task-definition nexus-sentry-daily-report \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_ID],securityGroups=[$SECURITY_GROUP_ID],assignPublicIp=ENABLED}" \
  --query 'tasks[0].taskArn'
```

Watch the logs:

```bash
aws logs tail $LOG_GROUP --follow --filter-pattern "sentry-daily-report"
```

You should see the same output you saw locally:

```
═══ Sentry Probe — Daily Report ═══
Targets: 2
─── Target: prod ───
   ▸ Tables: 373 ...
─── Target: staging ───
   ▸ Tables: 373 ...
═══ Combined Summary ═══
✓ Daily report sent to Lark webhook
```

And a Lark message in your ops channel.

### 6.2 Watch the long-running service

```bash
aws logs tail $LOG_GROUP --follow --filter-pattern "sentry-service"
```

Every 10 minutes you should see one cycle line; alerts only when rules fire.

---

## Step 7 — Optional CI/CD

For automatic image rebuild on `main` push, add a GitHub Actions job (template — adapt to your existing `deploy.yml`):

```yaml
sentry-build-and-deploy:
  runs-on: ubuntu-latest
  needs: [unit-test, sentry-test]
  permissions:
    id-token: write
    contents: read
  steps:
    - uses: actions/checkout@v4

    - name: Configure AWS credentials (OIDC)
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/GitHubActionsDeploy
        aws-region: us-east-1

    - name: Login to ECR
      uses: aws-actions/amazon-ecr-login@v2

    - name: Build and push
      run: |
        IMAGE=${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com/nexus-sentry:${{ github.sha }}
        docker build --platform linux/amd64 -f infra/sentry.Dockerfile -t $IMAGE .
        docker push $IMAGE
        docker tag $IMAGE ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com/nexus-sentry:latest
        docker push ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com/nexus-sentry:latest

    - name: Force ECS service redeploy
      run: |
        aws ecs update-service \
          --cluster nexus-sentry \
          --service sentry-service \
          --force-new-deployment
```

The daily-report task picks up the new image automatically on its next run because EventBridge always launches `:latest`.

---

## Step 8 — Cost estimate

| Resource | Sizing | Cost (us-east-1, 2026 prices) |
|---|---|---|
| ECS Fargate (sentry-service) | 0.25 vCPU + 512 MB, 24h × 30 = 720h/mo | ~$8.40/mo |
| ECS Fargate (sentry-daily-report) | 0.25 vCPU + 512 MB, 1 run × 90s × 30 = 0.75h/mo | ~$0.01/mo |
| ECR storage | ~50 MB image | <$0.01/mo |
| Secrets Manager | 5 secrets × $0.40 | ~$2.00/mo |
| CloudWatch Logs | ~30 MB/mo at $0.50/GB | ~$0.02/mo |
| S3 Standard | ~7 MB/day × 90 days = 630 MB | <$0.02/mo |
| Qwen API (DashScope) | ~30 alerts/mo × ~$0.05/call | ~$1.50/mo |
| **Total** | | **~$12/mo** |

---

## Teardown

```bash
# Stop and delete service
aws ecs update-service --cluster $ECS_CLUSTER --service sentry-service --desired-count 0
aws ecs delete-service --cluster $ECS_CLUSTER --service sentry-service

# Remove the schedule
aws events remove-targets --rule sentry-daily-report --ids 1
aws events delete-rule --name sentry-daily-report

# Delete task definitions (deregister latest revisions you care about)
aws ecs deregister-task-definition --task-definition nexus-sentry-service:1
aws ecs deregister-task-definition --task-definition nexus-sentry-daily-report:1

# Delete cluster, log group, ECR repo, S3 bucket, secrets, IAM roles
aws ecs delete-cluster --cluster $ECS_CLUSTER
aws logs delete-log-group --log-group-name $LOG_GROUP
aws ecr delete-repository --repository-name $ECR_REPO --force
aws s3 rb s3://${S3_BUCKET} --force
for s in tidb-prod probe-targets-json dashscope lark-webhook; do
  aws secretsmanager delete-secret --secret-id sentry/$s --force-delete-without-recovery
done
for r in SentryExecutionRole SentryTaskRole SentryEventBridgeInvokeRole; do
  aws iam delete-role-policy --role-name $r --policy-name SentrySecretsRead 2>/dev/null
  aws iam delete-role-policy --role-name $r --policy-name SentryTaskPermissions 2>/dev/null
  aws iam delete-role-policy --role-name $r --policy-name SentryRunTask 2>/dev/null
  aws iam detach-role-policy --role-name $r --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy 2>/dev/null
  aws iam delete-role --role-name $r
done
```

---

## Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| Task starts then immediately stops with exit 1 | Bad env var (e.g., missing `SENTRY_ENV`) | Check CloudWatch Logs first event |
| `Resource not found` on Secrets Manager | Wrong region or wrong ARN format | Verify `aws secretsmanager describe-secret --secret-id sentry/...` works |
| Task gets stuck in PROVISIONING | Subnet has no egress, can't pull from ECR | Use a public subnet with `assignPublicIp=ENABLED` or add NAT/VPC endpoints |
| Lark posts fail with 401 | Stale signing secret in Secrets Manager | Update `sentry/lark-webhook` and force task restart |
| EventBridge rule doesn't fire | Wrong cron syntax (use 6 fields, not 5) | `cron(0 1 * * ? *)` not `cron(0 1 * * *)` |
| Daily report fires twice on DST changeover | EventBridge runs in UTC | Not an issue — UTC has no DST |