# FinOps AWS MVP Superset Deployment Guide

This guide explains how to deploy Superset 6.0 for the current AWS MVP data model, connect to BigQuery, and build dashboards from existing marts.

Target users:
- Data platform engineers operating Superset
- FinOps/BI engineers building cost dashboards

Current data scope:
- Project: `gcp-rd-research-dev`
- Dataset: `finops_cloud_billing`
- Location: `us-west2`
- Core marts:
  - `mart_cost_monthly`

Dashboard workflow status (2026-02-12):
- Default workflow: manual Dashboard v2 guide
  - `dbt/design/superset-aws-dashboard-v2.md`
- Legacy API automation scripts:
  - Archived in `dbt/scripts/archive/superset_legacy/`
  - Kept for historical reference only (not standard entrypoint)

---

## 1. Scope And Non-Scope

In scope:
- Superset 6.0 deployment on single-host Docker Compose
  - Path A: custom compose in this repo guide
  - Path B: official Apache Superset `docker-compose-image-tag.yml`
- BigQuery connection using Service Account JSON
- Dataset connectivity and readiness checks for `mart_cost_monthly`
- Manual dashboard workflow handoff to v2 guide

Out of scope:
- Kubernetes deployment
- Superset SSO/OIDC full integration
- Finance/ledger marts (not in current MVP mart set)
- Multi-cloud unified dashboards

---

## 2. Prerequisites

Host prerequisites:
- Linux host with Docker and Docker Compose
- Recommended: 4 vCPU, 8 GB RAM, 30+ GB disk

GCP prerequisites:
- Service account JSON file available on host
- Service account has at least:
  - `roles/bigquery.dataViewer` on `gcp-rd-research-dev`
  - `roles/bigquery.jobUser` on `gcp-rd-research-dev`

Data prerequisites:
- `dbt build` has completed successfully
- The AWS MVP mart `mart_cost_monthly` exists in `finops_cloud_billing`

---

## 3. Deployment Layout

This guide supports two deployment paths.

### 3.1 Path A (Custom Compose) Layout

Create a dedicated directory:

```bash
mkdir -p superset/docker/pythonpath superset/secrets
cd superset
```

Expected files:

```text
superset/
  compose.yml
  Dockerfile
  .env
  docker/
    pythonpath/
      superset_config.py
  secrets/
    sa.json
```

Copy your GCP service account key:

```bash
cp /path/to/your-sa.json ./secrets/sa.json
chmod 400 ./secrets/sa.json
```

### 3.2 Path B (Official `docker-compose-image-tag.yml`) Layout

If you follow the official Superset quickstart repository:

```bash
git clone https://github.com/apache/superset.git
cd superset
mkdir -p docker/.secrets
cp /path/to/your-sa.json docker/.secrets/bigquery-sa.json
chmod 400 docker/.secrets/bigquery-sa.json
```

Then configure local overrides:

`docker/.env-local`

```env
GOOGLE_APPLICATION_CREDENTIALS=/app/docker/.secrets/bigquery-sa.json
```

`docker/requirements-local.txt`

```text
sqlalchemy-bigquery>=1.12.0
google-cloud-bigquery>=3.20.0
db-dtypes>=1.2.0
```

---

## 4. Compose Deployment (Superset 6.0)

### 4.1 Create `.env`

`superset/.env`

```env
SUPERSET_SECRET_KEY=CHANGE_ME_TO_A_LONG_RANDOM_SECRET
SUPERSET_DB_URI=postgresql+psycopg2://superset:superset@superset-db:5432/superset
REDIS_HOST=superset-redis
REDIS_PORT=6379
```

Generate a strong secret key:

```bash
openssl rand -base64 42
```

### 4.2 Create `Dockerfile`

`superset/Dockerfile`

```dockerfile
FROM apache/superset:6.0.0

USER root
RUN pip install --no-cache-dir \
  "sqlalchemy-bigquery>=1.12.0" \
  "google-cloud-bigquery>=3.20.0" \
  "db-dtypes>=1.2.0"
USER superset
```

### 4.3 Create Superset config

`superset/docker/pythonpath/superset_config.py`

```python
import os

from cachelib.redis import RedisCache

SECRET_KEY = os.environ["SUPERSET_SECRET_KEY"]
SQLALCHEMY_DATABASE_URI = os.environ["SUPERSET_DB_URI"]

WTF_CSRF_ENABLED = True

# Keep security defaults strict for MVP
AUTH_USER_REGISTRATION = False

# Query result cache
RESULTS_BACKEND = RedisCache(
    host=os.environ.get("REDIS_HOST", "superset-redis"),
    port=int(os.environ.get("REDIS_PORT", "6379")),
    key_prefix="superset_results",
)

# Metadata cache
CACHE_CONFIG = {
    "CACHE_TYPE": "RedisCache",
    "CACHE_DEFAULT_TIMEOUT": 300,
    "CACHE_KEY_PREFIX": "superset_cache_",
    "CACHE_REDIS_HOST": os.environ.get("REDIS_HOST", "superset-redis"),
    "CACHE_REDIS_PORT": int(os.environ.get("REDIS_PORT", "6379")),
}

DATA_CACHE_CONFIG = CACHE_CONFIG
```

### 4.4 Create `compose.yml`

`superset/compose.yml`

```yaml
services:
  superset-db:
    image: postgres:14
    container_name: finops-superset-db
    environment:
      POSTGRES_DB: superset
      POSTGRES_USER: superset
      POSTGRES_PASSWORD: superset
    volumes:
      - superset_db_data:/var/lib/postgresql/data
    restart: unless-stopped

  superset-redis:
    image: redis:7
    container_name: finops-superset-redis
    restart: unless-stopped

  superset:
    build:
      context: .
      dockerfile: Dockerfile
    image: finops-superset:6.0.0
    container_name: finops-superset
    env_file:
      - .env
    environment:
      SUPERSET_ENV: production
      SUPERSET_CONFIG_PATH: /app/pythonpath/superset_config.py
      GOOGLE_APPLICATION_CREDENTIALS: /app/secrets/sa.json
      TZ: UTC
    volumes:
      - ./docker/pythonpath:/app/pythonpath
      - ./secrets/sa.json:/app/secrets/sa.json:ro
    ports:
      - "8088:8088"
    depends_on:
      - superset-db
      - superset-redis
    restart: unless-stopped

volumes:
  superset_db_data:
```

### 4.5 Initialize and start

```bash
docker compose -f compose.yml build
docker compose -f compose.yml up -d superset-db superset-redis

docker compose -f compose.yml run --rm superset superset db upgrade
docker compose -f compose.yml run --rm superset superset fab create-admin \
  --username admin \
  --firstname Superset \
  --lastname Admin \
  --email admin@example.com \
  --password 'CHANGE_ME'
docker compose -f compose.yml run --rm superset superset init

docker compose -f compose.yml up -d superset
```

Access Superset:
- URL: `http://<host>:8088`
- Login with admin credentials created above

Health check:

```bash
docker compose -f compose.yml ps
curl -I http://localhost:8088/health
```

### 4.6 Official Compose Path (`docker-compose-image-tag.yml`)

If you are using the official file `docker-compose-image-tag.yml`, use this flow:

```bash
docker compose -f docker-compose-image-tag.yml up -d
```

If the official docs/version you use requires explicit init step, run:

```bash
docker compose -f docker-compose-image-tag.yml run --rm superset-init
```

Verify SA file is visible in container:

```bash
docker compose -f docker-compose-image-tag.yml exec superset \
  ls -l /app/docker/.secrets/bigquery-sa.json
```

If file is missing, re-check `docker/.secrets` path and `GOOGLE_APPLICATION_CREDENTIALS` in `docker/.env-local`.

---

## 5. BigQuery Connection (Service Account JSON)

### 5.1 Add database in UI

In Superset UI:
- `Settings` -> `Database Connections` -> `+ Database`
- Select engine: `Google BigQuery`

Recommended SQLAlchemy URI:

```text
bigquery://gcp-rd-research-dev/finops_cloud_billing?location=us-west2
```

Notes:
- Why this is recommended:
  - Credential material stays outside URI (less leak risk in logs/metadata exports).
  - Superset uses Application Default Credentials via `GOOGLE_APPLICATION_CREDENTIALS`.
  - Rotation only requires replacing the SA file, not editing the DB connection URI.
- Path A (custom compose): set `GOOGLE_APPLICATION_CREDENTIALS=/app/secrets/sa.json`.
- Path B (official compose): set `GOOGLE_APPLICATION_CREDENTIALS=/app/docker/.secrets/bigquery-sa.json`.
- With either path, no extra credential JSON is required in Superset UI if env var + file mount are valid.
- If needed, fallback URI:

```text
bigquery://gcp-rd-research-dev/finops_cloud_billing?credentials_path=<SA_PATH_IN_CONTAINER>&location=us-west2
```

Example fallback values:
- Path A: `credentials_path=/app/secrets/sa.json`
- Path B: `credentials_path=/app/docker/.secrets/bigquery-sa.json`

Click `Test Connection` and save.

### 5.2 Validate with SQL Lab

Run:

```sql
SELECT COUNT(*) AS cnt
FROM `gcp-rd-research-dev.finops_cloud_billing.mart_cost_monthly`;
```

If query succeeds, connection and permissions are correct.

---

## 6. Dashboard Delivery Workflow (v2)

Current default:
- Follow `dbt/design/superset-aws-dashboard-v2.md`.
- Build one dashboard only: `AWS Executive Overview`.
- Use environment-first analysis (`account_purpose`) and no KPI cards in v2 first release.

What this guide covers:
- Superset deployment and BigQuery connectivity.
- Pre-dashboard readiness checks.

What v2 guide covers:
- Virtual dataset SQL with fixed `list_cost_usd` baseline for trend/distribution charts.
- Native filter and chart-by-chart manual setup.
- Dashboard acceptance checklist.
- Optional local bootstrap script:
  - `dbt/scripts/create_superset_dashboard_v2.sh`

---

## 7. Legacy Automation Scripts (Archived)

The previous API automation scripts are archived and are no longer standard entrypoints:
- `dbt/scripts/archive/superset_legacy/register_superset_mvp_datasets.sh`
- `dbt/scripts/archive/superset_legacy/configure_superset_mvp_metrics.sh`
- `dbt/scripts/archive/superset_legacy/create_superset_mvp_dashboards.sh`
- `dbt/scripts/archive/superset_legacy/create_superset_mvp_charts.sh`

Policy:
- Use v2 manual guide for new dashboard delivery.
- Use archived scripts only for historical troubleshooting or controlled rollback.

---

## 8. Validation Checklist

After initial setup:

1. Data connectivity
- Run `COUNT(*)` on `mart_cost_monthly` in SQL Lab.

2. v2 guide completeness
- Confirm `dbt/design/superset-aws-dashboard-v2.md` is followed end-to-end.

3. Dashboard health
- Dashboard `AWS Executive Overview` loads without query errors.
- Global native filters update all 8 components.

4. Cost baseline consistency
- Trend/distribution charts are based on `selected_cost_usd` (fixed to `list_cost_usd` in v2 first release).

5. Semantics
- Environment-first analysis works via `account_purpose`.
- No KPI cards (`weighted_savings_rate`, `sp_coverage_rate`) in v2 first release.

---

## 9. Daily Operations

Recommended routine:

1. Refresh data (outside Superset)
- Run dbt refresh:
  - `cd dbt && dbt build`

2. Refresh Superset metadata
- For each dataset, refresh metadata if columns changed

3. Validate key chart
- Open Executive dashboard and confirm newest month appears

4. Backup metadata weekly
- Export dashboards and datasets

Example backup commands:

```bash
docker compose -f compose.yml exec superset superset export-dashboards -f /tmp/dashboards.zip
docker compose -f compose.yml exec superset superset export-datasources -f /tmp/datasources.yaml
docker cp finops-superset:/tmp/dashboards.zip ./backup/dashboards-$(date +%F).zip
docker cp finops-superset:/tmp/datasources.yaml ./backup/datasources-$(date +%F).yaml
```

If your Superset image does not expose these CLI export commands, use UI export/import from `Settings` -> `Import/Export`.

---

## 10. Troubleshooting

### 10.1 `403 Access Denied` in BigQuery query

Cause:
- Service account missing required IAM roles

Fix:
- Grant `bigquery.dataViewer` and `bigquery.jobUser` on `gcp-rd-research-dev`

### 10.2 `Dataset ... not found` or table not visible

Cause:
- Wrong project/schema in DB connection

Fix:
- Ensure URI uses `gcp-rd-research-dev/finops_cloud_billing`

### 10.3 Location mismatch error

Cause:
- Querying datasets in different locations

Fix:
- Keep all queried marts in `us-west2`
- Keep URI `location=us-west2`

### 10.4 `No module named pybigquery` or BigQuery engine missing

Cause:
- Driver not installed in Superset image

Fix:
- Rebuild image after installing `sqlalchemy-bigquery` in Dockerfile

### 10.5 Credential file not found

Cause:
- `sa.json` path/mount mismatch

Fix:
- Verify compose volume mount:
  - `./secrets/sa.json:/app/secrets/sa.json:ro`
- Verify env:
  - `GOOGLE_APPLICATION_CREDENTIALS=/app/secrets/sa.json`
- For official compose path:
  - host file exists at `docker/.secrets/bigquery-sa.json`
  - container path resolves to `/app/docker/.secrets/bigquery-sa.json`
  - `GOOGLE_APPLICATION_CREDENTIALS=/app/docker/.secrets/bigquery-sa.json`

---

## 11. Production Enhancements (Appendix)

For post-MVP hardening:

1. Security and access
- Put Superset behind HTTPS reverse proxy (Nginx/ALB)
- Enable SSO/OIDC
- Disable admin password login in public networks

2. Reliability
- Use managed Postgres and managed Redis
- Add scheduled backups for metadata DB
- Add monitoring for container health and query latency

3. Scale
- Split web and async workers (Celery)
- Tune cache TTL by dashboard criticality
- Precompute heavy slices in dbt marts

4. Governance
- Add dataset owners and review workflow
- Add naming standard for charts/dashboards
- Track semantic changes when dbt mart schema evolves

---

## 12. Mapping To Current AWS MVP Model

The dashboard design in this guide is aligned with current MVP definitions:
- Usage-only marts from `int_aws_billing`
- Dual perspective metrics:
  - financial view: `actual_cost_usd`
  - R&D list-price view: `list_cost_usd`
- Optimization indicators:
  - `savings_amount`
  - `weighted_savings_rate`
  - `sp_coverage_rate`
- Dashboard v2 first release uses trend/distribution + detail table only (no KPI cards).

If finance/ledger marts are added later, create a separate Superset dashboard domain rather than mixing usage and invoice semantics in the same charts.