# AGENTS.md — o11ymetering

> Metering service for TiDB Cloud Observability. Separate Go module.
> Module path: `github.com/tidbcloud/observability/o11ymetering`
> Parent: see [root AGENTS.md](../AGENTS.md) for shared conventions (logging, errors, imports, CI).

---

## Quick Reference

```bash
# All commands run from o11ymetering/ directory
make mock              # Regenerate mocks (required before tests)
make ut                # Unit tests + coverage HTML at /tmp/o11ymetering/cover.html
make ci_ut             # CI-style tests (JUnit XML + coverage XML)
make lint              # golangci-lint v2 + goimports (runs mock first)
make metering-server   # Build binary -> build/bin/observability-regional-metering-server
make docker            # Build Docker image (linux/amd64)
```

Single test: `go test -v -run TestFunctionName ./pkg/path/to/package/...`

Build uses `-tags codes`. Requires `EDITION=Enterprise` (enforced by Makefile.common).

---

## Architecture

Layered + Facade/Adapter pattern:

```
cmd/main.go                    # Entry: flags, kube config, config load, start schedulers + HTTP
  |
  +-- pkg/schedule/            # Background tickers (statistic, merge, report, proportion)
  |     Task.AsyncTicker() -> Task.Execute() -> checkpoint + execute + upload
  |
  +-- pkg/httpapi/             # Gin HTTP server, routes, request/response helpers
  |     POST /api/v1/metering/statistic/tasks/
  |     PUT  /api/v1/metering/statistic/tasks/:id/execution/
  |     POST /api/v1/metering/statistic/tasks/replay
  |
  +-- pkg/service/statistic/   # Business logic: generate tasks, execute, report
  |     +-- handler/           # TaskHandler implementations (metrics/loki/s3 x serverless/dedicated/cloud)
  |
  +-- pkg/facade/              # Adapter layer
        +-- iface/             # Interface definitions (ports)
        +-- factory.go         # Wires implementations via global Get/Set accessors
        +-- database/          # GORM-based RW (SQLite per-period + MySQL/TiDB Serverless)
        +-- s3storage/         # S3/GCS/Aliyun OSS client, checkpoint, data sync
        +-- o11ymeta/          # GS API client for cluster metadata
        +-- o11ymetrics/       # Regional DS API client for metrics queries
```

---

## Scheduler Tasks in `cmd/main.go`

`cmd/main.go` starts four background workflows:

```go
schedule.StatisticTask().AsyncTicker()
schedule.StatisticMergeTask().AsyncTicker()
schedule.ReportTask().AsyncTicker()
schedule.MonthlyProportionSumGlobally()
```

### What each task does + related DB tables

| Task | Main Logic | Trigger/Gate | Related Tables |
|------|------------|--------------|----------------|
| `StatisticTask().AsyncTicker()` | Generates and executes raw statistic tasks per 5-minute period. Writes per-period local SQLite and uploads task data to object storage. | `NextExecuteTimestamp` + `ExecuteReady` in `pkg/schedule/executestatistictask.go` | `statistic_tasks`, `statistic_records` (in per-period SQLite, e.g. `data/metering/statistic/<timestamp>.sqlite`) |
| `StatisticMergeTask().AsyncTicker()` | Merges many period-level statistic files into monthly merge SQLite by copying records from `statistic` data files. | `ReadyToMerge` requires `statistic` checkpoint to pass next-month boundary (`pkg/schedule/statisticmerge.go`) | `statistic_records` (target monthly merge SQLite `data/metering/merge_statistic/<timestamp>.sqlite`; same model set also includes `statistic_tasks`) |
| `ReportTask().AsyncTicker()` | Builds monthly shared-cost reports from merged statistic data and inserts reporting rows into report DB. | `ReadyToReport` checks merge checkpoint + `o11y_cost` readiness (`IfCostReady`) | Reads `statistic_records` (merge SQLite), reads `o11y_cost`, writes `monthly_statistic_reports` (GORM model `MonthlyStatisticReport`) |
| `MonthlyProportionSumGlobally()` | Computes tenant proportion for previous month, inserts proportion rows, then uploads CSV to S3 (`metering/split/o11y_usage_proportion_<YYYY-MM>.csv`). | Immediate-on-start + hourly ticker; condition is `(4th day 00:00 && monthlyProportion) OR (renewMonthlyProportionTs > 0 && now > renewMonthlyProportionTs)` | Reads `monthly_statistic_reports`, writes `monthly_proportion_sum` |

### Checkpoint note (important)

- `StatisticTask` / `StatisticMergeTask` / `ReportTask` use `Task.Execute()` flow and S3 checkpoint files via `CheckpointReaderWriter`.
- `MonthlyProportionSumGlobally` **does not** use the `Task` checkpoint pipeline; it is config-driven (`monthlyProportion`, `renewMonthlyProportionTs`) in `pkg/schedule/monthlyproportionsum.go`.

---

## Package Map

| Package | Purpose | Key Types |
|---------|---------|-----------|
| `cmd/` | Entry point. Flag parsing, kube/config init, scheduler start | — |
| `pkg/httpapi/` | Gin server, routes, request binding, response envelope | `StartService()`, `HandleRequest()` |
| `pkg/service/statistic/` | Task generation, execution orchestration, reporting | `GenerateStatisticTask()`, `ExecuteStatisticTask()` |
| `pkg/service/statistic/handler/` | Concrete metric/log collectors per cluster type | `TaskHandler` interface, 15 handler structs |
| `pkg/schedule/` | Background periodic task execution | `Task` struct, `AsyncTicker()` |
| `pkg/facade/iface/` | Interface definitions (ports) | `StatisticReaderWriter`, `O11yMetaQuery`, `O11yMetricsQuery`, `CheckpointReaderWriter` |
| `pkg/facade/factory.go` | Singleton registry wiring implementations | `SwitchStatisticReaderWriter()`, `GetO11yMetaQuery()` |
| `pkg/facade/database/` | GORM adapters: SQLite (per-period local files) + MySQL | `GormRW`, `GetSqliteDB()` |
| `pkg/facade/s3storage/` | Multi-cloud storage: AWS S3, GCP GCS, Aliyun OSS | `Client` interface, `S3Checkpoint` |
| `pkg/facade/o11ymeta/` | GS API client (Auth0 token, cluster metadata cache) | `O11yMetaAPI` |
| `pkg/facade/o11ymetrics/` | Regional DS metrics API client | `O11yRegionalDataApi` |
| `pkg/common/models/` | Domain models | `StatisticTask`, `StatisticRecord`, `MonthlyStatisticReport`, `Period` |
| `pkg/common/config/` | YAML config loading + env var secret override | `LoadMeteringConfig()`, `GetDatabases()`, `GetAuth0Config()` |
| `pkg/common/constants/` | Enums: `StatisticClass`, `MeteringGroup`, `Category`, `StatisticName` | — |
| `pkg/common/message/` | API request/response payload types | — |
| `mock/pkg/facade/` | Generated mocks (mockgen) | `mock_o11ymetering` package |

---

## Key Interfaces

All in `pkg/facade/iface/`. Mocks generated by `make mock` into `mock/pkg/facade/`.

| Interface | File | Implementations |
|-----------|------|-----------------|
| `StatisticReaderWriter` | `dbreaderwriter.go` | `database.GormRW` (SQLite) |
| `StatisticMergeReaderWriter` | `dbreaderwriter.go` | `database.GormRW` (SQLite) |
| `StatisticReportReaderWriter` | `dbreaderwriter.go` | `database.GormRW` (MySQL/SQLite) |
| `ProportionReaderWriter` | `dbreaderwriter.go` | `database.GormRW` |
| `O11yMetaQuery` | `clustermeta.go` | `o11ymeta.O11yMetaAPI` |
| `O11yMetricsQuery` | `metricsquery.go` | `o11ymetrics.O11yRegionalDataApi` |
| `CheckpointReaderWriter` | `scheduletask.go` | `s3storage.S3Checkpoint` |
| `ScheduleTaskDataSync` | `scheduletask.go` | `s3storage.S3ScheduleTaskData` |
| `TaskHandler` | `handler/taskinterface.go` | 15 handlers in `handler/` (metrics/loki/s3 x cluster type) |
| `Client` (S3) | `s3storage/client.go` | `S3Client`, `GCSClient`, `AlicloudClient` |

---

## Testing

- Framework: `testify/assert` + `go.uber.org/mock/gomock`
- DB tests use in-memory SQLite via `glebarez/sqlite`
- No build tags — all unit tests, no e2e in this module
- `UT_IGNORE_FILES`: `replay.go` excluded from coverage
- `PACKAGES` (Makefile.common): `pkg/service/statistic/`, `pkg/common/models/`, `pkg/common/config/`, `pkg/facade/database/`

Test pattern:
```go
ctrl := gomock.NewController(t)
defer ctrl.Finish()
mockDB := mock_o11ymetering.NewMockStatisticReaderWriter(ctrl)
facade.SetStatisticReaderWriter(mockDB)  // inject via facade setter
mockDB.EXPECT().BatchCreateTask(gomock.Any(), gomock.Any()).Return(tasks, nil)
// ... call service function, assert results
```

---

## Deviations from Root Conventions

| Area | Root Convention | o11ymetering Actual |
|------|----------------|---------------------|
| Logging | `pingcap/log` + `zap` only | Mixed: `logrus` (facade/database, s3, report), `pingcap/log` + `zap` (service), stdlib `log` (cmd) |
| Config | `spf13/viper` | Direct YAML parsing (`gopkg.in/yaml.v3`) + env var override |
| Error handling | `pkg/common/errors` for business errors | HTTP layer uses it; business/facade code returns plain `errors.New`/`fmt.Errorf` |
| Cyclomatic complexity | Threshold 25 | Module uses threshold 30 (`.golangci.yml` missing gocyclo; Makefile `cyclo` target uses `-over 30`) |

---

## Anti-Patterns (Known Issues)

- **Panics in production code**: `facade/factory.go` panics on DB init failure, missing config, nil singletons. `schedule/schedule.go` panics on checkpoint/execute/upload errors. Should return errors.
- **Unimplemented methods**: `database/gorm.go` has `DeleteRecordsByPeriod()` and `DeleteTasksByPeriod()` with `panic("implement me")`.
- **Global mutable singletons**: `facade/factory.go` uses package-level vars (`statisticRW`, `statisticMergeRW`, etc.) switched in-place. Not concurrency-safe. Test injection via `SetXxx()` functions.
- **`context.TODO()` usage**: `schedule/schedule.go` and `s3storage/` use `context.TODO()` instead of propagating context from callers.
- **Per-period SQLite files**: Created at `data/metering/statistic/<timestamp>.sqlite`. Managed via global state — potential file descriptor leaks if not closed properly.

---

## Adding a New Metering Handler

1. Create `pkg/service/statistic/handler/<name>_collection.go`
2. Implement `TaskHandler` interface (`Build` + `Execute`)
3. Register in `pkg/service/statistic/statisticservice.go` → `pickTaskHandler()` switch
4. Add constants in `pkg/common/constants/metering.go` if new `StatisticClass`/`MeteringGroup`
5. Write tests using mocks: `mock_o11ymetering.NewMockO11yMetricsQuery(ctrl)` etc.
6. Run `make mock && make ut && make lint`

---

## Config

Loaded from YAML file (`--meteringConfig` flag, default `./config/meteringconfig.yaml`).
Secrets overridden by env vars in `UpdateMeteringConfigSecret()`:
- `AUTH0_CLIENT_SECRET` → Auth0 config
- `DATABASE_DSN` → DB connection string

Account info from `--accountInfo` flag (default `./config/accountinfo.yaml`).

---

## Deployments

K8s manifests in `deployments/server/{dev,staging,prod}/` across AWS, GCP, Alicloud.
Docker image: `build/package/metering/Dockerfile` (Alpine-based, single binary).
CI/CD: `.github/workflows/metering-ci.yml` and `.github/workflows/build-metering.yaml`.