# AGENTS.md — Observability Platform

> TiDB Cloud Observability platform. Go monorepo (go.work).
> Module path: `github.com/tidbcloud/observability`

---

## Quick Reference

```bash
make build              # Build all servers
make test               # Unit tests (local)
make lint               # Lint (golangci-lint v2.2.2 + goimports)
make mock               # Regenerate mocks after interface changes
make swag               # Regenerate Swagger docs
make manifests          # Regenerate CRD/webhook manifests
make reviewable         # Run generation + lint + diff-clean checks (CI gate)
make e2e                # Build FT test binary -> build/bin/cases.test
```

Build individual servers: `make global_server`, `make regional_server`, `make regional_data_server`, `make ops_server`. Binaries go to `build/bin/`. Build uses `-tags codes`.

---

## Project Structure

```
cmd/                     # Server entrypoints (globalserver, regionalserver, dataserver, opsserver)
pkg/
  common/                # Shared: errors, constants, configs, metrics, structs, interceptors
  globalapi/             # Global server HTTP handlers
  globalservice/         # Global server business logic
  regionalapi/           # Regional server HTTP handlers
  regionalservice/       # Regional server business logic
  dataapi/               # Data server API handlers
  dataservice/           # Data service logic
  resourcemanager/       # K8s resource management interfaces (16 ReaderWriter interfaces)
  resourceworker/        # Cloud resource provisioning + K8s controllers/reconcilers
  metamanager/           # Metadata management (etcd, S3)
  diagcontroller/        # Diagnosis task orchestration
api/                     # CRD type definitions (regionalserver/resourceworker/v1/)
agent-control-server/    # Separate module — K8s operator for agent management
data-collector/          # Separate module — data collection service
o11ydiagnosis/           # Diagnosis and analysis packages
o11ymetering/            # Metering packages (has own AGENTS.md)
evaluation/              # AI-powered evaluation services
test/
  mock*/                 # Generated mocks
  ft/                    # Functional (e2e) tests
    cases/               # Main FT test suite (cluster, integration task, serverless)
    utils/               # Shared test utilities (auth, HTTP helpers, config)
    datacheck/           # Data verification tests
    datalink/            # Data link tests
    dedicatede2e/        # Dedicated cluster e2e tests
    logscheck/           # Log verification tests
    performance/         # Performance tests
deployments/             # K8s manifests, Helm charts, scripts
```

Pattern: `{service}api/` for HTTP handlers, `{service}service/` for business logic.

---

## Testing

### Unit Tests

```bash
make test                                                    # All unit tests
go test -v -run TestFunctionName ./pkg/path/to/package/...   # Single test
go test -v -run "TestName/subtest" ./pkg/path/to/package/... # Single subtest
WITH_RACE=1 go test -v ./pkg/path/to/package/...             # With race detector
```

- Framework: `github.com/stretchr/testify/assert`
- Mocking: `go.uber.org/mock/gomock` — generated via `make mock`
- Test files alongside source: `foo.go` -> `foo_test.go`
- Coverage threshold: 86% warning, 90% target

```go
func Test_CreateCluster(t *testing.T) {
    ctrl := gomock.NewController(t)
    defer ctrl.Finish()
    mockRW := mockresourcemanager.NewMockClusterReaderWriter(ctrl)
    resourcemanager.MockClusterReaderWriter(mockRW)
    t.Run("success case", func(t *testing.T) {
        mockRW.EXPECT().Create(gomock.Any(), gomock.Any()).Return(nil)
        // ... test logic
        assert.NoError(t, err)
    })
}
```

### Functional Tests (E2E)

FT tests live in `test/ft/` and run against live staging environments. They use build tag `e2e`.

**Build & run:**

```bash
make e2e                                         # Build binary -> build/bin/cases.test
go test -v -tags e2e -run TestName ./test/ft/cases/...  # Run specific test locally
```

**Configuration:** Copy `test/ft/cases/e2e_config_tmpl.yaml` to `test/ft/cases/e2e_config.yaml` and fill in:

| Field | Purpose |
|-------|---------|
| `service.serverURL` | Global Server (GS) URL |
| `service.regionalURL` | Regional Server (RS) URL |
| `service.dsURL` | Data Server (DS) URL |
| `service.vendor` / `service.region` | Cloud provider and region |
| `auth0.*` | Auth0 M2M credentials (client_credentials grant) |
| `dsAuth.url` / `dsAuth.githubToken` | DS auth for integration task tests (GitHub PAT) |

**Test lifecycle** (`TestMain` in `main_test.go`):

1. Load config from `e2e_config.yaml`
2. Obtain Auth0 token (for GS/RS APIs) and DS token (for integration task APIs)
3. Initialize K8s client
4. Clear leftover FT namespaces
5. Create test clusters (unless `skipPrepare: true`)
6. Run tests
7. Clean up namespaces

**Auth patterns:**

- **GS/RS APIs**: Auth0 token via `utils.PrepareToken()` -> `utils.WithHeader(req)` sets `Authorization: Bearer <token>`
- **DS/Integration task APIs**: GitHub PAT -> DS `/client/auth` -> bearer token via `utils.PrepareDSToken()` -> `utils.WithDSAuthHeader(req)` sets Authorization + vendor/region headers

**Key utilities** (`test/ft/utils/utils.go`):

| Function | Purpose |
|----------|---------|
| `CreateCluster(req)` | POST to GS, returns `*CreateClusterResp` |
| `DeleteCluster(id)` | DELETE via GS |
| `GetCluster(id)` / `GetClusterIfExisted(id)` | GET cluster detail |
| `QueryClusters(req)` | List clusters with filters |
| `CheckStatus(t, id, getFn, intermediate, final)` | Poll until resource reaches target status |
| `HandleWithDefault(fn)` | Apply customization to default shared cluster request |
| `WithHeader(req)` | Inject Auth0 auth headers |
| `WithDSAuthHeader(req)` | Inject DS auth + vendor/region headers |
| `UnmarshalRespData(data, v)` | Unwrap `{"code":0,"data":...}` response envelope |

**Writing a new FT test:**

```go
//go:build e2e
// +build e2e

package cases

func Test_MyFeature(t *testing.T) {
    s := utils.GetService()
    // Use utils helpers to build requests
    req, _ := http.NewRequest("GET", fmt.Sprintf("%s/api/v1/...", s.ServerURL), nil)
    utils.WithHeader(req) // Auth0 token
    resp, err := http.DefaultClient.Do(req)
    assert.NoError(t, err)
    defer resp.Body.Close()
    assert.Equal(t, http.StatusOK, resp.StatusCode)
    // Parse response using utils.UnmarshalRespData or utils.HandleResp
}
```

**BizType constants** (used in test cluster creation):

| Constant | Value | Usage |
|----------|-------|-------|
| `DefaultBizType` | `ft-biz` | Standard FT test clusters |
| `NexgenBizType` | `dev-tidbcloud-nextgen` | NextGen cluster tests |
| `ServerlessBizType` | `dev-devtier` | Serverless cluster tests |
| `DedicatedBizType` | `tidbcloud` | Dedicated cluster tests |

---

## Code Style & Conventions

**Go version:** 1.25+ | **Formatter:** goimports | **GOPRIVATE:** github.com/tidbcloud

**Imports** — group with blank lines between:

1. Standard library
2. Third-party
3. Internal (`github.com/tidbcloud/observability/...`)

**Logging:** `github.com/pingcap/log` + `go.uber.org/zap`. Never use `fmt.Println` or stdlib `log`.

```go
log.Info("message", zap.String("key", value), zap.Error(err))
```

**Error handling:** Use `pkg/common/errors` for business logic errors:

```go
errors.New(errors.ParameterInvalid, "invalid cluster ID")
errors.Wrap(errors.K8SClientError, "failed to create namespace", err)
if errors.Is(err, errors.ClusterNotFound) { ... }
```

Error codes defined in `pkg/common/errors/code.go`. For non-business errors, use standard Go errors.

**Naming:** Files `snake_case.go`, packages lowercase, constants `PascalCase`, interfaces in `interface.go`, constructors `NewXxx()` or `GetHandler()` singleton.

**HTTP handlers:** Swagger-annotated, request/response types in `pkg/common/message/` and `pkg/common/structs/`.

---

## CI Pipeline

On PR to `main`/`release`:

1. `make reviewable` + `git diff --exit-code` (auto-fixed changes must be committed)
2. `make ci_test` with JUnit reporting
3. Coverage via codecov (86%/90%)
4. Chinese character check (no Chinese in code/comments, excluding .md/.json/.yaml)
5. `gosec` security scan

Linters: bodyclose, errcheck, gocyclo (threshold 25), gosec, misspell, prealloc, predeclared, ineffassign, govet, staticcheck, unconvert, unused.

Separate CI pipelines for `agent-control-server/`, `data-collector/`, `o11ymetering/`.

---

## Key Dependencies

| Purpose | Library |
|---------|---------|
| HTTP | `github.com/gin-gonic/gin` |
| Logging | `github.com/pingcap/log` + `go.uber.org/zap` |
| Testing | `github.com/stretchr/testify` + `go.uber.org/mock` |
| K8s | `sigs.k8s.io/controller-runtime`, `k8s.io/client-go` |
| ORM | `gorm.io/gorm` |
| Config | `github.com/spf13/viper` |
| Validation | `github.com/go-playground/validator/v10` |
| Cloud | AWS SDK v2, GCP client libs, Azure SDK |

---

## Rules

- **No Chinese characters** in Go code or comments — CI rejects.
- **No type error suppression** (`as any` / `@ts-ignore` equivalents).
- `make reviewable` must pass before pushing and in PR CI.
- Commit with semantic commit messages and Pull Request with the description of `.github/PULL_REQUEST_TEMPLATE.md`.
- Keep cyclomatic complexity below 25.
- All new interfaces need mocks: add to `make mock` targets.