# Cluster Service Architecture

## Deployment Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                         Regional Control Plane                               │
│                                                                              │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    Kong Gateway (API Gateway)                        │    │
│  └──────┬──────────────┬──────────────┬──────────────┬─────────────────┘    │
│         │              │              │              │                       │
│         ▼              ▼              ▼              ▼                       │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌──────────────────┐          │
│  │  Cluster   │ │  ticdc-   │ │ dataplane- │ │  k8s-version-    │          │
│  │  Service   │ │  service  │ │  manager   │ │  manager         │          │
│  │  ◄──THIS───│ │           │ │            │ │                  │          │
│  └─────┬──────┘ └─────┬─────┘ └──────▲─────┘ └────────┬─────────┘          │
│        │              │               │                │                     │
│        │    Direct gRPC (internal)    │                │                     │
│        └──────────────┴───────────────┘                │                     │
│                                                        │                     │
└────────────────────────────────────────────────────────┼─────────────────────┘
                                                         │ Manages
                                                         ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           Data Plane (K8s Clusters)                          │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    Kubernetes Cluster (EKS/GKE/AKS/ACK)              │    │
│  │  ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ ┌─────────────┐  │    │
│  │  │tidb-operator │ │cluster-agent │ │service-proxy│ │ K8s Addons  │  │    │
│  │  └──────────────┘ └──────────────┘ └─────────────┘ └─────────────┘  │    │
│  │  ┌──────────────────────────────────────────────────────────────┐   │    │
│  │  │              TiDB Clusters / TiCDC Clusters                   │   │    │
│  │  └──────────────────────────────────────────────────────────────┘   │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘
```

### Service Roles

| Service | Role | Location |
|---------|------|----------|
| **Cluster Service** | TiDB cluster lifecycle (SharedPool/TiDBCluster/TiDBGroup) | `cmd/`, `domain/`, `pkg/` |
| **ticdc-service** | TiCDC cluster and Changefeed management | `internal/ticdc-service/` |
| **dataplane-manager** | K8s cluster and cloud resource management (Crossplane) | `internal/dataplane-manager/` |
| **k8s-version-manager** | K8s version upgrade orchestration | `internal/k8s-version-manager/` |
| **cluster-agent** | In-cluster agent (metrics, pd-ctl) | `internal/cluster-agent/` |

**Communication**: Cluster Service and ticdc-service call dataplane-manager via **internal gRPC**, bypassing Kong.

## Key Concepts

### 3-Tier Architecture

```
Global Control Plane (Global metadata)
        ↓
Regional Control Plane (Cluster Service + Dataplane Manager)
        ↓
Dataplane Layer (K8s Clusters)
```

### Data Model

```
SharedPool (n:m K8s Cluster)
  └─ DataplaneName: "eks-prod-us-east-1"
      └─→ TiDBCluster (N)
          └─→ TiDBGroup (N)
```

### API Design (Per-Region Deployment)

- ❌ No `cloud` parameter (service already in specific cloud)
- ❌ No `region` parameter (service already in specific region)
- ✅ `resource_pool_id` optional: empty=entire region, value=specific SharedPool

## Conventions

These are the repo-level rules most likely to break CI if ignored. The source of truth is the codebase and `.golangci.yml`; this section is the short operational map.

### Entity Type Aliases

- Add new shared protobuf types to `domain/entity/entity.go` first.
- Use `entity.*` across repo/service/server boundaries instead of raw `apiv1.*` types.
- Keep `apiv1` imports for enum constants or protobuf-local logic.

### Import Order and Formatting

- `make fmt` runs `gofmt` and `gci`.
- Keep imports deterministic: stdlib first, then external packages, then repo-internal packages.

```go
import (
    "context"

    "go.uber.org/zap"

    "github.com/tidbcloud/cluster-service-ng/domain/entity"
)
```

### Naming and Test Imports

- Follow existing file naming: `snake_case.go`.
- Keep receiver names short and local (`s`, `r`, `a`); use `ctx` for context parameters.
- In tests, follow nearby alias patterns already used in this repo; `pkg/storage/repo/mocks` is commonly imported as `repomocks`.

```go
import (
    repomocks "github.com/tidbcloud/cluster-service-ng/pkg/storage/repo/mocks"
)
```

### Error Handling and Logging

- Wrap internal errors with `fmt.Errorf(... %w)` so callers keep context.
- Return gRPC-facing errors with `status.Errorf(...)` when the API contract needs a specific code.
- Prefer contextual logs through `tilog` plus structured fields.

```go
return fmt.Errorf("failed to load sharedPool %s: %w", id, err)
return status.Errorf(codes.NotFound, "shared pool not found: %v", err)
tilog.Info(ctx, "scaling sharedPool", zap.String("sharedPoolId", id))
```

### NOLINT Guardrails

- Do not use bare `//nolint`.
- Keep `nolint` directives specific and explained, for example: `//nolint:errcheck // test helper intentionally ignores cleanup error`.

## Critical Patterns

### Load SharedPool → Get K8s Cluster

```go
sharedPool, err := sharedPoolRepo.Load(ctx, sharedPoolId, false)
clusterName := sharedPool.DataplaneName  // K8s cluster name
```

### UpdateNodePool (Replace Semantics!)

```go
// MUST preserve existing labels!
req := &dplanemgrv1.UpdateNodePoolRequest{
    DataPlaneId: dataplaneName,
    NodePoolId:  nodePool.GetName(),
    NodePool: &dplanemgrv1.NodePool{
        Labels: nodePool.GetLabels(),  // ← Preserve!
        Taints: updatedTaints,
    },
}
_, err := dataplaneManager.UpdateNodePool(ctx, req)
if err != nil {
    return status.Errorf(codes.Internal, "failed to update nodepool %s taint: %v", nodePool.GetName(), err)
}
```

Rules:
- `UpdateNodePool` behaves like a replace operation for the embedded `NodePool` payload.
- Preserve existing labels unless you intentionally want to rewrite them.

### Idempotency

```go
_, err := dataplaneManager.CreateNodePool(ctx, req)
if status.Code(err) == codes.AlreadyExists {
    return nil
}
if err != nil {
    return fmt.Errorf("failed to create nodepool: %w", err)
}
```

### TiDBGroup Type → Label Semantics

```go
// TiDBGroupType drives TiDB server labels.
switch tidbGroup.GetType() {
case apiv1.TiDBGroupType_INTERNAL:
    comp.Labels["internal.tidbcloud.com"] = "true"
    // Do not set keyspace label for INTERNAL.
case apiv1.TiDBGroupType_DEFAULT:
    comp.Labels["keyspace"] = tidbGroup.GetClusterId()
}
```

Rules:
- `TiDBGroupType` is immutable after creation in current design.
- `DEFAULT`: use keyspace label for tenant isolation.
- `INTERNAL`: use `internal.tidbcloud.com=true`, skip keyspace label.
- Validate type in `CreateTiDBGroup` and default unspecified to `DEFAULT`.

## Workflow System

Cluster Service uses a DAG (Directed Acyclic Graph) based workflow engine for orchestrating complex operations like cluster creation, updates, and deletion.

### Core Concepts

```
Plan (DAG Root)
  └─ Action Nodes (parallel/sequential execution)
       ├─ Action: PreCreateSharedPool
       ├─ Action: CreateDataplane
       ├─ Action: ApplyCluster
       └─ ...
```

| Concept | Description | Location |
|---------|-------------|----------|
| **Plan** | A named DAG definition | `domain/workflow/dag/node.go` |
| **Action** | Single unit of work | `domain/workflow/action/*.go` |
| **Executor** | Runs DAGs with retries | `pkg/runtime/executor/` |

### Adding a New Action

```go
// 1. Define action name constant
const MyActionName = "my-action"

// 2. Implement Action interface
type MyAction struct {
    sharedPoolRepo repo.SharedPoolRepo
    dataplaneManager dataplane.Manager
}

func NewMyAction(...) actions.Action {
    return &MyAction{...}
}

func (a *MyAction) Name() string {
    return MyActionName
}

func (a *MyAction) Invoke(args actions.ActionArgs) error {
    ctx := utils.GenerateContext(args)
    
    // Get arguments
    sharedPoolId, err := args.GetString(workflow.SharedPoolIdKey)
    if err != nil {
        return fmt.Errorf("failed to get sharedPool id: %w", err)
    }
    
    // Implementation
    // ...
    
    return nil
}

// 3. Register in domain/workflow/dag/dag.go
func registerActions(actionLib *actions.ActionLibrary, ...) {
    must(actionLib.Register(action.NewMyAction(...)))
}

// 4. Add to DAG in domain/workflow/dag/node.go
func createSharedPoolDag(dagPlans *dag.Plans) *dag.Plans {
    createSharedPool := dagPlans.NewPlanWithRootNode(PlanCreateSharedPool)
    createSharedPool.
        SafeAddDependNode(action.MyActionName).  // ← Add here
        SafeAddDependNode(action.CreateDataplaneActionName)
    return dagPlans
}
```

### Action Arguments

```go
// Passing arguments to actions
args := actions.ActionArgs{
    workflow.SharedPoolIdKey: actions.ActionArgString("pool-123"),
    workflow.CountKey:        actions.ActionArgInt(3),
}

// Reading arguments in action
sharedPoolId, _ := args.GetString(workflow.SharedPoolIdKey)
```

### Built-in Action Keys

| Key | Type | Description |
|-----|------|-------------|
| `SharedPoolIdKey` | string | SharedPool identifier |
| `TiDBClusterIdKey` | string | TiDBCluster identifier |
| `TiDBGroupIdKey` | string | TiDBGroup identifier |
| `CountKey` | int | Numeric parameter |

## Database

### Schema Definition

```go
// pkg/storage/mysql/schema/schema.go
type MyEntity struct {
    ID        string `gorm:"type:varchar(128);primarykey"`
    Name      string `gorm:"type:varchar(256);not null;index"`
    Config    string `gorm:"type:text"`              // JSON stored as text
    Status    string `gorm:"type:enum('pending','running');default:'pending'"`
    CreatedAt time.Time
    UpdatedAt time.Time
    DeletedAt gorm.DeletedAt `gorm:"index"`          // Soft delete
}

// Register model
var Models = []any{
    &SharedPool{},
    &MyEntity{},  // ← Add here
}
```

### Repository Pattern

```go
// Define interface
type MyEntityRepo interface {
    Load(ctx context.Context, id string) (*schema.MyEntity, error)
    Create(ctx context.Context, entity *schema.MyEntity) error
    Update(ctx context.Context, entity *schema.MyEntity) error
    Delete(ctx context.Context, id string) error
}

// Implementation
func (r *myEntityRepo) Load(ctx context.Context, id string) (*schema.MyEntity, error) {
    var entity schema.MyEntity
    if err := r.db.WithContext(ctx).First(&entity, "id = ?", id).Error; err != nil {
        return nil, fmt.Errorf("failed to load myEntity %s: %w", id, err)
    }
    return &entity, nil
}
```

### Transaction Handling

```go
// Using transaction helper
err := db.Transaction(func(tx *gorm.DB) error {
    if err := repo.Create(tx, entity); err != nil {
        return err
    }
    if err := otherRepo.Update(tx, other); err != nil {
        return err
    }
    return nil
})
```

### DB Migration Behavior

- Registration: `pkg/storage/mysql/migrate/db/migrate_*.go` via `AddDBMigrateRecord(version, func)`.
- Sorted by `version`, executed by gormigrate table `cluster_next_gen_server_migrations`.
- Runs with `UseTransaction: false` — write idempotent, restart-safe SQL.
- Prefer forward-only, additive changes. New non-null columns need safe defaults.
- For MySQL DDL, prefer `ADD COLUMN IF NOT EXISTS` guards.
- Version format: `YYYYMMDDNNN` (e.g. `20260305001`).

Example:
```go
func migrate20260305001(db *gorm.DB) error {
    return db.AutoMigrate(&schema.TidbGroup{})
}
```

### Deploy Migration Behavior

- Location: `pkg/storage/mysql/migrate/deploy/*.go`.
- Each migration is a method on `*migrator` returning `gormigrate.MigrateFunc`, registered by `registerMigrateRecord((*migrator).methodName)`.
- Migration ID derived from Go method name (reflection), sorted lexicographically.
- State stored in gormigrate table `cluster_next_gen_deploy_migrations`.
- Also `UseTransaction: false` — must be idempotent and retry-safe.
- Naming: keep method names monotonic (e.g. `migrate20260227`, `updateClusterAgent_20260303041758`).

## Template System

Templates render Kubernetes CRDs (TiDBCluster, TiKVGroup, etc.) with variable substitution.

### Template Location

```
deployments/templates/
├── tidb_cluster.templ      # Main TiDBCluster CR
├── tikv_group.templ        # TiKVGroup CR
├── tidb_group.templ        # TiDBGroup CR
└── ...
```

### Template Syntax

```yaml
# tidb_cluster.templ
apiVersion: pingcap.com/v1alpha1
kind: TiDBCluster
metadata:
  name: {{.ClusterName}}
  labels:
    app.kubernetes.io/component: tidb-cluster
spec:
  version: {{.TiDBVersion}}
  tikv:
    replicas: {{.TiKVReplicas}}
    requests:
      storage: {{.TiKVStorage}}Gi
```

### Template Rendering

```go
rendered, err := templateLib.Render("tidb_cluster", map[string]string{
    "ClusterName":  "my-cluster",
    "TiDBVersion":  "v8.1.0",
    "TiKVReplicas": "3",
    "TiKVStorage":  "100",
})
err = dataplaneManager.ApplyYAML(ctx, clusterName, rendered)
```

## Cron Jobs

### Interface

```go
type CronJob interface {
    Name() string           // Unique identifier
    CronExpression() string // "0 */6 * * * *" (with seconds)
    Execute(ctx context.Context) error
}
```

### Adding a Cron Job

```go
// 1. Implement the job
type MyCronJob struct {
    sharedPoolRepo repo.SharedPoolRepo
}

func (j *MyCronJob) Name() string           { return "my-cron-job" }
func (j *MyCronJob) CronExpression() string  { return "0 0 */6 * * *" }
func (j *MyCronJob) Execute(ctx context.Context) error {
    // Job logic
    return nil
}

// 2. Register in cmd/executor/main.go or cmd/cluster_ng_server/main.go
cronManager.AddJob(cron.NewMyCronJob(sharedPoolRepo))
```

## Protobuf / gRPC

### File Responsibilities

- `api/cluster-service-ng/v1beta/resources.proto`: shared resource models and reusable enums/messages.
- `api/cluster-service-ng/v1beta/services.proto`: service definitions and endpoint-specific request/response wrappers.
- Reused types or domain resource shapes → `resources.proto`.

### Adding a New API Endpoint

```protobuf
// 1. Define in services.proto
service SharedPoolService {
  rpc MyOperation(MyRequest) returns (MyResponse) {
    option (google.api.http) = {
      post: "/sharedPools/{shared_pool_id}:myAction"
      body: "*"
    };
  }
}

// 2. Define messages in resources.proto
message MyRequest {
  string shared_pool_id = 1 [
    (buf.validate.field).string.uuid = true,
    (google.api.field_behavior) = REQUIRED
  ];
  int32 count = 2 [
    (buf.validate.field).int32.gt = 0,
    (google.api.field_behavior) = REQUIRED
  ];
}

message MyResponse {
  SharedPool shared_pool = 1;
}
```

### Generation

```bash
make proto           # buf lint + format + generate
make generate-mocks  # if Go interfaces changed
make generate        # proto + manifests (full codegen)
```

## Testing

### Table-Driven Pattern

```go
tests := []struct {
    name          string
    setupMocks    func(*repomocks.SharedPoolRepo, ...)
    expectedCode  codes.Code
    validate      func(*testing.T, *Response, error)
}{...}

for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) { ... })
}
```

### Mock Imports

```go
repomocks "github.com/tidbcloud/cluster-service-ng/pkg/storage/repo/mocks"
```