# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Quick Start

```bash
# Build all components
make build

# Run lint and tests
make reviewable

# Build and push to ECR
make VAR_IMG_HUB=ecr push

# Run e2e tests
make VAR_INFRA_KUBECONFIG=infra-api-kubeconfig.yaml e2e

# Format code
make fmt
```

## Architecture Overview

The AWS Shared Provider is a **Kubernetes-based infrastructure controller** for TiDB Cloud DBaaS free tier on AWS. It manages multi-tenant TiDB clusters using a serverless architecture with shared resources and keyspace isolation.

### Core Components

#### **Manager** - The Central Controller
The **manager** is the primary controller that orchestrates the entire TiDB Cloud infrastructure. It's a Kubernetes operator that:

- **Manages CRDs**: Watches and reconciles Serverless and Gateway CRDs
- **Orchestrates Controllers**: Coordinates 18+ specialized controllers for different aspects
- **Handles Multi-Cloud**: Supports AWS and AliCloud with region-specific configurations
- **Provides gRPC API**: Exposes management APIs for external clients
- **Manages TiDB Lifecycle**: Handles cluster creation, deletion, scaling, and upgrades
- **Integrates Services**: Connects with PD, TiKV, tidb-worker, and external AWS services

**Key Capabilities**:
- Cluster lifecycle management (create/scale/delete)
- Branch operations (fork/restore databases)
- Import/export job coordination
- Keyspace allocation and binding
- Node management (draining, rebalancing, recovery)
- Autoscaling based on metrics
- Security and secret management
- Zone-redundant pod eviction for high availability

**Other Components**:
- **tidb-worker**: Background tasks (GC, DDL, batch processing, TTL, auto-analyze)
- **tidb-gateway**: MySQL protocol gateway for client connections
- **tidb-http-gateway**: HTTP API access layer
- **audit-log**: Audit log collection and processing
- **datacleaner**: Data cleanup and archival
- **vpa-agent**: Vertical Pod Autoscaler agent
- **csi-node-taint**: CSI node taint controller
- **eks-qps**: EKS QPS monitoring
- **etcd-qps**: Etcd QPS monitoring
- **gateway-controller**: Gateway controller
- **meta-server**: Metadata server
- **ops-service**: Operations service

### Key Abstractions

- **Keyspace**: Logical database isolation unit for multi-tenancy
- **Cluster/Branch**: Customer-facing database instances
- **TiDBPool**: Physical compute resources allocated to keyspaces
- **Regional pools**: Geographic distribution across AWS regions
- **NodeScore**: Resource allocation scoring for keyspaces
- **TiDBAutoscaling**: Automatic scaling configuration for TiDB instances
- **Gateway**: Gateway configuration for load balancing and routing

## Development Commands

### Build & Test

```bash
make build              # Build all binaries
make lint               # Run golangci-lint
make unit-test          # Run unit tests
make e2e                # Run end-to-end tests
make reviewable         # Build + lint + test (PR ready)
make fmt                # Format Go code (alias for fmt-modified)
make fmt-modified       # Format only modified Go files (compared to HEAD)
make fmt-staged         # Format only staged Go files (git add'd files)
make fmt-all            # Format all Go files in the project
make fmt-check          # Check if Go files need formatting (CI-friendly)
make setup-hooks        # Install Git hooks for automatic Go formatting
make generate           # Generate code (CRDs, protobuf, etc.)
make tidy               # Tidy go modules (go mod tidy)
make coverage           # Generate coverage html under coverage/ after running tests
make crd                # CustomResourceDefinition objects
make kustomize          # Download kustomize locally if necessary
make help               # Display help
```

### Docker & Registry

```bash
make build-in-docker   # Build in Docker with SSH for private deps
make docker-build      # Docker build (alias for build-in-docker)
make docker-push       # Push Docker images
make VAR_IMG_HUB=ecr push   # Build and push to ECR
make VAR_IMG_HUB=ecr VAR_PLATFORMS=linux/arm64,linux/amd64 push  # Multi-arch
```

### Environment Setup

```bash
# Get kubeconfigs for testing
aws eks update-kubeconfig --name dev-base-us-west-2 --region us-west-2
aws eks update-kubeconfig --name dev-us-east-1-f01 --region us-east-1

# Infra API access
kubectl get secret gardener-provider-infra-kubeconfig --template={{'.data.config'}} -n infra | base64 -d > infra-api-kubeconfig.yaml
```

### Testing Options

```bash
# Focus specific tests
FOCUS=cluster make e2e

# Custom cluster name
RANDOM_CLUSTER=my-test make e2e

# Skip deployment
SKIP_DEPLOY=1 make e2e

# Skip cleanup (keep resources after test)
SKIP_CLEANUP=1 make e2e

# Cleanup
CLEANUP=1 make e2e

# Debug mode
DEBUG=1 make e2e

# Parallel execution
PARALLEL=4 make e2e
```

## Directory Structure

```bash
├── apis/                 # Kubernetes CRD type definitions and schemas
│   ├── serverless/      # Serverless CRDs (Keyspace, NodeScore, TiDBAutoscaling)
│   │   └── v1alpha1/    # v1alpha1 API versions
│   └── gateway/         # Gateway CRDs (Gateway configurations)
│       └── v1alpha1/    # v1alpha1 API versions
├── cmd/                 # Main application entry points and binaries
│   ├── manager/         # Central Kubernetes controller manager
│   ├── tidb-worker/     # Background task processor (GC, DDL, batch operations)
│   ├── tidb-gateway/    # MySQL protocol gateway for client connections
│   ├── tidb-http-gateway/ # REST API gateway for HTTP clients
│   ├── audit-log/       # Audit log collection and processing service
│   ├── datacleaner/     # Data cleanup and lifecycle management service
│   ├── vpa-agent/       # Vertical Pod Autoscaler recommendation agent
│   ├── csi-node-taint/  # CSI node taint controller
│   ├── eks-qps/         # EKS QPS monitoring service
│   ├── etcd-qps/        # Etcd QPS monitoring service
│   ├── gateway-controller/ # Gateway controller
│   ├── meta-server/     # Metadata server
│   └── ops-service/     # Operations service
├── pkg/                 # Reusable library code and shared implementations
│   ├── controllers/     # Kubernetes controller implementations (18+ controllers)
│   │   ├── branch/          # Branch operations controller
│   │   ├── cluster/         # Cluster lifecycle management
│   │   ├── export/          # Export job coordination
│   │   ├── imports/         # Import job coordination
│   │   ├── keyspace/        # Keyspace management
│   │   ├── keyspacepoolconfig/ # Keyspace pool configuration
│   │   ├── noderebalance/   # Node rebalancing
│   │   ├── noderecover/     # Node recovery
│   │   ├── secretsyncer/    # Secret synchronization
│   │   ├── tidbnodescaler/  # TiDB node scaling
│   │   ├── tidbpod/         # TiDB pod management
│   │   ├── tidbpodevicter/  # TiDB pod eviction
│   │   ├── tidbpodrescheduler/ # TiDB pod rescheduling
│   │   ├── tidbservertemplate/ # TiDB server templates
│   │   └── zoneredundantevicter/ # Zone-redundant eviction
│   ├── tidbworker/      # Background task framework and processors
│   │   ├── admin/           # Admin task handlers
│   │   ├── autoanalyze/     # Automatic table analysis
│   │   ├── bgtask/          # Background task framework
│   │   ├── gc/              # Garbage collection v1
│   │   ├── gcv2/            # Garbage collection v2
│   │   ├── ttl_task/        # TTL task handlers
│   │   └── schema/          # Schema synchronization
│   ├── gateway/         # Gateway service implementations
│   ├── gatewaycontroller/ # Gateway controller implementations
│   ├── httpgateway/     # HTTP API gateway implementations
│   ├── auditlog/        # Audit log processing and storage logic
│   ├── datacleaner/     # Data cleanup algorithms and policies
│   ├── metaserver/      # Metadata server implementations
│   ├── opsservice/      # Operations service implementations
│   ├── managerserver/   # Manager server implementations
│   ├── dmscaler/        # DM (Data Migration) scaler logic
│   ├── ali/             # Alibaba Cloud specific implementations
│   ├── infra/           # Infrastructure management
│   ├── storage/         # Storage abstraction layer
│   ├── client/          # API clients for external services
│   ├── metrics/         # Prometheus metrics collection (20+ metric files)
│   ├── mysql/           # MySQL protocol implementation
│   ├── nodedrainer/     # Node draining utilities
│   ├── nodescorer/      # Node scoring for resource allocation
│   ├── notifier/        # Notification system
│   ├── o11y/            # Observability utilities
│   ├── tidbmanager/     # TiDB cluster management
│   ├── tidbpdb/         # TiDB/PD integration
│   ├── tidbpool/        # TiDB resource pooling
│   ├── tidbreclaimer/   # TiDB resource reclamation
│   ├── tidbscaler/      # TiDB scaling logic
│   ├── tidbupgrader/    # TiDB upgrade management
│   ├── tidbautoscaling/ # TiDB autoscaling logic
│   ├── vpa/             # VPA (Vertical Pod Autoscaler) integration
│   ├── version/         # Version information
│   ├── versionupdater/  # Version update utilities
│   ├── types/           # Shared type definitions
│   ├── scheme/          # Kubernetes scheme definitions
│   └── utils/           # Shared utilities and helper functions
├── cd/                  # Continuous deployment manifests and configurations
├── projects/            # Pulumi infrastructure as code definitions
├── tests/               # End-to-end test suites and integration tests
├── hack/                # Development scripts and build utilities
└── docs/                # Project documentation and guides
```

## Key Patterns

### Controller Architecture

- **Reconciler pattern** for state management
- **Event-driven** updates via controller-runtime
- **Leader election** for high availability
- **Finalizers** for cleanup guarantees

### Multi-Tenancy Model

- **Shared TiDB instances** with keyspace isolation
- **Dynamic resource allocation** based on demand
- **Regional deployment** across AWS zones
- **Automatic scaling** via TiDBAutoscaling CRD
- **Zone redundancy** for high availability

### Configuration Management

- **Kustomize** for environment-specific configs
- **Pulumi** for AWS infrastructure
- **ConfigMaps/Secrets** for runtime configuration
- **Regional overrides** per AWS region
- **Helm** for deployment packaging

## Common Tasks

### Adding a New Controller

1. Define CRD in `apis/`
2. Generate code: `make generate`
3. Implement controller in `pkg/controllers/`
4. Add to `cmd/manager/app/cmd.go` (see manager setup process)
5. Update manifests in `cd/manifests/`
6. Add RBAC rules if needed in `cd/manifests/rbac/`
7. Update e2e tests in `tests/e2e/`

### Adding a New Component

1. Create directory structure in `cmd/<component-name>/`
2. Implement main.go and Dockerfile
3. Add to Makefile targets
4. Create Helm chart in `cd/charts/<component-name>/`
5. Update deployment manifests
6. Add to CI/CD pipeline

### Manager Development

The **manager** is the central orchestrator. When working on it:

**Key Files**:
- `cmd/manager/main.go`: Main entry point and controller setup
- `cmd/manager/app/cmd.go`: CLI configuration and manager initialization
- `cmd/manager/app/config/`: Configuration structures
- `cmd/manager/app/options/`: CLI flags and options

**Controller Registration**:
Controllers are registered in `cmd/manager/app/cmd.go:Run()` - follow the pattern used for existing controllers:
```go
if err = (&yourcontroller.Reconciler{
    Client: mgr.GetClient(),
    // ... other dependencies
}).SetupWithManager(mgr); err != nil {
    return fmt.Errorf("unable to create controller: %w", err)
}
```

**Available Controllers** (18+):
- Branch Controller: Database branch operations (fork/restore)
- Cluster Controller: Cluster lifecycle management
- Export Controller: Export job coordination
- Import Controller: Import job coordination
- Keyspace Controller: Keyspace management
- KeyspacePoolConfig Controller: Keyspace pool configuration
- NodeRebalance Controller: Node rebalancing
- NodeRecover Controller: Node recovery
- SecretSyncer Controller: Secret synchronization
- TiDBNodeScaler Controller: TiDB node scaling
- TiDBPod Controller: TiDB pod management
- TiDBPodEvicter Controller: TiDB pod eviction
- TiDBPodRescheduler Controller: TiDB pod rescheduling
- TiDBServerTemplate Controller: TiDB server templates
- ZoneRedundantEvicter Controller: Zone-redundant eviction

**Dependencies**:
- Managers use TLS configs for secure connections to PD, TiKV, tidb-worker
- AWS/AliCloud credentials for infrastructure access
- Etcd for distributed state management
- Redis for caching and messaging
- Kubernetes API server for CRD management
- Prometheus for metrics collection

### Testing Changes

1. Build: `make build`
2. Format: `make fmt`
3. Tidy: `make tidy`
4. Unit tests: `make unit-test`
5. Lint: `make lint`
6. E2E tests: `make e2e` (requires proper kubeconfig setup)
7. Review: `make reviewable`

## Upgrade Operations

### TiDB Component Upgrades

The system includes an automated **TiDB Upgrader** that manages rolling upgrades of TiDB pods:

**Key Features:**
- **Rolling upgrades**: Creates new pods before deleting old ones for zero-downtime upgrades
- **Batch processing**: Processes one pod per cycle per token to maintain stability
- **Stability monitoring**: Tracks cache sizes to ensure cluster stability during upgrades
- **Abnormal pod cleanup**: Automatically handles old abnormal pods
- **Metrics integration**: Exports Prometheus metrics for monitoring upgrade progress

**Upgrade Triggers:**
Changes to these ConfigMaps trigger automatic upgrades:
- `TidbServerPodTemplateCm` - Pod template configuration changes
- `serverlessPoolConfig` - Resource configuration changes
- `serverless-tidb-config` - TiDB configuration changes

**Monitoring Upgrades:**
```bash
# Check upgrade metrics
kubectl get --raw /metrics | grep tidbupgrader

# View upgrader logs
kubectl logs -n tidb-system -l app.kubernetes.io/name=manager | grep upgrader

# Check pod revision status
kubectl get pods -n tidb-system -l app.kubernetes.io/component=tidb -o jsonpath='{.items[*].metadata.labels.serverless\.pingcap\.com/revision}'
```

### EKS Cluster Upgrades

**Process for upgrading Kubernetes clusters:**
1. Modify Pulumi configuration files for EKS version and AMI ID
2. Submit PR and merge to trigger upgrade
3. Upgrade takes ~50 minutes for 39 nodes
4. Control plane upgrades first (~40 minutes), then node groups

**Commands:**
```bash
# Check current EKS version
aws eks describe-cluster --name dev-us-east-1-f01 --query cluster.version

# Update kubeconfig after upgrade
aws eks update-kubeconfig --name dev-us-east-1-f01 --region us-east-1
```

### Release and Deployment Upgrades

**Environment Promotion:**
- **Dev**: PRs to main branch trigger automatic dev environment upgrades
- **Staging**: Create `release-*` branches for staging environment upgrades
- **Production**: Create tags from release branches for production upgrades

**Manual Deployment Commands:**
```bash
# Build and push specific component
make VAR_IMG_HUB=ecr VAR_TARGETS=manager push

# Deploy to specific environment
kubectl apply -k cd/overlays/dev-us-east-1-f01

# Rollback deployment
kubectl rollout undo deployment/manager -n tidb-system
```

**Upgrade Verification:**
```bash
# Check deployment status
kubectl rollout status deployment/manager -n tidb-system

# Verify pod versions
kubectl get pods -n tidb-system -o jsonpath='{.items[*].spec.containers[*].image}'

# Check component health
kubectl get pods -n tidb-system -l app.kubernetes.io/name=manager
```

## Key Features

### Branch Operations
- **Database Forking**: Create branches from existing databases
- **Database Restore**: Restore databases from backups
- **Branch Lifecycle**: Manage branch creation, deletion, and merging

### Import/Export
- **Data Import**: Import data from external sources
- **Data Export**: Export data for backup or migration
- **Job Coordination**: Orchestrate import/export jobs across clusters

### High Availability
- **Zone Redundancy**: Distribute pods across availability zones
- **Pod Eviction**: Intelligent pod eviction during maintenance
- **Node Recovery**: Automatic node recovery and rebalancing

### Observability
- **Metrics Collection**: Comprehensive Prometheus metrics (20+ metric files)
- **Notification System**: Event-driven notifications for critical events
- **Audit Logging**: Complete audit trail for compliance

### Multi-Cloud Support
- **AWS**: Primary cloud provider with regional deployments
- **AliCloud**: Alibaba Cloud support with region-specific configurations
- **Cloud Abstraction**: Unified interface for multi-cloud operations

## Debugging Common Issues

**Build Issues**:
- Run `make tidy` to resolve dependency issues
- Check `make lint` output for code style issues
- Use `make generate` if CRD changes aren't reflected

**E2E Test Issues**:
- Ensure proper kubeconfig setup for test clusters
- Check cluster availability: `kubectl get nodes`
- Verify CRDs are installed: `kubectl get crds | grep tidb`
- Use `DEBUG=1` for verbose test output

**Runtime Issues**:
- Check pod status: `kubectl get pods -n tidb-system`
- View logs: `kubectl logs -n tidb-system -l app.kubernetes.io/name=manager`
- Check events: `kubectl get events -n tidb-system --sort-by='.lastTimestamp'`

### Debugging E2E Failures

- Check pod logs: `kubectl logs -n <namespace> <pod-name>`
- View events: `kubectl get events -n <namespace>`
- Port-forward for debugging: `kubectl port-forward -n <namespace> <pod-name> 8080:8080`
- Get pod details: `kubectl describe pod <pod-name> -n <namespace>`
- Check resource usage: `kubectl top pods -n <namespace>`

## Environment Variables

### Kubeconfig Variables

- `VAR_KUBECONFIG`: Free tier EKS access (default: ~/.kube/config)
- `VAR_INFRA_KUBECONFIG`: Infra API access (default: ~/.kube/config)
- `VAR_CONTEXT`: EKS context name (default: arn:aws:eks:us-east-1:385595570414:cluster/dev-us-east-1-f01)
- `VAR_INFRA_CONTEXT`: Infra API context name (default: infra-dev)
- `VAR_AWS_REGION`: AWS region for testing

### Build Variables

- `VAR_IMG_HUB`: Registry (ecr, gcr.io/pingcap-public)
- `VAR_PLATFORMS`: Target platforms (linux/amd64, linux/arm64)
- `TARGETS`: Components to build (manager, tidb-gateway, etc.)
- `VAR_DEBUG`: Enable debug mode (1/0)

### Testing Variables

- `FOCUS`: Focus specific e2e tests (default: ".*")
- `RANDOM_CLUSTER`: Custom cluster name for tests (default: git username)
- `SKIP_DEPLOY`: Skip deployment during e2e tests (default: 0)
- `SKIP_CLEANUP`: Skip cleanup after e2e tests (default: 0)
- `CLEANUP`: Force cleanup after e2e tests (default: 0)
- `DEBUG`: Enable debug output for e2e tests (default: 0)
- `PARALLEL`: Number of parallel test executions
- `MANAGER_NS`: Manager namespace (auto-generated)
- `TIDB_POOL_NS`: TiDB pool namespace (auto-generated)
- `VIP_TIDB_POOL_NS`: VIP TiDB pool namespace (auto-generated)

## Testing Infrastructure

### Required Setup

1. **AWS CLI** configured with appropriate permissions
2. **kubectl** with access to test EKS clusters
3. **Pulumi** for infrastructure management
4. **Docker** for container builds
5. **Go 1.24+** for development
6. **golangci-lint v1.64.8** for code linting
7. **Kubebuilder** for CRD generation

### Test Environments

- **dev-us-east-1-f01**: Primary development cluster
- **dev-base-us-west-2**: Base infrastructure cluster
- **Regional clusters**: ap-southeast-1, eu-central-1, etc.
- **Staging environments**: staging-us-east-1, staging-ap-southeast-1
- **Production**: Managed by SRE team

### Quick Health Checks

```bash
# Check manager status
kubectl get pods -n tidb-system -l app.kubernetes.io/name=manager

# Verify CRDs
kubectl get crds | grep tidb

# Check keyspace status
kubectl get keyspaces -A

# Monitor logs
kubectl logs -f deployment/manager -n tidb-system
```

### Infrastructure Access

**AWS Services**:
- EKS for Kubernetes clusters
- ECR for container registry
- S3 for backups and artifacts
- RDS for metadata storage
- CloudWatch for logging and monitoring
- IAM for service authentication

**Kubernetes Resources**:
- Custom Resource Definitions (CRDs)
- ConfigMaps for configuration
- Secrets for sensitive data
- ServiceAccounts for pod permissions
- NetworkPolicies for security isolation