# TiDB Cloud Data Service Infrastructure

A comprehensive enterprise-grade multi-cloud database service platform for managing TiDB data operations including backup, restore, data replication, and migration across AWS, GCP, and Azure.

## Overview

This repository contains the data services infrastructure that power **TiDB Cloud** - a fully managed cloud-native database service. The platform focuses on data operations including TiCDC replication, backup & restore, and data import/export with comprehensive multi-cloud support.

### Key Capabilities

- **Data Replication**: TiCDC-based change data capture and cross-cluster synchronization
- **Backup & Restore**: Full backup, incremental backup, and point-in-time recovery (PITR)
- **Data Import/Export**: Large-scale data migration from/to object storage (S3/GCS/ABS)
- **Multi-Cloud Support**: Unified operations across AWS, GCP, and Azure
- **Infrastructure as Code**: GitOps-based deployment using ArgoCD and Kubernetes operators
- **Enterprise Security**: TLS encryption, IAM integration, and private connectivity support
- **Observability**: Comprehensive monitoring with Prometheus and Grafana

## Architecture

```mermaid
graph TB
    subgraph "Control Plane"
        A[TiDB Cloud API] --> B[Dataflow Service]
        A --> C[TCIDM]
    end

    subgraph "Dataflow Service - Controllers"
        B --> B1[Backup Controller<br/>(BR)]
        B --> B2[Import Controller]
        B --> B3[CDC Controller]
        B --> B4[DM Controller]
    end

    subgraph "Dataflow Service - Workers"
        B --> C1[Agent Service]
        B --> C2[Backup Worker]
        B --> C3[Import Worker]
        B --> C4[Worker Service]
    end

    subgraph "External Integrations"
        B1 --> S3[S3]
        B1 --> GCS[GCS]
        B1 --> ABS[Azure Blob]
        B2 --> S3
        B2 --> GCS
        B2 --> ABS
    end

    subgraph "Data Plane"
        C --> D1[TiDB Clusters<br/>AWS]
        C --> D2[TiDB Clusters<br/>GCP]
        C --> D3[TiDB Clusters<br/>Azure]
    end
```

## Core Components

### 1. Dataflow Service

The primary service providing data migration, replication, backup, and import capabilities.

**Key Features:**
- TiCDC data replication between clusters
- Backup and restore operations using BR tool
- Large-scale data import from object storage
- Asynchronous task processing and scheduling
- Multi-cloud object storage abstraction (S3/GCS/ABS)

**Dataflow-Related Controllers (in infra-provider):**
- `controllers/backup`: Backup task management and BR integration
- `controllers/br`: Backup & Restore tool operations
- `controllers/cdc`: TiCDC controller for data replication
- `controllers/dm`: Data migration controller
- `controllers/compactbackup`: Compact backup operations

**Dataflow Service Components:**
- `Agent`: HTTP/gRPC API service for job management
- `Backup Worker`: Backup task execution engine
- `Import Worker`: Data import operations
- `Worker Service`: Asynchronous task scheduling

### 2. TCIDM (TiCDC Data Management)

Specialized service for managing TiCDC clusters and data replication tasks.

**Key Features:**
- TiCDC cluster deployment and configuration
- Data replication task management
- Private link connection management

**Components:**
- `regional_tcidm_service`: Regional-level TiCDC management

## Project Structure

```
data-service-infra-repo-set/
├── .gitmodules                    # Git submodules configuration
├── .trae/                         # AI-DLC skills
├── .gitignore
├── README.md
│
├── dataflow-service/              # Dataflow service (primary)
│   ├── app/                       # Application core
│   │   ├── cmd/                   # Command entry points
│   │   ├── models/                # Data models
│   │   ├── repo/                  # Data repository layer
│   │   │   ├── billing.go
│   │   │   ├── bucket.go
│   │   │   └── ticdc_spec.go
│   │   ├── utils/                 # Utility functions
│   │   └── worker/                # Worker service
│   ├── cmd/                       # Command entry points
│   ├── idl/                       # Interface definitions (protobuf)
│   ├── internal/                  # Internal implementation
│   │   ├── cmd/                   # Sub-commands
│   │   ├── config/                # Configuration
│   │   ├── db/                    # Database operations
│   │   ├── jobs/                  # Background jobs
│   │   └── metrics/               # Metrics
│   ├── pkg/                       # Shared packages
│   │   ├── agent/                 # Agent client
│   │   ├── aws/                   # AWS SDK wrapper
│   │   ├── gcs/                   # GCP Storage client
│   │   ├── azblob/                # Azure Blob client
│   │   ├── blob/                  # Storage abstraction
│   │   ├── billing/               # Billing service
│   │   ├── grpc/                  # gRPC utilities
│   │   ├── infra-api/             # Infrastructure API client
│   │   └── storage/               # Storage abstraction
│   └── test/                      # Tests
│
├── infra-provider/                # Infrastructure provider
│   ├── api/                       # CRD API definitions
│   ├── cluster-controller/        # Main controller
│   │   ├── controllers/
│   │   │   ├── backup/           # Backup controller (used)
│   │   │   ├── br/               # BR tool integration (used)
│   │   │   ├── cdc/              # TiCDC controller (used)
│   │   │   ├── dm/               # DM controller (used)
│   │   │   ├── compactbackup/    # Compact backup (used)
│   │   │   ├── cluster/          # Cluster management
│   │   │   └── autoops/          # Auto operations
│   │   └── pkg/                   # Shared libraries
│   └── [other components]
│
└── tcidm/                         # TiCDC Data Management
    ├── api/                       # API definitions
    ├── cmd/                       # Command entry points
    ├── commons/                   # Common utilities
    └── internal/                  # Internal implementation
```

### Relevant Components for Dataflow

**Dataflow Service** is the primary service, while **infra-provider** contains dataflow-related controllers:

| Component | Location | Purpose |
|-----------|----------|---------|
| **Agent** | dataflow-service | Job API service |
| **Backup Worker** | dataflow-service | Backup execution |
| **Import Worker** | dataflow-service | Import execution |
| **Backup Controller** | infra-provider/cluster-controller | Backup CRD management |
| **BR Controller** | infra-provider/cluster-controller | BR tool integration |
| **CDC Controller** | infra-provider/cluster-controller | TiCDC operations |
| **DM Controller** | infra-provider/cluster-controller | Data migration |

## Technology Stack

| Category | Technology |
|----------|------------|
| **Language** | Go 1.21+ |
| **Orchestration** | Kubernetes (EKS/GKE/AKS) |
| **Database** | TiDB 6.5 LTS, PostgreSQL |
| **Object Storage** | AWS S3, GCP GCS, Azure Blob Storage |
| **Data Tools** | TiCDC, BR (Backup & Restore), Dumpling, Lightning |
| **Monitoring** | Prometheus, Grafana |
| **Deployment** | ArgoCD, Helm, Kustomize |
| **API** | gRPC, Protocol Buffers |
| **Testing** | Ginkgo, Gomega, envtest |

## Getting Started

### Prerequisites

- Go 1.21 or later
- Kubernetes cluster (1.27+)
- kubectl configured
- Access to AWS, GCP, or Azure
- Object storage bucket (S3/GCS/ABS)

### Development Setup

1. **Clone the repository with submodules:**
```bash
git clone --recurse-submodules https://github.com/your-org/data-service-infra-repo-set.git
cd data-service-infra-repo-set
```

2. **Install dependencies:**
```bash
# Dataflow Service
cd dataflow-service
go mod download

# Infra Provider (for dataflow controllers)
cd ../infra-provider
go mod download
```

3. **Build the project:**
```bash
# Build all components
make build

# Build dataflow-related components
cd dataflow-service && go build ./...
cd ../infra-provider/cluster-controller && go build ./...
```

4. **Run tests:**
```bash
# Dataflow Service tests
cd dataflow-service
go test ./...

# Infra Provider controller tests
cd ../infra-provider/cluster-controller
go test ./controllers/backup/...
go test ./controllers/cdc/...
go test ./controllers/dm/...
```

### Deployment

The project uses GitOps deployment with ArgoCD. See the deployment documentation for detailed instructions.

```bash
# Deploy to Kubernetes
make deploy

# Generate manifests
make generate
```

## Data Operations

### Backup & Restore

```mermaid
flowchart TB
    A[Backup Request] --> B{Backup Type}
    B -->|Snapshot| C[Create Cloud Snapshot]
    B -->|Log| D[Stream binlog to S3]
    B -->|Full| E[BR Tool Export]
    
    C --> F[Store Metadata]
    D --> F
    E --> F
    
    F --> G[Record in DB]
    
    H[Restore Request] --> I{Find Backup}
    I --> J[Download Data]
    J --> K[BR Tool Import]
    K --> L[Verify Data]
    L --> M[Complete]
```

### Data Import

Supported formats:
- CSV files
- SQL dumps
- Mydumper output

Supported sources:
- AWS S3
- Google Cloud Storage (GCS)
- Azure Blob Storage

### TiCDC Replication

- Real-time change data capture
- Cross-cluster data synchronization
- DDL statement replication
- Filter-based replication

## API Reference

### Kubernetes CRDs (Dataflow-Related)

| CRD | Version | Group | Purpose |
|-----|---------|-------|---------|
| `Backup` | v1alpha1 | infra.pingcap.com | Backup task definition |
| `TiCDC` | v1alpha1 | infra.pingcap.com | TiCDC cluster management |
| `DMJob` | v1alpha1 | infra.pingcap.com | Data migration tasks |
| `CompactBackup` | v1alpha1 | infra.pingcap.com | Compact backup tasks |

### gRPC Services

```protobuf
service DataflowService {
    rpc CreateBackup(CreateBackupRequest) returns (CreateBackupResponse);
    rpc GetBackup(GetBackupRequest) returns (GetBackupResponse);
    rpc RestoreBackup(RestoreBackupRequest) returns (RestoreBackupResponse);
    rpc ImportData(ImportDataRequest) returns (ImportDataResponse);
}
```

## Monitoring & Observability

### Metrics

Prometheus metrics are exposed by all components:
- `backup_tasks_total` - Total backup tasks
- `backup_tasks_completed_total` - Completed backup tasks
- `backup_task_duration_seconds` - Backup task duration
- `ticdc_replication_lag_seconds` - Replication lag
- `import_progress_bytes` - Import progress

### Logging

Structured logging with support for:
- Elasticsearch
- Loki
- CloudWatch Logs

### Tracing

Distributed tracing with OpenTelemetry and Jaeger support.

## Security

### Authentication & Authorization
- RBAC-based access control
- JWT token validation
- Cloud IAM integration

### Network Security
- VPC isolation
- Private Link support
- TLS/mTLS encryption

### Data Security
- Encryption at rest
- Secure credential management
- Audit logging

## Contributing

We welcome contributions! Please see our contributing guidelines:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

### Code Quality

We enforce code quality through:
- golangci-lint for linting
- go fmt for formatting
- Unit tests with 70%+ coverage
- Integration tests with envtest

```bash
# Run linter
make lint

# Format code
make fmt

# Run tests
make test
```

## Documentation

Detailed documentation is available in the `aidlc-docs/` directory:

- [Architecture](aidlc-docs/inception/reverse-engineering/architecture.md) - System architecture
- [Business Overview](aidlc-docs/inception/reverse-engineering/business-overview.md) - Business capabilities
- [Code Structure](aidlc-docs/inception/reverse-engineering/code-structure.md) - Code organization
- [Component Inventory](aidlc-docs/inception/reverse-engineering/component-inventory.md) - Component details
- [Technology Stack](aidlc-docs/inception/reverse-engineering/technology-stack.md) - Technology choices

## License

This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.

## Support

- **Documentation**: [docs.tidb.cloud](https://docs.tidb.cloud)
- **Issues**: GitHub Issues
- **Slack**: TiDB Community Slack
- **Forum**: [TiDB Forum](https://ask.pingcap.com)

## Acknowledgments

- [TiDB](https://github.com/pingcap/tidb) - Distributed SQL database
- [TiKV](https://github.com/tikv/tikv) - Distributed KV engine
- [TiCDC](https://github.com/pingcap/ticdc) - Change data capture
- [BR](https://github.com/pingcap/br) - Backup & Restore tool
- [Kubernetes](https://kubernetes.io/) - Container orchestration
- [ArgoCD](https://argoproj.github.io/argo-cd/) - GitOps delivery

## Status

| Component | Status |
|-----------|--------|
| Dataflow Service | Production |
| Backup/BR Controller | Production |
| TiCDC Controller | Production |
| DM Controller | Production |
| TCIDM | Production |

---

**Built with ❤️ by the TiDB Cloud Team**