# System Architecture

## System Overview

dataflow-service is a comprehensive data management service for TiDB Cloud that provides backup/restore, TiCDC (change data capture), data import, and data migration capabilities. It is designed with a classic MVC architecture pattern, with clear separation between API entry points, business logic, data access, and background processing.

## Architecture Diagram

```mermaid
graph TD
    A[HTTP API] --> B[Controllers]
    C[gRPC API] --> B
    B --> D[Models]
    D --> E[Repository]
    E --> F[Database]
    D --> G[Event Bus]
    G --> H[Workers]
    H --> I[Workflow Engine]
    
    subgraph "Controllers"
        B1[Console API]
        B2[Internal API]
        B3[Open API]
        B4[Operations API]
    end
    
    subgraph "Models"
        D1[Backup]
        D2[TiCDC]
        D3[Import]
        D4[Migration]
    end
    
    subgraph "Background Processing"
        H1[Backup Worker]
        H2[Restore Worker]
        H3[Import Worker]
        H4[Migration Worker]
    end
```

## Component Descriptions

### dataflow-service

- **Purpose**: Provide core data management services for TiDB Cloud
- **Responsibilities**:
  - Handle HTTP and gRPC API requests
  - Process business logic for data operations
  - Manage data access and storage
  - Handle background worker tasks
  - Orchestrate workflows

- **Dependencies**:
  - TiDB clusters for data storage
  - Cloud storage for backups
  - External APIs for integration

- **Type**: Application

### Controllers

- **Purpose**: API entry points for different user groups
- **Responsibilities**:
  - Receive and validate requests
  - Call appropriate model methods
  - Return responses

- **Subcomponents**:
  - Console API: For management console users
  - Internal API: For other TiDB Cloud services
  - Open API: For external users
  - Operations API: For operations personnel

### Models

- **Purpose**: Business logic layer
- **Responsibilities**:
  - Implement core data operations
  - Handle business rules and validation
  - Interact with data access layer

- **Subcomponents**:
  - Backup: Backup-related business logic
  - TiCDC: Change data capture and replication
  - Import: Data import operations
  - Migration: Data migration logic

### Repository

- **Purpose**: Data access layer
- **Responsibilities**:
  - Encapsulate database operations
  - Provide reusable data access methods
  - Interact with GORM ORM library

### Workers

- **Purpose**: Background task processing
- **Responsibilities**:
  - Handle time-consuming operations
  - Process asynchronous jobs
  - Manage workflow execution

- **Subcomponents**:
  - Backup Worker: Handles backup operations
  - Restore Worker: Handles restore operations
  - Import Worker: Handles data import
  - Migration Worker: Handles data migration

### Event Bus

- **Purpose**: Internal system communication
- **Responsibilities**:
  - Publish system events
  - Subscribe to and handle events
  - Enable loose coupling between components

## Data Flow

```mermaid
sequenceDiagram
    participant Client as API Client
    participant Ctrl as Controller
    participant Mdl as Model
    participant Repo as Repository
    participant DB as Database
    participant Wkr as Worker
    
    Client->>Ctrl: HTTP/gRPC Request
    Ctrl->>Mdl: Business Logic Call
    Mdl->>Repo: Data Access Request
    Repo->>DB: Database Query
    DB-->>Repo: Query Result
    Repo-->>Mdl: Data
    Mdl-->>Ctrl: Processed Result
    Ctrl-->>Client: Response
    
    Mdl->>Wkr: Background Task
    Wkr->>Mdl: Task Status Update
```

## Integration Points

### External APIs
- **TiDB Cloud Console API**: For console operations
- **TiDB Cloud Internal APIs**: For service-to-service communication
- **Cloud Storage APIs**: For backup and restore operations

### Databases
- **TiDB Cluster**: Main data storage
- **Internal Database**: For system metadata

### Third-party Services
- **Amazon S3/Azure Blob/GCP Cloud Storage**: For backup storage
- **Other TiDB Cloud Services**: For integration and coordination

## Infrastructure Components

- **Deployment**: Argo CD managed Kubernetes deployment
- **Networking**: Kubernetes cluster network with VPC isolation
- **Scalability**: Horizontal pod autoscaling based on load
- **Monitoring**: Prometheus metrics and Grafana dashboards
- **Logging**: Structured logging with Elasticsearch and Kibana