# IDL Module

## Overview

The IDL (Interface Definition Language) module contains the protobuf definitions for the APIs and services provided by dataflow-service. This module defines the structure of requests and responses for all API endpoints, as well as the gRPC service definitions.

## Directory Structure

```
idl/
├── idl/              # Protobuf definition files
│   ├── agent/        # Agent service protobuf definitions
│   ├── dataflow/     # Main service protobuf definitions
│   ├── dedicated/    # Dedicated service protobuf definitions
│   ├── buf.gen.yaml  # Buf generation configuration
│   └── buf.work.yaml # Buf workspace configuration
└── pbgen/            # Generated code
    └── proto/        # Generated Go code from protobuf definitions
```

## Core Components

### 1. idl/agent/

Protobuf definitions for the Agent service, which is deployed in various environments to perform local operations such as backup and restore.

### 2. idl/dataflow/

Protobuf definitions for the main dataflow service, including:

- **adminapi/**: Admin API definitions
- **common/**: Common message definitions
- **dataflow/**: Main service API definitions
- **dm/**: Data migration API definitions
- **openapi/**: Open API definitions
- **premium_dm/**: Premium data migration API definitions
- **replay/**: Replay API definitions

### 3. idl/dedicated/

Protobuf definitions for dedicated services, such as recovery group.

### 4. pbgen/

Generated code from the protobuf definitions, including:

- Go language bindings
- gRPC service definitions
- HTTP/gRPC gateway code
- Swagger JSON files for API documentation

## How to Use

### Generating Code

To generate code from the protobuf definitions, use the `buf` tool:

```bash
# Install buf if not already installed
# go install github.com/bufbuild/buf/cmd/buf@latest

# Generate code
buf generate
```

This will generate the Go code, gRPC service definitions, and Swagger JSON files in the `pbgen/` directory.

### Adding New API Definitions

1. Create a new protobuf file in the appropriate directory under `idl/`
2. Define your API messages and services
3. Run `buf generate` to generate the code
4. Implement the service in the appropriate controller

### Example: Adding a New API Endpoint

1. Add a new method to an existing service in a protobuf file:

```protobuf
// in idl/idl/dataflow/dataflow/dataflow.proto
service DataflowService {
  // Existing methods...
  
  // New method
  rpc NewEndpoint(NewEndpointRequest) returns (NewEndpointResponse) {
    option (google.api.http) = {
      post: "/v1beta1/dataflow/new-endpoint"
      body: "*"
    };
  }
}

// Request message
message NewEndpointRequest {
  string name = 1;
  // Other fields...
}

// Response message
message NewEndpointResponse {
  string id = 1;
  // Other fields...
}
```

2. Generate the code:

```bash
buf generate
```

3. Implement the method in the controller:

```go
// in app/controllers/console/dataflow.go
func (c *Controller) NewEndpoint(ctx context.Context, req *dataflowpb.NewEndpointRequest) (*dataflowpb.NewEndpointResponse, error) {
  // Implementation...
}
```

## Dependencies

The IDL module depends on the following tools and libraries:

- **buf**: For managing and generating code from protobuf definitions
- **protoc**: Protocol Buffers compiler
- **protoc-gen-go**: Go code generator for Protocol Buffers
- **protoc-gen-go-grpc**: gRPC code generator for Go
- **protoc-gen-grpc-gateway**: HTTP/gRPC gateway generator
- **protoc-gen-openapiv2**: OpenAPI (Swagger) generator
- **protoc-gen-validate**: Protobuf message validation generator

## Best Practices

- Follow the [Google API Design Guide](https://cloud.google.com/apis/design/) when defining APIs
- Use consistent naming conventions for messages and services
- Add clear comments to protobuf definitions
- Version APIs appropriately (e.g., v1beta1, v1)
- Use the `google.api.http` option to define RESTful HTTP mappings

## Notes

- The protobuf definitions are the single source of truth for API interfaces
- Generated code should not be modified directly
- When making changes to protobuf definitions, always regenerate the code
- Ensure backward compatibility when modifying existing API definitions