# Pkg Module

## Overview

The Pkg module contains reusable packages and utilities for the dataflow-service project. These packages provide common functionality that can be used across different parts of the application. The module includes various utilities for working with cloud services, storage, authentication, and more.

## Directory Structure

```
pkg/
├── account/           # Account management utilities
├── agent/             # Agent-related utilities
├── aws/               # AWS service utilities
├── azblob/            # Azure Blob storage utilities
├── billing/           # Billing utilities
├── bizerr-wrapper/    # Business error wrapper
├── blob/              # Blob storage utilities
├── clusteragentcli/   # Cluster agent client
├── encryptor-wrapper/ # Encryptor wrapper
├── error-handler/     # Error handling utilities
├── error-wrapper/     # Error wrapper
├── event/             # Event utilities
├── fakedb/            # Fake database implementation
├── gcs/               # Google Cloud Storage utilities
├── grpc/              # gRPC utilities
├── http/              # HTTP utilities
├── id_generator/      # ID generator utilities
├── import-config/     # Import configuration utilities
├── infra-api/         # Infrastructure API client
├── log/               # Log utilities
├── parse/             # Parsing utilities
├── redact/            # Data redaction utilities
├── regional-api/      # Regional API client
├── s3/                # S3 storage utilities
├── storage/           # Storage utilities
├── string-utils/      # String utilities
└── version/           # Version utilities
```

## Core Components

### 1. account/

Account management utilities, including account database access and user checking.

### 2. agent/

Agent-related utilities, including file parsing, gRPC services, HTTP services, and Lightning integration.

### 3. aws/

AWS service utilities, including EC2, EKS, Route53, and S3.

### 4. azblob/

Azure Blob storage utilities for working with Azure storage services.

### 5. billing/

Billing utilities for handling billing-related functionality.

### 6. bizerr-wrapper/

Business error wrapper for handling business-specific errors.

### 7. blob/

Blob storage utilities for working with various blob storage services.

### 8. clusteragentcli/

Cluster agent client for communicating with cluster agents.

### 9. encryptor-wrapper/

Encryptor wrapper for handling encryption and decryption.

### 10. error-handler/

Error handling utilities for handling various types of errors.

### 11. error-wrapper/

Error wrapper for wrapping and handling errors consistently.

### 12. event/

Event utilities for working with events.

### 13. fakedb/

Fake database implementation for testing purposes.

### 14. gcs/

Google Cloud Storage utilities for working with GCS.

### 15. grpc/

gRPC utilities, including client implementations and middleware.

### 16. http/

HTTP utilities for working with HTTP requests and responses.

### 17. id_generator/

ID generator utilities for generating unique IDs.

### 18. import-config/

Import configuration utilities for handling import configurations.

### 19. infra-api/

Infrastructure API client for communicating with infrastructure services.

### 20. log/

Log utilities for logging messages.

### 21. parse/

Parsing utilities for parsing various types of data.

### 22. redact/

Data redaction utilities for redacting sensitive information.

### 23. regional-api/

Regional API client for communicating with regional services.

### 24. s3/

S3 storage utilities for working with Amazon S3.

### 25. storage/

Storage utilities for working with various storage services.

### 26. string-utils/

String utilities for working with strings.

### 27. version/

Version utilities for handling version information.

## Key Features

- **Cloud Storage Integration**: Utilities for working with AWS S3, Google Cloud Storage, and Azure Blob Storage
- **Agent Communication**: Tools for communicating with agents deployed in various environments
- **Error Handling**: Consistent error handling across the application
- **gRPC and HTTP Utilities**: Helpers for working with gRPC and HTTP services
- **Infrastructure API**: Client for interacting with infrastructure services
- **Regional API**: Client for interacting with regional services
- **Data Redaction**: Tools for redacting sensitive information
- **ID Generation**: Utilities for generating unique IDs

## Usage Guidelines

### Importing Packages

To use a package from the Pkg module, import it using the full path:

```go
import (
	"github.com/tidbcloud/dataflow-service/pkg/s3"
	"github.com/tidbcloud/dataflow-service/pkg/grpc/grpcli"
)
```

### Adding New Packages

To add a new package to the Pkg module:

1. Create a new directory in the `pkg/` directory with a descriptive name
2. Implement the package functionality
3. Add appropriate tests for the package
4. Update this README file to include the new package

### Best Practices

- **Keep packages focused**: Each package should have a single, well-defined purpose
- **Write comprehensive tests**: All packages should have thorough test coverage
- **Document packages**: Add comments and documentation to explain package functionality
- **Use interfaces**: Define interfaces for better testability and flexibility
- **Avoid circular dependencies**: Ensure packages don't have circular dependencies

## Notes

- The Pkg module is intended for reusable code that can be used across different parts of the application
- Packages in this module should be self-contained and not depend on application-specific code
- Changes to packages in this module may affect multiple parts of the application
- Ensure backward compatibility when modifying existing packages