# pkg/ - Shared Packages

> Reusable, self-contained packages providing common functionality across the codebase.

## Package Categories

| Category | Packages | Purpose |
|----------|----------|---------|
| **Cloud Storage** | `s3/`, `gcs/`, `azblob/`, `blob/`, `storage/` | Provider-specific and abstract storage clients |
| **API Clients** | `infra-api/`, `regional-api/`, `grpc/grpcli/` | External service communication |
| **Error Handling** | `bizerr-wrapper/`, `error-wrapper/`, `error-handler/` | Consistent error handling patterns |
| **Utilities** | `id_generator/`, `redact/`, `parse/`, `string-utils/`, `log/` | Common helpers |
| **Testing** | `fakedb/` | In-memory MySQL for tests |
| **Account/Auth** | `account/` | User checking and authorization |

## Core Patterns

### Interface + Implementation
All major packages define interfaces first:

```go
// storage/interface.go
type StorageClient interface {
    ListObjects(ctx context.Context, bucket, prefix string) ([]string, error)
}

// storage/s3.go
type S3Client struct { s3iface.S3API }
var _ StorageClient = &S3Client{} // Compile-time check
```

### Constructor Pattern
Standard naming: `New<Type>` or `MustNew<Type>` (panic on error):

```go
func NewChangefeedClient(regionalCli RegionalClient) ChangefeedClient {
    return &ChangefeedClientImpl{regionalCli: regionalCli}
}
```

### Error Handling
**Always return `bizerr.BizError`** for business errors:

```go
import bizerr_wrapper "github.com/tidbcloud/dataflow-service/pkg/bizerr-wrapper"

// Helper functions
bizerr_wrapper.NewInvalidInputError("invalid param: %s", param)
bizerr_wrapper.NewInternalErr("operation failed")
bizerr_wrapper.NewNotFoundErr("resource %d not found", id)
bizerr_wrapper.WrapGormError(err, "query failed")
```

gRPC error wrapping:
```go
if err != nil {
    return nil, bizerr.WrapGrpcError(err, "MethodName")
}
```

HTTP error conversion (regional-api):
```go
// Converts HTTP status to bizerr.BizError
err := ErrorResponse2BizErr(resp.StatusCode, body)
```

### Mock Generation
Interfaces have mocks in `mock/` subdirectory:

| Package | Mock Location |
|---------|---------------|
| `regional-api/` | `regional-api/mock/` |
| `infra-api/` | `infra-api/mock/` |
| `grpc/grpcli/` | `grpc/grpcli/mock/` |

Generate with `make generate4test`.

## Key Utilities

### ID Generator
```go
type IDGenerator interface { NextID() uint64 }
idGen := MustNewSnowflakeIDGenerator(machineIDGenerator)
```

### Redact (Sensitive Data)
**ALWAYS use before logging sensitive data:**

```go
import "github.com/tidbcloud/dataflow-service/pkg/redact"

// Redact known sensitive fields
safeJSON, _ := redact.RedactKeysToJSON(redact.PasswordFields, req)
log.Info("request", zap.ByteString("body", safeJSON))

// Custom fields
redact.RedactKeys([]string{"token", "secret"}, dataMap)
```

### Parse (URL Path Parsing)
```go
import "github.com/tidbcloud/dataflow-service/pkg/parse"

orgID, ok := parse.ParseOrgIDFromPath(path)
clusterID, ok := parse.ParseClusterIDFromPath(path)
```

### Logging
```go
import "github.com/tidbcloud/dataflow-service/pkg/log"

logger := log.GetLogger(ctx)  // Get logger from context
ctx = log.WithLogger(ctx, logger.With(zap.String("key", "value")))
```

### FakeDB (Testing)
```go
import "github.com/tidbcloud/dataflow-service/pkg/fakedb"

db, err := fakedb.StartFakeDB4Test(cfg)  // In-memory MySQL
```

### HTTP Client
Functional options pattern:

```go
import pkghttp "github.com/tidbcloud/dataflow-service/pkg/http"

cli := pkghttp.NewRestyCli(
    pkghttp.RestyWithBaseURL(baseURL),
    pkghttp.RestyCliWithDefaultRetry(),
    pkghttp.RestyWithProtoJson(),
)
```

## Client Wrapper Pattern

`regional-api` and `infra-api` use facade pattern:

```go
// regional-api/changefeed.go
type ChangefeedClientImpl struct {
    regionalCli RegionalClient  // Composed base client
}

func (c *ChangefeedClientImpl) CreateChangefeed(
    ctx context.Context, region *Region, req *ticdc_v1.CreateChangefeedRequest,
) (*ticdc_v1.Changefeed, bizerr.BizError) {
    pathParams := map[string]string{ticdcClusterIDPathParam: req.TicdcClusterId}
    changefeed := new(ticdc_v1.Changefeed)
    if err := c.regionalCli.Post(ctx, region, createChangefeedPath, pathParams, nil, req.Changefeed, changefeed); err != nil {
        return nil, err
    }
    return changefeed, nil
}
```

## Best Practices

1. **Always use interfaces** - Define in `interface.go`, implement in separate files
2. **Return `bizerr.BizError`** - Use helper wrappers from `bizerr-wrapper`
3. **Functional options** - For configurable constructors
4. **Mock generation** - Place in `mock/` directory
5. **Redact secrets** - Always before logging
6. **Context propagation** - Pass `ctx` to all methods
7. **Constructor naming** - `New<Type>` or `MustNew<Type>`