# Dataflow Service Agent Instructions

> **Single Source of Truth** for AI coding agents. See nested `AGENTS.md` files in subdirectories for layer-specific patterns.

## Overview

Go-based microservice for TiDB Cloud data integration:
- **Backup & Restore (BR)**: Snapshot/PiTR backup and restore
- **Data Migration (DM)**: MySQL-to-TiDB migration
- **Data Import**: Bulk import from S3/GCS/Azure (CSV, Parquet)
- **CDC**: Real-time replication via TiCDC

**Binaries**: `cmd/dataflow` (main service), `cmd/agent` (agent)

**For Next-Gen architecture**: `/dataflow-agent-ng`, `/dataflow-service-ng` (separate Go modules)

**LEGACY**: `/internal` contains DM/Import code - in-use but being migrated to `app/`

## Architecture

```
app/
├── container/      # Dependency injection (dig) - container.Global()
├── controllers/    # gRPC/HTTP entry points (thin validation layer)
├── models/         # Core business logic (Interface + Impl pattern)
├── repo/           # Data persistence (GORM + functional options)
├── workflow/       # Async DAG workflows (fastflow engine)
├── worker/         # Scheduled jobs (gocron)
└── internal/       # Shared utilities (feature gates, facades)
```

See `app/AGENTS.md` for detailed layer patterns.

## Generated Code - DO NOT EDIT

| Pattern | Generator |
|---------|-----------|
| `idl/pbgen/**`, `api/*.swagger.json` | `make generate-idl` |
| `*/entity/autogenerated_entity.go` | `make generate-entity` |
| `*/mock/*.go` | `make generate4test` |
| `test/e2e/internal/tidbcloud/*/model/` | `make -C test/e2e generate-api-client` |

## Commands

Select build target by local environment:
- macOS (darwin): use `make build-dataflow-darwin` (`make build-dataflow-agent-darwin` for agent)
- Linux or CI default: use `make build-dataflow` (`make build-dataflow-agent` for agent)
- Linux amd64 cross-build: use `make build-dataflow-amd64` (`make build-dataflow-agent-amd64` for agent)

```bash
# Build
make build                  # Build all binaries
make build-dataflow         # Main service only
make build-dataflow-darwin  # Main service on macOS (CGO)

# Test
make app-unit-test              # All tests with race detector
make app-unit-test-local    # Quick app tests (no coverage)

# Generate
make generate               # All (IDL + entity)
make generate-idl  # Generate IDL
make generate-entity  # Generate entity
make generate4test          # Mocks only

# Lint
make check                  # Full CI check (lint + format)
make lint                   # Linters only
make fmt                    # Auto-fix formatting
```

## Core Patterns (Quick Reference)

### Interface/Impl
```go
// interface.go
type BackupModel interface { CreateBackup(ctx context.Context, req *domain.Req) bizerr.BizError }

// impl/backup.go
var _ backup.BackupModel = &backupModelImpl{} // Compile-time check
func NewBackupModel(...) backup.BackupModel { return &backupModelImpl{...} }
```

### NG Display Events

The `dataflow-service-ng` activity feed publishes operation boundaries and
reliable terminal outcomes only: `started`, `succeeded`, `failed`, `canceled`,
and `skipped`. Internal states such as `running`, `paused`, `resumed`, and
retry transitions are not display events. Producers should pass an explicit
operation and result when available; do not infer display events from an
internal state alone when no reliable terminal boundary exists. Asynchronous
commands may publish only `started` when the API confirms acceptance but no
durable completion signal is available.
The shared NG EventCenter is transport-only and must not enforce a global
allowlist that could reject future event types; intermediate-state filtering
belongs to each event producer.

### Error Handling
```go
// Business errors - use bizerr package
return bizerr.New(errcode.BackupNotFound, "backup %d not found", id)
return bizerr_wrapper.NewInternalErr("operation failed")

// Error chaining
return err.Chain("context: load backup %d", id)
```

### Repo Filtering
```go
// Functional options pattern
backups, err := repo.List(ctx,
    WithClusterID(clusterID),
    WithStatus(entity.BackupStatusRunning),
    WithPager(page, perPage),
)
```

### Transactions
```go
err := transProvider.Transaction(ctx, func(ctx context.Context) error {
    // All operations share same transaction
    if err := repo1.Create(ctx, entity1); err != nil { return err }
    if err := repo2.Update(ctx, entity2); err != nil { return err }
    return nil
})
```

### Logging
```go
log.GetLogger(ctx).Info("backup created",
    zap.Uint64("backupID", backupID),
    zap.Uint64("clusterID", clusterID))
// ALWAYS redact secrets (passwords, tokens, keys)
```

## Testing Patterns

### Unit Tests
- **Structure**: Table-driven with `map[string]struct{}`, testify suite
- **Mocks**: Generated via mockery, `mock.NewXxx(s.T())`
- **Fake DB**: `repo.GetFakeDB()` for integration tests

```go
tests := map[string]struct {
    input    string
    expected string
    wantErr  bool
}{
    "valid case": {input: "a", expected: "b"},
    "error case": {input: "", wantErr: true},
}
for name, tc := range tests {
    s.Run(name, func() { ... })
}
```

### E2E Tests
See `.github/instructions/e2e-test.instructions.md`

## Migrations

Path: `migrate/record_YYYYMMDDHHMM_desc.go`

```go
var _ = AddDataflowDBMigrateRecord(
    "202602031000",
    func(db *gorm.DB) error {
        return db.AutoMigrate(&entity.NewTable{})
    },
)
```
- Use `gormigrate` with `UseTransaction: false`
- Auto-runs on worker startup

## Best Practices

1. **Context**: Always pass `ctx` for tracing/cancellation
2. **Controllers**: Keep thin - delegate to models
3. **Idempotency**: Essential for workers/workflows
4. **Domain types**: Use `domain/` types in model interfaces, not gRPC/DB entities
5. **Versioning**: Use `SaveWithVersioning()` for optimistic locking
6. **Pitfalls**: Before touching, read `.domain-knowledge/pitfalls.md` for pitfalls (e.g., `ImportDataSource` AZBLOB is `4` in the public API but `3` in entity constants).
7. **Postmortems**: Review team postmortem documentation (runbooks, incident reports, or wiki) for known pitfalls
8. **Verify before done**: After any code change, run the appropriate verification command before reporting completion. Never assume a change is correct without evidence:
   - Go code changes → `make app-unit-test-local`
   - Generated code → `make generate4test` / `make generate-entity` / `make generate-idl`
   - Scripts/tools → execute the script directly and confirm no errors
   - Build-only changes → `make build`

## Nested Agent Instructions

| Path | Purpose |
|------|---------|
| `app/AGENTS.md` | App layer overview and patterns |
| `app/models/AGENTS.md` | Business logic patterns |
| `app/repo/AGENTS.md` | Data access patterns |
| `app/controllers/AGENTS.md` | Controller patterns |
| `app/workflow/AGENTS.md` | Workflow/action patterns |
| `app/worker/AGENTS.md` | Scheduled jobs and workers |
| `pkg/AGENTS.md` | Shared packages (storage, API clients, utilities) |
| `internal/AGENTS.md` | Legacy DM/Import patterns |
| `migrate/AGENTS.md` | Migration patterns |
| `test/e2e/AGENTS.md` | E2E test framework and cases |

## Automation Rules

When users provide new domain logic or architectural decisions:
1. Update relevant `/.domain-knowledge/*.md` files
2. If introducing new patterns, update relevant `AGENTS.md`