# dataflow-service

Some of the services available on [tidbcloud](https://tidbcloud.com), including but not limited to

* Backup&Restore
* TiCDC
* Data Import
* Data Migration

## Project-layout

see more in [app/readme.md](app/readme.md)

### Running Service

* http, serve HTTP traffic

* grpc, serve gRPC traffic

* worker/workflow, do some async jobs

## How to deploy

We currently use [argocd](https://argo-cd.readthedocs.io/en/stable//) to manage our deployment.
the deploy yaml scripts is in [app-dellivery](https://github.com/tidbcloud/app-delivery/tree/main/dataflow-svc)

## How to migrate database

In most cases, the Worker will [automatically](https://github.com/tidbcloud/dataflow-service/blob/14d70535fabdba40fddf039836803ddc5cd980da/internal/cmd/worker.go#L39) migrate the db to the latest state on startup.

However, if you find that this migration is taking so long that it is interfering with the worker's normal work, you will need to migrate manually.

```bash
./dataflow migrate --config config.yaml
```

see more in <https://pingcap.feishu.cn/wiki/wikcnZwdxvvAxGGmsrPNdycYJcd>

## How to run

```bash
❯ ./dist/dataflow -h
Usage:
  dataflow [command]

Available Commands:
  grpc        start grpc server
  help        Help about any command
  http        start http server
  migrate     be careful when run this cmd, THIS IS DB MIGRATIONS!!!
  run_job     run some custom jobs
  worker      start background db workers

Flags:
      --config string   config file (default "config.yaml")
  -h, --help            help for dataflow
  -v, --version         version for dataflow

Use "dataflow [command] --help" for more information about a command.
```