# Project Structure

## Why use this structure?

Currently (and for the foreseeable future) our business scenario is not complex, and the classic **MVC** model is sufficient
to handle our business needs

In general, there are **only seven package** inside app package: container, config, controllers, models, internal, and worker

* container register all external extensions, such as mysql, kafka, and some third-party services, which are all
  global single instances, so you need to make sure all extension is thread-safe

* config is responsible for storing all the configuration information of the project

* controllers is responsible for receiving requests, processing requests, returning responses, and some data validation

* models is responsible for storing, reading, updating, and deleting data and business logic

* internal is responsible for the internal helper func/pkg for business logic, such an new event bus, metrics, and so on

* repo encapsulates gorm, which needs to have certain reusability. Domain uses the method of repo encapsulation to
  access the database, which can make the domain layer focus more on the orchestration and organization of business logic

* worker is responsible for some time-consuming operations, such as backup, restore, and some asynchronous operations

* workflow is responsible for managing the workflow of workers

## Points to note

* *Only* important modules of the main process mentioned above are putted into the level 1(app) package, you MUST not add any other package in this level

* Other business-related tools and functions and non-main logic can be added into internal/..pkg package, such as infra-api client, cm client, account client, metrics

* don't write any business logic inside the controller, all interfaces in the controller need to be tested (**UT**)

* all the business logic should in models, this will be the core and the heaviest layer, so the functions in this layer
  need to take into account the reusability and testability (written in the mode of golang interface)

* domain in models defines the input and output parameters of the model life interface. Since different business entries will
  call the same model, the definitions of input and output parameters will be different. If you directly use the entity
  definition of the definition or database generated by grpc, you cannot flexibly increase and control the impact area.

## About worker

We use `gocron` to manage the workflow of workers, The system operates with three key components - the Scheduler, Job, and Executor.

* The Scheduler acts as a coordinator, keeping track of all the jobs assigned to it and making sure that each job is passed on to the Executor when it's ready to run. the Scheduler can control various aspects of job behavior, such as limiting the number of concurrent jobs.

* The Job, on the other hand, is focused on the specific task it has been assigned. It has limited awareness and can only perform actions related to that task, such as preventing itself from running over another job that is taking too long.

* The Executor's role is to simply execute the task handed to it by the Job. The Scheduler sends the Job to the Executor when it's ready to run, and the Executor carries out the task without any further management required.

Please help me to improve the above paragraph to help others understand it better
![design-diagram](https://user-images.githubusercontent.com/19351306/110375142-2ba88680-8017-11eb-80c3-554cc746b165.png)

see more in gocron [readme](https://github.com/go-co-op/gocron)

* An example of how to organize the worker part of the code

```bash
app/worker
├── biz_changefeed # worker package
│   └── jobs.go # all jobs about biz_changefeed
├── ticdc # worker package
│   ├── jobs.go # all jobs about ticdc
└── server.go # worker server, will be started by cmd `dataflow-worker`
```

* code in jobs.go

The job described in the above paragraph will only be performed when the trigger logic has been included. The implementation of the business logic should still be implemented in the model layer.

## About workflow

we use `fastflow` to manager the workflow, https://github.com/ShiningRush/fastflow