# data-trans

`data-trans` is the active private Go service for finance and revenue operations workflows. It exposes authenticated HTTP endpoints that trigger data sync jobs, file-based finance operations, and CRM-related repair flows used by internal operators.

The canonical source repository for this service is
[`amber-lzz/data-trans`](https://github.com/amber-lzz/data-trans). The previous
`qq851185223/data-trans` SSH remote redirects to this repository. The
`pingcap-inc/netforce-hub` repository is a stale snapshot and must not be used
as the source for current builds or deployments. The Go module and binary name
remain `data-platform`.

## What the service does

- Runs an HTTP API on port `8868` by default.
- Orchestrates finance sync jobs backed by MySQL, BigQuery, Salesforce, Fivetran, and related cloud services.
- Provides ad-hoc operational endpoints for selective Salesforce sync, last-invoice-month sync, AR table generation, cloud breakdown generation, downloads, uploads, event-center jobs, and fix-up flows.
- Ships with Docker and Cloud Run deployment assets.

## API surface

The current server exposes these routes:

- `GET /healthz`
- `GET /internal/health`
- `POST /v1/sf_single`
- `POST /v1/sf_invoice_month`
- `POST /v1/ar_table`
- `POST /v1/cloud_breakdown`
- `POST /v1/download`
- `POST /v1/upload`
- `POST /v1/event_center`
- `POST /v1/fix`

`/v1/*` and `/internal/health` are protected by `Authorization: Bearer <secret.token>`. The unauthenticated readiness endpoint is `GET /healthz`.

### Last-invoice-month Order batches

`POST /v1/sf_invoice_month` remains asynchronous. Order sync accepts three
mutually exclusive scope modes:

```json
{"type": 1}
```

Runs the existing full last-month Order batch.

```json
{"type": 1, "limit": 3}
```

Selects at most three active, unsynced, CRM-eligible last-month Order rows in
ascending TiDB ID order. Numeric tenant-placeholder Accounts are excluded from
the limited sample and therefore do not consume the limit.

```json
{"type": 1, "order_ids": [123, 456]}
```

Runs an exact sample. Every requested ID must be an active, unsynced
last-month Order row and must be eligible for CRM creation. Missing, already
synced, out-of-month, duplicate, zero, or placeholder-Account IDs reject the
entire sample before any CRM write. `limit` and `order_ids` cannot be combined,
and neither option is accepted for RevRec or Collection sync.

For a controlled create-first workflow, add `"defer_activation": true` to an
Order request:

```json
{"type": 1, "limit": 50, "defer_activation": true}
```

This creates or reuses the NexusCRM Order, ensures exactly one standard
OrderItem, persists the CRM Order ID and Order Number, and deliberately leaves
the finance row unsynced for a separately controlled activation step. The flag
is rejected for RevRec and Collection requests. Account mapping for this path
uses the live `salesforce_account.sf_id` projection; `bq_account` is not the
authority for NexusCRM Account IDs.

The HTTP response confirms only asynchronous acceptance. Require the terminal
success/failure receipt plus TiDB and CRM readback before expanding the batch.

Cloud-breakdown reruns preserve a generated Order's original `type=0` identity
after its fake reference has been resolved to a real CRM Order number. The
resolver requires an exact persisted Order reference; it does not fall back to
account-only matching, which could incorrectly collapse a normal commitment
plan into an on-demand Order. If an already-published generated RevRec retains
an older CRM Order number, a second-stage match is allowed only when bill month,
tenant, account, type, recognition type, and raw revenue amount identify exactly
one synced finance fact; ambiguous candidates remain separate.

Collection upserts treat `updated_at` as bookkeeping, not business payload. A
rerun with identical collection fields leaves the row untouched; `updated_at`
advances only when a persisted collection value actually changes.

## Local development

Prerequisites:

- Go `1.22.x`
- A reachable MySQL instance for most business flows
- Optional Redis, depending on the environment you are reproducing
- Valid external credentials for the endpoints you intend to exercise

Build the binary:

```bash
go build ./cmd/data-platform
```

Run the server with an explicit config file:

```bash
go run ./cmd/data-platform -c /path/to/config.yaml
```

At minimum, your config should define:

- `secret.token` for bearer authentication
- `insecure.bind-address` and `insecure.bind-port`
- `mysql.*` when the route needs database access

Most sync endpoints also depend on environment-specific credentials and service settings that are intentionally not stored in this repository.

## Container and deployment

Build a local image:

```bash
docker build -t netforce-hub:local .
```

The image expects a runtime config file mounted at `/app/config/config.yaml`. In Cloud Run, `deploy.sh` mounts that path from Secret Manager and deploys the service with the existing internal image naming convention.

## Repository layout

- `cmd/` contains the executable entrypoint.
- `internal/dataplatform/` contains the application wiring, controllers, services, and stores.
- `internal/pkg/` contains shared HTTP, option, middleware, and server helpers.
- `model/` contains storage and payload models.
- `pkg/` contains shared application infrastructure.
- `config/migrations/` contains checked-in SQL migrations that are still part of the codebase.
- `script/` contains build and packaging helpers.

## Notes for maintainers

- This repository no longer carries the old unserved static web assets or placeholder controllers that had no runtime entrypoint.
- Local secrets, local config files, generated binaries, editor metadata, and PDFs are intentionally ignored and should stay out of Git.
- If you later want to rename the Go module from `data-platform` to `netforce-hub`, treat that as a separate refactor from this repository migration.