# tidb-fivetran-connector

TiDB Cloud source connector and deployer runtime for Fivetran Connector SDK.

## Connector development

```bash
cd tidb-fivetran-connector
uv python pin 3.13
uv sync
uv run ruff check .
uv run ruff format --check .
uv run pyright
uv run pytest
```

Runtime dependencies live in `[project].dependencies`; local-only tooling lives in `[dependency-groups]`.
The Fivetran runtime provides `fivetran_connector_sdk`, so `fivetran-connector-sdk` is kept out of the
published connector dependencies and installed only for local development, tests, and the deployer image CLI.

Install Git hooks after syncing the environment:

```bash
uv run pre-commit install
```

## Deployer image

The repository also contains a one-shot Kubernetes Job runtime that:

1. reads `DESTINATION_NAME`, `CONNECTION_NAME`, `FIVETRAN_API_KEY_BASE64`, and `CONNECTOR_CONFIG_JSON`
2. runs `fivetran deploy --force`
3. resolves `connection_id` through the Fivetran REST API with `groups(name) -> connections(group_id, schema)`
4. writes the result JSON to stdout and `/dev/termination-log`

### Build

From `tidb-fivetran-connector/`:

```bash
docker build -f Dockerfile.deployer -t tidb-fivetran-deployer:local .
```

Or use the helper script:

```bash
AWS_PROFILE=full-manager-service-role \
AWS_REGION=us-east-1 \
AWS_ACCOUNT_ID=385595570414 \
ECR_REPOSITORY=tidb-fivetran-deployer \
IMAGE_TAG=0.1.0 \
tidb-fivetran-connector/scripts/build_deployer_image.sh
```

### Run locally

```bash
docker run --rm \
  -e FIVETRAN_API_KEY_BASE64=... \
  -e DESTINATION_NAME=tidb_snowflake \
  -e CONNECTION_NAME=tidb_job_example \
  -e CONNECTOR_CONFIG_JSON='{"sync_task_id": "task-001", "s3_url":"s3://bucket/prefix","snapshot_prefix":"snapshot","incremental_prefix":"increment"}' \
  tidb-fivetran-deployer:local
```

### Kubernetes

See [k8s/deployer-job.example.yaml](./k8s/deployer-job.example.yaml).

## Table-level resync

Table-level resync is controlled through the Fivetran connection state. External callers should not write
connector snapshot files or marker files. Control Plane prepares a request-scoped Dumpling snapshot prefix,
then patches one top-level `resync` marker into the full Fivetran state object.

State key format is `<source_schema>.<source_table>`, for example `test.worker`:

```json
{
  "version": 1,
  "sync_task_id": "task-001",
  "snapshot": {
    "done": true,
    "snapshot_tso": "100"
  },
  "incremental": {
    "checkpoint_ts": "200",
    "ddl_watermark": {}
  },
  "resync": {
    "request_id": "table-resync-001",
    "phase": "snapshot",
    "tables": ["test.worker"],
    "snapshot_tso": "150",
    "snapshot_prefix": "snapshot/table-resync-001",
    "truncate_done": false,
    "current_file": null,
    "next_row": 0
  }
}
```

Recommended external flow:

1. Pause the Fivetran connection with `PATCH /v1/connections/{connectionId}` and `{"paused": true}`.
2. Read current state with `GET /v1/connections/{connectionId}/state`.
3. Back up the current state.
4. Merge the top-level `resync` marker into the full state object.
5. Write the full merged state with `PATCH /v1/connections/{connectionId}/state`.
6. Unpause with `PATCH /v1/connections/{connectionId}` and `{"paused": false}`.
7. Let the next connector `update()` consume the patched state.

If `incremental.scan` is present when the state is patched, the connector finishes that frozen scan before
truncating or replaying table resync snapshot data.