# 2026-05-25 dlt and Databend Evaluation

Historic note: this evaluation informed the TiDB Cloud ingestion MVP, but it is
not an active platform design document.

## Decision

Do not use dlt as the primary TiDB Cloud Essential to TiDB Cloud Lake loader for
this MVP.

Use the reusable TiDB Cloud ingestion engine to:

```text
TiDB Cloud export -> S3 Parquet -> Databend/TiDB Cloud Lake COPY INTO
```

Keep dlt as an experimental option for smaller Python-native sources or future
custom destination work.

## What The Docs Say

- dlt's supported destination list does not include Databend as a first-class
  destination. It does include a generic SQLAlchemy destination.
- dlt's SQLAlchemy destination can target databases with SQLAlchemy dialects,
  but only MySQL and SQLite are described as fully supported and tested in dlt
  CI. Other dialects are expected to generally work but are not tested.
- dlt's SQLAlchemy destination supports `append`, `replace`, and `merge`
  dispositions, plus state sync in destination tables.
- Fast Parquet loading in the SQLAlchemy destination is documented for MySQL via
  an ADBC driver. Without a supported fast path, dlt falls back to batch insert.
- dlt's filesystem destination can write Parquet to S3 and manage dlt state
  there, but that would duplicate the TiDB Cloud export-to-S3 path rather than
  improve it for this workload.
- Databend has a SQLAlchemy dialect through `databend-sqlalchemy`, and that
  dialect supports `COPY INTO` operations.

## Probe Result

The repo-local `uv` environment can install:

```text
uv sync --extra dlt-probe
```

The Databend SQLAlchemy dialect can connect to TiDB Cloud Lake with the same DSN
after translating `lake://` to `databend://`.

A minimal dlt SQLAlchemy destination probe did not load cleanly into a fresh
Databend dataset. dlt tried to read `_dlt_pipeline_state` and `_dlt_loads`
before they existed, and the Databend dialect surfaced this as an unknown
database/table error during pipeline sync. That may be fixable with dialect
capability registration or manual state bootstrapping, but it is not a good MVP
dependency.

## Implication

dlt is not the right first loader for this particular path:

- The source already has a managed export mechanism that writes Parquet to S3.
- Cloud Lake already supports `COPY INTO` from S3 Parquet.
- The data volume is large enough that row-wise SQLAlchemy inserts are the wrong
  default.
- The MVP needs source-owned contracts and control tables more than it needs a
  general Python extraction framework.

## References

- dlt destinations:
  <https://dlthub.com/docs/dlt-ecosystem/destinations>
- dlt SQLAlchemy destination:
  <https://dlthub.com/docs/dlt-ecosystem/destinations/sqlalchemy>
- dlt filesystem destination:
  <https://dlthub.com/docs/dlt-ecosystem/destinations/filesystem>
- dlt Parquet support:
  <https://dlthub.com/docs/dlt-ecosystem/file-formats/parquet>
- Databend SQLAlchemy package:
  <https://pypi.org/project/databend-sqlalchemy/>