# DSBulk Support

This document describes the current support status for [DataStax Bulk Loader (DSBulk)](https://github.com/datastax/dsbulk) in this project.

The current implementation is aimed at `dsbulk load`. Support statements below are based on the current working tree and local validation with DSBulk `1.11.1`.

## Support Summary

| Capability | Status | Notes |
| --- | --- | --- |
| `dsbulk load` with CSV | Supported | Validated with simple scalar tables, composite keys, UUID/timestamp, decimal/date/boolean, and partial-column custom queries. |
| `dsbulk load` with JSON | Supported | Validated with scalar columns and collection types such as `list`, `set`, and `map`. |
| `-k/-t` schema-driven load | Supported | DSBulk can infer the target `INSERT` from proxy-exposed schema metadata. |
| Custom `-query` / `--schema.query` for `INSERT` | Supported | Named bind markers such as `:id` and `:name` are exposed in prepared metadata. |
| Partial-column load with explicit mapping | Supported | Use a custom query plus `-m` / `--schema.mapping`. |
| Wide-table / larger imports | Supported with tuning | Prefer `--engine.maxConcurrentQueries` to avoid hitting the proxy's backend connection limit. |
| `dsbulk unload` | Not supported | DSBulk relies on token-range splitting via `TOKEN(...)`, which is not implemented with Cassandra-compatible semantics here. |
| `dsbulk count` | Not supported | Same `TOKEN(...)` limitation as `unload`. |
| `preserveTimestamp` | Not supported | The proxy rejects `USING TIMESTAMP`. |
| `preserveTtl` | Not supported | The proxy rejects `USING TTL`, and TTL-related read paths are not implemented. |
| Parquet connector | Not available | DSBulk itself provides CSV and JSON connectors; it does not ship a Parquet connector. |

## Recommended Usage

For current production-style usage:

- Use `dsbulk load`.
- Prefer CSV or JSON input.
- For larger imports, set `--engine.maxConcurrentQueries` explicitly.
- If you use a custom query, use named bind markers and explicit field mapping.
- If you are diagnosing load failures, start with `--batch.mode DISABLED`.

## Known Gaps

The current implementation can run tested `load` workflows, but it is not yet a full Cassandra-compatible DSBulk target.

- Prepared metadata currently returns `pk_count = 0`, so partition-key routing metadata is incomplete.
- `system_schema.keyspaces` currently advertises `SimpleStrategy`; Java driver logs a token-map refresh warning. This did not block tested `load` flows.
- Default DSBulk concurrency can exceed the proxy's tenant backend connection budget on wider or larger imports. When that happens, DSBulk reports `tenant backend connection limit exceeded`.
- `TOKEN(...)` is translated as a plain column expression, which blocks `unload` and `count`.

## Validated Scenarios

The following scenarios were validated locally against the current project state:

- CSV load with at least 5 different schemas.
- JSON load with at least 5 different schemas.
- JSON load for collections: `list<text>`, `set<text>`, and `map<text,int>`.
- Custom `INSERT` query with named bind markers.
- Partial-column load via custom query and explicit mapping.
- A `20`-column CSV schema with `10,000` rows. Default DSBulk concurrency hit the proxy's backend connection limit; the same import succeeded with `--engine.maxConcurrentQueries 4`.

## Example: CSV Load

Schema:

```sql
CREATE KEYSPACE IF NOT EXISTS app
WITH replication = {'class':'SimpleStrategy','replication_factor':1};

CREATE TABLE app.simple_people (
  id int PRIMARY KEY,
  name text,
  score int
);
```

CSV file:

```csv
id,name,score
1,Alice,10
2,Bob,20
3,Carol,30
```

Command:

```bash
dsbulk load \
  -h 127.0.0.1 \
  -port 9142 \
  -u appuser \
  -p apppass \
  -k app \
  -t simple_people \
  -url /path/to/simple_people.csv \
  --engine.maxConcurrentQueries 4
```

## Example: JSON Load

Schema:

```sql
CREATE TABLE app.profile_collections (
  id int PRIMARY KEY,
  tags list<text>,
  scores map<text, int>
);
```

JSON lines file:

```json
{"id":1,"tags":["red","blue"],"scores":{"math":90,"eng":88}}
{"id":2,"tags":["green"],"scores":{"math":77,"eng":91}}
```

Command:

```bash
dsbulk load \
  -h 127.0.0.1 \
  -port 9142 \
  -u appuser \
  -p apppass \
  --connector.name json \
  -k app \
  -t profile_collections \
  -url /path/to/profile_collections.json \
  --engine.maxConcurrentQueries 4
```

## Example: Custom Query With Partial Columns

Schema:

```sql
CREATE TABLE app.custom_partial (
  id int PRIMARY KEY,
  name text,
  score int,
  note text
);
```

CSV file:

```csv
id,name,score
41,PartialA,410
42,PartialB,420
43,PartialC,430
```

Command:

```bash
dsbulk load \
  -h 127.0.0.1 \
  -port 9142 \
  -u appuser \
  -p apppass \
  -url /path/to/custom_partial.csv \
  -query "INSERT INTO app.custom_partial (id, name, score) VALUES (:id, :name, :score)" \
  -m "id = id, name = name, score = score" \
  --batch.mode DISABLED \
  --engine.maxConcurrentQueries 4
```

## Example: Wide Table / 10k Rows

For wider tables or larger files, explicit concurrency control is strongly recommended.

Schema excerpt:

```sql
CREATE TABLE app.wide20_csv_10k (
  id int PRIMARY KEY,
  tenant text,
  name text,
  email text,
  city text,
  region text,
  segment text,
  status text,
  created_at timestamp,
  updated_at timestamp,
  signup_date date,
  age int,
  visits bigint,
  score int,
  level int,
  active boolean,
  balance decimal,
  ratio float,
  rating double,
  note text
);
```

Command:

```bash
dsbulk load \
  -h 127.0.0.1 \
  -port 9142 \
  -u appuser \
  -p apppass \
  -k app \
  -t wide20_csv_10k \
  -url /path/to/wide20_csv_10k.csv \
  --engine.maxConcurrentQueries 4 \
  --log.directory /tmp/dsbulk-logs
```

Observed behavior during validation:

- Default DSBulk concurrency could fail with `tenant backend connection limit exceeded`.
- The same `20`-column, `10,000`-row CSV load succeeded with `--engine.maxConcurrentQueries 4`.

## Unsupported Features To Avoid

Do not rely on the following with the current implementation:

- `dsbulk unload`
- `dsbulk count`
- `--schema.preserveTimestamp true`
- `--schema.preserveTtl true`
- TTL or writetime preservation workflows
- Token-range reads based on `TOKEN(...)`

## Troubleshooting

If a load fails:

- Reduce concurrency first: `--engine.maxConcurrentQueries 4`
- Disable batching while diagnosing: `--batch.mode DISABLED`
- Enable a dedicated DSBulk log directory: `--log.directory /path/to/logs`
- Check for driver warnings related to token-map refresh; these are currently expected and non-fatal for tested `load` workflows