# TiCI Import Module Design

## Overview
tici_import converts TiDB "import into" files into Tantivy fragments. It is a
read-and-decode pipeline: it parses the import file header, decodes each KV
pair into a `RawDoc`, then builds and uploads fragment files via the shard
writer/uploader pipeline. The module is stateless; all persistence happens in
the input import file and the output fragment storage managed by `tici_shard`.

Current index-type support in import is:
- `Fulltext`
- `Hybrid` (with sharding columns)

## Inputs and Outputs
Inputs:
- Import file stored on S3 or local filesystem.
- TiDB table metadata embedded in the import file header.

Outputs:
- Fragment files written by `FragWriter` and uploaded by `FragUploader`.
- Metadata updates stored via `MetadataStore` (owned by `tici_shard`).

## Import File Format (Version 1)
Header:
- `version: u8`
- `table_info_len: u64`
- `table_info_json: [u8; table_info_len]`
- `commit_ts: u64`

Body (repeated):
- `key_len: u64`
- `val_len: u64`
- `key: [u8; key_len]`
- `value: [u8; val_len]`

`table_info_json` is parsed as `tici_grpc::TiDBTableInfo`.

## High-Level Flow
```
import file (S3/local)
  -> File::open / File::open_local
  -> File::next (read KV, size checks)
  -> Decoder::decode
  -> RawDoc
  -> TantivyDoc
  -> FragWriter
  -> FragUploader
```

## Core Components

### types.rs
- Defines `JobMeta`, `Task`, and `TaskParams` used by scheduling/worker layers.
- `TaskParams::split_with_job_info` expands one input range into per-index tasks.

### file.rs
- `File::new` reads the header, parses `TiDBTableInfo`, builds `IndexMeta`, and
  constructs a `Decoder` with necessary info.
- `File::next` reads a single KV pair, enforces `max_kv_bytes`, and returns a
  decoded `RawDoc`.
- `collation_name_to_id` maps collation names from table metadata to TiDB IDs.

### decoder.rs
- `DecoderOptions::from_table_info` derives `key_columns_len` by index type:
  - Fulltext index: key columns count = PK columns count.
  - Hybrid index: key columns count = sharding columns count.
- Hybrid index without sharding columns is rejected in import.
- Index types other than Fulltext/Hybrid are rejected in import.
- `Decoder` caches decoding metadata and reuses `LazyBatchColumnVec`.

### index_scan_executor.rs
- Forked subset of TiKV index scan executor code.
- Adds `key_columns_len` to support Fulltext/Hybrid key layouts.
- Adds `force_restore_index_columns` so Fulltext/Hybrid index columns are
  decoded from restore data.
- Handles both old and new index value encodings, including restore data
  (v4/v5) and common handle decoding.

### writer.rs
- `Writer::write` opens the import file, builds a `FragWriter`, converts
  `RawDoc` to `TantivyDoc`, and sends the fragment to `FragUploader`.

## Decode Flow Details
```
KV bytes
  -> File::next (read lengths, fill key/value buffers)
  -> Decoder::decode
       -> keyspace prefix handling
       -> IndexScanExecutorImpl::process_kv_pair
       -> decode index columns to strings
       -> merge handle columns (if needed)
       -> build sharding key by index type
  -> RawDoc (handle = derived sharding key, version = commit_ts)
```

### Keyspace Handling
If `keyspace_id == NULLSPACEID`, the key is used as-is.
Otherwise, the decoder removes the keyspace prefix and returns an error if the
prefix is missing or the key is too short.

### Handle Decoding Strategies
`DecodeHandleStrategy` is derived from table metadata:
- `NoDecode`: no handle columns are appended.
- `DecodeIntHandle`: int handle decoded from key (non-unique) or value (unique).
- `DecodeCommonHandle`: common handle decoded from key (non-unique) or value
  (unique).

For Fulltext/Hybrid indexes, `IndexScanExecutorImpl` skips `key_columns_len`
encoded columns before reading handle bytes.

### Sharding Key Construction
`RawDoc.handle` (the internal sharding key) is built by index type:
- Fulltext: `[keyspace]t[tableID]_r[handle]`
- Hybrid: `[keyspace]t[tableID]_i[indexID][encoded sharding columns][handle]`

### Index Value Versions and Restore Data
`IndexScanExecutorImpl` sets `index_version` on the first KV pair and assumes it
is constant within the file. It supports:
- Old encoding (short value, legacy collation).
- New encoding (value includes tail length, options, restore data).
- Restore data v4/v5 to reconstruct original column values.

For Fulltext/Hybrid, index columns are decoded directly from restore data
(`force_restore_index_columns = true`), matching TiDB's current encoding path.

### Task expansion and DXF file lifecycle
- A single DXF import file may describe multiple target `index_id`s for the same
  table.
- TiCI expands that input into multiple import tasks, one task per `index_id`.
- This keeps each task focused on building one fragment for one index, which
  makes task size and worker memory consumption easier to reason about and
  control.
- The trade-off is that multiple derived tasks may read the same DXF input file
  independently.
- DXF input file GC must therefore be coordinated at the job level: do not
  delete an input file until all tasks derived from that file have finished and
  no worker still needs to read it.

## RawDoc Construction
`RawDoc` fields:
- `handle`: sharding key derived from index type (see "Sharding Key
  Construction"), not always equal to the raw index KV key bytes.
- `version`: `commit_ts` from the file header.
- `op_type`: always `Insert`.
- `cols`: decoded `(column_id, Option<String>)` values.
- `is_local_timezone`: `false` (import data uses UTC timestamps).

## Error Handling and Limits
- Unknown file version -> error.
- Missing index metadata in table info -> error.
- Unsupported index type for import -> error.
- Hybrid index without sharding columns -> error.
- Key/value length overflow or `max_kv_bytes` exceeded -> error.
- Keyspace prefix mismatch -> error.
- Missing column values during decode -> error.

## Tests
- `components/tici_import/src/file.rs`: end-to-end file parsing and decoding
  with real table metadata JSON.
- `components/tici_import/src/decoder.rs`: handle decoding, keyspace prefix.
- `components/tici_import/src/index_scan_executor.rs`: encoding variants and
  handle decoding rules.

## Related Design Docs
- `docs/design/tici_import/pk_handle_synthetic_index_info.md` documents how
  synthetic PK index metadata is derived for `pk_is_handle` tables.
- `docs/design/tici_import/hybrid_index_kv_format.md` documents the HYBRID index
  KV encoding rules used by TiCI import.