## Summary

We want to support multiple columns in one FTS index so joint search across
multiple text columns can be served by one logical index instead of multiple
independent indexes.

Today, the FTS implementation hardcodes the Tantivy field name to `"body"`.
That works for the current single-column design, but it prevents one logical
FTS index from safely storing more than one indexed column. Tantivy supports
multiple fields, but our write path, query path, and shard metadata currently
assume one fixed field.

This proposal introduces immutable per-index physical field layout metadata at
the shard level. Existing tracked indexes keep their current physical layout
(`"body"`). New tracked indexes use deterministic field names derived from
column IDs (`str(col_id)`), which allows multiple columns to coexist in one
Tantivy index without conflicts.

## Problem Statement

### Current limitation

- `TantivyIndexWriter` builds a schema with a constant field name `"body"`.
- `clara_fts` query construction and BM25 statistics assume one fixed field.
- FTS read and write paths assume the first query column or the first schema
  column is the only indexed text column.
- `FtsLevels` only tracks `(table_id, index_id)` and does not persist the
  physical Tantivy field layout of an existing tracked index.

### Why this is a problem

- A logical FTS index cannot safely contain multiple columns because the
  current field naming scheme would collide.
- If we simply switch from `"body"` to `str(col_id)`, existing tracked indexes
  become unreadable unless all old data is rebuilt at once.
- If we re-derive field layout from the latest schema during reads and writes,
  we risk mixing old and new physical layouts under the same tracked index.

## Goals

- Support multiple columns in one logical FTS index.
- Preserve readability of existing tracked indexes without requiring immediate
  rebuild.
- Make physical field layout explicit and stable for the lifetime of a tracked
  index.
- Keep doc ID alignment unchanged: one source row maps to one Tantivy document.
- Support snapshot, apply, split, and merge for the new metadata.
- Provide a staged rollout plan with low-risk intermediate milestones.

## Non-Goals

- In-place conversion of existing tracked indexes from `"body"` to
  `str(col_id)`.
- Supporting mixed physical layouts inside one tracked `(table_id, index_id)`.
- Solving unrelated FTS refactors in the same work.
- Changing user-visible SQL semantics beyond enabling joint search on multiple
  columns.

## Background and Current State

### Current logical model

- One FTS logical partition is identified by `(table_id, index_id)`.
- `FullTextIndexDef` currently stores one `col_id`.
- `FtsLevels` tracks only whether an index is tracked; it does not store how
  that index is physically encoded in Tantivy.

### Current physical model

- Tantivy schema is built with one text field named `"body"`.
- Queries construct terms against a global `FIELD_BODY`.
- Dedicated-file helpers and merge helpers validate that the Tantivy schema has
  only one field and that its field ID matches the hardcoded `"body"` field.

### Consequence

The current model is safe only as long as each logical FTS index maps to one
indexed column and one physical Tantivy field.

## Proposed Design

### Design overview

Introduce immutable per-index tracked metadata that records:

1. which logical columns belong to the tracked index, and
2. which physical Tantivy field layout is used for that tracked index.

Existing tracked indexes are loaded as `LegacyBody`. New tracked indexes use
`ColIdString`.

### Logical schema vs physical layout

We intentionally separate two concerns:

- Logical schema definition:
  - comes from `SchemaFile.fulltext_indexes`
  - tells us what columns the index should logically contain
- Physical field layout:
  - comes from shard-level tracked FTS metadata
  - tells us how the current on-disk Tantivy files are encoded

The logical schema may evolve over time. The physical layout of an already
tracked index must not change in place.

### New tracked metadata

Suggested Rust model:

```rust
enum TrackedFtsFieldLayout {
    LegacyBody,
    ColIdString,
}

struct TrackedFtsIndexMeta {
    table_id: i64,
    index_id: i64,
    col_ids: Vec<i64>,
    field_layout: TrackedFtsFieldLayout,
}
```

Suggested protobuf model:

```proto
message FullTextIndexDef {
  int64 index_id = 1;
  int64 col_id = 2;            // legacy fallback
  repeated int64 col_ids = 3;  // canonical logical columns
  string parser_type = 10;
}

enum FtsFieldLayout {
  FTS_FIELD_LAYOUT_UNSPECIFIED = 0;
  FTS_FIELD_LAYOUT_LEGACY_BODY = 1;
  FTS_FIELD_LAYOUT_COL_ID_STRING = 2;
}

message FtsTrackedIndexMeta {
  int64 table_id = 1;
  int64 index_id = 2;
  repeated int64 col_ids = 3;
  FtsFieldLayout field_layout = 4;
}
```

### Field naming rules

- `LegacyBody`
  - valid only when `col_ids.len() == 1`
  - `col_id -> "body"`
- `ColIdString`
  - valid for one or more columns
  - `col_id -> col_id.to_string()`

This keeps old tracked indexes unchanged while giving new tracked indexes a
deterministic multi-field naming scheme.

### Invariants

- `TrackedFtsIndexMeta.col_ids` must be non-empty.
- `TrackedFtsIndexMeta.col_ids` must be unique and preserve DDL order.
- `LegacyBody` must have exactly one column.
- One tracked `(table_id, index_id)` must have exactly one physical layout.
- Existing tracked indexes never change physical layout in place.
- A logical schema change that changes column set or order requires rebuild.

## Read Path Design

### Query validation

`Query::validate_info` validates the request-level invariants before any shard
reader is created. `Query::new_with_index_layout`, the only `Query`
constructor, also calls it so construction cannot bypass these checks:

- the query type is supported,
- the query has an `index_id`,
- a legacy `query_text` request has exactly one string-like column with a
  column ID.

Match-expression requests identify source fields through
`match_expr.fields[].col_id` and may leave the legacy `FtsQueryInfo.columns`
field empty. Those field references are validated against the tracked physical
index metadata when the complete query is constructed.

After selecting a shard, `SnapAccessCore::new_fts_reader` requires its tracked
index layout and calls `Query::new_with_index_layout` to bind the request to the
persisted physical fields. A missing or incompatible tracked layout is an error
for both indexed and brute-force execution. There is no unresolved `Query`
state: construction produces the executable Tantivy query, physical schema,
tokenizers, and source-column bindings together.

### Indexed query construction

- `LegacyBody`
  - build Tantivy terms against the one field `"body"`
- `ColIdString`
  - for each token, build an OR across all mapped fields of this tracked index
  - one logical token matches if any indexed field contains it

### Indexed BM25 scoring

BM25 today is effectively single-field. For multi-column indexes, the indexed
path must aggregate statistics across all mapped fields in the same logical
index.

This is a meaningful behavior change and should be implemented in a dedicated
phase. Before that phase, multi-column support should be limited to `NoScore`
or otherwise explicitly gated.

### Fallback readers

Brute-force readers must stop assuming only `columns[0]` matters.

For multi-column fallback:

- build inner schemas that include all indexed text columns,
- evaluate match across all indexed columns,
- keep score handling separate from no-score handling,
- preserve one output row per source row.

## Write Path Design

### Schema ingestion

`FullTextIndexDef` should evolve from single-column to multi-column by adding
`col_ids`.

Rules:

- `col_ids` is the canonical logical column list
- `col_id` remains populated as `col_ids[0]` for backward compatibility

### Choosing tracked layout

Tracked layout is selected only when the index becomes tracked.

- Old snapshots or old tracked indexes with no layout metadata:
  - load as `LegacyBody`
- New tracked index creation:
  - use `ColIdString`

This choice is persisted and reused on all future incremental updates.

### Building Tantivy documents

For both historical build and incremental build:

- one source row must create exactly one Tantivy document,
- each indexed column writes into its mapped field name,
- NULL or empty values should not shift doc IDs,
- if all indexed columns are NULL or empty, write an empty/null Tantivy
  document to preserve row-to-doc alignment.

### L0 and delta builds

The write-side code should build one Tantivy writer per tracked logical index,
not one writer per column. All columns of that tracked index should be written
into the same document.

## Metadata Lifecycle

### Snapshot

Snapshot serialization must persist the full tracked metadata, not only
`(table_id, index_id)`.

Backward compatibility:

- new code should read old snapshots that only contain `fts_indexes`
- these should be materialized as `LegacyBody`

### Apply

`FtsUpdate` should support adding tracked index metadata, not only table/index
pairs.

Backward compatibility:

- if a change set only contains legacy `add_tracked_indexes`, materialize them
  as `LegacyBody`

### Split

Split must copy tracked metadata exactly for indexes that remain in scope for
the child shard. Physical layout must not change during split.

### Merge

Merge must preserve tracked metadata exactly. If source and target somehow
contain incompatible tracked metadata for the same `(table_id, index_id)`, that
should be treated as a hard error or forced rebuild condition, not silently
merged.

## DDL and Rebuild Rules

### Safe cases

- existing single-column tracked index remains single-column:
  - continue to use its current layout
- new index added:
  - tracked as `ColIdString`

### Rebuild-required cases

- indexed column set changes
- indexed column order changes
- tracked layout changes
- logical schema says the index is multi-column but the tracked meta is
  `LegacyBody`

### Drop behavior

If logical schema no longer contains a tracked FTS index, existing cleanup
behavior should drop tracked metadata and stale files.

## Compatibility and Rollout

### Backward compatibility

Supported:

- new code reading old snapshots
- new code reading old tracked indexes encoded with `"body"`
- new code building new tracked indexes with `str(col_id)`

Not supported without gating:

- old binaries reading new tracked metadata
- old binaries reading new multi-field Tantivy schemas

### Rollout rule

Do not emit `ColIdString` tracked metadata or multi-column tracked indexes until
all nodes that can read the shard are upgraded to a version that understands the
new metadata and layout.

## Alternatives Considered

### Alternative 1: Global switch from `"body"` to `str(col_id)`

Rejected because it breaks existing tracked indexes unless all old data is
rebuilt immediately.

### Alternative 2: Re-derive field naming from latest schema on every read/write

Rejected because it can mix old and new physical layouts under one tracked
index.

### Alternative 3: Store raw `col_id -> field_name` strings in shard metadata

Possible, but less desirable than storing a compact layout enum plus ordered
`col_ids` because:

- `ColIdString` is deterministic
- raw strings increase duplication
- raw strings make it easier for metadata and actual naming policy to drift

## Risks and Mitigations

### Risk: mixed physical layouts under one tracked index

Mitigation:

- persist tracked layout explicitly
- never infer layout from the latest schema once an index is tracked
- validate layout consistency during apply, split, merge, and rebuild

### Risk: multi-column scoring drift between indexed and brute-force paths

Mitigation:

- stage multi-column `NoScore` before multi-column scored search
- add dedicated scored tests before enabling scored multi-column by default

### Risk: file-format helper code assumes one Tantivy field

Mitigation:

- audit dedicated-file builder and merge helpers
- relax validation first
- fall back to conservative whole-file handling for multi-field indexes if
  needed before optimizing sub-offset computation

## Formal Implementation Plan

### Phase 1: Metadata and compatibility scaffolding

#### Objective

Introduce canonical logical column lists and tracked physical layout metadata
without changing current single-column behavior.

#### Scope

- protobuf changes
- `SchemaFile` ingestion changes
- `FtsLevels` metadata changes
- snapshot/apply/meta/util/split/merge plumbing

#### Concrete changes

1. Extend `FullTextIndexDef` with `col_ids`.
2. Add `FtsFieldLayout` and `FtsTrackedIndexMeta`.
3. Add snapshot and `FtsUpdate` fields for tracked metadata.
4. Replace tracked index set in `FtsLevels` with tracked metadata map.
5. Keep compatibility helpers:
   - `has_tracked_index`
   - `iter_tracked_indexes`
   - `track_legacy_index`
   - `track_index_meta`
6. Load old metadata as `LegacyBody`.
7. Serialize and deserialize tracked metadata through:
   - snapshot creation
   - snapshot apply
   - `FtsUpdate`
   - split
   - merge
   - `ShardMeta`

#### Files

- `components/kvenginepb/src/fts.proto`
- `components/kvenginepb/src/changeset.proto`
- generated protobuf outputs
- `components/cloud_worker/src/schema_manager.rs`
- `components/kvengine/src/table/fts/level.rs`
- `components/kvengine/src/read.rs`
- `components/kvengine/src/apply.rs`
- `components/kvengine/src/meta.rs`
- `components/kvengine/src/util.rs`
- `components/kvengine/src/split.rs`
- `components/kvengine/src/shard.rs`
- `components/kvengine/src/compaction.rs`

#### Acceptance criteria

- old snapshots load successfully
- old tracked indexes are materialized as `LegacyBody`
- snapshot/apply/split/merge preserve tracked metadata
- current single-column tests remain green

### Phase 2: Layout-aware single-field refactor

#### Objective

Remove the hard dependency on the literal `"body"` while preserving current
single-column behavior.

#### Scope

- `clara_fts` schema/query/read/write validation
- single-field layout awareness
- no multi-column semantics yet

#### Concrete changes

1. Replace global `FIELD_BODY` assumptions with per-index layout resolution.
2. Make index writer build schemas from tracked layout metadata.
3. Make index reader validate schemas against tracked layout metadata.
4. Make query construction field-aware for single-field cases.
5. Update dedicated-file and merge helpers to validate single-field layout
   generically instead of hardcoding `"body"`.

#### Files

- `components/clara_fts/src/lib.rs`
- `components/clara_fts/src/index_writer.rs`
- `components/clara_fts/src/index_reader.rs`
- `components/clara_fts/src/query/mod.rs`
- `components/clara_fts/src/query/bm25.rs`
- `components/kvengine/src/table/fts/dedicated_file/builder.rs`
- `components/kvengine/src/table/fts/compact/merge_utils.rs`

#### Acceptance criteria

- old `"body"` indexes still read correctly
- new single-column indexes can use `str(col_id)` safely
- no change in behavior for `LegacyBody`

### Phase 3: Multi-column build and `NoScore` read path

#### Objective

Enable one logical FTS index to index multiple columns with deterministic field
names, starting with `NoScore` support.

#### Scope

- L0 historical build
- delta build
- indexed query matching
- brute-force fallback matching

#### Concrete changes

1. Build one writer per tracked logical index, not one per column.
2. Populate all mapped fields in one Tantivy document per row.
3. Update indexed query construction to OR terms across mapped fields.
4. Update fallback readers to read and match across all indexed columns.
5. Validate that query field references match tracked metadata.

#### Files

- `components/kvengine/src/table/fts/compact/l0.rs`
- `components/kvengine/src/table/fts/delta_cache.rs`
- `components/kvengine/src/table/fts/reader/misc.rs`
- `components/kvengine/src/table/fts/reader/brute_force.rs`
- `components/kvengine/src/table/fts/reader/brute_force_cond.rs`
- `components/kvengine/src/table/fts/reader/index_columnar.rs`
- `components/kvengine/src/read.rs`
- `components/clara_fts/src/query/mod.rs`

#### Acceptance criteria

- multi-column `NoScore` works on indexed data
- multi-column `NoScore` works on fallback paths
- doc ID alignment remains correct

### Phase 4: Multi-column scored search

#### Objective

Make scored search behave correctly and consistently for multi-column indexes.

#### Scope

- BM25 stats aggregation across fields
- brute-force scored behavior
- indexed vs fallback score consistency

#### Concrete changes

1. Aggregate BM25 statistics across all mapped fields of one logical index.
2. Build multi-field scored queries in `clara_fts`.
3. Update brute-force scored search to use the same logical field set.
4. Decide whether dedicated-file term-aligned splitting needs layout-specific
   handling for multi-field indexes.

#### Files

- `components/clara_fts/src/query/bm25.rs`
- `components/clara_fts/src/index_reader.rs`
- `components/clara_fts/src/brute_searcher.rs`
- `components/kvengine/src/table/fts/dedicated_file/builder.rs`
- `components/kvengine/src/table/fts/compact/merge_utils.rs`

#### Acceptance criteria

- scored multi-column search works end to end
- indexed and fallback scored behavior are consistent enough for existing query
  semantics

## Formal Test Plan

### Metadata tests

- old snapshot -> `LegacyBody`
- new snapshot roundtrip preserves tracked metadata
- split preserves tracked metadata
- merge preserves tracked metadata
- `FtsUpdate` add/remove merges preserve tracked metadata

### Read/write compatibility tests

- existing old index with `"body"` remains searchable
- new single-column tracked index with `str(col_id)` is searchable
- multi-column `NoScore` search works on indexed files
- multi-column `NoScore` search works on fallback data
- mixed old and new tracked indexes in the same shard both work

### Rebuild tests

- changing `col_ids` forces rebuild
- changing column order forces rebuild
- trying to reuse `LegacyBody` for multi-column logical schema is rejected

### Scoring tests

- indexed multi-column score ordering is stable
- fallback multi-column score ordering is stable
- BM25 stats aggregation across fields does not panic or silently drop fields

## Open Questions

1. Should multi-column scored search be enabled in the same release as
   multi-column `NoScore`, or should it remain gated until scoring behavior is
   validated more thoroughly?
2. Do we want a dedicated version/feature gate at the shard metadata level to
   prevent partially upgraded clusters from creating `ColIdString` indexes too
   early?
3. For dedicated-file sub-offset computation, do we want a conservative
   correctness-first implementation for multi-field indexes before adding
   field-aware offset optimization?

## Decision Summary

- Persist immutable tracked physical layout metadata per FTS index.
- Load old tracked indexes as `LegacyBody`.
- Use `str(col_id)` for new tracked indexes.
- Keep physical layout stable for the lifetime of a tracked index.
- Treat logical column-set or column-order changes as rebuild-required.
- Roll out in phases: metadata, single-field refactor, multi-column `NoScore`,
  then multi-column scored search.