# Columnar Pack Clean Read

## Problem

Columnar scan currently reads the handle and version columns even when the
requested columns do not contain handle, primary key handle, or MVCC version.
The handle is needed by the normal path for range seek, multi-source merge, and
MVCC visibility filtering. However, some bottom-level columnar packs are already
MVCC-clean:

- no delete marker exists in the pack
- each handle has only one version in the pack
- the table or file max version is visible to the current `read_ts`

For those packs, reading handle and version only to prove that no MVCC filtering
is needed wastes IO, decompression, CPU, and serialization work.

TiFlash has a similar clean-read optimization based on pack-level metadata. This
document proposes adding equivalent pack-level metadata to TiKV columnar files
and using it to skip handle/version reads for eligible packs.

## Goals

- Add pack-level MVCC-clean metadata to columnar table metadata.
- Keep old columnar files readable by falling back to the existing path.
- Allow the read path to skip handle and version pack reads for clean packs when
  the query does not need handle or version output.
- Preserve existing correctness for MVCC, delete markers, common handle tables,
  int handle tables, vector search, FTS, and predicate pushdown.
- Keep the optimization local to eligible packs and easy to disable.

## Non-goals

- Do not remove the handle and version columns from the columnar file format.
- Do not use constant handles with `ColumnarMvccReader`; that would make
  different rows look like versions of one handle.
- Do not support clean read when the result needs handle, `pk_handle`, or version
  output.
- Do not require old files to be rewritten before reads can continue.
- Do not change SQL-visible semantics or TiFlash FFI data layout.

## Current Columnar Read Path

The normal path is:

```text
ColumnarTableReader
    -> ColumnarMergeReader
    -> ColumnarMvccReader
    -> CloudColumnarReader / CloudColumnarReaders
```

Handle and version are used by different layers:

- `ColumnarTableReader::seek` uses handle index and the handle pack to find the
  exact row offset, then aligns version and data columns to that row offset.
- `ColumnarMergeReader` orders sources by `(handle, version)`.
- `ColumnarMvccReader` filters by `read_ts`, skips older versions for the same
  handle, filters delete markers, and checks range end.
- `CloudColumnarReader` serializes handle when the requested column is the
  handle column or a primary key handle column.

Therefore the optimization must bypass merge/MVCC for proven-clean packs. It
cannot simply fill a constant handle into the existing MVCC reader.

## Current File Format

Each columnar table stores special handle and version columns. Each column meta
contains `PackOffsets`, column info, optional min-max index, and column
properties:

```text
num_packs(4) | pack_offsets | row_offsets
col_info_len(4) | col_info
min_max_idx_len(4) | min_max_idx
column_props_len(4) | column_props
```

Each table meta then stores a compressed handle index followed by table
properties:

```text
num_columns(4)
handle_column_meta
version_column_meta
normal_column_metas...
compressed_handle_index_len(4) | compressed_handle_index
table_props_len(4) | table_props
```

Important compatibility detail:

- Table properties are safe to extend because they are at the end of table meta.
  Current code reads the length and ignores the payload.
- Column properties should not be used for this feature yet. Current parser only
  skips the 4-byte length field and assumes the payload is empty, so non-empty
  column properties would break parsing of following column metadata.

## Proposed Metadata

Add a table-level property:

```rust
const PROP_KEY_PACK_MVCC_STATS: &[u8] = b"pack_mvcc_stats_v1";

const PACK_HAS_DELETE: u8 = 1 << 0;
const PACK_HAS_MULTI_VERSION: u8 = 1 << 1;
```

The property value is encoded as:

```text
version: u8 = 1
num_packs: u32
flags: [u8; num_packs]
```

`num_packs` must equal `table_meta.handle_column.pack_offsets.num_packs()`.
Otherwise the reader must ignore the property and use the normal path.

The property intentionally does not store per-pack max version. Reads usually
use a `read_ts` newer than the columnar table/file max version. The clean-read
path uses `max_version <= read_ts` as a coarse visibility gate. If the gate
fails, fall back to the normal MVCC path for the table/file instead of trying to
rescue individual clean packs.

The file-level `max_version` is already parsed from file properties. The builder
also writes table-level `max_ver` into table properties today, but the current
parser ignores table properties. Use file-level `max_version` as the gate by
default; parse table-level `max_ver` if table granularity is needed for
multi-table columnar files.

The stats are defined on handle packs, not on each requested data column pack.
This matches the existing invariant that all columns are aligned by row offset
and the handle pack determines the MVCC row group.

## Metadata Semantics

For handle pack `i`:

- `PACK_HAS_DELETE` is set if any row in the pack has a delete marker
  (`version_column` null mark).
- `PACK_HAS_MULTI_VERSION` is set if any adjacent rows in the pack have the same
  handle.

A pack is clean for a read only if:

```rust
table_or_file_max_version <= read_ts
    && flags[i] & (PACK_HAS_DELETE | PACK_HAS_MULTI_VERSION) == 0
```

The existing builder avoids splitting multiple versions of the same handle
across handle packs: when the handle pack reaches the row limit, it only
finishes the pack if the next handle differs from the last handle in the pack.
This makes adjacent-handle detection inside one pack sufficient for the
multi-version flag.

## Building The Metadata

Add a small `PackMvccStatsBuilder` owned by `ColumnarTableBuilder`.

When `ColumnarTableBuilder::append_block` appends rows into the handle builder,
it already receives the final `[start_offset, end_offset)` appended to the
current handle pack. The stats builder should scan the same row range:

- compare each handle with the previous row handle in the current pack
- set `PACK_HAS_MULTI_VERSION` if equal
- set `PACK_HAS_DELETE` when `block.versions.is_null(row)` is true
- finalize the current pack when the handle builder finishes a pack

When `finish_table` finishes the last handle pack, finalize the last stats pack
and write `PROP_KEY_PACK_MVCC_STATS` into table properties.

Columnar files created before this property exists have no pack stats and will
use the normal path.

## Reader Changes

Parse table properties into `TableMeta`:

```rust
pub(crate) struct TableMeta {
    ...
    pub(crate) max_version: Option<u64>,
    pub(crate) pack_mvcc_stats: Option<PackMvccStats>,
}
```

Add helper methods:

```rust
impl TableMeta {
    fn is_pack_clean_for_read(&self, pack_idx: usize) -> bool;
    fn pack_row_range(&self, pack_idx: usize) -> (usize, usize);
}
```

Add a clean-pack reader path that does not wrap the data in
`ColumnarMergeReader` or `ColumnarMvccReader`.

Enable clean read only when all of these conditions hold:

- only enable for L2 columnar files
- only enable when the selected files for the table do not need merging
- only enable when the requested range covers whole clean packs
- only enable when projection and filter do not need handle, `pk_handle`, or
  version
- only enable when table/file `max_version <= read_ts`
- only enable when pack stats exist and every selected pack is clean for
  delete and multi-version flags

For clean packs, read only the requested data columns by row range. The internal
`Block` may still need row count. If the current ABI requires handle/version
buffers, fill dummy constant buffers only inside the clean-read result object.
Those dummy values must not be consumed by MVCC logic or returned for requested
handle/version columns.

Boundary packs that are only partially covered by the query range must use the
normal path unless the implementation can prove the range covers the full pack.
Dirty packs must also use the normal path.

## Range And Pack Selection

Use the existing handle index and pack offsets:

- pack start row and end row come from `handle_column.pack_offsets`
- pack start handle comes from `handle_index[i]`
- pack end handle can be derived from `handle_index[i + 1]` when available, or
  from the table/file bound for the last pack

For a read range `[start_handle, end_handle)`, a pack is fully covered when:

```text
start_handle <= pack_start_handle
pack_end_handle <= end_handle
```

Clean read works at pack granularity. Each selected pack is classified
independently:

```text
clean pack  -> data-only read
dirty pack  -> normal ColumnarTableReader + ColumnarMvccReader
boundary    -> normal path
```

Mixed-mode output must preserve row order.

## Predicate Pushdown

For predicates that only reference requested data columns and existing min-max
indexes, clean read can still use the same rough pack filter before loading data
columns.

Disable clean read if the predicate references:

- handle or primary key handle
- MVCC version
- columns that require normal MVCC filtering before predicate evaluation

This keeps predicate handling explicit and avoids coupling the clean path to
every filter expression case.

## Concurrency And Serialization

`CloudColumnarReaders` currently materializes a `BlockResult` by serializing
handle, version, and all schema columns after each block. Clean read should avoid
that eager serialization.

Change `BlockResult` toward projection-aware serialization:

- serialize only requested columns
- skip handle and version when they are not requested
- keep physical table id and row count separately
- return dummy handle/version only if the external FFI requires a buffer and the
  caller did not request those values

This is useful even without clean read, because it avoids unnecessary
serialization in multi-table concurrent reads.

## Correctness Invariants

- Clean read is an optimization only. Missing, malformed, or inconsistent stats
  must fall back to the normal path.
- Constant handle/version buffers must never enter `ColumnarMvccReader`.
- A pack with any delete marker is dirty.
- A pack with repeated adjacent handles is dirty.
- A table/file with `max_version > read_ts` must use the normal MVCC path.
- A pack that is not fully covered by the read range uses the normal MVCC path.
- Existing columnar compaction must preserve the invariant that multiple
  versions of one handle are not split across handle packs.

## Compatibility

Old readers:

- Can read files with non-empty table properties because the property payload is
  at the end of table meta.
- Cannot use pack clean read and will ignore the property.

New readers:

- Use pack clean read only when `PROP_KEY_PACK_MVCC_STATS` exists and validates.
- Fall back to the normal path for old files or invalid stats.

File format version:

- The table property is backward-compatible with the current parser shape, so a
  format version bump is not required for safety.
- This design stores pack stats in table properties. Storing stats in column
  properties is intentionally out of scope because column property parsing would
  require a format-safe parser change.

## Metrics

Add counters and histograms to validate the rollout:

- number of packs eligible for clean read
- number of packs rejected by reason: missing stats, range boundary, delete,
  multi-version, table-max-version, projection, predicate
- bytes read for handle/version columns avoided
- clean-read block rows and latency
- fallback count

These metrics should not include table id or keyspace id labels by default to
avoid high cardinality.

## Testing

Unit tests:

- Encode/decode `PackMvccStats`.
- Build stats for packs with no deletes and no repeated handles.
- Build stats for packs with repeated handles.
- Build stats for packs with delete markers.
- Validate fallback when stats length does not match handle pack count.
- Verify old files without stats still read through the normal path.

Reader tests:

- Clean L2 pack, full range, projection excludes handle/version: read data
  columns without loading handle/version packs.
- Same table/file with `read_ts < max_version`: fallback.
- Same file with handle or `pk_handle` projected: fallback.
- Dirty pack with delete marker: fallback and returns correct rows.
- Dirty pack with multiple versions: fallback and returns latest visible row.
- Boundary range that starts or ends inside a clean pack: fallback.
- Common handle and int handle tables.
- Concurrent `CloudColumnarReaders` returns the same data as the normal path.

Integration tests:

- Columnar major compaction creates pack stats.
- Reads after schema update, split, merge, and trim-over-bound still match the
  normal path.
- Columnar files in IA storage still prefetch/read only necessary segments.

## Rollout

1. Add pack stats encoding and parsing.
2. Start writing pack stats for newly built columnar files.
3. Add validation tests comparing stats against full handle/version scans.
4. Add the pack-granular clean-read path behind a config or failpoint.
5. Enable clean read for eligible L2 packs.

## Open Questions

- Does the TiFlash FFI caller ever consume version data for non-projected
  columns, or can clean read return an empty/dummy version buffer?
- Should vector index and FTS readers share this metadata or keep independent
  MVCC visibility checks?