[![Language](https://img.shields.io/badge/Language-Rust-orange.svg)](https://www.rust-lang.org/)

# TiCI

TiCI is a distributed full-text indexing system for TiDB ecosystem workloads.  
It consumes CDC (Change Data Capture) logs and continuously builds and updates full-text indexes, inverted indexes.

## Table of Contents

- [What is TiCI](#what-is-tici)
- [Key Features](#key-features)
- [Architecture Overview](#architecture-overview)
- [Quick Start](#quick-start)
- [Build from Source](#build-from-source)
- [Testing](#testing)
- [Configuration](#configuration)
- [Observability](#observability)
- [Project Layout](#project-layout)
- [Contributing](#contributing)
- [License](#license)

## What is TiCI

TiCI provides distributed indexing services for large-scale text data.

- Decouples index build/maintenance from OLTP query execution.
- Uses CDC logs for incremental index updates.
- Persists index data in S3-compatible object storage.

## Key Features

- **Distributed architecture**: scales with a Meta + Worker model.
- **Incremental indexing**: continuously ingests CDC changes and updates shards.
- **High-efficiency search**: support high-efficiency full-text and custom indexes for searching.
- **gRPC interface**: gRPC service for index creation and management.

## Architecture Overview

TiCI includes the following core roles:

- **Meta service**: metadata management, scheduling, and cluster-level operations.
- **Worker service**: index build, incremental ingestion, and shard compaction.
- **Index libraries**: indexing data structures and retrieval primitives.
- **External systems**: TiDB/TiKV ecosystem, CDC logs, and S3-compatible object storage (MinIO is commonly used for local runs).

## Quick Start

### Prerequisites

- Docker or Podman (for test image and integration workflows)
- Optional: TiUP, MinIO, MySQL client (for local end-to-end playground)

### Local playground

1. Start MinIO:

```bash
docker run -d -p 9000:9000 -p 9001:9001 \
  -v /mnt/data:/data \
  quay.io/minio/minio server /data --console-address ":9001"

CONTAINER_ID=$(sudo docker ps -q --filter ancestor=quay.io/minio/minio)
sudo docker exec "$CONTAINER_ID" mc alias set myminio http://localhost:9000 minioadmin minioadmin
sudo docker exec "$CONTAINER_ID" mc mb myminio/ticidefaultbucket
```

2. Start TiUP playground (example):

- Use default online binary:

```bash
tiup mirror set http://tiup.pingcap.net:8988
tiup playground:v1.16.2-feature.fts --mode tidb-fts --s3.bucket ticidefaultbucket
```

- (Or) Use local build binary:

```bash
tiup playground:v1.16.2-feature.fts --mode tidb-fts --s3.bucket ticidefaultbucket --tici.binpath <tici-server-binary-path>
```

3. Create a full-text index and verify:

- Download test dataset:

```bash
wget "https://quickwit-datasets-public.s3.amazonaws.com/hdfs-logs-multitenants.json.gz"
gzip -d -c "hdfs-logs-multitenants.json.gz" > "hdfs-logs-multitenants.json"
```

- Import dataset:

```bash
python3 ci/read_hdfs.py hdfs_10w --max_rows 100000 --tidb_host 127.0.0.1 --tidb_port 4000 --input "path/to/hdfs-logs-multitenants.json"
```

- Create full-text index, check shard process and query:

```bash
mysql --host 127.0.0.1 --port 4000 -u root \
  -e "ALTER TABLE test.hdfs_10w ADD FULLTEXT INDEX ft_index (body) WITH PARSER standard;"

mysql --host 127.0.0.1 --port 4000 -u root \
  -e 'SELECT * FROM tici.tici_shard_meta;'

mysql --host 127.0.0.1 --port 4000 -u root \
  -e "select count(*) from test.hdfs_10w where fts_match_word('this_is_tici',body) or not fts_match_word('this_is_tici',body)";
+----------+
| count(*) |
+----------+
|   100000 |
+----------+
```

## Build from Source

### Prerequisites

- Rust toolchain (version pinned by `rust-toolchain.toml`)

### Clone

Note: since `tici` is now a private repository, you need to fork it to your github repository first (Access to `tici` has been granted).

```bash
git clone git@github.com:<your_github_account>/tici.git
```

### Build

```bash
cd tici && make build
```

- `make build`: builds workspace with `openssl-vendored` and `failpoints`.
- `make dev`: runs `fmt`, `check`, `build`, and integration-test target build.
- `make release`: builds release binaries with `openssl-vendored`.

## Testing

### Pre-work

#### Fast local checks

```bash
make fmt
make check
```

#### Build docker image (option)

This image contains all the necessary runtime environment and dependencies, such 
as Rust, Minio, etc. You can save the created image for later use.

```bash
sudo make build-test-image
```

### Unit tests

#### Method 1：One-shot test

Outside docker environment, for one-shot creating docker container and test, then remove container.

```bash
make ut
```
#### Method 2：Local Test

You can run it in a local environment or in a Docker environment, running it in a Docker environment is recommended.

```bash
make ci-ut
```

- Crate-scoped tests

```bash
cargo test --features "openssl-vendored"
# or
cargo test -p tici-search-lib
cargo test -p tici_shard
...
```

- Specific test

```bash
cargo test --features "openssl-vendored testexport failpoints" -- meta::reader::shard_cache::manager --test-threads 1
```

- Smaller-scale test

```bash
cargo test -p tici_shard --lib -- utils::key::tests::test_key_from_and_to --exact --nocapture
```

#### Searchlib (Reader)

The Searchlib related tests are run as follows: 

```bash
# Run tests for the searchlib component
cargo test -p tici-search-lib
cargo test -p test_process_usage
```

### Integration tests

#### Method 1：One-shot test

```bash
make it
```

#### Method 2：Local Test

```bash
make ci-prepare-it
make ci-run-it
```

- Sequential testing

```bash
cargo test -p tici_integration_tests --features "openssl-vendored testexport failpoints" -- sequential --test-threads 1
```

- Parallel testing

```bash
cargo test -p tici_integration_tests --features "openssl-vendored testexport failpoints" -- meta
```

- Smaller-scale test

```bash
cargo test -p tici_integration_tests --features "openssl-vendored testexport failpoints" -- meta::compaction::test_remove_node --exact
```

### End-to-end tests

#### Method 1：One-shot test

```bash
make e2e
```

The e2e runner runs multiple `.test` files in parallel by default:

- Directory runs use `--parallel 4` by default.
- A single `.test` file still runs serially, even with the default parallel value.
- Each parallel `.test` file runs in an isolated physical schema, so test cases can keep using the logical `test` schema in SQL and wait directives.

To force serial execution:

```bash
sudo -E E2E_TEST_RUNNER_ARGS="--parallel 1" make e2e
```

To choose a different parallelism:

```bash
sudo -E E2E_TEST_RUNNER_ARGS="--parallel 2" make e2e
sudo -E E2E_TEST_RUNNER_ARGS="--parallel 8" make e2e
```

#### Method 2：Local Test

```bash
make ci-e2e
```

## Configuration

- Configuration templates are under `ci/`.
- For local deployment and CI, start from `ci/*.toml` and override per environment.

## Observability

### Status server

Both Meta and Worker can expose status endpoints. The URL of the status server is configurable:

```toml
advertise-addr = "172.0.1.1:8500"
status-addr = "0.0.0.0:8501"
```

### Dashboards

- Grafana sources are in `grafana/*.jsonnet`.
- After editing `.jsonnet`, run `grafana/generate_json.sh` to regenerate `.json` files.

## Project Layout

- `src/`: `tici-server` binary and core modules (`meta/`, `worker/`, `status_server/`, etc.)
- `components/`: workspace crates (`searchlib`, `tici_shard`, `tici_common`, `tici_grpc`, `tici_import`, etc.)
- `components/tici_integration_tests/tests/`: Rust integration and e2e tests
- `tests/`: script-based test runners and CI helper scripts
- `docs/design/`: design documents
- `grafana/`: dashboards
- `ci/`: Dockerfile and configuration templates

## Contributing

Before opening a PR, make sure you run:

```bash
make format
make check

make ci-ut
make ci-prepare-it
make ci-run-it
make ci-e2e
```

Recommended PR checklist:

- Describe impacted components and config/dashboard changes.
- Include the exact test commands and results.
- Keep changes scoped and add/update tests for behavior changes.

## License

This project is licensed under the MIT License - see the LICENSE file for details.