# Cloud Storage Engine

Cloud Storage Engine (CSE) is the storage engine behind the cloud-native,
shared-storage architecture of TiDB X. It keeps the transactional model
and Raft-based replication semantics, while moving the persistent copy of
data to object storage and offloading heavy background work to elastic
worker services.

Its major building blocks are a Raft-based transactional storage layer, a
shared object-storage-backed persistence model, a per-shard LSM-forest data
engine, and worker services for heavier background tasks.

This repository contains the storage engine, the Raft integration layer, the
server runtime, worker services, operational tools, tests, and design notes
for that architecture.

> For the product-level overview of the managed architecture, see
> [TiDB X Architecture](https://docs.pingcap.com/tidbcloud/tidb-x-architecture/).

## Why CSE exists

Classic shared-nothing storage can deliver strong consistency and horizontal
scalability, but it couples online serving traffic with local compaction,
rebalancing, file migration, and other maintenance work. CSE evolves that model
for cloud environments by making object storage the durable backbone and
separating lightweight online execution from heavier background workflows.

In practice, that means CSE is designed to improve:

- **elastic scaling**: new nodes can attach to shared storage instead of waiting
  for large physical data migration
- **workload isolation**: heavy background work can run outside the hot OLTP
  request path
- **predictable performance**: foreground traffic is less exposed to compaction,
  data loading, and maintenance interference
- **cost control**: local SSD is used as cache and working storage instead of
  holding the only persistent copy of all data

## Architecture at a glance

At a high level, CSE combines:

- a **Raft-based transactional storage layer** for correctness and availability
- a **shared object storage layer** as the durable data backbone
- a **per-shard LSM-forest engine** for row-store data and file lifecycle
- **separate worker services** for heavier workflows such as compaction,
  load-data, schema work, remote coprocessor execution, and selected backup or
  encryption tasks

For the architecture overview, see [`doc/architecture/overview.md`](doc/architecture/overview.md).

## Main components

### `cloud_server`
Main server runtime and service wiring.

### `rfstore`
Raft integration layer and replicated metadata control plane.

### `kvengine`
Per-shard storage engine and file lifecycle manager.

### `rfengine`
Raft-log durability engine.

### `cloud_worker`
Worker runtime for heavier background workflows.

### `cse-ctl`
Operational CLI.

## What you can find in this repository

- `cmd/tikv-server`: server binary entry point
- `cmd/tikv-worker`: worker binary entry point
- `cmd/cse-ctl`: operator CLI
- `components/cloud_server`: process/runtime wiring
- `components/rfstore`: Raft layer
- `components/kvengine`: storage engine
- `components/rfengine`: Raft log engine
- `components/cloud_worker`: background workers
- `doc/architecture/overview.md`: high-level architecture overview
- `doc/features`: feature-oriented design and code-path notes
- `tests/`: integration, failpoint, random, and workload tests

## Read this first

If you want a user-facing introduction to the architecture, start here:

- [`doc/architecture/overview.md`](doc/architecture/overview.md)

Useful feature docs for deeper reading:

- [`doc/deploy.md`](doc/deploy.md)
- [`doc/features/txn_file.md`](doc/features/txn_file.md)
- [`doc/features/remote_coprocessor.md`](doc/features/remote_coprocessor.md)
- [`doc/features/schema_manager.md`](doc/features/schema_manager.md)
- [`doc/features/rfengine_design.md`](doc/features/rfengine_design.md)
- [`doc/features/keyspace_level_cpu_throttle.md`](doc/features/keyspace_level_cpu_throttle.md)
- [`doc/features/blob_store.md`](doc/features/blob_store.md)

And for a deeper internal component view:

- `components/cloud_server/MAINTAINER_GUIDE.md`
- `components/rfstore/MAINTAINER_GUIDE.md`
- `components/kvengine/MAINTAINER_GUIDE.md`
- `components/rfengine/MAINTAINER_GUIDE.md`
- `components/cloud_worker/MAINTAINER_GUIDE.md`

## Licensing

- The original top-level license text is now preserved in [`THIRD-PARTY-LICENSE`](THIRD-PARTY-LICENSE).
- Source code and other material incorporated from third-party projects are listed in the generated [`ThirdPartyNotices.txt`](ThirdPartyNotices.txt). See the [`third_party` maintenance guide](third_party/README.md) for its reviewed inputs and update workflow.
- The current commercial notice is available in [`COMMERCIAL-LICENSE`](COMMERCIAL-LICENSE).

## Preserved Upstream Project Documents

Original upstream community documents that are no longer directly applicable to this repository root have been moved to:

- [`doc/Original-Project`](doc/Original-Project)