--- title: S3 Express Append Log Server Specification status: final date: 2026-09-05 --- # S3 Express Append Log Server Specification ## 1. Scope This specification owns the server half of the SQLite WAL append optimization. It applies to `tidbcloud/fs` only: persistent metadata, S3 Express access, the HTTP contract, logical read bounds, object lifecycle, and configuration. Drive9 client and FUSE behavior are specified separately in the `drive9` repository. The objective is to make a Drive9 server acknowledge a WAL append only after the new tail is durable in S3 Express and the corresponding file revision is committed. It resolves the growing full-rewrite write amplification in #875. ### In scope 1. `single` and `append_log` content layouts. 2. A server-owned, streaming `POST ?append-log` API. 3. S3 Express append-at-offset, server-proxied range reads and rewrites, CopyObject rebase, and layout-aware object deletion. 4. Metadata CAS, zero-length creation, rewrite/reset handling, and fail-closed recovery after an incomplete append. 5. Capability advertisement and deployment configuration. 6. A durable, immutable S3 Express binding per storage namespace, including fail-closed tenant deletion when the required Express cleanup target cannot be proven. ### Out of scope 1. Drive9 FUSE WAL discovery, snapshotting, and retry policy. 2. Direct FUSE/client access to the Directory Bucket. 3. #876's stale FUSE payload-generation / durable-watermark fix. 4. Multiple independent writers, `append_log_ops`, durable request idempotency, or automatic finalization after a server crash. 5. A generic segmented-object or manifest storage format. 6. `append_log` objects larger than 5,000,000,000 bytes (5 GB) or server-side multipart rebase. 7. Native fork append-log write support or append-log copy-on-write. Inherited append-log objects remain readable in forks, but append and rewrite are explicitly unsupported until that protocol is separately designed. 8. Automatic relocation of an already-bound namespace between Directory Buckets, multi-target namespace cleanup, or data-copy migration. A bound namespace requires matching Express configuration until it is deleted. The base append-log implementation estimate is **~650-850 net LoC**. The durable Express-binding extension adds **~250-400 net LoC**. The combined server production estimate is **~900-1,250 net LoC** (tests excluded). ## 2. Model and metadata `storage_type` remains `s3`; it does not encode a cloud service choice. `content_layout` selects the physical client and object semantics. ```sql ALTER TABLE contents ADD COLUMN content_layout VARCHAR(32) NOT NULL DEFAULT 'single'; ALTER TABLE file_gc_tasks ADD COLUMN content_layout VARCHAR(32) NOT NULL DEFAULT 'single'; -- Required while legacy files dual-write remains supported. ALTER TABLE files ADD COLUMN content_layout VARCHAR(32) NOT NULL DEFAULT 'single'; -- Central MetaDB: required for durable physical-client routing. ALTER TABLE object_gc_candidates ADD COLUMN content_layout VARCHAR(32) NOT NULL DEFAULT 'single'; -- Central MetaDB: immutable physical target for append-log objects. ALTER TABLE storage_namespaces ADD COLUMN append_log_binding_state VARCHAR(16) NOT NULL DEFAULT 'unbound', ADD COLUMN append_log_bucket VARCHAR(255) NOT NULL DEFAULT '', ADD COLUMN append_log_prefix VARCHAR(2048) NOT NULL DEFAULT '', ADD COLUMN append_log_region VARCHAR(64) NOT NULL DEFAULT ''; ``` | Layout | `storage_type` | `content_blob` | Physical object | |---|---|---|---| | `single` | current `db9` / `s3` selection | current behavior | current object-store route | | `append_log` | always `s3` | always `NULL` | one Directory Bucket object | `append_log` uses the same encryption metadata as the default `single` S3 policy: `storage_encryption_mode='none'` and an empty `storage_encryption_key_id`. Here `none` means the server sends no per-object SSE headers; the Directory Bucket's default encryption still applies. Create writes these values explicitly instead of relying on the datastore's legacy zero-value fallback. Append retains them, and rewrite or rebase copies them to the replacement generation. `contents.content_layout` is authoritative. The legacy `files` column is a dual-write mirror. `file_gc_tasks.content_layout` is copied at deletion time because its content row may no longer exist when file GC runs. `object_gc_candidates.content_layout` is copied whenever file GC, overwrite, rebase, fork cleanup, or failed-write cleanup creates a physical-object candidate. The object-GC worker selects the physical client from that durable field; it must not infer the bucket from the object key. ### 2.1 Durable S3 Express namespace binding `storage_namespaces` is the control-plane source of truth for a tenant's ordinary S3 target. It also owns the append-log physical target. A namespace has exactly one of these binding states: | State | Target columns | Meaning | |---|---|---| | `unbound` | all empty | Created after this change and has never been made append-log capable. No Express cleanup is required. | | `bound` | bucket and region are non-empty; base prefix is normalized and may be empty | The immutable Directory Bucket target for every append-log generation in this namespace. | The base prefix is the configured optional prefix before the fixed `append-log//` suffix. `append_log_bucket`, `append_log_prefix`, and `append_log_region` are immutable once `bound`. The base prefix is optional, matching `DRIVE9_S3_EXPRESS_PREFIX`. Credentials and role configuration are intentionally not persisted; an unavailable current identity causes an operation to fail and retry rather than changing the physical target. During S3 backend cold-open, a complete Express configuration atomically transitions an `unbound` namespace to `bound`, before the pool injects an Express client. This matches ordinary S3 namespace allocation: the namespace is bound when its backend is first opened, not when its first object is written. The transition is compare-and-set guarded, so concurrent cold opens converge on one target. The pool creates an Express client only when the current complete Express configuration exactly matches a `bound` namespace's bucket, base prefix, and region. An empty, incomplete, or different configuration makes append-log unavailable; it must not redirect operations to a new target or construct a client from the persisted target alone. This is the same fail-closed availability rule used by ordinary S3 namespace cleanup. An empty `DRIVE9_S3_EXPRESS_BUCKET` disables the S3 Express append-log feature: the server creates no Express client and accepts no append-log operation. A bound namespace is not silently rehomed or forgotten; until matching configuration is restored, its Express reads, GC, and tenant cleanup are unavailable and a required cleanup retries without tenant finalization. Append-log is new in this release: there are no pre-existing append-log objects or historical Express targets to recover. The MetaDB schema migration therefore initializes all existing normal storage namespaces as `unbound`. No legacy backfill state, admin migration command, fallback to current configuration for a bound namespace, or automatic relocation is needed. An `append_log` object key is `append-log///`. The key change is the physical generation. Namespace scoping lets tenant deletion remove exactly that tenant's Directory Bucket objects before finalizing metadata. Do not add `append_generation` or `append_part_count`: the server rebases after S3 returns `TooManyParts`. The full-file `checksum_sha256` is cleared after an append. Calculating it would require rereading the growing WAL and defeat the hot-path objective. Semantic text extraction and embedding work are skipped for `append_log`. Every active tenant schema source and managed-schema export must carry the columns with `DEFAULT 'single'`: TiDB app, TiDB auto, TiDB shared, db9, and the legacy files compatibility schema. The central MetaDB schema and migration must add the object-GC column with the same default. Existing content and GC candidates remain `single`. ## 3. Configuration and capability Add optional server configuration: ```text DRIVE9_S3_EXPRESS_BUCKET= DRIVE9_S3_EXPRESS_PREFIX= ``` The Express client uses the namespace's bound bucket, base prefix, and region only after verifying they match the current server configuration. It uses the server's current AWS credential configuration, but not a custom S3-compatible endpoint or path-style addressing. The AWS SDK resolves the Directory Bucket's zonal virtual-hosted endpoint and manages S3 Express sessions. The runtime identity needs `s3express:CreateSession`; an SSE-KMS bucket also requires the corresponding KMS permissions. Express objects rely on the Directory Bucket's configured default encryption. Append-log requests do not inherit general-bucket per-object encryption headers, including DSSE-KMS or a tenant-specific encryption context. An empty bucket setting disables the feature; it does not redirect or orphan an existing bound namespace. There is no client credential, presigned object URL, or implicit fallback from an existing `append_log` object to the default bucket. Expose the enabled server capability from `GET /v1/status`: ```json { "storage_capabilities": { "append_log_v1": true } } ``` The field is omitted or false unless the requesting namespace is `bound`, the current Express configuration matches that binding, its Express client initializes successfully, and layout-aware object GC is available. Older clients ignore it; newer clients must treat absence as disabled. `HEAD /v1/fs/{path}` also returns `X-Dat9-Content-Layout: single|append_log` for a file, alongside its existing revision, size, and storage-type headers. ## 4. HTTP API Add a versioned streaming action: ```text POST /v1/fs/{path}?append-log Content-Length: X-Dat9-Expected-Revision: X-Dat9-Expected-Size: 200 {"revision": , "size_bytes": } ``` The request body is exactly one immutable tail; the endpoint does not accept a multipart plan or a full file. `Content-Length` is required, chunked transfer is rejected, and the server streams exactly that many bytes without buffering the whole tail. The expected revision and size headers are required and must be non-negative. A zero-length body is valid only for an expected-revision `0`, expected-size `0` create. The endpoint is authorized as `FSOpWrite` before any metadata lookup. An `append_log` object has a hard maximum logical size of `5_000_000_000` bytes (5 GB). The server checks `expected size + Content-Length` for overflow and against this limit before any S3 mutation. The same limit applies to create, full rewrite, truncate/reset replacement, and rebase. The server also applies the normal per-tenant file-size, storage, file-count, and request upload-size limits before any Express mutation. Quota failures use the existing `507` response contract. | Request state | Result | |---|---| | missing path, expected revision `0`, expected size `0` | create a new `append_log` object and file after object write | | existing `append_log`, matching revision and size | append body at that size and CAS metadata | | existing `single` / unsupported backend | `400 append_log_unsupported` | | stale revision or logical size | `409 append_log_conflict` | | S3 rebase completed | `409 append_log_rebased` | | resulting logical size exceeds 5,000,000,000 bytes | `413 append_log_too_large` | | S3 error or metadata commit error | request failure; never report success | Domain error responses use a JSON `code` field in addition to `error`; the machine-readable code is part of this contract. Only `append_log_rebased` permits the Drive9 client to re-stat and retry its unchanged tail. A normal conflict must not enter a last-writer-wins path. ## 5. Append commit protocol ```text validate layout + expected revision/size/ref │ ▼ PutObject(WriteOffsetBytes=expected size, body=tail) │ ▼ UPDATE contents/inodes ... WHERE revision, size, ref, layout match │ ┌─────┴─────┐ ▼ ▼ success failure acknowledge fail closed ``` The S3 offset is the physical compare-and-set. The metadata update is the logical compare-and-set and updates `size_bytes`, `revision`, and `mtime`, clears the checksum, and retains the same `storage_ref`. Quota accounting uses only the appended byte delta. If the metadata commit outcome is unknown, the request returns a server error and does not claim success. A later request first re-reads metadata: a committed revision causes a normal conflict, while unchanged metadata plus a longer physical object enters the divergence rebase below. Reads remain bounded by the metadata version visible to that request. Every operation that can create a new Express key — create, full rewrite, truncate/reset replacement, prefix rebase, or CopyObject rebase — uses the existing fenced failed-write ownership protocol before the S3 mutation: ```text prepare object-GC candidate(content_layout=append_log) │ ▼ write or copy the new Express object │ ▼ claim candidate for metadata publishing │ ▼ metadata insert/CAS ┌─────┼──────────────┐ ▼ ▼ ▼ success known failure outcome unknown clear GC clean/retry retain candidate candidate new object for reachability check ``` Preparing the candidate after the object write is forbidden: a process crash in that gap would leave an object with no durable cleanup owner. A commit with unknown outcome must not directly delete the new object. Object GC first checks whether the reference became reachable and either clears the failed-write candidate or deletes the unreferenced object. For an expected-revision `0` create, write the first (possibly zero-byte) object through the Express client under a pre-created failed-write candidate, then insert the file/node/content metadata. A known metadata failure deletes or retries cleanup of the unreferenced object; an unknown outcome leaves the candidate for reachability-based recovery. This lets the first WAL header use `append_log` instead of first being stored inline. The conditional full-body `PUT` fallback is the only non-tail mutation of an existing `append_log`: it writes a new Express object and atomically replaces `storage_ref` while retaining `content_layout=append_log`. Before the metadata CAS, the server registers a delayed, layout-aware GC candidate for the old reference. Reachability checking prevents deletion while that reference remains current. A known CAS failure cleans the new-object candidate but retains the old-reference candidate: it may have been published by a concurrent winning CAS, and reachability safely clears it if still current. Existing nonempty `single` objects are not migrated in place. The conditional append-log `PUT` fallback rejects `X-Dat9-Tag` and `X-Dat9-Description`. It does not silently ignore requested metadata or introduce a separate atomic rewrite-plus-metadata update protocol. The ordinary `PUT` handler looks up an existing target's layout before its current inline-versus-presigned-upload split. For `append_log`, it always streams the full body through the server into a new Express key, even when the body exceeds the normal inline threshold. Existing direct-upload, V2 upload, PATCH, and legacy `POST ?append` plan endpoints must not issue an Express URL; when aimed at an `append_log` target they return `400 append_log_unsupported`. The Drive9 fallback for a reset or non-tail change is the conditional full-body `PUT` path. The companion FUSE design must select that path for an existing `append_log` file instead of retrying a V2 or PATCH plan. All other generic mutation paths — V1/V2 multipart upload, batch-write, PATCH, legacy append plans, and generic backend overwrite — reject an existing `append_log` with `append_log_unsupported`; they never convert it to `single` or DB9 storage. LayerFS likewise explicitly rejects append-log reads, writes, snapshots, commits, and rollback paths in this release. ## 6. Reads, rebase, and GC `append_log` reads are server-proxied and bounded by committed metadata; they never return a redirect or presigned Express URL. A GET without `Range` issues an Express range request for `[0, size_bytes)` and returns `200` with exactly the committed `Content-Length`. A zero-length file returns an empty response without issuing an invalid S3 range. The read plan carries the committed `storage_ref`, `size_bytes`, and revision from one metadata read. Range parsing, response headers, and the Express range request all use that same generation; the range-open path must not re-stat by logical filesystem path. A ranged GET accepts one HTTP byte range, intersects it with `[0, size_bytes)`, and issues one inclusive Express range request. It returns `206` with `Content-Range` and the bounded `Content-Length`. An unsatisfiable, malformed, or multi-range request returns `416`; the server must never widen a range to include a physical suffix. This prevents data written by S3 but not committed in metadata from becoming visible and avoids downloading the entire WAL for a SQLite page read. `single` S3 files retain the existing redirect behavior. When S3 rejects an append because its actual size differs from committed `size_bytes`, the server treats the object as divergent: 1. Range-read the committed prefix `[0, size_bytes)`. 2. Prepare a failed-write candidate for a new `append-log///` object. 3. Stream the prefix into the new object and claim it for publishing. 4. Register a delayed Express-routed GC candidate for the old object. 5. CAS the new `storage_ref`, keep size unchanged, and increment revision. 6. Return `409 append_log_rebased`; do not append the request body in that request. This discards an unacknowledged tail rather than guessing its identity. It is the intentional first-release crash policy. S3 Express returns `TooManyParts` after 10,000 append parts. The server uses `CopyObject` to a new key, pre-registers the old key for delayed GC, CASes the new reference, and returns the same retryable `append_log_rebased` result. The client re-stats and retries the retained tail. The common path adds no part-count metadata write or per-append HeadObject request. The 5,000,000,000-byte layout limit guarantees that the source fits the single-call `CopyObject` contract; multipart rebase is not implemented. Both rebase paths use the fenced candidate protocol. A metadata CAS conflict after the new object is written cleans or releases the new-object candidate, retains the old-object candidate, and returns `append_log_conflict`, not `append_log_rebased`. A commit with unknown outcome retains both candidates and returns a server error; reachability decides which object is authoritative. All deletion paths — file deletion, overwrite replacement, rebase cleanup, and failed-write cleanup — select the default S3 client for `single` and the Express client for `append_log`. File GC copies the layout into the central object-GC candidate before its tenant row can disappear. Object GC uses only the candidate's durable layout and routes existing candidates to `single` via the schema default. Fork reachability protection that currently covers shared `blobs/` and `layers/` references must also cover `append-log/` references. Tenant deletion resolves its durable `storage_namespace_id` from the delete job, then derives `append-log//` under that namespace's bound bucket/base-prefix/region. It deletes the normal S3 target and, for a `bound` namespace, the bound Directory Bucket prefix before tenant metadata is finalized. Both deletes are part of the same retryable job; the job does not finalize after either target alone. `unbound` means no append-log client could have been issued and safely skips Express cleanup. For a `bound` namespace, a malformed binding, absent or mismatched current Express configuration, an unavailable bound client, or an Express delete error is a hard cleanup failure: the job records the error, retries, and must not call `FinalizeTenantDelete`. The worker must never recompute an Express cleanup target from a different current global bucket or prefix. As with ordinary S3, retain matching Directory Bucket configuration until bound namespaces have completed deletion. ## 7. S3 client boundary Keep ordinary `S3Client` behavior unchanged. Add a narrow append-log storage interface implemented by the AWS S3 client and injected separately into the backend: 1. append at an explicit write offset; 2. range get with an inclusive start/end bound; 3. copy object to a new key; and 4. streaming put/delete for create, rewrite, rebase, and GC. The boundary returns typed offset-mismatch and too-many-parts errors so backend logic does not parse AWS error strings. It exposes no presign operation. All write/copy methods accept an exact content length, enforce the 5,000,000,000-byte object limit before calling AWS, and keep SDK-specific session behavior inside the implementation. The AWS SDK's S3 Express session support remains inside this server-side client. Local/mock implementations can provide deterministic equivalents for unit tests; local mode advertises no append-log capability by default. ## 8. Required tests and rollout 1. Schema/default compatibility across every tenant provider, including DB9 and app-managed upgrades from tables without `content_layout`, legacy dual-write, and central `object_gc_candidates` migration. 2. HTTP append tests for authorization, missing/malformed headers, chunked and short bodies, zero create, append, conflict, structured rebase, and 5,000,000,000-byte overflow rejection before S3 mutation. 3. HTTP read tests for whole, single-range, zero-length, `416`, committed-bound enforcement, a read-plan generation race, and absence of an Express redirect; rewrite tests must prove that `append_log` bypasses direct and multipart upload plans. 4. Backend tests for offset append, metadata CAS, bounded reads, checksum invalidation, explicit default encryption metadata, zero reset, old-object GC routing, active-fork protection, generic-write and LayerFS rejection, namespace-scoped keys, and failed-write cleanup for every new-key path. 5. Fault tests at S3 success before metadata commit, known CAS failure with a retained old-reference candidate, unknown commit outcome, prefix rebase, CopyObject rebase, cleanup failure, and GC failure. Acknowledged bytes must remain readable. 6. Durable binding tests: concurrent cold-open binding, immutable target reuse, absent/mismatched configuration fail-closed deletion, malformed-binding failure, and normal-plus-Express cleanup before finalization. 7. Production S3 Express validation: 10,000 appends, `TooManyParts`, rebase, one successful follow-up append, tenant namespace cleanup, binding/default configuration changes, and bucket/KMS configuration compatibility. Enable the setting only after the Directory Bucket, bucket encryption policy, server IAM permission, and metrics are deployed. It is safe to ship the server first: unbound namespaces remain incapable until configuration is complete, and old clients continue current behavior. Retain matching configuration until all bound namespaces have completed deletion; changing or removing it earlier intentionally leaves their append-log operations and tenant cleanup retrying.