# Compaction Worker

Status: Frozen architecture note
Owner: Search Platform Architecture

## 1. Scope

This document defines the execution design of `Compaction Worker`, the background-compute executor that rewrites immutable object sets to reduce read amplification and space amplification while preserving manifest-first correctness.

This document is subordinate to the canonical owner stack:
- machine-readable contract: `api/proto/searchfabric/v1/search_fabric.proto`
- internal owner file for this boundary: `api/proto/searchfabric/v1/compaction_worker.proto`
- public ingest/query/control semantics: `docs/specs/core/search-fabric-api.md`
- canonical error registry: `docs/specs/core/error-codes.md`
- system topology and state ownership: `docs/specs/core/search-fabric-architecture.md`

It must not redefine:
- public `MutationBatch`, `MutationAck`, `GetReadiness`, or query-token semantics,
- committed-manifest publication or `manifest_committed_head.json` ownership,
- CCS-owned lifecycle policy semantics or lifecycle-window hot projection truth,
- WAL replay-anchor ownership (`wal_replay_start_anchor`, `wal_durable_anchor`),
- GC delete authority or public `SF_*` error mappings.

## 2. Canonical Position in the System

`Compaction Worker` is a stateless background executor behind the manifest-driven scheduling path:

```text
Background Compute Coordinator / manifest-driven scheduler
  -> Compaction Worker
       -> Object Store (read old objects, write replacement objects)
       -> scheduler-facing `ManifestRewriteSuccess`
```

The checked-out topology in `docs/specs/core/search-fabric-architecture.md` places the component in:

```text
Background Compute Coordinator -> Compaction Workers -> Object Store -> Background Compute Coordinator -> Manifest Publisher (CAS swap)
```

The worker exists to rewrite already-published immutable objects. It is not:
- a public RPC surface,
- a second visibility authority,
- the owner of background-job admission or predictive pacing,
- the owner of query continuation safety,
- the owner of physical deletion or tombstone reservation release,
- the owner of schema-serving readiness in CCS.

Primary source anchors:
- `docs/specs/core/search-fabric-architecture.md` Section `3.1`, Section `3.2`, Section `4.4`, Section `10.1`, Section `10.2`, Section `10.5`, Section `12.2`, Section `12.3`, Section `12.7`
- `docs/specs/core/search-fabric-api.md` Section `4.1`, Section `6.8`, Section `6.11`
- `api/proto/searchfabric/v1/search_fabric.proto` (`DataTier`, `EngineWatermarks`, `IngestMode`, `FieldAccessIntent`, `VectorRetrievalIntent`, `IndexSortSpec`, `LifecyclePolicy`)
- `docs/architecture/manifest-publisher.md`
- `docs/architecture/query-broker.md`
- `docs/architecture/gc-worker.md`

## 3. Design Anchors

The design below is constrained by the following checked-out facts:

1. Compaction is defined as object rewrite plus manifest swap, and heavy rewrite/delete operations are prohibited on ingest/query plane nodes.
2. `Compaction Workers` are a stateless fleet in the background-compute plane; `Background Compute Coordinator` owns compaction/GC priority computation, lifecycle scheduling, and isolation budgets.
3. Trigger classes are workload-aware rather than wall-clock-only:
   - read amplification when query fan-in crosses configured segment-count thresholds,
   - space amplification when obsolete/tombstone density exceeds configured thresholds,
   - collection-scoped consolidation pressure when tiny fragmented objects cause high object-store overhead.
4. If concurrent writes advance during a long-running compaction, workers must apply newer delta tombstones before final CAS so stale rewrites do not resurrect logically deleted rows.
5. Compaction workers perform deterministic schema upcasting when merging compatible mixed-version segments; incompatible schema changes remain fenced schema-evolution work rather than opportunistic compaction behavior.
6. Lifecycle-managed manifests publish window bounds plus effective `DataTier`; a new tier becomes query-visible only after manifest CAS, and physical reclaim remains a later GC concern.
7. Vector rewrite policy is governed by the effective three-axis tuple `(DataTier, FieldAccessIntent, VectorRetrievalIntent)` rather than by tier alone:
   - `HIGH_FIDELITY` rewrites must preserve the payload needed for exact rerank or audit-grade replay,
   - `STANDARD` rewrites may choose balanced layouts and may drop residual payload in colder windows only when the effective quality contract allows it,
   - `ECONOMY` rewrites may choose the densest admitted layout and may omit residual/raw rerank payload.
8. Query Broker pins `(committed_manifest_version, manifest_checksum_or_etag)` for the full request and continuation lifetime, so compaction only performs logical dereference; delayed GC preserves old manifests and objects long enough for valid continuation to finish.
9. `Manifest Publisher` serializes per-collection committed-manifest progression; stale expected-head or stale replacement assumptions must fail closed instead of silently rebasing to a different canonical result.
10. Object discovery is manifest-first and pointer-driven. Correctness must not depend on object-store `LIST`.
11. The current architecture names collection-scoped consolidation only as a compaction trigger. The active owner stack still keeps manifests, object layout, lifecycle, authorization, and GC collection-scoped, so this note does not define cross-collection shared immutable objects as a correctness-bearing contract.
12. `GetCollectionStatistics` remains CCS-owned; its optimizer-statistics projector may consume compaction outputs as one source input, but compaction does not publish canonical statistics snapshots by itself.

## 4. Responsibilities and Non-Responsibilities

### 4.1 Responsibilities

`Compaction Worker` is responsible for:

1. Re-validating an admitted compaction candidate against the active committed-manifest lineage before rewriting.
2. Rewriting manifest-referenced immutable metadata-search, semantic-search, and related sidecar objects into a lower-amplification replacement set.
3. Preserving canonical winner order inside rewritten coverage:
   - highest `commit_ts`,
   - then `source_partition`,
   - then `source_epoch`,
   - then `source_seq`.
4. Applying deterministic schema upcasting when merging compatible mixed-version objects.
5. Realizing allowed tier- and quality-aware physical rewrites without weakening the admitted schema/lifecycle contract.
6. Producing the scheduler-facing `ManifestRewriteSuccess` completion facts needed for later manifest cutover, including replacement artifacts, replaced objects, affected lifecycle windows, and optional target-tier annotation.
7. Returning that same trusted pre-cutover completion carrier to the lease-owning background workflow so it can report completion and perform the later `Manifest Publisher` handoff without translating executor-only result types.
8. Emitting internal observability needed for background scheduling, retry, and backlog management.

### 4.2 Non-Responsibilities

`Compaction Worker` is not responsible for:
- public write admission or `MutationAck` timing,
- public `safe_ts` / readiness computation,
- query token interpretation or continuation lifetime tracking,
- source-object discovery through CCS head state or object-store `LIST`,
- opportunistic "latest READY schema" lookup in CCS to redefine rewrite semantics,
- advancing `applied_ts_*`, `cdc_resolved_ts`, `wal_replay_start_anchor`, or `wal_durable_anchor` as a worker-local authority,
- publishing CCS-owned optimizer-statistics snapshots directly,
- mutating `NRT Delta Store`,
- physical deletion of superseded objects,
- collection/namespace reservation release,
- cross-collection shared immutable-object contracts in the current revision.

## 5. Rewrite Classes

Compaction work in this architecture falls into four rewrite classes.

### 5.1 Read-Amplification Reduction

This is the steady-state "too many objects per query" path:
- too many metadata-search segments for one collection/window,
- too many semantic objects or sidecars for one retrieval path,
- repeated query plans that need large fan-in from S3 for the same committed snapshot.

The goal is to reduce object fan-in while preserving the exact same committed row winners.

### 5.2 Space-Amplification Reduction

This path rewrites immutable objects whose obsolete-version or tombstone density is high enough that keeping them unmerged wastes storage and query work.

The precise thresholds are service policy, not public contract. The checked-out core architecture names a representative tombstone-density threshold only as an example, not as a public compatibility guarantee.

### 5.3 Lifecycle-Driven Tier Rewrite

Some background rewrites are triggered because a lifecycle-managed window crossed a transition deadline:
- the source of truth remains the admitted `LifecyclePolicy` plus committed manifests,
- CCS `collection_lifecycle_windows` is a scheduling hint only,
- the new `DataTier` becomes query-visible only after committed-manifest CAS publishes the new object references and tier annotation.

The same rewrite engine may be reused here, but lifecycle policy ownership remains outside the worker.

### 5.4 Schema and Layout Consolidation

Compaction may also:
- upcast compatible older metadata objects into the currently admitted manifest-visible schema generation,
- rewrite metadata-bearing objects into the admitted `IndexSortSpec` physical order,
- rewrite semantic objects into a different admitted quantization or residual-preservation layout when the effective quality contract allows it.

These are layout realizations of already admitted control-plane state. They must not invent a new schema generation or silently weaken an existing one.

## 6. State Model and Safety Rules

### 6.1 Durable Truth vs Ephemeral Worker State

`Compaction Worker` must remain stateless for correctness.

Durable truth lives in:
- committed manifests and `manifest_committed_head.json`,
- immutable objects referenced by those manifests,
- admitted schema/lifecycle policy already reflected in committed manifest state,
- replacement objects that have been durably written but are not yet visible until manifest cutover.

Worker-local queues, caches, spill files, or partial merge outputs are execution aids only. They may be rebuilt after crash from the owner state above.

### 6.2 Canonical Rewrite Predicate

For a replacement candidate to be publishable, all of the following must hold:

```text
publishable(rewrite) =
  source_set_resolves_from_committed_manifest
  AND replacement_artifacts_are_durably_written
  AND merged_output_preserves_canonical_winner_order
  AND rewrite_is_still_compatible_with_current_committed_head
  AND newer_tombstones_or_invalidations_since_base_are_absorbed_or_cutover_is_rejected
  AND admitted_schema_lifecycle_quality_contract_is_not_weakened
```

The checked-out owner set now pins that handoff split explicitly: worker execution returns the same scheduler-facing `ManifestRewriteSuccess` carrier later accepted by `BackgroundComputeCoordinatorInternalService.ReportBackgroundTaskCompletion`, and `Background Compute Coordinator` performs the later publisher handoff. The safety conditions above are therefore machine-visible rather than comment-only.

### 6.3 What Must Never Be Used as Sole Authority

The worker must never decide a rewrite or cutover from any of the following in isolation:
- CCS `manifest_head_version`,
- wall-clock age alone,
- object-store `LIST`,
- local cache contents,
- a stale scheduler queue entry,
- a newer CCS schema-serving generation that is not already reflected in the committed manifest lineage for the target rewrite.

## 7. Execution Pipeline

### 7.1 Candidate Pinning and Input Discovery

The scheduler admits a job on a collection-scoped committed base:
- collection identity,
- rewrite objective,
- target object/window set,
- expected committed-manifest base version or equivalent fence.

Execution then stays manifest-first:

```text
admitted compaction job
  -> open exact committed manifest base
  -> enumerate source objects from that manifest only
  -> rewrite replacement objects
  -> return replacement facts to Background Compute Coordinator
```

If lifecycle projections or query-side compaction-pressure signals helped choose the job, they remain hints only. Final input discovery still comes from the committed manifest.

### 7.2 Modality-Specific Rewrite Rules

For metadata-bearing objects, compaction may:
- merge fragmented metadata-search segments,
- rewrite tombstone/visibility sidecars together with their surviving row versions,
- apply deterministic schema upcasting for additive or approved widening changes,
- emit metadata-bearing objects in the admitted `IndexSortSpec` order when the active schema generation requires that physical layout.

For semantic-bearing objects, compaction may:
- merge fragmented semantic segments,
- rewrite vector payloads into another admitted quantization/layout,
- preserve or relocate rerank/audit payload according to the effective quality contract,
- realize colder-tier encodings only when the admitted three-axis tuple allows them.

For both modalities:
- local implementation may use bounded RAM or NVMe spill,
- spill remains disposable,
- the resulting committed object set must stay collection-scoped in the current revision.

### 7.3 Long-Running Job Revalidation

Long-running compaction must assume the committed head may advance during execution.

Required behavior:
1. Re-open the current committed head before cutover.
2. Preserve unchanged manifest-owned metadata that the worker does not own.
3. Apply newer delta tombstones and equivalent invalidation state needed to avoid zombie resurrection.
4. Reject cutover when the newer committed head changed overlapping coverage in a way that the replacement candidate cannot incorporate without changing the canonical result.

This is optimistic background execution, not blind overwrite.

### 7.4 Manifest Cutover

Cutover remains manifest-first:

```text
replacement objects durably written
  -> trusted replacement facts reported to Background Compute Coordinator
  -> coordinator submits publisher handoff
  -> Manifest Publisher writes manifest_vN+1
  -> CAS manifest_committed_head.json
  -> replacement becomes query-visible
```

Important rules:
- the worker does not advance visibility by writing objects alone,
- `Manifest Publisher` serializes committed-manifest progression,
- stale expected-head or incompatible replacement candidates fail closed and are retried from a fresh committed base,
- unchanged frontier metadata and replay anchors remain carried by the owning committed-manifest path rather than by a worker-local side protocol.

## 8. Delayed GC, Lifecycle, and Query Continuity

Successful compaction performs logical dereference, not immediate physical deletion.

After cutover:
- old objects stop being part of the newest committed snapshot,
- valid continuations may still pin and reopen older manifests directly,
- `GC Worker` later decides physical deletion using the canonical delayed-GC rule.

That means compaction must assume superseded objects remain protected until GC proves all relevant gates:
- not referenced by the latest committed reachability,
- delayed-GC horizon compatible with query token lifetime,
- `evict_commit_ts < global_safe_ts - retention_window`,
- lifecycle retention deadline has passed when the collection is lifecycle-managed.

Compaction therefore never eagerly deletes:
- old immutable objects,
- no-longer-current manifests,
- lifecycle-retired windows,
- replay-related objects.

## 9. Failure Handling and Recovery

### 9.1 Failure Cases

If replacement-object write fails:
- no visibility changes,
- the job may retry or be re-admitted later,
- partial outputs remain unpublished artifacts rather than committed truth.

If manifest cutover fails because the expected head is stale or the replacement is incompatible with the current head:
- no visibility changes,
- the worker must retry from a fresh committed base rather than silently rebasing.

If committed-manifest CAS succeeds but worker-local completion bookkeeping is lost:
- recovery follows the committed manifest, not local state,
- duplicate execution remains safe because visibility truth is the committed-manifest pointer.

If object-store throttling occurs:
- workers back off and preserve ingest/query object-store budgets first,
- scheduler-level pacing remains owned by `Background Compute Coordinator`.

### 9.2 Recovery Rules

Recovery is pointer-driven:
- read the current committed manifest lineage,
- trust only manifest-referenced objects plus durably written replacement objects that can still be proven part of an admitted replacement attempt,
- discard local spill or cache state,
- avoid `LIST`-based live-object discovery.

Unpublished orphan outputs are cleanup debt for later GC or retry logic. They are never query truth by themselves.

## 10. Resource Isolation and Observability

`Background Compute Coordinator` owns predictive pacing, admission budgets, and autoscaling policy. `Compaction Worker` consumes those budgets; it does not redefine them.

Useful worker-facing signals include:
- queue depth by rewrite class,
- bytes read and bytes written,
- object-count reduction and segment-fan-in reduction,
- estimated read-amplification and space-amplification change,
- stale-head retry rate,
- cutover CAS conflict rate,
- delta-tombstone rebase count,
- lifecycle tier-rewrite count,
- quality-intent-specific rewrite count (`HIGH_FIDELITY` / `STANDARD` / `ECONOMY`),
- S3 GET/PUT latency and throttling rate,
- orphan unpublished output backlog.

These are internal observability signals only. Public failure projection remains owned by the public/control surfaces already defined in the canonical error registry.