# GC Worker

Status: Frozen architecture note
Owner: Search Platform Architecture

## 1. Scope

This document defines the execution design of `GC Worker`, the background-compute component that performs physical object reclamation after the canonical owner stack has already made data logically unreachable.

This document is subordinate to the canonical owner stack:
- machine-readable contract: `api/proto/searchfabric/v1/search_fabric.proto`
- internal owner file for this boundary: `api/proto/searchfabric/v1/gc_worker.proto`
- public ingest/query/control semantics: `docs/specs/core/search-fabric-api.md`
- canonical error registry: `docs/specs/core/error-codes.md`
- system topology and state ownership: `docs/specs/core/search-fabric-architecture.md`

It must not redefine:
- public `DropCollection`, `GetCollectionMetadata`, `GetCollectionSchema`, `GetReadiness`, or `GetCollectionStatistics` semantics,
- committed-manifest publication or `manifest_committed_head.json` ownership,
- query token lifetime, continuation legality, or pinned-manifest behavior,
- lifecycle policy semantics or `CollectionSettings.lifecycle_policy`,
- CCS-owned tombstone state and name-reservation release semantics,
- public `SF_*` error mappings.

## 2. Canonical Position in the System

`GC Worker` is a background executor behind the manifest-driven scheduling path:

```text
Background Compute Coordinator / manifest-driven scheduler
  -> GC Worker
       -> Object Store
       -> scheduler-facing `GcTaskSuccess` / `BackgroundTaskStale` / `BackgroundTaskFailure`
  -> Catalog Control Service tombstone/lifecycle workflow
```

The topology in `docs/specs/core/search-fabric-architecture.md` places the component in:

```text
Background Compute Coordinator -> GC Workers -> Object Store (DeleteObjects, throttled/backoff)
```

The worker exists to reclaim physical objects only after upstream owners have already established that those objects are no longer part of reader-visible truth.

The worker is not:
- a public RPC surface,
- a second manifest-visibility authority,
- a query-session registry,
- the owner of collection/namespace lifecycle transitions,
- the owner of collection-name or namespace-name reservation release,
- the owner of WAL replay anchors.

## 3. Design Anchors

The design below is constrained by the following checked-out facts:

1. `GC Workers` are a stateless fleet in the background-compute plane; `Background Compute Coordinator` owns compaction/GC priority computation and isolation budgets.
2. Query-visible truth is the committed-manifest pointer `manifest_committed_head.json`; CCS `manifest_head_version` is derivative governance metadata only.
3. Core architecture defines two-phase GC:
   - logical dereference after manifest change,
   - physical sweep only for safe candidates.
4. Canonical physical deletion requires both:
   - candidate is inactive in latest manifests,
   - `evict_commit_ts < global_safe_ts - retention_window`.
5. Lifecycle-managed data has an extra delete gate:
   - the current committed manifest/window projection must show that the affected window/object set is beyond the admitted `LifecyclePolicy.retention_ms` deadline.
6. Query continuity depends on delayed GC:
   - valid continuations may reopen older pinned manifest versions directly,
   - the configured GC eviction horizon must therefore be greater than or equal to the maximum admitted token lifetime.
7. Collection tombstones remain CCS-owned private control state until tombstone GC completes and CCS records `collection_name_reservation_release_intent`.
8. Namespace tombstone release follows the analogous CCS-owned `namespace_name_reservation_release_intent`, but namespace tombstoning is admitted only after child collections are already hidden.
9. Published manifests persist `wal_replay_start_anchor` and `wal_durable_anchor`; replay uses only those anchored bounds, and unanchored WAL tails are orphan/uncommitted state rather than durable truth.
10. S3-facing delete execution is explicitly rate-controlled:
   - batched `DeleteObjects`,
   - token-bucket limited,
   - exponential backoff on `503 Slow Down`,
   - preserve ingest/query S3 budgets first.

## 4. Responsibilities and Non-Responsibilities

### 4.1 Responsibilities

`GC Worker` is responsible for:

1. Re-checking physical-delete eligibility against current canonical owner state before deleting any object.
2. Reclaiming immutable objects that are no longer referenced by the active committed-manifest frontier.
3. Respecting delayed-GC horizons required by query continuation and search-context token lifetimes.
4. Respecting lifecycle-retention deadlines for lifecycle-managed windows before reclaiming cold or retired objects.
5. Reclaiming WAL or unpublished/orphan artifacts only when canonical replay/session/control owners no longer require them.
6. Executing object-store deletes with S3-aware throttling, batching, retry, and backoff.
7. Reporting object-cleanup evidence back to the owning background-control workflow so BCC/CCS-owned tombstone state can decide any later reservation-release step.

The checked-out owner set now pins GC completion on the same scheduler-facing carrier family used by `BackgroundComputeCoordinatorInternalService.ReportBackgroundTaskCompletion`:
- truthful bounded delete/retry/block/release evidence is returned as `GcTaskSuccess`,
- safe no-delete rebuild outcomes use `BackgroundTaskStale`,
- malformed or internally inconsistent task requests use `BackgroundTaskFailure{kind=BACKGROUND_TASK_FAILURE_KIND_TASK_REJECTED, rejected_reason=...}` rather than an executor-only invalid branch.

### 4.2 Non-Responsibilities

`GC Worker` is not responsible for:
- publishing manifests,
- computing or advancing `safe_ts` / `global_safe_ts`,
- deciding public visibility or durability,
- interpreting query tokens or tracking reader sessions,
- using CCS `manifest_head_version` as an independent correctness source,
- inventing new public tombstone states or user-visible success/error codes,
- releasing collection or namespace reservations without CCS durable control intent,
- mutating schema/control-plane state outside the owning workflow.

## 5. Cleanup Domains

GC work in this architecture falls into four internal domains.

### 5.1 Manifest-Dereferenced Immutable Objects

These are immutable objects that were once referenced by committed manifests and became unreachable after:
- compaction,
- lifecycle tier transition,
- manifest-published object replacement,
- retirement of previously visible segment or sidecar artifacts.

Typical examples include:
- metadata-search segments,
- semantic-search segments and payload sidecars,
- visibility/tombstone bitmaps,
- superseded manifest-version objects that are no longer needed by continuation safety.

### 5.2 Lifecycle-Retention Reclaim

Lifecycle-managed collections may retain retired window/object sets after they stop being query-visible.

For these collections, GC must treat lifecycle state as an additional gate:
- dereference from the latest committed manifest is necessary but not sufficient,
- reclaim is blocked until the admitted `retention_ms` deadline for the affected window has passed,
- CCS lifecycle-window projection may drive scheduling, but manifests remain the correctness authority.

### 5.3 WAL and Orphan Cleanup

Replay ownership is anchor-driven:
- manifests publish `wal_replay_start_anchor` and `wal_durable_anchor`,
- recovery rebuilds only from the anchored replay window,
- WAL chunks not covered by durable anchors are orphan/uncommitted tails and are not replay truth.

The same domain also covers unpublished or abandoned staging artifacts left behind by writers or sessionized import flows, provided they are proven unreachable from:
- the active committed-manifest lineage,
- authoritative replay anchors,
- active admitted session/control state owned elsewhere.

### 5.4 Tombstone-Completion Cleanup

After `DropCollection` makes a lineage tombstoned, physical cleanup still proceeds asynchronously.

That cleanup domain includes work needed before same-name recreation becomes admissible, such as:
- deleting obsolete manifest lineage and data objects belonging only to the dropped lineage,
- deleting stale schema/control artifacts that are no longer needed after the tombstone retention horizon,
- proving that the tombstoned collection's referenced cleanup preconditions have succeeded.

This domain does not directly release the name reservation. Instead, successful cleanup produces object-cleanup evidence that BCC/CCS may use as one input to the CCS-owned `collection_name_reservation_release_intent` workflow.

## 6. State Model and Safety Rules

### 6.1 Durable Truth vs Ephemeral Worker State

`GC Worker` must remain stateless for correctness.

Durable truth lives in:
- committed manifests and `manifest_committed_head.json`,
- admitted lifecycle policy plus committed manifest window metadata,
- CCS tombstone/lifecycle control records and reservation-release intents,
- manifest-published replay anchors (`wal_replay_start_anchor`, `wal_durable_anchor`).

Worker-local queues, caches, leases, or candidate batches are scheduling aids only. They may be rebuilt after crash from the owner state above.

### 6.2 Canonical Delete Predicate

For any candidate object, physical deletion is legal only when every applicable gate passes:

```text
deletable(candidate) =
  not_referenced_by_current_committed_reachability
  AND evict_commit_ts < global_safe_ts - retention_window
  AND lifecycle_deadline_passed_if_applicable
  AND tombstone_release_horizon_passed_if_applicable
  AND replay_or_session_owners_no_longer_require_it
```

Notes:
- `not_referenced_by_current_committed_reachability` must be evaluated from committed-manifest lineage, authoritative anchors, and CCS-owned tombstone/session state as applicable.
- `evict_commit_ts` is the conservative GC horizon carried or recomputed for the candidate; exact derivation is an internal implementation detail, but the canonical inequality above is not.
- lifecycle-managed windows MUST honor `retention_ms` in addition to ordinary GC horizon checks.
- tombstone cleanup MUST honor CCS-owned reservation horizons such as `gc_after_unix_ms` before release-completion work can finish.

### 6.3 What Must Never Be Used as Sole Authority

The worker must never decide deletion from any of the following in isolation:
- CCS `manifest_head_version`,
- wall-clock age alone,
- local cache contents,
- a stale scheduler queue entry,
- object-store presence/absence without re-checking canonical reachability.

In particular, a fixed age gate such as "older than 24 hours" is not the canonical safety rule in the checked-out owner stack.

## 7. Core Workflows

### 7.1 Manifest-Dereference Reclaim

This is the steady-state path after compaction or lifecycle object replacement:

```text
new committed manifest published
  -> old object set becomes logically dereferenced
  -> background scheduler enqueues reclaim candidates
  -> GC worker rechecks latest committed reachability
  -> GC worker waits until delayed-GC horizon passes
  -> GC worker issues batched delete
```

Important rules:
- logical dereference happens when the committed manifest changes, not when physical delete succeeds,
- active query continuations may still reopen older pinned manifests, so non-head manifests and their objects remain protected until the delayed-GC horizon passes,
- the worker must re-check current committed reachability at execution time because newer manifest changes may have changed the candidate set.

### 7.2 Lifecycle-Retention Reclaim

For lifecycle-managed windows:

```text
window object set becomes non-current
  -> lifecycle deadline eventually passes
  -> GC worker verifies latest committed manifests no longer reference retired objects
  -> GC worker deletes only after both retention and ordinary GC safety checks pass
```

`RECLAIMABLE` lifecycle projection in CCS is a scheduling hint, not a direct delete order. Final delete authority still comes from committed manifests plus the canonical GC horizon.

### 7.3 WAL / Orphan Cleanup

For WAL and unpublished/orphan artifacts:

1. Read the latest authoritative replay/session/control owners for the collection.
2. Preserve any chunk/object still needed by `wal_replay_start_anchor` / `wal_durable_anchor` or by another admitted owner workflow.
3. Treat unanchored tails and unreachable staging artifacts as cleanup candidates only after that proof succeeds.
4. Delete them under the same S3 rate-control and retry rules as ordinary object reclaim.

This keeps replay correctness manifest/anchor-driven and avoids treating abandoned object-store data as durable truth.

### 7.4 Tombstone Object Cleanup and Reservation Release

Collection drop cleanup has a distinct handoff:

```text
DropCollection makes lineage TOMBSTONED
  -> CCS keeps name reservation active
  -> background cleanup removes dropped-lineage-only artifacts after gc_after_unix_ms and other preconditions
  -> GC Worker reports object-cleanup evidence only
  -> BCC/CCS combines owner-state facts with accepted cleanup evidence
  -> CCS records collection_name_reservation_release_intent if release is still legal
  -> same-name recreation becomes admissible
```

Key boundary rule:
- GC Worker may supply only object-cleanup facts: `object_cleanup_complete`, `tombstone_horizon_passed`, `replay_anchors_clear`, and `ready_for_release_handoff=false`,
- GC Worker must not decide whether a collection name can be released, whether an allocation epoch still matches, or whether a name can be returned to the allocator,
- BCC/CCS must read collection name, collection-name allocation epoch, drop revision, tombstone state, same-name allocation state, and release ordering from owner state before recording any durable release intent.

Until that handoff occurs:
- duplicate drop requests still resolve to `already_dropped`,
- same-name recreation must still fail with the canonical lifecycle-conflict detail keys,
- collection-scoped reads remain tombstone- or GC-state-dependent according to the public API.

## 8. Delete Execution Pipeline

For one scheduled reclaim batch, a production-quality worker pipeline should be:

```text
scheduled reclaim batch
  -> validate internal task coherence
  -> load current committed-manifest / lifecycle / tombstone / anchor state
  -> recompute safe candidate subset
  -> split into object-store delete batches
  -> apply token-bucket and backoff policy
  -> issue DeleteObjects
  -> record per-key outcome and residual retry set
  -> emit completion or retry state upstream
```

### 8.1 Internal Task Validation

Before deleting, the worker should validate that the task still names:
- one collection lineage or one tombstone lineage,
- one candidate class,
- a reproducible reachability source,
- the conservative horizon metadata needed for the canonical delete inequality.

Stale or incomplete tasks should be rejected back to the scheduler rather than executed optimistically.

### 8.2 Execution-Time Revalidation

Revalidation must reopen authoritative owner state at execution time because:
- manifest head may have advanced,
- lifecycle projection may have changed,
- tombstone cleanup preconditions may still be incomplete,
- replay anchors may have moved.

This revalidation step is required even when the scheduler already computed a candidate set earlier.

### 8.3 Batched Delete and Partial Failure

Delete execution should:
- batch object keys into backend-appropriate `DeleteObjects` requests,
- treat object-store throttling as a retryable operational condition,
- preserve a residual retry set when only a subset of keys failed,
- keep deletion idempotent so safe retries do not require expensive compensation logic.

### 8.4 Completion Signaling

Completion output is internal workflow state, not a public API contract.

Typical completion signals include:
- reclaimed object count/bytes,
- keys still blocked by delayed-GC horizon,
- residual retry set,
- tombstone object-cleanup evidence. This evidence never grants collection-name release by itself; release handoff remains a BCC/CCS owner-state decision.

## 9. Failure Handling and Recovery

### 9.1 Crash Recovery

Because the worker is stateless for correctness, restart recovery must be pointer-driven:
- rebuild pending work from committed manifests, lifecycle projections, tombstone records, and scheduler-owned task state,
- never trust local caches or partially executed delete batches as durable truth,
- re-run execution-time reachability checks before retrying deletion.

### 9.2 Throttling and Dependency Outage

The architecture already fixes the S3-protection response:
- token-bucket limit delete rate,
- exponential backoff on `503 Slow Down`,
- preserve ingest/query S3 budgets first.

This is internal runtime behavior. The worker does not introduce a public `SF_GC_HALT` or similar user-visible status family.

### 9.3 Correctness Under Stalled Cleanup

Stalled cleanup must not change visibility truth:
- old objects remaining in S3 after logical dereference are waste, not correctness drift,
- delayed manifest/object retention is expected while valid continuations still exist,
- tombstone backlogs intentionally keep same-name recreation blocked until CCS records release.

If tombstone cleanup is delayed, public behavior remains the existing CCS-owned split:
- duplicate drop -> `already_dropped`,
- same-name recreate before release -> `SF_VERSION_CONFLICT`,
- fully released lineage after completion -> `SF_RESOURCE_NOT_FOUND` on old collection-scoped reads.

## 10. Operational Constraints

### 10.1 Multi-Tenant Isolation

GC must remain subordinate to `Background Compute Coordinator` scheduling and must not starve online planes.

Implementations should therefore assume:
- per-tenant or per-pool delete budgets,
- bounded concurrent delete fanout,
- cooperative cancellation and retry on budget exhaustion,
- scheduler ownership of priority across compaction, lifecycle transition, and GC debt.

### 10.2 Observability Guidance

Useful production metrics include:
- pending reclaim bytes and object counts by cleanup domain,
- age of oldest logically dereferenced but not yet physically deleted object set,
- age of oldest tombstone waiting for reservation release,
- delete request rate, batch sizes, and per-batch partial-failure counts,
- object-store `503 Slow Down` frequency and backoff duration,
- count of candidates deferred by delayed-GC horizon,
- count of candidates deferred by lifecycle retention deadline.