# Background Compute Coordinator

Status: Frozen architecture note
Owner: Search Platform Architecture

## 1. Scope

This document defines the stateful background-compute control owner of the checked-out architecture.
`Background Compute Coordinator` is the canonical service name in the current design set.
Within that service, the manifest-observation path is the internal scheduling flow that turns committed-manifest and lifecycle changes into background work.

This document is subordinate to the canonical owner stack:
- machine-readable contract: `api/proto/searchfabric/v1/search_fabric.proto`
- internal owner file for this boundary: `api/proto/searchfabric/v1/background_compute_coordinator.proto`
- public ingest/query/control semantics: `docs/specs/core/search-fabric-api.md`
- canonical error registry: `docs/specs/core/error-codes.md`
- system topology and state ownership: `docs/specs/core/search-fabric-architecture.md`

It must not redefine:
- public query consistency, `safe_ts`, `effective_read_ts`, `GetReadiness`, or query waiting semantics,
- continuation/search-context token legality or delayed-GC lifetime rules owned by the query plane,
- committed-manifest publication or `manifest_committed_head.json` ownership,
- CCS-owned lifecycle, tombstone, and reservation-release semantics,
- public `SF_*` error mappings,
- source-facing ingest, snapshot-import, or schema-control RPC behavior.

## 2. Canonical Position in the System

The checked-out topology places this component only on the background-compute path:

```text
Committed manifest / lifecycle inputs
  -> Background Compute Coordinator
       -> Compaction Workers -> Object Store -> scheduler-facing `ManifestRewriteSuccess`
       -> GC Workers -> Object Store
       -> CCS tombstone / reservation-release workflow
```

The service-responsibility and governance documents fix the owner split around that path:

- `Background Compute Coordinator` is a high-availability scheduling owner with rebuild-only recovery authority in the current revision.
- `Compaction Workers` and `GC Workers` are stateless executors.
- `Manifest Publisher` remains the only owner of committed-manifest version progression and committed-pointer CAS.
- `Catalog Control Service` remains the only owner of lifecycle/tombstone control state and release intents.
- `Query Gateway` and `Query Broker` remain the only owners of query readiness, watermark interpretation, and continuation semantics.
- Subordinate compaction / GC executor RPCs reuse the same completion carrier family later accepted by `ReportBackgroundTaskCompletion`, so the lease owner does not translate executor-only result messages into a second scheduler vocabulary.

This component is therefore not a bridge between `Manifest Publisher` and readers.
It is a scheduler and fence owner for background work that must stay subordinate to committed-manifest truth.

## 3. Design Anchors

The design below is constrained by the following checked-out facts.

1. `Background Compute Coordinator` subscribes to manifest evolution, computes compaction/GC priorities, schedules background jobs, schedules lifecycle tier-transition and retention work, and enforces isolation budgets.
2. The topology explicitly places the manifest-driven coordinator in front of `Compaction Workers` and `GC Workers`, not on the query hot path.
3. `Manifest Publisher` owns immutable manifest publication and the atomic committed-pointer CAS on `manifest_committed_head.json`.
4. `Catalog Control Service` advances `manifest_head_version` only as derivative control metadata after committed-pointer CAS succeeds; CCS head is not the read-visibility boundary.
5. Query first-page execution resolves the committed manifest through `manifest_committed_head.json`, and continuation reopens the exact pinned manifest version directly from token state.
6. Delayed GC is required for valid continuation: pinned manifests and referenced objects must survive for at least the maximum admitted token lifetime.
7. Physical deletion safety is defined by the core GC rule, not by scheduler-local heuristics:
   - candidate is inactive in the latest committed manifests,
   - `evict_commit_ts < global_safe_ts - retention_window`,
   - lifecycle-managed data additionally respects `LifecyclePolicy.retention_ms`.
8. `Catalog Control Service` owns lifecycle-policy materialization into hot projections for transition scheduling and retention tracking, but those projections remain derivative state; manifests remain the query-time truth.
9. Collection and namespace same-name release remain CCS-owned control intents:
   - `collection_name_reservation_release_intent`
   - `namespace_name_reservation_release_intent`
10. Recovery authority for `Background Compute Coordinator` is rebuild-only in the current revision: authoritative restart inputs are the active committed-manifest lineage plus CCS lifecycle/tombstone scheduling state; coordinator-owned task/fence records, queues, and leases are explicitly disposable derivatives rather than durable authority.
11. Stale-head protection is explicit in the core architecture: background tasks must verify that the target manifest version exists and is active before compaction or GC actions.
12. The public proto and public API define no external `Background Compute Coordinator` RPC surface. Concrete task transport remains internal-only.

## 4. Responsibilities and Non-Responsibilities

### 4.1 Responsibilities

`Background Compute Coordinator` is responsible for:

1. Observing committed-manifest evolution for active collections and converting that evolution into background-work opportunities.
2. Joining committed-manifest state with CCS-owned lifecycle/tombstone scheduling inputs to derive background candidates.
3. Computing priority, pacing, and fairness across compaction, lifecycle transition, retention reclaim, and tombstone-cleanup debt.
4. Assigning leased, fenced background tasks to `Compaction Workers` and `GC Workers`.
5. Revalidating tasks against current committed-manifest lineage before destructive or visibility-affecting completion paths proceed.
6. Keeping background work collection-scoped so one collection's backlog does not require global ordering state.
7. Rebuilding schedulable background state after failover from committed-manifest lineage and CCS lifecycle/tombstone scheduling inputs rather than from worker-local memory or durable coordinator-owned task records.
8. Converting worker completion into the next owner handoff:
   - compaction/lifecycle replacement -> `Manifest Publisher`
   - tombstone cleanup completion -> CCS release-intent workflow

### 4.2 Non-Responsibilities

`Background Compute Coordinator` is not responsible for:

- serving `GetReadiness` or any query-facing readiness API,
- computing or publishing `safe_ts`,
- collecting reader ACKs or maintaining reader subscription registries,
- interpreting continuation/search-context tokens,
- publishing manifests or CAS-updating `manifest_committed_head.json`,
- advancing CCS `manifest_head_version`,
- releasing collection or namespace name reservations directly,
- defining public throttling or quota policy on the query surface,
- relying on broad object-store listing as a correctness source.

## 5. Input Boundaries and State Model

### 5.1 Canonical Inputs

The component consumes four input families.

1. Committed-manifest lineage:
   - `manifest_committed_head.json`
   - immutable committed manifests referenced by that pointer
2. CCS-owned private control state:
   - lifecycle-policy hot projections
   - tombstone state
   - reservation horizons and cleanup dependencies
3. Background-governance policy:
   - fairness envelopes
   - concurrency budgets
   - compaction / GC pacing policy
4. Worker execution signals:
- lease heartbeats or timeout
- success/failure completion
- completion metadata needed for the next owner handoff

For manifest-rewrite success, completion metadata includes the exact
`ManifestPublishedArtifact` replacement coverage. Identity-only object refs are
not sufficient for deterministic multi-output cutover.

Only the first two families can change correctness.
Governance policy and worker signaling change scheduling, not truth.

### 5.2 Durable Truth vs Disposable Derivatives

Durable truth lives in:

- committed manifests and `manifest_committed_head.json`,
- CCS-owned lifecycle/tombstone state and release-intent records.

Disposable derivative state includes:

- priority queues,
- in-memory debt scores,
- cache of previously observed committed heads,
- short-lived leases,
- worker routing tables,
- coordinator-owned task/fence records and retry queues.

Losing derivative state must not require operator repair.
The coordinator must be able to rebuild its schedulable view solely from committed-manifest lineage and CCS control state.
The current revision therefore does not treat coordinator-owned task/fence records as part of the authoritative recovery set.

### 5.3 Suggested Internal Task Fence Shape

The public contract does not define this shape, but the checked-out owner stack requires every background task to stay tied to one collection and one committed-manifest assumption.

An implementation-friendly internal lease should therefore carry at least:

```text
BackgroundTaskLease {
  task_id
  task_kind
  collection_ref
  base_manifest_version
  base_manifest_checksum_or_etag
  lease_epoch
  lease_expiry_unix_ms
}
```

The important contract point is not the exact field names.
It is that destructive or cutover-capable work must remain version-fenced against a known committed lineage.

## 6. Task Domains

The coordinator has four distinct work domains.

### 6.1 Manifest Observation and Reconciliation

Normal scheduling begins when the committed manifest advances or when CCS updates lifecycle/tombstone scheduling facts.

For each affected collection, the coordinator:

1. loads the current committed head,
2. reopens the committed manifest named by that head,
3. compares the new state with the previously observed schedulable state,
4. updates the collection's background debt model,
5. enqueues or reprioritizes work.

The implementation may use event delivery, shard-local polling of committed-head objects, or periodic reconciliation.
What it must not do is treat a blind prefix scan of object storage as the authoritative way to discover live state.

### 6.2 Compaction Scheduling

Compaction work exists to replace one committed object layout with a better one.
The scheduler may derive compaction debt from committed-manifest facts such as:

- excessive object fan-out,
- obsolete-object density,
- tier-local fragmentation,
- repeated read-amplification signals from previous background analysis.

This document intentionally does not freeze the compaction scoring formula.
It only fixes the ownership rule:

- the coordinator may decide that a rewrite is worth attempting,
- the worker may build replacement immutable objects,
- only `Manifest Publisher` may make the replacement query-visible.

### 6.3 Lifecycle Transition and Retention Scheduling

For collections that admit `CollectionSettings.lifecycle_policy`, CCS materializes due-window projections for transition and retention scheduling.
The coordinator consumes those projections as scheduling input, then binds execution back to manifest truth:

- a new `DataTier` becomes query-visible only after the affected committed manifest advances,
- retention reclaim may be scheduled when a window is due,
- physical deletion still waits for the canonical GC gates and does not happen solely because a CCS projection says "due".

This keeps CCS hot projections useful for scheduling without turning them into a second visibility source.

### 6.4 GC and Tombstone-Cleanup Scheduling

GC candidates arise from three common situations:

1. a newer committed manifest logically dereferences older objects,
2. lifecycle retention deadlines make retired windows eligible for reclaim,
3. a tombstoned lineage accumulates cleanup debt that blocks same-name release.

The coordinator may queue those candidates, but the queue entry is only a hint.
`GC Worker` must still re-check the latest committed reachability and all applicable safety horizons before deletion.

For tombstoned lineages, the coordinator also tracks cleanup completion as a dependency graph:

- manifest/object cleanup,
- stale schema/control artifact cleanup,
- reservation-horizon passage,
- any other CCS-declared cleanup precondition.

When those dependencies are satisfied, the coordinator hands completion to the CCS-owned release-intent workflow.
It does not release the reservation itself.

## 7. Scheduling, Leasing, and Stale-Head Defense

### 7.1 Collection-Scoped Serialization

The checked-out architecture does not publish a mandatory sharding algorithm for this component.
The safe consequence from the owner stack is still clear:

- one collection's background tasks should be coordinated by one active coordinator shard at a time,
- leases should be version-fenced to one committed-manifest baseline,
- cross-collection ordering is unnecessary for correctness and should not be introduced accidentally.

This keeps the scheduler compatible with the manifest-publisher rule that manifest progression is serialized per collection.

### 7.2 Lease Rules

Every leased task should follow five rules:

1. Lease against a specific committed-manifest baseline.
2. Treat worker timeout or leader failover as lease loss, not as proof of task failure or success.
3. Revalidate the baseline before destructive completion or visibility-affecting handoff.
4. Make duplicate completion harmless through existing owner-side idempotency:
   - manifest cutover is serialized by `Manifest Publisher` CAS,
   - CCS release is serialized by deterministic internal intent identity,
   - object deletion remains rechecked by `GC Worker`.
5. After coordinator failover, rebuild and reissue work from current durable owner state instead of attempting lease recovery from coordinator-owned task records.

### 7.3 Stale-Head Handling

The core architecture explicitly requires stale-head defense.
The coordinator must therefore treat any task as stale when one of the following becomes true:

- its `base_manifest_version` is no longer the active committed version,
- the pinned manifest object cannot be reopened with the pinned checksum or etag,
- lifecycle/tombstone assumptions carried into the task are no longer compatible with current CCS state.

Required response:

- do not silently rebase the task onto the new head if that would change semantics,
- drop or quarantine the stale result,
- rebuild a fresh task from the current committed-manifest lineage.

That rule applies especially to compaction-style replacement tasks.

## 8. Core Workflows

### 8.1 Compaction or Lifecycle Rewrite

```text
committed manifest vN observed
  -> coordinator computes rewrite candidate pinned to vN
  -> worker lease issued
  -> worker writes replacement immutable objects
  -> worker reports rewrite completion facts, including replacement artifact coverage, to coordinator
  -> coordinator submits accepted cutover candidate to Manifest Publisher
  -> Manifest Publisher CAS-advances to vN+1
  -> coordinator observes vN+1 and marks old objects as logically dereferenced
```

Important consequences:

- writing replacement objects is not visibility,
- scheduler completion is not visibility,
- worker completion must carry full replacement coverage for deterministic multi-output cutover,
- only committed-manifest CAS is visibility.

If the publisher rejects the candidate because the head moved or assumptions drifted, the coordinator rebuilds from the new committed head rather than pretending the old result is still correct.

### 8.2 GC After Logical Dereference

```text
manifest advances or retention deadline becomes due
  -> coordinator enqueues GC candidate
  -> GC worker reopens latest committed lineage
  -> GC worker checks canonical delete predicate
  -> GC worker executes throttled batched delete
  -> worker reports completion
```

The scheduler's queue entry is not deletion authority.
Deletion authority remains the canonical GC rule enforced at execution time.

### 8.3 Tombstone Completion and Name Release

```text
DropCollection tombstones lineage
  -> coordinator tracks cleanup dependencies
  -> GC / cleanup work removes lineage-specific artifacts
  -> reservation horizon passes and dependencies clear
  -> coordinator calls CCS internal release-intent handoff
  -> CCS records collection_name_reservation_release_intent
```

The same pattern applies to namespace release, except namespace tombstoning is admitted only after child collections are already hidden.

## 9. Failure Handling and Recovery

### 9.1 Coordinator Restart

On restart, the coordinator must rebuild from durable owner state:

- current committed-manifest lineage,
- CCS lifecycle/tombstone scheduling state.

It must not recover by trusting:

- worker-local in-memory queues,
- prior coordinator-owned task/fence records as authoritative recovery input,
- stale CCS `manifest_head_version` alone,
- prefix listing of data objects.

### 9.2 Worker Failure or Duplicate Completion

Worker failure leaves background debt outstanding; it does not change visibility truth.
Duplicate completion must be safe because:

- manifest cutover is still guarded by publisher CAS,
- deletion is still guarded by GC revalidation,
- CCS release remains an idempotent internal-intent path.

### 9.3 Partial Dependencies

If one dependency source is unavailable:

- loss of committed-manifest access blocks any task that needs current lineage validation,
- loss of CCS lifecycle/tombstone state blocks tasks that depend on those control facts,
- policy loss may stop new scheduling, but it must not allow unsafe work to continue on stale assumptions.

Fail closed is preferred to speculative background mutation.

## 10. Observability

Useful coordinator-local signals include:

- queue depth by task class (`compaction`, `lifecycle_transition`, `retention_reclaim`, `tombstone_cleanup`),
- age of the oldest due item in each class,
- stale-head task invalidation rate,
- lease timeout and reassignment rate,
- publish-conflict / rebuild rate for rewrite tasks,
- GC deferral count due to delayed-GC or lifecycle-retention gates,
- age of oldest tombstone waiting for release-intent eligibility,
- background budget saturation and throttling pressure,
- lag between latest committed head and latest scheduler-observed head.

These are internal observability signals.
They do not create a new public readiness or status API.

## 11. Rejected Ownership Expansions

The checked-out owner stack explicitly rejects the following designs for this component:

- a query-facing `Readiness Oracle`,
- reader subscription tables or reader ACK-based visibility computation,
- a second `safe_ts` source derived from reader state,
- a metadata-notification bus used as query truth,
- namespace-level WRR admission for public query traffic,
- broad manifest-log or data-path prefix scans as the primary correctness mechanism,
- direct release of collection or namespace names outside CCS intents.

Those ideas either duplicate existing owners or weaken the manifest-first correctness model.