# Aggressive Bootstrap Incident Report

## Summary

Staging experienced repeated table `DROP` and `CREATE` waves during deployment retries. The affected objects included business tables such as `Account`, `Lead`, `Contact`, and others. The behavior was caused by runtime bootstrap failure handling that uses destructive rollback semantics.

## Impact

- Repeated destructive DDL churn during unstable deploy windows.
- Elevated risk of metadata and permission cleanup side effects because rollback uses `DropTable`, not only physical table drop.
- Startup instability amplification because orchestration retries re-enter bootstrap on each task restart.

## Verified Findings

1. Runtime startup executes bootstrap schema initialization by default:
   - `backend/cmd/server/main.go` calls `bootstrap.InitializeSchema(db)` unless `SKIP_BOOTSTRAP=true`.
2. Batch table creation includes destructive compensation:
   - `backend/internal/infrastructure/persistence/schema_ddl_ops.go`
   - `BatchCreatePhysicalTables` calls `DropTable` on `newlyCreatedTables` after DDL or registration failure.
3. Strict metadata create path has a separate destructive cleanup:
   - `CreateTableWithStrictMetadata` defers `DROP TABLE IF EXISTS` on returned error.
4. The `newlyCreatedTables` guardrail is based on a point-in-time snapshot and does not provide a hard safety guarantee under retries/concurrency.
5. Deployment restart loops amplify the issue:
   - bootstrap fails -> process exits -> task restarts -> bootstrap re-runs -> compensation can re-trigger.

## Root Cause

The runtime path combines:

- mutating bootstrap DDL,
- destructive rollback,
- process restarts under orchestration.

This creates a system where a transient/bootstrap error can repeatedly trigger destructive DDL behavior.

## Why Existing Guardrails Were Insufficient

- Guardrail logic depends on pre-snapshot table existence.
- Snapshot assumptions are weaker than hard ownership proofs.
- Failure handling is destructive by default instead of fail-closed/preserve-data.

## Corrective Direction

1. Remove destructive rollback behavior from runtime bootstrap failure paths.
2. Convert bootstrap to single-writer checkpointed mutation with resumable phases.
3. Decouple normal app startup from broad mutating bootstrap.
4. Add regression coverage for failure/retry/concurrency behavior.

## Confidence

- High confidence on destructive code paths and restart amplification.
- High confidence that removing runtime destructive rollback is required.
- Medium-to-high confidence on full operational redesign details (to be validated in staging with restart simulations).