# Issue 961 ChildCloneRules Ownership Architecture Status: architectural review for PR #965. ## Relationship To Issue 979 Issue #979 is the design authority for copy, dynamic inherit, and child fan-out on create. Under that decision, new create-from-source behavior is authored in Flow: `_System_ObjectFieldMapping` for flat snapshot copy, Flow expressions and formula metadata for dynamic inherit, record lookup / loop / recordCreate for child fan-out, and standard-action transaction semantics for required atomic behavior. `ChildCloneRules` is not the primary product authoring model for new create-from-source behavior. It is retained here as a compatibility/install primitive for existing or transitional child-clone metadata that already reaches `_System_Object.ChildCloneRules`. This PR's job is therefore ownership safety: make the retained executable fact attributable, verifiable, repairable, and L2-override-aware without blessing it as the recommended new authoring surface. ## Problem `ChildCloneRules` is executable object metadata stored on `_System_Object.ChildCloneRules`. The runtime uses it during record insert to clone child rows from a source record to the newly inserted parent. That makes the column closer to `DefaultChildRows` or `LockRule` than to display metadata: an incorrect owner can change write-time behavior inside a transaction. PR #965 started as a persistence fix: `ObjectSpec.childCloneRules` existed in metadata but was not written to `_System_Object`. The PR then grew into an ownership cutover because the same column was also writable from tenant overlay metadata. Review feedback has converged on one architectural issue: There must be one authoritative ownership model for the column. Pack apply, tenant apply, verify, and upgrade repair cannot each infer ownership locally. ## How The System Works There are four moving pieces: 1. Source-backed app packs provide L1 metadata through `object_spec` components. Each component has manageability, usually `locked` or `defaults_only`. 2. Tenant overlay / admin metadata is L2. Under `defaults_only`, tenants may own allowed override surfaces. Under `locked`, the pack owns the surface. 3. `_System_Object.ChildCloneRules` is the live runtime value read by insert execution. 4. Pack verify checks whether the live database still matches the installed source-backed pack, except where a valid L2 override owns that exact fact. Historically, tenant overlay wrote `ChildCloneRules` directly. The PR adds pack payload as another writer, plus these ownership sidecars: - `ChildCloneRulesPackName` - `ChildCloneRulesPackBusinessKey` - `ChildCloneRulesPackManageability` - `ChildCloneRulesPackValue` - `ChildCloneRulesCustomized` That is the right compatibility shape for a retained executable fact. The remaining work is to make every path use those columns with the same state machine and in the same order. ## Review History Synthesis The blockers have not been random. They have been the same design gap showing up at different boundaries: - Two write authorities: pack `object_spec` and tenant overlay both wrote the live column without a shared ownership decision. - Coarse ownership: row-level `IsCustomized` was too broad for an executable column; label edits must not transfer `ChildCloneRules`. - Missing L2 path: removing tenant overlay without an explicit replacement made legitimate defaults-only customizations impossible. - Missing preflight: invalid `childCloneRules` payloads could fail after pack mutation instead of before. - Tenant apply coupling: L2 apply tried to resolve active source-backed pack transports instead of reading persisted ownership. - Custom-object bypass: standard-object and custom-object override paths did not enforce the same pack manageability contract. - Reapply/backfill order: ownership repair sometimes ran after strict verify, or a legacy classifier ran during steady-state apply. - Empty pack value ambiguity: an owned object whose pack omitted `childCloneRules` stored a `NULL` pack sidecar, which was indistinguishable from "ownership not hydrated" to later classifiers. My conclusion: continue with the column-scoped ownership model, but stop adding local classifiers. Make `ChildCloneRules` use a single ownership oracle and a two-phase repair/apply sequence. ## Target Contract For every source-backed `object_spec` row, pack ownership of `ChildCloneRules` is explicit even when the pack omits the property. Canonical pack value: - Authored rules: canonical JSON array of rules. - Omitted or empty rules: canonical JSON `[]`. - Never use `NULL` in `ChildCloneRulesPackValue` to mean "the pack owns an empty value". `NULL` should mean "ownership is not hydrated / unknown". Runtime live value: - `_System_Object.ChildCloneRules` may remain `NULL` or `[]` for no rules if the read path treats them equivalently. - Every comparison must canonicalize both `NULL` and empty string to `[]`. Ownership decisions: - `locked` + any live mismatch is pack drift. Tenant apply must reject an override before writing it. - `defaults_only` + `ChildCloneRulesCustomized=0` means pack-owned. Pack apply may replace the live value with the current pack value. - `defaults_only` + `ChildCloneRulesCustomized=1` means L2-owned. Pack apply preserves live value; verify skips exact comparison for this column. - `ChildCloneRulesCustomized=1` under `locked` is invalid state, not an override. Verify should fail and repair should require an explicit restore. - Missing pack ownership for an existing source-backed object is unresolved state. Tenant apply should fail closed with a repair instruction instead of guessing from `IsCustom`. This ownership decision should live in one helper and be called by pack apply, pack verify, tenant apply, cleanup, and backfill. ## Preferred Fix For Current Blockers Blocker 1: `ApplyAll` verifies active source-backed packs before `ChildCloneRules` ownership repair runs. Preferred solution: Move `ChildCloneRules` ownership hydration and classification before any strict source-backed pack verify that consumes the column. The order should be: 1. Ensure schema columns exist. 2. Adopt source-backed identities for legacy installed app packs. 3. Resolve active source-backed packs in the control-plane repair/reconcile phase. 4. Hydrate `ChildCloneRulesPack*` sidecars for every active `object_spec`. 5. Classify legacy live values against the hydrated canonical pack value. 6. Only then run strict active-pack verify / reconcile. An acceptable alternative is a first-pass verify mode that suppresses only `ChildCloneRules` drift until sidecars are hydrated, followed by strict verify after hydration. I prefer moving hydration earlier because it keeps strict verify simple and avoids a special "half verified" mode. Blocker 2: rows whose pack omits `childCloneRules` get an empty/`NULL` `ChildCloneRulesPackValue`, so live L2 rules on those objects are never marked customized. Preferred solution: Persist canonical `[]` into `ChildCloneRulesPackValue` for every source-backed object spec that owns the surface but omits `childCloneRules`. Then classify: - live canonical `[]` vs pack canonical `[]` -> pack-owned, `ChildCloneRulesCustomized=0`. - live non-empty vs pack canonical `[]` under `defaults_only` -> L2-owned, `ChildCloneRulesCustomized=1`. - live non-empty vs pack canonical `[]` under `locked` -> locked drift, fail closed or restore through an explicit repair path. If the implementation keeps `NULL` for compatibility in the short term, the classifier must still treat `PackName/BusinessKey present + PackValue NULL` as canonical pack value `[]`. But that is a weaker shape; it leaves future code one `NULL` check away from repeating this bug. ## Apply And Verify Flow Pack apply for an `object_spec` component: 1. Preflight validate `childCloneRules` against the object/field graph before any DB mutation. 2. Snapshot targeted sidecars before the source-backed install plan begins so a failed install can restore the previous ownership tuple. 3. During object-spec apply, validate existing sidecar ownership and classify unresolved legacy `defaults_only` rows before live object metadata is upserted. This step must not advance the sidecar restore source yet. 4. Upsert object metadata: - if ownership says exact required, write live `ChildCloneRules` from pack; - if ownership says live override allowed, preserve live value and `ChildCloneRulesCustomized=1`. 5. Persist pack ownership sidecars for the targeted specs, including canonical `ChildCloneRulesPackValue`. 6. If any later install/finalize phase fails before the pack reaches active, restore the sidecar snapshot so a failed pack version cannot become the tenant-override release value. 7. Verify using the same ownership helper. Pack verify: 1. Load live value, pack sidecar, manageability, and customized bit. 2. Canonicalize spec, sidecar, and live values. 3. Fail if sidecar identity disagrees with the component being verified. 4. Fail if sidecar value or manageability disagrees with the current component, even when the live value is a valid L2 override. 5. Apply ownership decision: - exact required: live canonical must equal spec canonical; - live override allowed: skip exact comparison for this column; - unresolved: fail with repair instruction. Tenant apply: 1. Read persisted ownership only for targeted objects. 2. Do not resolve source-backed pack transports during ordinary tenant apply. 3. For each requested override: - no pack owner and truly tenant/custom object: allow; - no pack owner on an existing source-backed object: fail unresolved; - locked pack owner: reject; - defaults-only pack owner: write live value and set `ChildCloneRulesCustomized=1`. 4. When an override is removed, restore live value from the persisted pack sidecar and set `ChildCloneRulesCustomized=0`. This keeps L2 apply independent from L1 source transport and gives admins a deterministic error when ownership has not been repaired yet. ## Backfill Strategy Even though there have been no deployments yet, the repository already has a substantial compatibility/backfill framework. `RunRepair` and bootstrap code include migrations and repair passes for scheduled run reservations, workflow-alert template IDs, navigation visibility, permission SSoT rows, setup object permissions, polymorphic discriminators, field capabilities, lookup references, trigger subscriptions, flow source, action input guard keys, and related-list payloads. Adding a focused `ChildCloneRules` backfill is consistent with the existing system. The backfill should be explicit and closure-checked: 1. Add nullable sidecar columns and `ChildCloneRulesCustomized` with safe defaults. 2. Hydrate sidecars from active installed source-backed packs. 3. Use canonical pack value `[]` for omitted rules. 4. Classify live rows: - live equals pack -> customized `0`; - live differs and manageability `defaults_only` -> customized `1`; - live differs and manageability `locked` -> fail closed or restore pack value only under an explicit repair command. 5. Check closure before stamping migration complete: - no source-backed row with live rules and missing ownership; - no `locked` row with `ChildCloneRulesCustomized=1`; - no defaults-only override without sidecar identity. Because no production deployments exist yet, this can be implemented as a normal repair/backfill without a long-running compatibility window. But the code should still be idempotent, because tests and local developer databases will exercise warm-upgrade states. ## Tests Required Add tests at the ownership helper boundary and at each caller boundary: - Canonical sidecar: source-backed object specs that omit `childCloneRules` persist `ChildCloneRulesPackValue=[]`. - Empty-pack classification: pack `[]` + live non-empty + defaults-only marks `ChildCloneRulesCustomized=1`. - ApplyAll ordering: active-pack verify cannot run strict `ChildCloneRules` comparison before sidecar hydration/classification. - Steady-state pack update: defaults-only pack value changes land when `ChildCloneRulesCustomized=0`. - L2 override preservation: defaults-only tenant override survives pack reapply and verify. - Override removal: deleting the L2 override restores pack value and clears the customized bit. - Locked standard and custom objects: tenant overlay rejects before writing. - Unresolved ownership: tenant apply fails with a targeted repair message and does not resolve pack source. - Verify oracle parity: pack verify, tenant apply, and pack apply agree on the same row state. ## Implementation Shape Introduce a small ownership module, likely under `backend/internal/controlplane/packs`, that owns: - canonical encoding / comparison; - sidecar persistence; - row classification; - ownership decision; - restore-from-pack behavior for override removal; - verify decision. Existing callers should become thin: - `source_pack_ops.go` snapshots sidecars before source-backed install and restores them if the install fails. - `source_backed_pack.go` classifies unresolved rows, applies object specs, persists sidecars, and verifies. - `pack_schema_verify.go` delegates `ChildCloneRules` drift decisions. - `tenantoverlay/insert_rules_loader.go` delegates override allow/reject and restore behavior. - `bootstrap/child_clone_rules_backfill.go` delegates classification. - `operations.go` runs hydration/classification before strict active-pack verify. The important design rule is that no caller should re-derive ownership from `IsCustom`, row-level `IsCustomized`, source availability, or whether JSON is `NULL`. Those are implementation facts, not ownership facts. ## Review Verdict The PR is converging on the right architecture only if it finishes the column-scoped ownership cutover. The current blockers show that the design is still split between local repair guesses and the intended sidecar model. My preferred path is: 1. Canonicalize pack-owned empty values as `[]`. 2. Hydrate and classify ownership before strict active-pack verify. 3. Snapshot targeted ownership sidecars before source-backed install, then classify only unresolved legacy rows before the live `ChildCloneRules` upsert can run. 4. Make tenant apply use persisted ownership only. 5. Route apply, verify, tenant override, cleanup, and backfill through one helper. That is the robust path. It is a little more work than another targeted classifier patch, but it removes the class of bugs reviewers keep finding. ## Implemented Follow-Up The first blocker fix following this review implemented these repair edges: - source-backed object specs now persist canonical `ChildCloneRulesPackValue` for every owned object, including `[]` when the pack omits rules; - legacy classification treats `PackName/BusinessKey present + PackValue NULL/empty` as canonical `[]`, but only stamps `ChildCloneRulesCustomized=1` for `defaults_only` ownership; locked mismatches remain strict pack drift; - active installed-pack reconciliation now hydrates/classifies `ChildCloneRules` ownership before strict source-backed pack verify. The second blocker fix closes the source-backed apply race that remained: - `sourceBackedObjectSpecComponent.Apply` now calls `prepareObjectSpecChildCloneRulePackOwnershipForApply` before `applyObjectSpecsWithResolver`; - that preparation snapshots existing target object rows before classifying unresolved legacy ownership; - rows already owned by another pack/component fail closed; - rows with no prior pack sidecar under `defaults_only` are classified as L2-owned when their live canonical `ChildCloneRules` differs from the current canonical pack value; - rows already owned by the same pack/component are not reclassified when the pack value changes, so legitimate pack updates still land when `ChildCloneRulesCustomized=0`; - `locked` rows are never converted into L2 ownership by this repair path. The third blocker fix makes the sidecar contract atomic and verifiable: - source-backed install snapshots all targeted `ChildCloneRulesPack*` sidecars before marking the pack installing; - if the install plan, cache refresh, insert-rule validation, final verify, or derived-artifact materialization fails, the snapshot is restored so failed pack values cannot become tenant override restore state; - pack verify now checks `ChildCloneRulesPackName`, `ChildCloneRulesPackBusinessKey`, `ChildCloneRulesPackManageability`, and canonical `ChildCloneRulesPackValue` against the current object-spec component before applying the live-value ownership decision; - L2-owned live overrides remain allowed only after the sidecar tuple itself is proven current. The fourth blocker fix closes tenant custom-object gaps on the same contract: - tenant `tenant_objects_spec.json` child-clone preflight now uses the merged overlay + live metadata snapshot, matching the standard-object `object_child_clone_rules.json` path, so partial standard-object overlays do not hide existing base fields from valid custom-object rules; - custom-object ownership lookup treats missing `ChildCloneRulesPack*` as unresolved when the target row has pack-owned fields, even if the live `ChildCloneRules` column is empty. That keeps upgraded locked pack-owned custom objects from being seized by tenant overlay before sidecar backfill. The fifth blocker fix closes stale standard-object overlay deletion: - stale-overlay cleanup now clears unresolved legacy standard-object live `ChildCloneRules` rows when the tenant file entry disappears and the row has no `ChildCloneRulesCustomized` bit and no pack sidecar identity; - the cleanup path still preserves current tenant payload keys and sidecar-owned pack restore values, so it removes orphaned executable metadata without making `apply-tenant` resolve active pack source transports. The sixth blocker fix closes source-backed multi-component apply validation: - install preflight already validates `ChildCloneRules` against the whole source-backed pack, but apply-time validation was still resolving references against only the current `object_spec` component plus live metadata; - object-spec apply now validates the current component's rules against a pack-wide object metadata snapshot, so a rule in one component may reference source/target objects declared by another component in the same pack without failing after install mutation has started; - the rule subject and the lookup snapshot are separate inputs, which avoids widening the component's mutation scope while still giving validation the same reference graph as preflight. The seventh blocker fix closes tenant custom-object omission writes: - `tenant_objects_spec.json` may omit `childCloneRules` while editing other custom-object metadata, and omission must mean "do not author this retained compatibility executable fact" rather than "write NULL"; - tenant custom-object metadata save now skips `ChildCloneRules` and `ChildCloneRulesCustomized` in the generic object duplicate-update path; - `buildTenantObjectMetadata` no longer carries `ChildCloneRules` at all. Explicit custom-object rules continue through `applyTenantObjectSpecChildCloneRules`, which is the only tenant writer for that fact and has the locked/defaults-only ownership checks. The eighth blocker fix attempted to close legacy custom-object stale cleanup: - the retained standard-object `object_child_clone_rules.json` compatibility surface historically could also carry custom-object rules; - the first implementation cleared unresolved legacy live rows with no current authoring key and no `ChildCloneRulesPack*` sidecar regardless of `IsCustom`; - sidecar-owned rows and current `tenant_objects_spec.json` rules remain kept through the existing keep/restore path. The ninth blocker fix replaces ambiguous sidecar-less auto-cleanup with a fail-closed guard: - a sidecar-less, non-empty live `ChildCloneRules` row is ambiguous during warm upgrades regardless of `ChildCloneRulesCustomized`, because it could be stale tenant residue or a valid pack-authored rule before ownership sidecar backfill/reconcile has run; - ordinary `apply-tenant` therefore no longer nulls those rows from the steady-state cleanup path; - if such a row is outside the current tenant keep-set, `apply-tenant` now fails with the unresolved object names and requires either active-pack ChildCloneRules ownership backfill/reconcile or explicit current-metadata compatibility reauthor/clear before the legacy source is removed; - rows explicitly marked `ChildCloneRulesCustomized=1` remain safe to clear only when current metadata keeps them or pack sidecar/restore state proves the underlying pack value can be deterministically restored. The tenth blocker fix makes the fail-closed guard mutation-safe: - unresolved sidecar-less detection now runs before any restore or cleanup mutation, so a failed `apply-tenant` cannot partially change live `ChildCloneRules` behavior; - `MissingPackOwnership` custom-object rows are not treated as kept and do not hide sidecar-less live rules from the guard; - restore bookkeeping records only rows actually restored from proven sidecar-backed pack ownership. The eleventh blocker fix closes the final customized-row cleanup gap: - unresolved sidecar-less detection no longer filters on `ChildCloneRulesCustomized=0`; - customized-but-sidecarless rows outside the current keep-set now fail closed before cleanup, matching the same ownership proof required for non-customized warm-upgrade rows. The twelfth blocker fix makes rollout restart-safe: - standalone `setup repair` hydrates source-backed `ChildCloneRulesPack*` sidecars before running the legacy `ChildCloneRulesCustomized` classifier; - pack verify still checks exact sidecar identity for the whole object-spec component once any sidecar evidence is present, but it defers sidecar-exact failures for wholly sidecar-less legacy components and verifies the live executable values instead; - this keeps warm-upgraded `setup verify` and server readiness from failing solely because the new sidecars have not yet been hydrated, while preserving exact verification after repair/apply has established pack ownership.