# Layout System Research — Current State, Salesforce Reference, and Gap Analysis

**Date**: 2026-03-31
**Status**: Research complete, pending design decisions
**Authors**: Ian + Claude Code analysis agents

---

## Table of Contents

1. [Context and Motivation](#1-context-and-motivation)
2. [Current System: Reverse Engineering](#2-current-system-reverse-engineering)
3. [Salesforce Reference: How It Works](#3-salesforce-reference-how-it-works)
4. [Gap Analysis: Side-by-Side Comparison](#4-gap-analysis-side-by-side-comparison)
5. [Database Reality: What's Actually in Production](#5-database-reality-whats-actually-in-production)
6. [Data vs Code Classification](#6-data-vs-code-classification)
7. [Open Design Questions](#7-open-design-questions)

---

## 1. Context and Motivation

NexusCRM is a config-driven CRM where configuration lives in three places:

- **Backend database** — what objects exist, their schema, page layouts, picklist values (org/team-specific)
- **Code** — frontend specs, Go backend handlers (git-versioned)
- **Bootstrapped system tables** — `system_tables.json`, `system_data.json` (JSON in git, applied on every startup)

The layout system is a concrete example of a recurring problem: **testers find issues and cannot determine if the root cause is code or data**. LLMs frequently misdiagnose data problems as code problems (e.g., "missing layout" → creates bootstrap files instead of data-spec). This research documents the current layout system fully, compares it to Salesforce's design, and identifies gaps to inform design decisions.

### User Requirements

There are two distinct user personas with different needs:

**End users (sales reps, managers)** need to:
- Open any record (Account, Lead, Quote, etc.) and see a well-organized detail page with meaningful sections, correct field grouping, and logical field ordering — not 200 fields dumped alphabetically
- Edit a record using a form that shows the right fields in the right sections, with required fields marked and read-only fields excluded
- See a highlights panel (compact layout) at the top with the most important fields at a glance

**Admins (system administrators, CRM managers)** need to:
- **View** the current layout configuration for any object
- **Create** a new page layout (from scratch, or by cloning an existing one)
- **Edit** an existing page layout using a drag-and-drop editor — reorganize sections, add/remove fields, reorder fields within sections
- **Assign** different layouts to different user profiles (and ideally to different RecordType × Profile combinations, so different business processes show different forms)
- **Delete** layouts that are no longer needed (with protection so the default layout cannot be accidentally removed)
- **Set a default** layout that applies when no specific assignment exists

**The system itself** needs to:
- Always provide a functional layout for every object, even if no admin has configured one yet (sensible default behavior)
- Protect admin-customized layouts from being overwritten by system upgrades or algorithm changes
- Automatically add newly created custom fields to existing layouts so they don't "disappear"
- Serve layouts efficiently at runtime without requiring admin intervention for every new object

### Long-Term Vision: Personalized CRM Per User ("千人千面")

The layout system is not just about "showing fields in sections." The long-term vision is **personalized experiences per user** — each person sees only the information relevant to them, their role, their region, and their business process.

This vision is informed by real operational pain from the team's Salesforce experience:

**The NA vs APAC conflict**: In Salesforce, NA Ops and APAC Ops shared the same page layouts because Salesforce's assignment model is `(Profile × RecordType) → Layout`. A single Profile (e.g., "Sales Rep") can only have one layout per RecordType — there is no region or team dimension. When APAC Ops changed a field on the Opportunity layout, NA Ops would complain. When NA added a section, APAC said it was irrelevant. This created constant friction.

In 2024, a hacky workaround was implemented in Salesforce to give NA and APAC different layouts (likely using extra Profiles or RecordTypes as a region proxy), but it was fragile and hard to maintain. **Salesforce's (Profile × RecordType) model is the floor, not the ceiling.**

**What NexusCRM needs**: An assignment model with at least one additional dimension beyond Salesforce:

```
Salesforce:  (Profile × RecordType)               → Layout
NexusCRM:    (Region/Team × Profile × RecordType)  → Layout
```

This enables:
- NA Ops defines the Opportunity page for NA Sales Reps
- APAC Ops defines the Opportunity page for APAC Sales Reps
- They operate independently — no more conflicts
- Future: individual Sales Managers customize their team's views

**Ownership evolution**:
- **Short-term**: Ops teams own layout configuration (NA Ops, APAC Ops, each independently)
- **Long-term**: Sales leaders own their team's layouts (self-service customization)
- **Always**: Different regions/teams must be able to work independently without stepping on each other

This is NexusCRM's opportunity to be genuinely better than Salesforce at layout personalization. The architecture must be designed to support this from the start, even if the full multi-dimension assignment is implemented incrementally.

### The core problem today

The layout CRUD infrastructure is largely complete — two drag-and-drop editors exist, Save/Delete/SetDefault/Assign APIs all work, the 3-tier resolution logic is correct. **The actual problem is that no initial layout data exists for the core standard objects.** 8 out of 11 core objects (Contact, Opportunity, Lead, Campaign, Case, Contract, Order, Event) have zero layout data in the database. When users open these records, the system falls back to auto-generation, which produces generic sections with 70% of fields in a catch-all "Additional Details" bucket. This is not a code bug — it is a data gap that no one (neither bootstrap nor an admin) has filled.

---

## 2. Current System: Reverse Engineering

### 2.1 Data Model

#### Table: `_System_PageLayout`

Defined in `backend/internal/bootstrap/system_tables.json:2128-2206`.

| Column | Type | Purpose |
|--------|------|---------|
| `Id` | VARCHAR(255) PK | Layout identifier (e.g., `default_Account` for auto-generated, UUID for admin-created) |
| `ObjectApiName` | VARCHAR(255) NOT NULL | Which object this layout belongs to |
| `LayoutName` | VARCHAR(255) NOT NULL | Human-readable name (e.g., "Account Layout") |
| `Type` | VARCHAR(255) | Layout type: Detail, Edit, Create, List |
| `IsDefault` | TINYINT(1) default 0 | Whether this is the default layout for the object |
| `IsDeleted` | TINYINT(1) default 0 | Soft-delete flag |
| `Config` | JSON NOT NULL | Full layout structure (sections, fields, related lists, actions) |
| `IsCustomized` | TINYINT(1) default 0 | If 1, bootstrap/version-upgrade will not overwrite |
| `LayoutVersion` | INT default 0 | Tracks auto-generation algorithm version |
| `CreatedDate` | DATETIME | |
| `LastModifiedDate` | DATETIME | |

#### Table: `ProfileLayout`

Defined in `system_tables.json:2948`. Foreign key to `_System_PageLayout(Id) ON DELETE CASCADE`.

Enables per-profile layout assignment (used in Tier 1 resolution). **Does NOT have a RecordType column** — this is a key gap vs Salesforce.

#### Config JSON Structure

The `Config` column stores the entire `models.PageLayout` struct, defined in `shared/pkg/models/metadata.go:172-223`:

```json
{
  "id": "default_Account",
  "objectApiName": "Account",
  "layoutName": "Default Layout",
  "type": "Detail",
  "isDefault": true,
  "compactLayout": ["Name", "Type", "Industry", "Phone", "OwnerId"],
  "tabs": ["Details", "Related", "Activity"],
  "sections": [
    {
      "id": "uuid",
      "label": "Information",
      "columns": 2,
      "fields": ["Name", "OwnerId", "Type", "Industry"],
      "fieldLabels": {"AccountId.Name": "Account"},
      "visibilityCondition": null
    }
  ],
  "relatedLists": [
    {
      "id": "Contact-AccountId",
      "label": "Contacts",
      "objectApiName": "Contact",
      "lookupField": "AccountId",
      "fields": ["Name", "CreatedDate"]
    }
  ],
  "headerActions": [{"name":"Edit","label":"Edit","type":"UpdateRecord"}],
  "quickActions": []
}
```

Key observations:
- **Flat field list**: Sections contain `fields: string[]` (API names), not a Salesforce-style nested LayoutRow → LayoutItem → LayoutComponent hierarchy
- **Compact layout embedded**: Not a separate entity — embedded as `compactLayout: string[]` in the same JSON
- **Related lists embedded**: Also in the same JSON, not a separate table
- **Dual representation**: Frontend types (`frontend/src/types/layout.ts:90-139`) support both flat `fields[]` (for editors) and nested `layoutRows` (for Salesforce wire-format rendering)

### 2.2 Bootstrap Flow

At server startup (`cmd/server/main.go:154-162`):

```go
// line 157: Delete all std_layout_* rows from _System_PageLayout
bootstrap.CleanupHardcodedLayouts(db.DB())
// line 162: Clean up system fields in ListFields
bootstrap.CleanupSystemListFields(db.DB())
```

**CleanupHardcodedLayouts** (`bootstrap/layouts.go:42-53`):
```go
rawDB.Exec("DELETE FROM `_System_PageLayout` WHERE `Id` LIKE 'std_layout_%%'")
```

This was intentionally added to undo PR #466 which seeded 9 curated layouts that "contained only 8-27 standard fields, hiding 200+ custom fields" (comment at `layouts.go:36-39`).

**`standard_layouts.json`** (`bootstrap/standard_layouts.json`):
- Contains 9 curated Salesforce-parity layouts (Lead, Case, Campaign, Order, Contract, Account, Contact, Opportunity, Quote)
- Each has proper section names, field ordering, compact layouts
- **This file is dead code** — no Go code loads it. No `//go:embed`, no `json.Unmarshal`, no `os.ReadFile` anywhere in the codebase.

**Key takeaway**: There is **no layout seeding at startup**. The system relies entirely on lazy auto-generation.

### 2.3 Read Path — Layout Resolution

When a user views a record, the layout is resolved via `GetLayout()` in `metadata_layouts.go:28-126`:

```
GET /ui-api/record-ui/:recordIds
  → uiService.GetRecordUI()
    → [parallel] GetLayout(ctx, objectApiName, &profileID)
      → Tier 1: repo.GetLayoutIDForProfile(ctx, apiName, profileID)
                If found → repo.GetLayout(ctx, layoutID) → return
      → Tier 2: repo.GetLayouts(ctx, apiName)
                Pick IsDefault=true, or first found → return
      → Tier 2b: Version check
                If !IsCustomized && LayoutVersion < currentLayoutVersion(3)
                → DELETE stale layout, fall through to Tier 3
      → Tier 3: generateSmartLayout(apiName, schema)
                → persist via UpsertLayout() → return
    → transformLayout() → Salesforce UI API wire format
```

**Tier 1** (Profile-specific, `metadata_layouts.go:41-49`): Queries `ProfileLayout` for a layout assigned to this profile + object. Requires admin to have explicitly assigned a layout via Tooling API.

**Tier 2** (Object default, `metadata_layouts.go:52-65`): Queries all non-deleted layouts for the object. Picks the one marked `IsDefault`, or falls back to the first found.

**Tier 2b** (Version check, `metadata_layouts.go:69-76`): If the found layout is auto-generated (`IsCustomized=false`) and its `LayoutVersion` is behind `currentLayoutVersion` (currently 3), the layout is deleted and Tier 3 runs. This ensures auto-generation algorithm improvements propagate.

**Tier 3** (Auto-generate, `metadata_layouts.go:79-106`): Calls `generateSmartLayout()` which classifies fields into sections, builds compact layout, and persists the result so subsequent requests hit the DB.

### 2.4 Auto-Generation Algorithm

`generateSmartLayout()` at `metadata_layouts.go:442-588`:

**Field classification** (`classifyField()`):

| Section | Rule | Examples |
|---------|------|---------|
| Key Information | Name field, required fields, references, explicit key names (Status, Type, OwnerId, etc.) | Name, OwnerId, Type, Industry |
| Address | Fields matching Billing*/Shipping*/Mailing*/Other* + address suffixes | BillingStreet, ShippingCity |
| Description | textarea or richtextarea type | Description, Notes |
| Additional Details | Everything else (catch-all) | All remaining fields |
| System Information | CreatedById, CreatedDate, LastModifiedById, LastModifiedDate | System audit fields |

**Custom field sub-grouping**: Custom fields (`*__c`) with shared prefixes (≥3 fields) get their own section. E.g., `ARR_Amount__c`, `ARR_Type__c`, `ARR_Date__c` → "ARR" section.

**Compact layout** (`buildCompactLayout()`, lines 628-681): Selects up to 7 fields in priority order: Name field → required key fields → reference fields → remaining key fields.

**Output**: `PageLayout` with ID `"default_" + apiName`, persisted immediately via `UpsertLayout()`.

### 2.5 Write Path — Admin Creates/Edits Layout

**Save endpoint**: `POST /ui-api/layout/Global` → `HandleSaveLayout` (`ui_api_handler.go:207-248`)

1. Bind JSON body to `models.PageLayout`
2. If `Id` is empty, generate UUID
3. Call `metadataSvc.SaveLayout()` → `repo.SaveLayout()` (upsert with `ON DUPLICATE KEY UPDATE`)
4. Call `metadataSvc.MarkLayoutCustomized()` → sets `IsCustomized=1`
5. Return 201 (created) or 200 (updated)

**Other write operations**:
- `PATCH /ui-api/layout/:layoutId/default` — set as default (transactional: clears all others, sets target)
- `DELETE /ui-api/layout/:layoutId` — soft-delete (blocks deletion of default layout)
- `POST /tooling/sobjects/ProfileLayout` — assign layout to profile

### 2.6 Frontend Editors

Two layout editors exist:

**LayoutEditor** (`frontend/src/features/admin/components/LayoutEditor.tsx:16-292`):
- Full-featured editor with 4 tabs: Detail Layout, List View, Compact Layout, Related Lists
- Drag-and-drop field palette, section management, lookup display config
- Accessible at `/setup/objects/:objectApiName/layout[/:layoutId]`
- Uses `useLayoutState` hook for state management

**StudioLayoutEditor** (`frontend/src/features/studio/components/StudioLayoutEditor.tsx:23-368`):
- Simpler editor embedded in the Studio object editor
- Drag-and-drop sections and fields
- Creates default layout if none exists (lines 44-65)

Both save via `metadataAPI.saveLayout(layout)` → `POST /ui-api/layout/Global`.

### 2.7 Frontend Rendering

**LayoutRenderer** (`frontend/src/components/runtime/LayoutRenderer.tsx:24-273`):
- Iterates `layout.sections` as collapsible cards
- System sections collapsed by default
- Iterates nested `layoutRows → layoutItems → layoutComponents`
- Handles compound name fields, lookup expansions, empty spaces
- Each field rendered as `InlineEditField` with permission checks

**MetadataRecordForm** (`frontend/src/features/record/components/MetadataRecordForm.tsx`):
- Accepts optional `layout?: PageLayout` prop
- If layout provided: renders fields grouped by sections
- If no layout: renders all fields as flat 2-column alphabetical grid (the "100+ fields" problem)

**ObjectView** layout resolution (`frontend/src/pages/ObjectView.tsx:156-213`):
- Tries NexusCRM flat format: `{ sections: [...] }`
- Falls back to Salesforce nested format: `{ Full: { View: { sections } } }`
- Merges related lists from describe endpoint if missing

### 2.8 Complete API Surface

| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/sobjects/:sobject/describe/layouts` | RecordType → Layout mappings |
| GET | `/sobjects/:sobject/describe/layouts/:recordTypeId` | Full layout in SF REST format |
| GET | `/ui-api/record-ui/:recordIds` | Aggregated record + objectInfo + layout |
| GET | `/ui-api/layout/:objectApiName` | Single layout (profile-aware resolution) |
| GET | `/ui-api/layouts/:objectApiName` | List all layouts for object |
| POST | `/ui-api/layout/Global` | Create or update layout |
| PATCH | `/ui-api/layout/:layoutId/default` | Set as default |
| DELETE | `/ui-api/layout/:layoutId` | Soft-delete |
| POST | `/tooling/sobjects/ProfileLayout` | Assign layout to profile |

---

## 3. Salesforce Reference: How It Works

### 3.1 Data Model

Salesforce stores layouts as **metadata** (not data), deployable via Change Sets and Metadata API as XML.

**Hierarchy**:
```
Layout
  └── LayoutSection (label, style: OneColumn/TwoColumnsTopToBottom/TwoColumnsLeftToRight)
        └── LayoutColumn
              └── LayoutItem (behavior: Required/Edit/Readonly/Blank)
                    └── LayoutComponent (type: Field/Separator/SControl/Canvas/...)
```

**Key entities**:
- **PageLayout**: The full detail page layout. Includes sections, mini layout, related lists, feed view, quick action list, platform action list.
- **CompactLayout**: A separate metadata object. Controls the highlights panel (top of record page). Up to 10 fields. Can be different per RecordType.
- **SearchLayout**: Controls which fields appear in search results, lookup dialogs, list views.
- **MiniLayout**: Deprecated subset of PageLayout used for hover/popup details.

**Layout Assignment**: A separate metadata type `ProfileLayoutAssignment` maps `(Profile, RecordType) → Layout`. This is a **matrix**: for each profile, every record type can map to a different layout.

### 3.2 Layout Lifecycle

**Create**:
- Setup > Object Manager > {Object} > Page Layouts > New
- Options: blank layout or clone from existing
- New layout starts with all fields in an unnamed section
- Admin organizes fields into sections using drag-and-drop editor

**Edit**:
- Drag-and-drop editor with two panels:
  - Top: field palette (available fields, related lists, blank spaces, sections)
  - Bottom: layout canvas (sections with fields)
- Required fields shown with red asterisk (cannot be removed from layout)
- Section properties: label, 1-column or 2-column, detail/edit page visibility
- Field properties per layout: required, read-only

**Clone**:
- "Clone" option creates a copy with a new name
- Essential for creating RecordType-specific layouts: clone the default, then customize

**Assign**:
- "Page Layout Assignment" button in Object Manager
- Matrix UI: rows = Profiles, columns = RecordTypes
- Each cell selects which layout to use
- "Master" RecordType serves as the default column

**Delete**:
- Cannot delete if it's the only layout or if it's assigned to any Profile-RecordType combination
- Must reassign first

**New field auto-add**:
- When a custom field is created, Salesforce auto-adds it to **ALL page layouts** for that object
- Added at the bottom of the first section
- Admin must manually reposition or remove from layouts where it's not needed

### 3.3 Runtime Resolution

```
User logs in → has a Profile
User views record → record has a RecordTypeId
System looks up: (Profile, RecordType) → Layout in assignment matrix
  Found → use that layout
  Not found → use Master RecordType assignment for that Profile
  Still not found → use the first-created (default) layout
```

**Key principle**: Resolution is always deterministic. There is no "auto-generation" — every object has at least one layout from the moment it's created.

### 3.4 APIs

**Describe Layouts** (`GET /sobjects/{obj}/describe/layouts`):
- Returns `recordTypeMappings[]` — which layouts are assigned to which record types
- Each mapping includes `layoutId`, `recordTypeId`, `name`, `available`, `defaultRecordTypeMapping`
- Does NOT return the layout body itself (just the mapping)

**Describe Layout by RecordType** (`GET /sobjects/{obj}/describe/layouts/{rtId}`):
- Returns the full layout body:
  - `detailLayoutSections[]` with nested `layoutRows → layoutItems → layoutComponents`
  - `editLayoutSections[]` (same structure, different field behaviors for edit mode)
  - `relatedLists[]`
  - `buttonLayoutSection` (standard/custom buttons)

**UI API Layout** (`GET /ui-api/layout/{obj}`):
- Returns layout for the **current user's profile and record type context**
- Response wrapped in: `{ layouts: { Full: { View: {...}, Edit: {...}, Create: {...} } } }`
- Each mode has its own section structure with `layoutRows → layoutItems → layoutComponents`
- Includes `objectApiName`, `recordTypeId`, `layoutType`, `mode`

**UI API Record UI** (`GET /ui-api/record-ui/{recordId}`):
- Aggregated response: `{ records: {...}, layouts: {...}, objectInfos: {...} }`
- Layout embedded under `layouts[objectApiName][layoutKey]`

**Metadata API** (`Layout` type):
- Full XML representation deployable via Change Sets, ANT Migration Tool, SFDX
- Includes all sections, items, related lists, actions, feed view config
- Used for CI/CD promotion between environments

### 3.5 Default Layout Behavior

- Every new object gets a default layout automatically
- The default layout contains ALL fields in a single unnamed section
- This is the "starting point" — admins customize from there
- The default layout is assigned to all profiles for the Master RecordType
- New fields are auto-added to all layouts (including the default)

### 3.6 Key Design Decisions in Salesforce

1. **Layouts are metadata, not data**: Deployable between environments via Change Sets. This is how Salesforce solves the "promote config from sandbox to prod" problem.
2. **Always at least one layout**: No object can exist without a layout. No "auto-generation from schema" needed.
3. **RecordType is a first-class dimension**: Different business processes = different record types = different layouts. This is fundamental.
4. **New fields auto-added**: Prevents the "invisible field" problem. Fields always appear somewhere.
5. **Clone as the creation pattern**: Don't build from scratch — clone the default and customize.

### 3.7 Salesforce Design Philosophy — Why It Works

Understanding the *reasoning* behind Salesforce's design choices is as important as understanding the mechanics.

**"Something is always better than nothing" principle**

When a new object is created, Salesforce immediately generates a default layout containing ALL fields in a single unnamed section. This layout is ugly — one giant list of every field — but it is *functional*. Users can see every field, enter data, and get work done from day one.

The admin's job is to *improve* this starting point: organize fields into sections, remove irrelevant fields, set required/read-only behaviors. This is a refinement workflow (start messy, make better), not a creation workflow (start from nothing, build up).

NexusCRM's `generateSmartLayout()` follows this principle by classifying fields into sections automatically. However, it produces generic section names ("Information", "Additional Details") that cannot match Salesforce's curated names ("Account Information", "TiDB Cloud", "Intent Insights"). The algorithm is smarter than Salesforce's "all fields in one section" default, but less controllable.

**"Fields must never disappear" principle**

When a new custom field is created, Salesforce auto-adds it to ALL page layouts for that object. This is a deliberate tradeoff:

- A field appearing in the wrong position → admin asks "why is this here?" → easy to fix (drag to correct section)
- A field not appearing at all → user asks "where is my data?" → hard to diagnose (is it a permission issue? layout issue? field deleted?)

The second scenario is far more expensive in support time. Salesforce chooses the lower-cost failure mode.

NexusCRM's `addFieldToLayout()` only adds to the first layout (`layouts[0]`), violating this principle. Fields silently disappear on non-default layouts.

**"RecordType is a business process boundary" principle**

Salesforce treats RecordType as the primary mechanism for business process differentiation:

- An Opportunity might be "New Business" (needs fields: Discovery Date, Competitor, POC Status) or "Renewal" (needs fields: Contract End Date, Churn Risk Score, Renewal Amount)
- An Account might be "Business Account" (needs fields: Industry, Employee Count, Annual Revenue) or "Person Account" (needs fields: Birthdate, Personal Email, Mailing Address)

Without RecordType-aware layouts, all Opportunities show the same form regardless of sales motion. This forces admins to include ALL fields on one layout, recreating the "200 fields on a page" problem that layouts are supposed to solve.

The (Profile × RecordType) → Layout matrix is Salesforce's answer to: "Different users doing different types of work need different views of the same object."

**"Layout is metadata, not data" principle**

Salesforce classifies layouts as *metadata* — the same category as code, fields, and objects. This means:

- Layouts are version-controlled (included in Change Sets and deployable via Metadata API)
- Layouts move between environments (sandbox → production) using the same promotion pipeline as code
- Layouts have a lifecycle: created in a sandbox, tested, then deployed

NexusCRM stores layouts as database rows (`_System_PageLayout`), which means they:
- Are not version-controlled (not in git)
- Cannot be promoted between environments (the project's `data-spec` system is a workaround for this)
- Can be lost if the database is reset

This is the root of the "Data vs Code" classification challenge described in section 1. In Salesforce, this challenge doesn't exist because layouts are unambiguously metadata.

### 3.8 Fundamental Difference: Deterministic Lookup vs Fallback Chain

The deepest architectural difference between the two systems:

**Salesforce**: Layout resolution is a **deterministic lookup** in a pre-configured matrix. Given (Profile, RecordType), there is exactly one answer. No generation, no fallback chain, no algorithm. If the matrix cell is empty, it falls back to the Master RecordType column — but even that is a configured value, not a computed one.

**NexusCRM**: Layout resolution is a **fallback chain** with three tiers. The system tries increasingly general strategies (profile-specific → object default → auto-generate) until one produces a result. Tier 3 (auto-generation) is a computation, not a lookup.

This means:
- In Salesforce, layout resolution is O(1) — a hash lookup
- In NexusCRM, layout resolution can be O(N) in the worst case (Tier 3 reads all fields, classifies them, generates sections, persists to DB)
- In Salesforce, the result is always admin-curated
- In NexusCRM, the result may be algorithm-generated with no admin involvement

Neither approach is inherently wrong. Salesforce's deterministic model requires upfront admin work but guarantees quality. NexusCRM's fallback model works with zero configuration but produces generic results. The question for design is: **which tradeoff is right for NexusCRM's target users?**

---

## 4. Gap Analysis: Side-by-Side Comparison

### 4.1 Feature Comparison

| Capability | Salesforce | NexusCRM | Status |
|------------|-----------|----------|--------|
| Layout storage | XML metadata (deployable) | JSON in DB table | ✅ Different approach, both work |
| Section/field structure | Nested: Section > Column > Item > Component | Flat: Section > Fields[] | ⚠️ Less flexible, but sufficient for field-only layouts |
| (Profile, RecordType) → Layout | Full matrix assignment | Profile-only, no RecordType dimension | ❌ Missing |
| Runtime resolution | Deterministic lookup in matrix | 3-tier: Profile → Default → Auto-generate | ⚠️ Works, but Tier 3 is a fallback Salesforce doesn't need |
| Default layout on object creation | Auto-created with all fields | None until first access triggers Tier 3 | ⚠️ Lazy vs eager |
| Layout editor (drag-and-drop) | Setup UI | Two editors (LayoutEditor, StudioLayoutEditor) | ✅ |
| Clone layout | Built-in | Not implemented | ❌ |
| New field auto-add to ALL layouts | Yes | Only adds to `layouts[0]` | ⚠️ Partial |
| Compact layout | Separate metadata object | Embedded in PageLayout JSON | ⚠️ Less flexible |
| Related lists in layout | Configurable per layout | Auto-discovered from schema | ⚠️ Different approach |
| Layout assignment UI (matrix) | Profile × RecordType grid | No matrix UI | ❌ |
| Metadata API (deploy/retrieve) | Full XML deploy | Not implemented | ❌ (not needed short-term) |
| Layout versioning/migration | Change Sets between orgs | LayoutVersion + auto-regeneration | ⚠️ Different approach |
| Delete protection | Cannot delete if assigned | Cannot delete default layout | ✅ |
| Set default | Implicit via assignment matrix | Explicit endpoint | ✅ |
| Save with customization guard | N/A (all layouts are admin-managed) | IsCustomized=1 prevents auto-overwrite | ✅ NexusCRM-specific |

### 4.2 Rated Gaps

| # | Gap | Severity | Impact |
|---|-----|----------|--------|
| 1 | No RecordType dimension in layout assignment | **P0** | Multi-business-process orgs cannot differentiate layouts |
| 2 | No seed/default layout for standard objects | **P1** | 8 core objects have no layout data; auto-generate produces generic sections |
| 3 | UI API response format missing `recordTypeMappings` wrapper | **P1** | Frontend expects Salesforce wire format; contract tests flag red |
| 4 | New field only added to first layout | **P2** | Fields disappear on non-default layouts |
| 5 | No clone layout | **P2** | Admin productivity loss when creating RT-specific layouts |
| 6 | Compact layout not independent | **P2** | Cannot vary compact layout per RecordType |
| 7 | No Metadata API for layouts | **P3** | Blocks CI/CD layout promotion (not needed for single-env) |

---

## 5. Database Reality: What's Actually in Production

Verified 2026-03-31 against TiDB Cloud production database.

### Layout Data

| Metric | Value |
|--------|-------|
| Total objects in `_System_Object` | 355 |
| Objects with layout in `_System_PageLayout` | **8** (2.3%) |
| All layouts `IsCustomized` | **0** (none customized) |
| All layouts `LayoutVersion` | **0** (all will be regenerated on next access due to currentLayoutVersion=3) |

**Which core objects have layouts**:

| Object | Has Layout? | Source |
|--------|------------|--------|
| Account | ✅ Yes | Auto-generated (`default_Account`) |
| Quote | ✅ Yes | Auto-generated (`default_Quote`) |
| Task | ✅ Yes | Auto-generated |
| invoice_collection__c | ✅ Yes | Auto-generated |
| org__c | ✅ Yes | Auto-generated |
| pricebook2 | ✅ Yes | Auto-generated |
| product2 | ✅ Yes | Auto-generated |
| Flow | ✅ Yes | Auto-generated |
| **Contact** | ❌ No | — |
| **Opportunity** | ❌ No | — |
| **Lead** | ❌ No | — |
| **Campaign** | ❌ No | — |
| **Case** | ❌ No | — |
| **Contract** | ❌ No | — |
| **Order** | ❌ No | — |
| **Event** | ❌ No | — |

### Layout Quality (Account example)

Account layout has 4 sections with field distribution:

| Section | Field Count |
|---------|------------|
| Information | 8 |
| Address Information | 20 |
| Description | 22 |
| Additional Details | **139** (catch-all) |

**Total**: 198 fields, with 139 (70%) dumped into the generic "Additional Details" section.

In Salesforce, Account has named sections like "Account Information", "TiDB Cloud", "Intent Insights", "Account Plan Strategy", "SLA" — none of which auto-generation can produce.

### Field Label Quality

| Metric | Value |
|--------|-------|
| Total fields in `_System_Field` | 9,119 |
| Fields where `Label = ApiName` (not humanized) | **1,027** (11.3%) |
| Fields with `*` prefix in Label | **29** |
| Fields with `IsCustomized = 1` | **0** |

Worst objects: Lead (16 bad labels), User (16), CampaignMember (13), Asset (12).

---

## 6. Data vs Code Classification

Per the project's `data-vs-code` skill (`.agent/skills/data-vs-code/SKILL.md`):

| Layout Problem | Classification | Rationale |
|----------------|---------------|-----------|
| Standard objects have no seed layout data | **DATA_REQUEST** | Layouts are org/team-specific config, not universal schema. Per skill: "Layout only in prod → DATA_REQUEST (data-spec to update `_System_PageLayout`)" |
| Auto-generate algorithm produces generic sections | **CODE_FIX** | `generateSmartLayout()` logic in `metadata_layouts.go` |
| `addFieldToLayout` only updates first layout | **CODE_FIX** | Bug in `metadata_layouts.go:238-271` |
| RecordType dimension missing from assignment | **CODE_FIX** + **SCHEMA** | Requires `ProfileLayout` schema change + `GetLayout` signature change |
| UI API response format wrong | **CODE_FIX** | Response wrapper missing in `HandleLayout` |
| `standard_layouts.json` is dead code | **DECISION NEEDED** | Either delete it, activate as bootstrap, or use as data-spec source |
| `CleanupHardcodedLayouts` deletes seed data | **CODE_FIX** (if we decide to seed) | Remove or gate the cleanup |

### The Ownership Gap

The current system has a gap in the data-vs-code framework: **who is responsible for creating initial layout data for standard objects?**

- Bootstrap code doesn't do it (intentionally — `CleanupHardcodedLayouts` was added to undo previous seeding)
- No data-spec has been created for it
- The auto-generation algorithm fills the gap at runtime, but produces generic results

This is not a bug — it's an unresolved design decision.

---

## 7. Design Recommendation

### 7.1 Why the Current Architecture Needs Redesign

The current 3-tier fallback (Profile → Default → Auto-generate) was a reasonable starting point, but it has a fundamental problem: **it mixes three different concerns into one code path** — content storage, routing rules, and resolution logic are all tangled together in `GetLayout()`.

More critically, adding the Region/Team dimension required for the "千人千面" vision (see section 1) would mean adding yet another tier to an already complex fallback chain. The current Tier 1 (`ProfileLayout`) only supports Profile — extending it to support Role, RecordType, and future dimensions would require rewriting the resolution logic each time.

The correct approach is to **separate concerns cleanly** so that new routing dimensions can be added without changing the resolution algorithm.

### 7.2 Architecture: Three Components, Each Does One Thing

```
┌──────────────────┐      ┌──────────────────────┐      ┌──────────────┐
│  Layout Content   │◄─────│  Layout Assignment    │─────►│  Resolution   │
│                   │      │                       │      │  Engine       │
│  _System_         │      │  _System_             │      │              │
│  PageLayout       │      │  LayoutAssignment     │      │  Pure func:  │
│                   │      │  (NEW TABLE)          │      │  (user, obj, │
│  Stores:          │      │                       │      │   record)    │
│  - sections       │      │  Stores routing:      │      │     ↓        │
│  - fields         │      │  - ObjectApiName      │      │  LayoutId    │
│  - compactLayout  │      │  - LayoutId           │      │              │
│  - relatedLists   │      │  - RoleId     (nullable)     │  No match?   │
│  - headerActions  │      │  - ProfileId  (nullable)     │  → auto-gen  │
│                   │      │  - RecordTypeId(nullable)    │              │
│  No routing logic │      │  - Specificity (computed)    │              │
└──────────────────┘      └──────────────────────┘      └──────────────┘
```

**1. Layout Content (`_System_PageLayout`)** — stores what a layout looks like. Just JSON. Does not know or care who sees it. Unchanged from current schema.

**2. Layout Assignment (`_System_LayoutAssignment`)** — stores routing rules. "Given this combination of role/profile/recordType, use this layout." Pure data, no behavior. **Replaces `ProfileLayout`.**

**3. Resolution Engine** — a pure function. Given (user context, object, record), find the best matching assignment, return the layout. If nothing matches, auto-generate and create a default assignment. **Replaces the current 3-tier `GetLayout()` logic.**

### 7.3 The Key Insight: Tiers Are Just Specificity Levels

The current 3-tier fallback is not three different mechanisms — they are the **same mechanism at different specificity levels**:

| Current Tier | What It Really Is | Specificity |
|-------------|-------------------|-------------|
| Tier 1: Profile-specific | An assignment row with `ProfileId` set | 1 |
| Tier 2: Object default | An assignment row with all dimensions NULL | 0 |
| Tier 3: Auto-generate | Fallback when zero assignments exist | N/A |

The new design makes this explicit. There are no "tiers" — just assignments with different specificity, and a single resolution query.

### 7.4 `_System_LayoutAssignment` Schema

```sql
CREATE TABLE _System_LayoutAssignment (
    Id               VARCHAR(255) PRIMARY KEY,
    ObjectApiName    VARCHAR(255) NOT NULL,
    LayoutId         VARCHAR(255) NOT NULL,    -- FK to _System_PageLayout

    -- Routing dimensions (all nullable; NULL = wildcard / "any")
    RoleId           VARCHAR(255),             -- organizational position (inherits via Role Hierarchy)
    ProfileId        VARCHAR(255),             -- permission role (Salesforce compatibility)
    RecordTypeId     VARCHAR(255),             -- business process type

    -- Computed: number of non-NULL dimensions (more = more specific)
    Specificity      INT AS (
        (RoleId IS NOT NULL) + (ProfileId IS NOT NULL) + (RecordTypeId IS NOT NULL)
    ),

    CreatedDate      DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    LastModifiedDate DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    CreatedById      VARCHAR(255),

    INDEX idx_object (ObjectApiName),
    INDEX idx_specificity (ObjectApiName, Specificity DESC)
);
```

**Design decisions**:

- **All routing dimensions are nullable.** `NULL` means "matches any value for this dimension." An assignment with `RoleId=NULL, ProfileId=NULL, RecordTypeId=NULL` is the global default (specificity 0). An assignment with all three set is the most specific (specificity 3).

- **RoleId uses Role Hierarchy for inheritance.** When resolving, the engine checks the user's Role AND all ancestor Roles. If APAC VP has an assignment, all roles under APAC VP inherit it automatically. No need to create separate assignments for every APAC role.

- **Specificity is computed, not manually set.** No admin needs to think about priority. The system counts non-NULL dimensions. More specific always wins.

- **Future dimensions** (AppId, TeamId, etc.) are added as nullable columns. Specificity computation and resolution SQL automatically include them. No architecture change needed.

### 7.5 Resolution Algorithm

```sql
-- Single query: find the best matching layout for this user + record
-- Tie-breaking priority: RecordTypeId > ProfileId > RoleId > CreatedDate (see Decision 3)
SELECT la.LayoutId
FROM _System_LayoutAssignment la
WHERE la.ObjectApiName = :objectApiName
  AND (la.RoleId IS NULL OR la.RoleId IN (:userRoleAncestors))
  AND (la.ProfileId IS NULL OR la.ProfileId = :userProfileId)
  AND (la.RecordTypeId IS NULL OR la.RecordTypeId = :recordTypeId)
ORDER BY
  la.Specificity DESC,                          -- most dimensions matched wins
  (la.RecordTypeId IS NOT NULL) DESC,           -- RT = business process (most specific context)
  (la.ProfileId IS NOT NULL) DESC,              -- Profile = permission role
  (la.RoleId IS NOT NULL) DESC,                 -- Role = organizational position
  la.CreatedDate ASC                            -- oldest assignment wins as final tiebreak
LIMIT 1;
```

Where `:userRoleAncestors` is the user's Role ID + all ancestor Role IDs from the Role Hierarchy (computed per-request from `PermissionService` in-memory cache — see Decision 8).

**For users with no Role** (`user.RoleID == nil`), the Role clause becomes `AND la.RoleId IS NULL` — only wildcard-role assignments match (see Decision 9).

**If no rows match**: auto-generate a layout from schema, persist it as a `PageLayout`, create an assignment with all NULLs (specificity 0), return it. This is a one-time operation per object.

**In Go pseudocode**:

```go
func ResolveLayout(ctx context.Context, objectApiName string, user User, recordTypeId string) *PageLayout {
    // 1. Find best assignment
    assignment := repo.FindBestAssignment(ctx, objectApiName, user.RoleAncestors, user.ProfileId, recordTypeId)

    if assignment != nil {
        return repo.GetLayout(ctx, assignment.LayoutId)
    }

    // 2. No assignment exists — auto-generate and create default assignment
    schema := metadataCache.GetSchema(objectApiName)
    layout := generateSmartLayout(objectApiName, schema)
    repo.UpsertLayout(ctx, layout)
    repo.CreateAssignment(ctx, objectApiName, layout.Id, nil, nil, nil)  // all NULL = global default
    return layout
}
```

No tiers. No fallback chain. One path.

### 7.6 How This Supports the "千人千面" Vision

**Example: NA vs APAC Opportunity layouts**

Setup by admins:

| # | Assignment | RoleId | ProfileId | RecordTypeId | Specificity | Effect |
|---|-----------|--------|-----------|--------------|-------------|--------|
| 1 | Global default | NULL | NULL | NULL | 0 | Everyone's fallback |
| 2 | NA Opportunity | `NA_VP_Sales` | NULL | NULL | 1 | All NA users see this |
| 3 | APAC Opportunity | `APAC_VP_Sales` | NULL | NULL | 1 | All APAC users see this |
| 4 | APAC New Business | `APAC_VP_Sales` | NULL | `NewBusiness` | 2 | APAC users on New Business deals |
| 5 | APAC Sales Rep + New Biz | `APAC_Sales_Mgr` | `SalesRep` | `NewBusiness` | 3 | Narrow: APAC Sales Reps on New Business |

**Runtime resolution for Charlie (APAC Sales Rep) opening a New Business Opportunity**:

Charlie's context: Role = `APAC_Sales_Rep`, RoleAncestors = [`APAC_Sales_Rep`, `APAC_Sales_Mgr`, `APAC_VP_Sales`, `CEO`], Profile = `SalesRep`, RecordType = `NewBusiness`

Matching assignments: #1 (specificity 0), #3 (specificity 1, APAC_VP_Sales ∈ ancestors), #4 (specificity 2), #5 (specificity 3)

**Winner: #5** (highest specificity). Charlie sees the layout tailored for APAC Sales Reps working New Business deals.

**Ownership model**:
- APAC Ops Manager creates assignments #3, #4, #5 — controls what APAC sees
- NA Ops Manager creates assignment #2 — controls what NA sees
- They never touch each other's assignments
- Future: APAC Sales Manager overrides #5 with a more specific assignment for their sub-team

### 7.7 Salesforce Compatibility

If only `ProfileId` and `RecordTypeId` are used (RoleId always NULL), the behavior is **identical to Salesforce's (Profile × RecordType) → Layout model**:

| ProfileId | RecordTypeId | Specificity | Salesforce Equivalent |
|-----------|--------------|-------------|----------------------|
| `SalesRep` | `NewBusiness` | 2 | Cell in Profile × RecordType matrix |
| `SalesRep` | NULL | 1 | Profile's default layout (Master RT) |
| NULL | NULL | 0 | Object's default layout |

Salesforce API compatibility maintained:
- `GET /sobjects/{obj}/describe/layouts` → query assignments grouped by RecordType
- `GET /sobjects/{obj}/describe/layouts/{rtId}` → resolve for current user's Profile + this RT
- `GET /ui-api/layout/{obj}` → resolve for current user's full context (Role + Profile + RT)

### 7.8 Auto-Generation as Initialization, Not Fallback

Auto-generation is no longer "Tier 3 fallback" — it is **initialization**. It runs exactly once per object (when the first user accesses an object with zero assignments), produces a layout, and creates a specificity-0 assignment. After that, resolution is always a pure database lookup.

The auto-generation algorithm should be improved with object-specific section names:

```go
var standardObjectSections = map[string][]SectionTemplate{
    "Account":     {{"Account Information", keyRules}, {"Address Information", addressRules}, ...},
    "Opportunity": {{"Opportunity Information", keyRules}, ...},
    // ...9 standard objects
}
```

These templates define section names and **classification rules** (not static field lists), so new fields are automatically classified into the correct section. This avoids the `standard_layouts.json` problem (static data drifting from dynamic schema).

### 7.9 Version History

Add `_System_PageLayoutHistory` for content versioning:

```sql
CREATE TABLE _System_PageLayoutHistory (
    Id              VARCHAR(255) PRIMARY KEY,
    LayoutId        VARCHAR(255) NOT NULL,    -- FK to _System_PageLayout
    VersionNumber   INT NOT NULL,             -- auto-increment per layout
    Config          JSON NOT NULL,            -- snapshot of layout content
    ChangedById     VARCHAR(255),             -- NULL = system/auto-generate
    ChangeReason    VARCHAR(255),             -- 'admin_edit', 'auto_regenerate', 'field_added'
    CreatedDate     DATETIME NOT NULL,

    UNIQUE KEY (LayoutId, VersionNumber)
);
```

Written on every `SaveLayout()` and every auto-regeneration. Last 20 versions kept. Restore via `POST /ui-api/layout/:layoutId/restore/:versionNumber`.

This is separate from `LayoutVersion` (algorithm version). `LayoutVersion` controls when auto-generated layouts are regenerated. `PageLayoutHistory.VersionNumber` tracks content changes for undo/rollback.

### 7.10 Bug Fixes (Clean Up Current System)

**a) Delete `CleanupHardcodedLayouts` and `standard_layouts.json`**

Both are artifacts of the old seed-data approach. With improved auto-generation, neither is needed. Remove `main.go:157` call, delete the function and the dead JSON file.

**b) `addFieldToLayout` must update ALL layouts for the object**

Current: only updates `layouts[0]`. Fix: iterate all layouts. Follows Salesforce's "fields must never disappear" principle.

**c) Replace `ProfileLayout` with `_System_LayoutAssignment`**

Migrate any existing profile-layout data to the new table. Drop the old table. One clean routing mechanism instead of two.

### 7.11 Design Decisions (Resolved)

The following decisions were resolved through a 5-reviewer team review + critic challenge:

#### Decision 1: Region/Team dimension — Role Hierarchy, not LayoutGroupId

**Choice**: Use existing Role Hierarchy for layout personalization. Do NOT create a separate LayoutGroupId.

**Rationale**: A `LayoutGroupId` on the User record creates a parallel organizational structure that must be manually kept in sync with the real org structure. When a new hire joins APAC, someone must remember to set their LayoutGroupId — but their Role is already set by HR as part of onboarding. Role Hierarchy already encodes region/team boundaries (APAC VP → APAC Sales Manager → APAC Sales Rep). Using it for layout avoids dual maintenance.

**Edge cases handled by specificity**: A contractor in APAC's role hierarchy who needs a restricted layout gets it via Profile dimension — `(Role=APAC, Profile=Contractor)` at specificity 2 overrides `(Role=APAC, Profile=NULL)` at specificity 1.

**Coupling concern**: Layout resolution only READS the Role Hierarchy tree (one-way dependency). It does not write to it or depend on permission semantics. The role_hierarchy.go `getRoleAncestors()` function is a pure tree traversal on an in-memory cache — sub-microsecond, no DB hit.

#### Decision 2: Source + GenerationVersion replace IsCustomized + LayoutVersion

**Choice**: Add `Source VARCHAR(20)` (values: `auto_generated`, `admin_created`, `cloned`) and `GenerationVersion INT` to `_System_PageLayout`. Drop `IsCustomized` and `LayoutVersion`.

**Rationale**: A boolean `IsCustomized` conflates three distinct origins into two states. `Source` makes the "should I regenerate?" check explicit:
- `Source = 'auto_generated' AND GenerationVersion < current` → regenerate in-place
- `Source = 'admin_created'` → NEVER touch
- `Source = 'cloned'` → NEVER touch

`GenerationVersion` replaces `LayoutVersion` with clearer semantics: it tracks which algorithm version produced this layout, not a generic "version number."

VARCHAR(20) chosen over ENUM because TiDB ENUMs require ALTER TABLE to add values. VARCHAR allows adding new source types (e.g., `imported`, `template`) without schema migration.

#### Decision 3: Specificity tie-breaking — RecordTypeId > ProfileId > RoleId

**Choice**: Fixed priority `RecordTypeId > ProfileId > RoleId`, with `CreatedDate ASC` as final deterministic tiebreaker.

**Rationale**: RecordType represents a business process boundary (the most specific context — "this is a New Business deal"). Profile represents a permission role ("this user is a Sales Rep"). Role represents organizational position ("this user is in APAC"). Business process > permission role > org position is the correct specificity order.

```sql
ORDER BY
  la.Specificity DESC,
  (la.RecordTypeId IS NOT NULL) DESC,
  (la.ProfileId IS NOT NULL) DESC,
  (la.RoleId IS NOT NULL) DESC,
  la.CreatedDate ASC
LIMIT 1;
```

#### Decision 4: Role depth — NOT needed, defer

**Choice**: Do not add role depth to resolution. Accept that same-specificity matches from different ancestor depths are resolved by CreatedDate.

**Rationale**: The scenario (two assignments at same specificity but different ancestor depths) requires an admin to create overlapping role-based assignments — which the UNIQUE constraint prevents if all other dimensions are identical. If needed later, adding `FIELD(la.RoleId, :role0, :role1, ...) ASC` is a purely additive one-line change.

#### Decision 5: UNIQUE constraint via COALESCE sentinels

**Choice**: Use `GENERATED ALWAYS AS (COALESCE(col, '')) STORED` columns with a UNIQUE index.

```sql
`_RoleKey`         VARCHAR(255) GENERATED ALWAYS AS (COALESCE(`RoleId`, '')) STORED,
`_ProfileKey`      VARCHAR(255) GENERATED ALWAYS AS (COALESCE(`ProfileId`, '')) STORED,
`_RecordTypeKey`   VARCHAR(255) GENERATED ALWAYS AS (COALESCE(`RecordTypeId`, '')) STORED,

UNIQUE INDEX `uq_assignment_dims` (`ObjectApiName`, `_RoleKey`, `_ProfileKey`, `_RecordTypeKey`)
```

**Rationale**: TiDB treats `NULL != NULL` in unique indexes, so `UNIQUE(ObjectApiName, RoleId, ProfileId, RecordTypeId)` would allow duplicate all-NULL rows. COALESCE sentinels convert NULLs to empty strings for uniqueness purposes only. The `_` prefix signals "internal/computed, do not query directly." STORED (not VIRTUAL) is required by TiDB for indexed generated columns.

#### Decision 6: Specificity as GENERATED STORED column

**Choice**: Database-computed, not application-computed.

```sql
`Specificity` INT GENERATED ALWAYS AS (
    (`RoleId` IS NOT NULL) + (`ProfileId` IS NOT NULL) + (`RecordTypeId` IS NOT NULL)
) STORED
```

**Rationale**: A generated column cannot be wrong — the database recomputes it on every write. Application-computed specificity could have bugs or be inconsistent during rolling deploys. The column is read-only from Go's perspective.

#### Decision 7: In-place UPDATE for auto-regeneration, CASCADE as safety net

**Choice**: When `GenerationVersion` is stale, UPDATE the existing layout row (preserve ID). Add `ON DELETE CASCADE` to LayoutAssignment FK as safety net for explicit admin deletions.

```go
if layout.Source == "auto_generated" && layout.GenerationVersion < currentGenerationVersion {
    newLayout := generateSmartLayout(apiName, schema)
    newLayout.ID = layout.ID            // preserve existing ID → all FK references intact
    newLayout.Source = "auto_generated"
    newLayout.GenerationVersion = currentGenerationVersion
    repo.UpsertLayout(ctx, newLayout)   // UPDATE, not DELETE+INSERT
}
```

**Rationale**: In-place UPDATE preserves the layout ID and all LayoutAssignment FK references. DELETE would cascade-remove assignments, requiring re-initialization. If UPDATE fails, the user gets the stale layout (graceful degradation) — strictly better than the current behavior where DELETE succeeds but regeneration fails, leaving the user with nothing.

#### Decision 8: Role ancestors computed per-request, not cached in session

**Choice**: Call `permissionService.GetRoleAncestors(roleId)` on every layout resolution request.

**Rationale**: The Role Hierarchy cache is an in-memory `map[string]*string` populated eagerly at server startup (`NewPermissionService` calls `refreshRoleHierarchy` synchronously). `getRoleAncestors()` walks this map from child to root — O(depth), typically 3-7 hops, sub-microsecond. Caching in JWT/session would inflate token size and create stale-data bugs when the hierarchy changes mid-session.

#### Decision 9: No-Role users see only wildcard assignments

**Choice**: When `user.RoleID == nil`, resolution uses `AND la.RoleId IS NULL` instead of `IN (:ancestors)`.

**Rationale**: A user with no Role is not in any organizational unit — they should see default/profile-specific layouts, never role-specific ones. This matches current behavior (the existing system doesn't check roles at all).

### 7.12 Version Management

The system has two distinct version concepts:

#### GenerationVersion (algorithm version)

Tracks which version of the `generateSmartLayout()` algorithm produced a layout.

| Event | Behavior |
|-------|----------|
| Developer improves `generateSmartLayout()` | Bumps `currentGenerationVersion` constant in code |
| User accesses an auto-generated layout | Resolution checks `GenerationVersion < currentGenerationVersion` |
| Stale auto-generated layout found | In-place UPDATE: regenerate Config, update GenerationVersion, write to history |
| Admin-created layout | `Source='admin_created'` — GenerationVersion check is SKIPPED, never regenerated |

#### Content Version History (undo/rollback)

Every layout content change is recorded in `_System_PageLayoutHistory`.

| Event | History Record |
|-------|---------------|
| Auto-generation (first access) | `ChangeReason='auto_generate'`, `ChangedById=NULL` |
| Auto-regeneration (version bump) | `ChangeReason='auto_regenerate_v{N}'`, `ChangedById=NULL` |
| Admin edits layout in editor | `ChangeReason='admin_edit'`, `ChangedById={userId}` |
| New field auto-added to layout | `ChangeReason='field_added'`, `ChangedById=NULL` |
| Admin restores a previous version | `ChangeReason='restored_from_v{N}'`, `ChangedById={userId}` |

**Retention**: Last 20 versions per layout (configurable). Oldest versions pruned on write.

**Restore**: `POST /ui-api/layout/:layoutId/restore/:versionNumber` — copies historical Config to live layout, creates a new history entry. The restored layout keeps its current Source (if admin_created, stays admin_created).

### 7.13 Deployment Migration Path

The new architecture requires a one-time data migration. Four deployment scenarios are covered below.

#### Migration Version Tracking

Use a `_System_Migration` table instead of fragile column-existence detection:

```sql
CREATE TABLE IF NOT EXISTS _System_Migration (
    Id          VARCHAR(100) PRIMARY KEY,
    Version     INT NOT NULL,
    AppliedAt   DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    Description VARCHAR(500)
);
```

```go
func MigrateLayoutSystem(db *sql.DB) error {
    const migrationID = "layout_system"
    const targetVersion = 1

    var currentVersion int
    err := db.QueryRow(
        "SELECT Version FROM _System_Migration WHERE Id = ?", migrationID,
    ).Scan(&currentVersion)
    if err == sql.ErrNoRows {
        currentVersion = 0
    }
    if currentVersion >= targetVersion {
        return nil  // already migrated
    }
    // Run migration steps...
}
```

#### Migration Steps

**Step 1**: `InitializeSchema()` automatically creates new tables (`_System_LayoutAssignment`, `_System_PageLayoutHistory`, `_System_Migration`) and adds new columns (`Source DEFAULT 'auto_generated'`, `GenerationVersion DEFAULT 0`) to `_System_PageLayout` from updated `system_tables.json`.

**Step 2**: Populate new columns from old (idempotent — `WHERE Source IS NULL` prevents re-processing):
```sql
UPDATE _System_PageLayout SET
  Source = CASE WHEN IsCustomized = 1 THEN 'admin_created' ELSE 'auto_generated' END,
  GenerationVersion = LayoutVersion
WHERE Source IS NULL OR Source = 'auto_generated' AND GenerationVersion = 0;
```

**Step 3**: Create LayoutAssignment for every existing layout (idempotent via `ON DUPLICATE KEY`):
```sql
INSERT INTO _System_LayoutAssignment (Id, ObjectApiName, LayoutId, CreatedDate, LastModifiedDate)
SELECT
  CONCAT('migrated_', Id),
  ObjectApiName,
  Id,
  CreatedDate,
  LastModifiedDate
FROM _System_PageLayout
WHERE IsDeleted = 0
ON DUPLICATE KEY UPDATE LastModifiedDate = VALUES(LastModifiedDate);
```

**Step 4**: Migrate `ProfileLayout` data (table name is `ProfileLayout`, NOT `_System_ProfileLayout` — verified in production DB):
```sql
INSERT INTO _System_LayoutAssignment (Id, ObjectApiName, LayoutId, ProfileId, CreatedDate, LastModifiedDate)
SELECT Id, ObjectApiName, LayoutId, ProfileId, CreatedDate, LastModifiedDate
FROM ProfileLayout
ON DUPLICATE KEY UPDATE ProfileId = VALUES(ProfileId);
```
Note: Production currently has 0 rows in `ProfileLayout`. This step is a no-op but handles non-zero cases safely.

**Step 5**: Record migration version:
```sql
INSERT INTO _System_Migration (Id, Version, Description)
VALUES ('layout_system', 1, 'Layout system redesign: LayoutAssignment + Source/GenerationVersion')
ON DUPLICATE KEY UPDATE Version = 1, AppliedAt = NOW();
```

**Step 6 (DEFERRED — do NOT run in v1 release)**: Drop old columns and table:
```sql
-- ONLY run in a LATER release after confirming stability and no rollback needed
ALTER TABLE _System_PageLayout DROP COLUMN IsCustomized;
ALTER TABLE _System_PageLayout DROP COLUMN LayoutVersion;
DROP TABLE IF EXISTS ProfileLayout;
```

> **CRITICAL**: Old columns (`IsCustomized`, `LayoutVersion`) and `ProfileLayout` table are KEPT in the first release. New code ignores them. Old code can still read them. This provides a safe rollback window. Drop them in migration v2 after confirming stability for at least one release cycle.

#### Deployment Scenarios

**Scenario 1: Fresh environment (empty DB)**
- `InitializeSchema()` creates all tables with new schema (including `Source`/`GenerationVersion`; `system_tables.json` still defines `IsCustomized`/`LayoutVersion` for backward compat)
- `MigrateLayoutSystem()` — migration table doesn't exist yet, so it creates it. No rows in `_System_PageLayout`, Steps 2-4 are no-ops. Records version=1.
- First user access triggers `ResolveLayout()` → auto-generate + create specificity-0 assignment.

**Scenario 2: Upgrade (current production DB with 8 layouts)**
- `InitializeSchema()` adds `Source`/`GenerationVersion` columns (DEFAULT values prevent NULL window), creates new tables.
- `MigrateLayoutSystem()` — version=0 < 1, runs Steps 2-5.
  - 8 layouts get `Source='auto_generated'`, `GenerationVersion=0`.
  - 8 assignment rows created (specificity 0, all dimensions NULL).
  - 0 `ProfileLayout` rows migrated (no-op).
  - Migration recorded as version=1.
- Next user access: `ResolveLayout()` finds assignment → fetches layout → checks `Source='auto_generated', GenerationVersion=0 < currentGenerationVersion(3)` → in-place UPDATE regenerates with improved algorithm. Writes to history table.
- User sees: fresh layout with latest algorithm, immediately.

**Scenario 3: Rollback (revert to old code after migration)**
- Old code deploys. Old `scanLayout` reads `IsCustomized`/`LayoutVersion` columns — **they still exist** (not dropped in v1). No SQL error.
- Old code reads `ProfileLayout` table — **still exists** (not dropped in v1). Returns 0 rows as before.
- Old `GetLayout()` 3-tier resolution works as before. The new `_System_LayoutAssignment` table is ignored by old code (it doesn't know about it).
- **Safe rollback.** Zero data loss, zero errors.

**Scenario 4: Partial failure (crash mid-migration)**
- Server restarts. `MigrateLayoutSystem()` runs again.
- Checks `_System_Migration` — if version=1 already recorded (Step 5 succeeded before crash), skips everything. **Safe.**
- If version=0 (crash before Step 5): Steps 2-4 re-run. All use `WHERE Source IS NULL` guards or `ON DUPLICATE KEY UPDATE`, so they are idempotent. Step 5 records version=1. **Safe.**
- If `_System_Migration` table doesn't exist yet: `InitializeSchema()` creates it first. Migration proceeds from scratch. **Safe.**

#### Post-Migration Validation

Run these queries after deployment to verify:

```sql
-- 1. Every non-deleted layout has exactly one assignment
SELECT pl.Id, pl.ObjectApiName, COUNT(la.Id) AS cnt
FROM _System_PageLayout pl
LEFT JOIN _System_LayoutAssignment la ON la.LayoutId = pl.Id
WHERE pl.IsDeleted = 0
GROUP BY pl.Id HAVING cnt != 1;
-- Expected: 0 rows

-- 2. No layout has Source=NULL
SELECT Id FROM _System_PageLayout WHERE Source IS NULL;
-- Expected: 0 rows

-- 3. All assignments have valid FK
SELECT la.Id FROM _System_LayoutAssignment la
LEFT JOIN _System_PageLayout pl ON pl.Id = la.LayoutId
WHERE pl.Id IS NULL;
-- Expected: 0 rows

-- 4. Migration version recorded
SELECT * FROM _System_Migration WHERE Id = 'layout_system';
-- Expected: 1 row, Version=1
```

#### Bootstrap Sequence After Migration

```
main.go startup:
  InitializeSchema()              ← creates new tables/columns from system_tables.json
  MigrateLayoutSystem()           ← one-time (idempotent, skips if version >= target)
  // CleanupHardcodedLayouts()    ← REMOVED from code
  ...rest of bootstrap...
```

### 7.14 Edge Cases and Defensive Design

The following edge cases were analyzed and must be handled in the implementation:

#### Critical (prevent crash or data loss)

**EC-1: Assignment points to soft-deleted layout** — `ResolveLayout()` finds a matching assignment, calls `GetLayout(assignmentLayoutId)`, gets nil (layout filtered by `IsDeleted=false`). Without a nil check, returns nil to API layer → 404.

```go
// In ResolveLayout:
if assignment != nil {
    layout := repo.GetLayout(ctx, assignment.LayoutId)
    if layout == nil {
        // Layout was deleted — remove orphaned assignment, fall through to auto-generate
        repo.DeleteAssignment(ctx, assignment.Id)
        assignment = nil
    }
}
if assignment == nil {
    // auto-generate path
}
```

**EC-2: Schema not found** — `generateSmartLayout` dereferences `obj.Fields`. If schema lookup returns nil (non-existent object), panic occurs.

```go
// Preserve existing nil guard from metadata_layouts.go:98-99:
if schema == nil {
    return nil, nil  // API handler returns 404
}
```

**EC-3: Concurrent first-access** — Two users access same object simultaneously, both see 0 assignments, both auto-generate. `CreateAssignment` MUST use `ON DUPLICATE KEY UPDATE` (not plain INSERT) to handle the race.

#### High (wrong behavior visible to user)

**EC-4: Admin save overwrites another admin's changes** — Classic lost-update. Admin A opens editor, Admin B saves, Admin A saves → B's work lost silently.

```go
// In HandleSaveLayout: add optimistic concurrency check
if layout.LastModifiedDate != "" {
    existing := repo.GetLayout(ctx, layout.ID)
    if existing != nil && existing.LastModifiedDate.After(parsedClientDate) {
        return 409 Conflict, "Layout was modified by another user"
    }
}
```

**EC-5: SaveLayout ON DUPLICATE KEY UPDATE drops IsDefault/IsDeleted** — Current `repo_layout.go:103-110` omits these columns. Fix by adding them to the update map.

**EC-6: describe/layouts/{rtId} ignores recordTypeId** — `sobject_describe.go:384` captures rtId from URL but never passes it to resolution. With new design, pass to `ResolveLayout()`.

**EC-7: Malformed Config JSON silently skipped** — `scanLayout` at `metadata_scan_ui.go:79` catches unmarshal error but only logs at WARN. Corrupt layouts become invisible forever. Change to ERROR log with layout ID.

**EC-8: Ghost fields in layout** — Layout Config references a field that was deleted from schema. `transformLayout` renders it with raw API name as label, editable. Fix: filter fields not in schema during `transformLayout`.

#### Medium (degraded experience)

**EC-9: Cache not invalidated after admin save** — `uiapi.layoutCache` not evicted when admin saves. Users see stale layout until TTL expires. Fix: add cache eviction in `SaveLayout`.

**EC-10: Orphaned assignments after layout delete** — Soft-deleting a layout leaves assignment rows pointing to it. Fix: cascade-delete (or soft-delete) matching assignments in `DeleteLayout`.

**EC-11: addFieldToLayout races with admin editor** — Admin has stale copy, auto-field-add modifies the same layout, admin save overwrites the auto-added field. Fix: same as EC-4 (optimistic concurrency).

### 7.15 Implementation Phases

| Phase | Scope | Effort | Risk |
|-------|-------|--------|------|
| **Phase 1: Foundation** | Create `_System_LayoutAssignment` + `_System_PageLayoutHistory` tables in `system_tables.json`. Add `Source`/`GenerationVersion` columns to `_System_PageLayout`. Implement `ResolveLayout()` replacing `GetLayout()`. Write `MigrateLayoutSystem()`. Fix `addFieldToLayout` for all layouts. Delete `CleanupHardcodedLayouts` + `standard_layouts.json`. Fix `SaveLayout` repo (add IsDefault/IsDeleted to ON DUPLICATE KEY UPDATE). Eliminate duplicate fallback in `sobject_describe.go:414-462`. | Medium (1-2 weeks) | Medium — core resolution change |
| **Phase 2: Better defaults** | Extract section templates from `standard_layouts.json` (before deleting). Add `standardObjectSections` to `generateSmartLayout()` for 9 standard objects. Bump `currentGenerationVersion` to trigger regeneration. | Medium (3-5 days) | Low — existing algorithm is fallback |
| **Phase 3: Version history** | Write-on-save to history table. Restore endpoint. Retention pruning. | Small (2-3 days) | Low — append-only |
| **Phase 4: Role-based resolution** | Integrate Role Hierarchy ancestor traversal in `ResolveLayout()`. Build admin UI for creating role-based assignments. | Medium (1-2 weeks) | Low — assignment table already supports it |
| **Phase 5: Assignment matrix UI** | Build Profile × RecordType × Role matrix UI. Assignment write authorization (role-subtree scoping). | Medium (1-2 weeks) | Low — UI only |

Phase 1 is the architectural foundation + cleanup. Phase 2 improves layout quality. Phase 3 adds safety. Phase 4-5 unlock "千人千面."

### 7.16 Data vs Code Classification (Final)

| Component | Classification | Rationale |
|-----------|---------------|-----------|
| `generateSmartLayout()` + `standardObjectSections` | **CODE** | Algorithm + classification rules. Ships with every instance. |
| `ResolveLayout()` engine | **CODE** | Resolution logic. Pure function. |
| `MigrateLayoutSystem()` | **CODE** | One-time migration. Idempotent. |
| `_System_PageLayout` rows (Source=auto_generated) | **DATA** (system-generated) | Runtime artifact of auto-generation. |
| `_System_PageLayout` rows (Source=admin_created) | **DATA** (admin-managed) | Created via editor UI. Never auto-overwritten. |
| `_System_LayoutAssignment` rows | **DATA** (admin-managed) | Routing rules. Org-specific. |
| `_System_PageLayoutHistory` rows | **DATA** (audit trail) | Content snapshots for undo/rollback. |

**LLM/tester classification guide**:
- "Layout sections have wrong names for Account" → **CODE** (fix `standardObjectSections` template)
- "NA should see different layout than APAC" → **DATA** (create LayoutAssignment with RoleId)
- "New field doesn't appear on layout" → **CODE** (bug in `addFieldToLayout`)
- "Layout editor save doesn't persist" → **CODE** (bug in save handler)
- "Account layout should have TiDB Cloud section" → **DATA** (admin uses editor, or create data-spec)
- "Admin edited layout and wants to undo" → **DATA** (restore from `_System_PageLayoutHistory`)

### 7.17 Adding a New Dimension (Checklist)

When a future requirement demands a new routing dimension (e.g., AppId, TeamId):

1. **Schema**: Add nullable `VARCHAR(255)` column to `_System_LayoutAssignment`. Update `Specificity` generated column expression to include `+ (NewDimId IS NOT NULL)`. Add COALESCE sentinel column + update UNIQUE index.
2. **Migration**: Create data-spec with `ALTER TABLE`. Existing rows get NULL (wildcard) — preserves current behavior.
3. **Bootstrap**: Add column to `system_tables.json` table definition.
4. **Resolution SQL**: Add `AND (la.NewDimId IS NULL OR la.NewDimId = :userNewDimId)` to `ResolveLayout()`.
5. **Tie-breaking**: Decide where the new dimension ranks in the ORDER BY priority.
6. **User context**: Ensure `UserSession` includes the dimension value. Add to JWT or fetch from User record.
7. **API**: Update assignment creation endpoint to accept the new dimension.
8. **Admin UI**: Add the dimension as a filter/column in the assignment management UI.
9. **Tests**: Add resolution test with two assignments differing only in the new dimension.
10. **Docs**: Update this design document and the `data-vs-code` skill.

---

## 8. Resolved Decisions

| # | Question | Decision |
|---|----------|----------|
| 1 | Region/Team dimension: Role Hierarchy or LayoutGroupId? | **Role Hierarchy** — avoids parallel org structure, inheritance is automatic |
| 2 | Replace IsCustomized with what? | **Source VARCHAR(20) + GenerationVersion INT** — more expressive, same protection |
| 3 | Tie-breaking priority? | **RecordTypeId > ProfileId > RoleId > CreatedDate** |
| 4 | Role depth in resolution? | **Defer** — same-specificity resolved by CreatedDate; additive if needed later |
| 5 | UNIQUE constraint approach? | **COALESCE sentinel generated columns** — handles TiDB NULL semantics |
| 6 | Specificity: computed or application? | **GENERATED STORED column** — DB guarantees correctness |
| 7 | Auto-regeneration strategy? | **In-place UPDATE** + FK CASCADE as safety net |
| 8 | Role ancestors: session or per-request? | **Per-request** from in-memory PermissionService cache |
| 9 | No-Role user behavior? | **`AND la.RoleId IS NULL`** — only matches wildcard assignments |
| 10 | Eager or lazy bootstrap? | **Lazy** — auto-generate on first access, persist, done |
| 11 | Extract from standard_layouts.json? | **Yes** — extract section templates before deleting |
| 12 | Version history retention? | **20 versions** per layout, configurable |
| 13 | Assignment write authorization? | **Defer to Phase 5** — role-subtree scoping |
| 14 | Compact layout independence? | **Defer** — keep embedded in PageLayout JSON |
| 15 | Migration strategy? | **One-step idempotent migration** in bootstrap (0 rows in ProfileLayout) |
| 16 | Edit form required/read-only? | **Handled by existing field metadata** — `field.required` and `field.updateable` from describe API drive form rendering. Not a layout-layer concern. |
| 17 | Layout create/clone → assignment workflow? | **Separate operations** — creating a layout does NOT auto-create an assignment. Admin must explicitly assign. UI may offer a "create and assign" shortcut. |
| 18 | Default layout deletion protection? | **Reject deletion** if layout is the target of the only specificity-0 assignment for that object. |
| 19 | Assignment deletion API? | **`DELETE /ui-api/layout-assignment/:id`** — admin can remove an assignment. Users affected fall back to next-best match or auto-generate. |
| 20 | Cache invalidation after admin save? | **Evict `uiapi.layoutCache` in `SaveLayout`** — admin sees changes immediately. |

---

## Appendix: File References

| File | Purpose |
|------|---------|
| `backend/internal/bootstrap/system_tables.json:2128-2206` | `_System_PageLayout` table definition |
| `backend/internal/bootstrap/standard_layouts.json` | Dead code: 9 curated layouts |
| `backend/internal/bootstrap/layouts.go:42-53` | `CleanupHardcodedLayouts()` |
| `backend/cmd/server/main.go:154-162` | Bootstrap startup layout steps |
| `backend/internal/features/metadata/metadata_layouts.go:28-126` | GetLayout 3-tier resolution |
| `backend/internal/features/metadata/metadata_layouts.go:200-211` | SaveLayout + MarkLayoutCustomized |
| `backend/internal/features/metadata/metadata_layouts.go:238-271` | addFieldToLayout (only first layout) |
| `backend/internal/features/metadata/metadata_layouts.go:442-588` | generateSmartLayout algorithm |
| `backend/internal/features/metadata/metadata_layouts.go:628-681` | buildCompactLayout |
| `backend/internal/infrastructure/persistence/repo_layout.go:17-236` | Repository CRUD operations |
| `backend/internal/interfaces/rest/ui_api_handler.go:121-319` | UI API layout handlers |
| `backend/internal/interfaces/rest/sobject_describe.go:309-564` | SF REST layout endpoints |
| `backend/internal/features/uiapi/primitives.go:146-319` | Layout transform to SF wire format |
| `backend/internal/features/uiapi/aggregator.go:17-97` | RecordUI parallel aggregator |
| `shared/pkg/models/metadata.go:172-223` | Go model structs |
| `frontend/src/types/layout.ts:90-139` | Frontend TypeScript types |
| `frontend/src/pages/ObjectView.tsx:156-213` | Layout resolution in page component |
| `frontend/src/components/runtime/LayoutRenderer.tsx:24-273` | Runtime rendering engine |
| `frontend/src/features/record/components/MetadataRecordForm.tsx` | Form with optional layout |
| `frontend/src/features/admin/components/LayoutEditor.tsx:16-292` | Full layout editor |
| `frontend/src/features/studio/components/StudioLayoutEditor.tsx:23-368` | Studio layout editor |
| `frontend/src/core/hooks/useMetadata.ts:14-84` | useLayout, useLayouts hooks |
| `frontend/src/infrastructure/api/schemas.ts:408-521` | Frontend API client functions |
| `.agent/skills/data-vs-code/SKILL.md` | Data vs Code classification guide |
| `.agent/skills/data-change-spec/SKILL.md` | Data-spec format definition |