# Load Modeling

## Goal

Model data for future load simulation in a way that preserves workload behavior, not just row counts.

Priority order:

1. Joinability
2. Filter selectivity
3. Group or hotspot distribution
4. History depth
5. File layout and importability
6. Only then total volume

## Determine the Driving Dimension

Before choosing any tenant-like model, decide whether the workload actually has a grouping dimension.

Sometimes there is no stable grouping column at all. In that case, model skew directly on:

- time ranges
- status buckets
- owner or assignee pools
- hot IDs

Use these sources together:

- schema
- SQL
- plans or statistics
- user-provided business context

If the user explicitly corrects your assumption, override your heuristic.

Business context is not optional decoration. Use it to infer:

- whether the workload is consumer, enterprise, internal, or partner-facing
- whether IDs and names should look transactional, conversational, financial, document-centric, etc.
- whether hotspots should come from a few very active actors or broad background traffic
- whether optional JSON or metadata fields should be sparse or frequently present

## Multi-tenant or Grouped Modeling

Do not default to a tiny fixed group set.

Also do not default to a flat average such as:

- `distinct group count = total rows / constant`

Instead, use size buckets only when the evidence supports a grouped model.

Recommended generic buckets:

- `enterprise`
- `large`
- `medium`
- `small`
- `micro`

For each bucket define:

- percentage of total rows
- approximate rows per group
- resulting group count

Example reasoning:

- very large groups own a large share of rows
- tiny groups are numerous but individually small
- large and medium tenants create the long tail between those extremes

The exact ratios should be driven by:

- schema semantics
- workload SQL
- observed plans or statistics
- user expectations
- customer background and product behavior

not copied from a previous case.

## Join-heavy Table Families

When several tables form a chain, for example:

- `contacts`
- `inbox_conversations`
- `dialogs`
- `inbox_conversation_queue`

prefer generating each table independently from the same deterministic mapping formula:

- group bucket
- group index
- group-local ordinal
- stable contact id / conversation id formulas

This avoids the file-layout distortion that can happen with deep chain generation.

## Queue / History Modeling

History tables should not be generated only from the first N parent rows unless the workload truly behaves that way.

Instead, sample or spread them across the whole parent population.

Examples:

- queue rows should span hot, warm, and tail groups when grouped
- history depth can vary by group size bucket
- some rows may have one history record, others two or three

## Assignee Modeling

Avoid a tiny global assignee pool when the workload evidence implies many owners or operators.

Better defaults:

- very large groups:
  large assignee pool
- medium groups:
  medium assignee pool
- small or tiny groups:
  small assignee pool, sometimes `NULL`

Keep these fields aligned:

- `assignee_id`
- `first_response_operator_id`
- `first_response_operator_name`
- `team_ids`

## JSON Modeling

Classify JSON columns before generating.

### Class A: usually populated

Examples:

- `channels`
- `team_ids`

These should rarely be empty unless the schema or workload implies optionality.

### Class B: frequently populated

Examples:

- `tags`
- `tag_ids`

Use realistic partial fill rates such as 20% to 60%, depending on table role and group size when relevant.

### Class C: sparse enrichment

Examples:

- `custom_attrs`
- `additional_attributes`

Use lower fill rates but still make them present often enough to exercise storage and query behavior.

## File Layout

For large generation jobs:

- keep one root table per template where practical
- generate each table to its own predictable shard set
- verify row counts per file after generation

This matters because a mathematically correct total row count can still produce a structurally bad dataset if all child rows pile into one or two files.

## Practical Checks

## Default Scale

If the user does not provide a size target, pick a medium scale by default.

Medium scale should be:

- big enough to exercise representative joins and skew
- small enough to regenerate and re-import during iteration
- appropriate for the complexity of the workload rather than a fixed universal number

Before import:

- count rows per file
- count distinct grouping keys in CSV when a grouping dimension exists
- inspect top groups by row count
- inspect representative JSON fill rates

After import:

- count rows per table
- count distinct grouping keys when relevant
- inspect representative joins
- inspect distribution sanity:
  top groups, assignee diversity, JSON population