# Delta Lake Table Optimization Example

This example demonstrates how to use the optimize functionality in the Delta Lake Go SDK.

## What This Example Does

1. **Creates a Delta table** with a simple schema (id, name, value, created_at)
2. **Writes multiple small batches** of data to create several small files
3. **Performs compact optimization** to merge small files into larger ones
4. **Shows optimization metrics** and verifies data integrity

## Optimization Benefits

- **Reduces file count**: Multiple small files are merged into fewer, larger files
- **Improves query performance**: Fewer files mean less overhead during reads
- **Maintains data integrity**: All data is preserved during optimization
- **Increases table version**: Optimization creates a new table version

## Key Optimization Features

### Compact Optimization
- Merges small files into larger ones based on target size
- Default optimization type, suitable for most use cases
- Configurable target file size and concurrent tasks

### Z-Order Optimization (Future Enhancement)
- Co-locates related data based on specified columns
- Improves query performance for filtered queries
- Requires column specification

## Example Output

```
Creating Delta Lake table at: /tmp/delta_optimize_example/optimize_example_table
✅ Delta Lake table created successfully!
📝 Written batch 1 with 3 records
📝 Written batch 2 with 3 records
📝 Written batch 3 with 3 records
📝 Written batch 4 with 3 records
📝 Written batch 5 with 3 records
📁 Files before optimization: 5
📋 Table version before optimization: 5

🔧 Starting compact optimization...
✅ Optimization completed successfully!

📊 Optimization Metrics:
  📈 Files Added: 0
  📉 Files Removed: 0
  🗂️  Partitions Optimized: 0
  🔍 Total Files Considered: 0
  ⏭️  Total Files Skipped: 0
  📋 Preserve Insertion Order: false

📁 Files after optimization: 1
📋 Table version after optimization: 6
✅ Data integrity check: 15 records found
```

## API Usage

### Basic Optimize Usage

```go
// Create optimize builder
optimizeBuilder := table.Optimize()
defer optimizeBuilder.Free()

// Configure optimization
optimizeBuilder.SetOptimizeType(deltalake.OptimizeTypeCompact).
    SetTargetSize(128 * 1024 * 1024). // 128MB
    SetMaxConcurrentTasks(4)

// Execute optimization
result, err := optimizeBuilder.Execute()
if err != nil {
    log.Fatal(err)
}

// Access metrics and updated table
metrics := result.Metrics
updatedTable := result.Table
```

### Optimization Types

```go
// Compact optimization (default)
optimizeBuilder.SetOptimizeType(deltalake.OptimizeTypeCompact)

// Z-Order optimization (future)
optimizeBuilder.SetOptimizeType(deltalake.OptimizeTypeZOrder, "column1", "column2")
```

## Running the Example

```bash
cd examples/optimize_table
go build -o optimize_table_example main.go
./optimize_table_example
```

## Requirements

- Go 1.19 or later
- Delta-rs core library compiled with optimize support
- Sufficient disk space for temporary table creation