# Open Delta Table Example This example demonstrates how to open an existing Delta table from S3 and retrieve its metadata using the Delta Lake Go bindings. ## Prerequisites 1. AWS credentials configured (using AWS SSO or environment variables) 2. Access to an S3 bucket with Delta tables 3. Go 1.24+ installed ## Environment Variables - `AWS_PROFILE`: AWS profile name (default: "ob-dev") - `S3_BUCKET`: S3 bucket name (default: "o11y-dev-shared-us-west-2") ## Running the Example ```bash # Set your AWS profile and S3 bucket export AWS_PROFILE="your-profile" export S3_BUCKET="your-bucket" # Run the example go run main.go ``` ## What This Example Does 1. **AWS Authentication**: Retrieves AWS credentials from the specified profile using AWS SDK v2 2. **Open Delta Table**: Opens an existing Delta table from S3 using storage options 3. **Table Version**: Retrieves the current version of the Delta table 4. **List Files**: Gets a list of all files in the Delta table 5. **Display Results**: Shows table metadata and file information ## Expected Output ``` Delta Lake Go Binding - Open Table Example Using AWS Profile: your-profile Target S3 Bucket: your-bucket Retrieving AWS credentials from profile... Successfully retrieved credentials for AWS Account: 123456789012 User ARN: arn:aws:sts::123456789012:assumed-role/YourRole/session AWS Region: us-west-2 Opening Delta table from S3... Table Path: s3://your-bucket/jinweitest/delta-tables/products Storage Options: AWS_ACCESS_KEY_ID: AKIA... AWS_SECRET_ACCESS_KEY: ***hidden*** AWS_SESSION_TOKEN: ***hidden*** AWS_REGION: us-west-2 AWS_S3_ALLOW_UNSAFE_RENAME: true Attempting to open Delta table... ✅ Successfully opened Delta table from S3! Getting table version... Table version: 0 Getting table files... Number of files: 3 Sample files: - part-00000-uuid.snappy.parquet - part-00001-uuid.snappy.parquet - _delta_log/00000000000000000000.json ✅ Successfully demonstrated Delta table operations! 📝 Table opened, version retrieved, and files listed from S3 ``` ## API Usage This example demonstrates the following Delta Lake Go binding functions: - `deltalake.OpenDeltaLakeTable(tablePath, storageOptions)`: Open an existing Delta table - `table.Version()`: Get the current table version - `table.GetFiles()`: List all files in the table - `table.Close()`: Clean up table resources ## Integration Notes To integrate this functionality into your application: 1. Import the deltalake package 2. Set up AWS credentials (environment variables, IAM roles, or AWS profiles) 3. Create storage options map with your AWS credentials 4. Use `OpenDeltaLakeTable` to access existing Delta tables 5. Call `Version()` and `GetFiles()` to retrieve table metadata deltalake "path/to/deltalake" ) func main() { // 打开本地 Delta 表 table, err := deltalake.OpenDeltaLakeTable("/path/to/delta/table", nil) if err != nil { log.Fatal(err) } defer table.Close() // 获取表版本 version, err := table.Version() if err != nil { log.Fatal(err) } fmt.Printf("Table version: %d\n", version) // 获取表文件 files, err := table.GetFiles() if err != nil { log.Fatal(err) } fmt.Printf("Table has %d files\n", len(files)) } ``` ### S3 存储使用示例 ```go package main import ( "fmt" "log" deltalake "path/to/deltalake" ) func main() { // 配置 S3 存储选项 storageOptions := map[string]string{ "AWS_ACCESS_KEY_ID": "your-access-key", "AWS_SECRET_ACCESS_KEY": "your-secret-key", "AWS_REGION": "us-west-2", // 如果使用临时凭证，还需要添加： // "AWS_SESSION_TOKEN": "your-session-token", } // 打开 S3 上的 Delta 表 table, err := deltalake.OpenDeltaLakeTable("s3://bucket/path/to/table", storageOptions) if err != nil { log.Fatal(err) } defer table.Close() // 获取表版本 version, err := table.Version() if err != nil { log.Fatal(err) } fmt.Printf("Table version: %d\n", version) // 获取表文件 files, err := table.GetFiles() if err != nil { log.Fatal(err) } fmt.Printf("Table has %d files:\n", len(files)) for i, file := range files { if i < 5 { // 只显示前5个文件 fmt.Printf(" %s\n", file) } else if i == 5 { fmt.Printf(" ... and %d more files\n", len(files)-5) break } } } ``` ### AWS SSO 集成示例 如果你使用 AWS SSO，可以结合 AWS SDK v2 来获取凭证： ```go package main import ( "context" "fmt" "log" "github.com/aws/aws-sdk-go-v2/config" deltalake "path/to/deltalake" ) func getAWSCredentials(ctx context.Context, profileName string) (map[string]string, error) { cfg, err := config.LoadDefaultConfig(ctx, config.WithSharedConfigProfile(profileName)) if err != nil { return nil, err } credentials, err := cfg.Credentials.Retrieve(ctx) if err != nil { return nil, err } storageOptions := map[string]string{ "AWS_ACCESS_KEY_ID": credentials.AccessKeyID, "AWS_SECRET_ACCESS_KEY": credentials.SecretAccessKey, "AWS_REGION": cfg.Region, } if credentials.SessionToken != "" { storageOptions["AWS_SESSION_TOKEN"] = credentials.SessionToken } return storageOptions, nil } func main() { ctx := context.Background() // 从 SSO profile 获取凭证 storageOptions, err := getAWSCredentials(ctx, "your-sso-profile") if err != nil { log.Fatal(err) } // 打开 Delta 表 table, err := deltalake.OpenDeltaLakeTable("s3://bucket/path/to/table", storageOptions) if err != nil { log.Fatal(err) } defer table.Close() // 使用表... } ``` ## 支持的存储选项 ### AWS S3 - `AWS_ACCESS_KEY_ID`: AWS 访问密钥 ID - `AWS_SECRET_ACCESS_KEY`: AWS 秘密访问密钥 - `AWS_SESSION_TOKEN`: AWS 会话令牌（临时凭证时需要） - `AWS_REGION`: AWS 区域 ### Azure Blob Storage - `AZURE_STORAGE_ACCOUNT_NAME`: 存储账户名称 - `AZURE_STORAGE_ACCOUNT_KEY`: 存储账户密钥 - `AZURE_STORAGE_SAS_TOKEN`: SAS 令牌 ### Google Cloud Storage - `GOOGLE_SERVICE_ACCOUNT`: 服务账户 JSON 密钥文件路径 - `GOOGLE_SERVICE_ACCOUNT_KEY`: 服务账户 JSON 密钥内容 ## 编译和运行 1. 确保已经编译了 Rust 核心库： ```bash cd /path/to/delta-rs/golang cargo build --release ``` 2. 编译 Go 代码： ```bash go build -o example main.go ``` 3. 运行示例： ```bash # 设置环境变量 export DELTA_TABLE_PATH="s3://your-bucket/path/to/table" export AWS_PROFILE="your-sso-profile" # 运行示例 ./example ``` ## 错误处理 所有函数都返回适当的错误信息。常见错误类型包括： - 表路径不存在 - 权限不足 - 网络连接问题 - 凭证无效 - 表格式错误 建议在生产代码中实现适当的错误处理和重试逻辑。 ## 性能考虑 - `Version()` 方法通常很快，因为它只需要读取元数据 - `GetFiles()` 方法可能较慢，特别是对于包含大量文件的表 - 使用 `defer table.Close()` 确保资源被正确释放 - 对于频繁操作，考虑重用表句柄而不是重复打开 ## 兼容性 此功能与 delta-rs 0.28.0 兼容，支持 Delta Lake 协议的所有版本。