# backup-worker

A CLI tool for backing up TiDB Cloud observability data. Supports metric export, log replication, and Delta Lake table synchronization across multi-cloud environments.

## Features

| Feature | Description                                                                              |
|---------|------------------------------------------------------------------------------------------|
| **Three Backup Modes** | Metric export (VictoriaMetrics), Log replication, DeltaLake table sync                   |
| **Multi-Cloud Storage** | AWS S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS                      |
| **Resumable** | Checkpoint-based progress tracking; resumes from last completed shard after interruption |
| **Idempotent** | Object existence checks ensure safe re-runs                                              |
| **Concurrent** | Configurable parallelism for concurrent shard processing                                  |
| **Server-Side Copy** | Log and DeltaLake modes use cloud-native copy APIs (data never passes through local)     |

## Architecture

```mermaid
flowchart TB
    Config(["TOML Config"]) --> Controller["Controller"]
    Controller --> Mode{Mode}

    Mode -->|metric| MetricWorker["MetricWorker"]
    Mode -->|log/deltalake| FileWorker["FileWorker"]

    MetricWorker -->|export| VictoriaMetrics[("VictoriaMetrics")]
    MetricWorker -->|upload| Sink[("Cloud Storage")]

    FileWorker --> Syncer["Syncer"]
    Syncer -->|list/read| Source[("Source Storage")]
    Syncer -->|server-side copy| Dest[("Destination Storage")]

    Checkpoint[("Checkpoint File")] -.->|resume| Controller
```

### Components

- **Controller**: Parses configuration, creates appropriate Worker based on mode, coordinates shard scheduling
- **MetricWorker**: Exports metrics from VictoriaMetrics, writes time-sharded data to cloud storage
- **FileWorker**: Performs server-side copy via Syncer; supports same-account, cross-account, and cross-region scenarios
- **Syncer**: Wraps cloud provider server-side copy APIs (S3 CopyObject, GCS Rewrite, Azure CopyBlob, OSS CopyObject)
- **Checkpoint**: Local file tracking completed shards for resume capability

## Quick Start

### 1. Install

```bash
go build -o backup-worker .
```

### 2. Configure

Copy the example configuration and modify:

```bash
cp config.example.toml config.toml
```

### 3. Run

```bash
./backup-worker -config config.toml
```

## Backup Modes

### Metric Mode

Export time-series data from VictoriaMetrics.

```toml
mode = "metric"

[sources.victoriametrics]
endpoint = "http://victoriametrics:8428"
timeInterval = "1h"
match = '{__name__=~".*"}'
startTime = 1774799588
endTime = 1774842788

[sinks.s3]  # Supported: s3, gcs, azblob, oss (see config.example.toml)
bucket = "my-backup-bucket"
region = "us-west-2"
pathPrefix = "metrics/2024"
```

**Sharding Strategy**: Splits `[startTime, endTime]` into intervals based on `timeInterval`. Each interval is exported as a separate object.

> [!NOTE]
> For log/deltalake mode, the syncer also supports all storage providers (`s3`, `gcs`, `azblob`, `oss`). See `config.example.toml` for full configuration options.

### Log Mode

Replicate TiDB component logs, sharded by hour.

```toml
mode = "log"

[syncers.s3]
clusterId = "cluster1"
components = ["tidb", "tikv", "pd"]
startTime = 1753779580
endTime = 1753782018

[syncers.s3.source]
bucket = "source-bucket"

[syncers.s3.dest]
bucket = "dest-bucket"
pathPrefix = "diagnosis/data-backup"
```

**Path Mapping**:
- Source: `diagnosis/data/{clusterId}/merged-logs/{YYYYMMDDHH}/{component}/`
- Dest: `{pathPrefix}/{clusterId}/{YYYYMMDDHH}/{component}/`

### DeltaLake Mode

Replicate Delta Lake tables, sharded by day.

```toml
mode = "deltalake"

[syncers.azblob]
clusterId = "cluster1"
orgId = "org1"
components = ["slowlogs"]
startTime = 1753779580
endTime = 1753782018

[syncers.azblob.source]
container = "source-container"

[syncers.azblob.dest]
container = "dest-container"
pathPrefix = "deltalake/data-backup"
```

**Path Mapping**:
- Source: `deltalake/{orgId}/{clusterId}/{component}/date={YYYYMMDD}/`
- Dest: `{pathPrefix}/{orgId}/{clusterId}/{component}/date={YYYYMMDD}/`

## Authentication

### AWS S3

- **EKS IRSA**: Configure `roleArn` for WebIdentity token authentication
- **Static Credentials**: Set `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` environment variables

### Azure Blob

- **Connection String**: `connectionString = "DefaultEndpointsProtocol=https;AccountName=..."`
- **Service Principal**: `credentialsFile = "/path/to/credentials.json"`
- **Managed Identity**: Leave empty to use DefaultAzureCredential

### Google Cloud Storage

- **Service Account**: `credentialsFile = "/path/to/service-account.json"`
- **Workload Identity**: Runs in GKE, automatically uses Workload Identity

### Alibaba Cloud OSS

- **Access Keys**: Configure `accessKeyId` + `accessKeySecret`
- **STS Temporary Credentials**: Additionally configure `securityToken`

## Advanced Configuration

### Concurrency Control

```toml
[execution]
concurrency = 2  # Number of shards to process concurrently
```

### Retry Policy

```toml
[retryPolicy]
retryMaxAttempts = 3
retryInterval = "5s"
```

### Checkpoint

```toml
[checkpoint]
path = "/tmp/checkpoints"  # Directory for checkpoint files
```