# VictoriaMetrics Backup Manager

A service for automating and managing VictoriaMetrics backups.

## Features

- **Automated Backup Scheduling**
  - Incremental backups performed every 10 minutes by default
  - Full backups performed every VM retention period (3 months by default)
  - Configurable schedule for both backup types using cron expressions
  - **Coordinated backups**: Incremental backups are automatically skipped when a full backup is running

- **HTTP API**
  - List all backups
  - Get details of a specific backup
  - Trigger an immediate full backup
  - Health check endpoint
  - Prometheus metrics endpoint

- **Snapshot Management**
  - Automatic snapshot creation and cleanup
  - Handles cleanup of failed backups and leftover snapshots

- **Backup Retention Management**
  - Configurable backup retention period (13 months by default)
  - Automatic cleanup of expired backup data
  - Can be disabled with `-disableBackupRetention` flag

- **Backup Migration Support**
  - Support for migrating from old directory structures
  - Server-side copy for efficient migration

- **Backup Failure Handling**
  - Automatic retry with configurable attempts
  - Leftover snapshot files cleanup
  - Resuming backup after restart
  - Queue-based task management

- **Monitoring and Metrics**
  - Prometheus metrics for backup status and operations
  - Structured JSON logging with configurable levels

## Installation

### Building from Source

```bash
git clone https://github.com/tidbcloud/observability.git
cd observability/vmbackupmanager
go build -o vmbackupmanager .
```

## Configuration

### Command-Line Flags

| Flag | Description | Default |
|------|-------------|---------|
| `-storageDataPath` | Path to VictoriaMetrics data | `victoria-metrics-data` |
| `-snapshot.createURL` | VictoriaMetrics create snapshot URL | (required) |
| `-dst` | Backup destination | (required) |
| `-concurrency` | Number of concurrent workers for incremental backups | `10` |
| `-fullBackupConcurrency` | Number of concurrent workers for full backups | `100` |
| `-maxBytesPerSecond` | Maximum upload speed | `0` (no limit) |
| `-httpListenAddr` | HTTP API listen address | `:8825` |
| `-incrementalBackupSchedule` | Cron expression for incremental backups | `10 * * * *` (every 10 minutes) |
| `-fullBackupSchedule` | Cron expression for full backups | "" (uses VM retention period) |
| `-backupRetention` | Time to keep full backups in remote storage | `13` days |
| `-vmRetentionPeriod` | VM retention period for full backups | `3` days |
| `-vmRetentionTimeZoneOffset` | VM retention period time zone offset | `0` |
| `-disableFullBackup` | Disable full backups | `false` |
| `-disableBackupRetention` | Disable backup retention check | `false` |
| `-migrate` | Migrate backups to latest folder using server-side copy | `false` |
| `-logLevel` | Log level (debug, info, warn, error) | `info` |
| `-version` | Print version information and exit | `false` |

## API Endpoints

### List All Backups

```
GET /api/v1/backups
```

Response:
```json
{
  "backups": [
    {
      "id": "incremental-20240601-120000",
      "type": "incremental",
      "dest_path": "s3://your-bucket/backups/latest/incremental-20240601-120000",
      "created_at": "2024-06-01T12:00:00Z",
      "started_at": "2024-06-01T12:00:05Z",
      "completed_at": "2024-06-01T12:05:00Z",
      "status": "completed",
      "data_retention": "72h0m0s"
    }
  ],
  "count": 1
}
```

### Get Backup Details

```
GET /api/v1/backups/{id}
```

Response:
```json
{
  "message": "backup details",
  "backup": {
    "id": "incremental-20240601-120000",
    "type": "incremental",
    "dest_path": "s3://your-bucket/backups/latest/incremental-20240601-120000",
    "created_at": "2024-06-01T12:00:00Z",
    "scheduled_at": "2024-06-01T12:00:00Z",
    "started_at": "2024-06-01T12:00:05Z",
    "completed_at": "2024-06-01T12:05:00Z",
    "status": "completed",
    "data_retention": "72h0m0s"
  }
}
```

### Trigger a Full Backup

```
POST /api/v1/backups
```

Response:
```json
{
  "message": "backup created",
  "backup": {
    "id": "full-20240601-120000",
    "type": "full",
    "dest_path": "s3://your-bucket/backups/full/full-20240601-120000",
    "created_at": "2024-06-01T12:00:00Z",
    "status": "pending",
    "data_retention": "72h0m0s"
  }
}
```

> **Note**: Attempting to start a backup while another backup is in progress will result in a 409 Conflict error.

### Health Check

```
GET /health
```

Response:
```json
{
  "status": "ok"
}
```

### Metrics

```
GET /metrics
```

Returns Prometheus metrics including backup status and operation metrics.

## Backup Coordination

The backup manager includes intelligent coordination between incremental and full backups:

- **Scheduled incremental backups** (every 10 minutes by default) are automatically skipped when a full backup is running.
- **Manual backups** triggered via the API will be rejected with a 409 Conflict status if another backup is running.
- Only one backup can run at a time to prevent resource contention and ensure backup integrity.
- Failed backups are automatically retried up to 2 times with a 30-second delay between attempts.

## Backup Types and Statuses

### Backup Types
- **incremental**: Incremental backup that only backs up changed data
- **full**: Full backup that backs up all data
- **migrated**: Backup that has been migrated from an old directory structure

### Backup Statuses
- **pending**: Backup is queued and waiting to start
- **running**: Backup is currently in progress
- **completed**: Backup completed successfully
- **failed**: Backup failed after all retry attempts
- **deleting**: Backup is being deleted due to retention policy

## Storage Path Structure

```
destination/
├── full/                    # Full backups
│   ├── full-20240601-120000/
│   └── full-20240602-120000/
├── latest/                  # Incremental backups
│   ├── incremental-20240601-120000/
│   └── incremental-20240601-121000/
└── backupmgr_metadata/      # Backup metadata files
    ├── full-20240601-120000.json
    └── incremental-20240601-120000.json
```

## Monitoring

The service exposes Prometheus metrics at `/metrics` endpoint, including:
- Backup status metrics with labels for ID, destination, type, timestamps, retention, and status
- Operation metrics for backup operations

Logs are structured in JSON format with configurable log levels.

## Dev Guide

### Local Testing

1. Download and Start VictoriaMetrics (single-node version recommended)
2. Download the `vmbackup` binary and put it into your PATH
3. Test with local filesystem:

```bash
bin/vmbackupmanager \
  -storageDataPath=/path/to/victoria-metrics-data \
  -snapshot.createURL=http://127.0.0.1:8428/snapshot/create \
  -dst=fs:///tmp/backup/
```

### Migration

To migrate existing backups to the new structure:

```bash
bin/vmbackupmanager \
  -storageDataPath=/path/to/victoria-metrics-data \
  -snapshot.createURL=http://127.0.0.1:8428/snapshot/create \
  -dst=s3://your-bucket/backups \
  -migrate=true
```

This will use server-side copy to efficiently migrate backups to the latest folder structure.