# Manual Backup Metering Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Support `manual-backup` metering records and bill them as GB under `backup-storage:{provider}:{region}:{plan}:manual`.

**Architecture:** Add manual backup as a separate nextgen category with a dedicated SKU generator and premium/reserved handler registration. Keep existing auto backup `auto` and `T1/T2` paths unchanged, and update final billing conversion so only `backup-storage:*:*:*:manual` bypasses bytes-to-MiB conversion.

**Tech Stack:** Go, `testing`, `testify`, `metering_sdk/common`, existing nextgen metering registry and SKU generator.

---

## File Structure

- Modify `pkg/common/models/nextgenmeteringrule.go`: add constants for `manual-backup`, `backup_size_gb`, `backupID`, and `manual`.
- Modify `pkg/common/models/billing.go`: detect manual backup storage SKUs and skip bytes-to-MiB conversion for them.
- Modify `pkg/common/models/billing_test.go`: add reporter conversion tests for manual backup GB billing.
- Modify `pkg/nextgenmetering/sku_rules.go`: add `GenerateManualBackupStorageSKUs`.
- Modify `pkg/nextgenmetering/sku_rules_test.go`: add generator tests for valid, zero, missing, and malformed manual backup rows.
- Modify `pkg/nextgenmetering/premium_handler.go`: add a manual backup handler that delegates to the new generator.
- Modify `pkg/nextgenmetering/sku_registry.go`: register manual backup for premium and reserved plans only.
- Create `pkg/nextgenmetering/sku_registry_test.go`: assert plan/category registration behavior.
- Modify `pkg/nextgenmetering/metering_test.go`: add tests for `generateSKURecords` and `generateBillingRecords` with real metadata.

## Task 1: Preserve GB Billing For Manual Backup Reporter Conversion

**Files:**
- Modify: `pkg/common/models/billing_test.go`
- Modify: `pkg/common/models/billing.go`

- [ ] **Step 1: Write the failing reporter conversion test**

Append this test to `pkg/common/models/billing_test.go` after `TestBillingReportV2Deltalake_ConvertBytesSkusToMiB`:

```go
func TestBillingReportV2Deltalake_ConvertManualBackupStorageAsGB(t *testing.T) {
	report := BillingReportV2Deltalake{
		SchemeVersion: "v2",
		TenantID:      12345,
		CostID:        "manual-backup-cost",
		Cost:          2,
		CostTimestamp: 1696543200,
		Unit:          NextGenSKUBackupStorage + ":aws:ap-south-1:premium:manual",
		UnitValue:     "128",
		Labels:        `{"sourceUnit":"gb","backupType":"manual","backupID":"backup-1"}`,
	}

	billingReport, err := report.Convert2BillingReport()

	require.NoError(t, err)
	assert.Equal(t, int64(256), billingReport.Cost)
	assert.Equal(t, report.Unit, billingReport.Unit)
	assert.Equal(t, report.Labels, billingReport.Labels)

	var labels map[string]string
	require.NoError(t, json.Unmarshal([]byte(billingReport.Labels), &labels))
	assert.Equal(t, "gb", labels[NextGenSKULabelKeySourceUnit])
	assert.Equal(t, "manual", labels[NextGenSKULabelKeyBackupType])
	assert.NotContains(t, labels, NextGenSKULabelKeyOriginalBytes)
}
```

- [ ] **Step 2: Run test to verify it fails**

Run:

```bash
go test ./pkg/common/models -run TestBillingReportV2Deltalake_ConvertManualBackupStorageAsGB -count=1
```

Expected: FAIL because current `requiresMiBConversion` treats every `backup-storage` unit as bytes and returns cost `2` with an `originalBytes` label.

- [ ] **Step 3: Implement manual backup SKU detection**

In `pkg/common/models/billing.go`, replace the existing `requiresMiBConversion` function with:

```go
func requiresMiBConversion(unit string) bool {
	return strings.HasPrefix(unit, NextGenSKUDataTransferOut) ||
		strings.HasPrefix(unit, NextGenSKUDataTransferCrossAZ) ||
		(strings.HasPrefix(unit, NextGenSKUBackupStorage) && !isManualBackupStorageSKU(unit)) ||
		strings.HasPrefix(unit, NextGenSKUDiskStorageSize) ||
		strings.HasPrefix(unit, NextGenSKUServerlessRowStorage) ||
		strings.HasPrefix(unit, NextGenSKUServerlessColumnStorage) ||
		strings.HasPrefix(unit, NextGenSKUServerlessRowStorageDualEncryption) ||
		strings.HasPrefix(unit, NextGenSKUServerlessColumnStorageDualEncryption)
}

func isManualBackupStorageSKU(unit string) bool {
	parts := strings.Split(unit, ":")
	return len(parts) == 5 &&
		parts[0] == NextGenSKUBackupStorage &&
		parts[4] == NextGenBackupTypeManual
}
```

In `pkg/common/models/nextgenmeteringrule.go`, add these constants:

```go
const (
	NextGenBackupTypeAuto   = "auto"
	NextGenBackupTypeManual = "manual"
)
```

- [ ] **Step 4: Run test to verify it passes**

Run:

```bash
go test ./pkg/common/models -run 'TestBillingReportV2Deltalake_ConvertManualBackupStorageAsGB|TestBillingReportV2Deltalake_ConvertBytesSkusToMiB' -count=1
```

Expected: PASS. The existing bytes-to-MiB conversion test must still pass, including the existing `backup-storage` minimum MiB case.

- [ ] **Step 5: Commit**

```bash
git add pkg/common/models/billing.go pkg/common/models/billing_test.go pkg/common/models/nextgenmeteringrule.go
git commit -m "fix: preserve gb billing for manual backup storage"
```

## Task 2: Add Manual Backup SKU Generation

**Files:**
- Modify: `pkg/common/models/nextgenmeteringrule.go`
- Modify: `pkg/nextgenmetering/sku_rules_test.go`
- Modify: `pkg/nextgenmetering/sku_rules.go`

- [ ] **Step 1: Write failing SKU generator tests**

Append these tests to `pkg/nextgenmetering/sku_rules_test.go` after the existing `GenerateBackupStorageSKUs` tests:

```go
func TestGenerateManualBackupStorageSKUs(t *testing.T) {
	generator := newTestSKUGenerator(t)
	now := time.Unix(1700000, 0)

	t.Run("generates manual backup storage sku in gb", func(t *testing.T) {
		dataPoint := map[string]interface{}{
			models.NextGenMeteringLogicalClusterID: "cluster-1",
			models.NextGenMeteringBackupID:         "backup-1",
			models.NextGenMeteringBackupSizeGB: map[string]interface{}{
				"value": uint64(128),
				"unit":  "GB",
			},
		}

		records, err := generator.GenerateManualBackupStorageSKUs(
			dataPoint,
			"cluster-1",
			"shared-1",
			6000,
			1700000,
			"aws",
			"ap-south-1",
			"premium",
			map[string]interface{}{},
			now,
		)

		require.NoError(t, err)
		require.Len(t, records, 1)

		record := records[0]
		assert.Equal(t, "backup-storage:aws:ap-south-1:premium:manual", record.Unit)
		assert.Equal(t, "128", record.UnitValue)
		assert.Equal(t, int64(1), record.Cost)
		assert.Equal(t, models.NextGenSKUTypePublic, record.SKUType)
		assert.Equal(t, uint64(6000), record.TenantID)
		assert.Equal(t, int64(1700000), record.CostTimestamp)

		var labels map[string]string
		require.NoError(t, json.Unmarshal([]byte(record.Labels), &labels))
		assert.Equal(t, "gb", labels[models.NextGenSKULabelKeySourceUnit])
		assert.Equal(t, models.NextGenBackupTypeManual, labels[models.NextGenSKULabelKeyBackupType])
		assert.Equal(t, "backup-1", labels[models.NextGenSKULabelKeyBackupID])
		assert.Equal(t, "cluster-1", labels[models.NextGenSKULabelKeyClusterID])
		assert.Equal(t, "shared-1", labels[models.NextGenSKULabelKeySharedPoolID])
	})

	t.Run("skips zero backup size", func(t *testing.T) {
		dataPoint := map[string]interface{}{
			models.NextGenMeteringBackupID: "backup-zero",
			models.NextGenMeteringBackupSizeGB: map[string]interface{}{
				"value": uint64(0),
				"unit":  "GB",
			},
		}

		records, err := generator.GenerateManualBackupStorageSKUs(
			dataPoint,
			"cluster-1",
			"shared-1",
			6000,
			1700000,
			"aws",
			"ap-south-1",
			"premium",
			map[string]interface{}{},
			now,
		)

		require.NoError(t, err)
		assert.Empty(t, records)
	})

	t.Run("skips missing backup size", func(t *testing.T) {
		dataPoint := map[string]interface{}{
			models.NextGenMeteringBackupID: "backup-missing-size",
		}

		records, err := generator.GenerateManualBackupStorageSKUs(
			dataPoint,
			"cluster-1",
			"shared-1",
			6000,
			1700000,
			"aws",
			"ap-south-1",
			"premium",
			map[string]interface{}{},
			now,
		)

		require.NoError(t, err)
		assert.Empty(t, records)
	})

	t.Run("skips malformed backup size", func(t *testing.T) {
		dataPoint := map[string]interface{}{
			models.NextGenMeteringBackupID:     "backup-bad-size",
			models.NextGenMeteringBackupSizeGB: "128",
		}

		records, err := generator.GenerateManualBackupStorageSKUs(
			dataPoint,
			"cluster-1",
			"shared-1",
			6000,
			1700000,
			"aws",
			"ap-south-1",
			"premium",
			map[string]interface{}{},
			now,
		)

		require.NoError(t, err)
		assert.Empty(t, records)
	})
}
```

- [ ] **Step 2: Run tests to verify they fail**

Run:

```bash
go test ./pkg/nextgenmetering -run TestGenerateManualBackupStorageSKUs -count=1
```

Expected: FAIL because `NextGenMeteringBackupID`, `NextGenMeteringBackupSizeGB`, `NextGenSKULabelKeyBackupID`, `NextGenBackupTypeManual`, and `GenerateManualBackupStorageSKUs` do not exist yet.

- [ ] **Step 3: Add constants**

In `pkg/common/models/nextgenmeteringrule.go`, add these constants near the other metering field constants:

```go
NextGenMeteringBackupID     = "backup_id"
NextGenMeteringBackupSizeGB = "backup_size_gb"
```

Add this label constant near the other SKU label keys:

```go
NextGenSKULabelKeyBackupID = "backupID"
```

- [ ] **Step 4: Implement the generator**

Add this function to `pkg/nextgenmetering/sku_rules.go` near the existing backup storage generators:

```go
// GenerateManualBackupStorageSKUs generates manual backup storage SKUs from GB usage.
func (g *SKUGenerator) GenerateManualBackupStorageSKUs(dataPoint map[string]interface{}, clusterID, sharedPoolID string, tenantID uint64, timestamp int64, cloudProvider, region, plan string, enrichmentData map[string]interface{}, now time.Time) ([]models.BillingReportV2Deltalake, error) {
	var billingRecords []models.BillingReportV2Deltalake

	backupSizeGB, ok := g.ConvertToMeteringValue(dataPoint[models.NextGenMeteringBackupSizeGB])
	if !ok || backupSizeGB == nil || backupSizeGB.Value == 0 {
		return billingRecords, nil
	}

	skuUnit := fmt.Sprintf("%s:%s:%s:%s:%s", models.NextGenSKUBackupStorage, cloudProvider, region, plan, models.NextGenBackupTypeManual)
	unitValue := strconv.FormatUint(backupSizeGB.Value, 10)
	costID := uuid.New().String()
	sourceUnit := strings.ToLower(backupSizeGB.Unit)

	labelsMap := map[string]string{
		models.NextGenSKULabelKeySourceUnit: sourceUnit,
		models.NextGenSKULabelKeyBackupType: models.NextGenBackupTypeManual,
	}
	if backupID, ok := dataPoint[models.NextGenMeteringBackupID].(string); ok && strings.TrimSpace(backupID) != "" {
		labelsMap[models.NextGenSKULabelKeyBackupID] = backupID
	}
	if clusterID != "" && clusterID != models.NextGenMeteringSystemID {
		labelsMap[models.NextGenSKULabelKeyClusterID] = clusterID
	}
	if sharedPoolID != "" {
		labelsMap[models.NextGenSKULabelKeySharedPoolID] = sharedPoolID
	}

	labelsJSON, _ := json.Marshal(labelsMap)

	skuType := models.NextGenSKUTypePublic
	if clusterID == models.NextGenMeteringSystemID || clusterID == "" {
		skuType = models.NextGenSKUTypePrivate
	}

	record := models.BillingReportV2Deltalake{
		SchemeVersion: "v2",
		TenantID:      tenantID,
		CostID:        costID,
		CostTimestamp: timestamp,
		Unit:          skuUnit,
		UnitValue:     unitValue,
		Cost:          1,
		Labels:        string(labelsJSON),
		CreatedAt:     now.Unix(),
		SKUType:       skuType,
		HourDimension: utils.GenerateHourDimension(now.Unix()),
	}

	billingRecords = append(billingRecords, record)
	return billingRecords, nil
}
```

- [ ] **Step 5: Run tests to verify they pass**

Run:

```bash
go test ./pkg/nextgenmetering -run TestGenerateManualBackupStorageSKUs -count=1
```

Expected: PASS.

- [ ] **Step 6: Commit**

```bash
git add pkg/common/models/nextgenmeteringrule.go pkg/nextgenmetering/sku_rules.go pkg/nextgenmetering/sku_rules_test.go
git commit -m "feat: generate manual backup storage skus"
```

## Task 3: Register Manual Backup Category For Premium And Reserved

**Files:**
- Create: `pkg/nextgenmetering/sku_registry_test.go`
- Modify: `pkg/nextgenmetering/premium_handler.go`
- Modify: `pkg/nextgenmetering/sku_registry.go`

- [ ] **Step 1: Write failing registry tests**

Create `pkg/nextgenmetering/sku_registry_test.go`:

```go
package nextgenmetering

import (
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
	"github.com/tidbcloud/metering/pkg/common/models"
	"go.uber.org/zap"
)

func TestSKUHandlerRegistry_ManualBackup(t *testing.T) {
	registry := NewSKUHandlerRegistry(newTestSKUGenerator(t), zap.NewNop())

	t.Run("premium supports manual backup", func(t *testing.T) {
		handler, ok := registry.GetHandler(models.PlanPremium, models.NextGenCategoryManualBackup)
		require.True(t, ok)
		assert.IsType(t, &PremiumManualBackupHandler{}, handler)
	})

	t.Run("reserved supports manual backup", func(t *testing.T) {
		handler, ok := registry.GetHandler(models.PlanReserved, models.NextGenCategoryManualBackup)
		require.True(t, ok)
		assert.IsType(t, &PremiumManualBackupHandler{}, handler)
	})

	t.Run("essential does not support manual backup", func(t *testing.T) {
		_, ok := registry.GetHandler(models.PlanEssential, models.NextGenCategoryManualBackup)
		assert.False(t, ok)
	})

	t.Run("byoc does not support manual backup", func(t *testing.T) {
		_, ok := registry.GetHandler(models.PlanBYOC, models.NextGenCategoryManualBackup)
		assert.False(t, ok)
	})
}
```

- [ ] **Step 2: Run test to verify it fails**

Run:

```bash
go test ./pkg/nextgenmetering -run TestSKUHandlerRegistry_ManualBackup -count=1
```

Expected: FAIL because `NextGenCategoryManualBackup` and `PremiumManualBackupHandler` do not exist.

- [ ] **Step 3: Add category constant**

In `pkg/common/models/nextgenmeteringrule.go`, add this category constant:

```go
NextGenCategoryManualBackup = "manual-backup"
```

- [ ] **Step 4: Add the handler**

In `pkg/nextgenmetering/premium_handler.go`, add:

```go
type PremiumManualBackupHandler struct {
	gen *SKUGenerator
}

func NewPremiumManualBackupHandler(gen *SKUGenerator) *PremiumManualBackupHandler {
	return &PremiumManualBackupHandler{gen: gen}
}

func (h *PremiumManualBackupHandler) GenerateSKUs(dataPoint map[string]interface{}, params *GenerateSKUParams) ([]models.BillingReportV2Deltalake, error) {
	return h.gen.GenerateManualBackupStorageSKUs(
		dataPoint, params.ClusterID, params.SharedPoolID, params.TenantID, params.Timestamp,
		params.CloudProvider, params.Region, params.Plan,
		params.EnrichmentData, params.Now,
	)
}
```

- [ ] **Step 5: Register premium and reserved handlers**

In `pkg/nextgenmetering/sku_registry.go`, add `models.NextGenCategoryManualBackup: NewPremiumManualBackupHandler(gen),` to the handler maps in `registerPremiumHandlers` and `registerReservedHandlers`.

- [ ] **Step 6: Run test to verify it passes**

Run:

```bash
go test ./pkg/nextgenmetering -run TestSKUHandlerRegistry_ManualBackup -count=1
```

Expected: PASS.

- [ ] **Step 7: Commit**

```bash
git add pkg/common/models/nextgenmeteringrule.go pkg/nextgenmetering/premium_handler.go pkg/nextgenmetering/sku_registry.go pkg/nextgenmetering/sku_registry_test.go
git commit -m "feat: register manual backup metering handler"
```

## Task 4: Verify Manual Backup Through NextGen Metering Flow

**Files:**
- Modify: `pkg/nextgenmetering/metering_test.go`

- [ ] **Step 1: Write failing `generateSKURecords` test**

Append this subtest inside `TestNextGenMetering_GenerateSKURecords_DetailedCoverage` in `pkg/nextgenmetering/metering_test.go`:

```go
t.Run("manual backup category with gb size", func(t *testing.T) {
	dataPoint := map[string]interface{}{
		models.NextGenMeteringLogicalClusterID: "manual-backup-cluster",
		models.NextGenMeteringBackupID:         "backup-1",
		models.NextGenMeteringBackupSizeGB: map[string]interface{}{
			"value": uint64(64),
			"unit":  "GB",
		},
	}

	records, err := ng.generateSKURecords(
		dataPoint,
		models.NextGenCategoryManualBackup,
		"manual-backup-cluster",
		"manual-backup-pool",
		12345,
		timestamp,
		"aws",
		"ap-south-1",
		"premium",
		map[string]interface{}{},
		nil,
		now,
	)

	require.NoError(t, err)
	require.Len(t, records, 1)
	assert.Equal(t, "backup-storage:aws:ap-south-1:premium:manual", records[0].Unit)
	assert.Equal(t, "64", records[0].UnitValue)
})
```

- [ ] **Step 2: Run test to verify behavior**

Run:

```bash
go test ./pkg/nextgenmetering -run TestNextGenMetering_GenerateSKURecords_DetailedCoverage -count=1
```

Expected after Tasks 2 and 3: PASS.

- [ ] **Step 3: Write failing `generateBillingRecords` integration test**

Append this test to `pkg/nextgenmetering/metering_test.go`:

```go
func TestNextGenMetering_GenerateBillingRecords_ManualBackup(t *testing.T) {
	ng, writers := setupTestNextGenMeteringWithWriters(t)
	defer ng.Close()
	defer writers.Close()

	ctx := context.Background()
	timestamp := getMeteringUniqueTimestamp()
	clusterID := "manual-backup-cluster"
	sharedPoolID := "manual-backup-pool"

	metadata := map[string]interface{}{
		models.NextGenMetaTenantID:      "12345",
		models.NextGenMeteringRegion:    "ap-south-1",
		models.NextGenMetaPlan:          models.PlanPremium,
		models.NextGenMetaRUVersion:     "1",
		models.NextGenMetaMinRCU:        "5000",
		models.NextGenMetaMaxRCU:        "20000",
		models.NextGenMetaDiskType:      "gp3",
		models.NextGenMetaStorageEncryptionEnabled: false,
	}
	metaData := ng.loader.CreateMetaData(clusterID, timestamp, metadata)
	require.NoError(t, writers.WriteMetaData(ctx, clusterID, sdkcommon.MetaTypeLogic, timestamp, metaData))
	time.Sleep(100 * time.Millisecond)

	meteringData := []*sdkcommon.MeteringData{
		{
			SelfID:       "manual-backup-self",
			Timestamp:    timestamp,
			Category:     models.NextGenCategoryManualBackup,
			SharedPoolID: sharedPoolID,
			Data: []map[string]interface{}{
				{
					models.NextGenMeteringLogicalClusterID: clusterID,
					models.NextGenMeteringBackupID:         "backup-1",
					models.NextGenMeteringBackupSizeGB: map[string]interface{}{
						"value": uint64(128),
						"unit":  "GB",
					},
				},
			},
		},
	}

	records, err := ng.generateBillingRecords(ctx, meteringData, timestamp)

	require.NoError(t, err)
	require.Len(t, records, 1)
	record := records[0]
	assert.Equal(t, "backup-storage:local:ap-south-1:premium:manual", record.Unit)
	assert.Equal(t, "128", record.UnitValue)
	assert.Equal(t, uint64(12345), record.TenantID)

	var labels map[string]string
	require.NoError(t, json.Unmarshal([]byte(record.Labels), &labels))
	assert.Equal(t, "gb", labels[models.NextGenSKULabelKeySourceUnit])
	assert.Equal(t, models.NextGenBackupTypeManual, labels[models.NextGenSKULabelKeyBackupType])
	assert.Equal(t, "backup-1", labels[models.NextGenSKULabelKeyBackupID])
	assert.Equal(t, clusterID, labels[models.NextGenSKULabelKeyClusterID])
	assert.Equal(t, sharedPoolID, labels[models.NextGenSKULabelKeySharedPoolID])
}
```

- [ ] **Step 4: Run integration test**

Run:

```bash
go test ./pkg/nextgenmetering -run TestNextGenMetering_GenerateBillingRecords_ManualBackup -count=1
```

Expected: PASS after Tasks 2 and 3.

- [ ] **Step 5: Commit**

```bash
git add pkg/nextgenmetering/metering_test.go
git commit -m "test: cover manual backup metering flow"
```

## Task 5: Run Regression Tests

**Files:**
- No source changes expected.

- [ ] **Step 1: Run focused package tests**

Run:

```bash
go test ./pkg/common/models ./pkg/nextgenmetering -count=1
```

Expected: PASS.

- [ ] **Step 2: Run reporter package tests**

Run:

```bash
go test ./pkg/nextgenrepoter -count=1
```

Expected: PASS.

- [ ] **Step 3: Inspect git status**

Run:

```bash
git status --short
```

Expected: no unstaged source changes after all task commits.

## Self-Review Notes

- Spec coverage: Tasks 1 through 4 cover GB conversion, SKU format, category handling, registry registration, metadata-based flow, and T1/T2 isolation.
- Type consistency: The plan consistently uses `NextGenCategoryManualBackup`, `NextGenMeteringBackupSizeGB`, `NextGenMeteringBackupID`, `NextGenSKULabelKeyBackupID`, and `NextGenBackupTypeManual`.
- Existing behavior preservation: Task 1 reruns the existing bytes-to-MiB backup conversion test; Task 5 runs broader package tests.