# NodePoolGCFailed

## Impact

The periodic garbage collector could not safely complete processing for a
Dedicated Backup, Restore, or CompactBackup nodepool. The nodepool is retained,
so an active workload is not removed, but stale infrastructure and its cloud
cost can remain until the failure is resolved.

## Meaning

`infra_br_nodepool_gc_status` uses these values:

- `0`: not expired
- `1`: expired and waiting for approval
- `2`: deletion approved
- `3`: deleted
- `4`: failed

This alert fires only for `failed`. Expiration and deletion approval are normal
lifecycle states and do not trigger this alert.

The `reason` label is a fixed, low-cardinality classification of the failure.
An unregistered APPBox or unstarted dataplane cache skips the current GC scan
without recording a failed metric and therefore does not fire this alert.

A failed state can mean:

- `reason=unknown`: the failure is not one of the two specific conditions
  below. Check the regional-server error log for metadata, dataplane, patch,
  or other processing errors.
- `reason=havingNodes`: matching dataplane Nodes still exist. This is
  an intentional safety block and must be treated as an invalid operational
  state.
- `reason=failToDelete`: the Kubernetes Delete operation for the nodepool or a
  companion resource failed.

## Diagnosis

1. Use the alert labels to identify the regional control plane and resource:
   `provider`, `region`, `namespace`, `name`, `tenant_id`, `project_id`, and
   `cluster_id`.
2. Read the `reason` label first, then check regional-server logs for the
   nodepool name and the latest GC round. Look for one of these messages:
   - `failed to evaluate BR nodepool GC state`
   - `failed to patch BR nodepool GC metadata`
   - `failed to double check BR nodepool nodes`
   - `an expired nodegroup still has running nodes`
   - `failed to delete BR nodepool`
3. Inspect the cloud-specific Crossplane nodepool resource and these
   annotations:
   - `tidbcloud.com/nodepool-gc-last-used-time`
   - `tidbcloud.com/nodepool-gc-expired-at`
   - `tidbcloud.com/nodepool-gc-approved-at`
   - `tidbcloud.com/nodepool-gc-last-scan-at`
4. If matching Nodes exist, identify their remaining Backup, Restore, or
   CompactBackup workloads. Do not approve or force nodepool deletion while any
   workload is active.
5. For `reason=unknown`, use the matching error log to identify whether the
   failure is metadata validation, metadata patching, dataplane resolution, or
   dataplane querying.
6. For `reason=failToDelete`, inspect the Crossplane resource conditions and
   regional-server logs before retrying.

Use approved production access procedures when running Kubernetes commands.

## Resolution

- `havingNodes`: allow or repair workload termination and node scale-in.
- `failToDelete`: resolve the Kubernetes API, permission, or finalizer error.
- `unknown`: follow the regional-server error log. Repair malformed metadata,
  APPBox/dataplane connectivity, or metadata-patch access as applicable.

The alert clears after a later GC scan records a non-failed state. The previous
failure-reason series is removed when the current state is recorded, so a
recovered nodepool does not leave a stale failed series behind. The default
scan interval is one hour, so recovery is not necessarily reflected
immediately.

## Post-Check

- Confirm that the metric for the exact nodepool is no longer `4`.
- Confirm that no active BR workload lost its nodepool.
- If deletion was expected, confirm that the logical nodepool and any companion
  cloud resources are gone.
- Confirm that no replacement failure series exists for the same cluster.