# TiKVRepeatedDiskExpandFallback

## Impact

The autoscaler attempted to replace TiKV disk expansion with a capacity
reservation at least twice for one resource pool within 30 minutes. Each
successful fallback may add TiKV nodes, so repeated fallback can grow the pool
beyond the intended single scale-out increment and increase customer cost.

## Meaning

When TiKV needs more capacity, the autoscaler may first try to expand disk. If
the cloud volume is still in its modification cooldown, the engine falls back
to a capacity reservation that can add nodes. One fallback can be legitimate;
two within 30 minutes indicates that the same condition is being acted on
repeatedly before the pool has converged.

The alert counts the event
`OutputCapacityReservationV1.ExpandDisk.Fallback` by `shared_pool_id`. It fires
on the second event in 30 minutes and clears when fewer than two matching events
remain in that window.

## Diagnosis

1. Use `shared_pool_id` and `cluster` from the alert to open the regional Loki
   datasource.
2. Query the affected pool and inspect the full fallback sequence:

   ```logql
   {cluster_env="prod", cluster="<cluster>", app="autoscaler-engine"}
     |= `<shared_pool_id>`
     |~ `OutputCapacityReservationV1.(ExpandDisk.Fallback|CreateReservation)`
   ```

3. Count `OutputCapacityReservationV1.ExpandDisk.Fallback` events and record
   their timestamps and reservation IDs.
4. For each fallback, check whether a matching
   `OutputCapacityReservationV1.CreateReservation.Success` event exists. A
   success means that fallback completed and may have added TiKV nodes.
5. Compare the resource pool's TiKV node count before and after each successful
   reservation. Also check the requested `MaxAdditionalNodeCount` in the
   fallback event.
6. Check autoscaler-engine restarts and deployment timestamps. A restart or
   rollout between events can explain lost in-memory state, but does not make
   repeated capacity reservations safe.

## Resolution

- If a reservation is still running, do not start another manual scale-out.
  Wait for it to reach a terminal state and confirm the resource pool has
  converged.
- If TiKV nodes were added unexpectedly, stop the repeated trigger first. Use
  the approved Premium operational process to correct capacity; do not remove
  nodes solely to clear the alert.
- If the fallback repeated after the autoscaler fixes were deployed, preserve
  the event sequence and reservation IDs and escalate it as an autoscaler
  invariant violation.

## Post-Check

- Confirm that no new disk fallback event appears for the pool during the next
  30 minutes.
- Confirm that there is no overlapping capacity reservation for the pool.
- Confirm that the TiKV node count and storage capacity match the intended
  resource pool state.