# Troubleshooting kubectl exec command failures when a Pod is in the CrashLoopBackoff state

## Problem summary

When a Pod is in the `CrashLoopBackoff` state, the containers in the Pod exit continually. As a result, you cannot use `kubectl exec` normally, making it inconvenient to diagnose issues

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

## Diagnosis

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

You can see what is going on by using the `kubectl describe pod <<myContainer>>` command:

```json
`Containers:
   myContainer:
        Image:         busybox
             ...
            Args:       false
                State:          Waiting
                Reason:`       `CrashLoopBackOff`
    `Last State:     Terminated       Reason:       Error       Exit Code:    1`
```

The `Reason: CrashLoopBackOff` tells you the pod is in a crash/restart loop.

## Solution and next steps

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

To solve this problem, TiDB Operator provides the Pod diagnostic mode for PD, TiKV, and TiDB components. In this mode, the containers in the Pod hang directly after they are started, and will not repeatedly crash. Then you can use `kubectl exec` to connect to the Pod containers for diagnosis.

To use the diagnostic mode for troubleshooting:

1. Add an annotation to the Pod to be diagnosed:

    ```{.bash .copyable}
    kubectl annotate pod ${pod_name} -n ${namespace} runmode=debug
    ```

    When the container in the Pod is restarted again, it will detect this annotation and enter the diagnostic mode.

2. Wait for the Pod to enter the Running state.

    ```{.bash .copyable}
    watch kubectl get pod ${pod_name} -n ${namespace}
    ```

    Here's an example of using `kubectl exec` to get into the container for diagnosis:

    ```{.bash .copyable}
    kubectl exec -it ${pod_name} -n ${namespace} -- /bin/sh
    ```

3. After finishing the diagnosis and resolving the problem, delete the Pod.

    ```{.bash .copyable}
    kubectl delete pod ${pod_name} -n ${namespace}
    ```

After the Pod is rebuilt, it automatically returns to the normal mode.