# Debugging cert-manager

`cert-manager` can be a little finicky to get working as it has a lot of moving parts. It can also be helpful to map the relationship between k8s resources that get created. You can explore cert-manager components by the following commands:

```sh
kubectl get order -n ${PROJECT} # valid state means the tls certificate is issued by letsencrypt
kubectl get issuer -n ${PROJECT}
kubectl get certificate -n ${PROJECT}
kubectl get secret -n ${pr} dbaascert -ojsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -in /dev/stdin -text -noout
```

## GCP Service Account Secret

As `cert-manager` needs to make changes to Cloud DNS, the service account is needed

Needed by:
* `cert-manager` pod
* `Cluster Issuer` resource

## Cluster Issuer

This is where the DNS provider is specified. This is where the secret name and key are for the service account are specified. A common bug is having the wrong name for either of them. The service account script appends a `-key` suffix to the name of the secret, and uses `key.json` as the key.

Needed by:
* `cert-manager` pod

## `cert-manager` pod

This is where the controller for the CRDs runs. It is very helpful to look at logs to see what is going on

```bash
kubectl -n cert-manager logs cert-manager-b9748ff89-pj7cx -f
```

## CRDs

### Certificate

The certificate resource is created by the user and kicks off the process for getting a TLS certificate from Let's Encrypt

### Order

When the controller sees that a new certificate is created, it will create an `Order` resource. This is, presumably, what hits the Let's Encrypt API to get the information needed for a DNS challenge.

### Challenge

When an `Order` is created and successfully completed, the `Challenge` is created. This is usually done by a TXT record at `_acme_challenge.whatever.domain.something`. When the challenge is successfully completed, the certificate `Ready` status transitions to `True`.

### Debugging

Given the dependency, if one of them gets stuck in a bad state then the process might not be able to proceed.

For example: the service account is no longer valid. What might happen is that the certificate attempts to renew itself but can't complete the challenge. Once the service account is fixed and properly updated (if necessary) in the k8s secret we can work on re-starting the process.

Tailing the logs for the `cert-manager` pod we see

```
I0912 18:05:07.698706       1 sync.go:274] Need to create 0 challenges
I0912 18:05:07.698730       1 sync.go:319] Waiting for all challenges for order "dbaasoperatorcert-1041921957" to enter 'valid' state
```

Since it didn't create a challenge, that means one already exists.

```bash
$ kubectl -n operations get challenge
NAME                             STATE     DOMAIN                                AGE
dbaasoperatorcert-1041921957-0   pending   us-west1.operator.gke.tidbcloud.com   36d
```

Looking at the logs further

```bash
I0912 18:07:34.520457       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="dbaasoperatorcert-1041921957" "related_resource_namespace"="operations" "resource_kind"="Certificate" "resource_name"="dbaasoperatorcert" "resource_namespace"="operations"
```

The solution is to restart the process. We check the order

```bash
$ kubectl -n operations get order
NAME                           STATE     AGE
dbaasoperatorcert-1041921957   invalid   36d
```
Delete the order and then examine it

```bash
$ kubectl -n operations delete order dbaasoperatorcert-1041921957 
order.certmanager.k8s.io "dbaasoperatorcert-1041921957" deleted
$ kubectl -n operations get order
NAME                           STATE     AGE
dbaasoperatorcert-1041921957   pending   2s
$ kubectl -n operations describe order dbaasoperatorcert-1041921957 
.
.
.
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  9s    cert-manager  Created Challenge resource "dbaasoperatorcert-1041921957-0" for domain "us-west1.operator.gke.tidbcloud.com"
```

Checking logs again

```bash
I0912 18:10:49.094076       1 logger.go:58] Calling FinalizeOrder
I0912 18:10:49.577243       1 controller.go:219] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="operations/dbaasoperatorcert-1041921957-0" 
I0912 18:10:49.577287       1 controller.go:213] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="operations/dbaasoperatorcert-1041921957-0" 
I0912 18:10:49.577404       1 controller.go:219] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="operations/dbaasoperatorcert-1041921957-0" 
I0912 18:10:50.156311       1 logger.go:43] Calling GetOrder
I0912 18:10:50.291249       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="operations/dbaasoperatorcert-1041921957" 
I0912 18:10:50.292115       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="operations/dbaasoperatorcert" 
I0912 18:10:50.292277       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="operations/dbaasoperatorcert-1041921957" 
I0912 18:10:50.300303       1 controller.go:213] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="operations/dbaasoperatorcert-1041921957-0" 
I0912 18:10:50.303451       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="operations/dbaasoperatorcert-1041921957" 
I0912 18:10:50.303478       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="operations/dbaasoperatorcert-1041921957" 
I0912 18:10:50.311261       1 controller.go:219] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="operations/dbaasoperatorcert-1041921957-0" 
E0912 18:10:50.312791       1 controller.go:200] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="challenges.certmanager.k8s.io \"dbaasoperatorcert-1041921957-0\" not found" "key"="operations/dbaasoperatorcert-1041921957" 
I0912 18:10:50.312832       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="operations/dbaasoperatorcert-1041921957" 
I0912 18:10:50.313713       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="operations/dbaasoperatorcert-1041921957" 
I0912 18:10:50.313756       1 controller.go:213] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="operations/dbaasoperatorcert-1041921957-0" 
E0912 18:10:50.313807       1 controller.go:238] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"dbaasoperatorcert-1041921957-0\" not found"  
I0912 18:10:50.313819       1 controller.go:219] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="operations/dbaasoperatorcert-1041921957-0" 
I0912 18:10:50.391409       1 sync.go:288] cert-manager/controller/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="1438h59m58.608647793s" "related_resource_kind"="Secret" "related_resource_name"="dbaascert" "related_resource_namespace"="operations" "resource_kind"="Certificate" "resource_name"="dbaasoperatorcert" "resource_namespace"="operations" 
I0912 18:10:50.394803       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="operations/dbaasoperatorcert" 
I0912 18:10:50.395858       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="operations/dbaasoperatorcert" 
I0912 18:10:50.396565       1 conditions.go:143] Found status change for Certificate "dbaasoperatorcert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2019-09-12 18:10:50.396464015 +0000 UTC m=+6728609.258234378
```