<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*

- [DBaaS Operations Runbook](#dbaas-operations-runbook)
  - [Introduction](#introduction)
  - [Alerts](#alerts)
    - [KubeEtcdFullBackupFailed](#kubeetcdfullbackupfailed)
      - [Linked issue](#linked-issue)
      - [Cause](#cause)
      - [Resolution](#resolution)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

# DBaaS Operations Runbook

## Introduction

This is where the SRE team lists paging alerts and how to resolve them. The template is
```markdown
### <Alert name>, ie. KubePodCrashLooping
#### Linked issue
the_linked_issue_if_any
#### Cause
It is caused because a kube pod is crashing
#### Resolution
Turn it off and on again
```

## Alerts

### KubeEtcdFullBackupFailed

#### Linked issue

https://github.com/tidbcloud/dbaas/issues/2008

#### Cause

Copying aylei's comment in the github issue:
```
alert: KubeEtcdFullBackupFailed
expr: (time()
  - etcdbr_snapshot_latest_timestamp{job="kube-etcd3-backup-restore",kind="Full",role="main"}
  > bool 86400) + (etcdbr_snapshot_required{kind="Full",role="main"}
  >= bool 1) == 2
```

> After restart, the timestap drops to 0 until the next fullbackup, so the alert will be triggered unexpectedly.
>

#### Resolution

To resolve we must confirm that there was a restart that occured within the last 24 hours. The full alert will be like this:
```
[FIRING:1] KubeEtcdFullBackupFailed critical (shoot--prod--a7799fb5 false kube-etcd3-backup-restore Full a7799fb5 etcd-main-0 prod main https://10.124.0.1:443 gcp us-west1 etcd gcp a7799fb5 seed operator)
```
which gives the etcd instance for the shoot (shoot--prod--a7799fb5) and on which seed (gcp us-west1)

1. Follow instructions in operations.md to ssm into the bastion machine
2. Use `kubectl` to interact with the seed
```
$ kubectl -n shoot--prod--a7799fb5 get pod etcd-main-0 -o json | jq -re '.status.containerStatuses | .[] | .lastState'
{
  "terminated": {
    "containerID": "docker://ac73a073c94c0c21fdea7f8707a564537b0e9d0587ff328beba2f0881df45d87",
    "exitCode": 1,
    "finishedAt": "2020-07-14T18:37:00Z",
    "reason": "Error",
    "startedAt": "2020-06-29T07:17:46Z"
  }
}
{}
```
Here we see that the first container, which is the etcdbr-ctrl one, has been restarted. The `finishedAt` field is when the container was terminated.
3. Verify that it has been less than 24 hours since the container was restarted.
4. (Optional) In GCP it is possible to browse the backup bucket and check that backup data has been written to since the time of the container restart