# Backup & Restore (BR)

> - **Objective:** Learn to back up & restore a TiDB cluster on AWS (with Kubernetes)
> - **Prerequisites:** 
>     - Background knowledge of TiDB components 
>     - Background knowledge of Kubernetes and TiDB Operator 
>     - Background knowledge of [BR](https://pingcap.com/docs/stable/reference/tools/br/br/#command-line-description)
>     - AWS account
>     - TiDB cluster on AWS
> - **Optionality:** Required
> - **Estimated time:** 1 hour

This document describes how to perform physical backups of a TiDB cluster and use them to restore/recover a TiDB cluster.

## Prepare

> **Optionality:** Required

If you already created an S3 bucket and Kubernetes secrets as part of the [Import & Export](/hc/en-us/articles/360050033133) guide, you can skip this section.

### Grant AWS Account Permissions

> - **Optionality:** Required

Before you perform backup, AWS account permissions need to be granted to the Backup Custom Resource (CR) object. There are three methods to grant AWS account permissions:

- [Grant permissions by importing AccessKey and SecretKey](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-to-aws-s3-using-br/#three-methods-to-grant-aws-account-permissions)
- [Grant permissions by associating IAM with Pod](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-to-aws-s3-using-br/#grant-permissions-by-associating-iam-with-pod)
- [Grant permissions by associating IAM with ServiceAccount](https://pingcap.com/docs/tidb-in-kubernetes/stable/backup-to-aws-s3-using-br/#grant-permissions-by-associating-iam-with-pod)

In this doc, we use `grant permissions by importing accesskey and secretkey` to grant AWS account permissions.

> **Note**
> 
> Grant permissions by associating IAM with Pod or Grant permissions by associating IAM with ServiceAccount is recommanded in production enviroment.

### Create S3 Bucket

If you don't have an S3 bucket for backup, you can create an S3 bucket in the same AWS region of your EKS cluster:

![create S3 bucket](https://i.imgur.com/4Prh6yx.png)

You can skip this section if you already have a S3 bucket to store backups.

### Install RBAC

Download [backup-rbac.yaml](https://github.com/pingcap/tidb-operator/blob/master/manifests/backup/backup-rbac.yaml), and execute the following command to create the role-based access control (RBAC) resources in your namespace:

```{.bash .copyable}
kubectl apply -f backup-rbac.yaml -n ${namespace}
```

```{.output}
serviceaccount/tidb-backup-manager created
rolebinding.rbac.authorization.k8s.io/tidb-backup-manager created
```

### Create Secrets

#### Create s3-secret

TiDB operator needs to access S3 when performing backup operations. To do that, you can create the s3-secret secret which stores the credential used to access S3:

```{.bash .copyable}
kubectl create secret generic s3-secret --from-literal=access_key=${aws_access_key} --from-literal=secret_key=${aws_secret_key} --namespace=${namespace}
```

```{.output}
secret/s3-secret created
```

This s3-secret will be used in your Backup CR.

Verfify that the secret is properly created:

```{.bash .copyable}
kubectl get secrets -n ${namespace}
```

#### Create export-secret

TiDB operator needs to access TiDB when performing backup operations. To do that, you can create a secret which stores the password of the user account needed to access the TiDB cluster. 

```{.bash .copyable}
kubectl create secret generic export-secret --from-literal=password=${password} --namespace=${namespace}
```

```{.output}
secret/backup-secret created
```

> **Note**
> 
> If there's no password for the user account, leave it blank in ${password}.

#### Verify Secrets

Verfify that the secret is properly created:

```{.bash .copyable}
kubectl get secrets -n ${namespace}
```

## Ad-hoc Full Backup

> - **Optionality:** Required

This section describes how to perform full backup. We use Backup Custom Resource (CR) to desbribe an ad-hoc full backup. TiDB Operator performs backup operation based on the specification in the Backup CR.

### Configure Backup CR

The following is an example Backup CR to save as `backup-aws-s3.yaml`:

You should replace values in `{}` with specific variables in your envrioment and save in `backup-aws-s3.yaml`.

```{.bash .copyable}
namespace=<namespace>
cluster_name=<cluster_name>
region=<region>
bucket=<bucket>
prefix=<prefix>
cat > backup-aws-s3.yaml<<EOF
apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
  name: backup01
  namespace: ${namespace}
spec:
  backupType: full
  br:
    cluster: ${cluster_name}
    sendCredToTikv: true
  from:
    host: ${cluster_name}-tidb
    secretName: export-secret
  s3:
    provider: aws
    secretName: s3-secret
    region: ${region}
    bucket: ${bucket}
    prefix: ${prefix}
EOF
```

### Perform Backup

You can perform an ad-hoc full backup using the following command:

```{.bash .copyable}
kubectl apply -f backup-aws-s3.yaml
```

```{.output}
backup.pingcap.com/backup01 created
```

### Verify Backup

#### Check Backup Status

You can use the following command to check the backup status:

```{.bash .copyable}
kubectl get bk -n ${namespace} -o wide
```

```{.output}
NAME       BACKUPPATH            BACKUPSIZE   COMMITTS             STARTED   COMPLETED   AGE
backup01   s3://bucket/prefix/   1611872      416522333306486785   3m58s     3m55s       15m
```

#### Inspect Backup Log

You can use the following command to find the pod for the ad-hoc full backup:

```{.bash .copyable}
kubectl get po -n ${namespace}
```

You can then use the following command to check the backup progress:

```{.bash .copyable}
kubectl logs -n ${namespace} -f ${backup_pod}
```

```{.output}
...
I0508 02:56:48.029966       1 backup.go:92] [2020/05/08 02:56:48.029 +00:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=2.586075ms]
I0508 02:56:48.030625       1 backup.go:92] [2020/05/08 02:56:48.030 +00:00] [INFO] [collector.go:203] ["Full backup Success summary: total backup ranges: 16, total success: 16, total failed: 0, total take(s): 1.98, total kv: 16000, total size(MB): 1.92, avg speed(MB/s): 0.97"] ["backup fast checksum"=1.561152ms] ["backup checksum"=21.194566ms] ["backup total regions"=16]
I0508 02:56:48.032898       1 backup.go:92]
I0508 02:56:48.033027       1 backup.go:107] Backup data for cluster anthony/backup01 successfully
I0508 02:56:48.040218       1 manager.go:207] reset cluster anthony/backup01 tikv_gc_life_time to 10m0s success
I0508 02:56:48.040241       1 manager.go:218] backup cluster anthony/backup01 data to s3://anthonybr/anthony_01/ success
I0508 02:56:48.136334       1 manager.go:232] Get size 1611872 for backup files in s3://anthonybr/anthony_01/ of cluster anthony/backup01 success
I0508 02:56:48.156090       1 manager.go:244] get cluster anthony/backup01 commitTs 416522333306486785 success
I0508 02:56:48.179889       1 backup_status_updater.go:66] Backup: [anthony/backup01] updated successfully
```

#### Check Backup Files

You can use the S3 console or `aws s3` command to check backup files:

```{.bash .copyable}
aws s3 ls s3://${bucket}/${prefix}/
```

```{.output}
2020-05-17 17:06:03          0
2020-05-17 17:59:48      11598 1_100_25_3283d3a03adc06548749d97c17e127615132cbbf66b60da73a957847f19b62f7_write.sst
2020-05-17 17:59:48     187058 1_100_25_80992061af3e5194c3f28a5b79d486c5e9db2feda1afb3f84b4ca229ddce9932_write.sst
2020-05-17 17:59:48     187374 1_104_26_6d17617cf027713cbd7e33a2a124fe83bc2c35035a776d0c11e63551cbae7815_write.sst
2020-05-17 17:59:48      11487 1_104_26_a48ed62e9b0b2959f804d7850a940f2fc34e1382be1ab74d1518368e2512aba2_write.sst
2020-05-17 17:59:49      11544 1_108_27_2334a18b4455213018fb96ec3c92e951dcc4b10106e91896d882a5ee0d2dbea4_write.sst
2020-05-17 17:59:49     187315 1_108_27_4d4bf84bd30a3ae10b645ad9354c2c26cbc5ff2a95b338b93b820db95359617b_write.sst
2020-05-17 17:59:49      11551 1_116_29_96deb4607345137477d0eea19a6e7a200a37064f87c259d5fa8b08f3c52ed426_write.sst
2020-05-17 17:59:49     187206 1_116_29_9fb8afb3f7322e93f506f0d9d11e9b1569bc90b7c3779da9bb7e35137e8e6597_write.sst
2020-05-17 17:59:49     187222 1_2_29_393d3575060c0d616300c1199ef1c015784fdd3d13e950a6251007bbcbaf2c06_write.sst
2020-05-17 17:59:49      11640 1_2_29_e738a916dbc786a9aef2a69b15289ac934265a4fe851e1dc11140d2ac17b28e8_write.sst
2020-05-17 17:59:48      11556 1_92_23_c882bdf96118f3fcc64e32ac9040b3194ab67d910d1f5232b928416088211ba5_write.sst
2020-05-17 17:59:48     187312 1_92_23_f34f254bf742607feee767bcba259d92794f5f373e6fd415af086e4b42689491_write.sst
2020-05-17 17:59:48     187683 1_96_24_38e4059222579a22c8c937f6506dd8e2198a19c3d7fb4a245c0a4f804cd85adc_write.sst
2020-05-17 17:59:48      11487 1_96_24_675ff68620bc779c8c35ff210de4be67260ad904c9d18c17fed517a4fcf4226b_write.sst
2020-05-17 17:59:49      11564 5_112_28_20ea02d2b95115ebc0c5516aee14058da5e27a1f5bbc545fb831e6c3446fda82_write.sst
2020-05-17 17:59:49     187310 5_112_28_2550d727ffe3e408af41f541b134d5e064440c25988bbecfb665933d1195a45d_write.sst
2020-05-17 17:59:49      20770 backupmeta
```

## Scheduled Full Backup

> - **Optionality:** Optional

You can set up a backup policy to perform scheduled backups for a TiDB cluster, and set a backup retention policy. A scheduled full backup is described by a BackupSchedule CR object.

### Configure BackupSchedule CR

The following is an example BackupSchedule CR. You should replace values including `<>` with specific variables in your envrioment and save in `backup-scheduler-aws-s3.yaml`.

```{.bash .copyable}
namespace=<namespace>
cluster_name=<cluster_name>
aws_region=<aws_region>
bucket=<bucket>
prefix=<prefix>
cat > backup-scheduler-aws-s3.yaml<<EOF
apiVersion: pingcap.com/v1alpha1
kind: BackupSchedule
metadata:
  name: demo-backup-schedule-s3
  namespace: ${namespace}
spec:
  #maxBackups: 5
  #pause: true
  maxReservedTime: "3h"
  schedule: "*/2 * * * *"
  backupTemplate:
    backupType: full
    br:
      cluster: ${cluster_name}
      clusterNamespace: ${namespace}
      sendCredToTikv: true
    from:
      host: ${cluster_name}-tidb
      secretName: export-secret
    s3:
      provider: aws
      secretName: s3-secret
      region: ${aws_region}
      bucket: ${bucket}
      prefix: ${prefix}
EOF
```

### Perform Scheduled Backup

You can perform scheduled full backup using the following command:

```{.bash .copyable}
kubectl apply -f backup-scheduler-aws-s3.yaml
```

```{.output}
backupschedule.pingcap.com/demo-backup-schedule-s3 created
```

### Verify

### Check Backup Status

You can use the following command to check the backup status:

```{.bash .copyable}
kubectl get bk -n ${namespace} -o wide
```

```{.output}
NAME                                          BACKUPPATH                                                                 BACKUPSIZE   COMMITTS             STARTED   COMPLETED   AGE
backup02                                      s3://bucket/anthony_02/                                                 1611578      416550273190199297   5m1s      4m58s       5m12s
demo-backup-schedule-s3-2020-05-09t08-36-00   s3://bucket/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00/   1611578      416550320389226497   2m1s      118s        2m2s
demo-backup-schedule-s3-2020-05-09t08-38-00                                                                              0                                                       2s
```

#### Inspect Backup Log

You can use the following command to find the pod for the scheduled full backup:

```{.bash .copyable}
kubectl get pod -n ${namespace} 
```

You can then use the following command to check the backup progress:

```{.bash .copyable}
kubectl logs -n anthony -f ${backup_pod} 
```

```{.output}
...
I0509 08:36:10.257977       1 backup.go:92] [2020/05/09 08:36:10.257 +00:00] [INFO] [client.go:149] ["save backup meta"] [path=s3://anthonybr/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00] [jobs=0]
I0509 08:36:10.258477       1 backup.go:92] [2020/05/09 08:36:10.258 +00:00] [INFO] [progress.go:102] [Checksum] [progress=100.00%!](MISSING)
I0509 08:36:10.296059       1 backup.go:92] [2020/05/09 08:36:10.295 +00:00] [INFO] [ddl.go:407] ["[ddl] DDL closed"] [ID=061caa48-f309-422b-a53e-11623f55e1a8] ["take time"=973.143µs]
I0509 08:36:10.296130       1 backup.go:92] [2020/05/09 08:36:10.296 +00:00] [INFO] [ddl.go:301] ["[ddl] stop DDL"] [ID=061caa48-f309-422b-a53e-11623f55e1a8]
I0509 08:36:10.297872       1 backup.go:92] [2020/05/09 08:36:10.297 +00:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=2.820476ms]
I0509 08:36:10.298207       1 backup.go:92] [2020/05/09 08:36:10.298 +00:00] [INFO] [collector.go:203] ["Full backup Success summary: total backup ranges: 16, total success: 16, total failed: 0, total take(s): 2.00, total kv: 16000, total size(MB): 1.92, avg speed(MB/s): 0.96"] ["backup checksum"=16.581592ms] ["backup fast checksum"=1.633146ms] ["backup total regions"=16]
I0509 08:36:10.300690       1 backup.go:92]
I0509 08:36:10.300730       1 backup.go:107] Backup data for cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 successfully
I0509 08:36:10.328351       1 manager.go:207] reset cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 tikv_gc_life_time to 10m0s success
I0509 08:36:10.328368       1 manager.go:218] backup cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 data to s3://anthonybr/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00/ success
I0509 08:36:10.491981       1 manager.go:232] Get size 1611578 for backup files in s3://anthonybr/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00/ of cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 success
I0509 08:36:10.520015       1 manager.go:244] get cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 commitTs 416550320389226497 success
I0509 08:36:10.541470       1 backup_status_updater.go:66] Backup: [anthony/demo-backup-schedule-s3-2020-05-09t08-36-00] updated successfully
```

#### Check Backup Files

You can use the S3 console or `aws s3` command to check backup files:

```{.bash .copyable}
aws s3 ls s3://${bucket}/${prefix}/
```

## Restore

> - **Optionality:** Required

In this section, we will demonstrate how to use the backup created in previous sections to restore a database. 

### Cleanup

> **Note:** that, BR only supports to restore to an empty cluster. To do that，you first need to login to the tidb cluster and drop the database `sbtest`:

```
MySQL [(none)]> drop database sbtest;
Query OK, 0 rows affected (0.31 sec)
```

### Configure Restore CR

Similar to Backup, we use a Restore Custom Resource (CR) to describe a restore operation.

The following is an example Restore CR:

You should replace values including `<>` with specific variables in your envrioment and save in `restore-aws-s3.yaml`.

```{.bash .copyable}
namespace=<namespace>
cluster_name=<cluster_name>
aws_region=<aws_region>
bucket=<bucket>
prefix=<prefix>
cat > restore-aws-s3.yaml<<EOF
apiVersion: pingcap.com/v1alpha1
kind: Restore
metadata:
  name: demo-restore-s3
  namespace: ${namespace}
spec:
  br:
    cluster: ${clustername}
    clusterNamespace: ${namespace}

  to:
    host: ${cluster_name}-tidb
    secretName: export-secret
  s3:
    provider: aws
    secretName: s3-secret
    region: ${aws_region}
    bucket: ${bucket}
    prefix: ${prefix}
EOF
```

### Perform Restore

You can perform restore using the following command:

```{.bash .copyable}
kubectl apply -f resotre-aws-s3.yaml
```

```{.output}
restore.pingcap.com/demo-restore-s3 created
```

### Verify

#### Check Restore Status

You can use the following command to check the restore status:

```{.bash .copyable}
kubectl get rt -n ${namespace} -o wide
```

```{.output}
NAME              STARTED   COMPLETED   AGE
demo-restore-s3   43s       25s         49s
```

#### Inspect Restore Log

You can use the following command to check the retore process:

```{.bash .copyable}
kubectl logs ${restore_pod} -n ${namespace}
```

```{.output}
...
I0508 09:04:46.106204       1 restore.go:86] [2020/05/08 09:04:46.106 +00:00] [INFO] [collector.go:203] ["Full restore Success summary: total restore files: 16, total success: 16, total failed: 0, total take(s): 0.51, total kv: 16000, total size(MB): 1.92, avg speed(MB/s): 3.79"] ["split region"=110.963551ms] ["restore checksum"=19.994152ms] ["restore ranges"=16]
I0508 09:04:46.108742       1 restore.go:86]
I0508 09:04:46.108804       1 restore.go:101] Restore data for cluster anthony/demo-restore-s3 successfully
I0508 09:04:46.120192       1 manager.go:206] reset cluster anthony/demo-restore-s3 tikv_gc_life_time to 10m0s success
I0508 09:04:46.120219       1 manager.go:217] restore cluster anthony/demo-restore-s3 from  succeed
I0508 09:04:46.143491       1 restore_status_updater.go:66] Restore: [anthony/demo-restore-s3] updated successfully
```

#### Check Database

Examine the state of the tables in TiDB after the restore completes:

```{.sql .copyable}
show databases;
use sbtest;
show tables;
```

```{.output}
mysql> show databases; 
+--------------------+
| Database           |
+--------------------+
| INFORMATION_SCHEMA |
| METRICS_SCHEMA     |
| PERFORMANCE_SCHEMA |
| mysql              |
| sbtest             |
| test               |
+--------------------+
6 rows in set (0.00 sec)

mysql> use sbtest; 
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+------------------+
| Tables_in_sbtest |
+------------------+
| sbtest1          |
| sbtest2          |
| sbtest3          |
| sbtest4          |
| sbtest5          |
| sbtest6          |
| sbtest7          |
| sbtest8          |
+------------------+
8 rows in set (0.00 sec)

mysql> select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
|     1000 |
+----------+
1 row in set (0.01 sec)
```