# test-plan

## 如何更新 test-plan

⚠ 本 repo 下的分支和 TiDB 的发版分支是相互对应的，因此除管理员外**不应**在该 repo 下直接创建或推送一个新分支，应基于 fork 的 repo 进行修改，最后提交一个 PR 到该 repo 下。

<details>
  <summary>展开</summary>

```text
# 添加 upstream remote
git remote add upstream https://github.com/pingcap/test-plan.git

# 拉取 upstream 上的所有分支
git fetch upstream

# 合并 upstream main 的内容到本地
git rebase upstream/main

# 创建一个新的分支 XXX 用于提交 PR
git checkout -b XXX

修改..., 最后选择要提交的修改

# 提交

git commit -sm "xxx"

git push --set-upstream origin XXX
```

</details>

### Yaml 格式

Yaml 格式能够满足大多数场景的测试计划的编写，其基本形式如下。

<details>
  <summary>展开</summary>

```yaml
apiVersion: api/v1
kind: Plan
name: test_infra_tidb_001
type: GITHUB
schedule: "@daily"
tags: []
testbed:
    generateName: test-infra-
    items:
        - name: tc
          type: TIDB_CLUSTER
          spec: ...
steps:
    - name: test-infra-001
      caseName: test_infra_001
      arguments:
          - name: tc
            value: "{{testbed.tc}}"
failFast: false
```

</details>

### tpl.jsonnet 格式

> jsonnet 入门参考 https://jsonnet.org/

对于需要根据参数动态生成测试计划等复杂场景, 提供 tpl.jsonnet 文件后缀格式以动态生成测试计划。

tpl.jsonnet 求值的结果必须为单个 test-plan json 对象或者 json 数组。

<details>
  <summary>展开</summary>

以升级测试为例，升级测试是多条路径进行的，每一条路径由一个二元组 (source, target) 表示以 source_version 作为初始版本，升级到 target 版本。则可以在 meta.yaml 里头配置一个参数：

```yaml
parameters:
    upgrade_strategy: |
        source: [v4.0.0,v4.0.16,v5.0.0,v5.1.0,v5.3.1]
        target: [v5.3.1,v5.3.2,v5.3.3,v5.3.4,v5.4.0]
```

并编写如下 tpl.jsonnet 文件:

```jsonnet
local upgrade_strategy = std.parseYaml("{{meta.parameters.upgrade_strategy}}");

local genUpgradePlans(source, target) =
  local plan_name = "upgrade_test_" + source + "-" + target;
  std.parseYaml(|||
    apiVersion: api/v1
    kind: Plan
    name: %(plan_name)s
    type: GITHUB
    schedule: "@daily"
    tags: []
    testbed:
    ...
||| % {
  source: source,
  target: target,
  plan_name: plan_name,
  }
);

[ genUpgradePlans(source, "{{meta.parameters.upgrade_target_version}}") for source in upgrade_strategy["source"] ] +
[ genUpgradePlans("{{meta.parameters.upgrade_src_verion}}", target) for target in upgrade_strategy["target"] ]
```

</details>

## 元信息

根目录下存在一个名为 meta.yaml 的配置文件, 用于进行测试计划的相关配置.

> whitelist 和 blacklist 不能同时配置. 根据分支的类型（见后文）选择进行配置.

-   whitelist: 白名单配置, 用于生效测试计划。非在白名单中的任务不会被日常调度. 只在非主干分支起作用.
-   blacklist: 黑名单配置, 用于失效测试计划。在黑名单中的任务不会被日常调度. 只在主干分支起作用.
-   release: 发版白名单配置.
-   parameters: 公共变量, 可在测试计划文件中被引用, 例如: `{{meta.parameters.tidb_image}}` 会被替换成 `hub.pingcap.net/qa/tidb:master`
-   resourcePool: 在 CI 提交时，用于覆盖 plan 本身设置的 resourcePool 字段.

```yaml
whitelist:
    - test-infra/test-infra-001.daily.yaml
#blacklist:
#    - test-infra/test-infra-002.daily.yaml
parameters:
    tidb_image: hub.pingcap.net/qa/tidb:master
    pd_image: hub.pingcap.net/qa/pd:master
    tikv_image: hub.pingcap.net/qa/tikv:master
    ticdc_image: hub.pingcap.net/qa/ticdc:master
    tiflash_image: hub.pingcap.net/qa/tiflash:master
```

## 测试计划调度策略

### 主干分支

master 分支采用黑名单机制, 不能跑的测试计划需要加到黑名单中。

不在黑名单里的根据测试计划中的 `schedule` 的设定来运行:

-   `schedule: @daily`: 每天调度一次
-   `schedule: @weekly`: 每周调度一次
-   `schedule: '0 0 * * 1,3,5'`: 每周一、三、五调度一次
-   `schedule: '0 0 * * 2,4,6'`: 每周二、四、六调度一次
-   `schedule: ''`: 或者不设置 schedule: 不调度

**`schedule` 暂不支持其他 cron 表达式**

meta.yaml

```yaml
blacklist:
    - "test-infra/test-infra-001.daily.yaml"
    - "test-infra/test-infra-002.daily.yaml"
```

### 发版分支

发版分支例如 release-4.0, release-5.0, release-5.1 是从主干分支切出来的, 并且不会再 merge 会主干。 和 master 不同的是, 发版分支不会周期性被调度。在发版测试时去触发，根据 meta.yaml 中 release 白名单列表过滤出需要调度的。

### Hotfix 分支

Hotfix 分支从发版分支切出来, 大致上和发版分支相同。

### Feature 分支

Feature 分支从主分支切出来，当开发完成之后，会合并到主分支中。需要 daily run，但是通常只需要运行新增的用例。

因此采用白名单方式来控制 daily run 的任务。

meta.yaml

```yaml
whitelist:
    - "test-infra/test-infra-001.daily.yaml"
    - "test-infra/test-infra-002.daily.yaml"
```

当 feature 分支需要合并到主分支时, 因为 meta.yaml 有更变(main 是黑名单机制, feature 是白名单机制), 因此必须要先 rebase 一下 main 的 meta.yaml。 此时, 白名单设置被移除,
feature 分支不会再被 daily run。

### 镜像

目前同步了一下官网的镜像到内部站点 hub.pingcap.net/qa 上, 如需更多，请联系 @mahjonp

| tag     | tidb                            | tikv                            | pd                            | ticdc                            | tiflash                            | tidb-binlog                            |
| ------- | ------------------------------- | ------------------------------- | ----------------------------- | -------------------------------- | ---------------------------------- | -------------------------------------- |
| v4.0.0  | hub.pingcap.net/qa/tidb:v4.0.0  | hub.pingcap.net/qa/tikv:v4.0.0  | hub.pingcap.net/qa/pd:v4.0.0  | hub.pingcap.net/qa/ticdc:v4.0.0  | hub.pingcap.net/qa/tiflash:v4.0.0  | hub.pingcap.net/qa/tidb-binlog:v4.0.0  |
| v4.0.1  | hub.pingcap.net/qa/tidb:v4.0.1  | hub.pingcap.net/qa/tikv:v4.0.1  | hub.pingcap.net/qa/pd:v4.0.1  | hub.pingcap.net/qa/ticdc:v4.0.1  | hub.pingcap.net/qa/tiflash:v4.0.1  | hub.pingcap.net/qa/tidb-binlog:v4.0.1  |
| v4.0.10 | hub.pingcap.net/qa/tidb:v4.0.10 | hub.pingcap.net/qa/tikv:v4.0.10 | hub.pingcap.net/qa/pd:v4.0.10 | hub.pingcap.net/qa/ticdc:v4.0.10 | hub.pingcap.net/qa/tiflash:v4.0.10 | hub.pingcap.net/qa/tidb-binlog:v4.0.10 |
| v4.0.11 | hub.pingcap.net/qa/tidb:v4.0.11 | hub.pingcap.net/qa/tikv:v4.0.11 | hub.pingcap.net/qa/pd:v4.0.11 | hub.pingcap.net/qa/ticdc:v4.0.11 | hub.pingcap.net/qa/tiflash:v4.0.11 | hub.pingcap.net/qa/tidb-binlog:v4.0.11 |
| v4.0.12 | hub.pingcap.net/qa/tidb:v4.0.12 | hub.pingcap.net/qa/tikv:v4.0.12 | hub.pingcap.net/qa/pd:v4.0.12 | hub.pingcap.net/qa/ticdc:v4.0.12 | hub.pingcap.net/qa/tiflash:v4.0.12 | hub.pingcap.net/qa/tidb-binlog:v4.0.12 |
| v4.0.13 | hub.pingcap.net/qa/tidb:v4.0.13 | hub.pingcap.net/qa/tikv:v4.0.13 | hub.pingcap.net/qa/pd:v4.0.13 | hub.pingcap.net/qa/ticdc:v4.0.13 | hub.pingcap.net/qa/tiflash:v4.0.13 | hub.pingcap.net/qa/tidb-binlog:v4.0.13 |
| v4.0.14 | hub.pingcap.net/qa/tidb:v4.0.14 | hub.pingcap.net/qa/tikv:v4.0.14 | hub.pingcap.net/qa/pd:v4.0.14 | hub.pingcap.net/qa/ticdc:v4.0.14 | hub.pingcap.net/qa/tiflash:v4.0.14 | hub.pingcap.net/qa/tidb-binlog:v4.0.14 |
| v4.0.15 | hub.pingcap.net/qa/tidb:v4.0.15 | hub.pingcap.net/qa/tikv:v4.0.15 | hub.pingcap.net/qa/pd:v4.0.15 | hub.pingcap.net/qa/ticdc:v4.0.15 | hub.pingcap.net/qa/tiflash:v4.0.15 | hub.pingcap.net/qa/tidb-binlog:v4.0.15 |
| v4.0.16 | hub.pingcap.net/qa/tidb:v4.0.16 | hub.pingcap.net/qa/tikv:v4.0.16 | hub.pingcap.net/qa/pd:v4.0.16 | hub.pingcap.net/qa/ticdc:v4.0.16 | hub.pingcap.net/qa/tiflash:v4.0.16 | hub.pingcap.net/qa/tidb-binlog:v4.0.16 |
| v4.0.2  | hub.pingcap.net/qa/tidb:v4.0.2  | hub.pingcap.net/qa/tikv:v4.0.2  | hub.pingcap.net/qa/pd:v4.0.2  | hub.pingcap.net/qa/ticdc:v4.0.2  | hub.pingcap.net/qa/tiflash:v4.0.2  | hub.pingcap.net/qa/tidb-binlog:v4.0.2  |
| v4.0.3  | hub.pingcap.net/qa/tidb:v4.0.3  | hub.pingcap.net/qa/tikv:v4.0.3  | hub.pingcap.net/qa/pd:v4.0.3  | hub.pingcap.net/qa/ticdc:v4.0.3  | hub.pingcap.net/qa/tiflash:v4.0.3  | hub.pingcap.net/qa/tidb-binlog:v4.0.3  |
| v4.0.4  | hub.pingcap.net/qa/tidb:v4.0.4  | hub.pingcap.net/qa/tikv:v4.0.4  | hub.pingcap.net/qa/pd:v4.0.4  | hub.pingcap.net/qa/ticdc:v4.0.4  | hub.pingcap.net/qa/tiflash:v4.0.4  | hub.pingcap.net/qa/tidb-binlog:v4.0.4  |
| v4.0.5  | hub.pingcap.net/qa/tidb:v4.0.5  | hub.pingcap.net/qa/tikv:v4.0.5  | hub.pingcap.net/qa/pd:v4.0.5  | hub.pingcap.net/qa/ticdc:v4.0.5  | hub.pingcap.net/qa/tiflash:v4.0.5  | hub.pingcap.net/qa/tidb-binlog:v4.0.5  |
| v4.0.6  | hub.pingcap.net/qa/tidb:v4.0.6  | hub.pingcap.net/qa/tikv:v4.0.6  | hub.pingcap.net/qa/pd:v4.0.6  | hub.pingcap.net/qa/ticdc:v4.0.6  | hub.pingcap.net/qa/tiflash:v4.0.6  | hub.pingcap.net/qa/tidb-binlog:v4.0.6  |
| v4.0.7  | hub.pingcap.net/qa/tidb:v4.0.7  | hub.pingcap.net/qa/tikv:v4.0.7  | hub.pingcap.net/qa/pd:v4.0.7  | hub.pingcap.net/qa/ticdc:v4.0.7  | hub.pingcap.net/qa/tiflash:v4.0.7  | hub.pingcap.net/qa/tidb-binlog:v4.0.7  |
| v4.0.8  | hub.pingcap.net/qa/tidb:v4.0.8  | hub.pingcap.net/qa/tikv:v4.0.8  | hub.pingcap.net/qa/pd:v4.0.8  | hub.pingcap.net/qa/ticdc:v4.0.8  | hub.pingcap.net/qa/tiflash:v4.0.8  | hub.pingcap.net/qa/tidb-binlog:v4.0.8  |
| v4.0.9  | hub.pingcap.net/qa/tidb:v4.0.9  | hub.pingcap.net/qa/tikv:v4.0.9  | hub.pingcap.net/qa/pd:v4.0.9  | hub.pingcap.net/qa/ticdc:v4.0.9  | hub.pingcap.net/qa/tiflash:v4.0.9  | hub.pingcap.net/qa/tidb-binlog:v4.0.9  |
| v5.0.0  | hub.pingcap.net/qa/tidb:v5.0.0  | hub.pingcap.net/qa/tikv:v5.0.0  | hub.pingcap.net/qa/pd:v5.0.0  | hub.pingcap.net/qa/ticdc:v5.0.0  | hub.pingcap.net/qa/tiflash:v5.0.0  | hub.pingcap.net/qa/tidb-binlog:v5.0.0  |
| v5.0.1  | hub.pingcap.net/qa/tidb:v5.0.1  | hub.pingcap.net/qa/tikv:v5.0.1  | hub.pingcap.net/qa/pd:v5.0.1  | hub.pingcap.net/qa/ticdc:v5.0.1  | hub.pingcap.net/qa/tiflash:v5.0.1  | hub.pingcap.net/qa/tidb-binlog:v5.0.1  |
| v5.0.2  | hub.pingcap.net/qa/tidb:v5.0.2  | hub.pingcap.net/qa/tikv:v5.0.2  | hub.pingcap.net/qa/pd:v5.0.2  | hub.pingcap.net/qa/ticdc:v5.0.2  | hub.pingcap.net/qa/tiflash:v5.0.2  | hub.pingcap.net/qa/tidb-binlog:v5.0.2  |
| v5.0.3  | hub.pingcap.net/qa/tidb:v5.0.3  | hub.pingcap.net/qa/tikv:v5.0.3  | hub.pingcap.net/qa/pd:v5.0.3  | hub.pingcap.net/qa/ticdc:v5.0.3  | hub.pingcap.net/qa/tiflash:v5.0.3  | hub.pingcap.net/qa/tidb-binlog:v5.0.3  |
| v5.0.4  | hub.pingcap.net/qa/tidb:v5.0.4  | hub.pingcap.net/qa/tikv:v5.0.4  | hub.pingcap.net/qa/pd:v5.0.4  | hub.pingcap.net/qa/ticdc:v5.0.4  | hub.pingcap.net/qa/tiflash:v5.0.4  | hub.pingcap.net/qa/tidb-binlog:v5.0.4  |
| v5.0.5  | hub.pingcap.net/qa/tidb:v5.0.5  | hub.pingcap.net/qa/tikv:v5.0.5  | hub.pingcap.net/qa/pd:v5.0.5  | hub.pingcap.net/qa/ticdc:v5.0.5  | hub.pingcap.net/qa/tiflash:v5.0.5  | hub.pingcap.net/qa/tidb-binlog:v5.0.5  |
| v5.0.6  | hub.pingcap.net/qa/tidb:v5.0.6  | hub.pingcap.net/qa/tikv:v5.0.6  | hub.pingcap.net/qa/pd:v5.0.6  | hub.pingcap.net/qa/ticdc:v5.0.6  | hub.pingcap.net/qa/tiflash:v5.0.6  | hub.pingcap.net/qa/tidb-binlog:v5.0.6  |
| v5.1.0  | hub.pingcap.net/qa/tidb:v5.1.0  | hub.pingcap.net/qa/tikv:v5.1.0  | hub.pingcap.net/qa/pd:v5.1.0  | hub.pingcap.net/qa/ticdc:v5.1.0  | hub.pingcap.net/qa/tiflash:v5.1.0  | hub.pingcap.net/qa/tidb-binlog:v5.1.0  |
| v5.1.1  | hub.pingcap.net/qa/tidb:v5.1.1  | hub.pingcap.net/qa/tikv:v5.1.1  | hub.pingcap.net/qa/pd:v5.1.1  | hub.pingcap.net/qa/ticdc:v5.1.1  | hub.pingcap.net/qa/tiflash:v5.1.1  | hub.pingcap.net/qa/tidb-binlog:v5.1.1  |
| v5.1.2  | hub.pingcap.net/qa/tidb:v5.1.2  | hub.pingcap.net/qa/tikv:v5.1.2  | hub.pingcap.net/qa/pd:v5.1.2  | hub.pingcap.net/qa/ticdc:v5.1.2  | hub.pingcap.net/qa/tiflash:v5.1.2  | hub.pingcap.net/qa/tidb-binlog:v5.1.2  |
| v5.1.3  | hub.pingcap.net/qa/tidb:v5.1.3  | hub.pingcap.net/qa/tikv:v5.1.3  | hub.pingcap.net/qa/pd:v5.1.3  | hub.pingcap.net/qa/ticdc:v5.1.3  | hub.pingcap.net/qa/tiflash:v5.1.3  | hub.pingcap.net/qa/tidb-binlog:v5.1.3  |
| v5.1.4  | hub.pingcap.net/qa/tidb:v5.1.4  | hub.pingcap.net/qa/tikv:v5.1.4  | hub.pingcap.net/qa/pd:v5.1.4  | hub.pingcap.net/qa/ticdc:v5.1.4  | hub.pingcap.net/qa/tiflash:v5.1.4  | hub.pingcap.net/qa/tidb-binlog:v5.1.4  |
| v5.2.0  | hub.pingcap.net/qa/tidb:v5.2.0  | hub.pingcap.net/qa/tikv:v5.2.0  | hub.pingcap.net/qa/pd:v5.2.0  | hub.pingcap.net/qa/ticdc:v5.2.0  | hub.pingcap.net/qa/tiflash:v5.2.0  | hub.pingcap.net/qa/tidb-binlog:v5.2.0  |
| v5.2.1  | hub.pingcap.net/qa/tidb:v5.2.1  | hub.pingcap.net/qa/tikv:v5.2.1  | hub.pingcap.net/qa/pd:v5.2.1  | hub.pingcap.net/qa/ticdc:v5.2.1  | hub.pingcap.net/qa/tiflash:v5.2.1  | hub.pingcap.net/qa/tidb-binlog:v5.2.1  |
| v5.2.2  | hub.pingcap.net/qa/tidb:v5.2.2  | hub.pingcap.net/qa/tikv:v5.2.2  | hub.pingcap.net/qa/pd:v5.2.2  | hub.pingcap.net/qa/ticdc:v5.2.2  | hub.pingcap.net/qa/tiflash:v5.2.2  | hub.pingcap.net/qa/tidb-binlog:v5.2.2  |
| v5.2.3  | hub.pingcap.net/qa/tidb:v5.2.3  | hub.pingcap.net/qa/tikv:v5.2.3  | hub.pingcap.net/qa/pd:v5.2.3  | hub.pingcap.net/qa/ticdc:v5.2.3  | hub.pingcap.net/qa/tiflash:v5.2.3  | hub.pingcap.net/qa/tidb-binlog:v5.2.3  |
| v5.3.0  | hub.pingcap.net/qa/tidb:v5.3.0  | hub.pingcap.net/qa/tikv:v5.3.0  | hub.pingcap.net/qa/pd:v5.3.0  | hub.pingcap.net/qa/ticdc:v5.3.0  | hub.pingcap.net/qa/tiflash:v5.3.0  | hub.pingcap.net/qa/tidb-binlog:v5.3.0  |
| v5.4.0  | hub.pingcap.net/qa/tidb:v5.4.0  | hub.pingcap.net/qa/tikv:v5.4.0  | hub.pingcap.net/qa/pd:v5.4.0  | hub.pingcap.net/qa/ticdc:v5.4.0  | hub.pingcap.net/qa/tiflash:v5.4.0  | hub.pingcap.net/qa/tidb-binlog:v5.4.0  |

| dm                           |
| ---------------------------- |
| hub.pingcap.net/qa/dm:v2.0.0 |
| hub.pingcap.net/qa/dm:v2.0.1 |
| hub.pingcap.net/qa/dm:v2.0.2 |
| hub.pingcap.net/qa/dm:v2.0.3 |
| hub.pingcap.net/qa/dm:v2.0.4 |
| hub.pingcap.net/qa/dm:v2.0.5 |
| hub.pingcap.net/qa/dm:v2.0.6 |
| hub.pingcap.net/qa/dm:v2.0.7 |
| hub.pingcap.net/qa/dm:v5.3.0 |
| hub.pingcap.net/qa/dm:v5.4.0 |

#### ECR 镜像同步

镜像同步指的是在 test-plan 中, plan 编写者配置的镜像仍为 hub.pingcap.net 镜像, ECR 对使用者透明，由 test-infra 进行同步和改写。镜像同步有两种方式: 显式同步、自动同步。

显式同步, 即在 meta.yaml 中手动配置需要同步的镜像列表, 如下所示:

```yaml
syncToECRImages:
    - hub.pingcap.net/qa/go-tpc:v1.0.7
```

自动同步为 test-plan 提交时, 会扫描所有会调度到 AWS 上的测试计划并搜集测试计划中 testbed 包含的镜像以及 pipeline 中 case 使用镜像，并自动同步到 ECR 中。