# Placement rules in SQL works with tables with TiFlash replica

- Author: [Yunyan Hong](https://github.com/hongyunyan)

## Introduction

This RFC enables tables to support both setting placement rules and setting tiflash replicas.

## Motivation or Background

Currently, there is a conflict between setting tiflash replicas and setting placement rules. For example, if a table has a tiflash replica, it is not supported to set placement rules for this table. Similarly, if a table has set some placement rules, it is not allowed to set tiflash replica for this table. The conflict between tiflash replicas and placement rules has a negative impact on user experience. Thus, this RFC will resolve this conflict, to support setting tiflash replicas and placement rules on tables at the same time.

The conflict between tiflash replicas and placement rules setting is because setting tiflash replicas and placement rules will both generate placement configuration in PD. 

- In the configuration generated by placement rules, `group_index = 40, group_override = true`.
- In the configuration generated by tiflash replicas, `group_index = 0`, `group_override = false`.

According to the rules in PD, if `group_override` is true , the configuration will override the configuration of other groups whose `group_index` is smaller; While if `group_override` is false, other configurations will not be overridden. Therefore, placement rules will overwrite the setting of tiflash replicas. To avoid this problem, the coexistence of placement rules and tiflash replicas was prohibited in the original development.

## Detailed Design

Because placement rules and tiflash replica do not affect each other in their own rules and work independently, we can give a largest value to group_index for the configuration generated by tiflash replicas. We set `RuleIndexTiFlash = 120`, which is larger than all other placement configurations. 

In this way, other placement rules can not override the settings of tiflash replicas. Besides, due to `group_override = false` in tiflash replicas' configuration, tiflash replicas configuration will not override other placement rules. This allows the tables to support both placement rules and tiflash replicas settings.

Therefore, before setting tiflash replicas(`POST /pd/api/v1/config/rule`), we call `POST /pd/api/v1/config/rule-group` to modify the group_index value of the tiflash group as RuleIndexTiFlash.

For the already existing placement configuration of the tiflash replica setting in the old version, we will call `POST /pd/api/v1/config/rule-group` in the background task of `PollTiFlashRoutine` to update the group index of the placement configuration generated by the existing tiflash replicas. In this way, we can ensure normal settings for subsequent placement rules.

## Investigation & Alternatives

1. Add setting group_index in the api `POST /pd/api/v1/config/rule` called by the tiflash replica settings, so that the group_index is `RuleIndexTiFlash` when the tiflash replica is set.
    - This practice changes the semantics of `/pd/api/v1/config/rule`, which is inappropriate.
2. tiflash uses the `POST /pd/api/v1/config/placement-rule` API to replace the original API.
    - This API will require setting the full number of rules in the group. All the settings of tiflash are in a group, which will lead to a large package of http requests when there are many tiflash tables.
    - Compared with the original code, this change is relatively large, and the introduction of concurrency ddl may cause concurrent coverage problems.