# APIs for adding node profiles ## TLDR: Real-world examples ### How to add a new node profile? > From @Dan: 我新加一个 tikv 16c 64g 的机型，有对应的 arm /x86 机型，然后有对应的 PD，然后 storage 有最大最小值，默认存储 > gp3，给所有region开，要支持白名单，一开始不给所有客户用。 #### 1. List Instance SKUs `GET /cloudProviders/aws/instanceSkus` Request: empty Response: ```json { "instanceSkus": [ { "id": 1, "cloudProvider": "aws", "availabilityRegionPricings": [ { "region": "us-east-1", "pricePerHour": 0.768 }, { "region": "us-west-1", "pricePerHour": 0.768 } ], "key": "aws:m5.4xlarge:2C:8G:arm64", "instanceType": "m5.4xlarge", "cpuCores": 2, "memoryGi": 8, "cpuArch": "arm64" }, ... ] } ``` #### 2. Add 64C instance SKU, both arm64 and amd64 Find that there is no 64C instance SKU, then add it. `POST /cloudProviders/aws/instanceSkus:batchUpsert` Request: ```json { "adds": [ { "cloudProvider": "aws", "availabilityRegionPricings": [ { "region": "us-east-1", "pricePerHour": 0.768 }, { "region": "us-west-1", "pricePerHour": 0.768 } ], "instanceType": "m5.4xlarge", "cpuCores": 64, "memoryGi": 64, "cpuArch": "arm64" }, { "cloudProvider": "aws", "availabilityRegionPricings": [ { "region": "us-east-1", "pricePerHour": 0.768 }, { "region": "us-west-1", "pricePerHour": 0.768 } ], "instanceType": "m5.4xlarge", "cpuCores": 64, "memoryGi": 64, "cpuArch": "amd64" } ] } ``` Response: ```json { "adds": [ ... ] } ``` #### 3. List Storage SKUs `GET /cloudProviders/aws/storageSkus` Request: empty Response: ```json { "storageSkus": [ { "id": 1, "cloudProvider": "aws", "availabilityRegionPricings": [ { "region": "us-east-1", "pricePerGiPerMonth": 0.768, "pricePerIopsPerMonth": 0.768, "pricePerThroughputMbpsPerMonth": 0.768 }, { "region": "us-west-1", "pricePerGiPerMonth": 0.768, "pricePerIopsPerMonth": 0.768, "pricePerThroughputMbpsPerMonth": 0.768 } ], "key": "aws:gp3", "volumeType": "gp3" }, ... ] } ``` #### 4. Add io2 storage SKU Find that there is no io2 storage SKU, then add it. `POST /cloudProviders/aws/storageSkus:batchUpsert` Request: ```json { "adds": [ { "cloudProvider": "aws", "availabilityRegionPricings": [ { "region": "us-east-1", "pricePerGiPerMonth": 0.768, "pricePerIopsPerMonth": 0.768, "pricePerThroughputMbpsPerMonth": 0.768 }, { "region": "us-west-1", "pricePerGiPerMonth": 0.768, "pricePerIopsPerMonth": 0.768, "pricePerThroughputMbpsPerMonth": 0.768 } ], "storageType": "m5.4xlarge" } ] } ``` Response: ```json { "adds": [ ... ] } ``` #### 5. Add the new tier unit Add both arm64 and amd64 tier units. We assume associated PD tier unit is already exist here. `POST /cloudProviders/aws/componentTierUnits:batchUpsert` Request: ```json { "adds": [ { "cloudProvider": "aws", "componentType": "tikv", "instanceSkuId": 1, "defaultNodeCount": 3, "storages": [ { "storageType": "data", "storageSkuId": 1, "sizeGiDefault": 1000, "sizeGiMin": 1000, "sizeGiMax": 1000, "iopsMax": 20000, "iopsMin": 1000, "throughputMbpsMax": 1000, "throughputMbpsMin": 1000, // For data storage, use new iopsFactor and throughputMbpsFactor rule. "iopsFactor": 0.111, "throughputMbpsFactor": 0.111 }, { "storageType": "raftlog", "storageSkuId": 2, "sizeGiDefault": 1000, "sizeGiMin": 1000, "sizeGiMax": 1000, "iopsMax": 20000, "iopsMin": 1000, "throughputMbpsMax": 1000, "throughputMbpsMin": 1000, // For raft-log storage, use old rule. "iopsDefault": 999, "throughputMbpsDefault": 999 } ] }, { "cloudProvider": "aws", "componentType": "tikv", "instanceSkuId": 2, "defaultNodeCount": 3, "storages": [ { "storageType": "data", "storageSkuId": 1, "sizeGiDefault": 1000, "sizeGiMin": 1000, "sizeGiMax": 1000, "iopsMax": 20000, "iopsMin": 1000, "throughputMbpsMax": 1000, "throughputMbpsMin": 1000, // For data storage, use new iopsFactor and throughputMbpsFactor rule. "iopsFactor": 0.111, "throughputMbpsFactor": 0.111 }, { "storageType": "raftlog", "storageSkuId": 2, "sizeGiDefault": 1000, "sizeGiMin": 1000, "sizeGiMax": 1000, "iopsMax": 20000, "iopsMin": 1000, "throughputMbpsMax": 1000, "throughputMbpsMin": 1000, // For raft-log storage, use old rule. "iopsDefault": 999, "throughputMbpsDefault": 999 } ] } ] } ``` Response: ```json { "adds": [ ... ] } ``` #### 6. Add a customer specific tier unit rule Then we need to add tier unit rules for each customer-required region. For one customer input tuple: ```json { "cloudProvider": "aws", "region": "us-east-1", "componentType": "tikv", "cpuCore": 64, "memoryGi": 64 } ``` Only one tier unit can be selected for this tuple. `POST /cloudProviders/aws/tenantTierUnitMappings:batchUpsert` Request: ```json { "adds": [ { "organizationId": "123", "input": { "cloudProvider": "aws", "region": "us-east-1", "componentType": "tikv", "cpuCore": 64, "memoryGi": 64 }, "output": { "tierUnitId": 1, "pdTierUnitId": 99 } }, { "organizationId": "123", "input": { "cloudProvider": "aws", "region": "us-west-2", "componentType": "tikv", "cpuCore": 64, "memoryGi": 64 }, "output": { "tierUnitId": 2, "pdTierUnitId": 100 } }, ... // One record per region. All records of customer required regions need to be added. ] } ``` Response: ```json { "adds": [ ... ] } ``` ### How to add multiple node profiles for a new region? The step 1–5 are the same as the previous example. The only difference is that step 6 is replaced by adding some default tier unit rules. #### 6. Add default tier unit rules `POST /cloudProviders/aws/defaultTierUnitMappings:batchUpsert` Request: ```json { "adds": [ { "input": { "cloudProvider": "aws", "region": "us-east-1", "componentType": "tikv", "cpuCore": 64, "memoryGi": 64 }, "output": { "tierUnitId": 1, "pdTierUnitId": 99 } }, { "input": { "cloudProvider": "aws", "region": "us-west-2", "componentType": "tidb", "cpuCore": 64, "memoryGi": 64 }, "output": { "tierUnitId": 2 } }, ... // One record per region per component type. // All records of (regions * component types) need to be added. ] } ``` Response: ```json { "adds": [ ... ] } ``` --- There are two main roles in this use case: PM and OPS. And two other roles: billing team and customers. - [Add brand-new node specs for Azure](https://pingcap-cn.feishu.cn/wiki/MNaWwr599iKbkMkxKRJcbr7snud) - [Add 64C TiKV node spec for AWS](https://pingcap-cn.feishu.cn/wiki/HKlVwwEKMiOF2FkyAiZcUvTQn0f) - [Add 8C node specs for GCP](https://pingcap-cn.feishu.cn/wiki/SSlcw9FleiyBRokUorfc1QQTnQb) After reviewing these PRD specs, the standard requirements are summarized as follows. > Note: All regions mentioned in this doc are cloud-provider-defined regions, not logical regions. ## PM PM needs to submit a form including the following information: - Cloud provider - SKUs - Tier units - Default tier units ### Cloud provider Cloud provider must be selected first. And each form submission can only contain one cloud provider. ### What actual resources are provisioned, from 3rd-party cloud providers? In the end, for the control plane, it needs to provision resources from cloud providers. There are two types of resources: - Instance: CPU, Memory and CPU arch - Storage: Disk type Each instance type and storage type defines a "**SKU**"("库存单位"). Instance **SKU**s are uniquely identified by cloud provider instance type names, while storage **SKU**s are uniquely identified by cloud provider disk types. Each **SKU** is bound to several available regions. These regions are defined by cloud providers. In this section, PM needs to provide the following information for each **SKU**: > Refer to global api proto. #### The billing team should update pricing for each **SKU** later. It is not in the scope of TiDB management service but belongs to the billing service. However, Providing the pricing for each **SKU** here is more behavior-consistent. When PM submits the **SKU** form, only available regions need to be provided. After this, the billing team can update the pricing for each **SKU**. In this section, the billing team needs to provide the following information for each **SKU**: > Refer to global api proto. ### How do SKUs compose to useful nodes? Each node must be a combination of instance and storage so that different TiDB cluster components can run on. There are several business rules for composing nodes, we cannot list all of them here. However, we can summarize the following already-known rules: - TiDB only requires instance SKU. While TiKV and TiFlash require both instance SKU and storage SKU. - Instance SKUs and storage SKUs have relationships. One instance SKU can only be combined with some specific storage SKU. - For a specific TiDB component, let's say TiKV, a node can have multiple storage SKUs. Such as: - Storage for data usage. - Storage for raft-logs usage. - For a specific TiDB component, let's say TiKV, some storage parameters of nodes require complicated rules, such as: - Storage size depends on the instance SKU and storage SKU. - Storage iops depends on the storage size. - Storage throughput depends on the storage iops. - Storage size, iops, and throughput all have a max / min limit. This relationship defines a "**tier unit**". A **tier unit** is uniquely identified by component type, instance type SKU and storage type SKU. Each **tier unit** is also bound to several available regions, like SKUs. The available regions of a **tier unit** are the intersection of the available regions of its instance SKU and storage SKU. ` In this section, PM needs to provide the following information for each **tier unit**: > Refer to global api proto. ### How do customer-input parameters map to a tier unit? Customers input several parameters: - Cloud provider, such as "AWS" - Region, such as "us-west-2" - Component type, such as "TiKV" - Required CPU cores and memory Gigabytes, such as "64C 128G" Backend logic will select a single tier unit for these parameters. It first selects a single tier unit by processing default tier unit rules. And then it overrides the default tier unit by processing tenant-specific tier unit rules. #### Default tier unit rules For each customer input, there is one and only one corresponding tier unit. There is a default tier unit for most of the tenants. Each default tier unit is uniquely defined by the following fields: - Cloud provider - Region - Component type - CPU cores - Memory Gi Note that PM needs to provide the default tier unit for the PD component, inside the default tier unit for the TiKV component. In this section, PM needs to provide the following information for default tier units: > Refer to global api proto. ## OPS OPS sometimes needs to override default tier unit rules for specific customers. #### Specific tier unit rules for some Customers For some customers, there are special tier unit requirements, which cannot be satisfied by default tier unit rules. For example, organization "W" requires using 64C 9999IOPS TiKV component combined with 32C PD component. In this section, OPS first selects an organization and optional projects, then provides the following information: > Refer to global api proto. ## Customer ### What can customers see actually? Customers can see the "pricing **tier**"("价格层级"). Each **tier** is uniquely identified by component type, CPU cores, and memory Gigabytes. All other cloud resource specs are hidden from customers. And also notice that there is no **tier** of a PD component. The **tier** is a concept for internal only. It is not exposed to customers. The same concept is called **NodeType** in the TiDB Cloud Open API. There is one more validation layer for customers' **tier** input. Any business validation logic not changed frequently can be added here. Currently, it contains the following rules: - 2C TiDB must be combined with 2C TiKV and not allowed to be combined with TiFlash. - ... To be added. ## Interaction process (API one-by-one in order) Each previously mentioned resource has two APIs: List and BatchUpdate. > Refer to global api proto.