# Development Environment The purpose of this documentation is to help you get started and set up your own DBaaS backend. ## 1. Set up Dev Environment Access Follow steps in [DBaaS Operation 101] to learn and set up necessary credentials and configurations to access the dev environment. If everything is OK, you should be able to access the dev environment by using `kubectl`. Example: ```shell # Switch to use the `dev-base` cluster. # You can also execute `kubectl ctx` to select a cluster (context) interactively. kubectl ctx `kubectl config get-contexts --output=name | grep dev-base` # List all pods in the nightly DBaaS backend in dev-base cluster. kubectl get po -n nightly ``` Sample output: ```text NAME READY STATUS RESTARTS AGE Running 0 11h billing2-s7v1jzec-7469f84dfd-9ztqp 1/1 Running 0 11h billing2-s7v1jzec-7469f84dfd-fk5q7 1/1 Running 0 11h billing2-worker-65ie77yv-54f9c6b696-rcpc8 1/1 Running 0 11h central-lzqm1kgd-7d8d479c4b-bl8lk 1/1 Running 0 11h central-lzqm1kgd-7d8d479c4b-t29fw 1/1 Running 0 11h core-server-ge571lp8-6f685967c5-jjq4b 1/1 Running 0 11h core-server-ge571lp8-6f685967c5-nhjkv 1/1 Running 0 11h management-portal-server-rntpx6ha-56fc94db6c-hvm6k 1/1 Running 0 11h management-portal-server-rntpx6ha-56fc94db6c-ncwc5 1/1 Running 0 11h worker-isa7h7uh-6d9c5bf875-pw7lq 1/1 Running 0 11h ``` ## 2. Install Prerequisites In addition to prerequisites listed in [DBaaS Operation 101], these tools are also needed for development: - [Docker](https://www.docker.com) - [Golang](https://golang.org/dl/) 1.15.2 + - [Pulumi](https://www.pulumi.com/docs/get-started/install/versions/) v2.14.0 - [Nodejs](https://nodejs.org/en/download/). **The version of nodejs must be `LTS`** - [Helm](https://helm.sh/docs/intro/install/) ## 3. Create a Gardener Project DBaaS backend requires a Gardener project. To create your own Gardener project if you don't have one: ```shell # We usually use first name as the project name, e.g. PROJECT=dengshuan export PROJECT= # Switch to the gardener cluster kubectl ctx garden # Create the gardener project hack/gardener-new-project.sh ``` ## 4. Create A MetaDB Account Manually trigger [the GitHub Action](https://github.com/tidbcloud/infra/actions/workflows/add-user-to-metadb.yml) in `infra` repo by clicking the `Run workflow` and entering the project name. Then a PR (like [this](https://github.com/tidbcloud/infra/pull/1478)) will be created automatically, and you can approve it. After the PR is merged, you can fetch the password via: ```shell PROJECT= hack/sharedb-password.sh ``` Refer to the [Access database](database.md#access-database) documentation: You are using a shared database, `use $PROJECT_central`; ```text mysql> show databases; +----------------------------+ | Database | +----------------------------+ | INFORMATION_SCHEMA | | METRICS_SCHEMA | | PERFORMANCE_SCHEMA | | $PROJECT_central | | $PROJECT_billing | +----------------------------+ mysql> use $PROJECT_central; ``` ## 5. Create Pulumi Stack DBaaS use [`pulumi`](https://www.pulumi.com/docs/get-started/) to manage infrastructures and deployments. To create a new pulumi stack if you don't have one: ```shell # We usually use first name as the project name, e.g. PROJECT=dengshuan PROJECT= aws-vault exec dev -- ./hack/pulumi-new-project.sh ``` If you want create TiDB Cluster in GCP provider, you need to create a [PR](https://github.com/tidbcloud/infra/pull/1187/files) like this in the tidbcloud/infra repository. **⚠️ WARNING ⚠️**: If you meet any problems with the pulumi stack and want to re-initialize a new one, remember to execute `pulumi -C ${PULUMI_DIR} destroy` first to destroy previously created resources!!! ## 6. Build Backend and Deploy After you make changes to the source code, or if you have not yet deployed any backend: These changes require aws-vault. For a long running command such as `pulumi up`, the session token may expire. In this case you can run `aws-vault exec dev --no-session --`. ```shell # In `tidbcloud/dbaas` repository root: ./build.sh # Deploy export PULUMI_CONFIG_PASSPHRASE=wearepingcap cd deploy/aws # Note: below is printed in the output of the last command pulumi config set dbaas:image_tag pulumi config set dbaas:useSharedMetadb true pulumi config set dbaas:sharedMetadbAddress tidb.e23bdccc.cb17aaf2.us-west-2.prod.aws.tidbcloud.com pulumi config set dbaas:sharedMetadbUser pulumi config set --secret dbaas:sharedMetadbPassword pulumi config set managementPortalInitUserEmail # Disable sentry on development env pulumi config rm dbaas:sentryDsn pulumi config rm dbaas:sentryEnv pulumi config rm dbaas:sentryLogLevel # Optional: reduce costs pulumi config set managementReplicas 1 pulumi config set billingReplicas 1 pulumi config set centralReplicas 1 pulumi config set metricsReplicas 1 pulumi config set coreReplicas 1 # Done pulumi up ``` If it shows the following error after running `pulumi up`, you need to run `pulumi refresh`, and then run `pulumi up` again. ``` Diagnostics: pulumi:pulumi:Stack (dbaas-): error: no status for central load balancer {"status":{"loadBalancer":{}}} ... ``` **Optional: Version Control:** By default, `build.sh` builds docker images with tag `git rev-parse --short HEAD`. To build with a specific tag, for example: ```shell IMAGE_TAG= ./build.sh ``` To update/revert to a specific version manually: ```shell pulumi -C ${PULUMI_DIR} config set dbaas:image_tag pulumi -C ${PULUMI_DIR} up ``` **Optional: Fast Deploy:** ``` GCR_REGISTRY=NONE # If you don't need to push a new image to GCP, set UPDATE_PULUMI=1 # Go ahead and run pulumi up SKIP_PROXY # skip updating the proxy image # only build the server component (actually includes the worker) # only supported for server right now in build.sh BUILD_COMPONENTS=server # all together now GCR_REGISTRY=NONE UPDATE_PULUMI=1 SKIP_PROXY=1 BUILD_COMPONENTS=server aws-vault exec dev --no-session ./build.sh ``` **Destroy:** If you do not need your pulumi stack anymore, destroy it with: ```shell pulumi -C ${PULUMI_DIR} destroy ``` Note: Though this will destroy all resources created by Pulumi, the indirectly created resources are not destroyed: - TiDB Cluster PVC: `tikv-db-tikv-0` and `pd-db-pd-0` - Shoot clusters - Per customer S3 bucket and some IAM roles and policies, vpc peering (manually created bastion machine if exist) ## 7. Configure Frontend to Use Your Backend You may want to test your change end-to-end, then configure a frontend for your DBaaS would be useful: 1. Open https://dev.tidbcloud.com (DBaaS Portal) and https://ops-dev.tidbcloud.com/ (DBaaS Management Portal). 2. If you are asked to sign-in, sign-in use your Google account first. 3. Open **Developer Tools** (in Chrome: press **F12** or **Tools > View > Developer > Developer Tools**). 4. Switch to the **Console** tab. 5. Execute the following JavaScript code in the Console: ```js localStorage.setItem( "dbaasUrl", "https://...aws.tidbcloud.com" ); ``` Example: ```js localStorage.setItem( "dbaasUrl", "https://us-west-2.$PROJECT.shared.aws.tidbcloud.com" ); ``` 6. Refresh the web page. You may observe that your own backend is indeed used in the **Network** tab in the **Developer Tools**. **Optional: Reset frontend to use the default backend**: To reset the API endpoint in , execute `localStorage.removeItem('dbaasUrl')`. > **⚠️ CAUTION ⚠️**: `make verify` depends on `make generate`, which will generate several `.go` files from metadata, such as proto, openapi, opentelemetry, etc. > If you merge code from other branches or face bizarre build errors, run `make verify` (or `make generate`) first. [dbaas operation 101]: https://github.com/tidbcloud/infra/blob/master/docs/operation-101.md ## 8. Manage Node Profile and SKU Before creating the cluster，you need to initialize the Node Profile and SKU, otherwise you won't see the region list and the node profile list of the tidb cluster components. ### Create an API KEY for Node Profile and SKU management Create api-keys.json file. ``` # cat @/Users/louis/Desktop/api-keys.json { "creator_user_id": "1", "key_name": "profile_mng", "tenant_id": "1" } ``` Replace it with your own `backend URL` and `Bearer Token` (You could get it from Authorization Header of [DBaaS Management Portal]( https://ops-dev.tidbcloud.com/ ) API request.). Remember the `API KEY` from the response of the API request. ``` # curl --location --request POST 'https://us-west-2.$PROJECT.shared.aws.tidbcloud.com/api/v1/admin/api_key' \ --header 'Authorization: Bearer {Bearer Token}' \ --header 'Content-Type: application/json' \ --data-binary '@/Users/louis/Desktop/api-keys.json' ``` ### Initialize the Node Profile and SKU Download `node.yaml` and `sku.yaml` files from https://github.com/tidbcloud/cluster-tier-config/tree/main/data . Replace it with your own `backend URL` and `API KEY` (not `Bearer Token`). ``` curl --location --request POST 'https://us-west-2.$PROJECT.shared.aws.tidbcloud.com/external-api/v1/orgs/profile/replace_node_and_sku' \ --header 'Authorization: Bearer APIKEY_{API KEY}' \ --form 'commitId="2022"' \ --form 'nodes.yaml=@"/Users/louis/Desktop/node.yaml"' \ --form 'skus.yaml=@"/Users/louis/Desktop/sku.yaml"' ``` If the status of the profile version table becomes using, it means that node profile and SKU are in effect. ``` mysql> select * from profile_versions; +----+---------------------+---------------------+-----------+---------+ | id | created_at | updated_at | commit_id | status | +----+---------------------+---------------------+-----------+---------+ | 1 | 2022-03-21 05:39:07 | 2022-03-21 05:39:07 | 2022 | using | +----+---------------------+---------------------+-----------+---------+ ``` ### Become an on_demand user If you need to create a dedicated cluster, you need to [add a credit card](https://dev.tidbcloud.com/console/billing/payments) to become an on_demand user. For dev environment, you could use [Stripe test cards](https://stripe.com/docs/testing#cards). ---------------------------------------------------------------------------------------------------- ## Other FAQs ### Static Check Make sure all static checks and unit tests are passed before filing your PR: ```sh make verify ``` ### No GCP Service Account If your local development depends on GCP, create a service account first. Please refer to the [example](https://github.com/tidbcloud/infra/commit/30e4a5f34fb0a7ace8639b51a94e5195e34b08ec). ### Modify `go.mod` If you need to add new dependencies, you might use commands like `go get`, `go mod tidy`, etc. Don't forget to add `GOPRIVATE=github.com/tidbcloud` first. ```shell export GOPRIVATE=github.com/tidbcloud ``` TiDB Cloud has some private repos under `github.com/tidbcloud`, you need `GOPRIVATE=github.com/tidbcloud` to bypass them. For more info, please refer to [Go command: Configuration for downloading non-public code](https://pkg.go.dev/cmd/go#hdr-Configuration_for_downloading_non_public_code)