# Make Changes to Infrastructure

## Through Continuous Delivery (CD)

DBaaS follow the infrastructure as code principle and all changes to the infrastructure should be approved in Github PR and 
be applied after it is merged to certain branch.

1. Project structure:

    Projects:
    
    - `deploy/eks`: provision an EKS cluster as "Seed" or "Base"
    - `deploy/gke`: provision a GKE cluster as "Seed"
    - `deploy/monitoring`: add monitoring components to a Kubernetes cluster 
    - `deploy/garden`: given a "Base" and a set of "Seeds", ensure a Gardener cluster on these
      - Note: this project is not managed by pulumi yet, we use a wrapper script `garden-setup.sh` to manipulate it
    - `deploy/aws`: given a gardener cluster, deploy a DBaaS backend stack to it
      - You may deploy multiple DBaaS backend stacks to one gardener cluster

    We use pulumi's [`StackReference`](https://www.pulumi.com/docs/intro/concepts/organizing-stacks-projects/#inter-stack-dependencies) to encode the dependencies between these projects. It is recommend to add new project if you change can be abstracted as a separated module (ref: [micro-stacks](https://www.pulumi.com/docs/intro/concepts/organizing-stacks-projects/#micro-stacks)).
    
    When a new project is added, the [CI script](../Jenkinsfile) should be updated accordingly to process the new project.

2. Local development:

    Create a new stack for local development:
    
    ```
    pulumi login --local
    pulumi stack init <stack-name>
    pulumi up
    ```
    
    Apply some code changes and preview change:
    
    ```
    pulumi preview
    ```
    
    Test the change locally:
    
    ```
    pulumi up
    ```
   
3. Open a pull request to request infrastructure change:

    When a pull request is opened, the CI system will detect if there is any change to the code of infrastructure and
    add the change preview in comments, e.g.
    
    ```
        Name                              Type                   Operation
    ~   dev-seed-us-west-2-vpc-private-1  aws:ec2/subnet:Subnet  update
    ~   dev-seed-us-west-2-vpc-public-1   aws:ec2/subnet:Subnet  update
    ~   dev-seed-us-west-2-vpc-public-2   aws:ec2/subnet:Subnet  update
    ~   dev-seed-us-west-2-vpc-private-0  aws:ec2/subnet:Subnet  update
    ~   dev-seed-us-west-2-vpc-public-0   aws:ec2/subnet:Subnet  update
    ~   dev-seed-us-west-2-vpc            aws:ec2/vpc:Vpc        update
    ~   dev-seed-us-west-2-vpc-private-2  aws:ec2/subnet:Subnet  update
    ```

4. If the infrastructure change is get the production involved, then a jira ticket must be filed, you can find an [example](https://jira.tidbcloud.com/projects/CMP/issues/CMP-1) here. Include the ticket in your pull request and get the approvals from at least 2 SREs. For code 

## Manually

Manual write operations to the infra should be avoid unless one of the following conditions meets:

- The operation need be done as quick as possible, e.g. when emergency happens
- The operation cannot be done through CD, e.g. adding a prometheus silence rule

Manual change steps:

1. Create a jira incident ticket on our [cloud jira](https://jira.tidbcloud.com/) before taking the manual operation. The ticket must describe the purpose, change steps and expected results including verify steps of the operation([example](https://jira.tidbcloud.com/projects/CMP/issues/CMP-1)).
2. Test the change steps in staging environment and document the outcome under the incident ticket.
2. Request approvals from the Product Manager and a member of the SRE team in this ticket.
4. Apply the change to production after get the approvals from the Product Manager and at least one SRE.
5. Verify purpose of the change is accomplished and document the outcome of verify steps under the incident ticket.