# Shared-Tier Prod Checklist

Author: chenyiwen@pingcap.com

Freshness: 2021/11/5

Docs:
- https://docs.google.com/document/d/1CqsLZoKTWyjCXuhpEGk2JaC5QQOCFF1QgygVsPmRXbo/edit

Generally, You can verify the updates in prod environment in these steps:

1. Check the EKS, the cd pipeline should work as expected.
2. Check the whitelist, thanos and infra apiserver should allow the traffic from the newly created EKS. The whitelist should include the NAT IP of the EKS cluster.
3. Check the monitoring system, the PR in tidbcloud/infra should be merged, and the ci/cd pipeline should work as expected.
4. Check the alarming system, the PR in tidbcloud/runbooks should be merged, and the ci/cd pipeline should work as expected.
5. Check the shared-provider, the PR in tidbcloud/shared-cd-prod should be merged, and the ci/cd pipeline should work as expected.

There's a checklist for you

- [ ] Is there any pod working abnormally? Such as pending pod, crashloopbackoff pod?
- [ ] Is the whitelist allow the traffic from the newly added EKS cluster? You can check the PR like the followings. If the whitelist is not set as the system need, the pod will failed in launching.
    1. Infra apiserver [https://github.com/tidbcloud/infra-cd/pull/329](https://github.com/tidbcloud/infra-cd/pull/329)
    2. Thanos [https://github.com/tidbcloud/infra-cd/pull/329](https://github.com/tidbcloud/infra-cd/pull/329)
- [ ] Is the alarming system and the monitoring system deployed?

## Some checks after the changes in prod environment take effect

After the deployment, you can do some tests, such as creating a cluster and check the monitoring system, logging system, alarming system.

1. Connect the infra-apiserver in prod environment, create a TiDB cluster, you can use this YAML [https://internal-docs.shared.aws.tidbcloud.com/storage/developer/chenyiwen/dev-tier-test-2021-11-5/verify.yaml](https://internal-docs.shared.aws.tidbcloud.com/storage/developer/chenyiwen/dev-tier-test-2021-11-5/verify.yaml).

2. Get the status of the cluster and the network, you can get the components version, component status in cluster.

    ```
    kubectl get cluster -A, kubectl get network -A
    ```
    ![](media/prod-checklist-1.png)


3. Connect to the TiDB cluster

    get the cluste endpoint

    ```
    kubectl get cluster -A -o custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,"Endpoint":".status.sqlEndpoints.publicEndpoint.host"
    ```
    ![](media/prod-checklist-2.png)

    connect to the cluster

    ```
    mysql -h tidb.1a664dd1.931c51cd.ap-northeast-1.prod.aws.tidbcloud.com -P4000 -u root
    ```

4. Check the monitoring, logging, alarming system, and the component version:

    1. monitoring

        [https://grafana.oauth.prod.aws.tidbcloud.com/d/XnuGx8N7z/free-tier-clusters-overview?orgId=1](https://grafana.oauth.prod.aws.tidbcloud.com/d/XnuGx8N7z/free-tier-clusters-overview?orgId=1)

        EKS cluster status

        ![alt_text](media/prod-checklist-3.png)

        TiDB cluster status

        ![alt_text](media/prod-checklist-4.png)

        The dashboard of Dev Tier includes these:

        1. One Screen/Free Tier Clusters Overview(there's only EKS Overview before you created a cluster).
        2. General/Resource Overview
        3. One Screen/Free-Tier TiDB in One Screen(check this after you have a TiDB cluster)
    2. logging

        searching log in loki or ES:
        - [https://grafana.oauth.prod.aws.tidbcloud.com/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22loki%22,%7B%22expr%22:%22%7Bcluster%3D%5C%22freetier%2Feks%2Fprod-ap-northeast-1-s01%5C%22%7D%22%7D%5D](https://grafana.oauth.prod.aws.tidbcloud.com/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22loki%22,%7B%22expr%22:%22%7Bcluster%3D%5C%22freetier%2Feks%2Fprod-ap-northeast-1-s01%5C%22%7D%22%7D%5D)

            ![](media/prod-checklist-5.png)
        - search user_cluster_id: 99999: [https://search-dbaas-logs-prod-lmyaohiqaw3x2vlizvt5qty23a.us-west-2.es.amazonaws.com/_plugin/kibana/app/discover ](https://search-dbaas-logs-prod-lmyaohiqaw3x2vlizvt5qty23a.us-west-2.es.amazonaws.com/_plugin/kibana/app/discover)(You can get the dns in [https://github.com/tidbcloud/infra/blob/08c1c94f898dc1a32de5cc6feaa7637c174ec9c8/deploy/monitoring/Pulumi.prod-freetier-us-east-1-s01-monitoring.yaml#L3](https://github.com/tidbcloud/infra/blob/08c1c94f898dc1a32de5cc6feaa7637c174ec9c8/deploy/monitoring/Pulumi.prod-freetier-us-east-1-s01-monitoring.yaml#L3) monitoring:aws_es_host)

    3. Alarming system

        conenct to the EKS cluster, check if the prometheusrule is created.

        ```
        AWS_REGION=us-east-1 aws eks update-kubeconfig --name prod-us-east-1-s01

        kubectl get prometheusrule -n monitoring

        output:
        NAME                           AGE
        prod-us-east-1-s01-k8s         3h50m
        prod-us-east-1-s01-watch-dog   3h50m
        ```


5. **Delete test cluster***, use `kubectl delete -f prod-shared-tier-verify.yaml`.