# Reliability Guide

This repo depends on application code and infrastructure code moving together.
Reliability work should account for local development, async orchestration, CI,
and production deployment.

## Runtime Dependencies

- TiDB on port `4000`
- Redis on port `6379`
- `pingcap-inc/tidb-labs` mounted or cloned into `backend/labs/labs`
- Cloud provider credentials for AWS, Alicloud, and TiDB Cloud when exercising
  real provisioning flows

## Local Bring-Up

### Local Dependencies

For backend validation and most local integration work, start TiDB and Redis
first:

```bash
docker run --rm -d --name tidb -p 4000:4000 \
  wangdi4zm/tind:v8.5.1-standalone
docker run --rm -d --name redis -p 6379:6379 redis:latest
```

### Backend API And Worker

```bash
cd backend
uv sync
uv run python manage.py migrate
uv run python manage.py runserver 0.0.0.0:8000
```

Worker processes:

```bash
cd backend
uv run celery -A labs worker --concurrency=5 --loglevel=INFO
uv run celery -A labs beat -S redbeat.RedBeatScheduler
```

Notes:

- `uv run python manage.py generate_lab_list` depends on the external
  `pingcap-inc/tidb-labs` checkout being available at `backend/labs/labs`.
- Backend local `.env` is typically loaded from `backend/labs/.env`.
- Local TiDB often needs `DB_ENABLE_SSL=false`.

### Admin UI

The Django admin frontend has its own optional Vite dev server:

```bash
cd backend/labs/admin/frontend
npm ci
npm run dev
```

## Operationally Sensitive Paths

- `POST /api/v2/lab-instances` kicks off cloud orchestration and Celery task
  graphs.
- Browser terminal access depends on backend JWT signing and `ssh-proxy`
  availability.
- Reverse-proxy URLs depend on Caddy installation and token handoff on the lab
  VM.

## Local Validation

### Frontend

```bash
cd frontend
pnpm run ts-check
pnpm run test
pnpm run lint
pnpm run i18n:check
```

### Backend

```bash
cd backend
uv sync
uv run python manage.py generate_lab_list
uv run python manage.py test
```

Backend caveats:

- `backend/labs/labs` must contain the lab specs repo.
- TiDB and Redis must be available for normal backend validation.
- Local TiDB often needs `DB_ENABLE_SSL=false`.
- `uv run python manage.py generate_lab_list` should run before full backend
  test coverage when the lab-spec checkout changes.

### SSH Proxy

```bash
cd ssh-proxy
go test ./...
```

## CI And Deploy

- Backend CI: `.github/workflows/backend.yml`
- Frontend CI: `.github/workflows/frontend.yml`
- Backend image build: `.github/workflows/backend-image.yml`
- Deploy entrypoints:
  - `make deploy_labs_service`
  - `make deploy_monitoring`
  - `make deploy_metabase`

Labs production deployment can also be started manually with the `Deploy TiDB
Labs` GitHub Actions workflow. The job uses the protected `Production`
environment and GitHub OIDC to obtain short-lived AWS credentials. It deploys
the two application hosts sequentially and waits for both load-balancer target
groups to report each host healthy before continuing.

### Production Host Access

The AWS dynamic inventory selects running instances in `labs-server-asg` and
uses each EC2 instance ID as its Ansible host. SSH is tunneled through AWS
Systems Manager Session Manager, so Labs application deployment does not
depend on the former VPN-only bastion host. Operators need a valid AWS profile,
the Session Manager plugin, and the project-scoped `tidb-labs-deploy` private
key. Replacement instances receive the matching public key from Launch
Template User Data.

Changing the Launch Template default only affects instances launched later.
Do not start an Instance Refresh as part of an application deployment.

## Known Reliability Caveats

- The frontend local `/api/*` proxy is hard-coded to the production API base.
- The backend is tightly coupled to an external lab-spec repository.
- Full lab-start validation is not realistic without cloud credentials and a
  suitably provisioned environment.