# Scripts Maintainer Guide

This guide covers the design, operation, and maintenance of the build, test,
and operational scripts under `scripts/`.

It focuses on cross-cutting patterns, CI integration, and the architectural
boundaries between the three script subsystems. For component-specific details,
prefer the READMEs in `scripts/coverage-site/` and `scripts/docker-random/`.

## Contents

1. Purpose and scope
2. Architecture overview
3. CI and build integration
4. File layout
5. Operational procedures
6. Change management guidance
7. Failure modes and recovery

## 1. Purpose and scope

The `scripts/` directory is a single toolkit for build orchestration, test
dispatch, quality gates, coverage reporting, and long-running random tests.

Goals:
- Provide thin, single-purpose wrappers that keep the Makefile and CI YAML small.
- Enforce consistent shell patterns (fail-fast, env validation, atomic ops).
- Isolate complex or long-running test harnesses (Docker-based random tests,
  coverage upload pipelines) from the main build path.

Non-goals:
- Replacing `cargo`, `make`, or GitHub Actions native features.
- Hosting business logic; scripts are pure orchestration.
- Supporting non-Linux/non-Docker environments for the random-test harness.

## 2. Architecture overview

```
+---------------+      +-------------------+      +------------------+
| GitHub Actions|----->| Makefile          |----->| scripts/         |
| (workflows)   |      | (build/test/ci    |      | (wrappers,       |
|               |      |  dispatch layer)  |      |  checkers,       |
+---------------+      +-------------------+      |  harnesses)      |
                          |                       +------------------+
                          |                              |
              +-----------+------------+-----------------+
              |                        |                 |
     +--------v--------+    +----------v----------+  +--v----------------+
     | Build/test gates |    | Coverage pipeline   |  | Docker random tests|
     | (clippy, test)   |    | (CI upload + site)  |  | (long-running)     |
     +------------------+    +---------------------+  +--------------------+
```

### Three subsystems

1. **Build / test / quality gates**
   - `scripts/test`, `scripts/test-all` — cargo test wrappers.
   - `scripts/clippy`, `scripts/clippy-all` — cargo clippy wrappers.
   - `scripts/check-*` — standalone lint/check scripts invoked by `make clippy`.
   - `scripts/run-cargo.sh` — Makefile-driven cargo profile helper for dist builds.

2. **Coverage pipeline**
   - `scripts/coverage/*` — CI-side packaging, manifest generation, and MinIO upload.
   - `scripts/coverage-site/*` — operator-side static site that syncs and serves
     historical reports. See `scripts/coverage-site/README.md` for operator docs.

3. **Random testing**
   - `scripts/docker-random/*` — Docker harness for long-running random tests
     (all, TiDB, replication, CDC, upgrade scenarios).
   - `scripts/random-all.sh`, `scripts/random-tidb.sh` — ad hoc local random loops.
   - `scripts/merge-test.sh`, `scripts/br-test.sh` — legacy focused stress loops.

### Key design patterns

- **Thin wrappers**: Top-level scripts are orchestration only. Heavy logic lives
  in cargo, Python, or Docker.
- **Fail-fast shells**: All shell scripts use `set -euo pipefail`.
- **Env-driven config**: Required variables are validated with `:${VAR:?}`.
  No hardcoded credentials or bucket names.
- **Atomic operations**: Report extraction uses directory swaps (`<dir>.old.<pid>`)
  to minimize races.
- **Immutability**: Generated artifacts (coverage tarballs, manifests, dashboard
  JSON) are treated as immutable once written.

## 3. CI and build integration

### Makefile as the primary dispatch layer

The top-level `Makefile` hides script details from CI and local developers.
Most workflows invoke `make` targets rather than scripts directly.

Key target → script mappings:

| Make target | Script(s) invoked |
|-------------|-------------------|
| `make test` | `scripts/test-all -- --nocapture` |
| `make test_with_nextest` | `scripts/test-all` |
| `make clippy` | `scripts/check-redact-log`, `scripts/check-docker-build`, `scripts/check-license`, `scripts/check-workspace-hack`, `scripts/clippy-all` |
| `make dist_release` | `python scripts/check-bins.py --check-release` |
| `make x-build-dist` / `x-build-dist-debug` | `bash scripts/run-cargo.sh` |
| `make coverage-report` | `scripts/coverage/*` pipeline |
| `make dev` | `make format` + `make clippy` + `make test` |

### Workflow touchpoints

- `ci.yml` — fast PR gate: `make check-protobuf`, `make clippy`
- `unit-test.yml` — `make test-cloud-engine`
- `integrations.yml` — `make test-cloud-engine-integration`
- `random-test.yml` — `cargo nextest` directly (no scripts/ wrapper)
- `coverage.yml` — `make coverage-report`, then direct invocations of:
  - `scripts/coverage/prepare_run.sh`
  - `scripts/coverage/check_duplicate.sh`
  - `scripts/coverage/package_report.sh`
  - `scripts/coverage/build_manifest.py`
  - `scripts/coverage/upload_to_minio.sh`

### Dependency chains

```
Makefile test -> scripts/test-all -> scripts/test -> cargo test
Makefile clippy -> scripts/clippy-all -> scripts/clippy -> cargo clippy
Makefile dist_release -> build -> scripts/check-bins.py --check-release
scripts/test-all -> scripts/check-bins.py --check-tests
coverage.yml -> scripts/coverage/prepare_run.sh -> make coverage-report
  -> scripts/coverage/package_report.sh
  -> scripts/coverage/build_manifest.py
  -> scripts/coverage/upload_to_minio.sh
scripts/coverage-site/cron-refresh.sh -> Docker Compose -> refresh.sh
  -> generate_index.py
scripts/docker-random/all.sh -> make-bin.sh -> docker-run-random.sh
  -> run-random.sh
```

## 4. File layout

```
scripts/
├── br-test.sh                 # Legacy BR random stress loop
├── check-bins.py              # Binary sanity checks (jemalloc, SSE4.2)
├── check-build-opts.py        # Standalone feature-combination checker
├── check-dashboards           # Dashboard validation (currently disabled in Makefile)
├── check-docker-build         # Verify cargo targets have explicit `path =`
├── check-license              # License-header checker
├── check-redact-log           # Log-redaction lint
├── check-workspace-hack       # Hakari / workspace-hack verifier
├── clippy                     # cargo clippy wrapper
├── clippy-all                 # clippy driver with broader targets/features
├── coverage/                  # CI-side coverage packaging and upload
│   ├── build_manifest.py      # Creates manifest from lcov.info
│   ├── check_duplicate.sh     # Scheduled-run dedupe gate
│   ├── package_report.sh      # Packages HTML + lcov tarball
│   ├── prepare_run.sh         # Coverage job metadata / setup
│   └── upload_to_minio.sh     # Uploads tarball + manifest to MinIO
├── coverage-site/             # Static site for historical coverage reports
│   ├── compose.yml            # Docker Compose: nginx + refresh container
│   ├── cron-refresh.sh        # One-shot refresh wrapper for crontab
│   ├── deploy.sh              # Build refresh image and start nginx
│   ├── down.sh                # Tear down the Compose stack
│   ├── Dockerfile.refresh     # Python 3.12 + awscli + jq image
│   ├── generate_index.py      # Manifest scanner and HTML generator
│   ├── nginx.conf             # nginx virtual host
│   ├── refresh.sh             # Core sync / extract / index logic
│   ├── .env.example           # Required environment variables
│   └── README.md              # Operator quick-start
├── docker-random/             # Docker harness for long-running random tests
│   ├── all.sh                 # Cron entry: "all" scenario
│   ├── tidb.sh                # Cron entry: TiDB scenario
│   ├── replication.sh         # Cron entry: replication scenario
│   ├── upgrade.sh             # Cron entry: upgrade scenario
│   ├── cdc.sh                 # Cron entry: multi-tenant CDC scenario
│   ├── docker-run-random.sh   # Container launcher / orchestrator
│   ├── docker-stop-random.sh  # Stop matching containers
│   ├── make-bin.sh            # Build / copy random test binary
│   ├── run-random.sh          # In-container random test worker loop
│   ├── Dockerfile.tidb        # TiDB/PD/TiFlash/TiCDC test image
│   ├── Dockerfile.upgrade     # Upgrade test image
│   ├── Dockerfile.profile     # Heap profile analysis image
│   └── README.md              # Docker random test operator docs
├── env                        # Tiny `make run` wrapper
├── gen-tikv-details-dashboard # Regenerates Grafana dashboard JSON + SHA256
├── merge-test.sh              # Brute-force random merge test loop
├── random-all.sh              # Ad hoc random test loop (all scenarios)
├── random-tidb.sh             # Ad hoc random test loop (with TiDB)
├── run-cargo.sh               # Makefile-driven cargo profile helper
├── test                       # Main cargo test wrapper
└── test-all                   # Test suite driver with extra env / bin checks
```

## 5. Operational procedures

### Local development

The standard local loop is via `make`:

```bash
make dev      # format + clippy + test
make build    # dev build
make test     # full test suite
make clippy   # lints + checks
make format   # rustfmt
```

Direct script invocation is supported but not required:

```bash
./scripts/test-all -- --nocapture
./scripts/clippy-all
```

### Coverage pipeline operation

**CI side** (automated by `.github/workflows/coverage.yml`):
1. `make coverage-report` generates HTML + LCOV.
2. `scripts/coverage/package_report.sh` creates a gzipped tarball.
3. `scripts/coverage/build_manifest.py` parses `lcov.info` into JSON.
4. `scripts/coverage/upload_to_minio.sh` pushes both to MinIO.

**Site side** (manual operator steps):
```bash
cd scripts/coverage-site
cp .env.example .env
# Edit .env with real credentials and bucket name
./deploy.sh
./cron-refresh.sh   # Run one sync immediately
```

Automated refresh via crontab:
```bash
*/30 * * * * /path/to/scripts/coverage-site/cron-refresh.sh >> /var/log/cse-coverage-refresh.log 2>&1
```

### Random test harness operation

See `scripts/docker-random/README.md` for full details.

Quick start:
```bash
cd scripts/docker-random
./all.sh    # or tidb.sh, replication.sh, cdc.sh, upgrade.sh
```

### Grafana dashboard regeneration

When changing `metrics/grafana/*.dashboard.py`:
```bash
./scripts/gen-tikv-details-dashboard
# Commit the resulting tracked .json and .json.sha256 files only
```

The command also generates serverless JSON and checksum files as ignored local
artifacts. Their checksums are validated by `scripts/check-dashboards` when the
files are present locally.

## 6. Change management guidance

When modifying any script in this directory, keep the following invariants:

- **Makefile parity**: If a script is invoked from the Makefile, ensure the
  target behavior remains identical for local dev and CI.
- **Idempotency**: `refresh.sh`, `generate_index.py`, and `check-*` scripts
  must be safe to run repeatedly. Use `cmp -s` or equivalent for no-op detection.
- **Graceful degradation**: Network errors, corrupted files, or missing inputs
  must not crash loops or leave partial state. Log warnings and continue.
- **Env validation**: All required environment variables must be validated
  with a clear error message. No silent defaults for credentials or buckets.
- **Immutable artifacts**: Once a coverage tarball, manifest, or dashboard JSON
  is written, do not modify it in place. Write new versions and swap atomically.
- **Generated file hygiene**: Do not hand-edit generated dashboard JSON or
  coverage-site `public/` content. Regenerate via the approved scripts.

Testing script changes:
1. Run the corresponding `make` target locally.
2. For coverage scripts, point `.env` at a test MinIO bucket and run
   `./cron-refresh.sh`.
3. For check scripts, verify they fail appropriately when the checked condition
   is violated.
4. Re-run to confirm idempotency (no-op on second run when nothing changed).

## 7. Failure modes and recovery

### Script exits early with unset variable error
All shell scripts use `set -u` and `:${VAR:?}` for required variables.
Fix: export the missing variable or create `scripts/coverage-site/.env`.

### Corrupted manifest JSON during refresh
`generate_index.py` logs a warning and skips corrupted manifests.
The refresh continues for all other manifests. Delete the corrupted file from
`public/raw/` to force a re-download.

### tarball download or extraction failure
`refresh.sh` logs a warning, skips the report, and does **not** copy the new
manifest into the local cache. The old report (if any) remains intact.

### Manifest upload fails after tarball upload
`upload_to_minio.sh` uploads the tarball first, then the manifest. If the
manifest upload fails, the script deletes the orphaned tarball before exiting.

### nginx 404 during refresh
Report extraction uses an atomic directory swap. The window where the path does
not exist is minimal. If it occurs, wait for `refresh.sh` to finish.

### Local disk growth on coverage site
`refresh.sh` enforces a 45-day local retention window. If disk is still tight,
reduce the window or add an external cleanup job.