# CI/CD Current-State Assessment — NexusCRM

**Date:** 2026-04-20
**Dataset:** 600 merged PRs on `pingcap-inc/MetaDrivenPlatform4AI`, 2026-03-22 → 2026-04-20 (28 days)
**Scope:** Describe — quantitatively — how code gets from a PR to `main` today: what runs, what blocks, what reviews, what slips through. Do not prescribe.
**Methodology:** Raw GraphQL snapshots, the file-overlap cache, and the analysis scripts are retained in the assessment author's local working copy under `local_docs/cicd-current-state-assessment/data/` (that path is in `.git/info/exclude` and is intentionally not committed — the dataset is a point-in-time snapshot, not a long-lived artifact). See section 11 for reproducing the snapshot from scratch.

---

## 1. Executive summary

The repo has a busy PR pipeline and an almost entirely bot-run review process. Humans rarely look at PRs, branch protection is effectively off, and ~1 in 4 PRs merges without having been tested against the base it lands on — of which a small but non-zero fraction actually touch overlapping runtime code. The team already shipped auto-revert infrastructure, which is the clearest signal that post-merge breakage is a known cost.

- **Throughput:** 21 merges/day over the window. Peak of 28 concurrent open PRs; 46.8% of wall-clock time has ≥3 PRs open simultaneously.
- **Speed:** Median PR open-to-merge is 43 minutes. 56% merge within 1 h, 81% within 8 h, 95.5% within 24 h.
- **Review reality:** 5.5% of PRs got any human formal review; 8.7% got any non-author human comment. **91.3% of PRs merged with zero human feedback.**
- **De-facto reviewer:** `copilot-pull-request-reviewer` (125 formal reviews, 352 inline review-thread comments across the window — 88 unique PRs, 14.7%). `github-actions` posts 703 summary comments (E2E results + similar drops).
- **Base-drift exposure:** 23.3% of PRs merged with ≥1 other PR having landed between their last-push and their merge. Of those, 11.4% touched a runtime source file that an intervening merge also touched — **2.7% of all merges** (16 of 600) are in the "tested on a stale base AND in a collision zone" bucket.
- **Evidence this cost is real:** the repo already contains `PR #957 "ci(deploy): aggressive auto-revert + Feishu alert"` and `PR #1000 "only trigger auto-revert on PR-merge commits"`, plus visible reverts `#387, #389, #590, #663, #703` within the window.
- **Branch protection is effectively off.** Live `required_status_checks.contexts` is `[]`; `required_approving_review_count` is `0`; `required_conversation_resolution.enabled` is `false`. The *locked* configuration in `.agent/skills/gh-branch-protection/SKILL.md` lists four required contexts, but none are currently enforced on GitHub.

---

## 2. Dataset & methodology

- Pulled via `gh api graphql` in twelve batches of 50 PRs, ordered `CREATED_AT DESC`, state `MERGED`. Earliest PR in the sample is `#308` (2026-03-22).
- Every PR carries its commits, reviews, issue comments, and review-thread comments with author login + timestamp.
- For the 140 stale-base PRs, file-path lists were fetched for the target and every intervening PR (289 unique PRs total) and cached in `files.json`.
- Bot vs human classification is heuristic: any login in `BOT_LOGINS`, or matching `[bot]`, `copilot`, or `github-actions`, is treated as a bot. A human with a login containing `bot` would be misclassified; spot-checked against the top-20 commenter list and found clean.
- All numbers in this document are reproducible by running the analysis scripts (retained locally; see section 11) against the twelve 50-PR GraphQL snapshots that backed this assessment.

### Known sample limits

- **Merged-only.** Abandoned PRs are invisible.
- **No CI-result data.** The assessment does not pull check-runs or commit statuses. Questions like "CI first-attempt pass rate" and "flake rate of individual required checks" are not answerable from this sample — this is the single biggest evidence gap (see Gap G4).
- **File overlap is path-level**, not line-level. The 2.7% collision number is an upper bound on semantic conflicts.
- **Sample ordered by `CREATED_AT`, not `MERGED_AT`.** A PR created before the window start (2026-03-22) but merged inside the window is excluded. Long-lived PRs are exactly the ones most likely to be stale-based at merge time; the 2.7% runtime-overlap figure is therefore a slight undercount of the true base-drift exposure.
- **28-day window.** Bot mix and team composition may have been different earlier.

---

## 3. Workflow inventory

All files under `.github/workflows/` as of 2026-04-20:

| Workflow file | Purpose | Triggers | Emits required check? |
|---|---|---|---|
| `ci.yml` | Backend + frontend tests, coverage gate, E2E API stable scenes | `push [main, agent_dev]`, `pull_request` | **Yes** — 4 check-run names: `Unit Tests`, `Frontend Tests`, `Integration Tests`, `E2E API Stable Scenes`. Additional non-required: `Coverage Gate`, `Sentry Unit Tests`, `Detect Changes`. |
| `e2e-tests.yml` | Playwright E2E browser tests | `pull_request` (scoped paths), `push [main, develop]` | No (informational) |
| `contract-test.yml` | Salesforce API contract tests | `pull_request [main]` | No — **disabled** via `if: false` at the job level |
| `deploy.yml` | Docker build + ECS deploy | `push [main]`, `release`, `workflow_dispatch` | No |
| `feishu-pr-notify.yml` | Feishu chat notifications | `pull_request`, `workflow_run: CI` | No (notification-only) |
| `e2e-case-reminder.yml` | Sticky PR comment if backend CRUD paths touched without e2e-api case | `pull_request [opened, synchronize, reopened, edited]` | No — warning-only (reference pattern) |
| `staging-db-refresh.yml` | TiDB Cloud staging branch reset + SSM rotation | `schedule` (2×/day), `workflow_dispatch` | No (operational) |
| `ci-test-db-refresh.yml` | TiDB Cloud ci-test branch reset + setup apply | `schedule`, `workflow_run: Staging DB Refresh`, `workflow_dispatch` | No (operational) |

**Important correction to prior analysis.** `E2E API Stable Scenes` is emitted by `ci.yml` (job `api-tests` at line 177), **not** by a separate `e2e-api-tests.yml`. The gh-branch-protection SKILL currently lists `e2e-api-tests.yml` as its source workflow; no such file exists. The check itself is real — only the path reference in the SKILL is wrong.

**Job disablement.** `Integration Tests` (job `integration-test` in `ci.yml` at line 100) is disabled via `if: false` with a `# TEMPORARILY DISABLED - autonumber format issue in ci-test` comment. The locked list expects this check to gate `main`; if the live protection were flipped to match the lock, `Integration Tests` would never report and would block every PR indefinitely.

---

## 4. Branch protection state (live vs locked)

### Live protection on `main` (read 2026-04-20 via `gh api .../branches/main/protection`)

- `required_status_checks.contexts`: **`[]`** — zero checks required.
- `required_status_checks.strict`: `false` — "require branches to be up to date" is **off** (recently disabled per user statement).
- `required_pull_request_reviews.required_approving_review_count`: `0` — no approvals required.
- `required_conversation_resolution.enabled`: `false`.
- `enforce_admins.enabled`: `true`.
- `allow_force_pushes.enabled`: `false`.
- `allow_deletions.enabled`: `false`.
- `required_linear_history.enabled`: `false`.

### Locked configuration in `.agent/skills/gh-branch-protection/SKILL.md` (v1, 2026-04-18)

Four contexts MUST be required: `Frontend Tests`, `Integration Tests`, `Unit Tests`, `E2E API Stable Scenes`. Live reality: **zero** of them are required. The lock and the live state are fully out of sync.

---

## 5. Throughput & concurrency

### PR open-to-merge latency (n=600)

| Metric | Value |
|---|---|
| Median | **43 min** |
| Mean | 6.9 h |
| p75 | 4.5 h |
| p90 | 15.4 h |
| Max | 99.3 h |

| Window | % merged within |
|---|---|
| 30 min | 43% |
| 1 h | 56% |
| 2 h | 63% |
| 4 h | 72% |
| 8 h | 81% |
| 24 h | 95.5% |

### Concurrency

- **Peak concurrent open PRs:** 28 (at 2026-04-11 13:44 UTC).
- **% of wall-clock time at N simultaneously-open PRs:**
  - ≥1: 89.7%
  - ≥2: 66.0%
  - ≥3: 46.8%
  - ≥5: 34.4%
  - ≥10: 23.0%
  - ≥20: 11.4%

### Iteration

- 28% of PRs merge with **zero follow-up commits** (push, land, done).
- 28% have exactly 1 follow-up.
- 44% iterate 2+ times. Median among iterated PRs: 3 follow-ups; max observed: 49.
- Median last-commit-to-merge: 9 min — essentially just current CI runtime.

### Authors (top 10 by merged-PR count in the window)

| Login | Merged PRs |
|---|---|
| `ywang-official` | 55 |
| `sydpz` | 45 |
| `amber-lzz` | 45 |
| `IANTHEREAL` | 40 |
| `Daemonxiao` | 9 |
| `github-actions` | 6 |

---

## 6. Review coverage

### Humans barely review anything

- PRs with **any human formal review**: 33 / 600 = **5.5%**.
- PRs with any non-author human feedback (review OR issue comment OR review-thread comment): 52 / 600 = **8.7%**.
- **PRs merged with zero human feedback:** 548 / 600 = **91.3%**.
- Human review activity across the window — total events (not unique PRs): `ywang-official` 31 formal reviews + 40 issue comments; `amber-lzz` 15 + 17; `pantheon-ai` 14 + 17 + 26 thread comments; `IANTHEREAL` 24 issue comments. Concentrated among a handful of reviewers, almost entirely on LARGE / XL PRs.

### Where humans *do* show up (by PR size)

| Size bucket | n | PRs with any human feedback | Median latency when feedback exists | Median total open-to-merge |
|---|---|---|---|---|
| TRIVIAL (≤10 LOC) | 102 | 3.9% | 3.72 h | 0.20 h |
| SMALL (11–100) | 172 | 9.3% | 0.59 h | 0.71 h |
| MEDIUM (101–500) | 178 | 7.9% | 0.57 h | 0.75 h |
| LARGE (>500) | 148 | 12.2% | 0.72 h | 1.21 h |
| XL (>2000) | 32 | 21.9% | 1.01 h | 2.95 h |

Humans gate toward large PRs. Everything else effectively skates past.

### Bot activity (counts across the 600-PR window)

- `copilot-pull-request-reviewer`: 125 formal reviews, 352 inline review-thread comments. 88 unique PRs (14.7%) received any Copilot formal review. In the most recent 200 PRs of the window the same rate is 40% — Copilot adoption ramped up during the period.
- `github-actions`: 703 non-author issue comments (E2E result drops, workflow summaries).
- PRs with any bot feedback: 514 / 600 = **85.7%**. Bots are the de-facto external reviewer.

---

## 7. Base-drift exposure

The question: how often does a PR's last CI run happen against a base that is *not* the commit it merges into?

### Stale-base rate (n=600)

| Intervening merges between last-push and merge | % of PRs |
|---|---|
| ≥1 | **23.3%** (140) |
| ≥2 | 12.8% (77) |
| ≥3 | 8.5% (51) |
| ≥5 | 4.5% (27) |
| ≥10 | 1.8% (11) |

Maximum intervening merges for a single PR: 33.

### File-overlap between stale PRs and the PRs that merged in front of them (n=140)

| Overlap category | Count | % of stale | % of all merges |
|---|---|---|---|
| No file overlap | 115 | 82.1% | — |
| Any file overlap | 25 | 17.9% | 4.2% |
| **Runtime source overlap** (excludes tests/docs/yaml) | **16** | **11.4%** | **2.7%** |

2.7% of all merges (16 out of 600) landed in the zone where (a) the tests on record ran against an earlier base, AND (b) some PR that merged in front of them touched the same runtime file.

### Named examples (runtime-file overlap)

- **PR #526** (+312, −7): intervening `#393, #528, #527` all touched `backend/internal/interfaces/rest/metadata_deploy_handler.go`
- **PR #522** (+130, −2): intervening `#520, #521, #393` all touched `backend/internal/features/metadata/validation_service.go`
- **PR #495** (+121, −26): intervening `#494, #493, #436` all touched `backend/internal/infrastructure/persistence/repo_flows.go`
- **PR #478** (+106, −2): intervening `#461, #475, #481` all touched `backend/internal/features/records/persistence_service.go`
- **PR #434** (+5, 0): intervening `#431, #432, #435` all touched `backend/pkg/soql/parser.go`
- **PR #426** vs **#427**: both touched `backend/pkg/soql/parser.go`

File-overlap here is path-level, not hunk-level. The true semantic-conflict rate is ≤ 2.7%.

---

## 8. Evidence of past breakage

The team has already built and shipped infrastructure to mitigate post-merge breakage:

- **`PR #957`** — `ci(deploy): aggressive auto-revert + Feishu alert`
- **`PR #1000`** — `ci(deploy): only trigger auto-revert on PR-merge commits`

Post-hoc reverts within the 28-day window:

- **`PR #387`** — `Revert "Feature/sync fork"`
- **`PR #389`** — `Revert revert: re-apply sync fork changes (PR #384)`
- **`PR #590`** — `refactor(layout): revert to single-layout model (Salesforce parity)` (design revert)
- **`PR #663`** — `fix: revert bootstrap violation — remove org-specific custom fields from standard_listfields.json (#422)`
- **`PR #703`** — `Revert "[codex] Align account list view defaults"`
- **`PR #1181`** — `hotfix prod deployment`

Observed clear-breakage revert rate: 6 of 600 = **~1.0%**. This is a lower bound on actual breakages — reverts caught by the auto-revert pipeline land as ordinary commits on `main` and may not show up as distinct reverts. Additionally, the "Re-implement" pattern (e.g. `#1063, #1087, #1107, #1109, #1116`) implies prior reverts that don't match on title `^Revert` — these are not counted above and represent a systematic gap in the methodology.

---

## 9. Existing bot landscape

| Bot | Role | Coverage in window |
|---|---|---|
| `copilot-pull-request-reviewer` | GitHub Copilot Code Review. Auto-reviews PRs, posts formal reviews + inline comments. | 125 formal reviews across 88 unique PRs (14.7%), 439 comments total. Most recent 200 PRs: 40%. |
| `github-actions` | CI workflows. Posts summary comments on `pull_request` / `workflow_run` events (E2E drops, test-result summaries). | 703 issue comments across 600 PRs. Combined with Copilot, any-bot-feedback covers 85.7% of PRs. |
| Feishu integration (`feishu-pr-notify.yml`) | Out-of-band notifications to chat. Does not comment on PRs. | Not captured in the GraphQL dataset (posts to external chat). |
| Local `/review` slash command (`.claude/commands/review.md`) | Runs Claude + Codex dual-model review when invoked manually by a developer. | Zero automatic PR coverage — local-only. |

Copilot is the only external reviewer operating at scale. It comments on ~86% of PRs; its formal-review rate is lower and depends on the PR size/type.

---

## 10. Identified gaps

Not yet prescribing solutions — the [design doc](design/cicd-evolution-design.md) addresses these.

- **G1 — No merged-state verification.** 2.7% of merges land in a base-drift collision zone and were never tested against the actual merged state. Known mitigation (merge queue) not in use; known alternative mitigation ("require branches up-to-date") was intentionally disabled because of contributor-UX cost at 21 merges/day.
- **G2 — No human review culture.** 91.3% of PRs merge with zero human feedback. Copilot is carrying the entire external-review load. Quality claims about merged code rest on Copilot + CI + author self-review.
- **G3 — Branch-protection live/locked drift.** The `.agent/skills/gh-branch-protection/SKILL.md` locked list names four required checks. Live protection requires zero. The SKILL also names a source workflow (`e2e-api-tests.yml`) that does not exist. `Integration Tests` is in the locked list but disabled in the workflow.
- **G4 — No CI health telemetry.** No measurement of required-check first-attempt pass rate or per-check flake rate. This is the single biggest decision-blocker for anything that routes PRs through a merge queue, because queue pain is directly proportional to flake rate.
- **G5 — Assessment has no durable home.** The only prior visibility into this data was ad-hoc investigation. There is no dashboard, no scheduled re-measurement, and no agreed-upon threshold for acting. This document is the first snapshot; the `data/` directory is designed to make re-snapshots diffable.

---

## 11. Reproducing this assessment

The assessment's raw data is deliberately **not** checked into the repo. It lives in the author's local working copy under `local_docs/cicd-current-state-assessment/data/` (that path is locally-excluded via `.git/info/exclude`). Reasons: the dataset is a point-in-time snapshot, not a living artifact; re-running is the correct way to see today's numbers, not diffing committed JSON; and the files (~700 KB of public PR metadata) are not worth the repo weight.

### Rerun recipe

To regenerate the dataset and every number in sections 5–8 from scratch:

1. **Fetch merged PRs in batches** via `gh api graphql`. Page size 50; a larger size tends to HTTP 504 on this repo. Order `CREATED_AT DESC`, state `MERGED`, 12 pages = 600 PRs. Each PR node needs `number`, `createdAt`, `mergedAt`, `additions`, `deletions`, `changedFiles`, `author`, `commits(first:50)`, `reviews(first:30)`, `comments(first:30)`, and `reviewThreads(first:30)` with nested `comments(first:5)`.
2. **Fetch file paths for the stale-base set.** First identify PRs where any other merge fell between the PR's last commit timestamp and its own `mergedAt`; then pull `files(first:100).nodes.path` for each target PR and each intervening PR (batches of ~40 per query).
3. **Run the analyses** in dependency order: top-line timing → size slice → commenter breakdown → bot-filtered timing → concurrency / stale-base → file-overlap. Every number in this document comes from one of those six passes.
4. **Classify bots conservatively.** The heuristic is: login in a short denylist (`github-actions`, `copilot-pull-request-reviewer`, `dependabot` and their `[bot]` variants), plus any login ending in `[bot]` or containing `bot` / `copilot` / `github-actions`. Spot-check the top-20 commenter list against it before trusting a new snapshot — a human login containing `bot` would be misclassified.

### Future snapshots

For a fresh snapshot (e.g. 60 days after a design-doc rollout, to measure whether it moved the needle), run the recipe against a new dated directory under `local_docs/cicd-current-state-assessment/` and diff the output against the numbers quoted here.