# Design: Global Control Plane — Agent-Era Infrastructure (v2)

Codename: **controlplane9** | Repo: [tidbcloud/controlplane9](https://github.com/tidbcloud/controlplane9)

Supersedes: v1 (2026-03-28) | Status: DEPRECATED | Last updated: 2026-03-31

> Historical command-surface note (2026-06-01): this deprecated design predates the command cleanup train. Its command examples may mention removed ideas such as `sync`, `onboard`, `config`, `persona`, `crd`, `migrate`, `test`, `cost`, `reaper`, `stub-gen`, `overlay`, `verify`, `feature-cycle`, `topology`, or `up --persona` / `up --discover` / `up --profile` / `up --remote`. Current runnable surfaces are `setup`, `doctor`, `up`, `status`, `logs`, `deploy`, `destroy`, direct `pause`, and `db dump-init`.

---

## Problem Statement

TiDB Cloud global control plane has three compounding problems:

1. **Infrastructure is fragile and wasteful (O1).** Dev AWS account runs significant infrastructure 24/7. Developers share one EKS cluster (`dev-base-us-west-2`), causing flaky tests, config collisions, and wasted time. No disaster recovery. No way to stand up a control plane from scratch. See [Cost Model](#cost-model) for breakdown.

2. **Delivery is too slow (O2).** Features like CMEK M2 take 2+ weeks from spec to passing E2E. Agents write the code, but humans remain the orchestration layer — injecting context, triggering tasks, coordinating across 8+ repos and 6+ teams.

3. **Agents work solo (O3).** One human + one agent per session. Complex cross-team features can't be decomposed for parallel agent execution.

These problems are layered — you can't skip levels:

```
O3: Multi-Agent Orchestration (future, mentioned only)
 |  requires
O2: Agent-Driven E2E Delivery + Agent Plugin
 |  requires
O1: Local Ephemeral Dev Environment
```

**Chosen approach: Bottom-Up (O1 → O2 → O3).** O1 delivers immediate cost savings and developer autonomy. O2 layers agent tooling and the agent plugin on proven infrastructure. O3 and O4 (Remote Environment & DR) are future enhancements, mentioned but not detailed here.

**End goal: decommission dev-base EKS and dev dataplane.** After O2, the shared controlplane cluster and all dev dataplane EKS clusters shut down. Staging retained for PM QA and performance tests. See [Milestone: Shut Down dev-base EKS + Dev Dataplane](#milestone-shut-down-dev-base-eks--dev-dataplane).

**Scope: organization-wide.** All services deployed to dev-base — from tidb-management, billing, account, auth-server, to every other team — must be covered. This is not a single-team project.

---

## Architecture Overview

```
+=========================================================================+
|                       DEVELOPER WORKSTATION (Mac)                       |
|                                                                         |
|  +----------------------------+   +----------------------------------+  |
|  | Code Agent                 |   | k3d cluster (k3s in Docker)      |  |
|  | (Copilot / Claude Code)    |   |                                  |  |
|  |                            |   | REAL (feature code path):        |  |
|  | cp9-plugin installed:      |   |   Built from source              |  |
|  |   skills/                  |   |   Developer specifies branch/    |  |
|  |     up, deploy, test,      |   |   tag/commit per service         |  |
|  |     logs, destroy, ...     |   |                                  |  |
|  |   agents/                  |   | STUB (dependencies of real):     |  |
|  |     feature-dev.agent.md   |   |   Proto/swagger-generated stubs  |  |
|  |   hooks/                   |   |   Structurally correct stubs     |  |
|  |     post-deploy-verify     |   |                                  |  |
|  |                            |   | SKIP (everything else):          |  |
|  +-------------+--------------+   |   Not running. No image pull.    |  |
|                |                  +----------------------------------+  |
|                | invokes                            |                   |
|                v                                    |                   |
|  +----------------------------+                     |                   |
|  | cp9 CLI (Go binary)        |                     |                   |
|  | ticat-based (cobra backup) |                     |                   |
|  |                            |                     |                   |
|  |   cp9 up / cp9 doctor      |                     |                   |
|  |   cp9 deploy <svc>         |                     |                   |
|  |   cp9 sync / cp9 migrate   |                     |                   |
|  |   go test / repo checks    |                     |                   |
|  |   cp9 logs [--trace-id X]  |                     |                   |
|  |   cp9 destroy              |                     |                   |
|  |   cp9 status --json        |                     |                   |
|  +----------------------------+                     |                   |
|                                                     |                   |
+=========================================================================+
                                                      |
              +-------------------+-------------------+
              |                   |
       +------+-------+    +------+---------+
       | TiDB Cloud   |    | AWS            |
       | Zero         |    | (per-developer)|
       | (trusted)    |    |                |
       | Free 30d TTL |    | Local AWS creds|
       | MySQL compat |    | Resource track.|
       +--------------+    | Track + reap   |
       Fallback:           +----------------+
       - local MySQL
       - TiDB Starter
```

### Agent-Native Design

CLI is the universal interface for both humans and AI agents:

| Principle | Implementation |
|---|---|
| **Structured & Composable** | Text commands match LLM format and chain for complex workflows |
| **Lightweight & Universal** | Minimal overhead, works across all systems without dependencies |
| **Self-describing** | `cp9 --help`, `cp9 <cmd> --help` — agents auto-discover capabilities. Complete command/subcommand reference with get-started examples. |
| **Structured output** | `cp9 --json <cmd>` returns machine-parseable JSON — no output parsing |
| **Deterministic & Reliable** | Consistent results enable predictable agent behavior |
| **Cross-repo focused** | cp9 does NOT duplicate single-repo makefile targets. Each repo's makefile must be complete and safe. cp9 owns cross-repo orchestration and agent-native functionality only. |
| **Plugin-packaged** | Distributed as a code agent plugin (Copilot + Claude Code compatible) — ships with O2. |

---

## O1: Local Ephemeral Dev Environment

**Goal**: Any developer runs `cp9 up` on Mac and gets a working control plane in < 2 minutes. Only feature-related services run as real. Dependencies get proto/swagger stubs. Everything else skipped. **Target: shut down dev-base-us-west-2 and eliminate dev dataplane.**

Cost per developer: near-zero (local k3d compute, free TiDB Cloud Zero database, minimal AWS resources). See [Cost Model](#cost-model) for details.

### Smart Service Classification

The service list is NOT hardcoded. It's computed from three inputs:

1. **app-delivery manifests**: `app-delivery/app-registry/*.yaml` and `app-delivery/staging/app-registry/*.yaml` (ArgoCD ApplicationSet files)
2. **Pre-computed api-level-service-topology**: A static analysis tool scans control plane service source code across repos to generate API-level call dependency graphs. The tool lives in controlplane9 as `cp9 topology` (see [API-Level Service Topology Generation](#api-level-service-topology-generation) for full design). Reference repo: [tidbcloud/pingcap-kb-codebase](https://github.com/tidbcloud/pingcap-kb-codebase) (early stage).
3. **Developer's feature scope**: Which services the developer is actively changing.

**Classification algorithm:**

```
VISUAL: How the graph walk classifies services

  Developer's active service: [tidb-mgmt-service]

                     ┌───────────────────┐
                     │ tidb-mgmt-service │ ◄── REAL (active)
                     └────┬─────┬────────┘
           calls          │     │          calls
         ┌────────────────┘     └─────────────────┐
         v                                        v
  ┌─────────────┐                          ┌─────────────┐
  │ billing-svc │ ◄── REAL (downstream)    │ account-svc │ ◄── REAL (downstream)
  └──────┬──────┘                          └─────────────┘
         │ calls
         v
  ┌─────────────────┐
  │ payment-gateway │ ◄── REAL (downstream)
  └─────────────────┘

  ┌──────────────┐
  │ auth-server  │ ◄── STUB (referenced in config, not in topology)
  └──────────────┘

  ┌──────────────────┐   ┌───────────────┐
  │ monitoring-agent │   │ cost-metering │ ◄── SKIP (unreachable)
  └──────────────────┘   └───────────────┘
    (upstream caller,
     not followed)
```

```
INPUTS:
  - app-delivery/app-registry/*.yaml          (service registry)
  - app-delivery/staging/app-registry/*.yaml   (staging manifests)
  - api-level-service-topology                 (pre-computed call graph)
  - cp9.local.yaml sources: entries            (developer's active services)

ALGORITHM:
  1. Parse app-delivery manifests → full service list
  2. Read developer's active services from cp9.local.yaml sources:
  3. Walk api-level-service-topology starting from active services:
     "Feature code path" = the transitive closure of all services
     reachable via OUTGOING API calls (downstream callees only)
     from the developer's active services. Upstream callers are
     NOT included — only services that active services call.
     Graph walk uses a visited-set to handle cycles.
  4. Classify each service:

     REAL (feature code path):
       - Services the developer is actively changing (from sources:)
       - Services on the feature's downstream call chain (from topology walk)
       → Build from source. Developer specifies branch, tag, or commit.

     STUB (dependency of real, but NOT on feature code path):
       - Services called by REAL services that are NOT themselves
         on the feature code path (e.g., a billing check endpoint
         called by a REAL service, but billing itself isn't being changed)
       → Run as proto/swagger-generated stub in k3s

     SKIP (everything else):
       - Services not reachable from the feature code path
       → Not deployed. No image pull. No resource usage.

  PRECEDENCE: Manual overrides in cp9.yaml services.skip take
  precedence over topology classification. A service in the
  skip list is always skipped, even if the topology walk
  reaches it. Use with caution — skipping a needed dependency
  will cause runtime errors (cp9 doctor will warn).

OUTPUT: .cp9/service-graph.json
  { "name": "tidb-mgmt-service",
    "classification": "real",
    "source": "../tidb-management-service",
    "ref": "feature/cmek-m2",   # branch, tag, or commit
    "namespace": "tidb-management-service" }
  { "name": "billing-svc",
    "classification": "stub",
    "stub_source": "proto",     # or "swagger"
    "namespace": "billing" }
  { "name": "cost-metering",
    "classification": "skip" }
```

**Validation:** `cp9 doctor` validates the computed service graph against the api-level-service-topology. If a REAL service calls a service that is classified as SKIP, `cp9 doctor` flags it as a potential missing dependency.

### Deploy Strategy

For each service, cp9 determines how to deploy based on classification:

```
Deploy decision (per service):
  REAL:
    1. Check out source at specified branch/tag/commit
    2. Build from source (find Dockerfile in repo root, override via cp9.yaml)
    3. docker build (linux/arm64 on Apple Silicon)
    4. k3d image import
    5. Deploy with Kustomize overlay

  STUB:
    1. Generate stub from proto definition or swagger JSON
    2. Build minimal container with stub server
    3. Deploy to k3s

  SKIP:
    → Nothing. Not deployed, not pulled, not referenced.
```

**Image builds:** Each repo has a `Dockerfile` at repo root (confirmed across repos). cp9 auto-detects it. Override: `dockerfile: build/Dockerfile` in cp9.yaml sources entry. All images build as `linux/arm64` — no cross-compilation on Apple Silicon.

### Stub Generation

Server-side stubs, not client-side mocks. Stubs run as real k3s pods with real DNS names — real services call them over the network, same as production. No build tags, no compile-time wiring, no uber/gomock.

```
Stub generation pipeline:

  .proto files ──────> protoc + grpc-go ──────> gRPC stub server ───┐
                                                                    │
  swagger.json ──────> kin-openapi parse ─────> HTTP stub server ───┤
                                                                    ├──> docker build
  (no definition) ──> permissive stub ────────> 200 OK, {} ─────────┤     + k3d import
                      (cp9 doctor warns)                            │     + deploy to k3s
                                                                    │
  cache hit? ────────> skip rebuild ────────────────────────────────┘
  (SHA256 of proto/swagger content)
```

```
For each STUB service:
  1. Find API definition:
     - Proto files (.proto) → generate gRPC stub server
     - Swagger/OpenAPI JSON → generate HTTP stub server
     - No definition → permissive stub (200 OK, empty JSON body).
       cp9 doctor flags: "billing-svc stub is permissive (no API definition)."

  2. Generate structurally correct responses:
     - Default: zero-value responses matching the schema
     - Improved: staging request logs for realistic data
     - Best: agent analyzes source code for behavior-accurate stubs

  3. Build minimal container and deploy to k3s
```

**Improving stub accuracy:** Auto-generated stubs default to zero-value responses, which may cause cascading failures (e.g., a billing check that must return "allowed"). Two paths to fix:

1. **Edit stub code directly.** Generated stubs live in `.cp9/stubs/<service>/`. Manual edits persist across `cp9 sync` via git merge.
2. **Agent-assisted.** The code agent uses proto/swagger definitions, service topology, and staging logs to improve stubs automatically.

The stub source code *is* the configuration — no fixture config layer.

**Stub image caching:** Stub images are cached locally by content hash. Cache key = SHA256 of the transitive closure of all proto files (including imports) or the full swagger JSON file. On `cp9 up`, if the cache key matches an existing image, stub generation and build are skipped. This saves 15-30s on repeated runs with unchanged API definitions.

**Swagger stub generator:** Proto stubs use protoc + grpc-go (standard toolchain). For swagger/OpenAPI stubs, cp9 uses [kin-openapi](https://github.com/getkin/kin-openapi) for schema parsing and generates a lightweight HTTP server that returns schema-valid responses. Tool selection rationale to be finalized in a spike evaluating kin-openapi vs oapi-codegen vs go-swagger.

**Contract drift:** `cp9 sync` regenerates stubs when proto/swagger definitions change (cache key mismatch triggers rebuild). `cp9 doctor` flags schema drift.

### API-Level Service Topology Generation

The `topology.json` file is the foundation of smart service classification. It captures **which service API calls which other service API** — at method-level granularity, across all control plane repos. `cp9 topology` generates it via static analysis.

**Why this matters:** When a developer tests `CreateCluster`, they need to know not just "tidb-mgmt-service calls dedicated-global-service" but specifically which downstream APIs fire, recursively, so the right services are classified as REAL/STUB/SKIP.

#### Scope (4 repos)

| Repo | Services | Client Location Pattern |
|------|----------|------------------------|
| tidb-management-service | tidb-mgmt-service, serverless-global, dedicated-global, nextgen-global, portal | `tidb-mgmt-service/infra/*cli/client.go` — each subdir = one external service |
| account | Central, ACL, Admin, Worker | `thirdparty/*/impl.go` — each subdir = one external service |
| billing | Server, Worker | `handler/*/impl.go` — each subdir = one external service |
| dataflow-service | dataflow-service, dataflow-agent, dataflow-service-ng, dataflow-agent-ng | `pkg/grpc/grpcli/*.go` — each file = one external service client |

**Key finding:** All four repos follow a consistent pattern — cross-service calls are isolated in dedicated client directories with one module per external service. All gRPC clients use `tigrpc.NewClientConn()` + `pb.NewXxxServiceClient(conn)`. This uniformity makes AST-based pattern matching reliable.

#### Global Service Call Graph (Service Level)

```
tidb-mgmt-service (OpenAPI entry point)
  ├──→ serverless-global-service     (gRPC, infra/globalserverlesscli/)
  ├──→ dedicated-global-service      (gRPC, infra/globaldedicatedcli/)
  ├──→ nextgen-global-service        (gRPC, infra/globalnextgencli/)
  ├──→ dataflow-service              (gRPC, infra/dataflowcli/)
  ├──→ account-service               (gRPC, infra/accountcli/)
  ├──→ core-svc                      (gRPC, infra/tempcorecli/)
  └──→ billing-service               (HTTP, infra/billingcli/)

account.Central
  ├──→ core-svc                      (gRPC, thirdparty/infra/)
  ├──→ dataflow-service              (gRPC, thirdparty/dataflow/)
  ├──→ devops-service                (gRPC, thirdparty/devops/)
  ├──→ data-api-service              (gRPC, thirdparty/dataapi/)
  ├──→ auth-server                   (gRPC, thirdparty/authserver/)
  ├──→ tidb-mgmt-service             (gRPC, thirdparty/mgmt/)
  ├──→ account.ACL                   (gRPC, thirdparty/acl/)
  └──→ account (self)                (gRPC, thirdparty/account/)

billing.Server
  ├──→ core-svc                      (gRPC, handler/infrahandler/)
  ├──→ tidb-mgmt-service             (gRPC, handler/tidbmgmtsvchandler/)
  ├──→ account.Central               (HTTP, handler/centralhandler/)
  └──→ Stripe / Cloud Marketplace    (HTTP, leaf nodes)

dataflow-service
  ├──→ core-svc                      (gRPC, pkg/grpc/grpcli/core.go)
  ├──→ account-service               (gRPC, pkg/grpc/grpcli/accountsvc.go)
  ├──→ account.ACL                   (gRPC, pkg/grpc/grpcli/accountacl.go)
  ├──→ tidb-mgmt-service             (gRPC, pkg/grpc/grpcli/mgmt_svc.go)
  ├──→ dedicated-cluster-service     (gRPC, pkg/grpc/grpcli/dedicated_cluster.go)
  ├──→ regional-api                  (HTTP, pkg/regional-api/)
  └──→ cluster-agent                 (HTTP, pkg/clusteragentcli/)
```

Note: Circular dependency exists: tidb-mgmt-service → dataflow-service → tidb-mgmt-service. The graph walk uses a visited-set to handle cycles.

#### topology.json Schema

```json
{
  "version": "1.0",
  "generated_at": "2026-04-05T13:00:00Z",
  "generator": "cp9 topology v0.1.0",
  "repos": ["tidbcloud/tidb-management-service", "tidbcloud/account", "tidbcloud/billing", "tidbcloud/dataflow-service"],
  "services": {
    "tidb-mgmt-service": {
      "repo": "tidbcloud/tidb-management-service",
      "apis": {
        "ClusterService.CreateCluster": {
          "type": "grpc",
          "proto": "api/spec/tidb_mgmt_service/v1/services.proto",
          "calls": [
            {"service": "dedicated-global-service", "api": "ClusterService.CreateCluster", "protocol": "grpc"},
            {"service": "account-service", "api": "AccountAPIService.GetProject", "protocol": "grpc"},
            {"service": "billing-service", "api": "POST /api/v1/unc/tenants/{tenant_id}/bulk_trigger", "protocol": "http"}
          ]
        },
        "POST /v1beta1/clusters": {
          "type": "http",
          "handler": "app/openapi/dedicated/cluster.go",
          "calls": [
            {"service": "dedicated-global-service", "api": "ClusterService.CreateCluster", "protocol": "grpc"}
          ]
        }
      }
    },
    "dedicated-global-service": {
      "repo": "tidbcloud/tidb-management-service",
      "apis": { }
    },
    "account.Central": {
      "repo": "tidbcloud/account",
      "apis": { }
    }
  }
}
```

#### Static Analysis Pipeline (`cp9 topology`)

```
┌─────────────────────────────────────────────────────────────┐
│                     cp9 topology                             │
│                                                               │
│  INPUT: repo paths (--repos ../tidb-management-service,      │
│                            ../account, ../billing,            │
│                            ../dataflow-service)               │
│                                                               │
│  ┌──────────────┐ ┌──────────────┐ ┌───────────────────────┐│
│  │ Proto Scanner │ │ Route Scanner│ │ Client Call Scanner   ││
│  │              ││               ││                       ││
│  │ .proto files ││ Gin router.go││ infra/*cli/           ││
│  │ → services   ││ → HTTP API   ││ thirdparty/           ││
│  │   & methods  ││   surface    ││ handler/              ││
│  │              ││               ││ pkg/grpc/grpcli/      ││
│  │              ││               ││ → outbound calls      ││
│  └──────┬───────┘ └──────┬───────┘ └───────────┬──────────┘│
│         │                │                      │           │
│         └────────┬───────┘──────────────────────┘           │
│                  ▼                                           │
│         ┌────────────────────────┐                          │
│         │   Handler → Client     │                          │
│         │   Call Chain Linker    │                          │
│         │                        │                          │
│         │ AST walk: handler func │                          │
│         │ → service layer        │                          │
│         │ → client wrapper       │                          │
│         │ → target service.api   │                          │
│         └────────────┬───────────┘                          │
│                      ▼                                      │
│              topology.json                                  │
└─────────────────────────────────────────────────────────────┘
```

**Step 1 — Proto Scanner:** Parse all `.proto` files across repos. Extract every `service { rpc Method(...) }` definition. Build the API surface registry: `{service_name → [method1, method2, ...]}`.

**Step 2 — Route Scanner:** Parse Gin `router.go` files (account/admin, billing/api). Pattern-match `router.GET("/path", handler)` registrations. Build HTTP API surface: `{service_name → [GET /path1, POST /path2, ...]}`.

**Step 3 — Client Call Scanner:** Scan the four client directory patterns:

- For each client wrapper function, use Go AST to find `pb.NewXxxServiceClient(conn)` → identifies target service
- Find `client.MethodName(ctx, req)` calls → identifies target API method
- Output: `{client_wrapper_func → {target_service, target_api, protocol}}`

**Step 4 — Handler → Client Chain Linker:** From each API handler function, AST-walk the call chain through the service layer to identify which client wrapper functions are reachable. This connects: `{this_service.this_api → [{target_service.target_api, protocol}]}`.

**Step 5 — Graph Assembly:** Merge results across all repos. Resolve cross-repo proto imports. Handle cycles (visited-set). Output `topology.json`.

#### Query Mode

```bash
# What services must I run to test CreateCluster end-to-end?
$ cp9 topology query --service=tidb-mgmt-service \
                     --api="ClusterService.CreateCluster" \
                     --depth=all

Required services (transitive):
  1. tidb-mgmt-service              (entry point)
  2. dedicated-global-service       (gRPC, called for cluster creation)
  3. account-service                (gRPC, called for auth/project check)
  4. account.ACL                    (gRPC, called for RBAC)
  5. billing-service                (HTTP, called for UNC event + spend limit)
  6. core-svc                       (gRPC, called for legacy cluster info)

Leaf nodes (can be safely stubbed):
  7. Stripe                         (external payment, non-critical)
  8. Salesforce                     (CRM sync, non-critical)

# Validate current cp9 config covers all dependencies
$ cp9 topology validate --config=cp9.yaml

OK: All REAL service dependencies are covered (REAL or STUB).
WARN: core-svc classified as SKIP but called by dedicated-global-service.
      Consider: stub or add to skip exceptions.

# Output formats
$ cp9 topology graph --format=mermaid --service=tidb-mgmt-service --api="ClusterService.CreateCluster"
$ cp9 topology graph --format=dot --service=tidb-mgmt-service --api="ClusterService.CreateCluster"
$ cp9 topology graph --format=json  # full topology.json
```

#### Combined Approach: Static Analysis + Dynamic Validation

Neither static analysis nor dynamic discovery alone is sufficient. The recommended approach combines both:

```
COMBINED APPROACH OVERVIEW

  Static Analysis (topology.json)          Dynamic Validation (runtime)
  ─────────────────────────────            ─────────────────────────────
  Scans source code across repos           Observes actual API calls at runtime
  Finds ALL possible call paths            Finds ACTUALLY-USED call paths
  Over-estimates (includes error           Precise for specific test scenario
    paths, admin-only, rare branches)      Under-estimates (misses alternate
  Fast: ~30s, no services running            input paths)
  Safe: won't miss critical deps           Slow: iterative start/fail/retry
                                           
  USE FOR: initial service classification  USE FOR: refining classification
           (REAL/STUB/SKIP baseline)                per test scenario

  ┌─────────────────────────────────────────────────────────────────┐
  │                    HOW THEY COMBINE                             │
  │                                                                 │
  │  1. cp9 topology → topology.json (static, all possible edges)  │
  │                        │                                        │
  │  2. cp9 up → classify REAL/STUB/SKIP from topology.json        │
  │             → deploy services                                   │
  │                        │                                        │
  │  3. go test → run E2E test against local cluster               │
  │                        │                                        │
  │  4. cp9 topology refine → compare actual calls vs static       │
  │                           → mark untriggered edges as optional  │
  │                           → save per-API profile                │
  │                        │                                        │
  │  5. Next cp9 up → use refined profile if available             │
  │                   → fewer STUB services, faster startup         │
  └─────────────────────────────────────────────────────────────────┘
```

**Why both:**

- Static gives a **safe starting set** — you won't hit "service unavailable" errors mid-test
- Dynamic **prunes over-estimation** — for a specific test, you may need 4 services instead of 6
- cp9's STUB mode **buffers the gap** — over-estimated services run as lightweight stubs (~50MB each), so the cost of static over-estimation is low

#### Phase 1: Static Analysis (V1 — Ship First)

This is the `cp9 topology` pipeline described above. It produces `topology.json` with all possible `service.api → service.api` edges. cp9 uses this for initial REAL/STUB/SKIP classification.

**Trade-off accepted:** Static analysis includes edges that only fire on error paths, admin-only paths, or rare input conditions. This means some services classified as STUB could theoretically be SKIP. The cost: a few extra lightweight stub pods (~50MB each). The benefit: no surprise "connection refused" errors during testing.

#### Phase 2: Dynamic Optimistic Discovery

When topology.json is incomplete or unavailable (e.g., new service not yet scanned), cp9 falls back to **dynamic incremental discovery** — the "optimistic execution" approach:

```
DYNAMIC DISCOVERY FLOW (cp9 up --discover)

  Developer specifies: service=tidb-mgmt-service, api=CreateCluster
         │
         ▼
  Step 1: Start ONLY tidb-mgmt-service (REAL)
         │
         ▼
  Step 2: Send test request (CreateCluster)
         │
         ├── SUCCESS → done, record call graph
         │
         └── FAILURE → analyze error
                │
                ▼
  Step 3: Error analysis
         │
         ├── gRPC error: "connection refused" to dedicated-global-service:50051
         │   → Identify: dedicated-global-service is needed
         │   → Action: start dedicated-global-service as STUB
         │
         ├── HTTP error: 502 from billing-service
         │   → Identify: billing-service is needed
         │   → Action: start billing-service as STUB
         │
         └── Application error: "account service unavailable" in log
             → Pattern match against known service names
             → Action: start account-service as STUB
                │
                ▼
  Step 4: Retry test request
         │
         ├── SUCCESS → record discovered dependency graph
         │             save to .cp9/discovered-topology.json
         │
         └── FAILURE → goto Step 3 (next missing service)
                       Max iterations: 10 (configurable)
                       On budget exceeded: report partial graph + failures

  OUTPUT: .cp9/discovered-topology.json
          (can be merged back into topology.json)
```

**Error pattern matching:** cp9 recognizes these failure patterns to identify missing services:

| Error Pattern | Source | Identifies |
|---|---|---|
| `connection refused` on known gRPC port | gRPC dial error | Target service name from address/DNS |
| `no such host` for `*.svc.cluster.local` | k8s DNS resolution | Service name from DNS name |
| `context deadline exceeded` on gRPC call | gRPC timeout | Target service from interceptor logs |
| HTTP 502/503 to known service URL | HTTP proxy error | Target service from URL path |
| `"service_name" unavailable` in structured log | Application log (tilog) | Service name from log field |
| Pod CrashLoopBackOff with missing env var | k8s event | Dependency from config reference |

**Log analysis:** Services use tilog (structured JSON logging). cp9 parses log lines for:

```json
{"level":"error","msg":"failed to call account service","error":"connection refused","target":"account-central:50051","caller":"thirdparty/acl/impl.go:42"}
```

From this, cp9 extracts: target service = `account-central`, port = `50051`, caller = `thirdparty/acl/impl.go`.

#### Phase 3: Dynamic Refinement of Static Topology

After a successful test run (static topology → cp9 up → direct go test), cp9 can refine the topology by observing which calls actually fired:

```
REFINEMENT FLOW (cp9 topology refine)

  Prerequisites:
    - topology.json exists (from static analysis)
    - cp9 up completed (services running)
    - E2E test executed successfully
         │
         ▼
  Step 1: Collect actual call data
         │
         ├── Source A: gRPC interceptor logs
         │   Services already log every gRPC call via tigrpc interceptors:
         │   {"method":"/dedicated.ClusterService/CreateCluster",
         │    "target":"dedicated-global:50051","duration_ms":42}
         │
         ├── Source B: HTTP access logs
         │   Gin middleware logs every HTTP request:
         │   {"method":"POST","path":"/api/v1/unc/tenants/123/bulk_trigger",
         │    "status":200,"target":"billing:18888"}
         │
         └── Source C: Trace ID correlation
             All calls tagged with trace ID from original request.
             cp9 logs --trace-id <id> shows the actual call tree.
         │
         ▼
  Step 2: Build actual call graph from logs
         │
         For trace ID abc123:
           tidb-mgmt-service.CreateCluster
             → dedicated-global.ClusterService.CreateCluster ✓
             → account.AccountAPI.GetProject ✓
             → billing.POST /api/v1/unc/.../bulk_trigger ✓
             ✗ core-svc (in static topology, but NOT called)
             ✗ dataflow-service (in static topology, but NOT called)
         │
         ▼
  Step 3: Compare static vs actual
         │
         Static topology for CreateCluster says 6 services needed.
         Actual execution used 3 services.
         │
         Output: .cp9/profiles/CreateCluster.json
         {
           "api": "ClusterService.CreateCluster",
           "service": "tidb-mgmt-service",
           "static_deps": ["dedicated-global", "account", "billing",
                           "core-svc", "dataflow", "account.ACL"],
           "actual_deps": ["dedicated-global", "account", "billing"],
           "optional":    ["core-svc", "dataflow", "account.ACL"],
           "test_input":  "CreateCluster with region=us-west-2, plan=dedicated",
           "recorded_at": "2026-04-06T08:00:00Z"
         }
         │
         ▼
  Step 4: Future cp9 up uses profile (optional optimization)
         │
         cp9 up --profile=CreateCluster
           → REAL: tidb-mgmt-service (active)
           → REAL: dedicated-global (actual dep)
           → STUB: account, billing (actual dep, not active)
           → SKIP: core-svc, dataflow (optional, not needed)
           → Faster startup: 2 fewer stubs to generate
```

#### Integration with cp9 Lifecycle

```
LIFECYCLE INTEGRATION

  cp9 topology                     ← Generate/update topology.json (static)
       │
  cp9 up                           ← Classify from topology.json
       │                              (or --discover for dynamic fallback)
       │
  go test ./test/e2e -run X        ← Run E2E tests with cp9 session env
       │
  cp9 topology refine              ← Compare actual vs static
       │                              Save per-API profile
       │
  cp9 up --profile=X               ← Use refined profile (optional)
       │                              Fewer services, faster startup
       │
  cp9 topology validate            ← Verify config covers all deps
```

**New cp9 commands for combined approach:**

```
cp9 topology                       # Generate topology.json from source (static)
cp9 topology query --service=X     # Query dependencies for an API
  --api=Y --depth=all
cp9 topology validate              # Validate cp9.yaml covers all deps
  --config=cp9.yaml
cp9 topology refine                # Refine topology from actual test execution
  --trace-id=abc123                # (uses trace ID to collect actual calls)
cp9 topology profiles              # List saved per-API profiles
cp9 up --discover                  # Dynamic discovery mode (no topology.json)
cp9 up --profile=CreateCluster     # Use refined profile for faster startup
```

#### When to Use Which Approach

| Scenario | Approach | Why |
|---|---|---|
| Normal development | Static (topology.json) | Fast, safe, comprehensive |
| New service not yet in topology | Dynamic (--discover) | No static data available |
| Optimizing test startup time | Refine + profile | Prune unnecessary stubs |
| CI/CD pipeline | Static only | Deterministic, reproducible |
| Debugging "why is this service needed?" | Query + refine | Trace actual call path |
| First-time setup (no topology.json) | Dynamic → export to static | Bootstrap the topology |

#### Bootstrap: From Zero to topology.json

For first-time setup when no topology.json exists:

```bash
# Option A: Static analysis (preferred, requires repo access)
$ cp9 topology --repos ../tidb-management-service,../account,../billing,../dataflow-service

# Option B: Dynamic discovery (slower, but works without repo analysis)
$ cp9 up --discover --entry=tidb-mgmt-service --api=CreateCluster
# ... iterative discovery completes ...
$ cp9 topology export  # export discovered topology to topology.json

# Option C: Combine both
$ cp9 topology --repos ...          # static baseline
$ cp9 up && go test ./test/e2e      # run tests
$ cp9 topology refine --trace-id X  # refine from actual execution
```

#### tiinf Module Dependencies

tiinf modules (`tigrpc`, `tieventbus`, `auth`, `rbac`, `auditlog`) are infrastructure libraries, not services. They provide transport and middleware but do not themselves make cross-service API calls. The cross-service calls happen through the repo-specific client directories listed above, which use tiinf's `tigrpc.NewClientConn()` for gRPC connection management.

Exception: `tiinf/sdk/auditlog` may implicitly call an audit log collection service. `cp9 topology` scans tiinf source in the Go module cache to detect any hidden service calls.

---

### cp9.yaml Schema

```yaml
# cp9.yaml — checked into repo, shared by all developers
# Override locally with cp9.local.yaml (gitignored)
apiVersion: cp9/v1               # required first field. cp9 validates
                                 # version on load, rejects unknown versions.
                                 # Migration path for future schema changes.
cluster:
  name: "cp9-${USER}"         # required, string, env vars expanded
  k3d:
    agents: 2                    # optional, int, default 2
    memory: "8G"                 # optional, string with unit (G/M), default 8G

topology:
  # Inputs for smart service classification
  app_delivery: ../app-delivery          # path to app-delivery repo
  service_topology: ./topology.json      # pre-computed api-level-service-topology
  staging_logs_api: https://logs.internal/api/v1  # optional: staging log query endpoint

services:
  # NOTE: Services are auto-classified by the topology algorithm.
  # Manual overrides below for edge cases only.
  skip:                          # force-skip these services regardless of topology
    - o11y-collector
    - cost-metering
    - monitoring-agent

aws:
  profile: "${USER}"             # maps to ~/.aws/config profile
  region: us-west-2
  ttl: 72h                       # max 168h, reaper enforced
  # Services create their own AWS resources (SQS, S3, DynamoDB)
  # using developer's local CLI identity — same as prod

database:
  provider: tidb-cloud-zero      # trusted, stable
  # One Zero instance per cp9 up. Each REAL service gets its own
  # database (CREATE DATABASE <service_name>) within that instance.
  # cp9 migrate dumps schema from staging and seeds per-service databases.
  # DSN per service: same host:port, different database name.
  # Fallbacks (in order):
  #   1. tidb-cloud-starter (free tier, 5GiB, 50M RU/month)
  #   2. local-mysql (MySQL 8.0 in k3d, lose TiDB compatibility)

readiness:                       # per-service boot validation
  tidb-mgmt-service:
    type: api                    # call a real business API
    method: GET
    path: /api/v1beta1/projects  # list projects — proves full stack works
    expect_status: 200

retry:                           # agent retry budgets (per feature-cycle)
  build: 3                       # max retries for build failures
  deploy: 2                      # max retries for deploy failures
  test: 3                        # max retries for test failures
  total: 5                       # max retries across ALL operations combined
                                 # (total is a cap — even if per-op budget remains,
                                 #  total=5 means at most 5 retries in one feature-cycle)
  on_budget_exceeded: draft-pr   # open draft PR with failure context
```

```yaml
# cp9.local.yaml — gitignored, per-developer overrides
# Agent or developer creates this per feature

sources:                         # REAL services: build from source
  tidb-mgmt-service:
    path: ../tidb-management-service
    ref: feature/cmek-m2         # branch, tag, or commit hash
  billing-svc:
    path: ../billing
    ref: main                    # use main branch
  account-svc:
    path: ../account
    ref: v2.3.1                  # pin to a tag

overrides:                       # topology overrides for edge cases
  my-new-service:
    namespace: my-namespace
    classification: real
    source: ../my-new-service
    ref: feature/my-branch
```

### Get Started

```yaml
# cp9.yaml — minimal config
apiVersion: cp9/v1
cluster:
  name: "cp9-${USER}"
topology:
  app_delivery: ../app-delivery
  service_topology: ./topology.json
database:
  provider: tidb-cloud-zero
```

```yaml
# cp9.local.yaml — your feature work
sources:
  tidb-mgmt-service:
    path: ../tidb-management-service
    ref: feature/cmek-m2
```

Then run:

```bash
cp9 up    # creates k3d, classifies services, builds real, stubs deps, done
```

### CRD Installation

Custom K8s controllers are treated as normal microservices — classified as REAL/STUB/SKIP
by the topology engine. CRDs are analogous to database table schemas, and CRs (custom
resource instances) are analogous to table data. Just as `cp9 migrate` dumps schema from
staging instead of re-running each service's migration tool, **CRD install dumps definitions from the staging cluster instead of discovering them from source repos**.

```
CRD install (during cp9 up, parallel with db_provision/aws_inject/build/stubs):
  For each controller service in crds.mappings:
    1. kubectl get crd from staging (filtered by API group)
    2. Cache in .cp9/crd-dumps/<service>/crds/ (24h TTL)
    3. kubectl apply -f to local k3d
    4. Optionally seed CRs from staging (--seed=config-crs|full)

  Secrets injection:
    For each service in cp9.local.yaml secrets:
    1. Parse .env.dev file or inline values
    2. Generate K8s Secret YAML (Opaque, stringData)
    3. kubectl apply -f to local k3d

  # cp9.yaml
  crds:
    staging_access:
      kubeconfig: ~/.kube/staging
      context: staging-cluster
    mappings:
      tidb-operator:
        api_groups: ["pingcap.com"]
      infra-api-controller:
        api_groups: ["infra.tidbcloud.com"]
        exclude: true   # explicitly excluded
    seed:
      default: crds-only

  # cp9.local.yaml
  secrets:
    tidb-mgmt-service:
      env_file: ../tidb-management-service/.env.dev
```

Package: `internal/crd/` (mirrors `internal/migrate/` structure).

### Cluster Lifecycle

```
cp9 up                           # Re-entrant: tracks state in .cp9/state.json
  |                              # Re-running skips completed steps, retries failed ones
  |
  +-> 1. [~15s] Create k3d cluster (k3s in Docker, arm64 native)
  |      k3d cluster create cp9-${USER} \
  |        --agents 2 --k3s-arg "--disable=traefik"
  |
  |   STATE MACHINE: Each step is a named state in .cp9/state.json.
  |   States: cluster_create | crd_install | db_provision | aws_inject |
  |           classify | build_real | gen_stubs | config_gen |
  |           db_seed | deploy | readiness
  |   Transitions:
  |     - SUCCESS → next state
  |     - TRANSIENT_FAILURE → retry (up to 3x per step)
  |     - PERMANENT_FAILURE → abort, print failed state + error
  |     - PARTIAL (e.g., 5/8 services deployed) → record progress,
  |       re-run resumes from failed service, skips completed ones
  |   On abort: `cp9 up` re-run resumes from the failed state.
  |   On kill (SIGINT/SIGTERM): current state marked "interrupted",
  |     next `cp9 up` retries that state.
  |
  +-> 2. [~5s] Classify services (smart topology algorithm)
  |      Read app-delivery + api-level-service-topology
  |      Walk dependency graph from cp9.local.yaml sources:
  |      Output: .cp9/service-graph.json
  |      Print: "REAL: 5 services | STUB: 8 services | SKIP: 29 services"
  |
  |   ┌── PARALLEL ───────────────────────────────────────────────┐
  |   │                                                           │
  +-> 3. [~30-120s] Provision TiDB Cloud Zero database            │
  |      POST zero.tidbapi.com/v1beta1/instances                  │
  |      (no auth required, returns connectionString directly)    │
  |      Fallback chain (explicit triggers):                      │
  |        Zero HTTP error/timeout → TiDB Starter                 │
  |        Starter failure → local MySQL 8.0 in k3d               │
  |      (this is the bottleneck — everything else parallels it)  │
  |   │                                                           │
  +-> 4. [~10s] Inject AWS config                                 │
  |      Mount ~/.aws/config and ~/.aws/credentials into pods     │
  |      Set AWS_PROFILE=${USER} in pod env                       │
  |      Tag all resources: cp9-session=<id>, cp9-owner=${USER}   │
  |   │                                                           │
  +-> 5. [~30-60s] Build REAL services from source                │
  |      For each REAL service:                                   │
  |        git checkout <ref> in source dir                       │
  |        docker build (linux/arm64)                             │
  |        k3d image import                                       │
  |   │                                                           │
  +-> 6. [~15-30s] Generate STUB services                         │
  |      For each STUB service:                                   │
  |        Find proto/swagger definition                          │
  |        Generate stub server (gRPC or HTTP)                    │
  |        Build minimal container                                │
  |        k3d image import                                       │
  |   │                                                           │
  +-> 7. [~5s] Install CRDs + secrets from staging                 │
  |      Dump CRDs from staging (by API group), apply to k3d     │
  |      Parse .env.dev → K8s Secrets, apply to k3d              │
  |   └───────────────────────────────────────────────────────────┘
  |
  +-> 8. [~10s] Generate config overlays
  |      Kustomize overlays from real staging configs:
  |        DB -> TiDB Cloud Zero connection string
  |        AWS -> inherited from local profile
  |        gRPC/HTTP -> k3d internal K8s DNS names
  |        STUB services -> stub pod DNS names
  |
  +-> 9. [~30s] Seed databases from staging
  |      cp9 migrate (dump staging schema + seed for REAL services)
  |
  +-> 10. [~60s] Deploy services to k3d
  |       Deploy REAL services (from source builds)
  |       Deploy STUB services (from generated stubs)
  |       SKIP services: nothing deployed
  |
  +-> 11. [~30s] Readiness check
          For each REAL service:
            if readiness.type == "api":
              Call business API, expect readiness.expect_status
            else:
              Call /healthz, expect 200
          For each STUB service:
            Call /healthz on stub server
          Failure behavior:
            REAL service readiness failure → PARTIAL state.
              cp9 up continues, marks service as degraded.
              Report shows which services failed readiness.
              Developer can fix and re-run `cp9 up` (resumes from readiness).
            STUB service readiness failure → TRANSIENT_FAILURE, retry 3x.
              After 3 retries → PERMANENT_FAILURE, abort.
              (Stub failures indicate a stub generation bug, not a code issue.)
          Report: "5 real + 8 stubs ready" or "4/5 real ready, 1 failing: ..."

  Total: ~3-5 minutes (lighter than v1 — fewer services to build/pull)
  Timing breakdown printed at end of cp9 up

cp9 destroy
  |
  +-> k3d cluster delete cp9-${USER}
  +-> Query Resource Groups Tagging API for all resources tagged cp9-session=<id>
  +-> Delete ALL discovered AWS resources (SQS, S3, DynamoDB, whatever)
  +-> Delete TiDB Cloud Zero cluster
  +-> Print cost of destroyed resources (automatic)
  +-> Report: "Cleaned up 14 AWS resources, 1 database, 1 k3d cluster"
```

**Request flow through the local cluster:**

```
  E2E Test (go test)
       │
       │ POST /v1beta1/cmekKeys
       v
  ┌─────────────────────┐    gRPC     ┌─────────────────────┐
  │ tidb-mgmt-service   │ ────────── >│ billing-svc         │
  │ (REAL, from source) │             │ (STUB, proto-gen)   │
  │                     │             │ returns: {"ok":true}│
  │                     │    HTTP     ├─────────────────────┤
  │                     │ ────────── >│ account-svc         │
  │                     │             │ (REAL, from source) │
  └──────────┬──────────┘             └─────────────────────┘
             │
             │ Response: 200, key_id=xyz
             │ Header: X-Trace-Id: abc123
             v
  E2E Test verifies response + polls for state=ACTIVE
  On failure: cp9 logs --trace-id abc123
```

**Service discovery:** k3s includes CoreDNS. All services (real and stub) resolve via standard K8s DNS (`<service>.<namespace>.svc.cluster.local:<port>`). Host access for agent-driven tests: `k3d cluster create ... -p "8080:80@loadbalancer"` + `kubectl port-forward`.

### Config Generation

No Go templates. cp9 generates Kustomize overlays from real staging ArgoCD configs per service:

1. **Input:** Real staging config from app-delivery repo kustomize paths (primary). Fallback: `kubectl get application` from staging cluster if app-delivery repo is unavailable.
2. **Patch:** DB DSN → Zero connection string, AWS → developer's profile, gRPC/HTTP → k3d DNS names (including stub service DNS names)
3. **Output:** Kustomize overlay applied at deploy time. Regenerated on every `cp9 up`.

Benefits: real configs as baseline (not synthetic), no per-service template maintenance, config drift caught automatically.

### AWS Resource Management

Services provision their own AWS resources at runtime (SQS queues, S3 buckets, DynamoDB tables, etc.) using the developer's local CLI identity — same credential chain as production. cp9 doesn't pre-create resources — it tracks and cleans them.

```
AWS resource lifecycle:

  cp9 up
    │
    └─> Inject AWS creds into pods (mount ~/.aws/*)
        │
        └─> Services create resources at runtime
            (SQS, S3, DynamoDB — tagged cp9-session=<id>)
                │
  cp9 status    │     cp9 destroy
    │           │       │
    └─> Tagging │       └─> Query Tagging API (cp9-session=<id>)
        API  <──┘           └─> Delete ALL discovered resources
                                └─> Report: "Cleaned up 14 resources"

  destroy reaper stage (safety net)
    └─> TTL/session cleanup: destroy envs older than policy
        └─> AWS Budget alarm (last resort)
```

```
IAM Setup (cp9 setup):
  - Shared IAM role `cp9-developer` (created once by infra team, no Terraform)
  - ABAC with session tags: cp9-owner=${USER} scopes all resource access
  - Permission boundary: cp9-developer role is restricted to the dev AWS account only.
    Cannot access staging or production accounts. Resource-level conditions enforce
    that only resources tagged cp9-owner=${USER} can be modified/deleted.
    Services that need cross-account access (rare) use a separate staging-read-only
    role with explicit opt-in.
  - `cp9 setup` validates assume-role. Does NOT create IAM resources.

Config Injection:
  - Mount ~/.aws/config + ~/.aws/credentials into pods
  - Set AWS_PROFILE, AWS_REGION. Standard SDK credential chain — same as prod.

Resource Tracking + Cleanup:
  - All resources tagged: cp9-session=<id>, cp9-owner=${USER}
  - Tagging API (GetResources) for instant inventory
  - `cp9 destroy`: query tags → delete all resources → report
  - `cp9 destroy`: tag-scoped cleanup plus internal force-purge stage
  - `cp9 status`: session/resource/cost reporting when available
  - CloudTrail for audit forensics only
  - AWS Budget alarm
```

### Secrets Management

Services in staging use K8s secrets and AWS Secrets Manager for API keys, service tokens, and TLS certificates. Locally, secrets are injected as part of the CRD install step (`internal/crd/secrets.go`):

```
Secrets strategy (per service):
  1. AWS secrets: injected via local AWS credentials (same as prod)
  2. K8s secrets: cp9 generates an Opaque Secret per service from
     cp9.local.yaml secrets entries (.env.dev files or inline values)
  3. TLS: k3s generates self-signed certs for internal service communication
  4. Missing secrets: cp9 doctor warns

  # cp9.local.yaml (gitignored)
  secrets:
    tidb-mgmt-service:
      env_file: ../tidb-management-service/.env.dev
    billing-svc:
      values:
        STRIPE_KEY: sk_test_xxx
        WEBHOOK_SECRET: whsec_xxx

  Env file format: KEY=VALUE, KEY="VALUE", KEY='VALUE', export KEY=VALUE,
  blank lines and # comments are ignored. Inline values override env file.
```

Services that hard-depend on external secrets (e.g., third-party API keys) must have those keys in the developer's `.env.dev` file. cp9 does not manage or provision external secrets.

### Database Seeding

Instead of discovering and running migration tools per service, cp9 takes a simpler approach: **dump schema and seed data directly from the staging database**.

Most TiDB Cloud services use gorm with `github.com/go-gormigrate/gormigrate/v2` for migrations. Replicating each service's migration toolchain locally adds complexity for little benefit — the staging database already has the correct schema and representative data.

```
Strategy: Mirror staging DB → local TiDB Cloud Zero

  cp9 migrate
    |
    +-> For each REAL service's database:
    |
    |   1. Connect to staging DB (read-only credentials)
    |      - Staging DSN per service discovered from staging K8s secrets
    |        or specified in cp9.yaml
    |
    |   2. Dump schema (DDL only):
    |      mysqldump --no-data --routines --triggers <staging_db>
    |      → creates all tables, indexes, constraints
    |
    |   3. Dump seed data (configurable):
    |      - Default: schema-only (empty tables)
    |      - With --seed: dump selected rows from staging
    |        (config tables, enum tables, system records)
    |      - With --seed=full: full data mirror (large, slow)
    |
    |   4. Apply to local TiDB Cloud Zero:
    |      mysql -h <zero_host> <service_db> < schema.sql
    |      mysql -h <zero_host> <service_db> < seed.sql
    |
    +-> Cache dumps in .cp9/db-dumps/<service>/ with timestamp
        Re-use cached dump if < 24h old (--fresh to force re-dump)

  cp9 migrate --reset
    → DROP DATABASE + re-create + re-apply schema + seed
```

```yaml
# cp9.yaml — database seeding config
database:
  provider: tidb-cloud-zero
  staging_access:
    method: port-forward           # kubectl port-forward to staging cluster
    kubeconfig: ~/.kube/staging    # kubeconfig for staging cluster
    # Alternative: direct connection if VPN/bastion available
    # method: direct
    # host_override: staging-db.internal:4000

  seed:
    default: schema-only           # schema-only | config-tables | full
    overrides:
      account-svc:
        tables:                    # specific tables to seed with data
          - sys_config
          - permission_definitions
          - role_definitions
      billing-svc:
        tables:
          - currency_config
          - pricing_plans
```

**Why this approach:**

- **No migration tool replication.** Services use gorm/gormigrate — cp9 doesn't need to know which.
- **Schema is always correct.** Staging DB is the source of truth. No risk of migration ordering bugs.
- **Seed data solves empty-table failures.** Services often crash with empty config/lookup tables. Mirroring staging data for these tables avoids cascading failures.
- **Fast.** Schema-only dump for a typical service is < 1MB, applies in seconds.

**Caveats:**

- Requires read access to staging databases (via port-forward or direct connection).
- Seed data from staging may contain PII — `cp9 migrate --seed` should sanitize or filter as needed.
- If staging schema is ahead of a feature branch, the local schema may include extra columns/tables. In practice this rarely matters (additive schema changes are backward-compatible).

### Cost Model

**Background:** TiDB Cloud dedicated clusters create a per-(tenant,project,region) EKS dataplane cluster. When the dev controlplane moves local, dev TiDB clusters are no longer needed for controlplane-focused feature development, eliminating the entire dev dataplane stack.

```
CURRENT STATE: Dev environment infrastructure
  dev-base-us-west-2 (controlplane cluster):
    27 nodes (SPOT + ON_DEMAND mix)
  Dev dataplane:
    Per-(tenant,project,region) EKS clusters for dev TiDB
    EC2 nodes, EBS volumes, NAT gateways, ELBs
    These clusters exist solely to test controlplane features in dev.
  Staging (RETAINED):
    staging-base-us-west-2 (controlplane) + staging dataplane
    Retained for PM QA + performance testing

PROPOSED STATE: cp9 (per-developer ephemeral)
  Eliminated:
    dev-base controlplane cluster
    Dev dataplane EKS clusters + resources
    Dev share of shared costs (KMS, ACM, VPC endpoints, ECR, etc)

  New costs (per-developer ephemeral):
    Local k3d compute: no cloud cost (developer's Mac)
    TiDB Cloud Zero: free tier
    AWS resources created by services: minimal per developer

  + No developer contention (isolated environments)
  + No shared cluster instability
  + Lighter local footprint (only feature-path services run real)
  + Spot interruption risk eliminated for dev (local k3d is stable)
```

### Command Reference

```
cp9 — Local ephemeral control plane for TiDB Cloud development

USAGE:
  cp9 <command> [flags]

CROSS-REPO COMMANDS (cp9-native):
  up              Stand up local k3d cluster with smart service classification
  destroy         Tear down cluster and clean up all resources (local + AWS)
  sync            Refresh service topology from app-delivery + api-level-topology
  deploy <svc>    Build and deploy a specific service to local k3d
  status          Show environment health and service classification
  doctor          Pre-flight checks, validate service graph, warn on issues
  migrate         Seed databases from staging schema dump (--reset to drop + re-seed, --seed for data)
  logs            Query logs across services (--trace-id, --service, --level)
  test            Run E2E tests against local k3d (--feature <name>)
  cost            Show AWS resource spend for current session
  reap            TTL enforcement — destroy expired environments
  setup           Validate AWS assume-role and prerequisites
  onboard <repo>  Generate AGENTS.md + TOPOLOGY.md + cp9.yaml for a repo
  topology        Generate TOPOLOGY.md from app-delivery + proto/swagger + source
  stub-gen <svc>  Regenerate proto/swagger stub for a specific service

SINGLE-REPO COMMANDS: Use `make` in each repo.
  cp9 does NOT duplicate makefile targets (build, lint, fmt, generate, etc.).
  Each repo's Makefile must be complete. cp9 focuses on cross-repo orchestration.

FLAGS:
  --json          Machine-parseable JSON output (all commands)
  --help          Show help for any command
  --verbose       Detailed output for debugging

GET STARTED:
  1. cp9 setup                    # validate prerequisites
  2. cp9 doctor                   # pre-flight checks
  3. cp9 up                       # stand up environment (~3-5 min)
  4. cp9 status                   # see what's running
  5. cp9 logs --service <svc>     # check a service
  6. cp9 destroy                  # clean up when done

EXAMPLES:
  cp9 up                                    # default: classify + build + deploy
  cp9 deploy tidb-mgmt-service              # rebuild and redeploy one service
  go test ./test/e2e -run TestCMEKM2        # run feature E2E tests
  cp9 logs --trace-id abc123                # cross-service trace
  cp9 stub-gen billing-svc                  # regenerate billing stub
  cp9 onboard ../new-service-repo           # onboard a new repo
```

### Deliverables (O1)

| # | Deliverable | Description |
|---|---|---|
| D1 | `cp9` CLI | Go binary, `go install` from GitHub. CLI framework: [ticat](https://github.com/innerr/ticat) for native command chaining (`:` operator) and session management; [cobra](https://github.com/spf13/cobra) as fallback if ticat stalls (single maintainer, 26 stars). `--json` output, `--help` self-describing. Error categories: transient/permanent/config/resource with exit codes. CI: GitHub Actions builds on tag push. |
| D2 | `cp9.yaml` + `cp9.local.yaml` | `apiVersion: cp9/v1`. Complete reference doc + get-started example. Smart topology inputs, manual overrides, readiness checks, retry budgets. `cp9.local.yaml` (gitignored) for source paths with branch/tag/commit + overrides. |
| D3 | Smart service classifier | Topology algorithm: app-delivery + topology.json → real/stub/skip classification. `cp9 doctor` validates graph. |
| D3.1 | `cp9 topology` — topology.json generator | Static analysis tool: scans 4 repos (tidb-management-service, account, billing, dataflow-service) via Go AST. Proto scanner + route scanner + client call scanner + handler-client linker → topology.json. Query mode: given service + API, list all transitive service dependencies. |
| D4 | Proto/swagger stub generator | Generate server-side stubs from API definitions. Accuracy inputs: staging logs, source code. Deployed as real k3s pods. |
| D5 | AWS resource tracker + reaper | Tagging API inventory + tagged cleanup on destroy. Internal destroy reaper for TTL/session cleanup, `cp9 status` for reporting, `cp9 setup` to validate assume-role. |
| D6 | TiDB Cloud Zero integration | Fallback chain: Zero → Starter → local MySQL. Explicit triggers documented. |
| D7 | Kustomize overlay generator | Config overlays from real staging ArgoCD configs. Patches DB/AWS/gRPC/stub addresses. No templates. |
| D8 | `cp9 doctor` | Pre-flight checks. Validates service graph against topology. Warns on low RAM, missing dependencies, stale stubs. |
| D9 | `cp9 migrate` | Staging DB schema dump + seed. Dumps DDL from staging, optionally seeds config/lookup tables. `--seed` for data, `--reset` for drop + re-apply. Cached dumps with 24h TTL. |

### Success Criteria (O1)

- `cp9 up` creates a working control plane in < 2 minutes on M1/M2/M3 Mac
- Smart classification runs only feature-path services as real (not all 42)
- Stub services handle API calls from real services without crashes
- Feature service handles a real API request through the full RPC chain (real → real, real → stub where appropriate)
- `cp9 destroy` leaves zero resources (local + ALL AWS resources tracked)
- Services from ALL teams (not just tidb-management) can be classified and run
- **dev-base EKS cluster is technically replaceable for local development** (full decommission requires O2 for E2E workflow replacement)

### cp9 Self-Test Strategy

cp9 itself is tested at three levels:

| Level | Scope | What's Tested |
|---|---|---|
| **Unit** | `internal/` packages | Config parsing, topology classification algorithm, state machine transitions, stub generation, Kustomize overlay generation, tag filtering. Standard `go test ./internal/...`. |
| **Integration** | 3 real + 5 stub services on k3d | `cp9 up` with tidb-mgmt-service (real), billing-svc (real), account-svc (real), plus stub services. Verifies full lifecycle: classify → build → stub-gen → deploy → readiness → destroy. Runs in ~5 min. |
| **CI** | On PR to cp9 repo | Unit tests on every PR. Integration test on merge to main (needs Docker-in-Docker CI runner). |

---

## O2: Agent-Driven E2E Delivery + Agent Plugin

**Goal**: Agent delivers a feature from spec to passing E2E in < 60 minutes. Human intervention limited to: spec approval, design doc review, test plan review, and code review. Agent plugin ships as part of O2.

### Agent Delivery Pipeline

```
Feature spec (PRD / design doc)
        |
        | Human: "implement this feature"
        v
  1. Agent reads spec + AGENTS.md + TOPOLOGY.md
        |
  ┌─ GATE 1 ───────────────────────────────────────────────┐
  │ Agent produces: Detail Design Doc                      │
  │ >>> HUMAN REVIEW REQUIRED — agent STOPS <<<            │
  └────────────────────────────────────────────────────────┘
        |
  2. Agent makes changes (proto, handler, config, etc.)
        |
  3. Validate API safety (uses repo Makefile, NOT cp9):
     +-> proto changed?   → make dev (generate, lint, AIP)
     +-> swagger changed? → regenerate JSON, diff for breaks
     +-> CRD changed?     → validate YAML schema
     +-> always:          → make fmt, unit tests on changes
        |
        +-> FAIL? Agent reads error, fixes, retry
        |
  ┌─ GATE 2 ───────────────────────────────────────────────┐
  │ Agent produces: Detail Test Plan + Go test code        │
  │ (in proto/swagger repo)                                │
  │ >>> HUMAN REVIEW REQUIRED — agent STOPS <<<            │
  └────────────────────────────────────────────────────────┘
        |                          +-------------------+
  4. cp9 deploy <service>  <-----> | cp9 environment   |
  5. go test ./test/e2e -run X     | (from O1)         |
        |                          +-------------------+
        +-> FAIL? cp9 logs --trace-id <id>
        |         Agent reads correlated logs, fixes, goto 3
        |
  6. All tests pass → Agent opens PR
        |
        v
  Human review → PR → Staging → Prod
```

**Two human gates, no more:** Design doc (before code) and test plan (before execution). Single-repo operations (build, lint, fmt, generate) use each repo's Makefile — cp9 is not invoked for these.

### Tracing & Logs

Services already propagate trace IDs in HTTP response headers and gRPC metadata. cp9 leverages this — no Jaeger, no OpenTelemetry collector. Just structured logs + grep.

```
How it works:

1. E2E test sends request -> response includes trace ID header
   (e.g., X-Trace-Id: abc123 or grpc-metadata: trace-id=abc123)

2. All services tag every log line with trace ID
   (already implemented: services use tilog with trace context)

3. cp9 logs --trace-id abc123
   |
   +-> For each pod in k3d:
   |     kubectl logs <pod> | grep "abc123"
   |
   +-> Aggregate, sort by timestamp, show cross-service flow:
         [10:00:00.001] tidb-mgmt-portal    | POST /v1beta1/cmekKeys -> handler
         [10:00:00.012] tidb-mgmt-service    | CreateCmekKey RPC received
         [10:00:00.015] nextgen-global-svc   | CreateEncryptionKey called (STUB)
         [10:00:00.020] billing-svc          | BillingCheck called (STUB)
         [10:00:00.025] tidb-mgmt-service    | Response: 200, key_id=xyz

Other log commands:
  cp9 logs --all                  # all services, last 100 lines each
  cp9 logs --service tidb-mgmt    # single service
  cp9 logs --since 5m             # last 5 minutes
  cp9 logs --level error          # errors only

Output: --json for agents, table for humans
```

### AGENTS.md + TOPOLOGY.md

The `tidb-management-service/AGENTS.md` pattern is proven. `cp9 onboard <repo>` auto-generates a draft AGENTS.md + TOPOLOGY.md + cp9.yaml for any repo.

**TOPOLOGY.md generation inputs:**

1. **app-delivery manifests** — service registry and deployment topology
2. **Proto/swagger analysis** — API definitions, import graphs, dependency chains
3. **Source code** — actual service implementation for deeper understanding
4. **Pre-computed api-level-service-topology** — the canonical call graph (see [API-Level Service Topology Generation](#api-level-service-topology-generation))

`cp9 topology` combines all four inputs to produce TOPOLOGY.md, mapping each service's API type (gRPC/HTTP/CRD), dependencies, callers, and CRDs. The same static analysis pipeline that generates `topology.json` powers TOPOLOGY.md generation — `topology.json` is the machine-readable output, TOPOLOGY.md is the human-readable output.

### E2E Test Harness

Standard Go test code, driven by `go test` / `go test -bench`. No YAML test definitions — tests are plain Go using the [`testing`](https://pkg.go.dev/testing) package. Test code lives in the same repo as the proto/swagger definitions (the service's API contract and its tests co-locate).

**Agent workflow:**

1. Agent produces a Detail Test Plan (human review gate)
2. After approval, agent generates Go test code
3. Test code is submitted as part of the feature PR in the proto/swagger repo

**Example:** `e2e/cmek_m2_test.go` in the service repo:

```go
func TestCMEKM2_CreateAndWaitReady(t *testing.T) {
    client := cp9test.NewClient(t) // connects to local k3d
    resp := client.POST(t, "/v1beta1/projects/{project_id}/cmekKeys", map[string]any{
        "region": "us-west-2", "spec": map[string]any{...},
    })
    cp9test.RequireStatus(t, resp, 200)
    cp9test.RequireJSONPath(t, resp, "state", "CREATING")
    traceID := resp.Header.Get("X-Trace-Id")

    cp9test.PollUntil(t, client, fmt.Sprintf("/v1beta1/projects/{project_id}/cmekKeys/%s", keyID),
        cp9test.JSONPathEquals("state", "ACTIVE"), 5*time.Minute)
    t.Logf("trace_id=%s", traceID)
}
```

Feature E2E tests run as direct `go test` invocations against the local k3d environment, using cp9 session environment and `pkg/cp9test` helpers for kubeconfig, API endpoint, and credentials.

**Failure diagnostics:** Go test output includes trace IDs captured from response headers. On failure, `cp9 logs --trace-id <id>` correlates cross-service logs.

### Agent Plugin

The agent plugin ships with O2. It's the delivery interface for single-agent workflows.

```
cp9-plugin/
  .claude-plugin/
    plugin.json
  plugin.json                     # VS Code Copilot format
  skills/
    up/SKILL.md                   # Stand up local control plane
    deploy/SKILL.md               # Build and deploy a service
    test/SKILL.md                 # Run E2E tests
    logs/SKILL.md                 # Query logs (supports --trace-id)
    destroy/SKILL.md              # Tear down and cleanup
    feature/SKILL.md              # Full feature delivery: spec → PR
    status/SKILL.md               # Show env health (--json)
    sync/SKILL.md                 # Refresh service topology
    stub-gen/SKILL.md             # Regenerate a stub service
  agents/
    feature-dev.agent.md          # Single-feature delivery agent
                                  # (includes human review gates)
  hooks/
    hooks.json
      PostToolUse:
        - matcher: "Write|Edit"
          # After code edits, run repo-owned compatibility checks
        - matcher: "cp9 deploy"
          # After deploy, auto-run readiness check
  settings.json                   # Default agent: feature-dev
```

**Installation:**

```bash
# Claude Code
claude --plugin-dir ./cp9-plugin

# VS Code (settings.json)
"chat.pluginLocations": { "./cp9-plugin": true }
```

### Deliverables (O2)

| # | Deliverable | Description |
|---|---|---|
| D10 | `cp9 onboard <repo>` | Auto-generate draft AGENTS.md + TOPOLOGY.md + cp9.yaml. Inputs: app-delivery + proto/swagger + source code + api-level-service-topology. |
| D11 | TOPOLOGY.md generator | `cp9 topology` from all four inputs. Maps API type, dependencies, callers, CRDs per service. |
| D12 | E2E test harness | Go test code using `testing` package, driven by `go test`. Tests co-locate with proto/swagger definitions. Agent generates test code after human-approved test plan. |
| D14 | `cp9 logs --trace-id` | Aggregate logs from all pods, filter by trace ID, cross-service timeline. |
| D15 | Repo-owned compatibility checks | Cross-repo API breaking-change checks belong in repo-owned CI/tooling; the cp9 product CLI no longer ships a `verify` command. |
| D16 | cp9-plugin package | Full plugin: skills, agents, hooks. Compatible with Claude Code and VS Code Copilot. Single-agent feature delivery with human review gates. |
| D17 | feature-dev agent | Single-feature delivery agent: spec → design doc (gate) → code → test plan (gate) → deploy → test → PR. |

### Success Criteria (O2)

- Agent delivers a CMEK M2-equivalent feature from spec to passing E2E in < 60 minutes of agent time
- Human review gates enforced: agent produces design doc + test plan, waits for approval
- Agent delivers a real backlog feature E2E
- Human intervention limited to: feature spec, design doc review, test plan review, code review
- `cp9 logs --trace-id` shows cross-service request flow for any failing test
- Cross-repo breaking changes detected through repo-owned CI/tooling and the service topology contract
- cp9-plugin installable in one command (both Claude Code and VS Code)
- Agent does NOT invoke cp9 for operations that belong in a repo's Makefile

---

## Milestone: Shut Down dev-base EKS + Dev Dataplane

> **This is the primary deliverable of controlplane9.**
> Everything above (O1 + O2) exists to reach this point.

After O2, dev-base and dev dataplane have zero remaining use cases for controlplane-focused development:

| What dev infrastructure provides today | Replaced by |
|---|---|
| Shared K8s for manual testing | `cp9 up` — per-developer local k3d with smart classification (O1) |
| Long-lived dev environment | Ephemeral clusters, created/destroyed in minutes (O1) |
| Cross-service integration testing | Direct `go test` + E2E harness with trace IDs (O2) |
| "Deploy and check" workflow | Agent-driven feature-delivery workflow with human review gates (O2) |
| Shared database state | TiDB Cloud Zero per-developer instances, 30-day TTL (O1) |
| Dev TiDB clusters for controlplane testing | No longer needed — local cp9 stubs the dataplane interface |

**Shutdown sequence:**

```
1. All teams confirm cp9 up runs their services locally
   (tracked per-team in migration dashboard)
        |
2. Freeze new deployments to dev-base
        |
3. Final parallel validation: every team's E2E suite passes on cp9
        |
4. Decommission dev-base EKS (27 nodes) + dev dataplane
   (per-tenant EKS clusters, EC2, EBS, NAT, ELBs)
        |
5. Delete associated IAM roles, ECR policies, Route53 entries
        |
6. Retain staging-base + staging dataplane (PM QA + perf testing)
```

**Savings:** Significant (full scope including dev dataplane). See [Cost Model](#cost-model) for details.

This gate is the go/no-go for O3.

---

## O3: Multi-Agent Orchestration (Future)

Multiple agent sessions work on the same codebase in parallel. Shared k3d cluster with namespace-per-agent isolation. DNS rewriting for cross-agent service resolution. Git worktrees for isolated feature branches.

**Not detailed here.** O3 is an enhancement built on top of O1 + O2 infrastructure. Dev-base is already dead by this point. O3 design will be produced in a future /office-hours session after O2 ships.

---

## O4: Remote Environment & DR (Future)

`cp9 up --remote` provisions a real EKS cluster instead of local k3d. Enables disaster recovery, CI environments, and developers on low-spec machines.

**Not detailed here.** O4 design will be produced after O2 ships.

---

## Implementation Scope

Developers are empowered by coding agents (Claude Code, Copilot). Agent-assisted development compresses traditional timelines.

```
Phase 1: Foundation + Classification + Topology
  [ ] cp9 CLI skeleton (ticat-based, cobra fallback)
  [ ] cp9.yaml + cp9.local.yaml schema + parser
  [x] cp9 topology: static analysis of 4 repos → topology.json
      [x] Proto scanner (extract gRPC service/method definitions)
      [x] Route scanner (extract HTTP API surface from Gin routers)
      [ ] Client call scanner (AST analysis of infra/*cli, thirdparty/, handler/, pkg/grpc/grpcli/)
      [ ] Handler → client call chain linker
      [ ] Graph assembly + topology.json output
      [ ] Query CLI (cp9 topology query --service=X --api=Y)
  [ ] Smart service classifier (app-delivery + topology.json → real/stub/skip)
  [ ] k3d cluster lifecycle (sequential steps, retry-from-scratch on failure)
  [ ] Proto stub generator (gRPC, basic zero-value responses)

  ──── CRITICAL PATH DEMO: ────
  ──── 1 real service + 2 stubs running in k3d, ────
  ──── API request through real → stub chain    ────

Phase 2: Stubs + Deploy
  [ ] Proto stub generator (gRPC server from .proto files)
  [ ] Swagger stub generator (HTTP server from swagger JSON)
  [ ] Stub accuracy: staging log integration for realistic responses
  [ ] Real service deployment: build from source at specified ref
  [x] CRD install from staging + secrets injection (internal/crd/, PR #TBD)
  [ ] cp9 migrate (staging DB schema dump + seed for REAL services)
  [ ] TiDB Cloud Zero integration + fallback chain

Phase 3: AWS + Config + Integration
  [ ] cp9 setup (validate assume-role on shared cp9-developer role)
  [ ] AWS config injection + resource tracking + tagged cleanup
  [ ] internal destroy reaper (TTL/session cleanup) + cp9 status reporting
  [ ] Kustomize overlay generation (real staging configs, stub DNS names)
  [ ] Readiness checks (business API for real, healthz for stubs)
  [ ] cp9 unit + integration tests (3 real + 5 stubs on k3d)
           ──── O1 COMPLETE → dev-base shutdown gate ────

Phase 4: Agent Tooling + Plugin
  [ ] cp9 onboard <repo> (AGENTS.md + TOPOLOGY.md + cp9.yaml)
  [ ] TOPOLOGY.md generator (all four inputs: app-delivery, proto/swagger, source, topology)
  [ ] repo-owned compatibility checks (cross-repo API breaking change detection via service topology)
  [ ] feature-delivery skills/workflow (with retry budgets + human review gates)
  [ ] cp9-plugin package (skills, agents, hooks — Claude Code + VS Code)
  [ ] feature-dev agent (single-feature delivery with gates)

Phase 5: E2E + Validation
  [ ] cp9 logs --trace-id (grep-based, cross-service timeline)
  [ ] E2E test harness (Go test code in proto/swagger repo, trace ID capture, failure diagnostics)
  [ ] CMEK M2 redo (known answer test, agent-driven with gates)
  [ ] Real backlog feature (unknown answer test)
  [ ] O2 demo to leadership
           ──── O2 COMPLETE → agent delivery demo ────
           ──── MILESTONE: SHUT DOWN dev-base EKS ────
```

---

## Maintenance & Ownership

**Owner**: controlplane9 team (initially 1 person, grows to 2-3 as adoption spreads).

| Task | Frequency | Effort |
|---|---|---|
| Service topology re-classification | On every `cp9 up` (automated) | Automated |
| Stub regeneration on API changes | On `cp9 sync` when proto/swagger changes | Automated |
| New service onboarding | As teams add services | Minimal (`cp9 onboard`, review) |
| TTL reaping | Periodic operator cleanup through cp9 tag/session contracts | Automated |
| cp9 CLI bug fixes | As reported | Variable |
| CLI framework maintenance | As needed | Variable (cobra fallback if ticat stalls) |

**Key principle**: When a team adds a new service to staging, the topology algorithm auto-classifies it. When a proto/swagger changes, stubs auto-regenerate. No manual manifest updates needed.

---

## Risks

| Risk | Impact | Mitigation |
|---|---|---|
| k3d too resource-heavy on Mac (16GB RAM) | Blocks adoption | Smart classification means fewer services run (typically 3-5 real + 5-10 stubs vs 42 real in v1). Estimated memory: ~300-500MB per real Go service, ~50MB per stub. 5 real + 8 stubs ≈ 2.5GB service memory + 1.5GB k3s overhead = ~4GB total. 16GB Mac is viable. `cp9 doctor` warns on low RAM. Recommend 32GB for 8+ real services. BestEffort QoS. |
| Topology misses a dependency | Runtime errors that look like bugs | `cp9 doctor` validates graph. Stub services return valid (empty) responses — callers degrade gracefully. |
| Proto/swagger stubs diverge from real behavior | False-positive local E2E | Staging log integration for realistic responses. `cp9 sync` regenerates stubs on API changes. `cp9 doctor` flags schema drift. |
| ticat maturity (single maintainer, 26 stars) | CLI framework abandoned | Cobra is the documented fallback. Command surface area is framework-independent. |
| Cross-team resistance | Can't shut down dev-base | Start with tidb-management as proof. Show cost savings. Get leadership mandate. |
| AWS resource leaks | Cost overrun | Triple safety: Tagging API + destroy reaper TTL/session cleanup + AWS budget alarm. `cp9 status` provides visibility. |
| Human review gates slow agent delivery | Agent blocked waiting for human | Gates are at design doc and test plan only — not at every code change. Async review via PR-style workflow. |

## Dependencies

- **api-level-service-topology**: `cp9 topology` generates this via static analysis of 4 repos (tidb-management-service, account, billing, dataflow-service). See [API-Level Service Topology Generation](#api-level-service-topology-generation). Must produce accurate call graphs — this is the foundation for smart classification.
- **Infra team**: Pre-provision shared `cp9-developer` IAM role.
- **Platform team**: ECR read access for source checkout references.
- **All service teams**: Proto/swagger definitions must be up-to-date (for stub generation).
- **R&D leadership**: Org-wide mandate to adopt cp9 and decommission dev-base.
- **TiDB Cloud Zero team**: Confirm API stability and rate limits.

## Open Questions

1. **Topology accuracy.** `cp9 topology` regenerates `topology.json` from source code via static analysis. Recommended: run on every `cp9 up` (adds ~30s) or on every PR merge via CI. If a new RPC call is added but topology hasn't refreshed, a REAL service might call a SKIP service (no pod running). `cp9 doctor` catches this at startup and warns.

2. **Stub fidelity vs. effort.** For stubs generated from proto/swagger alone, responses are structurally correct but semantically empty (zero-value fields). Staging log integration improves this but adds complexity. Where's the right default? Start with zero-value stubs and add staging log accuracy only where tests fail?

3. **Cross-team adoption.** Shutting down dev-base requires ALL teams to contribute proto/swagger definitions and validate their stubs. Need a champion per team.