# CP9 API-Level Service Topology Implementation Design

Status: draft
Date: 2026-04-16
Related PRD: `prd-api-level-service-topology.md`

## 1. Objective

Build a reproducible topology pipeline that turns existing KB scan artifacts into an API-level service dependency graph for cp9.

The primary consumer is `cp9 up`, which needs to classify deployed services as:

- `REAL`: services on the active feature path and therefore built/run locally.
- `STUB`: services referenced by real services but not on the active API path.
- `SKIP`: deployed services outside the current local development graph.

The first implementation should optimize for correctness, debuggability, and explicit coverage gaps. It should not pretend the KB can answer paths that have not been deep-scanned.

## 2. Scope

### In Scope

- Build deployed-service coverage from `app-delivery` registry data and KB repo metadata.
- Build a name map from scan names and dependency names to canonical Kubernetes service names.
- Assemble method-level dependency edges from `knowledge/repos/*/interfaces/*.json` and `knowledge/repos/*/service-mapping/*.json`.
- Preserve API-level edge metadata: source method, target interface, operation type, conditions, source evidence where available.
- Apply tiinf shared-library implied edges through explicit templates.
- Provide a query CLI that returns REAL/STUB/SKIP for an entry service and optional API list.
- Emit machine-readable validation reports and refuse to silently drop unresolved names.

### Out Of Scope For V0

- Full in-process call graph reconstruction inside a method.
- Accurate topology for binaries that have not been deep-scanned to method level.
- Automatic LLM rescanning of stale repos.
- cp9 stub generation itself.
- Environment-specific runtime routing beyond canonical deployed service identity.

## 3. Directory Layout

Implementation files should live under `construction/cp9/`:

```text
construction/cp9/
├── prd-api-level-service-topology.md
├── root-call-graph-task.md
├── implementation-design.md
├── build-coverage-matrix.py
├── build-name-map.py
├── assemble-topology.py
├── query-topology.py
├── detect-stale.sh
├── rescan-stale.py
└── fixtures/
```

Generated V0 artifacts should live under `construction/cp9/artifacts/`:

```text
construction/cp9/artifacts/
├── topology.json
├── coverage-matrix.json
├── name-map.json
└── tiinf-edge-templates.json
```

These files are construction outputs, not committed KB facts yet. They can be promoted into `knowledge/` later only after the format and validation rules are stable.

## 4. Source Inputs

### Deployment Source

Use local `app-delivery` data if present. The intended source roots are:

- `{app-delivery}/app-registry/*.yaml`
- `{app-delivery}/staging/app-registry/*.yaml`

Resolve `{app-delivery}` with this order:

1. `.local/repos/app-delivery/path.json` field `local_path`
2. future submodule path `repos/app-delivery`
3. explicit `--app-delivery-path` CLI argument

The deployment source provides the canonical service universe. Services absent from the app registry are not topology nodes unless explicitly modeled as infrastructure.

### KB Source

Use only canonical KB artifacts:

- `knowledge/repos/{repo}/meta.json`
- `knowledge/repos/{repo}/interfaces/*.json`
- `knowledge/repos/{repo}/service-mapping/*.json`
- `knowledge/repos/{repo}/deep-scan-state.json`

For repo list operations, apply `.local/review-history/repos-filtered.txt` first.

### tiinf Source

Use explicit template data, not ad hoc inference in the assembler:

- `construction/cp9/artifacts/tiinf-edge-templates.json`

If this file does not exist yet, ship an empty template list and mark `tiinf.status = "not_configured"` in `topology.json`.

## 5. Data Model

### `artifacts/coverage-matrix.json`

Purpose: show every deployed service and whether the KB can contribute topology data.

```json
{
  "version": "1.0",
  "generated_at": "2026-04-16T00:00:00Z",
  "summary": {
    "total_deployed": 0,
    "scanned": 0,
    "missing_kb": 0,
    "missing_method_mapping": 0
  },
  "services": [
    {
      "service": "account-svc",
      "repo": "tidbcloud/account",
      "registry_file": "repos/app-delivery/app-registry/account.yaml",
      "kb_repo": "account",
      "scan_status": "approved",
      "method_level": true,
      "gap": null
    }
  ]
}
```

`gap` values:

- `missing_registry_source`
- `missing_repo_mapping`
- `missing_kb_entry`
- `missing_interfaces`
- `missing_service_mapping`
- `legacy_mapping_format`
- `not_approved`

### `artifacts/name-map.json`

Purpose: canonicalize source scan names and dependency target names into Kubernetes service names.

```json
{
  "version": "1.0",
  "generated_at": "2026-04-16T00:00:00Z",
  "canonical_services": ["account-svc"],
  "aliases": {
    "account-service": "account-svc",
    "account-provider": "account-svc"
  },
  "unresolved": [
    {
      "name": "billing2",
      "source": "knowledge/repos/foo/service-mapping/bar.json",
      "reason": "no registry match"
    }
  ]
}
```

The assembler may use aliases only when they map to a canonical service. Unresolved names must be retained in validation output.

### `artifacts/topology.json`

Use the PRD schema with a few implementation-oriented additions:

```json
{
  "version": "1.0",
  "generated_at": "2026-04-16T00:00:00Z",
  "coverage": {
    "total_deployed": 85,
    "scanned": 70,
    "missing": []
  },
  "nodes": [],
  "edges": [],
  "tiinf_templates": [],
  "tiinf": {
    "status": "not_configured"
  },
  "validation": {
    "errors": [],
    "warnings": []
  }
}
```

Edge shape:

```json
{
  "from": "tidb-mgmt-service",
  "to": "account-svc",
  "apis": [
    {
      "from_api": "ClusterService.CreateCluster",
      "to_api": "AccountAPIService.GetProjectByID",
      "type": "grpc_unary",
      "source": "direct",
      "conditions": ["Always"],
      "evidence": {
        "repo": "tidbcloud/tidb-management-service",
        "binary": "tidb-mgmt-service",
        "mapping_file": "knowledge/repos/tidb-management-service/service-mapping/tidb-mgmt-service-cmd-tidb-mgmt-service.json"
      }
    }
  ]
}
```

## 6. Script Responsibilities

### `build-coverage-matrix.py`

Inputs:

- app-delivery registry YAML files
- `knowledge/repos/*/meta.json`
- `knowledge/repos/*/deep-scan-state.json`
- `knowledge/repos/*/interfaces/*.json`
- `knowledge/repos/*/service-mapping/*.json`

Responsibilities:

1. Load canonical deployed service list.
2. Match deployed service to source repo using registry data first, then KB metadata.
3. Determine whether each service has approved method-level artifacts.
4. Emit `construction/cp9/artifacts/coverage-matrix.json`.
5. Exit non-zero only on malformed input, not on coverage gaps.

### `build-name-map.py`

Inputs:

- `construction/cp9/artifacts/coverage-matrix.json`
- deployment registry names
- dependency names found in `service-mapping/*.json`
- `target_repo` and `target_interface` fields when present

Responsibilities:

1. Build canonical service list from deployment registry.
2. Collect aliases from:
   - service dependency `name`
   - `target_repo`
   - binary names
   - repo names
   - known app registry names
3. Resolve aliases to canonical Kubernetes service names.
4. Emit `construction/cp9/artifacts/name-map.json`.
5. Keep unresolved names explicit for human review.

Resolution policy:

1. Exact canonical service name match.
2. Exact known alias from registry metadata.
3. Repo-to-service mapping from coverage matrix.
4. Conservative suffix normalization only when it is unambiguous.
5. Otherwise unresolved.

### `assemble-topology.py`

Inputs:

- `construction/cp9/artifacts/coverage-matrix.json`
- `construction/cp9/artifacts/name-map.json`
- `construction/cp9/artifacts/tiinf-edge-templates.json`
- `knowledge/repos/*/interfaces/*.json`
- `knowledge/repos/*/service-mapping/*.json`

Responsibilities:

1. Load all approved method-level KB artifacts.
2. Create one node per deployed service/binary represented in coverage.
3. For each service-mapping dependency:
   - `from` = canonical service for the mapping owner.
   - `from_api` = dependency object key.
   - `to` = canonical target service from `services[].name`, `target_repo`, or template.
   - `target_kind` = `deployed_service`, `infrastructure`, `external_endpoint`, or `unresolved`.
   - `to_api` = `target_interface` if present.
   - `type` = normalized `operation_type`.
4. Apply tiinf templates to repos importing matching packages.
5. Deduplicate edges by `(from, to)` and merge API entries.
6. Validate:
   - no edge target points to an unknown service unless marked `infrastructure`, `external_endpoint`, or `unresolved`.
   - every source mapping key exists in paired interface `name_key`.
   - every node has deployment coverage.
7. Write `construction/cp9/artifacts/topology.json`.

### `query-topology.py`

Inputs:

- `construction/cp9/artifacts/topology.json`
- `--entry <service>`
- optional `--apis <comma-separated-api-list>`
- optional `--services-only`
- optional `--dot`

Responsibilities:

1. Resolve entry service via `name_map`.
2. Traverse outgoing edges using BFS.
3. If `--apis` is provided:
   - include edges whose `from_api` matches the active API set.
   - once a downstream service is reached through a matched edge, continue traversal from the downstream `to_api` when available.
   - if no `to_api` is known, continue at service level and mark precision as `service_level_fallback`.
4. Compute:
   - `real`: reached services.
   - `stub`: services directly called by real services but not reached by the active path.
   - `skip`: deployed services outside real/stub.
5. Emit JSON, service-only text, or DOT.

Precision markers:

- `method_exact`
- `service_level_fallback`
- `missing_mapping`
- `unresolved_target`

### `detect-stale.sh`

Responsibilities:

1. Resolve local repo paths through `.local/repos/{repo}/path.json`, future `repos/{repo}` submodules, or explicit CLI arguments.
2. Compare local repo HEAD with `knowledge/repos/{repo}/meta.json.scan_commit` when `scan_commit` exists.
3. If `scan_commit` is missing, mark the repo as `stale_unknown` instead of failing. This is the current expected state for most KB entries.
4. Print stale, stale_unknown, and clean repos.
5. Exit `0` when no stale repos exist, `2` when stale or stale_unknown repos exist, and `1` for script/input errors.

### `rescan-stale.py`

V0 should be a coordinator stub rather than an LLM runner.

Responsibilities:

1. Read stale repo list from `detect-stale.sh` logic.
2. Support `--dry-run`, `--repo`, and `--all`.
3. For V0, print the exact scan task that must be performed and exit with a clear `not_implemented` status for automatic LLM rescans.
4. Later versions can call the scan workflow and then re-run topology assembly.

## 7. Query Algorithm

```text
resolve entry service
seed active queue with (entry, requested APIs or wildcard)

while queue not empty:
  pop (service, api_filter)
  if seen exact state: continue
  mark service real

  for each outgoing edge from service:
    matching_apis = edge.apis filtered by api_filter
    if matching_apis is empty: continue

    mark edge.to real
    enqueue (edge.to, to_api values from matching_apis)

stub = services directly referenced by real services but not in real
skip = deployed services - real - stub
```

When no API list is provided, traversal is service-level and all outgoing edges match.

When the graph lacks `to_api`, the traversal must continue at service level and include a warning in output.

## 8. Validation Strategy

### Structural Validation

Run on every build:

- `jq empty` for generated JSON.
- `construction/run-format-lint.py --paths knowledge/repos/<repo>` for participating repos.
- Topology validation inside `assemble-topology.py`.

### Semantic Validation

Minimum PRD acceptance validation:

1. 30 random topology edges manually checked against source scan evidence.
2. 3 cp9 feature scenarios compared against manual tracing.
3. 0 unresolved aliases before declaring v1.0 complete.
4. 0 legacy string-format service-mapping files included in topology.

### V0 Acceptance

V0 is acceptable when:

- It builds a topology from currently approved canonical artifacts.
- It can answer at least one root from `tidb-management-service`.
- It reports coverage gaps honestly.
- It has deterministic generated output.
- It does not silently discard unresolved targets.

## 9. Phased Delivery Plan

### Phase 0: Reorganize Requirements

Deliverables:

- `construction/cp9/prd-api-level-service-topology.md`
- `construction/cp9/root-call-graph-task.md`
- `construction/cp9/implementation-design.md`

### Phase 1: Deterministic Inventory

Deliverables:

- `build-coverage-matrix.py`
- `build-name-map.py`
- `construction/cp9/artifacts/coverage-matrix.json`
- `construction/cp9/artifacts/name-map.json`

Goal:

- Know exactly which deployed services are covered, missing, or unresolved.

### Phase 2: Topology Assembly V0

Deliverables:

- `assemble-topology.py`
- `construction/cp9/artifacts/topology.json`
- validation output embedded in `topology.json`

Goal:

- Assemble method-level edges from current KB without LLM rescanning.

### Phase 3: Query CLI V0

Deliverables:

- `query-topology.py`
- JSON output
- `--services-only`
- `--dot`

Goal:

- Support cp9 experiments using known deep-scanned roots.

### Phase 4: Gap Closure

Deliverables:

- legacy mapping migration queue
- tiinf templates
- stale detection
- manual validation report

Goal:

- Move from best-effort V0 to PRD-level completeness.

## 10. Risks And Decisions

### Risk: Name Mapping Can Produce False Edges

Mitigation:

- Prefer unresolved over guessed.
- Require explicit aliases for ambiguous names.
- Emit unresolved names into `name-map.json`.

### Risk: Empty `services: []` Does Not Mean Terminal

Mitigation:

- Add precision markers.
- Include coverage and scan status in query output.
- Do not claim full completeness for roots with incomplete downstream scans.

### Risk: tiinf Calls Are Hidden From Direct Scan

Mitigation:

- Keep tiinf templates as explicit data.
- Mark template-generated edges with `source = "tiinf_template"`.

### Decision: V0 Should Not Auto-Rescan

Rationale:

- Automatic LLM rescans are harder to make reproducible.
- The first useful milestone is deterministic topology assembly from existing KB.

## 11. Immediate Next Step

Implement `build-coverage-matrix.py` first. It defines the deployed service universe and exposes whether the current KB can support the PRD before topology assembly begins.