# Scan All Repos Task

## Overview

Scan tracks for each binary:
1. All API/gRPC endpoints
2. Per-method service dependencies
3. Conditions for external service calls
4. Implementation details

### Goal

Build a **service-level dependency graph** for each binary. Combined with deployment configuration, this enables:
- **Service-to-service relationships** in production
- **High-level data flow** between deployed components
- **Deployment ordering** based on dependency chains

Example insight:
```
dm-master → depends on etcd, mysql, dm-worker
dm-worker → depends on mysql, kafka

=> In production: dm-master must start after etcd + mysql, dm-worker after mysql + kafka
```

**Scope:**
- ✅ Service/binary level dependencies
- ✅ Interface/method level dependencies (with `target_interface`)

## Objective

Scan all private repositories, generate complete interface and dependency mappings for each repo, and use multi-party review to ensure quality.

## Test Repository

**First repo to scan and validate manually before processing others:**

| Property | Value |
|----------|-------|
| Repo | `tidbcloud/tidb-management-service` |
| Local Path | `/Users/lc/coding/tidb-management-service` |
| Files | 3672 |
| Binaries | 12 (6 primary_service + 6 CLI tools) |
| Interfaces | gRPC + CLI |

**Primary services to scan:**
| Binary | Path | gRPC Services |
|--------|------|---------------|
| tidb-mgmt-service | tidb-mgmt-service/cmd/... | Grpc |
| serverless-global-service | serverless-global-service/cmd/... | 7 services |
| dedicated-global-service | dedicated-global-service/cmd/... | 11 services |
| nextgen-global-service | nextgen-global-service/cmd/... | 11 services |
| regional_proxy | serverless-global-service/cmd/... | RegionalServerProxyService |
| tempo-proxy | tempo-proxy/cmd/... | CoreService |

**CLI tools to skip:** mgmt-cli, cli, rewrite_enum_stringer, ui_swagger_generate, tempo-wrapper, tidb-mgmt-portal

## Prerequisites

**Load skills:**
- `rule-agents-group-review` - Group review protocol
- `rule-review-address` - Review interaction protocol

**Required files:**
- `knowledge/meta/repos-private.json` - Repository list
- `.local/review-history/repos-filtered.txt` - Filtered repos
- `.local/repos/{repo-name}/path.json` - Local repo paths

## Directory Structure

```
knowledge/repos/{repo-name}/
├── meta.json                        # Basic metadata
├── deep-scan-state.json             # Completion marker (tracked in git)
├── interfaces/
│   └── {path-based-name}.json       # e.g., cmd-foo-server.json
└── service-mapping/
    └── {path-based-name}.json       # e.g., cmd-foo-server.json

.local/
├── deep-scan-all-repos.md           # This task file
├── deep-scan-reviews/               # Review records (not tracked)
│   └── {repo-name}.md
└── deep-scan-failed.json            # Failed repos list (not tracked)
```

**Collision-safe Naming:** Binary path `cmd/foo-server` → Output file `cmd-foo-server.json`. This handles duplicate binary names within a repo.

## File Format Definitions

### meta.json

Records basic repository information as part of the scan outputs.

**Full Example (multi_service repo):**
```json
{
  "repo": "pingcap/tiflow",
  "scan_version": "4.4",
  "scan_status": "completed",
  "scan_at": "2026-03-25T05:20:32.928988",
  "file_count": 4488,
  "language": "Go",
  "repo_kind": "multi_service",
  "type": "multi_service",
  "binaries": [
    {
      "name": "oauth2-server",
      "path": "cmd/oauth2-server",
      "language": "Go",
      "interfaces": {
        "cli": {"framework": "cobra"},
        "http_api": {"source": "cmd"}
      },
      "role": "primary_service"
    }
  ],
  "summary": {
    "total_binaries": 12,
    "has_cli": true,
    "has_http": true,
    "has_grpc": false,
    "binary_names": ["oauth2-server", "dm-master", "dm-worker"],
    "interfaces": ["cli", "http_api"],
    "build_manifests": ["dockerfile", "makefile"]
  }
}
```

**Field Definitions:**

| Field | Description |
|-------|-------------|
| `repo` | Full repo ID (e.g., `pingcap/tiflow`) |
| `scan_version` | Scanner version (e.g., `4.4`) |
| `scan_status` | `completed` / `nearly_empty` |
| `scan_at` | ISO timestamp of scan |
| `file_count` | Number of non-git files |
| `language` | Primary language (`Go`, `Rust`, `TypeScript`, etc.) |
| `repo_kind` | Repository classification (see table below) |
| `type` | Usually same as `repo_kind` |
| `binaries[]` | List of build targets |
| `summary` | Aggregate info about the repo |
| `scan_note` | (Optional) Notes like "Local clone has only 5 files" |

**binary[] element:**

| Field | Description |
|-------|-------------|
| `name` | Binary name (e.g., `dm-master`) |
| `path` | Path to binary (e.g., `cmd/dm-master`) |
| `language` | Language of this binary |
| `interfaces` | Detected interfaces (`cli`, `http_api`, `grpc`) |
| `role` | `primary_service` / `admin_cli` / `tool` / `example` / `unknown` |

**repo_kind / type values:**

| Value | Description | Note |
|-------|-------------|------|
| `service` | Single service binary | Route by binary.interfaces |
| `multi_service` | Multiple service binaries | Route by binary.interfaces |
| `multi_binary` | Multiple binaries (mixed types) | Route by binary.interfaces |
| `multi_tool` | Multiple CLI tools | Usually skip |
| `cli` | Single CLI tool | Usually skip |
| `library` | No binaries, library code | Skip |
| `deployment_config` | Config-only repo (Pulumi, Helm) | Skip |
| `docs` | Documentation only | Skip |
| `nearly_empty` | Very few files | Skip |
| `unknown` | Unable to classify | Route by binary.interfaces |

**Note:** `repo_kind` is informational only. Actual scan decision is based on each binary's `interfaces` field.

### .local/repos/{repo-name}/path.json

Local-only file (not tracked in git) containing local path information.

```json
{
  "repo": "tidbcloud/jiralert",
  "local_path": "/Users/lc/coding/jiralert",
  "scan_at": "2026-03-24T18:00:00+08:00"
}
```

### deep-scan-state.json

Tracks scan progress and status (tracked in git).

```json
{
  "repo": "{repo-name}",
  "full_name": "{owner}/{repo-name}",
  "status": "approved",
  "scan_started_at": "<iso-timestamp>",
  "scan_completed_at": "<iso-timestamp>",
  "review_completed_at": "<iso-timestamp>",
  "expected_binaries": [
    {"name": "server", "path": "cmd/server", "status": "completed", "role": "primary_service"},
    {"name": "cli", "path": "cmd/cli", "status": "skipped", "role": "admin_cli", "skip_reason": "cli_tool"}
  ],
  "scanned_binaries": [
    {"name": "server", "output_file": "cmd-server"}
  ],
  "skipped_binaries": [
    {"name": "cli", "reason": "cli_tool"}
  ],
  "review_status": "approved",
  "review_participants": ["opencode", "kimi", "codex"],
  "review_cycles": 1,
  "files_generated": [
    "interfaces/cmd-server.json",
    "service-mapping/cmd-server.json"
  ]
}
```

**Status values:**
| Status | Description |
|--------|-------------|
| `scan_in_progress` | Currently scanning |
| `scan_complete` | Scan done, review pending |
| `review_in_progress` | Active group review |
| `approved` | Triple consensus reached ✅ |
| `blocked` | Review blocked, needs user decision |
| `skipped` | Repo not applicable for scan |

**Binary status values:**
| Status | Description |
|--------|-------------|
| `pending` | Not yet scanned |
| `in_progress` | Currently scanning |
| `completed` | Scan completed |
| `skipped` | CLI tool, not scanned |

## Key Concepts

### Dependency Categories

Dependencies are split into two categories:

| Category | Description | Examples | Has target_interface |
|----------|-------------|----------|---------------------|
| **infrastructure** | Non-service infrastructure | mysql, redis, kafka, s3 | No (operation-based) |
| **services** | Service-to-service calls | auth-svc, worker-svc | Yes (method-based) |

**infrastructure examples:**
| Type | operations | client |
|------|------------|--------|
| mysql | SELECT, INSERT, UPDATE, DELETE | gorm.io/gorm |
| redis | GET, SET, DEL | go-redis/redis |
| kafka | PUBLISH, SUBSCRIBE | confluent-kafka-go |
| s3 | GET, PUT, DELETE | aws-sdk-go-v2 |

### Condition Types

| Type | Description |
|------|-------------|
| `Always` | Always executes |
| Conditional | Based on configuration or data |
| `error_path` | Only executes on error |
| `fallback` | Backup if primary fails |

### Implementation Field

Points to the actual code:
```json
"implementation": "github.com/tidbcloud/account/pkg/service/user.UserService.GetByEmail()"
```

## Cross-Binary Dependency Tracking

When a binary depends on another binary within the same repository, it is tracked as a service dependency with `target_interface`:

### Same-Repo Binary Dependencies

Dependencies between binaries in the same repo (e.g., `dm-master` → `dm-worker`) are recorded as:

```json
// service-mapping/cmd-dm-master.json
{
  "dependencies": {
    "MasterService.CreateTask": {
      "infrastructure": {},
      "services": [
        {
          "name": "dm-worker",
          "target_interface": "WorkerService.DispatchTask",
          "operation_type": "grpc_unary",
          "conditions": ["Always"],
          "operations": [{
            "operation": "gRPC DispatchTask",
            "trigger": "Dispatch task to dm-worker",
            "implementation": "s.workerClient.DispatchTask(ctx, req)"
          }]
        }
      ]
    }
  }
}
```

**Key:** Cross-binary calls use the same array format with `target_interface` as external service calls.

### Cross-Binary Reference Convention

| Aspect | Convention |
|--------|------------|
| Service name | Use the binary name (e.g., `dm-worker`, not `WorkerService`) |
| Cross-repo refs | Use `{repo}/{binary}` format (e.g., `pingcap/tiflow/dm-worker`) |
| Implementation | Document the client/connection initialization code path |

### Building the Complete Dependency Graph

To resolve cross-binary dependencies within a repo:

1. **Load all binary mappings** from `service-mapping/*.json`
2. **Identify internal refs** - Look for dependency keys matching binary names in `meta.json`
3. **Merge at service level** - Combine interface dependencies across binaries

Example merged view:
```json
{
  "repo": "pingcap/tiflow",
  "deployment_dependencies": {
    "dm-master": ["etcd", "dm-worker"],
    "dm-worker": ["mysql", "tidb"]
  }
}
```

## Step 0: Bootstrap (Required on First Run)

### 0.1 Generate Local Path Inventory

**Problem:** `.local/repos/{repo}/path.json` files don't exist yet.

**Solution:** Generate them based on standard convention:

```bash
# For each repo in knowledge/repos/
for repo in $(ls knowledge/repos/); do
  mkdir -p ".local/repos/$repo"
  cat > ".local/repos/$repo/path.json" << EOF
{
  "repo": "$repo",
  "local_path": "$HOME/coding/$repo",
  "scan_at": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
}
EOF
done
```

**Verification:**
```bash
# Check count matches
ls -1d knowledge/repos/*/ | wc -l
ls -1d .local/repos/*/ | wc -l
```

### 0.2 Initialize Directories

```bash
mkdir -p .local/deep-scan-reviews
[ ! -f .local/deep-scan-failed.json ] && echo '[]' > .local/deep-scan-failed.json
```

## Step 1: Prepare Repository List

### 1.1 Load and Filter Repos

```bash
# Read filtered repo names (first column)
FILTERED=$(cut -d' ' -f1 .local/review-history/repos-filtered.txt)

# Load private repos, filter, and sort by priority
jq -r --arg f "$FILTERED" '
  map(select(.archived == false)) |
  map(select(.name as $n | ($f | split("\n") | index($n) | not))) |
  sort_by(
    if .owner == "tidbcloud" then 0 else 1 end,
    .name
  ) |
  .[].name
' knowledge/meta/repos-private.json
```

**Priority:**
1. `tidbcloud` owner repos only
2. Filter by `.local/review-history/repos-filtered.txt`
3. Route by binary.interfaces (has http_api/grpc → SCAN)
4. **Total: ~89 repos eligible**

### 1.2 Binary-Based Routing

**Critical:** Scan decision is based on each binary's `interfaces` field, not `repo_kind`.

**Routing Logic (per binary):**

| interfaces | Action | Reason |
|------------|--------|--------|
| Has `http_api` or `grpc` | **SCAN** | Exposes network API |
| Only `cli` | **SKIP** | CLI tool, no network dependencies |
| Empty `{}` | **SKIP** | No detectable interfaces |

**Why binary-based routing:**
- `role` field is unreliable (e.g., `br` is `primary_service` but only has CLI)
- `repo_kind` doesn't reflect individual binary capabilities
- A repo can have mixed binaries (some to scan, some to skip)

**Examples:**
| Binary | role | interfaces | Action |
|--------|------|------------|--------|
| tidb-mgmt-service | primary_service | cli + grpc | SCAN |
| mgmt-cli | admin_cli | cli only | SKIP |
| br | primary_service | cli only | SKIP |
| vmctl | admin_cli | cli + http_api | SCAN |

**Implementation:**
```bash
# For each binary in repo
for binary in $(jq -c '.binaries[]' knowledge/repos/{repo}/meta.json); do
  binary_name=$(echo "$binary" | jq -r '.name')
  has_http=$(echo "$binary" | jq 'has("http_api")')
  has_grpc=$(echo "$binary" | jq 'has("grpc")')
  has_cli=$(echo "$binary" | jq 'has("cli")')
  is_empty=$(echo "$binary" | jq '(.interfaces | keys | length) == 0')
  
  if [ "$has_http" = "true" ] || [ "$has_grpc" = "true" ]; then
    echo "SCAN: $binary_name (has network API)"
  elif [ "$is_empty" = "true" ]; then
    echo "SKIP: $binary_name (empty interfaces)"
  else
    echo "SKIP: $binary_name (CLI only)"
  fi
done
```

**Record skip reason in `deep-scan-state.json`:**
```json
{
  "repo": "{repo-name}",
  "status": "skipped",
  "skip_reason": "library",
  "skip_at": "<iso-timestamp>"
}
```

### 1.3 Check Completion Status

Before processing each repo, check if already completed **by status**:

```bash
STATE_FILE="knowledge/repos/{repo-name}/deep-scan-state.json"

if [ -f "$STATE_FILE" ]; then
  STATUS=$(jq -r '.status // "unknown"' "$STATE_FILE")
  case "$STATUS" in
    approved|skipped)
      echo "SKIP"  # Completed
      ;;
    scan_complete|review_in_progress|blocked)
      echo "RESUME"  # Needs continuation
      ;;
    *)
      echo "REPROCESS"  # Unknown state
      ;;
  esac
else
  echo "PROCESS"  # New repo
fi
```

**If SKIP:** Move to next repo

**If RESUME or REPROCESS:** Continue from appropriate step

**If PROCESS:** Continue to Step 2

## Step 2: Scan Single Repo

### 2.1 Load Repo Metadata

```bash
# Get local path
LOCAL_PATH=$(jq -r '.local_path' .local/repos/{repo-name}/path.json)

# Get binary list
BINARIES=$(jq -c '.binaries[]' knowledge/repos/{repo-name}/meta.json)
```

### 2.2 Scan Process (Per Binary)

**For each binary:**

 1. **Identify Inbound Interfaces** (HTTP/gRPC - what the service exposes)
    ```bash
    cd $LOCAL_PATH/{binary-path}
    # Find HTTP handlers
    grep -r "HandleFunc\|Handle\|router\." . --include="*.go"
    # Find gRPC services
    find . -name "*.proto" -type f
    ```
    **Note:** Dependencies on external services are tracked in `service_dependencies` with `operation_type` field covering both outbound calls (gRPC, HTTP, MQ publish) and inbound subscriptions (MQ subscribe).

2. **Find Implementations**
   - Trace route → handler function
   - Trace gRPC method → service implementation

 3. **Identify Dependencies (with target_interface)**
     - Database calls (MySQL, Redis, etc.)
     - External API calls (AWS, GCP, etc.)
     - Internal service calls - **record target_interface** for ALL transports:
       - gRPC calls → `ServiceName.MethodName`
       - HTTP calls → `METHOD /path`
       - MQ calls → `TopicName`
       - WebSocket → `/ws/path`
       - Webhook → `EventType`
     - Message queue operations
     
     **For all service-to-service calls, must record:**
     - Service name (e.g., `dm-worker`)
     - **`target_interface`**: Varies by transport (see table in Section 2.2.2)
     - **`operation_type`**: grpc_unary, http_get, mq_publish, etc.
     - Operation details (call type, trigger, implementation)

 4. **Determine Conditions**
    - Configuration flags
    - Environment variables
    - Input parameters
    - Data conditions

### 2.2.1 Per-Interface Call Chain Tracing (CRITICAL)

**⚠️ MUST trace each interface's complete call chain to ensure no dependencies are missed.**

**For each interface method, follow this process:**

```
Interface Method (e.g., MasterService.GetTask)
     ↓
Handler/Implementation Function
     ↓
Trace ALL function calls within:
     ├─ Database calls (SQL operations)
     ├─ Service calls (ALL transports: gRPC/HTTP/MQ/WebSocket/Webhook)
     │    └─ Record target_interface for each!
     ├─ Cache operations (Redis)
     ├─ Message queue operations
     └─ Helper function calls
          └─ Recursively trace their dependencies
```

**Common Missing Dependencies (be careful):**
- Implicit dependencies via helper functions
- Conditional branches (if/else paths)
- Error handling paths
- Background goroutines/async operations
- Middleware/interceptor injected dependencies

**Verification Checklist (per interface):**
- [ ] Traced main execution path
- [ ] Traced all conditional branches
- [ ] Traced error handling paths
- [ ] Traced async operations
- [ ] Recorded all service calls with `target_interface`
- [ ] Recorded all database operations
- [ ] Recorded all external API calls

### 2.2.2 Scan Output Format Example

**Field Definitions for `services` array:**

| Field | Required | Description |
|-------|----------|-------------|
| `name` | Yes | Target service name (e.g., `worker-svc`, `dm-worker`) |
| `target_interface` | Yes | Target endpoint - format varies by `operation_type` (see below) |
| `operation_type` | Yes | Communication pattern (see `operation_type` values below) |
| `target_repo` | Optional | Full repo path for stable identity (e.g., `tidbcloud/worker-svc`) |
| `conditions` | Yes | Array of conditions (e.g., `["Always"]`, `["cache miss"]`) |
| `operations` | In service-mapping only | Detailed operation info |

**`target_interface` Format by `operation_type`:**

| operation_type | target_interface format | Example |
|----------------|------------------------|---------|
| `grpc_*` | `ServiceName.MethodName` | `WorkerService.DispatchTask` |
| `http_get/post/put/delete` | `METHOD /path` | `GET /api/v1/tasks` |
| `mq_publish` | `TopicName` | `task.events` |
| `mq_subscribe` | `TopicName` | `task.events` |
| `websocket` | `/ws/path` | `/ws/tasks` |
| `webhook` | `EventType` | `task.completed` |

**`operation_type` Values:**

| Value | Description |
|-------|-------------|
| `grpc_unary` | gRPC unary call |
| `grpc_server_stream` | gRPC server streaming |
| `grpc_client_stream` | gRPC client streaming |
| `grpc_bidi_stream` | gRPC bidirectional streaming |
| `http_get` | HTTP GET |
| `http_post` | HTTP POST |
| `http_put` | HTTP PUT |
| `http_delete` | HTTP DELETE |
| `mq_publish` | MQ publish (outbound) |
| `mq_subscribe` | MQ subscribe (inbound) |
| `websocket` | WebSocket |
| `webhook` | Webhook callback |

**File: `knowledge/repos/{repo}/interfaces/cmd-binary1.json`**

Interface definitions only - no dependencies (to avoid redundancy with service-mapping).

```json
{
  "repo": "tidbcloud/core-svc",
  "binary": "binary1",
  "binary_path": "cmd/binary1",
  "services": [
    {
      "name": "TaskService",
      "package": "task.pb",
      "methods": [
        {
          "name": "GetTask",
          "input": "GetTaskReq",
          "output": "GetTaskResp"
        },
        {
          "name": "CreateTask",
          "input": "CreateTaskReq",
          "output": "CreateTaskResp"
        }
      ]
    }
  ]
}
```

**File: `knowledge/repos/{repo}/service-mapping/cmd-binary1.json`**

Complete dependency mapping for each method.

```json
{
  "repo": "tidbcloud/core-svc",
  "binary": "binary1",
  "binary_path": "cmd/binary1",
  "dependencies": {
    "TaskService.GetTask": {
      "infrastructure": {
        "mysql": {
          "conditions": ["Always"],
          "operations": [
            {
              "operation": "SELECT",
              "trigger": "Fetch task from database",
              "conditions": ["Always"],
              "implementation": "s.db.Where(\"id = ?\", req.Id).First(&task)"
            }
          ]
        },
        "redis": {
          "conditions": ["s.cacheConfig.Enabled == true"],
          "operations": [
            {
              "operation": "GET",
              "trigger": "Check cache before DB",
              "conditions": ["cache enabled", "cache key: task:{id}"],
              "implementation": "s.redisClient.Get(ctx, cacheKey).Result()"
            }
          ]
        }
      },
      "services": [
        {
          "name": "auth-svc",
          "target_interface": "AuthService.VerifyToken",
          "operation_type": "grpc_unary",
          "conditions": ["Always"],
          "operations": [
            {
              "operation": "gRPC VerifyToken",
              "trigger": "Verify user authentication",
              "conditions": ["Always"],
              "implementation": "s.authClient.VerifyToken(ctx, &pb.VerifyTokenReq{Token: req.Token})"
            }
          ]
        }
      ]
    },
    "TaskService.CreateTask": {
      "infrastructure": {
        "mysql": {
          "conditions": ["Always"],
          "operations": [
            {
              "operation": "INSERT",
              "trigger": "Persist new task",
              "conditions": ["Always"],
              "implementation": "s.db.Create(&task)"
            }
          ]
        }
      },
      "services": [
        {
          "name": "auth-svc",
          "target_interface": "AuthService.VerifyToken",
          "operation_type": "grpc_unary",
          "conditions": ["Always"],
          "operations": [
            {
              "operation": "gRPC VerifyToken",
              "trigger": "Verify user authentication",
              "conditions": ["Always"],
              "implementation": "s.authClient.VerifyToken(ctx, &pb.VerifyTokenReq{Token: req.Token})"
            }
          ]
        },
        {
          "name": "worker-svc",
          "target_interface": "WorkerService.GetStatus",
          "operation_type": "grpc_unary",
          "conditions": ["Check worker health before dispatch"],
          "operations": [
            {
              "operation": "gRPC GetStatus",
              "trigger": "Check worker availability",
              "conditions": ["Always"],
              "implementation": "s.workerClient.GetStatus(ctx, &pb.GetStatusReq{})"
            }
          ]
        },
        {
          "name": "worker-svc",
          "target_interface": "WorkerService.DispatchTask",
          "operation_type": "grpc_unary",
          "conditions": ["Worker available"],
          "operations": [
            {
              "operation": "gRPC DispatchTask",
              "trigger": "Dispatch task to worker",
              "conditions": ["Worker status == healthy"],
              "implementation": "s.workerClient.DispatchTask(ctx, &pb.DispatchTaskReq{TaskId: task.Id})"
            }
          ]
        }
      ]
    }
  }
}
```

 5. **Generate Output Files**
    - **Collision-safe naming:** Use path-based filename to handle duplicate binary names
    - Convert binary path to safe filename: `cmd/foo-server` → `cmd-foo-server.json`
    - `knowledge/repos/{repo-name}/interfaces/{path-based-name}.json`
    - `knowledge/repos/{repo-name}/service-mapping/{path-based-name}.json`

### 2.3 Per-Binary Scan with Checkpoint

**For repos with many binaries, checkpoint after each binary to support intra-repo resume.**

```bash
# Step 2.3.1: Initialize state file before scanning
cat > knowledge/repos/{repo-name}/deep-scan-state.json << 'EOF'
{
  "repo": "{repo-name}",
  "full_name": "{owner}/{repo-name}",
  "status": "scan_in_progress",
  "scan_started_at": "<iso-timestamp>",
  "expected_binaries": [
    {"name": "server", "path": "cmd/server", "status": "pending", "role": "primary_service"},
    {"name": "cli", "path": "cmd/cli", "status": "pending", "role": "admin_cli"}
  ],
  "scanned_binaries": [],
  "skipped_binaries": [],
  "files_generated": [],
  "current_binary": null,
  "scan_progress": "0/2"
}
EOF

# Step 2.3.2: Scan each binary with checkpoint
for binary in "${BINARIES[@]}"; do
  binary_name=$(jq -r '.name' <<< "$binary")
  binary_path=$(jq -r '.path' <<< "$binary")
  binary_role=$(jq -r '.role // "unknown"' <<< "$binary")
  safe_name=$(echo "$binary_path" | sed 's|/|-|g')
  
  # Check if already completed (resume support)
  status=$(jq -r --arg bn "$binary_name" '.expected_binaries[] | select(.name == $bn) | .status' knowledge/repos/{repo-name}/deep-scan-state.json)
  if [ "$status" == "completed" ] || [ "$status" == "skipped" ]; then
    echo "Skipping $binary_name - already $status"
    continue
  fi
  
  # Skip CLI tools (admin_cli, tool, example)
  if [[ "$binary_role" =~ ^(admin_cli|tool|example)$ ]]; then
    jq --arg bn "$binary_name" --arg reason "cli_tool" '
      .expected_binaries = [.expected_binaries[] | if .name == $bn then .status = "skipped" else . end] |
      .skipped_binaries += [{"name": $bn, "reason": $reason}]
    ' knowledge/repos/{repo-name}/deep-scan-state.json > tmp.json && mv tmp.json knowledge/repos/{repo-name}/deep-scan-state.json
    echo "Skipped $binary_name (role: $binary_role)"
    continue
  fi
  
  # Update current binary being scanned
  jq --arg bn "$binary_name" --arg bp "$binary_path" '
    .current_binary = {"name": $bn, "path": $bp, "started_at": now | todate}
  ' knowledge/repos/{repo-name}/deep-scan-state.json > tmp.json && mv tmp.json knowledge/repos/{repo-name}/deep-scan-state.json
  
  # Scan the binary (follow Scan Process steps above)
  deep-scan-binary "$binary_path"
  
  # Checkpoint: mark binary as completed
  jq --arg bn "$binary_name" --arg safe "$safe_name" '
    .expected_binaries = [.expected_binaries[] | if .name == $bn then .status = "completed" else . end] |
    .scanned_binaries += [{"name": $bn, "output_file": $safe}] |
    .files_generated += ["interfaces/\($safe).json", "service-mapping/\($safe).json"] |
    .scan_progress = ((.scanned_binaries | length | tostring) + "/" + (.expected_binaries | length | tostring)) |
    .current_binary = null
  ' knowledge/repos/{repo-name}/deep-scan-state.json > tmp.json && mv tmp.json knowledge/repos/{repo-name}/deep-scan-state.json
done

# Step 2.3.3: Mark scan complete after all binaries
jq '.status = "scan_complete" | .scan_completed_at = (now | todate)' knowledge/repos/{repo-name}/deep-scan-state.json > tmp.json && mv tmp.json knowledge/repos/{repo-name}/deep-scan-state.json
```

**Per-Binary Checkpoint Benefits:**
- **Intra-repo resume:** If interrupted mid-repo, continue from last completed binary
- **Progress visibility:** `scan_progress: "3/10"` shows current status
- **Failure isolation:** One binary failure doesn't lose progress on others
- **Large repo friendly:** Repos with 10+ binaries can be safely interrupted

**State File Example (mid-scan):**
```json
{
  "repo": "core-svc",
  "status": "scan_in_progress",
  "scan_started_at": "2026-03-25T10:00:00Z",
  "expected_binaries": [
    {"name": "server", "path": "cmd/server", "status": "completed", "role": "primary_service"},
    {"name": "worker", "path": "cmd/worker", "status": "completed", "role": "primary_service"},
    {"name": "scheduler", "path": "cmd/scheduler", "status": "in_progress", "role": "primary_service"},
    {"name": "cli", "path": "cmd/cli", "status": "skipped", "role": "admin_cli"}
  ],
  "scanned_binaries": [
    {"name": "server", "output_file": "cmd-server"},
    {"name": "worker", "output_file": "cmd-worker"}
  ],
  "skipped_binaries": [
    {"name": "cli", "reason": "cli_tool"}
  ],
  "current_binary": {"name": "scheduler", "path": "cmd/scheduler", "started_at": "2026-03-25T10:15:00Z"},
  "scan_progress": "2/4"
}
```

**Resume Logic Enhancement (Step 1.3):**
```bash
# Check for in-progress scan (intra-repo resume)
if [ "$STATUS" == "scan_in_progress" ]; then
  current=$(jq -r '.current_binary.name // empty' "$STATE_FILE")
  if [ -n "$current" ]; then
    echo "RESUME_FROM_BINARY:$current"
  else
    # Find next pending binary
    next=$(jq -r '.expected_binaries[] | select(.status == "pending") | .name' "$STATE_FILE" | head -1)
    echo "RESUME_FROM_BINARY:$next"
  fi
fi
```

**Collision-safe Naming Convention:**
- Binary path: `cmd/foo-server` → Output file: `cmd-foo-server.json`
- Handles duplicate binary names (e.g., `core-svc` has two `server` binaries)
- Output files: `interfaces/{path-based-name}.json`

**Note:** Do NOT update `meta.json` yet. Wait for Review Approval.

## Step 3: Group Review

### 3.1 Trigger Group Review

After scan completion, invoke group review:

**Channel:** Local Doc (`.local/deep-scan-reviews/{repo-name}.md`)

Follow `rule-agents-group-review` workflow:

```
Phase 1: Opencode + Kimi
  ├─ kimi review (no VPN)
  ├─ Iterate until consensus
  └─ Record to review file
         ↓
Phase 2: Codex Review (need VPN)
  ├─ codex review
  └─ Codex agrees?
       ├─ YES → ✅ Triple consensus
       └─ NO  → Back to Phase 1
```

### 3.2 Review Commands

**Kimi (no VPN):**
```bash
kimi --print -p "Please review scan results for {repo-name}. Review file: .local/deep-scan-reviews/{repo-name}.md"
```

**Codex (need VPN):**
```bash
ALL_PROXY=socks5h://10.0.0.175:7890 codex exec "Please review scan results for {repo-name}. Review file: .local/deep-scan-reviews/{repo-name}.md" --timeout 1800000
```

### 3.3 Review Record Format

File: `.local/deep-scan-reviews/{repo-name}.md`

**Must include** (per `rule-agents-group-review`):
- Phase iterations with cycle logs
- Dissent handling
- Timeout logs
- User decision (when triple consensus cannot be reached)

```markdown
# Scan Review - {repo-name}

## Scan Summary
- **Repo**: {repo-name}
- **Binaries**: {list}
- **Scan Date**: {date}
- **Files Generated**: {list}

## Review Scope
Reviewers should verify:
1. Completeness of interface detection
2. Accuracy of dependency mapping
3. **`target_interface` for service calls** - Service-to-service dependencies must include target interface (e.g., `WorkerService.DispatchTask`)
4. Correctness of condition analysis
5. File format compliance

---

## Cycle 1

### Phase 1: Opencode + Kimi

#### Round 1
**Kimi Feedback:**
<kimi comments>

**Opencode Response:**
<response>

#### Consensus
- **Date**: {date}
- **Status**: ✅ APPROVED / 🔄 MORE_ITERATIONS_NEEDED

### Phase 2: Codex Review

#### Round 1
**Codex Feedback:**
<codex comments>

**Verdict:** APPROVED / CHANGES_REQUESTED

### Cycle Decision
- ✅ Complete / 🔄 Continue to Cycle 2

---

## [Cycle N if needed...]

---

## Timeout Log (if any)

- **Reviewer**: kimi / codex
- **Attempt**: N/5
- **Partial Result**: <extracted findings>
- **Action**: Retry / User decision

---

## Dissent Log (if any)

- **Reviewer**: codex
- **Concerns**: <list>
- **Resolution**: <how addressed>

---

## User Decision (if triple consensus unreachable)

- **Date**: {date}
- **Decision**: <user choice>
- **Final Status**: COMPLETED (with conditions) / CANCELLED

---

## Group Review Complete - TRIPLE CONSENSUS

- **Status**: ✅ COMPLETED
- **Participants**: opencode ✅, kimi ✅, codex ✅
- **Date**: {date}
- **Total Cycles**: N
```

## Step 4: Mark Completion

### 4.1 Update State File (After Review Approved)

File: `knowledge/repos/{repo-name}/deep-scan-state.json`

```json
{
  "repo": "{repo-name}",
  "full_name": "{owner}/{repo-name}",
  "status": "approved",
  "scan_completed_at": "<iso-timestamp>",
  "review_completed_at": "<iso-timestamp>",
  "expected_binaries": [
    {"name": "server", "path": "cmd/server", "status": "completed", "role": "primary_service"},
    {"name": "cli", "path": "cmd/cli", "status": "skipped", "role": "admin_cli", "skip_reason": "cli_tool"}
  ],
  "scanned_binaries": [
    {"name": "server", "output_file": "cmd-server"}
  ],
  "skipped_binaries": [
    {"name": "cli", "reason": "cli_tool"}
  ],
  "review_status": "approved",
  "review_participants": ["opencode", "kimi", "codex"],
  "review_cycles": 1,
  "files_generated": [
    "interfaces/cmd-server.json",
    "service-mapping/cmd-server.json"
  ]
}
```

**State Machine:**
- `scan_complete` → Scan done, review pending
- `review_in_progress` → Active group review
- `approved` → Triple consensus reached ✅
- `blocked` → Review blocked, needs user decision
- `skipped` → Not applicable for scan

### 4.2 Update meta.json (Only After Approval)

```bash
jq '. + {
  "scan_status": "deep_scanned",
  "scan_at": "<iso-timestamp>"
}' knowledge/repos/{repo}/meta.json > tmp.json && mv tmp.json knowledge/repos/{repo}/meta.json
```

**Review record location:** `.local/deep-scan-reviews/{repo-name}.md` (derived by convention)

## Step 5: Handle Failures

### 5.1 Failure Categories

| Category | Action | Follow GR Protocol |
|----------|--------|-------------------|
| `missing_prerequisite` | Record to failed.json, skip | No |
| `scan_failed` | Record to failed.json, skip | No |
| `review_blocked` | **Require user decision** | **YES** |

### 5.2 Initialize Failed List

Create `.local/deep-scan-failed.json` if not exists:
```json
[]
```

### 5.3 Record Failed Repo

**For `missing_prerequisite` or `scan_failed`:**
```bash
jq '. + [{
  "repo": "{repo-name}",
  "failed_at": "<iso-timestamp>",
  "category": "scan_failed",
  "reason": "<description>"
}]' .local/deep-scan-failed.json > tmp.json && mv tmp.json .local/deep-scan-failed.json
```

**For `review_blocked`:**
- **Do NOT auto-record to failed.json**
- Follow `rule-agents-group-review` Step 5: ask user
- Only record after explicit user decision

```markdown
## User Decision Required

Review blocked for {repo-name}. Please choose:
1. Retry review
2. Accept partial (2-party) and continue
3. Skip this repo
4. Stop entire task
```

### 5.4 Continue to Next Repo

After recording failure (or user decision to skip), continue to next repo.

## Step 6: Context Compact

After completing a repo (success or failure):

1. **Verify completion marker exists** (if success)
2. **Execute compact command:** `/compact`
3. **Move to next repo** in the list

## Execution Loop

```
# Step 0: Bootstrap (first run only)
generate-path-inventory()
initialize-directories()

FOR each repo in priority-sorted-list:
  # Step 1.3: Check completion (state-based)
  IF exists(knowledge/repos/{repo}/deep-scan-state.json):
    status = get_state_status(repo)
    IF status IN ["approved", "skipped"]:
      CONTINUE  # Skip completed repos
    ELSE IF status == "scan_complete":
      GOTO Step 3  # Resume to review
    ELSE IF status == "review_in_progress":
      GOTO Step 3  # Resume to review
    ELSE IF status == "blocked":
      GOTO Step 5  # Resume to user decision
  # New repo - continue to Step 1.2

  # Step 1.2: Binary-based routing
  binaries = get_binaries(repo)
  scan_binaries = filter(binaries, has http_api or grpc)
  
  IF len(scan_binaries) == 0:
    mark_skipped(repo, "no_network_binaries")
    CONTINUE
  
  # Step 2: Scan (only binaries with http_api or grpc)
  TRY:
    deep-scan(repo, scan_binaries)  # CLI tools skipped internally
    mark_scan_complete(repo)
  CATCH failure:
    record_failure(repo, "scan_failed")
    CONTINUE
  
  # Step 3: Group Review
  TRY:
    group-review(repo)  # May require user decision if blocked
  CATCH review_blocked:
    user_decision = ask_user()
    IF user_decision == "skip":
      record_failure(repo, "review_blocked")
    CONTINUE
  
  # Step 4: Mark Complete
  mark_approved(repo)
  update_meta(repo)
  
  # Step 6: Compact
  compact-context()
  NEXT repo

END

REPORT:
  - Total repos: ~89 (tidbcloud owner, filtered)
  - Scanned: N
  - Skipped (no network binaries): M
  - Skipped (filtered): F
  - Failed: (see .local/deep-scan-failed.json)
```

## Resume After Interruption

Simply re-run this task. The script will:

1. Load repo list
2. Check `knowledge/repos/{repo}/deep-scan-state.json` for existing markers
3. Skip completed repos
4. Continue from first incomplete repo

## Commands Quick Reference

**Check if repo completed:**
```bash
[ -f "knowledge/repos/{repo}/deep-scan-state.json" ] && echo "DONE" || echo "TODO"
```

**Get repo local path:**
```bash
jq -r '.local_path' .local/repos/{repo}/path.json
```

**List completed repos:**
```bash
find knowledge/repos -name "deep-scan-state.json" -exec dirname {} \; | xargs -I{} basename {}
```

**Count remaining repos:**
```bash
TOTAL=$(jq 'map(select(.archived == false)) | length' knowledge/meta/repos-private.json)
DONE=$(find knowledge/repos -name "deep-scan-state.json" 2>/dev/null | wc -l | tr -d ' ')
echo "Remaining: $((TOTAL - DONE))"
```

**Initialize directories:**
```bash
mkdir -p .local/deep-scan-reviews
[ ! -f .local/deep-scan-failed.json ] && echo '[]' > .local/deep-scan-failed.json
```

## Notes

- **Filter Rule:** Must apply `.local/review-history/repos-filtered.txt` before processing
- **VPN Requirement:** Codex review requires `ALL_PROXY=socks5h://10.0.0.175:7890`
- **No Code Changes:** Review phase is read-only, no modifications to scanned repos
- **Triple Consensus:** All three agents (opencode, kimi, codex) must APPROVED (no sampling)
- **Binary-Based Routing:** Scan decision based on `binary.interfaces` (has `http_api` or `grpc` → SCAN; only `cli` or empty → SKIP)
- **CLI Tools:** Binaries with only `cli` interface are skipped but recorded in `skipped_binaries`
- **Empty Interfaces:** Binaries with empty `{}` interfaces are skipped
- **State Machine:** `deep-scan-state.json` tracks status: `scan_complete` → `review_in_progress` → `approved`/`blocked`
- **Per-Binary Checkpoint:** For repos with many binaries, checkpoint after each binary to support intra-repo resume
- **Target Interface:** Service-to-service dependencies must include `target_interface` field (e.g., `WorkerService.DispatchTask`)
- **Output Separation:** `interfaces/` contains only interface definitions; `service-mapping/` contains complete dependencies
- **Infrastructure vs Services:** Infrastructure (mysql, redis, kafka) has no `target_interface`; Services (other microservices) have `target_interface`
- **Collision-safe Naming:** Binary outputs use path-based names (`cmd-foo-server.json`) to handle duplicate binary names
- **Meta Update Timing:** Update `meta.json` only after review approval, not before
- **Review Blocked:** When triple consensus unreachable, require explicit user decision (per `rule-agents-group-review`)
- **Failure Categories:** `missing_prerequisite`, `scan_failed` → auto-skip; `review_blocked` → user decision required
- **Compact Context:** Essential for long-running task to manage token usage
- **Git Tracking:** `deep-scan-state.json` is tracked; review records in `.local/` are not
- **Bootstrap Required:** Run Step 0 on first execution to generate path inventory
- **Review Doc Convention:** Review location is derived by repo name (`.local/deep-scan-reviews/{repo-name}.md`), NOT stored in git-tracked state file