# KB Sources External Repo Mirror — Batch 0 Baseline

Date: 2026-06-12
Branch: `feat/kb-source-repo-mirror`
Base: `origin/main` at `9dca7399543eb510c1d38d821605eda601b462ba`

## Scope locked for v1

- Implement **pull mirror only** for external Git repositories.
- Flow: external Git repo → AGS source repository → KB source indexing / reconcile / code-wiki generation.
- No push mirror or outbound publication in v1.
- Mirror credentials are represented as `credential_ref` / secret references; no token is stored directly in source `config_json`.
- Mirror sources explicitly store `upstream_type`; do not rely only on URL guessing.
- Manual and scheduled mirror sync must share one worker pipeline.
- Mirror status and errors must be persisted and visible from the KB source UI.

## Gitea reference patterns to reuse

Gitea separates mirror metadata from repository identity:

- Pull mirror metadata lives in `models/repo/mirror.go` and is attached to a repository with `Repository.IsMirror`.
- Mirror creation flows through repository migration/import, including `git clone --mirror` setup in `services/repository/migrate.go`.
- Sync is dispatched through queue/cron code in `services/mirror/queue.go`, `services/mirror/mirror.go`, and `services/cron/tasks_basic.go`.
- Pull sync in `services/mirror/mirror_pull.go` serializes per repository, performs fetch/prune and downstream reconciliation, and persists `last_update`, `next_update`, and error state.
- Push mirror is a separate model in `models/repo/pushmirror.go`; v1 AgentTeam KB mirror should not copy push mirror behavior unless outbound publishing becomes a product requirement.

Reusable implementation rules:

1. Store mirror config/status separately from the generic KB source row.
2. Use a unique queue job plus per-source lock to prevent concurrent sync for the same source.
3. Persist `last_sync_at`, `next_sync_at`, `last_success_at`, and `last_error`.
4. Separate sanitized remote URL from secret-bearing credential references.
5. Validate remote URLs before git access and isolate all git working directories.

## Current AgentTeam baseline

### Backend model/store

- `meshx/internal/store/kb/models.go`
  - `Source` already has `SourceType`, repo identity fields, `DefaultRef`, `IndexingMode`, `SyncStatus`, `IndexStatus`, `Config`, `LastError`, `LastSyncAt`, and `LastIndexAt`.
  - `Job` already backs asynchronous KB jobs.
- `meshx/internal/store/kb/store.go`
  - Existing source CRUD and `ClaimNextQueuedJob` can be extended for mirror metadata and sync jobs.

### KB service/API

- `meshx/internal/service/kb_sources.go`
  - `CreateSource` supports `file_repository`, `codebase`, `external_connector`, and `database_schema`.
  - Repo-backed sources currently support `repo_mode=existing|create`.
  - Materialized sources require `repo_mode=kb_repo`.
- `meshx/internal/api/kb/api_handler.go`
  - Existing routes include source list/create and `generate-code-wiki`.
- `meshx/internal/api/dto/kb.go`
  - DTOs already expose source status, repo identity, and config fields.

### Existing worker and AGS boundary

- `meshx/internal/service/kb_code_wiki.go`
  - Existing KB async job worker can serve as the template for mirror sync worker lifecycle and progress updates.
- `meshx/internal/service/kb_ags_wiki.go`
  - `AGSHTTPKBWikiStore` can create KB/source repositories and reconcile wiki pages, but it has no mirror provisioning/sync method today.

### Frontend

- `web/src/meshx/api/knowledge-base.ts`
  - Source API client already serializes repo-backed source fields.
- `web/src/meshx/components/knowledge-base/KnowledgeBaseSourceDialog.tsx`
  - Current UI supports existing/create repo modes but no mirror URL, interval, or credential fields.
- `web/src/meshx/components/knowledge-base/workbench/KnowledgeBaseWorkbench.tsx`
  - Source list already displays status derived from source sync/index/status fields.
- `web/src/meshx/components/knowledge-base/workbench/KnowledgeBaseSourceDetail.tsx`
  - Repo-backed source details reuse AGS repo layout and can host mirror status/actions.

## Design decisions for implementation batches

### Source shape

Use `source_type=codebase` or `source_type=file_repository` with a new `repo_mode=mirror` instead of adding a fifth source type. This keeps source detail and indexing semantics aligned with existing repo-backed sources.

### Upstream provider type

Add an explicit `upstream_type` field to mirror metadata, DTOs, API requests, and frontend forms. V1 must expose presets for every upstream type supported by the inspected Gitea migration enum.

The v1 worker should still use a shared Git clone/fetch pipeline. `upstream_type` is for provider-specific form presets, URL/credential validation, error messages, and future metadata import; it should not create hard-coded worker branches unless a provider truly requires a different git transport.

Gitea's current migration enum is defined in `modules/structs/repo.go` as:

| AgentTeam value | Gitea reference | Notes |
|---|---|---|
| `generic_git` | `PlainGitService` | Default for arbitrary Git remotes |
| `github` | `GithubService` | GitHub / GitHub Enterprise style |
| `gitea` | `GiteaService` | Gitea-compatible |
| `gitlab` | `GitlabService` | GitLab / self-hosted GitLab style |
| `gogs` | `GogsService` | Gogs-compatible |
| `onedev` | `OneDevService` | OneDev |
| `gitbucket` | `GitBucketService` | GitBucket |
| `codebase` | `CodebaseService` | Codebase |
| `codecommit` | `CodeCommitService` | AWS CodeCommit |

Code evidence from `/Users/lonng/devel/oss/gitea`:

- `modules/structs/repo.go:347-356` enumerates `PlainGitService`, `GithubService`, `GiteaService`, `GitlabService`, `GogsService`, `OneDevService`, `GitBucketService`, `CodebaseService`, and `CodeCommitService`.
- `modules/structs/repo.go:434-444` defines `SupportedFullGitService` as GitHub, GitLab, Gitea, Gogs, OneDev, GitBucket, Codebase, and CodeCommit.
- `services/migrations/migrate.go:149` selects downloader factories by `GitServiceType`, and the registered factories correspond to the enum above plus the plain Git fallback.

Do not add non-Gitea-supported upstreams in v1 unless the target Gitea reference gains a concrete downloader/import preset. For example, Bitbucket is not in the inspected enum or `SupportedFullGitService`, so it is not part of the v1 parity list.

Provider registry requirements:

- Define a central registry/preset table for labels, URL examples, allowed auth modes, and validation rules.
- Default to `generic_git`.
- Keep the enum extensible so future provider-specific metadata import can be added without changing the source/mirror table shape.

### Dedicated metadata

Add a dedicated mirror metadata table, tentatively `knowledge_base_source_mirrors`, keyed by `source_id`.

Minimum fields:

| Field | Purpose |
|---|---|
| `source_id` | KB source identity |
| `upstream_type` | explicit provider type, default `generic_git` |
| `remote_url` | sanitized external URL for display |
| `credential_ref` | secret reference for credentials |
| `default_ref` | branch/ref to mirror/index by default |
| `interval_seconds` | scheduled sync interval |
| `enabled` | enables scheduled sync |
| `sync_status` | idle/queued/running/success/failed/disabled |
| `last_sync_at` | last attempt time |
| `last_success_at` | last successful sync time |
| `next_sync_at` | next scheduled sync time |
| `last_error` | last error summary |
| `last_mirrored_commit` | optional incremental sync marker |

### Job pipeline

Add job type `sync_source_mirror` using the existing KB job table and claim/lease pattern.

Shared pipeline for manual and scheduled sync:

1. Validate source and mirror metadata.
2. Acquire per-source sync lock / lease.
3. Fetch from external repo in an isolated workspace.
4. Update AGS source repository.
5. Persist mirror/source sync state.
6. Trigger downstream KB index/reconcile and, for codebase sources, code-wiki generation when configured.

### API shape

Prefer resource-style endpoints:

| Operation | Endpoint |
|---|---|
| create mirror source | `POST /api/v1/kb/{kb_id}/sources` with `repo_mode=mirror` and `mirror` config |
| read source mirror summary | included in `GET /api/v1/kb/{kb_id}` source payload and source detail responses |
| update mirror config | `PATCH /api/v1/kb/{kb_id}/sources/{source_id}/mirror` |
| trigger manual sync | `POST /api/v1/kb/{kb_id}/sources/{source_id}/mirror-syncs` |
| list sync history | `GET /api/v1/kb/{kb_id}/sources/{source_id}/mirror-syncs` |

### Safety requirements

- Reject unsupported protocols and unsafe hosts before invoking git.
- Block localhost, link-local, private network ranges, and `file://` by default.
- Keep credentials out of source `config_json`, API responses, logs, and job payloads.
- Enforce git command timeout and output truncation.
- Clean temporary clone/fetch directories after each run.
- Record failures in mirror metadata and source `last_error`.

## Batch implementation map

| Batch | Task | Expected repo areas |
|---|---|---|
| 1 | mirror metadata / store / job model | `meshx/internal/store/kb/*`, migrations if present |
| 2 | service / worker / scheduler / AGS sync | `meshx/internal/service/kb_*`, `meshx/internal/server/server.go`, config |
| 3 | API / DTO / AgentTeam forwarder | `meshx/internal/api/kb/*`, `meshx/internal/api/dto/*`, web API client |
| 4 | frontend mirror mode/status UI | `web/src/meshx/api/knowledge-base.ts`, KB source dialog/workbench/detail components |
| 5 | end-to-end verification and PR | targeted Go tests, web tests/typecheck, route/API scan |

## Batch 0 verification commands

- `git diff --check`
- `git status --short`

No production code is changed in Batch 0; later batches must add focused unit tests per changed layer.