# GitHub Import Mode

Status: Draft implementation

## Summary

GitHub-to-AGS migration must preserve collaboration history, not just Git
objects. Normal GitHub-compatible create endpoints are not sufficient because
they derive the author from the current token, allocate new issue/PR numbers,
and stamp timestamps with the current time.

This design introduces a batch-first import mode under `/api/ext/v1` plus a
headless `ags-migrate` client. `ags-migrate` extracts GitHub collaboration data
in bulk, writes a portable snapshot, and uploads idempotent batches. AGS stores
durable migration jobs, logs, source-user placeholders, staged raw objects, and
source-to-target mappings, then applies supported batches into AGS' existing
repository, issue, pull request, release, wiki, team, webhook, and
branch-protection tables.

When GitHub Stacked Pull Requests is enabled for a source repository, the
importer also preserves each Stack's explicit, bottom-to-top PR membership.

## Goals

- Treat migration as a durable, observable job.
- Preserve original GitHub numbers, authors, timestamps, IDs, URLs, and states.
- Keep a mapping from GitHub objects to AGS objects.
- Support dry-run inventory before any writes.
- Keep Git mirroring separate from collaboration-data import.
- Support batch upload, checkpoint/resume, and idempotent retry.
- Keep source extraction outside the AGS server process so migrations can run
  from operator machines, CI, or dedicated migration workers.
- Make source-user mapping explicit through placeholders before account claims.
- Keep raw GitHub tokens out of import job rows.

## Non-goals

- Full GitHub.com parity for products AGS does not yet model.
- Importing raw secret values, packages, projects, stars, watchers, security
  scanning results, audit logs, or replaying GitHub Actions run execution.
- Running arbitrary source-host imports beyond GitHub.
- Rewriting all Markdown links in the initial importer pass.

## API

```text
POST /api/ext/v1/imports/github
GET  /api/ext/v1/imports/{import_id}
POST /api/ext/v1/imports/{import_id}/cancel
POST /api/ext/v1/imports/{import_id}/batches
GET  /api/ext/v1/imports/{import_id}/batches
POST /api/ext/v1/imports/{import_id}/apply
GET  /api/ext/v1/imports/{import_id}/verify
GET  /api/ext/v1/imports/{import_id}/logs
GET  /api/ext/v1/imports/{import_id}/mapping
```

The create route requires the authenticated AGS human or agent to be allowed
to create the target repository: the personal namespace owner for a personal
target, or an organization Owner for an organization target. Every subsequent
job route rechecks target repository `Admin` access. This uses AGS's existing
organization and repository permissions rather than an importer-specific global
`SiteAdmin` role.

Example request:

```json
{
  "source_owner": "pingcap",
  "source_repo": "agent-git-service",
  "source_api_url": "https://api.github.com",
  "source_html_url": "https://github.com/pingcap/agent-git-service",
  "target_owner": "pingcap",
  "target_repo": "agent-git-service",
  "target_mode": "create",
  "target_repository": {
    "description": "GitHub-compatible Git service",
    "visibility": "private",
    "private": true,
    "default_branch": "main",
    "has_issues": true,
    "has_wiki": true
  },
  "options": {
    "mirror_git": true,
    "import_labels": true,
    "import_milestones": true,
    "import_issues": true,
    "import_issue_comments": true,
    "import_pull_requests": true,
    "import_pr_reviews": true,
    "import_pr_review_comments": true,
    "import_wiki": true,
    "import_releases": true,
    "import_branch_protection": true,
    "import_webhooks": true,
    "import_collaborators": true,
    "import_teams": true,
    "import_secret_names": true,
    "import_actions_metadata": true
  }
}
```

The create request does not accept `source_token`, `source_token_ref`, or
`start`. `ags-migrate` keeps the source GitHub token locally for source API and
Git access; the AGS API receives only staged import batches and target metadata.
It also does not accept `dry_run`: `ags-migrate plan` is the non-mutating local
preflight, while creating an ImportJob may create or reserve a real target.
For `target_mode: create`, `ags-migrate` sends non-secret source repository
settings so AGS creates the destination with the same basic visibility,
default branch, and Issue/Wiki capability. An existing-empty target keeps the
settings its AGS administrator already chose.
`target_mode` defaults to `create`. `existing_empty` is available only to a
target repository Admin and refuses a target that already has Git refs or
repository data that could conflict with the import. Preconfigured AGS
collaborators, team grants, and pending invitations are allowed for cutover and
are never overwritten by the importer. Replacing a nonempty repository requires
a separately designed, explicit destructive workflow and is intentionally
unsupported here.

## Data Model

- `import_jobs`: job source/target, status, stage, options, inventory, errors.
- `import_logs`: append-only user-readable log messages.
- `import_item_mappings`: GitHub object identity to AGS object identity.
- `import_source_users`: GitHub users mapped to AGS users or placeholders.

The mapping table is required for link rewriting, verification, retries, and
support questions such as "where did GitHub PR #1558 go?"

`import_batches` and `import_staging_items` are the bulk boundary:

- `import_batches`: one idempotent uploaded shard with `batch_id`,
  `object_type`, `schema_version`, `item_count`, `checksum`, and status.
- `import_staging_items`: raw source-shaped JSON objects for one batch. Apply
  writers convert these rows into AGS-native tables and then mark them applied.

## ags-migrate

`ags-migrate` is a headless migration CLI/worker. It is intentionally separate
from the AGS server process:

```text
ags-migrate github plan
ags-migrate github snapshot
ags-migrate ags create
ags-migrate ags upload
ags-migrate ags apply
ags-migrate ags verify
```

The client is responsible for GitHub-side concerns: REST/GraphQL pagination,
release asset capture, Git/wiki mirror commands, and portable snapshot files.
AGS is responsible for trusted bulk ingest, staging, native apply, mapping,
logs, and verification.

The implemented batch flow is:

```bash
export GITHUB_TOKEN=...
export AGS_TOKEN=...

ags-migrate github plan \
  --repo pingcap/agent-git-service

ags-migrate github snapshot \
  --repo pingcap/agent-git-service \
  --objects all \
  --out /tmp/ags-snapshots/pingcap-agent-git-service

ags-migrate ags create \
  --target https://ags.example.com \
  --source-repo pingcap/agent-git-service \
  --target-repo pingcap/agent-git-service

ags-migrate ags upload \
  --target https://ags.example.com \
  --import-id 123 \
  --snapshot /tmp/ags-snapshots/pingcap-agent-git-service

ags-migrate ags apply \
  --target https://ags.example.com \
  --import-id 123

ags-migrate ags verify \
  --target https://ags.example.com \
  --import-id 123
```

`ags create` creates a staged target-authorized job only. The external
executor mirrors Git and Wiki with its locally held source credential, then
uploads collaboration batches through that same job. AGS never receives a
source credential or starts a server-side GitHub worker.

Snapshot layout is source-neutral so future importers can target the same AGS
bulk ingest contract:

```text
manifest.json
git/repository.bundle
wiki/wiki.bundle
labels-000001.ndjson
milestones-000001.ndjson
issues-000001.ndjson
issue_comments-000001.ndjson
pull_requests-000001.ndjson
pull_request_stacks-000001.ndjson
pull_request_reviews-000001.ndjson
pull_request_review_comments-000001.ndjson
releases-000001.ndjson
release_assets-000001.ndjson
branch_protections-000001.ndjson
webhooks-000001.ndjson
workflows-000001.ndjson
secret_names-000001.ndjson
unsupported-data-report.json
```

Each shard uploads as one `/batches` request with a stable `batch_id` and
checksum. Reuploading the same batch is a no-op; reusing the same `batch_id`
with different content is rejected.

The current CLI shards every object type with `--shard-size` (default 1000,
server maximum 10000). Release assets up to `--max-asset-bytes` are embedded in
the snapshot as base64 plus SHA-256. Larger assets are staged as metadata with a
mapping note that content was skipped.

Implemented native batch apply writers:

- `labels`
- `milestones`
- `issues`
- `issue_comments`
- `pull_requests`
- `pull_request_stacks`
- `pull_request_reviews`
- `pull_request_review_comments`
- `releases`
- `release_assets`
- `branch_protections`
- `webhooks`
- `workflows`
- `secret_names`

Each batch applies in its own transaction. A failed batch is marked `failed`
with the error message while already-applied batches remain committed. Replaying
the same upload is idempotent through `import_batches.checksum`; replaying
apply uses `import_item_mappings` to update existing target rows where the
target object already exists.

## Stage Order

1. `inventory`: fetch counts and capability warnings from GitHub.
2. `base_data`: create or update the target repo metadata, labels, and
   milestones. The target repo must exist before git refs can be mirrored.
3. `git_mirror`: clone/fetch/push refs, tags, and branches.
4. `teams`: collaborators, repository teams, and team members.
5. `issues`: issues, comments, state, labels, milestones, assignees.
6. `pull_requests`: PR records, reviews, review comments, merged state, then
   Stack relationships after every source PR has a durable mapping.
7. `wiki`: wiki pages and history when available.
8. `releases`: releases and release assets.
9. `settings`: branch protection and webhooks where AGS supports them.
10. `secrets`: secret names and workflow metadata inventory. Secret values are
    not readable from GitHub and must be recreated by admins.
11. `verify`: compare source counts with imported counts and write the report.

## Imported Objects

- Git refs, branches, tags, and GitHub pull refs through a mirror fetch.
- Repository metadata and feature flags.
- Source users as existing AGS users or placeholder users.
- Labels and milestones.
- Issues, PRs, issue-style comments, labels, assignees, milestones, state,
  timestamps, closed/merged metadata, and lock metadata.
- PR reviews and line review comments, including reply threading when source
  parents are available.
- GitHub Stacks from `GET /repos/{owner}/{repo}/stacks`, including the ordered
  bottom-to-top PR sequence and an object mapping for each source Stack.
- Releases and release assets up to `max_release_asset_bytes`.
- Wiki git repository plus Wiki V2 reconcile when the source wiki exists.
- Branch protection rules.
- Webhooks as inactive hooks without secrets.
- Direct collaborators, repository teams, and team members.
- Workflow definitions and secret names as metadata/manifest records.

## Verification

Every run produces logs and mappings. Dry-run populates `inventory_json` and
warnings without writing target repo collaboration objects. Batch upload stages
raw source objects first; apply writers then write AGS-native rows. Verification
compares snapshot/staging counts, applied rows, mappings, and unsupported data
reports before cutover.

### GitHub Stack Capability And Apply Order

GitHub exposes [Stacks as a public-preview REST surface](https://docs.github.com/en/pull-requests/reference/stacked-pull-requests-apis-and-webhooks).
Preflight probes the documented `GET /repos/{owner}/{repo}/stacks` endpoint
with GitHub's current Stack API version. A `404` means the feature is not enabled for that source
repository and is recorded as `github_stacks_not_enabled`; it is not treated as
a GitHub-wide export limitation. Permission or other API failures are a
blocking `github_stacks_unreadable` finding by default. The only bypass is the
per-repository `stack_api: skip_if_unreadable` waiver, which is recorded in the
source freeze and migration report so an intentional Stack omission cannot look
like a complete migration.

The snapshot keeps every Stack resource as one `pull_request_stacks` record.
GitHub supplies `pull_requests` in bottom-to-top order. AGS applies all
`pull_requests` batches ahead of Stack batches, even if an operator uploaded
the Stack shard first. The Stack writer resolves each source PR through the
durable PR mapping, validates the retained GitHub branch chain, creates the target Stack
and ordered entries in one transaction, then writes a source-to-target Stack
mapping. Reapplying a successful Stack batch verifies the same membership and
order rather than duplicating the Stack. Historical Stack records may include
merged PRs, which are preserved by this importer-only writer even though the
normal live Stack-creation API rejects a new Stack containing merged PRs.
If a lower member has already merged, GitHub retargets the next open member to
the Stack root; that partial-merge shape remains valid during import.

Post-apply verification compares source and target Stack counts, object
mappings, PR membership, member order, and the gh-web Stack route stored in
each mapping. Any discrepancy is a cutover-blocking recovery condition.

## Cut Line

This importer is complete enough when one real GitHub repository can be dry-run,
imported into AGS, inspected through logs/mappings, and then used for ongoing
issue/PR review work inside AGS. Secret values, webhook secrets, projects,
stars, packages, audit logs, and Actions run replay remain manual or future AGS
product work because GitHub either does not expose the original values or AGS
does not yet have equivalent behavior.