# GitHub Migration Importer Design

For organization-wide plans, batch identity mapping, preflight, and cutover
reports, see [GitHub Organization Migration Design](design/github-organization-migration.md).

## Goal

AGS should support repeatable GitHub-to-AGS migrations for repositories that are
moving from GitHub.com to AGS as their system of record. The importer must be
usable by operators and by future external users, not only by one-off scripts.

## Migration Planes

Git data and GitHub collaboration metadata are migrated through separate paths:

- Git history, branches, tags, and pull refs use normal Git mirror operations:
  `git clone --mirror` from GitHub and `git push --mirror` to AGS.
- Collaboration metadata uses `ags-migrate` to snapshot GitHub API data into
  sharded NDJSON, upload shards to AGS import batches, and apply those batches
  into native AGS tables.

Keeping these planes separate makes the migration debuggable: Git can complete
even when metadata needs retries, and metadata batches can be inspected and
replayed without fetching GitHub again.

The native Git mirror implementation lives in `internal/sync/gitmirror` and is
shared with repository sync. `ags-migrate` remains a supported caller while the
new sync control plane is introduced; this avoids replacing the validated
large-repository transport with a slower row-oriented path.

## Architecture

The importer has two cooperating pieces:

- `ags-migrate`, a CLI for operators and future external users. It is
  responsible for reading from GitHub, writing local snapshots, uploading
  snapshots to AGS, and driving resumable apply loops.
- AGS import APIs and writers. They are responsible for validating uploaded
  batches, staging raw source objects, converting supported object types into
  native AGS tables, and recording source-to-target mappings.

The CLI owns source-system concerns such as GitHub pagination, rate limits,
parallel fetches, local manifests, and resumable upload/apply commands. The AGS
server owns target-system concerns such as database transactions, idempotency,
native row creation, mapping tables, and verification. This keeps GitHub-specific
fetch logic out of the server while still giving AGS a reusable import substrate
for its own data model.

The importer does not require a console UI. A console can call the same APIs
later, but the migration path must remain scriptable so large organization or
repository sets can be migrated unattended.

## Batch Model

`ags-migrate github snapshot` writes source objects as adaptive snapshot shards.
`ags-migrate ags upload` uploads non-empty shards as idempotent import batches,
using the AGS bulk staging API by default. `ags-migrate ags apply` converts
staged batches into native AGS rows through bounded apply windows.

Applied batches are the retry boundary. Staged batches remain pending until
applied successfully. Failed batches are never silently skipped: the final
verification blocks cutover until an operator resets or replaces the affected
target import through an explicit recovery procedure. The apply endpoint accepts an optional
`batch_limit` so one API request only advances a bounded number of supported
staged batches. `ags-migrate ags apply` defaults to `--batch-limit=1` and loops
until no supported staged batches remain, which keeps each database transaction
and HTTP request small enough to retry independently. Shards are bounded by both
item count and encoded byte size so small metadata objects can move in large
batches while large release assets or comments do not create oversized
transactions.

AGS computes upload checksums from the uploaded payload by default. The snapshot
manifest still records file checksums for local integrity checks, but the upload
path avoids client/server JSON canonicalization drift. Operators can opt into
strict manifest checksum submission with `--send-manifest-checksum`.

The default snapshot mode is `core`, which prioritizes the data needed for daily
repo collaboration: labels, milestones, issues, issue comments, pull requests,
pull request Stacks, pull request reviews, pull request review comments, releases metadata, branch
protections, and workflow metadata. Operators can pass `--objects full` or
explicit object names to include heavier or more privileged data such as release
asset content, webhooks, and secret names.

The default performance posture is optimized for large repositories:

- `--shard-size` defaults to 500 items, with `--max-shard-bytes` defaulting to
  8 MiB.
- Issue comments and pull request review comments use repository-level GitHub
  endpoints instead of one request per issue or pull request.
- Pull request reviews are fetched with bounded parallelism controlled by
  `--fetch-concurrency`.
- Repository-level pull request review comments are filtered to pull requests
  present in the pull request snapshot. GitHub can return historical review
  comments whose parent pull request API now returns 404; those orphan comments
  must not fail the whole migration.
- Uploads use `/imports/{id}/batches/bulk` in groups controlled by
  `--bulk-size` and `--bulk-max-bytes`, with a compatibility fallback to
  single-batch uploads when the server does not support bulk staging.
- Apply uses `/imports/{id}/apply` with `batch_limit` and `skip_verify` controls
  so operators can run resumable windows during high-volume imports, then run a
  full `ags-migrate ags verify` at the end.

Batching exists in both layers:

- The CLI batches GitHub reads where GitHub provides repository-level endpoints,
  and uses bounded parallelism for nested endpoints that still require
  per-pull-request reads.
- The CLI batches local snapshot files by item count and encoded byte size.
- The CLI uploads many snapshot shards per HTTP request through
  `/batches/bulk`, bounded by shard count and payload bytes.
- The server applies batches in bounded windows through `batch_limit`, so each
  request has a retryable transaction boundary.
- The server writers use bulk lookup, bulk upsert, and bulk mapping writes for
  core collaboration object types instead of row-by-row apply loops.
- NDJSON input is read as records rather than through a fixed scanner token
  limit, so unusually large source metadata fields are not rejected merely
  because one line exceeds a default scanner buffer.
- Release asset bytes are not encoded into NDJSON or batch JSON. They are
  stored as source-addressed, checksum-verified local snapshot files and uploaded through a
  dedicated raw-byte staging endpoint before the release-asset batch applies.
  Target-side staging uses the durable, shared `GIT_REPO_DIR` storage already
  required for AGS Git repositories.
- Apply first atomically claims a staged batch. A worker that did not obtain
  that claim skips the batch; it cannot turn another worker's completed batch
  into a failed batch. Expired claims are returned to `staged` for recovery.

## Supported Initial Scope

The importer currently targets GitHub source repositories and AGS targets. The
initial metadata surface includes labels, milestones, issues, issue comments,
pull requests, pull request Stacks, pull request reviews, pull request review comments, releases,
release assets, branch protections, webhooks, workflow metadata, and secret
names. Secret values, Actions run replay, packages, projects, stars, watchers,
security scan results, and audit logs are out of scope.

When the source uses GitHub Stacks, the organization migration plan probes the
Stack API before cutover. An unreadable Stack surface blocks by default; an
administrator must explicitly set `stack_api: skip_if_unreadable` to waive it.
After apply, the report verifies Stack counts, durable Stack mappings, ordered
PR membership, and the gh-web Stack URL before declaring cutover ready.

## Non-goals

- Full GitHub.com product parity is not required for the first importer.
- Secret raw values are not migrated because GitHub does not expose them.
- Packages, Projects, stars, watchers, security scan results, audit logs, and
  Actions run replay are not migrated in the initial scope.
- Arbitrary forge import is not part of this design; the first supported source
  is GitHub.
- Markdown link rewriting is not required for the initial importer. Source URL
  mappings are recorded so link rewriting can be added later as a separate step.

## Operator Flow

1. Create the target AGS repository.
2. Mirror Git refs from GitHub to AGS with Git.
3. Run `ags-migrate github snapshot` for the source repository.
4. Create an AGS import job with `ags-migrate ags create`.
5. Upload snapshot shards with `ags-migrate ags upload`; this uses bulk staging
   unless `--bulk-size=1` is set.
6. Apply staged batches with `ags-migrate ags apply`. The command advances one
   supported batch per request by default and resumes automatically until the
   server reports no supported staged batches remain. Operators can increase
   `--batch-limit` for faster trusted environments, use `--once` for manual
   stepping, or use `--max-rounds` for scheduler-controlled runs.
7. Verify batch state and native object counts with `ags-migrate ags verify`
   and the AGS API/console.

Example core migration:

```sh
git clone --mirror https://github.com/source-owner/source-repo.git
git -C source-repo.git push --mirror https://ags.example.com/target-owner/target-repo.git

ags-migrate github snapshot source-owner/source-repo --out ./snapshot
ags-migrate ags create \
  --api https://ags.example.com \
  --owner target-owner \
  --repo target-repo \
  --source-owner source-owner \
  --source-repo source-repo
ags-migrate ags upload --api https://ags.example.com --import-id <id> --snapshot ./snapshot
ags-migrate ags apply --api https://ags.example.com --import-id <id>
ags-migrate ags verify --api https://ags.example.com --import-id <id>
```

## Validation Notes

A local validation against `tinylibs/tinypool` confirmed that Git mirror and
585 collaboration metadata objects can be imported into AGS native tables:
labels, issues, issue comments, pull requests, pull request reviews, pull
request review comments, releases, and workflows. `ags-migrate ags verify`
reported no pending or failed writes after apply.

The high-volume collaboration writers now apply one staged shard with bulk
lookup and batched writes: issues, issue comments, pull requests, pull request
reviews, and pull request review comments all avoid per-item mapping/user lookup
and row-by-row insert/update loops. Label replacement for imported issues and
pull requests is also batched per shard.

The remaining core metadata writers use the same bulk apply posture. Release
metadata is parsed per shard, authors are resolved in one pass, releases are
upserted in batches, and standalone release assets are applied through a shared
bulk helper. A release asset is accepted only when its downloaded content and
checksum are present; an absent or oversized asset fails the import instead of
creating a misleading empty mapping. Source-to-target mappings are written with
the bulk mapping path. Branch protections and workflow metadata are also batch
upserted and mapped after a single key-based readback.

A second validation pass against `sindresorhus/ky` showed that the old importer
shape was dominated by serial GitHub nested-object fetches, hundreds of small
upload requests, and many small apply transactions. The current design addresses
those bottlenecks by using repository-level comment endpoints, parallel PR
review fetches, adaptive shards, bulk upload/staging, and bulk apply writers for
all core collaboration object types.

A TiDB Cloud Starter validation on 2026-08-04 against `sindresorhus/ky`
imported 5,684 core metadata objects across 17 batches: 13 labels, 1
milestone, 447 Issues, 3,072 Issue comments, 332 pull requests, 852 reviews,
869 review comments, 96 releases, and 2 workflows. Final verification reported
no pending or failed writes, and all 5,684 source-object mappings were present.
The target default-branch Git commit exactly matched the source GitHub commit.

That repository has no Wiki, no Git LFS data, and no release assets, so this
validation does not replace the required Wiki, LFS, or asset-specific tests.
Its preflight also reported 102 external attachment URLs and lacked permission
to read collaborators, teams, webhooks, and secret names; those findings remain
explicit operator actions rather than evidence that those source data classes
were migrated.

The same validation exercised the historical-identity flow. A source identity
remained a suspended placeholder until an administrator created an attribution
invitation and the invited AGS user accepted it. On acceptance, 17 Issues, 56
pull requests, 249 reviews, and 319 review comments were reattributed to that
user, and the stable GitHub numeric identity was recorded for future imports.

A TiDB Cloud Starter Wiki validation on 2026-08-05 against
`dullage/flatnotes` imported 1,380 core metadata objects in 10 batches with no
pending or failed writes. The source and target code `develop` heads matched
exactly. Its separate Wiki `master` head also matched exactly; the target kept
39 Wiki commits and six pages, including GitHub-style page titles, in the Wiki
V2 index. This validation fixed two compatibility defects: Git mirrors now set
bare `HEAD` from the source remote's advertised default branch, and safe
GitHub-style page paths can be listed, navigated, and read through the
read-only V2 index. Public Wiki writes and catalog Git-ingest retain AGS's
strict canonical slug grammar; imported noncanonical page names cannot be
created or edited through those mutation APIs.

The validation also exposed reliability requirements:

- Uploads should not require manifest checksum submission, because AGS can
  compute payload checksums authoritatively.
- Apply must be bounded and resumable. A single long-running apply request can
  be interrupted by database stalls or client cancellation even though completed
  batches have already committed. Batch-limited apply windows let the next run
  continue from the remaining staged batches instead of replaying the entire
  import.
- TiDB-backed deployments may already have schema prepared; operators can
  set `AGS_SKIP_AUTO_MIGRATE=1` to avoid startup-time AutoMigrate scans during
  migration workers or validation runs.

## Future Work

The next performance step is to turn bounded apply into a worker pipeline:
process independent shards concurrently after dependency ordering is satisfied,
add explicit checkpoint/resume summaries, and improve failed-batch error
recording when the database connection is lost. For TiDB-class repositories, the
importer should target core metadata migration in a few hours, with full
high-fidelity migration taking longer only when large assets, attachments, or
historical Actions artifacts are explicitly included.