# GitHub Organization Migration Design ## Status Implemented locally in this branch, pending review and merge. This extends the repository importer described in [`../github-migration-importer.md`](../github-migration-importer.md). It is designed for PingCAP's AGS and ClawMem migration, but the configuration and CLI are reusable by other AGS operators. The 2026-08-04 TiDB Cloud Starter validation against `sindresorhus/ky` completed the Git, core collaboration, verification, resume, and attribution-invitation paths. A 2026-08-05 TiDB Cloud Starter validation against `dullage/flatnotes` completed the required Wiki path: source and target Wiki heads matched, 39 Wiki commits and six pages were preserved, and all six pages were readable through the AGS Wiki API. Neither repository covered Git LFS or release assets, which remain separate required validation coverage. ## Goal Move one or many GitHub repositories to AGS without silently losing data or granting access to the wrong people. A finished migration has all of the following: - Git refs and LFS usage have been checked and moved deliberately. - Issues, PRs, comments, reviews, releases, workflow metadata, and Wiki data are migrated or called out as unsupported. - GitHub collaborators and teams are represented as an access handoff plan, not converted into active AGS permissions by accident. - Historical GitHub activity is attributed to non-login placeholder identities until an organization member accepts a reattribution invitation. - Operators receive one machine-readable and one human-readable report per repository, plus an organization rollup. - Interruption is resumable from durable local state and idempotent AGS jobs. ## Operator Interface `ags-migrate` is the migration interface. It remains scriptable; a future AGS console may display its reports and call its APIs, but it is not required to run a migration. ```text ags-migrate plan validate --file migration.yaml ags-migrate plan preflight --file migration.yaml ags-migrate plan run --file migration.yaml --state .ags-migrate-state.json --confirm-source-frozen ags-migrate plan report --file migration.yaml --state .ags-migrate-state.json ags-migrate plan identity-template --file migration.yaml --state .ags-migrate-state.json --out identity-mappings.csv ags-migrate plan apply-identities --file migration.yaml --state .ags-migrate-state.json --csv identity-mappings.csv ``` `migration.yaml` is the organization-level unit of work. It supplies the repositories, target mapping, source and target credentials by environment variable name, concurrency, explicit source-to-target identity mappings, and cutover requirements. The state file stores only IDs, checkpoints, counters, and report paths. It never stores raw GitHub or AGS tokens. ```yaml version: 1 source: provider: github api_url: https://api.github.com token_env: GITHUB_TOKEN target: api_url: https://ags.example.com token_env: AGS_TOKEN execution: repository_concurrency: 4 github_fetch_concurrency: 8 apply_batch_limit: 8 release_asset_max_bytes: 104857600 cutover: require_wiki: true require_lfs_preflight: true require_release_assets: false block_on_unresolved_findings: true identity: invitation_required: true mappings_csv: ./identity-mappings.csv repositories: - source: pingcap/agent-git-service target: ags/agent-git-service target_mode: create mode: core wiki: required stack_api: required access: handoff - source: clawmem-ai/clawmem target: ags/clawmem target_mode: create mode: full wiki: required stack_api: required access: handoff ``` `mode: core` includes day-to-day collaboration data. `mode: full` additionally includes release assets and other explicitly selected large data. A selected release asset is downloaded into a source-addressed, checksum-verified binary file beneath the local snapshot directory; the NDJSON metadata records its source ID, size, and SHA-256 but never embeds Base64 bytes. The uploader verifies that local file then streams it to the AGS import-asset endpoint before metadata apply. An asset that exceeds `release_asset_max_bytes`, has no readable download URL, does not match its checksum, or cannot be staged makes the migration fail; it is never recorded as a successful empty mapping. `target_mode: create` is the safe default and requires the authenticated AGS human or agent to own the personal namespace or be an Owner of the destination organization. `target_mode: existing_empty` is for a deliberately pre-created empty repository and requires that identity to have repository Admin. A nonempty target is never silently replaced. Target-side staged asset files live under AGS `GIT_REPO_DIR`; that directory must be durable and, in a multi-instance deployment, shared by the API and apply workers just like the Git repositories themselves. ## Execution Model The migration has two data planes and one organization controller: 1. The Git plane mirrors normal refs, tags, pull refs, and the separate `*.wiki.git` repository through Git. Git is the correct transport for Git history and Wiki history. 2. The collaboration plane snapshots GitHub API objects to bounded NDJSON shards, uploads them to idempotent AGS import batches, and applies native rows with bulk writers. Release binaries use a separate raw-byte staging channel, not JSON batches, so the API body limit and Base64 expansion cannot reject an otherwise permitted asset. This keeps high-volume PR and comment imports resumable and avoids one long database transaction. 3. The plan controller expands `migration.yaml` into repository runs, enforces repository-level concurrency, records checkpoints, and combines preflight and verification results into reports. Each repository uses one durable AGS ImportJob. The local executor first mirrors Git and Wiki with its local source and target tokens, then uploads and applies the collaboration snapshot to that same job. It does not create a second server-side control import or transmit a GitHub token to AGS. The controller does not replace the existing snapshot, bulk upload, and apply commands. It composes them. A repository may resume after Git mirror, after snapshot creation, after batch upload, or after any bounded apply window. Each repository uses a readable source name plus a stable hash for its snapshot and report path, so repositories such as `foo-bar/baz` and `foo/bar-baz` cannot share a checkpoint directory. A resumed snapshot must also declare the same source repository as its plan entry. ## Preflight And Cutover Gates Preflight runs before any target mutation and creates a durable report. It checks API access and inventories all supported source data. The report records counts, byte sizes where available, and a finding for every data class that cannot be imported or verified. `plan run` writes the final report even when a repository is blocked or fails, so an operator never has to infer its cutover state from a terminal error. Required checks are: - repository visibility, Git ref parity after mirror import, and branch protections; - Wiki enabled state and `*.wiki.git` reachability; - Git LFS use, detected from `.gitattributes`; - release count, release asset count, and total release asset bytes; - issue and PR attachment URLs found in imported body and comment payloads; - webhooks, secret names, Actions metadata, and source-token permission failures; - source collaborator and team counts; - target repository existence and target organization readiness. `plan run` is a cutover command, not a live background sync. The operator must first stop source writes and pass `--confirm-source-frozen`. AGS creates the collaboration snapshot before target mutation and records its deterministic content digest alongside source Git-ref and inventory digests. Before any target mutation and again after metadata apply, AGS re-reads the source and compares all three digests. A content-only edit to an existing Issue, PR, comment, Review, Release, settings record, or full-mode release asset therefore blocks success even when object counts have not changed. The flag is an audited operator attestation, while the fences provide machine-verifiable evidence that the source stayed stable during the run. The final staged-import verification is also a hard gate. A repository cannot be successful while any staged write is pending or failed. Apply workers claim each batch atomically as `applying` before writing and may change it to `applied` or `failed` only while holding that claim. A concurrent worker skips a batch it did not claim; it must never overwrite an already-applied batch as failed. Claims expire and return to `staged` after a bounded lease so an interrupted worker can be recovered safely. If a failure or a post-write source drift is found, AGS marks the target as requiring explicit recovery and refuses to resume against it. AGS does not auto-delete that target: an operator must deliberately reset or replace the target repository and staged import job, then start with a new state file. This avoids deleting any manual work made after the failed attempt. LFS, attachments, and release assets are not considered successful merely because their metadata exists. AGS does not currently transfer Git LFS media; an LFS finding blocks cutover when `require_lfs_preflight` is enabled until the operator has migrated and checked out the media through an explicit LFS path. If a required class cannot be copied, the plan creates a blocking finding and the repository cannot be marked ready for cutover. Operators can explicitly accept a finding in the plan only when the cutover policy permits it; the acceptance remains in the final report. ## Wiki GitHub Wiki is a separate Git repository. For each source repository with a Wiki, AGS mirrors `.wiki.git` and reconciles its current tree into the read-only Wiki V2 index. The mirror sets the target bare Git `HEAD` to the source remote's advertised default Wiki branch, rather than leaving it at the target initialization default. The V2 index preserves safe GitHub page titles such as `Home`, `Advanced-Searching`, and `Release Notes` without rewriting their paths or history; those pages can be listed, navigated, and read through the Wiki API. This external-path grammar is read-only. AGS's public Wiki create, update, move, delete, and catalog Git-ingest paths retain the strict canonical slug grammar, so importing a GitHub Wiki does not widen normal AGS write behavior. The plan verifies that AGS wrote a durable `wiki_repository` mapping. A required Wiki that is disabled, private without token access, missing, not mirrored, or not reconciled is a blocking finding. ## Historical Identity And Access This follows GitHub Enterprise Importer's mannequin model. ### Historical activity Issues, comments, PRs, reviews, and releases initially point to a deterministic `imported-github-*` placeholder. The placeholder is suspended, cannot log in, is not an organization member, and has no repository access. It preserves the source GitHub numeric ID, login, URL, and source-host scope. An AGS account is auto-linked only when it already has a durable identity for the same GitHub host and immutable GitHub numeric user ID. Matching a GitHub login, display name, or commit email is never enough to auto-link activity. Organization administrators can generate a CSV of placeholders, map rows to existing AGS organization members, and submit the mapping in bulk. By default, AGS creates an attribution invitation. The selected member must accept before the historical activity is reattributed. A managed-identity deployment may enable an explicit `skip_invitation` policy, which is recorded in the audit and the migration report. Once accepted, the durable GitHub-ID-to-AGS-user mapping is reused by later migrations in that organization. Git commit authorship is separate from placeholders. It is resolved from the email recorded in the commit, after the AGS account has a verified matching email. The importer does not use a commit email to reassign Issue or PR history, or to grant access. ## GitHub Stacks GitHub now provides a [documented public-preview Stack REST API](https://docs.github.com/en/pull-requests/reference/stacked-pull-requests-apis-and-webhooks). During preflight, `ags-migrate` probes `GET /repos/{owner}/{repo}/stacks` with the current Stack API version and records the Stack count. A `404` is the documented signal that Stacks is not enabled for that repository; it means there is no Stack data to migrate, rather than a global GitHub export gap. A permission or other API error is a blocking `github_stacks_unreadable` finding by default: an unreadable Stack surface must not be treated as empty. An administrator can set `stack_api: skip_if_unreadable` only as an explicit waiver; the source freeze records the exclusion and the final report says that Stack relationships were intentionally not migrated. For an enabled source, the snapshot stores each Stack's documented ordered `pull_requests` list. AGS applies PR batches first, resolves every source PR number through its durable import mapping, and only then writes the target Stack in the identical bottom-to-top order. It writes a Stack mapping as well, so historical GitHub Stack URLs can later redirect to their AGS equivalents. The importer preserves retained historical Stacks containing already-merged PRs without exposing that as a normal user-facing Stack creation path. When a lower Stack member has already merged, GitHub retargets the next open PR to the Stack root; the importer accepts that retained history instead of requiring the deleted lower branch. After staged apply, the migration report compares source and target Stack counts, every source-to-target Stack mapping, bottom-to-top PR membership and order, and the target gh-web Stack URL. Any mismatch blocks cutover and marks the target for explicit recovery rather than silently shipping a partial relationship graph. ### Access handoff Collaborator and team membership are authorization data, not historical attribution. The importer snapshots all source collaborators, teams, and team members into `access_handoff` records. It never creates a target collaborator, team, team membership, or team-repository permission, including when a source identity already has a verified AGS match. Team definitions and repository permissions can be prepared in AGS; individual membership and direct collaborator grants require existing AGS organization members and are applied only through a separate, explicit access step. This matches the important GitHub Enterprise Importer behavior: history reattribution does not migrate repository access. ## Reports Every repository receives `reports/--.json` and `.md`; the plan also writes `reports/summary.json` and `.md`. They contain: - source inventory and target-native counts by object type; - Git source/target ref parity, Wiki mirror state, and required-Wiki result; - upload and apply batch counts, retries, failed items, resume checkpoint, and source-freeze attestation/content fence digests; - mappings from source URLs and IDs to target URLs and IDs; - LFS, attachment, release asset, secret, webhook, and Actions findings; - pending placeholders, attribution invitations, and access handoff items; - explicit operator actions needed before and after cutover; - any explicit target reset or replacement required after a failed cutover; - a final `cutover_ready` boolean with the exact blocking findings when false. The report is the acceptance artifact for AGS and gh-web. The UI should display it read-only; it is not an importer control panel. ## API Surface The existing admin import APIs continue to own staged batches. The completed design adds: - an admin-only raw-byte release-asset staging endpoint. It requires source asset ID, expected size, and SHA-256 headers, writes to durable import staging storage atomically, and is idempotent only for the same checksum; - import-batch claim leases so multiple schedulers or operators can call apply without corrupting batch status; - source-user listing, CSV export, bulk invitation creation, invitation status, accept, and reject endpoints; - preflight findings and a structured import verification report; - access-handoff rows separate from active collaborators and team members; - import job retry from the last durable stage; - source/target URL mappings for later legacy URL redirects. All source credentials remain process-only or external secret references. They are never persisted in import job rows, state files, reports, or logs. ## Validation Tests must cover YAML validation, source-freeze confirmation, count drift, and content-only drift, collision-proof repository workspace keys, state resume, duplicate repository rejection, bounded concurrency, preflight blockers, required Wiki failure, LFS detection, release asset limits and missing-content rejection, placeholder creation, trusted identity auto-linking, bulk invitation creation, accept/reject behavior, failed/pending staged-write rejection, and confirmation that imported collaborators and team members have no organization membership or repository permission until the separate access step runs. Asset tests must include an asset larger than the normal JSON API limit and verify raw-byte upload, checksum rejection, and recovery-safe retry. Batch tests must run two concurrent apply callers against one batch and prove that exactly one claims it while the other leaves the final status unchanged. An end-to-end validation imports a repository containing code, Issues, PRs, comments, reviews, releases, and Wiki pages, then verifies those objects through the AGS API and gh-web. The completed `sindresorhus/ky` validation imported 5,684 collaboration objects in 17 batches with zero pending or failed writes and exact default-branch Git-head parity. It also verified the invitation path by accepting a single historical identity claim and reattributing the corresponding Issues, pull requests, reviews, and review comments. Preflight warnings for externally hosted attachments and unreadable collaboration configuration were intentionally retained in the report; a production plan must treat those findings according to its own cutover policy. The completed `dullage/flatnotes` Wiki validation imported 1,380 collaboration objects in ten batches with zero pending or failed writes: 9 labels, 1 milestone, 315 Issues, 922 Issue comments, 55 pull requests, 12 reviews, 14 review comments, 51 releases, and 1 workflow. The target code `develop` head matched the source exactly. Its separate Wiki `master` head also matched exactly, with 39 commits and six GitHub-style page paths. AGS listed all six pages and read `Home` through the Wiki API. The source token could not read collaborators, teams, webhooks, or secret names, and the report retained those as warnings rather than treating them as migrated data.